The second risk-limiting audit under California AB 2023 was conducted
on 6 May 2011, in Monterey County.The contest was a Special all-mail election for Monterey Peninsula
Water Management District Director, Division 1. Monterey uses Sequoia
equipment. There were two candidates: Brenda Lewis and Thomas M. Mancini, and
write-ins.
2111 ballots were cast in all. The reported totals were 1353 reported for
Lewis, 742 for Mancini, and 13 write-ins. The remaining 3 ballots were recorded
as undervotes and overvotes. Lewis was reported to have 64.18% of the valid
votes.
The outcome was confirmed with a risk limit of 10%. That is, if the outcome
was wrong, there was at least a 90% chance that the audit would lead to a
full hand count, which would correct the outcome. The audit required looking at
89 individual ballots. The auditing method also ensured that if Lewis had at
least 64.08% of the vote (0.1% less than reported, but still a win for Lewis), there was
at most a 1% chance that the audit would lead to a full hand count.
Two members of the public observed the entire audit process, which
took roughly 90 minutes including some preliminary explanation of the procedure.
They confirmed that their interpretation of the ballots agreed with mine
and the elections officials', and they helped roll the dice used to select
ballots at random. In conversations afterward, they seemed quite satisfied with
the transparency of the procedure (although perhaps not utterly convinced by
the mathematics that justified the details).
The audit was performed as follows. After the ballots had been
tabulated officially, elections officials Bates-stamped each with a unique serial
number (1962 ballots that were scanned had been stamped prior to audit day;
the remaining 149 were stamped as part of the audit). It is my understanding
that stamping the ballots took about 5 person-hours in all.
The particular risk-limiting auditing method used was extremely simple,
although statistically inefficient. It does not require exporting any data at all
from the voting system. All it relies on is the audit trail. The calculations required
are also very
simple (only multiplication). Thus, it might be useful in a wide variety
of settings, although the hand-counting burden can get high if the margin (as a
percentage of valid votes) is small.
Conventional vote-tabulation audits generally have two parts: (1)
confirming that batch subtotals as reported by the voting system add up to the contest
totals as reported by the voting system, then (2) confirming that the subtotals
are
sufficiently accurate to give the right answer (by checking a random
sample of subtotals against hand counts for those ballots).
A risk-limiting audit at 10% risk based on checking the accuracy of
precinct-level reports from the voting system would have required hand counting the
majority of the ballots.
If the voting system reported cast vote records for individual ballots,
roughly 30 ballots would have sufficed (if the CVRs were all accurate). I had
hoped to use the Trachtenberg Election Verification System (TEVS) to obtain cast
vote records from ballot scans and perform a "transitive audit" along the lines
suggested by Calendrino et al., but modified to be risk-limiting. Mitch Trachtenberg
was extremely helpful in getting TEVS software working for Sequoia ballots. It performed
correctly on
a set of 25 ballots we had for testing. Moreover, Monterey County
Elections scanned all their ballots on an office scanner so that I could process them with
TEVS. As it turned out, however, when I tried to process the ballot images this morning
to extract CVRs, roughly 10% of the ballot images could not be processed
automatically (most likely because of the quality of the sheet-feeding in the scanner).
Time did not permit further tweaking of the software settings, so I used a
backup plan:
*blind ballot polling*. This is the first time ballot polling has been
used to perform a risk-limiting audit.
To perform ballot polling, physical ballots were selected at random
(using the Bates stamp number to identify them) and interpreted by hand.
This selection continued until the fraction of ballots in the sample was
sufficiently high to give strong evidence that a full hand count would show that Lewis
actually won. If strong evidence that Lewis won had not been forthcoming, or if
the sample gave strong evidence that Lewis had not won, there would have been
a full hand count. Blind ballot polling ignores the voting system completely
and makes its own statistical assessment of who won directly from a random sample of
the audit trail.
The method works as follows: Lewis really won if her share of votes
among the ballots that showed valid votes was larger than the share of Mancini
or write-ins. Her reported margin was so large that I could treat Mancini and
write-ins as a single candidate: Lewis won if her share of the valid votes was
greater than 50%. I used Wald's sequential probability ratio test (published in 1945)
to audit incrementally by sequentially testing the hypothesis that Lewis
received less than 50% of the valid votes. To reject that hypothesis is to conclude
that Lewis really won.
The algorithm is simple:
1. Set T = 1.
2. Select a ballot at random from the 2111 cast in the contest.
3. If the ballot is an undervote, overvote, or invalid ballot, go back to
step 2.
4. If the ballot shows a vote for Lewis, multiply T by 64.08%/50%.
5. If the ballot shows a vote for Mancini or a write-in, multiply T by
(100%-64.08%)/50%.
6. If T > 9.9, stop the audit and declare Lewis to be the winner.
7. If T < 0.011, stop the audit and perform a full hand count.
8. Go back to step 2.
It is a theorem that the chance is at most 10% that this stops short of a
full hand count if Lewis did not really win.
This method could terminate by looking at as few as 10 ballots if the first
10 allreported votes for Lewis. The expected number of ballots to examine if
Lewis reallygot at least 64.08% of the vote is 58. (The expected number depends only on
the margin and the fraction of ballots that show valid votes, not on the size of the
contest.) In this case, it took drawing 92 ballots (89 of which were distinct) before
T exceeded 9.9 (it ended up at 10.16), because there were a surprising number of votes
for
Mancini in the early draws. Among the 92, one was a deliberate undervote
(a write-in with no name written in), 56 were for Lewis, and 35 were for Mancini.
Among the 56 for Lewis, two ballots had been selected twice; among the 35 for
Mancini, one had been selected twice. We thus looked at 89 ballots in all before the audit
was done.
This method was practical for this contest because the margin was so large.
The work load for a risk-limiting audit using ballot polling grows rapidly as
the margin shrinks, but ballot polling has the advantage of requiring nothing from the
voting
system other than the audit trail--no "data plumbing." On the other hand,
while such an audit confirms the outcome with low risk, it does not directly
check the accuracy of the voting system, only of the outcome: The voting system might
have
found the right outcome through a fortuitous cancellation of errors, rather
than because it is intrinsically accurate.
I'm very grateful to Monterey County Elections for participating in this
pilot, and to Mitch Trachtenberg for his help with TEVS.
Philip B. Stark | Professor of Statistics | University of California