End-to-End Standards for Accuracy in Paper-Based Systems

Notes on a talk to the
2002 Workshop on Election Standards and Technology
January 31 2002, Washington DC

Douglas W. Jones

Associate Professor of Computer Science, University of Iowa
Chair, Iowa Board of Examiners for Voting Machines and Electronic Voting Systems
Member, 2001 Iowa Election Reform Task Force

Executive Summary

The measure of voting system accuracy that ultimately matters to the success of our democracy is the following:
How close does the outcome of the election, as reflected in the official canvass, come to the actual intent of the voters who participated in the election?
This is an end-to-end measure of performance, in the sense used by Saltzer, Reed and Clark in their 1984 paper, End to End Arguments in System Design. It is extremely difficult to connect the kinds of accuracy standards found in the current (1990) or draft FEC standards with this ultimate standard.

Both recounts and exit poll data give a useful approximation of this measure, and we should use them regularly! I have not looked at exit poll data, but by the recount measure, current paper-based election systems routinely achieve overall accuracies bettern than 1 in 5,000. This is a far cry from the 1 in 500,000 or 1 in 10,000,000 figures cited in standards, but those are not end-to-end measures, and in fact, they verge on being meaningless because they ignore many issues of both law and voting technology.

In, my tutorial on optical-mark sense voting, I have documented in fair detail the interaction of law, human factors and technology that we must address if we are to correctly deal with these issues.

The relationship between law and questions of vote-counting accuracy is stunningly illustrated by the machine model of what constitutes a valid vote; this model is central to the arguments and appeals in Touchston and Shepperd vs. McDermott (U.S. Court of Appeals, 11th District, Dec 6, 2000). The machine model defines, as valid, those votes that the machine counts. Interpreted rather foolishly, this standard makes the entire question of voting machine accuracy moot!

We must formulate state laws that distinguish between the prescribed mark on a ballot, the mark voters are instructed to make to cast a vote, and the class of acceptable marks, those marks that the law counts as votes. For each ballot tabulating system, vendors must document the class of reliably counted and reliably ignored marks. For all real systems, there will also be marginal marks, marks that may be counted under some circumstances and ignored under others.

Only when all of these issues are pinned down can we reasonably begin to ask the key questions that allow us to grapple with the end-to-end accuracy of the voting system: How likely are voters to express their intent by making acceptable marks, a question of both human factors and law, and how likely is the machine to detect these acceptable marks and count them as votes, a question of technology.