Connecting Usability and Accessibility in Elections EAC Usability and…

Connecting Usability and Accessibility
in Elections
EAC Usability and Accessibility Roundtable, 27 March 2008

Whitney Quesenbery
Usability Professionals' Association (past-President)
Technical Guidelines Development Committee (chair for Human Factors and Privacy)



What is the connection between usability and accessibility?
Historically, usability (or human factors) has worked to maximize the number of people who can
use a product without difficulty. Accessibility work has focused on minimizing the number of
people who are unable to use a product at all, or can only use it with great difficulty. The people
who use any product are on a continuum, with many overlaps: a person with a disability may be
a "power user", while someone who is simply uncomfortable or unfamiliar with a technology may
only stumble through its use.
Removing the absolute barriers to access is a critical step, but it does not guarantee that the
product will now be usable to people with disabilities. The many reports of voters taking up to 45
minutes to vote with the accessible audio interface is an excellent example of access provided
without usability.
In other words, it takes access plus usability to provide accessible usability to all.
Usability and accessibility go hand in hand. Improving the accessibility of a product can improve
its usability for everyone. This is as true in ballot design and voting systems as it is in the
physical environment. For example, curb cuts and automatic door openers that make streets
and buildings accessible also make them easier to use for people with baby carriages or
luggage carts. Large, clearly defined controls or check boxes help people with dexterity
disabilities, but also help people whose hands are just a little unsteady or are not familiar with
using touchscreen technology.
Improving the usability of a product can also improve the accessibility of that product. This is
particularly true for cognitive, language and learning disabilities. In a presentation to the Access
Board's Telecommunications, Electronic and Information Technologies Advisory Committee
(TEITAC), Dr. Clayton Lewis suggested that cognitive disabilities should be seen as part of a
spectrum of abilities that includes all people, rather than a special class of people with distinct
behaviors, abilities and access needs. Almost any cognitive disability may be mild or severe.
For example, literacy extends from those who cannot read at all through levels of proficiency to
expert readers. His research suggests that many barriers to cognitive accessibility are the
same as usability problems for a general user audience... but more severe. We can think about
cognitive disabilities as amplifiers: problems that are simply a mild annoyance to many people
can be an absolute barrier to people with specific types of cognitive disabilities.
The usability and accessibility communities are starting to work together. Although still new, the
trend is clear. Leading usability and design experts now include expertise in accessibility in their
Connecting Usability and Accessibility in Elections

"portfolio" of skills. There is a growing awareness in the accessibility community of the
importance of usability for access features. These leaders are going beyond "access for screen
readers" to consider how to provide usable accessibility for all people. This trend is driven by
several forces:
 The emergence of international standards for accessibility (such as the W3C's Web
 Accessibility Initiative) and national laws (including "Section 508" here in the US). This has
 forced designers to consider a wider range of users, including those who use assistive
 technology.
 The growing number of products that function in a global marketplace, forcing designers to
 consider cultural differences in the user experience. This has opened a rich area of research
 to understand how to create products that meet different needs, and work in different
 contexts.
 The need to create information and communication technology (ICT) products that work
 across platforms. This push for "standards-based" design also benefits people who use
 assistive technology, as these products are intended to be used in many different modes
 (including small and large screens, visual and audio interfaces and with different input
 devices).
This change does not happen in a vacuum. One of the roles of standards and legislation is to
raise the bar, creating a demand that stimulates innovation to meet social needs. This is
especially true in situations like elections, where there is not a mass market to create demand
for new products.


Why are usability benchmarks important for voting systems?
The overarching concern for both usability and accessibility of voting systems is that voters are
able to vote accurately. The introduction to the draft VVSG Part 1, Chapter 3, Section 3.2 says
"The voting system should support a process that provides a high level of usability for all voters.
The goal is for voters to be able to negotiate the process effectively, efficiently, and
comfortably." This is a paraphrase of the standard definition of usability from ISO 9241-11:1999
which defines usability as "ability of specified users to complete tasks in a specified context of
use with effectiveness, efficiency, and satisfaction." In the case of voting system standards, the
"users" are the broad group of all voters, including people with disabilities, those who speak
other languages, and the full range of human abilities.
Historically, efforts to improve usability have focused on designing and developing usable voting
systems. A user centered design process, and usability testing during development, is key, but
does not offer a way to independently evaluate the final product. The VVSG cannot dictate
design process, but it can--and does--require that products meet benchmarks. The current
draft VVSG simply adds usability and accessibility benchmarks to the core reliability,
mechanical, electrical, and other technical requirements.



Whitney Quesenbery                                                                           Page 2
Usability Professionals' Association
Chair, Human Factors and Privacy, TGDC
Connecting Usability and Accessibility in Elections

Testing systems against human performance in using them may be new, but it does not make
these measurements any less valid. Usability benchmarks are the only way to test a system for
one of the most basic requirements: the ability of voters to cast their ballot as they intend, with
accuracy (as well as efficiency and confidence).



Why not rely on simpler design guidelines?
Design guidelines are a useful part of any standard. They document best practices and features
known to support the goals of the standard. At a minimum, they ensure that systems are
capable of meeting known requirements for good usability and accessibility. But, they do not
directly address the ultimate goal of the VVSG: voting systems that allow all voters to cast their
ballots with a high level of usability. It is possible for a product to meet a set of design guidelines
and still fail to achieve the usability goal. Many of the recent problems in elections have been
powerful illustrations.
It is tempting to try to tighten the design requirements, adding new guidelines to address each
problem reported. This not only adds complexity to the guidelines, but it risks constraining
systems so severely that there is no room for innovation. In the end, usability testing is the
summative proof that the real goals of the guidelines are met.
Two objections have been raised to the performance benchmarks:
 They are too difficult and expensive to test
 Manufacturers will "design to the test"
We simply don't believe that testing these benchmarks is too difficult or too expensive. It is new,
and therefore unknown, but this is not a difficult test to administer or evaluate. Even with all the
complexity of elections, the range of tasks is not large. Even with a long test ballot, voting is a
relatively short task. This makes it possible to include a large number of participants in a
reasonable test schedule.
As for the danger that we will see systems that work well in the test lab, but fail in the real world,
frankly, anything that helps voting systems designers focus on improving usability and
accessibility performance is not a bad thing.


Are the benchmarks high enough?
Voting is an area where the benchmark has to be almost at the top. Every citizen has the right
to vote. Any problem in design, writing, usability, or accessibility that keeps a voter from voting
the voter's intention accurately and completely means that the system has failed. We have to
produce systems that don't fail the voters. If the standards raise the bar, manufacturers will find
a way to meet that new level. That's what standards and regulations do. And the American free
enterprise system responds to the opportunities created by standards.



Whitney Quesenbery                                                                              Page 3
Usability Professionals' Association
Chair, Human Factors and Privacy, TGDC
Connecting Usability and Accessibility in Elections

Is the test protocol appropriate for testing the performance benchmarks?
The NIST staff has worked carefully, methodically, and thoroughly to develop the Voting
Performance Protocol. This attention to detail and rigorous process is appropriate to a test
method that breaks new ground, and will have such wide implications. Several aspects of the
protocol are particularly important in ensuring that the test is reliable:
 The use of standard voting instructions, and "realistic, but not real" candidate and party
 names eliminates concerns that test participants will not take care to vote as intended, or
 will reject instructions that go against their own political preferences.
 The "black box" approach, including standard instructions with no additional assistance,
 removes the need to carefully observe the test participants' actions, and limits any variation
 caused by differences in test moderators.
 The test includes no qualitative analysis. The results can be quickly and easily calculated,
 with no expert interpretation.
We wish the work would proceed faster, especially the benchmarking of the accessible
interfaces, but we understand the challenges of this work.




Will the results of the performance tests be misinterpreted?
A final objection to the performance tests is that the results will be misinterpreted as the number
of people who can vote accurately. Like energy efficiency ratings, the benchmarks are designed
to allow comparison, not predict actual performance. The role of the test is to create a way to
compare any system to a standard measurement. This gives any system, whether conventional
or innovative, an equal and fair way to prove itself. It will take careful education to ensure that
election officials, the media, election advocates and the public understand the meaning of the
test results.


How can we recognize, and certify, experts in usability and accessibility?
The "problem" of not having certifications for usability or accessibility testing is in part a
"chicken-and-the-egg" issue: there have been no public testing programs that required such
certification, so there has been no market need for such programs. This does not mean,
however, that experts do not exist and cannot be found when needed, or that suitable criteria
could not be created. It does mean that test labs will have to reach out beyond their usual
community into a new source of experts.
Even without certification, it is still possible to recognize an appropriate expert in usability,
accessibility, or human performance testing.
The last balloted draft of the IEEE standard for voting systems included this definition of a
usability expert, and includes a similar definition of a plain-language expert:


Whitney Quesenbery                                                                                  Page 4
Usability Professionals' Association
Chair, Human Factors and Privacy, TGDC
Connecting Usability and Accessibility in Elections

      "Usability experts should have experience in conducting summative, quantitative,
      performance-based usability tests. They should also have experience with
      systems for the general public and in accessibility. They should have professional
      qualifications or significant experience in human factors, human computer
      interaction, cognitive psychology, usability, or a related field. There is no standard
      industry certification for a usability expert."
      IEEE P1583: Voting System Standards - Section 6.3.3.1


There are few degrees with a named focus on usability, but experts often have advanced
degrees or relevant professional experience in fields such as:
 Human computer interaction
 Human factors and ergonomics
 Cognitive psychology
 Social or psychological research methods


The Board of Certification in Professional Ergonomics (BCPE) (http://www.bcpe.org/) offers
professional certifications in ergonomics. The BCPE also adheres to the criteria and policies for
competency assessment set by NOCA, the National Organization for Competency Assurance,
and has been endorsed by the IEA, the International Ergonomics Association, as an accredited
ergonomics certifying body.
The BCPE offers two professional certifications as well as two lower-level associate
certifications. (http://bcpe.org/info/default.asp)
 Certified Professional Ergonomist (CPE)
 Certified Human Factors Professional (CHFP)
The criteria for certification as a CPE (or CHFP) are:
 A master's degree in human factors/ergonomics, or an equivalent educational background in
 the life sciences, engineering sciences and behavioral sciences to comprise a professional
 level of ergonomic education.
 Three (3) years of full-time professional practice in human factors/ergonomics.
 A passing score on the CPE/CHFP written examination.


Usability and human factors experts are often members of one or more of the following
professional organizations:
 Human Factors and Ergonomics Society (HFES) (www.hfes.org)

Whitney Quesenbery                                                                             Page 5
Usability Professionals' Association
Chair, Human Factors and Privacy, TGDC
Connecting Usability and Accessibility in Elections

 Usability Professionals' Association (UPA) (www.usabilityprofessionals.org)
 ACM SIGCHI (ACM Special Interest Group on Computer Human Interaction)
 (www.acm.org/sigchi)
There are also human factors and testing experts in the IEEE and the IEEE Computer Society
(www.ieee.org).

Experts In Usability Testing With Human Participants
One of the important differences between usability testing and other quality testing is that it uses
human participants as part of the "test instrument." This places some burdens on the people
conducting the test.
There are several professional society ethical guidelines or codes of conduct that are relevant.
Experts should be aware of these guidelines and should be able to demonstrate that they follow
them.
 Usability Professionals' Association
 http://www.usabilityprofessionals.org/about_upa/leadership/code_of_conduct.html
 Human Factors and Ergonomics Society
 http://www.hfes.org/web/AboutHFES/ethics.html
 American Psychological Association
 http://www.apa.org/ethics/code2002.html
There is also educational material on human participant protections from federal agencies, and
a self-certification program used by the National Institutes of Health:
 National Cancer Institute - Human Participant Protections Education for Research Teams.
 "This free, web-based course presents information about the rights and welfare of human
 participants in research. The two-hour tutorial is designed for those involved in conducting
 research involving human participants. It satisfies the NIH human subjects training
 requirement for obtaining Federal Funds. You will have the option of printing a certificate of
 completion from your computer upon completing the course."
 http://cme.cancer.gov/clinicaltrials/learning/humanparticipant-protections.asp
Graduate programs and academic research institutions all have similar guidelines and
education, usually part of their Institutional Review Board.




Whitney Quesenbery                                                                           Page 6
Usability Professionals' Association
Chair, Human Factors and Privacy, TGDC