Information about http://www.csl.sri.com/users/neumann/ccs06.pdf
System and Network Trustworthiness in Perspective …
Tags: complexity, computer science laboratory, computer systems, development recommendations, environments, insiders, menlo, menlo park, menlo park ca, modes, neumann computer, outsiders, peter g neumann, principled, problem areas, security vulnerabilities, societal dependence, sri international, systems group, trustworthiness,Pages: 5
Language: english
Created: Thu Sep 21 15:53:00 2006
Display cached document
Page 1
Page 2
Page 3
Page 4
Page 5
System and Network Trustworthiness in Perspective
Peter G. Neumann
Computer Science Laboratory, Principled Systems Group
SRI International, Menlo Park CA 94025-3493
ABSTRACT erating, maintaining, and using complex highly distributed systems
Characteristic problem areas experienced in the past are considered and networks -- and especially those with extensive requirements
here, as well as some of the challenges that must be confronted in for trustworthiness.
trying to achieve greater trustworthiness in computer systems and ˇ "The future isn't what it used to be."1 It is becoming ever
networks and in the overall environments in which they must oper- more difficult to keep up with critical needs for trustwor-
ate. Some system development recommendations for the future are thy applications in the face of increasing system and network
also discussed. security vulnerabilities, increasing development complexity,
multiple fault modes, sophisticated threats from insiders and
Category outsiders, ubiquitously increasing societal dependence on in-
D.4.6 Security and Protection formation technology, increased dependence on the roles of
administrators and users, and increasing costs and disrup-
General Terms tions associated with the resulting risks. All these factors --
and more -- tend to exacerbate the problems of developing
Design, security, reliability, verification trustworthy intensively computer-based enterprises.
Keywords ˇ "We need to go back to the future." Past research and devel-
opment efforts present many important lessons for the future,
Computer systems, networks, trustworthiness, security, reliability,
although many of those lessons are either forgotten or never
survivability, assurance, threats, vulnerabilities, risks
learned. Thus, it is useful to reconsider how we arrived at
the present state of the art, and extrapolate on what might be
1. INTRODUCTION needed in the future. We consider here both short-term and
In the context of this paper, trustworthiness means simply that long-term approaches, and outline some of the hard problems
something is worthy of being trusted to satisfy its specified require- that must be more systematically addressed.
ments, typically relating to overall information system properties
such as security, reliability, human safety, and survivability in the ˇ "Progress is always slower than you'd think." Proactive sys-
presence of a wide range of adversities, subject to some sort of as- tem development is often shunned, but has enormous poten-
surance measures. In this context, security can be broadly thought tial payoffs. Mistakes of the past are frequently repeated,
of as encompassing everything that promotes the prevention, detec- many of which could be easily surmounted -- for example,
tion, and remediation of deleterious system behavior (for example, using approaches such as evolvable system architectures, de-
including actions of people, information technology, and physical velopment that facilitates pervasive and predictable subsys-
environments). tem composability, intelligent interface design, sound system
Trustworthiness is relevant throughout system life cycles, from embeddings of strong cryptography, sound analysis tools,
conception to requirements to detailed architectural designs to im- and up-front attention to assured trustworthiness.
plementation, and then continuing into use, operation, maintenance,
and recycling through revisions as appropriate. Trustworthiness 2. PRINCIPLED SYSTEMS
is very difficult to achieve unless it is systematically integrated For many years, the security community has been evolving var-
throughout the development cycle. ious principles for the development of trustworthy systems, such
We begin with a few high-level remarks that might help moti- as [14] and the Saltzer-Schroeder principles [28], both of which
vate some of the problems that arise in designing, developing, op- grew out of the Multics development.
An extended approach to principled development [16] also pro-
vides some analysis of how these principles and others can affect
trustworthiness and assurance of systems as a whole. Several as-
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are pects of such an approach are highlighted here, notably the princi-
not made or distributed for profit or commercial advantage and that copies ple of least privilege; minimization of the extent to which portions
bear this notice and the full citation on the first page. To copy otherwise, to of a system or network must be trusted, thereby limiting what is
republish, to post on servers or to redistribute to lists, requires prior specific 1
permission and/or a fee. I first heard this almost-Yogi-Berra-like statement in 1969 in a
CCS'06, October 30November 3, 2006, Alexandria, Virginia, USA. keynote address by Arthur Clarke, who lamented that it was be-
Copyright 2006 ACM 1-59593-518-5/06/0010 ...$5.00. coming ever more difficult to write good science fiction.
vulnerable to accidents and misuse; modular abstraction with en- ˇ The 1980 ARPANET collapse: As a result of unchecked bits
capsulation and complete mediation; and thorough nonbypassable dropped in the highest-priority status messages in the mem-
auditing, In addition, various principles under the rubric of good ory of one ARPANET router (a so-called Interface Message
software engineering practice can add significantly to assured trust- Processor) and a garbage-collection algorithm that could sub-
worthiness, which -- in addition to the aforementioned properties sequently not eliminate any nonrecent status messages, a mem-
relating to security, reliability, safety, and survivability -- can also ory overflow occurred in every node in the network. It took 4
encompass other concepts such as predictable composability, as- hours to diagnose the unprecedented failure mode, telephone
sured interoperability, and real-time performance. the administrator of every node and request that it be shut
Multics was perhaps the most principled of system developments. down manually, and finally, upon confirmation that all nodes
It broke some important new ground with hardware-software sup- had been shut down, manually initiate restarting every node.
port for virtual memory, segmentation, paging, protection domains, (See [24].)
stacks in which anything outside of the current stack frame was
nonexecutable and nonaddressable, hierarchical directories, access ˇ The 1990 AT&T long-lines collapse: An untested upgrade to
control lists, dynamic linking, the use of a stark subset of PL/I that the fault-tolerant recovery software had been installed in 114
naturally prevented certain flaws, and so on. However, most in- ESS Number 4 telephone switches. (The C program con-
structive may have been its development discipline [5, 6]. tained a break statement within an if clause nested within
Other principled efforts from the 1960s and 1970s include the a switch clause.) On 15 January 1990, one node crashed,
T.H.E. system and its use of a deadlock-avoiding hierarchical lock- and auto-recovered. However, as a result of the flaw, all
ing strategy described by Dijkstra [7], and a seminal series of pa- the immediate neighbor nodes crashed, followed by iterated
pers by Parnas (e.g., [20]). The 1977 RobinsonLevitt paper [22] crashes of every node in the network repeatedly for some-
on hierarchical formal specifications introduced the concept of for- thing approaching half a day. Completing long-distance tele-
mal mappings between different layers of functional specifications phone calls became almost impossible. (See the ACM Risks
that represent abstract implementations of each layer as a function Forum, vol 9, number 61, and Telephony, 22 January 1990,
of the lower layers, as the foundation for the design of the Provably p. 11.)
Secure Operating System (PSOS) [8, 17, 18]. In both of these cases, the developers of the networking technol-
The desire for multilevel security (MLS) prompted considerable ogy seemed to have believed that network-wide outages could not
research and development in the 1970s and 1980s, although those possibly result from single-point failure modes.
efforts did not bear much fruit with respect to commercially avail- Computer-controlled electrical-power networks have also been
able MLS systems with strong assurance. Various efforts that were implicated in numerous widespread propagating power outages,
countercultural at the time (e.g., Rushby-Randell [27] and Proctor- evidently with some a priori disbelief on the part of developers
Neumann [21]) are gaining new currency with respect to the con- and operators that widespread outages could result. (References
cept of Multiple Independent Levels of Security (MILS) within the for these and other cases noted below can be found in [13], with
Secure Global Information Grid. some of the older cases documented in [15].)
In addition, there is renewed interest in Rushby's 1981 notion of
a separation kernel [26] and also in virtual machine monitors (such ˇ Northeast U.S. blackout, November 1965: a threshold was
as VMware and Xen), which have a basic need for disciplined but exceeded that had been set too low, resulting in a 13-hour
limited interpartition communications [3]. outage that affected New York, Pennsylvania, Vermont, Con-
Principles that are generally accepted in principle are often not necticut and Massachusetts.
widely used in practice. There are several possible explanations
ˇ Lower New York State blackout, July 1977: recovery took in
for this. For example, principles are not absolute; they are also not
excess of 26 hours.
independent, and may sometimes interfere with one another; and
their effective use requires appropriate educational stimuli, expe- ˇ Parts of ten Western U.S. states blacked out, October 1984:
riential feedback, acquired discipline, and (above all) foresight -- a faulty computer reading (the actual power loss was off by
which often runs counter to short-term motives such as profits and a factor of two) caused a chain reaction, although the trig-
marketplace. gering loss was reportedly an insignificant routine event that
To illustrate the relevance of principled developments, we con- was supposed to have been handled seamlessly.
sider some particularly problematic real-world uses of computers
and communications in the next two sections. ˇ Western U.S., July 1996: a heat-wave expanded power lines,
which came in contact with a tree. Resulting power outages
propagated over at least 12 states.
3. NETWORK DISRUPTIONS ˇ Western U.S./Canada/Baja California (Mexico), August 1996:
Past networking difficulties offer lessons that need to be assim- a summer heat-wave triggered cascading power outages, re-
ilated by researchers and system developers. This section revisits sulting in air-traffic slowdowns and many collateral prob-
some old and new history relating to widespread network outages lems.
and other extreme network behavior. The earlier history may be of ˇ Northeast U.S., August 2003: a software race condition and
particular interest to younger researchers and developers who were an alarm-system failure caused a computer crash and widespread
not active in the olden days. power outages; recovery took almost
Various examples of widespread propagation effects are illus- 2 days.
trative of some of the complexities arising in networked systems.
Of particular interest are two cases in which total network failure ˇ Circuit breakers tripped in Maryland, in Queens, New York,
modes resulted from local faults -- namely the 1980 ARPANET and in Philadelphia on 25 May 2006, shutting down Amtrak,
collapse and the 1990 AT&T long-distance collapse -- as well as NJ Transit, and MARC trains in the morning rush-hour for
numerous instances of domino-like propagating power outages. the day. Five trains were stuck in tunnels.
ˇ Queens, New York, July 2006: century-old wiring failed over rigorously controlled interactions among partitions (e.g., [3]), care-
a wide local region; the outage lasted a week. fully configured fine-grained access control policies (e.g., [9]), good
software engineering practice with safe uses of sensible program-
What is perhaps most alarming about this list of outages is that
ming languages and static analysis tools for software. With these
such cases continue to occur. In some cases, this happens despite
and other approaches, the challenge of preventing exploitations of
efforts to take remedial measures; in other cases, such measures are
malware and security flaws such as buffer overflows could be much
not even taken. Again, we often hear that a particular propagating
less problematic than it is today. (See [1, 10, 19, 31] for taxonomies
outage is impossible because of the system design. Then, after it
of security flaws.)
happens, we are typically told that the system or software or algo-
With respect to confining the propagation of undesirable behav-
rithms have been changed to prevent the problem from ever hap-
ior in networks and distributed systems, the principles of security
pening again. And then something similar happens again. Clearly,
and good software development could contribute considerably. For
more proactive efforts are needed to analyze systems for potential
example, the principle of least privilege suggests the development
widespread fault modes and ensuing risks, and to take appropri-
of architectures that enforce confined-domain or virtual-machine
ate timely actions. Of course, similar conclusions apply to proac-
process isolation, with intercomponent interfaces and network pro-
tive defenses against environmental disasters, as in the case of tidal
tocols that are well defined, carefully controlled, and carefully an-
waves, global warming, and anticipation of hurricanes (which was
alyzed. The principle of complete mediation suggests an archi-
lacking in New Orleans before Katrina).
tecture in which only a subset of the system needs to be highly
Leslie Lamport has remarked that a distributed system is a sys-
trustworthy, and in which the integrity of that subset cannot be cir-
tem in which you have to trust components whose existence is com-
cumvented.
pletely unknown to you. In the present context, distributed con-
trol of distributed systems with distributed sensors and distributed
actuators is typically more vulnerable to propagated outages and 4. COMPUTERIZED ELECTIONS
other perverse failure modes such as deadlocks and other unrecog- The problems of trying to ensure the fairness, accuracy, and over-
nized hidden interdependencies (particularly on components that all integrity of elections necessitate many requirements that span
are untrustworthy and perhaps even hidden), race conditions and the entire election process. Various end-to-end requirements are
other timing quirks, coordinated denial-of-service attacks, and so needed to address system integrity, reliability, and survivability,
on. Achieving reliability, fault tolerance, and system survivability vote integrity, overall accountability and oversight, nonsubvertible
in highly distributed systems is problematic; both formal analyses audit trails, impartial resolution of disputes, and uncompromised
and system testing are highly desirable, but also potentially more voter privacy (to identify just a few requirements). Simultaneously
complex than in nondistributed systems. satisfying both vote integrity and voter privacy requires special care
Furthermore, security, reliability, and system survivability are in design and implementation of operational procedures and rele-
closely linked. A system is not likely to be secure if it is unre- vant computer systems.
liable; similarly, a system is not likely to be reliable if it is not Since the mid-1980s, various efforts have been made to increase
secure. A system is not likely to be survivable in the face of natu- the automation of the voting process, particularly with the wider
ral, accidental, and malicious adversities if it is not secure, reliable, use of computer-based systems. (See Mercuri's doctoral thesis [11]
fault tolerant, people tolerant, usable, and easily administered, with for an extensive set of requirements casting the computer system re-
sufficient performance even under crises. quirements into the framework of the Common Criteria.) However,
To illustrate this point, although the ARPANET outage and the the need for trustworthy computing is only part of the problem.
AT&T long-lines collapse were triggered spontaneously by mech- Trustworthiness is required throughout the end-to-end process.
anisms that required no human intervention, each could have been Ideally, the total-process combination of computer system archi-
equally well caused by (possibly remote) human activity, either in- tectures and operational procedures should seek strength in depth,
tentionally (by anyone who knew about the fault mode) or acciden- to prevent isolated failures or single-point attacks from compromis-
tally. In the ARPANET case, the insertion of two bogus network ing the end results. Unfortunately, the entire election process today
status messages could have created the same effect as the dropped represents weakness in depth: every step is a potential weak link
bits. In the AT&T case, inadvertent action by maintenance person- -- from registration to voter authentication to vote casting, vote
nel or intruders on one telephone switch could have initiated the counting, and transmission of the partial and final results (perhaps
same sequence of propagating crashes. (There is some reason to even via the Internet or wireless communications, perhaps unen-
suspect that this might actually have happened.) Similar reason- crypted or weakly protected, and subject to denial-of-service at-
ing also applies to the distribution of electrical power, where the tacks). Indeed, each stage in the voting process represents potential
control systems are heavily dependent on computer systems, with vulnerabilities that can be compromised in many ways -- for exam-
considerable functionality accessible from the Internet. Thus, se- ple, accidentally or intentionally, detectably or undetectably, tech-
curity, reliability, and survivability need to be considered together nologically or procedurally. The potential risks of faults, errors,
rather than separately. failures, and misuse encompass human, environmental, and tech-
Any discussion of network disruptions must of course mention nological causes. Each step must be safeguarded from the outset
viruses, worms, propagating Trojan horses, and other forms of dis- with extensive cross-checks, and should be noncompromisibly au-
tributed malware. It is worth revisiting the 1988 Internet Worm [23, dited. These requirements become especially important whenever
29], which exploited four different system weaknesses -- a buffer election results are contested.
overflow in the finger daemon, systems configured with overly In today's all-electronic paperless voting machines, there is ef-
permissive .rhosts access, the sendmail debug option, and fectively no independent confirmation that the vote that is cast is
dictionary attacks on encrypted password files; however, it ran amok the same as the vote that is counted, and that the results remain
because of an overly aggressive algorithm for keeping the worm unchanged throughout. Furthermore, there is no independent audit
alive. Preventing malware might be aided by a system environment trail that would enable an irrefutable recount. Today's unauditable
that provides a combination of sound computer system architec- proprietary touch-screen systems (e.g., [25]) are typically evaluated
tures with confined execution environments or virtual machines, under a proprietary process that is commissioned by the vendors,
relative to voluntary standards that are inherently weak. The result delivery delays, easier system operation, and much less need
is fundamentally unconvincing from the perspective of trustworthi- for recurring patch management. Systems should be designed
ness. to be predictably composable out of evaluated subsystems,
One potential remedy for having to trust today's unauditable pro- with correspondingly predictable properties.
prietary all-electronic voting systems is to add an independent voter-
verified paper trail that is the vote of record [11], as some vendors ˇ Many characteristic system vulnerabilities can be system-
are now attempting to do. However, this introduces further com- atically avoided or at least considerably diminished, espe-
plexities, and appears to be only a palliative addition to an already cially in newly developed systems, but also through judicious
overly expensive PC-based solution when compared with less ex- use of well-conceived static analysis tools applied to existing
pensive paper-based alternatives that are much less dependent on systems.
untrustworthiness of computer systems, such as optical scan tech- ˇ Although there are always many difficulties that arise in try-
nologies (but which of course have their own potential risks). ing to retrofit wisdom into legacy systems, many of the prin-
Of great potential interest to security researchers are possible ciples can also be applied to incremental evolutions -- for
voting systems and associated procedures that are designed, imple- example, incorporating an architecturally sound combination
mented, and rigorously evaluated openly to be demonstrably able of old and new subsystems, or attempting rather humbly to
to satisfy stringent end-to-end requirements. One very promising encapsulate or otherwise mask the weaknesses of older sys-
direction involves cryptographic approaches (e.g., [2, 4, 12]) that tems (albeit with somewhat lowered expectations of trust-
could be formally verified. Of course, a fundamental challenge worthiness).
for the cryptography community is to provide convincing evidence
that the resulting total systems (that is, not just the cryptography in Trustworthiness (especially with respect to overall security, re-
isolation) are demonstrably resistant to internal tampering, external liability, system survivability, and human safety) is inherently a
manipulation, and undetected system errors, including compromise weak-link phenomenon. In poorly designed systems, one flaw may
from any underlying operating systems (which do not even have be enough to result in compromise. Simplistic solutions are likely
to be considered under today's voluntary evaluation guidelines) or to be untrustworthy. Merely patching a few of the more obvious
from compiler subversions (such as in [30]). Such evidence would flaws is not likely to be very successful. Therefore, considerable
need to be convincing to independent cryptographers, system secu- pessimism should accompany any efforts to make silk purses out
rity experts, and -- to a considerable extent -- the general public. of sows' ears. However, trying to minimize what must be trust-
Although the problem of adding up a bunch of votes might in- worthy, avoiding violations of the principle of least privilege, and
tuitively seem easy, the complexity of the overall election process wisely applying some of the other principles can help significantly.
is considerable, particularly in operational practice. With respect This paper addresses just the tip of a huge iceberg that is lurking
to computerized elections, serious observance of the principles of in our path. It should not be surprising. Its message is clearly not
Section 2 could (in principle) provide dramatic improvements in to- novel (for example, see [14, 28]), but nevertheless still fundamental
day's voting technology, but would need to be applied end-to-end and still underappreciated. In addition, more of the less visible
to detect, prevent, and recover from software-hardware failures and portions of the iceberg are examined in [16], along with how to
power outages, as well as surreptitious manipulations and other un- attain predictable composability of trustworthy systems with some
expected irregularities in human procedures throughout the election meaningful measures of assurance.
process. The increasing complexity of modern systems, the critical de-
Of particular relevance here is the principle of least privilege. As pendence on computer-communication technology, and the ever
an extension of that principle, what is needed are system architec- greater need for trustworthiness make it imperative that we reflect
tures, network protocols, and operational procedures that minimize on mistakes of the past and attempt to avoid them in the future.
the areas of vulnerability, for reasons similar to those in Section 3, The road to disciplined system development is clearly paved with
but with rather different threat models. Also needed are indepen- good intentions, and the suggested course is riddled with practi-
dent cross-checks that can ensure the integrity of the process de- cal pitfalls. However, designing and implementing systems with
spite accidental failures and intentional manipulations. In addition, trustworthiness as an integral fundamental goal is a very important
the principle of pervasive nonbypassable auditing is critical, with challenge, and is well worth pursuing vigorously in research and
nonalterable audit trails and oversight sufficient to enable incontro- development practice.
vertible reconstruction of all votes as cast.
6. ACKNOWLEDGMENTS
5. CONCLUSIONS This paper was prepared in part under National Science Founda-
If we take the principled advice of Section 2 and the examples of tion Grant Number 0524111. The author thanks Douglas Maughan,
Section 3 and Section 4 seriously, several benefits can result. who sponsored the work culminating in [16] when he was a Pro-
gram Manager in the U.S. Defense Advanced Research Projects
ˇ The development process could be dramatically improved. Agency (DARPA). (Doug is now a Program Manager in the Sci-
Principled system developments and selective uses of formal ence and Technology Directorate in the Department of Homeland
methods are increasingly needed. Greater up-front empha- Security, where he spearheads research and development relating
sis on requirements, sound architectures, and good software to cybersecurity.) The author is grateful to Rebecca Wright for her
engineering practice can result in fewer cost overruns, fewer very helpful feedback.
7. REFERENCES [16] P.G. Neumann. Principled assuredly trustworthy composable
[1] R.P. Abbott et al. Security analysis and enhancements of architectures. Technical report, Computer Science
computer operating systems. Technical report, National Laboratory, SRI International, Menlo Park, California,
Bureau of Standards, 1974. Order No. S-413558-74. December 2004.
[2] B. Adida and C.A. Neff. Ballot casting assurance. In http://www.csl.sri.com/neumann/chats4.html, .pdf, and .ps.
Workshop on Electronic Voting Technology Workshop, [17] P.G. Neumann, R.S. Boyer, R.J. Feiertag, K.N. Levitt, and
Vancouver, BC, Canada, August 2006. USENIX. L. Robinson. A Provably Secure Operating System: The
[3] Steven M. Bellovin. Virtual machines, virtual security? system, its applications, and proofs. Technical report,
Communications of the ACM, 49(10), October 2006. Inside Computer Science Laboratory, SRI International, Menlo
Risks column. Park, California, May 1980. 2nd edition, Report CSL-116.
[4] J. Benaloh. Simple verifiable elections. In Workshop on [18] P.G. Neumann and R.J. Feiertag. PSOS revisited. In
Electronic Voting Technology Workshop, Vancouver, BC, Proceedings of the 19th Annual Computer Security
Canada, August 2006. USENIX. Applications Conference (ACSAC 2003), Classic Papers
section, pages 208216, Las Vegas, Nevada, December 2003.
[5] F.J. Corbat´ . On building systems that will fail (1990 Turing
o
IEEE Computer Society. http://www.acsac.org/ and
Award Lecture, with a following interview by Karen
http://www.csl.sri.com/neumann/psos03.pdf.
Frenkel). Communications of the ACM, 34(9):7290,
September 1991. [19] P.G. Neumann and D.B. Parker. A summary of computer
misuse techniques. In Proceedings of the Twelfth National
[6] F.J. Corbat´ , J. Saltzer, and C.T. Clingen. Multics: The first
o
Computer Security Conference, pages 396407, Baltimore,
seven years. In Proceedings of the Spring Joint Computer
Maryland, 1013 October 1989. NIST/NCSC.
Conference, volume 40, Montvale, New Jersey, 1972. AFIPS
Press. [20] D.L. Parnas. On the criteria to be used in decomposing
systems into modules. Communications of the ACM, 15(12),
[7] E.W. Dijkstra. The structure of the THE multiprogramming
December 1972.
system. Communications of the ACM, 11(5), May 1968.
[21] N.E. Proctor and P.G. Neumann. Architectural implications
[8] R.J. Feiertag and P.G. Neumann. The foundations of a
of covert channels. In Proceedings of the Fifteenth National
Provably Secure Operating System (PSOS). In Proceedings
Computer Security Conference, pages 2843, Baltimore,
of the National Computer Conference, pages 329334.
Maryland, 1316 October 1992.
AFIPS Press, 1979.
(http://www.csl.sri.com/neumann/ncs92.html).
http://www.csl.sri.com/neumann/psos.pdf.
[22] L. Robinson and K.N. Levitt. Proof techniques for
[9] P.A. Karger. Limiting the damage potential of discretionary
hierarchically structured programs. Communications of the
Trojan horses. In Proceedings of the 1987 Symposium on
ACM, 20(4):271283, April 1977.
Security and Privacy, pages 3237, Oakland, California,
April 1987. IEEE Computer Society. [23] J.A. Rochlis and M.W. Eichin. With microscope and
tweezers: The Worm from MIT's perspective.
[10] C.E. Landwehr, A.R. Bull, J.P. McDermott, and W.S. Choi.
Communications of the ACM, 32(6):689698, June 1989.
A taxonomy of computer program security flaws, with
examples. Technical report, Center for Secure Information [24] E. Rosen. Vulnerabilities of network control protocols. ACM
Technology, Information Technology Division, Naval SIGSOFT Software Engineering Notes, 6(1):68, January
Research Laboratory, Washington, D.C., November 1993. 1981.
[11] R. Mercuri. Electronic Vote Tabulation Checks and Balances. [25] A. Rubin. Brave New Ballot. Random House, 2006.
PhD thesis, Department of Computer Science, University of [26] J.M. Rushby. The design and verification of secure systems.
Pennsylvania, 2001. In Proceedings of the Eighth ACM Symposium on Operating
http://www.notablesoftware.com/evote.html. System Principles, pages 1221, Asilomar, California,
[12] C.A. Neff. A verifiable secret shuffle and its application to December 1981. (ACM Operating Systems Review, 15(5)).
e-voting. In Proceedings of the ACM Conference on [27] J.M. Rushby and B. Randell. A distributed secure system
Computer and Communications Security, pages 116125, (extended abstract). In Proceedings of the 1983 IEEE
Philadelphia, Pennsylvania, November 2001. Symposium on Security and Privacy, pages 127135,
[13] P.G. Neumann. Illustrative risks to the public in the use of Oakland, California, April 1983. IEEE Computer Society.
computer systems and related technology, index to RISKS [28] J.H. Saltzer and M.D. Schroeder. The protection of
cases. Technical report, Computer Science Laboratory, SRI information in computer systems. Proceedings of the IEEE,
International, Menlo Park, California. Updated regularly at 63(9):12781308, September 1975.
http://www.csl.sri.com/neumann/illustrative.html; also in .ps [29] E.H. Spafford. The Internet Worm: crisis and aftermath.
and .pdf form for printing in a denser format. Communications of the ACM, 32(6):678687, June 1989.
[14] P.G. Neumann. The role of motherhood in the pop art of [30] K.L. Thompson. Reflections on trusting trust.
system programming. In Proceedings of the ACM Second Communications of the ACM, 27(8):761763, August 1984.
Symposium on Operating Systems Principles, Princeton, [31] K. Tsikpenyuk, B. Chess, and G. McGraw. Seven pernicious
New Jersey, pages 1318. ACM, October 1969. kingdoms: A taxonomy of software security errors. IEEE
http://www.multicians.org/pgn-motherhood.html. Security and Privacy, 3(6), November-December 2005.
[15] P.G. Neumann. Computer-Related Risks. ACM Press, New
York, and Addison-Wesley, Reading, Massachusetts, 1995.