Information about http://www.medien.ifi.lmu.de/pubdb/publications/pub/butz2003peephole/butz2003peephole.pdf
A Generalized Peephole Metaphor for Augmented Reality and…
Tags: andreas butz, cken, coordinate system, core idea, current research, environments, human perception, integral aspect, kr, krueger, magic lenses, magnifying glass, metaphor, previous research, prototype, saarland university, scarce resources, screen space, torch, university computer science,Pages: 4
Language: english
Created: Sun Sep 14 21:15:01 2003
Display cached document
Page 1
Page 2
Page 3
Page 4
A Generalized Peephole Metaphor for Augmented
Reality and Instrumented Environments
Andreas Butz, Antonio Kr¨ ger,
u
Saarland University, Computer Science, 66123 Saarbr¨ cken, Germany
u
email:(butz,krueger)@cs.uni-sb.de
Abstract-- We present the generalized peephole metaphor, a PDA by moving it in space. A corresponding prototype is
model of interaction in augmented reality and instrumented described in [4], introducing the concept of peephole displays.
environments. After briefly introducing previous research, which The prototype described can act within a coordinate system
led to the model, we describe a specific scenario in which we
introduce the core idea and some implications. The model nicely fixed in space, e.g., over a table, or in a coordinate system
matches research in human perception, which is also discussed fixed relative to the user of the device. In the first case, it can
briefly. A short outlook describes our current research, of which be used just like a magnifying glass or a torch in the dark, to
the generalized peephole metaphor has become an integral aspect. reveal information bound to positions in space. in the second
case, it uses the strong sense that humans have about their
I. I NTRODUCTION own body coordinate system, to position certain information
or applications relative to the user's body.
A. Peepholes, Toolglasses and Magic Lenses
Historically, screen space and screen resolution have always B. Instrumented Environments and Augmented Reality
been scarce resources, and this has remained true until today. Instrumented environments provide a sandbox in which ubi-
In order to make better use of existing screens, and enhance quitous computing scenarios can be investigated. Instruments
the (logical, not physical) focus of a graphical display, the and devices embedded in the environment give things and
interface metaphor of the magic lens[1] was developed. The places the power to display or sense information. They provide
magic lens is the electronic equivalent of a magnifying glass the connecting points between the physical world, in which
and allows the display of more detail or alternative types of they are located, and the virtual world to which they are
information in screen areas, over which it is (metaphorically) connected. In this sense, the instrumentation of an environment
held. To allow interaction and input through the lens, it is augments it without the use of classical AR tools, such as
combined with the toolglass, a filter modifying the meaning of projection or head-worn displays (see also [5]).
the mouse cursor acting through it. Bier et al. give a systematic While in the vision of Ubiquitous Computing as devised
overview of tools and types of interaction with these see- in [6], objects or spaces have the ability to make themselves
through interface widgets in [2]. known, display information, or sense input, augmented reality
While the magic lens metaphor was designed for 2D research mostly looks at the external augmentation of objects
screens, there is an equivalent within physical 3D environ- or spaces. For a single user, the result is ideally the same,
ments. In [3], George Fitzmaurice devised the vision of creat- no matter whether an object contains a display, information
ing situated information spaces by making palmtop computers is projected onto it, or overlaid to its visual appearance in a
situation aware. Making Information situated or localized in video-based or see-through head-worn display: The user sees
space provides a very natural access to it. By mapping virtual the information on the object. For multiple users, care has
information to the physical world, many types of information to be taken to provide a common perception of the virtual
can be provided in places, where they would belong, were they overlay, if this is created with head-worn displays [7], but
physical entities. This transforms the question how to access a their experience will be shared automatically in the case of
given information into the question, where to access it and thus projection or embedded displays. Issues arising from shared
makes use of the very well developed spatial memory humans experiences, such as privacy, synchronization and common
possess. A spatially aware palmtop device is just held to the reference points are increasingly addressed in the multi-user
physical location in question, and the associated information VR and AR literature, such as for example [8].
appears. Fitzmaurice's prototype was called the chameleon and If we want to create environments, in which multiple users
used a portable TV monitor as the palmtop device. can effortlessly experience the same augmentation, instru-
Palmtop computers have become commonplace today in the mentation and projection provide the ideal means for this.
form of electronic organizers. They are powerful computing Mark Weiser and Daniel Russel [9] make clear, that creating
devices and provide access to various types of information. a harmonized view on multiple devices and modalities is
Therefore, a logical step was to combine them with 3D a prerequisite for the successful exploitation of ubiquitous
tracking to further work on Fitzmaurice's vision and create computing and AR/MR in the future. In the following sections
a large personal information space accessible with a small we will describe a specific instrumented environment and
then propose such a harmonized and generalized view on III. T HE G ENERALIZED P EEPHOLE M ETAPHOR
instrumented environments, which we call the generalized A. Peephole Output
peephole metaphor. The idea of looking at AR and instrumented environments
in a way consistent with peephole displays [4] first became
obvious, when we thought about how to control the ceiling
II. E LEMENTS OF A S PECIFIC I NSTRUMENTED
mounted projector. In accordance with [10], a 3D model of
E NVIRONMENT
the room is used to eliminate spatial distortion in the projected
images. Technically, images are texture-mapped onto surfaces
The instrumented environment which we are currently set-
in an otherwise dark 3D space, and then the virtual camera
ting up, is called the SUPIE (Saarland University Pervasive
(mounted on the virtual ceiling) is aligned with the physical
Instrumented Environment). It currently consists of a single
projector. If we point the physical projector to a certain
(office) room containing a desk and a chair. The desk is
position in space, it will display whatever information is there
augmented by a projector in order to create visual elements
at the same position in the virtual 3D model. The projector
on its surface, and by a camera providing both a steady video
thus makes visible the information assigned to a given position
stream and high resolution pictures upon request. On the wall
is a large touch screen, corresponding to an instrumented
in space, only when it is pointed at this position. Information
blackboard. A small touch screen display in the shape of a in other positions is still there in the virtual model, but not
picture frame can display a decorative image by default, but visible in the physical room. The projector effectively provides
also provide information or serve as an area of interaction a peephole into the virtual space. In theory, with fast enough
when needed. In addition, The whole room can be augmented head and gaze tracking and projector control, this could be
used to constantly align the projector with the user's gaze,
which would make all information visible to the user by just
looking at it. In practice, however, this becomes somewhat less
compelling (to say the least) because of the lag between user
tracking and the mechanical motion of the projector.
B. Peephole Input
Symmetrically to the peephole view on output modalities,
input can be looked at through a peephole. Just as IBM's
ED projector is equipped with a camera, so is SUPIE's
steerable projector. Similarly to the desk, a camera is mounted
aligned with the projector and provides a permanent low
resolution video stream as well as high resolution images on
demand. This camera provides marker recognition via the AR
Toolkit [11] library and thus serves as an input modality by
Fig. 1. A conceptual sketch of SUPIE recognizing tagged objects and tracking them in space. This
input modality is, of course, only available in the area at which
the steerable projector is currently pointed, which conveniently
by projection from a ceiling-mounted movable projector, serv- creates an interactive area in the room by simultaneously
ing the same purpose as IBM's Everywhere Display [10]. providing input and output there.
This steerable projector effectively turns the whole room into
a large continuous display, since it can display onto any C. Multiple Peepholes
surface with direct line of sight to its position. Spatial audio An ideal display continuum would provide display capa-
is provided by speakers distributed in the SUPIE environment. bilities at any time on any given surface in the room. An
This allows the generation of spatialized sound in any location electronic wallpaper would be a good step in this direction.
of the room. In reality, the steerable projector can not point at all surfaces
Furthermore, the room contains regular contemporary com- simultaneously. It has to share display availability between
puters of various sizes, such as desktop PCs, laptops, tablet different places. In addition to the peephole provided by
PCs and PDAs. These provide display space in more or less the projector, all available screens can serve as additional
well known positions, as well as input facilities of sorts peepholes into the same information space. They just become
(keyboards, mice, pens, touch screens). A common software islands of high resolution and permanent availability within an
layer called the environment manager will connect these otherwise time-shared and low resolution display continuum.
devices and take over management of their input and output Similarly, all available input channels and devices in the room
modalities, similar to a window manager on conventional 2D can be seen as additional islands of permanently available
screens. While it is too early to discuss technical details of input. Touch screens, keyboards, mice, or interaction objects
this infrastructure, its design will be guided by the model of such as the Tuister [12] provide additional peepholes for
interaction described below. input to the virtual layer of the environment. This implies,
that basically all devices in the environment can be used as observers, often make them fail to notice big changes in the
peepholes of sorts into the virtual realm. It also means, that by image while looking at the image at the same time. This
peeping into the same virtual layer, they provide a consistent effect, known as change blindness, provides strong arguments
view on it and thus a shared experience for multiple users. against the existence of a visual buffer for highly detailed
information everywhere. The model described by Rensink
D. Privacy Implications [13] explains the role of attention to establish the visual
The peephole metaphor for instrumented environments also coherence between objects. By guiding our attention to a
allows a consistent treatment of privacy issues. Public displays, certain object, we retrieve highly detailed visual information
such as large wall-mounted displays provide shared peepholes, which is lost when our attention is guided to other parts of
while small and private displays, such as PDAs or head-worn the environment. A much coarser representation of the object
displays provide private views into the virtual. The issue of remains in memory which allows us, for example, to remember
access control to information in shared environments leads the object's position in space.
to the need to formulate a set of rules, which displays will This view on the cognitive representation for visual pro-
become blind for which types of information. The various cesses has some striking similarities to the generalized peep-
peepholes can simply contain filters, which work similarly to hole metaphor. Instead of displaying every visual information
magic lenses to make sure that only information appropriate to permanently, we rely on guided attention to (re-)establish
that specific display is displayed. This is also a generalization spatio-visual consistency between objects in the environment.
of the vampire mirror and lens concept introduced in [7]. The According to Rensink's model only the objects needed for
filters can then be switched or adjusted in order to make, for the visual task at hand are represented in high detail. The
example, public displays show content, which would normally argumentation for our peephole metaphor is very similar: Only
be considered private, if this is wanted by the content's owner. the real objects needed for the user's actual task are augmented
with virtual information. In the same way the brain is saving
E. The Information Space
processing power and memory, the instrumented environment
The generalized notion of peepholes and peephole interfaces can save similar resources when presenting and obtaining
allows a consistent integration of all available devices in a information to and from users. Rensink's model implies, that
room. Every information has a position in space and a set even if only a small part of the environment is visible in
of access restrictions assigned to it. Displays with known high detail, users will still be able to interact with the whole
positions and level of privacy can display corresponding infor- environment effectively.
mation. Movable displays can be moved to where information Another related cognitive model is the concept of active
is available and be used to display it. Conversely, information vision. If a user's attention is directed at a given object,
objects can make themselves known by opening a peephole it already perceives much more detail than in the visual
which shows them. A virtual alarm clock attached to an object periphery, because of an area of higher resolution in the center
can, for example, request the steerable projector to display it of the retina, called fovea. On top of this, humans actively
when it goes off and simultaneously play back an audio signal looking at an object become even more sensitive to changes
from its position. Searching a specific object then just means by building up hypotheses of expected events and actively
to ask it to actively open a peephole to make itself known. The adapting their vision, looking for these events. If we extend
spatial audio channel itself is a second display continuum with this concept to other sensory channels, we can speak of active
perfect simultaneous availability in the whole environment and sensing, if we also include output, we might have to speak
can be used to guide the user's attention to given locations. of active communication. The generalized peephole metaphor
Both the visual and the audio channel can be used separately can strongly support this cognitive model by providing detailed
to open a peephole. In the example of the alarm clock or information only in areas of the user's attention.
search function it might also be sufficient to ring an alarm
bell at the position of the object and then display it only on B. Active vs. Passive Peeping
explicit request of the user. This may avoid interruptions, if Two styles of interaction can be distinguished in an instru-
the projector is already used to display other information at mented environment adopting the peephole metaphor: In active
the same time. peeping the user accesses information directly at a certain
location in space whenever he or she wants. Users can actively
IV. I MPLICATIONS FOR HCI IN I NSTRUMENTED
open a peephole by controlling the steerable projector directly
E NVIRONMENTS
via gestures or a 3D-input device. This allows the user to
A. Cognitive Peepholes refresh or look up information which is expected at a certain
While traditional models of visual attention assume that position in space.
everything we see is accumulated in a visual buffer and that In passive peeping the attention of the user is guided
all subsequent cognitive processing relies on this buffer, newer by means of a peephole to support the user's actual task.
research seems to disprove the existence of a homogeneously Attention can be guided from one place to another, e.g.,
detailed buffer. Experiments have shown that visual distur- by rendering a virtual sound source at a certain position in
bances in an image, which attract the visual attention of the real environment or by using the projector to follow a
virtual object that moves from one location to another. Passive understanding and interacting with localized information. The
peeping involves knowledge about the user's current tasks peephole metaphor also supports concepts of privacy and
and goals in the instrumented environment and, of course, shared experience in multi user augmented realities.
information on all accessible peepholes (e.g. high resolution While most of the hardware of our instrumented environ-
displays). The instrumented environment needs to rely on ment SUPIE is set up at the time of writing, we're currently
sensor information received through peephole input channels, designing a software environment for the management of
as well as a sophisticated model containing information on all all elements of instrumentation. This software will provide
devices available in the environment. consistent management of input and output resources in the
environment, similar to the device driver layer of an operating
C. Saving Cognitive and Technical Resources system. On top of this layer, an environment manager will
As mentioned before, electronic wallpapers would be an be built, serving the same purpose as a window manager on
excellent, but currently technically still unfeasible solution for 2D screens. Together, this will provide a sandbox in which a
visually presenting information in an instrumented environ- variety of interaction techniques and usage metaphors can be
ment. However, in the light of the discussed cognitive models, implemented, evaluated, and refined.
even if we assume availability of simultaneous ubiquitous A fundamental guideline in building this environment will
display capability, maintaining the peephole metaphor will be the peephole model of interaction. While this paper pro-
have a number of advantages over displaying all information vides mainly the formal framework, supported by a number of
simultaneously. One example are permanently and rapidly evidences from research on visual perception, the generalized
changing environments. As known from theories of preatten- peephole metaphor remains to be tested and verified in user
tive vision [14], small changes in the field of view are easily studies to prove its scientific validity. We would like to discuss
attracting our attention (known as pop-out effect). Too many it at the STARS workshop in order to get additional opinions,
of those changes will lead to a chaotic environment distracting experiences, and feedback.
users, thus counteracting the vision of calm computing.
R EFERENCES
As described in [15], this problem can be overcome by
using the change blindness effect again, if visual information [1] E. A. Bier, M. C. Stone, K. Pier, W. Buxton, and T. D. DeRose,
"Toolglass and magic lenses: The see-through interface," Computer
is only updated while the user's gaze is absent. Unfortunately Graphics, vol. 27, no. Annual Conference Series, pp. 7380, 1993.
this requires high fidelity tracking of the user's gaze which [2] E. A. Bier, M. C. Stone, K. Fishkin, W. Buxton, and T. Baudel, "A
currently is only possible for well defined settings in limited taxonomy of see-through tools," in Proc. ACM CHI '94. ACM Press,
1994, pp. 358364.
spaces (e.g., in front of a desktop computer). For scenarios [3] G. W. Fitzmaurice, "Situated information spaces and spatially aware
adopting the peephole metaphor, only a small part of the palmtop computers," CACM, vol. 36, no. 7, pp. 3849, July 1993.
environment is augmented with information and even without [4] K.-P. Yee, "Peephole displays: Pen interaction on spatially aware hand-
held computers," in Proceedings of CHI 2003. ACM Press, 2003.
gaze tracking most of the environment remains calm most of [5] "Augmenting buildings with infrared information," in Proceedings of
the time. the International Symposium on Augmented Reality ISAR 2000. IEEE
In some situations, users should be notified about changes Computer Society Press, 2000.
[6] M. Weiser, "The computer for the 21st century," Scientific American,
in the environment occurring outside of all currently open vol. 3, no. 265, pp. 94104, 1991.
peepholes. How this can be done depends on the importance [7] "Enveloping users and computers in a collaborative 3d augmented
of the information. About rather important changes, the user reality," in Proceedings of the International Workshop on Augmented
Reality IWAR '99. IEEE Computer Society Press, 1999.
should be actively notified, for example by directing his or [8] D. Schmalstieg, A. Fuhrmann, G. Hesina, Z. Szalavari, L. M. Encar-
her attention to the position of change itself (as described nacao, M. Gervautz, and W. Purgathofer, "The studierstube augmented
above). Less important changes may be summarized when the reality project," Presence, vol. 11, no. 1, pp. 3354, February 2002.
[9] D. M. Russell and M. Weiser, "The future of integrated design of
user peeps at a location again, for example either by small ubiquitous computing in combined real & virtual worlds [late-breaking
animations or temporal diagrams (similar to stock charts) that results]," in CHI '98. ACM Press, 1998, pp. 275276.
visualize the changes since the last visit. [10] C. Pinhanez, "The everywhere displays projector: A device to create
ubiquitous graphical interfaces," in Proc. of Ubiquitous Computing 2001
(Ubicomp'01), Atlanta, Georgia, 2001.
V. C ONCLUSIONS AND O NGOING W ORK [11] M. Billinghurst, S. Weghorst, and T. Furness, "Shared space: An
We have introduced the generalized peephole metaphor, augmented reality approach for computer supported collaborative work,"
Virtual Reality, vol. 3, no. 1, pp. 2536, 1998.
providing a consistent way of describing input and output [12] A. Butz, M. Groß, and A. Kr¨ ger, "Tuister: a tangible ui for hierarchical
u
in instrumented environments and augmented reality environ- structures," in Proceedings of PI03: Workshop on Real World User
ments with partial augmentation. The metaphor provides a Interfaces, held at Mobile HCI 2003, Udine, Italy, 2003.
[13] R. Rensink, "Internal vs external information in visual representation,"
consistent way to describe some limitations of current AR in Proceedings of Smart Graphics 2002. ACM Press, 2002, pp. 6370.
technologies, namely the limited availability and capabilities [14] A. Treisman, "Preattentive processing in vision," CVGIP, vol. 31, pp.
of display and input devices. It is also consistent with current 156177, 1985.
[15] S. Intille, "Cange blindness information displays for ubiquitous comput-
models of visual perception and allows the user to build up ing environments," in Proceedings of UbiComp 02. Springer, 2002, pp.
a spatial model of the virtual layer which is overlaid to the 91106.
physical environment. Simultaneously, it makes the environ-
ment calm by eliminating unwanted distraction, and thus eases