Image-Based Modeling and Rendering …

                Image-Based Modeling and Rendering
                                      Paul Debevec (organizer)
                                  University of California at Berkeley

                                          Christoph Bregler
                                          Stanford University

                                           Michael Cohen
                                         Microsoft Corporation

                                        Leonard McMillan
                                 Massachusetts Institute of Technology

                                         François Sillion
                                      iMAGIS - GRAVIR/IMAG

                                           Richard Szeliski
                                         Microsoft Corporation


                                 SIGGRAPH 99 Course 39 (Full Day)
                                     Tuesday, August 10, 1998



                                        Course Abstract
Image-based modeling and rendering differs from traditional graphics in that both the geometry and
appearance of the scene are derived from real photographs. The techniques often allow for shorter
modeling times, faster rendering speeds, and unprecedented levels of photorealism. In this course we will
explain and demonstrate a variety of ways of turning images into models and then back into renderings,
including movie maps, panoramas, image warping, photogrammetry, light fields, and 3D scanning. This
course overviews the relevant topics in computer vision, and show how these methods relate to image-
based rendering techniques. The course shows ways of applying the techniques to animation as well as to
3D navigation, and to both real and synthetic scenes. One underlying theme is that the various modeling
techniques make tradeoffs between navigability, geometric accuracy, manipulability, ease of acquisition,
and level of photorealism; another theme is the close connection between image-based modeling and
rendering and global illumination. The course shows how image-based lighting techniques allow
photorealistic additions and modifications to be made to image-based models. The described techniques are
illustrated with results from recent research, pioneering projects, and creative applications in art and
cinema.

Note: This course and SIGGRAPH Course #28, 3D Photography, cover related topics and are designed to
be complimentary.




                                                                                                     0-1
                                           Presenters

Christoph Bregler
Assistant Professor
Computer Science Department
Gates 138, 353 Serra Mall
Stanford University
Stanford, CA 94305
(650) 725-6359
(650) 725-1449 Fax
bregler@cs.stanford.edu
http://www.cs.stanford.edu/~bregler

Chris Bregler is an Assistant Professor in Computer Science at Stanford University. He received his
Diplom in Computer Science from Karlsruhe University in 1993 and his M.S. and Ph.D. in Computer
Science from U.C. Berkeley in 1995 and 1998. He also worked for several companies including IBM,
Hewlett Packard, and Interval. He is a member of the Stanford Computer Graphics and the Robotics
Laboratory. His research interests are in the areas of Computer Vision, Graphics, and Learning. Currently
he focuses on topics in visual motion capture, human face, speech, and body gesture recognition and
animation, and image based modeling and rendering.

Michael F. Cohen
Senior Researcher
Manager, Graphics Group
Microsoft Research
One Microsoft Way
Redmond WA 98052
(425) 703-0134
(425) 936-0502 Fax
mcohen@microsoft.com
http://www.research.microsoft.com/graphics/cohen/

Dr. Michael F. Cohen, senior researcher and manager of the Microsoft graphics research group, joined
Microsoft Research in 1994 from Princeton University where he was an Assistant Professor of Computer
Science. Dr. Cohen received his Ph.D. in 1992 from the University of Utah. He also holds undergraduate
degrees in Art and Civil Engineering from Beloit College and Rutgers University respectively, and an M.S.
in Computer Graphics from Cornell. Dr. Cohen also served on the Architecture faculty at Cornell
University and was an adjunct faculty member at the University of Utah. His work at the University of
Utah focused on spacetime control for linked figure animation. He is perhaps better known for his work on
the radiosity method for realistic image synthesis as discussed in his recent book "Radiosity and Image
Synthesis" (co-authored by John R. Wallace). Dr. Cohen has published and presented his work
internationally in these areas. At Microsoft, Dr. Cohen has worked on a number of projects, including the
IBMR projects "The Lumigraph" and "Layered Depth Images". He is also involved in the "Virtual
Cinematographer" project to create automatic camera placement and sequencing of shots for interactive
visual experiences, and in adding expressive refinements to the work in linked figure animation. Dr. Cohen
served as the papers chair for SIGGRAPH 98, where he was also awarded the 1998 Computer Graphics
Achievement Award for the development of practical radiosity methods for realistic image synthesis.




                                                                                                        0-2
Paul Debevec
Research Scientist
University of California at Berkeley
387 Soda Hall #1776
Computer Science Division, UC Berkeley
Berkeley, CA 94720-1776
(510) 642-9940
(510) 642-5775 Fax
debevec@cs.berkeley.edu
http://www.cs.berkeley.edu/~debevec/

Paul Debevec earned degrees in Math and Computer Engineering at the University of Michigan in 1992
and completed his Ph.D. at the University of California at Berkeley in 1996, where he is now a research
scientist. Debevec has worked on a number of image-based modeling and rendering projects, beginning in
1991 in deriving a 3D model of a Chevette from photographs for an animation project. Debevec has
collaborated on projects at Interval Research Corporation in Palo Alto that used a variety of image-based
techniques for interactive applications; the "Immersion '94" project done with Michael Naimark and John
Woodfill developed an image-based walkthrough of the Banff national forest and his art installation
"Rouen Revisited" done with Golan Levin showed at the SIGGRAPH 96 art show. His Ph.D. thesis under
Jitendra Malik in collaboration with C.J. Taylor presented an interactive method of modeling architectural
scenes from sparse sets of photographs and for rendering these scenes realistically. Debevec led the
creation of an image-based model of the Berkeley campus for "The Campanile Movie" shown at the
SIGGRAPH 97 Electronic Theater, and directed the animation "Rendering with Natural Light" at the
SIGGRAPH 98 ET which demonstrated image-based lighting from high dynamic range photography. With
Steve Gortler, Debevec organized the course "Image-Based Modeling and Rendering" at SIGGRAPH 98.

Leonard McMillan
Assistant Professor
Massachusetts Institute of Technology
545 Technology Square
Cambridge, MA 02139
 (617) 258-0381
 (617) 253-6652 Fax
mcmillan@graphics.lcs.mit.edu
http://graphics.lcs.mit.edu/~mcmillan/

Leonard McMillan is an assistant professor of Electrical Engineering and Computer Science at the
Massachusetts Institute of Technology. He received B.S. and M.S. degrees in Electrical Engineering from
the Georgia Institute of Technology in 1983 and 1984, and his Ph.D. in computer science in 1997 from the
University of North Carolina at Chapel Hill. His experiences designing digital signal processing hardware
have fueled his interest in making image-based rendering run at interactive speeds. His plenoptic modeling
work from SIGGRAPH'95 demonstrated how the optical flow information derived from panoramic images
could be used to simulate a three-dimensional immersive environments. Leonard is currently exploring new
algorithms and hardware designs for the accelerating image-based rendering methods. He currently teaches
introductory computer graphics and computer architecture and lectures on a wide range of issues related to
image-based rendering.




                                                                                                       0-3
François X. Sillion
Senior Researcher
French National Institute for Computer Science and Control (INRIA)
iMAGIS - GRAVIR/IMAG
B.P. 53, 38041 Grenoble Cedex 9
France
+33 4 76 51 43 54
+33 4 76 63 55 80 Fax
Francois.Sillion@imag.fr
http://www-imagis.imag.fr/~Francois.Sillion/

François Sillion is a senior researcher at the Institute for Research in Computer Science and Control
(INRIA), working in the iMAGIS project in Grenoble, France. He received undergraduate and graduate
degrees (1986) in Physics at the Ecole Normale Supérieure in paris, France, and a PhD in Computer
Science from the University of Paris-XI/Orsay (1989). Dr. Sillion worked for two years as a post-doc at
Cornell's Program of Computer Graphics, before joining France's National Center for Scientific Research
(CNRS), working first in Paris, then in Grenoble (1993). His research interest include the simulation of
illumination for realistic image synthesis (he worked on several extensions to the radiosity method,
including non-diffuse reflection and hierarchical techniques using clusters); progressive rendering
techniques allowing a continuous trade-off between quality and speed for interactive applications; image-
based techniques for the acceleration of rendering; and the application of computer graphics techniques to
the simulation of non-visible radiation (botanical studies and radio waves). Dr. Sillion published, with
Claude Puech, a comprehensive book on radiosity and global illumination, and co-authored several papers
on all the above subjects. In addition to participating in many conference program committees, he is an
associate editor of ACM Transactions on Graphics, serves on the editorial board of Computer Graphics
Forum, and chairs the EUROGRAPHICS working group on rendering, organizing a yearly workshop on
rendering.

Richard Szeliski
Senior Researcher
Microsoft Corporation, Vision Technology Group
One Microsoft Way
Redmond, WA 98052-6399
 (425) 936-4774
 (425) 936-0502 Fax
szeliski@microsoft.com
http://www.research.microsoft.com/research/vision/szeliski/

Richard Szeliski is a Senior Researcher in the Vision Technology Group at Microsoft Research, where he is
pursuing research in 3-D computer vision, video scene analysis, and image-based rendering. His current
focus in on constructing photorealistic 3D scene models from multiple images and video, and on
automatically parsing video for editing and retrieval applications. Dr. Szeliski received a B. Eng. degree in
Honours Electrical Engineering from McGill University, Montreal, in 1979, a M. Appl. Sc. degree in
Electrical Engineering from the University of British Columbia, Vancouver, in 1981, and a Ph. D. degree in
Computer Science from Carnegie Mellon University, Pittsburgh, in 1988. He joined Microsoft Research in
1995. Prior to Microsoft, he worked at Bell-Northern Research, Montreal, at Schlumberger Palo Alto
Research, Palo Alto, at the Artificial Intelligence Center of SRI International, Menlo Park, and at the
Cambridge Research Lab of Digital Equipment Corporation, Cambridge. Dr. Szeliski has published over 60
research papers in computer vision, computer graphics, medical imaging, neural nets, and parallel
numerical algorithms, as well as the book Bayesian Modeling of Uncertainty in Low-Level Vision. He is a
member of the Association of Computing Machinery, the Institute of Electrical and Electronic Engineers,
and Sigma Xi. He was an organizer of the first Workshop on Image-Based Modeling and Rendering, and is
currently an Associate Editor of the IEEE Transactions on Pattern Analysis and Machine Intelligence.




                                                                                                         0-4
                         Course Schedule and Syllabus
                                             Morning


1. 08:30 - 08:50, 20 minutes (Debevec)

Introduction and Overview
  1. What is image-based modeling and rendering (IBMR)
  2. Differences between image-based modeling and rendering and traditional 3D graphics
  3. Why this is a promising area
  4. Some Examples
  5. Advantages and disadvantages
  6. The spectrum of IBMR - from image indexing to 3D scanning

2. 08:50 - 10:00, 70 minutes (Sillion)

Image Formation Fundamentals and Using IBMR to Accelerate Rendering
 1. What is an image?
 2. Simple projective geometry, and the pin-hole camera model
 3. How light interacts with matter
 4. The relationship of global illumination to IBMR
 5. Challenges posed by non-diffuse reflectance
 6. Image caching techniques
 7. Affine sprite warping

Break

3. 10:15 - 11:00, 45 Minutes (Szeliski)

Determining Geometry from Images
 1. Why geometry is useful for image-based rendering
 2. Computer Vision as Inverse Computer Graphics
 3. Notes on camera calibration
 4. Computing depth maps with stereo and multi-baseline stereo
 5. Image correspondence techniques
 6. Structure from Motion
 7. Overview of other methods: Photogrammetric Modeling, 3D Scanning

  Note: Additional material on determining geometry from images is available in the course notes for
  Course #28, 3D Photography. Topics covered in detail include photogrammetric modeling, silhouette-
  based methods, 3D laser scanning, and other active sensing methods.

4. 11:00 - 12:00, 60 Minutes (McMillan)

2D and 3D Image Warping
 1. Image mosaicing and cylindrical panoramic viewing
 2. Explanation of a depth map
 3. Ways to warp an image based on depth
 4. Panoramic image warping
 5. Turning images and depth into a navigable environment

Lunch (12:00 ­ 01:30)



                                                                                                   0-5
                                            Afternoon
5. 01:30 - 02:20, 50 Minutes (Cohen)

LDI and Lightfield / Lumigraph representations
 1. What is an image versus what is a model?
 2. Layered depth images (LDIs)
 3. The plenoptic function
 4. Reduction to 4D
 5. Light field rendering and the Lumigraph
 6. Combining light fields with geometry
   - Silhouette models (Lumigraph)
   - View-dependent texture-mapping (Façade)

6. 02:20 - 03:00, 40 Minutes (Debevec)

Image-Based Lighting
 1. Recovering lighting information from photographs
   - High dynamic range photography / light probes / inverse lighting
 2. Illuminating synthetic objects with real light
 3. Making additions and modifications to image-based models maintaining correct global illumination
 4. Inverse global illumination: recovering material properties of real scenes from photographs
 5. Communicating the sense of brightness using post-processing operations
 6. The Light Stage: illuminating real objects/people with recorded light for compositing

Break

7. 03:15 - 04:05, 50 Minutes (Bregler)

Applications of IBMR in human animation
 1. How IBMR generalizes from 3D navigation to kinematic domains
 2. Facial animation with image-based rendering
 3. Human figure animation with image-based modeling

8. 04:05 - 04:40, 35 Minutes (Debevec)

Applications of IBMR in Art and Cinema
 1. Matte paintings vs. 3D Models in Movies (Gone with the Wind / Star Wars)
 2. The Aspen and San Francisco Movie Map projects (Lippman)
 3. Naimark's "Displacements" - physically projecting images onto geometry
 4. Dayton Taylor's Timetrack system & "jump morphing"
 5. Rouen Revisited (SIGGRAPH 96 art show), Mona Lisa Morph (SIGGRAPH 96)
    Buf Compagnie's Like a Rolling Stone (SIGGRAPH 96),
    Tour into the Picture (SIGGRAPH 97); What Dreams May Come (1998),
    The Matrix (1999); Prince of Egypt (1999)

9. 04:40 - 05:00, 20 Minutes (Everyone)

Questions and Dialog




                                                                                                       0-6
                                    Table of Contents
1. Introduction and Overview
   Notes:       What is Image-based Modeling and Rendering? (Debevec)
   Slides:      Introduction to Image-Based Modeling, Rendering, and Lighting (Debevec)
2. Fundamentals of Image Formation and Re-Use
   Notes:       Fundamentals of image formation and re-use (Sillion)
   Slides:      Fundamentals of image formation and re-use (Sillion)
   Paper:       Rendering With Coherent Layers
                Jed Lengyel and John Snyder, Proc. SIGGRAPH 97
   Paper:       Multi-layered impostors for accelerated rendering
                Xavier Decoret, Gernot Schaufler, François Sillion, and Julie Dorsey, Proc. Eurographics
                1999
   Paper:       A Three Dimensional Image Cache for Virtual Reality
                Gernot Schaufler and Wolfgang Stürzlinger, Proc. Eurographics 1996
3. Determining Geometry from Images
   Slides:      Determining Geometry form Images (Szeliski)
   Paper:       From images to models (and beyond): a personal retrospective
                Richard Szeliski, Proc. Vision Interface 1997
   Paper:       Modeling and Rendering Architecture from Photographs:
                A hybrid geometry- and image-based approach
                Paul E. Debevec Camillo J. Taylor, and Jitendra Malik, Proc. SIGGRAPH 96
  Note: Additional material on determining geometry from images is available in the course notes for
  Course #28, 3D Photography. Topics covered in detail include photogrammetric modeling, silhouette-
  based methods, 3D laser scanning, and other active sensing methods.

4. 2D and 3D Image Warping
   Notes:       Image-Based Rendering using Image Warping (McMillan)
   Notes:       Computing Visibility without Depth (McMillan)
   Slides:      Image-Based Rendering using Image Warping (McMillan)
   Paper:       Plenoptic Modeling
                Leonard McMillan and Gary Bishop, Proc. SIGGRAPH 95
   Paper:       View Morphing
                Steven M. Seitz and Charles R. Dyer, Proc. SIGGRAPH 96




                                                                                                     0-7
5. LDI and Lightfield / Lumigraph representations
   Slides:      The Lumigraph (Cohen)
                Slides prepared by Steven P. Gortler
   Paper:       Layered Depth Images
                Jonathan Shade, Steven Gortler, Li-wei Hey, and Richard Szeliski, Proc. SIGGRAPH 97
   Paper:       Light Field Rendering
                Marc Levoy and Pat Hanrahan, Proc. SIGGRAPH 96
   Paper:       The Lumigraph
                S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, Proc. SIGGRAPH 96
   Paper:       Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping
                Paul Debevec, George Borshukov, and Yizhou Yu, 9th Eurographics Rendering
                Workshop, 1998
6. Image-Based Lighting
   Slides:      Image-Based Lighting (Debevec)
   Paper:       Recovering High Dynamic Range Radiance Maps from Photographs.
                Paul E. Debevec and Jitendra Malik, Proc. SIGGRAPH 97
   Paper:       Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-Based
                Graphics with Global Illumination and High Dynamic Range Photography
                Paul Debevec, Proc. SIGGRAPH 98
7. Applications of IBMR in Human Animation
   Notes:       Video Based Animation Techniques for Human Motion (Bregler)
   Slides:      IBMR Techniques for Animating People (Bregler)
   Paper:       Video Rewrite: Driving Visual Speech with Audio
                Christoph Bregler, Michele Covell, Malcolm Slaney, Proc. SIGGRAPH 97
   Paper:       Synthesizing Realistic Facial Expressions from Photographs
                Frédéric Pighin, Jamie Hecker, Dani Lischinski, Richard Szeliski, and David H. Salesin, Proc.
                SIGGRAPH 98
   Paper:       Making Faces
                Brian Guenter, Cindy Grimm, Daniel Wood, Henrique Malvar, and Fredrick Pighin, Proc.
                SIGGRAPH 98
   Paper:       Video Motion Capture
                Christoph Bregler and Jitendra Malik
8. Applications of IBMR in Art and Cinema
   Slides:      Applications of IBMR in Art and Cinema (Debevec)
   Notes:       Rouen Revisited
                Golan Levin and Paul Debevec




                                                                                                                0-8