The goal of this work is to construct a layered and panoramic representation of a large-scale 3D scene with occlusion from translating video sequences. We are actually solving four problems: 1) how to generate seamless panoramic view images (PVIs) and epipolar plane images (EPIs) from video under a more general motion than a pure translation; 2) how to analyze the huge amount of data in EPIs robustly and efficiently to obtain dense depth information; 3) how to enhance resolution and recover occlusions in a PVI representation; and 4) how to represent a large scale 3D scene with occlusions efficiently and compactly.

1.  3D LAMP representation

A compact representation, 3D Layered, Adaptive-resolution and Multi-perspective Panorama (LAMP), is proposed for representing large scale and 3D scenes with occlusion. This work was inspired by the combination of classical Chinese Paintings - scrolls with multiple viewpoints and the western  bas-relief sculpture. Two kinds of 3D LAMP representations are constructed, i.e. the relief-like LAMP and the image-based LAMP, both of which concisely represent almost all the information from a long image sequence. The relief-like LAMP is basically a single extended multi-perspective panoramic view image with both texture and depth values, but each pixel has multiple values to represent results of occlusion recovery and resolution enhancement. The image-based LAMP, on the other hand, consists of a set of multi-perspective layers, each of which has both texture and depth maps, with adaptive time-sampling scales depending on depths of scene points.  Several examples of 3D LAMP construction for real image sequences are given. The 3D LAMP is a concise and powerful representation for image-based rendering.

• Relief-like LAMP representation

 

 
The name “relief-like” comes from the fact that the appearance of the front surface (relief surface) of this representation is somewhat like a relief sculpture, in which forms and figures (with image velocities v > 1) are distinguished from a surrounding plane surface (PVI surface).
 
The  2D "PVI surface" is the supporting (back) surface of the base layer in the relief-like LAMP, which makes the LAMP representation very efficient for archives (modeling) and retrieval (rendering).
 
The internal data of the base layer is displayed here as the sequential head-tail arrangement of all the multi-pixel segments of the base layer in each row. It can be found that both the texture and the depth map show almost seamless connections between successive segments, and it is obvious that higher resolutions than in the corresponding PVI are recovered for nearer objects.  We can also generate a Multi-viewpoint panoramic mosaic (JPEG 226kB) with varying time scales, which are selected  according to the dominant depth along each vertical line in the PVI
 
The internal data of the relief-like LAMP, including all the occluding layers, is displayed here the similar head-tail arrangements of all the  multi-pixel segments in all the layers, where the multi-pixel segments in the occluded layers are inserted into the locations of their corresponding depth boundaries (shown as red dots in the figure)
 

• Image-based LAMP representation
 
===> 

An Original video sequence (MPEG, 567KB) is first converted into an extended panoramic image with depth information, then it is further segmented as a image-based LAMP representation with multiple layers - background layer and foreground layer for for flower garden sequence.
The motivation for layering is to represent occluded regions and the different spatial resolutions of objects with different depth ranges; meanwhile the model is represented in the form of a seamless multi-perspective mosaic with viewpoints spanning a large distance. Synthetic sequences is shown here (MPEG, 4.21MB) with a virtual camera of 6 DOF motion generated from the LAMP model (with and without the object layer) of the flower garden sequence
 

2. Image Stabilization and 3D reconstruction

We present a systematic approach for the automatic construction of the 3D panoramic model of a natural scene from video sequences. Despite camera vibration from a platform moving on a common road surface, dense depth maps are derived from image sequences. First, a 3D image stabilization method is proposed which de-couples the vibration from the vehicle's motion so that good EPIs and panoramic view images (PVIs) can be generated. For the second problem, we propose an efficient panoramic EPI analysis (PEPIA) method in which only one scanline of each EPI is processed. The PEPIA combines advantages of PVIs and EPIs and consists of three important steps: locus orientation detection, motion boundary localization, and occlusion/resolution recovery. Since camera calibration, image segmentation, feature extraction and matching are avoided, all the proposed algorithms are fully automatic and rather general.

Automatic image stabilization

MPEG sequence of the original and the stabilized video (1.39MB)

3D panorama from a video sequence of a building scene

PVI (x=0)without video stabilization (JPEG, 124KB)

PVI (x=0, JPEG 118KB)  and EPI (y=0, JPEG 100K)  with video stabilization

Panoramic depth map (JPEG 30KB)

Parallel projection of the 3D panorama (JPEG 30KB)

Related Publications

• Z. Zhu, G. Xu and X. Lin, Efficient Fourier-Based Approach for Detecting Orientations and Occlusions in Epipolar Plane Images for 3D Scene Modeling, International Journal of Computer Vision, 61 (3): 1-26, February - March, 2005 (IJCV online link)
• Z. Zhu, and A. R. Hanson, LAMP: 3D Layered, Adaptive-resolution and Multi-perspective Panorama - a New Scene Representation, Computer Vision and Image Understanding, Special Issue on Model-based and Image-based 3D Scene Representation for Interactive Visualization, 2004, Volume 96, Issue 3 , December 2004, Pages 294-326 (CVIU online link)
• Z. Zhu, Full View Spatio-Temporal Visual Navigation - Imaging, Modeling and Representation of Real Scenes, China Higher Education Press, December 2001, First Hundred National Excellent Doctorate Dissertations Series.
•  Z. Zhu and A. R. Hanson, 3D LAMP:  a New Layered Panoramic Representation, The Eighth IEEE  International Conference on Computer Vision, Vancouver, Canada, July 2001, vol II, 723-730 (MPEG movies: Input => Rendering ).
• Zhigang Zhu, Guangyou Xu, Xueyin Lin, Panoramic EPI Generation and Analysis of Video from a Moving Platform with Vibration, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 23-25 June, 1999, Fort Collins, Colorado. (cvpr99.ps.gz: 2.8 MB )
• Zhigang Zhu, Guangyou Xu, Xueyin Lin, "Constructing 3D natural scene from video sequences with vibrating motions," 1998 IEEE Virtual Reality Annual International Symposium (VRAIS-98), March 14-18,1998, pp 105-112
•  Zhigang Zhu, Guangyou Xu, Yudong Yang, Jesse S. Jin, Camera stabilization based on 2.5D motion estimation and inertial motion filtering, IEEE Internaltional Conference on Intelligent Vehicles, Oct 28-30, 1998, Stuttgart, Germany. (iv98.ps.gz: 244 KB)  ( MPEG Demo: 1.3 M)
  
• Zhigang Zhu, Guangyou Xu, Xueyin Lin, Panoramic epipolar plane analysis for 3D scene modeling, Journal of Software, vol 10, no 5, May 1999, pp. 489-495 (in Chinese)
• Zhigang Zhu, Guangyou Xu, Xueyin Lin, Kinetic occlusion modeling and motion orientation detection in epipolar plane images , Journal of Computers, vol 22, no 3, Mar 1999, pp 283-289 (in Chinese)
• Zhigang Zhu, Guangyou Xu, Xueyin Lin, Dingji Shi, Constructing 3D panoramic model of natural scene from real images, Journal of Tsinghua University , vol 38, no s1, April 1997, pp 84-88 (in Chinese)

Guangyou Xu, Professor, Department of Computer Science and Technology, Tsinghua University, Beijing
Xueyin Lin, Professor, Department of Computer Science and Technology, Tsinghua University, Beijing

Allen R. Hanson, Professor, Department of Computer Science, UMass-Amherst

China National Science Foundation Project (No. 69805003), Evolution of Vision Algorithms for the Constructing of Layered Scene Models, 01/99-12/01, Principal Investigator (Zhu).

China National High-Tech Project (863 Plan), Image-based VR Scene Modeling and Image Retrieval for Internet/Intranet, 7/1997-6/1998, Co-PI (Zhu).