LAMP: 3D Layered, Adaptive-resolution and Multi-perspective Panorama

Back to Zhigang's Homepage | Computer Science | School of Engineering | CCNY

The goal of this work is to construct a layered and panoramic representation of a large-scale 3D scene with occlusion from translating video sequences. We are actually solving four problems: 1) how to generate seamless panoramic view images (PVIs) and epipolar plane images (EPIs) from video under a more general motion than a pure translation; 2) how to analyze the huge amount of data in EPIs robustly and efficiently to obtain dense depth information; 3) how to enhance resolution and recover occlusions in a PVI representation; and 4) how to represent a large scale 3D scene with occlusions efficiently and compactly.

1.  3D LAMP representation

A compact representation, 3D Layered, Adaptive-resolution and Multi-perspective Panorama (LAMP), is proposed for representing large scale and 3D scenes with occlusion. This work was inspired by the combination of classical Chinese Paintings - scrolls with multiple viewpoints and the western  bas-relief sculpture. Two kinds of 3D LAMP representations are constructed, i.e. the relief-like LAMP and the image-based LAMP, both of which concisely represent almost all the information from a long image sequence. The relief-like LAMP is basically a single extended multi-perspective panoramic view image with both texture and depth values, but each pixel has multiple values to represent results of occlusion recovery and resolution enhancement. The image-based LAMP, on the other hand, consists of a set of multi-perspective layers, each of which has both texture and depth maps, with adaptive time-sampling scales depending on depths of scene points.  Several examples of 3D LAMP construction for real image sequences are given. The 3D LAMP is a concise and powerful representation for image-based rendering.
Relief-like LAMP of the building scene : PVI surface, relief_surface and adaptive-resolution mosaic
 The modeling and rendering of a flower garden scene

2. Image Stabilization and 3D reconstruction

We present a systematic approach for the automatic construction of the 3D panoramic model of a natural scene from video sequences. Despite camera vibration from a platform moving on a common road surface, dense depth maps are derived from image sequences. First, a 3D image stabilization method is proposed which de-couples the vibration from the vehicle's motion so that good EPIs and panoramic view images (PVIs) can be generated. For the second problem, we propose an efficient panoramic EPI analysis (PEPIA) method in which only one scanline of each EPI is processed. The PEPIA combines advantages of PVIs and EPIs and consists of three important steps: locus orientation detection, motion boundary localization, and occlusion/resolution recovery. Since camera calibration, image segmentation, feature extraction and matching are avoided, all the proposed algorithms are fully automatic and rather general.
  • Automatic image stabilization
  • 3D panorama from a video sequence of a building scene
  • PVI (x=0)without video stabilization (JPEG, 124KB)

    PVI (x=0, JPEG 118KB)  and EPI (y=0, JPEG 100K)  with video stabilization
    Panoramic depth map (JPEG 30KB)
    Parallel projection of the 3D panorama (JPEG 30KB)

    Related Publications

    Guangyou Xu, Professor, Department of Computer Science and Technology, Tsinghua University, Beijing
    Xueyin Lin, Professor, Department of Computer Science and Technology, Tsinghua University, Beijing

    Allen R. Hanson, Professor, Department of Computer Science, UMass-Amherst

    Supported by
  • China National Science Foundation Project (No. 69805003), Evolution of Vision Algorithms for the Constructing of Layered Scene Models, 01/99-12/01, Principal Investigator (Zhu).
  • China National High-Tech Project (863 Plan), Image-based VR Scene Modeling and Image Retrieval for Internet/Intranet, 7/1997-6/1998, Co-PI (Zhu).