Virtualized Classroom - the Next Generation e-Learning System

Department of Computer Science, City College of New York /CUNY
Back to Zhigang's Homepage | Computer Science | School of Engineering | CCNY

Research Team

Supported by

    NSF CRI: Center of Perceptual Robotics, Intelligent Sensors and Machines (PRISM) at CCNY
    The CUNY Institute for Software Design and Development (CISDD)
    The CUNY Graduate Research Technology Initiative (GRTI)

Project Goal

A domain of primary importance in the future of web-based technology and digital libraries is distance and electronic education (e-learning).  Multimedia materials from classrooms and seminars are rich sources of information. Today's first generation e-learning systems primarily adopt a “record-and-playback” approach, which does not leverage the processing capabilities (during live capture, after-capture post-processing, and during later user interaction with archived materials) that we believe will underlie the next generation of more automated, flexible, comprehensible, and individualized e-learning systems. The goal of the Virtualized Classroom project is to design novel software systems to address fundamental research problems occurring in different components of a next-generation, processing-enhanced e-learning system for college and graduate courses.  It includes the following key components:
In the Virtualized Classroom project, we aim to create a natural environment that could give students interactive experiences equivalent to or even better than the real classroom, which is enabled by virtual instructor presence (generated by computer vision, intelligent question-answering and virtual reality techniques), user-selectable and fully-indexed visual aids and materials, full communications between students-students, students-instructor and students-virtual instructors (computers), all organized in the Virtualized Classroom environment that are more natural and friendly.

Phase I Progress

In the Phase I project supported by CISDD (for software development) and GRTI (for equipment purchase), we have developed the following basic components for the Virtualized Classroom project: automated data collection, intelligent algorithms and user interface designs.

Sensors - Automated Data Collection

The commonly used classroom /lecture presentation tools are PPT slides, overhead projections, and blackboards /whiteboards.  In order to digitize the classroom/lecture contents into computers, we use a low-cost Mimio digital whiteboards system to substitute the use of the blackboard.  The "sensors" we are using for the classroom setting are PPT slide capturer, Mimio Virtual Ink and Sony video camera with an omnidirectional lens. All the sensors can be easily managed by the instructor/lecturer himself/herself.

(1) PowerPoint Slide Capture
PPT Add-In The PowerPoint Slide Capturer was modified from the Berkeley PPT Recording Add-In through Prof. Parviz  Kermani  from his PINY ELN: e-Learning Lecture Navigator .  We added the start date and  time of the  presentationin in order to synchronize the recorded PPT slides with the accompanying whiteboard pages. In aadditionto the pages of slides in one of the image formats (e.g. Jpeg), a PPT log file is automatically generated with timing information and the titles of all the slides in the presentation.

(2)Whiteboard Capture
Mimio Virtual Ink Everything an instructor writes on a normal whiteboard can be captured by the Mimio Virtual Ink , a hardware and software package for digitizing whiteboards. The whiteboard presentation will be saved in a series of html files, one html file (with the time information of the page) for each whiteboard page (in JPEG). An index.html file is also generated, which will be used to retrieve the white pages.

The Virtual Ink system can also be used as a remote mouse and/or  a remote  keyboard, in that the lecturer could control his/her PPT presentation or even generate the PPT presentation just in front of the whiteboard, without touching the computer after you start the system.

(3) Video / Audio Capture
OmniCam We use an omnidirectional lens developed by RemoteReality with and a DV camera  to capture the entire classroom in 360 degrees - both the instructor and students will be in the field of view. Of course the lecturer could just setup the DV camera to get a normal video capture. Start recording right when you click “OK” button in the PPT Recording Add-In pop-up panel so that the video stream will be synchronized with the slides. Right now the video stream should be saved either in an .avi or in .mpg file.

If the lecturer only needs to save the audio stream, you just need to setup the computer microphone, and in the PPT recording Add-In, simply check the “Audio” check box. A .wav file will be saved, and it is synchronized with the slides.
----------------------------------------------------------------------------
In general we assume that the instructor will use a computer projector to project PPT slides on a whiteboard in his/her class. The handwriting contents written on the whiteboard are on top of the slides projections and will be captured by the Mimio Virtual Ink system. The instructor's video and audio streams will be captured by an omnidirectional camcorder.  However our Virtualized Classroom  authoring tool (creator) and presentation tool (player) also work with PPT slides only presentation, Mimio Virtual Ink whiteboard presentation,  the audio/video only presentation, or the combination of any of them. After the instructor set up the sensors and start the presentation, almost everything is automatically saved for him/her.

Algorithms - Data Processing

The algorithms are still under development. Till now we have developed the following basic algorithms to support our Virtualized Classroom presentation.

(1) Omnidirectional Video Transformation

Circular Image The omnidirectional camera is used to capture in real-time the 360-degree FOV classroom scene. In the video stream, both the instructor and the students can be seen. Currently we only transform an original circular video stream into a cylindrical panoramic video stream for better viewing. In the future, we will study the best way to present the classroom setting using omnidirectional vision technology, for example, instructor tracking,  students tracking (for question-answering and other interaction), integration of video and slide/ whiteboard presentation, and immersive presentation in the 3D Virtualized Classroom.

Panoramic Image

(2) Media Synchronization

The synchronization of PPT slides, whiteboard pages and audio/video streams is enabled by our stream synchronization algorithm using the timing information in the PPT log file and whiteboard log files, and the video/audio stream in Java .There are three issues in synchronize PPT and whiteboard page presentation.  First each PPT slides has a start time in the presentation, while each whiteboard page has a time indicating when the page is done.  Second the number of whiteboard pages will most probably be different from the number of the PPT slides. Sometime the instructor may generate more than one white pages within the presentation of a single slide. Sometime he/she may use the same whiteboard page for several PPT slides. In other occasions, he/she may not generate any whiteboard pages for some slides.  Third, the starts of the corresponding PPT slides and the whiteboard pages may be slightly or largely different. Our algorithm matches up the two timing sequences, taking care of all these problems.

(3) Registration of PPT Slides and Whiteboard Images

PPT slides and the whiteboard system can be used together to better present a lecture in a classroom. When the PPT projection and the whiteboard  are in two separate areas, we do not need to register PPT slide images and the whiteboard images in the Virtualized Classroom presentation.  However when we project the PPT slides on the whiteboard, where we are going to annotate, modify or expend the PPT presentation, we need to geometrically register the images from the two sources. For this purpose, right after he/she starts the classroom presentation, the instructor needs to mark at least the four corners of the PPT projection in the whiteboard (see Figure (a) below). The PPT slide image will be captured by PPT recorder, and the whiteboard image with the four markers will be captured by the Mimio Virtual Ink. Since both of them are images of the same plane, we use a projective transformation to register the two images using at least four point pairs. Usually the slide projector will stay still during the presentation, therefore the rest of the PPT-whiteboard pairs can be integrated using the same transformation parameters (see Figure (b) below for an example).

Calibration PPT _whiteboard

                       (a) Calibration                                                (b) Registration

(4) Registration of PPT Slides and Instructor Video Image

We have done some preliminary research on automatic instructor extraction from the video streams in order to perform content-based video compression and slide-video integration. As a result, the text will be always presented in sharp resolution (using the digital slides), much higher than the standard type of video. The image of the lecturer will be merged with the synthesized slide projection rather than displayed in two different windows.

Slide in video  Augmented Slide

                     (a) A slide in video                                                (b) Digital slide + video image

An example of automatic integration of the real and high-resolution virtual images is shown above. (a). A frame from the original video at low resolution with undesired shadows; (b) The extracted real image of the lecturer and the synthetic image of the high resolution digital slide are merged in one view, giving the user a strong feeling that the lecturer is the Virtualized Classroom. The slide image from video and the synthetic image from digital slide have been aligned so that the lecturer can point to the exact point that he pointed in the slide projection.

Interfaces - Data Presentation

Our Virtualized Classroom (VC) presentation system is designed as both an authoring tool and a presentation interface for different kinds of lectures, and with user-selectable interface. The VC presentation system thus includes two parts - the VC Creator and the VC Player.

(1) Virtualized Classroom Creator
The instructor or a student who uses the system can customize the presentation by using the VC Creator to include different media in windows with user-selected sizes and positions. After an user opens a new Creator page, there will be a floating frame that has checkboxes on it.  Use these checkboxes to add or remove presentation components.  Each time the user clicks on one of the checkboxes a popup window will appear directing him/her to click on a certain file so that the program can load the proper information. 
After the user adds presentation components, the application window will contain frames encompassing the presentation video or slides.  The user can resize and reposition any of the frames (including the ones representing the buttons).  Since the Creator works in a WYSIWYG fashion, the frame window will be larger than the component will be when it is played.  So, even if frames overlap a bit, if the content in the frame is positioned as he/she wants, the presentation will play as he wants. The following two figures show two examples with different contents, number of windows, and sizes/positions of the windows. You can click the figures to see the details.

Once the user has included all of the components in his/her presentation he/she need to save the presentation to a .vcp file.  Go to File and click on save.  Give the file any name and click on save in the popup window. 

PPT+Whiteboard   OmniVdieo+Slides
 (a) Integrated presentation of PPT and whiteboard pages; (b) PPT slides + panoramic video

(2) Virtualized Classroom Player

When you want to play a presentation all you have to do is click open under the Player menu.  It will take a few minutes for the presentation to load and to transform into the right sizes.  Once it has loaded you can click on the “Start” button to start the presentation from the beginning, or, if there are PowerPoint based slides, you can right click to get a slide based Table of Contents (ToC)  to guide the presentation to a specific topic in the presentation. All the media contents are synchronized by using the timing information ( from PPT log file,  whiteboard html file, and video/audio timing information). Using a pop-up ToC, we can make full use of the space in the player.

PPT_Whiteboard  PPT+OminVideo
 (a) Integrated presentation of PPT and whiteboard pages; (b) PPT slides + panoramic video

Demos (Under Construction)

For trying the VC presentation system, you first need to have JMStudio installed in your Windows system. Then download the VCPlayer.jar (a Java Executable for both lecture creation and playing) and a DLL file (for image transformation) and save them in the same folder. Then you can download and unzip each of the the following demo presentations. Make sure materials for each presentation are saved in the same folder.  More detailed information on how to use the system may be found in the CUNY Virtualized Classroom (tm) User's Guide .

The system is still under development. We are integrating the video/image processing capabilities developed in Matlab/C++ into the Java program of the VC Creator and Player. But you could try our current version of the VCPlayer. Stay tuned for updating. 

Phase II Plan

What's next?  The following is a list of what we want to do in Phase II of the project - in order to make to Virtualized Classroom smarter, more attractive and more flexible.

Content-based Multimedia Data Collection and Presentation

The ultimate goal of the project is to create e-lecture presentations that include active objects in various kinds of media - PPT slides, handwritings, video and audio so that every meaningful object in the presentation is active - terms, pictures, instructor images, and students. We intent to create a Virtualized Classroom environment that support interactive learning even though the instructor and students are neither in the same physical space nor presented in the same time. Therefore content-based data collection and presentation is required. In the Phase I version of our program, both the PPT slides and the whiteboard pages are saved as image files (jpeg). This does not allow us to create links in the pages. In the next step, we are going to study ways to enable the implementation of the active object concept. For example we will save PPT slides into HTML files. We will save handwritings on whiteboard in their dynamic forms, and then further with handwriting recognition. We will also extract the subjects (instructor and students) and the relevant events ( e.g. question-answering) from the video sequence.

Interactive  User Interface Design

With the content-based data representation, we will study a new user interface that better supports interactive learning, for example non-linear course navigation, student-instructor remote discussions, intelligent question -answering by machines, etc. We will also perform a feasibility study on the impact of 3D Virtualized Classroom environment on the active learning experience.

Usability Study and Case Study

We will study several typical classroom examples using different instructing technologies. For example, we will apply our VC presentation system to different typed of courses (CS/EE/Arts etc.), to different levels (Undergrad/Graduate), using different  presentation methods (PPT presentation/ whiteboard/ blackboard/speech-only). In different cases, we will study the data collection strategy and user interface design. We will develop algorithms that can automatically collect user information in using our VC Creator and Player.

Real-time Presentation Creation

The work to create a good PPT presentation is tedious, particularly when you need to create some fine drawings and animation. Therefore we will study the feasibility of using Mimio Virtual Ink to generate or to revise the PPT presentation in real time. For doing that,  we need to design better interfaces than just the Mimio mouse for interactive creation of multimedia contents, and to develop algorithms for converting /combining handwriting text /drawings into PPT digital presentations.

Collaborators:

Edward M. Riseman, Professor, Department of Computer Science, UMass-Amherst
Rick Adrion, Professor, Department of Computer Science, UMass-Amherst
Jim Kurose, Professor, Department of Computer Science, UMass-Amherst

Related Piblications
  1. W. Li, H. Tang and Z. Zhu, Vision-Based Projection-Handwriting Integration in Classroom, IEEE International Workshop on Projector-Camera Systems (PROCAMS’06), New York City, June 17, 2006 (in conjunction with IEEE CVPR 2006), accepted.
  2. W. Li, H. Tang, C. McKittrick and Z. Zhu, Classroom Multimedia Integration, accepted to IEEE International Conference on Multimedia & Expo (ICME), Toronto, Canada, July 9-12 2006, poster presentation, acceptance rate 51%
  3. Z. Zhu, C. McKittrick and W. Li, Virtualized Classroom – automated production, media integration and user-customized presentation, The 4th Int. Workshop on Multimedia Data and Document Engineering (with CVPR’04),  July 2, 2004, Washington DC 
  4. W. Li, H. Tang and Z. Zhu, Automated registration of high-resolution images from slide presentations and whiteboard handwritings via a low-cost digital video camera. The 2nd IEEE Workshop on Image and Video Registration (with CVPR’04), July 2, 2004, Washington DC 
  5. W. R. Adrion, J. Kurose, E. Riseman, Z. Zhu, The Multimedia Virtual Classroom – Developing and Evaluating A Constructivist Learning Environment, Fifth International Conference on Information Communication Technologies in Education, July 1-3, 2004, on Samos Island, Greece
Links to Related Work:
PINY ELN: e-Learning Lecture Navigator 

UMass Multimedia Asynchronous Networked Individualized Courseware (MANIC)

Copyright @ Zhigang Zhu, City College of New York, 2003- 2006