Remote Hearing: Multimodal Human Signature Detection via PTZ, IR and LDV Sensors
Back to Zhigang's Homepage | Computer Science | School of Engineering | CCNY
Laser Doppler vibrometer (LDV) is a non-contact, remote and high resolution voice detector. Vibration of the objects caused by voice reflects the voice itself. After the enhancement with a Gaussian bandpass filtering and an adaptive volume scaling, the LDV voice signals were mostly intelligible from targets without retro-reflective finishes at short or medium distances (< 100m). By using retro-reflective tapes, the distance could be as far as 300 meters. Infrared (IR) imaging for target selection and localization was also discussed for LDV listening. A system has been set up with three types of sensors (IR cameras, PTZ color cameras and LDVs) for performing integration of multimedia sensors in human signature detection. The basic idea is to provide an advanced augmented interface in order to give users the best cognitive understanding of the environment, the sensors and the events.

The research challenge is, without retro-reflective tape treatment, the LDV voice signals were still very noisy from targets at medium and large distances. Therefore, with the state-of-the-art sensor technology, more advanced signal enhancement techniques are needed. Further sensor improvement is also necessary. In addition, automatic targeting and intelligent refocusing is a technical issue that deserves research attention for long range LDV listening.

Hear some voice clips captured by the LDV sensor!

Related Publications and Links
  1. W. Li, M. Liu, Z. Zhu and T. S. Huang, LDV Remote Voice Acquisition and Enhancement, International Conference on Pattern Recognition (ICPR’06), Hong Kong, China, August 2006
  2. W. Li, Z. Zhu and G. Wolberg, Remote Voice Acquisition in Multimodal Surveillance, accepted to IEEE International Conference on Multimedia & Expo (ICME), Toronto, Canada, July 9-12 2006, oral presentation, acceptance rate 22%
  3. Z. Zhu, W. Li and G. Wolberg, Integrating LDV audio and IR video for remote multimodal surveillance, The 2nd Joint IEEE International Workshop on Object Tracking and Classification in and Beyond the Visible Spectrum (OTCBVS'05), San Diego, CA, US,  Monday June 20, 2005
  4. Z. Zhu, W. Li, Integration of laser vibrometer and infrared video for multimedia surveillance display, TR-2005006, Computer Science Department, Graduate Center, City University of New York, April 2005.  (more voice clips are availble by click the above link)


Collaborators:

Students:

Weihong Li,  Department of Computer Science, The CUNY Graduate Center

Related Grants