Speech and Gesture Interfaces for Multimedia

The working group "Multimedia NRW: The Virtual Knowledge Factory", founded in 1996, is a collaborative effort to bundle research from different areas of human-machine interaction. The SGIM project contributes to this overall goal by developing techniques for communicating with multimedia systems through the detection and interpretation of a user's verbal (speech) and coarse gestural input. The aim is to overcome the physical limitations of common computer displays and input devices; to enable enhanced interaction methods when using large-screen displays (wall projections, workbenches, caves), thus allowing the user to operate an application system in a more natural and convenient manner.

In an experimental application of virtual construction 'users' gestural communication is supported by simple speech input. Object types and locations can be referenced via verbal expressions, e.g. "...this wheel", to identify objects and locations in sole cases where gestural communication is unnatural or reaches technical limitations. Therefore, the gestural data gathered by electromagnetic 6DOF position- and orientation sensors (trackers) s integrated with output from a speech analyser to form multimodal inputs for the multimedia system.


The construction of a mobile platform ("City-Mobile"), based on large-screen projected CAD-models, serves as a shared testbed application which integrates the results achieved by the different research groups. M4I "Multi modal Man Machine Interaction" from the University of Bielefeld, the Institute for Media Communication in St. Augustin and the European Centre for Mechatronic in Aachen concentrate their efforts on the implementation of a demonstration system to evaluate the research results using a real world application. Project cooperation exists with several other institutions. The Graduate Study Program `Task-oriented Communications' as well as the Collaborative Research Center 'Situated Artificial Communicators' (SFB360) of the University of Bielefeld are in-house partners in the project.