Conceptual Processing of Spatial Perspectives and Construction Episodes

CODY ("Concept Dynamics") is a project in the Collaborative Research Centre "Situated Artificial Communicators" (SFB 360) approved by the Deutsche Forschungsgemeinschaft in 1993 and located at the University of Bielefeld. In an experimental setting of mechanical-object assembly, the CODY project is concerned with the development of knowledge representations and inference methods that are able to dynamically conceptualize the situation in the task environment. A central aim is to enable an artificial agent to understand and process natural-language instructions of a human partner. Instructions may build on the current perception of the assembly environment on the one hand, and on the other on the knowledge-based understanding of grouped structures in the developing construct. To this end, a dynamic conceptualization must integrate information not only describing the types of the objects and aggregates involved, but also their changing roles when becoming part of structured assemblies.

One of the main results of the first funding period (1993-1996) is the development of an operational knowledge representation formalism, COAR, by which processes of dynamic conceptualization in sequences of assembly steps can be formally reconstructed. Inferences concern the assertion or retraction of aggregate representations in a dynamic knowledge base, as well as the computation of role changes for individual objects associated herewith. The structural representations integrate situated spatial features and relations, such as position, size, distance, or orthogonality, which are inferred on need from a geometry description of the task environment.

During the second funding period (1996 - 1999) a new type of imaginal representations was developed. These imaginal prototypes are parametric spatial representations of objects and aggregates that capture - at different levels of abstraction - the generic 3D shapes of structured mechanical objects. Furthermore, additional spatial information, such as an object's intrinsic orientation, can be attached to imaginal prototypes and inherited to all geometric instances in the mechanical-object assembly environment. For example, the picture to the left shows the inheritance of the spatial orientation of the imaginal prototype of a bolt to three concrete bolts.

In the third funding period (2000 - 2002) anticipation has been realized as a facilitating feature in human computer interaction. Based on the available knowledge about the object to build, the actual interpretation of e.g. a wooden bar with three holes can be a propeller blade of an aeroplane in one situation, and in another situation such a piece can be part of the tail unit. The interpretation of a single part is always done with respect to other as yet unassembled parts as well as those parts already assembled to form an aggregate.

The fourth funding period (2003 - 2005) deals with another aspect of anticipation. We now work on making construction episodes available for the next level of user instructions. This is intended to cope with instructions like "and now do the same thing on the other side".

An artificial agent, MAX, takes on the role of a situated artificial communicator and interaction takes place in a fully immersive, large-scale virtual reality environment . The evaluation of interaction metaphors as well as the influence of the degree of immersion in virtual reality for various aspects of spacial descriptions are subject to the cooperation with the psycholinguistic part of this project.

project team:
Prof. Dr. Ipke Wachsmuth, Ian Voss, Dr. Stefan Kopp

student workers:
Marcel Richter, Robert Neumann

cooperation with the psycholinguistic part of C1
Prof. Dr. Gert Rickheit, Helmut Flitter

demos:
see: VR Lab Showcase

project publications:

• I. Voss: The Semantic of Episodes in Communication with the Anthropomorphic Interface Agent MAX. In N.J. Nunes & C. Rich (eds.) Proceedings of IUI 2004. pp. 343-345, ACM Press, New York 2004
• I. Voss, I. Wachsmuth: Extending Long-Term Knowledge on the Basis of Episodic Short-Term Knowledge. In F. Schmalhofer & R. Young (eds.) Proceedings of the EuroCogSci03. p. 445, Lawrence Erlbaum Associates, Publishers. London 2003
• T. Pfeiffer, I. Voss & M.E. Latoschik: Resolution of Multimodal Object References using Conceptual Short Term Knowledge. In F. Schmalhofer & R. Young (eds.) Proceedings of the EuroCogSci03. p. 426, Lawrence Erlbaum Associates, Publishers. London 2003
• I. Voss: Conceptualisation of Episodes in Interactive Virtual Construction. In C. Stephanidis (ed.) HCI International 2003 Adjunct Proceedings. pp. 185-186, Crete University Press, Heraklion June 2003
• I. Voss, I. Wachsmuth: Anticipation in a VR-based Anthropomorphic Construction Assistant. In J. Jacko & C. Stephanidis (eds.) Human - Computer Interaction, Theory and Practice (Part I) pp. 1283 - 1287, Lawrence Erlbaum Associates, Publishers. London 2003
• I. Voß: Gedächtnismodelle für die virtuelle Konstruktion in der virtuellen Realität. In C. Vorwerg & I. Voß (eds.) Gedächtnisprozesse in Interaktionen, pp. 41-46, SFB Report 03/02 , Universität Bielefeld. 2002.
• I. Voss: Anticipation in construction dialogues. In J. Vanderdonckt, A. Blandford & A. Derycke (eds.) "Interaction without frontiers", Proceedings of Joint AFIHM-BCS Conference on Human-Computer Interaction IHM-HCI'2001 (Lille, France, Sept. 2001), Vol. II, pp. 189-190, Cépaduès-Editions, Toulouse, 2001.
• C. Bauckhage, H. Rieser, B. Jung und I. Voß: Struktur, Semiotik und Dynamik - ein Überblick über Aspekte und Methoden der Modellierung mechanischer Aggregate. SFB 360 Report 01/01, Universität Bielefeld. Abstract.
• I.Wachsmuth, I. Voss, T. Sowa, M. Latoschik, S. Kopp & B. Jung: Multimodale Interaktion in der Virtuellen Realität. Mensch & Computer (pp. 265-274), Stuttgart: Teubner, 2001.
• M. Hoffhenke & I. Wachsmuth: Object Recognition with Shape Prototypes in a 3D Construction Scenario. In W. Burgard, Th. Christaller & A. B. Cremers (Eds.), KI-99: Advances in Artificial Intelligence, Berlin: Springer (LNAI 1701), 1999, 231-242.
• M. Hoffhenke & I. Wachsmuth: Objektrepräsentation mit imaginalen Prototypen. In KogWis99: Proceedings der 4. Fachtagung der Gesellschaft für Kognitionswissenschaft, Bielefeld, 28. September - 1. Oktober 1999. Sankt Augustin: Infix, 1999, 292-293.
• B. Jung, M. Hoffhenke, B. Lenzmann & I. Wachsmuth: Interaktive Montagesimulation in virtuellen Umgebungen. Erschienen in H. Szczerbicka und T. Uthmann (Hrsg.): Modellierung, Simulation und Künstliche Intelligenz, Delft: SCS Euroean Publishing House, 2000, 193-210.
• K. Kessler, M. Hoffhenke, G. Rickheit & I. Wachsmuth: Dynamische Konzeptverarbeitung mit imaginalen und assoziativen Strukturen. In Kognitionswissenschaft, 8 (3), 1999, 115-122.
• M. Hoffhenke & I. Wachsmuth: Dynamische Konzeptualisierung mit imaginalen Prototypen. SFB 360 Report 97/9, Universität Bielefeld, 1998.
• B. Jung: Wissensverarbeitung für Montageaufgaben in virtuellen und realen Umgebungen. Dissertationen der Künstlichen Intelligenz, Bd. 157, Sankt Augustin: Infix, 1997.
• I. Wachsmuth & B. Jung: Dynamic Conceptualization in a Mechanical-Object Assembly Environment, Artificial Intelligence Review, 10 (3-4) (pp 345-368).
  Reprinted in P. Mc Kevitt (ed.): Integration of Natural Language and Vision Processing (Vol. IV), (pp. 191-214), Dordrecht: Kluwer, 1996.
• B. Jung, M. Hoffhenke, I. Wachsmuth: Virtual Assembly with Construction Kits. Presented at 1997 Design for Manufacturing Conference (DFM'97), September 14-17, 1997, Sacramento, CA. Appeared in Proceedings of the 1998 ASME Design for Engineering Technical Conferences (DECT-DFM '98).
• B. Jung & I. Wachsmuth: Ein wissensbasiertes System für die 3D-computergraphische Montage-Simulation. In D. Ruland (ed.): Verteilte und intelligente CAD-Systeme: Tagungsband CAD '96 (pp. 107-119), Bonn: Ges. für Informatik; Kaiserslautern/Saarbrücken: DFKI, 1996.

internal documents