Activities

No upcoming events

Intelligent Systems Lab Project: Interactive Digital Attachments

Participants

• Katharina Suhre
• Markus Broelemann
• Max Peters
• Sebastian Rottschäfer
• Tim Pontzen

Associated Members

• Matthias Orlikowski

Supervisors

• Thies Pfeiffer
• Patrick Renner

Motivation

• Nowadays there is almost no interaction between physical and digital working spaces
• Growing influence of digitalisation of working environments require a balancing act of the user
• Documents, most of the time, are either digital or physical
• Sharing physical documents is complicated and time consuming
• Physical documents can not have meaningful, i.e. hierarchical connections to other documents
• Printed documents cannot display multimedia content

The systemInteractive Digital Attachments(INDIGA) aims at overcoming these limitations by using new technology in augmented reality to identify physical documents and link them to digital documents.

First term

Application scenario

Businesses that heavily depend on documents and want to improve their workflow and efficiency are predestined for a system that simplifies sharing of whole document structures (e.g. documents belonging to a certain contract) and enable linking meaningful attachments to a master document.

Specific scenario

A soap manufacturer asks an advertising agency to design an additional landing page for their website which is supposed to target male students. The agency uses INDIGA to manage their documents. One employee has drawn a design draft (physical document with an identifier for the system). He places the document on his table in range of the camera that detects the document. He then attaches it to the requirements document his supervisor sent to him. He then shares the requirements document with a web developer.

Objectives

The project goal is to extend the possibilites of a physical workspace by providing the following capabilities

• Recognize paper documents by marker IDs
• Manage digital attachements
• Link and group documents digitally
• Share documents with colleagues
• Intuitive interaction

Description

Setup

The setup uses a camera for document detection, placed above the user, facing down on the desktop and a camera that detects gestures to navigate through the system's menu. The down facing camera recognizes markers on documents to search a database and display them in the system. The user can then interact with them via gestures to group or link them with other documents or share them with other users. New documents can be added to the database via a second, more elaborate, interface that is operated with a mouse and keyboard.

Assets

• Bielefeld Augmented Reality Tracker (BART) to recognise documents with AR-markers
• JavaFX GUI with SQLite database to manage documents and display camera streams (connection to BART via RSB)
• Intel® RealSense™ SDK and 3D camera for gesture recognition
• Conventional webcam for tracking documents

Results

Working demo system, which includes:

• GUI-Interaction via mouse/keyboard or gesture camera (gesture recognition has stability flaws)
• stable AR-marker based detection of paper documents with reasonable performance
• local multiple-user SQLite database to manage document data
• grouping system to organise and share document data

The video shows a number of basic use cases including adding documents to the system, login for multiple users, display of recognised documents, and attaching and sharing documents via gestures/hand recognition.

Discussion and conclusion

• All planned basic features were successfully implemented:
• Document database structure
• Displaying additional information for recognized documents
• Features for collaborative work (Sharing and linking of documents)
• Menu navigation by gestures
• Managing document groups and attachments
• Advanced features that are not implemented yet:
• Displaying documents (e.g. text documents, images, videos) in the system
• Additional interaction possibilites on attachments of documents
• So far only basic gestures are available, advanced gestures that allow more precise interface handling are still missing

The implemented system is able to represent physical documents in a digital system. By this, it enables the user to work with collaborative concepts like sharing and grouping.
However, after implementing and testing the gesture navigation, we noticed the disadvantages of moving your hand freely in a 3 dimensional space in front of the gesture camera that has no noticable borders. As we also have no possibility of giving haptic feedback, this leads to inaccuracies in interface handling and triggers unintended actions of the system. Moreover the gesture camera and SDK are still in development and resulted in additional handling errors. Hence, we didn't fully achieve our goal of creating an intuitive interface.

Therefore, the focus in the next semester should be mainly on improving the gesture interface. This would also include refactoring the gesture recognition component (currently part of the Java application) into a seperated RSB-connected component.

Outlook

• Implement missing features for the conventional mouse-keyboard interface
• Improve gesture recognition and menu navigation
• Transition from AR-markers to QR-codes with the possibility of identifying a larger quantity of documents

Second term

Conclusion of first term

The results of the system, developed in the first term, revealed two major flaws in the design. The interface proved to be not as intuitive as expected and the digital and physical workspaces were too losely coupled. We took a new approach on implementing the interface and developed in a new direction to reach our intended goals. To prevent repeating our mistakes from the previous term we concentrated on fusing both workspaces from the beginning of development (see here) on.

Changed focus of objectives

• Use QR-codes (will be referred to as QR-markers ¹) for document identification instead of AR-markers for a more general application range
• Intuitive interaction (minimal spacial distance between physical and digital workspace and a design with self-explanatory actions)
• Improve code quality and structure of the underlying model

Description

New interface concept

To couple the two workspaces more tightly, we decided to fuse them by projecting the digital view directly onto the desk and establishing a connection via simultaneous tracking of multiple QR-markers printed on the physical documents. The tracking of all physical documents enables us to project virtual elements (buttons, overlays, etc.) on top of those documents and previously implemented actions like sharing, grouping and attaching with them. By this, we hope to improve the augmented reality experience and reduce the cognitive load of the user by eliminating their need to connect both workspaces in their mind.

Setup

The picture above depicts the currently used system setup. A projector as well as a Kinect system are attached to a tripod. They are rotated towards the table to be able to scan the surface for documents with QR-codes.

Assets

• TouchIO as a pointer input layer [1]
• Kinect for Windows v2, needed for TouchIO and QR-marker recognition
• LG PG60G LED Projector for overlaying the physical workspace with the digital one
• JavaFX GUI with SQLite database to manage documents and display the overlay

Discarded technologies:

• JFxtras (proved to be too unflexible)
• TouchGraph (not avaibable for JavaFX)
• Dasher (not avaibable for JavaFX, development stopped seven years ago)

Results

Working demo system, which includes:

• Projection of buttons and overlays on the desk, relative to paper documents
• Interaction via hands with physical documents and virtual menu
• QR-marker based detection and tracking of paper documents with reasonable performance
• Optimized functionality of model in comparison to last term

The video shows that scanning a QR-marker results in a projection of a virtual interaction menu on top of the documents. The menu includes sharing, grouping, and attaching other documents as functionalities. During any of these actions the documents selected are colour coded - green for selected, red for unselected documents. Due to some issues between gesture tracking in combination with a sheet of paper we were forced to us the mouse as an input.

Discussion and conclusion

• Most of the planned features were successfully implemented:
• Refined model/backend structure
• New interface by fusion of physical and digital workspace
• Projecting virtual interface directly onto the table
• Document-associated menus follow movements of documents
• Direct interaction on the desk (no mapping between information on screen and desk)
• Features that are not implemented yet:
• RSB communication for QR-marker still missing (current implementation with sockets)
• The currently used QR-marker library did not support smaller dimensions of markers so they had to be rather big (can be seen in the video)
• An alternative keyboardless input method proved to be too time consuming for the scope of the project
• Refinement of the document menu (as considered libraries were too unflexible or insufficient, refinement of the interface needs to be done "by hand")

The implemented system is able to track documents via QR-markers and projects virtual menus on them to extend the possibilities of physical documents with actions from the digital world, like grouping and attaching.
However, the search for and evaulation of libraries for displaying and arranging virtual menus and buttons nicely (TouchGraph, JFxtras) consumed alot of time and yielded no usable results. We noticably lacked the time spent on this task at the end of the project so crucial features could not be (satisfactorily) implemented; for example an alternative keyboardless input method. Sadly we could not invest any time into evaluating the new prototype but the subjective impression of all team members concluded that the new design felt much more intuitive and direct, as the digital and physical workspaces were tightly coupled in their visualisation and behaviour.

References

• Neumann, H., Renner, P., & Pfeiffer, T. (2015). Entwicklung und Evaluation eines Kinect v2-basierten Low-Cost Aufprojektionssystems. In A. Hinkenjann, J. Maiero, & R. Blach (Eds.), Berichte aus der Informatik. Virtuelle und Erweiterte Realität - 12. Workshop der GI-Fachgruppe VR/AR (pp. 22–33). Aachen: Shaker Verlag.

---

[1] As the QR-codes serve the additional purpose of locating documents on the table, we refer to them as QR-markers showing their close relationship to AR-markers.