Activities

No upcoming events

Intelligent Systems Lab Project: Wearable Sonic Feedback

Participants

• Kai Banasiak
• Florian Hofmann
• Eduard Wall

Project Supervisors

• Thomas Hermann
• Sebastian Zehe

Motivation

• In physiotherapeutic movement therapy patients are shown exercises they have to do regularly. It is difficult to do the exercises correctly. Even with the help of visual displays or proprioception some exercises are hard to perform.
• Sonification as a new interface offers a new type of help for patients: it allows to represent the patient's movements in real-time while performing the exercise as non-speech sound. This information can be used regardless of the current head orientation.
• It is sill unclear what sonification methods are best suited to enable patients to react to the acoustic feedback to use the information to adjust their movement. Thus, basic research in sonification of movements is necessary.

Application Scenario

• The user wants to perform a physiotherapeutic exercise consisting of three specific postures.
• The person puts on weson gloves and knee socks and starts the application which we developed in this project and show below.
• First, the available sensors (two sensors on each elbow, wrist, and knee) have to be calibrated.
• The user can then choose the 'recording' mode to get live feedback from actual movement and simultaneously recording.
• While performing the exercise our system determines the current posture and the user can hear a sound which corresponds to the body part that has the biggest error compared to the correct performance.
• The movement can so be adjusted to fit the requirements.
• If the user performs the exercise correctly, he won't hear any sound. So the user will - in order to reach the correct target for the movement - aim at 'silencing' all sounds in sequence.
• Later the recorded practise can be played back for evaluation.

Used Technologies

In our project we use the BRIX sensor system (link auf Sebastians paper und website) for rapid prototyping.

To implement inter-process communication the middleware architecture RSB (RSB documentation) is used.

The RSBag tools (RSBag documentation) provide efficient and flexible recording and replay of RSB events.

Objectives

The project goals are:

• To create an interface between the BRIX platform and the RSB middleware.
• To create a BRIX-based body tracking system.
• To combine these systems with sonifications to sonify a therapeutic exercise.

Description

The following figure shows a sketch of the architecture. Our system consists of two applications: 'Manager' and 'App'.

The WESON Manager is able to connect to three BRIX units the user is equipped with. It receives sensor data and publishes them with RSB. In record mode the application stores the data in a file that can later be played back with use of RSBag.
The WESON App listens to a specific RSB scope thus receives data send by the WESON Manager. This component can plot the incoming data oline or sonify them with pyo, a python sound synthesis tool.
Screenshots of the two applications are depicted in the next figures.


For tracking body motion we developed both gloves and knee socks. Goniometers on wrists, elbows and knees measure the joint angles. The data from these sensors are used for the sonification. The following pictures show the fully equipped arm and knee socks:

Results

In this project we build a prototypical implementation of a body motion tracking system and developed accompanying software. One of the early successes was the production of a low-tech FSR (force sensitive resistor) that was used in the first iteration of the system. Later on these sensors were exchanged by a commercial available strain gauge strip.

Due to technical difficulties with the BRIX platform, particularly concerning the stability of the bluetooth transmission and the batteries, and several subsequent hardware lab sessions, the first complete prototype was finished late in the semester. The project yielded a running system that can be used to track body motion.

Interaction video: The video shows a user trying to execute all three part of the exercise correctly with help of sonification. He switches through the three postures and performs them as well as possible.

Discussion and Conclusion

• A large amount of time was spend uselessly during downtimes of the BRIX system. Getting a bluetooth connection to several BRIX blocks proved to be a game of chance.
• Most of the recording sessions had to be done wired due to low battery life.
• In the end a running system was attained that worked most of the time.

Outlook

This project could have been a lot easier if the BRIX platform would have worked more reliably. Luckily there is a new generation in the pipeline that promises to deal with most of the shortcomings that have bugged us up to this point.

We will continue to work on the WESON project during the next semester. Most probably a different activity will be chosen to be analyzed and optimized.