Activities

No upcoming events

Intelligent Systems Lab Project: Rii - Recipes intelligent and integrated

Participants

• Christoph Goeker
• Kai Mismahl
• Anja Philippsen
• Sebastian Schuhmacher

Supervisors

• Thies Pfeiffer
• Nadine Pfeiffer-Leßmann

Motivation

Existing recipe management systems are only designed to be readable by humans. The ingredient list might probably follow some structure, but instruction steps are normally written in free text without any formatting limits. Programs to search for recipes that meet some constraints in the allowed ingredients therefore already can be build quite easily with general recipes. But it is difficult to handle the instruction steps of such recipes in a computer program.

It would be worthwhile having an exchangeable format that is well suited for humans as well as for machines. This would enable a machine to assist the user by optimizing the recipe before presenting it to the user. This optimization could be done to achieve time limits or to cope with sparse kitchen equipment.

Application Szenario

Tina has a new kitchen which is equipped with a large screen and even a kinect. She would like to cook her favourite dishes using the Rii cooking book application. Therefore she decides to create her recipes in the exchangeable recipe format Rii. To create a recipe that follows the Rii format specification she can simply use the Rii Editor

She starts the Rii Editor at her PC to create her recipes. Like in a normal recipe management system she first types in which ingredients she wants to use. After that instead of just writing an unstructured text about the cooking steps, she fills in every instruction separately in three steps:

• First she chooses the ingredients that will be consumed or used in this instruction.
• She chooses an action command like "cook" or "cut" from a list of predefined actions. If she misses some action she can use the menu to add the particular action to her library. For actions like "cook" that need a duration or a temperature specification the according text fields appear.
• Every instruction yields some result, e.g. sliced carrot or cooked noodles. Tina is free to rename the product as she wishes or to add more products to this instruction.

After the last instruction step Tina saves the recipe and also adds a nice picture.

A few days later Tina and her friend Jutta decide to cook this recipe together. Luckily the Rii cooking application facilitates the possibility of cooking with more than one user. They select "two" as the number of cooks. Now the screen is split. Tina sees her instruction in the first half, Jutta in the second half of the window. Using the kinect they are able to tell the system which cooking steps are finished. If one user finishes a step, the system dynamically decides which instruction is assigned next to this user. In this way, they can divide the cooking tasks in a natural way and they even save time in the cooking process.

Objectives

The goals of this project are stated as follows:

• Using XML scheme a recipe format shall be created. This format covers the most common cooking activities and is well suited to cope with optimization problems.
• To allow the user to create recipes in this format in a comfortable way a user interface is provided for this task.
• To demonstrate the benefits of the implemented system a prototypical cooking application is created that offers the possibility for several persons to cook in parallel.

Description

Recipe format

The recipe format is specified using XML scheme. The application flow of a recipe is only stated indirectly via the dependencies between the instruction steps. E.g. one cannot cook the peeled potatoes before having peeled them. This allows a program to dynamically choose which of the possible instruction steps should be executed next. An additional feature worth to be mentioned is the possibility to mark an intermediate product to decay quite fast denoting that it should be processed further as soon as possible. This is true for beaten egg whites or it might be necessary for warm dishes.

As a wrapper for this format a jar library is provided. Using this library, recipes can be loaded, changed and saved.

Rii Libraries

As ingredients as well as cooking actions should be well defined, these information are encapsulated into Rii libraries. Note that these have to be discriminated from java libraries. Rii libraries are XML files that contain information about ingredients and action names. They can be modified by the user before or while he creates the recipe. For the most common cooking actions and ingredients the RiiStandardLibray already exists. An user could use this or (additionally) he uses his own created libraries. This approach would for example allow food production companies to provide their own specified ingredient lists and to make available recipes that use their products via internet.

Implementation details

This picture shows a general overview over the implementation. The main components are the following:

• The recipe format wrapper is the wrapper class for creating recipes and loading them from and saving them to the file system. It uses rii libraries that contain the definitions for basic ingredients or action that are used in a recipe.
• The recipe editor is the GUI application to create recipes. It uses the format wrapper as a model.
• The cooking application interacts with the optimization module. This gets a recipe as input. The recipe that shall be cooked is selected by the user. Afterwards it is optimized regarding the number of cooks. This optimization is done online and adapts to the users. The cooking application interacts with the kinect interface provided by the project group SKI (Swipe kinect interaction).

Rii-User-Interface

The main interface contains options to create or edit recipes as well as to cook these recipes.

General handling

This screenshot shows the main user interface right after the start of the Rii program. The top buttons are there for creating, editing and cooking a recipe. The last button exits the program. The menu bar contains the menu "Datei" which contains options to return to the main menu, to start creating a new recipe or to load or cook an existing recipe as well as to exit the program. The "Bearbeiten" menu contains the option to edit the rii libraries and to reload the libraries if something has been changed there.

Creating a new recipe

If a new recipe shall be created, first some general information about the recipe can be inserted. This is a name, the number of dishes of the recipe and a description that can help users decide if they want to cook this recipe. Also here is decided which libraries are used. All libraries in the "libraries" folder in the working directory are proposed.

The next step is to insert the ingredients that are used in the recipe. In the left bar it can be chosen which type of ingredient shall be added. The button "weiter" adds all these ingredients to the recipe.

This screenshot shows the User-Interface for editing the individual production steps. On the left (A) the buttons to switch between the steps can be seen. On the top an instruction can be named. (B) A predefined name is proposed using one chosen ingredient and the specified action. Below that it has to be stated which ingredients have to be used. (C) At the right additional comments can be inserted. (D) Below the operation for the instruction has to be chosen. (E) Finally the user is able to name the result of each step and specify if the result has to be processed as soon as possible. (F) With the buttons at the bottom one can go back to change ingredients, add another production step or save the recipe.

Editing a recipe

To edit a recipe the according option can be chosen in the main menu. After selecting the XML file for the recipe in a file chooser the instruction step view is opened, where instructions can be edited or new instructions may be added. Details for this can be found in the section above (Creating a recipe).

Editing rii libraries

This picture shows the editor for rii libraries. The left screenshot shows the available libraries. Using the buttons below the list the ingredients or the actions defined in the selected library can be edited.

In the middle the window for editing the actions is shown. As an action type something like "cook" has to be inserted. Additionally the used equipment can be named, and it can also be told the system that the action takes place on the stove or in the oven. The automatic flag tells the program that this is an action like baking where even a single user could do something in parallel. The other two flags state if the action needs to be told a duration or a temperature.

On the right the editor window for ingredients is opened. For the verbalization, i.e. the creation of floating text from the instructions of a recipe, also genus and plural of an ingredient can be stated.

Cooking a recipe

This screenshot shows the user interface for cooking a recipe. For an interaction with the cooking application that is well suited to the kitchen environment this project cooperated with the project Swipe Kinect Interaction (SKI). This project enables the user to activate the buttons using five different gestures. The kinect interface can be seen on the right. The following actions can be done using the buttons or the kinect. Instead of clicking the buttons also the arrow keys and the enter key for the middle button can be used on the keyboard:

• The user can leaf through the instructions using the left and right buttons or a swiping gesture to the left or the right if the kinect is used.
• To change the active cook the up and down buttons can be used. This equates an up or down movement of the hand for the kinect.
• To confirm the completion of an instruction the button in the middle or as a gesture a forward movement of the user's hand is used.

The left side shows the different instructions for each cook individually. In this screenshot two cooks are participating. While cook 1 is busy cutting the rhubarb, cook 2 already started to prepare the dough. The instruction above is bordered in green. This means that cook 1 is currently the active cook. If cook 2 wants to progress he first has to activate the down button to become the active cook.

Results

The system that has been implemented achieves the goals as stated above. In detail this means:

• The Rii recipe format is ready to be used for many different recipes.
• The created GUI simplifies the creation process of recipes. The user just needs to type in the ingredients at the beginning and can use these as a guideline to create the recipe in simple steps. Nevertheless a field for notes keeps the flexibility to add free text hints to an instruction if necessary.
• The benefits of the format are shown in a cooking application that allows up to three cooks to work in parallel. During the cooking process the single cooking steps are assigned to the participating users, following an underlying optimization for time constraints.

A demonstration is available as Interaction Video (mp4, 5,46 Mb): The video shows a demonstration of the working system.

A Full-HD Version is available here:

Discussion and Conclusion

Discussion of results

The project Rii decided to focus on the recipe format. This includes the creator for recipes as a GUI application, a basic planning component that is able to cope with this recipe format and a cooking application to display the instructions and demonstrate the functionality of the format. Concerning these points the project yielded a format specification that is simple to use but also contains the necessary complexity. To demonstrate the created system this project concentrated on the recipe optimization concerning time constraints.

Other components that could be considered to be useful in connection with this project are a component to manage the storage in the kitchen or one component to enable the user to choose a recipe due to some constraint like using only the ingredients that the user has at home. These aspects were realized by the project groups IK (Intelligente Küche) and CAR (Countertop Assistant for Recipes).

Open issues

A problem that has not been fully solved in this implementation is the verbalization. As the recipes in our format do not use free text, the verbal instructions for the user have to be created by the computer. This is simple in many cases, but as German is a language rich of grammatical exceptions a lexicon would be needed to create completely correct output texts. The proposed names for results of an instruction are currently only named "Result from [instruction name]". Using a lexicon "Result from cooking noodles" could simply be converted into "cooked noodles".

Outlook

The planning component that has been built is limited to the essentials at the moment. It takes account of the chronology of steps and decay flags for products. A timer and notifications would be necessary to reach more effective parallelism. With this it would be possible to better handle cases like starting with an instruction five minutes before the cake base in the oven is ready.

It would be nice to optimize not only depending on time constraints but also with regard to the kitchen equipment. How many stoves can be used, how many pots exist? These information would offer valuable knowledge to check how far a recipe can be parallelized. Unfortunately there are many scenarios that have to be considered even for a simple version. This is e.g. how large the pots are or if two things can be put together into the same oven. Also some dishes scale in time if the number of dishes is multiplied, others would take exactly the same time.

Therefore the created format probably does not cover all aspects of cooking. It is sufficient for the kind of implementation this project has been aiming for, but if the format should describe the cooking process completely one cannot get around going in more detail.

This project demonstrates how a recipe format can offer much more flexibility to a recipe. The possibility to let a computer optimize the recipe and support the user during the cooking process is a promising idea for the future.