5. Testing Knowledge¶

5.1. General Approach and Guidelines¶

In general, integration tests are conducted using the following strategy: first of all, functional tests are defined using the FSMT framework. Secondly, these tests are added to the CSRA Test Distribution as mentioned in the Tooling section. Thirdly, so-called “run” jobs must be defined in order to automatically execute the previously defined tests on a CI Server, please also see Tooling Section: CSRA RUN Tests Jobs.

Functional tests can “assess” basal properties such as, is the sub-system starting correctly on localhost, the CI Server in this case. More advanced tests however, may test if the apartment currently provides the correct/required information/sensory data. Therefore, you may also interact with the sensors (hardware) in the apartment by connecting to it (from the CI Server). In these advanced tests you may evaluate “run time” data based on the apartment’s sensory output. See Workflow section.

5.2. Required Tools¶

1. FSMT Testing Framework (Already included in the CSRA-nightly Distribution)

git clone https://opensource.cit-ec.de/git/fsmt && cd fsmt && git checkout 0.18

2. CSRA Integration Tests Distribution

git clone https://opensource.cit-ec.de/git/citk

3. CSRA Test Git Repository (Already included in the Integration Tests Distribution)

git clone https://projects.cit-ec.uni-bielefeld.de/git/lsp-csra.fsmt-experiments.git

4. CSRA RUN Tests Jobs (Already included in the Integration Tests Distribution, see: runnable jobs)

git clone https://opensource.cit-ec.de/git/citk

5.3. Testing Concept Overview¶

Basically, the CSRA Testing Distribution contains two kinds of recipes: a) FSMT test definitions, and b) “runnables”. As soon as the build-generator finishes, the aforementioned build job types are available on the CI Server. While test definitions are installed using CMake (static), runnables are, in contrast, CITK recipes that include a shell fragment which actually invokes FSMT with a given test definition (that has just been installed by CMake), such as:

#!/bin/bash
sleep 10;
export DISPLAY=:0.0;
. ${toolkit.dir}/etc/vdemo_scripts/lsp-csra-base.sh;
$prefix/bin/fsmt -o $WORKSPACE -c ${toolkit.dir}/etc/fsmt-experiments/mixed/audio/audio-io-test-hallway.scxml;"

FSMT tests jobs (runnables) are executed on a dedicated build slave (mex) attached to the CSRA CI server (ayq). This build slave is part of the apartment’s network (spread ring) and may access all data sources/sensors/files that are provided by the apartment. This way tests may either test locally, that is, on the build slave with a local spread configuration for instance, or connect globally to any core machine or service. The tests results are saved in the CI Server’s workspace (per job) and are accesible by all devs.

5.4. Current Testing Infrastructure¶

The current testing infrastucture comprises a) our Jenkins CI server, b) a dedicated testing build slave, and c) the apartment, of course. The testing distribution is generated by executing this build job In case the tests changed, you need to update/install them by executing this job After that, you may trigger any job by executig one of the runnable jobs.

Currently, all integration tests are automatically triggered every morning at 3:15 AM, sending an email report to the CSRA-DEV mailing list.

5.5. Test Integration¶

TODO: Describe physical “Trigger Test” Button in Apartment

TODO: Describe Testing State Machine with Speech output

5.6. Test Templates¶

Please place this information on top of everything else in your .ini file.

[meta]

description = "Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam
 nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed
 diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet
 clita kasd gubergren, no # sea takimata sanctus est Lorem ipsum dolor sit amet.
 Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod
 tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero
 eos et accusam et justo duo dolores et ea # rebum. Stet clita kasd gubergren, no sea
 takimata sanctus est Lorem ipsum dolor sit amet."

input_data = "/rsb/scope/x/y, file_xy, /dev/urandom"

expected_result = "x is greater than y, time is lower than, message of type x
 was received ... Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed
 diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed
 diam voluptua."

5.7. Example Test INI¶

Example Ini

5.8. Overview Application¶

We also have a “constant” visual reminder of how many test are currently passing without actually opening a web-browser and looking at the CI Server. This application is meant to provide a quick overview.

5.9. Ground Truth Data¶

If you previously recorded ground truth data that you might test against in your integration test, files larger than 2MB are to be saved in...

$prefix/share/fsmt-experiments

...where $prefix is the current distribution, e.g., /vol/csra/releases/trusty/lsp-csra-nightly/

5.10. Current Tests¶

These are the currently implemented integration tests: