1.15. RGBD Grabber

The RGB-D Grabbing and streaming is done using a combination of OpenCV and Gstreamer in the Person Tracking components. This captures images from an ASUS Xtion Pro Live USB camera mounted at the apartment’s ceiling. For each camera two video streams and an audio stream are created and provided via the rtp protocol.

1.15.2. Depth image encoding

Depth images cannot be directly encoded in h264 so some transformations need to be done by the person-tracking before the images are passed into the gstreamer pipeline. The distances in millimetres are cut off at 4096mm and scaled by 1/16 to fit into an 8bit image.

In c++ code:

cv::Mat input_depth = grab_depth_image_from_camera();
cv::Mat transformed_depth;
// scaling distance from [0,4096) mm to [0,256) (factor = 1/16) everything >= 4096 is cut off
imput_depth.mat.convertTo(transformed_depth, CV_8UC1, 1.0 / 16);
// mask all cut-off values to zero.
transformed_depth.setTo(0,transformed_depth == 255);
cv_video_writer.write(transformed_depth);

1.15.3. Cameras and hosts

A RGB-D device (ASUS Xtion Pro Live) is connected to one camera host (Intel NUC barebone) at a time. Below the list of installed camera hosts:

Camera location Host Stat./Info UDP Group Color Port Depth Port Audio Port
Hallway (CAM1) dme available 224.2.1.1 5200 5250 5300
Hallway (CAM2) iom available 224.2.1.1 5201 5251 5301
Kitchen (CAM1) mpc available 224.2.1.1 5202 5252 5302
Kitchen (CAM2) lxr available 224.2.1.1 5203 5253 5303
Living (CAM1) bfe available 224.2.1.1 5204 5254 5304
Living (CAM2) bos available 224.2.1.1 5205 5255 5305
Living (CAM3) adl available 224.2.1.1 5206 5256 5306
Living (CAM4) bkk available 224.2.1.1 5207 5257 5307
Sports (CAM1) dxb !disabled!        
Sports (CAM2) atl !disabled!        
Sports (CAM3) pug !disabled!        
Sports (CAM4) bcn !disabled!        

1.15.4. Accessing the streams

A single rgb stream can be accessed via gstreamer with the following pipeline:

gst-launch-1.0 \
    udpsrc do-timestamp=true address=224.2.1.1 port=5200 \
  ! application/x-rtp ! queue ! rtph264depay ! h264parse ! avdec_h264 \
  ! videoconvert ! queue ! xvimagesink sync=false

The same pipeline can be used with OpenCV with Gstreamer support. Just drop the gst-launch-1.0, switch xvimagesink sync=false for ‘appsink’ and pass the whole thing as parameter to a cv::CaptureDevice.

Visualisations of the depth and color streams can be started with the scripts:

# shows color video streams from ceiling and webcams in a matrix
gst-show-csra.sh -m -r

# shows depth video streams from ceiling
gst-show-csra-depthcams.sh

1.15.5. Examples

Starting the Person-Tracking & Grabber component:

This example is the configuration used in the csra. The images can be flipped and get an overlay with the current time. The encoding is done with hardware support if available. As a fall back, software encoding is used. Colour and depth streams are encoded as h264 video streams. Audio is grabbed too and encoded in ac3. The streams are published separately via rtp using udp.

PORT=5000
FLIP=0
CLOCKOVERLAY=" ! clockoverlay \
  time-format=\"%Y-%m-%d %H:%M:%S %Z\" \
  halignment=left valignment=top \
  xpad=0 ypad=0 auto-resize=true \
  shading-value=255 shaded-background=true \
  font-desc=r\"Sans,\ 8\" draw-shadow=false \
  draw-outline=false "

# need the right display for hardware encoding
export DISPLAY=:0
# test if vaapi is supported
if gst-launch-1.0 videotestsrc num-buffers=1 ! vaapipostproc ! vaapih264enc ! fakesink; then
  CONVERT=" vaapipostproc ! vaapih264enc cabac=true tune=high-compression "
else
  CONVERT=" x264enc speed-preset=veryfast tune=zerolatency "
fi

# find PrimeSense audio device
AUDIO_DEVICE=$(arecord -l | grep "PrimeSense" | sed -r "s/^card[ ]([0-9]).*/\1/g")
AUDIO="hw:${AUDIO_DEVICE},0"
# set audio capture volume
amixer -c$AUDIO_DEVICE set 'Mic',0 50%
amixer -c$AUDIO_DEVICE set 'Mic',1 50%

export GST_DEBUG=4
rsb_monitor rsb-person-tracking -o $PH_SCOPE -c $CAM_ID -f Home \
    -r "appsrc ! queue ! videoconvert ! videoflip method=$FLIP $CLOCKOVERLAY \
               ! $CONVERT ! queue ! h264parse ! rtph264pay \
               ! udpsink host=224.2.1.1 port=$PORT auto-multicast=true \
               alsasrc device=\"$AUDIO\" ! audioconvert ! avenc_ac3 ! ac3parse \
               ! rtpac3pay  ! udpsink host=224.2.1.1 port=$(($PORT+100)) auto-multicast=true" \
    -D "appsrc ! queue ! videoconvert ! videoflip method=$FLIP $CLOCKOVERLAY \
               ! $CONVERT ! queue ! h264parse ! rtph264pay \
               ! udpsink host=224.2.1.1 port=$(($PORT + 50)) auto-multicast=true" \
  rsb-depth-sensors-grabber -s "PARAMETER"