spencer_people_tracking

SPENCER Multi-Modal People Detection & Tracking Framework    

Multi-modal ROS-based people and group detection & tracking framework for mobile robots developed within the context of the EU FP7 project SPENCER.

Features at a glance

• Multi-modal detection: Multiple RGB-D & 2D laser detectors in one common framework.
• People tracking: Efficient tracker based upon nearest-neighbor data association.
• Social relations: Estimate spatial relations between people via coherent motion indicators.
• Group tracking: Detection and tracking of groups of people based upon their social relations.
• Robustness: Various extensions such as IMM, track initiation logic and high-recall detector input make the people tracker work relatively robustly even in very dynamic environments.
• Real-time: Runs at 20-30 Hz on gaming laptops, tracker itself requires only ~10% of 1 CPU core.
• Extensible and reusable: Well-structured ROS message types and clearly defined interfaces make it easy to integrate custom detection and tracking components.
• Powerful visualization: A series of reusable RViz plugins that can be configured via mouse click, plus scripts for generating animated (2D) SVG files.
• Evaluation tools: Metrics (CLEAR-MOT, OSPA) for evaluation of tracking performance.
• ROS integration: All components are fully integrated with ROS and written in C++ or Python. ROS Kinetic, Melodic and Noetic are supported.

Motivation

The aim of the EU FP7 research project SPENCER was to develop algorithms for service robots that can guide groups of people through highly dynamic and crowded pedestrian environments, such as airports or shopping malls, while behaving in a socially compliant manner by e.g. not crossing in between families or couples. Exemplary situations that such a robot could encounter are visualized in below image on the right. To this end, robust and computationally efficient components for the perception of humans in the robot's surroundings needed to be developed.

  

Architecture

The following figure shows the real-time people and group detection and tracking pipeline developed in the context of the SPENCER project:

The entire communication between different stages of our pipeline occurs via ROS messages which encourage reuse of our components in custom setups. The modular architecture allows for easy interchangeability of individual components in all stages of the pipeline.

Overview of available components

Message definitions

We provide a set of reusable ROS message type definitions, which we have successfully applied across various people detection and tracking scenarios, over different sensor modalities and tracking approaches. Most relevant messages can be found inside the spencer_tracking_msgs package.

We highly encourage reuse of these messages to benefit from our rich infrastructure of detection, tracking, filtering and visualization components! Existing detection and tracking algorithms can often easily be integrated by publishing additional messages in our format, or by writing a simple C++ or Python node that converts between message formats.

People detection

We have integrated the following person detection modules:

• A reimplementation of a boosted 2D laser segment classifier, based upon the method by Arras et al. [3]
• An RGB-D upper-body detector described more closely in [2], which slides a normalized depth template over ROIs in the depth image
• A monocular-vision full-body HOG detector (groundHOG) [2], which based upon a given ground plane estimate determines the image corridor in which pedestrians can be expected. This detector is GPU-accelerated using CUDA. The contained cudaHOG library requires manual compilation and a recent CUDA SDK as well as an nVidia graphics card.
• An RGB-D detector from PCL, which extracts candidate ROIs on a groundplane and then applies a linear HOG classifier [4]

Further external detectors which output geometry_msgs/PoseArray or people_msgs/PositionMeasurementArray messages can easily be integrated into our framework using the scripts from this package. Examples of such detectors include:

• The laser-based leg detector from wg-perception, which might work better than our own laser detector if the sensor is located very close to the ground. See our wrapper package and leg_detectors.launch (replaces laser_detectors.launch).
• The deep learning-based DROW and DR-SPAAM person detectors for 2D laser range data by Beyer et al. [5] and Jia et al. [6], which fuse information from multiple successive frames.

Multi-modal detection and fusion

For detection-to-detection fusion, we have implemented a series of nodelets which can be used to flexibly compose a fusion pipeline by means of roslaunch XML files. Details can be found in the spencer_detected_person_association package. The following figure shows an example configuration which was used during experiments in SPENCER:

In case of detection-to-track fusion (currently not implemented), it is still advisable to publish a CompositeDetectedPerson message (via CompositeDetectedPersons) for each set of detections associated with a track, such that later on it is possible to go back to the original detections from a track, and lookup associated image bounding boxes etc. via the associated detection_id.

Person and group tracking

For person and group tracking, we currently provide exemplary code based upon a nearest-neighbor standard filter data association, which is robust enough in most use cases (especially if multi-modal detectors are being used). The people tracker has been enhanced with a track initiation logic and 4 different IMM-based motion models (constant velocity with low process noise, high process noise, coordinated turn and Brownian motion) to make tracking more robust.

The group tracker relies on social/spatial relations determined via the same coherent motion indicator features as described in [1].

Internally, we have already integrated more advanced methods, including a track-oriented multi-hypothesis person tracker [2], and a hypothesis-oriented multi-model multi-hypothesis person and group tracker [1]. These components use exactly the same ROS message definitions, however, they are not yet publicly available. The components available here were originally implemented as baseline methods for comparison.

Filtering of tracked persons & tracking metrics

The spencer_tracking_utils package contains a number of standalone ROS nodes that can filter an incoming set of TrackedPerson messages based upon different criteria, e.g. distance to the sensor/robot, visually confirmed tracks only, etc.

In spencer_tracking_metrics, we have wrapped publicly available implementations of different tracking metrics, such as CLEAR-MOT and OSPA, such that they are compatible with our message definitions. These are useful for evaluating tracking performance for a given groundtruth.

Import of old annotated logfiles

The srl_tracking_logfile_import package provides a Python script for importing old 2D laserscan logfiles in CARMEN format that have been annotated with groundtruth person tracks, such as these datasets.

Visualization

The srl_tracking_exporter package contains a useful Python script for rendering track trajectories, detections and robot odometry from a 2D top-down perspective as scalable vector graphics (SVGs). These can optionally be animated to visualize the evolution of one or multiple tracks over time.

One major highlight of our framework is a reusable and highly configurable set of custom RViz plugins for the visualization of:

• Detected persons
• Tracked persons (including occlusion state, associated detection ID, and covariance ellipses)
• Social relations
• Tracked groups

As an example, some features of the tracked persons display are:

• Different visual styles: 3D bounding box, cylinder, animated human mesh
• Coloring: 6 different color palettes
• Display of velocity arrows
• Visualization of the 99% covariance ellipse
• Display of track IDs, status (matched, occluded), associated detection IDs
• Configurable reduction of opacity when a track is occluded
• Track history (trajectory) display as dots or lines
• Configurable font sizes and line widths

All of the following screenshots have been generated using these plugins.

Example screenshots of our system in action

The following screenshots show our system in action, while playing back recorded data from a crowded airport environment:

Multi-modal people detection. In orange: 2D laser [3], cyan: upper-body RGB-D [2], yellow: monocular vision HOG [2], grey: fused detections (when using detection-to-detection fusion).

People tracking. In red: tracks visually confirmed by image-based detector

Group tracking via coherent motion indicator features, as described in [1]

Demo videos

Videos of the people detection and tracking system in action can be found on the SPENCER YouTube Channel:

• Real-Time Multi-Modal People Tracking in a Crowded Airport Environment (RGB-D and 2D laser)
• Single Person Guidance Scenario Prototype (2D laser only)
• Group Guidance Scenario Prototype (2D laser only)

Runtime performance

On the SPENCER robot platform, which is equipped with front and rear RGB-D sensors (Asus Xtion Pro Live) and two SICK LMS500 laser scanners, we distributed the people and group detection & tracking system over two high-end gaming laptops (Intel Core i7-4700MQ, nVidia GeForce 765M). The detectors for the frontal sensors were executed on one laptop along with the detection-fusion pipeline. The detectors for the rear-facing sensors and the people and group tracking modules were executed on the second laptop. Both laptops were connected with each other and the rest of the platform via gigabit ethernet.

With this configuration, the components run in real-time at 20-25 Hz (with visualization off-loaded to a separate computer), even in crowded environments where more than 30 persons are concurrently visible.

Installation from L-CAS package repository

A packaged version of the entire framework for ROS Kinetic (Ubuntu 16.04 Xenial) and ROS Melodic (Ubuntu 18.04 Bionic) is kindly provided by the Lincoln Res