Unity's privacy-preserving human-centric synthetic data generator
Unity's privacy-preserving human-centric synthetic data generator
PSP-HDRI+ Paper Poster PSP-HDRI+ Demo Video
Salehe Erfanian Ebadi, Saurav Dhakad, Sanjay Vishwakarma, Chunpu Wang, You-Cyuan Jhang,
Maciek Chociej, Adam Crespi, Alex Thaman, Sujoy Ganguly
Unity Technologies
@inproceedings{ebadi2022psphdri,
title={PSP-HDRI+: A Synthetic Dataset Generator for Pre-Training of Human-Centric Computer Vision Models},
author={Erfanian Ebadi, Salehe and Dhakad, Saurav and Vishwakarma, Sanjay and Wang, Chunpu and Jhang, You-Cyuan and
Chociej, Maciek and Crespi, Adam and Thaman, Alex and Ganguly, Sujoy},
booktitle={First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022},
year={2022},
}
Paper macOS and Linux Binaries Demo Video
Salehe Erfanian Ebadi, You-Cyuan Jhang, Alex Zook, Saurav Dhakad,
Adam Crespi, Pete Parisi, Steve Borkman, Jonathan Hogins, Sujoy Ganguly
Unity Technologies
We introduce PeopleSansPeople, a human-centric privacy-preserving synthetic data generator with highly parametrized domain randomization.
PeopleSansPeople contains simulation-ready 3D human assets, a parameterized lighting and camera system, and generates 2D and 3D bounding box, instance and semantic segmentation, and COCO pose labels.
We use naïve ranges for the domain randomization and generate a synthetic dataset with labels.
We provide some guarantees and analysis of human activities, poses, and context diversity on our synthetic data.
We found that pre-training a network using synthetic data and fine-tuning on target real-world data (COCO-person train) resulted in few-shot transfer keypoint AP of 60.37 ± 0.48 (COCO test-dev2017) outperforming models trained with the same real data alone (keypoint AP of 55.80) and pre-trained with ImageNet (keypoint AP of 57.50).
Abstract (click to expand) In recent years, person detection and human pose estimation have made great strides, helped by large-scale labeled datasets. However, these datasets had no guarantees or analysis of human activities, poses, or context diversity. Additionally, privacy, legal, safety, and ethical concerns may limit the ability to collect more human data. An emerging alternative to real-world data that alleviates some of these issues is synthetic data. However, creation of synthetic data generators is incredibly challenging and prevents researchers from exploring their usefulness. Therefore, we release a human-centric synthetic data generator PeopleSansPeople which contains simulation-ready 3D human assets, a parameterized lighting and camera system, and generates 2D and 3D bounding box, instance and semantic segmentation, and COCO pose labels. Using PeopleSansPeople, we performed benchmark synthetic data training using a Detectron2 Keypont R-CNN variant. We found that pre-training a network using synthetic data and fine-tuning on target real-world data (few-shot transfer to limited subsets of COCO-person train) resulted in a keypoint AP of 60.37 ± 0.48 (COCO test-dev2017) outperforming models trained with the same real data alone (keypoint AP of 55.80) and pre-trained with ImageNet (keypoint AP of 57.50). This freely-available data generator should enable a wide range of research into the emerging field of simulation to real transfer learning in the critical area of human-centric computer vision.
PeopleSansPeople executable binaries come with:
- 28 parameterized simulation-ready 3D human assets
- 39 diverse animation clips
- 21,952 unique clothing textures (from 28 albedos, 28 masks, and 28 normals)
- rameterized lighting
- Parameterized camera system
- Natural backgrounds
- Primitive occluders/distractors
A comparison between our benchmark generated data with PeopleSansPeople and the COCO person dataset.
#train#validation#instances (train)#instances w/ kpts (train)
COCO64,1152,693262,465149,813
PeopleSansPeople490,00010,000>3,070,000>2,900,000
PeopleSansPeople produces the following types of labels in COCO format: 2D bounding box, human keypoints, semantic and instance segmentation masks. In addition PeopleSansPeople generates 3D bounding boxes which are provided in Unity's Perception format.
Generated image and corresponding labels: 2D bounding box, human keypoints, semantic and instance segmentation masks in COCO format. 3D bounding box annotations are provided separately in Unity Perception format.
Here we show a comparison of gains obtained from pre-training on our synthetic data and fune-tuning on COCO person class over training from scratch and pre-training with ImageNet. For each dataset size we show the results of the best performing model.
bbox AP (COCO person val2017)
size of real datascratchw/ ImageNetw/ PeopleSansPeopleΔ / scratchΔ / ImageNet
64113.8227.6142.58+28.76+14.97
641137.8242.5349.04+11.22+6.51
3205752.1552.7555.04+2.89+2.29
6411556.7356.0957.44+0.71+1.35
keypoint AP (COCO person val2017)
size of real datascratchw/ ImageNetw/ PeopleSansPeopleΔ / scratchΔ / ImageNet
6417.4723.5146.40+38.93+22.89
641139.4845.9955.21+15.73+9.22
3205758.6860.2863.38+4.70+3.10
6411565.1265.1066.83+1.71+1.73
keypoint AP (COCO test-dev2017)
size of real datascratchw/ ImageNetw/ PeopleSansPeopleΔ / scratchΔ / ImageNet
6416.4021.9044.43+38.03+22.53
641137.3044.2052.70+15.40+8.50
3205755.8057.5060.37+4.57+2.87
6411562.0062.4063.47+1.47+1.07
Top row: our 3D human assets from RenderPeople with their original clothing textures.
Bottom row: using our Shader Graph randomizers we are able to swap out clothing texture albedos, masks, and normals, yielding very diverse-looking textures on the clothing, without needing to swap out the clothing items themselves.
Additional images generated with PeopleSansPeople. Notice the high diversity of lighting, camera perspectives, scene background and occluders, as well as human poses, their proximity to each other and the camera, and the clothing texture variations. Our domain randomization is done here with naïvely-chosen ranges with uniform distributions. It is possible to drastically change the look and the structure of the scenes by varying the randomizer parameters.
@article{ebadi2021peoplesanspeople,
title={PeopleSansPeople: A Synthetic Data Generator for Human-Centric Computer Vision},
author={Erfanian Ebadi, Salehe and Jhang, You-Cyuan and Zook, Alex and Dhakad, Saurav and
Crespi, Adam and Parisi, Pete and Borkman, Steve and Hogins, Jonathan and Ganguly, Sujoy},
year={2021},
eprint={2112.09290},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Unity Environment Template here
macOS and Linux binaries here
PeopleSansPeople is licensed under the Apache License, Version 2.0. See LICENSE for the full license text.
No open issues yet, or sync has not completed.