AI Technology: Vision (image recognition, image generation, text recognition OCR, etc.); Machine Learning.
This project aims at empowering computers with a deeper understanding of humans interacting with objects by enabling robust 3D and 4D (3D+time) reconstruction of human bodies, hands and objects during interactions (referred to as Human-Object Interactions - HOIs), purely from color camera images and videos.
Current methods for HOI reconstruction struggle with occlusions, low resolution of hands and objects in images, handling of unknown objects, lack of large-scale 3D datasets, inefficient representation of dynamic interactions, and lack of temporal consistency.
STRIPES tackles these challenges through a multi-faceted approach.
The study will develop novel generative 3D morphable shape models for both humans and objects, enabling fitting bodies and objects to images representing HOIs under occlusion featuring possibly unknown objects.
Moving beyond contacts and mesh-based HOI reconstruction methods, we will leverage a richer proximity-aware relative representation of objects and bodies that better captures intricacies of shape and position.
Finally, the research team will tackle spacetime-aware 4D HOI reconstruction, which remains largely underexplored.
The study will develop methods that ensure temporal consistency and physical plausibility, all based on a novel joint representation of body, fingers, and object trajectories.
Key to those advances is the use of common-sense knowledge representations from foundation models for mitigating the ubiquitous lack of 3D data.
The project will deliver open-source models and methods, significantly advancing perception of HOIs for intelligent systems like assistive robots and mixed reality applications.
Dimitrios Tzionas, University of Amsterdam, The Netherlands