TokenHSI

Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization

Liang Pan1,2,    Zeshi Yang3,    Zhiyang Dou2,    Wenjia Wang2,    Buzhen Huang4,
Bo Dai2,5,    Taku Komura2,    Jingbo Wang1,†
1Shanghai AI Laboratory,    2The University of Hong Kong,    3Independent Researcher,
4Southeast University,    5Feeling AI

(†: corresponding author)

CVPR 2025
arXiv Code (Coming Soon)
Your Image

Introducing TokenHSI, a unified model that enables physics-based characters to perform diverse human-scene interaction tasks.



Abstract

Synthesizing diverse and physically plausible Human-Scene Interactions (HSI) is pivotal for both computer animation and embodied AI. Despite encouraging progress, current methods mainly focus on developing separate controllers, each specialized for a specific interaction task. This significantly hinders the ability to tackle a wide variety of challenging HSI tasks that require the integration of multiple skills, e.g., sitting down while carrying an object. To address this issue, we present TokenHSI, a single, unified transformer-based policy capable of multi-skill unification and flexible adaptation. The key insight is to model the humanoid proprioception as a separate shared token and combine it with distinct task tokens via a masking mechanism. Such a unified policy enables effective knowledge sharing across skills, thereby facilitating the multi-task training. Moreover, our policy architecture supports variable length inputs, enabling flexible adaptation of learned skills to new scenarios. By training additional task tokenizers, we can not only modify the geometries of interaction targets but also coordinate multiple skills to address complex tasks. The experiments demonstrate that our approach can significantly improve versatility, adaptability, and extensibility in various HSI tasks.



Pipeline

Your Image

TokenHSI consists of two stages:
(left) foundational skill learning and (right) policy adaptation.



Foundational HSI Skills


TokenHSI excels at seamlessly unifying multiple foundational HSI skills within a single transformer.


Path-following


Sitting

Climbing

Carrying



Challenging New Tasks


The learned skills can be flexibly and efficiently adapted to challenging new tasks through our transformer-based policy adaptation.

(1) Skill Composition

(2) Object Shape Variation

(3) Terrain Shape Variation

(4) Long-horizon Task Completion in A Complex Dynamic Environment

BibTeX

@inproceedings{pan2025tokenhsi,
  author    = {Pan, Liang and Yang, Zeshi and Dou, Zhiyang and Wang, Wenjia and Huang, Buzhen and Dai, Bo and Komura, Taku and Wang, Jingbo},
  title     = {TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenization},
  journal   = {CVPR},
  year      = {2025},
}