Unified Multimodal Interaction Control for Human-Object Interaction Controllable Generation
Yichao MA , Guohui LI , Mingjie MA , Zhong Yang
Text-to-Image (T2I) diffusion models have achieved significant progress in spatial comprehension and controllable generation. However, generating semantically accurate human-object interactions remains a major challenge, especially under diverse layout conditions. Existing approaches that incorporate interaction conditions often suffer from semantic confusion and fail to generalize correctly across varying spatial configurations. To address these limitations, we construct the Human-Object Single Interaction (HOSI) dataset, which supports supervised training for fine-grained interaction semantic control. Building on this, we propose the Unified Multimodal Interaction Control (UMIC) framework, which establishes a unified representation space through a multimodal encoder. Within this space, the mapping between spatial layouts and interaction semantics is explicitly learned, thereby enhancing both semantic precision and generalization ability of interaction conditions. Furthermore, we introduce an interaction-aware self-attention mechanism trained on the proposed semantic conditions to acquire interaction controllability. To enable a comprehensive and systematic evaluation, we design the HOICG Benchmark based on HOSI, encompassing a diverse set of object layouts to assess both interaction controllability and spatial generalization performance.
multimodal interaction control / text-to-image diffusion model / human object interaction image generation
Higher Education Press 2026
/
| 〈 |
|
〉 |
PDF
(13460KB)
