Talk, Render, Act: Integrating Social Gesture and Digital Face with Synchronized Speech for Conversational Humanoid Robot

Talk, Render, Act

Integrating Social Gesture and Digital Face with Synchronized Speech for Conversational Humanoid Robot

Jin Jiang¹ Kun Li¹ Jiancong Ma¹ Shengcai Liao^1,†

¹United Arab Emirates University (UAEU) ^†Corresponding author

Paper Demo video Dialogues Code (soon)

Talk: Streaming speech
Render: Audio-driven digital face
Act: Semantically planned gesture

Physical Unitree G1, answering in real time.

Overall demo video

Physical-robot experiment first, then the simulation dialogues with every condition we compare.

Method

One offline agent builds the motion vocabulary; three online agents turn each spoken reply into speech, a face, and gesture.

Overview of TRABot: offline motion atom construction, online planning, and real-time embodiment with Talk, Render and Act lanes. — **Overview of TRABot.** The offline Motion Atom Construction Agent converts G1-retargeted BEAT2 motion into an approved catalog of Robot-Ready Semantic Motion Atoms. Online, the Dialogue Agent generates streaming reply audio and a semantic motion request. The Motion Planning Agent composes approved atoms and constructs state-dependent transitions, while the Facial Animation Agent generates facial frames from the same audio stream. Speech, facial animation, and robot body motion are then executed concurrently.

Qualitative results

The highlighted words are what the gesture should correspond to. Play the same turns as video in the section below.

Body-motion keyframes for two dialogue turns under Simple Motion, Random Motion and the full TRABot system. — **Action–speech alignment.** Each row shows a user utterance, the robot’s spoken reply, and body-motion keyframes under Simple Motion, Random Motion, and Full (TRABot). Simple and Random Motion can produce actions weakly related to the highlighted speech; the full system produces actions that reflect its meaning.

Twelve dialogues, five conditions

The same reply audio, replayed under each condition from the user study. All five clips play in sync; the score underneath is the timing the full system actually logged for that reply.

User

Robot

0.0 s

Talk

Render

Act

Time is measured from the first sample of reply audio. Hatched segments are minimum-jerk bridges from the measured pose into each atom; the outlined block is the return to neutral.

What the numbers say

Twenty participants ranked the five conditions on each of the twelve turns, then picked the one they preferred overall.

84.6%

of participant-turn decisions preferred the full system 203 / 240. Random composition from the same catalog, with the same amount of motion, was chosen 13.75% of the time.

User study · mean within-turn rank, 1 = worst, 5 = best

Condition	Naturalness	Expressiveness	Coherence	Gesture–speech	Preferred

End-to-end timing

Speech–body overlap: 84.72%
Turns completed without failure: 100%
Face offset from reply audio: 37.5 ms
Body offset from reply audio: −666.6 ms

Averages over the 12 turns. The negative body offset means the first gesture begins before the first audio sample, as a human preparation phase does.

Beat alignment

Method	Binary BAS ↑	Gaussian BAS ↑
Simple Motion	0.297 ± 0.206	0.210 ± 0.129
Random Motion	0.281 ± 0.197	0.190 ± 0.145
TRABot	0.308 ± 0.141	0.225 ± 0.130

Gesture apexes against emphasised speech intervals, 12 utterances per condition, σ = 0.30 s for the Gaussian variant.

Planner, on 200 fixed replies

Method	Atoms per reply ↑	Unused budget ↓
Random Match	1.451	2.376 s
TRABot	1.705	0.993 s

Same catalog and budget for both. The planner fits 17.5% more atoms into a reply and leaves 58.2% less of the budget unused.

Cite

@article{jiang2026trabot,
  title   = {Talk, Render, Act: Integrating Social Gesture and Digital Face
             with Synchronized Speech for Conversational Humanoid Robot},
  author  = {Jiang, Jin and Li, Kun and Ma, Jiancong and Liao, Shengcai},
  journal = {arXiv preprint arXiv:2610.06153},
  year    = {2026}
}