Can meaningful gestures improve communication?
People use hand movements to add meaning to speech. Most AI systems only add simple beat gestures that follow the rhythm, not meaningful ones. This work asks whether we can generate gestures that carry semantic meaning along with speech.
What they did
They built a system that looks at an image and a spoken sentence, pulls out visual details like shape and symmetry, matches them to the words, and then uses a motion engine to create iconic or pointing gestures that fit the speech, adding natural beat gestures for a smooth mix.
What they found
A user study showed that when the gestures matched the content, participants understood the message more quickly and clearly than when only speech was present.