From K‑pop finger hearts to Taylor Swift’s iconic hand hearts, simple hand gestures have become a playful way to show love and affection.
But as they become part of popular culture, how can researchers study them at scale?
“Heart emblems, or gestures that form a heart shape, carry distinct cultural meanings across generations and regions,” says Associate Professor Lauren Gawne. “But unlike words, which we can easily track using digital search methods, gestures can be difficult to study.”
To address this, an interdisciplinary research team combined their expertise in linguistics, AI, computer vision and data science.
The team included Associate Professor Gawne and Dr Judith Bishop (School of Humanities and Social Sciences), with Dr Yang Zhao and Master of Data Science student Joyce Nguyen (School of Computing, Engineering and Mathematical Sciences).
Together, the researchers showed that existing computer vision models can be adapted to recognise heart gestures with only a small amount of additional training data.
Their research resulted in the Heart Gesture Classification Tool, a publicly available tool that uses computer vision to recognise and classify different types of heart gestures.
Associate Professor Gawne says the technology creates new opportunities for gesture studies research.
“This means we can collect large amounts of data on who is using heart hand gestures, how they use them and what they might mean,” she says. “This project also demonstrates how interdisciplinary collaboration can bring new tools and approaches to humanities research.”
The Heart Gesture Classification tool is currently live and will be presented at the upcoming conference of the Alliance of Digital Humanities Organizations in South Korea.
