Grand Challenges
EVAC
As the prevalence of autonomous interactive agents is growing incredibly fast, it becomes increasingly clear that these virtual agents must not only comprehend and respond to our verbal content but also engage with our emotions, which is crucial for enabling more profound interactions. While recent advancements in AI have significantly improved the automatic recognition and understanding of human speech, challenges persist in accurately identifying and addressing the nuances of human emotions. We assume that an empathic virtual agent should excel in at least three key tasks: i) recognising human’s spontaneous emotional expressions alongside understanding the verbal content, ii) generating appropriate responses in terms of timing and style, and iii) providing insightful feedback while comprehending user responses. In order to accelerate the development of empathic agents, we introduce the first Empathic Virtual Agent Challenge: EVAC. In its inaugural edition, the focus is set on the robust recognition of spontaneous human expressions during interactions with a virtual agent, using the recently introduced THERADIA WoZ dataset. Participants will have to predict the intensity of dimensional or categorical attributes of affect, from audiovisual sequences of human interactions in French with a virtual agent. We encourage the participation from both academics and the industry.
Organisers
ERR
Human-Robot Interaction (HRI) research is currently placing a greater emphasis on the development of autonomous robots that can be deployed in real-world scenarios to understand the implications of integrating such robots in our lives. However, past literature has shown that such autonomous robots are often characterised by making mistakes, for example when the robot interrupted people or when the robot took a very long time to respond. Such robot failures may disrupt the interaction and negatively impact the perception of people towards the robot. To overcome this problem, robots should be able to detect HRI failure.
The ERR@HRI challenge aims at addressing the problem of failure detection in human-robot interaction (HRI) by providing the community with the means to benchmark efforts for mono-modal vs. multi-modal robot failure detection in HRI. Upon participants acceptance of the ERR@HRI terms and condition by signing the End User Licence Agreement (EULA) , we will share with them a dataset that includes multimodal non-verbal feature statistics (i.e., facial, speech, and pose features) of interaction clips where individuals interact with a robotic coach delivering positive psychology exercises, and labels. Audio-video recordings will not be provided due to anonymity and ethical requirements. The feature statistics and labels will be used to train and evaluate the predictive models. The dataset has been annotated as a time-series with the following labels: robot mistake (e.g., interruption or non-responding, (0) absent, (1) present), user awkwardness (e.g., when the coachee feels uncomfortable interacting with the robot without any robot mistakes, (0) absent, (1) present), and interaction ruptures (i.e., either when the user displays some cues of awkwardness towards the robot and/or when the robot makes some mistakes; (0) absent, (1) present).
We invite participants to collaborate in teams to submit their multi-modal ML model for evaluation, which will be benchmarked based on various performance metrics, including accuracy, precision, recall, F1 score, and timing-based metrics in detecting robot failures.
Organisers
- Micol Spitale
Assistant Professor, DEIB, Politecnico di Milano & Visiting Affiliated Researcher, Department of Computer Science and Technology, University of Cambridge - Maria Teresa Parreira
PhD Student at the Information Science Department at Cornell University - Maia Stiber
PhD Student at Johns Hopkins University, USA - Chien-Ming Huang
Assistant Professor at Johns Hopkins University, USA - Wendy Ju
Associate Professor of Information Science at the Jacobs Technion-Cornell Institute at Cornell Tech - Malte Jung
Associate Professor in Information Science at Cornell University - Hatice Gunes
Full Professor of Affective Intelligence & Robotics at the University of Cambridge, United Kingdom
EmotiW 2024
This challenge has been cancelled.
The Tenth Emotion Recognition in the Wild 2024 Grand Challenge consists of a half day event with a focus on affective sensing in unconstrained conditions and an audio-video based news-reader emotion classification sub-challenge and an engagement prediction sub-challenge, which mimic the real-world conditions. The details on the challenge can be accessed at sites.google.com/view/emotiw2024
Organisers
- Abhinav Dhall, Flinders University
- Shreya Ghosh, Curtin University
- Roland Goecke, UNSW
- Tom Gedeon, Curtin University