ICMI 2017- Proceedings of the 19th ACM International Conference on Multimodal Interaction

ICMI 2017- Proceedings of the 19th ACM International Conference on Multimodal Interaction

Full Citation in the ACM Digital Library

SESSION: Invited Talks

Gastrophysics: using technology to enhance the experience of food and drink (keynote)

  • Charles Spence

Collaborative robots: from action and interaction to collaboration (keynote)

  • Danica Kragic

Situated conceptualization: a framework for multimodal interaction (keynote)

  • Lawrence Barsalou

Steps towards collaborative multimodal dialogue (sustained contribution award)

  • Phil Cohen

SESSION: Oral Session 1: Children and Interaction

Tablets, tabletops, and smartphones: cross-platform comparisons of children’s touchscreen interactions

  • Julia Woodward
  • Alex Shaw
  • Aishat Aloba
  • Ayushi Jain
  • Jaime Ruiz
  • Lisa Anthony

Toward an efficient body expression recognition based on the synthesis of a neutral movement

  • Arthur Crenn
  • Alexandre Meyer
  • Rizwan Ahmed Khan
  • Hubert Konik
  • Saida Bouakaz

Interactive narration with a child: impact of prosody and facial expressions

  • Ovidiu Șerban
  • Mukesh Barange
  • Sahba Zojaji
  • Alexandre Pauchet
  • Adeline Richard
  • Emilie Chanoni

Comparing human and machine recognition of children’s touchscreen stroke gestures

  • Alex Shaw
  • Jaime Ruiz
  • Lisa Anthony

SESSION: Oral Session 2: Understanding Human Behaviour

Virtual debate coach design: assessing multimodal argumentation performance

  • Volha Petukhova
  • Tobias Mayer
  • Andrei Malchanau
  • Harry Bunt

Predicting the distribution of emotion perception: capturing inter-rater variability

  • Biqiao Zhang
  • Georg Essl
  • Emily Mower Provost

Automatically predicting human knowledgeability through non-verbal cues

  • Abdelwahab Bourai
  • Tadas Baltrušaitis
  • Louis-Philippe Morency

Pooling acoustic and lexical features for the prediction of valence

  • Zakaria Aldeneh
  • Soheil Khorram
  • Dimitrios Dimitriadis
  • Emily Mower Provost

SESSION: Oral Session 3: Touch and Gesture

Hand-to-hand: an intermanual illusion of movement

  • Dario Pittera
  • Marianna Obrist
  • Ali Israr

An investigation of dynamic crossmodal instantiation in TUIs

  • Feng Feng
  • Tony Stockman

“Stop over there”: natural gesture and speech interaction for non-critical spontaneous intervention in autonomous driving

  • Robert Tscharn
  • Marc Erich Latoschik
  • Diana Löffler
  • Jörn Hurtienne

Pre-touch proxemics: moving the design space of touch targets from still graphics towards proxemic behaviors

  • Ilhan Aslan
  • Elisabeth André

Freehand grasping in mixed reality: analysing variation during transition phase of interaction

  • Maadh Al-Kalbani
  • Maite Frutos-Pascual
  • Ian Williams

Rhythmic micro-gestures: discreet interaction on-the-go

  • Euan Freeman
  • Gareth Griffiths
  • Stephen A. Brewster

SESSION: Oral Session 4: Sound and Interaction

Evaluation of psychoacoustic sound parameters for sonification

  • Jamie Ferguson
  • Stephen A. Brewster

Utilising natural cross-modal mappings for visual control of feature-based sound synthesis

  • Augoustinos Tsiros
  • Grégory Leplâtre

SESSION: Oral Session 5: Methodology

Automatic classification of auto-correction errors in predictive text entry based on EEG and context information

  • Felix Putze
  • Maik Schünemann
  • Tanja Schultz
  • Wolfgang Stuerzlinger

Cumulative attributes for pain intensity estimation

  • Joy O. Egede
  • Michel Valstar

Towards the use of social interaction conventions as prior for gaze model adaptation

  • Rémy Siegfried
  • Yu Yu
  • Jean-Marc Odobez

Multimodal sentiment analysis with word-level fusion and reinforcement learning

  • Minghai Chen
  • Sen Wang
  • Paul Pu Liang
  • Tadas Baltrušaitis
  • Amir Zadeh
  • Louis-Philippe Morency

IntelliPrompter: speech-based dynamic note display interface for oral presentations

  • Reza Asadi
  • Ha Trinh
  • Harriet J. Fell
  • Timothy W. Bickmore

SESSION: Oral Session 6: Artificial Agents and Wearable Sensors

Head and shoulders: automatic error detection in human-robot interaction

  • Pauline Trung
  • Manuel Giuliani
  • Michael Miksch
  • Gerald Stollnberger
  • Susanne Stadler
  • Nicole Mirnig
  • Manfred Tscheligi

The reliability of non-verbal cues for situated reference resolution and their interplay with language: implications for human robot interaction

  • Stephanie Gross
  • Brigitte Krenn
  • Matthias Scheutz

Do you speak to a human or a virtual agent? automatic analysis of user’s social cues during mediated communication

  • Magalie Ochs
  • Nathan Libermann
  • Axel Boidin
  • Thierry Chaminade

Estimating verbal expressions of task and social cohesion in meetings by quantifying paralinguistic mimicry

  • Marjolein C. Nanninga
  • Yanxia Zhang
  • Nale Lehmann-Willenbrock
  • Zoltán Szlávik
  • Hayley Hung

Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks

  • Terry T. Um
  • Franz M. J. Pfister
  • Daniel Pichler
  • Satoshi Endo
  • Muriel Lang
  • Sandra Hirche
  • Urban Fietzek
  • Dana Kulić

SESSION: Poster Session 1

Automatic assessment of communication skill in non-conventional interview settings: a comparative study

  • Pooja Rao S. B
  • Sowmya Rasipuram
  • Rahul Das
  • Dinesh Babu Jayagopi

Low-intrusive recognition of expressive movement qualities

  • Radoslaw Niewiadomski
  • Maurizio Mancini
  • Stefano Piana
  • Paolo Alborno
  • Gualtiero Volpe
  • Antonio Camurri

Digitising a medical clerking system with multimodal interaction support

  • Harrison South
  • Martin Taylor
  • Huseyin Dogan
  • Nan Jiang

GazeTap: towards hands-free interaction in the operating room

  • Benjamin Hatscher
  • Maria Luz
  • Lennart E. Nacke
  • Norbert Elkmann
  • Veit Müller
  • Christian Hansen

Boxer: a multimodal collision technique for virtual objects

  • Byungjoo Lee
  • Qiao Deng
  • Eve Hoggan
  • Antti Oulasvirta

Trust triggers for multimodal command and control interfaces

  • Helen Hastie
  • Xingkun Liu
  • Pedro Patron

TouchScope: a hybrid multitouch oscilloscope interface

  • Matthew Heinz
  • Sven Bertel
  • Florian Echtler

A multimodal system to characterise melancholia: cascaded bag of words approach

  • Shalini Bhatia
  • Munawar Hayat
  • Roland Goecke

Crowdsourcing ratings of caller engagement in thin-slice videos of human-machine dialog: benefits and pitfalls

  • Vikram Ramanarayanan
  • Chee Wee Leong
  • David Suendermann-Oeft
  • Keelan Evanini

Modelling fusion of modalities in multimodal interactive systems with MMMM

  • Bruno Dumas
  • Jonathan Pirau
  • Denis Lalanne

Temporal alignment using the incremental unit framework

  • Casey Kennington
  • Ting Han
  • David Schlangen

Multimodal gender detection

  • Mohamed Abouelenien
  • Verónica Pérez-Rosas
  • Rada Mihalcea
  • Mihai Burzo

How may I help you? behavior and impressions in hospitality service encounters

  • Skanda Muralidhar
  • Marianne Schmid Mast
  • Daniel Gatica-Perez

Tracking liking state in brain activity while watching multiple movies

  • Naoto Terasawa
  • Hiroki Tanaka
  • Sakriani Sakti
  • Satoshi Nakamura

Does serial memory of locations benefit from spatially congruent audiovisual stimuli? investigating the effect of adding spatial sound to visuospatial sequences

  • Benjamin Stahl
  • Georgios Marentakis

ZSGL: zero shot gestural learning

  • Naveen Madapana
  • Juan Wachs

Markov reward models for analyzing group interaction

  • Gabriel Murray

Analyzing first impressions of warmth and competence from observable nonverbal cues in expert-novice interactions

  • Beatrice Biancardi
  • Angelo Cafaro
  • Catherine Pelachaud

The NoXi database: multimodal recordings of mediated novice-expert interactions

  • Angelo Cafaro
  • Johannes Wagner
  • Tobias Baur
  • Soumia Dermouche
  • Mercedes Torres Torres
  • Catherine Pelachaud
  • Elisabeth André
  • Michel Valstar

Head-mounted displays as opera glasses: using mixed-reality to deliver an egalitarian user experience during live events

  • Carl Bishop
  • Augusto Esteves
  • Iain McGregor

SESSION: Poster Session 2

Analyzing gaze behavior during turn-taking for estimating empathy skill level

  • Ryo Ishii
  • Shiro Kumano
  • Kazuhiro Otsuka

Text based user comments as a signal for automatic language identification of online videos

  • A. Seza Doğruöz
  • Natalia Ponomareva
  • Sertan Girgin
  • Reshu Jain
  • Christoph Oehler

Gender and emotion recognition with implicit user signals

  • Maneesh Bilalpur
  • Seyed Mostafa Kia
  • Manisha Chawla
  • Tat-Seng Chua
  • Ramanathan Subramanian

Animating the adelino robot with ERIK: the expressive robotics inverse kinematics

  • Tiago Ribeiro
  • Ana Paiva

Automatic detection of pain from spontaneous facial expressions

  • Fatma Meawad
  • Su-Yin Yang
  • Fong Ling Loy

Evaluating content-centric vs. user-centric ad affect recognition

  • Abhinav Shukla
  • Shruti Shriya Gullapuram
  • Harish Katti
  • Karthik Yadati
  • Mohan Kankanhalli
  • Ramanathan Subramanian

A domain adaptation approach to improve speaker turn embedding using face representation

  • Nam Le
  • Jean-Marc Odobez

Computer vision based fall detection by a convolutional neural network

  • Miao Yu
  • Liyun Gong
  • Stefanos Kollias

Predicting meeting extracts in group discussions using multimodal convolutional neural networks

  • Fumio Nihei
  • Yukiko I. Nakano
  • Yutaka Takase

The relationship between task-induced stress, vocal changes, and physiological state during a dyadic team task

  • Catherine Neubauer
  • Mathieu Chollet
  • Sharon Mozgai
  • Mark Dennison
  • Peter Khooshabeh
  • Stefan Scherer

Meyendtris: a hands-free, multimodal tetris clone using eye tracking and passive BCI for intuitive neuroadaptive gaming

  • Laurens R. Krol
  • Sarah-Christin Freytag
  • Thorsten O. Zander

AMHUSE: a multimodal dataset for HUmour SEnsing

  • Giuseppe Boccignone
  • Donatello Conte
  • Vittorio Cuculo
  • Raffaella Lanzarotti

GazeTouchPIN: protecting sensitive data on mobile devices using secure multimodal authentication

  • Mohamed Khamis
  • Mariam Hassib
  • Emanuel von Zezschwitz
  • Andreas Bulling
  • Florian Alt

Multi-task learning of social psychology assessments and nonverbal features for automatic leadership identification

  • Cigdem Beyan
  • Francesca Capozzi
  • Cristina Becchio
  • Vittorio Murino

Multimodal analysis of vocal collaborative search: a public corpus and results

  • Daniel McDuff
  • Paul Thomas
  • Mary Czerwinski
  • Nick Craswell

UE-HRI: a new dataset for the study of user engagement in spontaneous human-robot interactions

  • Atef Ben-Youssef
  • Chloé Clavel
  • Slim Essid
  • Miriam Bilac
  • Marine Chamoux
  • Angelica Lim

Mining a multimodal corpus of doctor’s training for virtual patient’s feedbacks

  • Chris Porhet
  • Magalie Ochs
  • Jorane Saubesty
  • Grégoire de Montcheuil
  • Roxane Bertrand

Multimodal affect recognition in an interactive gaming environment using eye tracking and speech signals

  • Ashwaq Alhargan
  • Neil Cooke
  • Tareq Binjammaz

SESSION: Demonstrations 1

Multimodal interaction in classrooms: implementation of tangibles in integrated music and math lessons

  • Jennifer Müller
  • Uwe Oestermeier
  • Peter Gerjets

Web-based interactive media authoring system with multimodal interaction

  • Bok Deuk Song
  • Yeon Jun Choi
  • Jong Hyun Park

Textured surfaces for ultrasound haptic displays

  • Euan Freeman
  • Ross Anderson
  • Julie Williamson
  • Graham Wilson
  • Stephen A. Brewster

Rapid development of multimodal interactive systems: a demonstration of platform for situated intelligence

  • Dan Bohus
  • Sean Andrist
  • Mihai Jalobeanu

MIRIAM: a multimodal chat-based interface for autonomous systems

  • Helen Hastie
  • Francisco Javier Chiyah Garcia
  • David A. Robb
  • Pedro Patron
  • Atanas Laskov

SAM: the school attachment monitor

  • Dong-Bach Vo
  • Mohammad Tayarani
  • Maki Rooksby
  • Rui Huan
  • Alessandro Vinciarelli
  • Helen Minnis
  • Stephen A. Brewster

The Boston Massacre history experience

  • David Novick
  • Laura Rodriguez
  • Aaron Pacheco
  • Aaron Rodriguez
  • Laura Hinojos
  • Brad Cartwright
  • Marco Cardiel
  • Ivan Gris Sepulveda
  • Olivia Rodriguez-Herrera
  • Enrique Ponce

Demonstrating TouchScope: a hybrid multitouch oscilloscope interface

  • Matthew Heinz
  • Sven Bertel
  • Florian Echtler

The MULTISIMO multimodal corpus of collaborative interactions

  • Maria Koutsombogera
  • Carl Vogel

Using mobile virtual reality to empower people with hidden disabilities to overcome their barriers

  • Matthieu Poyade
  • Glyn Morris
  • Ian Taylor
  • Victor Portela

SESSION: Demonstrations 2

Bot or not: exploring the fine line between cyber and human identity

  • Mirjam Wester
  • Matthew P. Aylett
  • David A. Braude

Modulating the non-verbal social signals of a humanoid robot

  • Amol Deshmukh
  • Bart Craenen
  • Alessandro Vinciarelli
  • Mary Ellen Foster

Thermal in-car interaction for navigation

  • Patrizia Di Campli San Vito
  • Stephen A. Brewster
  • Frank Pollick
  • Stuart White

AQUBE: an interactive music reproduction system for aquariums

  • Daisuke Sasaki
  • Musashi Nakajima
  • Yoshihiro Kanno

Real-time mixed-reality telepresence via 3D reconstruction with HoloLens and commodity depth sensors

  • Michal Joachimczak
  • Juan Liu
  • Hiroshi Ando

Evaluating robot facial expressions

  • Ruth Aylett
  • Frank Broz
  • Ayan Ghosh
  • Peter McKenna
  • Gnanathusharan Rajendran
  • Mary Ellen Foster
  • Giorgio Roffo
  • Alessandro Vinciarelli

Bimodal feedback for in-car mid-air gesture interaction

  • Gözel Shakeri
  • John H. Williamson
  • Stephen A. Brewster

A modular, multimodal open-source virtual interviewer dialog agent

  • Kirby Cofino
  • Vikram Ramanarayanan
  • Patrick Lange
  • David Pautler
  • David Suendermann-Oeft
  • Keelan Evanini

Wearable interactive display for the local positioning system (LPS)

  • Daniel M. Lofaro
  • Christopher Taylor
  • Ryan Tse
  • Donald Sofge

SESSION: Grand Challenge

From individual to group-level emotion recognition: EmotiW 5.0

  • Abhinav Dhall
  • Roland Goecke
  • Shreya Ghosh
  • Jyoti Joshi
  • Jesse Hoey
  • Tom Gedeon

Multi-modal emotion recognition using semi-supervised learning and multiple neural networks in the wild

  • Dae Ha Kim
  • Min Kyu Lee
  • Dong Yoon Choi
  • Byung Cheol Song

Modeling multimodal cues in a deep learning-based framework for emotion recognition in the wild

  • Stefano Pini
  • Olfa Ben Ahmed
  • Marcella Cornia
  • Lorenzo Baraldi
  • Rita Cucchiara
  • Benoit Huet

Group-level emotion recognition using transfer learning from face identification

  • Alexandr Rassadin
  • Alexey Gruzdev
  • Andrey Savchenko

Group emotion recognition with individual facial emotion CNNs and global image based CNNs

  • Lianzhi Tan
  • Kaipeng Zhang
  • Kai Wang
  • Xiaoxing Zeng
  • Xiaojiang Peng
  • Yu Qiao

Learning supervised scoring ensemble for emotion recognition in the wild

  • Ping Hu
  • Dongqi Cai
  • Shandong Wang
  • Anbang Yao
  • Yurong Chen

Group emotion recognition in the wild by combining deep neural networks for facial expression classification and scene-context analysis

  • Asad Abbas
  • Stephan K. Chalup

Temporal multimodal fusion for video emotion classification in the wild

  • Valentin Vielzeuf
  • Stéphane Pateux
  • Frédéric Jurie

Audio-visual emotion recognition using deep transfer learning and multiple temporal models

  • Xi Ouyang
  • Shigenori Kawaai
  • Ester Gue Hua Goh
  • Shengmei Shen
  • Wan Ding
  • Huaiping Ming
  • Dong-Yan Huang

Multi-level feature fusion for group-level emotion recognition

  • B. Balaji
  • V. Ramana Murthy Oruganti

A new deep-learning framework for group emotion recognition

  • Qinglan Wei
  • Yijia Zhao
  • Qihua Xu
  • Liandong Li
  • Jun He
  • Lejun Yu
  • Bo Sun

Emotion recognition in the wild using deep neural networks and Bayesian classifiers

  • Luca Surace
  • Massimiliano Patacchiola
  • Elena Battini Sönmez
  • William Spataro
  • Angelo Cangelosi

Emotion recognition with multimodal features and temporal models

  • Shuai Wang
  • Wenxuan Wang
  • Jinming Zhao
  • Shizhe Chen
  • Qin Jin
  • Shilei Zhang
  • Yong Qin

Group-level emotion recognition using deep models on image scene, faces, and skeletons

  • Xin Guo
  • Luisa F. Polanía
  • Kenneth E. Barner

SESSION: Doctoral Consortium

Towards designing speech technology based assistive interfaces for children's speech therapy

  • Revathy Nayar

Social robots for motivation and engagement in therapy

  • Katie Winkle

Immersive virtual eating and conditioned food responses

  • Nikita Mae B. Tuanquin

Towards edible interfaces: designing interactions with food

  • Tom Gayler

Towards a computational model for first impressions generation

  • Beatrice Biancardi

A decentralised multimodal integration of social signals: a bio-inspired approach

  • Esma Mansouri-Benssassi

Human-centered recognition of children's touchscreen gestures

  • Alex Shaw

Cross-modality interaction between EEG signals and facial expression

  • Soheil Rayatdoost

Hybrid models for opinion analysis in speech interactions

  • Valentin Barriere

Evaluating engagement in digital narratives from facial data

  • Rui Huan

Social signal extraction from egocentric photo-streams

  • Maedeh Aghaei

Multimodal language grounding for improved human-robot collaboration: exploring spatial semantic representations in the shared space of attention

  • Dimosthenis Kontogiorgos

SESSION: Workshop Summaries

ISIAA 2017: 1st international workshop on investigating social interactions with artificial agents (workshop summary)

  • Thierry Chaminade
  • Fabrice Lefèvre
  • Noël Nguyen
  • Magalie Ochs

WOCCI 2017: 6th international workshop on child computer interaction (workshop summary)

  • Keelan Evanini
  • Maryam Najafian
  • Saeid Safavi
  • Kay Berkling

MIE 2017: 1st international workshop on multimodal interaction for education (workshop summary)

  • Gualtiero Volpe
  • Monica Gori
  • Nadia Bianchi-Berthouze
  • Gabriel Baud-Bovy
  • Paolo Alborno
  • Erica Volta

Playlab: telling stories with technology (workshop summary)

  • Julie Williamson
  • Tom Flint
  • Chris Speed

MHFI 2017: 2nd international workshop on multisensorial approaches to human-food interaction (workshop summary)

  • Carlos Velasco
  • Anton Nijholt
  • Marianna Obrist
  • Katsunori Okajima
  • Rick Schifferstein
  • Charles Spence






ICMI 2017 ACM International Conference on Multimodal Interaction. Copyright © 2017-2017