DOI: 10.1145/1125451.1125759 | CITEULIKE: 882184 | REFERENCE: BibTex, Endnote, RefMan | PDF

Lyons, M., & Bartneck, C. (2006). HCI and the Face. Proceedings of the Conference on Human Factors in Computing Systems (CHI2006), Montreal, pp 1671 - 1674.

HCI and the Face

Michael J. Lyons

ATR Intelligent Robotics and Communication Labs
2-2-2 Hikaridai, Seika-cho Soraku-gun
Kyoto 619-0288 Japan

Christoph Bartneck

Department of Industrial Design
Eindhoven University of Technology
Den Dolech 2, 5600MB Eindhoven, NL

Abstract - The workshop aims at a general assessment of facial information processing in HCI. We will discuss why certain areas of face-based HCI, such as facial expression recognition and robotic facial display, have lagged others, such as eye gaze tracking, facial identity recognition, and conversational characters. Our goal is to collectively identify research strategies to bring the more slowly developing areas up to speed.

Keywords: Face, recognition, eye gaze, emotion, expression

Introduction and Motivation

The human face plays an important role in many aspects of verbal and non-verbal communication. The face is involved in speech, the facial expression of emotions, gestures such as nods, winks, and other human communicative acts. Major subfields of neuroscience, cognitive science, and psychology are devoted to study of the information conveyed by the human face. Increasingly, computer scientists and engineers have worked on the face from the viewpoints of graphics, animation, computer vision, and pattern recognition. By contrast, in the past, relatively few works on human-computer interaction have concerned the human face. This is more likely to reflect technical limitations to the development of practical applications rather than any intrinsic lack of relevance to HCI. Many problems in HCI involve aspects of human identity, affect, and tacit communication, which are all informed by information conveyed by the face. In fact, human-computer interaction is one of the most commonly stated application domains for works reporting basic research into facial information processing.

Eye gaze has occupied the greatest share of the attention of HCI researchers [10]. Eye gaze tracking technology has become sufficiently advanced that several companies offer complete, working systems which are usable even without special technical expertise. Gaze tracking has become a widely used technique in studies of interface usability, machine-mediated human communication, as well as alternative input devices for disabled users. This area can therefore be viewed as a successful, mature sub-field related to face-based HCI.

Research on face classification [6], mostly by the pattern recognition community, is perhaps the only other case of a relatively mature sub-field of facial information processing, with several companies offering access control and security systems to end users.

Compared to the state of the art in screen-based characters, such as Embodied Conversational Agents [3], the field of robot’s facial expressions is underdeveloped. Much attention has been paid to robots’ motor skills, such as walking and gesturing, but little work has been done on their facial expression.

Advanced humanoid robots such as the Honda Asimo or the Sony Quiro have barely any facial features at all. The Phillips iCat and the most realistic android developed to date, Ishiguro’s Repliee Q1Expo, represent substantial progress in this respect, but even the Repliee Q1Expo is only able to convince humans of the naturalness of its expressions for at best a few seconds. In summary, natural robotic expressions remain in their infancy [5].

Some fundamental theoretical aspects of facial information processing relevant to HCI applications remain unresolved. The representation of the space of emotional expressions is a prime example [9]. The space of expressions is often modeled either with continuous dimensions, such as valence and arousal [8] or with a categorical approach [4]. This controversial issue has broad implications for all HCI applications involving facial expression [9]. The same can be said for other fundamental aspects of facial information processing, such as the believability of synthetic facial expressions by characters and robots [1].

In this workshop we would like to examine the status of these and other aspects of facial information input and output within the context of HCI. Why have several areas of research, such as facial expression processing, fallen short of usable, practical applications? Are there technical barriers or does the general approach need revision? What is the potential of areas of the face, other than the eyes, for gestural input? What can be learned from ongoing research in pattern recognition, computer graphics, and robotics to aid the progress of face-based HCI?

Topics for the Workshop

The workshop will explore a broad range of topics related to facial information processing and its relevance to HCI. Below we list some of the topics of potential interest.

Facial Output

Facial Input

Theory and Research Paradigms

Application areas

Further Research, Dissemination, Education

Goals of the Workshop

Our main purpose in organizing this workshop is to take stock of the current situation with regards to HCI and human face, by bringing together researchers on several aspects of face processing, who are interested in human-computer interaction, with HCI experts who are interested in the human face and to stimulate exchange with following aims:

  1. To review current and recent works in human-computer interaction, as well as important works in the past, which have involved the use of techniques for analysis and display of information in the human face.
  2. To summarize the state of the art with regards to currently used techniques, and to identify significant outstanding issues or limitations of these which are relevant to HCI applications.
  3. To list domains of HCI which are widely recognized as being application domains of face processing technology and to identify novel potential applications.
  4. To pool the collective experience of the workshop participants to develop a strategy to move this field ahead. To plan public presentation of the shared knowledge and wisdom in the form of a special journal issue, co-authored book, and/or future meetings.

Format and organization

The workshop will consist of a day long highly interactive format encouraging small group dialogue and knowledge transfer. The overall goal is to elicit research issues and findings related to the design and implementation of HCI applications involving facial information processing as well as fundamental research which will further progress of this important field.


  1. Bartneck, C. (2001). How convincing is Mr. Data's smile: Affective expressions of machines. User Modeling and User-Adapted Interaction, 11, 279-295. | DOI: 10.1023/A:1011811315582 | DOWNLOAD
  2. Suzuki, N., & Bartneck, C. (2003). Subtle Expressivity of Characters and Robots. Proceedings of the CHI2003, Fort Lauderdale (Extended Abstracts), pp 1064 - 1065. | DOI: 10.1145/765891.766150 | DOWNLOAD
  3. Cassell, J., Sullivan, J., Prevost, S., & Churchill, E. (2000). Embodied Conversational Agents. Cambridge: MIT Press. | view at
  4. Ekman, P., & Friesen, W. V. (1975). Unmasking the Face. Englewood Cliffs: Prentice Hall. | view at
  5. Fong, T., Nourbakhsh, I., & Dautenhahn, K. (2003). A survey of socially interactive robots. Robotics and Autonomous Systems, 42, 143-166. | DOI: 10.1016/S0921-8890(02)00372-X
  6. Lyons, M. J., Budynek, J., & Akamatsu, S. (1999). Automatic Classification of Single Facial Images. IEEE Transactions Pattern Analysis and Machine Intelligence, 21(12), 1357-1362. | DOI: 10.1109/34.817413
  7. Lyons, M. J. (2004). Facial Gesture Interfaces for Expression and Communication. Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, The Hague. | DOI: 10.1109/ICSMC.2004.1398365
  8. Russell, J. A. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110(1), 145-172. | DOI: 10.1037/0033-295X.110.1.145
  9. Schiano, D. J. (2004). Categorical Imperative NOT: Facial Affect is Perceived Continously. Proceedings of the CHI2004, Vienna. | DOI: 10.1145/985692.985699
  10. Zhai, S., Morimoto, C., & Ihde, S. (1999). Manual and gaze input cascaded (MAGIC) pointing . Proceedings of the Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, Pittsburgh, pp 246-253. | DOI: 10.1145/302979.303053

© ACM, 2006. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in Proceedings of the Conference on Human Factors in Computing Systems (CHI2006), Montreal, pp 1671 - 1674. | last updated January 30, 2008 | All Publications