Archive (2016–2006)

Real-time Feedback on Nonverbal Clinical Communication

Journal: Methods of Information in Medicine
Subtitle: A journal stressing, for more than 50 years, the methodology and scientific fundamentals of organizing, representing and analyzing data, information and knowledge in biomedicine and health care
ISSN: 0026-1270

Focus Theme
Pervasive Intelligent Technologies for Health
Guest Editors: O. Mayora, P. Lukowicz, M. Marschollek

Issue: 2014 (Vol. 53): Issue 5 2014
Pages: 389-405

Real-time Feedback on Nonverbal Clinical Communication

Theoretical Framework and Clinician Acceptance of Ambient Visual Design

Focus Theme - Pervasive Intelligent Technologies for Health

A. L. Hartzler (1), R. A. Patel (2), M. Czerwinski (3), W. Pratt (1, 2), A. Roseway (3), N. Chandrasekaran (3), A. Back (4)

(1) The Information School, University of Washington, Seattle, Washington, USA; (2) Department of Biomedical Informatics and Medical Education, Division of Biomedical and Health Informatics, University of Washington, Seattle, Washington, USA; (3) Microsoft Research, Redmond, Washington, USA; (4) Department of Medicine, Division of Oncology, University of Washington, Seattle, Washington, USA


user-computer interface, feedback, patient-provider communication, Nonverbal communication, information display, social signal processing, formative evaluation


Introduction: This article is part of the Focus Theme of Methods of Information in Medicine on “Pervasive Intelligent Technologies for Health”.

Background: Effective nonverbal communication between patients and clinicians fosters both the delivery of empathic patient-centered care and positive patient outcomes. Although nonverbal skill training is a recognized need, few efforts to enhance patient-clinician communication provide visual feedback on nonverbal aspects of the clinical encounter.

Objectives: We describe a novel approach that uses social signal processing technology (SSP) to capture nonverbal cues in real time and to display ambient visual feedback on control and affili-ation – two primary, yet distinct dimensions of interpersonal nonverbal communication. To examine the design and clinician acceptance of ambient visual feedback on nonverbal communication, we 1) formulated a model of relational communication to ground SSP and 2) conducted a formative user study using mixed methods to explore the design of visual feedback.

Methods: Based on a model of relational communication, we reviewed interpersonal communication research to map nonverbal cues to signals of affiliation and control evidenced in patient-clinician interaction. Corresponding with our formulation of this theoretical framework, we designed ambient real-time visualizations that reflect variations of affiliation and control. To explore clinicians’ acceptance of this visual feedback, we conducted a lab study using the Wizard-of-Oz technique to simulate system use with 16 healthcare professionals. We followed up with seven of those participants through interviews to iterate on the design with a revised visualization that addressed emergent design considerations.

Results: Ambient visual feedback on non- verbal communication provides a theoretically grounded and acceptable way to provide clinicians with awareness of their nonverbal communication style. We provide implications for the design of such visual feedback that encourages empathic patient-centered communication and include considerations of metaphor, color, size, position, and timing of feedback.

Conclusions: Ambient visual feedback from SSP holds promise as an acceptable means for facilitating empathic patient-centered nonverbal communication.

You may also be interested in...


Y. Mori 1, T. Igarashi 1, R. Haraguchi 2, K. Nakazawa2

Methods Inf Med 2007 46 6: 694-699


D. W. Glasspool 1, A. Oettinger 1, J. H. Smith-Spark 2, F. D. Castillo 1, V. E. L. Monaghan 1, J. Fox1

Methods Inf Med 2007 46 6: 636-640


M. M. Kuilboer 1 , M. A. M. van Wijk1 , M. Mosseveld1 , E. van der Does2 , J. C. de Jongste3 , S. E. Overbeek3 , B. Ponsioen2 , J. van der Lei1

Methods Inf Med 2006 45 4: 447-454