Enabling Empathic Communication in Ubiquitous Computing Environments to Improve Interaction between People

Abstract

Empathy is crucial for the establishment of building consensus and seeing a situation from the other person’s emotional point of view. Being able to empathize with our conversational partners can lead to better interpretation of the intended messages as well as improve social and working relationships. We envision that enabling empathic communication through the use of computers is possible with recent advances in Ubiquitous computing (Ubicomp) technology and affective computing. For instance, mediating systems can provide empathic feedback to influence one’s own decision-making process through the registration of the user’s affective state, which is captured by sensors embedded in the Ubicomp environment. However, contemporary emotion recognition techniques have been reported to be effective for a set of basic affective states and also expect users to be seated or remain in a stationary position. Therefore, there is a need to identify a diverse spectrum of affective states through nonverbal means and without constraining the users in order to allow empathic communication with the conversational parties in the Ubicomp environment. In this dissertation, a framework for empathic communication was proposed to provide emotion awareness in Ubicomp environments. The framework, which comprises of vision-based posture recognition, affective state inference and visual representation, is applicable in both face-to-face interaction as well as remote collaboration. The vision-based posture recognition uses a depth camera for determining the whole body articulation in real-time. Sets of kinematical and semantical features are extracted for classifying eleven subtle postures that can be related to an affective state of the user. The results of these postures can also be used to discriminate different openness level of the users. We refer to openness as the flexibility of one’s behaviors and personal tendency to take into consideration the possibility of accepting another person.ii The affective state inference was developed to use association rules for inferring the affective states solely from the body postures. By combining the posture information with physiological measurements we were able to mine a set of association rules. This is done by relating postures to a new affective categorization alongside the fuzzy logic-based affective dimensions of arousal and valence (in an Arousal-Valence-Openness model) for an extended representation of emotions. We show that our system is equivalent with using physiological measurements for each openness level. Evaluation on third-party affective dataset shows the validations of our approach to generate fine-grained emotion inference. In addition, body postures are also identified from the third-party affective dataset. Bro-cam was presented as a proof-of-concept implementation of the visionbased posture recognition and affective state inference. Bro-cam uses the postures as an indicator of their openness and profiles the players into different personality types ranging from introvert to extrovert. Empathic feedback, which is matched to the preferences of the personality type, is then generated automatically. A simple and intuitive visualization was developed to support empathic communication during remote collaboration. The visual representation shows the psychophysiological measurements in an easy to read and understandable format. In this way, the remote person is able to convey physiological information related to their emotional behavior during remote tasks. The effects of using the visual representation are investigated for videomediated collaboration in which an instructor remotely guides a worker in a confined work environment. Different display configurations of visual representation, facial, and task views are used to study the users perceived emotional (positive affect and group cohesiveness) and cognitive effects (stress and mental workload). The study shows that integrating physiological reading on a visual representation and as part of the video-mediated interface can improve remote task collaboration. The stress level (i.e. increased task engagement and decreased worry) can be reduced for the instructors and the mental workload (i.e. heightened performance) can be lowered for the workers. These findings provide both emotional and cognitive insights to develop interactive interfaces for Ubicomp systems to enable empathic communication in workrelated context.