Multimodal selection in virtual environments: Enhancing the user experience and facilitating development

Abstract

The usage of virtual environments is manifold, from visualisation and interaction with molecular data to medical applications. Virtual environments are represented in three dimensions and display technology to visualise such environments has significantly improved. Besides visual feedback, other types of feedback in virtual environments, such as haptics and audio, are gaining importance. We explore the possible benefits of these types of feedback. In this dissertation we focus on one of the four primary interaction techniques which must be supported for interactive virtual environments, namely selection. Our efforts during this dissertation are twofold. Firstly, we perform research into selection techniques augmented with multimodal feedback. Several newly designed selection techniques are compared taking into account new factors such as the environment density and occlusion. Results show that our techniques could manage these factors and that multimodal feedback provides small non-significant improvements. In our experiments the force feedback was designed using a pilot test or a trial-and-error approach, therefore we try to formulate a guideline to design force feedback. We perform a series of multidirectional pointselect experiments which show that the definite integral of the force profile can be used as a guideline to know when a certain force feedback value will cause the performance of the user to deteriorate during his movement. Secondly, we propose approaches to integrate semantic and contextual knowledge in a model-based user interface design process for virtual environments. Semantic information is added through the addition of a data type to the interaction description model (NiMMiT) and the automatic generation of a speech grammar. With regard to context, we propose a context system based on the ‘Event-Condition-Action’ paradigm which is realised at the dialog level instead of the usual task level. Several case studies show that these additions can be useful during the design of interaction techniques including selection techniques.