A User and Designer Perspective on Multimodal Interaction in 3D environments

Abstract

Creating user interfaces that fully support the three dimensions, is not a simple process. Although 3D interfaces have been successfully applied in several domains, they often support a poor interaction, compared to the possibilities of our human body. Current solutions often limit to visual output (e.g. via a screen) and 2D or 3D input via mice or trackers. As a consequence, those 3D environments are often far from the intuitive and user friendly application they are intended to be.

In our daily life, humans have several possibilities to communicate, such as speech, the hearing, the vision, touch and gestures. Seen from a user’s perspective, the research described in this thesis is focused on improving the interaction within such a 3D environment. Therefore, we describe several user experiments describing the possible benefits of force feedback, speech input and two-handed interaction.

Since finding the best interface for a 3D environment is often a process of trialand-error, the development cycle of such applications is more often than not long and expensive. Seen from a designer’s perspective, this thesis shows how the design process may be facilitated. We propose NiMMiT, a visual notation, that can be used to describe interaction with the environment as a part of a model-based development approach. In this context, we also show how an application framework can abstract from the physical devices, supporting easy testing of the environment.

We believe that the results of this thesis will contribute to an easier and more intuitive interaction in 3D environments.