Programming Tangible Proxy Interfaces using GUI task demonstrations and follow-up questions

Abstract

The work presented in this thesis investigates novel techniques for automating GUI actions for users without any technical background. The thesis presents a toolkit which has been developed to proxy user interface behaviour to and from the physical world. To accomplish this, novel pixel-based reverse engineering approaches are developed which leverage programming-by-demonstration concepts as well as auto- mated follow-up questions to allow easy specification of user interface behaviour. In addition to the toolkit, a novel tangible kit with an accompanying tangible program- ming paradigm was developed which is specifically designed to allow easy linking of a tangible to the demonstrated user-interface behaviour to provide input from the physical world to the digital environment, or output from the digital environment to the physical world. To overcome the inherent limitation of pixel-based reverse engineering approaches, which require pixels to be visible in order to be analysed, a novel solution is presented which is leveraged to process non-visible pixels in addition to visible pixels.