Please use this identifier to cite or link to this item:
http://hdl.handle.net/1942/31300
Title: | Rataplan: Resilient Automation of User Interface Actions with Multi-modal Proxies | Authors: | VEUSKENS, Tom LUYTEN, Kris RAMAKERS, Raf |
Issue Date: | 2020 | Publisher: | Source: | Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4 (2) (Art N° 60) | Abstract: | We present Rataplan, a robust and resilient pixel-based approach for linking multi-modal proxies to automated sequences of actions in graphical user interfaces (GUIs). With Rataplan, users demonstrate a sequence of actions and answer human-readable follow-up questions to clarify their desire for automation. After demonstrating a sequence, the user can link a proxy input control to the action which can then be used as a shortcut for automating a sequence. Alternatively, output proxies use a notification model in which content is pushed when it becomes available. As an example use case, Rataplan uses keyboard shortcuts and tangible user interfaces (TUIs) as input proxies, and TUIs as output proxies. Instead of relying on available APIs, Rataplan automates GUIs using pixel-based reverse engineering. This ensures our approach can be used with all applications that offer a GUI, including web applications. We implemented a set of important strategies to support robust automation of modern interfaces that have a flat and minimal style, have frequent data and state changes, and have dynamic viewports. | Keywords: | tangible user interfaces;pixel-based reverse engineering;UI automation;programming-by-demonstration | Document URI: | http://hdl.handle.net/1942/31300 | e-ISSN: | 2474-9567 | DOI: | 10.1145/3397329 | Rights: | 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM. | Category: | A1 | Type: | Journal Contribution | Validations: | vabb 2022 |
Appears in Collections: | Research publications |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
3397329.pdf Restricted Access | Published version | 11.25 MB | Adobe PDF | View/Open Request a copy |
WEB OF SCIENCETM
Citations
2
checked on Oct 15, 2024
Page view(s)
112
checked on Jun 9, 2022
Download(s)
42
checked on Jun 9, 2022
Google ScholarTM
Check
Altmetric
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.