Please use this identifier to cite or link to this item:
http://hdl.handle.net/1942/31300
Full metadata record
DC Field | Value | Language |
---|---|---|
dc.contributor.author | VEUSKENS, Tom | - |
dc.contributor.author | LUYTEN, Kris | - |
dc.contributor.author | RAMAKERS, Raf | - |
dc.date.accessioned | 2020-06-17T13:20:40Z | - |
dc.date.available | 2020-06-17T13:20:40Z | - |
dc.date.issued | 2020 | - |
dc.date.submitted | 2020-06-16T12:53:23Z | - |
dc.identifier.citation | Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4 (2) (Art N° 60) | - |
dc.identifier.uri | http://hdl.handle.net/1942/31300 | - |
dc.description.abstract | We present Rataplan, a robust and resilient pixel-based approach for linking multi-modal proxies to automated sequences of actions in graphical user interfaces (GUIs). With Rataplan, users demonstrate a sequence of actions and answer human-readable follow-up questions to clarify their desire for automation. After demonstrating a sequence, the user can link a proxy input control to the action which can then be used as a shortcut for automating a sequence. Alternatively, output proxies use a notification model in which content is pushed when it becomes available. As an example use case, Rataplan uses keyboard shortcuts and tangible user interfaces (TUIs) as input proxies, and TUIs as output proxies. Instead of relying on available APIs, Rataplan automates GUIs using pixel-based reverse engineering. This ensures our approach can be used with all applications that offer a GUI, including web applications. We implemented a set of important strategies to support robust automation of modern interfaces that have a flat and minimal style, have frequent data and state changes, and have dynamic viewports. | - |
dc.description.sponsorship | ACKNOWLEDGMENTS This research was supported by the Special Research Fund (BOF) of Hasselt University and by the Research Foundation - Flanders (FWO), project G0E7317N End-User Development of Intelligible Internet-of-Things Objects and Applications. | - |
dc.language.iso | en | - |
dc.publisher | - | |
dc.rights | 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM. | - |
dc.subject.other | tangible user interfaces | - |
dc.subject.other | pixel-based reverse engineering | - |
dc.subject.other | UI automation | - |
dc.subject.other | programming-by-demonstration | - |
dc.title | Rataplan: Resilient Automation of User Interface Actions with Multi-modal Proxies | - |
dc.type | Journal Contribution | - |
dc.identifier.epage | 23 | - |
dc.identifier.issue | 2 | - |
dc.identifier.spage | 1 | - |
dc.identifier.volume | 4 | - |
local.bibliographicCitation.jcat | A1 | - |
local.type.refereed | Refereed | - |
local.type.specified | Article | - |
local.bibliographicCitation.artnr | 60 | - |
dc.identifier.doi | 10.1145/3397329 | - |
dc.identifier.eissn | - | |
local.provider.type | CrossRef | - |
local.uhasselt.uhpub | yes | - |
local.uhasselt.international | no | - |
item.contributor | VEUSKENS, Tom | - |
item.contributor | LUYTEN, Kris | - |
item.contributor | RAMAKERS, Raf | - |
item.fullcitation | VEUSKENS, Tom; LUYTEN, Kris & RAMAKERS, Raf (2020) Rataplan: Resilient Automation of User Interface Actions with Multi-modal Proxies. In: Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4 (2) (Art N° 60). | - |
item.accessRights | Restricted Access | - |
item.fulltext | With Fulltext | - |
item.validation | vabb 2022 | - |
crisitem.journal.eissn | 2474-9567 | - |
Appears in Collections: | Research publications |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
3397329.pdf Restricted Access | Published version | 11.25 MB | Adobe PDF | View/Open Request a copy |
WEB OF SCIENCETM
Citations
2
checked on May 1, 2024
Page view(s)
112
checked on Jun 9, 2022
Download(s)
42
checked on Jun 9, 2022
Google ScholarTM
Check
Altmetric
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.