Filtered by themeCombinatorial optimization8 papersclear ✕

User-Driven Constraints for Layout Optimisation in Augmented RealityAziz Niyazov · 2023Automatic layout optimisation allows users to arrange augmented reality content in the real-world environment without the need for tedious manual interactions. This optimisation is often based on modelling the intended content placement as constraints, defined as cost functions. Then, applying a cost minimization algorithm leads to a desirable placement. However, such an approach is lim- ited by the lack of user control over the optimisation results. In this paper we explore the concept of user-driven constraints for augmented reality layout optimisation. With our approach users can define and set up their own constraints directly within the real-world environment. We first present a design space composed of three dimensions: the constraints, the regions of interest and the constraint parameters. Then we explore which input gestures can be employed to define the user-driven constraints of our design space through a user elicitation study. Using the results of the study, we propose a holistic system design and implementation demon- strating our user-driven constraints, which we evaluate in a final user study where participants had to create several constraints at the same time to arrange a set of virtual contents. CCS CONCEPTS • Human-centered computing →Interactive systems and tools. ACM Reference Format: Aziz Niyazov, Barrett Ens, Kadek Ananta Satriadi, Nicolas Mellado, Loïc Barthe, Tim Dwyer, and Marcos Serrano. 2023. User-Driven Constraints Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. CHI ’23, April 23–28, 2023, Hamburg, Germany © 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-9421-5/23/04...$15.00 https://doi.org/10.1145/3544548.3580873 for Layout Optimisation in Augmented Reality. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23), April 23–28, 2023, Hamburg, Germany. ACM, New York, NY, USA, 16 pages. https: //doi.org/10.1145/3544

UX of AI & PersonalizationResearch Papers

SemanticAdapt: Optimization-based Adaptation of Mixed Reality Layouts Leveraging Virtual-Physical Semantic ConnectionsYifei Cheng∗ · 2021We present an optimization-based approach that automatically adapts Mixed Reality (MR) interfaces to different physical envi- ronments. Current MR layouts, including the position and scale of virtual interface elements, need to be manually adapted by users ∗This work was done while Yifei Cheng was an intern at Tsinghua University. †The corresponding author. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. UIST’21, October 2021, Virtual © 2021 Association for Computing Machinery. ACM ISBN 978-1-4503-8635-7/21/10...$15.00 https://doi.org/10.1145/3472749.3474750 whenever they move between environments, and whenever they switch tasks. This process is tedious and time consuming, and ar- guably needs to be automated for MR systems to be beneficial for end users. We contribute an approach that formulates this challenge as a combinatorial optimization problem and automatically decides the placement of virtual interface elements in new environments. To achieve this, we exploit the semantic association between the virtual interface elements and physical objects in an environment. Our optimization furthermore considers the utility of elements for users’ current task, layout factors, and spatio-temporal consistency to previous layouts. All those factors are combined in a single linear program, which is used to adapt the layout of MR interfaces in real time. We demonstrate a set of application scenarios, showcasing the versatility and applicability of our approach. Finally, we show that compared to a naive adaptive baseline approach that does not 282 UIST’21, October 2021, Virtual Cheng and Yan, et al. take semantic associations into account, our approach decreased the number of manual interface adaptations by 33%. CCS CONCEPTS • Human-centered computing →Mixed / augmented reality; Virtual reality; User interface management systems.

UX of AI & PersonalizationResearch Papers

Combinatorial Optimization of Graphical User Interface DesignsAntti Oulasvirt · 2020| The graphical user interface (GUI) has become the prime means for interacting with computing systems. It lever- ages human perceptual and motor capabilities for elementary tasks such as command exploration and invocation, informa- tion search, and multitasking. For designing a GUI, numerous interconnected decisions must be made such that the out- come strikes a balance between human factors and technical objectives. Normally, design choices are specified manually and coded within the software by professional designers and developers. This article surveys combinatorial optimization as a flexible and powerful tool for computational generation and adaptation of GUIs. As recently as 15 years ago, applications were limited to keyboards and widget layouts. The obstacle has been the mathematical definition of design tasks, on the one hand, and the lack of objective functions that capture essential aspects of human behavior, on the other. This article presents definitions of layout design problems as integer programming tasks, a coherent formalism that permits identification of problem types, analysis of their complexity, and exploitation of known algorithmic solutions. It then surveys advances in Manuscript received April 8, 2019; revised October 9, 2019 and January 12, 2020; accepted January 17, 2020. This work was supported in part by the European Research Council (ERC) through the European Union’s Horizon 2020 Research and Innovation Program under Grant 637991 and in part by the Academy of Finland projects Bayesian Artefact Design (BAD) and Human Automata. (Corresponding author: Antti Oulasvirta.) Antti Oulasvirta is with the Department of Communications and Networking, School of Electrical Engineering, Aalto University, 02150 Espoo, Finland, and also with the Finnish Center for Artificial Intelligence (FCAI), 02015 Espoo, Finland (e-mail: antti.oulasvirta@aalto.fi). Niraj Ramesh Dayama and Morteza Shiripour are with the Department of Communications and Networking, School of Electrical Engineering, Aalto University, 02150 Espoo, Finland. Maximilian John and Andreas Karrenbauer are with the Max Planck Institute for Informatics, 66123 Saarbrücken, Germany. Digital Object Identifier 10.1109/JPROC.2020.2969687 formulating evaluative functions for common design-goal foci such as user performance and experience. The convergence of these two advances has expanded the range of solvable problems. Approaches to practical deployment are outlined with a wide spectrum of applica

UX of AI & PersonalizationResearch Papers

I see, you design: user interface intelligent design system with eye tracking and interactive genetic algorithmCheng · 2019User interfaces are important for streamlining the interactions between humans and computers. However, there are few effective approaches for collecting users’ preferences implicitly and objectively for the purpose of user interface (UI) design optimization. This paper presents an effective approach to interactive genetic algorithm (IGA) optimization-based UI design via eye tracking, including eye-movement data based users’ preference inferring, gene coding for real UI components and design features, and the visualization and interaction mechanisms. Then we design and build a prototype system that applies IGA to generate and optimize music player UI solutions automatically. An evaluation of our prototype system suggests that it can generate and identify personalized UI designs reliably with minimal user intervention and high efficiency and user satisfaction.

UX of AI & PersonalizationResearch Papers

Individualising Graphical Layouts with Predictive Visual Search ModelsKRIS LUYTEN · 2019In domains where users are exposed to large variations in visuo-spatial features among designs, they often spend excess time searching for common elements (features) on an interface. This article contributes individualised predictive models of visual search, and a computational approach to restructure graphical layouts for an individual user such that features on a new, unvisited interface can be found quicker. It explores four technical principles inspired by the human visual system (HVS) to predict expected positions of features and create individualised layout templates: (I) the interface with highest frequency is chosen as the template; (II) the interface with highest predicted recall probability (serial position curve) is chosen as the template; (III) the most probable locations for features across interfaces are chosen (visual statistical learning) to generate the template; (IV) based on a generative cognitive model, the most likely visual search locations for features are chosen (visual sampling modelling) to generate the template. Given a history of previously seen interfaces, we restructure the spatial layout of a new (unseen) interface with the goal of making its features more easily findable. The four HVS principles are implemented in Familiariser, a web browser that automatically restructures webpage layouts based on the visual history of the user. Evaluation of Familiariser (using visual statistical learning) with users provides first evidence that our approach reduces visual search time by over 10%, and number of eye-gaze fixations by over 20%, during web browsing tasks.

UX of AI & PersonalizationResearch Papers

Context-Aware Online Adaptation of Mixed Reality InterfacesDavid Lindlbauer, Anna Maria Feit et al. · 2019We present an optimization-based approach for Mixed Reality (MR) systems to automatically control when and where appli­ cations are shown, and how much information they display. Currently, content creators design applications, and users then manually adjust which applications are visible and how much information they show. This choice has to be adjusted every time users switch context, i.e., whenever they switch their task or environment. Since context switches happen many times a day, we believe that MR interfaces require automation to alleviate this problem. We propose a real-time approach to automate this process based on users’ current cognitive load and knowledge about their task and environment. Our system adapts which applications are displayed, how much informa­ tion they show, and where they are placed. We formulate this problem as a mix of rule-based decision making and combina­ torial optimization which can be solved effciently in real-time. We present a set of proof-of-concept applications showing that our approach is applicable in a wide range of scenarios. Finally, we show in a dual-task evaluation that our approach decreased secondary tasks interactions by 36%. Author Keywords Mixed Reality, Context-Awareness, UI Optimization CCS Concepts •Human-centered computing → Mixed / augmented real­ ity; Virtual reality; User interface management systems; Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for proft or commercial advantage and that copies bear this notice and the full citation on the frst page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specifc permission and/or a fee. Request permissions from permissions@acm.org. UIST’19, The 32nd Annual ACM Symposium on User Interface Software and Tech­ nology, October 20–23, 2019. New Orleans, LA, USA © 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM. ISBN 978-1-4503-6816-2/19/10...$15.00 DOI: 10.1145/3332165.3347945

UX of AI & PersonalizationResearch Papers

Familiarisation: Restructuring Layouts with Visual Learning ModelsKashyap Todi · 2018In domains where users are exposed to large variations in visuo-spatial features among designs, they often spend excess time searching for common elements (features) in familiar locations. This paper contributes computational approaches to restructuring layouts such that features on a new, unvisited interface can be found quicker. We explore four concepts of fa- miliarisation, inspired by the human visual system (HVS), to automatically generate a familiar design for each user. Given a history of previously visited interfaces, we restructure the spatial layout of the new (unseen) interface with the goal of making its elements more easily found. Familiariser is a browser-based implementation that automatically restruc- tures webpage layouts based on the visual history of the user. Our evaluation with users provides first evidence favouring familiarisation. Author Keywords Visual search; Graphical layouts; Computational design; Adaptive user interfaces

UX of AI & PersonalizationResearch Papers

User Interface Design with Combinatorial Optimization1977ptimization methods in user interface (UI) design is a long-standing topic in human-computer interaction (HCI) research. Currently, user-centered design is largely focused on human creativity, sensemaking, empathy, and creation of meaning, but optimization methods have been explored as supplemental ways to help speed up the design cycle and improve design quality (for more information about computational UI design, see the sidebar). Unlike any other design method in HCI, optimization methods offer a greater-than-zero chance of finding an optimal design, and some exact methods even offer mathematical guarantees for solutions. More importantly, if the design task and its

UX of AI & PersonalizationResearch Papers