Yifei Cheng∗
We present an optimization-based approach that automatically adapts Mixed Reality (MR) interfaces to different physical envi- ronments. Current MR layouts, including the position and scale of virtual interface elements, need to be manually adapted by users ∗This work was done while Yifei Cheng was an intern at Tsinghua University. †The corresponding author. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. UIST’21, October 2021, Virtual © 2021 Association for Computing Machinery. ACM ISBN 978-1-4503-8635-7/21/10...$15.00 https://doi.org/10.1145/3472749.3474750 whenever they move between environments, and whenever they switch tasks. This process is tedious and time consuming, and ar- guably needs to be automated for MR systems to be beneficial for end users. We contribute an approach that formulates this challenge as a combinatorial optimization problem and automatically decides the placement of virtual interface elements in new environments. To achieve this, we exploit the semantic association between the virtual interface elements and physical objects in an environment. Our optimization furthermore considers the utility of elements for users’ current task, layout factors, and spatio-temporal consistency to previous layouts. All those factors are combined in a single linear program, which is used to adapt the layout of MR interfaces in real time. We demonstrate a set of application scenarios, showcasing the versatility and applicability of our approach. Finally, we show that compared to a naive adaptive baseline approach that does not 282 UIST’21, October 2021, Virtual Cheng and Yan, et al. take semantic associations into account, our approach decreased the number of manual interface adaptations by 33%. CCS CONCEPTS • Human-centered computing →Mixed / augmented reality; Virtual reality; User interface management systems.