Bringing full-featured mobile phone interaction into virtual reality. (June 2021)

• Record Type: Journal Article
• Authors: Bai, Huidong; Zhang, Li; Yang, Jing; Billinghurst, Mark
• Is Part Of: Computers & graphics. Volume 97(2021)
• Page Start: 42
• Page End: 53
• Publication Date: 2021-06
• Subjects: Virtual reality -- Augmented virtuality -- Head-mounted displays -- Smartphone interaction -- Text input; Computer graphics -- Periodicals; 006.6
• DOI: 10.1016/j.cag.2021.04.004
• Languages: English
• ISSNs: 0097-8493

Abstract

Highlights: We develop an Augmented Virtuality (AV) system that brings the full-featured smartphone into a Virtual Reality (VR) environment. Using a mobile phone with our AV interfaces in VR was stressful than the regular smartphone touch interaction. We explored augmented hand visualization for the AV mobile touch input to improve the on-screen content interaction, especially in typing. A skin-colored hand visualization method provided better usability and immersiveness than other hand rendering styles. Graphical abstract: Abstract: Virtual Reality (VR) Head-Mounted Display (HMD) technology immerses a user in a computer generated virtual environment. However, a VR HMD also blocks the users' view of their physical surroundings, and so prevents them from using their mobile phones in a natural manner. In this paper, we present a novel Augmented Virtuality (AV) interface that enables people to naturally interact with a mobile phone in real time in a virtual environment. The system allows the user to wear a VR HMD while seeing his/her 3D hands captured by a depth sensor and rendered in different styles, and enables the user to operate a virtual mobile phone aligned with their real phone. We conducted a formal user study to compare the AV interface with physical touch interaction on user experience in five mobile applications. Participants reported that our system brought the real mobile phone into the virtual world. Unfortunately, the experiment results indicated that using a phone with our AV interfaces in VR was more difficult than the regular smartphone touch interaction, with increased workload and lower system usability, especially for a typing task. We ran a follow-up study to compare different hand visualizations for text typing using the AV interface. Participants felt that a skin-colored hand visualization method provided better usability and immersiveness than other hand rendering styles.