Integrated multimodal interaction using texture representations. (April 2016)

• Record Type: Journal Article
• Authors: Sterling, Auston; Lin, Ming C.
• Is Part Of: Computers & graphics. Volume 55(2016)
• Page Start: 118
• Page End: 129
• Publication Date: 2016-04
• Subjects: Multimodal interaction -- Texture mapping -- Perception -- Haptics -- Physics -- Sound synthesis; Computer graphics -- Periodicals; 006.6
• DOI: 10.1016/j.cag.2015.10.010
• Languages: English
• ISSNs: 0097-8493

Abstract

Abstract: In this paper, we explore texture mapping as a unified representation for enabling realistic multimodal interaction with finely detailed surfaces. We show how both normal maps and relief maps can be adopted as unified representations to handle collisions with textured rigid body objects, synthesize complex sound effects from long lasting collisions and perform rendering of haptic textures. The resulting multimodal display system allows a user to see, hear, and feel complex interactions with textured surfaces. By using texture representations for seamlessly integrated multimodal interaction instead of complex triangular meshes otherwise required, this work is able to achieve up to 25 times performance speedup and reduce up to six orders of magnitude in memory storage. We further validate the results through user studies to demonstrate the effectiveness of texture representations for integrated multimodal display. Abstract : Graphical abstract: Abstract : Highlights: We present a system displaying fine details of visuals, haptics, and audio. We present a new method for rigid-body physics on normal-mapped surfaces. Multimodal interaction with texture maps reduces sensory conflict. Users prefer multimodal relief map interaction over normal maps.