Dark‐Mode Human–Machine Communication Realized by Persistent Luminescence and Deep Learning. (29th April 2022)

• Record Type: Journal Article
• Authors: Timilsina, Suman; Shin, Ho Geun; Sohn, Kee-Sun; Kim, Ji Sik
• Is Part Of: Advanced intelligent systems. Volume 4:Number 7(2022)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2022-04-29
• Subjects: dark-mode human–machine communication -- deep learning -- human–machine communication -- luminescence -- visual speech recognition; Artificial intelligence -- Periodicals; Robotics -- Periodicals; Control theory -- Periodicals; 006.3
• DOI: 10.1002/aisy.202200036
• Languages: English
• ISSNs: 2640-4567

Abstract

Abstract : Increasing ubiquitous collaborative intelligence between humans and machines requires human–machine communication (HMC) that is more human and less machine‐like to accomplish given tasks. Although speech signals are considered the best modes of communication in HMC, background noise often interferes with these signals. Therefore, research focused on integrating lip‐reading technology into HMC has gained significant attention. However, lip‐reading functions effectively only in well‐lit environments. In contrast, HMC may occur daily in dark environments owing to potential energy shortages, increased exploration in darkness, nighttime emergencies, etc. Herein, a possible method for HMC in the dark mode is presented, which is realized based on deep learning motion patterns of persistent luminescence (PL) of the skin surrounding the lips. An ultrasoft PL–polymer composite patch is used to record the motion pattern of the skin during speech in the dark. It is found that visual geometric group network (VGGNET‐5) and residual neural network (ResNet‐34) could predict spoken words in darkness with test accuracies of 98.5% and 98.75%, respectively. Furthermore, these models could effectively distinguish similar‐sounding words such as "around" and "ground." Dark‐mode communication can allow a wide range of people, including disabled people with limited dexterity and voice tremors, to communicate with artificial intelligence machines. Abstract : A possible method for human‐machine communication in the dark can be realized based on deep learning motion patterns of persistent luminescence of the skin surrounding the lips. It is found that deep convolutional neural networks can predict spoken words in the darkness with ≈98.5% accuracy. Dark‐mode communication can be useful for all people seeking to communicate with artificial intelligence machines.