AGNet: Automatic generation network for skin imaging reports. (February 2022)

• Record Type: Journal Article
• Authors: Wu, Fan; Yang, Haiqiong; Peng, Linlin; Lian, Zongkai; Li, Mingxin; Qu, Gang; Jiang, Shancheng; Han, Yu
• Is Part Of: Computers in biology and medicine. Volume 141(2022)
• Publication Date: 2022-02
• Subjects: Attention mechanism -- Deep learning -- Image caption -- Medical imaging -- Skin imaging report generation; Medicine -- Data processing -- Periodicals; Biology -- Data processing -- Periodicals; 610.285
• DOI: 10.1016/j.compbiomed.2021.105037
• Languages: English
• ISSNs: 0010-4825

Abstract

Abstract: Medical imaging has been increasingly adopted in the process of medical diagnosis, especially for skin diseases, where diagnoses based on skin pathology are extremely accurate. The diagnostic reports of skin pathology images has the distinguishing features of extreme repetitiveness and rigid formatting. However, reports written by inexperienced radiologists and pathologists can have a high error rate, and even experienced clinicians can find the reporting task both tedious and time-consuming. To address this challenge, this paper studies the automatic generation of diagnostic reports based on images of skin pathologies. A novel deep learning-based image caption framework named the automatic generation network (AGNet), which is an effective network for the automatic generation of skin imaging reports, is proposed. The proposed AGNet consists of four parts: (1) the image model that extracts features and classifies images; (2) the language model that codes data and generates words using comprehensible language; (3) the attention module that connects the "tail" of the image model and the "head" of the language model, and computes the relationship between images and captions; (4) the embedding and labeling module that processes the input caption data. In case study, The AGNet is verified on a skin pathological image dataset and compared with several state-of-the-art models. The results show that the AGNet achieves the highest scores of the evaluation metrics of image caption among all comparison models, demonstrating the promising performance of the proposed method. Highlights: This research focuses on the automatic generation of reports based on images of skin pathologies. There are three principal challenges when generating such reports: accuracy, efficiency, and visualization. We propose AGNet, a novel and joint deep learning model to solve above challenges. For the attention module, we propose a "similarity-attention" mechanism.