Image2emmet: Automatic code generation from web user interface image. Issue 8 (28th June 2021)

• Record Type: Journal Article
• Authors: Xu, Yong; Bo, Lili; Sun, Xiaobing; Li, Bin; Jiang, Jing; Zhou, Wei
• Is Part Of: Journal of software. Volume 33:Issue 8(2021)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2021-06-28
• Subjects: code generation -- HTML‐CSS code -- UI component -- web development; Software engineering -- Periodicals; Computer software -- Development -- Periodicals; Software maintenance -- Periodicals; 005.1
• DOI: 10.1002/smr.2369
• Languages: English
• ISSNs: 2047-7473

Abstract

Abstract: Web development usually follows with analyzing the functionality, designing the user interface (UI) prototype, implementing the UI by front‐end (FE) developers and implementing the REpresentational State Transfer (RESTful) application programming interface (API) by back‐end (BE) programmers. Unfortunately, web development is a tedious, cumbersome, and time‐consuming task, which makes it a challenge for the FE programmers to work in an efficient way. In this paper, we propose an approach, image2emmet, to assist FE programmers in implementing the UI. First, we collect HyperText Markup Language, Cascading Style Sheets (HTML‐CSS) dataset in an automatic and efficient way. The HTML‐CSS dataset used for model training consists of HTML‐CSS code and its display images. Second, the faster region‐based convolutional neural network (CNN) (R‐CNN) is utilized to detect the UI component. Finally, we build a model combining CNN and long short‐term memory (LSTM) to transform the UI component into the HTML‐CSS code. The empirical study demonstrates that image2emmet can achieve a precision of 80% on the UI component detection and 60% on the transformation of UI component into HTML‐CSS code. Abstract : We collect HTML‐CSS dataset both with HTML‐CSS code and its display images in an automatic and efficient way. We utilize the deep learning object detection technique in computer vision (CV) to detect UI components from UI images, and the experiment shows that the UI components can be correctly detected at a precision of 80%. We directly transform UI components into HTML‐CSS code, and the experiment shows that the accuracy of code generation can achieve 60%.