Mixing effects of SEM imaging conditions on convolutional neural network-based low-carbon steel classification. (August 2022)

• Record Type: Journal Article
• Authors: Tsutsui, Kazumasa; Matsumoto, Kazushi; Maeda, Masaki; Takatsu, Terusato; Moriguchi, Koji; Hayashi, Kohtaro; Morito, Shigekazu; Terasaki, Hidenori
• Is Part Of: Materials today communications. Volume 32(2022)
• Publication Date: 2022-08
• Subjects: Convolutional neural network -- ResNet50 -- Low-carbon steel -- Scanning electron microscopy -- Bainite -- Martensite; Materials science -- Periodicals; 620.11
• DOI: 10.1016/j.mtcomm.2022.104062
• Languages: English
• ISSNs: 2352-4928

Abstract

Abstract: In this study, we address a practical issue encountered in the identification of in situ steel microstructures using convolutional neural networks (CNNs) and propose a possible method for managing it. Traditionally, to identify steel microstructures, micrographs obtained from optical microscopy (OM) and scanning electron microscopy (SEM) are used. Micrograph textures are known to be affected by imaging conditions, such as polishing, etching, and probe diameter. Hence, acquiring several micrographs under unified conditions from multiple samples and/or regions is challenging. When attempting to build large datasets that are trainable for CNN models, micrographs from different conditions mix between classes as well as within each class. In extreme cases, the imaging conditions of the training images will not correspond to those of the test data. Such a bias of the imaging conditions in the dataset, in other words "inherent imbalance in each class, " can be a non-negligible factor for the impaired performance of CNN models. Here, we examined the effect of inherent imbalance in each class on the performance using ResNet50, a well-known CNN model. As the classification target, we selected SEM images of low-carbon steel with identical compositions but implemented eight different heat treatments. Subsequently, we constructed two datasets based on emission electron sources: field emission (FE) and tungsten (W). Then, we mixed these datasets at several ratios and trained them on independent ResNet50s to assess the changes in the accuracy of their microstructural classification. As a result, the average accuracies were more than 0.98 in the cases of equal mixing of both sources. In contrast, when ResNet50 was trained using images from only one SEM source, the average accuracy decreased to approximately 0.64. In particular, the accuracies were assessed only from the images of one SEM source that excluded the training data mark less than 0.28. Moreover, to detect the inherent imbalance in each class in the dataset mixed with different imaging conditions, we investigated the identification of the FE and W sources by combining textural features based on the gray-level co-occurrence matrix (GLCM) method and five traditional machine learning algorithms, such as the support vector machine (SVM), k -nearest neighbor, random forest, gradient boosting machine, and multi-layer perceptron (MLP). SVM and MLP showed better performance than the others. Specifically, when only three categories of microstructural images in the dataset ( i.e ., upper bainite, lower bainite, and martensite) were trained, the SVM and MLP classifiers identified the SEM sources for the images of the microstructures belonging to the other five categories with accuracies of more than 0.96 and 0.91, respectively. Graphical Abstract: ga1 Highlights: ･CNN-based classifications of low-carbon steels are demonstrated. ･Two types of SEM sources are used and performances are compared at mixing. ･The CNN classifications are compared by tSNE-based dimensional reductions. ･Classification of the two sources by traditional ML classifiers is also attempted.