Scanflow: A multi-graph framework for Machine Learning workflow management, supervision, and debugging. (15th September 2022)

• Record Type: Journal Article
• Authors: Bravo-Rocca, Gusseppe; Liu, Peini; Guitart, Jordi; Dholakia, Ajay; Ellison, David; Falkanger, Jeffrey; Hodak, Miroslav
• Is Part Of: Expert systems with applications. Volume 202(2022)
• Publication Date: 2022-09-15
• Subjects: Machine Learning -- Symbolic knowledge -- Graph -- Robustness -- Containerization -- Concept drift; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2022.117232
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Machine Learning (ML) is more than just training models, the whole workflow must be considered. Once deployed, a ML model needs to be watched and constantly supervised and debugged to guarantee its validity and robustness in unexpected situations. Debugging in ML aims to identify (and address) the model weaknesses in not trivial contexts. Several techniques have been proposed to identify different types of model weaknesses, such as bias in classification, model decay, adversarial attacks, etc., yet there is not a generic framework that allows them to work in a collaborative, modular, portable, iterative way and, more importantly, flexible enough to allow both human- and machine-driven techniques. In this paper, we propose a novel containerized directed graph framework to support and accelerate end-to-end ML workflow management, supervision, and debugging. The framework allows defining and deploying ML workflows in containers, tracking their metadata, checking their behavior in production, and improving the models by using both learned and human-provided knowledge. We demonstrate these capabilities by integrating in the framework two hybrid systems to detect data drift distribution which identify the samples that are far from the latent space of the original distribution, ask for human intervention, and whether retrain the model or wrap it with a filter to remove the noise of corrupted data at inference time. We test these systems on MNIST-C, CIFAR-10-C, and FashionMNIST-C datasets, obtaining promising accuracy results with the help of human involvement. Highlights: Machine Learning is more than just training, the whole view must be considered. Security vulnerabilities, concept/data drift and lack of explainability are issues. Model needs to be supervised and debugged to guarantee its validity and robustness. A hybrid symbolic (human knowledge) and learning (abnormal behavior) system is needed.