A rule-based automated machine learning approach in the evaluation of recommender engine. Issue 10 (14th August 2020)

• Record Type: Journal Article
• Authors: Behera, Rajat Kumar; Bala, Pradip Kumar; Jain, Rashmi
• Is Part Of: Benchmarking. Volume 27:Issue 10(2020)
• Page Start: 2721
• Page End: 2757
• Publication Date: 2020-08-14
• Subjects: Recommendation system -- Recommender engine -- Automated recommender engine evaluation -- Recommender engine evaluation metrics -- Machine learning in software testing -- Personalization; Benchmarking (Management) -- Periodicals; Total quality management -- Periodicals; 658.562
• DOI: 10.1108/BIJ-01-2020-0051
• Languages: English
• ISSNs: 1463-5771

Abstract

Abstract : Purpose: Any business that opts to adopt a recommender engine (RE) for various potential benefits must choose from the candidate solutions, by matching to the task of interest and domain. The purpose of this paper is to choose RE that fits best from a set of candidate solutions using rule-based automated machine learning (ML) approach. The objective is to draw trustworthy conclusion, which results in brand building, and establishing a reliable relation with customers and undeniably to grow the business. Design/methodology/approach: An experimental quantitative research method was conducted in which the ML model was evaluated with diversified performance metrics and five RE algorithms by combining offline evaluation on historical and simulated movie data set, and the online evaluation on business-alike near-real-time data set to uncover the best-fitting RE. Findings: The rule-based automated evaluation of RE has changed the testing landscape, with the removal of longer duration of manual testing and not being comprehensive. It leads to minimal manual effort with high-quality results and can possibly bring a new revolution in the testing practice to start a service line "Machine Learning Testing as a service" (MLTaaS) and the possibility of integrating with DevOps that can specifically help agile team to ship a fail-safe RE evaluation product targeting SaaS (software as a service) or cloud deployment. Research limitations/implications: A small data set was considered for A/B phase study and was captured for ten movies from three theaters operating in a single location in India, and simulation phase study was captured for two movies from three theaters operating from the same location in India. The research was limited to Bollywood and Ollywood movies for A/B phase, and Ollywood movies for simulation phase. Practical implications: The best-fitting RE facilitates the business to make personalized recommendations, long-term customer loyalty forecasting, predicting the company's future performance, introducing customers to new products/services and shaping customer's future preferences and behaviors. Originality/value: The proposed rule-based ML approach named "2-stage locking evaluation" is self-learned, automated by design and largely produces time-bound conclusive result and improved decision-making process. It is the first of a kind to examine the business domain and task of interest. In each stage of the evaluation, low-performer REs are excluded which leads to time-optimized and cost-optimized solution. Additionally, the combination of offline and online evaluation methods offer benefits, such as improved quality with self-learning algorithm, faster time to decision-making by significantly reducing manual efforts with end-to-end test coverage, cognitive aiding for early feedback and unattended evaluation and traceability by identifying the missing test metrics coverage.