Is the suggested food your desired?: Multi-modal recipe recommendation with demand-based knowledge graph. (30th December 2021)

• Record Type: Journal Article
• Authors: Lei, Zhenfeng; Ul Haq, Anwar; Zeb, Adnan; Suzauddola, Md; Zhang, Defu
• Is Part Of: Expert systems with applications. Volume 186(2021)
• Publication Date: 2021-12-30
• Subjects: Recipe recommender -- Interpretation generation -- Multi-modality -- Knowledge graph -- BERT; Expert systems (Computer science) -- Periodicals; Systèmes experts (Informatique) -- Périodiques; Electronic journals; 006.33
• DOI: 10.1016/j.eswa.2021.115708
• Languages: English
• ISSNs: 0957-4174

Abstract

Abstract: Personalized recipe recommender systems help users mine certain dishes they want to find and even really desire, which play a significant role in matching dishes, balancing nutrients, and preventing non-communicable diseases. Generally, customer's preferences or needs vary from person to person, and people are often reluctant to accept recommended food without reasonable explanation, especially when their demands are not explicitly addressed. In this paper, we are devoted to providing recipe suggestions accompanied by rational interpretations generated from images or videos. First, we construct a recipe knowledge graph (RcpKG) through the use of multi-modality and hierarchical thought, which focuses on the underlying demands of users and the consideration of multiple fine-grained factors. On this basis, a novel multi-modal recipe recommendation method via the knowledge graph (RcpMKR) is proposed, which represents nodes in multiple aspects and performs multi-relational graph structure extraction of the RcpKG. It not only takes into account local associations within the graph but also global information, and incorporates user concerns at different levels. Then, we adopt BERT-based multi-modal models and generative adversarial networks to generate interpretations. Additionally, dynamic convolution and random synthetic attention are utilized in our work to discriminate among features. Experimental results show that the proposed method and BERT-based fusion models improve recipe recommendation performance and explanation generation. Specifically, the precision of the RcpMKR method through RcpKG, user concerns and graph convolutional network improves by 7.82%, and the viExpCBTBERT method via 2D&3D convolutional neural networks for developing text interpretations enhances the F1-score by 10% compared with the baseline. Highlights: Propose an extendable multi-modal recipe fusion framework based on knowledge graph. Constructing a recipe knowledge graph based on user preferences. Use multi-modality to improve recommendation results. Utilizing deep learning to generate explanations from multimedia data.