Integration and Analysis of Agricultural Market Information Based on Web Mining. Issue 17 (2018)

• Record Type: Journal Article
• Authors: Zhou, Jianghui; Cheng, Chunming; Kang, Li; Sun, Ruizhi
• Is Part Of: IFAC-PapersOnLine. Volume 51:Issue 17(2018)
• Page Start: 778
• Page End: 783
• Publication Date: 2018
• Subjects: Agricultural big data -- Decision support -- Web crawler -- Web mining -- Data visualization; Automatic control -- Periodicals; 629.805
• DOI: 10.1016/j.ifacol.2018.08.101
• Languages: English
• ISSNs: 2405-8963

Abstract

Abstract: Agricultural big data can be used to guide agricultural production, forecast agricultural market demands, and support agricultural decisions. How to effectively extract and use the information on the Internet, which contains a large amount of agricultural information, has become a huge challenge. This paper proposes three kinds of automatic data acquisition strategies based on (focused, incremental, custom) Web crawler technologies, which are better suited to different types of agricultural websites than traditional Web crawlers. In addition to solving asynchronous processing, dynamic page rendering, distribution, and data-persistent problems encountered during data acquisition, this paper also proposes to combine the Aho-Corasick algorithm to improve the text matching efficiency. Finally, the acquired agricultural market data was visually analyzed by using key technologies of Web mining. This study takes Chinese agricultural official websites, agricultural products wholesale market websites, and e-commerce websites as examples to integrate, process, visualize, and analyze the data acquired by using the three automatic data acquisition strategies proposed in this paper.