Research on Key Technologies of E-commerce Big Data Analysis Platform. Issue 5 (March 2019)

• Record Type: Journal Article
• Authors: Maalla, Allam; Jia, Qiwei
• Is Part Of: IOP conference series. Volume 242:Issue 5(2019)
• Publication Date: 2019-03
• Subjects: Earth sciences -- Periodicals; Environmental sciences -- Congresses; Environmental sciences -- Periodicals; 550.5
• DOI: 10.1088/1755-1315/242/5/052038
• Languages: English
• ISSNs: 1755-1307

Abstract

Abstract: This paper designs and implements the E-commerce big data analysis platform, which is mainly based on the data of commodities and sales of E-commerce sales platforms, providing a platform for intelligent analysis. The platform provides an interface for users to operate. It can efficiently obtain the necessary business information for e-commerce users and provide decision support. The system can be divided into four parts: crawler system, storage system, offline data analysis system and user interaction system. The storage system adopts the Hadoop ecosystem. Hadoop's HDFS are highly fault-tolerant, suitable for processing GB, TB, or even PB-level data, and can be expanded horizontally, and deployed on multiple inexpensive machines, the platform is Hadoop-based. HBase and Hive will handle and use data more efficiently. The offline analysis system adopts the Spark framework. That Spark is based on RDD (Elastic Distributed) set data processing. It can connect various data sources of HBase and Hive. Because the data are loaded into the memory of the cluster host, and quickly iterated, it is suitable for multiple rounds of computing tasks such as machine learning. Spark-SQL can provide SQL-like operations on structured or semi-structured data, which can greatly improve the efficiency of off-line data analysis.