Scale out databases for CERN use cases. Issue 4 (December 2015)

• Record Type: Journal Article
• Authors: Baranowski, Zbigniew; Grzybek, Maciej; Canali, Luca; Garcia, Daniel Lanza; Surdy, Kacper
• Is Part Of: Journal of physics. Volume 664:Issue 4(2015)
• Publication Date: 2015-12
• Subjects: Physics -- Congresses; 530.5
• DOI: 10.1088/1742-6596/664/4/042002
• Languages: English
• ISSNs: 1742-6588

Abstract

Abstract: Data generation rates are expected to grow very fast for some database workloads going into LHC run 2 and beyond. In particular this is expected for data coming from controls, logging and monitoring systems. Storing, administering and accessing big data sets in a relational database system can quickly become a very hard technical challenge, as the size of the active data set and the number of concurrent users increase. Scale-out database technologies are a rapidly developing set of solutions for deploying and managing very large data warehouses on commodity hardware and with open source software. In this paper we will describe the architecture and tests on database systems based on Hadoop and the Cloudera Impala engine. We will discuss the results of our tests, including tests of data loading and integration with existing data sources and in particular with relational databases. We will report on query performance tests done with various data sets of interest at CERN, notably data from the accelerator log database.