IceProd 2: A Next Generation Data Analysis Framework for the IceCube Neutrino Observatory. Issue 6 (December 2015)

• Record Type: Journal Article
• Authors: Schultz, D
• Is Part Of: Journal of physics. Volume 664:Issue 6(2015)
• Publication Date: 2015-12
• Subjects: Physics -- Congresses; 530.5
• DOI: 10.1088/1742-6596/664/6/062056
• Languages: English
• ISSNs: 1742-6588

Abstract

Abstract: We describe the overall structure and new features of the second generation of IceProd, a data processing and management framework. IceProd was developed by the IceCube Neutrino Observatory for processing of Monte Carlo simulations, detector data, and analysis levels. It runs as a separate layer on top of grid and batch systems. This is accomplished by a set of daemons which process job workflow, maintaining configuration and status information on the job before, during, and after processing. IceProd can also manage complex workflow DAGs across distributed computing grids in order to optimize usage of resources. IceProd is designed to be very light-weight; it runs as a python application fully in user space and can be set up easily. For the initial completion of this second version of IceProd, improvements have been made to increase security, reliability, scalability, and ease of use.