Integrating Apache Airavata with Docker, Marathon, and Mesos. (13th November 2015)

• Record Type: Journal Article
• Authors: Saha, Pankaj; Govindaraju, Madhusudhan; Marru, Suresh; Pierce, Marlon
• Other Names: Olabarriaga Silvia Delgado guestEditor.; Wilkins‐Diehr Nancy guestEditor.; Smari Waleed W. guestEditor.; Bakhouya Mohamed guestEditor.; Fiore Sandro guestEditor.; Aloisio Giovanni guestEditor.
• Is Part Of: Concurrency and computation. Volume 28:Number 7(2016)
• Page Start: 1952
• Page End: 1959
• Publication Date: 2015-11-13
• Subjects: Apache Airavata -- Distributed Systmes -- Mesos -- Docker -- Marathon; Parallel processing (Electronic computers) -- Periodicals; Parallel computers -- Periodicals; 004.35
• DOI: 10.1002/cpe.3708
• Languages: English
• ISSNs: 1532-0626

Abstract

Summary: Science Gateways provide scientists with tools for creating, executing, and monitoring scientific experiments on multiple resource infrastructures. Apache Airavata abstracts interactions between gateways and distributed computing infrastructures like Extreme Science and Engineering Discovery Environment, international grids, and campus clusters. Airavata consists of several component services such as the API server, Orchestrator, Workflow Interpreter, Credential Store, and Application Factory. In addition, Airavata uses third party software, including RabbbitMQ for messaging, MySQL for production database management, and Apache Zookeeper for internal communications. In this paper, we discuss our initial experiences with leveraging open source technologies to manage Airavata and its dependent components to deploy, detect, and restart failed components in an auto‐scaling platform. Such capabilities will allow Airavata services to be deployed in a wide area, large Virtual Machine (VM) based cluster, and a developer's laptop. The emerging technologies in cloud computing and Big Data that address these needs are the following: Docker, Marathon, and Apache Mesos. Docker is a Linux‐based lightweight container that allows different applications to run isolated from each other but safely share the machine's resources. Docker images of applications can be published in registries and retrieved for execution in the target infrastructures. Marathon provides a cluster‐wide init and control system for services, including Docker containers. Mesos provides a cluster‐wide framework to schedule tasks based on fine‐grained resource needs. Mesosphere provides the packages, scripts, and web interface to ease the use of these technologies. We present the design, experience, and lessons learned from integrating Mesos, Docker, and Marathon with Apache Airavata. Copyright © 2015 John Wiley & Sons, Ltd.