Web technologies for environmental Big Data. (January 2015)

• Record Type: Journal Article
• Authors: Vitolo, Claudia; Elkhatib, Yehia; Reusser, Dominik; Macleod, Christopher J.A.; Buytaert, Wouter
• Is Part Of: Environmental modelling & software. Volume 63(2015:Jan.)
• Page Start: 185
• Page End: 198
• Publication Date: 2015-01
• Subjects: Web-based modelling -- Big Data -- Web services -- OGC standards; Environmental monitoring -- Computer programs -- Periodicals; Ecology -- Computer simulation -- Periodicals; Digital computer simulation -- Periodicals; Computer software -- Periodicals; Environmental Monitoring -- Periodicals; Computer Simulation -- Periodicals; Environnement -- Surveillance -- Logiciels -- Périodiques; Écologie -- Simulation, Méthodes de -- Périodiques; Simulation par ordinateur -- Périodiques; Logiciels -- Périodiques; Computer software; Digital computer simulation; Ecology -- Computer simulation; Environmental monitoring -- Computer programs; Periodicals; Electronic journals; 363.70015118
• DOI: 10.1016/j.envsoft.2014.10.007
• Languages: English
• ISSNs: 1364-8152

Abstract

Abstract: Recent evolutions in computing science and web technology provide the environmental community with continuously expanding resources for data collection and analysis that pose unprecedented challenges to the design of analysis methods, workflows, and interaction with data sets. In the light of the recent UK Research Council funded Environmental Virtual Observatory pilot project, this paper gives an overview of currently available implementations related to web-based technologies for processing large and heterogeneous datasets and discuss their relevance within the context of environmental data processing, simulation and prediction. We found that, the processing of the simple datasets used in the pilot proved to be relatively straightforward using a combination of R, RPy2, PyWPS and PostgreSQL. However, the use of NoSQL databases and more versatile frameworks such as OGC standard based implementations may provide a wider and more flexible set of features that particularly facilitate working with larger volumes and more heterogeneous data sources. Highlights: We review web service related technologies to manage, transfer and process Big Data. We examine international standards and related implementations. Many existing algorithms can be easily exposed as services and cloud-enabled. The adoption of standards facilitate the implementation of workflows. Use of web technologies to tackle environmental issues is acknowledged worldwide.