Data transformation as a means towards dynamic data storage and polyglot persistence. Issue 4 (4th May 2017)

• Record Type: Journal Article
• Authors: Vanhove, Thomas; Sebrechts, Merlijn; Van Seghbroeck, Gregory; Wauters, Tim; Volckaert, Bruno; De Turck, Filip; Diao, Yixin; Zincir‐Heywood, A. Nur
• Is Part Of: International journal of network management. Volume 27:Issue 4(2017)
• Page Start: n/a
• Page End: n/a
• Publication Date: 2017-05-04
• Subjects: big data -- data transformation -- dynamic data storage -- lambda architecture -- polyglot persistence; Computer networks -- Management -- Periodicals; 004.6
• DOI: 10.1002/nem.1976
• Languages: English
• ISSNs: 1055-7148

Abstract

Summary: Legacy applications have been built around the concept of storing their data in one relational data store. However, with the current differentiation in data store technologies as a consequence of the NoSQL paradigm, new and possibly more performant storage solutions are available to all applications. The concept of dynamic storage makes sure that application data are always stored in the most optimal data store at a given time to increase application performance. Additionally, polyglot persistence aims to push this performance even further by storing each different data type of an application in the data store technology best suited for it. To get legacy applications into dynamic storage and polyglot persistence, schema and data transformations between data store technologies are needed. This usually infers application redesigns as well to support the new data stores. This paper proposes such a transformation approach through a canonical model. It is based on the Lambda architecture to ensure no application downtime is needed during the transformation process, and after the transformation, the application can continue to query in the original query language, thus requiring no application code changes. Abstract : Big data sets are often stored in suboptimal data store technologies, but data transformations is are costly operations. To that end, this article introduces a transformation algorithm through a centralized canonical model without the need for data store or application downtime. The transformation optimizes the representation of the data for each specific technology ultimately aiming at decreasing query latency for applications. Results show data sets can be correctly transformed and optimized within a reasonable time frame effectively increasing application performance.