Transactions across serverless functions leveraging stateful dataflows. Issue 108 (September 2022)

• Record Type: Journal Article
• Authors: de Heus, Martijn; Psarakis, Kyriakos; Fragkoulis, Marios; Katsifodimos, Asterios
• Is Part Of: Information systems. Issue 108(2022)
• Publication Date: 2022-09
• Subjects: Serverless -- Transactions -- FaaS -- Two-phase commit -- Sagas -- Streaming dataflows; Database management -- Periodicals; Electronic data processing -- Periodicals; Bases de données -- Gestion -- Périodiques; Informatique -- Périodiques; Database management; Electronic data processing; Periodicals; 005.7
• DOI: 10.1016/j.is.2022.102015
• Languages: English
• ISSNs: 0306-4379

Abstract

Abstract: Serverless computing is currently the fastest-growing cloud services segment. The most prominent serverless offering is Function-as-a-Service (FaaS), where users write functions and the cloud automates deployment, maintenance, and scalability. Although FaaS is a good fit for executing stateless functions, it does not adequately support stateful constructs like microservices and scalable, low-latency cloud applications. Recently, there have been multiple attempts to add first-class support for state in FaaS systems, such as Microsoft Orleans, Azure Durable Functions, or Beldi. These approaches execute business code inside stateless functions, handing over their state to an external database. In contrast, approaches such as Apache Flink's StateFun follow a different design: a dataflow system such as Apache Flink handles all state management, messaging, and checkpointing by executing a stateful dataflow graph providing exactly-once state processing guarantees. This design relieves programmers from having to "pollute" their business logic with distributed systems error checking, management, and mitigation. Although programmers can easily develop applications without worrying about messaging and state management, executing transactions across stateful functions remains an open problem. In this paper, we introduce a programming model and implementation for transaction orchestration of stateful serverless functions. Our programming model supports serializable distributed transactions with two-phase commit, as well as eventually consistent workflows with Sagas. We design and implement our programming model on Apache Flink StateFun to leverage Flink's exactly-once processing and state management guarantees. Our experiments show that the approach of building transactional systems on top of dataflow graphs can achieve very high throughput, but with latency overhead due to checkpointing mechanism guaranteeing the exactly-once processing. We compare our approach to Beldi that implements two-phase commit on AWS lambda functions backed by DynamoDB for state management, as well as an implementation of a system that makes use of CockroachDB as its backend. Highlights: Dataflow systems can serve as execution engines for cloud applications. Transactional guarantees can be implemented on stateful functions. Implementing transaction protocols is very simple under exactly-once processing guarantees. Further work is required for dataflow systems to reach their full potential as backends for cloud applications. The performance of transactions on stateful functions is comparable to the one of distributed databases.