DynLW: balancing and scalability for heavy dynamic stream-DB workloads. (1st January 2014) - Martins, Pedro; Furtado, Pedro
In the past, data management research has concentrated in separate data processing issues: heavy database like query processing, and throughput of stream data processing over high-rate data (CEP). However, in many practical contexts, high-rate stream and heavy data processing work together, for correlation, lookup, aggregation, merging or comparison with large amounts of previous data. We refer to these as stream-DB workloads. One way to provide scalability with any off-the-shelf engine is to have multiple machines and/or processor cores, and to parallelise the load (external scheduler), but nodes can still overload. We propose automated control for balancing and scalability over stream-DB workloads. The approach, called DynLW, offers scalability with an integrated mechanism that manages overload (re)scheduling, automated elasticity, shedding, admission control and overload alerts when resources are insufficient. As a result, the approach provides continuous and totally balanced operation, and avoids overload-related problems.