TU Berlin

Database Systems and Information Management GroupPublications

Logo FG DIMA-new  65px

Page Content

to Navigation

Publications

Analyzing Efficient Stream Processing on Modern Hardware
Citation key ZeuchBRdMKLuRTM2019
Author Steffen Zeuch, Sebastian Breß, Tilmann Rabl, Bonaventura Del Monte, Jeyhun Karimov, Clemens Lutz, Manuel Renz, Jonas Traub, Volker Markl
Pages 516-530
Year 2019
DOI /10.14778/3303753.3303758
Journal Proceedings of the VLDB Endowment
Volume 12(5)
Month January
Abstract Modern Stream Processing Engines (SPEs) process large data volumes under tight latency constraints. Many SPEs execute processing pipelines using message passing on shared-nothing architectures and apply a partition-based scale-out strategy to handle high-velocity input streams. Further-more, many state-of-the-art SPEs rely on a Java Virtual Ma-chine to achieve platform independence and speed up system development by abstracting from the underlying hardware. In this paper, we show that taking the underlying hard-ware into account is essential to exploit modern hardware efficiently. To this end, we conduct an extensive experimen-tal analysis of current SPEs and SPE design alternatives optimized for modern hardware. Our analysis highlights po-tential bottlenecks and reveals that state-of-the-art SPEs are not capable of fully exploiting current and emerging hard-ware trends, such as multi-core processors and high-speed networks. Based on our analysis, we describe a set of design changes to the common architecture of SPEs to scale-up on modern hardware. We show that the single-node throughput can be increased by up to two orders of magnitude compared to state-of-the-art SPEs by applying specialized code genera-tion, fusing operators, batch-style parallelization strategies, and optimized windowing. This speedup allows for deploy-ing typical streaming applications on a single or a few nodes instead of large clusters.
Link to original publication Download Bibtex entry

Navigation

Quick Access

Schnellnavigation zur Seite über Nummerneingabe