Inhalt des Dokuments
Publications
Citation key | DBLP:journals/pvldb/ZeuchBRMKLRTM19 |
---|---|
Author | Steffen Zeuch, Sebastian Breß, Tilmann Rabl, Bonaventura Del Monte, Jeyhun Karimov, Clemens Lutz, Manuel Renz, Jonas Traub, Volker Markl |
Pages | 516-530 |
Year | 2019 |
DOI | /10.14778/3303753.3303758 |
Journal | Proceedings of the VLDB Endowment |
Volume | 12(5) |
Month | January |
Abstract | Modern Stream Processing Engines (SPEs) process large data volumes under tight latency constraints. Many SPEs execute processing pipelines using message passing on shared-nothing architectures and apply a partition-based scale-out strategy to handle high-velocity input streams. Further-more, many state-of-the-art SPEs rely on a Java Virtual Ma-chine to achieve platform independence and speed up system development by abstracting from the underlying hardware. In this paper, we show that taking the underlying hard-ware into account is essential to exploit modern hardware efficiently. To this end, we conduct an extensive experimen-tal analysis of current SPEs and SPE design alternatives optimized for modern hardware. Our analysis highlights po-tential bottlenecks and reveals that state-of-the-art SPEs are not capable of fully exploiting current and emerging hard-ware trends, such as multi-core processors and high-speed networks. Based on our analysis, we describe a set of design changes to the common architecture of SPEs to scale-up on modern hardware. We show that the single-node throughput can be increased by up to two orders of magnitude compared to state-of-the-art SPEs by applying specialized code genera-tion, fusing operators, batch-style parallelization strategies, and optimized windowing. This speedup allows for deploy-ing typical streaming applications on a single or a few nodes instead of large clusters. |
Zusatzinformationen / Extras
Quick Access:
Schnellnavigation zur Seite über Nummerneingabe