Haupt-Navigation

Inhalt des Dokuments

Publications

Analyzing Efficient Stream Processing on Modern Hardware
Citation key	DBLP:journals/pvldb/ZeuchBRMKLRTM19
Author	Steffen Zeuch, Sebastian Breß, Tilmann Rabl, Bonaventura Del Monte, Jeyhun Karimov, Clemens Lutz, Manuel Renz, Jonas Traub, Volker Markl
Pages	516-530
Year	2019
DOI	/10.14778/3303753.3303758
Journal	Proceedings of the VLDB Endowment
Volume	12(5)
Month	January
Abstract	Modern Stream Processing Engines (SPEs) process large data volumes under tight latency constraints. Many SPEs execute processing pipelines using message passing on shared-nothing architectures and apply a partition-based scale-out strategy to handle high-velocity input streams. Further-more, many state-of-the-art SPEs rely on a Java Virtual Ma-chine to achieve platform independence and speed up system development by abstracting from the underlying hardware. In this paper, we show that taking the underlying hard-ware into account is essential to exploit modern hardware eﬃciently. To this end, we conduct an extensive experimen-tal analysis of current SPEs and SPE design alternatives optimized for modern hardware. Our analysis highlights po-tential bottlenecks and reveals that state-of-the-art SPEs are not capable of fully exploiting current and emerging hard-ware trends, such as multi-core processors and high-speed networks. Based on our analysis, we describe a set of design changes to the common architecture of SPEs to scale-up on modern hardware. We show that the single-node throughput can be increased by up to two orders of magnitude compared to state-of-the-art SPEs by applying specialized code genera-tion, fusing operators, batch-style parallelization strategies, and optimized windowing. This speedup allows for deploy-ing typical streaming applications on a single or a few nodes instead of large clusters.