TU Berlin

Database Systems and Information Management GroupPublications

Logo FG DIMA-new  65px

Page Content

to Navigation


Dynamic Parameter Allocation in Parameter Servers
Citation key DBLP:journals/pvldb/Renz-WielandGZM20
Author Alexander Renz-Wieland, Rainer Gemulla, Steffen Zeuch, Volker Markl
Year 2020
Journal Proceedings of the VLDB Endowment
Volume Volume 13
Number 11
Note A recording of the presentation is available here: https://www.youtube.com/watch?v=aMSjPW8Dmc0

Presentation slides are available here: https://www.user.tu-berlin.de/alexrenz/pub/2020-dpa-slides.pdf
Abstract To keep up with increasing dataset sizes and model complexity, distributed training has become a necessity for large machine learning tasks. Parameter servers ease the implementation of distributed parameter management–-a key concern in distributed training–-, but can induce severe communication overhead. To reduce communication overhead, distributed machine learning algorithms use techniques to increase parameter access locality (PAL), achieving up to linear speed-ups. We found that existing parameter servers provide only limited support for PAL techniques, however, and therefore prevent efficient training. In this paper, we explore whether and to what extent PAL techniques can be supported, and whether such support is beneficial. We propose to integrate dynamic parameter allocation into parameter servers, describe an efficient implementation of such a parameter server called Lapse, and experimentally compare its performance to existing parameter servers across a number of machine learning tasks. We found that Lapse provides near linear scaling and can be orders of magnitude faster than existing parameter servers.
Link to original publication Download Bibtex entry


Quick Access

Schnellnavigation zur Seite über Nummerneingabe