DBT: Scalable Data Processing
Format and Credit: Lecture Series with Exercises (VL+UE, 3+1 SWS, 6 ECTS)
Goals: The global data volume is increasing
dramatically each year. Understanding how to store, process and manage
these huge amounts of data efficiently is a key requirement for
software engineers and data analysts in the modern IT world. This
course will teach students both the fundamentals of data processing in
traditional single-node database systems and how to scale out these
techniques to huge amounts of data in large-scale, distributed
The lecture is accompanied by an optional implementation project, in which students will get hands-on experience with the main data processing techniques, by implementing several components of a relational database system and by using parallel programming platforms like Apache Hadoop or Nephele/PACT.
Audience: This course is the base course for master students with focus on database systems and information management and should be attended in the first semester of the master program. In contrast to the introduction of database systems (MPGI5/DBS), which looks database systems from an application programmers point of view, this class focuses on the internals of database systems. To participate, students are required to have successfully completed a Bachelor in computer science with a focus on database systems (participation in the Datenbankpraktikum, Datenbankprojekt). Knowledge of data modeling, relational algebra, and SQL as well as a very good command of Java, or possibly C/C++/C#, programming is required to participate in the course. Due to capacity reasons, the class is limited to at most 60 participants.
Content: The course is split into two parts, each
covering roughly one half of the semester. During the first part, the
students become acquainted with the fundamentals of query processing
in traditional relational database systems. This includes the general
architecture of a DBMS, file- & buffer management, query
processing, indexing, metadata management, query optimization,
locking, recovery and transaction management. In the second half of
the lecture, the lecture will cover the basics of parallel data
processing, with a focus on large-scale, distributed systems and
“cloud computing”. Topics include parallel processing platforms
like MapReduce, distributed data storage and retrieval – e.g., via
DHTs –, techniques for distributed locking and transaction handling,
multi-tenancy and software as a service. The course consists of a
lecture and theoretical, written exercises.
In the optional project, students will implement components of a relational database system and get hands-on experience with a parallel data processing platform. The actual components implemented may vary each year, but will include parsing, query optimizer, execution engine, index structures and storage system.
- Successful completion of all theoretical exercises
- Final Exam
 Hector Garcia-Molina, Jeffrey D. Ullman, Jennifer Widom: Database Systems - The Complete Book, Pearson Education International, 2002.
 Garcia-Molina, Ullman, Widom: “Database Systems: The Complete Book,” Prentice Hall, 2000