DL-Store: A Distributed Hybrid OLTP and OLAP Data Processing Engine

Kaiwen Zhang, Mohammad Sadoghi, and Hans-Arno Jacobsen.

In ICDCS Demos, 2016.

Abstract

There has been a recent push in the database community towards supporting real-time analytical queries (OLAP) while sustaining a large volume of fine-grained updates (OLTP). Supporting these types of workloads require both an efficient data storage layer as well as a distributed architecture. In this demo, we address the latter point with our Distributed Lineage-based Data Store (DL-Store), which is a distributed data processing engine. DL-Store is built on top of L-store, which is a lineage-based storage architecture designed to handle mixed OLTP and OLAP workloads, and provides scalability and elasticity by supporting multiple L-Store nodes. To maintain the desired consistency semantics, DL-Store employs a distributed transaction handler component which can horizontally scaled by provisioning additional transaction manager nodes. We leverage partitioning in the record space of the transactions to minimize communication across transaction managers while ensuring consistent execution. The demo shows our implementation of DL-Store over Apache Spark using a variety of use cases.

Download


Tags: olap/oltp, data store, distributed transactions, spark


Readers who enjoyed the above work, may also like the following:


  • Grand Challenge: High Performance Stream Queries in Scala.
    Dantong Song, Kaiwen Zhang, Tilmann Rabl, Prashanth Menon, and Hans-Arno Jacobsen.
    In DEBS, 2015.
    Tags: grand challenge, spark, scala, taxi monitoring
  • MADES - A Multi-Layered, Adaptive, Distributed Event Store.
    Tilmann Rabl, Mohammad Sadoghi, Kaiwen Zhang, and Hans-Arno Jacobsen.
    In Proceedings of the 7th ACM International Conference on Distributed Event-Based Systems, 2013.
  • Grand Challenge: The BlueBay Soccer Monitoring Engine.
    Hans-Arno Jacobsen, Kianoosh Mokhtarian, Tilmann Rabl, Mohammad Sadoghi, Reza Sherafat Kazemzadeh, Young Yoon, and Kaiwen Zhang.
    In The 7th ACM International Conference on Distributed Event-Based Systems, 2013.
    Acceptance rate 40%.
    Tags: event processing systems, live monitoring, soccer, content-based matching