Lightning fast and space efficient inequality joins

Zuhair Khayyat, William Lucia, Meghna Singh, Mourad Ouzzani, Paolo Papotti, Jorge-Arnulfo Quiané-Ruiz, Nan Tang, Panos Kalnis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

31 Scopus citations

Abstract

Inequality joins, which join relational tables on inequality conditions, are used in various applications. While there have been a wide range of optimization methods for joins in database systems, from algorithms such as sort-merge join and band join, to various indices such as B+-tree, R*-tree and Bitmap, inequality joins have received little attention and queries containing such joins are usually very slow. In this paper, we introduce fast inequality join algorithms. We put columns to be joined in sorted arrays and we use permutation arrays to encode positions of tuples in one sorted array w.r.t. the other sorted array. In contrast to sort-merge join, we use space efficient bit-arrays that enable optimizations, such as Bloom filter indices, for fast computation of the join results. We have implemented a centralized version of these algorithms on top of PostgreSQL, and a distributed version on top of Spark SQL. We have compared against well known optimization techniques for inequality joins and show that our solution is more scalable and several orders of magnitude faster.
Original languageEnglish (US)
Title of host publicationProceedings of the VLDB Endowment
PublisherVLDB Endowment
DOIs
StatePublished - Sep 1 2015
EventProceedings of the VLDB Endowment - Proceedings of the 41st International Conference on Very Large Data Bases - Kohala Coast, Hawaii
Duration: Sep 1 2015 → …

Conference

ConferenceProceedings of the VLDB Endowment - Proceedings of the 41st International Conference on Very Large Data Bases
Period09/1/15 → …

Fingerprint

Dive into the research topics of 'Lightning fast and space efficient inequality joins'. Together they form a unique fingerprint.

Cite this