Distributed in-memory analytics for big temporal data

Bin Yao; Wei Zhang; Zhi Jie Wang; Zhongpu Chen; Shuo Shang; Kai Zheng; Minyi Guo

doi:10.1007/978-3-319-91452-7_36

Distributed in-memory analytics for big temporal data

Bin Yao, Wei Zhang, Zhi Jie Wang, Zhongpu Chen, Shuo Shang^*, Kai Zheng, Minyi Guo

^*Corresponding author for this work

Computer, Electrical and Mathematical Sciences and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Scopus citations

Abstract

The temporal data is ubiquitous, and massive amount of temporal data is generated nowadays. Management of big temporal data is important yet challenging. Processing big temporal data using a distributed system is a desired choice. However, existing distributed systems/methods either cannot support native queries, or are disk-based solutions, which could not well satisfy the requirements of high throughput and low latency. To alleviate this issue, this paper proposes an In-memory based Two-level Index Solution in Spark (ITISS) for processing big temporal data. The framework of our system is easy to understand and implement, but without loss of efficiency. We conduct extensive experiments to verify the performance of our solution. Experimental results based on both real and synthetic datasets consistently demonstrate that our solution is efficient and competitive.

Original language	English (US)
Title of host publication	Database Systems for Advanced Applications - 23rd International Conference, DASFAA 2018, Proceedings
Editors	Yannis Manolopoulos, Jianxin Li, Shazia Sadiq, Jian Pei
Publisher	Springer Verlag
Pages	549-565
Number of pages	17
ISBN (Print)	9783319914510
DOIs	https://doi.org/10.1007/978-3-319-91452-7_36
State	Published - 2018
Event	23rd International Conference on Database Systems for Advanced Applications, DASFAA 2018 - Gold Coast, Australia Duration: May 21 2018 → May 24 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10827 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	23rd International Conference on Database Systems for Advanced Applications, DASFAA 2018
Country/Territory	Australia
City	Gold Coast
Period	05/21/18 → 05/24/18

Keywords

Apache Spark
Big temporal data
Distributed in-memory analytics
Temporal queries

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-319-91452-7_36

Cite this

Yao, B., Zhang, W., Wang, Z. J., Chen, Z., Shang, S., Zheng, K., & Guo, M. (2018). Distributed in-memory analytics for big temporal data. In Y. Manolopoulos, J. Li, S. Sadiq, & J. Pei (Eds.), Database Systems for Advanced Applications - 23rd International Conference, DASFAA 2018, Proceedings (pp. 549-565). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10827 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-319-91452-7_36

Yao, Bin ; Zhang, Wei ; Wang, Zhi Jie et al. / Distributed in-memory analytics for big temporal data. Database Systems for Advanced Applications - 23rd International Conference, DASFAA 2018, Proceedings. editor / Yannis Manolopoulos ; Jianxin Li ; Shazia Sadiq ; Jian Pei. Springer Verlag, 2018. pp. 549-565 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{c3cb20cb359d41d59b6213542df5b1bb,

title = "Distributed in-memory analytics for big temporal data",

abstract = "The temporal data is ubiquitous, and massive amount of temporal data is generated nowadays. Management of big temporal data is important yet challenging. Processing big temporal data using a distributed system is a desired choice. However, existing distributed systems/methods either cannot support native queries, or are disk-based solutions, which could not well satisfy the requirements of high throughput and low latency. To alleviate this issue, this paper proposes an In-memory based Two-level Index Solution in Spark (ITISS) for processing big temporal data. The framework of our system is easy to understand and implement, but without loss of efficiency. We conduct extensive experiments to verify the performance of our solution. Experimental results based on both real and synthetic datasets consistently demonstrate that our solution is efficient and competitive.",

keywords = "Apache Spark, Big temporal data, Distributed in-memory analytics, Temporal queries",

author = "Bin Yao and Wei Zhang and Wang, {Zhi Jie} and Zhongpu Chen and Shuo Shang and Kai Zheng and Minyi Guo",

note = "Publisher Copyright: {\textcopyright} Springer International Publishing AG, part of Springer Nature 2018.; 23rd International Conference on Database Systems for Advanced Applications, DASFAA 2018 ; Conference date: 21-05-2018 Through 24-05-2018",

year = "2018",

doi = "10.1007/978-3-319-91452-7_36",

language = "English (US)",

isbn = "9783319914510",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "549--565",

editor = "Yannis Manolopoulos and Jianxin Li and Shazia Sadiq and Jian Pei",

booktitle = "Database Systems for Advanced Applications - 23rd International Conference, DASFAA 2018, Proceedings",

address = "Germany",

}

Yao, B, Zhang, W, Wang, ZJ, Chen, Z, Shang, S, Zheng, K & Guo, M 2018, Distributed in-memory analytics for big temporal data. in Y Manolopoulos, J Li, S Sadiq & J Pei (eds), Database Systems for Advanced Applications - 23rd International Conference, DASFAA 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10827 LNCS, Springer Verlag, pp. 549-565, 23rd International Conference on Database Systems for Advanced Applications, DASFAA 2018, Gold Coast, Australia, 05/21/18. https://doi.org/10.1007/978-3-319-91452-7_36

Distributed in-memory analytics for big temporal data. / Yao, Bin; Zhang, Wei; Wang, Zhi Jie et al.
Database Systems for Advanced Applications - 23rd International Conference, DASFAA 2018, Proceedings. ed. / Yannis Manolopoulos; Jianxin Li; Shazia Sadiq; Jian Pei. Springer Verlag, 2018. p. 549-565 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10827 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Distributed in-memory analytics for big temporal data

AU - Yao, Bin

AU - Zhang, Wei

AU - Wang, Zhi Jie

AU - Chen, Zhongpu

AU - Shang, Shuo

AU - Zheng, Kai

AU - Guo, Minyi

N1 - Publisher Copyright: © Springer International Publishing AG, part of Springer Nature 2018.

PY - 2018

Y1 - 2018

N2 - The temporal data is ubiquitous, and massive amount of temporal data is generated nowadays. Management of big temporal data is important yet challenging. Processing big temporal data using a distributed system is a desired choice. However, existing distributed systems/methods either cannot support native queries, or are disk-based solutions, which could not well satisfy the requirements of high throughput and low latency. To alleviate this issue, this paper proposes an In-memory based Two-level Index Solution in Spark (ITISS) for processing big temporal data. The framework of our system is easy to understand and implement, but without loss of efficiency. We conduct extensive experiments to verify the performance of our solution. Experimental results based on both real and synthetic datasets consistently demonstrate that our solution is efficient and competitive.

AB - The temporal data is ubiquitous, and massive amount of temporal data is generated nowadays. Management of big temporal data is important yet challenging. Processing big temporal data using a distributed system is a desired choice. However, existing distributed systems/methods either cannot support native queries, or are disk-based solutions, which could not well satisfy the requirements of high throughput and low latency. To alleviate this issue, this paper proposes an In-memory based Two-level Index Solution in Spark (ITISS) for processing big temporal data. The framework of our system is easy to understand and implement, but without loss of efficiency. We conduct extensive experiments to verify the performance of our solution. Experimental results based on both real and synthetic datasets consistently demonstrate that our solution is efficient and competitive.

KW - Apache Spark

KW - Big temporal data

KW - Distributed in-memory analytics

KW - Temporal queries

UR - http://www.scopus.com/inward/record.url?scp=85048051760&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-91452-7_36

DO - 10.1007/978-3-319-91452-7_36

M3 - Conference contribution

AN - SCOPUS:85048051760

SN - 9783319914510

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 549

EP - 565

BT - Database Systems for Advanced Applications - 23rd International Conference, DASFAA 2018, Proceedings

A2 - Manolopoulos, Yannis

A2 - Li, Jianxin

A2 - Sadiq, Shazia

A2 - Pei, Jian

PB - Springer Verlag

T2 - 23rd International Conference on Database Systems for Advanced Applications, DASFAA 2018

Y2 - 21 May 2018 through 24 May 2018

ER -

Yao B, Zhang W, Wang ZJ, Chen Z, Shang S, Zheng K et al. Distributed in-memory analytics for big temporal data. In Manolopoulos Y, Li J, Sadiq S, Pei J, editors, Database Systems for Advanced Applications - 23rd International Conference, DASFAA 2018, Proceedings. Springer Verlag. 2018. p. 549-565. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-91452-7_36

Distributed in-memory analytics for big temporal data

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this