A Comparison of the Efficiencies of various Structured and Semi- Structured Data Formats in Data Analysis and Big Data Analytic Development

Peng, Taoxin; Graham, Heather

Abstract

As data volumes grow, so too does our need and ability to analyse it. Cloud computing technologies offer a wide variety of options for analysing big data and make this ability available to anyone. However, the monetary implications for doing this in an inefficient fashion could surprise those who may be used to an on-premises solution to big data analysis, as they move from a model where storage is limited and processing power has little cost implications, to a model where storage is cheap but compute is expensive. This paper investigates the efficiencies gained or lost by using each of five data formats, CSV, JSON, Parquet, ORC and Avro, on Amazon Athena, which uses SQL as a query language over data at rest in Amazon S3, and on Amazon EMR, using the Pig language over a distributed Hadoop architecture. Experiment results suggest that ORC is the most efficient data format to use on the platforms tested against, based on time and monetary costs.

Citation

Peng, T., & Graham, H. (2024, July). A Comparison of the Efficiencies of various Structured and Semi- Structured Data Formats in Data Analysis and Big Data Analytic Development. Presented at DATA 2024: 13th International Conference on Data Science, Technology and Applications, Dijon, France

• Presentation Conference Type: Conference Paper (published)
• Conference Name: DATA 2024: 13th International Conference on Data Science, Technology and Applications
• Start Date: Jul 9, 2024
• End Date: Jul 11, 2024
• Acceptance Date: Apr 15, 2024
• Publication Date: 2024
• Deposit Date: May 14, 2024
• Publisher: Scitepress Digital Library
• Peer Reviewed: Peer Reviewed
• Pages: 301-308
• ISBN: 9789897587078
• DOI: https://doi.org/10.5220/0000178300003756
• Keywords: Athena, Cloud, Big Data, Data Format, Semi-Structured, Efficiency, EMR
• Public URL: http://researchrepository.napier.ac.uk/Output/3642612
• Publisher URL: http://www.scitepress.org/DigitalLibrary/Default.aspx
• External URL: https://data.scitevents.org/