Performance evaluation of Map-reduce jar pig hive and spark with machine learning using big data
- Title
- Performance evaluation of Map-reduce jar pig hive and spark with machine learning using big data
- Creator
- Santosh Kumar J.; Raghavendra B.K.; Raghavendra S.; Meenakshi
- Description
- Big data is the biggest challenges as we need huge processing power system and good algorithms to make a decision. We need Hadoop environment with pig hive, machine learning and hadoopecosystem components. The data comes from industries. Many devices around us and sensor, and from social media sites. According to McKinsey There will be a shortage of 15000000 big data professionals by the end of 2020. There are lots of technologies to solve the problem of big data Storage and processing. Such technologies are Apache Hadoop, Apache Spark, Apache Kafka, and many more. Here we analyse the processing speed for the 4GB data on cloudx lab with Hadoop mapreduce with varing mappers and reducers and with pig script and Hive querries and spark environment along with machine learning technology and from the results we can say that machine learning with Hadoop will enhance the processing performance along with with spark, and also we can say that spark is better than Hadoop mapreduce pig and hive, spark with hive and machine learning will be the best performance enhanced compared with pig and hive, Hadoop mapreduce jar. Copyright 2020 Institute of Advanced Engineering and Science. All rights reserved.
- Source
- International Journal of Electrical and Computer Engineering, Vol-10, No. 4, pp. 3811-3818.
- Date
- 2020-01-01
- Publisher
- Institute of Advanced Engineering and Science
- Subject
- Cloudxlab; Flink; Hadoop; Hbase; HDFS; Hive; Map-reduce; Pig; Spark
- Coverage
- Santosh Kumar J., Department of Computer Science and Engineering, KSSEM, Affiliated to VTU Belagavi, Bangalore, India; Raghavendra B.K., Department of Computer Science and Engineering, BGSIT (ACU) Deemed to be University, India; Raghavendra S., Department of Computer Science and Engineering, Christ Deemed to be University, India; Meenakshi, Department of Computer Science and Engineering, Jain Deemed to be University, India
- Rights
- All Open Access; Gold Open Access; Green Open Access
- Relation
- ISSN: 20888708
- Format
- Online
- Language
- English
- Type
- Article
Collection
Citation
Santosh Kumar J.; Raghavendra B.K.; Raghavendra S.; Meenakshi, “Performance evaluation of Map-reduce jar pig hive and spark with machine learning using big data,” CHRIST (Deemed To Be University) Institutional Repository, accessed February 24, 2025, https://archives.christuniversity.in/items/show/16502.