Map Reduce Example In Python

Hadoop Streaming Writing A Hadoop MapReduce Program In Python from www.edureka.co

Introduction

MapReduce is a programming model that allows for processing and generating big data sets on a distributed system. It is widely used in big data applications to handle massive amounts of data. Python is a powerful programming language that can be used to implement MapReduce. In this article, we will discuss a map reduce example in Python.

What is MapReduce?

MapReduce is a programming model that allows for processing and generating big data sets on a distributed system. It was developed by Google and is widely used in big data applications. The model consists of two basic functions: map and reduce. The map function takes input data and converts it into a set of key-value pairs. The reduce function then takes the output of the map function and combines the values associated with the same key.

Why use MapReduce?

MapReduce is used in big data applications to handle massive amounts of data. It allows for processing and generating big data sets on a distributed system, which means that it can handle large amounts of data in a short amount of time. Additionally, MapReduce is fault-tolerant, which means that if one node fails, the system can continue to operate.

MapReduce Example in Python

Let's take a look at a simple MapReduce example in Python. In this example, we will count the number of occurrences of each word in a text file.

Map Function

The map function takes a key-value pair as input and outputs a set of key-value pairs. In this example, the input key is the line number, and the input value is the line of text. The output key is the word, and the output value is the number of occurrences of the word in the line. ```python def mapper(key, value): for word in value.split(): yield word, 1 ```

Reduce Function

The reduce function takes a key and a set of values as input and outputs a single value. In this example, the input key is the word, and the input values are the number of occurrences of the word in each line. The output is the total number of occurrences of the word in the entire text file. ```python def reducer(key, values): yield key, sum(values) ```

Putting it All Together

We can put the map and reduce functions together to create a MapReduce job. ```python from mrjob.job import MRJob class WordCount(MRJob): def mapper(self, key, value): for word in value.split(): yield word, 1 def reducer(self, key, values): yield key, sum(values) if __name__ =='__main__': WordCount.run() ```