What Is MapReduce?

Google's MapReduce framework is a programming model that makes it easy to create parallel programs. MapReduce has been used to power Google's Web page indexing, but several other large companies, universities, and research labs have also adopted it. The MapReduce framework consists of a map function and a reduce function. The map function takes an input key-value pair and produces zero or more output key-value pairs by applying some transformation to the input pair. The reduce function takes the output from all map tasks and produces a result for the entire job. MapReduce is a programming framework that runs on a large cluster of commodity machines and is highly scalable. It has several forms of implementation provided by multiple programming languages, like Java, C# and C++. It's designed to take advantage of parallelism to process large datasets. The system uses two phases: Map, which divides the input into chunks, and reduces, which merges them into output. MapReduce's main advantage is its fault tolerance, where periodic reports from each node in the cluster are expected when work is completed. If a node fails, the framework will re-allocate that task to another node. This allows for greater scalability and reliability as well as fewer bottlenecks. MapReduce is used in a lot of different ways. In distributed grep, it's used to find patterns in documents. It's also used to sort documents and web links. Web link-graph reversal is used to reverse a graph's link structure. Web access log stats it's used to calculate statistics about web surfing habits. Document clustering uses MapReduce to cluster documents into groups based on their content similarity. Machine learning and statistical machine translation both use MapReduce so that they can be scaled across many machines and perform more efficiently than they would if they were running on just one machine alone!