Introduction to Hadoop

1. Definition

Apache Hadoop is an open-source software framework designed for distributed storage and distributed processing of very large datasets across clusters of commodity hardware using simple programming models.

---

2. History and Evolution of Hadoop

Year	Milestone	Description
2003-2004	Google Papers Published	Google published research papers on Google File System (GFS) and MapReduce, laying theoretical foundation for Hadoop.
2005	Hadoop Created	Doug Cutting and Mike Cafarella created Hadoop while working on Apache Nutch search engine project.
2006	Yahoo Adoption	Yahoo! hired Doug Cutting and dedicated team to develop Hadoop. Became separate Apache project.
2008	Hadoop 1.0 Released	First stable release. Yahoo ran 4000-node Hadoop cluster, becoming largest production deployment.
2012	Hadoop 2.0 with YARN	Introduction of YARN (Yet Another Resource Negotiator) for better resource management.
2017	Hadoop 3.0 Released	Major improvements in scalability, efficiency, and addition of erasure coding for storage optimization.

---

3. Origin of the Name "Hadoop"

Source: Doug Cutting's son had a yellow stuffed toy elephant named "Hadoop."

Why This Name:

• Easy to pronounce and remember
• Unique and not a technical acronym
• Represented something approachable and friendly
• No prior meaning, so could define it independently

Logo: The official Hadoop logo features a yellow elephant, honoring the toy that inspired the name.

---

4. Core Problems Hadoop Solves

4.1 Storage Challenge

Problem: Traditional databases cannot economically store petabytes of data.

Hadoop Solution: HDFS (Hadoop Distributed File System) distributes data across many inexpensive commodity servers, providing:

• Scalable storage capacity
• Cost-effective expansion
• Fault tolerance through replication

Example: Storing 100 terabytes of log files would cost ₹50 lakhs+ in traditional storage, but only ₹10-15 lakhs using Hadoop with commodity hardware.

---

4.2 Processing Challenge

Problem: Processing large datasets sequentially takes prohibitively long time.

Hadoop Solution: MapReduce programming model enables parallel processing across cluster nodes:

• Divides data into smaller chunks
• Processes chunks simultaneously across multiple nodes
• Combines results efficiently

Example: Analyzing 10 TB of transaction data that would take 100 hours on single server can be completed in 1-2 hours on 100-node Hadoop cluster.

---

4.3 Cost Challenge

Problem: Enterprise-grade storage and processing systems are extremely expensive.

Hadoop Solution: Uses commodity hardware and open-source software:

• No expensive proprietary servers required
• No licensing costs (100% open-source)
• Linear cost scaling (add more cheap nodes as needed)

Cost Comparison:

Loading comparison…

---

5. Key Characteristics of Hadoop

5.1 Open Source

Definition: Hadoop is developed and maintained by Apache Software Foundation as open-source project.

Implications:

• Free to download and use
• Source code publicly available
• Large community of contributors
• No vendor lock-in
• Continuous improvements from global developers

---

5.2 Distributed Storage and Processing

Storage: Data stored across multiple machines in cluster rather than single location.

Processing: Computations performed in parallel across cluster nodes where data resides.

Benefit: Both storage capacity and processing power scale linearly by adding more nodes.

---

5.3 Scalability

Horizontal Scalability: Add more machines to cluster rather than upgrading existing machines.

Practical Example:

Initial Setup: 10 nodes handling 50 TB
Data Growth: Need to store 200 TB
Solution: Add 30 more nodes (total 40 nodes)
Cost: 30 × ₹50,000 = ₹15 lakhs
Result: Seamless capacity expansion

---

5.4 Fault Tolerance

Definition: Hadoop continues operation despite hardware failures by automatically handling node failures.

Mechanisms:

1. Data Replication: Each data block stored on 3 different nodes by default
2. Automatic Recovery: NameNode detects failed nodes and re-replicates data from remaining copies
3. Task Re-execution: If processing task fails on one node, automatically reassigned to another

Example: In 1000-node cluster, if 10 nodes fail, Hadoop automatically:

• Reads data from replica nodes
• Re-creates missing replicas on healthy nodes
• Continues processing without interruption

---

5.5 Data Locality

Principle: Move computation to data rather than moving data to computation.

Traditional Approach:

Data (Storage) → Network Transfer → Processing Node → Results
Problem: Network becomes bottleneck

Hadoop Approach:

Processing Task → Executed on Node Where Data Resides → Results
Benefit: Minimal network usage, faster processing

Impact: 10x-100x performance improvement for large datasets by eliminating data transfer overhead.

---

6. Hadoop Ecosystem Overview

While Hadoop core consists of HDFS and MapReduce, ecosystem includes many complementary tools:

Core Components:

• HDFS: Distributed file storage
• MapReduce: Batch processing engine
• YARN: Resource management and job scheduling

Supporting Tools:

• Hive: SQL-like queries on Big Data
• Pig: Data flow scripting language
• HBase: NoSQL database for real-time access
• Spark: Fast in-memory processing
• Kafka: Stream processing platform

---

7. When to Use Hadoop

Hadoop is Ideal For:

1. Large Data Volumes: Terabytes to petabytes of data
2. Batch Processing: Processing historical data in bulk
3. Unstructured Data: Logs, text files, images, videos
4. Cost-Sensitive Projects: Budget constraints
5. Scalability Requirements: Rapidly growing data

Hadoop is NOT Suitable For:

1. Small Datasets: Less than 1 TB (overhead not justified)
2. Real-Time Processing: Requires immediate responses
3. Random Data Access: Frequent read/write operations
4. Complex Transactions: ACID compliance needs

---

8. Real-World Use Case

Loading case study…

---

Exam Pattern Questions and Answers

Question 1: "Define Hadoop and explain its key characteristics." (8 Marks)

Answer:

Definition (2 marks):
Apache Hadoop is an open-source software framework designed for distributed storage and distributed processing of very large datasets across clusters of commodity hardware using simple programming models. It was created by Doug Cutting and Mike Cafarella in 2005, inspired by Google's research papers on GFS and MapReduce.

Key Characteristics (6 marks):

Open Source (1 mark): Hadoop is developed by Apache Software Foundation and freely available without licensing costs. This eliminates vendor lock-in and benefits from continuous community improvements.

Distributed Architecture ( mark): Data is stored across multiple machines in HDFS rather than single location, and processing is performed in parallel across cluster nodes, enabling both storage and processing to scale linearly.

Scalability (1 mark): Hadoop supports horizontal scaling by adding more commodity servers to cluster rather than expensive vertical upgrades, allowing seamless expansion from gigabytes to petabytes without system redesign.

Fault Tolerance (1.5 marks): Hadoop automatically handles hardware failures through data replication (each block stored on 3 nodes by default), automatic failure detection, and task re-execution, ensuring continuous operation despite node failures.

Data Locality (1.5 marks): Hadoop follows the principle of moving computation to data rather than moving data to computation, executing processing tasks on nodes where data resides, minimizing network traffic and maximizing performance.

---

Question 2: "Explain the evolution of Hadoop with timeline." (6 Marks)

Answer:

Origins (2003-2005) (2 marks): Hadoop's foundation was laid when Google published research papers on Google File System and MapReduce in 2003-2004. In 2005, Doug Cutting and Mike Cafarella, working on Apache Nutch search engine project, created Hadoop to implement concepts from these papers for distributed data storage and processing.

Growth Phase (2006-2008) (2 marks): Yahoo! hired Doug Cutting in 2006 and invested in Hadoop development, making it a separate Apache project. In 2008, Hadoop 1.0 was released as first stable version, with Yahoo running world's largest production deployment on 4000-node cluster, validating Hadoop's enterprise readiness.

Maturity (2012-Present) (2 marks): Hadoop 2.0 released in 2012 introduced YARN for improved resource management, separating resource management from programming model. Hadoop 3.0 released in 2017 brought major improvements in scalability, storage efficiency through erasure coding, and enhanced performance, establishing Hadoop as industry-standard Big Data platform.

---

Summary

Key Points for Revision:

1. Hadoop: Open-source framework for distributed storage and processing
2. Creator: Doug Cutting (2005), named after his son's toy elephant
3. Core Problems Solved: Storage scalability, processing speed, cost reduction
4. Key Characteristics: Open-source, distributed, scalable, fault-tolerant, data locality
5. Evolution: 2005 (Created) → 2008 (v1.0) → 2012 (v2.0 YARN) → 2017 (v3.0)
6. Use Cases: Large data volumes, batch processing, unstructured data, cost-sensitive projects

---

Quiz Time! 🎯

Loading quiz…