Week 1 Recap

What we covered:

• Big data landscape and the V's (Volume, Velocity, Variety)
• The Hadoop ecosystem overview
• Docker environment setup and first contact with the cluster

Key insight: Traditional databases hit walls at scale — we need new architectures designed for distributed commodity hardware.

Any environment setup issues? Quick show of hands — we'll troubleshoot during break.

Today's Path

HDFS is the answer — and understanding why means understanding what NFS got wrong.

By end of session you'll be able to trace a read request through the cluster and explain every design decision along the way.

The Scale Problem

A typical enterprise challenge:

• 10TB of web server logs generated per day
• Need to analyze 90 days of history (900TB)
• Query: "Find all sessions from users who later converted to paid"

On a single machine:

• 900TB doesn't fit on one disk
• Even if it did: reading at 500MB/sec → 21 days to scan once
• RAID adds capacity but not parallel throughput

The fundamental constraint: I/O bandwidth is physical, not a software problem.

FS Assumptions

Local file systems (ext4, NTFS, APFS) assume:

• One machine owns the storage
• Random access is common (seek + read small records)
• Files change frequently (overwrite, append, truncate)
• Concurrency is managed with locks
• Failure is rare — hardware is enterprise-grade

These assumptions break completely at petabyte scale on commodity clusters.

DFS Requirements

What a file system for big data must provide:

GFS (2003)

GFS Paper (Ghemawat, Gobioff, Leung) — the blueprint for HDFS

Key observations Google made:

1. Component failures are the norm, not the exception
2. Files are mostly large (multi-GB), not millions of tiny files
3. Files are written once (or appended), then read many times
4. Co-designing the file system and applications unlocks performance

The insight: Don't try to make distributed storage look like local storage. Embrace the difference and design accordingly.

Design Principles

HDFS is a direct implementation of GFS ideas in open-source Java:

• Write-once, read-many: Files are immutable after creation (append possible but limited)
• Large files: Optimized for 100MB–TB files, not millions of KB files
• Streaming access: Sequential reads dominate; random access is slow by design
• Commodity hardware: Runs on standard x86 servers; failures expected and handled
• Simple consistency: No POSIX semantics; relaxed model enables throughput

The trade-off: HDFS is spectacular at what it does and terrible at what it doesn't do.

HDFS vs. Local FS

Architecture

     ┌──────────────┐
     │  NameNode    │  ← metadata only
     └──────┬───────┘
    ┌───────┼───────┐
    ▼       ▼       ▼
┌────────┐┌────────┐┌────────┐
│DataNode││DataNode││DataNode│
└────────┘└────────┘└────────┘

Clients contact NameNode once for metadata, then read/write DataNodes directly.

The NameNode

Responsibilities:

• Maintains the namespace tree (directories and files)
• Stores metadata for every file (permissions, timestamps, block list)
• Tracks block locations on DataNodes (in memory — not on disk)
• Authorizes all filesystem operations

What NameNode does NOT do:

• Store any actual file data
• Directly participate in reads or writes (after the initial handshake)

Critical detail: All metadata lives in RAM. A cluster with billions of files needs a very large NameNode.

NameNode Memory

Each file requires ~150 bytes of NameNode RAM (name + block list + locations)

Same data volume, wildly different metadata cost.

The DataNode

Responsibilities:

• Store actual data blocks on local disk
• Send heartbeats to NameNode every 3 seconds (I'm alive!)
• Send block reports every hour (here's what I have)
• Serve read requests from clients directly
• Participate in replication pipeline for writes

On failure: NameNode detects missed heartbeat after 10 min, schedules re-replication automatically — transparent to users.

Block-Based Storage

HDFS stores files as fixed-size blocks distributed across DataNodes

Default block size: 128 MB

A 400MB file becomes:

Block 1: 128MB → DataNode 2, 5, 7
Block 2: 128MB → DataNode 1, 3, 8
Block 3: 128MB → DataNode 4, 6, 2
Block 4:  16MB → DataNode 3, 7, 1

Why 128MB? Not arbitrary — it's a careful trade-off.

Block Size

The trade-off between block size and system behavior:

The winner: 128MB minimizes NameNode memory while preserving enough parallelism for large analytical jobs.

Rule of thumb: Each block can be processed by one mapper. Too few blocks = too little parallelism.

Replication

Default: 3 copies of every block on different DataNodes:

Block A → [DataNode 1, DataNode 4, DataNode 7]
Block B → [DataNode 2, DataNode 5, DataNode 8]
Block C → [DataNode 3, DataNode 6, DataNode 9]

• Up to 2 DataNodes can fail simultaneously without data loss
• Any surviving DataNode can serve read requests
• Re-replication happens automatically when nodes fail

Storage cost: 3× raw storage, but commodity disks are cheap.

Rack Awareness

The problem: If all 3 replicas land in the same rack, a power failure kills all copies.

HDFS rack-aware placement strategy (default):

Replica 1: Writer's node (or random node in writer's rack)
Replica 2: Different rack from replica 1
Replica 3: Same rack as replica 2, different node

Why this works:

• Survives individual node failures (replicas on different nodes)
• Survives entire rack failure (replicas on 2+ racks)
• Keeps replica 2 and 3 on same rack to reduce cross-rack write traffic

Read Operation

Client reads /data/logs/access.log:

1. Client → NameNode: "I want to read /data/logs/access.log"
2. NameNode → Client: "Here are the block IDs and their DataNode locations"
3. Client → DataNode (nearest with Block 1): "Send me Block 1"
4. DataNode → Client: streams block data
5. Client → DataNode (nearest with Block 2): "Send me Block 2"
6. ... (repeat for all blocks, potentially from different DataNodes)

Key insight: NameNode is consulted once for metadata, then the client talks directly to DataNodes. NameNode never becomes a data bottleneck.

Write Operation

Client writes a new file:

1. Client → NameNode: "I want to create /data/new_file.txt"
2. NameNode → Client: "Write Block 1 to DN2; DN2 will forward to DN5; DN5 to DN8"
3. Client → DN2: streams data (DN2 simultaneously forwards to DN5, DN5 to DN8)
4. DN8 → DN5 → DN2 → Client: acknowledgments flow back through the pipeline
5. Client → NameNode: "Block 1 written successfully"
6. Repeat for each block

Pipeline replication: Data flows in one direction; acknowledgments flow back. Efficient use of network.

HDFS Read Trace

Pair up. Scenario: Client reads /data/logs/access.log

• 512MB file, 128MB blocks, replication=3, 8 DataNodes across 2 racks

Tasks:

1. How many blocks does this file have?
2. Draw the Client NameNode DataNode message sequence
3. Which messages carry metadata? Which carry data?
4. Bonus: DataNode 3 fails mid-read — what happens?

One pair presents to the class.

Discussion Points

Expected answers:

• Blocks: ⌈512 / 128⌉ = 4 blocks
• Metadata flow: Client → NameNode (once), then client reads directly from DataNodes
• Failure recovery: Client retries on a different DataNode holding a replica — transparent to the application

The big insight: NameNode handles metadata only — the metadata layer never becomes a data throughput bottleneck.

Move Computation

The core insight of data-local scheduling:

Locality Tiers

The scheduler tries to place tasks in this priority order:

1. Node-local — task runs on the exact DataNode holding the block

   • Read from local disk: ~500 MB/sec

2. Rack-local — task runs on a different node in the same rack

   • Read via rack switch: ~1 Gbps = ~125 MB/sec

3. Off-rack — task runs on any available node

   • Read via core switch: ~1 Gbps shared among all rack traffic

Real impact: Node-local vs. off-rack can be a 4× throughput difference on a busy cluster.

MapReduce Locality

Why data locality matters for MapReduce:

• Each mapper processes one HDFS block
• YARN scheduler checks which nodes have that block
• Launches the mapper task on a node-local DataNode when possible
• If all local nodes are busy: waits briefly, then falls back to rack-local

The payoff: On a 1,000-node cluster analyzing 1PB of data with good locality:

• Local reads at 500 MB/sec: ~24 hours of total I/O
• Off-rack reads at 50 MB/sec: 240 hours of total I/O

Locality turns an infeasible job into a practical one.

HDFS CLI Preview

Session 2 focus: Hands-on HDFS operations in the Docker cluster

# Navigation
hdfs dfs -ls /
hdfs dfs -mkdir -p /user/student/data

CLI Preview (2)

# File operations
hdfs dfs -put local_file.txt /user/student/
hdfs dfs -get /user/student/file.txt ./

# Inspection
hdfs dfs -cat /user/student/file.txt
hdfs dfs -stat "%n %b %r" /user/student/file.txt

# Administration
hdfs dfs -setrep -w 2 /user/student/file.txt
hdfs fsck /user/student/ -files -blocks

Session 1 Takeaways

What you should now understand:

1. Why HDFS exists: Traditional file systems can't scale horizontally; HDFS was designed from scratch for distributed commodity hardware

2. NameNode: Metadata only, in RAM — never touches data; single point of coordination (and historical SPOF)

3. DataNodes: Store blocks, send heartbeats, serve clients directly after metadata lookup

Takeaways (2)

4. Block size (128MB): Balances NameNode memory, parallelism, and seek overhead

5. Replication (3×) with rack awareness: Tolerates node and rack failures automatically

6. Data locality: Schedule computation where data lives — turns network bottleneck into disk throughput

HDFS Gaps

• No computation model — HDFS is a file system, not a processor; knowing where your 10PB of log data lives tells you nothing about how to count, filter, or aggregate it
• No query interface — there's no SQL, no API for "give me all rows where status=500"; data locality is valuable only when paired with a computation engine that exploits it
• The small files problem — pipelines that stream many small files per second will saturate NameNode RAM even if total data volume is modest; you need an ingestion strategy
• Write-once limitations — HDFS is optimized for immutable files; updating a single record requires rewriting the entire file, making it unsuitable for transactional workloads

What Comes Next

HDFS is the right tool for distributed, fault-tolerant storage of large files — but the moment you want to compute anything, you need a processing engine that understands data locality.

Next Session

Session 2: HDFS Operations and Programming

• HDFS CLI: navigate, upload, download, inspect, administer
• Scavenger hunt: hands-on command practice in the cluster
• Python hdfs library: programmatic access from Jupyter
• PySpark + HDFS: reading/writing DataFrames
• Homework walkthrough and work time

Before Session 2: Make sure Docker cluster is running — docker compose up -d then verify with hdfs dfs -ls /.