Session 1 Recap

Architecture foundations we covered:

• NameNode: metadata master (namespace + block map in RAM)
• DataNodes: block storage workers, heartbeat to NameNode
• Block size 128MB: balances NameNode memory and parallelism
• Replication factor 3 + rack awareness: fault tolerance
• Data locality: move computation to data, not data to computation

Today: Stop talking about HDFS — start using it.

Today's Answer

Both are right — and the difference is data locality.

You leave this session with CLI confidence and a notebook that reads and writes HDFS data without leaving Python.

Connect to Cluster

All HDFS commands go through the NameNode container:

# Shell into the NameNode
docker exec -it namenode bash

# Verify HDFS is running
hdfs dfs -ls /

You should see /data, /tmp, and /user listed.

Note: hdfs dfs vs hadoop fs — both work; hdfs dfs is HDFS-specific.

Navigation Commands

# List directory contents
hdfs dfs -ls /
hdfs dfs -ls /user

# List with human-readable sizes
hdfs dfs -ls -h /data

# Recursive listing
hdfs dfs -ls -R /user

HDFS paths are always absolute. There's no cd — include the full path every time.

Make Directories

# Create a single directory
hdfs dfs -mkdir /user/student

# Create a directory and all parents (like mkdir -p)
hdfs dfs -mkdir -p /user/student/hw1/input
hdfs dfs -mkdir -p /user/student/hw1/output
hdfs dfs -mkdir -p /user/student/hw1/backup

# Verify
hdfs dfs -ls /user/student/hw1

Uploading Files

# Create a test file locally
echo "Hello HDFS" > /tmp/hello.txt

# Upload to HDFS
hdfs dfs -put /tmp/hello.txt /user/student/

# Equivalent: copyFromLocal (more explicit name)
hdfs dfs -copyFromLocal /tmp/hello.txt /user/student/hello2.txt

# Verify upload and view content
hdfs dfs -ls /user/student/
hdfs dfs -cat /user/student/hello.txt

Downloading Files

# Download from HDFS to local filesystem
hdfs dfs -get /user/student/hello.txt /tmp/downloaded.txt

# Verify
cat /tmp/downloaded.txt

# getmerge: combine multiple HDFS files into one local file
# (Very useful for MapReduce output — splits across many part-* files)
hdfs dfs -getmerge /user/student/output/ /tmp/combined_results.txt

Why -getmerge? MapReduce jobs write output as part-00000, part-00001, etc.
-getmerge combines them into a single file in the correct order.

Inspecting Files

# Print file contents (like cat)
hdfs dfs -cat /user/student/hello.txt

# Print last N lines (like tail)
hdfs dfs -tail /user/student/large_file.txt

# File stats: %n=name, %b=bytes, %r=replication, %o=block size
hdfs dfs -stat "%n  size:%b  repl:%r  block:%o" /user/student/hello.txt

Block Health Check

# Check file integrity and block locations
hdfs fsck /user/student/hello.txt -files -blocks -locations

hdfs fsck output (sample):

/user/student/hello.txt: Under replicated BP...
 Total size: 11 B
 Total blocks: 1 (avg block size 11 B)
 No. of blocks: 1
Status: HEALTHY

File Operations

# Copy within HDFS
hdfs dfs -cp /user/student/hello.txt /user/student/hello_backup.txt

# Move/rename within HDFS
hdfs dfs -mv /user/student/hello_backup.txt /user/student/hw1/backup/

# Remove a file (goes to Trash by default)
hdfs dfs -rm /user/student/hello.txt

# Remove a directory and all contents (use with care!)
hdfs dfs -rm -r /user/student/old_data/

HDFS Trash: Deleted files go to /user/<name>/.Trash/ and are permanently deleted after a configurable interval (usually 24h).

Space Commands

# Disk usage (how much space a directory uses)
hdfs dfs -du /user/student/
hdfs dfs -du -h /user/student/         # human-readable
hdfs dfs -du -s /user/student/         # summary (total only)

# Disk free (cluster-wide storage)
hdfs dfs -df -h /

# Change replication factor
hdfs dfs -setrep 2 /user/student/hello.txt
hdfs dfs -setrep -w 2 /user/student/hello.txt   # wait for completion

Admin Commands

# Full cluster health check
hdfs fsck / -files -blocks

# NameNode status and metrics
hdfs dfsadmin -report

# Safe mode (cluster read-only for maintenance)
hdfs dfsadmin -safemode enter
hdfs dfsadmin -safemode leave

Web UI: NameNode dashboard at http://localhost:9870 — block reports, DataNode status, live metrics.

Hunt Rules

Work in pairs. First team to complete all tasks wins +1% on homework.

Open a shell to the NameNode:

docker exec -it namenode bash

Tasks on the next slide — go!

Hunt Tasks

1. Discovery: Find the total size of everything under /user/ (hint: -du -s)
2. Setup: Create /user/student/yourname/ with 3 subdirectories
3. Upload: Generate a 1000-line file locally (seq 1 1000 > /tmp/numbers.txt) and upload it
4. Inspect: What is the replication factor of your uploaded file? What DataNodes hold it?
5. Replication: Change replication to 2, wait for completion, verify with fsck

Hunt Tasks (2)

6. View: Use -tail to see the last 10 lines of your file
7. Merge: Create 3 small files and use -getmerge to combine them locally
8. Cleanup: Delete everything you created in /user/student/yourname/

Hunt Debrief

Common discoveries:

• Small files still occupy a full block on each DataNode (replication × block overhead)
• hdfs fsck shows exactly which DataNodes hold your blocks
• -setrep -w can take a minute — replication happens in the background
• -getmerge order is alphabetical by filename — important for part-0000N output files

Discussion: Why did we set replication to 2? When would you do this in production?

(Answer: testing environments, intermediate data, cost reduction when durability is less critical)

CLI vs. API

The CLI is great for:

• Manual exploration and debugging
• One-off data transfers
• Administrative tasks

APIs are necessary for:

• Automated pipelines (ETL workflows)
• Application integration (upload user data directly)
• Complex conditional logic (upload only if file doesn't exist)
• Testing and validation in CI/CD pipelines

Python Library

Already in Docker Jupyter — connect via WebHDFS:

from hdfs import InsecureClient

# Connect via WebHDFS (port 9870 in our Docker setup)
client = InsecureClient('http://namenode:9870', user='hadoop')

# List a directory
print(client.list('/user/student'))
# → ['hello.txt', 'hw1']

The hdfs library uses the WebHDFS REST interface — no Java required.

Read and Write

from hdfs import InsecureClient
client = InsecureClient('http://namenode:9870', user='hadoop')

# Write a file to HDFS
with client.write('/user/student/data.csv', encoding='utf-8') as writer:
    writer.write("id,value\n1,alpha\n2,beta\n3,gamma\n")

# Read a file from HDFS
with client.read('/user/student/data.csv', encoding='utf-8') as reader:
    content = reader.read()
    print(content)

DataFrame Upload

import pandas as pd
from hdfs import InsecureClient

client = InsecureClient('http://namenode:9870', user='hadoop')

# Upload a Pandas DataFrame directly to HDFS
df = pd.DataFrame({'id': [1, 2, 3], 'value': ['alpha', 'beta', 'gamma']})
with client.write('/user/student/df.csv', encoding='utf-8') as writer:
    df.to_csv(writer, index=False)

DataFrame Reads

from hdfs import InsecureClient
import pandas as pd

client = InsecureClient('http://namenode:9870', user='hadoop')

# Read CSV from HDFS into Pandas DataFrame
with client.read('/user/student/df.csv', encoding='utf-8') as reader:
    df = pd.read_csv(reader)

print(df.shape)        # (3, 2)
print(df.columns.tolist())  # ['id', 'value']

File Management

client = InsecureClient('http://namenode:9870', user='hadoop')

# Create a directory
client.makedirs('/user/student/pipeline_output')

# Check if file exists
if client.status('/user/student/data.csv', strict=False):
    print("File exists")
else:
    print("File not found")

File Management (2)

# Get file metadata
status = client.status('/user/student/data.csv')
print(f"Size: {status['length']} bytes")
print(f"Replication: {status['replication']}")
print(f"Block size: {status['blockSize']}")

# Delete a file
client.delete('/user/student/old_data.csv')

Live Demo

Open Jupyter: http://localhost:8888

Demo notebook flow:

1. Connect to HDFS NameNode
2. Create a directory structure
3. Generate sample log data (10,000 lines)
4. Upload via client.write()
5. Read back, parse HTTP status codes, count each
6. Save results as CSV to HDFS

Follow along in your own notebook.

PySpark + HDFS

PySpark reads/writes HDFS natively — no separate library needed:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("HDFS Demo").getOrCreate()

# Read CSV from HDFS (namenode:9000 = RPC port, not 9870)
df = spark.read.csv("hdfs://namenode:9000/user/student/data.csv",
                    header=True, inferSchema=True)
df.show()

Writing DataFrames

# Write DataFrame back to HDFS as Parquet
df.write.parquet("hdfs://namenode:9000/user/student/output/data.parquet")

# Write as CSV with header
df.write.csv("hdfs://namenode:9000/user/student/output/data_csv",
             header=True, mode="overwrite")

HDFS output: Each Spark partition writes one file (part-00000-abc123.csv). Use coalesce(1) or -getmerge for a single output file.

RDDs and HDFS

# Read text file as RDD (one element per line)
rdd = spark.sparkContext.textFile(
    "hdfs://namenode:9000/user/student/logs.txt"
)
print(f"Total lines: {rdd.count()}")

# Filter for ERROR lines
errors = rdd.filter(lambda line: "ERROR" in line)
print(f"Error lines: {errors.count()}")

HDFS Path Patterns

# Read all CSV files in a directory
df = spark.read.csv("hdfs://namenode:9000/user/student/data/*.csv",
                    header=True)

# Read with glob patterns
df = spark.read.parquet(
    "hdfs://namenode:9000/data/logs/year=2025/month=*/")

Path Patterns (2)

# Read multiple specific paths
df = spark.read.text([
    "hdfs://namenode:9000/data/jan.log",
    "hdfs://namenode:9000/data/feb.log",
    "hdfs://namenode:9000/data/mar.log"
])

# Default HDFS (configured in core-site.xml)
df = spark.read.csv("/user/student/data.csv")

File Organization

/data/raw/
  access_logs/
    year=2025/month=01/    ← partition directories
    year=2025/month=02/
/user/student/hw1/input/
/tmp/                      ← temporary intermediate data

Partition directories (Hive-style: key=value) enable partition pruning in Spark — huge performance win for filtering.

Format Guide

Small Files Problem

Problem: 1 million 1KB files = 150MB NameNode RAM + slow MapReduce

Solution 1: HAR (Hadoop Archive)

hadoop archive -archiveName logs.har \
  -p /user/student/small_files/ /user/student/

Solution 2: Sequence Files — bundle many key-value pairs into one binary file; splittable and indexed

Solution 3: Consolidate at ingestion — batch-merge into large HDFS files hourly via Kafka/S3

HDFS Homework

Three parts — due Sunday 11:59 PM

Part 1: Command-Line Operations (40 pts)

• Build a directory structure in HDFS
• Upload a 100MB log dataset
• Run a replication experiment (change factor, verify with fsck)
• File manipulation (copy, rename, tail)
• Space analysis (why does -du show 3× the file size?)

HDFS Homework (2)

Part 2: Programmatic Access (30 pts)

• Python hdfs library: list, upload, parse, save results
• PySpark: load file as RDD, count lines, filter errors, save output

Part 3: Analysis Questions (30 pts)

• Draw a block diagram for a 512MB file with replication=3
• Failure scenario analysis
• Why HDFS is wrong for tiny files and random access

HW Walkthrough

Let's do Part 1 Task 1 together:

# Enter NameNode container
docker exec -it namenode bash

# Create the directory structure
hdfs dfs -mkdir -p /user/student/hw1/input
hdfs dfs -mkdir -p /user/student/hw1/output

Your turn: Complete Tasks 2–5 using the commands from today's session.

Submission: ZIP containing commands.txt, hw1_notebook.ipynb, analysis.pdf

Work Time

Use the remaining time to begin Part 1.

I'm circulating — flag me if you hit:

• Docker container not running (docker compose up -d)
• Permission errors (you may need sudo in some commands)
• NameNode in safe mode (hdfs dfsadmin -safemode leave)

# Verify all containers are running
docker compose ps

Session 2 Takeaways

HDFS CLI skills you now have:

• Navigate, create, upload, download, inspect, copy, delete
• Check replication with fsck, modify with -setrep
• Measure space with -du and -df

Takeaways (2)

Programmatic access:

• Python hdfs library: connect via WebHDFS, read/write files, manage directories
• PySpark: native HDFS paths (hdfs://namenode:9000/...), read/write DataFrames and RDDs

Design principles:

• Large files > many small files
• Columnar formats (Parquet, ORC) for analytics
• Partition directories for query performance

HDFS Operation Gaps

• No parallel computation — you can upload 900TB and retrieve any block, but aggregating it requires something that can run simultaneously on all those blocks across all those nodes
• No structured access — HDFS gives you raw bytes; there's no concept of rows, columns, or schemas; every application must parse the format itself
• Batch-only paradigm — HDFS is optimized for write-once sequential reads; updating a record, streaming in new events, or running low-latency queries all require additional layers
• No job orchestration — uploading data and computing on it are completely separate concerns; someone has to coordinate which nodes run which tasks against which blocks

What Comes Next

HDFS is the foundation — every computation engine you'll study this semester reads and writes through exactly the commands you practiced today.

Next Week Preview

Week 3: MapReduce — Design Rationale and Implementation

• The MapReduce programming model (map, shuffle, reduce)
• Why MapReduce was a breakthrough (and where it falls short)
• Writing MapReduce jobs with Python and mrjob
• Running jobs on the Docker cluster

Before next session:

• Read the Google MapReduce paper (Dean & Ghemawat, 2004) — especially Sections 1–3
• Review Python functional programming: map(), filter(), reduce()
• Think about this: How would you count words in a 1TB text file?