Robert C. Green II Ph.D.

CS 4170/5170: Introduction to Parallel Programming - Fall 2026

Interactive companions to the lectures. Each one is a single page: concept map, animations, predict-then-reveal questions, step-by-step runs on OSC, and a self-check. Nothing is recorded; progress is kept in your own browser.

MPI

Module Page What it covers
MPI I Processes, Messages, and the Machine Six separate rooms: SPMD, the four-call skeleton, Send/Recv, Probe before Recv, barrier-fenced timing, Slurm and one-process-per-node
MPI II Splitting the Work, Timing It, Talking to It The trapezoid split by rank, linear vs tree merge, the cost model and when splitting pays, whose clock counts, who does I/O
One File, Three Questions Code walk: the hand merge, designing MPI_Reduce and MPI_Bcast, the n/p bug that prints a plausible answer, the fix, timing rules
MPI III Moving the Data, Not Just the Answer Scatter/Gather and the bytes bill, Scatterv counts and displs, the collectives family and rules, t(m) = α + m·β and the 40 KB crossover
Five Ways to Add Two Vectors Code walk: the N/p and ANY_SOURCE bugs, counts and displs by hand, the same in three calls, cyclic deals and the pack loop
MPI IV Describe the Data, Don't Copy It Struct padding and offsets, size vs extent, Type_vector for a column, resized types, Boost.MPI and what the wrapper costs
One Record, One Column, One Deal Code walk: the guessed-offsets bug byte by byte, pack and position, Boost serialize, the column, the resized cyclic scatter
MPI V Who Am I Talking To? Comm_split color and key, context and dup, the ragged two-stage sum, groups, Cartesian grids, Cart_shift and PROC_NULL, strips vs blocks
MPI VI Hang, Wrong, Crash, Silent Sorting every bug by symptom, eager vs rendezvous, choosing p from the shape, tracing a hang, error handlers, the silent Bcast, one model for R01–R04
MPI Review MPI Quiz/Exam Review Messages, costs, and the bugs that pass. Weeks 3–8: SPMD, collectives, the cost model, derived types, communicators, hang/wrong/crash/silent, metrics

OpenMP

Module Page What it covers
OpenMP I One Process, Many Threads Fork/join, parallel for and the schedule, shared/private/firstprivate, the race, critical vs atomic vs reduction, running it on Cardinal
The Trapezoid, Three Ways, in Threads Code walk: TrapRule1/2/3, the int-coordinate and remainder bugs, what a critical section holds, the fixed split, floating-point order, running it on Cardinal
OpenMP II A Must Precede B Flow, anti and output dependence, what removing each costs, the prefix-sum scan, odd–even sort and the barrier, why a racy loop passes k of 100 runs, how to detect it
Who Gets the Next i? Callback: static, static,c, dynamic and guided on triangular work, what a chunk grab costs, the same decision as an MPI manager–worker, the load-balance row of the payoff table
OpenMP III When the Parallelism Isn’t a Loop Sections and the busiest thread, the parallel/single task idiom, quicksort as tasks, a task reads a shared variable when it runs, task data scoping, taskwait vs taskgroup, the cutoff curve, size vs depth cutoffs, running it on Cardinal
fib, What a Task Costs, and the Cutoff Code walk: fib.cpp timed at five cutoffs, the cost of one task by subtraction and why it grows with threads, the two conditions for a cutoff, when no cutoff works, the quicksort curve on 48 threads with the formulas
OpenMP IV Reflection 6: A Dependence Removed, and the Honest Baseline Step by step on Cardinal: build at -O2 and -O3, a loop-carried dependence, the wrong fix and the right fix (a two-pass scan), what the fix costs, and which serial time a speedup is measured against

Foundations

Module Page What it covers
Foundations I Where the Data Lives The memory ladder measured, the 64-byte line, row vs column traversal, first touch and NUMA, the roofline and where AXPY sits