Week 8 · MPI VI · Companion to lecture 08a

Hang, Wrong,
Crash, Silent

Every MPI bug you have met this term, sorted by what you see. A method for finding the next one. Then the one cost model under R01–R04.

Concept map

The last MPI lecture. Nothing new to call, except MPI_Comm_set_errhandler. Click a node to jump to it.

todaylater

There are only four things a broken MPI program can look like, and each one has a cheap test. A hang tells you it failed. A crash tells you where. Wrong-at-some-p and silent tell you nothing unless you test for them.

Mark this section done

Four symptoms. Which is worst?

A dozen broken programs from Weeks 4–7 and today. Each was a separate surprise. Sort them by what you see, then read what the deck says about each.

Click one of the four for every row. The fact under each row is the deck's; it appears once you commit.

What finds each one

You see	What finds it
Hang	`timeout`; a size past the eager limit; print before/after
Wrong at some p	a p that does not divide the work; p off the grid; `--shuffle`
Crash	the abort message names the call; p < 8 and p ≥ 8
Silent	CHECK against serial at p ≥ 2, not only at p = 1

Hold your answer to "which is worst" until the silent section. Week 4's Friday question (quiet vs loud) already argued for one of them.

Mark this section done

A hang: everyone sends right, then receives from the left

ring_SendThenRecv(). Every rank: MPI_Send m doubles to rank r + 1 (mod p), then MPI_Recv m doubles from r − 1. m = 100: CHECK ok at p = 2 and p = 4. It passes review.

// every rank, in this order
MPI_Send(sendBuf, m, MPI_DOUBLE, (myRank + 1) % numProcs,             0, MPI_COMM_WORLD);
MPI_Recv(recvBuf, m, MPI_DOUBLE, (myRank + numProcs - 1) % numProcs, 0, MPI_COMM_WORLD, &status);

There is a call named MPI_Sendrecv for exactly this. Why bother? Commit: the smallest m where this stops working, or "never". One number.

Never. Every Send has a matching Recv, so the counts match and it always completes. m = 1. Everyone is sending and nobody is receiving; it only worked at m = 100 by luck. Somewhere in the hundreds. At some size the message stops fitting wherever the library was putting it, and Send stops returning early. Around m = 100,000. Only a message that fills the network's buffers can block.

MPI_Send is allowed to return as soon as the message has been copied somewhere (the library's buffer), or only once the matching receive has been posted. The standard lets the library choose. Libraries choose by size.

$ timeout 10 mpiexec -n 2 ./MPI ring_SendThenRecv 505
ring_SendThenRecv  m = 505 doubles   CHECK ok

$ timeout 10 mpiexec -n 2 ./MPI ring_SendThenRecv 506
(nothing, for 10 seconds)             exit 124

$ ompi_info --param btl vader --level 9 | grep eager_limit
btl_vader_eager_limit  (current value: "4096")

$ mpiexec -n 4 ./MPI ring_Sendrecv 100000
ring_Sendrecv  m = 100000 doubles    CHECK ok

505 × 8 = 4,040 B + header fits the 4,096 B eager limit: Send copies it out and returns. 506 doubles do not. Send now waits for a posted receive. Every rank is in Send; none is in Recv. Exit 124 is timeout's code for "I killed it." Test machine, Open MPI 4.1.6. Cardinal (MVAPICH, two nodes) will differ.

Eager or rendezvous: step through the ring

ranks p 2 m doubles 100

The threshold is the test machine's: m ≤ 505 goes eagerly, m ≥ 506 by rendezvous (4,096 B btl_vader_eager_limit, slide 5). The arrows show who each rank is waiting for. Cardinal's limits, one node and two nodes, are unmeasured; the job script on the branch sweeps m to find them.

Three standard fixes

Fix	Mechanism, in one sentence
`MPI_Sendrecv`	The library pairs the send and the receive for you (Week 7, `Cart_shift`). `ring_Sendrecv` on the branch: CHECK ok at m = 100,000.
Even/odd ordering	Evens send first, odds receive first: half the ranks receive first, so every Send finds a posted Recv.
`Isend`/`Irecv` + `Wait`	Neither call waits, so both are posted before anyone blocks in Wait (Week 7).

What each one costs, and which is best, is not written here on purpose. That depends on what each one costs; it is yours to reason about.

Why this is the dangerous kind of hang: it is correct at every p you test with small messages, and it appears when the problem gets bigger. That is usually on the cluster, the night before it is due.

Mark this section done

Choose p from the problem's shape

Method, 1 of 2. Powers of two hide remainders. The bug that needs a special p is found by choosing that p on purpose.

p = 1 first. Every path you wrote must still run with rank 0 alone. Necessary, not sufficient: the guessed offsets pass at p = 1.
A p that does not divide the work: 3 and 7. The trapezoid and the vector sum both broke there.
A p that breaks the shape: 6 on a 4-wide grid; 8 = 2 × 2 × 2 for a 3-D grid.
A p larger than the work: more ranks than rows, batches, or blocks. Some rank gets nothing. (Next section.)
A size past the eager limit, not only m = 100.

A test set

p = 1, 2, 3, 4, 7, 8

m = 100 and 100,000

Six runs, oversubscribed on a laptop, seconds each. Every run prints CHECK.

Why 7 and not 5: both are primes that divide neither 2^k nor a 4-wide grid. 7 also appears in R01's sweep for the same reason.

Build your own test set

Toggle values of p. The table shows which of the term's bugs each p would have exposed, using the facts the deck states for each bug. Rows marked "model" extend a stated mechanism to a p the deck did not run.

include m = 100,000

Mark this section done

A hang: idle ranks go home early

reduce_SkipIdle(). Six batches of work. Batch k goes to rank k mod p. A rank sums its batches. A rank with no batches returns ("nothing to do"). Everyone else calls MPI_Reduce to rank 0.

long mySum = 0;
int  myBatches = 0;

for (int k = myRank; k < 6; k += numProcs) {
    mySum += batchSum(k);
    myBatches++;
}

if (myBatches == 0) {                       // looks like good practice
    trace("no batches, returning early");
    return;
}

trace("before MPI_Reduce");
MPI_Reduce(&mySum, &total, 1, MPI_LONG, MPI_SUM, 0, MPI_COMM_WORLD);
trace("after MPI_Reduce");

Six batches, p ranks

p 8

Which of p = 4, 6, 7, 8 finish? Commit before you reveal.

All four. The Reduce simply has fewer contributions at 7 and 8, so the sum is still right. 4 and 6 finish. 7 and 8 finish with a wrong sum: the idle ranks' zeros are missing. 4 and 6 finish. 7 and 8 never complete: the Reduce waits for a rank that left. Only 4. At 6, rank 5 has one batch and rank 0 has one, which the tree cannot balance.

A collective must be called by every rank of the communicator. Rank 0 does not know rank 6 has nothing to send; it waits for rank 6's part.

$ timeout 10 mpiexec -n 7 ./MPI reduce_SkipIdle --trace
[r0] before MPI_Reduce
[r1] before MPI_Reduce
[r1] after MPI_Reduce
[r2] before MPI_Reduce
   ... r3, r4, r5: before, after
[r6] no batches, returning early
exit 124

$ gdb -p <pid> -batch -ex bt        (each rank)
r0, r2:     PMPI_Reduce <- reduce_SkipIdle  main.cpp:138
r1, r3-r6:  ompi_mpi_finalize <- main       main.cpp:311

Ranks 0 and 2 are in the Reduce, waiting for rank 6's part of the tree. Rank 6 left. Five ranks finished their part and wait in Finalize. Open MPI used a binary tree here: rank 6's parent is 2, 2's parent is 0. On MVAPICH the stuck set may differ; the method does not.

p = 4 and 6 finish. p = 7, 8 hang: p > 6 batches. Fix (reduce_EveryRank): an idle rank contributes 0, and still calls. CHECK ok at every p (1–9 tested).

The two drawings

An arrow from each waiting rank to the rank it waits for. Every hang this term was one of these two.

A cycle is a deadlock. The ring at p = 2: 0 waits for 1, 1 waits for 0.

An arrow to a rank that left is a missing call. reduce_SkipIdle at p = 7: 0 and 2 wait for 6, which returned.

A collective is a promise every rank of the communicator makes. If any rank can skip it, all the others can wait forever. Same for if (rank has data) MPI_Bcast(...).

Mark this section done

A hang: find who is waiting on whom

Method, 2 of 2. The procedure, in the order to use it.

Bound it first
timeout 60 srun … in the script; #SBATCH --time=0:05:00 for a test job. A hang should cost minutes, not your allocation.
Trace
Print [rK] before/after every MPI call on the suspect path, then fflush. Each rank's last line is where it is.
Or attach
On a node running your job, gdb -p <pid> -batch -ex bt, once per rank. No rebuild.
Draw it
An arrow from each waiting rank to the rank it waits for. A cycle is a deadlock. An arrow to a rank that left is a missing call.
Count them
For every Send, a Recv. For every collective, every rank, in the same order. Sizes past the eager limit. Then look for the rank that took a different branch.

// --trace on the command line
void trace(const char* what) {
    if (!TRACE) {
        return;
    }

    std::printf("[r%d] %s\n", rank, what);
    std::fflush(stdout);          // a hung rank never flushes
}

trace("before MPI_Reduce");
MPI_Reduce(...);
trace("after MPI_Reduce");

// every method ends in CHECK
void report(const char* name, std::string detail, bool myOk) {
    int bad      = myOk ? 0 : 1;
    int totalBad = 0;

    MPI_Reduce(&bad, &totalBad, 1, MPI_INT, MPI_SUM, 0, WORLD);

    if (rank == 0) {
        totalBad == 0 ? print("CHECK ok")
                      : print("CHECK k of p WRONG");
    }
}

The fflush rule. stdout is buffered. Without the flush, a rank that hangs holds its last lines in a buffer that is never written, and you look in the wrong place. In the container, runs killed by timeout without --trace lost even rank 0's header line. std::endl flushes too; "\n" does not. Rank 0 recomputes serially and compares exactly. Every line starts [rK].

Read a trace

The p = 7 trace again, every line this time. Click each rank whose last line says it is still inside a call. Then click the rank it needed.

Stuck inside MPI_Reduce:

The rank they needed, which never called:

Every MPI hang is a rank waiting for a call that another rank will never make.

Mark this section done

A crash tells you where

ring_OffTheEnd(). Rank r passes a token to r + 1. The last rank sends to rank p, which does not exist.

MPI_Comm_set_errhandler(MPI_COMM_WORLD, MPI_ERRORS_RETURN);       // only with --return

int rc = MPI_Send(&token, 1, MPI_LONG, r + 1, 0, MPI_COMM_WORLD);  // last rank: r + 1 == p

if (rc != MPI_SUCCESS) {
    MPI_Error_string(rc, msg, &len);
    printf("[r%d] ... %s\n", r, msg);
}

p = 4, default error handler. What happens when rank 3 calls MPI_Send with destination 4?

The send goes to rank 0: destinations wrap around the communicator. MPI_Send returns an error code, which the program ignores, and the run ends with CHECK WRONG. The whole job aborts, and the message names the call, the rank, and the reason. Rank 3 hangs in Send forever: nobody will ever receive from it.

Every communicator starts with MPI_ERRORS_ARE_FATAL. The job dies and the message names the call (MPI_Send), the rank (process [...,3]) and the reason. That is a gift: read it before anything else.

$ mpiexec -n 4 ./MPI ring_OffTheEnd                     (default: MPI_ERRORS_ARE_FATAL)
*** An error occurred in MPI_Send ... reported by process [...,3]
*** MPI_ERR_RANK: invalid rank
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort ...)     exit 6

$ mpiexec -n 4 ./MPI ring_OffTheEnd --return
[r3] MPI_Send to rank 4 returned: MPI_ERR_RANK: invalid rank
ring_OffTheEnd   MPI_ERRORS_RETURN        CHECK ring_OffTheEnd 1 of 4 WRONG

MPI_ERRORS_RETURN hands the code back to you instead. Useful only if you check it. Week 7 callback: boost::mpi::environment switches WORLD to ERRORS_RETURN (so it can throw), which is why lecture07 uses MPI_Init; under the environment, the prime-group abort turned into a segfault at p = 4 and a silent "success" at p = 8. The fix (ring_ProcNull): the ends talk to MPI_PROC_NULL, Week 7's Cart_shift idea. CHECK ok at every p. Real output, Open MPI 4.1.6, test machine; MVAPICH's abort text on Cardinal will look different.

Leave FATAL on while you develop: the abort names the call and the rank. RETURN is for code that checks every return code.

Crash at some p: `scatterv_FloorRecv()`

Week 5's counts and displacements are right. The receive is sized by N / p.

int recvCount = N / p;                    // BUG: counts[r] may be N/p + 1

MPI_Scatterv(x,    counts,    displs, MPI_DOUBLE,
             mine, recvCount,         MPI_DOUBLE, 0, WORLD);

N = 20. At which p in 1 … 9 does this abort?

None. A short receive just gets the first N/p values; the rest are dropped. Every p ≥ 2: the receive count never matches the sender's count array. p = 3, 6, 7, 8, 9: exactly the p that do not divide 20. p = 8 and 9 only: more ranks than the data can fill.

N = 20:
p = 1, 2, 4, 5   CHECK scatterv_FloorRecv ok
p = 3            *** MPI_ERR_TRUNCATE: message truncated
p = 6, 7, 8, 9   aborts the same way
fix: recvCount = counts[r]        CHECK ok at every p

The first 20 mod p ranks (rank 0 among them) get N/p + 1 whenever p does not divide 20. A message longer than the posted receive is an error. A message shorter than the receive is fine (the status tells you how much came). That asymmetry is why a receive sized by the floor fails exactly at the p that leave a remainder. p = 4 hides it. p = 3 finds it. Real output, Open MPI 4.1.6; MVAPICH's text will differ.

Mark this section done

Silent: the wrong type

bcast_WrongType(). One line from Monte Carlo π, isolated from an older debugging branch and made deterministic by putting the int next to the seed.

struct Params {
    int nSamples;
    int seed;          // seed = 1000 + rank on every rank
};

// rank 0 sets nSamples, then:
MPI_Bcast(&P.nSamples, 1, MPI_DOUBLE, 0, comm);

// each rank draws its share of darts from its seed.  p = 4 prints pi = 3.142584

Is the program right? If not, whose darts did rank 3 throw?

Right. The types do not match, but nSamples is 4 bytes and arrives intact, and π looks fine. Wrong. Rank 3 threw rank 0's darts: the 8-byte broadcast carried rank 0's seed along with nSamples. Wrong. Rank 3's nSamples is garbage: a double reinterpreted as an int. It crashes: the library detects that an int buffer was described as MPI_DOUBLE.

$ mpiexec -n 1 ./MPI bcast_WrongType
pi = 3.142514  seeds distinct 1 of 1   CHECK ok

$ mpiexec -n 4 ./MPI bcast_WrongType
pi = 3.142584  seeds distinct 1 of 4   CHECK 4 of 4 WRONG

$ mpiexec -n 4 ./MPI bcast_RightType            (MPI_INT)
pi = 3.141876  seeds distinct 4 of 4   CHECK ok

MPI_DOUBLE moves 8 bytes: rank 0's nSamples and seed. Every rank gets seed 1000. π is plausible and wrong. "4 of 4 WRONG": ranks 1–3 fail their own check (their seed is not 1000 + rank), and rank 0 fails because its total differs from the serial reference, which uses seeds 1000–1003. At p = 1 there is nobody to overwrite: CHECK ok, as Week 6's guessed offsets did.

Why π still looks right: four identical streams of 500,000 darts each is still 2,000,000 darts' worth of sample mean, with the variance of 500,000. The error is inside the noise, so no eyeball catches it. That is the definition of silent. And not "π equals the p = 1 run": a correct run differs from it (3.141876 against 3.142514; different seeds, different darts, both right). CHECK re-throws every share serially from seed 1000 + q and matches the hit count exactly. Real output, Open MPI 4.1.6, test machine.

Walk the bytes

P is 8 bytes: nSamples at 0–3, seed at 4–7. Pick the type rank 0 passes; the outlined bytes are what the broadcast moves into rank 3's P.

watching rank

rank 0's P

rank 3's P, after the Bcast

In a declaration you wrote, the compiler sees int; MPI only sees a pointer and the type you pass. The MPI type must describe your buffer: the library cannot see your declaration. Same lesson as Week 6's type map. With no struct, the 4 bytes after an int on the stack are whatever the compiler put there: another variable, padding, or a saved register. Undefined behavior; it may even pass.

Mark this section done

One model, four reflections

T_p ≈ compute/p + comm(p). Not a new model: Week 4's T_p ≈ (n/p)·t_f + C(p)·t_m, with Week 5's t(m) ≈ α + m·β put in for t_m.

	You varied	The terms that could matter	Ask your own data
R01	p; nothing divided	(p − 1) messages of a few bytes: α, p − 1 times	Was the cost per message flat across p?
R02	p; 2 × 10⁸ samples fixed	(n/p)·t_f + one Reduce	Did the Reduce ever show up? At what p?
R03	p and N (10³, 10⁷)	hand: (p − 1) × t(m) in a row · Gatherv: a schedule	Which of α and m·β won at each N?
R04	n records, p = 2	the same bytes on the wire · plus whatever touches them	If the gap is not the wire, what is it?

α ≈ 2 µs, β ≈ 1/(20 GB/s): the Week 5 planning numbers. Every reflection measured one corner of the same two lines. The last column is not answered here. Those are the questions to ask of your own numbers; R04 is not measured yet, so its question is a prediction prompt for Friday.

Which term wins: t(m) = α + m·β

Model, not a measurement. Slide one message's size and watch the two terms trade places. The deck's two anchors: N = 10³ doubles is 8,000 B, 0.4 µs of β-time against 2 µs of α; N = 10⁷ doubles is 8 × 10⁷ B, 4 ms of β-time.

N doubles, 10^ 3 p 4

Open each repository and put the predicted number next to the measured one. A specific belief, the measured number that replaced it, and the mechanism: that is what the synthesis asks for. "Communication is expensive" says nothing.

Mark this section done

Run it on Cardinal

Branch lecture08: one method per run, every method ends in CHECK. Two nodes, so the ring can cross InfiniBand. About 20 minutes.

Clone once, check out the lecture branch

ssh <your-username>@cardinal.osc.edu
git clone -o class git@gitlab.com:cs4540-spring-2019/cs-4170-5170-fall-2026/mpi.git mpi
cd mpi
git fetch class
git checkout -b lecture08 class/lecture08

Load modules, build, get two nodes

Modules do not persist between sessions: load them every time. The login node is shared; nothing runs there.
```
module load mvapich/4.1 boost/1.83.0
cd Default && make OSC && cd ..
salloc --nodes=2 --ntasks-per-node=4 --cpus-per-task=2 --time=00:15:00 -A PCS0313
```
Every run: timeout 20 srun -c 2 -n p ./Default/MPI <method> [n] [--trace] [--return]. Methods: ring_SendThenRecv ring_Sendrecv reduce_SkipIdle reduce_EveryRank ring_OffTheEnd ring_ProcNull scatterv_FloorRecv scatterv_Counts bcast_WrongType bcast_RightType. The timeout 20 is the bound from the hang method: a hang costs 20 seconds, not the allocation. Exit 124 means it hung.

Experiments

A · The ring, past the eager limit, on one node and across two

timeout 20 srun -c 2 -n 2 ./Default/MPI ring_SendThenRecv 506
timeout 20 srun -c 2 -N 2 --ntasks-per-node=1 -n 2 ./Default/MPI ring_SendThenRecv 16000

What you should see

Cardinal's eager limits are unmeasured; the deck says only that they will differ from the test machine's 505 doubles. Expect the ring to hang at some size (exit 124 after 20 s), and expect the one-node and two-node thresholds to differ: the eager threshold is an MVAPICH parameter and is not the same within a node as across InfiniBand. If 506 passes on one node, raise m until it does not. Then run ring_Sendrecv at the same m: CHECK ok. The batch script sweeps m = 100 … 100,000 on both paths; in lec08/results/ring_1node_m*.txt and ring_2node_m*.txt, the first EXIT 124 is Cardinal's eager limit on that path.

B · The missing collective, traced

timeout 20 srun -c 2 -n 7 ./Default/MPI reduce_SkipIdle --trace
timeout 20 srun -c 2 -n 7 ./Default/MPI reduce_EveryRank --trace

What you should see

The first hangs at p = 7 (exit 124). Read the trace: [r6] no batches, returning early, and some ranks with a before MPI_Reduce and no after. Which ranks are stuck may differ from the test machine's 0 and 2 (MVAPICH's reduce tree is not Open MPI's); the method is the same. The second finishes with CHECK ok.

C · A crash that names the call

srun -c 2 -n 4 ./Default/MPI ring_OffTheEnd
srun -c 2 -n 4 ./Default/MPI ring_OffTheEnd --return

What you should see

Without --return, the job aborts: MVAPICH's abort text looks different from Open MPI's, but it still names MPI_Send, the rank, and an invalid-rank error. With --return, rank 3 prints an MPICH-style error string and the run ends CHECK 1 of 4 WRONG.

D · The silent one

srun -c 2 -n 4 ./Default/MPI bcast_WrongType
srun -c 2 -n 4 ./Default/MPI bcast_RightType
srun -c 2 -n 3 ./Default/MPI scatterv_FloorRecv

What you should see

bcast_WrongType is CHECK WRONG at p = 4, with a π that looks fine and seeds distinct 1 of 4: the bug is in the program, not the library. bcast_RightType is ok. scatterv_FloorRecv at p = 3 aborts with a truncation error in MVAPICH's words; at p = 4 it would pass. Then exit to release the allocation.

Before Friday Oct 16 (R04 lab): make the lab repository and clone it on Cardinal; that is homework, not lab time. On paper: where each field of Particle sits and sizeof(Particle). Prediction committed at 8:30, before any code. Job submitted by 9:20. Due Sun Oct 18.

Mark this section done

Concept map

Four symptoms. Which is worst?

What finds each one

A hang: everyone sends right, then receives from the left

Eager or rendezvous: step through the ring

Three standard fixes

Choose p from the problem's shape

A test set

Build your own test set

A hang: idle ranks go home early

Six batches, p ranks

The two drawings

A hang: find who is waiting on whom

Bound it first

Trace

Or attach

Draw it

Count them

Read a trace

A crash tells you where

Crash at some p: scatterv_FloorRecv()

Silent: the wrong type

Walk the bytes

One model, four reflections

Which term wins: t(m) = α + m·β

Run it on Cardinal

Clone once, check out the lecture branch

Load modules, build, get two nodes

Experiments

A · The ring, past the eager limit, on one node and across two

B · The missing collective, traced

C · A crash that names the call

D · The silent one

Check yourself

Crash at some p: `scatterv_FloorRecv()`