Week 6 · MPI IV · Companion to lecture 06b

One Record,
One Column, One Deal

Five ways to send three numbers; one is wrong on every run at every p but 1. A column, Week 5's cyclic deal with no pack loop, then all of it again in Boost.

Concept map

Monday gave you type maps, signatures, extents and Boost's words. Today is one file, lecture06/src/main.cpp, that uses all of them. Click a node to jump to it.

todaylater

Every method ends in CHECK. A type map is a list of (type, offset): ask which bytes it covers, not which field names it gets wrong. Types describe layout; they do not fix the remainder, and they do not send a length.

Mark this section done

Guessed offsets: it compiles, it runs, it prints

Three Bcasts become one. struct TrapInput { int n; double a; double b; }, set on rank 0. The two highlighted lines add up sizes.

TrapInput in{};                     // {} zeroes n, a and b
if (myRank == 0) {
    in.n = 1024000;
    in.a = 0.0;
    in.b = 5.0;
}

int          lens[3]  = {1, 1, 1};
MPI_Aint     displ[3] = {0, sizeof(int),
                         sizeof(int) + sizeof(double)};
MPI_Datatype types[3] = {MPI_INT, MPI_DOUBLE, MPI_DOUBLE};

MPI_Type_create_struct(3, lens, displ, types, &T);
MPI_Type_commit(&T);

MPI_Bcast(&in, 1, T, 0, MPI_COMM_WORLD);

Walk it

Count 1, type T: the type describes the whole record.
The offsets this code builds: 0, 4, 12. The compiler put the fields at 0, 8, 16 (Monday's padding: a double wants an 8-byte boundary).
At p = 1: CHECK ok. No error, no warning, no crash.
Both sides build the same T. Both sides are wrong in the same way.

Now p = 4. The receiver's in starts as all zeros. Field by field: which of n, a, b arrive on rank 3?

None of them: a wrong type map is an error, and MPI aborts. n arrives; a and b are both garbage, since their offsets are wrong. n and a arrive; b arrives as 0. All three: the same wrong map on both sides cancels out.

Real output from lecture06, abridged, on the course's test machine:

input_ThreeBcasts    rank 3 got n = 1024000, a = 0, b = 5
                     CHECK ok
input_GuessedOffsets rank 3 got n = 1024000, a = 0, b = 0
                     CHECK WRONG  (result 0.6510416667)
input_Struct         rank 3 got n = 1024000, a = 0, b = 5
                     CHECK ok
input_Pack           ... b = 5   CHECK ok
input_Boost          ... b = 5   CHECK ok

Wrong on every run, at every p ≥ 2. Right at p = 1. The result is exactly rank 0's share of the integral: 0.651 = (5/4)³ / 3. The other three ranks integrated x² over [0, 0].

Option (d) is half right, and the half that is right is the whole lesson of the next screen: the same wrong map on both sides copies every byte the map covers to the place it came from. So the question is not "which offsets are wrong" but which bytes does this map describe, and which does it leave out.

Mark this section done

Byte by byte: bytes 20–23 never travel

The struct is 24 bytes. Lay the type map over it and see what each map covers. Little-endian throughout (x86-64 and ARM: Cardinal and every laptop in the room).

type map a = b =

n (bytes 0–3)pad (4–7)a (8–15)b (16–23)

The struct, as the compiler laid it out:

The bytes the map sends (faded = the map never mentions them):

What arrives on a receiver that started as zeros

Computed here from the deck's mechanism: copy exactly the bytes the map covers, leave the receiver's zeros everywhere else, read the 8 bytes back as a double. The deck confirmed a = 1.0 arrives intact and b = 5.1 arrives as a denormal of about 8e-315 on the test machine; try both.

Same wrong map, p = 1. Why does it pass?

With one rank, the library skips the type map and copies sizeof(TrapInput) bytes. A broadcast never writes into the root's own buffer, so the wrong map is never used to receive anything. The padding bytes happen to hold b's exponent on rank 0. It does not pass; the deck's p = 1 run was a different method.

Rank 0's in was set by rank 0 and is never overwritten. Nothing is received anywhere, so no map is applied to a destination. A test at p = 1 passes every bug on this branch that matters. Test at p = 3 and 7.

A type map is a list of (type, offset). Both sides use the same wrong list, so every byte the list covers is copied to the same place it came from. The guessed list covers bytes 0–19 of 24. It never mentions 20–23: the sign and exponent of b. a arrives for any value. b arrives as 0 for any value whose low four bytes are zero.

Mark this section done

Ask the compiler

input_Struct(). Three lines changed. That is the whole fix.

MPI_Aint displ[3] = {
    offsetof(TrapInput, n),     // 0
    offsetof(TrapInput, a),     // 8
    offsetof(TrapInput, b) };   // 16

MPI_Type_create_struct(3, lens, displ, types, &T);
MPI_Type_commit(&T);                // once, then reuse
MPI_Bcast(&in, 1, T, 0, MPI_COMM_WORLD);
MPI_Type_free(&T);

// The old Lecture_DerivedDataType did the same with MPI_Get_address:
//   displ[1] = addr(&t.a) - addr(&t.n);  ...

Checked

CHECK ok at p = 1 to 8. The first line of the program's output prints sizeof(TrapInput) and the three offsets, so the answer to Monday's question is on screen every run:

sizeof(TrapInput) = 24   offsets 0 / 8 / 16

offsetof or MPI_Get_address, never a sum of sizeof. The compiler knows where it put things; you are guessing.

MPI_Get_address is the portable general tool: it works for fields of different objects, and with MPI_BOTTOM. offsetof is enough for one plain struct.

Lay the right map over the same strip

Set the type map to offsetof on the previous screen and the faded bytes move to 4–7: the padding. Nothing in the struct lives there, so leaving it out costs nothing. The map covers 20 bytes either way; what matters is which 20.

npad, not sentab, all eight bytes

Mark this section done

Pack: position is a cursor

input_Pack(). No type this time. Copy the fields into bytes, send the bytes, copy them back out in the same order.

// every rank sizes the buffer: no length message
MPI_Pack_size(1, MPI_INT,    comm, &si);
MPI_Pack_size(1, MPI_DOUBLE, comm, &sd);
int len = si + 2 * sd;
std::vector<char> buf(len);

int position = 0;
if (myRank == 0) {
    MPI_Pack(&in.n, 1, MPI_INT,    buf.data(), len, &position, comm);
    MPI_Pack(&in.a, 1, MPI_DOUBLE, buf.data(), len, &position, comm);
    MPI_Pack(&in.b, 1, MPI_DOUBLE, buf.data(), len, &position, comm);
}

MPI_Bcast(buf.data(), len, MPI_PACKED, 0, comm);      // one call

if (myRank != 0) {
    position = 0;
    MPI_Unpack(buf.data(), len, &position, &in.n, 1, MPI_INT,    comm);
    MPI_Unpack(buf.data(), len, &position, &in.a, 1, MPI_DOUBLE, comm);
    MPI_Unpack(buf.data(), len, &position, &in.b, 1, MPI_DOUBLE, comm);
}

The receivers need len before the Bcast, and only rank 0 knows what it packed. Does rank 0 have to send len first? (Week 5's question again.)

Yes: one MPI_Bcast of an int, then the bytes. Two calls. No: every rank calls MPI_Pack_size for the same three fields and computes the same len. One call. No: MPI_PACKED receives carry their own length, so receivers pass 0. Yes, and it must be a separate MPI_Send to each rank, as in Week 5's _Uneven_Send_Recv.

The count message in Week 5's _Uneven_Send_Recv that no one needed, again. The fields and their order are fixed in the program, so every rank can size the buffer itself. Pack: 1 Bcast, and a copy on each side. CHECK ok at every p.

Step the cursor

Each MPI_Pack writes at position and advances it; each MPI_Unpack reads the same way. Sizes here are the ones the guessed offsets added up, 4 and 8 (MPI_Pack_size returns an upper bound; that is all a buffer needs).

Notice the packed buffer has no padding: n, then a at byte 4, then b at 12. Those are the guessed offsets, and here they are right, because this buffer was built that way on both sides.

Mark this section done

Boost: name the fields, in order

input_Boost(). What would Boost need from you to pack, size and unpack by itself? A list of the fields and their order. That is serialize().

// Boost: name the fields, in order. Nothing about bytes.
struct TrapInput {
    int    n;
    double a;
    double b;

    template <class Archive>
    void serialize(Archive& ar, unsigned) { ar & n; ar & a; ar & b; }
};

bmpi::communicator world;
bmpi::broadcast(world, in, 0);

Count the calls. input_Pack made one MPI_Bcast. How many does bmpi::broadcast(world, in, 0) make for this TrapInput under Boost 1.83?

One: Boost builds a derived type from serialize() and sends the struct. One: Boost packs into a buffer and broadcasts it as MPI_PACKED, exactly like input_Pack. Two: the size of the packed archive, then the bytes. Three: one per field, like input_ThreeBcasts.

Verified in boost/mpi/collectives/broadcast.hpp for 1.83, the version OSC loads: for a type that is not an MPI datatype, broadcast packs it into a packed_oarchive, broadcasts the archive size, then broadcasts the bytes as MPI_BYTE.

Pack:        1 Bcast, and a copy on each side.
Boost 1.83:  2 Bcasts. The size, then the bytes.

Pack needed one call only because every rank could compute the length. Boost cannot assume that in general: a std::vector member's size is known only to the sender. R04 gives you a Particle with a std::vector inside. That is why Boost is in this week.

If asked: BOOST_IS_MPI_DATATYPE(TrapInput) would make Boost build a derived type from serialize() and send it in one call. Not used here; it would turn input_Boost into input_Struct. Only safe for fixed layouts, never with a std::vector member.

Five ways, one record

method	what you wrote	MPI calls	p = 4
input_ThreeBcasts	three Bcasts, one per field	3	ok
input_GuessedOffsets	struct type, offsets 0 / 4 / 12	1	WRONG: b = 0
input_Struct	struct type, `offsetof`	1	ok
input_Pack	Pack_size, 3 Pack, Bcast, 3 Unpack	1	ok
input_Boost	`serialize()`, one line	2	ok

Four are right. Which of the derived type and the pack is faster, and why, is R04's measurement (Oct 16). Today shows only that both are right.

Mark this section done

One column, two layouts

column_Vector(). Column 3 of an 8 × 8 matrix, M[i][j] = 10*i + j. One send. Receive it as 8 doubles.

double M[8][8];                     // M[i][j] = 10*i + j

MPI_Datatype column;
MPI_Type_vector(8, 1, 8, MPI_DOUBLE, &column);
MPI_Type_commit(&column);
MPI_Send(&M[0][3], 1, column, 1, 0, comm);

// rank 1
double col[8];
MPI_Recv(col, 8, MPI_DOUBLE, 0, 0, comm, MPI_STATUS_IGNORE);

The three numbers

MPI_Type_vector(count, blocklen, stride, …) = (8, 1, 8): eight blocks of one double, each 8 doubles after the last. The type describes positions relative to the pointer you pass.

Output

rank 1 got column 3:
3 13 23 33 43 53 63 73
CHECK ok at every p ≥ 2.

Values are 10i + j, so a wrong column is obvious at a glance.

Which address goes to MPI_Send: M, M[3], or &M[0][3]? And is the receiver's 8 × MPI_DOUBLE allowed when the sender said 1 × column?

M[3], the third row; the receive is allowed. &M[0][3], the column's first element; the receive is allowed: same signature, eight doubles. &M[0][3]; but the receiver must also build and use column. M, and the type's stride finds column 3 by itself.

One column is eight doubles; eight MPI_DOUBLE is eight doubles. MPI matches the type signature; the layouts need not match. No buffer and no copy loop in your code. (Whether the library copies internally is Monday's question and R04's measurement.) M[3] is the common wrong answer: see it on the grid below.

Point the type somewhere

MPI_Send( , 1, column, …)

where the pointer pointsthe 8 elements the type pickspast the end of M

Flat index of M[i][j] is 8i + j. The type picks start, start + 8, …, start + 56. From M[3] (flat 24) that is M[3][0], M[4][0], … M[7][0], then three reads past the end of the matrix: column 0 of rows 3 onward, and then whatever is in memory next.

Mark this section done

The cyclic deal, typed

cyclic_Scatter(). Week 5's cyclic vector sum: scatter x, scatter y, add, gather z. x has 24 doubles. Three calls, no pack loop.

MPI_Type_vector(per, 1, p, MPI_DOUBLE, &vec);        // per = N / p
MPI_Type_create_resized(vec, 0, sizeof(double), &cyc);
MPI_Type_commit(&cyc);

MPI_Scatter(x, 1, cyc, myX, per, MPI_DOUBLE, 0, comm);
MPI_Scatter(y, 1, cyc, myY, per, MPI_DOUBLE, 0, comm);

for (int k = 0; k < per; k++) {
    myZ[k] = myX[k] + myY[k];
}

MPI_Gather(myZ, per, MPI_DOUBLE, z, 1, cyc, 0, comm);

Which type, which side

call	root side	every rank
Scatter x	1 × cyc	per × MPI_DOUBLE
Scatter y	1 × cyc	per × MPI_DOUBLE
Gather z	1 × cyc per rank	per × MPI_DOUBLE

Gather is the mirror of Scatter. One type, resized so rank r's piece starts at x[r]. Week 5's _Cyclic had a pack loop before every send and an unpack loop after every receive; those lines are gone.

Why resize

Scatter advances the root pointer by one extent of the send type per rank. The extent of vec runs from its first element to its last: (per − 1) × p + 1 doubles. Resized to one double, rank r's piece starts at x[r].

The program prints cyclic_Extent at every p ≥ 2 that divides N: the unresized extent and where rank 1 would have started. It does not run the unresized scatter, because that is an out-of-bounds read.

Deal 24 elements

p 4

Run it at p = 5. What happens?

It works: the resized type hands out 5, 5, 5, 5, 4 by itself. The method skips: 24 % 5 ≠ 0, and a vector type has one count for every rank. It runs and CHECK is wrong by the last four elements. MPI aborts: the type's extent does not divide the buffer.

p = 5: N % p != 0, skipped
24 = 5 × 4 + 4. Four ranks need 5 elements, one needs 4.
CHECK ok at p = 1, 2, 3, 4, 6, 8.

Types describe layout. They don't fix the remainder. The remainder needs different counts per rank; this one-type trick does not provide that. Padding x to a multiple of p is the usual workaround. What would you pad with, and who removes it on the way back? Leave that one open; the next screen shows what happens when a library handles it for you.

Mark this section done

Part C: the same program, wrapped

Weeks 4 and 5 again, in Boost. No counts, no displs, no types. Boost never asks for them. It still needs them.

// Week 4, wrapped
bmpi::broadcast(world, in, 0);                  // 2 Bcasts: size, bytes
bmpi::reduce(world, myPiece(in), total,
             std::plus<double>(), 0);           // 1 MPI_Reduce, MPI_SUM

// Week 5, wrapped: any N, any p
if (myRank == 0) {
    for (int i = 0; i < N; i++) {
        dealX[i % p].push_back(x[i]);           // the deal is still yours
    }
}
bmpi::scatter(world, dealX, myX, 0);
bmpi::scatter(world, dealY, myY, 0);

for (int k = 0; k < myX.size(); k++) {
    myZ[k] = myX[k] + myY[k];
}

bmpi::gather(world, myZ, parts, 0);

dealX is a std::vector<std::vector<double>> on rank 0: one inner vector per rank. bmpi::reduce with std::plus<double> on a double maps to MPI_SUM and calls MPI_Reduce: identical to Week 4, zero overhead.

Commit a call count. Rank r's share is its own vector, so lengths can differ. How many MPI calls does one bmpi::scatter(world, dealX, myX, 0) make, and what does each carry? Week 5 already wrote the answer.

One MPI_Scatterv: Boost computes counts and displs on rank 0 and the receivers do not need them. One MPI_Scatter of equal-size byte blocks, padded to the longest share. Two: MPI_Scatter of the sizes, then MPI_Scatterv of the bytes. p: one MPI_Send per rank, each with its own count.

Most of the room says one. Every receiver needs its count before it can receive, and only rank 0 knows it, so the counts come from a first collective on the sizes. Boost 1.83: scatter = MPI_Scatter (sizes) + MPI_Scatterv (bytes); gather = MPI_Gather (sizes) + MPI_Gatherv (bytes). That is exactly Week 5's Gatherv recipe, written for you. Verified Sep 28 with a PMPI call counter on Boost 1.83.

boost_CyclicAnyN at p = 5
share sizes 5 5 5 5 4   CHECK ok

No skip at p = 5. Boost removed the counts, the displs and the types. The deal loop on rank 0 did not go away, and every double is serialized into an archive and back out: Monday's γ term, twice. cyclic_Scatter's resized type may copy nothing. Which costs more is R04's measurement (Oct 16): time it, do not guess it.

Any N, any p

p 5

The deal is the same i % p as the previous screen. The difference is who copes with the ragged shares: here Boost sends the sizes first, so nothing skips.

The honest summary: Boost is a productivity tool, not a performance tool. It is the right default when data is ragged or heap-shaped (R04's Particle). For fixed, regular layouts, a derived type is usually tighter. Boost.MPI's API is MPI-1 shaped: no nonblocking collectives, no one-sided. KaMPIng and MPL are the modern C++ wrappers; mpi4py has the same split (comm.send pickles, comm.Send is a raw buffer).

Mark this section done

Run it on Cardinal

Branch lecture06. One binary runs every method, Part A then B then C; every method ends in CHECK. About 20 minutes.

Clone once, check out the lecture branch

ssh <your-username>@cardinal.osc.edu
git clone -o class git@gitlab.com:cs4540-spring-2019/cs-4170-5170-fall-2026/mpi.git mpi
cd mpi
git fetch class
git checkout -b lecture06 class/lecture06

Load the modules, build

Modules do not persist between logins: load them every time. mvapich/4.1 and srun, not mvapich/3.0 and mpiexec.
```
module load mvapich/4.1 boost/1.83.0
cd Default && make OSC && cd ..
```
Get six nodes, one process each

One process per node so every message crosses the network. srun does not inherit --cpus-per-task, hence -c 2 on each run. The login node is shared; run nothing there.
```
salloc --nodes=6 --ntasks-per-node=1 -c 2 --time=00:15:00 -A PCS0313
module load mvapich/4.1 boost/1.83.0
```

Experiments: predict first

A · p = 1: how many CHECK WRONG lines?

srun -c 2 -n 1 ./Default/MPI

What you should see

The first line prints sizeof(TrapInput) = 24 and offsets 0 / 8 / 16. Then every method ok, including input_GuessedOffsets; column_Vector is skipped (it needs p ≥ 2). Zero WRONG lines. A test at p = 1 passes every bug on this branch that matters.

B · p = 4: which field of the guessed struct is wrong, and what value?

srun -c 2 -n 4 ./Default/MPI

What you should see

input_GuessedOffsets: rank 3 got n = 1024000, a = 0, b = 0, CHECK WRONG, result 0.651. Every other Part A method ok with b = 5. column_Vector: 3 13 23 33 43 53 63 73. cyclic_Scatter ok (24 % 4 = 0). Part C ok.

C · p = 5: what skips, and what runs anyway?

srun -c 2 -n 5 ./Default/MPI

What you should see

cyclic_Scatter: p = 5: N % p != 0, skipped. Right below it, boost_CyclicAnyN: share sizes 5 5 5 5 4, CHECK ok. Same deal, two answers to the remainder.

D · Change b, and a, in the guessed version

In src/main.cpp, inside input_GuessedOffsets, set in.b = 5.1 and in.a = 1.0. Predict both arrivals with the byte strip on screen 3 before you rebuild. Then put the file back.

cd Default && make OSC && cd ..
srun -c 2 -n 4 ./Default/MPI | grep -A1 GuessedOffsets
git checkout src/main.cpp && cd Default && make OSC && cd ..

What you should see

a = 1 arrives intact: bytes 8–15 are fully covered, for any value. b arrives as a tiny denormal, about 8e-315: its low four bytes travelled, its exponent did not. Then exit to release the allocation.

E · Batch alternative

sbatch jobScript.slurm

What you should see

From the repository root. The job runs p = 1 to 6 and greps the CHECK lines: one WRONG per run at p ≥ 2 (the guessed offsets), one skipped line at p = 5.

Before 8:30 Friday (R03): CS4170_FA2026_R03_LASTNAME made, README box unchecked; fa2026-mpi/lab03 pulled, the three TODOs read, none written. The first five minutes are a prediction with no machine: bring Week 5's model, α ≈ 2 µs, β ≈ 1/(20 GB/s). No class Mon Oct 5 (Fall Break). R04 (Oct 16) times derived type against pack: today showed both are right; R04 asks which is faster, and why.

Mark this section done

Concept map

Guessed offsets: it compiles, it runs, it prints

Walk it

Byte by byte: bytes 20–23 never travel

What arrives on a receiver that started as zeros

Ask the compiler

Checked

Lay the right map over the same strip

Pack: position is a cursor

Step the cursor

Boost: name the fields, in order

Five ways, one record

One column, two layouts

The three numbers

Output

Point the type somewhere

The cyclic deal, typed

Which type, which side

Why resize

Deal 24 elements

Part C: the same program, wrapped

Any N, any p

Run it on Cardinal

Clone once, check out the lecture branch

Load the modules, build

Get six nodes, one process each

Experiments: predict first

A · p = 1: how many CHECK WRONG lines?

B · p = 4: which field of the guessed struct is wrong, and what value?

C · p = 5: what skips, and what runs anyway?

D · Change b, and a, in the guessed version

E · Batch alternative

Check yourself