Who Am I
Talking To?

Concept map

Every call since Week 3 has involved every rank. Real programs need a collective over some of them: a row of a matrix, the ranks on one node, the ranks a library owns. Click a node to jump to it.

Split: color picks the group, key orders it

12 ranks as a 3 × 4 grid. Every row needs its own sum. MPI_Allreduce takes one communicator; on WORLD it sums all 12.

Rank 6:  row 1, rowRank 2 of 4.

int color = rank / COLS;               // my row: 0, 1, 2
int key   = rank;                      // order inside the row
MPI_Comm row;

// collective on WORLD: every rank calls it
MPI_Comm_split(MPI_COMM_WORLD, color, key, &row);

MPI_Comm_rank(row, &rowRank);          // rank 6 -> 2
MPI_Comm_size(row, &rowSize);          //        -> 4

MPI_Allreduce(&v, &rowSum, 1, MPI_INT, MPI_SUM, row);

MPI_Comm_free(&row);                   // create, use, free

// columns: color = rank % COLS     rank 6 -> col 2, colRank 1
// key = -rank: each row numbered right to left

Same ranks, same tag, never crossed

context_Dup(). What a library does first: duplicate the communicator it is handed.

MPI_Comm lib;
MPI_Comm_dup(MPI_COMM_WORLD, &lib);     // what a library does first

if (rank == 0) {                        // Isend: Send could deadlock
    MPI_Isend(&x111, 1, MPI_INT, 1, 0, lib, &req);        // sent FIRST
    MPI_Send (&x222, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
    MPI_Wait(&req, &st);
}

if (rank == 1) {                        // wildcards on source AND tag
    MPI_Recv(&got, 1, MPI_INT, MPI_ANY_SOURCE, MPI_ANY_TAG,
             MPI_COMM_WORLD, &st);
}

got = 222
context_Dup   CHECK ok   (got == 222, and the lib receive then gets 111)

Why MPI_Isend for the first message: rank 1 receives on WORLD first. A blocking MPI_Send on lib may wait until its receive is posted, and that receive comes second: a Send-then-Recv deadlock. Eager vs rendezvous is Mon Oct 12.

The ragged two-stage sum

sum_RowThenCol(). v = rank + 1. Row Allreduce, then a column Allreduce of rowSum. At p = 12 (3 × 4) every rank prints 78 = 12·13/2.

world  0  total 21
world  1  total 21
world  2  total 10
world  3  total 10
world  4  total 21
world  5  total 21
sum_RowThenCol  expect 21 (ragged)  CHECK 2 of 6 WRONG
sum_RowLeaders  total 21            CHECK ok

Leaders only; everyone else opts out

sum_RowLeaders(). Three moves: reduce to the row leader, leaders-only allreduce, broadcast back down the row.

MPI_Comm_split(MPI_COMM_WORLD, rank / COLS, rank, &row);

int color = (rowRank == 0) ? 0 : MPI_UNDEFINED;
MPI_Comm_split(MPI_COMM_WORLD, color, rank, &leaders);

MPI_Reduce(&v, &rowSum, 1, MPI_INT, MPI_SUM, 0, row);

if (leaders != MPI_COMM_NULL) {
    MPI_Allreduce(&rowSum, &total, 1, MPI_INT, MPI_SUM, leaders);
}

MPI_Bcast(&total, 1, MPI_INT, 0, row);      // back down the row

if (leaders != MPI_COMM_NULL) {
    MPI_Comm_free(&leaders);
}
MPI_Comm_free(&row);

Boost: bmpi::communicator row = world.split(world.rank() / COLS); returns a communicator; its destructor frees it. The key defaults to the old rank. boost_Split in the file.

Groups: the hand-picked few

group_Hardcoded(), from the old Communicators code. When the ranks you want are not a function of your own rank (so no color formula), build a group and make a communicator from it.

MPI_Group worldGroup, primeGroup;
MPI_Comm_group(MPI_COMM_WORLD, &worldGroup);

const int ranks[4] = {2, 3, 5, 7};
MPI_Group_incl(worldGroup, 4, ranks, &primeGroup);

MPI_Comm primeComm;
MPI_Comm_create_group(MPI_COMM_WORLD, primeGroup, 0, &primeComm);

MPI_Comm_free(&primeComm);             // every rank
MPI_Group_free(&primeGroup);
MPI_Group_free(&worldGroup);

A group is a set of ranks, no context. You cannot send on it. MPI_Comm_create_group is collective over the group, not over WORLD: only the members have to call it. A non-member may call it too; it returns at once with MPI_COMM_NULL.

$ mpiexec -n 4 ./MPI 4 --hardcoded
*** An error occurred in MPI_Group_incl
*** MPI_ERR_RANK: invalid rank

$ mpiexec -n 8 ./MPI 4 --hardcoded
*** An error occurred in MPI_Comm_free
*** MPI_ERR_COMM: invalid communicator

// the fix, group_Primes():
for (int i = 0; i < p; i++) {
    if (isPrime(i)) primes.push_back(i);
}

if (primeComm != MPI_COMM_NULL) MPI_Comm_free(&primeComm);

Aside: lecture07 calls MPI_Init, not boost::mpi::environment. The Boost environment sets MPI_ERRORS_RETURN on WORLD so it can throw C++ exceptions. Under it these unchecked C calls did not abort: p = 4 segfaulted later with no message, and p = 8 printed "success". An error handler that returns is only as good as the code that reads the return value.

Grids: let MPI hand out the coordinates

MPI_Dims_create(p, ndims, dims) picks the shape. MPI_Cart_create builds the communicator. MPI_Cart_coords(cart, rank, maxdims, coords) fills in (x, y, z).

cart 0 (0,0)   cart 4 (1,0)
cart 1 (0,0)   cart 5 (1,0)
cart 2 (0,1)   cart 6 (1,1)
cart 3 (0,1)   cart 7 (1,1)

4 distinct (x,y) for 8 ranks
fine at p <= 7: dims[2] was 1

MPI_Cart_coords(..., maxdims=2, ...)
  failed
Number of dimensions 3 is too
  large (maximum is 2)

refused at every p, even p = 1

Ask for neighbours, not rank ± 1

cart_Shift(). A 1-D line. Every rank runs the same line; no if at the edge.

int dims[1] = {p}, periods[1] = {0};           // open line
MPI_Cart_create(MPI_COMM_WORLD, 1, dims, periods, 0, &line);

MPI_Cart_shift(line, 0, +1, &left, &right);   // source, dest

int got = -1;
MPI_Sendrecv(&me,  1, MPI_INT, right, 0,
             &got, 1, MPI_INT, left,  0, line, &st);

// open:     rank 0: left = MPI_PROC_NULL, got stays -1
// periodic: rank 0: left = p - 1,         got = p - 1

2-D: cart_Sub in the file does Dims_create(p, 2), Cart_create, then MPI_Cart_sub(grid, remain = {0, 1}) keeps dimension 1, so each rank lands in its row's communicator. No color math. When COLS divides p the file builds the grid as p/COLS × COLS, so these are exactly split_Rows' communicators, derived from the grid instead of from arithmetic. A Cartesian grid cannot be ragged, so for other p it falls back to Dims_create's shape.

Strips or blocks? Count the α's

Week 5's model, t(m) = α + mβ, applied to the grid's shape. n × n doubles over p ranks, one halo exchange per step, a rank's messages one after another (the conservative case). α ≈ 2 µs, β ≈ 0.05 ns/B.

What the model leaves out: with reorder = 1, MPI may place grid neighbours on the same node, where α is smaller. The standard permits that; it does not promise it. Whether Cardinal does is a measurement, not a slide: the question is left open here.

Run it on Cardinal

Branch lecture07. Every method ends in CHECK; the program prints a rank/row table. About 20 minutes.

1. Clone once, check out the lecture branch

   ssh <your-username>@cardinal.osc.edu
   git clone -o class git@gitlab.com:cs4540-spring-2019/cs-4170-5170-fall-2026/mpi.git mpi
   cd mpi
   git fetch class
   git checkout -b lecture07 class/lecture07

   copy

2. Load the modules, build

   Modules do not persist between logins. Load them every time.

   module load mvapich/4.1 boost/1.83.0
   cd Default && make OSC && cd ..

   copy

3. Get nodes of your own

   The login node is shared; run nothing there. Two nodes, so a message between ranks crosses the network. Load the modules again inside the allocation.

   salloc --nodes=2 --ntasks-per-node=6 -c 2 --time=00:15:00 -A PCS0313
   module load mvapich/4.1 boost/1.83.0

   copy

   Every run: srun -c 2 -n p ./Default/MPI [COLS] [--hardcoded]. Default COLS = 4. group_Hardcoded runs only with --hardcoded.

Experiments

srun -c 2 -n 12 ./Default/MPI

srun -c 2 -n 6 ./Default/MPI
srun -c 2 -n 8 ./Default/MPI

srun -c 2 -n 4 ./Default/MPI 4 --hardcoded
srun -c 2 -n 8 ./Default/MPI 4 --hardcoded

srun -c 2 -n 5 ./Default/MPI