Skip to content

SUMMA: operand tiles are pulled to the phase owner before being broadcast #604

Open

Description

@evaleev

What happens

ContEngine::init_distribution imposes ProcGrid::make_row_phase_pmap(K) / make_col_phase_pmap(K) (a CyclicPmap over the process grid) on both operand evaluators, and Summa roots every row/column broadcast at the phase owner: get_row_group_root picks grid position (rank_row, k % proc_cols) from k alone, and bcast_col_range_task obtains the tile to broadcast with left_.get(index), i.e. from the operand's ArrayEvalImpl, which for a tile the array holds elsewhere does array_.find(array_index).

So the path of an operand tile is: original owner → phase owner (point-to-point pull) → broadcast tree over the row (column) group rooted at the phase owner. The original location only decides whether the first hop is a no-op, and with a blocked operand map against a cyclic phase map it almost never is.

Why it matters

For a P = P_r × P_c grid and uniformly placed sources, the tile goes through the extra serialized hop with probability 1 - 1/P: 75% on a 2×2 grid, ~93% on 4×4. In transfer volume the surplus is one transfer for each tile whose source lies in the destination group but is not the root, so it is modest for dense SUMMA on few ranks, but it is not negligible in practice, and sparse SUMMA (pruned groups, fewer flops per tile) makes the comm/comp ratio worse. The hop also serializes the pull before the broadcast.

Candidate fixes

  • (a) construct broadcast groups that include the source rank: many groups, not attractive;
  • (b) broadcast into a group from a rank outside it: TA/MADNESS groups are not MPI subcommunicators, so this is possible in principle, but gop.bcast(key, value, root, group) names the root by group-local rank (Group::rank(world_rank) is -1 for a non-member), so it needs a push-from-source entry point rather than the current pull into the root;
  • (c) short-circuit remote references when passing them around: useful for pulls, and the broadcast is a push, so probably not applicable here.

Shelved for now; recorded so it is not forgotten. A TA_TRACE_GLOBAL_COMM_STATS build would quantify it (left_ntiles_used_, ntiles_pending_ counters).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions