[toresbe]

OUTPUT CHANGE (heatmap PNG only; no CNA values, calls or tables change).

Summary

Above 4,096 cells the heatmap draws an evenly spaced subset of the ordered cells, and the 20k-200k-cell ordering embedding is computed from the collapsed bins matrix. The image looks the same (same genome-wide structure and prediction sidebar), but individual pixel rows show different cells and the k-means block order can differ. The legend still lists every category present among the plotted cells.

Plotting was the run's memory peak by far: mat[:, cell_order] copied the float64 bins x cells matrix, and imshow copied it twice more (masked copy, then astype) before resampling it to the ~2,000-pixel-tall axes, where nearest-neighbour rendering shows one cell per pixel row anyway. The block ordering also made a float32 copy of the whole matrix for its TruncatedSVD.

Performance (32 threads, with the heatmap; development runs)

• Xenium 80k: plotting +24.2 GB -> +1.6 GB; whole-run peak 33.7 GB -> 11.1 GB.
• Xenium 30k: +9.1 GB -> +1.6 GB; run 98.9 s -> 86.1 s.

Tests

tests/test_heatmap_rows.py.

How to check

This script runs main and this branch on the built-in example dataset and lists which output files are byte-for-byte identical, then prints each step's runtime for both versions. The example has 302 cells, below the 4,096-cell cap, so every file is expected to be identical here. To try your own data, replace load_example_data() with a path to a genes x cells .csv/.tsv count matrix. It needs git and a Python environment with CopyKAT-Python's dependencies.

#!/usr/bin/env bash
# Compare CopyKAT-Python output between a base branch and a pull-request
# branch.
#
# Runs both versions on the built-in example dataset (302 cells) and reports
# which output files are byte-for-byte identical. Needs git and a Python
# environment with CopyKAT-Python's dependencies (e.g. the conda environment
# from environment.yml).

set -euo pipefail

REPO=${REPO:-https://github.com/toresbe/Copykat_python.git}
BASE_BRANCH=${BASE_BRANCH:-main}
PR_BRANCH=${PR_BRANCH:-perf/heatmap-subsample}
CORES=${CORES:-8}
WORK=$(mktemp -d)
echo "working in $WORK"

# 1. Check out both versions side by side.
git clone --quiet --branch "$BASE_BRANCH" "$REPO" "$WORK/base"
git clone --quiet --branch "$PR_BRANCH" "$REPO" "$WORK/pr"

# 2. Run the same analysis with each version. PYTHONPATH makes Python import
#    copykat_py from that checkout instead of any installed copy.
for version in base pr; do
    mkdir -p "$WORK/out_$version"
    echo "running $version ..."
    (
        cd "$WORK/out_$version"
        PYTHONPATH="$WORK/$version" python - > run.log 2>&1 <<PYEOF
from copykat_py import copykat, load_example_data
copykat(load_example_data(), sam_name="example", n_cores=$CORES, output_seg=True)
PYEOF
    )
done

# 3. Compare every output file. The timing report (runtime.json) and the run
#    log always differ, so they are skipped.
status=0
for file in "$WORK/out_base"/*; do
    name=$(basename "$file")
    [[ $name == *runtime.json || $name == run.log ]] && continue
    if cmp -s "$file" "$WORK/out_pr/$name"; then
        echo "identical: $name"
    else
        echo "DIFFERENT: $name"
        status=1
    fi
done

# 4. Show how long each step took in each version (seconds).
python - "$WORK/out_base/example_copykat_runtime.json" "$WORK/out_pr/example_copykat_runtime.json" <<'PYEOF'
import json, sys
base, pr = (json.load(open(path)) for path in sys.argv[1:])
print(f"\n{'step':30} {'base':>8} {'PR':>8}")
for a, b in zip(base["steps"], pr["steps"]):
    print(f"{a['step']:30} {a['seconds']:8.2f} {b['seconds']:8.2f}")
print(f"{'total':30} {base['total_seconds']:8.2f} {pr['total_seconds']:8.2f}")
PYEOF

exit $status

🤖 Generated with Claude Code