Skip to content

perf(csr): aggregate single-term unsorted rows without a scipy row gather - #1127

Merged
FabianHofmann merged 1 commit into
masterfrom
perf/aggregate-unsorted
Nov 4, 2026
Merged

FabianHofmann merged 1 commit into
masterfrom
perf/aggregate-unsorted

Conversation

@FabianHofmann

Copy link
Copy Markdown
Collaborator

Note

The following content was generated by AI.

CSRStore.aggregate (the kernel behind sparse groupby().sum(), DenseStore.aggregate and, on #1122, LinearExpression.from_coo) spent most of its unsorted-rows time in scipy's CSR row gather source[order]. When every source row holds exactly one term — the common case from from_labels / 1.0 * var — the stable cell order of the rows is also the order of their entries, so data/indices are now gathered with plain numpy fancy indexing and the existing indptr is reused. Multi-term rows and sorted rows keep their existing paths. Output is identical (arrays and dtypes) to before.

Related: #1124 (unifying the two grouped-sum kernels is out of scope here), #1122.

case before after
2M single-term rows → 200k cells, unsorted 101 ms 63 ms (1.6x)
500k rows × ~4 terms, unsorted 26 ms 23 ms (unchanged path)
2M single-term rows, sorted 15.6 ms 15.0 ms (unchanged path)
sparse (1.0*flow).groupby(bus).sum(), 10 × 200k → 20k 94 ms 61 ms (1.55x)
Approaches measured and rejected
  • Entry-level counting sort (np.repeat(rows, row_nnz) → _grouped/coo_to_csr): 1.36x on single-term rows but 2.1x slower on multi-term rows (grouping over nnz instead of rows).
  • numpy arange+repeat entry gather for multi-term rows: slower than scipy's row gather.
  • np.argsort(kind="stable") instead of _grouped: ~6x slower.

Correctness: randomized old-vs-new equivalence check (800 cases; int32/int64 index dtype, empty rows/grid, absent cells, NaN consts, explicit zeros, duplicates) — all identical; new test for stable term order with single- and multi-term unsorted rows.


Generated by Claude Code

…ther

When every source row holds exactly one term, the stable cell order of
the rows is also the order of their entries, so gather data and indices
with plain numpy fancy indexing instead of scipy's CSR row indexing.
Multi-term rows keep the existing path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F4nx8KhK8QzSzHhpYKvcSx
@codspeed

codspeed Bot commented Nov 3, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks1


Comparing perf/aggregate-unsorted (11cbdea) with master (22a1882)

Open in CodSpeed

Footnotes

  1. 181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@FabianHofmann
FabianHofmann marked this pull request as ready for review November 4, 2026 07:01
@FabianHofmann
FabianHofmann merged commit cfaea93 into master Nov 4, 2026
23 checks passed
@FabianHofmann
FabianHofmann deleted the perf/aggregate-unsorted branch November 4, 2026 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants