Skip to content

Track benchmarks over time in CI #298

Open

Description

@alxmrs

🌀 Claude: The ML demos' numbers are measured by hand and posted on #285, with the CPU and versions. Nothing records them over time, so a regression in xarray-sql, ddx, einfold or DataFusion shows up only when someone measures again.

Proposal: track benchmarks over time in CI, starting small:

  • pytest-benchmark for the Python benchmarks, including the ML demos at fixed sizes. Each step's time and its error against JAX are already in _bench.py's results table and CSV.
  • criterion.rs where the Rust code needs it.
  • github-action-benchmark to store the history and flag a regression on a PR.

Hosted runners vary from run to run, so the thresholds need to allow for that noise. Alternatives: codspeed.io, which counts instructions rather than measuring time, and bencher.dev. Both are free for open source today, but carry the risk of a SaaS changing its terms. asv is a reference for what a full history and dashboard could look like.

Raised in review of #269. The performance targets themselves are tracked on #285.

Activity

  1. added
    enhancementNew feature or request
    mlGradients in SQL, the ml extra and its demos
    on Oct 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestmlGradients in SQL, the ml extra and its demos

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions