You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
🌀 Claude: The ML demos' numbers are measured by hand and posted on #285, with the CPU and versions. Nothing records them over time, so a regression in xarray-sql, ddx, einfold or DataFusion shows up only when someone measures again.
Proposal: track benchmarks over time in CI, starting small:
pytest-benchmark for the Python benchmarks, including the ML demos at fixed sizes. Each step's time and its error against JAX are already in _bench.py's results table and CSV.
Hosted runners vary from run to run, so the thresholds need to allow for that noise. Alternatives: codspeed.io, which counts instructions rather than measuring time, and bencher.dev. Both are free for open source today, but carry the risk of a SaaS changing its terms. asv is a reference for what a full history and dashboard could look like.
Raised in review of #269. The performance targets themselves are tracked on #285.
Activity