Repository navigation
feat(ci): report-only AgentX speculative acceptance gate / feat(ci)：仅作报告的 AgentX 投机解码接受长度门禁 - #4654
Conversation
Milestone M1 of #1651. Before AgentX throughput, every AgentX speculative deployment and drafting configuration runs once with real verification on the SPEED-Bench coding prompts its golden curve was collected with. The generation engine's own counters give the realized acceptance length (AL), which the run summary compares with the golden AL. Report-only: gate jobs always finish green and throughput ignores the verdict. - infx/srt_slurm/al_gate.py plans one gate per deployment and drafting configuration and lists points it cannot gate as unresolved. - infx/bench/speedbench.py prepares the prompts once per run on a hosted runner with the pinned NeMo-Skills prepare.py and checks a pinned digest. - infx/bench/al_gate.py replays the golden collection through aiperf and reads AL from Prometheus counters (infx/results/agentic/spec_decode.py). - Golden curves record their collection settings in a slim collection block. - Real-verification runs strip forced acceptance wherever a recipe sets it. - run-sweep.yml adds al-gate-select, al-gate, al-gate-multinode and al-gate-collect; AgentX throughput waits for al-gate-collect. 中文： 都会以真实验证运行一次，使用其黄金曲线采集时的 SPEED-Bench coding 提示词。生成引擎 自身的计数器给出实际接受长度（AL），运行摘要将其与黄金 AL 对比。仅报告、不阻断： 门禁作业始终显示成功，吞吐量测试忽略其结论。 - infx/srt_slurm/al_gate.py 为每个部署与草稿配置规划一个门禁，并将无法设门禁的点 列为未解析。 - infx/bench/speedbench.py 在 GitHub 托管的 Runner 上用固定版本的 NeMo-Skills prepare.py 每次运行只准备一次提示词，并校验固定摘要。 - infx/bench/al_gate.py 通过 aiperf 重放黄金采集，并从 Prometheus 计数器读取 AL （infx/results/agentic/spec_decode.py）。 - 黄金曲线在精简的 collection 块中记录采集设置。 - 真实验证运行会移除配方中任何位置设置的强制接受。 - run-sweep.yml 新增 al-gate-select、al-gate、al-gate-multinode 和 al-gate-collect； AgentX 吞吐量测试等待 al-gate-collect。 (cherry picked from commit 0bd0e9360121cb0823a69dbcb794dac41430927e) (cherry picked from commit 7aa34ec9b599e22d7809409d26c838b4a4bb9d52)
Explicit cancellation and setup-success conditions allow the canary and AL planner to run on label-triggered sweeps whose optional reuse job is skipped. 中文：当标签触发的扫描跳过可选复用任务时，通过显式取消状态和 setup 成功条件，确保 canary 与接受长度规划任务继续执行。 (cherry picked from commit 43da4256aa52b2f96fa03e3a40ae6681ebb89ac1)
…pace Keep canary execution eligible after the optional reuse check is skipped. Free unused SDK storage on the hosted gate-planning runner before the pinned SPEED-Bench resolver materializes its coding source datasets. 中文：可选复用检查被跳过后，保持 canary 执行资格；在托管门禁规划 runner 上释放未使用的 SDK 空间，供固定 SPEED-Bench 解析器生成 coding 源数据集。 (cherry picked from commit 1faca8705c8b9af074dcfe13ff947905afbb4184)
Use the pinned harness AIPerf entrypoint and the current schema-2 recipe binder. Preserve native power collection for throughput jobs and record the upstream Qwen qualification sweep. 适配当前 InferenceX 的固定版本 harness 入口与 schema-2 配置绑定器，保留吞吐量作业的原生功耗采集，并记录上游 Qwen 验证扫描。
将接受长度门禁验证记录关联到上游 PR #4586。
…view # Conflicts: # inferencex-e2e/perf-changelog.yaml
…e-gate # Conflicts: # inferencex-e2e/perf-changelog.yaml
adibarra
added
the
full-sweep-fail-fast
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
Generally most InferenceX submissions don't need to add changes to the infx package. On the off chance they do, generally the diff is small. Please prompt your agent to clean your PR to keep your diff to only necessary & minimal changes too. We have an increased code quality bar for changes to the infx package. 中文大多数 InferenceX 提交通常不需要修改 infx 包。即使确实需要，改动通常也很小。请让你的 agent 清理 PR，使改动只包含必要且最小的内容。infx 包的改动有更高的代码质量要求。 |
|
Claude finished @adibarra's task in 1m 51s —— View job Review of PR #4654
Summary: I found one 🟡 warning and no blocking issues. The gate reuse key in Things I checked that look right:
I couldn't run |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41557099272 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41557099272 |
|
Claude finished @adibarra's task in 26s —— View job Re-review of PR #4654
Re-review: LGTM, no blocking issues found. The earlier 🟡 warning is fixed in I couldn't run |
There was a problem hiding this comment.
Findings marked 🟡 are optional suggestions and need no follow-up push.
Additional findings (outside the current diff — GitHub can't attach inline comments there):
-
🟡
inferencex-e2e/infx/workflows/al_gate_reuse.py— Conditional: on PR branches that have run full sweeps, each sweep's gate planning may make hundreds to thousands of API calls against the sharedgithub.tokenbudget. When no key is reused (the first sweep after any gate-code or recipe change),measured()pages through every artifact of up toLOOKBACK_RUNS= 30 runs. Only gate artifacts matter. Fix: publish one per-run index artifact (for example fromal-gate-collect) and fetch it with/actions/runs/{id}/artifacts?name=<index>. That keeps the lookup to about one listing call per run.Why this was flagged
The early break at al_gate_reuse.py:48 fires only when every key has been found. A PR that changes any file in
GATE_INPUTS, the curve, the recipe or the srt-slurm commit gets all-new keys, so the loop walks all 30 runs. For each run,github.paginaterequests/actions/runs/{id}/artifactsat 100 per page (infx/github.py:114). A full sweep uploads several artifacts per benchmark job across 1,000+ AgentX points plus fixed-sequence jobs, which is many pages per run. The GITHUB_TOKEN REST limit is per repository (1,000 requests per hour on non-Enterprise plans), so this lookup can drain the budget other jobs in the repo depend on, such as collect-job-events and klaud. It also makes the 60sgh api --paginatetimeout more likely. The base branch makes none of these calls. I could not count real artifacts per run from this checkout; the risk holds whenever recent runs on the branch were full sweeps.Verification: Line 47 of inferencex-e2e/infx/workflows/al_gate_reuse.py walks up to
LOOKBACK_RUNS = 30runs, and the only early exit is line 48:if found.keys() == keys: break. Every visited run gets a full artifact listing atper_page: 100(infx/github.py:114-116), and thekey not in keysfilter is applied only after that (line 55). The calls draw on the shared GITHUB_TOKEN (run-sweep.yml:462).
There was a problem hiding this comment.
Findings marked 🟡 are optional suggestions and need no follow-up push.
Still open from earlier reviews (3):
- 🔴
inferencex-e2e/infx/workflows/al_gate_reuse.py:46—PR sweeps can reuse gate verdicts from runs of a different pull request that has the same head-branch name. measured()… - Also unresolved: 2 minor or pre-existing.
If you have decided not to act on one of these findings, resolve its thread (a reply alone leaves it open) and the next review stops counting it. To review this commit again now, use Re-run on its "Claude Code Review" check.
adibarra
removed
the
full-sweep-fail-fast
…e-gate # Conflicts: # inferencex-e2e/perf-changelog.yaml
|
Claude finished @adibarra's task in 56s —— View job Re-review of PR #4654
Re-review: I found no new blocking issues. The latest commits remove the perf-changelog entry, which resolves the earlier 🔴 Still open: the earlier 🔴 branch-name reuse finding is unchanged. Perf changelog: I couldn't run |
|
Claude finished @adibarra's task in 24s —— View job Re-review of PR #4654
Re-review: LGTM, no blocking issues found. The same commit also fixes the earlier 🟡 lookup-robustness finding. Timeouts, bad zips and malformed JSON ( Not done: I couldn't run |
There was a problem hiding this comment.
I reviewed the latest push and found no new bugs. A human should still look, because this is a 2.4k-line change to the sweep workflows that handles HF secrets, artifact reuse and GPU scheduling.
What was reviewed:
al_gate_reuse.py: reuse now only uses runs whose head SHA is one of this PR's commits. Download, zip and JSON failures now fall back to "not reusable" instead of aborting the plan step.run-sweep.yml: the reuse key now hashes the wholeinfxtree plus the gate script and the srt-slurm submodule, so changes to gate code produce a new key.- The new prompt-cache workflow, the gate launch, collection and planning paths, and the golden-curve
collectionblocks.
Extended reasoning...
The PR adds a report-only speculative-acceptance gate to CI: new workflows and steps in run-sweep and the benchmark templates, gate planning and launch code under infx/srt_slurm, infx/bench and infx/launch, a reuse module that downloads earlier run artifacts, and collection settings for each golden curve. The security-sensitive parts are workflow steps that receive the read-only HF token and a reuse path that trusts JSON from earlier run artifacts. That path is now limited to runs whose head SHA belongs to this PR. Earlier inline concerns about cross-PR reuse and lookup errors look addressed in the current code. I deferred because of the change's size, its CI and secret-handling surface, and the fact that it has not been run on GPUs.
Still open from earlier reviews (3):
- Unresolved: 3 minor or pre-existing.
adibarra
added
full-sweep-fail-fast
adibarra
removed
full-sweep-fail-fast
|
Claude finished @adibarra's task in 32s —— View job Re-review of PR #4654
Re-review: LGTM, no blocking issues found. The only functional change is
The temporary GLM-5.2 perf-changelog entry is added in Not done: I couldn't run |
…ion settings on re-collection
|
Claude finished @adibarra's task in 33s —— View job Re-review of PR #4654
Re-review: LGTM, no blocking issues found.
Not done: I couldn't run the tests or |
|
Claude finished @adibarra's task in 16s —— View job Re-review of PR #4654
Re-review: LGTM, no blocking issues found.
Not done: I couldn't run the tests or |
Summary
Lands the report-only AgentX speculative acceptance gate from #4586 with review fixes. For each AgentX deployment and drafting configuration, a PR sweep replays the pinned SPEED-Bench coding prompts with real verification and compares the measured acceptance length (AL) with the golden curve. Verdicts never block throughput.
Changes on top of #4586:
llmd-vllmpoints are gated. The exclusion predatesllmd-vllmin the golden-acceptance engine map; planning now yields 4 gates for its 16 points with real verification. The planner and launcher share one exemption list (dynamo-trt,tilert).al-gate-prompts.ymlkeeps main's entry warm; a miss prepares them with dependencies resolved as of 2026-10-27, which reproduces the pinned prompt digest.minimaxm31flash_dsparkgains its collection settings.musespark_dflash(collected on SGLang's/generate) anddsv41flash_dsparkstay unresolved and are documented.Not validated on GPUs: this PR adds no perf-changelog entry, so it runs no sweep. #4586 carries the entry and the first GPU run of the gate.
Validation details
infx/tests: same results as the Withdrawn / 已撤回 #4586 merge with main, plus the new reuse and exemption tests. zizmor and actionlint report nothing new.AI model disclosure
PR checklist and change type
Type of Change
Checklist
python -m infx.matrix.changed --base origin/mainlists affected config keys, I have appended one new entry covering them to the physical end ofinferencex-e2e/perf-changelog.yaml; if it lists none, I have added none. I have not edited historical entriesinferencex-e2e/perf-changelog.yamlentry, it carries exactly one primary sweep label (a maintainer applies it on fork PRs):full-sweep-fail-fast(recommended),full-sweep-enabled, ornon-canary-full-sweep-enabled. Optional modifiersall-evals,evals-only, andagentx-fastrequire a primary label; the last two block reuse while applied.OWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the primary sweep label will no longer automatically kick off new sweeps. Remove and re-add the primary sweep label to force a new sweep.中文
概述
在 #4586 的基础上合入仅作报告的 AgentX 投机解码接受长度（AL）门禁，并加入评审修改。PR 扫描会针对每种 AgentX 部署和草稿配置，以真实验证重放固定的 SPEED-Bench coding 提示词，并将实测 AL 与黄金曲线比较。门禁结论不会阻断吞吐量测试。
相对 #4586 的改动：
llmd-vllm的点纳入门禁。原先的排除早于llmd-vllm被加入黄金接受引擎映射；现在其 16 个点规划为 4 个使用真实验证的门禁。规划器和启动器共用同一份豁免列表（dynamo-trt、tilert）。al-gate-prompts.yml在 main 上保持缓存可用；缓存未命中时使用 2026-10-27 的依赖解析结果准备提示词，可复现固定的提示词摘要。minimaxm31flash_dspark补充了收集设置。musespark_dflash（在 SGLang 的/generate上收集）和dsv41flash_dspark仍列为未解析，并已在文档中说明。尚未在 GPU 上验证：本 PR 没有添加 perf-changelog 条目，因此不运行扫描。#4586 包含该条目，并承担门禁的首次 GPU 运行。
验证详情：完整 agentic 扫描规划为 238 个门禁、380 个未解析点（原为 215 和 520），豁免框架没有门禁；全部 1,347 个 AgentX 投机解码点的配方与 main 相同，吞吐量运行不变；门禁复用查询已对本仓库 API 在现有 PR 分支上运行，238 个键均唯一，复用 0 个；
infx/tests结果与 #4586 合入 main 后一致，并新增复用和豁免测试；zizmor 和 actionlint 没有新问题。AI 模型使用说明：anthropic/claude-opus-5-5（运行环境提供的标识）负责评审 #4586、编写上述改动、本地验证并准备本 PR；另有只读 scout 子 agent 参与调查，其确切模型标识无法确认。来自 #4586 的提交保留原作者及该 PR 的说明。