Repository navigation
[AMD][MI325X] DSv4.1 Flash vLLM: re-sweep with --monokernel at TP4 and SWA bounded replay on the latest ROCm 10.0 nightly / [AMD][MI325X] DSv4.1 Flash vLLM：在最新 ROCm 10.0 nightly 上于 TP4 启用 --monokernel 与 SWA bounded replay 并重测 - #4617
maeehart wants to merge 8 commits into
Conversation
Move dsv41flash-fp4-mi325x-vllm-agentic-dspark from v0.32.0 to the 2026-10-28 ROCm 10.0 nightly edad8b03, which carries vllm-project/vllm#60731, its fix #63576 and #59039. Turn on --monokernel for the TP4 points and drop --no-swa-bounded-replay. Signed-off-by: Markus Hartikainen <markus.hartikainen@amd.com>
Signed-off-by: Markus Hartikainen <markus.hartikainen@amd.com>
…-vllm-monokernel # Conflicts: # inferencex-e2e/perf-changelog.yaml
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| - "Adaptive DSpark verification stays off. With --monokernel on gfx942 the server faults during its CUDA graph profile until vllm-project/vllm#63816 merges." | ||
| - "No point changes: concurrency 1, 2, 4, 8, 16 and 32 on TP8, TP4 and TP2 (18 points), all re-measured on the new image." | ||
| - "DSpark draft: vLLM builds it with the checkpoint's own quantization config and the target's fp8_ds_mla KV cache, and its weights are used as shipped (FP8 E4M3 with UE8M0 block scales, MXFP4 routed experts). On the TP4 points --monokernel also runs the routed-expert FFN of the three draft layers as one launch. That launch quantizes the FFN input to MXFP8 against the same MXFP4 expert weights, as it does for the target layers, where vLLM's gfx942 MoE multiplies bf16 activations with them. The draft weights, its KV cache and its attention are unchanged, and the target model's output does not depend on this launch, only how many drafts it accepts." | ||
| pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX |
There was a problem hiding this comment.
🔴 Readers of the changelog get a pr-link to pull/XXXX, which leads nowhere, so this entry cannot be traced back to its PR. The new block at the end of the file still has the placeholder. Fix: replace https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX with this PR's real number before merge. Edit only that line, because the file is append-only and byte-sensitive.
Why this was flagged
The appended block for config key dsv41flash-fp4-mi325x-vllm-agentic-dspark ends with pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX (perf-changelog.yaml:11504). Every earlier entry links to a real PR, for example pull/4571 just above it. The file is append-only, so after merge a fix would need another change to an existing byte. Until then, anyone following the link from the published perf history lands on a 404. On the base branch the file has no such placeholder.
Verification: Trigger: the PR merges with the appended block as written. The diff adds pr-link: https://github.com/SemiAnalysisAI/InferenceX/pull/XXXX as the last line of the new block in inferencex-e2e/perf-changelog.yaml, and the base file has no such placeholder. It uses four X's, and the repo's tooling only accepts three, per validate_perf_changelog.py:21-24.
| args: | ||
| tensor-parallel-size: 4 | ||
| engram-config: '{"cpu_offload":true}' | ||
| monokernel: true |
There was a problem hiding this comment.
🟡 (optional) The automatic GSM8K for this config never runs the new TP4 mono decode path. Its accuracy is unchecked on this sweep and on every later image bump. The kernel is switched by the CLI arg monokernel: true, not an env var. Agentic GSM8K groups split only on engine env, so TP4 shares one group with TP8 and TP2. That group's single pick is TP8 c32. Fix: have the automatic eval cover a TP4 point whose decode steps stay within 42 rows (e.g. c4). Either split mark_eval_entries groups on kernel-selecting args like monokernel (code outside these files), or schedule that eval with this sweep.
Why this was flagged
The agentic GSM8K pick in infx/matrix/generate.py:802-815 groups single-node rows by model, runner, framework, precision, spec, dp-attn, image and _engine_env. Only the max-conc row of each group runs GSM8K. engine_environment (infx/matrix/fingerprint.py:96-101) hashes only recipe environment and role env, never role args. infx/evals/EVALS.md:21-26 says a kernel path switched by an env var gets its own eval, but "CLI args" do not split groups. The TP4 overrides add monokernel: true as an arg (agentic.yaml:144, 156, 168, 180, 192, 204) and leave env equal to TP8 and TP2. So all 18 points form one group, and the pick is the first conc-32 row, TP8 c32 (amd-master.yaml:1194). The PR description says the mono FFN launch quantizes activations to MXFP8 where the regular gfx942 MoE uses bf16. On the base branch no point ran a mono kernel, so the shared eval covered every kernel path. After the merge, TP4's mono path gets no automatic accuracy check. The author's one-off TP4 c4 GSM8K run does not carry over to later image bumps of this config key.
Verification: The automatic agentic GSM8K never runs on the TP4 --monokernel path. engine_environment (infx/matrix/fingerprint.py:95-102) serializes only the recipe-wide environment and each role's env. Role args never enter it. monokernel: true is added under roles.agg.args (agentic.yaml:144), so all 18 points fall into one group and infx/matrix/generate.py:815 picks TP8 c32.
There was a problem hiding this comment.
Not changed in this PR, on purpose. Two reasons.
The grouping of the agentic GSM8K is shared code in infx/matrix/generate.py (mark_eval_entries), and #4587 changed it two days ago to split groups by engine environment. Splitting on kernel-selecting server arguments such as monokernel changes the eval picks of every config key that sets such an argument, the MI355X DeepSeek-V4.1-Flash entry included, which has monokernel: true on all its points. That belongs in its own PR with its own review, not in an image and flag update of one entry.
This PR does have a GSM8K run on the mono path: TP4 c4 with --monokernel, where every decode step runs on the mono kernels, 97.42% strict match on 1,319 samples (run 40718348873). The accuracy checks of the kernels themselves are in vllm-project/vllm#60731 (3 GSM8K runs per arm at TP4 c7, GPQA-Diamond, AIME25 and RULER NIAH). I agree that a later image update of this entry would not check the mono path again, and I will open a separate issue or PR for the eval grouping if the maintainers want it.
| # and stops capturing above 512 tokens. TP4 runs the gfx942 mono decode kernels | ||
| # (vllm-project/vllm#60731), which build for TP4 only: two persistent launches a | ||
| # layer for decode steps of up to 42 rows (7 requests with five drafts). Larger | ||
| # steps take the regular path. |
There was a problem hiding this comment.
🟡 nit (optional): The new TP4 sentences grow a section-header comment over the override variants. AGENTS.md (Comments) does not allow header comments that describe an override or section. They describe what #60731 does in general, not why a particular line is set. Fix: drop them, or put a single short line above the first monokernel: true (override_tp4_c1) saying the gfx942 mono decode kernels build for TP4 only (vllm-project/vllm#60731).
Why this was flagged
The diff adds four lines (agentic.yaml:67-70) to the comment above override_tp8_c1. They explain the TP4 mono decode kernels: two persistent launches per layer, at most 42 rows, and that larger steps take the regular path. AGENTS.md's mandatory Comments rule bans header comments that describe a file, config, recipe, override or section, and allows only a short technical reason attached to a specific line. Nothing breaks at runtime. The issue is that the repo's comment convention is broken in a file the PR changes.
Verification: Nothing breaks at runtime. The diff adds agentic.yaml:67-70 to the section-header comment at line 65, above override_tp8_c1, far from the monokernel: true lines it explains (lines 144-204). AGENTS.md:56 says "No header comments describing a file, topology, config, recipe, override, or section." The new sentences describe #60731 in general, not a specific line.
There was a problem hiding this comment.
Fixed in 7c37502. The four lines above override_tp8_c1 are removed, and one line above the first monokernel: true in override_tp4_c1 says that the gfx942 mono decode kernels build for TP4 only (vllm-project/vllm#60731). The change is in a comment only, and the 18 generated rows, recipe fingerprints included, are the same as before.
… flag AGENTS.md allows no comments that describe an override section, so the four lines above override_tp8_c1 become one line above the first monokernel: true. Comment only, the recipe fingerprints are unchanged. Signed-off-by: Markus Hartikainen <markus.hartikainen@amd.com>
chunfangamd
added
the
full-sweep-fail-fast
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=40744174186 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=40744174186 |
|
/reuse-sweep-run 40744174186 |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40744174186
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40744174186
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- Recipe: vllm-project/vllm#64589
- Check 2: #4617 (comment)
Signed: @chunfangamd
functionstackx
left a comment
functionstackx
left a comment
There was a problem hiding this comment.
@chunfangamd plz update the vllm recipes using the new way for agentx optmized configs also!
as i mentioned in slack, upstream maintainers have an new way to update recipes in vLLM for agentx. going forward plz point ur agents to look at these PRs on how to do it. plz forward to whoever at AMD is updating vLLM recipes
https://github.com/vllm-project/recipes/pull/1162/changes
https://github.com/vllm-project/vllm/pull/63560/changes
❌❌❌ REJECTED ❌❌❌@chunfangamd Two checks block this sign-off. On the six TP4 ❌ Check 2 (Evals pass): FAIL — No eval log that this verifier could inspect shows that the TP4 ❌ Check 13 (Draft precision): FAIL — The six TP4 points ( Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (In-PR sweep): PASS — In-PR commit ✅ Check 3 (Upstream recipe): PASS — vllm-project/vllm#64589 is MERGED (2026-11-02T17:46Z). It adds the ✅ Check 4 (Reuse command): PASS — ✅ Check 5 (Checklist template): PASS — All 20 items of the current template are present and checked. ✅ Check 6 (Upstream image): PASS — The image ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (No architecture hacks): PASS — The recipe sets no ✅ Check 9 (Chat templates): PASS — The AgentX harness replays traces through chat completions with the ✅ Check 10 (No engine patches): PASS — No patches, setup scripts, or source edits are reachable from this recipe. ✅ Check 11 (Golden AL): PASS — Throughput jobs set ➖ Check 12 (Append-only): N/A — No new changelog entry uses ✅ Check 14 (Pareto coverage): PASS — The curve dsv41flash/agentic-coding/mi325x vLLM/fp4/P90 E2EL from 40744174186 ( ✅ Check 15 (PR description): PASS — The image, flag scope, removed flag, 18 points, run links and eval figures match the pinned code and the inspected runs. ➖ Check 16 (NVIDIA overview): N/A — MI325X is AMD hardware only. ➖ Check 17 (Metrics publication): N/A — This is direct vLLM with no Dynamo TRT-LLM. ✅ Check 18 (Recipe consistency): PASS — Generation succeeds, the master TP8/4/2 matches the recipe Assessed commit: |
|
Check 2 evidence, on the current head I ran the entry's TP4 c4 eval-only job again on this head, where GSM8K strict match 0.97498 ± 0.0043, flexible extract 0.97422, 1,319 samples. The TP8 c32 eval of the sweep is 0.97422. The server log of that job ( So the accuracy number in this job comes from a server that ran the mono kernels on the target layers and on the DSpark draft layers' FFN, with decoder SWA bounded replay on. |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40744174186
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40744174186
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that
SGLANG_NVFP4_CKPT_FP8_NEXTN_MOEis not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison. - For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting
ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md. - For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- For official InferenceX (non-preview), verified that established models use upstream engine images built from
master(or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md. - If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
- Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.
Additional detail section:
- Recipe: vllm-project/vllm#64589
- Check 2: accuracy and TP4 mono-path coverage. The [full sweep](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40744174186) scored 97.4223% on TP8 c32. The supplemental [TP4 c4 eval](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40919069311/job/123607325199), at
c8befe0ed1f0, scored 97.4981% on 1,319 GSM8K samples using the same pinned image and real block rejection. Its recipe remains unchanged at assessed head92aa1fb282c9. Both scores satisfy the two-SE comparison against the [97.6497% matching cohort](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/40183897667).
The previously archived [server log is now directly readable](https://github.com/user-attachments/files/34286293/40919069311-tp4c4-server-log.txt) and matches the extracted artifact byte-for-byte. It records bothmono decode FFN launch activeandmono decode layer activeduring graph capture, followed by the scored evaluation with four running requests. With five draft tokens, c4 produces at most 24 verification rows, within the 42-row mono limit; the captured graph sizes include 24. Together with the [pinned dispatch guards](https://github.com/vllm-project/vllm/blob/edad8b03c5285a1c12e7e4f83c1af709b319a01b/vllm/models/deepseek_v41/amd/mono_decode.py#L615), this supports mono coverage on eligible decode-only steps. Prefill and mixed steps retain their intended fallback. - Check 13 (Draft prpecision): #4617 (comment)
Signed: @chunfangamd
❌❌❌ REJECTED ❌❌❌@chunfangamd Blocking: on the TP4 points, ❌ Check 13 (Draft precision): FAIL — Passed and not applicable checks✅ Check 0 (CODEOWNER): PASS — ✅ Check 1 (Sweep on in-PR commit): PASS — run 40744174186 ran on in-PR commit ✅ Check 2 (Evals): PASS — In run 40744174186, TP8 c32 GSM8K scored 0.9742 ± 0.0044 on the pinned image, a gap of 0.0023 against the 0.9765 exact cohort (threshold 0.0121). The supplemental TP4 c4 ✅ Check 3 (Upstream recipe): PASS — vllm-project/vllm#64589 is MERGED (2026-11-02). It adds the ✅ Check 4 (Reuse command): PASS — COLLABORATOR ✅ Check 5 (Checklist template): PASS — all 20 items of the current template are present and checked. ✅ Check 6 (Upstream image): PASS — ✅ Check 7 (Deprecated models): PASS — ✅ Check 8 (Architecture hacks): PASS — no hf-overrides or config edits. Decoder SWA bounded replay is the pinned image's default and skips only unused upper-layer prefill rows. The TP8 and TP4 evals cover the configs. ✅ Check 9 (Chat templates): PASS — the AgentX trace replay drives the chat endpoint, and evals use ✅ Check 10 (Engine patches): PASS — the PR changes only YAML args, image and changelog. No reachable patch, setup script or wheel install. ✅ Check 11 (Golden AL): PASS — the connector injected ➖ Check 12 (Append-only): N/A — the new changelog entry is not ✅ Check 14 (Pareto coverage): PASS — the single curve (dsv41flash / agentic-coding / MI325X vLLM / fp4 / run 40744174186 / p90 E2EL) has 5 frontier points of 18 measured. They are TP4 c8, TP4 c16, TP2 c16, TP4 c32 and TP2 c32, taken from the ✅ Check 15 (PR description): PASS — the image, flags, removed option, 18-point scope, adaptive-verification state and labeled validation runs ( ➖ Check 16 (NVIDIA overview): N/A — AMD MI325X only. ➖ Check 17 (Metrics publication): N/A — plain vLLM, with no Dynamo TRT-LLM path. ✅ Check 18 (Recipe/image consistency): PASS — the config generates at head, master TP matches recipe GPUs, and the image digest resolves. No orchestration pins or dispatch changes. The changelog accurately describes the image, Assessed commit: |
|
No accuracy difference shows between the two arms. Every paired difference is within run-to-run variation: the 95% intervals for tau2-bench, HLE and MRCR all contain zero, and the acceptance length, where a change in draft precision would show first, moves by 0.02 (0.6%). Paired accuracy A/B for Check 13 (draft precision), on MI325X with the pinned image
|
Re-sweeps the MI325X DeepSeek-V4.1-Flash vLLM DSpark AgentX recipe (
dsv41flash-fp4-mi325x-vllm-agentic-dspark) on a ROCm 10.0 nightly that carries vllm-project/vllm#60731 (gfx942 mono decode), its fix vllm-project/vllm#63576, and vllm-project/vllm#59039 (decoder SWA bounded replay on ROCm).Changes
v0.32.0([Klaud Cold] Update DeepSeek-V4.1-Flash FP4 MI325X vLLM AgentX DSpark image to v0.32.0 / [Klaud Cold] 将 DeepSeek-V4.1-Flash FP4 MI325X vLLM AgentX DSpark 镜像更新至 v0.32.0 #4496) tonightly-rocm100-edad8b03c5285a1c12e7e4f83c1af709b319a01b@sha256:009169c114af5b15e25eddc808afb3c1fc3f12c94e746472b16f39fcf39cd716, the 2026-10-28 ROCm 10.0 nightly. It is 162 commits past #63576 (9744ada58292), so it contains #60731 (1a74cc7bb24e), #63576 and #59039 (c077dc663d92).v0.32.0(5df63a72) contains none of the three.monokernel: true(--monokernel) to the six TP4 variants (override_tp4_c1throughoverride_tp4_c32). On gfx942 #60731 runs each backbone layer of a decode step as two persistent launches, and the routed-expert FFN of the three DSpark draft layers as one launch, for decode steps of up to 42 rows (7 requests with five DSpark drafts). Larger steps take the regular path. The gfx942 kernels build for TP4 only (mono_decode.pydeclines any other TP on gfx942), so the TP8 and TP2 variants stay withoutmonokernel: trueand change only by the image and by the removal ofno-swa-bounded-replay: true. The upstream vLLM recipe turns--monokernelon for MI325X TP4 in vllm-project/vllm#62713.no-swa-bounded-replay: true, which turns on #59039's decoder SWA bounded replay at every point: in eager prefill steps (every prefill step here, sincecudagraph_modeisFULL_DECODE_ONLY), layer 20 writes its KV for every token, and the rest of layer 20 and layers 21-39 run on each request's last 128 tokens only. Prefix-cache SWA replay, the path whose missing window clamp the flag worked around, stays off: vLLM still forces it off on ROCm.enable_adaptive_verification: false. With--monokernelon gfx942 the server faults during the adaptive-verification CUDA graph profile at startup until vllm-project/vllm#63816 merges.FULL_DECODE_ONLYwithVLLM_USE_BREAKABLE_CUDAGRAPH=0.get_draft_quant_config) and the target'sfp8_ds_mlaKV cache, and its weights are used as shipped (FP8 E4M3 with 32x32 UE8M0 block scales, MXFP4 routed experts, BF16 norms and heads). On the TP4 variants--monokernelalso runs the routed-expert FFN of the three draft layers as one launch, which quantizes the FFN input to MXFP8 against the same MXFP4 expert weights, as it does for the target layers, where vLLM's gfx942 MoE multiplies bf16 activations with them. The draft's weights, its KV cache and its attention are unchanged, and the target model's output does not depend on this launch, only how many drafts it accepts.Validation
Five points of this entry at commit
3df335bfe629, throughe2e-tests.ymloncluster:mi325x-amd, 3,600 s each, scored by InferenceX. The reference is the publishedv0.32.0run of the same points (40183897667). Interactivity in tokens/s/user, throughput in total tokens/s per GPU.--monokernel--monokernel--monokernelThe three TP4 server logs show
DSv4.1 mono decode layer active (decode steps of <= 42 rows), the FFN launch for the DSpark draft layers andDecoder SWA bounded replay, and no line that turns the kernels off. TP4 c8 has steps of 48 rows at full load, which take vLLM's path, and gains from the steps of up to 42 rows. TP8 and TP2 change only by the image and bounded replay. At most 2 request errors per point, as in the v0.32.0 run.GSM8K at TP4 c4 with
--monokernel, where every decode step runs on the mono path: 97.42% strict match (± 0.44%), 97.35% flexible extract, 1,319 samples (40718348873). The v0.32.0 sweep has 97.65% at TP8 c32. The full sweep of all 18 points follows with the sweep label.For reviewers
v0.32.0), and the Validation table compares against them.AI model disclosure
Prepared through a model router that sends each request to Claude Opus 5.5 (
Claude-Opus-5.5), Gemini 4 Argon (gemini-4-argon) or Grok 4.7 (Grok-4.7), whichever is up: the recipe, master-config andperf-changelog.yamledits, the local matrix and changelog checks, and this description. The runtime does not record which of the three answered each step, so the model of each step could not be verified. Reviewed and edited by the author.