[Fangzhou-Ai]

Summary

Adds a TP8, concurrency-1 monokernel point for DeepSeek-V4.1-Flash on MI355X with vLLM. Removes INT4 QuickReduce from the shared recipe by explicitly setting VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: NONE, so integer QuickReduce is disabled for TP2, TP4 and TP8.

The TP8 key uses vllm/vllm-openai-rocm:nightly-335dd77b09161c5d63139b4e7ae51bb9928305b1 pinned to digest sha256:3c634cc2c51eda7eb840a236b1f09a5ecf6a4b0cb92c4b8486e2158770507291. The existing TP4/TP2 key retains its nightly-e68efc685ab1e6f1eda004ced7157e3ca411da2f image.

• Adds dsv41flash-fp4-mi355x-vllm-agentic-dspark-tp8-c1 and override_tp8_c1, with GPU-resident Engram tables, max-num-seqs: 1, max-cudagraph-capture-size: 8 and max-num-batched-tokens: 32768.
• The shared QuickReduce change affects both config keys. The appended changelog entry includes the existing TP4/TP2 grid and the TP8 point in the full sweep, superseding the earlier TP8-only scope.
• Monokernel decode, five-token DSpark, checkpoint, KV-cache configuration and draft precision remain unchanged.

The pinned TP8 nightly includes vllm-project/vllm#64101, which enables mono decode at TP8 on gfx950. The earlier nightly failed at startup outside TP2/TP4. The TP8 key remains a temporary scan to retire after collecting the point.

Earlier validation

Run 41098938196 on fa9d486 passed its TP8 benchmark and eval-only jobs. That run predates removal of INT4 QuickReduce and does not validate this revised recipe.

Co-authored-by: @okorzh-amd

中文

新增 DeepSeek-V4.1-Flash 在 MI355X 上使用 vLLM 的 TP8、并发 1 monokernel 测试点。共享 recipe 中将 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION 从 INT4 改为 NONE，为 TP2、TP4 和 TP8 显式禁用整数 QuickReduce。

TP8 使用固定 digest 的 nightly-335dd77b09161c5d63139b4e7ae51bb9928305b1，现有 TP4/TP2 保留 nightly-e68efc685ab1e6f1eda004ced7157e3ca411da2f。新增 changelog 条目将两个配置键均纳入完整 sweep，取代此前仅运行 TP8 的范围。Monokernel、五 token DSpark、checkpoint、KV cache 配置及草稿精度保持不变。

此前 41098938196 在 fa9d486 上的 TP8 基准测试和评测通过，但该结果早于禁用 INT4 QuickReduce，不能验证修改后的 recipe。

共同作者：@okorzh-amd

AI model disclosure

• Model/version: claude-sonnet-5-5 (Claude Sonnet 5.5)
• Role: prepared the original config, recipe variant, changelog entry and PR.
• Model/version: claude-opus-5-5 (Claude Opus 5.5)
• Role: rewrote the original perf-changelog block, checked draft data types at vLLM 335dd77b0 and updated the earlier description.
• Model/version: GPT-6 Astra
• Role: rebased onto upstream main, disabled integer QuickReduce, appended the affected-config changelog entry and revised this description.

[github-actions]

Thanks for the contribution!

• Review: If this PR changes files owned by someone other than a repository admin or @SemiAnalysisAI/core, ask one eligible CODEOWNER to complete the latest PR_REVIEW_CHECKLIST.md before contacting a core maintainer on Slack. Follow the template exactly, including As a PR reviewer and CODEOWNER, I have reviewed this and have, so sign-off verification triggers.
• PR verification: Sweeps run only when the PR appends an inferencex-e2e/perf-changelog.yaml entry and carries exactly one primary label: full-sweep-fail-fast (strongly recommended; canary plus per-matrix fail-fast), full-sweep-enabled (canary; matrix jobs continue after a failure), or non-canary-full-sweep-enabled (no canary or fail-fast). The modifiers all-evals, evals-only, and agentx-fast require a primary label. On fork PRs, a maintainer applies the label. See sweep labels and reuse.
• After merging: PR authors must ensure all GitHub Actions jobs pass. Transient failures often pass on rerun; see how to rerun failed jobs.

中文

感谢你的贡献！

• **审阅：**如果 PR 修改的文件归属于仓库管理员及 @SemiAnalysisAI/core 之外的 CODEOWNER，请先联系一位有资格的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，再通过 Slack 联系核心维护者。必须严格遵循模板，并保留 As a PR reviewer and CODEOWNER, I have reviewed this and have，才能触发签核验证。
• **PR 验证：**只有当 PR 在 inferencex-e2e/perf-changelog.yaml 末尾追加了条目，并且恰好带有一个主标签时，才会运行扫描：full-sweep-fail-fast（强烈推荐；canary 加逐矩阵 fail-fast）、full-sweep-enabled（有 canary；矩阵任务在失败后继续运行）或 non-canary-full-sweep-enabled（无 canary，也无 fail-fast）。修饰标签 all-evals、evals-only 和 agentx-fast 必须与主标签一起使用。fork PR 的标签由维护者添加。参见扫描标签与复用。
• **合并后：**PR 作者必须确保所有 GitHub Actions 任务通过。临时性失败通常可以通过重新运行恢复；参见重新运行失败任务的说明。

[claude]

Beyond the inline findings, I checked two other things and neither is a problem. First, override_tp8_c1 does not need its own monokernel setting, because it inherits monokernel: true from the shared base args at agentic.yaml:52, and c1 with five drafts is well under the 48-row mono decode limit. Second, the pull/XXX changelog link follows a placeholder pattern that existing entries already use, for example perf-changelog.yaml:10792.

Extended reasoning...

Two concerns beyond the inline comments were checked and ruled out. Mono decode is set in the base recipe args that every override inherits. The placeholder pr-link matches existing changelog entries.

Findings marked 🟡 are optional suggestions and need no follow-up push.

[claude]

🔴 Merging this would likely replace the published MI355X vLLM DSv4.1-Flash AgentX curve (TP4 c1-c2, TP2 c1-c64) with one TP8 c1 point on a different image. The dashboard groups by model, hardware+framework, precision and run, not config key. A regular (non-append-only) run of a new key in that series becomes the newest curve. The comment at :1206-1207 assumes a separate key isolates the scan, but it does not. Fix: keep the existing curve as the newest published run. Either collect the scan from the PR sweep without merging, or add TP8 c1 to dsv41flash-fp4-mi355x-vllm-agentic-dspark as append-only: true on its current image.

Why this was flagged

The new key at amd-master.yaml:1208 has the same model-prefix, runner, framework and precision as dsv41flash-fp4-mi355x-vllm-agentic-dspark (:1183). It generates one point: tp 8, conc 1 (:1221). The changelog entry at perf-changelog.yaml:11673-11681 has no append-only: true. After /use and merge, that one-job run is ingested as a normal run of the same app series. .github/klaud-candidate-prompt.md:63 and infx/klaud/main.py:169-171 say "the newest normal run replaces that whole curve". CONTRIBUTING.md:177-182 says append-only is the only way to extend a curve instead of showing a partial run as its own curve. On the base branch, the dashboard shows the 9-point TP4/TP2 frontier on the e68efc68 image. After merge it shows one TP8 c1 point on the 3966924d image. append-only cannot fix this as written, because it requires the curve's same non-null image (CONTRIBUTING.md:191-192), and this key pins a new image.

Verification: The new changelog block (perf-changelog.yaml, appended at the tail) has no append-only: true. So once merged and ingested, the newest run of this MI355X/vLLM/fp4 DSv4.1-Flash AgentX series has one point (TP8 c1, on a different nightly image). It replaces the published multi-point TP4/TP2 frontier, per inferencex-e2e/infx/klaud/main.py:169-171.

[claude]

🟡 nit (optional): two new comments describe a config and an override instead of explaining a value, which AGENTS.md forbids. sweep:^# (Single-point TP8 c1 scan|TP8 c1 mirrors TP4 c1) The block at amd-master.yaml:1205-1207 summarizes the key and adds dated "latest nightly (2026-10-29)" context. agentic.yaml:177 restates the override's own args. Fix: drop both comments; the perf-changelog.yaml entry already records the scan, image and rationale.

Why this was flagged

AGENTS.md "Comments (mandatory, every file)" bans header comments describing a config, recipe or override, and bans history or past context; comments may only explain a specific value and must not restate code. inferencex-e2e/configs/amd-master.yaml:1205-1207 is a header for the new key dsv41flash-fp4-mi355x-vllm-agentic-dspark-tp8-c1. It also carries a dated "latest ... nightly (2026-10-29)" claim that goes stale. inferencex-e2e/benchmarks/single_node/srt-slurm-recipes/dsv41flash/vllm/mi355x-fp4-mtp/agentic.yaml:177 ("TP8 c1 mirrors TP4 c1: one sequence at a time, Engram tables GPU-resident.") restates max-num-seqs: 1 and engram-config: '{"cpu_offload":false}' in the lines below it. Merging this adds two convention violations that the base branch does not have. There is no runtime impact.

Verification: nit. AGENTS.md:54-58 says: "No header comments describing a file, topology, config, recipe, override, or section". The diff adds two. inferencex-e2e/configs/amd-master.yaml:1205-1207 is a header for the new key that also carries dated provenance ("latest ... nightly (2026-10-29)"). agentic.yaml:177 describes the override_tp8_c1 block. Nothing breaks at runtime.

[github-actions]

View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41394320841

View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41394320841

[Fangzhou-Ai]

Upstream vLLM recipe PR for this lane: vllm-project/vllm#64804 (adds the mi355x_agentx_tp8 preset to the DeepSeek-V4.1-Flash recipe). Still open; the checklist requires it to be MERGED before this PR merges.

中文

本 lane 对应的上游 vLLM recipe PR：https://github.com/vllm-project/vllm/pull/64804（在 DeepSeek-V4.1-Flash recipe 中新增 mi355x_agentx_tp8 preset）。目前仍为 open；检查清单要求其在本 PR 合并前已 MERGED。

[seungrokj]

Draft data types: no change. The DSpark draft (mtp.0-2 in deepseek-ai/DeepSeek-V4.1-Flash) is stored exactly like the target's layers (read from the safetensors headers of shards 44-46):

Part	Data type
Attention projections (wq_a, wq_b, wkv, wo_a, wo_b), shared experts, main_proj	FP8 E4M3 with 32×32 UE8M0 block scales
Routed experts	MXFP4 (packed I8 with E8M0 scales)
Norms, router gate, Markov and confidence heads	BF16
Attention sinks, router bias, hc mixers	F32

At vLLM 335dd77b0 (the image's revision), the AMD DSpark model builds its layers with the draft config's quant config (dspark.py#L310). That config is the checkpoint's own FP8 config (fp8, block 32×32, ue8m0, expert_dtype: fp4), via get_draft_quant_config. With no kv_cache_dtype in speculative-config, the draft inherits the target's auto KV dtype, which DSv4.1 resolves to fp8_ds_mla (attention.py#L160). The mono decode kernels identify the draft layers at mono_decode.py#L397 and read the loaded weights in place; vllm-project/vllm#64101 only widens the tensor-parallel check to TP8 (mono_decode.py#L357) and pads the 8 heads per rank into one MFMA tile. The recipe sets no online quantization and no draft dtype.

Between e68efc685 (the TP4/TP2 key's image) and 335dd77b0, the only draft-relevant change is vllm-project/vllm#61875: the MXFP4 MoE now declares MXFP8 activations (quant_config.py#L261). With --moe-backend aiter, no AITER backend matches that key, so _filter_by_activation falls back to the same AITER_MXFP4_BF16 backend as before, for the target and the draft alike.

Sign-off evidence (PR review checklist)

The CODEOWNER sign-off links this comment as its additional detail section. Evidence for each item, assessed at 537d3f5 (head) / fa9d486 (swept); the upstream recipe row was updated on 2026-11-02 after vllm-project/vllm#64804 merged:

Item	Status	Evidence
PR validation	✅	Run 41098938196 on fa9d486: the one-point matrix (TP8 c1) ran as the canary agentic job, full 3600 s. Later commits change only the changelog text and merge main.
Evals	✅	Same run, agentic eval: gsm8k strict-match 0.9719, flexible-extract 0.9712 (threshold 0.9), TP8 with --monokernel.
Chat templates	✅	AgentX replays the traces through /v1/chat/completions, so the served chat template applies.
Draft precision	✅	Above: the mtp.0-2 draft runs as shipped (FP8 E4M3 32×32 UE8M0, MXFP4 experts, BF16 norms/heads) with an fp8_ds_mla KV cache; no draft quantization or dtype override. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not set (vLLM path).
Golden AL	✅	DSpark k=5 throughput runs use synthetic acceptance 3.51, the dsv41flash_dspark.yaml thinking_on value at k=5.
MODELS.md	✅	dsv41flash agentic coding is active.
Architecture hacks	✅	No --hf-overrides or layer skipping; --monokernel fuses the same computation.
Integer all-reduce	✅ (grandfathered)	The recipe sets VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4. dsv41flash is in the policy-baseline model matrix, so the exception applies.
Dynamo TRT-LLM metrics	N/A	Direct vLLM serving; no Dynamo.
Upstream image / main provenance / engine-first	✅	vllm/vllm-openai-rocm:nightly-335dd77b0…@sha256:3c634cc2…, the upstream vLLM org; 335dd77b0 is on vLLM main and carries vllm-project/vllm#64101. vLLM is the engine itself, so ordering does not apply.
Upstream recipe	✅	vllm-project/vllm#64804 was merged on 2026-10-31 (3f9d1847) and adds the mi355x_agentx_tp8 preset, "AgentX TP8 conc 1" (DeepSeek-V4.1-Flash.yaml#L1070-L1117), now live on the published recipe. It cites this PR at fa9d486, pins the same image and digest (nightly-335dd77b0…@sha256:3c634cc2…), and sets the same values as override_tp8_c1: tp: 8, --max-num-seqs 1, --max-num-batched-tokens 32768, --max-cudagraph-capture-size 8, --engram-config {"cpu_offload":false} and OMP_NUM_THREADS=1. The shared settings come from the recipe's MI355X single_node_tp override (L290-L312: --monokernel, the MXFP4 sparse-logits indexer, --block-size 128, the AITER Triton sparse MLA, INT4 quick all-reduce), the AMD hardware override (L215-L231: --moe-backend aiter, --gpu-memory-utilization 0.9, breakable CUDA graphs), and the AMD DSpark override (L91-L94: 5 tokens, probabilistic drafting, block rejection, adaptive verification off). These match the base recipe here.
Engine patches	✅	No patches, setup scripts, overlays or waivers; the pinned image runs as shipped.
Append-only	N/A	The changelog block does not set append-only: true; it adds a new config key.
Pareto counts	⚠️ 1/5	One curve: dsv41flash / agentic-coding / MI355X vLLM / fp4 / p90, run 41098938196 artifact results_bmk: 1 measured frontier point (TP8 c1, tput_per_gpu 3940.25, p90 E2EL 7.545 s). This is a single-point scan, so it needs a core maintainer's review and a recorded admin bypass before merge.

中文

草稿模型数据类型：无变化。 DSpark 草稿（deepseek-ai/DeepSeek-V4.1-Flash 中的 mtp.0-2）的存储方式与目标模型各层完全相同（读取自第 44-46 个分片的 safetensors 头）：

部分	数据类型
注意力投影（wq_a、wq_b、wkv、wo_a、wo_b）、共享专家、main_proj	FP8 E4M3，32×32 UE8M0 块缩放
路由专家	MXFP4（打包为 I8，E8M0 缩放）
Norm、路由门控、Markov 头与置信度头	BF16
注意力 sink、路由偏置、hc mixer	F32

在 vLLM 335dd77b0（镜像对应的版本）中，AMD DSpark 模型使用草稿配置的量化配置构建各层（dspark.py#L310）。该配置即 checkpoint 自带的 FP8 配置（fp8、32×32 块、ue8m0、expert_dtype: fp4），通过 get_draft_quant_config 获得。speculative-config 未设置 kv_cache_dtype，因此草稿沿用目标模型的 auto KV 类型，DSv4.1 将其解析为 fp8_ds_mla（attention.py#L160）。mono decode 内核在 mono_decode.py#L397 识别草稿层，并直接读取已加载的权重；vllm-project/vllm#64101 只是把张量并行检查放宽到 TP8（mono_decode.py#L357），并把每个 rank 的 8 个 head 填充进一个 MFMA tile。recipe 没有设置在线量化，也没有设置草稿数据类型。

在 e68efc685（TP4/TP2 配置键的镜像）与 335dd77b0 之间，唯一与草稿相关的改动是 vllm-project/vllm#61875：MXFP4 MoE 现在声明 MXFP8 激活（quant_config.py#L261）。在 --moe-backend aiter 下没有 AITER 后端匹配该类型，因此 _filter_by_activation 回退到与之前相同的 AITER_MXFP4_BF16 后端，目标模型与草稿均如此。

签核证据（PR 审查清单）

CODEOWNER 签核将本评论链接为其附加说明部分。各项证据如下（head 537d3f5，扫描提交 fa9d486）；上游配方一行已于 2026-11-02 在 vllm-project/vllm#64804 合并后更新：

项目	状态	证据
PR 验证	✅	fa9d486 上的运行 41098938196：单点矩阵（TP8 c1）以 canary agentic 任务完整运行 3600 s。之后的提交仅修改 changelog 文本并合并 main。
评测	✅	同一运行的 agentic eval：gsm8k strict-match 0.9719、flexible-extract 0.9712（阈值 0.9），TP8 并启用 --monokernel。
聊天模板	✅	AgentX 通过 /v1/chat/completions 回放 trace，使用服务端聊天模板。
草稿精度	✅	见上文：mtp.0-2 草稿按原样运行（FP8 E4M3 32×32 UE8M0、MXFP4 专家、BF16 norm/头），KV 缓存为 fp8_ds_mla；无草稿量化或 dtype 覆盖。未设置 SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE（vLLM 路径）。
Golden AL	✅	DSpark k=5 吞吐运行使用合成接受长度 3.51，即 dsv41flash_dspark.yaml thinking_on 在 k=5 的值。
MODELS.md	✅	dsv41flash agentic coding 处于活跃状态。
架构 hack	✅	无 --hf-overrides 或跳层；--monokernel 融合的是相同计算。
整数 all-reduce	✅（豁免）	配方设置 VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4；dsv41flash 位于政策基线模型矩阵中，适用豁免。
Dynamo TRT-LLM 指标	不适用	直接使用 vLLM 服务，无 Dynamo。
上游镜像 / main 来源 / 引擎优先	✅	vllm/vllm-openai-rocm:nightly-335dd77b0…@sha256:3c634cc2…，来自上游 vLLM 组织；335dd77b0 位于 vLLM main，包含 vllm-project/vllm#64101。vLLM 本身即引擎，顺序规则不适用。
上游配方	✅	vllm-project/vllm#64804 已于 2026-10-31 合并（3f9d1847），新增 mi355x_agentx_tp8 预设“AgentX TP8 conc 1”（DeepSeek-V4.1-Flash.yaml#L1070-L1117），现已上线已发布配方。该预设引用本 PR 的 fa9d486，固定相同的镜像与 digest（nightly-335dd77b0…@sha256:3c634cc2…），参数与 override_tp8_c1 相同：tp: 8、--max-num-seqs 1、--max-num-batched-tokens 32768、--max-cudagraph-capture-size 8、--engram-config {"cpu_offload":false} 和 OMP_NUM_THREADS=1。共享设置来自配方的 MI355X single_node_tp 覆盖（L290-L312：--monokernel、MXFP4 sparse-logits indexer、--block-size 128、AITER Triton sparse MLA、INT4 quick all-reduce）、AMD 硬件覆盖（L215-L231：--moe-backend aiter、--gpu-memory-utilization 0.9、可中断 CUDA graph）以及 AMD DSpark 覆盖（L91-L94：5 个 token、概率式草稿、块拒绝、关闭自适应验证），与本 PR 的基础配方一致。
引擎补丁	✅	无补丁、setup 脚本、overlay 或豁免；固定镜像按原样运行。
仅追加	不适用	changelog 条目未设置 append-only: true，新增的是独立配置键。
Pareto 点数	⚠️ 1/5	一条曲线：dsv41flash / agentic-coding / MI355X vLLM / fp4 / p90，运行 41098938196 的 results_bmk 产物：1 个实测前沿点（TP8 c1，tput_per_gpu 3940.25，p90 E2EL 7.545 s）。这是单点扫描，合并前需要核心维护者审查并记录管理员豁免。

[seungrokj]

As a PR reviewer and CODEOWNER, I have reviewed this and have:

Checklist items

• Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
• Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
• Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this.
• Verified that this PR passes evals. Please link to GitHub Action workflow that shows this.
• Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
• Verified that every draft model and draft head is served as it ships: the draft that ships with the served checkpoint, at its stored precision, through the pinned upstream image's default handling, with the shipped and effective draft precision recorded in the additional detail section. No submission-side quantization, dtype override, checkpoint substitution, or patch may lower draft precision below that default, regardless of eval results or AL. Explicitly verified that SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not enabled in the effective recipe, including inherited settings; enabling it is prohibited going forward, and historical runs do not grant an exception. See Draft-model precision for what counts as the default and the MLPerf comparison.
• For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in infx/golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
• Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
• Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; target/verifier FLOPs at lower precisions is fine, given that the config passes private evals, but this does not permit lowering draft-model or draft-head precision below what ships. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
• Verified that new models disable integer all-reduce, with SGLang ROCm explicitly setting ROCM_QUICK_REDUCE_QUANTIZATION=NONE. Existing models are grandfathered per MODELS.md.
• For affected Dynamo TRT-LLM benchmarks, verified that iteration/KV-cache metrics are collected and published under the pinned Dynamo implementation and effective engine settings. Resolve the installed Dynamo version separately from the container tag; use supported metrics controls without requiring KV events. Record the version and configuration evidence in the additional detail section. See verifier Check 17. N/A for unaffected serving paths or frameworks without these Dynamo settings.
• If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
• For official InferenceX (non-preview), verified that established models use upstream engine images built from master (or the equivalent default development branch), with new models transitioning within a reasonable time after release. See MODELS.md.
• If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
• Verified that every affected vLLM/SGLang production serving configuration in this PR, including Dynamo deployments, is documented in the official vLLM recipes and/or the SGLang cookbook, for both single-node and multi-node aggregated and disaggregated serving. The documentation covers production-useful flags, environment variables, and launch commands (including prefill, decode, and router/frontend commands for disaggregation); InferenceX-only benchmark/harness settings are excluded with reasoning in the additional detail section:
  • I linked the corresponding upstream PR in the vLLM recipe repo or SGLang cookbook source repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
• Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/<PR_NUMBER>.md — named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section.
• If this PR uses append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it.
• If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
• Reported measured throughput/E2EL Pareto counts and evidence per affected curve (≥5 points strongly recommended). Below 5 or unverifiable: tag a core maintainer for review; recorded admin bypass required before merge. N/A if no curves are affected. Details.

Additional detail section:

• insert any additional info here

• recipe vllm-project/vllm#64804

• #4666 (comment)

Signed: seungrokj

[github-actions]

⚠️ Verdict: WARN ⚠️

⚠️ Pareto coverage needs additional review. At least 5 points per affected throughput-versus-E2EL frontier are highly recommended. Below 5, or when coverage cannot be verified, merge only with an explicit, recorded admin bypass for the assessed commit; this advisory comment does not grant or enforce a bypass.

⚠️ Check 14 (Pareto coverage): WARN — Curve dsv41flash / agentic-coding / MI355X vLLM / fp4 / run 41098938196 (source fa9d486, image nightly-335dd77b0@sha256:3c634cc2) / P90 has 1 of 5 frontier points: TP8 c1, tput_per_gpu 492.53 (3940.25 total / 8 GPUs), p90_e2el 7.545 s, from artifact results_bmk; pareto_coverage confirms 1 frontier point. The TP4/TP2 key's points come from another run and image and cannot be pooled. No admin bypass is recorded for 9117fe6.

Passed and not applicable checks

✅ Check 0 (CODEOWNER): PASS — @seungrokj is a listed owner of inferencex-e2e/configs/amd-master.yaml and srt-slurm-recipes/**/mi355x-*/; perf-changelog.yaml falls to the catch-all.

✅ Check 1 (Sweep on in-PR commit): PASS — In-PR commit fa9d486, run 41098938196: the one-point agentic matrix ran as the canary job (full 3600 s via benchmark-tmpl, success), and agentic eval / succeeded. The single-node */ and eval / jobs are empty because the config is agentic-only. Later commits are not material: 9d17566 only rewrites the changelog text, and merges 537d3f5/9117fe6 bring CollectiveX, OperatorX, power_model, other models' recipes, a new qwen3.8next key, and classifier-enum/CI edits that do not reach this config's launch, benchmark, or eval path.

✅ Check 2 (Evals): PASS — The eval ran on the same image (digest 3c634cc2) with --tensor-parallel-size 8 --monokernel and real block rejection. GSM8K em_strict is 0.9719 ± 0.0045 (n=1319). The closest ingested MI355X vLLM DSpark cohort is 0.9712 ± 0.0046 (40161856449), and the best recent peer, 0.9742 (39844994441), is within 2 sigma. The run has no siblings.

✅ Check 3 (Upstream recipe): PASS — vllm-project/vllm#64804 is MERGED (2026-10-31, 3f9d1847) and published at recipes.vllm.ai. The mi355x_agentx_tp8 preset pins the same image and digest with TP8, --max-num-seqs 1, --max-num-batched-tokens 32768, --max-cudagraph-capture-size 8, resident Engram, and OMP_NUM_THREADS=1. It inherits the MI355X single_node_tp overrides (--monokernel, --block-size 128, MXFP4 sparse-logits indexer, sparse-MLA, INT4 quick reduce), the AMD overrides (--moe-backend aiter, AITER env, breakable CUDA graphs, gpu-mem 0.9), the AMD DSpark k=5 speculative config, and the Rust frontend.

✅ Check 4 (Reuse command): PASS — /reuse-sweep-run 41098938196 was posted on its own line by seungrokj (COLLABORATOR); that run is the Check 1 run.

✅ Check 5 (Checklist template): PASS — All 20 items in the current template, including the recipe sub-item, are present and checked.

✅ Check 6 (Upstream images): PASS — vllm/vllm-openai-rocm is the upstream vLLM org. 335dd77b0 is an ancestor of vLLM main (compare: behind_by 0). The entry is vLLM-only, so engine-first ordering does not apply.

✅ Check 7 (Deprecated models): PASS — The dsv41flash agentic-coding scenario is active in MODELS.md.

✅ Check 8 (No arch hacks): PASS — There are no --hf-overrides or FLOP-reducing settings, and --monokernel fuses the same computation. VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 is inherited, but dsv41flash is in the policy-baseline matrix (grandfathered), and the eval ran with it.

✅ Check 9 (Chat template): PASS — AgentX trace replay goes through the chat route via the unchanged shared client (infx/bench/agentic/replay.py); the PR does not change the client.

✅ Check 10 (No engine patches): PASS — The diff adds only a recipe override, a master entry, and a changelog entry. No patches, setup scripts, overlays, or wheel installs are reachable.

✅ Check 11 (Golden AL): PASS — The canary log shows rejection_sample_method: synthetic and synthetic_acceptance_length: 3.51; signoff_facts golden-al (dsv41flash DSpark, 5 tokens, probabilistic, thinking_on) expects 3.51.

➖ Check 12 (Append-only): N/A — The new changelog entry does not set append-only: true.

✅ Check 13 (Draft precision): PASS — At vLLM 335dd77b0, the DSpark draft (mtp.0-2 in the target checkpoint) is built with the draft config's checkpoint quant config (amd/dspark.py#L310). There is no draft quantization, dtype, or kv-cache override in speculative-config, and the draft KV is inherited from the target. SGLANG_NVFP4_CKPT_FP8_NEXTN_MOE is not set and does not apply to this vLLM path.

✅ Check 15 (PR description): PASS — The image, digest, TP8 c1 settings, single-job matrix, unchanged TP4/TP2 key image, and run 41098938196 on fa9d486 match the pinned code and CI evidence.

➖ Check 16 (NVIDIA overview): N/A — The only affected hardware is AMD MI355X.

➖ Check 17 (Metrics publication): N/A — This is direct vLLM serving, not Dynamo TRT-LLM.

✅ Check 18 (Recipe/image consistency): PASS — override_tp8_c1 has gpus: 8, matching master tp: 8 and conc [1]. Generation shows no errors, the pinned digest resolves, and no points are removed or re-tuned. The changelog matches the diff, and the pinned index's amd64 manifest 2b0c35e4 is the nightly-rocm100-335dd77b0 image.

Assessed commit: 9117fe6ce523873798d61b1350061ec71517d832.

[functionstackx]

/use 41394320841

[github-actions]

@functionstackx staged run 41394320841.

• DeepSeek-V4.1-Flash · agentic-traces · FP4 · mi355x_vllm · 2026-11-02

This run remains available across future /stage-results requests. Staging the same run ID again updates its staged data. Staging workflow

[functionstackx]

/reuse-sweep-run 41394320841

[github-actions]

Ingest report — ✅ Results ingested into the production database.

Ingest workflow · Merge Ingest run · Benchmark source run

Category	new	duplicate	skipped	unmapped	failed
benchmark	10	0	10	0	0
eval	2	2	2	0	0
trace	10	0	0	0	0
power	10	0	0	0	0

_{benchmarkFiles 11, rawBenchmarkRows 20, coveredByPerJobFiles 10, purgedPoints 0, failedRunRows 0, runStatsNew 0, runStatsDuplicate 0, evalSamplesNew 2638, gpuMetricSamplesNew 136212, powerPublicationPoints 10, changelogEntriesWritten 2}

Counts cover only this run. Rows: benchmark_results, eval_results, agentic rows linked to a trace, gpu_metrics series. This comment updates if the ingest is rerun.

中文

Ingest 报告 — ✅ 结果已写入 production 数据库。

上表仅统计本次运行写入的数据，各行单位依次为 benchmark_results 行、eval_results 行、已关联 trace 的 agentic 行和 gpu_metrics 序列。重新运行 ingest 时本评论会原地更新。