Skip to content

Phase 4: upload fixture eval results; show the never-approves line once - #15

Closed
irvingpop wants to merge 2 commits into
mainfrom
irvingpop/phase4-a
Closed

irvingpop wants to merge 2 commits into
mainfrom
irvingpop/phase4-a

Conversation

@irvingpop

Copy link
Copy Markdown
Member

Two phase 4 in-repo items.

  • eval.yml uploads the fixture eval results (results.jsonl, per-case summary.md) as an artifact, always, 14 days. The upload action uses the same SHA pin as claude-review.yml. The eval egress allowlist gains *.blob.core.windows.net:443 for the upload; the design doc records it.
  • The pass comment shows "This bot never approves a PR" once (provenance line), not twice. This changes reviewer output, so it takes effect only at the next release.

Unit tests: 2632 passed; release tests: 2 passed. Eval not run. Reviewed locally (one fix round), re-review clean.

🤖 Generated with Claude Code

irvingpop and others added 2 commits October 28, 2026 08:35
…ine once

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@irvingpop
irvingpop requested a review from a team as a code owner October 28, 2026 15:40
@github-actions

Copy link
Copy Markdown

🔴 FAIL: OC Claude review did not complete

Caution

The Review, Security and Supply chain reviews did not run, so this PR could not be judged. No waiver can clear this.
Next step: a maintainer re-runs the newest run of the required workflow once the cause is fixed. The author does not need to change anything.

Severity Effect Where Finding Found by
⚠️ Not run Blocks, no waiver Review pass authentication failed. The pass returned no result. Workflow (code, no model)
⚠️ Not run Blocks, no waiver Security pass authentication failed. The pass returned no result. Workflow (code, no model)
⚠️ Not run Blocks, no waiver Supply chain pass authentication failed. The pass returned no result. Workflow (code, no model)
⚠️ Not run Blocks, no waiver Gate A required job did not succeed: dependency-review: failure Workflow (code, no model)
⚠️ Not run Blocks, no waiver Gate Dependency review did not succeed: job result failure Workflow (code, no model)
⚠️ Not run Blocks, no waiver Gate Dependency review did not succeed: step outcome failure Workflow (code, no model)
File checks Review Security Supply chain Dependency review Coverage
✅ ok, 2 signals ❌ failed ❌ failed ❌ failed ❌ failed ✅ complete

Run 40718271983, attempt 1 for head eba8a00 into main. The required workflow in the merge box is the result that counts. This bot never approves a PR.

📋 Full review record: reasons, checks, signals, finding details, model summaries, token usage

Reviewed snapshot

Head SHA Merge base SHA Base branch Run
eba8a00c02a0fa3cb1d791df9739394349d67e94 03e1b8d09c322bf215d1699da0e793f0064f2974 main 40718271983, attempt 1

Reasons the gate failed

Reason Waivable? Detail
A required job did not succeed 🚫 no dependency-review: failure
Dependency review did not succeed 🚫 no job result failure
Dependency review did not succeed 🚫 no step outcome failure
A review pass failed 🚫 no pass review failed: authentication failed
A review pass failed 🚫 no pass security failed: authentication failed
A review pass failed 🚫 no pass supply-chain failed: authentication failed

No waiver line is shown. At least one reason cannot be waived, so a waiver would not change the result.

Checks (code, no model)

Check Result Detail
Hard blocks ✅ none
Coverage ✅ complete files in comparison 6, files collected 6, trees complete
Dependency changes ✅ none
Dependency review job ❌ failed

Signals (context only, never block)

  • ⚠️ Sensitive path changed: change to workflow, in .github/workflows/eval.yml
  • ℹ️ Very long line: lines longer than 500 characters, in docs/claude-review-design.md, lines 197 200 574

Dependency changes

None.

Finding details

Model output from untrusted input. Check it before you act on it. Verified means only that the quoted text is at that place. It does not mean the claim is true.

Blocking is decided by code from category and severity, never by the model. A finding in a blocking category blocks only at high or critical severity.

No findings from the passes that ran. The Review, Security, Supply chain pass did not run, so this is not a clean result.

Model summaries

Model output from untrusted input. Check it before you act on it. Verified means only that the quoted text is at that place. It does not mean the claim is true.

Pass Status Model Summary
Review ❌ failed No summary. Reason: authentication failed
Security ❌ failed No summary. Reason: authentication failed
Supply chain ❌ failed No summary. Reason: authentication failed

Token usage

Pass Input Output Cache write Cache read Attempts
Review not recorded not recorded not recorded not recorded not recorded
Security not recorded not recorded not recorded not recorded not recorded
Supply chain not recorded not recorded not recorded not recorded not recorded
Recorded total not recorded not recorded not recorded not recorded not recorded

@irvingpop

Copy link
Copy Markdown
Member Author

Combined into #20.

@irvingpop irvingpop closed this Oct 29, 2026
irvingpop added a commit that referenced this pull request Oct 29, 2026
* Upload fixture eval results as an artifact; show the never-approves line once

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Fix review round 1: eval allowlist doc, upload warn, tests, comment

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Plan bounded continuation comments for summary-only findings

Add docs/continuation-comments-plan.md (planned, not built): current
fail-closed behavior with file:line citations, limits (6 parts of 64,000
characters, severity first, fail closed past the bound), idempotent slot
reuse, security, gate interaction, tests, rollout and open questions.
Point build-plan phase 4 at it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix continuation plan after review B; record decisions Q1-Q7

Per-run part ownership: a run POSTs its own parts and retires another
run's parts only when that run attempt is done and no bot sticky links
them, so concurrent runs cannot remove a linked set. Add final-check
verification, a Publication result kept through the final update,
defined read/write outcomes, cleanup caps, a time budget, a total
per-finding order, the over-limit pass case, the first-run backlink
and interleaving tests. Move the open questions to Decisions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record pilot start (four repos) and the immutable OIDC subject prerequisite (P4-1 to P4-6)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Fix pilot dates, immutable-subject wording and stale back-end-only text; add open-PR timing lesson (P4 fix round 1)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Parse pnpm 12 two-document pnpm-lock.yaml (K-16)

pnpm 12 writes the env lockfile (pnpm itself) as a first YAML document
before the main lockfile. The single-document load raised ComposerError,
so every front-end PR that changed the lockfile got the false hard block
"incomplete dependency data".

Read every document with the same unique-key safe loader. Accept one
document, or exactly two with an env document of the known shape. Each
document needs a supported lockfileVersion, and every env dependency
needs a packages entry. Packages from both documents get every check.
Anything else stays a parse failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Close K-16 review gaps in the pnpm two-document parser

Review of 01d3cb3 found that an env package entry could take another
package's identity through its own name and version, so pnpm could pass
the env check and then skip every check. The env check now refuses an
entry whose name or version differs from its key, and checks each env
dependency against the packages the walker emits.

The main document of a two-document file had no shape check. It may now
hold only the top-level keys of pnpm 12's main lockfile. The
one-document form keeps its old check.

The shared YAML loader expands merge keys before the duplicate-key
check, so a merged key that repeats another key is a parse failure.

Tests cover both scenarios, cross-document aliases, unsafe tags in each
document, merge-key collisions, end markers, a BOM, a document marker in
a block scalar, and a malformed two-document base in collect (K-10).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Anchor long patches at unique lines so lockfile churn fits the size gate (K-17)

PR 2027 in OperationCode/front-end (a 16k-line pnpm-lock.yaml changed near
both ends) got "too large for automatic review": its middle was past
DIFF_BUDGET, so the patch was one hunk of every base and head line and the
bundle estimated 451,544 tokens against the 150,000 limit.

In collect only, a middle past the size budget is now split at lines that
occur once on each side (patience anchors), and each gap gets the bounded
diff with the same file and run work budgets. The patch stays a correct
edit script, reads no path, and a full rewrite still has no anchors, so the
size gate stays fail-closed. PR 2027 now estimates 93,786 tokens (115,961
with the K-16 parser and full registry data). The publisher is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
@irvingpop
irvingpop deleted the irvingpop/phase4-a branch October 29, 2026 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant