Skip to content

Flag response tests that never reach albedo's reflected body - #6

Merged
fzipi merged 1 commit into
mainfrom
feat/albedo-reflect-tests
Oct 28, 2026
Merged

fzipi merged 1 commit into
mainfrom
feat/albedo-reflect-tests

Conversation

@fzipi

@fzipi fzipi commented Oct 27, 2026

Copy link
Copy Markdown
Member

what

The go-ftw test instructions now flag response-phase tests (RESPONSE-95x) whose /reflect request is not an albedo reflection spec with a body (or encodedBody) field. They also tell the reviewer not to flag the request Content-Type, which albedo ignores.

why

albedo's /reflect parses the request body as JSON and returns only body. Against the pinned albedo:0.3.0, a plain string gets 400 Invalid JSON in request body and JSON without body returns an empty response, so the payload never reaches RESPONSE_BODY and a negative test passes without testing anything. CodeRabbit caught this on coreruleset/coreruleset#4833, but the learning it saved is scoped to that one file. Tests 1–6 in 953100.yaml have the same problem, so it is not a one-off.

refs

ai disclosure

  • tools used: Claude (Opus 5.5)
  • assisted with: wording of the new instruction paragraph and drafting this description
  • review performed: ran the pinned albedo image locally with a plain string, JSON without body and {"body": ...} to confirm the three responses; yamllint -d relaxed and check-jsonschema against the v2 schema pass

albedo's /reflect parses the request as a JSON reflection spec and returns only its body field, so a plain string or JSON without body leaves RESPONSE_BODY without the payload and a negative test passes vacuously. Seen on coreruleset/coreruleset#4833.
@fzipi
fzipi merged commit bc1de3e into main Oct 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant