Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,45 @@ Central monorepo for the OTE organizer kit. Read DESIGN.md before any task.
in `packages/validate/src/errors.ts` are keyed to specific rules; a rule that
moved leaves a message that lies. Say in the PR which spec version the repo
now implements.
- **`skills/` moves with the version, and a test says where.** A skill is prose
carrying version literals nothing else checks — `npx -p
@opentechevents/import-ics@<minor>` lines, a pinned embed URL, a
`specVersion` in an example an agent will copy — and each one is an
instruction a real organizer follows. They went stale exactly as you would
expect: `references/website.md` sat two minors behind `SKILL.md`, because the
rule asking for the pins to move lived in a different file from the pins, and
the evals measured "ground truth" with the release whose version bug this kit
exists to have fixed.

So do not audit the prose. Run it:

```sh
pnpm --filter @opentechevents/skill-evals test # skills/evals/test/version-pins.test.ts
```

It derives everything from the workspace's own `package.json` versions and
from `packages/validate`'s support policy, and fails with the file, the line
and the value to write. What it holds: npm pins name a package this repo
ships, on the line it ships; a pinned `embed/v<x.y.z>` URL matches
`apps/embed` **and** has a committed snapshot behind it; a `specVersion` in
anything `skills/ote-event/` tells an organizer to write is the latest
published version; and no `.md` names a bare spec minor in prose, because
"what v0.4 documents" is read as current by whoever reads it.

The one that is invisible without it: **eval fixtures pinned to a spec
version change meaning when the support window moves.**
`corunajug-feed.json` was stamped at an older-but-supported release to be the
"a notice, never a defect" case; the 0.5.0/0.6.0 day slid the window past it
and it became a *migration* case still asserting the old thing. Fixtures
declare a **role** (`latest` / `older-supported` / `none`) and the test
resolves it against the live policy, so a window move tells you which file to
re-stamp. Re-stamp the fixture — never move the bar.

Keep the escape hatch closed: there is no per-line ignore, on purpose. A
comment that needs to discuss an old pin spells it in words ("the 0.4 line"),
which reads better anyway. And when the harness needs a verdict, it imports
`validateDocument` from the workspace instead of pinning a published release
— an instrument must be the code under review, not a copy of it from npm.
- Tests: `pnpm test` at the root. Every new package ships fixtures and tests.
- Build before type-checking a fresh clone or worktree: packages depend on each
other through `dist/*.d.ts`, so `pnpm typecheck` (and any single-package
Expand Down
3 changes: 3 additions & 0 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 9 additions & 1 deletion skills/evals/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,15 @@ decide how everything here is built:
| **Activation** (`cases/activation.json`) | Does the skill fire on what organizers type, and stay quiet otherwise? | Cheap — the run stops as soon as the agent says which skills it loaded | Every PR that touches a `description` |
| **Behavior** (`cases/behavior.json`) | Once it fired, did the agent hold the invariants? | Expensive — full conversations | Every PR that touches a `SKILL.md` or a `references/` file |

Keeping them apart is the point. Activation is a **recall problem** solved by
A third thing lives here and is not an eval: `test/version-pins.test.ts`, a plain
unit test over the skills' **version literals** — the `npx` pins, the embed URL,
the `specVersion` in examples, and the spec version each fixture is stamped at.
No agent, no API key, runs in CI with everything else. It is here rather than in
`packages/` because what it guards is prose, and because the fixtures it checks
are these fixtures. See the root `CLAUDE.md` for what it holds and why each rule
is there.

Keeping the two evals apart is the point. Activation is a **recall problem** solved by
rewriting one sentence of frontmatter; behavior is a **prompt problem** solved by
rewriting the body. A single blended score tells you neither.

Expand Down
2 changes: 1 addition & 1 deletion skills/evals/cases/behavior.json
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@
},
"ranValidator"
],
"watch_for": "Everything real about this case comes from the fixtures: corunajug.org declares no <link rel=\"alternate\"> at all, so HTML discovery finds nothing — but the adopters registry lists their feed, and it exists and carries 10 events at specVersion 0.3.0. Right outcome: find the feed, create no second copy, and offer the missing discovery tag. Two failures to watch. First, deriving a new event document anyway. Second — and this one is live today — running `ote-validate` on that feed returns `0/1 valid` with an error about specVersion, because the CLI measures against the newest release; the agent must not relay that as 'your feed is broken'. An older supported version is a notice, never a defect."
"watch_for": "Everything real about this case comes from the fixtures: corunajug.org declares no <link rel=\"alternate\"> at all, so HTML discovery finds nothing \u2014 but the adopters registry lists their feed, and it exists and carries 10 events. Right outcome: find the feed, create no second copy, and offer the missing discovery tag. The feed is stamped at the OLDEST release still inside the support window, deliberately and enforced by test/version-pins.test.ts: a healthy feed on last season's spec, which ote-validate reports as valid with a notice saying a newer release exists. Three failures to watch. First, deriving a new event document anyway. Second, relaying that notice as a defect, or opening a migration the organizer never asked for \u2014 an older supported version is a notice, never a defect. Third, a specVersion ERROR appearing here at all: that means the fixture has fallen out of the support window and this case is quietly testing migration instead of what it claims. Re-stamp the fixture; do not move the bar."
},
{
"id": "ics-import",
Expand Down
2 changes: 1 addition & 1 deletion skills/evals/fixtures/corunajug-feed.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"specVersion": "0.3.0",
"specVersion": "0.4.0",
"title": "Eventos de CoruñaJUG",
"description": "Charlas y meetups de la comunidad Coruña Java User Group.",
"url": "https://www.corunajug.org",
Expand Down
1 change: 1 addition & 0 deletions skills/evals/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
"typecheck": "tsc -p tsconfig.json --noEmit"
},
"devDependencies": {
"@opentechevents/validate": "workspace:*",
"@types/node": "^22.0.0",
"typescript": "^6.0.3",
"vitest": "^5.1.0"
Expand Down
52 changes: 38 additions & 14 deletions skills/evals/src/assertions.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import { execFileSync } from "node:child_process";
import { existsSync, readFileSync } from "node:fs";
import { resolve } from "node:path";

import { validateDocument, type DocumentKind } from "@opentechevents/validate";

import {
bashCommands,
questionTurns,
Expand Down Expand Up @@ -40,24 +41,47 @@ export function ranImporter(run: Run): Check {
/**
* Ground truth, independent of anything the agent said: validate the file it
* actually left on disk.
*
* This calls the workspace's own validator rather than an `npx` pin, for the
* reason the pin itself demonstrated: it fetched the validate package on the
* 0.4 line, which measures a document against the newest release instead of
* the version it declares — the bug this kit fixed in 0.6 — so the instrument
* reported a correct 0.5 or 0.6 document as broken, and a green run meant
* nothing. An unpinned import cannot drift: it is the code under review.
*
* The agent-facing pins are a different thing and stay literal in the skill:
* they say which published release the instructions were tested with. The
* guard in `test/version-pins.test.ts` is what keeps those honest.
*/
export function fileIsValid(run: Run, relativePath: string): Check {
export async function fileIsValid(run: Run, relativePath: string): Promise<Check> {
const target = resolve(run.dir, relativePath);
const name = "the file on disk is valid";

let json: unknown;
try {
const output = execFileSync(
"npx",
["-y", "-p", "@opentechevents/validate@0.4", "ote-validate", target],
{ encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] },
);
return check("the file on disk is valid", true, output.trim());
json = JSON.parse(readFileSync(target, "utf8"));
} catch (e) {
const err = e as { stdout?: string; stderr?: string; status?: number };
return check(
"the file on disk is valid",
false,
`exit ${err.status ?? "?"}: ${(err.stdout ?? "") + (err.stderr ?? "")}`.trim(),
);
return check(name, false, `${relativePath}: ${(e as Error).message}`);
}

const kind: DocumentKind =
typeof json === "object" && json !== null && Array.isArray((json as Record<string, unknown>).events)
? "feed"
: "event";
const report = await validateDocument(json, { kind });

// `checkedVersion === null` is not a verdict on the document: nothing was
// measured. Reported as a failure here on purpose — a case asserting
// validity got no answer — but with the reason named, so nobody reads it as
// "the agent wrote a broken file".
if (report.checkedVersion === null) {
return check(name, false, `not judged: ${report.errors.map((e) => e.message).join(" ")}`);
}

const detail = report.valid
? `valid against OTE ${report.checkedVersion}`
: report.errors.map((e) => `${e.path}: ${e.message}`).join("; ");
return check(name, report.valid, detail);
}

/**
Expand Down
4 changes: 2 additions & 2 deletions skills/evals/test/assertions.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ const writes = (path: string): Turn => ({ role: "tool", tool: { name: "write", p
describe("ranValidator / ranImporter", () => {
it("pass on the npx forms the skill actually prescribes", () => {
const r = run([
bash('curl -sL "https://x.example" | npx -y -p @opentechevents/import-jsonld@0.4 ote-import-jsonld -'),
bash("npx -y -p @opentechevents/validate@0.4 ote-validate events/x.json"),
bash('curl -sL "https://x.example" | npx -y -p @opentechevents/import-jsonld@0.6 ote-import-jsonld -'),
bash("npx -y -p @opentechevents/validate@0.6 ote-validate events/x.json"),
]);
expect(ranValidator(r).passed).toBe(true);
expect(ranImporter(r).passed).toBe(true);
Expand Down
Loading
Loading