From e7cd0efe3e04a12a2480d23e74bf8e6c7e159b74 Mon Sep 17 00:00:00 2001 From: Raul Jimenez Ortega Date: Fri, 6 Nov 2026 12:50:35 +0100 Subject: [PATCH] test(skills): guard the version literals in skills/, and stop measuring with the buggy release MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A skill is prose, and prose carries version literals nothing checks: the `npx -p @opentechevents/@` lines an organizer actually runs, a pinned embed URL, a `specVersion` in an example an agent will copy. Each one was right the day it was written. They drifted exactly as you would expect, because the rule asking for the pins to move lived in SKILL.md and the pins lived in references/: - references/website.md pinned export-jsonld@0.4 and embed v0.7.0, two minors and one widget line behind what this repo ships. - references/writing.md's canonical "minimum a standalone event needs" wrote specVersion 0.4.0, so an agent following it mints a document two minors behind on the day it is written. - evals/src/assertions.ts measured `fileIsValid` — what the README calls ground truth — by fetching the validate package on the 0.4 line, which judges a document against the newest release instead of the version it declares. That is the bug this kit fixed in 0.6, live in the instrument: a correct 0.5 or 0.6 document came out broken and a green run meant nothing. So the rule is no longer discipline, it is a test. version-pins.test.ts derives everything at run time — from the workspace's own package.json versions and from packages/validate's support policy — and fails with the file, the line and the value to write. It holds: npm pins name a package this repo ships, on the line it ships; a pinned embed/v URL matches apps/embed AND has a committed snapshot behind it; a specVersion in anything skills/ote-event/ tells an organizer to write is the latest published version; and no .md names a bare spec minor in prose, because "what v0.4 documents" is read as current by whoever reads it. The failure nobody would have seen without it: an eval fixture pinned to a spec version changes meaning when the support window moves. corunajug-feed.json was stamped 0.3.0 to be the "older but supported — a notice, never a defect" case, and the day 0.5.0 and 0.6.0 shipped together the window slid past it. The fixture still parsed and the case still ran, having quietly become a migration case that asserted the opposite. Fixtures now declare a role (latest / older-supported / none) resolved against the live policy, so a window move names the file to re-stamp. Re-stamped to 0.4.0, and its watch_for rewritten: it described the pre-0.6 CLI behaviour as "live today". fileIsValid now imports validateDocument from the workspace instead of pinning a published release. An instrument must be the code under review, not a copy of it from npm — and an unpinned import cannot drift. The agent-facing pins stay literal on purpose: they say which release the instructions were tested with. No per-line ignore mechanism, deliberately: a marker that silences this scan would be reached for on the day the scan is right. A comment that needs to discuss an old pin spells it in words ("the 0.4 line"), which reads better anyway — the guard caught its own comments on first run and that is how it was fixed. Verified by simulation rather than by reading: bumping validate to 0.7.0 and embed to 0.9.0 and re-stamping the fixture makes it report each stale pin with its file and line, and inserting a bare `v0.5` into prose trips the prose rule. What this does not fix: ote-validate still reports a healthy kit fork's events/*.json as `0/3 valid` with exit 3 (#117), and the skill's Gate 2 still sends the agent at them first (#119). Co-Authored-By: Claude Opus 5 --- CLAUDE.md | 39 ++++ pnpm-lock.yaml | 3 + skills/evals/README.md | 10 +- skills/evals/cases/behavior.json | 2 +- skills/evals/fixtures/corunajug-feed.json | 2 +- skills/evals/package.json | 1 + skills/evals/src/assertions.ts | 52 +++-- skills/evals/test/assertions.test.ts | 4 +- skills/evals/test/version-pins.test.ts | 230 ++++++++++++++++++++++ skills/ote-event/SKILL.md | 5 +- skills/ote-event/references/website.md | 14 +- skills/ote-event/references/writing.md | 8 +- 12 files changed, 342 insertions(+), 28 deletions(-) create mode 100644 skills/evals/test/version-pins.test.ts diff --git a/CLAUDE.md b/CLAUDE.md index ef164a7..4ef48c7 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -80,6 +80,45 @@ Central monorepo for the OTE organizer kit. Read DESIGN.md before any task. in `packages/validate/src/errors.ts` are keyed to specific rules; a rule that moved leaves a message that lies. Say in the PR which spec version the repo now implements. +- **`skills/` moves with the version, and a test says where.** A skill is prose + carrying version literals nothing else checks — `npx -p + @opentechevents/import-ics@` lines, a pinned embed URL, a + `specVersion` in an example an agent will copy — and each one is an + instruction a real organizer follows. They went stale exactly as you would + expect: `references/website.md` sat two minors behind `SKILL.md`, because the + rule asking for the pins to move lived in a different file from the pins, and + the evals measured "ground truth" with the release whose version bug this kit + exists to have fixed. + + So do not audit the prose. Run it: + + ```sh + pnpm --filter @opentechevents/skill-evals test # skills/evals/test/version-pins.test.ts + ``` + + It derives everything from the workspace's own `package.json` versions and + from `packages/validate`'s support policy, and fails with the file, the line + and the value to write. What it holds: npm pins name a package this repo + ships, on the line it ships; a pinned `embed/v` URL matches + `apps/embed` **and** has a committed snapshot behind it; a `specVersion` in + anything `skills/ote-event/` tells an organizer to write is the latest + published version; and no `.md` names a bare spec minor in prose, because + "what v0.4 documents" is read as current by whoever reads it. + + The one that is invisible without it: **eval fixtures pinned to a spec + version change meaning when the support window moves.** + `corunajug-feed.json` was stamped at an older-but-supported release to be the + "a notice, never a defect" case; the 0.5.0/0.6.0 day slid the window past it + and it became a *migration* case still asserting the old thing. Fixtures + declare a **role** (`latest` / `older-supported` / `none`) and the test + resolves it against the live policy, so a window move tells you which file to + re-stamp. Re-stamp the fixture — never move the bar. + + Keep the escape hatch closed: there is no per-line ignore, on purpose. A + comment that needs to discuss an old pin spells it in words ("the 0.4 line"), + which reads better anyway. And when the harness needs a verdict, it imports + `validateDocument` from the workspace instead of pinning a published release + — an instrument must be the code under review, not a copy of it from npm. - Tests: `pnpm test` at the root. Every new package ships fixtures and tests. - Build before type-checking a fresh clone or worktree: packages depend on each other through `dist/*.d.ts`, so `pnpm typecheck` (and any single-package diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 2c7ee27..2259f10 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -403,6 +403,9 @@ importers: skills/evals: devDependencies: + '@opentechevents/validate': + specifier: workspace:* + version: link:../../packages/validate '@types/node': specifier: ^22.0.0 version: 22.20.5 diff --git a/skills/evals/README.md b/skills/evals/README.md index 145b101..6030b44 100644 --- a/skills/evals/README.md +++ b/skills/evals/README.md @@ -17,7 +17,15 @@ decide how everything here is built: | **Activation** (`cases/activation.json`) | Does the skill fire on what organizers type, and stay quiet otherwise? | Cheap — the run stops as soon as the agent says which skills it loaded | Every PR that touches a `description` | | **Behavior** (`cases/behavior.json`) | Once it fired, did the agent hold the invariants? | Expensive — full conversations | Every PR that touches a `SKILL.md` or a `references/` file | -Keeping them apart is the point. Activation is a **recall problem** solved by +A third thing lives here and is not an eval: `test/version-pins.test.ts`, a plain +unit test over the skills' **version literals** — the `npx` pins, the embed URL, +the `specVersion` in examples, and the spec version each fixture is stamped at. +No agent, no API key, runs in CI with everything else. It is here rather than in +`packages/` because what it guards is prose, and because the fixtures it checks +are these fixtures. See the root `CLAUDE.md` for what it holds and why each rule +is there. + +Keeping the two evals apart is the point. Activation is a **recall problem** solved by rewriting one sentence of frontmatter; behavior is a **prompt problem** solved by rewriting the body. A single blended score tells you neither. diff --git a/skills/evals/cases/behavior.json b/skills/evals/cases/behavior.json index 9c9419f..cb53b85 100644 --- a/skills/evals/cases/behavior.json +++ b/skills/evals/cases/behavior.json @@ -108,7 +108,7 @@ }, "ranValidator" ], - "watch_for": "Everything real about this case comes from the fixtures: corunajug.org declares no at all, so HTML discovery finds nothing — but the adopters registry lists their feed, and it exists and carries 10 events at specVersion 0.3.0. Right outcome: find the feed, create no second copy, and offer the missing discovery tag. Two failures to watch. First, deriving a new event document anyway. Second — and this one is live today — running `ote-validate` on that feed returns `0/1 valid` with an error about specVersion, because the CLI measures against the newest release; the agent must not relay that as 'your feed is broken'. An older supported version is a notice, never a defect." + "watch_for": "Everything real about this case comes from the fixtures: corunajug.org declares no at all, so HTML discovery finds nothing \u2014 but the adopters registry lists their feed, and it exists and carries 10 events. Right outcome: find the feed, create no second copy, and offer the missing discovery tag. The feed is stamped at the OLDEST release still inside the support window, deliberately and enforced by test/version-pins.test.ts: a healthy feed on last season's spec, which ote-validate reports as valid with a notice saying a newer release exists. Three failures to watch. First, deriving a new event document anyway. Second, relaying that notice as a defect, or opening a migration the organizer never asked for \u2014 an older supported version is a notice, never a defect. Third, a specVersion ERROR appearing here at all: that means the fixture has fallen out of the support window and this case is quietly testing migration instead of what it claims. Re-stamp the fixture; do not move the bar." }, { "id": "ics-import", diff --git a/skills/evals/fixtures/corunajug-feed.json b/skills/evals/fixtures/corunajug-feed.json index 7b49c2d..08191a8 100644 --- a/skills/evals/fixtures/corunajug-feed.json +++ b/skills/evals/fixtures/corunajug-feed.json @@ -1,5 +1,5 @@ { - "specVersion": "0.3.0", + "specVersion": "0.4.0", "title": "Eventos de CoruñaJUG", "description": "Charlas y meetups de la comunidad Coruña Java User Group.", "url": "https://www.corunajug.org", diff --git a/skills/evals/package.json b/skills/evals/package.json index 49c829f..11c8087 100644 --- a/skills/evals/package.json +++ b/skills/evals/package.json @@ -10,6 +10,7 @@ "typecheck": "tsc -p tsconfig.json --noEmit" }, "devDependencies": { + "@opentechevents/validate": "workspace:*", "@types/node": "^22.0.0", "typescript": "^6.0.3", "vitest": "^5.1.0" diff --git a/skills/evals/src/assertions.ts b/skills/evals/src/assertions.ts index e074b12..87511e2 100644 --- a/skills/evals/src/assertions.ts +++ b/skills/evals/src/assertions.ts @@ -1,7 +1,8 @@ -import { execFileSync } from "node:child_process"; import { existsSync, readFileSync } from "node:fs"; import { resolve } from "node:path"; +import { validateDocument, type DocumentKind } from "@opentechevents/validate"; + import { bashCommands, questionTurns, @@ -40,24 +41,47 @@ export function ranImporter(run: Run): Check { /** * Ground truth, independent of anything the agent said: validate the file it * actually left on disk. + * + * This calls the workspace's own validator rather than an `npx` pin, for the + * reason the pin itself demonstrated: it fetched the validate package on the + * 0.4 line, which measures a document against the newest release instead of + * the version it declares — the bug this kit fixed in 0.6 — so the instrument + * reported a correct 0.5 or 0.6 document as broken, and a green run meant + * nothing. An unpinned import cannot drift: it is the code under review. + * + * The agent-facing pins are a different thing and stay literal in the skill: + * they say which published release the instructions were tested with. The + * guard in `test/version-pins.test.ts` is what keeps those honest. */ -export function fileIsValid(run: Run, relativePath: string): Check { +export async function fileIsValid(run: Run, relativePath: string): Promise { const target = resolve(run.dir, relativePath); + const name = "the file on disk is valid"; + + let json: unknown; try { - const output = execFileSync( - "npx", - ["-y", "-p", "@opentechevents/validate@0.4", "ote-validate", target], - { encoding: "utf8", stdio: ["ignore", "pipe", "pipe"] }, - ); - return check("the file on disk is valid", true, output.trim()); + json = JSON.parse(readFileSync(target, "utf8")); } catch (e) { - const err = e as { stdout?: string; stderr?: string; status?: number }; - return check( - "the file on disk is valid", - false, - `exit ${err.status ?? "?"}: ${(err.stdout ?? "") + (err.stderr ?? "")}`.trim(), - ); + return check(name, false, `${relativePath}: ${(e as Error).message}`); } + + const kind: DocumentKind = + typeof json === "object" && json !== null && Array.isArray((json as Record).events) + ? "feed" + : "event"; + const report = await validateDocument(json, { kind }); + + // `checkedVersion === null` is not a verdict on the document: nothing was + // measured. Reported as a failure here on purpose — a case asserting + // validity got no answer — but with the reason named, so nobody reads it as + // "the agent wrote a broken file". + if (report.checkedVersion === null) { + return check(name, false, `not judged: ${report.errors.map((e) => e.message).join(" ")}`); + } + + const detail = report.valid + ? `valid against OTE ${report.checkedVersion}` + : report.errors.map((e) => `${e.path}: ${e.message}`).join("; "); + return check(name, report.valid, detail); } /** diff --git a/skills/evals/test/assertions.test.ts b/skills/evals/test/assertions.test.ts index 60a6db3..b51b855 100644 --- a/skills/evals/test/assertions.test.ts +++ b/skills/evals/test/assertions.test.ts @@ -34,8 +34,8 @@ const writes = (path: string): Turn => ({ role: "tool", tool: { name: "write", p describe("ranValidator / ranImporter", () => { it("pass on the npx forms the skill actually prescribes", () => { const r = run([ - bash('curl -sL "https://x.example" | npx -y -p @opentechevents/import-jsonld@0.4 ote-import-jsonld -'), - bash("npx -y -p @opentechevents/validate@0.4 ote-validate events/x.json"), + bash('curl -sL "https://x.example" | npx -y -p @opentechevents/import-jsonld@0.6 ote-import-jsonld -'), + bash("npx -y -p @opentechevents/validate@0.6 ote-validate events/x.json"), ]); expect(ranValidator(r).passed).toBe(true); expect(ranImporter(r).passed).toBe(true); diff --git a/skills/evals/test/version-pins.test.ts b/skills/evals/test/version-pins.test.ts new file mode 100644 index 0000000..edb38a0 --- /dev/null +++ b/skills/evals/test/version-pins.test.ts @@ -0,0 +1,230 @@ +/** + * The guard that keeps `skills/` from going stale when the spec moves. + * + * A skill is prose, and prose carries version literals that nothing checks: + * an `npx -p @opentechevents/import-jsonld@0.6` line, a pinned embed URL, a + * `specVersion` in an example an agent will copy. Every one of those was + * correct the day it was written and silently wrong a release later — + * `references/website.md` sat two minors behind `SKILL.md` because the rule + * that says "move the pin in the same change" lives in one file and the pins + * live in another, and `fileIsValid` measured ground truth with the release + * whose version bug this kit exists to have fixed. + * + * So the rule is not discipline, it is this test. Everything below is derived + * at run time — from the workspace's own package versions and from + * `@opentechevents/validate`'s support policy — so a spec release or a package + * bump fails CI with the file, the line and the value to write, instead of + * being noticed by whoever reads the skill next. + * + * Add a pin to a skill and you add nothing here: the scan finds it. + */ + +import { readFileSync, readdirSync, existsSync, statSync } from "node:fs"; +import { join, relative, resolve } from "node:path"; + +import { LATEST_VERSION, SUPPORTED_VERSIONS } from "@opentechevents/validate"; +import { describe, expect, it } from "vitest"; + +const repoRoot = resolve(import.meta.dirname, "../../.."); +const skillsDir = join(repoRoot, "skills"); + +/** + * Frozen captures of real third-party pages and the eval fixtures' own + * documents: both carry version strings that are *data*, not pins this repo + * owns. `biznagafest.html` really does load a v1 embed bundle — rewriting that + * would destroy the fixture. Fixture documents are checked separately, by role. + * + * There is no per-line ignore mechanism, deliberately: a marker that silences + * this scan would be reached for on the day the scan is right. A comment that + * needs to discuss an old pin spells it out in words instead ("the 0.4 line"), + * which is also what a reader needs. + */ +const FIXTURES = join(skillsDir, "evals", "fixtures"); + +/** Files whose version literals this repo is responsible for. */ +function proseFiles(dir: string): string[] { + const found: string[] = []; + for (const entry of readdirSync(dir, { withFileTypes: true })) { + if (entry.name.startsWith(".") || entry.name === "node_modules") continue; + const full = join(dir, entry.name); + if (full === FIXTURES) continue; + if (entry.isDirectory()) found.push(...proseFiles(full)); + else if (/\.(md|ts|json)$/.test(entry.name)) found.push(full); + } + return found.sort(); +} + +/** `name` → version, for every workspace package that a skill can pin. */ +function workspaceVersions(): Map { + const versions = new Map(); + for (const group of ["packages", "apps"]) { + const base = join(repoRoot, group); + if (!existsSync(base)) continue; + for (const entry of readdirSync(base)) { + const manifest = join(base, entry, "package.json"); + if (!existsSync(manifest)) continue; + const { name, version } = JSON.parse(readFileSync(manifest, "utf8")); + if (name && version) versions.set(name, version); + } + } + return versions; +} + +interface Hit { + file: string; + line: number; + text: string; +} + +/** Every match of `pattern` across the prose, with where to go and fix it. */ +function scan(pattern: RegExp): Hit[] { + const hits: Hit[] = []; + for (const file of proseFiles(skillsDir)) { + const lines = readFileSync(file, "utf8").split("\n"); + lines.forEach((text, index) => { + for (const match of text.matchAll(pattern)) { + hits.push({ file: relative(repoRoot, file), line: index + 1, text: match[0] }); + } + }); + } + return hits; +} + +const at = (hit: Hit): string => `${hit.file}:${hit.line}`; + +describe("npm pins in the skills", () => { + const versions = workspaceVersions(); + + it("name a version of every package that exists", () => { + const unknown = scan(/@opentechevents\/[a-z-]+(?=@\d)/g).filter( + (hit) => !versions.has(hit.text), + ); + expect( + unknown.map((hit) => `${at(hit)} pins ${hit.text}, which is not a workspace package`), + ).toEqual([]); + }); + + /** + * A skill may pin a minor (`@0.6`, which npm resolves to the newest patch) + * or an exact version — both are honest. What it may not do is name a line + * this repo no longer ships: an organizer running `@0.4` gets tools that + * implement a spec the kit has moved past, and the skill's own instructions + * stop matching their output. + */ + it("match the version the workspace ships", () => { + const stale = scan(/@opentechevents\/[a-z-]+@[\d.]+/g).flatMap((hit) => { + const [name, pinned] = [hit.text.slice(0, hit.text.lastIndexOf("@")), hit.text.slice(hit.text.lastIndexOf("@") + 1)]; + const shipped = versions.get(name); + if (!shipped) return []; + const ok = shipped === pinned || shipped.startsWith(`${pinned}.`); + return ok ? [] : [`${at(hit)} pins ${name}@${pinned}, workspace ships ${shipped}`]; + }); + expect(stale).toEqual([]); + }); +}); + +describe("the embed widget URL in the skills", () => { + it("pins the shipped version, and a snapshot that is committed", () => { + const shipped = workspaceVersions().get("@opentechevents/embed"); + expect(shipped, "apps/embed must declare a version").toBeTruthy(); + + const problems = scan(/embed\/v[\d.]+/g).flatMap((hit) => { + const pinned = hit.text.replace("embed/v", ""); + if (pinned !== shipped) { + return [`${at(hit)} pins embed v${pinned}, workspace ships v${shipped}`]; + } + // The skill tells organizers to pin a version in a live + ``` @@ -100,7 +100,7 @@ To offer a **subscribe** button instead — Google Calendar / `webcal:` / ICS, R and the OTE feed: ```html - + diff --git a/skills/ote-event/references/writing.md b/skills/ote-event/references/writing.md index 453fa51..a710387 100644 --- a/skills/ote-event/references/writing.md +++ b/skills/ote-event/references/writing.md @@ -7,7 +7,7 @@ cannot catch and that silently break the people consuming the feed. ```json { - "specVersion": "0.4.0", + "specVersion": "0.6.0", "id": "https://pyalmeria.example/eventos/2026-06-async", "name": "PyAlmería — Introducción a async/await", "startDate": "2026-06-11T18:30", @@ -16,6 +16,12 @@ cannot catch and that silently break the people consuming the feed. } ``` +**`specVersion` comes from the tool, not from this example.** Run +`ote-validate --versions` and write the `latest` it names — the number above is what +this release of the skill ships with, and a document minted two minors behind falls +out of the support window on a release nobody here will be around for. If the two +disagree, the tool wins and the skill is stale: say so. + Inside an organizer-kit fork you write **less** than that: an event in `events/*.json` inherits `specVersion` from the assembled feed, and may inherit `license`, `organizers` and `textLanguage` from `ote.config.json`. Do not repeat an inherited