Skip to content

ci(benchmark): refresh README and landing benchmarks on every main merge - #985

Draft
owjs3901 wants to merge 11 commits into
mainfrom
ci/benchmark-refresh
Draft

owjs3901 wants to merge 11 commits into
mainfrom
ci/benchmark-refresh

Conversation

@owjs3901

@owjs3901 owjs3901 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The benchmark numbers in the READMEs and on the landing homepage now refresh themselves after every merge to main. Every run is also kept in a per-commit history, so you can see which change made a benchmark worse.

What happens on a push to main

A new benchmark-refresh job in publish.yml runs after benchmark. It takes the benchmark-run artifact and does the following:

  1. Rewrites benchmark-results.json: run id, full SHA and short commit, date, runner, Next.js and Devup UI versions, and each row's library version, samples, median, range, sizes, CSS size and client size.
  2. Appends the run to benchmark/history.json. There is one entry per commit, in commit order, capped at 500 entries (about 2 KB each, so about 1 MB). Each entry holds the gate verdict (pass/fail), the rows that regressed, and [median, min, max, bytes, cssBytes, clientBytes] for all 22 rows.
  3. Regenerates the README sections: root, README_ko.md and the 7 package READMEs (added in docs: refresh package READMEs and agent metadata #911).
  4. Commits the changes as chore(benchmark): record run <id> of <sha> [skip ci] and pushes them to main.

Commit-back mechanism: a direct push to main, with a rolling PR as fallback

  • gh api …/branches/main/protection returns 404 (main has no branch protection). The only ruleset, main (id 9882973), has conditions.ref_name.include: [], so it currently matches no branch. Default workflow permissions are write. So GITHUB_TOKEN can push to main today.
  • A push made with GITHUB_TOKEN starts no workflow. The commit also carries [skip ci]. So the refresh cannot recurse, and it cannot trigger publish, release or changepacks.
  • If main moves while the job runs, it fetches, resets to the new HEAD, rebuilds the commit and retries (3 attempts).
  • If pushing keeps failing (for example, once the ruleset targets main without a bypass), it force-pushes the commit to chore/benchmark-refresh and keeps a single PR open for it. A PR opened with GITHUB_TOKEN gets no CI, and it only carries the latest run, so this path is a safety net, not the intended mode.

Safety

  • needs: [benchmark, release] with if: !cancelled() && push to main. The job starts only after release (changepacks) has finished, so the two never push at the same time. It runs no build, publish or changepacks step, and never touches changepacks/main (Update Versions #769).
  • It has its own concurrency: benchmark-refresh-main group (cancel-in-progress: false) on top of the workflow's own per-ref group.
  • Stale runs: benchmark-refresh.js uses git merge-base --is-ancestor.
    • A measured commit that is not on main is ignored.
    • A commit older than the published run.sha is added to the history only and never overwrites newer results.
    • A re-run of the same commit replaces that commit's history entry.
  • A run whose gate failed is still recorded, so the regression shows up in the history. A run with no artifact is skipped.

Gate vs. display. benchmark-gate.js, benchmark-baseline.json and benchmark-check.js keep their current behavior. The history is display and forensics only. Each entry's verdict is computed against the baseline checked in at that commit (git show <sha>:benchmark-baseline.json).

How to trace a regression to a commit

  1. Run bun benchmark-refresh.js --trend devup-ui-single-turbo. It prints one line per main commit: short SHA, commit time, gate verdict, median, ratio to the row's control in the same run, bytes, CSS bytes and client bytes.
  2. The first line where the ratio or a size jumps (or gate turns fail, with the row listed in regressions) names the merge. Compare it with the line before. git log <previous>..<sha> lists the commits in between, which is usually a single squash merge.
  3. The entry's run id links to the full log and the benchmark-run artifact. Workflow runs that a newer push cancelled while they were pending leave a gap. In that case the range covers more than one merge.

Size metrics (per the size investigation)

  • New column: Client Size, meaning the JavaScript and CSS shipped to the browser (.next/static/**/*.{js,css}, or vinext dist/client/**/*.{js,css}), uncompressed.
  • Every size is now measured on every sample, and sizes.{bytes,cssBytes,clientBytes} keep the samples with min and max.
  • Build Size is kept for continuity and labeled Build Size (incl. caches). The README text explains that it is mostly bundler cache.
  • The fixtures are unchanged.

Seeded numbers. benchmark-results.json and the first history entry come from the benchmark-run artifact of main run 43770976836 (commit c750d33c, after NAPI), at full precision. These match the median lines of job 132995530168. Library versions come from bun.lock at that commit. That run predates Client Size, so the column shows n/a until the first refresh.

Landing. apps/landing/src/app/Bench.tsx reads its numbers through benchmark-data.ts (see Screenshots), which imports benchmark-results.json. Bench is a server component, so no JSON ships to the client. The caption now links the run and the commit. Each crown shows only while Devup UI is the fastest or smallest card. Pages are deployed from the pushed commit, while the refresh lands right after it as a [skip ci] commit. So the homepage shows the newest numbers from the next deploy on, one merge behind the READMEs. Its run and commit links always match the numbers shown.

Screenshots: the e2e builds render frozen fixture data. A benchmark refresh on main must never change a screenshot, so the e2e builds don't read the live numbers at all:

  • e2e/fixtures/benchmark-results.json is a frozen copy of the current results (run 43770976836, commit c750d33c, Devup UI 1.0.44).
  • apps/landing/src/app/benchmark-data.ts picks the fixture when the build sets LANDING_BENCHMARK_DATA=fixture, and the live benchmark-results.json otherwise. Bench is prerendered at build time, so the choice is made by the build and ships no client code. The same module works under vinext and Next/Turbopack, with no bundler alias. The section carries data-benchmark-source="fixture|live".
  • CI sets the variable on every landing build that e2e serves: job-wide in landing-next-e2e, and on both landing builds in publish. On a push to main, publish rebuilds the singleCss site without it before uploading the Pages artifact and checks that the deployed index.html says live. The local Playwright webServer sets it too.
  • e2e/landing-benchmark-fixture.spec.ts fails if a build under test reads the live file. It checks the source marker, and that the section shows the fixture's run id, time and size.
  • The earlier masking (e2e/screenshot.css, stylePath, data-benchmark) is removed.
  • The Linux baselines that show the bench numbers were regenerated once from the actual images of landing-next-e2e run 44029448584: section-bench, dark-section-bench and components-benchmark-section-mobile. The images were identical on all three retries, and that run passed the fixture spec. The other baselines that include the section (full-page-*, dark-full-page-*, components-benchmark-section-desktop, dark-components-benchmark-section-desktop) stay within the 1% maxDiffPixelRatio, so CI produced no actual images for them and they are unchanged.

Owner decisions / required actions

  1. win32 baselines (local only). CI takes screenshots on Linux only, so the *-win32.png baselines are never checked in CI and nothing in CI can regenerate them. They are produced on a Windows machine with bun run test:e2e:update, which now builds the landing with the fixture through the Playwright webServer. Until then, a local Windows run shows diffs in section-bench, dark-section-bench and components-benchmark-section-mobile.
  2. Allow the bot to push to main. This works today only because the main ruleset targets no branch. If you fix the ruleset to target main (it requires the publish and benchmark checks with a strict policy), add a bypass for GitHub Actions or a GitHub App, or the job falls back to the rolling PR. A ruleset with strict required checks plus a bot commit after every merge would also mark every open PR out of date after each merge. Decide whether that is acceptable, or whether a path-based exception is preferable.
  3. Optional: keep "Read and write" default workflow permissions and "Allow GitHub Actions to create pull requests". The fallback PR needs the second one, and it is currently on.

Follow-ups (not in this PR)

  • Gzip size for Client Size.
  • Switch the landing cards from Build Size to Client Size once the first refresh has recorded it.
  • Fixture changes from the size investigation (for example the next.config.ts vs .mjs .tsbuildinfo difference).
  • The docs overview page (docs/overview/page*.mdx) still shows its labeled-historical run 33254030962. It could read the same JSON.

Test plan

  • bun test packages/plugin-utils/src/__tests__/benchmark-*.test.ts --coverage: 85 pass. benchmark-gate.js, benchmark-output.js and the new benchmark-record.js are at 100% functions and lines. The CLI is tested end to end in a temporary git repository: newer commit, older commit, a commit with no baseline, a commit not on main, --trend, and error cases.
  • Updated benchmark-output tests cover clientBytes.
  • bun render-benchmark-readme.js --check passes for all 9 READMEs.
  • formatJson output is byte-identical to the checked benchmark-results.json and benchmark/history.json (asserted in a test), and bunx prettier --single-quote --no-semi --check passes on every changed file. publish.yml already failed Prettier before this change.
  • ESLint passes on the changed scripts, tests and landing components. Local ESLint doesn't run for the later fixture commit's files, so I checked its rules by hand; prettier passes. CI's bun run lint is the final check.
  • Landing Bench.tsx, DevupUICard.tsx and OtherCard.tsx type-check with tsc (TS 7), including the JSON import.
  • actionlint 1.7.12 on publish.yml reports only the existing windows-11-vs2026-arm runner-label warnings.
  • CI landing-next-e2e run 44029448584: the fixture spec passed; the only failures were the three bench screenshots, which are now regenerated from that run's actual images.
  • CI: landing e2e with the new baselines, both Next and vinext.
  • After merge: the first main run pushes chore(benchmark): record run …, and the README Client Size column fills in.

요약

main에 병합될 때마다 README와 랜딩 홈페이지의 벤치마크 수치가 자동으로 갱신됩니다. 모든 실행은 커밋별 이력에도 남기 때문에, 어떤 변경 때문에 벤치마크가 나빠졌는지 확인할 수 있습니다.

main push 시 동작

publish.yml에 새로 추가한 benchmark-refresh job은 benchmark 다음에 실행됩니다. benchmark-run artifact를 받아 다음 작업을 합니다.

  1. benchmark-results.json을 다시 씁니다: run id, 전체 SHA와 짧은 커밋, 날짜, runner, Next.js와 Devup UI 버전, 행마다 라이브러리 버전, 샘플, 중앙값, 범위, 사이즈, CSS 사이즈, 클라이언트 사이즈.
  2. benchmark/history.json에 실행을 추가합니다. 커밋마다 항목 하나를 커밋 순서대로 남기며 최대 500개까지 보관합니다(항목당 약 2KB, 전체 약 1MB). 항목마다 gate 판정(pass/fail), 회귀한 행, 22개 행 전체의 [median, min, max, bytes, cssBytes, clientBytes]가 들어 있습니다.
  3. README 섹션을 다시 생성합니다: 루트, README_ko.md, 패키지 README 7개(#911에서 추가).
  4. chore(benchmark): record run <id> of <sha> [skip ci]로 커밋해 main에 push합니다.

커밋 반영 방식: main에 직접 push하고, 실패하면 rolling PR로 대체

  • gh api …/branches/main/protection은 404를 반환합니다(main에 branch protection이 없음). 유일한 ruleset main(id 9882973)은 conditions.ref_name.include: []라서 현재 어떤 브랜치에도 적용되지 않습니다. 기본 워크플로 권한은 write입니다. 따라서 지금은 GITHUB_TOKEN으로 main에 push할 수 있습니다.
  • GITHUB_TOKEN으로 한 push는 워크플로를 시작하지 않고, 커밋에 [skip ci]도 붙어 있습니다. 그래서 재귀 실행이 생기지 않고 publish, release, changepacks도 시작되지 않습니다.
  • job 도중 main이 움직이면 fetch 후 새 HEAD로 reset하고 커밋을 다시 만들어 재시도합니다(3회).
  • push가 계속 실패하면(예: ruleset이 bypass 없이 main을 대상으로 바뀐 경우) 커밋을 chore/benchmark-refresh에 force-push하고 PR 하나를 계속 열어 둡니다. GITHUB_TOKEN으로 연 PR에서는 CI가 돌지 않고 최신 실행만 담기 때문에, 이 경로는 비상용이지 기본 동작이 아닙니다.

안전장치

  • needs: [benchmark, release]와 if: !cancelled() && main push 조건을 씁니다. release(changepacks)가 끝난 뒤에만 시작하므로 둘이 동시에 push하지 않습니다. 빌드, publish, changepacks 단계는 실행하지 않고 changepacks/main(Update Versions #769)도 건드리지 않습니다.
  • 워크플로 자체의 ref별 concurrency 그룹에 더해 job 전용 concurrency: benchmark-refresh-main 그룹(cancel-in-progress: false)이 있습니다.
  • 오래된 실행 처리: benchmark-refresh.js가 git merge-base --is-ancestor로 확인합니다.
    • main에 없는 커밋의 측정은 무시합니다.
    • 게시된 run.sha보다 오래된 커밋은 이력에만 추가하고 더 새로운 결과를 덮어쓰지 않습니다.
    • 같은 커밋을 다시 실행하면 그 커밋의 이력 항목을 교체합니다.
  • gate가 실패한 실행도 기록해서 회귀가 이력에 남습니다. artifact가 없는 실행은 건너뜁니다.

gate와 표시. benchmark-gate.js, benchmark-baseline.json, benchmark-check.js의 동작은 그대로입니다. 이력은 표시와 원인 추적에만 쓰입니다. 각 항목의 판정은 그 커밋에 체크인된 baseline(git show <sha>:benchmark-baseline.json)을 기준으로 계산합니다.

회귀 원인 커밋 찾기

  1. bun benchmark-refresh.js --trend devup-ui-single-turbo를 실행합니다. main 커밋마다 짧은 SHA, 커밋 시각, gate 판정, 중앙값, 같은 실행 안에서 control 행과의 비율, bytes, CSS bytes, client bytes를 한 줄씩 출력합니다.
  2. 비율이나 사이즈가 처음 뛴 줄(또는 gate가 fail이 되고 regressions에 그 행이 나온 줄)이 원인 병합입니다. 바로 앞 줄과 비교해 보세요. git log <이전>..<sha>로 그 사이 커밋을 확인할 수 있으며, 보통 squash merge 하나입니다.
  3. 항목의 run id로 전체 로그와 benchmark-run artifact를 볼 수 있습니다. 대기 중에 더 새로운 push 때문에 취소된 워크플로 실행은 이력에 빈칸을 남깁니다. 이 경우 범위에 병합이 둘 이상 들어갈 수 있습니다.

사이즈 지표 (사이즈 조사 결과 반영)

  • 새 열 클라이언트 사이즈: 브라우저에 제공되는 JavaScript와 CSS(.next/static/**/*.{js,css} 또는 vinext dist/client/**/*.{js,css})의 압축 전 크기입니다.
  • 이제 모든 사이즈를 샘플마다 측정하며, sizes.{bytes,cssBytes,clientBytes}에 샘플과 min/max를 남깁니다.
  • 빌드 사이즈는 연속성을 위해 남기고 **빌드 사이즈(캐시 포함)**로 표시합니다. README 본문에 대부분이 번들러 캐시라는 설명을 넣었습니다.
  • fixture는 바꾸지 않았습니다.

초기 수치. benchmark-results.json과 첫 이력 항목은 main 실행 43770976836(커밋 c750d33c, NAPI 이후)의 benchmark-run artifact에서 전체 정밀도로 가져왔습니다. 이 값은 job 132995530168 로그의 median 줄과 같습니다. 라이브러리 버전은 그 커밋의 bun.lock에서 가져왔습니다. 이 실행은 클라이언트 사이즈 측정 이전이라 첫 갱신 전까지 그 열은 n/a로 표시됩니다.

랜딩. apps/landing/src/app/Bench.tsx는 benchmark-data.ts(스크린샷 항목 참고)를 거쳐 benchmark-results.json의 수치를 읽습니다. Bench는 서버 컴포넌트라 JSON이 클라이언트로 가지 않습니다. 캡션은 이제 실행과 커밋 링크를 보여 줍니다. 왕관은 Devup UI가 가장 빠르거나 가장 작을 때만 보입니다. Pages는 push된 커밋으로 배포되고, 갱신 커밋은 그 직후 [skip ci] 커밋으로 들어옵니다. 그래서 홈페이지는 다음 배포부터 새 수치를 보여 주며, README보다 병합 하나만큼 늦습니다. 실행/커밋 링크는 항상 보이는 수치와 일치합니다.

스크린샷: e2e 빌드는 고정된 fixture 데이터를 렌더링합니다. main의 벤치마크 갱신이 스크린샷을 바꾸면 안 되므로, e2e 빌드는 실시간 수치를 아예 읽지 않습니다.

  • e2e/fixtures/benchmark-results.json은 현재 결과(실행 43770976836, 커밋 c750d33c, Devup UI 1.0.44)를 고정한 사본입니다.
  • apps/landing/src/app/benchmark-data.ts는 빌드에 LANDING_BENCHMARK_DATA=fixture가 설정되면 fixture를, 아니면 실시간 benchmark-results.json을 고릅니다. Bench는 빌드 시 prerender되므로 선택은 빌드가 하고, 클라이언트 코드는 추가되지 않습니다. 번들러 alias 없이 vinext와 Next/Turbopack에서 같은 모듈이 동작합니다. 섹션에는 data-benchmark-source="fixture|live"가 붙습니다.
  • CI는 e2e가 서빙하는 모든 랜딩 빌드에 이 변수를 설정합니다: landing-next-e2e는 job 전체, publish는 두 랜딩 빌드. main push에서는 publish가 Pages artifact를 올리기 전에 이 변수 없이 singleCss 사이트를 다시 빌드하고, 배포될 index.html이 live인지 확인합니다. 로컬 Playwright webServer도 이 변수를 설정합니다.
  • e2e/landing-benchmark-fixture.spec.ts는 테스트 대상 빌드가 실시간 파일을 읽으면 실패합니다. source 표시와, 섹션에 fixture의 실행 id·시간·크기가 보이는지 확인합니다.
  • 이전의 마스킹(e2e/screenshot.css, stylePath, data-benchmark)은 제거했습니다.
  • 벤치마크 수치가 보이는 Linux baseline은 landing-next-e2e 실행 44029448584의 actual 이미지로 한 번 재생성했습니다: section-bench, dark-section-bench, components-benchmark-section-mobile. 세 번의 재시도 모두 같은 이미지였고, 그 실행에서 fixture spec도 통과했습니다. 섹션이 들어간 나머지 baseline(full-page-*, dark-full-page-*, components-benchmark-section-desktop, dark-components-benchmark-section-desktop)은 1% maxDiffPixelRatio 안에 들어서 CI가 actual 이미지를 만들지 않았으므로 그대로 두었습니다.

소유자 결정 / 필요한 작업

  1. win32 baseline(로컬 전용). CI는 Linux에서만 스크린샷을 찍으므로 *-win32.png는 CI에서 검사되지 않고, CI로 재생성할 수도 없습니다. Windows 머신에서 bun run test:e2e:update로 만듭니다. 이 명령은 이제 Playwright webServer를 통해 fixture로 랜딩을 빌드합니다. 그 전까지 로컬 Windows 실행에서는 section-bench, dark-section-bench, components-benchmark-section-mobile에 차이가 납니다.
  2. 봇의 main push 허용. 지금은 main ruleset이 어떤 브랜치도 대상으로 하지 않아서 동작합니다. ruleset을 main 대상으로 고치면(publish, benchmark 체크를 strict 정책으로 요구함) GitHub Actions나 GitHub App을 bypass에 추가해야 합니다. 그렇지 않으면 job이 rolling PR로 넘어갑니다. strict 필수 체크 ruleset에서는 병합 때마다 생기는 봇 커밋 때문에 열린 PR이 모두 out of date가 됩니다. 이를 받아들일지, 경로 기반 예외가 나을지 정해 주세요.
  3. 선택: 기본 워크플로 권한 "Read and write"와 "Allow GitHub Actions to create pull requests"를 유지해 주세요. 대체 PR에는 두 번째 설정이 필요하며, 현재 켜져 있습니다.

후속 작업(이 PR 범위 밖)

  • 클라이언트 사이즈의 gzip 크기.
  • 첫 갱신으로 클라이언트 사이즈가 기록되면 랜딩 카드를 빌드 사이즈에서 클라이언트 사이즈로 전환.
  • 사이즈 조사에서 나온 fixture 변경(예: next.config.ts와 .mjs 사이의 .tsbuildinfo 차이).
  • 문서 overview 페이지(docs/overview/page*.mdx)는 아직 과거 실행 33254030962를 표시합니다(과거 수치라고 표기됨). 같은 JSON을 읽게 할 수 있습니다.

테스트 계획

  • bun test packages/plugin-utils/src/__tests__/benchmark-*.test.ts --coverage: 85개 통과. benchmark-gate.js, benchmark-output.js, 새 benchmark-record.js 모두 함수/라인 100%. CLI는 임시 git 저장소에서 처음부터 끝까지 검증했습니다: 새 커밋, 오래된 커밋, baseline이 없는 커밋, main에 없는 커밋, --trend, 오류 케이스.
  • 수정한 benchmark-output 테스트가 clientBytes를 다룹니다.
  • README 9개 모두 bun render-benchmark-readme.js --check 통과.
  • formatJson 출력이 체크인된 benchmark-results.json, benchmark/history.json과 바이트 단위로 같습니다(테스트로 확인). 변경한 모든 파일에서 bunx prettier --single-quote --no-semi --check 통과. publish.yml은 이 변경 전부터 Prettier를 통과하지 못했습니다.
  • 변경한 스크립트, 테스트, 랜딩 컴포넌트에서 ESLint 통과. 이후 fixture 커밋의 파일은 로컬 ESLint가 실행되지 않아 규칙을 직접 확인했고, prettier는 통과했습니다. CI의 bun run lint가 최종 확인입니다.
  • 랜딩 Bench.tsx, DevupUICard.tsx, OtherCard.tsx가 JSON import를 포함해 tsc(TS 7) 타입 검사 통과.
  • publish.yml에 대한 actionlint 1.7.12 결과는 기존 windows-11-vs2026-arm runner label 경고뿐입니다.
  • CI landing-next-e2e 실행 44029448584: fixture spec 통과. 실패는 bench 스크린샷 세 개뿐이었고, 그 실행의 actual 이미지로 재생성했습니다.
  • CI: 새 baseline으로 Next와 vinext 랜딩 e2e.
  • 병합 후: 첫 main 실행이 chore(benchmark): record run …을 push하고, README의 클라이언트 사이즈 열이 채워지는지 확인.

🤖 Generated with Claude Code

owjs3901 and others added 7 commits November 15, 2026 09:51
- react and bundler plugin READMEs: take Features and How it works from
  the root README (current base-37 output instead of fixed d0/d1 names),
  replace the stale 1.0.40 benchmark copy with a link to the generated
  root tables, and point the Korean link at the root README_ko.md
- components and reset-css: replace the copied @devup-ui/react README
  with package-specific install, usage and API notes
- eslint-plugin: note that preset rules can report the same code and fix
  a no-duplicate-value "correct" example that no-useless-tailing-nulls
  reports
- AGENTS.md / libs/AGENTS.md: refresh commit, line counts and Oxc version

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Restore the React, plugin, components and reset-css READMEs to the main
branch structure and wording, then apply only the audit fixes:

- Keep the benchmark section in every package README that had it. It
  sits between the benchmark markers, and render-benchmark-readme.js now
  writes it into those READMEs from benchmark-results.json together with
  the root README, so the existing CI check fails when a copy goes stale.
  The generator also leaves a blank line after the start marker so the
  READMEs stay prettier-clean.
- Replace the fixed d0/d1 class names in the examples with the names the
  current build emits and add the note that generated names may change.
- Point the Korean README link at the repository README_ko.md.
- Components and reset-css: name their own package in the badges and
  install line and add a short usage section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After a push to main, the new benchmark-refresh job takes the run the
benchmark job measured. It rewrites benchmark-results.json, appends the
run to benchmark/history.json (one entry per commit, with the gate
verdict and the rows that regressed), regenerates the README sections
and pushes the result to main as a [skip ci] commit made with
GITHUB_TOKEN. It waits for the release job and runs no build, publish
or changepacks step. A commit older than the published results is only
added to the history. If pushing to main keeps failing, the job keeps
one rolling pull request on chore/benchmark-refresh instead.

The benchmark now measures client bytes (the JavaScript and CSS under
.next/static or dist/client), keeps every size sample with its range,
and records the runner, Next.js version, Devup UI version, the library
version of each row and the date. The README tables gain a Client Size
column and label Build Size as including caches.

The checked results and the first history entry are seeded from the
benchmark-run artifact of main run 43770976836 (commit c750d33, after
the native engine), which predates client bytes, so that column shows
n/a until the first refresh.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comparison cards now import the checked results instead of carrying
numbers from run 33254030962, so the homepage shows the same run as the
READMEs and moves with every refresh. The caption links the run and the
commit it measured, and the crowns only show while Devup UI is the
fastest or smallest card.

Every benchmark value carries data-benchmark, and the Playwright
screenshot config hides those elements through stylePath, so a refresh
never changes a visual baseline. The baselines that include the bench
section have to be regenerated once for this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #983, which records the styled identity wrapper in the extractor
snapshots that made the publish job fail.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
e2e/screenshot.css hides every [data-benchmark] element, so the bench
section, its dark variant and the mobile component showcase no longer show
the build times, sizes, versions or run line that the old baselines held.
These are the actual images from the landing-next-e2e run 44011106735 (the
same image on all three retries); they differ from the old baselines only in
the hidden benchmark text and in the header's language switcher, which the
old baselines predate and which the 1% ratio used to absorb.

Only the Linux baselines are recorded here; CI runs the screenshots on
Linux only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Files with missing lines Coverage Δ
benchmark-output.js 100.00% <100.00%> (ø)
benchmark-record.js 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

owjs3901 and others added 4 commits November 17, 2026 15:11
Hiding the numbers with [data-benchmark] and a Playwright stylePath still
left the screenshots tied to the shape of the live data. The e2e builds now
render a frozen copy of benchmark-results.json instead, so a benchmark refresh
on main can never change a screenshot.

- e2e/fixtures/benchmark-results.json is the current results, frozen.
- apps/landing/src/app/benchmark-data.ts picks the fixture when the build sets
  LANDING_BENCHMARK_DATA=fixture and the live file otherwise. Bench is a
  server component prerendered at build time, so the choice ships no client
  code, and it works the same under vinext and Next/Turbopack. The section
  carries data-benchmark-source="fixture|live".
- CI sets the variable on every landing build that e2e serves
  (landing-next-e2e for the whole job; the two publish builds). On a push
  to main, publish rebuilds the singleCss site without it before uploading
  the Pages artifact and checks the deployed index.html says "live".
- The local Playwright webServer sets the variable too.
- e2e/landing-benchmark-fixture.spec.ts fails when a build under test
  reads the live file: it checks the source marker and that the section
  shows the fixture's run id, time and size.
- e2e/screenshot.css, the stylePath and the data-benchmark markers are
  removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bench section, its dark variant and the mobile component showcase now
show the frozen e2e fixture (run 43770976836, commit c750d33, Devup UI
1.0.44) instead of hidden values. These are the actual images from the
landing-next-e2e run 44029448584, identical on all three retries; the same
run passed landing-benchmark-fixture.spec.ts, so they were rendered from the
fixture. Only the Linux baselines are recorded: CI takes screenshots on Linux
only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #911 (squash-merged as fc9a6f0), which this branch was stacked
on. The README conflicts were all inside the generated benchmark sections;
they were resolved by running `bun render-benchmark-readme.js`, and
`--check` passes for every README.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #963, which landed right after the previous merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant