Skip to content

Run the older tracks on the emulator when a pull request can change them - #27

Merged
sbokatuk merged 2 commits into
mainfrom
ci-track-device-coverage
Oct 30, 2026
Merged

sbokatuk merged 2 commits into
mainfrom
ci-track-device-coverage

Conversation

@sbokatuk

@sbokatuk sbokatuk commented Oct 30, 2026 •

Copy link
Copy Markdown
Owner

pr.yml never passed a native version, so the 7.1.x and 6.1.x tracks were never packed on a pull request, and release.yml packs a tag with verify: false. Those packages reached nuget.org without running on a device. Published 7.1.5.4 freezes on x86_64, and its Audio package can't load on API 31.

What changes

A track now runs on a pull request that does one of these:

  • adds a release note for it, because merging that note is the release
  • adds or replaces its row in build/native-versions.tsv, because that changes the .aar
  • changes the checks themselves (pr.yml, build.yml, run-device-tests.sh, run-sample-test.sh, tracks-to-verify.sh), which runs every track that still ships

The new .github/scripts/tracks-to-verify.sh decides this. The rows that ship today are 6.1.6 and 7.1.5, plus 8.1.3, which is the default and always runs. The 8.1.2 row is only kept so that 8.1.7 can be rebuilt, so it runs only if someone adds a note for it.

Each track gets package tests, both sample builds, Audio and Video on net8 on API 31 and 35, and run-sample-test.sh on net10. Its packages are versioned <FFmpeg>-beta.<pr>.<run> and stay in the run. The artifact-suffix input keeps them out of the artifact that publish pushes, so betas are still only the default track's. publish waits for the tracks and accepts them being skipped.

Merging a note is how a track ships, so "verified on its pull request" now holds for older tracks too, without touching release.yml. This PR adds no release note.

A second commit fixes run-sample-test.sh. The first run of track 6.1.6 failed with "the sample did not load within 180s" while the sample was fully loaded on screen. A "Pixel Launcher isn't responding" dialog sat on top of it, and uiautomator only reads the focused window. Nothing was wrong with the package. This could hit any sample run, and more sample legs would hit it more often.

Cost

Measured on the control runs below and on this PR's own run. The default track alone is 16 jobs and 68 to 76 job-minutes.

Pull request Extra jobs Extra job-minutes
Ordinary (most of them) 0 0
Adds a note or changes a row, one track 8 about 40
Changes the checks, both tracks 16 about 85
Every pull request, had I chosen that 16 on every push about 85 on every push, roughly double

This PR's run, with both tracks beside the default, took 13 minutes against 12 without them, because the tracks run side by side. The repository is public, so the minutes are free. What every-PR would really cost is queueing and flakes: 16 more jobs on each push, for pull requests that ship nothing on those tracks, and any one failing would hold up the betas. One leg in these controls died on an sdkmanager zip error, which has nothing to do with the code.

Evidence

This PR's own run took the third trigger: all 32 jobs passed, both tracks included, and it published 8.1.3-beta.27.38 and no 7.1.5 or 6.1.6 beta.

The controls ran the same build.yml inputs as older-tracks from a push to temporary branches, so nothing was published. The branches are deleted.

Control Result
Real tracks, with the launcher frozen to raise the real dialog before the sample (run) All 16 track jobs green. This is the first time FFmpegKit 7.1.7 and 6.0.4 ran on x86_64. The dialog stayed up through the build with hide_error_dialogs set, the script closed it, and the sample passed on both tracks.
7.1.5 bound to 7.1.6 and the SIGXCPU opt-out removed (run) Audio and Video on API 31 on 7.1.6: can't enable GNU RELRO protection, passed=1 failed=14. Other device legs: 14 pass, then hang in the GC check. Sample: ANR on both tracks.
The publish condition against stub jobs (run) Publishes with no tracks and with green tracks. Doesn't publish on a red track, a red build or a failed version job.

The second control's legs match the failures seen on the published packages, so these legs can catch both. One leg in it, 6.1.6 Video on API 35, is the sdkmanager failure above and tested nothing; the other three 6.1.6 legs hung as expected.

tracks-to-verify.sh also passed 30 cases over real commits in a scratch clone: each trigger, notes that must not trigger (three-part, edited, default track), a note for an unmapped FFmpeg, bad revisions, and a merge commit with the base moved on. The last two fail loudly, because a diff that can't read its commits must not look like nothing to verify. The harness isn't in this PR.

Not covered

  • A change to shared binding code that breaks an older track shows up on that track's next release note, not on the pull request that caused it. It still can't ship through a note. That's the price of running tracks this rarely.
  • A tag pushed by hand skips all of this. The README now says so.
  • Older tracks run Audio and Video on a device, and Full through the sample. The other five variants don't load on a device on those tracks.
  • The note and mapping triggers, and publish with no tracks, have not yet run on a real pull request. This PR changes the checks, so it takes the third trigger. The first ordinary pull request after this merges is the real test of the skipped path.

🤖 Generated with Claude Code

Sergei Bokatuk and others added 2 commits October 30, 2026 18:04
pr.yml never passed a native version, so 7.1.x and 6.1.x were never packed
on a pull request, and release.yml packs a tag without verifying it. A
package could reach nuget.org having never run on a device, and 7.1.5.4 did:
it freezes on x86_64, and Audio fails to load on API 31.

A track now runs when a pull request adds its release note, adds or replaces
its row in native-versions.tsv, or changes the checks themselves. It gets
package tests, Audio and Video on API 31 and 35, and the sample. Merging a
note is how a track ships, so the release's "verified on its pull request"
now holds for those tracks without touching release.yml. Ordinary pull
requests pay nothing.

The packages stay in the run: artifact-suffix keeps them out of the artifact
publish pushes. publish waits for the tracks and accepts them being skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The first run of track 6.1.6 failed with "the sample did not load within
180s", with the sample fully loaded on screen. A "Pixel Launcher isn't
responding" dialog sat over it: hide_error_dialogs stops new dialogs but does
not close one already up, and uiautomator only reads the focused window, so
the script could not see the text it was waiting for. Nothing about the
package was wrong, and more sample runs would hit it more often.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 30, 2026 15:19

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are blocking approval of the older-track verification changes.

No open issues 🎉

What changed in this PR

This PR adds targeted verification for older FFmpeg tracks before release, while keeping ordinary pull requests on the default track and publishing only its beta packages.

Changes:

  • Selects older tracks when release notes, native-version mappings, or verification checks change.
  • Adds API-level matrix inputs and isolated artifacts; beta publishing waits for older-track verification when required.
  • Handles leftover emulator dialogs that can obscure the sample and cause false startup timeouts.
File Description
README.md Documents older-track verification and the manual-tag limitation.
.github/​workflows/​pr.yml Selects and verifies older tracks, then gates beta publishing.
.github/​workflows/​build.yml Adds configurable API levels and artifact suffixes.
.github/​scripts/​tracks-to-verify.sh Determines which older tracks require verification.
.github/​scripts/​run-sample-test.sh Dismisses leftover dialogs during sample startup.
.github/​instructions/​workflows.instructions.md Records verification and artifact-isolation requirements.
.github/​instructions/​native-versions.instructions.md Documents emulator verification triggered by mapping changes.
.github/​copilot-instructions.md Summarises the older-track CI flow.

Review effort: Balanced

@sbokatuk
sbokatuk merged commit 63b2f37 into main Oct 30, 2026
33 checks passed
@sbokatuk
sbokatuk deleted the ci-track-device-coverage branch October 30, 2026 16:05

This branch was successfully deployed

1 active deployment
nuget.org — baaf4eb1 Deployed Oct 30, 2026 by sbokatuk via publish beta to nuget.org #38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants