Repository navigation
Run every variant and the sample on the emulator before publishing - #25
Merged
Merged
Conversation
ReportsBundledFFmpegVersion claimed a cross-check but only asserted the value was non-empty, so a package named for one FFmpeg and carrying another passed, and every gate had to read the ffmpeg= log line by hand. run-device-tests.sh now derives the FFmpeg from the package version, resolves it through build/native-versions.tsv so an unmapped version stops before the build, and compiles it into the test app, which compares it to what the library reports. An empty value fails rather than skips, so a build that stops passing it cannot turn this back into the old check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI ran one variant, and built the sample without ever deploying it, so a package that froze the sample on its first conversion passed everything. The e2e matrix now runs all eight variants on net10, whose assets the merge step grafts in per variant. net8 keeps a Video leg and gains Audio on API 31: Audio is the one variant whose load behaviour has depended on the ABI, and that RELRO failure only shows on an older Android, so API 35 alone hides it. run-sample-test.sh installs the sample, taps convert, requires the success text and fails on an ANR or a dead process. It builds the sample with a tiny GC nursery: the freeze needs a collection after FFmpeg has run, and without this the published 8.1.2.5 passed three runs out of three. With it, that package fails with the known ANR and 8.1.3.1 passes. Both are inside build.yml's verify gate, so they stop the beta publish and release runs skip them as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
The sample test script can reuse a cached NuGet package after repacking the same version, allowing tests to pass against older binaries unless the package cache is cleared or isolated.
1 open issue
What changed in this PR
This PR strengthens pre-publish validation for the Android bindings by exercising every native variant and running the MAUI sample to catch app freezes missed by smoke tests. Release publishing and its guard remain unchanged.
Changes:
- Expands emulator coverage to all eight variants on net10, plus Video and Audio on net8.
- Checks the runtime FFmpeg version against the package’s mapped version.
- Adds a sample conversion and responsiveness check under increased garbage-collection pressure, with diagnostic uploads.
| File | Description |
|---|---|
tests/FFmpegKit.Android.DeviceTests/SmokeTests.cs |
Fails on missing or mismatched expected FFmpeg versions. |
tests/FFmpegKit.Android.DeviceTests/FFmpegKit.Android.DeviceTests.csproj |
Embeds the expected version as assembly metadata. |
README.md |
Documents expanded coverage and local sample checks. |
.gitignore |
Ignores sample logs and screenshots. |
.github/workflows/build.yml |
Expands the emulator matrix and adds sample execution. |
.github/scripts/run-sample-test.sh |
Installs and drives the sample, checking conversion and responsiveness. |
.github/scripts/run-device-tests.sh |
Resolves and passes the expected FFmpeg version. |
.github/instructions/workflows.instructions.md |
Documents required emulator and sample gates. |
.github/copilot-instructions.md |
Updates CI and testing guidance. |
Review effort: Balanced
| echo "==> installing the sample (version=${VERSION}, tfm=${TARGET_FRAMEWORK}, sdk=${sdk_version})" | ||
| # A failed build leaves any earlier APK in place, and that would then report a verdict of its own. | ||
| adb uninstall "${PACKAGE_NAME}" > /dev/null 2>&1 || true | ||
| ( cd "${SDK_DIR}" && dotnet build "${PROJECT}" \ |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Published packages carried a permanent app freeze because CI ran one variant and never deployed the sample. This closes those gaps. It does not touch
release.ymlor its guard, and adds no release note, so merging releases nothing.The numbers below are from real runs: this PR's run and throwaway runs on a branch that has since been deleted. The freeze and RELRO results used packages already on nuget.org.
Total cost, measured against the last run of the previous PR: 68 runner-minutes instead of 18, and 11.8 minutes of wall-clock instead of 9.6.
1. Every variant on the emulator
.aar, and nothing else loads its native code. net10 assets are grafted in per variant by the merge step, so that is the framework where a variant can go wrong on its own.Why Audio on API 31. On 8.1.9 no variant differs by ABI: across all 8
.aars there is noTEXTREL, no writable and executable segment, identicalDT_NEEDED, and 16 KB alignment everywhere. The one ABI-specific load failure on record is RELRO: FFmpegKit 7.1.6 maps it past the end of a library in five variants on both ABIs, and in Audio on x86_64 only. That failure doesn't show on API 35, so eight API 35 legs would all pass. This leg's job is to catch that class of failure if it comes back.Why one job per leg rather than one emulator working through a list. Legs run side by side, so they add almost no wall-clock time, and a failed boot re-runs one leg rather than all of them.
Cost: 10 emulator jobs instead of 2, at 4.3 to 5.2 minutes each, so about 40 more runner-minutes per push. They're unbilled because the repository is public, and the slowest leg took 5.2 minutes against 4.9 before. Each extra emulator boot is one more chance of an infrastructure flake, which is what the per-leg re-run offsets.
2.
ReportsBundledFFmpegVersionnow fails on a mismatchrun-device-tests.shtakes the FFmpeg from the package version, resolves it throughbuild/native-versions.tsvso that an unmapped version stops before the build, and compiles it into the test app. The app compares it with what the library reports at runtime.Cost: no extra job or minutes. Two behaviour changes: a test app built outside the script now fails this check on purpose, and an upstream build reporting anything other than
n<version>will fail until someone looks.3. The sample runs on an emulator
run-sample-test.shinstalls the sample, waits until it has loaded, taps convert, requires the success text, then keeps tapping for 15 seconds to prove the app still takes input. An ANR or a dead process fails the run. Screenshot and full log are uploaded either way.One thing to review closely. The script builds the sample with
MONO_GC_PARAMS=nursery-size=64k, from a scratch file, so the sample's own project is unchanged. Without it, the published 8.1.2.5 passed this script three times out of three: the freeze needs a collection after FFmpeg has run, and with the default 4 MB nursery this small app may never have one. So this is still a Release build of the real package, but with far more frequent collections than an app would see. That's what makes the check meaningful.Cost: one job of 6 to 8.5 minutes, which becomes the slowest job in the run and accounts for the extra wall-clock time. It's also the most flake-prone check here, because it drives a real UI on a software-rendered emulator. It waits on what's on screen rather than fixed delays, reads tap positions from the UI tree, and hides system dialogs so that a "System UI isn't responding" can't swallow a tap.
Evidence
9.9.9.1can't enable GNU RELRO protection, 1 of 15 passn8.1.3The ANRs carry the known signature: main thread in
sem_wait,.NET TP Workerthreads in__rt_sigsuspend.The two hangs also show that the published 7.1.5.4 freezes on x86_64. That wasn't known before; 6.1.6.4 was not run.
Not covered
🤖 Generated with Claude Code