Skip to content

feat(build-logic): add opt-in PITest mutation testing support and skill - #16819

Open
borinquenkid wants to merge 3 commits into
8.0.xfrom
feat/mutation-testing-8.0.x
Open

borinquenkid wants to merge 3 commits into
8.0.xfrom
feat/mutation-testing-8.0.x

Conversation

@borinquenkid

Copy link
Copy Markdown
Member

Summary of Changes

  • PITest Support in build-logic:
    • Added grailsCodeAnalysis.enablePitest() opt-in and -Pgrails.code-analysis.enabled.pitest[.projects] overrides in GrailsCodeAnalysisPlugin and GrailsCodeAnalysisExtension.
    • Auto-configures the pitest task when invoked directly on the command line (e.g., ./gradlew :<module>:pitest).
    • Pre-excludes Groovy compiler-generated synthetic methods ($getStaticMetaClass, getMetaClass, setMetaClass, invokeMethod, getProperty, setProperty, $getLookup, pfget$*, pfset$*, super$*, this$*), closure classes (*$_closure*, *$_*), and runtime call sites.
    • Supports command-line class/test scoping via -Pgrails.code-analysis.pitest.targetClasses, -Pgrails.code-analysis.pitest.targetTests, -Pgrails.code-analysis.pitest.threads, and -Pgrails.code-analysis.pitest.mutators.
  • Specifications:
    • Extended GrailsCodeAnalysisPluginSpec to verify PITest opt-in, CLI property overrides, and Groovy synthetic exclusions.
  • Agent Skill:
    • Added .agents/skills/mutation-testing/SKILL.md and updated skill indexes in AGENTS.md, .agents/skills/README.md, and .agents/skills/test-fixer/SKILL.md.

Copilot AI balanced review requested due to automatic review settings November 1, 2026 02:01

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Mutation-testing configuration can silently skip intended classes and tests, blanket exclusions omit handwritten behavior, and the added tests do not verify actual mutant execution or configuration-cache reuse.

5 open issues
What changed in this PR

Adds opt-in PITest mutation testing to Grails’ shared build conventions, with targeted execution controls and contributor guidance for assessing test effectiveness.

Changes:

  • Adds PITest enablement through the analysis extension, Gradle properties, and direct task requests.
  • Configures mutation targets, reports, concurrency, and Groovy bytecode exclusions, with configuration-level tests.
  • Introduces a mutation-testing skill and links it from existing contributor guidance.
File Description
gradle.properties Pins PITest engine, Gradle plugin, and JUnit 5 plugin versions.
build-logic/​plugins/​src/​test/​groovy/​org/​apache/​grails/​buildsrc/​GrailsCodeAnalysisPluginSpec.groovy Tests enablement, target selectors, and exclusions.
build-logic/​plugins/​src/​main/​groovy/​org/​apache/​grails/​buildsrc/​GrailsCodeAnalysisPlugin.groovy Implements PITest activation and configuration defaults.
build-logic/​plugins/​src/​main/​groovy/​org/​apache/​grails/​buildsrc/​GrailsCodeAnalysisExtension.groovy Exposes the enablePitest() opt-in.
build-logic/​plugins/​build.gradle Adds the PITest Gradle plugin dependency.
AGENTS.md Adds mutation testing to the skill index.
.agents/​skills/​test-fixer/​SKILL.md Links assertion-depth work to the new skill.
.agents/​skills/​README.md Lists mutation testing as a contributor-only skill.
.agents/​skills/​mutation-testing/​SKILL.md Documents commands, reports, exclusions, and semantic mutation checks.

Review effort: Balanced

- Integrate info.solidsoft.pitest (1.19.0 / PITest 1.19.6 / pitest-junit5-plugin 1.2.2) into GrailsCodeAnalysisPlugin and GrailsCodeAnalysisExtension
- Auto-enable PITest when the pitest task is requested on the command line or via grails.code-analysis.enabled.pitest[.projects]
- Disable upstream blanket FGROOVY filtering so @CompileStatic Groovy classes are mutated alongside Java classes while excluding synthetic Groovy compiler helpers
- Lazily evaluate targetClasses defaults and default targetTests to * independently of targetClasses
- Add .agents/skills/mutation-testing/SKILL.md covering targeted PITest execution and the Java/Groovy semantic mutation checklist
- Reference the mutation-testing skill from AGENTS.md, CLAUDE.md, GEMINI.md, and companion skills
@borinquenkid
borinquenkid force-pushed the feat/mutation-testing-8.0.x branch from 4bc4260 to 95f25fe Compare November 1, 2026 15:52
PITest only mutates the classes of the module it runs in, so a class could not be
mutation-tested when its specs live in another module (JpaQueryBuilder is in
grails-datastore-core, its specs in grails-datamapping-core-test and the Hibernate modules).

-Pgrails.code-analysis.pitest.mutableProjects=:other-project[,...] adds the compiled main
classes of those projects as mutable code and runs their classes task first. A path that does
not exist fails the build with a message that names the property.
grails-datamapping-core-test, a test-only module, now applies grails-code-analysis so it can
run PITest.

Covered by a TestKit build with two projects: the class of the other project is mutated and
killed with the property, left alone without it, and a missing path is refused.
@borinquenkid

Copy link
Copy Markdown
Member Author

@jamesfredley added -Pgrails.code-analysis.pitest.mutableProjects (head 94ff174). PITest only mutates the module it runs in, so a class whose specs live in another module could not be checked; with this the specs of one module mutate the classes of another. Five PRs on 8.0.x (#16816, #16817, #16818, #16820, #16821) now rely on PITest to verify their tests, so a look at this one first would unblock them.

@jamesfredley jamesfredley left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approving. A mutation run of an extracted test never sees that test.

defaults
})
)
pitest.targetTests.set(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A module whose tests are extracted for the shared suite runs those tests normally. A mutation run of one of those tests reports no coverage. The extracted classes are not on the mutation classpath. Put them there.

A module that runs the shared TCK extracts its specs from the tck jar into a directory only its Test
tasks read, because JUnit does not find tests inside a jar. PITest finds tests the same way, so a mutation
run of a TCK spec examined no test and reported every mutant as having no coverage. The jar is extracted
again for PITest, apart from the Test tasks' directory (their cleanup task deletes it), and that
directory is added to the mutation classpath. A module without a tck configuration is unchanged.

Claude-Session: https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL
@borinquenkid

Copy link
Copy Markdown
Member Author

Fixed in 659f040. The extracted classes were not on the mutation classpath, as you say.

A module that runs the shared TCK extracts the specs from the tck jar into build/extracted-tck-classes for its Test tasks, because JUnit does not find tests inside a jar. PITest finds tests the same way, so a mutation run of a TCK spec examined none. The plugin now registers extractTckForPitest for any module that has a tck configuration, extracts the jar into build/pitest-tck-classes, adds that directory to the PITest task's classpath and makes the task depend on it. It is a separate directory because the cleanup task deletes the Test tasks' one after them, and the extraction task there is finalized by test, which a mutation run should not start. A module without a tck configuration is unchanged.

Run on Hibernate 5 with targetClasses=org.grails.orm.hibernate.query.HibernateQuery and targetTests=org.apache.grails.data.testing.tck.tests.RangeQuerySpec: before, 0 tests examined and 0 of 35 lines covered; now 1 test examined and 4 of 35 lines covered, with a mutant killed. A TestKit fixture (a module whose only spec comes from its tck jar) fails without the change and passes with it, with the configuration cache on, and a second feature checks that a module without tck gets no such task. The 300 tests of build-logic/plugins pass, codeStyle is clean, and the skill says which modules this covers.

Other shapes I did not run: targetTests=* on a data module (the jar and the directory both hold the TCK classes, so I expect each to run once, but I have not counted it), and the Hibernate 7, MongoDB and Neo4j modules, which share the same configuration. If a mutation run there still shows no coverage, the target class and spec would be enough for me to reproduce it.

https://claude.ai/code/session_01Dmki4RzWwreEC2GkkyFnvL

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants