Skip to content

Unresolved late indels - #9

Merged
alex-sandercock merged 6 commits into
developmentfrom
unresolved-late-indels
Nov 20, 2026
Merged

alex-sandercock merged 6 commits into
developmentfrom
unresolved-late-indels

Conversation

@alex-sandercock

Copy link
Copy Markdown
Contributor

This pull request updates the documentation and code to introduce and explain a new "unresolved end" state for rows in the alignment and classification process. This state occurs when a gapped alignment could explain a row with fewer edits but lacks enough bases after the last gap to verify it, so the end of the row is not confidently read. The changes clarify how such cases are handled, annotated, and reported throughout the tool and its outputs.

The most important changes are:

Evidence Model and Classification Logic:

  • Introduced the concept of an "unresolved end" for rows where a gapped alignment is plausible but cannot be verified due to insufficient bases after the last gap. Such rows are now marked as unresolved, and their target alleles may be classified as Other if the ambiguity affects the target position. [1] [2] [3]
  • When both Ref and Alt tag alignments require the same number of edits, the one with fewer separate gaps is now chosen, improving assignment accuracy for certain indel cases. [1] [2]

Output Formats and Annotation:

  • Added a new unresolved marker type and the unresolved=k token in the Variants field, indicating how many bases at the end of a row are not read due to ambiguity. Updated documentation for how this affects downstream consumers and guidance for handling such rows. [1] [2] [3] [4]
  • Updated the QC/provenance JSON to include counts of unresolved rows and the number of reads affected, ensuring users can track the impact of this new state.

Codebase and Architecture:

  • Refactored the alignment engine to introduce row_decision, a new structure that encapsulates both the accepted gapped reading (if any) and the unresolved end (if present). The GappedReading class now records the template used. [1] [2] [3]
  • Split the logic for accepting gapped readings: now, if a reading is plausible but not verifiable, the row is marked as unresolved rather than being forced into a possibly incorrect classification.

Documentation and Changelog:

  • Thoroughly updated documentation and the changelog to describe the new behavior, including the impact on output tables, sidecar files, and backward compatibility with previous versions. [1] [2]

These changes improve the accuracy and transparency of row classification and variant annotation, especially in ambiguous cases near the ends of reads.

…e verified

A gapped reading that explains a row with fewer edits than substitutions,
but has fewer than 12 bases after its last gap, was dropped and the row was
read at the fixed offsets. Its end was then written as substitutions no
sample carries, and an insertion or deletion before the target could put
the other allele's base on the target.

The row is now read up to the first base at which the two explanations
differ (row_decision). A target at or after that base is Other. Variants
closes with the new token unresolved=k (the last k Ref bases are not read
in this row), MarkerType has the new value unresolved, the sidecar the new
Type unresolved, and the QC JSON counts the rows under unresolved_ends.
The decision is made once on the full tag and shared by classification,
Variants, the pooled denoiser and primer removal. Every row still rebuilds
from its Ref row and its Variants.

call reads the token: positions in the unread end show no base. A Variants
cell with a token call does not know now leaves the row unread and counted.
It is no longer replaced by events derived from the sequence.

This changes the default table. On pecan, sweetpotato, alfalfa and
blackberry 2.2% to 3.2% of rows get the token (0.7% to 2.0% of reads) and
0.12% to 0.22% of reads move from RefMatch or AltMatch to Other. None moves
to the other allele. On three sets of simulated haplotypes the rows counted
for the wrong target allele fall from 1.2% to 0.13%, and 1.0% fewer are
counted for the right one. generate from cache takes as long as before.

On the new sweetpotato table call's read-model estimate moves from 40 to
30 and 10,409 target genotypes go missing. With the estimator fix on
call-estimator-stability it stays at 40.
…signed-indel markers

Two places wrote a row whose end is not read as if it were.

With --remove-primers on, a row cut at its right primer to another
length than the Ref row lost its unresolved=k token: the located primer
was taken to settle the gap.  It does not: the row's category was decided
on the full tag, where the gap was not verified.  On pecan 619 of 889
unread rows lost the token, 137 of them rows that are Other because
their target is not read and yet showed a definite reading of it.
trim_locus_rows now carries the unread stretch whatever length the row
was cut to; only a stretch that lay wholly in the primer cut from the
Ref row leaves nothing to mark.  Every row that is Other for an unread
target now keeps the token (pecan 228 of 228, sweetpotato 124 of 124).

On a marker designed as an indel no row got the token, because its rows
are described against the Ref tag through the designed gap, which is
neither of the two readings the decision compares.  unresolved_end now
places the stretch in the Ref tag's coordinates by the base of the row
it starts at.  On blackberry 8 rows gain the token.

Categories and read counts are unchanged, and every row still rebuilds
from its Ref row and its Variants.  The cost is in tables without
primers: on the simulated haplotypes the rows whose Variants are exact
go from 88.6% to 79.1%, and the planted indels found from 59,685 to
48,879 of 72,622, because rows whose gap a located primer had in fact
settled are now marked too.  Tables with primers are unchanged.
…ins a tie in edits

A row can fit the Ref tag and the Alt tag through gaps in the same
number of edits: an Alt haplotype with one insertion or deletion beside
its target reads against the Ref tag as two gaps around the target, or
with the target inside a deletion.  The Ref tag's reading was taken on
every such tie, and the row was counted as RefMatch or Other.

Readings of the two tags are now ranked by edits, then by gap openings
(_reading_cost), as gapped_alignment already ranks the alignments to one
tag.  Readings equal in both still take the Ref tag's.

On pecan, sweetpotato, alfalfa and blackberry 0.03% to 0.10% of rows
change category (0.002% to 0.04% of reads), all of them to AltMatch.

Variants is still written from the row's alignment to the Ref tag, so
on these rows it can show the target unchanged or inside a deletion
while the category is AltMatch.  The evidence model says so.
A row whose category was read from the Alt tag had its Variants written
from its own alignment to the Ref tag.  That alignment has no reason to
put the Alt base on the target: it can split one gap in two, or put the
target inside a deletion, and Variants then showed the target unchanged
or deleted on an AltMatch row.

Such a row is now aligned to the Alt tag and written in the Ref tag's
positions (_read_from_alt, the against argument of _describe), and a
GappedReading records the tag it was made against.  A gap right after a
substituted base is written without an anchor base (31:G>A;32:CACG>.),
as a gap on the first base of a tag already was, because the
substitution uses that base.  call reads a deletion without an anchor
from its own first base at any position.  trim_locus_rows carries which
tag a printed row was read from, so a table without primers is written
from the same reading.

With primers kept, the rows whose category and Variants put different
bases on the target go from 75, 44, 28 and 68 to none on pecan,
sweetpotato, alfalfa and blackberry.  Categories, rows and counts do
not change, every Variants cell parses back to the same events, and no
row disagrees that agreed before.  Without primers 31, 16, 21 and 0
such rows remain (112, 61, 46 and 68 before): rows cut at their right
primer to another length than the Ref row, whose category was read on
the full tag.  They predate this change.
… primers

A row cut at its right primer to another length than the Ref row is
aligned end to end, while its category was read on the full tag.  On
pecan, sweetpotato and alfalfa 16 to 31 such rows per panel put
different bases on the target; blackberry has none.  The output-formats
page and the changelog said no row does.
Brings in primer removal as merged (pull request #8, with the two fixes
from its review) and the fix to call's read-model estimate (#7).

One conflict, in generate.md, where both sides rewrote the paragraph on
comparing with v0.13.  It now says both: --leading-anchor-mismatches 0
belongs to the settings that restore the v0.13 assignment, and no option
restores the old reading of a row with an unresolved end.
@alex-sandercock alex-sandercock self-assigned this Nov 20, 2026
Copilot AI lite review requested due to automatic review settings November 20, 2026 21:43

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified, and the change is supported by extensive regression coverage and corresponding implementation and documentation updates.

No open issues 🎉

What changed in this PR

This PR introduces an unresolved end state for rows whose cheaper gapped alignment cannot be verified near the read terminus. It updates alignment decisions, variant annotations, primer removal, downstream calling, QC reporting, documentation, and regression coverage.

Changes:

  • Adds RowDecision and unresolved-end detection, including fewer-gap tie breaking between Ref and Alt alignments.
  • Adds unresolved=k annotations, an unresolved marker/sidecar type, primer-removal propagation, and QC counts.
  • Updates call behavior for unresolved rows and unknown future Variants tokens, with extensive tests and documentation updates.
File Description
tests/​test_unresolved.py Adds end-to-end coverage for unresolved classification, annotations, primer removal, denoising, QC, and calling.
tests/​test_remove_primers.py Extends variant reconstruction tests for unresolved events during primer removal.
tests/​test_reading_ties.py Covers Ref/Alt alignment tie breaking and variants generated from the selected reading.
tests/​test_madcats.py Updates classification and annotation expectations for unresolved late indels.
tests/​test_call.py Tests unsupported or future Variants tokens and unread-row behavior in call.
madcats.py Implements row decisions, unresolved alignment handling, variant serialization and parsing, primer propagation, and QC summaries.
madcats_call.py Reads unresolved annotations, handles unsupported variant tokens, and updates downstream calling behavior.
docs/​src/​reference/​output-formats.md Documents the unresolved marker, token, sidecar type, QC fields, and consumer behavior.
docs/​src/​reference/​evidence-model.md Explains unresolved-end classification and gapped-reading selection.
docs/​src/​development/​changelog.md Records the new unresolved behavior, compatibility changes, and observed impact.
docs/​src/​development/​architecture.md Updates the alignment-engine architecture description for row decisions and unresolved ends.
docs/​src/​commands/​generate.md Documents gapped-rescue behavior and related compatibility guidance.

Review effort: Lite

@alex-sandercock
alex-sandercock merged commit c7c3c16 into development Nov 20, 2026
17 checks passed
@alex-sandercock
alex-sandercock deleted the unresolved-late-indels branch November 20, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants