Skip to content

Implement read continuation for unresolved row targets - #10

Merged
alex-sandercock merged 2 commits into
developmentfrom
read-continuation
Nov 20, 2026
Merged

alex-sandercock merged 2 commits into
developmentfrom
read-continuation

Conversation

@alex-sandercock

Copy link
Copy Markdown
Contributor

Builds on unresolved-late-indels. Merge that one first.

What changes

A row whose target lies in an unresolved end is Other: the tag alone cannot tell a gap from substitutions there. Reads longer than the tag show what follows it. generate now uses those bases to decide such a row again.

  • Read pass: keeps the commonest bases after a tag, for the two canonical tags and for rows whose target is unread. They are stored in the per-sample cache.
  • Pooling: a tag has a continuation when 3 reads and 60% of its reads carry it.
  • Decision: the row, the Ref tag and the Alt tag are each read with their own bases after the tag. It is the same decision on a longer window. The answer counts when the window grows by 12 bases or more and reads the added bases with at most 2 substitutions.
  • Outcome: the gap stands, the substitutions stand, or the row stays unresolved. A settled row is written like any other row.
  • Not changed: tags, row identity, AlleleSequence, assignment and denoising. A row whose target the tag already reads is never asked. A target that an insertion pushed past the tag stays Other.
  • QC JSON: gains output_summary.read_continuation with bases_after_tag, rows_settled and reads_settled.

Effect

Three sets of simulated haplotypes:

Rows counted for Before After v0.13
the right allele 91.7% 94.8% 92.2%
the wrong allele 0.14% 0.14% 1.15%
no allele, target unread 3.2% 0.06% —

Cost

  • A run takes 4% to 7% longer where reads carry bases after the tag.
  • The cache grows by 2% to 3%.
  • The cache format is now 0.6.0. Existing caches are rebuilt once.

Copilot AI lite review requested due to automatic review settings November 20, 2026 21:46
@alex-sandercock alex-sandercock self-assigned this Nov 20, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An active moderate issue in madcats.py causes the tag continuation gate to omit the first two eligible reads, producing incorrect minimum-read and 60% share calculations.

1 open issue
What changed in this PR

Implements read-continuation rescue for unresolved row targets by caching and pooling post-tag bases, re-evaluating eligible rows on extended windows, and reporting continuation-related QC metrics while preserving existing assignment and allele identity behavior.

Changes:

  • Adds continuation collection, cache serialization, pooling thresholds, extended decisions, and settled-row annotation support.
  • Adds tests covering continuation decisions, pooling, cache reuse, primer removal, and unchanged short-read behavior.
  • Documents the continuation workflow, cache-format change, output fields, architecture, and performance effects.
File Description
tests/​test_read_continuation.py Adds coverage for continuation-based classification, pooling thresholds, cache reuse, primer removal, and unresolved cases.
madcats.py Implements continuation collection, pooling, extended row decisions, cache handling, annotation propagation, and QC reporting.
docs/​src/​reference/​output-formats.md Documents the output_summary.read_continuation QC fields.
docs/​src/​reference/​evidence-model.md Describes continuation-based row resolution and its safeguards.
docs/​src/​development/​changelog.md Records the feature, cache-format change, performance impact, and measured effects.
docs/​src/​development/​architecture.md Documents continuation data in the per-sample cache and processing architecture.
docs/​src/​commands/​generate.md Updates the generate pipeline and gapped-rescue behavior documentation.

Review effort: Lite

Comment thread madcats.py Resolved
Copilot AI lite review requested due to automatic review settings November 20, 2026 23:26

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A moderate active finding remains: the 60% threshold uses a truncated top-three continuation population rather than all reads reaching the window, allowing under-supported continuations to pass and miscalculating downstream counts.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Use all reads reaching the window in the 60% threshold denominator

madcats.py:2823

The 60% test is calculated only over at_length, which is reconstructed from the per-sample top three continuations. Reads of this tag that stop at the tag/selected length, and fourth-or-later continuation sequences discarded by CONTINUATIONS_KEPT, are absent from the denominator, so a continuation can pass this gate even when it is below 60% of the tag's reads (and the same truncated population also drives bases_after_tag). Retain the relevant total/length counts or otherwise include all reads that reach the chosen window when applying the threshold.

1 resolved since last review

Review effort: Lite

Base automatically changed from unresolved-late-indels to development November 20, 2026 23:31
…esolved end

A row whose end the tag leaves unresolved is decided again on a longer
window: the row and the Ref and Alt tags, each followed by the commonest
bases the cohort's reads carry after that tag.

- The read pass keeps, per sample, the 3 commonest strings of up to 40
  bases after a tag, from the tag's third read on. It keeps them for the
  two canonical tags and for rows whose target is unread. They are stored
  inside the cache's count entries; the cache format is 0.6.0.
- The kept strings are pooled over the samples. A tag has a continuation
  with 3 reads and 60% of the tag's reads.
- The longer window's answer is taken when the window grows by 12 bases
  or more, its own end is resolved, and it reads the added bases with at
  most 2 substitutions. The answer is cut back to the tag.
- Only a row whose target is unread is asked. A row the tag alone decides
  does not change. A row whose target an insertion pushed past the tag
  stays Other.
- Category, Variants, MarkerType and primer removal read the one shared
  decision. Variants is written from the alignment the row was settled
  with, in tables with and without primers.
- Tags, row identity, AlleleSequence, assignment and denoising are
  unchanged. Reads that end at the tag give byte-identical tables.
- The summary reports read_continuation: bases_after_tag, rows_settled,
  reads_settled.

Simulated haplotypes: 94.82% right, 0.14% wrong allele, 0.06% unread
(before: 91.69%, 0.14%, 3.18%). Pecan settles 149 rows and sweetpotato 45.
A run takes 4% to 7% longer and the cache grows by 2% to 3% where reads
carry bases after the tag.
The classification page said a continuation counts when 3 reads and 60%
of a tag's reads carry it.  The read pass keeps continuations from a
tag's third read in a sample on, so a sample's first two reads of a tag
are in neither count.  The page now says so.

No code changes.  Under the default evidence rule a new row has 10 reads
in each of two samples or 30 in one, so the reads left out never decide
whether it has a continuation.
@alex-sandercock
alex-sandercock merged commit a8e411a into development Nov 20, 2026
16 checks passed
@alex-sandercock
alex-sandercock deleted the read-continuation branch November 20, 2026 23:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants