From 5342bd881fc6f91e27f6483e40116455e788c1d5 Mon Sep 17 00:00:00 2001 From: Alexander Sandercock <39815775+alex-sandercock@users.noreply.github.com> Date: Fri, 20 Nov 2026 08:24:13 -0500 Subject: [PATCH 1/2] generate: read continuation settles a row whose target lies in an unresolved end A row whose end the tag leaves unresolved is decided again on a longer window: the row and the Ref and Alt tags, each followed by the commonest bases the cohort's reads carry after that tag. - The read pass keeps, per sample, the 3 commonest strings of up to 40 bases after a tag, from the tag's third read on. It keeps them for the two canonical tags and for rows whose target is unread. They are stored inside the cache's count entries; the cache format is 0.6.0. - The kept strings are pooled over the samples. A tag has a continuation with 3 reads and 60% of the tag's reads. - The longer window's answer is taken when the window grows by 12 bases or more, its own end is resolved, and it reads the added bases with at most 2 substitutions. The answer is cut back to the tag. - Only a row whose target is unread is asked. A row the tag alone decides does not change. A row whose target an insertion pushed past the tag stays Other. - Category, Variants, MarkerType and primer removal read the one shared decision. Variants is written from the alignment the row was settled with, in tables with and without primers. - Tags, row identity, AlleleSequence, assignment and denoising are unchanged. Reads that end at the tag give byte-identical tables. - The summary reports read_continuation: bases_after_tag, rows_settled, reads_settled. Simulated haplotypes: 94.82% right, 0.14% wrong allele, 0.06% unread (before: 91.69%, 0.14%, 3.18%). Pecan settles 149 rows and sweetpotato 45. A run takes 4% to 7% longer and the cache grows by 2% to 3% where reads carry bases after the tag. --- docs/src/commands/generate.md | 13 +- docs/src/development/architecture.md | 3 +- docs/src/development/changelog.md | 14 ++ docs/src/reference/evidence-model.md | 25 +- docs/src/reference/output-formats.md | 3 + madcats.py | 324 ++++++++++++++++++++---- tests/test_read_continuation.py | 355 +++++++++++++++++++++++++++ 7 files changed, 688 insertions(+), 49 deletions(-) create mode 100644 tests/test_read_continuation.py diff --git a/docs/src/commands/generate.md b/docs/src/commands/generate.md index 58cc28e..5fcf4ca 100644 --- a/docs/src/commands/generate.md +++ b/docs/src/commands/generate.md @@ -21,14 +21,16 @@ them: every marker get one more try with a gapped alignment under the same edit limit, so the counts include undesigned indels. Credit a read that fits several tiled markers of one amplicon equally to each of them, and drop other ties. Reads shorter than the tag count toward a full-length - allele whose start they match. Cache results per sample. + allele whose start they match. Keep the commonest bases that follow a tag in longer reads, for + stage 6. Cache results per sample. 3. Count exact tag sequences per sample. 4. Apply the evidence rules. Always write canonical Ref and Alt rows, and report any other sequence only if it passes an evidence route. 5. Denoise. Within each sample, merge rare sequences one edit away from a much more common one into it. A pooled pass then merges sequences that failed the evidence rules into a parent that carries the same target allele. -6. Classify each row as Ref, Alt, RefMatch, AltMatch or Other, and annotate it. +6. Classify each row as Ref, Alt, RefMatch, AltMatch or Other, and annotate it. A row whose + target the tag cannot read is read again with the bases that follow the tag. 7. Write the MADC, the QC/provenance JSON and the variants sidecar. The [Evidence Model](../reference/evidence-model.md) explains stages 2–6 in detail. @@ -64,7 +66,7 @@ The [Evidence Model](../reference/evidence-model.md) explains stages 2–6 in de | `--leading-anchor ` | none | Constant sequence before the amplicon in raw deliveries (for DArT, `GATGGATCCAGTGC`). MADCATS cuts each read where the anchor ends, and the anchor must end within 40 bases beyond its own length. `auto` detects it from a sample of up to 24 FASTQs. With one or two FASTQs the detected anchor may include the sample barcode, and the run warns about it | | `--leading-anchor-mismatches {0,1}` | 1 | Substitutions allowed in the leading anchor. With `1`, MADCATS cuts a read that has no exact copy after its only one-substitution match, and counts it as `reads_leading_anchor_mismatch`. A read with no such match, or more than one, counts as `reads_no_leading_anchor`. Anchors under 12 bases must match exactly. `0` requires an exact match for every anchor, as in 0.13.0 | | `--primer-mismatches {0,1}` | 1 | Mismatches allowed while assigning a primer/anchor. With `0`, reads from Match haplotypes whose distinguishing SNP falls inside the anchor are lost | -| `--gapped-rescue {on,off}` | `on` | Rescue indel reads and allow accepted gapped alignments to locate the target allele and annotate variants. A row whose end has a cheaper but unverified gap is read up to it and marked `unresolved`. With `off`, retained indel rows can have incorrect target categories; see [Classification](../reference/evidence-model.md#classification) | +| `--gapped-rescue {on,off}` | `on` | Rescue indel reads and allow accepted gapped alignments to locate the target allele and annotate variants. A row whose end has a cheaper but unverified gap is read up to it and marked `unresolved`, unless the bases after the tag settle it. With `off`, retained indel rows can have incorrect target categories; see [Classification](../reference/evidence-model.md#classification) | | `--max-indel-length ` | 10 | Widest gap (alignment band) a gapped alignment may use | | `--tile-groups {auto,declared,off}` | `auto` | Link tiled markers (markers that read the same tag window of one amplicon), so a read is credited to every tile it fits instead of being dropped as a tie. `auto` links tiles declared with `LocusGroup` and detects undeclared ones, `declared` links only the declared ones, and `off` links none. See [Tiled Markers](../getting-started/input-formats.md#tiled-markers) | | `--partial-read-mode {on,off}` | `on` | Add a read shorter than the tag to the one full-length allele (already passing the evidence rules) whose start it matches, only when it is longer than every prefix shared between that locus's alleles. `off` drops such reads | @@ -132,8 +134,9 @@ blanket guarantee of v0.13 output. Tile-group crediting was added after v0.13 an at a leading anchor, restores the v0.13 assignment. And a row whose end has a cheaper but unverified gap is now marked `unresolved`, where v0.13 read it at the fixed offsets (see [Classification](../reference/evidence-model.md#classification)): about 2 to 3 rows in 100 on the -cohorts measured, of which the ones with the target in that end become `Other`. No option restores -the old reading of those rows. Version/provenance metadata differs too. +cohorts measured, of which the ones with the target in that end become `Other` unless the bases +after the tag settle them. No option restores the old reading of those rows. Version/provenance +metadata differs too. What to know before using it: diff --git a/docs/src/development/architecture.md b/docs/src/development/architecture.md index f31dae8..fc8b336 100644 --- a/docs/src/development/architecture.md +++ b/docs/src/development/architecture.md @@ -77,7 +77,8 @@ annotation use the same `row_reading`, so those stages always agree. sequential passes. - The per-sample cache (`--cache-dir`) is keyed by the panel and every read-assignment parameter, so downstream thresholds never invalidate it. `CACHE_FORMAT_VERSION` is bumped whenever the - serialized cache or raw-read processing changes. + serialized cache or raw-read processing changes. The cache also holds the commonest bases + after each tag that read continuation can use (`keeps_continuation`). ## Vocabulary diff --git a/docs/src/development/changelog.md b/docs/src/development/changelog.md index d938958..46d7634 100644 --- a/docs/src/development/changelog.md +++ b/docs/src/development/changelog.md @@ -28,6 +28,20 @@ Notable changes by version. Measurements behind each change are in the 0.13% and the substitutions written that no haplotype carries from 108,592 to 8,312, while 1.0% fewer rows are counted for the right allele. Rows without the token are written as before, except a few whose counts change because the pooled denoiser groups rows by category. +- `generate` settles a row whose target lies in an unresolved end with the bases its reads carry + after the tag (read continuation). The read pass keeps the commonest of those bases for such + rows and for the two canonical tags. The row is then decided again on the longer window: it is + read through its gap or as substitutions and written like any other row, or it stays unresolved. + A target that an insertion pushed past the tag stays `Other`. Reads must run at least 12 bases + past the tag, so reads that end at the tag, and reads from `normalize-reads`, give the same table + as before. The QC JSON gains `read_continuation`. The cache format changed: existing caches are + rebuilt once. On three sets of simulated haplotypes the rows counted for the right allele rise + from 91.7% to 94.8% (v0.13: 92.2%) and unread rows fall from 3.2% to 0.06%; rows counted for the + wrong allele stay at 0.14% (v0.13: 1.15%). On pecan 149 rows are settled and 124 of them gain an + allele; on sweetpotato 45 and 31. No other row of those tables changes, and no read count. With + `--remove-primers on`, 7 of the settled pecan rows are cut at their right primer before the + target, like the rows named below. A run takes 4% to 7% longer on these cohorts and the cache + grows by 2% to 3%. - `generate` reads a row from the Alt tag when the Ref tag and the Alt tag explain it through gaps in the same number of edits and the Alt tag needs fewer separate gaps. Before, the Ref tag's alignment was used on every such tie. An Alt haplotype with one insertion or deletion beside its diff --git a/docs/src/reference/evidence-model.md b/docs/src/reference/evidence-model.md index 99757c4..c742894 100644 --- a/docs/src/reference/evidence-model.md +++ b/docs/src/reference/evidence-model.md @@ -144,8 +144,29 @@ to the first base at which the two explanations differ, and no further: Reading such a row at the fixed offsets, as versions before this outcome did, writes its end as substitutions that no sample carries, and an insertion or deletion before the target can put the base of the other allele on the target. Reading it through the unverified gap is right more often -but invents gaps in rows that have none. The row's own continuation past the tag would settle it, -and a tag does not hold it. +but invents gaps in rows that have none. + +**Read continuation.** A read longer than the tag shows what follows it, and that can settle a +row whose target lies in an unresolved end. `generate` keeps the commonest bases after such a row +and after the two canonical tags, and pools them over the samples. It then reads the row again on +a longer window: the row, the Ref tag and the Alt tag, each followed by its own bases. This is the +same decision on more bases, with three outcomes: + +- the gap stands, and the target is read through it; +- the substitutions stand, and the target is read at the fixed offset; +- the longer window settles nothing, and the row stays unresolved. + +A tag's continuation counts when 3 reads and 60% of its reads carry it. The window must grow by at +least 12 bases, and it must read the added bases with at most 2 substitutions. + +A settled row is written like any other row. Its `Variants` list the gap or the substitutions as +far as the row's own bases go. A target that an insertion pushed past the tag is not in the row, +so the row stays `Other`. A row whose target the tag already reads is never asked, so its category +cannot change. + +Read continuation needs reads that run at least 12 bases past the tag. Reads written by +`normalize-reads` end at the tag, so it does not act on them. The QC JSON reports the length +compared and the rows settled under `output_summary.read_continuation`. With `--gapped-rescue off`, both classification and annotation use the fixed offsets. Many indel reads fail assignment, but some still pass the substitution-only identity limit. Those rows can diff --git a/docs/src/reference/output-formats.md b/docs/src/reference/output-formats.md index 808e7b4..90005c1 100644 --- a/docs/src/reference/output-formats.md +++ b/docs/src/reference/output-formats.md @@ -124,6 +124,9 @@ unread stretch that lay wholly in the primer cut from the Ref row leaves nothing - `unresolved_ends`: the rows whose end is not read and their reads, and how many of them have the target in that end (those rows are `Other`). Counted on the full tags, so it is the same with and without `--remove-primers on` and with `--annotations off` +- `read_continuation`: the number of bases after the tag that reads were compared over + (`bases_after_tag`, 0 when reads end at the tag), and the rows and reads whose unread target + they settled ([Classification](evidence-model.md#classification)) - the variants sidecar path - with `--remove-primers on`, `primer_removal`: the panel's amplicon length, the markers by whether their right primer was located (and if not, why), the rows that were merged, the rows that lost diff --git a/madcats.py b/madcats.py index ffc01aa..69a5299 100644 --- a/madcats.py +++ b/madcats.py @@ -123,14 +123,15 @@ helpers). 2. Alignment engine: the Myers prescreen, the banded DP, and ``accepted_gapped_reading``/``row_decision`` — the one gapped reading a - row may be given, and where a row stops being read because its end has - two explanations. + row may be given, where a row stops being read because its end has + two explanations, and how the bases after the tag settle such an end. 3. Panel definition: design-file reconstruction, reference-tag tables, and primer-free inference. 4. Platform intake: pair merging, window location and freezing, tiling and routing, window-scan. 5. Per-sample read assignment: ``PrimerIndex``, ``tile_families``, - ``select_markers``, ``process_sample`` and its cache. + ``select_markers``, ``process_sample`` and its cache; the bases after + a tag (``keeps_continuation``, ``pooled_continuations``). 6. Evidence, denoising and partial-read salvage: ``EvidenceRule``, ``collect_evidence``, ``finalize_counts``, ``apply_partial_reads``. 7. Classification and annotation: ``classify_sequence``, ``variant_events``, @@ -184,7 +185,7 @@ # 0.4.0: fuzzy primer matches resolve across all matched depths by tag # identity instead of preferring the deepest primer, so caches built with # --primer-mismatches 1 (or prefix-nested panels) must be regenerated. -CACHE_FORMAT_VERSION = "0.5.0" +CACHE_FORMAT_VERSION = "0.6.0" TAG_LENGTH_DEFAULT = 81 QUALITY_THRESHOLDS = (10, 20, 30, 40) @@ -238,6 +239,12 @@ GAP_MAX_SUBSTITUTIONS = 6 # composition guard: an indel haplotype is its canonical plus gaps and FEW substitutions GAP_MAX_EVENTS = 6 # ... in few separate gap events; a co-amplified paralog needs many substitutions GAP_VERIFIED_SUFFIX = 12 # bases that must follow the last gap before a gapped reading may re-read the target +CONTINUATION_BASES = 40 # read continuation: bases after the tag kept from a read +CONTINUATIONS_KEPT = 3 # ... commonest continuations kept per tag and sample +CONTINUATION_REACHED_BY = 0.9 # ... all are cut to the longest length this share of the cohort's reads reach +CONTINUATION_MIN_READS = 3 # ... a tag's commonest continuation counts with this many reads (kept from as many in a sample) +CONTINUATION_MIN_SHARE = 0.6 # ... and this share of the tag's reads that carry one +CONTINUATION_MAX_MISMATCHES = 2 # ... a longer window settles a row only if it reads the added bases this cleanly PARTIAL_TAIL_DEFAULT = 9 # --min-partial-length defaults to --tag-length minus this: at most 9 unobserved tail bases LEADING_ANCHOR_SEARCH = 40 # --leading-anchor: the anchor must end within this many bases beyond its own length LEADING_ANCHOR_MISMATCHES_DEFAULT = 1 # --leading-anchor-mismatches: substitutions tolerated in the anchor @@ -291,6 +298,8 @@ class SampleResult: partial_quality_support: Dict[Tuple[int, str], Tuple[int, ...]] qc: Dict[str, int] cache_hit: bool = False + # (marker index, tag) -> [(bases after the tag, reads)], commonest first (keeps_continuation) + continuations: Dict[Tuple[int, str], List[Tuple[str, int]]] = dataclasses.field(default_factory=dict) class EvidenceSupport(NamedTuple): @@ -891,17 +900,21 @@ def _reading_cost(reading: GappedReading) -> Tuple[int, int]: return reading.edits, reading.gap_events -def row_reading(locus: Locus, sequence: str, gap_band: int) -> Optional[GappedReading]: +def row_reading( + locus: Locus, sequence: str, gap_band: int, continuations: Optional[Mapping[str, str]] = None +) -> Optional[GappedReading]: """The locus-level decision both consumers share: the accepted gapped reading of ``sequence`` against whichever canonical explains it most cheaply (_reading_cost), or ``None``. The bound a gapped reading must beat is the cheaper of the two substitution-only readings, and the composition guard is applied against the canonical the row is read from, so a designed substitution never counts against an AltMatch row's indel reading.""" - return row_decision(locus, sequence, gap_band).reading + return row_decision(locus, sequence, gap_band, continuations).reading -def row_decision(locus: Locus, sequence: str, gap_band: int) -> RowDecision: +def row_decision( + locus: Locus, sequence: str, gap_band: int, continuations: Optional[Mapping[str, str]] = None +) -> RowDecision: """How a row is read: its accepted gapped reading, and where it stops being read. Over both canonicals there is the cheapest gapped reading that is @@ -921,10 +934,96 @@ def row_decision(locus: Locus, sequence: str, gap_band: int) -> RowDecision: equal placements (_right_align), so every base both explanations agree on stays read. The decision is made once here and shared by classify_sequence, variant_events and the primer removal. + + ``continuations`` maps a tag of this marker to the bases its reads + carry after it (pooled_continuations). With them, a row whose target + lies in an unresolved end is decided again on a longer window + (_continued_decision). A row whose target the tag reads is never + asked, so its category cannot change. """ if gap_band <= 0 or sequence in (locus.ref_tag, locus.alt_tag): return RowDecision(None, None) - return _row_decision(locus.ref_tag, locus.alt_tag, sequence, gap_band, GAP_VERIFIED_SUFFIX) + decision = _row_decision(locus.ref_tag, locus.alt_tag, sequence, gap_band, GAP_VERIFIED_SUFFIX) + if not continuations or not _target_unread(locus, decision): + return decision + return _continued_decision(locus, sequence, gap_band, continuations) or decision + + +def _target_unread(locus: Locus, decision: RowDecision) -> bool: + """The marker's target lies in the end the decision leaves unresolved.""" + return decision.unresolved is not None and locus.variant_pos >= decision.unresolved[0] + + +def _continued_decision( + locus: Locus, sequence: str, gap_band: int, continuations: Mapping[str, str] +) -> Optional[RowDecision]: + """The decision of a row whose target is unread, made on a longer window; None when that settles nothing. + + The window is the row and the two canonical tags, each followed by the + bases its own reads carry after the tag (the Alt tag borrows the Ref + tag's when it has none). It is the same decision on more bases + (_row_decision), with no new way to align. Its answer is taken when + the window grows by at least GAP_VERIFIED_SUFFIX bases, its own end is + resolved, and it reads the added bases with at most + CONTINUATION_MAX_MISMATCHES substitutions. The last test keeps out a + row that only fits the longer tags through many substitutions. + + The answer is cut back to the tag (_cut_reading): the row is read as + far as its own bases go. A target the longer window finds past the tag + is not in the row, and the row stays Other. + """ + row_more = continuations.get(sequence) + ref_more = continuations.get(locus.ref_tag) + if not row_more or not ref_more: + return None + alt_more = continuations.get(locus.alt_tag) or ref_more + added = min(len(row_more), len(ref_more), len(alt_more)) + if added < GAP_VERIFIED_SUFFIX: + return None + row = sequence + row_more[:added] + ref = locus.ref_tag + ref_more[:added] + alt = locus.alt_tag + alt_more[:added] + reading, unresolved = _row_decision(ref, alt, row, gap_band, GAP_VERIFIED_SUFFIX) + if unresolved is not None: + return None + length = len(sequence) + if reading is None: + mismatches = min(tag_mismatches(row[length:], template[length:]) for template in (ref, alt)) + else: + mismatches = sum( + 1 for kind, i, j in reading.ops + if kind == "M" and i >= length and not _design_matches(row[i], reading.template[j]) + ) + if mismatches > CONTINUATION_MAX_MISMATCHES: + return None + return RowDecision(None if reading is None else _cut_reading(reading, sequence), None) + + +def _cut_reading(reading: GappedReading, sequence: str) -> GappedReading: + """A longer window's reading as the row itself shows it. + + Kept are the alignment steps that use the row's own bases and the tag's + own positions. The reading ends where an insertion pushes the tag's + bases past the row, or a deletion pulls in bases from past the tag, as + it does for a row read on its tag alone (gapped_alignment).""" + length = len(sequence) + template = reading.template[:length] + ops, end_read, end_ref = _cut_alignment(reading.ops, 0, length, length) + edits = sum(1 for kind, i, j in ops if kind != "M" or not _design_matches(sequence[i], template[j])) + return _reading_from_ops(sequence, template, ops, end_read, end_ref, edits) + + +def _cut_alignment(ops, start: int, row_length: int, ref_length: int): + """The steps of an alignment among ``row_length`` row bases and ``ref_length`` template bases from + ``start`` on, counted from there: ``(ops, end_read, end_ref)`` as gapped_alignment gives them.""" + kept = tuple( + (kind, i - start, j - start) + for kind, i, j in ops + if start <= i < start + row_length and start <= j < start + ref_length + ) + end_read = sum(1 for kind, _, _ in kept if kind != "D") + end_ref = sum(1 for kind, _, _ in kept if kind != "I") + return kept, end_read, end_ref @functools.lru_cache(maxsize=65536) @@ -2392,7 +2491,13 @@ def _load_cache(path: Path, sample: Sample, source_stat: os.stat_result) -> Opti # invalidates its cache without any content hashing on the hot path. if payload["source_size"] != source_stat.st_size or payload["source_mtime_ns"] != source_stat.st_mtime_ns: return None - counts = {(int(marker), sequence): int(count) for marker, sequence, count in payload["counts"]} + # A counts entry is [marker, tag, reads], followed by [bases, reads] for each continuation kept. + counts = {(int(marker), sequence): int(count) for marker, sequence, count, *_ in payload["counts"]} + continuations = { + (int(marker), sequence): [(bases, int(reads)) for bases, reads in kept] + for marker, sequence, _, *kept in payload["counts"] + if kept + } quality_support = { (int(marker), sequence): tuple(int(value) for value in values) for marker, sequence, *values in payload["quality_support"] @@ -2413,6 +2518,7 @@ def _load_cache(path: Path, sample: Sample, source_stat: os.stat_result) -> Opti partial_quality_support=partial_quality_support, qc=payload["qc"], cache_hit=True, + continuations=continuations, ) except (OSError, KeyError, TypeError, ValueError, json.JSONDecodeError): return None @@ -2425,7 +2531,10 @@ def _write_cache(path: Path, result: SampleResult, source_stat: os.stat_result) "source_size": source_stat.st_size, "source_mtime_ns": source_stat.st_mtime_ns, "qc": result.qc, - "counts": [[marker, sequence, count] for (marker, sequence), count in result.counts.items()], + "counts": [ + [marker, sequence, count, *result.continuations.get((marker, sequence), ())] + for (marker, sequence), count in result.counts.items() + ], "quality_support": [ [marker, sequence, *values] for (marker, sequence), values in result.quality_support.items() @@ -2666,6 +2775,56 @@ def add_quality_support( support[key][quality_index] += 1 +def keeps_continuation(locus: Locus, tag: str, gap_band: int) -> bool: + """Whether the read pass keeps the bases that follow ``tag`` in its reads. + + Only row_decision uses them, and only for a row whose target lies in + an unresolved end. It compares such a row with the two canonical tags, + so theirs are kept too.""" + if gap_band <= 0: + return False + return tag in (locus.ref_tag, locus.alt_tag) or _target_unread(locus, row_decision(locus, tag, gap_band)) + + +def pooled_continuations(results: Sequence[SampleResult]) -> Tuple[Dict[int, Dict[str, str]], int]: + """What follows each tag across the cohort: ``({marker index: {tag: bases}}, length)``. + + Reads end at different places, so one continuation arrives at several + lengths. All are cut to ``length``, the longest that + CONTINUATION_REACHED_BY of the reads reach; reads that stop short of it + are left out. A tag then has a continuation when its commonest one has + CONTINUATION_MIN_READS reads and CONTINUATION_MIN_SHARE of the tag's + reads. A tag followed by two sequences in turn (a paralog, a second + indel past the tag) has none. + """ + found: Dict[Tuple[int, str], Counter] = defaultdict(Counter) + reads_by_length: Counter = Counter() + for result in results: + for key, kept in result.continuations.items(): + for bases, reads in kept: + found[key][bases] += reads + reads_by_length[len(bases)] += reads + all_reads = sum(reads_by_length.values()) + length = reaching = 0 + for candidate in sorted(reads_by_length, reverse=True): + reaching += reads_by_length[candidate] + if reaching >= CONTINUATION_REACHED_BY * all_reads: + length = candidate + break + pooled: Dict[int, Dict[str, str]] = defaultdict(dict) + for (marker_index, tag), counts in found.items(): + at_length: Counter = Counter() + for bases, reads in counts.items(): + if len(bases) >= length: + at_length[bases[:length]] += reads + if not at_length: + continue + bases, reads = at_length.most_common(1)[0] + if reads >= CONTINUATION_MIN_READS and reads >= CONTINUATION_MIN_SHARE * sum(at_length.values()): + pooled[marker_index][tag] = bases + return dict(pooled), length + + def process_sample( sample: Sample, loci: Sequence[Locus], @@ -2691,6 +2850,11 @@ def process_sample( whole-tag-clean read tallies is kept so quality thresholds can change later without touching the FASTQs again. Reads shorter than the tag are collected separately for the optional partial-read stage. + + A read longer than the tag also shows what follows it. For the tags + row_decision can use that for (keeps_continuation), the commonest of + those continuations are kept with their read counts, from the tag's + third read in the sample on. """ fastq_path = Path(sample.fastq) source_stat = fastq_path.stat() @@ -2713,6 +2877,8 @@ def process_sample( # primer-less reads fail the trie walk cheaply and are not worth storing. full_decisions: Dict[str, Tuple[Tuple[int, ...], str]] = {} partial_decisions: Dict[str, Tuple[Tuple[int, ...], str]] = {} + continuations: Dict[Tuple[int, str], Counter] = defaultdict(Counter) + keepers: Dict[str, Tuple[int, ...]] = {} # per tag, the markers that keep its continuation family_of = {member: family for family in settings.tile_families for member in family} for sequence, quality in fastq_records(fastq_path): qc["reads_total"] += 1 @@ -2779,6 +2945,17 @@ def process_sample( key = (marker_index, tag) exact_counts[key] += 1 add_quality_support(exact_quality_support, key, quality[:settings.tag_length]) + # What follows the tag is kept from the tag's third read on: most tags are read once or + # twice, and deciding whether to keep theirs (keeps_continuation) would cost an alignment each. + continuation = sequence[settings.tag_length : settings.tag_length + CONTINUATION_BASES] + if len(continuation) >= GAP_VERIFIED_SUFFIX and exact_counts[(markers[0], tag)] >= CONTINUATION_MIN_READS: + if tag not in keepers: + keepers[tag] = tuple( + marker_index for marker_index in markers + if keeps_continuation(loci[marker_index], tag, settings.gap_band) + ) + for marker_index in keepers[tag]: + continuations[(marker_index, tag)][continuation] += 1 resolve_n_tags(exact_counts, exact_quality_support, loci, qc) qc["unique_full_tags_raw"] = len(exact_counts) @@ -2793,6 +2970,10 @@ def process_sample( }, qc=dict(qc), cache_hit=False, + # A tag with an N is counted under the tag it resolves to and keeps no continuation. + continuations={ + key: found.most_common(CONTINUATIONS_KEPT) for key, found in continuations.items() if key in exact_counts + }, ) if cache_path is not None: _write_cache(cache_path, result, source_stat) @@ -3497,7 +3678,7 @@ def marker_type_label(locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFA def _describe( ref: str, sequence: str, gap_band: int, gapped: bool, whole: bool = False, unresolved: Optional[Tuple[int, int]] = None, - against: Optional[str] = None, + against: Optional[str] = None, alignment=None, ): """Events of ``sequence`` against ``ref`` (the core of variant_events). @@ -3528,7 +3709,14 @@ def _describe( of these two sequences. The part before it is described end to end (both readings of the row agree on it), and one closing ``unresolved`` event holds the Ref bases not read and the row's bases that stand - there.""" + there. + + ``alignment`` is an alignment of these two sequences that the caller + already has (``ops``, ``end_read``, ``end_ref`` as gapped_alignment + gives them). It is written as it stands and none is made here: a row + that read continuation settled was read on more bases than it holds.""" + if alignment is not None: + return _alignment_events(*alignment, ref, sequence) if against is not None and len(against) != len(ref): against = None if unresolved is not None: @@ -3558,6 +3746,11 @@ def _describe( events.append(("overhang", len(ref) + 1, ".", sequence[shared:])) return events ops, end_read, end_ref, _ = gapped_alignment(sequence, ref if against is None else against, gap_band, whole) + return _alignment_events(ops, end_read, end_ref, ref, sequence) + + +def _alignment_events(ops, end_read: int, end_ref: int, ref: str, sequence: str): + """The events of one alignment of ``sequence``, written against ``ref`` (see _describe and gapped_alignment).""" ops = _left_align(ops, sequence, ref) events = [] k = 0 @@ -3600,15 +3793,19 @@ def _designed_events(ref: str, alt: str, gap_band: int): return tuple(_describe(ref, alt, gap_band, gapped=_reading(alt, ref, gap_band, guard=False) is not None)) -def _read_through_gaps(locus: Locus, sequence: str, gap_band: int) -> bool: +def _read_through_gaps( + locus: Locus, sequence: str, gap_band: int, continuations: Optional[Mapping[str, str]] = None +) -> bool: """Whether a row is described through gaps: every row of a marker whose own design is an indel, and otherwise a row with an accepted gapped reading (row_reading, the decision classification made).""" designed = _designed_events(locus.ref_tag, locus.alt_tag, gap_band) - return any(kind in ("ins", "del") for kind, *_ in designed) or row_reading(locus, sequence, gap_band) is not None + return any(kind in ("ins", "del") for kind, *_ in designed) or row_reading(locus, sequence, gap_band, continuations) is not None -def variant_events(locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFAULT): +def variant_events( + locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFAULT, continuations: Optional[Mapping[str, str]] = None +): """Every event a row carries against the Ref tag, in 1-based tag coordinates. Returns ``(kind, position, ref_allele, alt_allele)`` tuples in tag order: @@ -3634,15 +3831,26 @@ def variant_events(locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFAULT show the same base on the target. A gap right after a substituted base is written without an anchor base (``32:CACG>.``, ``32:.>GG``), as a gap on the first base of the tag is. + + A row that read continuation settled (row_decision with + ``continuations``) is written from that reading, cut to the row's own + bases. Its alignment was made on more bases than the row holds, so it + is taken from the decision and not made again on the tag. """ + reading, unresolved = row_decision(locus, sequence, gap_band, continuations) + from_alt = _read_from_alt(locus, sequence, gap_band, continuations) + if reading is not None and unresolved is None and (from_alt or reading.template == locus.ref_tag): + return _alignment_events(reading.ops, reading.end_read, reading.end_ref, locus.ref_tag, sequence) return _describe( - locus.ref_tag, sequence, gap_band, _read_through_gaps(locus, sequence, gap_band), - unresolved=unresolved_end(locus, sequence, gap_band), - against=locus.alt_tag if _read_from_alt(locus, sequence, gap_band) else None, + locus.ref_tag, sequence, gap_band, _read_through_gaps(locus, sequence, gap_band, continuations), + unresolved=unresolved_end(locus, sequence, gap_band, continuations), + against=locus.alt_tag if from_alt else None, ) -def _read_from_alt(locus: Locus, sequence: str, gap_band: int) -> bool: +def _read_from_alt( + locus: Locus, sequence: str, gap_band: int, continuations: Optional[Mapping[str, str]] = None +) -> bool: """Whether a row's accepted gapped reading was made against the Alt tag. row_decision reads a row from the Ref tag or from the Alt tag, whichever @@ -3651,13 +3859,15 @@ def _read_from_alt(locus: Locus, sequence: str, gap_band: int) -> bool: Never on a marker whose own design is an indel: there the tags' positions differ past the designed gap, and every row is described by its alignment to the Ref tag (_read_through_gaps).""" - reading = row_reading(locus, sequence, gap_band) + reading = row_reading(locus, sequence, gap_band, continuations) if reading is None or reading.template != locus.alt_tag or locus.alt_tag == locus.ref_tag: return False return not any(kind in ("ins", "del") for kind, *_ in _designed_events(locus.ref_tag, locus.alt_tag, gap_band)) -def unresolved_end(locus: Locus, sequence: str, gap_band: int) -> Optional[Tuple[int, int]]: +def unresolved_end( + locus: Locus, sequence: str, gap_band: int, continuations: Optional[Mapping[str, str]] = None +) -> Optional[Tuple[int, int]]: """row_decision's unresolved stretch in the Ref tag's coordinates, as a row's Variants show it, or None. The decision counts positions on the canonical tag each reading was @@ -3669,7 +3879,7 @@ def unresolved_end(locus: Locus, sequence: str, gap_band: int) -> Optional[Tuple written from, _read_through_gaps) puts under that base, moved back over any Ref bases deleted right before it. A known designed indel does not settle another gap near the row's end.""" - stretch = row_decision(locus, sequence, gap_band).unresolved + stretch = row_decision(locus, sequence, gap_band, continuations).unresolved if stretch is None or stretch[0] <= locus.variant_pos: return stretch if not any(kind in ("ins", "del") for kind, *_ in _designed_events(locus.ref_tag, locus.alt_tag, gap_band)): @@ -3760,7 +3970,9 @@ def parse_variants(text: str, ref: str, sequence: str): return events -def classify_sequence(locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFAULT) -> str: +def classify_sequence( + locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFAULT, continuations: Optional[Mapping[str, str]] = None +) -> str: """Classify an observed tag against the canonical Ref/Alt definitions. The Ref/Alt categories require exact sequence equality; per-row read @@ -3780,7 +3992,8 @@ def classify_sequence(locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFA bases at the target, and neither is taken. The variant-base comparison is IUPAC-aware on the design side only: an ambiguous observed base such as N is missing data, not evidence for - either allele, and classifies as Other. + either allele, and classifies as Other. ``continuations`` are handed + to row_decision, which may settle an unresolved end with them. """ if sequence == locus.ref_tag: return "Ref" @@ -3791,7 +4004,7 @@ def classify_sequence(locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFA # The cheapest substitution-only reading over both canonicals is the # bound a gapped reading must beat; the reading itself is the one # accepted_gapped_reading gives, shared with the row's Variants. - best, unresolved = row_decision(locus, sequence, gap_band) + best, unresolved = row_decision(locus, sequence, gap_band, continuations) if unresolved is not None and locus.variant_pos >= unresolved[0]: return "Other" if best is not None: @@ -3930,7 +4143,8 @@ def _cheapest_ops(row: str, template: str, gap_band: int, slack: Sequence[int] = def right_primer_cut( - locus: Locus, sequence: str, site: Optional[Tuple[int, str]], gap_band: int + locus: Locus, sequence: str, site: Optional[Tuple[int, str]], gap_band: int, + continuations: Optional[Mapping[str, str]] = None, ) -> Tuple[int, Optional[str]]: """How many bases after a row's left primer are the sample's own: ``(cut, kept)``. @@ -3990,7 +4204,7 @@ def primer_from(index: int, primer: str) -> bool: if primer_from(earlier, primer) and 2 <= sum( 1 for i in range(earlier, size) if i not in aligned or not _design_matches(body[i], template[aligned[i]]) ): - reading = row_reading(locus, sequence, gap_band) # where classification read the category + reading = row_reading(locus, sequence, gap_band, continuations) # where classification read the category category_base = locus.variant_pos if reading is None else reading.ref_to_read.get(locus.variant_pos) if category_base is None or earlier > category_base - after_left: cut, moved = earlier, True @@ -4037,6 +4251,7 @@ def trim_locus_rows( site: Optional[Tuple[int, str]], gap_band: int, tally: Counter, + continuations: Optional[Mapping[str, str]] = None, ): """One marker's rows without their primers: ``(locus, rows, counts, readings)`` as the table shows them. @@ -4056,9 +4271,11 @@ def trim_locus_rows( printed. ``readings`` gives ``_describe`` what the trimmed sequences no longer - show, as ``(gapped, whole, unresolved, against)`` per row (``against``: - the Alt row as printed when the category was read from the Alt tag, so - the row is described from the same reading). ``gapped`` is the + show, as ``(gapped, whole, unresolved, against, alignment)`` per row + (``against``: the Alt row as printed when the category was read from + the Alt tag, so the row is described from the same reading; + ``alignment``: for a row that read continuation settled, the alignment + it was read from, cut to the bases shown). ``gapped`` is the decision made on the full tag (_read_through_gaps): a primer that verified a gap is gone from the row, so the decision is carried, not made again. ``whole`` marks a row that was cut at its right primer on @@ -4071,19 +4288,31 @@ def trim_locus_rows( when the stretch lay in the primer that was cut from the Ref row. """ after_left = len(locus.left_primer) - cuts = {sequence: right_primer_cut(locus, sequence, site, gap_band) for _, _, sequence in rows} + cuts = {sequence: right_primer_cut(locus, sequence, site, gap_band, continuations) for _, _, sequence in rows} without_primers = lambda sequence: sequence[after_left : after_left + cuts[sequence][0]] if without_primers(locus.ref_tag) == without_primers(locus.alt_tag): # right_primer_cut keeps the target base, so the canonical rows stay # distinct; if they ever would not, the marker keeps its right end. cuts = {sequence: (len(sequence) - after_left, None) for sequence in cuts} tally["markers_right_primer_kept"] += 1 + def settled_alignment(sequence: str): + """The alignment read continuation settled a row with, among the bases shown; None for any other row.""" + reading, unresolved = row_decision(locus, sequence, gap_band, continuations) + if reading is None or unresolved is not None or row_decision(locus, sequence, gap_band).unresolved is None: + return None + if reading.template != locus.ref_tag and not _read_from_alt(locus, sequence, gap_band, continuations): + return None # read from the Alt tag of a designed indel: not in the Ref tag's positions + if any(kind != "M" for kind, i, j in reading.ops if i < after_left or j < after_left): + return None # a gap in the left primer: the row and the Ref row as shown start at different bases + return _cut_alignment(reading.ops, after_left, cuts[sequence][0], cuts[locus.ref_tag][0]) + merged: Dict[str, List] = {} gapped: Dict[str, bool] = {} from_alt: Dict[str, bool] = {} # the row whose category a printed row carries was read from the Alt tag + settled: Dict[str, object] = {} # ... and the alignment it was settled with (settled_alignment) for _, category, sequence in rows: trimmed = without_primers(sequence) - gapped[trimmed] = gapped.get(trimmed, False) or _read_through_gaps(locus, sequence, gap_band) + gapped[trimmed] = gapped.get(trimmed, False) or _read_through_gaps(locus, sequence, gap_band, continuations) reads = counts[sequence] total = sum(reads) tally["rows_right_primer_removed"] += after_left + len(trimmed) < len(sequence) @@ -4092,14 +4321,16 @@ def trim_locus_rows( entry = merged.get(trimmed) if entry is None: merged[trimmed] = [category, list(reads), total] - from_alt[trimmed] = _read_from_alt(locus, sequence, gap_band) + from_alt[trimmed] = _read_from_alt(locus, sequence, gap_band, continuations) + settled[trimmed] = settled_alignment(sequence) continue tally["rows_merged"] += 1 tally["rows_merged_across_allele_groups"] += ALLELE_GROUP[category] != ALLELE_GROUP[entry[0]] entry[1] = [kept + added for kept, added in zip(entry[1], reads)] if entry[0] not in ("Ref", "Alt") and total > entry[2]: entry[0] = category - from_alt[trimmed] = _read_from_alt(locus, sequence, gap_band) + from_alt[trimmed] = _read_from_alt(locus, sequence, gap_band, continuations) + settled[trimmed] = settled_alignment(sequence) entry[2] = max(entry[2], total) shown = dataclasses.replace( locus, @@ -4116,7 +4347,7 @@ def trim_locus_rows( ref_ends_at_primer = len(shown.ref_tag) < uncut or (site is not None and site[0] == len(locus.ref_tag)) unresolved: Dict[str, Tuple[int, int]] = {} for _, _, sequence in rows: - stretch = unresolved_end(locus, sequence, gap_band) + stretch = unresolved_end(locus, sequence, gap_band, continuations) if stretch is None: continue trimmed = without_primers(sequence) @@ -4138,7 +4369,7 @@ def trim_locus_rows( readings = { sequence: ( gapped[sequence], ref_ends_at_primer and len(sequence) < uncut, unresolved.get(sequence), - shown.alt_tag if from_alt[sequence] else None, + shown.alt_tag if from_alt[sequence] else None, settled[sequence], ) for sequence in merged } @@ -4371,6 +4602,10 @@ def write_madc( above, then shown without their primers (trim_locus_rows): sequences, consensus, Variants and MarkerType are those of the trimmed rows, and rows that become identical are one row. + + What the reads carry after their tags is pooled over the samples first + (pooled_continuations) and handed to every reading of a row, so a row + whose end the tag leaves unresolved can be settled (row_decision). """ output.parent.mkdir(parents=True, exist_ok=True) observed: Dict[int, Counter] = defaultdict(Counter) @@ -4396,6 +4631,8 @@ def write_madc( reads_in_emitted_rows = 0 per_sample_emitted = [0] * len(results) unresolved_ends = {"rows": 0, "reads": 0, "rows_target_not_read": 0, "reads_target_not_read": 0} + continuations, bases_after_tag = pooled_continuations(results) + read_continuation = {"bases_after_tag": bases_after_tag, "rows_settled": 0, "reads_settled": 0} primer_tally: Counter = Counter() if options.remove_primers: geometry = amplicon_geometry(loci) @@ -4410,6 +4647,7 @@ def write_madc( ] writer.writerow(header_columns + [result.sample.display_name for result in results]) for locus in loci: + after_tag = continuations.get(locus.index) forced = {locus.ref_tag: "Ref", locus.alt_tag: "Alt"} sequences = set(forced) for sequence, count in observed.get(locus.index, {}).items(): @@ -4424,17 +4662,20 @@ def write_madc( excluded[route]["reads"] += count rows = [] for sequence in sequences: - category = forced.get(sequence) or classify_sequence(locus, sequence, gap_band) + category = forced.get(sequence) or classify_sequence(locus, sequence, gap_band, after_tag) rows.append((ALLELE_TYPE_ORDER.get(category, 99), category, sequence)) - stretch = row_decision(locus, sequence, gap_band).unresolved + stretch = row_decision(locus, sequence, gap_band, after_tag).unresolved + reads = observed[locus.index][sequence] if stretch is not None: # Counted on the full tags, so the tally is the same with and without primers. - reads = observed[locus.index][sequence] target_in_it = locus.variant_pos >= stretch[0] unresolved_ends["rows"] += 1 unresolved_ends["reads"] += reads unresolved_ends["rows_target_not_read"] += target_in_it unresolved_ends["reads_target_not_read"] += reads * target_in_it + elif row_decision(locus, sequence, gap_band).unresolved is not None: + read_continuation["rows_settled"] += 1 # its target is unread on the tag alone + read_continuation["reads_settled"] += reads rows.sort(key=lambda row: (row[0], row[2])) counts_by_sequence = { sequence: [result.counts.get((locus.index, sequence), 0) for result in results] for _, _, sequence in rows @@ -4443,7 +4684,7 @@ def write_madc( if options.remove_primers: # From here on ``locus`` is the marker as the table shows it. locus, rows, counts_by_sequence, readings = trim_locus_rows( - locus, rows, counts_by_sequence, primer_sites[locus.marker][0], gap_band, primer_tally + locus, rows, counts_by_sequence, primer_sites[locus.marker][0], gap_band, primer_tally, after_tag ) designed = _describe(locus.ref_tag, locus.alt_tag, gap_band, *readings[locus.alt_tag]) type_serials = Counter() @@ -4478,7 +4719,7 @@ def write_madc( label = variants = "." if options.annotations: if readings is None: - events = variant_events(locus, sequence, gap_band) + events = variant_events(locus, sequence, gap_band, after_tag) else: events = _describe(locus.ref_tag, sequence, gap_band, *readings[sequence]) label = marker_type_label(locus, sequence, gap_band, events, designed) @@ -4542,6 +4783,7 @@ def write_madc( "novel_rows_by_evidence_route": dict(route_counts), "excluded_novel": excluded, "unresolved_ends": unresolved_ends, + "read_continuation": read_continuation, "reads_in_emitted_rows": reads_in_emitted_rows, "per_sample_reads_in_emitted_rows": per_sample_emitted, } diff --git a/tests/test_read_continuation.py b/tests/test_read_continuation.py new file mode 100644 index 0000000..57e41d8 --- /dev/null +++ b/tests/test_read_continuation.py @@ -0,0 +1,355 @@ +"""Tests of read continuation. + +A row whose target lies in an unresolved end has two readings and too few +bases to choose. Reads longer than the tag show what follows it. ``generate`` +keeps the commonest of those bases, pools them over the samples, and decides +such a row again on the longer window. These tests pin that: the row settled +through its gap, the row settled as substitutions, the rows that stay +unresolved, the guard on the added bases, and a cohort whose reads end at the +tag, where nothing changes. +""" +import contextlib +import csv +import gzip +import io +import json +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO)) # the repository's modules, never an installed madcats +sys.path.insert(0, str(Path(__file__).resolve().parent)) # the helpers of test_unresolved + +import madcats as mp # noqa: E402 +from test_unresolved import BAND, LEFT, REF, locus_at, replay # noqa: E402 + +# The amplicon goes on after the 81-base tag. +AFTER = "CATTGACCTGAAGTCCATCGGTAAGCTTGA" +SEEN = 28 # bases of it a read shows + +# The target is the tag's last base, G; the Alt allele is C. +LOCUS = locus_at(80, alt_base="C") +# The tag of a Ref haplotype that lost base 79: its last three bases are those of REF moved up by +# one. It is also the tag of an Alt haplotype with two substitutions before the target. The tag +# cannot tell the two apart; the bases after it can. +LATE = REF[:78] + REF[79:] + AFTER[0] +LOST_A_BASE = AFTER[1 : SEEN + 1] # what follows the tag of the haplotype that lost a base +SUBSTITUTED = AFTER[:SEEN] # what follows the tag of the one with substitutions + + +# A Ref haplotype with three bases inserted after base 76. They push the last three Ref bases, +# the target among them, out of the tag. On the tag alone the cheapest alignment is another one: +# two substitutions and a deletion that the haplotype does not have. +INSERTION = (REF[:76] + "ACG" + REF[76:])[:81] +AFTER_INSERTION = (REF[78:] + AFTER)[:SEEN] + + +def continuations(after_row, after_ref=AFTER[:SEEN], after_alt=AFTER[:SEEN], locus=LOCUS, row=LATE): + """What row_decision is handed for one marker: {tag: the bases its reads carry after it}.""" + found = {locus.ref_tag: after_ref, locus.alt_tag: after_alt, row: after_row} + return {tag: bases for tag, bases in found.items() if bases} + + +def cell(locus, row, found): + return mp.format_variants(mp.variant_events(locus, row, BAND, found)) + + +def substituted(bases, count): + """``bases`` with ``count`` bases changed, spread out.""" + changed = list(bases) + for index in (5, 11, 17, 23)[:count]: + changed[index] = "A" if changed[index] != "A" else "C" + return "".join(changed) + + +class DecisionTests(unittest.TestCase): + def test_the_module_under_test_is_the_repository_s(self): + self.assertEqual(Path(mp.__file__).resolve().parent, REPO) + + def test_the_tag_alone_leaves_the_target_unread(self): + self.assertEqual(mp.row_decision(LOCUS, LATE, BAND), mp.RowDecision(None, (78, 78))) + self.assertEqual(mp.classify_sequence(LOCUS, LATE, BAND), "Other") + self.assertEqual(cell(LOCUS, LATE, None), "unresolved=3") + + def test_a_row_settled_through_its_gap(self): + found = continuations(LOST_A_BASE) + reading, unresolved = mp.row_decision(LOCUS, LATE, BAND, found) + self.assertIsNone(unresolved) + self.assertEqual((reading.gap_bases, reading.end_read, reading.end_ref), (1, 80, 81)) + # Read through the deletion, the target shows the Ref base. + self.assertEqual(mp.classify_sequence(LOCUS, LATE, BAND, found), "RefMatch") + events = mp.variant_events(LOCUS, LATE, BAND, found) + text = mp.format_variants(events) + self.assertEqual(text, "78:TG>T;overhang=1") + self.assertEqual(mp.marker_type_label(LOCUS, LATE, BAND, events), "indel") + self.assertEqual(replay(REF, text, LATE), LATE) + self.assertEqual(mp.parse_variants(text, REF, LATE), events) + + def test_a_row_settled_as_substitutions(self): + found = continuations(SUBSTITUTED) + self.assertEqual(mp.row_decision(LOCUS, LATE, BAND, found), mp.RowDecision(None, None)) + # Read at the fixed offsets, the target shows the Alt base. + self.assertEqual(mp.classify_sequence(LOCUS, LATE, BAND, found), "AltMatch") + events = mp.variant_events(LOCUS, LATE, BAND, found) + self.assertEqual(mp.format_variants(events), "79:G>T;80:T>G;81:G>C") + self.assertEqual(mp.marker_type_label(LOCUS, LATE, BAND, events), "snp") + + def test_a_row_stays_unresolved_without_enough_bases_after_the_tag(self): + alone = mp.row_decision(LOCUS, LATE, BAND) + short = mp.GAP_VERIFIED_SUFFIX - 1 + cases = { + "no continuation of the row": continuations(""), + "no continuation of the Ref tag": continuations(LOST_A_BASE, after_ref=""), + "the row's is too short": continuations(LOST_A_BASE[:short]), + "the Ref tag's is too short": continuations(LOST_A_BASE, after_ref=AFTER[:short]), + "none at all": {}, + } + for name, found in cases.items(): + with self.subTest(name): + self.assertEqual(mp.row_decision(LOCUS, LATE, BAND, found), alone) + self.assertEqual(mp.classify_sequence(LOCUS, LATE, BAND, found), "Other") + self.assertEqual(cell(LOCUS, LATE, found), "unresolved=3") + + def test_the_shortest_window_that_settles_adds_twelve_bases(self): + enough = mp.GAP_VERIFIED_SUFFIX + found = continuations(LOST_A_BASE[:enough], after_ref=AFTER[:enough], after_alt=AFTER[:enough]) + self.assertEqual(mp.classify_sequence(LOCUS, LATE, BAND, found), "RefMatch") + + def test_the_alt_tag_borrows_the_ref_tag_s_continuation(self): + found = continuations(LOST_A_BASE, after_alt="") + self.assertEqual(mp.classify_sequence(LOCUS, LATE, BAND, found), "RefMatch") + + def test_the_added_bases_must_be_read_cleanly(self): + # Two substitutions in the bases after the tag are sequence variation; three are a row + # that does not follow this amplicon, and the longer window then settles nothing. + allowed = mp.CONTINUATION_MAX_MISMATCHES + for after_row, settled in ((LOST_A_BASE, "RefMatch"), (SUBSTITUTED, "AltMatch")): + with self.subTest(settled): + found = continuations(substituted(after_row, allowed)) + self.assertEqual(mp.classify_sequence(LOCUS, LATE, BAND, found), settled) + found = continuations(substituted(after_row, allowed + 1)) + self.assertEqual(mp.row_decision(LOCUS, LATE, BAND, found), mp.row_decision(LOCUS, LATE, BAND)) + self.assertEqual(cell(LOCUS, LATE, found), "unresolved=3") + + def test_a_target_pushed_past_the_tag_stays_other(self): + # Three inserted bases push the last three Ref bases out of the tag. The longer window + # reads the insertion and finds the target after the tag. The row does not hold it. + row = (REF[:74] + "ACG" + REF[74:])[:81] + after_row = (REF[78:] + AFTER)[:SEEN] + pushed_out = locus_at(79) + found = continuations(after_row, locus=pushed_out, row=row) + self.assertEqual(mp.row_decision(pushed_out, row, BAND).unresolved, (75, 75)) + self.assertIsNone(mp.row_decision(pushed_out, row, BAND, found).unresolved) + self.assertEqual(mp.classify_sequence(pushed_out, row, BAND, found), "Other") + self.assertEqual(cell(pushed_out, row, found), "74:T>TACG;clip=3") + self.assertEqual(replay(REF, "74:T>TACG;clip=3", row), row) + # A target the insertion left in the tag is read from the same row. + kept_in = locus_at(76, alt_base="T") + found = continuations(after_row, locus=kept_in, row=row) + self.assertEqual(mp.classify_sequence(kept_in, row, BAND), "Other") + self.assertEqual(mp.classify_sequence(kept_in, row, BAND, found), "RefMatch") + + def test_a_settled_row_is_written_from_the_longer_window_s_alignment(self): + found = continuations(AFTER_INSERTION, row=INSERTION) + self.assertEqual(cell(LOCUS, INSERTION, None), "77:G>A;78:T>C;unresolved=2") + # The alignment the tag alone would be given, were its gap believed: + self.assertEqual(mp.format_variants(mp._describe(REF, INSERTION, BAND, True)), "77:G>A;78:T>C;79:GT>G;overhang=1") + # The longer window reads the insertion. The target is past the tag, so the row stays Other. + self.assertEqual(cell(LOCUS, INSERTION, found), "76:A>AACG;clip=3") + self.assertEqual(replay(REF, "76:A>AACG;clip=3", INSERTION), INSERTION) + self.assertEqual(mp.classify_sequence(LOCUS, INSERTION, BAND, found), "Other") + + def test_a_row_whose_target_the_tag_reads_is_never_asked(self): + before = locus_at(40) + found = continuations(LOST_A_BASE, locus=before) + self.assertEqual(mp.row_decision(before, LATE, BAND, found), mp.row_decision(before, LATE, BAND)) + self.assertEqual(mp.classify_sequence(before, LATE, BAND, found), "RefMatch") + self.assertEqual(cell(before, LATE, found), "unresolved=3") + + def test_canonical_rows_and_rescue_off_are_never_asked(self): + found = continuations(LOST_A_BASE) + for row in (LOCUS.ref_tag, LOCUS.alt_tag): + self.assertEqual(mp.row_decision(LOCUS, row, BAND, found), mp.RowDecision(None, None)) + self.assertEqual(mp.row_decision(LOCUS, LATE, 0, found), mp.RowDecision(None, None)) + + +class PoolingTests(unittest.TestCase): + def pooled(self, *samples): + results = [mp.SampleResult(mp.Sample("s", "s", "unused.fastq", {}), {}, {}, {}, {}, {}, continuations=kept) for kept in samples] + return mp.pooled_continuations(results) + + def test_samples_add_up_and_the_commonest_is_taken(self): + key = (0, LATE) + found, length = self.pooled({key: [(LOST_A_BASE, 2), (SUBSTITUTED, 1)]}, {key: [(LOST_A_BASE, 5)]}) + self.assertEqual((found, length), ({0: {LATE: LOST_A_BASE}}, SEEN)) + + def test_continuations_are_cut_to_the_length_most_reads_reach(self): + # Nine reads in ten show 16 bases, one shows 28: all are compared over 16. + key = (0, LATE) + found, length = self.pooled({key: [(LOST_A_BASE[:16], 9), (LOST_A_BASE, 1)]}) + self.assertEqual((found, length), ({0: {LATE: LOST_A_BASE[:16]}}, 16)) + # Reads that stop short of that length are left out. + found, length = self.pooled({key: [(LOST_A_BASE, 9), (SUBSTITUTED[:16], 1)]}) + self.assertEqual((found, length), ({0: {LATE: LOST_A_BASE}}, SEEN)) + + def test_a_tag_followed_by_two_sequences_has_no_continuation(self): + key = (0, LATE) + self.assertEqual(self.pooled({key: [(LOST_A_BASE, 5), (SUBSTITUTED, 5)]})[0], {}) + self.assertEqual(self.pooled({key: [(LOST_A_BASE, 6), (SUBSTITUTED, 4)]})[0], {0: {LATE: LOST_A_BASE}}) + + def test_a_continuation_needs_three_reads(self): + key = (0, LATE) + self.assertEqual(self.pooled({key: [(LOST_A_BASE, 2)]})[0], {}) + self.assertEqual(self.pooled({key: [(LOST_A_BASE, 3)]})[0], {0: {LATE: LOST_A_BASE}}) + + def test_no_continuation_anywhere(self): + self.assertEqual(self.pooled({}, {}), ({}, 0)) + + +class GenerateTests(unittest.TestCase): + """``generate`` from FASTQ files: one marker, two samples, reads of the Ref, Alt and LATE tags.""" + + PLAIN = REF[:50] + "A" + REF[51:] # a Ref haplotype with one substitution: the tag decides it + + def write_reads(self, directory, name, after_tag, after_late): + """Two samples' reads, each tag followed by ``after_tag`` bases; LATE by those of ``after_late``.""" + reads = Path(directory) / f"reads_{name}" + reads.mkdir() + rows = [(LOCUS.ref_tag, AFTER, 40), (LOCUS.alt_tag, AFTER, 30), (self.PLAIN, AFTER, 12), (LATE, after_late, 12)] + for sample in ("s1", "s2"): + with gzip.open(reads / f"{sample}.fastq.gz", "wt") as handle: + number = 0 + for tag, following, copies in rows: + sequence = tag + following[:after_tag] + for _ in range(copies): + number += 1 + handle.write(f"@{sample}_{number}\n{sequence}\n+\n{'I' * len(sequence)}\n") + return reads + + def generate(self, directory, reads, cache=None): + """Run generate on ``reads``: (table path, {sequence: row}, the QC file's output summary).""" + panel = Path(directory) / "tags.csv" + mp.write_reference_tags(panel, [LOCUS]) + output = Path(directory) / f"{reads.name}.csv" + arguments = [ + "generate", "--fastq-dir", str(reads), "--reference-tags", str(panel), "--tag-length", "81", + "--jobs", "1", "--output", str(output), + ] + if cache is not None: + arguments += ["--cache-dir", str(cache)] + with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()): + self.assertEqual(mp.main(arguments), 0) + with open(output, newline="") as handle: + table = {row["AlleleSequence"]: row for row in csv.DictReader(handle)} + with open(str(output) + ".qc.json") as handle: + summary = json.load(handle)["output_summary"] + return output, table, summary + + def run_generate(self, directory, name, after_tag, after_late): + return self.generate(directory, self.write_reads(directory, name, after_tag, after_late)) + + def test_reads_that_end_at_the_tag_change_nothing(self): + # Reads that stop at the tag, or two bases after it, give the same table: the row's + # target stays unread and nothing is kept from after the tag. + with tempfile.TemporaryDirectory() as directory: + at_tag, table, summary = self.run_generate(directory, "at_tag", 0, LOST_A_BASE) + two_more, _, summary_two = self.run_generate(directory, "two_more", 2, LOST_A_BASE) + self.assertEqual(at_tag.read_bytes(), two_more.read_bytes()) + for found in (summary, summary_two): + self.assertEqual(found["read_continuation"], {"bases_after_tag": 0, "rows_settled": 0, "reads_settled": 0}) + self.assertEqual(found["unresolved_ends"]["rows_target_not_read"], 1) + self.assertEqual((table[LATE]["AlleleID"], table[LATE]["Variants"]), ("m1|Other", "unresolved=3")) + + def test_longer_reads_settle_the_row_and_only_that_row(self): + with tempfile.TemporaryDirectory() as directory: + _, before, _ = self.run_generate(directory, "at_tag", 0, LOST_A_BASE) + cache = Path(directory) / "cache" + reads = self.write_reads(directory, "longer", SEEN, LOST_A_BASE) + output, table, summary = self.generate(directory, reads, cache) + self.assertEqual( + (table[LATE]["AlleleID"], table[LATE]["Variants"], table[LATE]["MarkerType"]), + ("m1|RefMatch", "78:TG>T;overhang=1", "indel"), + ) + self.assertEqual(summary["read_continuation"], {"bases_after_tag": SEEN, "rows_settled": 1, "reads_settled": 24}) + self.assertEqual(summary["unresolved_ends"]["rows"], 0) + self.assertEqual(set(table), set(before)) + for sequence in set(table) - {LATE}: + self.assertEqual(table[sequence], before[sequence]) + self.assertEqual((table[LATE]["s1"], table[LATE]["s2"]), (before[LATE]["s1"], before[LATE]["s2"])) + # The read pass kept what follows the two canonical tags and the unread row, each from + # its third read on, and nothing of the row the tag decides. + with gzip.open(sorted(cache.glob("s1.*.json.gz"))[0], "rt") as handle: + kept = {tag: found for _, tag, _, *found in json.load(handle)["counts"]} + self.assertEqual( + kept, + {LOCUS.ref_tag: [[SUBSTITUTED, 38]], LOCUS.alt_tag: [[SUBSTITUTED, 28]], LATE: [[LOST_A_BASE, 10]], self.PLAIN: []}, + ) + # A second run takes them from the cache and writes the same table. + first = output.read_bytes() + with mock.patch.object(mp, "_write_cache", side_effect=AssertionError("the cache was not used")): + again, _, _ = self.generate(directory, reads, cache) + self.assertEqual(again.read_bytes(), first) + + def test_the_same_tag_settles_the_other_way_when_its_reads_say_so(self): + with tempfile.TemporaryDirectory() as directory: + _, table, _ = self.run_generate(directory, "substituted", SEEN, SUBSTITUTED) + self.assertEqual( + (table[LATE]["AlleleID"], table[LATE]["Variants"], table[LATE]["MarkerType"]), + ("m1|AltMatch", "79:G>T;80:T>G;81:G>C", "snp"), + ) + + def test_table_without_primers_carries_the_settled_reading(self): + # The rows without their primers are not aligned again: each settled row is written from + # the alignment it was settled with, in the positions of the Ref row as printed. + with tempfile.TemporaryDirectory() as directory: + results = [] + for sample in ("s1", "s2"): + counts = {(0, LOCUS.ref_tag): 40, (0, LOCUS.alt_tag): 30, (0, LATE): 12, (0, INSERTION): 12} + kept = { + (0, LOCUS.ref_tag): [(SUBSTITUTED, 40)], (0, LATE): [(LOST_A_BASE, 12)], (0, INSERTION): [(AFTER_INSERTION, 12)], + } + results.append(mp.SampleResult(mp.Sample(sample, sample, "unused.fastq", {}), counts, {}, {}, {}, {}, continuations=kept)) + output = Path(directory) / "madc.csv" + mp.write_madc( + output, [LOCUS], results, mp.EvidenceRule(1, 20, 0, 0, 0, 5, 0.0), mp.OutputOptions(remove_primers=True), BAND, + ) + with open(output, newline="") as handle: + table = {row["AlleleSequence"]: row for row in csv.DictReader(handle)} + row = table[LATE[LEFT:]] + self.assertEqual((row["AlleleID"], row["Variants"], row["MarkerType"]), ("m1|RefMatch", "58:TG>T;overhang=1", "indel")) + self.assertEqual(replay(REF[LEFT:], row["Variants"], LATE[LEFT:]), LATE[LEFT:]) + row = table[INSERTION[LEFT:]] + self.assertEqual((row["AlleleID"], row["Variants"], row["MarkerType"]), ("m1|Other", "56:A>AACG;clip=3", "indel")) + self.assertEqual(replay(REF[LEFT:], row["Variants"], INSERTION[LEFT:]), INSERTION[LEFT:]) + + def test_a_settled_row_with_a_gap_in_its_left_primer_is_described_from_the_bases_shown(self): + # Two bases inserted in the left primer shift the row against the Ref row from there on, so + # the two rows as printed start at different bases of the alignment. It is not carried + # then: the printed rows are aligned as they stand, and the cell still rebuilds the row. + whole = REF[:8] + "GG" + REF[8:75] + "ACG" + REF[75:] + AFTER + row, after_row = whole[:81], whole[81 : 81 + SEEN] + found = continuations(after_row, row=row) + self.assertEqual(cell(LOCUS, row, None), "unresolved=73") # the two readings part at the first inserted base + self.assertEqual(cell(LOCUS, row, found), "8:A>AGG;75:A>AACG;clip=5") + with tempfile.TemporaryDirectory() as directory: + results = [] + for sample in ("s1", "s2"): + counts = {(0, LOCUS.ref_tag): 40, (0, LOCUS.alt_tag): 30, (0, row): 12} + kept = {(0, LOCUS.ref_tag): [(SUBSTITUTED, 40)], (0, row): [(after_row, 12)]} + results.append(mp.SampleResult(mp.Sample(sample, sample, "unused.fastq", {}), counts, {}, {}, {}, {}, continuations=kept)) + output = Path(directory) / "madc.csv" + mp.write_madc( + output, [LOCUS], results, mp.EvidenceRule(1, 20, 0, 0, 0, 5, 0.0), mp.OutputOptions(remove_primers=True), BAND, + ) + with open(output, newline="") as handle: + table = {entry["AlleleSequence"]: entry for entry in csv.DictReader(handle)} + shown = table[row[LEFT:]] + self.assertEqual((shown["AlleleID"], shown["Variants"]), ("m1|Other", "1:.>AC;56:A>AC;58:T>A;clip=3")) + self.assertEqual(replay(REF[LEFT:], shown["Variants"], row[LEFT:]), row[LEFT:]) + + +if __name__ == "__main__": + unittest.main() From 224eeebf45f93ad2d583d6107b8104a8b810147d Mon Sep 17 00:00:00 2001 From: Alexander Sandercock <39815775+alex-sandercock@users.noreply.github.com> Date: Fri, 20 Nov 2026 18:24:04 -0500 Subject: [PATCH 2/2] Docs: a tag's continuation is counted from its third read in a sample on The classification page said a continuation counts when 3 reads and 60% of a tag's reads carry it. The read pass keeps continuations from a tag's third read in a sample on, so a sample's first two reads of a tag are in neither count. The page now says so. No code changes. Under the default evidence rule a new row has 10 reads in each of two samples or 30 in one, so the reads left out never decide whether it has a continuation. --- docs/src/reference/evidence-model.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/src/reference/evidence-model.md b/docs/src/reference/evidence-model.md index c742894..2082478 100644 --- a/docs/src/reference/evidence-model.md +++ b/docs/src/reference/evidence-model.md @@ -156,8 +156,10 @@ same decision on more bases, with three outcomes: - the substitutions stand, and the target is read at the fixed offset; - the longer window settles nothing, and the row stays unresolved. -A tag's continuation counts when 3 reads and 60% of its reads carry it. The window must grow by at -least 12 bases, and it must read the added bases with at most 2 substitutions. +The read pass keeps these bases from a tag's third read in a sample on, so a sample's first two +reads of a tag are not counted. A tag's continuation counts when 3 of the kept reads and 60% of +them carry it. The window must grow by at least 12 bases, and it must read the added bases with at +most 2 substitutions. A settled row is written like any other row. Its `Variants` list the gap or the substitutions as far as the row's own bases go. A target that an insertion pushed past the tag is not in the row,