diff --git a/docs/src/commands/generate.md b/docs/src/commands/generate.md index dabeb2a..58cc28e 100644 --- a/docs/src/commands/generate.md +++ b/docs/src/commands/generate.md @@ -64,7 +64,7 @@ The [Evidence Model](../reference/evidence-model.md) explains stages 2–6 in de | `--leading-anchor ` | none | Constant sequence before the amplicon in raw deliveries (for DArT, `GATGGATCCAGTGC`). MADCATS cuts each read where the anchor ends, and the anchor must end within 40 bases beyond its own length. `auto` detects it from a sample of up to 24 FASTQs. With one or two FASTQs the detected anchor may include the sample barcode, and the run warns about it | | `--leading-anchor-mismatches {0,1}` | 1 | Substitutions allowed in the leading anchor. With `1`, MADCATS cuts a read that has no exact copy after its only one-substitution match, and counts it as `reads_leading_anchor_mismatch`. A read with no such match, or more than one, counts as `reads_no_leading_anchor`. Anchors under 12 bases must match exactly. `0` requires an exact match for every anchor, as in 0.13.0 | | `--primer-mismatches {0,1}` | 1 | Mismatches allowed while assigning a primer/anchor. With `0`, reads from Match haplotypes whose distinguishing SNP falls inside the anchor are lost | -| `--gapped-rescue {on,off}` | `on` | Rescue indel reads and allow accepted gapped alignments to locate the target allele and annotate variants. With `off`, retained indel rows can have incorrect target categories; see [Classification](../reference/evidence-model.md#classification) | +| `--gapped-rescue {on,off}` | `on` | Rescue indel reads and allow accepted gapped alignments to locate the target allele and annotate variants. A row whose end has a cheaper but unverified gap is read up to it and marked `unresolved`. With `off`, retained indel rows can have incorrect target categories; see [Classification](../reference/evidence-model.md#classification) | | `--max-indel-length ` | 10 | Widest gap (alignment band) a gapped alignment may use | | `--tile-groups {auto,declared,off}` | `auto` | Link tiled markers (markers that read the same tag window of one amplicon), so a read is credited to every tile it fits instead of being dropped as a tie. `auto` links tiles declared with `LocusGroup` and detects undeclared ones, `declared` links only the declared ones, and `off` links none. See [Tiled Markers](../getting-started/input-formats.md#tiled-markers) | | `--partial-read-mode {on,off}` | `on` | Add a read shorter than the tag to the one full-length allele (already passing the evidence rules) whose start it matches, only when it is longer than every prefix shared between that locus's alleles. `off` drops such reads | @@ -126,11 +126,14 @@ classified on the full tags, as with `off`, and a row keeps the category it was read is gained or lost. The primer-removal option alone leaves the table unchanged when set to `off`. This is not a -blanket guarantee of v0.13 output: tile-group crediting was added after v0.13 and defaults to -`--tile-groups auto`, and a leading anchor now matches with one substitution by default. For -comparisons with v0.13, use the same panel, reads and analysis settings, plus -`--tile-groups off --remove-primers off`, and `--leading-anchor-mismatches 0` where reads are cut -at a leading anchor. Version/provenance metadata still differs. +blanket guarantee of v0.13 output. Tile-group crediting was added after v0.13 and defaults to +`--tile-groups auto`, and a leading anchor now matches with one substitution by default; +`--tile-groups off --remove-primers off`, with `--leading-anchor-mismatches 0` where reads are cut +at a leading anchor, restores the v0.13 assignment. And a row whose end has a cheaper but +unverified gap is now marked `unresolved`, where v0.13 read it at the fixed offsets (see +[Classification](../reference/evidence-model.md#classification)): about 2 to 3 rows in 100 on the +cohorts measured, of which the ones with the target in that end become `Other`. No option restores +the old reading of those rows. Version/provenance metadata differs too. What to know before using it: diff --git a/docs/src/development/architecture.md b/docs/src/development/architecture.md index a1f0de4..f31dae8 100644 --- a/docs/src/development/architecture.md +++ b/docs/src/development/architecture.md @@ -29,7 +29,7 @@ The module follows the order a read travels. Each section opens with a banner co |---------|----------| | 0. Constants, errors and data types | `Locus`, `Sample`, `SampleResult`, MADC column definitions | | 1. Sequence and file utilities | IUPAC matching, mismatch counts, CSV/FASTQ helpers | -| 2. Alignment engine | Myers bit-vector prescreen, banded DP, `accepted_gapped_reading` / `row_reading` | +| 2. Alignment engine | Myers bit-vector prescreen, banded DP, `accepted_gapped_reading` / `row_decision` | | 3. Panel definition | Design-file reconstruction (`build_panel`), reference-tag tables, primer-free inference | | 4. Platform intake | Pair merging, window location and freezing, tiling and routing, window-scan | | 5. Per-sample read assignment | `PrimerIndex`, `select_marker`, `process_sample` and its cache | diff --git a/docs/src/development/changelog.md b/docs/src/development/changelog.md index a61ebf8..d938958 100644 --- a/docs/src/development/changelog.md +++ b/docs/src/development/changelog.md @@ -13,6 +13,39 @@ Notable changes by version. Measurements behind each change are in the the left primer. Reads are still assigned, counted, denoised and classified on the full tags, so a row keeps its category. The QC JSON gains `primer_removal`. The default is `off`, and the table written with `off` is unchanged. +- `generate` leaves the end of a row unresolved when a gapped alignment explains the row with + fewer edits than substitutions but has fewer than 12 bases after its last gap. Such a row was + read at the fixed offsets: its end was written as substitutions, and an insertion or deletion + before the target could put the other allele's base on the target. It is now read up to the + first base at which the two explanations differ. A target at or after that base is `Other`. + `Variants` closes with the new token `unresolved=k` (the last `k` Ref bases are not read in this + row), `MarkerType` has the new value `unresolved`, the sidecar the new `Type` `unresolved`, and + the QC JSON counts the rows under `unresolved_ends`. `--remove-primers on` carries the decision + into the rows as printed. This changes the default table. On pecan, sweetpotato, alfalfa and + blackberry 2.2% to 3.2% of rows get the token (0.7% to 2.0% of reads), and 0.12% to 0.22% of + reads move from `RefMatch` or `AltMatch` to `Other`; none moves to the other allele. On three + sets of simulated haplotypes the rows counted for the wrong target allele fall from 1.2% to + 0.13% and the substitutions written that no haplotype carries from 108,592 to 8,312, while 1.0% + fewer rows are counted for the right allele. Rows without the token are written as before, + except a few whose counts change because the pooled denoiser groups rows by category. +- `generate` reads a row from the Alt tag when the Ref tag and the Alt tag explain it through gaps + in the same number of edits and the Alt tag needs fewer separate gaps. Before, the Ref tag's + alignment was used on every such tie. An Alt haplotype with one insertion or deletion beside its + target was then counted as `RefMatch` (the Ref tag with two gaps around the target) or as `Other` + (the target inside a deletion). Such rows are now `AltMatch`. Rows whose two alignments are equal + in edits and in gaps still use the Ref tag's. On pecan, sweetpotato, alfalfa and blackberry 0.03% + to 0.10% of rows change category (0.002% to 0.04% of reads), all of them to `AltMatch`. +- `Variants` of a row whose category was read from the Alt tag is written from that same alignment, + in the Ref tag's coordinates. Before, it was always the row's own alignment to the Ref tag. That + alignment has no reason to put the Alt base on the target: it could split one gap in two, or put + the target inside a deletion, and `Variants` then showed the target unchanged or deleted on an + `AltMatch` row. An insertion or deletion right after a substituted base is written without an + anchor base (`31:G>A;32:CACG>.`), as one on the first base of a tag already was, because the + substitution uses that base; `call` reads this form. `--remove-primers on` writes the rows from + the same reading. On pecan, sweetpotato, alfalfa and blackberry 0.07% to 0.25% of rows get other + `Variants` (28 to 75 rows), and with primers kept no row is left whose `Variants` and category + put different bases on the target. Without primers 16 to 31 rows per panel still do: rows cut at + their right primer to another length than the Ref row. Categories and counts do not change. - `Variants` writes an insertion or deletion on the very first base of a tag without an anchor base (`1:.>GG`, `1:TT>.`). Before, it borrowed the base after it, which another event could use as well, and the row then could not be rebuilt from its `Variants`. A tag that starts with a diff --git a/docs/src/reference/evidence-model.md b/docs/src/reference/evidence-model.md index 089005a..99757c4 100644 --- a/docs/src/reference/evidence-model.md +++ b/docs/src/reference/evidence-model.md @@ -116,18 +116,45 @@ explains the row with fewer edits, stays within the substitution and gap limits, at least 12 bases after its last gap, the target base comes from that alignment instead. A target inside a deletion, or pushed past the end of the read, is `Other`. +The row is aligned to the Ref tag and to the Alt tag, and the alignment with fewer edits is used. +When both need the same number of edits, the one with fewer separate gaps is used. An Alt haplotype +with one insertion beside its target is then read as that, and not as a Ref haplotype with two +insertions around the target. When the two are equal in gaps as well, the Ref tag's is used. + This applies even to a row whose reads passed the substitution-only assignment check. An upstream insertion moves the target farther along the observed sequence; a deletion moves it earlier. The `Variants` annotation uses the same decision about whether to read through gaps, and reports -events in the Ref tag's coordinates, with equivalent indel placements left-normalised. +events in the Ref tag's coordinates, with equivalent indel placements left-normalised. A row read +from the Alt tag is written from that alignment too, so its `Variants` list the designed +substitution and its gaps where the category was read, and the two show the same base on the +target. + +**An unresolved end.** A gapped alignment can explain a row with fewer edits and still have fewer +than 12 bases after its last gap. The row's end then has two explanations, a gap or a run of +substitutions, and too few bases to tell them apart. `generate` takes neither. It reads the row up +to the first base at which the two explanations differ, and no further: + +- a target at or after that base is `Other`, because the two explanations can put different bases + on it; +- a target before it is read as usual; +- the row's `Variants` close with `unresolved=k`, and its `MarkerType` is `unresolved` + (see [Output Formats](output-formats.md)); +- the QC JSON counts such rows and their reads under `output_summary.unresolved_ends`. + +Reading such a row at the fixed offsets, as versions before this outcome did, writes its end as +substitutions that no sample carries, and an insertion or deletion before the target can put the +base of the other allele on the target. Reading it through the unverified gap is right more often +but invents gaps in rows that have none. The row's own continuation past the tag would settle it, +and a tag does not hold it. With `--gapped-rescue off`, both classification and annotation use the fixed offsets. Many indel reads fail assignment, but some still pass the substitution-only identity limit. Those rows can receive the wrong biological category, and an indel can appear as several substitutions. Turning -rescue off is therefore not an indel-safe way to classify target alleles. Even with rescue on, -a gap that fails the acceptance checks falls back to fixed offsets; repeats and gaps near the -tag end can remain unresolved. Agreement between a category and its annotation is not independent -proof of the biological event. +rescue off is therefore not an indel-safe way to classify target alleles, and it writes no +`unresolved` rows. Even with rescue on, a gap that is no cheaper than substitutions or that fails +the substitution and gap limits is read at the fixed offsets, and so is a deletion longer than +`--max-indel-length`. Agreement between a category and its annotation is not independent proof of +the biological event. `--remove-primers on` acts after classification and preserves that decision. It changes the printed sequences and annotation coordinates, not how the target was identified. Identical diff --git a/docs/src/reference/output-formats.md b/docs/src/reference/output-formats.md index 7e017bd..808e7b4 100644 --- a/docs/src/reference/output-formats.md +++ b/docs/src/reference/output-formats.md @@ -45,11 +45,15 @@ column *positions* are unchanged, so positional consumers see the same 16-column | `snp` | Substitutions only (the Alt row of a designed SNP is `snp`) | | `indel` | Gaps only (the Alt row of a designed indel is `indel`) | | `both` | Substitutions and gaps | +| `unresolved` | The row's end is not read (`unresolved=k` in `Variants`), whatever the row carries before it. A gap there is likely but unverified | -To keep SNP-only microhaplotypes, drop `indel` and `both` rows by this label. +To keep SNP-only microhaplotypes, drop `indel`, `both` and `unresolved` rows by this label. **`Variants`** lists the row's events against the Ref tag, `;`-separated, in 1-based coordinates on -the tag as printed: +the tag as printed. They come from the alignment the row's category was read from, so the two show +the same base on the target. One exception: in a table without primers, a row cut at its right +primer to another length than the Ref row is aligned end to end, which can differ from the reading +its category came from. | Token | Meaning | |-------|---------| @@ -58,15 +62,28 @@ the tag as printed: | `70:TAG>T` | Deletion | | `clip=4` | An insertion pushed the last 4 Ref bases out of the window | | `overhang=3` | A deletion pulled 3 downstream bases into the window | +| `unresolved=9` | The last 9 Ref bases are not read in this row. Its end has two explanations, a gap or substitutions, with too few bases after the gap to tell them apart ([Classification](evidence-model.md#classification)). Always the last token | | `1:.>GG`, `1:TT>.` | An insertion before the very first base, or a deletion that starts on it. No base precedes it, so it is written without an anchor base, and `.` stands for no bases. Here `GG` is inserted before base 1, and bases 1 and 2 are deleted | +| `31:G>A;32:CACG>.`, `31:G>A;32:.>TT` | An insertion or deletion right after a substituted base. The substitution uses that base, so the gap is written without an anchor base as well, from its own first base. Here bases 32 to 35 are deleted, or `TT` is inserted before base 32. It appears on a row that carries the Alt base with a gap directly after it | An example value is `55:C>CAAAA;91:A>T;clip=4`. The Ref row is `.`, and the Alt row of a designed SNP is exactly the design (e.g. `46:C>G`). Each `position:REF>ALT` token reads "replace these Ref bases, starting at this position, by these bases". `clip=k` drops the last `k` Ref bases, and `overhang=k` appends the row's own last `k` -bases. No two tokens of a row use the same Ref base, so applying them in order rebuilds the row -from the Ref row. +bases. `unresolved=k` drops the last `k` Ref bases too, and then appends whatever of the row has +not been built yet. No two tokens of a row use the same Ref base, so applying them in order +rebuilds the row from the Ref row. + +On a marker whose own design is an insertion or deletion, every row is written through gaps +against the Ref row, and the unread end is counted on the Ref row as everywhere else. The designed +insertion or deletion does not settle another gap near the row's end. + +A row with `unresolved=k` says nothing about Ref positions from `length - k + 1` on: no +substitution, no insertion or deletion, and no agreement with the Ref row either. A consumer that +counts reads per position should leave the row's reads out there, as it does for `clip`. +A reader that does not know a token should not guess the row from its sequence: the token states +something about the row that the sequence alone does not show. With `--remove-primers on` the frame is the Ref row as printed, so position 1 is the first base after the left primer. Whether a row is read through an insertion or deletion is decided on its @@ -77,6 +94,11 @@ Every other row is a fixed-length window and keeps `clip` and `overhang`. A row the Ref row in length by more than `--max-indel-length` is compared base by base from its start, and the rest is `clip` or `overhang`. +An unresolved end is decided on the full tag as well, and carried: the row keeps `unresolved=k`, +counted on the Ref row as printed, whatever length it was cut to. A right primer found at another +place shows that the row gained or lost bases, not which reading of its end was right. Only an +unread stretch that lay wholly in the primer cut from the Ref row leaves nothing to mark. + ## Variants Sidecar `OUTPUT.variants.csv` (with `--annotations on`) lists the same events one per line, keyed by @@ -87,7 +109,7 @@ and the rest is `clip` or `overhang`. | `CloneID`, `AlleleID`, `AlleleSequence` | Row identity | | `Position` | 1-based position on the Ref tag | | `Ref`, `Alt` | Event alleles | -| `Type` | `snp`, `ins`, `del`, `clip` (Ref holds the unobserved bases) or `overhang` (Alt holds the pulled-in bases) | +| `Type` | `snp`, `ins`, `del`, `clip` (Ref holds the unobserved bases), `overhang` (Alt holds the pulled-in bases) or `unresolved` (Position is the first Ref base not read, Ref the Ref bases from it on, Alt the row's bases that stand there) | ## QC / Provenance JSON @@ -99,6 +121,9 @@ and the rest is `clip` or `overhang`. - the tile families (`tile_families`) and the extra credits they add (`tile_extra_credits_*`, described in [Read Accounting](evidence-model.md#read-accounting)) - row counts by category, by evidence route, and by `marker_types` +- `unresolved_ends`: the rows whose end is not read and their reads, and how many of them have the + target in that end (those rows are `Other`). Counted on the full tags, so it is the same with and + without `--remove-primers on` and with `--annotations off` - the variants sidecar path - with `--remove-primers on`, `primer_removal`: the panel's amplicon length, the markers by whether their right primer was located (and if not, why), the rows that were merged, the rows that lost diff --git a/madcats.py b/madcats.py index 285c6c1..ffc01aa 100644 --- a/madcats.py +++ b/madcats.py @@ -122,8 +122,9 @@ 1. Sequence and file utilities (IUPAC matching, mismatch counts, CSV/FASTQ helpers). 2. Alignment engine: the Myers prescreen, the banded DP, and - ``accepted_gapped_reading``/``row_reading`` — the one gapped reading a - row may be given. + ``accepted_gapped_reading``/``row_decision`` — the one gapped reading a + row may be given, and where a row stops being read because its end has + two explanations. 3. Panel definition: design-file reconstruction, reference-tag tables, and primer-free inference. 4. Platform intake: pair merging, window location and freezing, tiling and @@ -308,6 +309,7 @@ class GappedReading(NamedTuple): counts describe it; ``last_gap_end`` is the read index just after the last gap; ``ref_to_read`` maps aligned reference indices to read indices (reference bases inside a deletion, or past the aligned end, are absent). + ``template`` is the canonical tag the reading was made against. """ edits: int @@ -319,6 +321,20 @@ class GappedReading(NamedTuple): end_ref: int last_gap_end: int ref_to_read: Dict[int, int] + template: str = "" + + +class RowDecision(NamedTuple): + """How a row is read against its marker (row_decision). + + ``reading`` is the accepted gapped reading, or None when the row is read + at the fixed offsets. ``unresolved`` is None when the whole row is + read, or ``(ref_index, row_index)``: the Ref-tag position and the base + of the row from which, to their ends, the row is not read at all. + """ + + reading: Optional[GappedReading] + unresolved: Optional[Tuple[int, int]] @dataclasses.dataclass(frozen=True) @@ -796,10 +812,27 @@ def accepted_gapped_reading( substitution-only reading (``substitutions``, the caller's bound; by default the mismatch count against ``template``), well composed (``gapped_explanation_ok``) and verified by at least GAP_VERIFIED_SUFFIX - bases after its last gap. A reading that fails any test is not a - reading of the row at all: the row is then classified at the fixed - offset and described as substitutions, by both consumers alike. + bases after its last gap. A reading that is no cheaper or badly + composed is not a reading of the row at all: the row is then classified + at the fixed offset and described as substitutions, by both consumers + alike. One that only lacks the verifying bases is not taken either, + but it leaves the row's end unresolved (row_decision). """ + reading = _unverified_gapped_reading(sequence, template, gap_band, substitutions) + return reading if reading is not None and _gap_verified(reading, sequence) else None + + +def _gap_verified(reading: GappedReading, sequence: str) -> bool: + """At least GAP_VERIFIED_SUFFIX bases of the row follow the reading's last gap.""" + return reading.last_gap_end <= len(sequence) - GAP_VERIFIED_SUFFIX + + +def _unverified_gapped_reading( + sequence: str, template: str, gap_band: int, substitutions: Optional[int] = None +) -> Optional[GappedReading]: + """accepted_gapped_reading before its last test: the fewest-openings + alignment when it is strictly cheaper than substitutions and well + composed, whether or not enough bases follow its last gap to verify it.""" if gap_band <= 0 or len(template) != len(sequence): return None if substitutions is None: @@ -824,10 +857,7 @@ def accepted_gapped_reading( previous = kind if not gapped_explanation_ok(guard_subs, guard_bases, guard_events): return None - reading = _reading_from_ops(sequence, template, ops, end_read, end_ref, edits) - if reading.last_gap_end > len(sequence) - GAP_VERIFIED_SUFFIX: - return None - return reading + return _reading_from_ops(sequence, template, ops, end_read, end_ref, edits) def _reading_from_ops(sequence: str, template: str, ops, end_read: int, end_ref: int, edits: int) -> GappedReading: @@ -845,25 +875,127 @@ def _reading_from_ops(sequence: str, template: str, ops, end_read: int, end_ref: gap_events += previous != kind last_gap_end = max(last_gap_end, i + 1 if kind == "I" else i) previous = kind - return GappedReading(edits, substitutions, gap_bases, gap_events, ops, end_read, end_ref, last_gap_end, ref_to_read) + return GappedReading(edits, substitutions, gap_bases, gap_events, ops, end_read, end_ref, last_gap_end, ref_to_read, template) + + +def _reading_cost(reading: GappedReading) -> Tuple[int, int]: + """How the two canonicals' readings of one row are ranked: fewest edits, + then fewest gap openings. + + A row can fit the Ref tag and the Alt tag in the same number of edits: + six inserted bases as one insertion beside the Alt base, or split in two + around a Ref base. The reading with fewer separate gaps is the one + taken, as gapped_alignment already takes it among the alignments to one + tag. Readings equal in both leave the Ref tag's standing (it is tried + first).""" + return reading.edits, reading.gap_events def row_reading(locus: Locus, sequence: str, gap_band: int) -> Optional[GappedReading]: """The locus-level decision both consumers share: the accepted gapped reading - of ``sequence`` against whichever canonical explains it most cheaply, or - ``None``. The bound a gapped reading must beat is the cheaper of the two + of ``sequence`` against whichever canonical explains it most cheaply + (_reading_cost), or ``None``. The bound a gapped reading must beat is the cheaper of the two substitution-only readings, and the composition guard is applied against the canonical the row is read from, so a designed substitution never counts against an AltMatch row's indel reading.""" - if gap_band <= 0: - return None - substitutions = min(tag_mismatches(sequence, locus.ref_tag), tag_mismatches(sequence, locus.alt_tag)) - best: Optional[GappedReading] = None - for template in (locus.ref_tag, locus.alt_tag): - accepted = accepted_gapped_reading(sequence, template, gap_band, substitutions) - if accepted is not None and (best is None or accepted.edits < best.edits): - best = accepted - return best + return row_decision(locus, sequence, gap_band).reading + + +def row_decision(locus: Locus, sequence: str, gap_band: int) -> RowDecision: + """How a row is read: its accepted gapped reading, and where it stops being read. + + Over both canonicals there is the cheapest gapped reading that is + verified (row_reading: GAP_VERIFIED_SUFFIX bases follow its last gap) + and the cheapest one whether verified or not. Cheapest is fewest + edits, then fewest gap openings (_reading_cost). Where the two are the + same reading, or no gapped reading is cheaper than substitutions, the + whole row is read. Where they differ, the row's end has two + explanations and too few bases to tell them apart: a gap there, or the + reading without it. The row is then read up to the first Ref-tag + position at which the two explanations show a different base of the + row, and from there on it is unresolved. Neither explanation is taken + for that stretch: a target in it classifies as Other, and the row's + Variants close with ``unresolved`` in place of substitutions or a gap. + + The first such position is found with each gap at the rightmost of its + equal placements (_right_align), so every base both explanations agree + on stays read. The decision is made once here and shared by + classify_sequence, variant_events and the primer removal. + """ + if gap_band <= 0 or sequence in (locus.ref_tag, locus.alt_tag): + return RowDecision(None, None) + return _row_decision(locus.ref_tag, locus.alt_tag, sequence, gap_band, GAP_VERIFIED_SUFFIX) + + +@functools.lru_cache(maxsize=65536) +def _row_decision(ref_tag: str, alt_tag: str, sequence: str, gap_band: int, verified_suffix: int) -> RowDecision: + """row_decision, cached per row (classification and annotation both ask).""" + substitutions = min(tag_mismatches(sequence, ref_tag), tag_mismatches(sequence, alt_tag)) + verified = cheapest = None + verified_template = cheapest_template = "" + for template in (ref_tag, alt_tag): + reading = _unverified_gapped_reading(sequence, template, gap_band, substitutions) + if reading is None: + continue + cost = _reading_cost(reading) + if cheapest is None or cost < _reading_cost(cheapest): + cheapest, cheapest_template = reading, template + if reading.last_gap_end <= len(sequence) - verified_suffix and (verified is None or cost < _reading_cost(verified)): + verified, verified_template = reading, template + if cheapest is None or (verified is not None and verified.ops == cheapest.ops and verified_template == cheapest_template): + return RowDecision(verified, None) + # Two explanations of the row's end. Which base of the row stands at each + # Ref-tag position under each, gaps at their rightmost placement: + shown = {j: i for kind, i, j in _right_align(cheapest.ops, sequence, cheapest_template) if kind == "M"} + kept_ops = None if verified is None else _right_align(verified.ops, sequence, verified_template) + kept = None if kept_ops is None else {j: i for kind, i, j in kept_ops if kind == "M"} + first = next( + (j for j in range(len(sequence)) if (j if kept is None else kept.get(j)) != shown.get(j)), None + ) + if first is None: + return RowDecision(verified, None) + # The row bases the agreed part uses: an insertion placed right before the + # first disputed position belongs to the dispute. + start = first if kept_ops is None else sum(1 for kind, _, j in kept_ops if kind != "D" and j < first) + return RowDecision(verified, (first, start)) + + +def _right_align(ops, read: str, ref: str): + """Shift every gap run right across matching bases: the mirror of _left_align. + + Of the equal placements of a gap inside a repeat this is the one that + agrees with the reading without the gap for the longest, so it marks + where two readings of a row really part ways.""" + ops = list(ops) + k = len(ops) - 1 + while k >= 0: + kind = ops[k][0] + if kind == "M": + k -= 1 + continue + a = k + while a - 1 >= 0 and ops[a - 1][0] == kind: + a -= 1 + b = k + start = a + while b + 1 < len(ops) and ops[b + 1][0] == "M": + _, next_i, next_j = ops[b + 1] + if read[next_i] != ref[next_j]: + break + if kind == "I": + first_i = ops[a][1] + if read[next_i] != read[first_i]: + break + ops[a : b + 2] = [("M", first_i, next_j)] + [("I", ops[t][1] + 1, next_j + 1) for t in range(a, b + 1)] + else: + first_j = ops[a][2] + if ref[next_j] != ref[first_j]: + break + ops[a : b + 2] = [("M", next_i, first_j)] + [("D", next_i + 1, ops[t][2] + 1) for t in range(a, b + 1)] + a += 1 + b += 1 + k = start - 1 + return ops def _left_align(ops, read: str, ref: str): @@ -3333,8 +3465,10 @@ def marker_type_label(locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFA inherited indel and the SNP. 'snp' = substitutions only, 'indel' = gaps only, 'both' = gaps plus substitutions, '.' = no concrete event beyond the canonical (the Ref row; a row that merely realises an - ambiguity code). Downstream users can keep or drop undesigned indel - haplotypes by this label alone. ``events`` may be passed when already + ambiguity code), 'unresolved' = the row's end is not read + (row_decision), whatever the row carries before it: a gap there is + likely but unverified. Downstream users can keep or drop undesigned + indel haplotypes by this label alone. ``events`` may be passed when already computed, and ``designed`` (the Alt row's events) when the caller knows more than the two tags show, as for a marker written without primers. """ @@ -3351,6 +3485,8 @@ def marker_type_label(locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFA else: residual.append(event) events = residual + if any(event[0] == "unresolved" for event in events): + return "unresolved" kinds = {event[0] for event in events if event[0] in ("snp", "ins", "del")} if not kinds: return "." @@ -3359,12 +3495,22 @@ def marker_type_label(locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFA return "both" if "snp" in kinds else "indel" -def _describe(ref: str, sequence: str, gap_band: int, gapped: bool, whole: bool = False): +def _describe( + ref: str, sequence: str, gap_band: int, gapped: bool, whole: bool = False, unresolved: Optional[Tuple[int, int]] = None, + against: Optional[str] = None, +): """Events of ``sequence`` against ``ref`` (the core of variant_events). ``gapped`` is the caller's decision — row_reading for an observed row, the unguarded cheapest reading for a design — so the description never - re-decides what classification decided. + re-decides what classification decided. ``against`` is part of that + decision: the Alt tag, in the frame and at the length of ``ref``, for a + row that was read from it (_read_from_alt). The row is then aligned to + that tag and the events are written against ``ref`` column by column, + so the gaps stand where the row's category was read and the designed + substitution is listed. The row's own alignment to ``ref`` has no + reason to put the Alt base on the target: it can split one gap in two, + or delete the target, to avoid a substitution there. A sequence of another length than ``ref`` (a row whose primers were removed, see trim_locus_rows) is always aligned: the lengths themselves @@ -3376,7 +3522,26 @@ def _describe(ref: str, sequence: str, gap_band: int, gapped: bool, whole: bool primer from one an insertion pushed the primer out of. When the lengths differ by more than ``gap_band`` no alignment is attempted: the shared length is compared base by base and the rest is ``clip`` or - ``overhang``.""" + ``overhang``. + + ``unresolved`` is row_decision's ``(ref_index, row_index)`` in the frame + of these two sequences. The part before it is described end to end + (both readings of the row agree on it), and one closing ``unresolved`` + event holds the Ref bases not read and the row's bases that stand + there.""" + if against is not None and len(against) != len(ref): + against = None + if unresolved is not None: + first, start = unresolved + if first and not start: # nothing of the row before it: the Ref bases before it are deleted + events = [("del", 1, ref[:first], ".")] + else: + events = _describe( + ref[:first], sequence[:start], gap_band, gapped or first != start, whole=True, + against=None if against is None else against[:first], + ) + events.append(("unresolved", first + 1, ref[first:], sequence[start:])) + return events if sequence == ref: return [] difference = abs(len(sequence) - len(ref)) @@ -3392,7 +3557,7 @@ def _describe(ref: str, sequence: str, gap_band: int, gapped: bool, whole: bool if shared < len(sequence): events.append(("overhang", len(ref) + 1, ".", sequence[shared:])) return events - ops, end_read, end_ref, _ = gapped_alignment(sequence, ref, gap_band, whole) + ops, end_read, end_ref, _ = gapped_alignment(sequence, ref if against is None else against, gap_band, whole) ops = _left_align(ops, sequence, ref) events = [] k = 0 @@ -3409,14 +3574,18 @@ def _describe(ref: str, sequence: str, gap_band: int, gapped: bool, whole: bool inserted = sequence[i : ops[run_end][1] + 1] if kind == "I" else "" deleted = ref[j : ops[run_end][2] + 1] if kind == "D" else "" name = "ins" if kind == "I" else "del" - if j > 0: + before = ops[k - 1] if k else None + if before is not None and before[0] == "M" and _design_matches(sequence[before[1]], ref[before[2]]): events.append((name, j, ref[j - 1] + deleted, ref[j - 1] + inserted)) else: - # No base precedes a gap on the very first base, so it is written - # without an anchor: 1:.>GG inserts GG before base 1, 1:TT>. deletes - # bases 1 and 2. Borrowing the base after it as an anchor would - # use that base twice whenever another event starts there. - events.append((name, 1, deleted or ".", inserted or ".")) + # The base before the gap cannot anchor it: there is none (a gap + # on the very first base), or the row shows another base there and + # a substitution already uses it (a gap right after the Alt base of + # a row read from the Alt tag). The gap is then written without an + # anchor, from its own first base: 1:.>GG inserts GG before base 1, + # 32:CACG>. deletes bases 32 to 35. Anchoring it on that base would + # use the base twice, and the row could not be rebuilt in order. + events.append((name, j + 1, deleted or ".", inserted or ".")) k = run_end + 1 if end_ref < len(ref): events.append(("clip", end_ref + 1, ref[end_ref:], ".")) @@ -3455,8 +3624,68 @@ def variant_events(locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFAULT substitutions). Positions are on the tag as printed (design strand); a consumer that knows the strand flips them, as it does for substitutions already. + + A row whose end is unresolved (row_decision) lists its events up to + there and closes with ``unresolved``: the first Ref position not read, + the Ref bases from it on, and the row's own bases that stand there. + + A row read from the Alt tag (_read_from_alt) is aligned to that tag and + written in the Ref tag's positions, so its Variants and its category + show the same base on the target. A gap right after a substituted base + is written without an anchor base (``32:CACG>.``, ``32:.>GG``), as a + gap on the first base of the tag is. """ - return _describe(locus.ref_tag, sequence, gap_band, _read_through_gaps(locus, sequence, gap_band)) + return _describe( + locus.ref_tag, sequence, gap_band, _read_through_gaps(locus, sequence, gap_band), + unresolved=unresolved_end(locus, sequence, gap_band), + against=locus.alt_tag if _read_from_alt(locus, sequence, gap_band) else None, + ) + + +def _read_from_alt(locus: Locus, sequence: str, gap_band: int) -> bool: + """Whether a row's accepted gapped reading was made against the Alt tag. + + row_decision reads a row from the Ref tag or from the Alt tag, whichever + explains it most cheaply; the two tags have the same positions, so the + reading can be written against the Ref tag column by column (_describe). + Never on a marker whose own design is an indel: there the tags' positions + differ past the designed gap, and every row is described by its alignment + to the Ref tag (_read_through_gaps).""" + reading = row_reading(locus, sequence, gap_band) + if reading is None or reading.template != locus.alt_tag or locus.alt_tag == locus.ref_tag: + return False + return not any(kind in ("ins", "del") for kind, *_ in _designed_events(locus.ref_tag, locus.alt_tag, gap_band)) + + +def unresolved_end(locus: Locus, sequence: str, gap_band: int) -> Optional[Tuple[int, int]]: + """row_decision's unresolved stretch in the Ref tag's coordinates, as a row's Variants show it, or None. + + The decision counts positions on the canonical tag each reading was + made against. Those are the Ref tag's positions, except on a marker + whose own design is an indel, for a reading against the Alt tag past + the designed indel. There the stretch is placed by the base of the row + it starts at, which is the same in every frame: the Ref position is the + one the row's alignment to the Ref tag (the one its Variants are + written from, _read_through_gaps) puts under that base, moved back over + any Ref bases deleted right before it. A known designed indel does not + settle another gap near the row's end.""" + stretch = row_decision(locus, sequence, gap_band).unresolved + if stretch is None or stretch[0] <= locus.variant_pos: + return stretch + if not any(kind in ("ins", "del") for kind, *_ in _designed_events(locus.ref_tag, locus.alt_tag, gap_band)): + return stretch + start = stretch[1] + ops = _left_align(gapped_alignment(sequence, locus.ref_tag, gap_band)[0], sequence, locus.ref_tag) + at = next((k for k, (kind, i, _) in enumerate(ops) if kind != "D" and i >= start), None) + if at is None: + return None # the row's unread bases lie past the Ref tag's end: no Ref position stands for them + first = ops[at][2] + while at > 0 and ops[at - 1][0] == "D": + at -= 1 + first = ops[at][2] + if first >= len(locus.ref_tag) or abs(first - start) > gap_band: + return None + return first, start def events_either_way(ref: str, sequence: str, gap_band: int): @@ -3476,11 +3705,14 @@ def events_either_way(ref: str, sequence: str, gap_band: int): def format_variants(events) -> str: - """The Variants cell: ``;``-joined tokens, ``pos:REF>ALT`` per event plus ``clip=k`` / ``overhang=k``; '.' when empty.""" + """The Variants cell: ``;``-joined tokens, ``pos:REF>ALT`` per event plus ``clip=k`` / ``overhang=k`` / + ``unresolved=k`` (the last ``k`` Ref bases are not read in this row); '.' when empty.""" tokens = [] for kind, position, ref_allele, alt_allele in events: if kind == "clip": tokens.append(f"clip={len(ref_allele)}") + elif kind == "unresolved": + tokens.append(f"unresolved={len(ref_allele)}") elif kind == "overhang": tokens.append(f"overhang={len(alt_allele)}") else: @@ -3495,13 +3727,26 @@ def parse_variants(text: str, ref: str, sequence: str): what the ``clip`` and ``overhang`` tokens leave out. Lets a consumer of the table take a row's reading from it and not derive its own, which matters for a table written without primers: only ``generate`` - knows which rows end at their right primer. + knows which rows end at their right primer. A token this version does + not know raises ValueError: the cell then says something about the row + that cannot be read here, and must not be replaced by a guess. """ events = [] for token in text.split(";"): if token in ("", "."): continue - if token.startswith("clip="): + if token.startswith("unresolved="): + # The row's bases that stand there are what the tokens before it leave over. + first = len(ref) - int(token[11:]) + shift = sum( + len(added.strip(".")) - len(removed.strip(".")) + for kind, _, removed, added in events + if kind in ("ins", "del") + ) + if first < 0 or not 0 <= first + shift <= len(sequence): + raise ValueError(f"Variants token {token!r} does not fit the row") + events.append(("unresolved", first + 1, ref[first:], sequence[first + shift :])) + elif token.startswith("clip="): clipped = int(token[5:]) events.append(("clip", len(ref) - clipped + 1, ref[len(ref) - clipped :], ".")) elif token.startswith("overhang="): @@ -3529,7 +3774,10 @@ def classify_sequence(locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFA clustered errors at the read end must not masquerade as a shift). That reading is the same one variant_events describes, so a row's category and its Variants never disagree about where the target sits. A target inside a deletion, or - pushed past the read, is missing data and classifies as Other. The + pushed past the read, is missing data and classifies as Other. So does + a target in the unresolved end of a row (row_decision): a gapped reading + is cheaper there but unverified, the two readings can put different + bases at the target, and neither is taken. The variant-base comparison is IUPAC-aware on the design side only: an ambiguous observed base such as N is missing data, not evidence for either allele, and classifies as Other. @@ -3543,7 +3791,9 @@ def classify_sequence(locus: Locus, sequence: str, gap_band: int = GAP_BAND_DEFA # The cheapest substitution-only reading over both canonicals is the # bound a gapped reading must beat; the reading itself is the one # accepted_gapped_reading gives, shared with the row's Variants. - best = row_reading(locus, sequence, gap_band) + best, unresolved = row_decision(locus, sequence, gap_band) + if unresolved is not None and locus.variant_pos >= unresolved[0]: + return "Other" if best is not None: mapped = best.ref_to_read.get(locus.variant_pos) if mapped is None or mapped >= len(sequence): @@ -3806,12 +4056,19 @@ def trim_locus_rows( printed. ``readings`` gives ``_describe`` what the trimmed sequences no longer - show, as ``(gapped, whole)`` per row. ``gapped`` is the decision made - on the full tag (_read_through_gaps): a primer that verified a gap is - gone from the row, so the decision is carried, not made again. - ``whole`` marks a row that was cut at its right primer on a marker - whose Ref row ends at its own: the two are compared end to end, every - other row as a fixed-length window. + show, as ``(gapped, whole, unresolved, against)`` per row (``against``: + the Alt row as printed when the category was read from the Alt tag, so + the row is described from the same reading). ``gapped`` is the + decision made on the full tag (_read_through_gaps): a primer that + verified a gap is gone from the row, so the decision is carried, not + made again. ``whole`` marks a row that was cut at its right primer on + a marker whose Ref row ends at its own: the two are compared end to + end, every other row as a fixed-length window. ``unresolved`` is the + stretch the full tag's decision left unread (unresolved_end), counted in + the trimmed frame, and it is carried whatever length the row was cut + to: a row that is Other because its target was not read never shows a + reading of that target. It is None only when the whole row is read, or + when the stretch lay in the primer that was cut from the Ref row. """ after_left = len(locus.left_primer) cuts = {sequence: right_primer_cut(locus, sequence, site, gap_band) for _, _, sequence in rows} @@ -3823,6 +4080,7 @@ def trim_locus_rows( tally["markers_right_primer_kept"] += 1 merged: Dict[str, List] = {} gapped: Dict[str, bool] = {} + from_alt: Dict[str, bool] = {} # the row whose category a printed row carries was read from the Alt tag for _, category, sequence in rows: trimmed = without_primers(sequence) gapped[trimmed] = gapped.get(trimmed, False) or _read_through_gaps(locus, sequence, gap_band) @@ -3834,12 +4092,14 @@ def trim_locus_rows( entry = merged.get(trimmed) if entry is None: merged[trimmed] = [category, list(reads), total] + from_alt[trimmed] = _read_from_alt(locus, sequence, gap_band) continue tally["rows_merged"] += 1 tally["rows_merged_across_allele_groups"] += ALLELE_GROUP[category] != ALLELE_GROUP[entry[0]] entry[1] = [kept + added for kept, added in zip(entry[1], reads)] if entry[0] not in ("Ref", "Alt") and total > entry[2]: entry[0] = category + from_alt[trimmed] = _read_from_alt(locus, sequence, gap_band) entry[2] = max(entry[2], total) shown = dataclasses.replace( locus, @@ -3854,7 +4114,34 @@ def trim_locus_rows( uncut = len(locus.ref_tag) - after_left # The Ref row ends at its primer when it was cut, or when the primer starts exactly where the tag ends. ref_ends_at_primer = len(shown.ref_tag) < uncut or (site is not None and site[0] == len(locus.ref_tag)) - readings = {sequence: (gapped[sequence], ref_ends_at_primer and len(sequence) < uncut) for sequence in merged} + unresolved: Dict[str, Tuple[int, int]] = {} + for _, _, sequence in rows: + stretch = unresolved_end(locus, sequence, gap_band) + if stretch is None: + continue + trimmed = without_primers(sequence) + first, start = max(stretch[0] - after_left, 0), max(stretch[1] - after_left, 0) + # A row cut before the stretch begins lost bases both readings agree + # on; the Ref bases that stood for them are not read either. + cut_short = start - len(trimmed) + if cut_short > 0: + first, start = max(first - cut_short, 0), len(trimmed) + # Kept whatever the row's length: a right primer found at another + # place does not by itself say which reading of the end was right. + # Only a stretch that lay wholly in the primer cut from the Ref row + # leaves nothing to mark. + if first < len(shown.ref_tag) and (trimmed not in unresolved or first < unresolved[trimmed][0]): + unresolved[trimmed] = (first, start) + # The fourth entry is _describe's ``against``: the Alt row as printed, for a + # row whose category was read from the Alt tag, so that its Variants show + # the gaps where that reading has them. + readings = { + sequence: ( + gapped[sequence], ref_ends_at_primer and len(sequence) < uncut, unresolved.get(sequence), + shown.alt_tag if from_alt[sequence] else None, + ) + for sequence in merged + } return shown, trimmed_rows, {sequence: entry[1] for sequence, entry in merged.items()}, readings @@ -4108,6 +4395,7 @@ def write_madc( } reads_in_emitted_rows = 0 per_sample_emitted = [0] * len(results) + unresolved_ends = {"rows": 0, "reads": 0, "rows_target_not_read": 0, "reads_target_not_read": 0} primer_tally: Counter = Counter() if options.remove_primers: geometry = amplicon_geometry(loci) @@ -4138,6 +4426,15 @@ def write_madc( for sequence in sequences: category = forced.get(sequence) or classify_sequence(locus, sequence, gap_band) rows.append((ALLELE_TYPE_ORDER.get(category, 99), category, sequence)) + stretch = row_decision(locus, sequence, gap_band).unresolved + if stretch is not None: + # Counted on the full tags, so the tally is the same with and without primers. + reads = observed[locus.index][sequence] + target_in_it = locus.variant_pos >= stretch[0] + unresolved_ends["rows"] += 1 + unresolved_ends["reads"] += reads + unresolved_ends["rows_target_not_read"] += target_in_it + unresolved_ends["reads_target_not_read"] += reads * target_in_it rows.sort(key=lambda row: (row[0], row[2])) counts_by_sequence = { sequence: [result.counts.get((locus.index, sequence), 0) for result in results] for _, _, sequence in rows @@ -4244,6 +4541,7 @@ def write_madc( }, "novel_rows_by_evidence_route": dict(route_counts), "excluded_novel": excluded, + "unresolved_ends": unresolved_ends, "reads_in_emitted_rows": reads_in_emitted_rows, "per_sample_reads_in_emitted_rows": per_sample_emitted, } diff --git a/madcats_call.py b/madcats_call.py index 06369a6..7711771 100644 --- a/madcats_call.py +++ b/madcats_call.py @@ -2310,15 +2310,17 @@ class RowReading: the MADC's Variants column when the table has one, otherwise ``madcats.variant_events``, the reading that fills that column. A position is not shown when the row deleted it or when an insertion pushed - it out of the fixed-length window. A row that is not ``comparable`` - shows no position at all: one of another length than the Ref tag whose - events the table does not state, or whose length differs by more than - the gap limit. + it out of the fixed-length window, or when ``generate`` left the row's + end unresolved. A row that is not ``comparable`` shows no position at + all: one of another length than the Ref tag whose events the table does + not state, or whose length differs by more than the gap limit, or one + whose Variants cell this version cannot read (not ``understood``). """ - def __init__(self, ref_tag: str, events, comparable: bool = True) -> None: + def __init__(self, ref_tag: str, events, comparable: bool = True, understood: bool = True) -> None: self.ref_tag = ref_tag - self.comparable = comparable + self.comparable = comparable and understood + self.understood = understood self.substitutions: Dict[int, str] = {} self.positions_not_shown: Set[int] = set() for kind, position, ref_allele, alt_allele in events: @@ -2326,12 +2328,17 @@ def __init__(self, ref_tag: str, events, comparable: bool = True) -> None: self.substitutions[position] = alt_allele elif kind == "del": # Written from the base before the deleted ones: that base is still shown. - # A deletion at the very start of the tag has no base before it and is written - # without one (alt "."), or in older tables from the base after it; the forms - # cannot always be told apart there, so the whole span is treated as not shown. - first_missing = position if position == 1 else position + 1 + # A deletion without that base (alt ".") starts on its own first base: one at + # the very start of the tag, or one right after a substituted base, which a + # substitution already uses. Older tables wrote a deletion at the start of the + # tag from the base after it; the forms cannot always be told apart there, so + # the whole span is treated as not shown. + anchored = position != 1 and alt_allele.strip(".") != "" + first_missing = position + 1 if anchored else position self.positions_not_shown.update(range(first_missing, position + len(ref_allele))) - elif kind == "clip": + elif kind in ("clip", "unresolved"): + # Pushed out of the window by an insertion, or left unread by ``generate`` + # because the row's end has two explanations (madcats.row_decision). self.positions_not_shown.update(range(position, len(ref_tag) + 1)) def base_at(self, position: int) -> Optional[str]: @@ -2343,14 +2350,24 @@ def base_at(self, position: int) -> Optional[str]: return base if base in CONCRETE_BASES else None +NOT_UNDERSTOOD = "variants cell not understood" # table_events' answer for a cell this version cannot read + + def table_events(row: MadcRow, ref_tag: str): - """The events of a row's Variants cell, or None when the row has none or it cannot be read.""" + """The events of a row's Variants cell; None when the row has no cell. + + A cell that cannot be read gives NOT_UNDERSTOOD. It may hold a token a + later version of ``generate`` writes, which says something about the row + this version does not know, so the events are not derived from the + sequences in its place: that would put another reading where the table + stated one. The caller leaves such a row unread and counts it. + """ if row.variants is None: return None try: return parse_variants(row.variants, ref_tag, row.sequence) - except ValueError: # not written by ``generate``: derive the events instead - return None + except ValueError: + return NOT_UNDERSTOOD def read_row(locus: Locus, row: MadcRow, gap_band: int, use_variants: bool = True) -> RowReading: @@ -2365,10 +2382,13 @@ def read_row(locus: Locus, row: MadcRow, gap_band: int, use_variants: bool = Tru without primers, and whether it ends at its right primer cannot be told from its sequence: it is read only if its events are the same either way (``events_either_way``). Otherwise its reads are left out of every - off-target record. + off-target record. So are those of a row whose Variants cell cannot be + read (``table_events``). """ events = table_events(row, locus.ref_tag) if use_variants else None difference = abs(len(row.sequence) - len(locus.ref_tag)) + if events is NOT_UNDERSTOOD: + return RowReading(locus.ref_tag, [], understood=False) if events is not None: return RowReading(locus.ref_tag, events, comparable=difference <= gap_band) if difference: @@ -2765,10 +2785,13 @@ def add(vcf_type: str, line: str, tallies: Counter) -> None: # The Alt row's own Variants state the design when they were written against this Ref tag. alt_row = next((row for row in rows if row.category == "Alt" and row.sequence == alt_tag), None) stated = alt_row is not None and ref_tag == _first_sequence(rows, "Ref") - target, skip_reason = resolve_target( - ref_tag, alt_tag, task.declared_target_position, options.gap_band, - table_events(alt_row, ref_tag) if stated else None, - ) + designed = table_events(alt_row, ref_tag) if stated else None + if designed is NOT_UNDERSTOOD: + target, skip_reason = None, "variants_not_understood" + else: + target, skip_reason = resolve_target( + ref_tag, alt_tag, task.declared_target_position, options.gap_band, designed + ) if target is None: for vcf_type in site_types: output.tallies[(vcf_type, f"targets_skipped_{skip_reason}")] += 1 @@ -2832,7 +2855,10 @@ def _off_target_records( readings = [read_row(locus, row, options.gap_band, stated) for row in rows] rows_with_reads = [index for index, counts in enumerate(counts_by_row) if any(counts)] tallies[("markers", "rows_with_reads_not_read_for_their_length")] += sum( - 1 for index in rows_with_reads if not readings[index].comparable + 1 for index in rows_with_reads if readings[index].understood and not readings[index].comparable + ) + tallies[("markers", "rows_with_reads_whose_variants_were_not_understood")] += sum( + 1 for index in rows_with_reads if not readings[index].understood ) target_is_substitution = target is not None and len(target.ref_allele) == len(target.alt_allele) == 1 site_positions = sorted( @@ -3666,6 +3692,13 @@ def parallel_map(function, items): ), file=sys.stderr, ) + not_understood = tallies.get(("markers", "rows_with_reads_whose_variants_were_not_understood"), 0) + if not_understood and vcf_type == VCF_ALL: + print( + f" {not_understood:,} rows with reads have a Variants cell this version of call cannot read and " + "were not read; their reads are in XD. Call the table with the madcats version that wrote it", + file=sys.stderr, + ) if counts.get("records_not_converged"): print( f" {counts['records_not_converged']:,} records reached --max-iterations before their allele " diff --git a/tests/test_call.py b/tests/test_call.py index a38a68c..b929808 100644 --- a/tests/test_call.py +++ b/tests/test_call.py @@ -1785,16 +1785,23 @@ def test_trimmed_marker_without_an_alt_row_stays_in_the_frame_of_its_rows(self): ref_only = [call.MadcRow("M1|Ref", "Ref", REF_TAG[8:26], (1,))] self.assertEqual(call.design_as_written(design, ref_only), (REF_TAG[8:26], "", 4)) - def test_variants_cell_that_generate_did_not_write_is_not_used(self): + def test_variants_cell_that_cannot_be_read_is_not_replaced_by_a_guess(self): + # The cell may hold a token a later ``generate`` writes. The row is left unread and + # counted; its events are never derived from the sequences in the cell's place. locus = madcats.Locus(0, "M1", "", "", "G", "C", 3, REF_TAG[8:26], ALT_TAG[8:26]) with_site = REF_TAG_SITE_20[8:26] - for cell in ("NA", "0.25", "12:G", "x:G>T"): + for cell in ("NA", "0.25", "12:G", "x:G>T", "12:G>T;someday=3"): row = call.MadcRow("M1|RefMatch", "RefMatch", with_site, (1,), cell) - self.assertIsNone(call.table_events(row, locus.ref_tag), cell) - self.assertEqual(call.read_row(locus, row, 10).substitutions, {12: "T"}, cell) - # ... and a row of another length with such a cell is not read at all. + self.assertIs(call.table_events(row, locus.ref_tag), call.NOT_UNDERSTOOD, cell) + reading = call.read_row(locus, row, 10) + self.assertFalse(reading.understood, cell) + self.assertFalse(reading.comparable, cell) + self.assertIsNone(reading.base_at(12), cell) shorter = call.MadcRow("M1|RefMatch", "RefMatch", with_site[:-2], (1,), cell) self.assertFalse(call.read_row(locus, shorter, 10).comparable, cell) + # A row without the cell is still read from the sequences. + bare = call.MadcRow("M1|RefMatch", "RefMatch", with_site, (1,), None) + self.assertEqual(call.read_row(locus, bare, 10).substitutions, {12: "T"}) class HaplotypeVcfTests(CommandTestCase): diff --git a/tests/test_madcats.py b/tests/test_madcats.py index 5c62e07..df34c73 100644 --- a/tests/test_madcats.py +++ b/tests/test_madcats.py @@ -287,8 +287,9 @@ def test_partial_defaults_resolve_from_tag_length(self): self.assertEqual(max(1, args.tag_length - mp.PARTIAL_TAIL_DEFAULT), 100) def test_one_reading_for_category_and_variants(self): - # A deletion inside the last 12 bases is not a verified reading: both the - # category (fixed offset) and the Variants (substitutions) ignore it. + # A deletion inside the last 12 bases is not a verified reading, yet it is + # cheaper than substitutions: the row is read up to it and no further. The + # target lies before it, so the category still reads it at the fixed offset. random.seed(11) ref = "".join(random.choice("ACGT") for _ in range(109)) alt = ref[:60] + ("A" if ref[60] != "A" else "C") + ref[61:] @@ -296,8 +297,9 @@ def test_one_reading_for_category_and_variants(self): tail_deletion = ref[:100] + ref[102:] + "GT" self.assertEqual(mp.classify_sequence(locus, tail_deletion, 20), "RefMatch") events = mp.variant_events(locus, tail_deletion, 20) - self.assertTrue(events and all(kind == "snp" for kind, *_ in events)) - self.assertEqual(mp.marker_type_label(locus, tail_deletion, 20, events), "snp") + self.assertEqual(events, [("unresolved", 101, ref[100:], tail_deletion[100:])]) + self.assertEqual(mp.format_variants(events), "unresolved=9") + self.assertEqual(mp.marker_type_label(locus, tail_deletion, 20, events), "unresolved") # A verified upstream deletion shifts the target for both consumers alike. shifted = ref[:30] + ref[31:60] + alt[60] + ref[61:] + "G" self.assertEqual(mp.classify_sequence(locus, shifted, 20), "AltMatch") @@ -1760,7 +1762,7 @@ def test_tail_gap_keeps_the_substitution_reading(self): # Deletion at 37 pushes ref[40]='C' into index 39 and appends a T. read = ref[:37] + ref[38:] + "T" self.assertEqual(read[variant_pos], "C") - self.assertEqual(mp.classify_sequence(locus, read), "Other") # fixed-offset reading, not the gapped one + self.assertEqual(mp.classify_sequence(locus, read), "Other") # neither reading: the target is not read def test_denoise_prefers_closer_parent_before_deeper(self): ref = "A" * 20 diff --git a/tests/test_reading_ties.py b/tests/test_reading_ties.py new file mode 100644 index 0000000..d1076ec --- /dev/null +++ b/tests/test_reading_ties.py @@ -0,0 +1,392 @@ +"""A row that the Ref tag and the Alt tag both explain through gaps, and how it is written. + +``row_decision`` takes the reading with fewer edits, then the one with fewer gap openings +(``_reading_cost``), and the Ref tag's only when the two are equal in that as well. Before, the Ref +tag's reading stood on every tie: an Alt haplotype with one insertion or deletion beside its target +was read as a Ref haplotype with two, or as a row whose target was deleted. + +A row read from the Alt tag is also written from that reading (``_read_from_alt``): its Variants are +its alignment to the Alt tag in the Ref tag's positions. Before, they were always its alignment to +the Ref tag, which could show the target unchanged or deleted on an AltMatch row. +""" + +import csv +import random +import sys +import tempfile +import unittest +from pathlib import Path + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO)) + +import madcats as mp # noqa: E402 +import madcats_call as call # noqa: E402 + +BAND = 10 +# 0 1 2 3 4 5 6 7 8 +# 012345678901234567890123456789012345678901234567890123456789012345678901234567890 +REF = "GCTAAAGACAATTACATAACATACACGTCAGCACGAAACTTGATGGCCCAGTGTGAATCGCTTAAGGGTTAAGTAAGTGTG" +LEFT = 20 # bases of REF that are the left primer + + +def locus_at(target, alt_base, ref=REF): + """A marker on ``ref`` with its target at the 0-based ``target``.""" + return mp.Locus(0, "m1", ref[:LEFT], "", ref[target], alt_base, target, ref, ref[:target] + alt_base + ref[target + 1 :]) + + +def readings(locus, row): + """The gapped reading of ``row`` against the Ref tag and against the Alt tag, as row_decision compares them.""" + bound = min(mp.tag_mismatches(row, locus.ref_tag), mp.tag_mismatches(row, locus.alt_tag)) + return tuple(mp._unverified_gapped_reading(row, tag, BAND, bound) for tag in (locus.ref_tag, locus.alt_tag)) + + +def cell(locus, row, gap_band=BAND): + return mp.format_variants(mp.variant_events(locus, row, gap_band)) + + +def replay(ref, text, row): + """Rebuild a row from the Ref row and its Variants cell, by the rule the documentation gives. + + ``pos:REF>ALT`` replaces REF at POS by ALT (``.`` is no bases) and may not start on a Ref + base an earlier token used. ``clip=k`` and ``unresolved=k`` drop the last k Ref bases; + ``overhang=k`` then appends the row's last k bases, ``unresolved`` the rest of the row. + """ + out, cursor, drop, tail = "", 0, 0, None + for token in (token for token in text.split(";") if token not in ("", ".")): + if token.startswith("clip="): + drop = int(token[5:]) + elif token.startswith("overhang="): + tail = int(token[9:]) + elif token.startswith("unresolved="): + drop, tail = int(token[11:]), "rest" + else: + position, alleles = token.split(":") + removed, added = (allele.strip(".") for allele in alleles.split(">")) + start = int(position) - 1 + assert start >= cursor, f"two tokens use Ref base {position}: {text}" + assert ref[start : start + len(removed)] == removed, f"{token} does not name the Ref bases: {text}" + out += ref[cursor:start] + added + cursor = start + len(removed) + limit = len(ref) - drop + assert cursor <= limit, f"a token reaches into the bases the last one drops: {text}" + out += ref[cursor:limit] + if tail == "rest": + out += row[len(out) :] + elif tail: + out += row[len(row) - tail :] + return out + + +def shown_at(events, position): + """What a row's events put on a 1-based Ref position: the Ref base ("ref"), another base, or None (no base).""" + for kind, start, removed, added in events: + if kind == "snp" and start == position: + return added + if kind == "del": + first = start + 1 if added.strip(".") else start # an anchored deletion keeps its first base + if first <= position < start + len(removed.strip(".")): + return None + if kind in ("clip", "unresolved") and start <= position: + return None + return "ref" + + +def side_of(category, alt_base): + """What a category says stands on the target, in shown_at's terms; None for a category that names no allele.""" + return {"Ref": "ref", "RefMatch": "ref", "Alt": alt_base, "AltMatch": alt_base}.get(category) + + +def haplotypes(locus, count, seed): + """Ref and Alt haplotypes of a marker with one insertion or deletion, some with one more substitution. + + Half of the gaps lie within three bases of the target, where the two tags' readings tie most often; + inserted bases are copied from the neighbourhood half of the time, which makes repeats.""" + rng = random.Random(seed) + rows = set() + while len(rows) < count: + base = rng.choice([locus.ref_tag, locus.alt_tag]) + near = locus.variant_pos + rng.randint(-3, 4) + at = near if rng.random() < 0.5 else rng.randrange(LEFT + 2, len(base) - 14) + size = rng.randint(1, 6) + if rng.random() < 0.5: + row = base[:at] + base[at + size :] + "".join(rng.choice("ACGT") for _ in range(size)) + else: + piece = base[at : at + size] if rng.random() < 0.5 else "".join(rng.choice("ACGT") for _ in range(size)) + row = (base[:at] + piece + base[at:])[: len(base)] + if rng.random() < 0.3: + index = rng.randrange(LEFT, len(row)) + row = row[:index] + rng.choice([b for b in "ACGT" if b != row[index]]) + row[index + 1 :] + if row not in (locus.ref_tag, locus.alt_tag): + rows.add(row) + return sorted(rows) + + +class ReadingTieTests(unittest.TestCase): + def test_the_module_under_test_is_the_repository_s(self): + self.assertEqual(Path(mp.__file__).resolve().parent, REPO) + + def test_one_insertion_beside_the_alt_base_is_not_read_as_two_around_a_ref_base(self): + # The target is base 42, G in Ref and A in Alt, inside TGATG. An Alt haplotype (TAATG) that + # gained ATGATT after it reads TAATGATGATT: the Alt tag plus one insertion, or the Ref tag + # with TAA inserted before the T and ATT after the last G. Six inserted bases either way. + locus = locus_at(41, "A") + self.assertEqual((locus.ref_tag[40:45], locus.alt_tag[40:45]), ("TGATG", "TAATG")) + row = (locus.alt_tag[:45] + "ATGATT" + locus.alt_tag[45:])[: len(REF)] + self.assertEqual(row, (REF[:40] + "TAA" + REF[40:45] + "ATT" + REF[45:])[: len(REF)]) + on_ref, on_alt = readings(locus, row) + self.assertEqual((on_ref.edits, on_ref.gap_events), (6, 2)) + self.assertEqual((on_alt.edits, on_alt.gap_events), (6, 1)) + decision = mp.row_decision(locus, row, BAND) + self.assertIsNone(decision.unresolved) + self.assertEqual(decision.reading.gap_events, 1) + self.assertEqual(decision.reading.ref_to_read[41], 41) # the target is read where it stands + self.assertEqual(row[41], locus.alt_base) + self.assertEqual(mp.classify_sequence(locus, row, BAND), "AltMatch") + + def test_the_same_insertion_in_a_ref_haplotype_stays_with_the_ref_tag(self): + # The mirror marker: TAATG is the Ref tag and G the Alt base. The Ref tag's reading has the + # one insertion now, the Alt tag's the two. + ref = REF[:41] + "A" + REF[42:] + locus = locus_at(41, "G", ref) + row = (ref[:45] + "ATGATT" + ref[45:])[: len(ref)] + on_ref, on_alt = readings(locus, row) + self.assertEqual((on_ref.edits, on_ref.gap_events), (6, 1)) + self.assertEqual((on_alt.edits, on_alt.gap_events), (6, 2)) + self.assertEqual(mp.row_decision(locus, row, BAND).reading.gap_events, 1) + self.assertEqual(mp.classify_sequence(locus, row, BAND), "RefMatch") + + def test_one_deletion_after_the_alt_base_is_not_read_as_two_that_delete_the_target(self): + # The target is base 31, G in Ref and A in Alt, followed by CACG. An Alt haplotype without + # those four bases shows its A where the Ref tag has G C A: against the Ref tag the cheapest + # reading deletes G and C, keeps the A of CACG, and deletes C and G. Four deleted bases + # either way, in one piece against the Alt tag. + locus = locus_at(30, "A") + self.assertEqual(REF[30:35], "GCACG") + row = locus.alt_tag[:31] + locus.alt_tag[35:] + "ACGT" + on_ref, on_alt = readings(locus, row) + self.assertEqual((on_ref.edits, on_ref.gap_events), (4, 2)) + self.assertEqual((on_alt.edits, on_alt.gap_events), (4, 1)) + self.assertNotIn(30, on_ref.ref_to_read) # the Ref tag's reading deletes the target + decision = mp.row_decision(locus, row, BAND) + self.assertEqual(decision.reading.ref_to_read[30], 30) + self.assertEqual(mp.classify_sequence(locus, row, BAND), "AltMatch") + + def test_readings_equal_in_edits_and_openings_leave_the_ref_tag_s(self): + # With five bases gone (CACGA) the A that follows them is the Alt base again: the row is the + # Alt haplotype without CACGA and equally the Ref haplotype without GCACG. One deletion + # either way; no reading is better, and the Ref tag's has no base on the target. + locus = locus_at(30, "A") + row = locus.alt_tag[:31] + locus.alt_tag[36:] + "ACGTA" + self.assertEqual(row, REF[:30] + REF[35:] + "ACGTA") + on_ref, on_alt = readings(locus, row) + self.assertEqual((on_ref.edits, on_ref.gap_events), (5, 1)) + self.assertEqual((on_alt.edits, on_alt.gap_events), (5, 1)) + self.assertEqual(mp.row_decision(locus, row, BAND).reading.ops, on_ref.ops) + self.assertEqual(mp.classify_sequence(locus, row, BAND), "Other") + + def test_fewer_edits_come_before_fewer_openings(self): + def reading(edits, openings): + return mp.GappedReading(edits, 0, edits, openings, (), 0, 0, 0, {}) + + self.assertLess(mp._reading_cost(reading(5, 3)), mp._reading_cost(reading(6, 1))) + self.assertLess(mp._reading_cost(reading(6, 1)), mp._reading_cost(reading(6, 2))) + self.assertEqual(mp._reading_cost(reading(6, 2)), mp._reading_cost(reading(6, 2))) + + def test_rescue_off_reads_the_alt_base_at_the_fixed_offset(self): + # Without gapped readings there is nothing to tie: both rows show the Alt base on the target. + locus = locus_at(41, "A") + row = (locus.alt_tag[:45] + "ATGATT" + locus.alt_tag[45:])[: len(REF)] + self.assertEqual(mp.row_decision(locus, row, 0), mp.RowDecision(None, None)) + self.assertEqual(mp.classify_sequence(locus, row, 0), "AltMatch") + + +class VariantsFollowTheReadingTests(unittest.TestCase): + def test_insertion_beside_the_alt_base_is_written_as_one_insertion(self): + locus = locus_at(41, "A") + row = (locus.alt_tag[:45] + "ATGATT" + locus.alt_tag[45:])[: len(REF)] + self.assertTrue(mp._read_from_alt(locus, row, BAND)) + events = mp.variant_events(locus, row, BAND) + self.assertEqual(events, [("snp", 42, "G", "A"), ("ins", 45, "G", "GATGATT"), ("clip", 76, REF[75:], ".")]) + self.assertEqual(mp.format_variants(events), "42:G>A;45:G>GATGATT;clip=6") + self.assertEqual(replay(REF, "42:G>A;45:G>GATGATT;clip=6", row), row) + self.assertEqual(mp.marker_type_label(locus, row, BAND, events), "indel") + # The row's own alignment to the Ref tag, which was written before: two insertions and no Alt base. + self.assertEqual(mp.format_variants(mp._describe(REF, row, BAND, True)), "40:T>TTAA;45:G>GATT;clip=6") + + def test_gap_right_after_the_alt_base_is_written_without_an_anchor(self): + # The base before the gap is the target, and the row shows the Alt base there: the + # substitution uses that Ref base, so the gap starts on its own first base. + locus = locus_at(30, "A") + deleted = locus.alt_tag[:31] + locus.alt_tag[35:] + "ACGT" + events = mp.variant_events(locus, deleted, BAND) + self.assertEqual(events, [("snp", 31, "G", "A"), ("del", 32, "CACG", "."), ("overhang", 82, ".", "ACGT")]) + self.assertEqual(mp.format_variants(events), "31:G>A;32:CACG>.;overhang=4") + inserted = (locus.alt_tag[:31] + "TTT" + locus.alt_tag[31:])[: len(REF)] + self.assertEqual(mp.classify_sequence(locus, inserted, BAND), "AltMatch") + self.assertEqual(cell(locus, inserted), "31:G>A;32:.>TTT;clip=3") + for row in (deleted, inserted): + text = cell(locus, row) + self.assertEqual(replay(REF, text, row), row) + self.assertEqual(mp.parse_variants(text, REF, row), mp.variant_events(locus, row, BAND)) + self.assertEqual(mp.marker_type_label(locus, row, BAND), "indel") + + def test_call_reads_a_deletion_without_an_anchor_from_its_first_base(self): + locus = locus_at(30, "A") + row = locus.alt_tag[:31] + locus.alt_tag[35:] + "ACGT" + reading = call.RowReading(REF, mp.parse_variants("31:G>A;32:CACG>.;overhang=4", REF, row)) + self.assertEqual(reading.base_at(31), "A") + self.assertEqual([reading.base_at(position) for position in range(32, 36)], [None] * 4) + self.assertEqual(reading.base_at(36), REF[35]) + # An anchored deletion still shows the base it is written from. + anchored = call.RowReading(REF, [("del", 30, "AGCAC", "A")]) + self.assertEqual(anchored.base_at(30), "A") + self.assertEqual([anchored.base_at(position) for position in range(31, 35)], [None] * 4) + + def test_a_row_read_from_the_ref_tag_is_written_as_before(self): + ref = REF[:41] + "A" + REF[42:] + locus = locus_at(41, "G", ref) + row = (ref[:45] + "ATGATT" + ref[45:])[: len(ref)] + self.assertFalse(mp._read_from_alt(locus, row, BAND)) + self.assertEqual(mp.variant_events(locus, row, BAND), mp._describe(ref, row, BAND, True)) + self.assertEqual(cell(locus, row), "45:G>GATGATT;clip=6") + + def test_marker_designed_as_an_indel_keeps_its_alignment_to_the_ref_tag(self): + alt = REF[:40] + REF[41:] + "A" + locus = mp.Locus(0, "m1", REF[:LEFT], "", REF[40], alt[40], 40, REF, alt) + row = alt[:60] + alt[62:] + "GT" + self.assertFalse(mp._read_from_alt(locus, row, BAND)) + self.assertEqual(mp.variant_events(locus, row, BAND), mp._describe(REF, row, BAND, True)) + + def test_variants_and_category_show_the_same_base_on_the_target(self): + # Every row is rebuilt from its Variants with no Ref base used twice, and its Variants put + # on the target what its category says: the Ref base, the Alt base, or no allele. + for target, alt_base, seed in ((41, "A", 1), (30, "A", 2), (55, "C", 3)): + locus = locus_at(target, alt_base) + read_from_alt = 0 + for row in haplotypes(locus, 500, seed): + category = mp.classify_sequence(locus, row, BAND) + events = mp.variant_events(locus, row, BAND) + text = mp.format_variants(events) + self.assertEqual(replay(REF, text, row), row, text) + self.assertEqual(mp.parse_variants(text, REF, row), events, text) + shown = shown_at(events, target + 1) + if side_of(category, alt_base) is not None: + self.assertEqual(shown, side_of(category, alt_base), (category, text)) + else: + self.assertNotIn(shown, ("ref", alt_base), (category, text)) + read_from_alt += mp._read_from_alt(locus, row, BAND) + self.assertGreater(read_from_alt, 50) + + +class TableTests(unittest.TestCase): + """The table ``generate`` writes for Alt haplotypes with a gap beside the target, with and without primers.""" + + def setUp(self): + self.loci = [ + mp.Locus(0, "inserted", REF[:LEFT], "", REF[41], "A", 41, REF, REF[:41] + "A" + REF[42:]), + mp.Locus(1, "deleted", REF[:LEFT], "", REF[30], "A", 30, REF, REF[:30] + "A" + REF[31:]), + ] + ins, dele = self.loci + self.row_inserted = (ins.alt_tag[:45] + "ATGATT" + ins.alt_tag[45:])[: len(REF)] + self.row_deleted = dele.alt_tag[:31] + dele.alt_tag[35:] + "ACGT" + self.rows = { + 0: {ins.ref_tag: 40, ins.alt_tag: 30, self.row_inserted: 12}, + 1: {dele.ref_tag: 40, dele.alt_tag: 30, self.row_deleted: 9}, + } + for index, locus in enumerate(self.loci): + for row in haplotypes(locus, 150, 10 + index): + self.rows[index].setdefault(row, 6) + + def write(self, directory, **options): + counts = {(index, tag): reads for index, rows in self.rows.items() for tag, reads in rows.items()} + results = [mp.SampleResult(mp.Sample("s0", "s0", "unused.fastq", {}), counts, {}, {}, {}, {})] + output = Path(directory) / "madc.csv" + mp.write_madc( + output, self.loci, results, mp.EvidenceRule(1, 20, 0, 0, 0, 5, 0.0), + mp.OutputOptions(variants_path=output.with_suffix(".variants.csv"), **options), BAND, + ) + with open(output, newline="") as handle: + table = {(row["CloneID"], row["AlleleSequence"]): row for row in csv.DictReader(handle)} + with open(output.with_suffix(".variants.csv"), newline="") as handle: + sidecar = list(csv.DictReader(handle)) + return table, sidecar + + def check(self, table, shift): + """Every row is rebuilt from the marker's Ref row and its Variants, which show on the target what its category says.""" + checked = 0 + for locus in self.loci: + ref_row = next(seq for (marker, seq), row in table.items() if marker == locus.marker and row["AlleleID"].endswith("|Ref")) + for (marker, sequence), row in table.items(): + if marker != locus.marker: + continue + self.assertEqual(replay(ref_row, row["Variants"], sequence), sequence, row["Variants"]) + category = row["AlleleID"].split("|")[1].rstrip("_0123456789") + shown = shown_at(mp.parse_variants(row["Variants"], ref_row, sequence), locus.variant_pos - shift + 1) + if side_of(category, locus.alt_base) is not None: + self.assertEqual(shown, side_of(category, locus.alt_base), (category, row["Variants"])) + else: + self.assertNotIn(shown, ("ref", locus.alt_base), (category, row["Variants"])) + checked += 1 + return checked + + def test_table_with_primers(self): + with tempfile.TemporaryDirectory() as directory: + table, sidecar = self.write(directory) + inserted, deleted = table[("inserted", self.row_inserted)], table[("deleted", self.row_deleted)] + self.assertEqual((inserted["AlleleID"].split("_")[0], inserted["Variants"], inserted["MarkerType"]), ("inserted|AltMatch", "42:G>A;45:G>GATGATT;clip=6", "indel")) + self.assertEqual((deleted["AlleleID"].split("_")[0], deleted["Variants"], deleted["MarkerType"]), ("deleted|AltMatch", "31:G>A;32:CACG>.;overhang=4", "indel")) + gap = [row for row in sidecar if row["AlleleSequence"] == self.row_deleted and row["Type"] == "del"] + self.assertEqual([(row["Position"], row["Ref"], row["Alt"]) for row in gap], [("32", "CACG", ".")]) + self.assertGreater(self.check(table, 0), 300) + + def test_table_without_primers(self): + # The same reading in the rows as printed: positions count from the first base after the left primer. + with tempfile.TemporaryDirectory() as directory: + table, _ = self.write(directory, remove_primers=True) + inserted, deleted = table[("inserted", self.row_inserted[LEFT:])], table[("deleted", self.row_deleted[LEFT:])] + self.assertEqual((inserted["AlleleID"].split("_")[0], inserted["Variants"]), ("inserted|AltMatch", "22:G>A;25:G>GATGATT;clip=6")) + self.assertEqual((deleted["AlleleID"].split("_")[0], deleted["Variants"]), ("deleted|AltMatch", "11:G>A;12:CACG>.;overhang=4")) + self.assertGreater(self.check(table, LEFT), 300) + + +class PrimerCutTests(unittest.TestCase): + """Rows of a panel whose tags end with the first bases of the right primer, cut at that primer.""" + + def test_row_cut_at_its_primer_is_written_from_the_alt_tag_s_reading(self): + import test_remove_primers as primers + from unittest import mock + + checked = 0 + for seed in range(40): + rng = random.Random(seed) + left, insert, right = primers.dna(rng, primers.LEFT), primers.dna(rng, primers.INSERT), primers.dna(rng, primers.RIGHT) + target = primers.TARGET + alt_insert = primers.substituted(insert, target) + tag = lambda body: (left + body + right + primers.ADAPTER)[: primers.TAG] + locus = mp.Locus(0, "m", left, right, insert[target], alt_insert[target], primers.LEFT + target, tag(insert), tag(alt_insert)) + # An Alt haplotype without the three bases right after its target. + body = alt_insert[: target + 1] + alt_insert[target + 4 :] + row = tag(body) + category = mp.classify_sequence(locus, row, BAND) + if category != "AltMatch" or not mp._read_from_alt(locus, row, BAND): + continue + shown, rows, _, described = mp.trim_locus_rows( + locus, [(1, "Ref", locus.ref_tag), (2, "Alt", locus.alt_tag), (3, category, row)], + {locus.ref_tag: [5], locus.alt_tag: [5], row: [5]}, (primers.LEFT + primers.INSERT, right), BAND, mock.MagicMock(), + ) + self.assertEqual((shown.ref_tag, shown.alt_tag), (insert, alt_insert)) + self.assertIn(body, described) # cut at its primer: three bases shorter than the Ref row + self.assertEqual(described[body][3], alt_insert) + events = mp._describe(shown.ref_tag, body, BAND, *described[body]) + text = mp.format_variants(events) + self.assertEqual(replay(insert, text, body), body, text) + self.assertEqual(shown_at(events, target + 1), locus.alt_base, text) + self.assertEqual(events[0], ("snp", target + 1, insert[target], alt_insert[target]), text) + self.assertEqual(events[1], ("del", target + 2, insert[target + 1 : target + 4], "."), text) + checked += 1 + self.assertGreater(checked, 20) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_remove_primers.py b/tests/test_remove_primers.py index 428194d..3288f7a 100644 --- a/tests/test_remove_primers.py +++ b/tests/test_remove_primers.py @@ -79,6 +79,9 @@ def rebuild(ref, events): elif kind == "overhang": out.append(ref[done:] + alt_allele) done = len(ref) + elif kind == "unresolved": # the Ref bases from here on are not read: the row's own bases stand there + out.append(ref[done:start] + alt_allele) + done = len(ref) else: out.append(ref[done:start] + alt_allele.replace(".", "")) done = start + len(ref_allele.replace(".", "")) diff --git a/tests/test_unresolved.py b/tests/test_unresolved.py new file mode 100644 index 0000000..4ae038d --- /dev/null +++ b/tests/test_unresolved.py @@ -0,0 +1,494 @@ +"""Tests of the unresolved end of a row. + +A gap near the end of a tag can explain a row with fewer edits than +substitutions and still have too few bases after it to be believed. Such a +row has two readings of its end. ``generate`` takes neither: the row is read +up to the first base the two readings disagree on, its Variants close with +``unresolved=k``, and a target in that stretch is ``Other``. These tests pin +that one decision where it is used: the category, the Variants and their +replay, the table with and without primers, the pooled denoiser, and ``call``. +""" +import contextlib +import csv +import io +import json +import random +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO)) # the repository's modules, never an installed madcats +sys.path.insert(0, str(Path(__file__).resolve().parent)) # the helpers of test_remove_primers + +import madcats as mp # noqa: E402 +import madcats_call as call # noqa: E402 + +BAND = 10 +# 0 1 2 3 4 5 6 7 8 +# 012345678901234567890123456789012345678901234567890123456789012345678901234567890 +REF = "GCTAAAGACAATTACATAACATACACGTCAGCACGAAACTTGTTGGCCCAGTGTGAATCGCTTAAGGGTTAAGTAAGTGTG" +LEFT = 20 # bases of REF that are the left primer + + +def locus_at(target, alt_base=None, index=0, marker="m1"): + """A marker on REF with its target at the 0-based ``target``.""" + alt_base = alt_base or next(base for base in "ACGT" if base != REF[target]) + return mp.Locus(index, marker, REF[:LEFT], "", REF[target], alt_base, target, REF, REF[:target] + alt_base + REF[target + 1 :]) + + +def cell(locus, row, gap_band=BAND): + return mp.format_variants(mp.variant_events(locus, row, gap_band)) + + +def replay(ref, text, row): + """Rebuild a row from the Ref row and its Variants cell, by the rule the documentation gives. + + ``pos:REF>ALT`` replaces REF at POS by ALT (``.`` is no bases) and may not start on a Ref + base an earlier token used. ``clip=k`` and ``unresolved=k`` drop the last k Ref bases; + ``overhang=k`` then appends the row's last k bases, ``unresolved`` the rest of the row. + """ + out, cursor, drop, tail = "", 0, 0, None + for token in (token for token in text.split(";") if token not in ("", ".")): + if token.startswith("clip="): + drop = int(token[5:]) + elif token.startswith("overhang="): + tail = int(token[9:]) + elif token.startswith("unresolved="): + drop, tail = int(token[11:]), "rest" + else: + position, alleles = token.split(":") + removed, added = (allele.strip(".") for allele in alleles.split(">")) + start = int(position) - 1 + assert start >= cursor, f"two tokens use Ref base {position}: {text}" + assert ref[start : start + len(removed)] == removed, f"{token} does not name the Ref bases: {text}" + out += ref[cursor:start] + added + cursor = start + len(removed) + limit = len(ref) - drop + assert cursor <= limit, f"a token reaches into the bases the last one drops: {text}" + out += ref[cursor:limit] + if tail == "rest": + out += row[len(out) :] + elif tail: + out += row[len(row) - tail :] + return out + + +# A Ref haplotype that lost base 71 (0-based 70), with one more base of its amplicon at the end. +ONE_BASE_DELETION = REF[:70] + REF[71:] + "C" +# One that lost bases 73 and 74 (0-based 72 and 73). +TWO_BASE_DELETION = REF[:72] + REF[74:] + "GT" +# A verified two-base deletion after base 25, then one inserted base near the end. +DELETION_THEN_LATE_GAP = "GCTAAAGACAATTACATAACATACATCAGCACGAAACTTGTTGGCCCAGTGTGAATCGCTTAAGGGTTCAGGTAAGTGTGT" + + +class DecisionTests(unittest.TestCase): + def test_the_modules_under_test_are_the_repository_s(self): + self.assertEqual(Path(mp.__file__).resolve().parent, REPO) + self.assertEqual(Path(call.__file__).resolve().parent, REPO) + + def test_target_before_the_unread_end_keeps_its_category(self): + locus = locus_at(40) + # Nine bases follow the deletion, three fewer than a gapped reading needs. The gap costs + # two edits and substitutions six, so the row's end has two readings and is not read. + self.assertEqual(mp.tag_mismatches(TWO_BASE_DELETION, REF), 6) + decision = mp.row_decision(locus, TWO_BASE_DELETION, BAND) + self.assertIsNone(decision.reading) + self.assertEqual(decision.unresolved, (72, 72)) + # The target, base 41, is read where both readings put it. + self.assertEqual(mp.classify_sequence(locus, TWO_BASE_DELETION, BAND), "RefMatch") + events = mp.variant_events(locus, TWO_BASE_DELETION, BAND) + self.assertEqual(events, [("unresolved", 73, REF[72:], TWO_BASE_DELETION[72:])]) + self.assertEqual(mp.format_variants(events), "unresolved=9") + self.assertEqual(mp.marker_type_label(locus, TWO_BASE_DELETION, BAND, events), "unresolved") + + def test_target_in_the_unread_end_is_other(self): + # The target is base 77. The deletion moves base 78, a T, onto it, and T is the Alt allele: + # read at the fixed offset this Ref haplotype is an Alt allele, read through the gap it is + # a Ref allele. Ten bases follow the gap, so neither reading is taken. + locus = locus_at(76, alt_base="T") + self.assertEqual((REF[76], REF[77]), ("G", "T")) + self.assertEqual(ONE_BASE_DELETION[76], locus.alt_base) + self.assertEqual(mp.row_decision(locus, ONE_BASE_DELETION, BAND).unresolved, (71, 71)) + self.assertEqual(mp.classify_sequence(locus, ONE_BASE_DELETION, BAND), "Other") + self.assertEqual(cell(locus, ONE_BASE_DELETION), "unresolved=10") + # What each reading alone would say: without gapped readings, the Alt allele ... + self.assertEqual(mp.classify_sequence(locus, ONE_BASE_DELETION, 0), "AltMatch") + # ... and with every cheaper gapped reading believed, the Ref allele. + with mock.patch.object(mp, "GAP_VERIFIED_SUFFIX", 0): + self.assertEqual(mp.classify_sequence(locus, ONE_BASE_DELETION, BAND), "RefMatch") + self.assertEqual(cell(locus, ONE_BASE_DELETION), "70:TA>T;overhang=1") + + def test_a_gap_with_enough_bases_after_it_is_read_as_before(self): + locus = locus_at(76, alt_base="T") + row = locus.alt_tag[:30] + locus.alt_tag[32:] + "GT" # an Alt haplotype without bases 31 and 32 + decision = mp.row_decision(locus, row, BAND) + self.assertIsNotNone(decision.reading) + self.assertIsNone(decision.unresolved) + self.assertEqual(mp.classify_sequence(locus, row, BAND), "AltMatch") + self.assertEqual(cell(locus, row), "30:AGC>A;77:G>T;overhang=2") + + def test_substitutions_near_the_end_stay_substitutions(self): + locus = locus_at(40) + row = REF[:78] + "A" + REF[79:] + self.assertEqual(mp.row_decision(locus, row, BAND), mp.RowDecision(None, None)) + self.assertEqual(cell(locus, row), "79:G>A") + self.assertEqual(mp.marker_type_label(locus, row, BAND), "snp") + + def test_a_verified_gap_before_the_unread_end_is_kept(self): + # Against the Alt tag the row is the two-base deletion alone, verified; against the Ref tag + # a second gap near the end is cheaper. The two agree up to base 70: the deletion is + # written, the rest is not read, and the row's 11 last bases stand for the Ref row's 11. + locus = locus_at(70) + decision = mp.row_decision(locus, DELETION_THEN_LATE_GAP, BAND) + self.assertIsNotNone(decision.reading) + self.assertEqual(decision.unresolved, (70, 68)) + self.assertEqual(mp.classify_sequence(locus, DELETION_THEN_LATE_GAP, BAND), "Other") + events = mp.variant_events(locus, DELETION_THEN_LATE_GAP, BAND) + self.assertEqual( + events, [("del", 25, "ACG", "A"), ("unresolved", 71, REF[70:], DELETION_THEN_LATE_GAP[68:])] + ) + self.assertEqual(mp.format_variants(events), "25:ACG>A;unresolved=11") + self.assertEqual(replay(REF, "25:ACG>A;unresolved=11", DELETION_THEN_LATE_GAP), DELETION_THEN_LATE_GAP) + # A target before the stretch is read through the verified deletion. + earlier = locus_at(40) + self.assertEqual(mp.classify_sequence(earlier, DELETION_THEN_LATE_GAP, BAND), "RefMatch") + + def test_canonical_rows_and_rescue_off_have_nothing_unresolved(self): + locus = locus_at(76, alt_base="T") + for row in (locus.ref_tag, locus.alt_tag): + self.assertEqual(mp.row_decision(locus, row, BAND), mp.RowDecision(None, None)) + self.assertEqual(mp.row_decision(locus, ONE_BASE_DELETION, 0), mp.RowDecision(None, None)) + self.assertNotIn("unresolved", cell(locus, ONE_BASE_DELETION, 0)) + + def test_marker_designed_as_an_indel_marks_the_unread_end_in_ref_coordinates(self): + # The Alt tag lacks base 41, so every row is written through gaps against the Ref tag. + # A designed deletion does not settle a second gap near the row's end: that end is + # unread on such a marker too, counted on the Ref tag whichever tag the row was read from. + alt = REF[:40] + REF[41:] + "A" + locus = mp.Locus(0, "m1", REF[:LEFT], "", REF[40], alt[40], 40, REF, alt) + self.assertEqual(mp.format_variants(mp._designed_events(REF, alt, BAND)), "39:CT>C;overhang=1") + # A Ref haplotype without bases 73 and 74. + self.assertEqual(mp.unresolved_end(locus, TWO_BASE_DELETION, BAND), (72, 72)) + self.assertEqual(cell(locus, TWO_BASE_DELETION), "unresolved=9") + # The same two bases missing from an Alt haplotype: Alt base 72 is Ref base 73. + on_alt = alt[:71] + alt[73:] + "GT" + self.assertEqual(mp.row_decision(locus, on_alt, BAND).unresolved, (71, 71)) + self.assertEqual(mp.unresolved_end(locus, on_alt, BAND), (72, 71)) + self.assertEqual(cell(locus, on_alt), "39:CT>C;unresolved=9") + self.assertEqual(mp.classify_sequence(locus, on_alt, BAND), "AltMatch") # the target lies before it + for row in (TWO_BASE_DELETION, on_alt): + events = mp.variant_events(locus, row, BAND) + self.assertEqual(replay(REF, mp.format_variants(events), row), row) + self.assertEqual(mp.parse_variants(mp.format_variants(events), REF, row), events) + # ``call`` is shown no base in the unread end. + reading = call.RowReading(REF, events) + self.assertIsNone(reading.base_at(77)) + self.assertEqual(reading.base_at(60), REF[59]) + + +class VariantsCellTests(unittest.TestCase): + def rows(self): + """Ref and Alt haplotypes with zero or one gap well inside, then an edit in the last 15 bases.""" + rng = random.Random(5) + locus = locus_at(70) + rows = set() + while len(rows) < 600: + row = rng.choice([locus.ref_tag, locus.alt_tag]) + if rng.random() < 0.5: + cut, lost = rng.randrange(22, 50), rng.randrange(1, 4) + row = row[:cut] + row[cut + lost :] + "".join(rng.choice("ACGT") for _ in range(lost)) + for _ in range(rng.randrange(1, 3)): + at, kind = rng.randrange(66, 81), rng.choice("sdi") + if kind == "s": + row = row[:at] + rng.choice("ACGT") + row[at + 1 :] + elif kind == "d": + row = row[:at] + row[at + 1 :] + rng.choice("ACGT") + else: + row = (row[:at] + rng.choice("ACGT") + row[at:])[:81] + if row not in (locus.ref_tag, locus.alt_tag): + rows.add(row) + return locus, sorted(rows) + + def test_every_row_is_rebuilt_from_its_cell_and_the_cell_parses_back(self): + locus, rows = self.rows() + unresolved = after_a_gap = 0 + for row in rows: + events = mp.variant_events(locus, row, BAND) + text = mp.format_variants(events) + self.assertEqual(replay(REF, text, row), row, text) + self.assertEqual(mp.parse_variants(text, REF, row), events, text) + if "unresolved=" in text: + unresolved += 1 + after_a_gap += len(text.split(";")) > 1 and mp.row_decision(locus, row, BAND).reading is not None + self.assertTrue(text.split(";")[-1].startswith("unresolved="), text) + self.assertEqual(mp.marker_type_label(locus, row, BAND, events), "unresolved") + # The rows do exercise the outcome, with and without a verified gap before it. + self.assertGreater(unresolved, 100) + self.assertGreater(after_a_gap, 10) + + def test_category_and_cell_agree_about_the_target(self): + # A target at or after the first Ref base not read is Other; one before it is read. + _, rows = self.rows() + for target in (40, 70, 76): + locus = locus_at(target) + for row in rows: + stretch = mp.row_decision(locus, row, BAND).unresolved + if stretch is not None and target >= stretch[0]: + self.assertEqual(mp.classify_sequence(locus, row, BAND), "Other") + + def test_the_token_counts_ref_bases_from_the_end(self): + events = mp.parse_variants("25:ACG>A;unresolved=11", REF, DELETION_THEN_LATE_GAP) + self.assertEqual(events[-1], ("unresolved", 71, REF[70:], DELETION_THEN_LATE_GAP[68:])) + self.assertEqual( + mp.parse_variants("unresolved=9", REF, TWO_BASE_DELETION), [("unresolved", 73, REF[72:], TWO_BASE_DELETION[72:])] + ) + + def test_a_token_this_version_does_not_know_is_an_error(self): + for text in ("79:G>A;someday=3", "unresolved=200", "unresolved=x"): + with self.assertRaises(ValueError, msg=text): + mp.parse_variants(text, REF, TWO_BASE_DELETION) + + +class TableTests(unittest.TestCase): + """The table ``generate`` writes: two markers on REF, one row with an unread end at each.""" + + def setUp(self): + self.before = locus_at(40, index=0, marker="before") # target before the unread end + self.inside = locus_at(76, alt_base="T", index=1, marker="inside") # target inside it + self.loci = [self.before, self.inside] + self.rows = { + 0: {self.before.ref_tag: (40, 0), self.before.alt_tag: (0, 30), TWO_BASE_DELETION: (9, 11)}, + 1: {self.inside.ref_tag: (40, 0), self.inside.alt_tag: (0, 30), ONE_BASE_DELETION: (7, 6)}, + } + + def write(self, directory, **options): + results = [] + for sample in range(2): + counts = {(index, tag): reads[sample] for index, rows in self.rows.items() for tag, reads in rows.items() if reads[sample]} + results.append(mp.SampleResult(mp.Sample(f"s{sample}", f"s{sample}", "unused.fastq", {}), counts, {}, {}, {}, {})) + output = Path(directory) / "madc.csv" + summary = mp.write_madc( + output, self.loci, results, mp.EvidenceRule(1, 20, 0, 0, 0, 5, 0.0), + mp.OutputOptions(variants_path=output.with_suffix(".variants.csv"), **options), BAND, + ) + with open(output, newline="") as handle: + table = {(row["CloneID"], row["AlleleSequence"]): row for row in csv.DictReader(handle)} + sidecar = [] + if output.with_suffix(".variants.csv").exists(): + with open(output.with_suffix(".variants.csv"), newline="") as handle: + sidecar = [row for row in csv.DictReader(handle) if row["Type"] == "unresolved"] + return summary, table, sidecar + + TALLY = {"rows": 2, "reads": 33, "rows_target_not_read": 1, "reads_target_not_read": 13} + + def test_table_with_primers(self): + with tempfile.TemporaryDirectory() as directory: + summary, table, sidecar = self.write(directory) + before, inside = table[("before", TWO_BASE_DELETION)], table[("inside", ONE_BASE_DELETION)] + self.assertEqual((before["AlleleID"], before["Variants"], before["MarkerType"]), ("before|RefMatch", "unresolved=9", "unresolved")) + self.assertEqual((inside["AlleleID"], inside["Variants"], inside["MarkerType"]), ("inside|Other", "unresolved=10", "unresolved")) + self.assertEqual((before["s0"], before["s1"], inside["s0"], inside["s1"]), ("9", "11", "7", "6")) + self.assertEqual( + [(row["CloneID"], row["Position"], row["Ref"], row["Alt"]) for row in sidecar], + [("before", "73", REF[72:], TWO_BASE_DELETION[72:]), ("inside", "72", REF[71:], ONE_BASE_DELETION[71:])], + ) + self.assertEqual(summary["unresolved_ends"], self.TALLY) + self.assertEqual(summary["variants"]["events"], {"snp": 2, "unresolved": 2}) + self.assertEqual(summary["marker_types"]["unresolved"], 2) + self.assertEqual(summary["categories"], {"Ref": 2, "Alt": 2, "RefMatch": 1, "Other": 1}) + + def test_table_without_annotations_has_the_same_categories_and_tally(self): + with tempfile.TemporaryDirectory() as directory: + summary, table, sidecar = self.write(directory, annotations=False) + self.assertEqual(table[("before", TWO_BASE_DELETION)]["AlleleID"], "before|RefMatch") + self.assertEqual(table[("inside", ONE_BASE_DELETION)]["AlleleID"], "inside|Other") + self.assertEqual(summary["unresolved_ends"], self.TALLY) + self.assertEqual(sidecar, []) + + def test_table_without_primers_carries_the_decision(self): + # The same decision in the frame of the rows as printed: the categories and the token are + # those of the full tags, positions count from the first base after the left primer. + with tempfile.TemporaryDirectory() as directory: + summary, table, sidecar = self.write(directory, remove_primers=True) + before, inside = table[("before", TWO_BASE_DELETION[LEFT:])], table[("inside", ONE_BASE_DELETION[LEFT:])] + self.assertEqual((before["AlleleID"], before["Variants"], before["MarkerType"]), ("before|RefMatch", "unresolved=9", "unresolved")) + self.assertEqual((inside["AlleleID"], inside["Variants"], inside["MarkerType"]), ("inside|Other", "unresolved=10", "unresolved")) + self.assertEqual( + [(row["CloneID"], row["Position"], row["Ref"], row["Alt"]) for row in sidecar], + [("before", "53", REF[72:], TWO_BASE_DELETION[72:]), ("inside", "52", REF[71:], ONE_BASE_DELETION[71:])], + ) + for (marker, sequence), row in table.items(): + ref_row = next(shown for (other, shown), entry in table.items() if other == marker and entry["AlleleID"].endswith("|Ref")) + self.assertEqual(replay(ref_row, row["Variants"], sequence), sequence) + self.assertEqual(summary["unresolved_ends"], self.TALLY) + + def test_gapped_rescue_off_writes_no_unresolved_row(self): + with tempfile.TemporaryDirectory() as directory: + output = Path(directory) / "madc.csv" + results = [mp.SampleResult(mp.Sample("s0", "s0", "unused.fastq", {}), {(1, ONE_BASE_DELETION): 9}, {}, {}, {}, {})] + summary = mp.write_madc(output, self.loci, results, mp.EvidenceRule(1, 20, 0, 0, 0, 5, 0.0), mp.OutputOptions(), 0) + self.assertEqual(summary["unresolved_ends"]["rows"], 0) + self.assertNotIn("unresolved", summary["variants"]["events"]) + + +class PrimerCutTests(unittest.TestCase): + """Rows of a panel whose tags end with the first bases of the right primer.""" + + def trimmed_row(self, target): + """A marker with its target at this insert base, and a row without insert bases 30 and 31. + + One insert base and ten primer bases follow the gap on the full tag, so the row's end is + unread from insert base 30. The row's primer is found two bases early and cut off, which + leaves the row two bases shorter than the Ref row.""" + import test_remove_primers as primers + + rng = random.Random(11) + left, insert, right = primers.dna(rng, primers.LEFT), primers.dna(rng, primers.INSERT), primers.dna(rng, primers.RIGHT) + alt_insert = primers.substituted(insert, target) + tag = lambda body: (left + body + right + primers.ADAPTER)[: primers.TAG] + locus = mp.Locus(0, "m", left, right, insert[target], alt_insert[target], primers.LEFT + target, tag(insert), tag(alt_insert)) + row = tag(insert[:29] + insert[31:]) + self.assertEqual(mp.unresolved_end(locus, row, BAND), (49, 49)) + category = mp.classify_sequence(locus, row, BAND) + shown, _, _, readings = mp.trim_locus_rows( + locus, [(1, "Ref", locus.ref_tag), (2, "Alt", locus.alt_tag), (3, category, row)], + {locus.ref_tag: [5], locus.alt_tag: [5], row: [5]}, (primers.LEFT + primers.INSERT, right), BAND, mock.MagicMock(), + ) + trimmed = insert[:29] + insert[31:] + self.assertEqual((len(shown.ref_tag), len(trimmed)), (32, 30)) + self.assertIn(trimmed, readings) + text = mp.format_variants(mp._describe(shown.ref_tag, trimmed, BAND, *readings[trimmed])) + self.assertEqual(replay(shown.ref_tag, text, trimmed), trimmed) + return category, readings[trimmed], text + + def test_row_cut_at_its_primer_to_another_length_keeps_its_unread_end(self): + # The target, insert base 11, is read: the category stands. The row's other length shows + # that something is missing, but not which reading of its end was right, so the end stays + # unread in the row as printed: the last three Ref bases, where one base of the row stands. + category, reading, text = self.trimmed_row(target=10) + self.assertEqual(category, "RefMatch") + self.assertEqual(reading[2], (29, 29)) + self.assertEqual(text, "unresolved=3") + + def test_row_that_is_other_for_an_unread_target_never_shows_a_reading_of_it(self): + # The target is the last insert base, inside the unread end: the row is Other. Cut at + # its primer, the row must not be written through a deletion that puts a base on it. + category, reading, text = self.trimmed_row(target=31) + self.assertEqual(category, "Other") + self.assertEqual(text, "unresolved=3") + + +class PooledDenoiserTests(unittest.TestCase): + def test_row_with_an_unread_target_is_not_folded_into_an_allele(self): + # The row sits one base closer to the Alt tag than to the Ref tag, because the shifted + # base at the target is the Alt allele. Read at the fixed offset it was an Alt-allele row + # and the pooled pass could fold its reads into the Alt row. Its target is not read now, + # so it belongs to neither allele and stays where it is. + locus = locus_at(76, alt_base="T") + self.assertLess(mp.tag_mismatches(ONE_BASE_DELETION, locus.alt_tag), mp.tag_mismatches(ONE_BASE_DELETION, locus.ref_tag)) + counts = {locus.ref_tag: 10**6, locus.alt_tag: 10**6, ONE_BASE_DELETION: 3} + distance = mp.tag_mismatches(ONE_BASE_DELETION, locus.alt_tag) + self.assertEqual(distance, 8) + self.assertEqual(mp._pooled_moves(locus, counts, (), distance, 2.0, BAND), {}) + # With gapped readings off the same row is an Alt-allele row and is folded. + self.assertEqual(mp._pooled_moves(locus, counts, (), distance, 2.0, 0), {ONE_BASE_DELETION: locus.alt_tag}) + + +class CallTests(unittest.TestCase): + """``call`` on a table with an unresolved row: the row gives no reads to sites in its unread end.""" + + SAMPLES = ["s1", "s2", "s3", "s4"] + + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.dir = Path(self.temporary.name) + self.locus = locus_at(76, alt_base="T") + self.site_row = REF[:78] + "A" + REF[79:] # Ref at the target, G>A at base 79 + self.rows = [ + ("Ref", REF, [30, 12, 0, 20], "."), + ("Alt", self.locus.alt_tag, [0, 14, 25, 0], "77:G>T"), + ("RefMatch", self.site_row, [0, 0, 0, 16], "79:G>A"), + ("Other", ONE_BASE_DELETION, [0, 0, 0, 18], "unresolved=10"), + ] + + def tearDown(self): + self.temporary.cleanup() + + def table(self, variants=True, replace=None): + header = [mp.ANNOTATION_COLUMNS.get(column, column) if variants else column for column in mp.MADC_METADATA_COLUMNS] + path = self.dir / "madc.csv" + with open(path, "w", newline="") as handle: + writer = csv.writer(handle) + writer.writerow(header + self.SAMPLES) + for category, sequence, counts, text in self.rows: + cells = dict.fromkeys(header, ".") + cells.update(AlleleID=f"m1|{category}", CloneID="m1", AlleleSequence=sequence) + if variants: + cells["Variants"] = (replace or {}).get(sequence, text) + writer.writerow([cells[column] for column in header] + counts) + return path + + def records(self, path): + prefix = self.dir / "calls" + captured = io.StringIO() + with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(captured): + mp.main(["call", "--madc", str(path), "--output-prefix", str(prefix), "--jobs", "1", "--vcf", "all"]) + self.stderr = captured.getvalue() + with open(str(prefix) + ".qc.json") as handle: + self.qc = json.load(handle) + records = {} + with open(str(prefix) + call.VCF_FILE_SUFFIX["all"]) as handle: + for line in handle: + if not line.startswith("#"): + fields = line.rstrip("\n").split("\t") + keys = fields[8].split(":") + samples = [dict(zip(keys, column.split(":"))) for column in fields[9:]] + records[int(fields[1])] = (fields[3], fields[4], [(sample["AD"], sample["XD"]) for sample in samples]) + return records + + def test_the_cells_this_table_states_are_what_generate_writes(self): + for category, sequence, _, text in self.rows: + self.assertEqual(cell(self.locus, sequence), text) + if category not in ("Ref", "Alt"): + self.assertEqual(mp.classify_sequence(self.locus, sequence, BAND), category) + + def test_unread_end_gives_no_reads_to_the_target_or_to_sites_in_it(self): + records = self.records(self.table()) + # The target, base 77: the unresolved row is Other, so its 18 reads are for neither allele. + self.assertEqual(records[77], ("G", "T", [("30,0", "0"), ("12,14", "0"), ("0,25", "0"), ("36,0", "18")])) + # The site at base 79 lies in the unread end: the 18 reads show no base there. + self.assertEqual(records[79], ("G", "A", [("30,0", "0"), ("26,0", "0"), ("25,0", "0"), ("20,16", "18")])) + self.assertEqual(sorted(records), [77, 79]) + self.assertEqual(self.qc["markers"]["rows_with_reads_whose_variants_were_not_understood"], 0) + + def test_table_without_a_variants_column_is_read_the_same_way(self): + stated = self.records(self.table()) + self.assertEqual(self.records(self.table(variants=False)), stated) + + def test_fixed_offset_reading_would_have_invented_sites(self): + # What the unread end replaces: read base by base, the row's shifted end is a run of + # substitutions, each a site nobody carries. + events = mp._describe(REF, ONE_BASE_DELETION, BAND, gapped=False) + self.assertEqual(mp.format_variants(events), "72:A>G;73:G>T;74:T>A;76:A>G;77:G>T;78:T>G;79:G>T;80:T>G;81:G>C") + self.assertEqual(cell(self.locus, ONE_BASE_DELETION, 0), mp.format_variants(events)) + + def test_cell_with_a_token_call_does_not_know_leaves_the_row_unread(self): + # A later ``generate`` may write a token this version has never seen. The row is not read + # from its sequence in the cell's place: its reads show no base anywhere, and it is counted. + records = self.records(self.table(replace={self.site_row: "79:G>A;someday=3"})) + self.assertEqual(sorted(records), [77]) # the only row that showed the site at 79 is unread + self.assertEqual(self.qc["markers"]["rows_with_reads_whose_variants_were_not_understood"], 1) + self.assertIn("1 rows with reads have a Variants cell this version of call cannot read", self.stderr) + row = call.MadcRow("m1|RefMatch", "RefMatch", self.site_row, (1,), "79:G>A;someday=3") + self.assertIs(call.table_events(row, REF), call.NOT_UNDERSTOOD) + reading = call.read_row(self.locus, row, BAND) + self.assertFalse(reading.understood) + self.assertIsNone(reading.base_at(79)) + self.assertIsNone(reading.base_at(10)) + + +if __name__ == "__main__": + unittest.main()