Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions docs/src/commands/generate.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ The [Evidence Model](../reference/evidence-model.md) explains stages 2–6 in de
| `--leading-anchor <SEQ\|auto>` | none | Constant sequence before the amplicon in raw deliveries (for DArT, `GATGGATCCAGTGC`). MADCATS cuts each read where the anchor ends, and the anchor must end within 40 bases beyond its own length. `auto` detects it from a sample of up to 24 FASTQs. With one or two FASTQs the detected anchor may include the sample barcode, and the run warns about it |
| `--leading-anchor-mismatches {0,1}` | 1 | Substitutions allowed in the leading anchor. With `1`, MADCATS cuts a read that has no exact copy after its only one-substitution match, and counts it as `reads_leading_anchor_mismatch`. A read with no such match, or more than one, counts as `reads_no_leading_anchor`. Anchors under 12 bases must match exactly. `0` requires an exact match for every anchor, as in 0.13.0 |
| `--primer-mismatches {0,1}` | 1 | Mismatches allowed while assigning a primer/anchor. With `0`, reads from Match haplotypes whose distinguishing SNP falls inside the anchor are lost |
| `--gapped-rescue {on,off}` | `on` | Rescue indel reads and allow accepted gapped alignments to locate the target allele and annotate variants. With `off`, retained indel rows can have incorrect target categories; see [Classification](../reference/evidence-model.md#classification) |
| `--gapped-rescue {on,off}` | `on` | Rescue indel reads and allow accepted gapped alignments to locate the target allele and annotate variants. A row whose end has a cheaper but unverified gap is read up to it and marked `unresolved`. With `off`, retained indel rows can have incorrect target categories; see [Classification](../reference/evidence-model.md#classification) |
| `--max-indel-length <INT>` | 10 | Widest gap (alignment band) a gapped alignment may use |
| `--tile-groups {auto,declared,off}` | `auto` | Link tiled markers (markers that read the same tag window of one amplicon), so a read is credited to every tile it fits instead of being dropped as a tie. `auto` links tiles declared with `LocusGroup` and detects undeclared ones, `declared` links only the declared ones, and `off` links none. See [Tiled Markers](../getting-started/input-formats.md#tiled-markers) |
| `--partial-read-mode {on,off}` | `on` | Add a read shorter than the tag to the one full-length allele (already passing the evidence rules) whose start it matches, only when it is longer than every prefix shared between that locus's alleles. `off` drops such reads |
Expand Down Expand Up @@ -126,11 +126,14 @@ classified on the full tags, as with `off`, and a row keeps the category it was
read is gained or lost.

The primer-removal option alone leaves the table unchanged when set to `off`. This is not a
blanket guarantee of v0.13 output: tile-group crediting was added after v0.13 and defaults to
`--tile-groups auto`, and a leading anchor now matches with one substitution by default. For
comparisons with v0.13, use the same panel, reads and analysis settings, plus
`--tile-groups off --remove-primers off`, and `--leading-anchor-mismatches 0` where reads are cut
at a leading anchor. Version/provenance metadata still differs.
blanket guarantee of v0.13 output. Tile-group crediting was added after v0.13 and defaults to
`--tile-groups auto`, and a leading anchor now matches with one substitution by default;
`--tile-groups off --remove-primers off`, with `--leading-anchor-mismatches 0` where reads are cut
at a leading anchor, restores the v0.13 assignment. And a row whose end has a cheaper but
unverified gap is now marked `unresolved`, where v0.13 read it at the fixed offsets (see
[Classification](../reference/evidence-model.md#classification)): about 2 to 3 rows in 100 on the
cohorts measured, of which the ones with the target in that end become `Other`. No option restores
the old reading of those rows. Version/provenance metadata differs too.

What to know before using it:

Expand Down
2 changes: 1 addition & 1 deletion docs/src/development/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ The module follows the order a read travels. Each section opens with a banner co
|---------|----------|
| 0. Constants, errors and data types | `Locus`, `Sample`, `SampleResult`, MADC column definitions |
| 1. Sequence and file utilities | IUPAC matching, mismatch counts, CSV/FASTQ helpers |
| 2. Alignment engine | Myers bit-vector prescreen, banded DP, `accepted_gapped_reading` / `row_reading` |
| 2. Alignment engine | Myers bit-vector prescreen, banded DP, `accepted_gapped_reading` / `row_decision` |
| 3. Panel definition | Design-file reconstruction (`build_panel`), reference-tag tables, primer-free inference |
| 4. Platform intake | Pair merging, window location and freezing, tiling and routing, window-scan |
| 5. Per-sample read assignment | `PrimerIndex`, `select_marker`, `process_sample` and its cache |
Expand Down
33 changes: 33 additions & 0 deletions docs/src/development/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,39 @@ Notable changes by version. Measurements behind each change are in the
the left primer. Reads are still assigned, counted, denoised and classified on the full tags,
so a row keeps its category. The QC JSON gains `primer_removal`. The default is `off`, and the
table written with `off` is unchanged.
- `generate` leaves the end of a row unresolved when a gapped alignment explains the row with
fewer edits than substitutions but has fewer than 12 bases after its last gap. Such a row was
read at the fixed offsets: its end was written as substitutions, and an insertion or deletion
before the target could put the other allele's base on the target. It is now read up to the
first base at which the two explanations differ. A target at or after that base is `Other`.
`Variants` closes with the new token `unresolved=k` (the last `k` Ref bases are not read in this
row), `MarkerType` has the new value `unresolved`, the sidecar the new `Type` `unresolved`, and
the QC JSON counts the rows under `unresolved_ends`. `--remove-primers on` carries the decision
into the rows as printed. This changes the default table. On pecan, sweetpotato, alfalfa and
blackberry 2.2% to 3.2% of rows get the token (0.7% to 2.0% of reads), and 0.12% to 0.22% of
reads move from `RefMatch` or `AltMatch` to `Other`; none moves to the other allele. On three
sets of simulated haplotypes the rows counted for the wrong target allele fall from 1.2% to
0.13% and the substitutions written that no haplotype carries from 108,592 to 8,312, while 1.0%
fewer rows are counted for the right allele. Rows without the token are written as before,
except a few whose counts change because the pooled denoiser groups rows by category.
- `generate` reads a row from the Alt tag when the Ref tag and the Alt tag explain it through gaps
in the same number of edits and the Alt tag needs fewer separate gaps. Before, the Ref tag's
alignment was used on every such tie. An Alt haplotype with one insertion or deletion beside its
target was then counted as `RefMatch` (the Ref tag with two gaps around the target) or as `Other`
(the target inside a deletion). Such rows are now `AltMatch`. Rows whose two alignments are equal
in edits and in gaps still use the Ref tag's. On pecan, sweetpotato, alfalfa and blackberry 0.03%
to 0.10% of rows change category (0.002% to 0.04% of reads), all of them to `AltMatch`.
- `Variants` of a row whose category was read from the Alt tag is written from that same alignment,
in the Ref tag's coordinates. Before, it was always the row's own alignment to the Ref tag. That
alignment has no reason to put the Alt base on the target: it could split one gap in two, or put
the target inside a deletion, and `Variants` then showed the target unchanged or deleted on an
`AltMatch` row. An insertion or deletion right after a substituted base is written without an
anchor base (`31:G>A;32:CACG>.`), as one on the first base of a tag already was, because the
substitution uses that base; `call` reads this form. `--remove-primers on` writes the rows from
the same reading. On pecan, sweetpotato, alfalfa and blackberry 0.07% to 0.25% of rows get other
`Variants` (28 to 75 rows), and with primers kept no row is left whose `Variants` and category
put different bases on the target. Without primers 16 to 31 rows per panel still do: rows cut at
their right primer to another length than the Ref row. Categories and counts do not change.
- `Variants` writes an insertion or deletion on the very first base of a tag without an anchor
base (`1:.>GG`, `1:TT>.`). Before, it borrowed the base after it, which another event could use
as well, and the row then could not be rebuilt from its `Variants`. A tag that starts with a
Expand Down
37 changes: 32 additions & 5 deletions docs/src/reference/evidence-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,18 +116,45 @@ explains the row with fewer edits, stays within the substitution and gap limits,
at least 12 bases after its last gap, the target base comes from that alignment instead. A target
inside a deletion, or pushed past the end of the read, is `Other`.

The row is aligned to the Ref tag and to the Alt tag, and the alignment with fewer edits is used.
When both need the same number of edits, the one with fewer separate gaps is used. An Alt haplotype
with one insertion beside its target is then read as that, and not as a Ref haplotype with two
insertions around the target. When the two are equal in gaps as well, the Ref tag's is used.

This applies even to a row whose reads passed the substitution-only assignment check. An upstream
insertion moves the target farther along the observed sequence; a deletion moves it earlier.
The `Variants` annotation uses the same decision about whether to read through gaps, and reports
events in the Ref tag's coordinates, with equivalent indel placements left-normalised.
events in the Ref tag's coordinates, with equivalent indel placements left-normalised. A row read
from the Alt tag is written from that alignment too, so its `Variants` list the designed
substitution and its gaps where the category was read, and the two show the same base on the
target.

**An unresolved end.** A gapped alignment can explain a row with fewer edits and still have fewer
than 12 bases after its last gap. The row's end then has two explanations, a gap or a run of
substitutions, and too few bases to tell them apart. `generate` takes neither. It reads the row up
to the first base at which the two explanations differ, and no further:

- a target at or after that base is `Other`, because the two explanations can put different bases
on it;
- a target before it is read as usual;
- the row's `Variants` close with `unresolved=k`, and its `MarkerType` is `unresolved`
(see [Output Formats](output-formats.md));
- the QC JSON counts such rows and their reads under `output_summary.unresolved_ends`.

Reading such a row at the fixed offsets, as versions before this outcome did, writes its end as
substitutions that no sample carries, and an insertion or deletion before the target can put the
base of the other allele on the target. Reading it through the unverified gap is right more often
but invents gaps in rows that have none. The row's own continuation past the tag would settle it,
and a tag does not hold it.

With `--gapped-rescue off`, both classification and annotation use the fixed offsets. Many indel
reads fail assignment, but some still pass the substitution-only identity limit. Those rows can
receive the wrong biological category, and an indel can appear as several substitutions. Turning
rescue off is therefore not an indel-safe way to classify target alleles. Even with rescue on,
a gap that fails the acceptance checks falls back to fixed offsets; repeats and gaps near the
tag end can remain unresolved. Agreement between a category and its annotation is not independent
proof of the biological event.
rescue off is therefore not an indel-safe way to classify target alleles, and it writes no
`unresolved` rows. Even with rescue on, a gap that is no cheaper than substitutions or that fails
the substitution and gap limits is read at the fixed offsets, and so is a deletion longer than
`--max-indel-length`. Agreement between a category and its annotation is not independent proof of
the biological event.

`--remove-primers on` acts after classification and preserves that decision. It changes the
printed sequences and annotation coordinates, not how the target was identified. Identical
Expand Down
35 changes: 30 additions & 5 deletions docs/src/reference/output-formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,11 +45,15 @@ column *positions* are unchanged, so positional consumers see the same 16-column
| `snp` | Substitutions only (the Alt row of a designed SNP is `snp`) |
| `indel` | Gaps only (the Alt row of a designed indel is `indel`) |
| `both` | Substitutions and gaps |
| `unresolved` | The row's end is not read (`unresolved=k` in `Variants`), whatever the row carries before it. A gap there is likely but unverified |

To keep SNP-only microhaplotypes, drop `indel` and `both` rows by this label.
To keep SNP-only microhaplotypes, drop `indel`, `both` and `unresolved` rows by this label.

**`Variants`** lists the row's events against the Ref tag, `;`-separated, in 1-based coordinates on
the tag as printed:
the tag as printed. They come from the alignment the row's category was read from, so the two show
the same base on the target. One exception: in a table without primers, a row cut at its right
primer to another length than the Ref row is aligned end to end, which can differ from the reading
its category came from.

| Token | Meaning |
|-------|---------|
Expand All @@ -58,15 +62,28 @@ the tag as printed:
| `70:TAG>T` | Deletion |
| `clip=4` | An insertion pushed the last 4 Ref bases out of the window |
| `overhang=3` | A deletion pulled 3 downstream bases into the window |
| `unresolved=9` | The last 9 Ref bases are not read in this row. Its end has two explanations, a gap or substitutions, with too few bases after the gap to tell them apart ([Classification](evidence-model.md#classification)). Always the last token |
| `1:.>GG`, `1:TT>.` | An insertion before the very first base, or a deletion that starts on it. No base precedes it, so it is written without an anchor base, and `.` stands for no bases. Here `GG` is inserted before base 1, and bases 1 and 2 are deleted |
| `31:G>A;32:CACG>.`, `31:G>A;32:.>TT` | An insertion or deletion right after a substituted base. The substitution uses that base, so the gap is written without an anchor base as well, from its own first base. Here bases 32 to 35 are deleted, or `TT` is inserted before base 32. It appears on a row that carries the Alt base with a gap directly after it |

An example value is `55:C>CAAAA;91:A>T;clip=4`. The Ref row is `.`, and the Alt row of a designed SNP
is exactly the design (e.g. `46:C>G`).

Each `position:REF>ALT` token reads "replace these Ref bases, starting at this position, by these
bases". `clip=k` drops the last `k` Ref bases, and `overhang=k` appends the row's own last `k`
bases. No two tokens of a row use the same Ref base, so applying them in order rebuilds the row
from the Ref row.
bases. `unresolved=k` drops the last `k` Ref bases too, and then appends whatever of the row has
not been built yet. No two tokens of a row use the same Ref base, so applying them in order
rebuilds the row from the Ref row.

On a marker whose own design is an insertion or deletion, every row is written through gaps
against the Ref row, and the unread end is counted on the Ref row as everywhere else. The designed
insertion or deletion does not settle another gap near the row's end.

A row with `unresolved=k` says nothing about Ref positions from `length - k + 1` on: no
substitution, no insertion or deletion, and no agreement with the Ref row either. A consumer that
counts reads per position should leave the row's reads out there, as it does for `clip`.
A reader that does not know a token should not guess the row from its sequence: the token states
something about the row that the sequence alone does not show.

With `--remove-primers on` the frame is the Ref row as printed, so position 1 is the first base
after the left primer. Whether a row is read through an insertion or deletion is decided on its
Expand All @@ -77,6 +94,11 @@ Every other row is a fixed-length window and keeps `clip` and `overhang`. A row
the Ref row in length by more than `--max-indel-length` is compared base by base from its start,
and the rest is `clip` or `overhang`.

An unresolved end is decided on the full tag as well, and carried: the row keeps `unresolved=k`,
counted on the Ref row as printed, whatever length it was cut to. A right primer found at another
place shows that the row gained or lost bases, not which reading of its end was right. Only an
unread stretch that lay wholly in the primer cut from the Ref row leaves nothing to mark.

## Variants Sidecar

`OUTPUT.variants.csv` (with `--annotations on`) lists the same events one per line, keyed by
Expand All @@ -87,7 +109,7 @@ and the rest is `clip` or `overhang`.
| `CloneID`, `AlleleID`, `AlleleSequence` | Row identity |
| `Position` | 1-based position on the Ref tag |
| `Ref`, `Alt` | Event alleles |
| `Type` | `snp`, `ins`, `del`, `clip` (Ref holds the unobserved bases) or `overhang` (Alt holds the pulled-in bases) |
| `Type` | `snp`, `ins`, `del`, `clip` (Ref holds the unobserved bases), `overhang` (Alt holds the pulled-in bases) or `unresolved` (Position is the first Ref base not read, Ref the Ref bases from it on, Alt the row's bases that stand there) |

## QC / Provenance JSON

Expand All @@ -99,6 +121,9 @@ and the rest is `clip` or `overhang`.
- the tile families (`tile_families`) and the extra credits they add (`tile_extra_credits_*`,
described in [Read Accounting](evidence-model.md#read-accounting))
- row counts by category, by evidence route, and by `marker_types`
- `unresolved_ends`: the rows whose end is not read and their reads, and how many of them have the
target in that end (those rows are `Other`). Counted on the full tags, so it is the same with and
without `--remove-primers on` and with `--annotations off`
- the variants sidecar path
- with `--remove-primers on`, `primer_removal`: the panel's amplicon length, the markers by whether
their right primer was located (and if not, why), the rows that were merged, the rows that lost
Expand Down
Loading
Loading