Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 8 additions & 5 deletions docs/src/commands/generate.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,16 @@ them:
every marker get one more try with a gapped alignment under the same edit limit, so the counts
include undesigned indels. Credit a read that fits several tiled markers of one amplicon equally
to each of them, and drop other ties. Reads shorter than the tag count toward a full-length
allele whose start they match. Cache results per sample.
allele whose start they match. Keep the commonest bases that follow a tag in longer reads, for
stage 6. Cache results per sample.
3. Count exact tag sequences per sample.
4. Apply the evidence rules. Always write canonical Ref and Alt rows, and report any other sequence
only if it passes an evidence route.
5. Denoise. Within each sample, merge rare sequences one edit away from a much more common one
into it. A pooled pass then merges sequences that failed the evidence rules into a parent that
carries the same target allele.
6. Classify each row as Ref, Alt, RefMatch, AltMatch or Other, and annotate it.
6. Classify each row as Ref, Alt, RefMatch, AltMatch or Other, and annotate it. A row whose
target the tag cannot read is read again with the bases that follow the tag.
7. Write the MADC, the QC/provenance JSON and the variants sidecar.

The [Evidence Model](../reference/evidence-model.md) explains stages 2–6 in detail.
Expand Down Expand Up @@ -64,7 +66,7 @@ The [Evidence Model](../reference/evidence-model.md) explains stages 2–6 in de
| `--leading-anchor <SEQ\|auto>` | none | Constant sequence before the amplicon in raw deliveries (for DArT, `GATGGATCCAGTGC`). MADCATS cuts each read where the anchor ends, and the anchor must end within 40 bases beyond its own length. `auto` detects it from a sample of up to 24 FASTQs. With one or two FASTQs the detected anchor may include the sample barcode, and the run warns about it |
| `--leading-anchor-mismatches {0,1}` | 1 | Substitutions allowed in the leading anchor. With `1`, MADCATS cuts a read that has no exact copy after its only one-substitution match, and counts it as `reads_leading_anchor_mismatch`. A read with no such match, or more than one, counts as `reads_no_leading_anchor`. Anchors under 12 bases must match exactly. `0` requires an exact match for every anchor, as in 0.13.0 |
| `--primer-mismatches {0,1}` | 1 | Mismatches allowed while assigning a primer/anchor. With `0`, reads from Match haplotypes whose distinguishing SNP falls inside the anchor are lost |
| `--gapped-rescue {on,off}` | `on` | Rescue indel reads and allow accepted gapped alignments to locate the target allele and annotate variants. A row whose end has a cheaper but unverified gap is read up to it and marked `unresolved`. With `off`, retained indel rows can have incorrect target categories; see [Classification](../reference/evidence-model.md#classification) |
| `--gapped-rescue {on,off}` | `on` | Rescue indel reads and allow accepted gapped alignments to locate the target allele and annotate variants. A row whose end has a cheaper but unverified gap is read up to it and marked `unresolved`, unless the bases after the tag settle it. With `off`, retained indel rows can have incorrect target categories; see [Classification](../reference/evidence-model.md#classification) |
| `--max-indel-length <INT>` | 10 | Widest gap (alignment band) a gapped alignment may use |
| `--tile-groups {auto,declared,off}` | `auto` | Link tiled markers (markers that read the same tag window of one amplicon), so a read is credited to every tile it fits instead of being dropped as a tie. `auto` links tiles declared with `LocusGroup` and detects undeclared ones, `declared` links only the declared ones, and `off` links none. See [Tiled Markers](../getting-started/input-formats.md#tiled-markers) |
| `--partial-read-mode {on,off}` | `on` | Add a read shorter than the tag to the one full-length allele (already passing the evidence rules) whose start it matches, only when it is longer than every prefix shared between that locus's alleles. `off` drops such reads |
Expand Down Expand Up @@ -132,8 +134,9 @@ blanket guarantee of v0.13 output. Tile-group crediting was added after v0.13 an
at a leading anchor, restores the v0.13 assignment. And a row whose end has a cheaper but
unverified gap is now marked `unresolved`, where v0.13 read it at the fixed offsets (see
[Classification](../reference/evidence-model.md#classification)): about 2 to 3 rows in 100 on the
cohorts measured, of which the ones with the target in that end become `Other`. No option restores
the old reading of those rows. Version/provenance metadata differs too.
cohorts measured, of which the ones with the target in that end become `Other` unless the bases
after the tag settle them. No option restores the old reading of those rows. Version/provenance
metadata differs too.

What to know before using it:

Expand Down
3 changes: 2 additions & 1 deletion docs/src/development/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,8 @@ annotation use the same `row_reading`, so those stages always agree.
sequential passes.
- The per-sample cache (`--cache-dir`) is keyed by the panel and every read-assignment parameter,
so downstream thresholds never invalidate it. `CACHE_FORMAT_VERSION` is bumped whenever the
serialized cache or raw-read processing changes.
serialized cache or raw-read processing changes. The cache also holds the commonest bases
after each tag that read continuation can use (`keeps_continuation`).

## Vocabulary

Expand Down
14 changes: 14 additions & 0 deletions docs/src/development/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,20 @@ Notable changes by version. Measurements behind each change are in the
0.13% and the substitutions written that no haplotype carries from 108,592 to 8,312, while 1.0%
fewer rows are counted for the right allele. Rows without the token are written as before,
except a few whose counts change because the pooled denoiser groups rows by category.
- `generate` settles a row whose target lies in an unresolved end with the bases its reads carry
after the tag (read continuation). The read pass keeps the commonest of those bases for such
rows and for the two canonical tags. The row is then decided again on the longer window: it is
read through its gap or as substitutions and written like any other row, or it stays unresolved.
A target that an insertion pushed past the tag stays `Other`. Reads must run at least 12 bases
past the tag, so reads that end at the tag, and reads from `normalize-reads`, give the same table
as before. The QC JSON gains `read_continuation`. The cache format changed: existing caches are
rebuilt once. On three sets of simulated haplotypes the rows counted for the right allele rise
from 91.7% to 94.8% (v0.13: 92.2%) and unread rows fall from 3.2% to 0.06%; rows counted for the
wrong allele stay at 0.14% (v0.13: 1.15%). On pecan 149 rows are settled and 124 of them gain an
allele; on sweetpotato 45 and 31. No other row of those tables changes, and no read count. With
`--remove-primers on`, 7 of the settled pecan rows are cut at their right primer before the
target, like the rows named below. A run takes 4% to 7% longer on these cohorts and the cache
grows by 2% to 3%.
- `generate` reads a row from the Alt tag when the Ref tag and the Alt tag explain it through gaps
in the same number of edits and the Alt tag needs fewer separate gaps. Before, the Ref tag's
alignment was used on every such tie. An Alt haplotype with one insertion or deletion beside its
Expand Down
27 changes: 25 additions & 2 deletions docs/src/reference/evidence-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,31 @@ to the first base at which the two explanations differ, and no further:
Reading such a row at the fixed offsets, as versions before this outcome did, writes its end as
substitutions that no sample carries, and an insertion or deletion before the target can put the
base of the other allele on the target. Reading it through the unverified gap is right more often
but invents gaps in rows that have none. The row's own continuation past the tag would settle it,
and a tag does not hold it.
but invents gaps in rows that have none.

**Read continuation.** A read longer than the tag shows what follows it, and that can settle a
row whose target lies in an unresolved end. `generate` keeps the commonest bases after such a row
and after the two canonical tags, and pools them over the samples. It then reads the row again on
a longer window: the row, the Ref tag and the Alt tag, each followed by its own bases. This is the
same decision on more bases, with three outcomes:

- the gap stands, and the target is read through it;
- the substitutions stand, and the target is read at the fixed offset;
- the longer window settles nothing, and the row stays unresolved.

The read pass keeps these bases from a tag's third read in a sample on, so a sample's first two
reads of a tag are not counted. A tag's continuation counts when 3 of the kept reads and 60% of
them carry it. The window must grow by at least 12 bases, and it must read the added bases with at
most 2 substitutions.

A settled row is written like any other row. Its `Variants` list the gap or the substitutions as
far as the row's own bases go. A target that an insertion pushed past the tag is not in the row,
so the row stays `Other`. A row whose target the tag already reads is never asked, so its category
cannot change.

Read continuation needs reads that run at least 12 bases past the tag. Reads written by
`normalize-reads` end at the tag, so it does not act on them. The QC JSON reports the length
compared and the rows settled under `output_summary.read_continuation`.

With `--gapped-rescue off`, both classification and annotation use the fixed offsets. Many indel
reads fail assignment, but some still pass the substitution-only identity limit. Those rows can
Expand Down
3 changes: 3 additions & 0 deletions docs/src/reference/output-formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,9 @@ unread stretch that lay wholly in the primer cut from the Ref row leaves nothing
- `unresolved_ends`: the rows whose end is not read and their reads, and how many of them have the
target in that end (those rows are `Other`). Counted on the full tags, so it is the same with and
without `--remove-primers on` and with `--annotations off`
- `read_continuation`: the number of bases after the tag that reads were compared over
(`bases_after_tag`, 0 when reads end at the tag), and the rows and reads whose unread target
they settled ([Classification](evidence-model.md#classification))
- the variants sidecar path
- with `--remove-primers on`, `primer_removal`: the panel's amplicon length, the markers by whether
their right primer was located (and if not, why), the rows that were merged, the rows that lost
Expand Down
Loading
Loading