From b5461d852532c936013e597d03a2087e7ee19356 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Sun, 25 Oct 2026 14:46:21 -0400 Subject: [PATCH 01/65] docs: amend the L3/L4 spec to the re-checked L4 design --- docs/design/specs/v2-l3-l4-dataflow.md | 556 +++++++++++++++++------ docs/design/specs/v2-schema-structure.md | 55 ++- 2 files changed, 453 insertions(+), 158 deletions(-) diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index a36eebc..a2ce5bf 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -1,7 +1,8 @@ # Spec: Go L3/L4 dataflow (CFG/CDG/DDG + SDG) with SDK taint (codeanalyzer-go) -**Status:** design, awaiting approval · **Date:** 2026-10-23 · **Repos:** `codeanalyzer-go`, -`python-sdk` · **Supersedes:** the "L3/L4 deferred" entries in `docs/design/roadmap.md`. +**Status:** Phases 0–1 shipped (PR #16); L4 design amended by the 2026-10-25 re-check (§4.16) · +**Date:** 2026-10-23, amended 2026-10-25 · **Repos:** `codeanalyzer-go`, `python-sdk` · +**Supersedes:** the "L3/L4 deferred" entries in `docs/design/roadmap.md`. ## 1. Intent @@ -67,35 +68,46 @@ siblings is required too; and the id grammar has to change for polyglot apps any case is called out. - **Golden rule** (from the v2 migration): keep everything that computes facts; add L3/L4 as new computation; the v2 emitter stays the single serialization point. -- **Substrate:** `golang.org/x/tools v0.46.0` is already a dependency — `go/cfg`, `go/ssa`, - `go/ssa/ssautil`, `go/callgraph/vta`, `go/callgraph/cha` are available with no new module. +- **Substrate:** `golang.org/x/tools v0.52.0` (`go.mod`, `go 1.26.0`) is already a dependency — + `go/cfg`, `go/ssa`, `go/ssa/ssautil`, `go/callgraph/vta`, `go/callgraph/cha` are available with + no new module. VTA's package doc still says "experimental" and carries a TODO on generic + instantiation wrappers, so every VTA call runs under `recover` (§4.8, §4.15). `x/tools/go/pointer` is deprecated: **do not adopt**. - Level flag semantics (`cli-contract.md`): `-a 3` implies `-a 2`; `-a 4` implies `-a 3`; an `-a 3` run must not pay L4 cost; `--emit neo4j` runs at max implemented depth (now 4) and rejects `-a` and `--graph-field-depth`. - python-sdk rules: the Iron Rule (no public accessor changes name/signature/return type); new - capability only as new methods; models are `extra="forbid"` mirrors of v2. + capability only as new methods; models are `extra="ignore"` mirrors of v2 (PR #434's choice, + matching TS/Java since python-sdk #386 — an additive analyzer release parses without an SDK + edit; §7). ## 3. Current state (what exists) -- `codeanalyzer-go` L1/L2 on v2, Neo4j projection live. Analysis loads packages once via - `go/packages` (`NeedSyntax|NeedTypes|NeedTypesInfo|NeedDeps`); the internal v1-shaped model - (`internal/schema/schema.go`) is re-serialized by `internal/syntactic_analysis/v2emit`. -- No per-statement facts survive into the internal model; L3 must work from `pkg.Syntax` + - `pkg.TypesInfo` directly. -- `internal/schema/v2/ids.go` already defines `OrdinalID` (`@:`) and `TagID` - (`@`). `internal/neo4j/schema.go` already reserves the L3/L4 relationships. -- Known defects to fix on the way: `GO_DDG` discriminant is `var` only (must be `(var, prov)` per - CLAUDE.md K1 and siblings); `typeKind` never emits `alias`/`defined`; `is_deferred` is declared but - never populated; L2 call-site resolution re-walks the whole package AST per site (parked memory - `parked-callsite-resolution-perf`) — must be fixed before L3/L4 multiply call-site lookups. -- L2 dispatch: an interface-method call's callee is the interface method (CHA-style); no concrete - targets. This is insufficient for must-have flow (3). -- python-sdk `main` (2.0.0, c125d48) has **no Go support at all** — verified 2026-10-23: no tracked - Go files, `CLDK` has only `java`/`python`/`typescript` factories, the legacy `analysis()` shim - dispatches only those three. Branch `add-go-support` (July 2026, never merged, 332 commits behind) - is an old **v1-shaped** implementation and is obsolete. Since then PR #434 (unmerged) adds v2 - L1/L2 Go support; the SDK work builds on it (§7). +As of 2026-10-25 (code at `3504445`, branch `feat/l3-dataflow` = `feat/l4-sdg`): + +- **Phases 0 and 1 shipped together in PR #16** (§9). L1/L2/L3 on v2, Neo4j projection live through + L3. +- **Phase 0 (live):** `can:///go/…` ids (§4.1, `internal/schema/v2/ids.go`); L2 library calls + with `application.external_symbols` (§4.14); co-positioned call keys `l:c/n`; `is_deferred`; + `typeKind` emits `alias`/`defined`; `GO_DDG` `_k = var|prov`; the per-package call-site index + (`callsite_index.go`) replacing the per-site AST walk; body-node `id` from `-a 1`. +- **Phase 1 (live):** `-a 3` and `--graph-field-depth`; per-callable statement body nodes, `cfg` + (defer chain, panic edges), `cdg`, syntactic `ddg` (`prov:["ssa"]`); `k_limit`; Neo4j + `GO_CFG_NEXT`/`GO_CDG`/`GO_DDG`. Package `internal/dataflow`; join key to the tree is + `dataflow.L3Key{Path, Signature}` (`run.go`); ids minted only in the emitter. +- **Load:** `go/packages` once, `NeedSyntax|NeedTypes|NeedTypesInfo|NeedDeps` — every dependency + (stdlib included) is parsed and type-checked from source. That dominates RSS: the Kubernetes + root peaks at 7.1 GB at `-a 3` (`docs/design/notes/l3-dataflow-metrics.md`). +- **L2 dispatch** is still `go/types`: an interface-method call's callee is the bodiless interface + method (`collectInterfaceMethods`); no concrete targets. Insufficient for must-have flow (3); L4 + adds VTA (§4.8). +- **Not yet built (Phase 2):** `-a 4` is rejected by the CLI; no `LevelSDG` in `internal/options`; + no `--strict`; `--emit neo4j` forces `LevelDataflow` (L3); `v2.Analysis` has no `degraded` and + the package doc says L4 fields are not declared; `ExternalSymbol` has no receiver or body; + `GO_PARAM_IN`/`GO_PARAM_OUT`/`GO_SUMMARY` stay `Reserved`. +- **python-sdk:** Go v2 L1/L2 support is on PR #434 (branch `pr-434-v3`, unmerged); the SDK work + builds on it (§7). It cannot yet parse `-a 3` output (§7 prerequisites). `add-go-support` is + obsolete. ## 4. Decisions @@ -261,8 +273,13 @@ truncated). Match Java's `AccessPath` / Python's `access_paths` semantics: every argument `a` that is pointer-, slice-, map-, chan-, func- or interface-typed, and the receiver if it is a pointer receiver or reference-typed** (pointer, slice, map, chan, func or interface — e.g. `hs.Write(b)` on an interface, `h.Set(k, v)` on a map-typed `http.Header`; the - callee may write through it; Python's suffixed-weak-def precedent). Value-typed struct arguments and - receivers are not weak-defined (copy semantics). A **slice expression** argument (`r.Read(buf[:n])`, + callee may write through it; Python's suffixed-weak-def precedent). **Amended (D1, gap 2):** the + predicate is widened to **"may hold a reference"** — the reference kinds above, or a struct or + array whose fields/elements transitively contain one (cycle-safe). A value struct or array that + may hold a reference (`fill(cfg)` with `cfg.items []T`) weak-defines `a.*` (whole base, not per + field, to stay k-safe). A value struct or array that holds no reference is not weak-defined (copy + semantics). The **same** predicate function is the L4 by-ref `formal_out` rule (§4.6), so caller + weak defs and callee ports agree. A **slice expression** argument (`r.Read(buf[:n])`, `io.ReadFull(r, hdr[:])`, and the destination of `copy(dst[i:], src)` / `clear(s[:n])`) shares its elements with its base, so it weak-defines the base's elements **`base[*]`** — slice, array or pointer-to-array base alike (`&`, `*` and parentheses around the base are stripped). `[*]` rather @@ -298,29 +315,42 @@ truncated). Match Java's `AccessPath` / Python's `access_paths` semantics: Differential check (non-gating, reported): compare scalar def-use against `go/ssa` (the construction guide's answer key) on fixtures; mismatches are triaged, not asserted. -#### L3 precision (known gaps) +#### L3 precision (known gaps) — amended 2026-10-25 (D1) [user] The L3 `ddg` is **syntactic and intraprocedural**. It over-approximates where it models a flow -(weak defs, bidirectional prefix matching), but it is **not refutation-complete**: the absence of -a `ddg` path does not prove the absence of a flow. It does not model: - -- **Writes to captured variables inside closures.** `func(){ x = secret }()` or - `once.Do(func(){ cfg = load() })` writes `x` / `cfg` in the closure's own body; the enclosing - callable sees only a use of the captures at the closure expression, never a def, so a later - read of `x` in the enclosing callable does not link to the closure's write. -- **Value-typed struct arguments or receivers that contain slice, map or pointer fields.** A - value struct is copied (no weak def, §4.5 Calls), but its reference-typed fields alias the - caller's: `fill(cfg)` with `cfg.items []T` may write `cfg.items[*]`, which L3 does not record. -- ~~Deferred calls reading receiver or argument state at exit~~ — **closed at L3, except for - reassignment**: the deferred call node also uses every path it may write through - (`defer w.Flush()` uses `w.*`, `defer fill(b[1:])` uses `b[*]`), so a write after the - registration reaches it. The paths are keyed on the *variable* at exit, not on the value - bound at registration, so a reassignment after the `defer` (`defer fill(b); …; b = o`) can - add a spurious edge, and can kill an earlier def on the normal path. - -The first two are addressed at **L4**: capture formals (`@formal_in`/`@formal_out` for captured -variables, §4.6 closures rule) carry closure writes back to the enclosing callable, and the -points-to delta (§4.10) links writes through reference fields of copied structs. +(weak defs, bidirectional prefix matching). The re-check found that L4 as first designed closed +neither gap 2 (the §4.10 oracle aliases only reference-typed bases, and the by-ref predicate +excluded value structs) nor gap 3. **Decision (D1): all three gaps are closed at L3**, before L4 +port wiring (§9 Phase 2 prerequisites), and L4 still adds capture ports on top: + +- **Gap 1 — writes to captured variables inside closures.** `func(){ x = secret }()` or + `once.Do(func(){ cfg = load() })` writes `x` / `cfg` in the closure's own body. **L3 fix:** the + node containing the literal weak-defines each captured variable the literal (or a literal nested + in it) may write; the write-set is computed syntactically per literal. In sequential code a + closure cannot run before it exists, so this is complete apart from goroutine interleaving (a + non-goal). Expect more `ddg` edges at capture sites (measured on Kubernetes). **L4** then adds a + traceable source through capture ports (§4.6 closures rule). +- **Gap 2 — value structs or arrays with reference fields.** `fill(cfg)` with `cfg.items []T` may + write `cfg.items[*]`. **L3 fix:** the "may hold a reference" predicate (§4.5 Calls) weak-defines + `a.*`. Weak defs drive the `ddg` long tail (`l3-dataflow-metrics.md` takeaway 4), so the change + is gated on a Kubernetes measurement. **L4** reuses the same predicate for by-ref `formal_out` + (§4.6). +- **Gap 3 — deferred operands after reassignment.** The deferred call node already uses every path + it may write through (`defer w.Flush()` uses `w.*`, `defer fill(b[1:])` uses `b[*]`). The real + miss was `defer fill(b); b[0] = s; b = o`: `b = o` kills `b[*]` before the exit, losing the write + `fill` reads. **L3 fix:** every write whose base is an operand base of a registered `defer` in the + same callable also weak-defines `:`, which closes the false negative. L4 does not + touch gap 3. + +**Remaining documented over-approximation:** the deferred call's exit-time uses stay keyed on the +*variable* at exit, not on the value bound at registration (they are the points-to hook), so a +reassignment after the `defer` can still add a **spurious** edge. That is extra edges, not a +dropped flow. + +With these fixes the L3 `ddg` misses no flow it models, but L3 is still **intraprocedural**: a flow +through a callee is not an L3 fact, and the absence of an L3 `ddg` path does not prove the +absence of a flow. A `go/ssa` def-use differential (built with the L4 SSA join, §4.8) is +reported, non-gating. ### 4.6 L4 vertices [call; naming follows Python because the SDK reads it] @@ -328,19 +358,49 @@ Per callable (formals), all with `kind` + `of`, no span: | Vertex | `of` | When | |---|---|---| -| `@formal_in:` | real parameter name; receiver name for the receiver | receiver first (index 0) if method, then declared params in order (unnamed `_`/blank params get `_`) | +| `@formal_in:` | real parameter name; receiver name for the receiver | receiver first (index 0) if method, then declared params in order (unnamed or `_` params: `_`; unnamed or `_` receiver: `_recv` — D6) | | `@formal_in:` | `:` | each variable a closure captures (after declared params) | | `@formal_in:` | `:::` | each package-level variable the callable (transitively, via summary) reads | -| `@formal_out` / `@formal_out:` | `` | **one per result position** — Go's multiple returns: `` for a single result; for >1 results, `@formal_out:` with `of` = the named result's name if named else `:` | -| `@formal_out:` | parameter name | each pointer-/reference-typed parameter (incl. pointer receiver) the callable may write through (by-ref out) | -| `@formal_out:` | `:::` | each package-level variable written | +| `@formal_out` / `@formal_out:` | `` / `:` | **one per result position** — `` for a single result; `:` for each position of a multi-result, **never the user's result names** (D6) | +| `@formal_out:` | parameter name | each parameter (incl. the receiver, pointer or value) satisfying the **shared by-ref predicate** the callable may write through (by-ref out) | +| `@formal_out:` | `:` | each captured variable a closure writes (strong or weak def) | +| `@formal_out:` | `:::` | each package-level variable written (transitively, via summary) | + +**Indexing and order (amended, A9) [call].** Python's `IdentityMap` rule verbatim +(`codeanalyzer-python dataflow/identity.py`): `formal_in` and `actual_in` are always indexed; a bare +`@formal_out` / `/actual_out` **iff that group has exactly one vertex**, else every vertex in +the group is indexed — so a single-result function with a by-ref or global out has +`@formal_out:0` (``), `@formal_out:1`, …. `formal_out` order: results by position, then +by-ref params (receiver first) in declaration order, then captures, then globals sorted by `of`. +`formal_in` order: receiver, declared params, captures, globals sorted (matches `EntryBases`). + +**`of` naming (D6) [call].** +- Unnamed or `_` param: `_`, i = the **declared-param index, receiver excluded** — the same + spelling `external_symbols[].parameters[]` uses. (In-project `parameters[].name` stays `_`; the + vertex `of` is the authority.) +- Unnamed or `_` receiver: `_recv`. +- Captures `:`; globals `:::` (matches the L3 `ddg` base + spelling `pkgpath::Name`; the Go SDK matches on import-path segments, §7). +- **The SDK reads `of`, never an index into `parameters[]`**: `formal_in:0` is the receiver for a + method, so Java's `formal_in:` = `parameters[n]` convention does not hold for Go. + +**By-ref predicate (amended, A3/A10) [call].** By-ref `formal_out` (callee-side `mutated_params`) +uses **the same function** as the L3 caller-side call rule (§4.5 Calls): reference-typed (pointer, +slice, map, chan, func, interface) **or "may hold a reference"** (struct/array transitively +containing one, D1 gap 2), for parameters and the receiver alike — a pointer receiver, and a value +receiver of reference kind (map-typed `http.Header`) or one that may hold a reference. Caller weak +defs and callee ports therefore agree. Per call site with a resolved callee (in-project **or stand-in**, §6): | Vertex | `of` | `parent` | |---|---|---| -| `/actual_in:` | the bound callee formal's name (mirrors the callee's `formal_in:` `of`) | call-site local id | -| `/actual_out` / `…/actual_out:` | mirrors callee's `formal_out` `of` | call-site local id | +| `/actual_in:` | the bound callee formal's name (mirrors the callee's `formal_in:` `of`) | the call key | +| `/actual_out` / `…/actual_out:` | mirrors callee's `formal_out` `of` | the call key | + +Vertices are **per call node** (Java style), keyed on the L1 call key (A11). Python groups actuals +per statement; that is a recorded parity difference the SDK does not depend on. A call node the +CFG pruned (unreachable, kept in `body` with no edges) gets **no** vertices. **Why Python's naming, not the keystone's `$ret`/`argN` [call, the one non-trivial parity choice]:** the SDK's `resolve_value` resolves a source/sink `(name, within)` against `formal_in` `of`, and @@ -349,7 +409,7 @@ With Python's spelling the shared resolver works unchanged; Java/TS's `$ret`/`ar Go-specific translation layer. The `:` multi-result form is a Go-only leaf extension of ``. Sources/sinks address `formal_in` only, so `:` never reaches `value_candidate`; it does reach `body_node_kind` (which labels path hops), where `RETURN_MARKER` -is compared by equality — so an unnamed multi-result hop would be labelled `return` with name +is compared by equality — so a multi-result hop would be labelled `return` with name `:1`. The Go backend normalises this locally before calling the shared helper; shared `commons/` stays untouched. @@ -361,36 +421,69 @@ miss later assignments (false refutation). Rule: - The closure callable gets `@formal_in:` `of:":"` per captured variable and `@formal_out:` `of:":"` per captured variable it **writes** (strong or weak def). -- In the enclosing callable, the `FuncLit` has a **capture site**: the statement containing the - `FuncLit` is the host; vertices `/actual_in:` and `/actual_out:` - (`` = `line:col` of the `func` keyword; `parent` = host statement's local id). - `param_in`/`param_out` connect them to the closure's capture formals. No `summary` edges at a - capture site (it is not a call; flow through the closure is via its own body). +- In the enclosing callable, the `FuncLit` has a **capture site** (amended, A6 — a call's key is + `call.Pos()` = `Fun.Pos()` = `FuncLit.Pos()`, so keying the site at the `func` keyword collides + with the call node of `func(x){…}(v)`, `defer func(){…}()`, `go func(){…}()`): + - **Directly called literal:** the captures bind **at that call node**. Their + `/actual_in:` / `actual_out:` indices continue after the declared arguments' + (matching the closure's `formal_in` order: params, then captures). For `defer func(){…}()` this + is the deferred call node in the exit chain. + - **Any other literal** (stored, passed, returned): the capture site is keyed at the literal's + position, taking the next free `/n` suffix the way `BodyNodes.add` does, so it never moves an + L1 call key (chains such as `func() T {…}().M()` already shift to `l:c/2`). + - The site's vertices carry `parent` = the innermost CFG node that contains the literal (the + host); at a directly called literal, `parent` = the call key, as for any call (§4.6 table). + - `param_in`/`param_out` connect the site to the closure's capture formals. No `summary` edges at + a non-call capture site (it is not a call; flow through the closure is via its own body). - **Flow-insensitive wiring for captured variables** (port wiring, `prov:["reaching-defs"]`): *every* def of a captured variable anywhere in the enclosing callable → the capture site's `actual_in`; the capture site's `actual_out` → *every* use of that variable in the enclosing - callable. This over-approximates timing (covers defer, goroutines and callbacks alike) and is the - same weak-update posture as the rest of the design. + callable, **and** (amended, A7) the enclosing callable's own `@formal_out` port for that variable + when it is a named result, a by-ref param base or a global. Without the latter, + `defer func(){ err = wrap(err) }()` — whose deferred call node only *uses* named results, with + nothing defining them after it — never reaches `@formal_out(err)`: a false refutation on the + commonest Go error idiom. This over-approximates timing (covers defer, goroutines and callbacks + alike) and is the same weak-update posture as the rest of the design. - Calls *of* the closure value (`f := func(){…}; f()`) resolve to the closure callable via VTA at L4 and get ordinary call-site vertices for its declared parameters. Go specifics: - **Variadic** `f(xs ...T)`: excess actuals collapse onto the last formal (weak, as Java). - **Multi-value call as args** `f(g())`: each `g` result binds to the corresponding `f` formal. -- **Method value / expression** (`f := s.M; f(x)`; `T.M(s, x)`): resolved callee is `M`; receiver - binds `formal_in:0`. +- **Method expression** (`T.M(s, x)`): resolved callee is `M`; `s` binds `formal_in:0`. +- **Method value** (amended, A8) (`f := s.M; f(x)`): the receiver is not an argument of the call, + and VTA resolves the call to `M$bound` (unwrapped to `M`, §4.8). For a bound-method callee the + call's **function-value operand** (`f`) feeds `actual_in:0`, and the arguments bind + `formal_in:1…`. L3 already carries `s` into `f` (the `f := s.M` statement uses `s`). - **Global state as extra parameters** (keystone L4 rule): globals read/written are computed in the summary pass and propagated up the call graph, so a caller's `formal_in`/`formal_out` set includes globals its callees touch (Python precedent). -- **Global filter — only globals that can carry data [call, from builder check].** Python emits - 546k `:` formals on one real app; transitive propagation is quadratic at cockroach scale. - A package-level variable is modelled as an extra parameter only if it is **mutable in a way that - can carry a value**: it has at least one write (strong or weak def, incl. a method call through it - or `&v` escaping) in any project function body including `init`, **or** its initializer contains a +- **Transitive globals at hosts (amended, A4) [call].** L3 `@entry` defines only the package vars a + body touches **directly** (not those of nested literals), and a satellite call has an empty + DefUse, so a global G only a callee reads has no def reaching `/actual_in(G)`, and a global + only a callee writes never defines G at the host. Adopt Python's `augmented_facts` + re-solve + (`summaries.py`): at each host statement, add the callee's `global_reads` as uses and its + `global_writes` as **weak** defs; `@formal_in(G)` is the def of G for transitively read globals. + Re-run `BuildDDG` and emit only the **new** edges, `prov:["reaching-defs"]`. Weak defs kill + nothing, so the L3 `ssa` set is untouched and L3 ⊆ L4 holds. +- **Global filter — only globals that can carry data [call, from builder check; amended A5].** + Python emits 546k `:` formals on one real app; transitive propagation is quadratic at + Kubernetes scale. A **project** package-level variable is modelled as an extra parameter only if + it is **mutable in a way that can carry a value**: it has at least one write in any project + function body **including closures and `init`** — a strong def, a **weak def from the call rule** + (`mu.Lock()` on a global with a pointer receiver), or `&v` — **or** its initializer contains a call (e.g. `os.Getenv`). A global initialized only from literals/constants and never written - cannot carry taint, so dropping it removes no flow. Constants and functions are never globals. - This filter is sound under the over-approximation rule; output size is measured in the eval (§8) - and a size budget set from the first measurement. + cannot carry taint, so dropping it removes no flow. **Non-project (library) package vars always + pass the filter** (`os::Args`, `os::Stdin`, `flag` vars — the L3 `@entry` already defines them via + `isPkgVar`); without that they would be unnameable as sources. Constants and functions are never + globals. This filter is sound under the over-approximation rule; `:` volume is measured in + the eval (§8) and a size budget set from the first measurement. +- **Range-over-func (amended, A21; D8) [user].** `for v := range seq` has no `CallExpr` (no L1 call + node), and SSA routes the loop body through a synthetic yield function, so data the iterator + passes to `yield` (a `rows.Scan` result inside a DB iterator) never reaches `v` + interprocedurally. **Not modelled this train:** a program that uses range-over-func gets degraded + code `range_func` (§4.15), and a fixture pins the gap. Modelling `for … range f` as a call site + of `f` is a follow-up (§11). ### 4.7 L4 edges [call] @@ -408,6 +501,12 @@ Go specifics: not an access path); `/actual_out[:i] → host stmt` (`var` = the LHS base it defines). **This wiring is what the SDK's `_ports_carry_dependence` probe checks; without it taint certifies false refutations.** +- **Port host (amended, A11) [call].** Go records the host directly: host = the call node's + `parent` if set, else the call node itself (a bare call, a `go` call, a deferred call — each is + its own CFG node). For a deferred call the host is the deferred call node in the exit chain; its + DefUse already carries `:` plus the weak defs of §4.5. Wiring is Java's coarse + host → `actual_in` (every host use feeds every `actual_in`; sound). Python's reaching-def-matched + variant is optional. A pruned call node (no vertices, §4.6) gets no port wiring. - `summary` (callable scope, local ids): `/actual_in:i → /actual_out[:j]` iff the callee's summary says formal `i` may flow to output `j`. - Edge-only-when-resolved: a call site whose callee cannot be resolved at all (e.g. a call through a @@ -421,9 +520,9 @@ Go specifics: ### 4.8 Dispatch resolution: VTA for L2 and L4 [call] Must-have flow (3) needs concrete targets for interface and function-value calls. Use -`x/tools/go/callgraph/vta` over `go/ssa` (built with `ssautil.AllPackages` from the same -`go/packages` load, `ssa.InstantiateGenerics`), seeded with a CHA call graph -(`callgraph/cha`) — the construction guide's recommended Go oracle. +`x/tools/go/callgraph/vta` over `go/ssa` — the construction guide's recommended Go oracle — with SSA +built per D2 (amended, A12): `ssautil.Packages` over the **same** `go/packages` load, not +`AllPackages`. - **L2 refinement:** `call_graph` gains VTA-derived edges for interface/func-value call sites, `prov` `["vta"]`, merged with existing `["go/types"]` edges via `MergeEdges` (union prov, sum weight). @@ -437,22 +536,48 @@ Must-have flow (3) needs concrete targets for interface and function-value calls `actual_*` have a target for the static edge) and a **pass-through summary** (like a stand-in, §6), so a call whose VTA target set is empty still does not drop flow. When VTA finds concrete targets, `param_in`/`param_out` go to each concrete target **and** to the interface method; the - call-site `summary` edges are the union over targets. -- **SSA/VTA scope [call, from builder check]:** build SSA with `ssautil.AllPackages` over the - loaded packages (dependency SSA is needed for types and wrappers), but give VTA a function set of - **project functions only** (the SSA functions whose position lies in a project file, plus their - anonymous functions), seeded with a CHA graph restricted the same way. VTA needs no `main`/roots, - so libraries work. Library callbacks are therefore not discovered (§6, as intended) and cost stays - bounded by project size. Packages that fail to type-check get no SSA (`AllPackages` yields nil); - see §4.12 degradation. -- **SSA → `can://` mapping [call]:** map an SSA function to a callable by source position: - `fn.Origin()` for generic instantiations; synthetic wrappers (bound `$bound`, thunk `$thunk`, - promoted-method and pointer wrappers from embedding) unwrap to the method they wrap; anonymous - functions (`f$1`) map to the L1 closure whose `FuncLit` position equals `fn.Pos()` (L1 closure - naming `closure_N` stays; the position is the join key). A target that maps to nothing in-project - is an external stand-in (§6) if it has a package path outside the project, else dropped with a - debug log. Build an `ast` position → callable index once (also fixes the parked per-site AST walk). -- **VTA runs only at `-a 4`** (and `--emit neo4j`, which is `-a 4`): SSA construction is the + call-site `summary` edges are the union over targets. **D7:** an interface call *always* binds to + the interface method (pass-through) and every VTA target, even when every target is in-project — + per the anti-goal. Cost: every interface call (`io.Writer`, `error.Error()`) carries an + all-to-all summary. +- **VTA-only externals (amended, A19).** A VTA target outside the project that the `go/types` + resolver never saw (e.g. `(*bytes.Buffer).Write` reached through `io.Writer`) is **added** to + `external_symbols` at `-a 4`, with its stand-in body (§6). Monotone: `-a 3` ⊆ `-a 4`. +- **SSA construction (amended, A12; D2) [call].** The load is + `NeedSyntax|NeedTypesInfo|NeedDeps`, so `AllPackages` would create, and `Build` would compile, + **every dependency body** (stdlib included). Instead: + - `ssautil.Packages(roots, ssa.InstantiateGenerics|ssa.BuildSerially)` over `builder.Pkgs()`: + bodies only for project packages; dependencies come from types. + - Build **per package** (`pkg.Build()`) in our own goroutine under `recover` — `Program.Build` + and VTA panic inside their own goroutines, out of reach of the per-callable `recover`. A panic + degrades (`partial_ssa` / `no_ssa`, §4.15). + - Ill-typed packages (`pkg.IllTyped`) get no SSA and degrade to `partial_ssa`. +- **VTA scope [call, from builder check].** Give `vta.CallGraph(funcs, nil)` a function set of + **project functions only** (the SSA functions of project packages, plus their anonymous + functions). `initial = nil`: VTA uses its internal lazy CHA (no separate `callgraph/cha` seed). + VTA needs no `main`/roots, so libraries work. Library callbacks are therefore not discovered (§6, + as intended) and cost stays bounded by project size. VTA runs under `recover`; a panic degrades to + `no_vta`. Targets are sorted by `can://` id (determinism: never key on SSA names or map order). +- **SSA → L3 join (amended, A12) [call].** By **AST pointer identity**, never by position: + - Function: `fn.Syntax()` == the L3 unit's `Unit.Fn` (`l3units.go`); generic instances via + `fn.Origin()`. This holds because SSA is built from the same `*ast.File`s; positions break under + `//line`, and SSA's anonymous-function numbering (`f$1`) follows evaluation order, not + `ast.Inspect` order (L1 closure naming `closure_N` stays). + - Call instruction: `CallCommon.Pos()` == `CallExpr.Lparen` → the `*ast.CallExpr` → its L1 key. + Never by `line:col`. + - Synthetics unwrap: `$bound` / `$thunk` and promoted-method / pointer wrappers to the method + they wrap; instantiation wrappers to the origin; range-over-func `yield` synthetics fold into + `fn.Parent()`; `init#N` to its `init`; "from type information" synthetics have no syntax and map + to no callable. + - An SSA function or call that maps to nothing in-project is an external stand-in (§6) if it has a + package path outside the project; otherwise it is counted and degrades with `unmapped_ssa` + (§4.15). +- **Mapping to `can://` ids (amended, A13) [call].** The L4 IR keys vertices by `(L3Key, local key)` + and cross-callable edges by `L3Key` pairs and external sigs — no ids in the IR. The emitter mints + global ids from an `L3Key` → callable-id map built in the same walk as `flow.apply` (`emit.go`). + Do not reuse `sigIndex.bySigPath`: it keys on the **nesting** module path, not the declaring file. + (The per-site AST walk the old text tied to this is already fixed by `callsite_index.go`.) +- **VTA runs only at `-a 4`** (and `--emit neo4j`, which is `-a 4` per D4, §4.12): SSA construction is the dominant cost; `-a 2`/`-a 3` keep today's cheap `go/types` resolver (an `-a 3` run must not pay L4 cost). VTA edges therefore appear in `call_graph` only at `-a 4` (still monotone: L3 ⊆ L4). - Unsound spots (reflection, `unsafe`, cgo, plugins): no targets ⇒ handled by the unresolved-call @@ -470,10 +595,20 @@ Context-insensitive. Per callable compute: results, by-ref params, written globals — by forward reachability over the callable's `ddg ∪ cdg` from each input's `@entry`/`formal_in` def, bridging call sites via callees' summaries (`summary` edges). Including `cdg` (implicit flow) matches Python and over-approximates. -- `global_reads`, `global_writes`, `mutated_params` (written through, from the weak-def rule), - propagated up the SCC DAG. +- **What summaries refine (amended, A15).** At statement granularity the host already links every + use to every def (the L3 call rule), so a callee summary adds **no intra-caller precision**; it + contributes only the callee's global footprint (A4, §4.6) and its by-ref writes. Summary edges + will be close to all-to-all — a precision limit, not a bug (Python's stated posture, + `summaries.py`). +- `global_reads`, `global_writes`, `mutated_params` (written through, from the shared by-ref + predicate, §4.6), propagated up the SCC DAG and injected at hosts as uses / weak defs (A4). - No hammock regions (neither sibling built them; recorded as a known simplification). - Stand-ins (§6) contribute their modelled summary (default all-inputs → all-outputs). +- **Unanalysed callables (amended, A16).** A callable whose L3 result was dropped (a contained + panic or a call-count mismatch, `run.go`) has no `ddg`, so taint entering through `param_in` + would stop at its `formal_in`. Such callables get formals and a **pass-through summary** (like an + interface method) and are listed under degraded code `l3_skipped` (§4.15). The same applies to SSA + functions that map to no callable (`unmapped_ssa`). - Caching of summaries in `cache_dir` and SCC-wavefront parallelism: **builder's choice** whether to implement in this train; determinism is mandatory either way. @@ -505,46 +640,94 @@ Fill the reserved overlay, no contract renames: flat span props siblings use if the SDK backend needs them (builder confirms against `PyNeo4jBackend`). - `GO_HAS_BODY_NODE` callable → every body node (incl. synthetic). -- `GO_CFG_NEXT {kind}` `_k=kind`; `GO_CDG`; `GO_DDG {var, prov}` **`_k=var|prov`** (fix the - current `var`-only discriminant); `GO_SUMMARY`; `GO_PARAM_IN {var}` / `GO_PARAM_OUT {var}`. +- `GO_CFG_NEXT {kind}` `_k=kind`; `GO_CDG`; `GO_DDG {var, prov}` `_k=var|prov` (live since + Phase 0/1); `GO_SUMMARY`; `GO_PARAM_IN {var}` / `GO_PARAM_OUT {var}`. **Amended (A17):** today + `schema.go` declares the param rels with no props; Phase 2 adds `var` to both (Python parity, + `codeanalyzer-python neo4j/schema.py`). - Stand-ins (§6) project onto the existing `:GoExternal:GoSymbol:GoCanNode` ghost (same - `can:///@external/…` id, so today's call-graph ghosts and stand-ins are one node), gaining - `signature`/`module` props and `GO_HAS_BODY_NODE` to their `formal_*` vertices. Their vertex ids - are `@formal_in:`, so the SDK's `under_callable` id-prefix rule covers them. - `RelHasBodyNode.From` widens to `{GoCallable, GoExternal}` in `schema.go`. + `can:///@external/…` id, so today's call-graph ghosts and stand-ins are one node; it already + carries `signature`/`module`). Their vertex ids are `@formal_in:`, so the SDK's + `under_callable` id-prefix rule covers them. `GO_HAS_BODY_NODE` already allows a `GoExternal` + source. **Amended (A17):** the projector today creates `:GoExternal` only as call-endpoint ghosts + and never walks an external's body; Phase 2 adds a **stand-in pass** that emits each external's + `body` vertices as `:GoBodyNode` with `GO_HAS_BODY_NODE` from the `:GoExternal`. +- **`degraded_json`** (amended, A17) — a new `:GoApplication` property carrying the envelope's + `degraded[]` (§4.15). - Flip `Reserved:false`; conformance test must see every declared relationship emitted on the fixture; cross-projection gate: Neo4j counts == JSON counts modulo `HAS_*`. -- `--emit neo4j` now runs `-a 4`. +- `--emit neo4j` now runs `-a 4` (D4, §4.12). +- **Lazy Bolt push (amended, A17; D5) [call].** The lazy push diffs per module by `content_hash`, + but L4 facts are non-local: editing a callee changes callers' `summary` edges, actual vertices and + even `formal_in` indices (a transitive global set grows). Policy: at L4 a lazy push **upserts + every L4 row on every push** (additive — adds, never removes). Removing stale L4 rows needs + `--eager`. The README documents this under Neo4j and in Upgrading. App-wide L4 purge-and-rewrite + on a lazy push is deferred (§11). ### 4.12 CLI and envelope [call] - `-a 3`, `-a 4` accepted; `--graph-field-depth ` (default 3, error below `-a 3`, error with `--emit neo4j`). +- **Phase 2 CLI tasks (amended, A14).** Today `-a 4` is rejected, there is no `LevelSDG` in + `internal/options`, no `--strict`, `--emit neo4j` forces `LevelDataflow`, and the + `--graph-field-depth` help says "at -a 3 only". Phase 2 adds `LevelSDG`, accepts `-a 4`, adds + `--strict`, sets the neo4j level per D4, handles `-t` per D9, and updates the help to "L3+". +- **`--emit neo4j` level (D4) [user].** `--emit neo4j` runs at max implemented depth = **L4**, and + `-a` / `--graph-field-depth` with it stay an error — the same as Python and Java v2. (Letting + `--emit neo4j` accept `-a ≤ max` would be a CLI-contract change shared with the siblings; not + decided here.) +- **`--target-files` with L4 (D9) [user].** L4 is whole-program, so a targeted run cannot produce + correct summaries or `param_in` edges. Incremental L4 is deferred to the CLDK-wide + incremental-analysis design (§11). + - `-a 1..3 -t` (JSON): unchanged, still works. + - `-a 4 -t` (JSON): an error, exact text: + `--target-files is not supported at analysis level 4 (whole-program); run without -t, or use -a 3` + - `--emit neo4j -t`: runs at **L3**, not L4, and pushes the targeted modules' L1–L3 rows as today. + It writes no L4 rows: the targeted modules' L4 rows stay as the last full run left them until + the next full run (`--eager` removes stale ones, D5). It logs one warning on stderr, exact text: + `--target-files: projecting the graph at level 3 (level 4 is whole-program); L4 rows keep the last full run's facts until a full run` + - The README documents both rules under Neo4j and in Upgrading. - Envelope: emit `k_limit` at L3+ (Python precedent; harmless superset of the keystone's "at L4"). - `max_level` = requested level. -- `--strict`: fail non-zero when SSA/VTA construction fails at `-a 4` (default: degrade — keep L3, +- **Envelope and tree fields (amended, A18).** `v2.Analysis` gains `degraded` (§4.15); `BodyNode` + gains `of`; `Callable` gains `summary`; `Application` gains `param_in`/`param_out`; + `ExternalSymbol` gains `receiver` and `body` (§6). All absent below `-a 4` (the `schema.go` + package doc's "L4 fields … are not declared here" is retired). +- `--strict`: fail non-zero on any degradation at `-a 4` (§4.15) (default: degrade — keep L3, derived vertices and param edges from `go/types` dispatch, log a warning to stderr, and record - the `degraded` envelope marker, §4.15), Java precedent. + the `degraded` envelope marker), Java precedent. - `-j N`: parallelize L3 per callable and L4 per SCC wavefront if implemented; collect then sort by id; never mint ids or emit inside a worker. Byte-identical to `-j 1`. ### 4.13 Caps [call] No hard node/time caps (neither sibling has them; a cap silently drops edges ⇒ false refutation). -Termination comes from k-limiting and finite summary domains. Memory: build L3 per callable and -discard ASTs-derived scratch after emission; SSA program is shared and freed after L4. - -### 4.14 L2 external calls — fixing the current gap (Phase 0) [user: fix it; details are calls] - -Today the L2 resolver resolves library callees (`internal/semantic_analysis/call_graph.go:105-131`) -and then drops them: no `call_graph` edge, and the emitter leaves the call node's `callee` `null` -(`v2emit/emit.go:207-216`). Python and TS emit these; Java emits them behind `--external-calls` -(always under `--emit neo4j`). Go fixes this at L2: +Termination comes from k-limiting and finite summary domains. + +**Memory (amended, A20; D3) [user: budget].** Today L3 nodes keep their AST (`ir.go` `Stmt`), the +whole envelope is marshalled into one `[]byte` (`core/analyzer.go`), and most RSS is `go/packages` +type info (Kubernetes root: 7.1 GB at `-a 3`). L4 adds SSA bodies for project packages, VTA, +summaries and roughly 1.5–2.5× the body nodes (the ports). +- **Prerequisites:** #27 streaming JSON output and AST release (before the real-app L4 eval). +- L4 needs the ASTs to bind actuals, so each callable's L4 vertices are built **before** its AST + references are dropped. The SSA program is shared and freed after L4. +- **Budget (D3):** `-a 4` peak RSS ≤ **1.5× `-a 3`** on the Kubernetes root, measured by an early + SSA+VTA spike task. If the spike blows it, record it in the ledger and keep going. Loading + dependencies from export data (dropping `NeedDeps` syntax) is a separate follow-up, not this + train (§11). + +### 4.14 L2 external calls (Phase 0, shipped in PR #16) [user: fix it; details are calls] + +Before Phase 0 the L2 resolver resolved library callees (`internal/semantic_analysis/call_graph.go`) +and then dropped them: no `call_graph` edge, and the emitter left the call node's `callee` `null`. +Python and TS emit these; Java emits them behind `--external-calls` (always under +`--emit neo4j`). Go now emits them at L2: - **`application.external_symbols{}`** (the Python/Java name) — one entry per distinct external callee: `id` (`can:///@external//`, §6), `kind:"external"`, `name`, `module` (import path), `signature`. Parameters/return type are added here too (cheap from - `go/types`; span-less `{name, type, is_variadic?}`); the L4 `body{}` vertices come in Phase 2. + `go/types`; span-less `{name, type, is_variadic?}`, unnamed → `_`, receiver excluded). The + receiver is not recorded today (`recordExternal` drops it); Phase 2 adds the optional + `receiver {name, type}` leaf and the L4 `body{}` vertices (§6). An external id may name a library *interface* method (e.g. `@external/io/Writer.Write`), and a method on a generic library type records the declared (origin) signature. - **`call_graph`** keeps edges to external targets (`dst` = external id, `prov:["go/types"]`). @@ -559,29 +742,39 @@ and then drops them: no `call_graph` edge, and the emitter leaves the call node' statement defining the LHS uses `b`. (Builtins likewise: L3 "call uses every argument, defines its LHS".) - This is an intended **L2 output change shipped in Phase 0** (more `call_graph` edges, non-null - `callee` on library calls, a new `external_symbols` map). -- Neo4j: the existing `:GoExternal` ghost path in `project.go` (`callEndpoint`) starts receiving - rows; `GO_RESOLVES_TO` extends to external callees. `GO_CALLS` endpoint set already allows - `GoExternal`. -- Gate update: the L2 fixture gate (`gate_l2_test.go`) currently asserts external callees stay - `null`; it changes to assert the external id plus a matching `external_symbols` entry, and no - dangling endpoint. + `callee` on library calls, a new `external_symbols` map; README Upgrading). +- Neo4j: the `:GoExternal` ghost path in `project.go` (`callEndpoint`) receives these rows; + `GO_RESOLVES_TO` extends to external callees; `GO_CALLS` endpoints include `GoExternal`. +- Gate: the L2 fixture gate (`gate_l2_test.go`) asserts the external id plus a matching + `external_symbols` entry, and no dangling endpoint. ### 4.15 Degradation marker [user] -When `-a 4` cannot build what L4 needs — SSA/VTA construction fails, or some packages failed to -type-check and got no SSA — the analyzer still emits (L3 + derived vertices + `go/types` dispatch) -but marks the envelope: +When `-a 4` cannot build what L4 needs — SSA/VTA construction fails, some packages failed to +type-check and got no SSA, or some callables or constructs were not analysed — the analyzer still +emits (L3 + derived vertices + `go/types` dispatch, with pass-through summaries for unanalysed +callables) but marks the envelope: ```json "degraded": [{"code": "no_vta", "detail": "...", "packages": ["example.com/x/pkg"]}] ``` - Envelope leaf, sibling of `max_level`; **absent** when nothing degraded (absence = no fact). - Codes (open vocabulary, plain strings): `no_ssa` (SSA build failed entirely), `no_vta` (VTA - failed), `partial_ssa` (listed packages had type errors / no SSA), `partial_types` (packages with - `go/packages` errors at any level). `max_level` still states the requested level. -- Projected to Neo4j as a `degraded_json` property on `:GoApplication`. + Entry shape `{code, detail?, packages[]?, callables[]?}` (`callables[]` = `can://` ids). + `max_level` still states the requested level. +- Codes (open vocabulary, plain strings): + - `no_ssa` — SSA build failed entirely. + - `no_vta` — VTA failed (incl. a recovered VTA panic). + - `partial_ssa` — listed packages had type errors (`pkg.IllTyped`) or a recovered per-package + build panic, so no SSA. + - `partial_types` — packages with `go/packages` errors at any level. + - `unmapped_ssa` — SSA functions or call instructions that map to no callable or call node + (§4.8); counted, listed where nameable (A16). + - `l3_skipped` — callables whose L3 result was dropped (contained panic, call-count mismatch); + listed in `callables[]`; they get formals and a pass-through summary (§4.9, A16). + - `range_func` — the program uses range-over-func, which is not modelled (§4.6, D8); listed + in `callables[]`. +- Projected to Neo4j as a `degraded_json` property on `:GoApplication` (§4.11). - `--strict` turns any degradation into a non-zero exit. - **SDK:** the Go backends read it at attach. When present, `taint()` returns **no `exhausted` pairs** and adds an `unresolved` diagnostic naming the degradation (`complete=False`), so a @@ -591,17 +784,44 @@ but marks the envelope: shared `commons/` unchanged). - Go-only leaf now; candidate for the keystone alongside the cross-language taint revisit (§11). +### 4.16 Decisions of the 2026-10-25 re-check + +The L4 design was re-checked against the shipped L1–L3 code (`3504445`) and the siblings; the +amendments are folded into the sections above. Decisions: + +- **D1 (#22) [user]:** all three L3 gaps are closed at L3, and L4 still adds capture ports; the + `go/ssa` def-use differential is built with the L4 SSA join and does not gate (§4.5). +- **D2 [call]:** SSA comes from `ssautil.Packages` over the same load, built per package with + `ssa.InstantiateGenerics|ssa.BuildSerially` under `recover` (§4.8). +- **D3 [user]:** `-a 4` peak RSS ≤ 1.5× `-a 3` on the Kubernetes root, measured by the spike; if + blown, record it in the ledger and keep going; export-data loading is a separate follow-up (§4.13). +- **D4 [user]:** `--emit neo4j` runs at max implemented depth = L4; `-a`/`--graph-field-depth` + with it stay an error, as in Python and Java v2 (§4.12). +- **D5 [call]:** a lazy Bolt push upserts every L4 row on every push; removing stale L4 rows needs + `--eager`, documented in the README (§4.11). +- **D6 [call]:** `of` = `` / `:` (never user names); unnamed or `_` params + `_` (declared-param index, receiver excluded); unnamed or `_` receiver `_recv`; captures + `:`; globals `:::` (§4.6). +- **D7 [call]:** an interface call always binds to the interface method (pass-through summary) + **and** to every VTA target (§4.8). +- **D8 [user]:** range-over-func is not modelled; a program that uses it gets degraded code + `range_func`, and a fixture pins the gap (§4.6, §4.15). +- **D9 [user]:** L4 is whole-program; `-a 1..3 -t` unchanged; `-a 4 -t` (JSON) is an error; + `--emit neo4j -t` projects at L3 with a one-line stderr warning; exact texts and the full rule in + §4.12; incremental L4 deferred to the CLDK roadmap (§11). + ## 5. Flow coverage map (user goal 4 → mechanism) | Flow | Mechanism | Level | |---|---|---| | (1) through calls | formal/actual vertices, `param_in/out`, `summary`, port wiring | L4 | | (3) interface / func-value dispatch | VTA targets → per-target `param_in/out` | L4 | -| (2) fields & pointers | k-limited access paths; weak-def through reference args; type-based may-alias `points-to` delta | L3 + L4 | -| closures | captures as `:` formals; nested callable's own SDG; capture bound at the `FuncLit` statement as `actual_in` (Python precedent) | L4 | +| (2) fields & pointers | k-limited access paths; weak-def through reference args and "may hold a reference" value structs/arrays (D1 gap 2); type-based may-alias `points-to` delta | L3 + L4 | +| closures | L3: the literal's host weak-defines the captures it may write (D1 gap 1). L4: captures as `:` formals; nested callable's own SDG; capture site at the direct call node or the literal (A6), wired flow-insensitively incl. into `@formal_out` (A7) | L3 + L4 | | goroutines | `go f(x)` is a call for data purposes (vertices + edges); no interleaving | L4 | | channels | `ch[*]` container path; cross-goroutine via args/captures/fields | L3 + L4 | | library calls | stand-ins with default pass-through summary | L4 | +| range-over-func iterators | **not modelled** (D8): degraded code `range_func`, fixture pins the gap | — | ## 6. Library stand-ins (sinks/sources naming library functions) [user] @@ -617,14 +837,18 @@ addition now; the cross-language SDK taint revisit is deferred (§11). **extended** on Go with `callables`-like content: each external entry carries `kind:"external"`, `name`, `module` (import path), `signature`, `parameters[]` (real names from `go/types`; unnamed → `_`), `return_type`, and at L4 a `body{}` holding only its `@formal_in:` / - `@formal_out[:i]` vertices. The map itself lands at L2 in Phase 0 (§4.14); Phase 2 adds only - `body`. Reachable by containment from `application` ⇒ keystone-legal. + `@formal_out[:i]` vertices. The map itself landed at L2 in Phase 0 (§4.14). **Amended (A19):** + Phase 2 adds two optional leaves — `receiver {name, type}` (a method's receiver; it gives + `formal_in:0`'s `of`, `_recv` when unnamed) and `body{}`. `ExternalParameter` (span-less, `_`) + fits unchanged. Externals only VTA finds are added at `-a 4` (§4.8; monotone). Reachable by + containment from `application` ⇒ keystone-legal. - **Id:** `can:///@external//` where `` is the existing `calleeSignatureOf` canonical form minus the package path (`DB.Query`, `Sprintf`). Language-neutral `@external` per Python/TS/Java. -- **Vertices:** `formal_in` per receiver+param (`of` = real name), `formal_out` per result - (`` / `:`), plus by-ref `formal_out` for each pointer/slice/map/interface param - and pointer receiver (library may write through them — `json.Unmarshal(data, &v)`, `buf.Write`). +- **Vertices:** `formal_in` per receiver+param (`of` = real name, else `_recv` / `_`, D6), + `formal_out` per result (`` / `:`), plus by-ref `formal_out` for each param and + receiver satisfying the shared by-ref predicate (§4.6) (library may write through them — + `json.Unmarshal(data, &v)`, `buf.Write`). Indexing per §4.6 (A9). - **Stand-in summary (pass-through default):** every `formal_in` → every `formal_out`. Materialized as `summary` edges at each **caller's** call site (`actual_in:i → actual_out:j`), *not* as edges inside the stand-in. The stand-in has no outgoing `param_out` into unrelated callers beyond the @@ -648,10 +872,28 @@ addition now; the cross-language SDK taint revisit is deferred (§11). ## 7. SDK (python-sdk) [user: in scope; Claude may write it] Build on python-sdk PR #434 (L1/L2 Go support: models, ABC, local + Neo4j backends, facade, factory; -unmerged as of 2026-10-23), fixing it first against the v2-only analyzer. `add-go-support` is obsolete. - -- **Models** `cldk/models/go/`: `extra="ignore"` (PR #434's choice, matching TS #386) mirror of Go v2 (module/type/field/callable/body - node/edges/external symbols), mirroring how `cldk/models/python` and `java` model L3/L4. +branch `pr-434-v3`, unmerged as of 2026-10-25), fixing it first against the v2-only analyzer. +`add-go-support` is obsolete. + +- **Phase 3 prerequisites (amended, A22)** — on `pr-434-v3` today: + - `GoType.kind: Literal["struct","interface"]` **rejects** `alias`/`defined`, which the analyzer + emits today (`extra="ignore"` does not help with a Literal mismatch). Fix first, so the SDK can + parse current output at all. + - `GoBodyNode` lacks `id`/`parent`/`of`/`is_deferred`; there is no `external_symbols`/ + `param_in`/`summary`/`degraded`, and no `_ports_carry_dependence`. Add the L3 fields first (so + `-a 3` output parses), then L4. + - `extra` policy is reconciled to **`ignore`** (§2 previously said `forbid`). +- **Id splitting (A22).** External ids contain `/@external/` before the vertex `@`, so the Go + backend must **never split a body-node id on its first `@`**; strip the known callable id prefix + instead. +- **Global names (A22).** `value_candidate` turns `:example.com/lib::V` into + `example.com/lib.V`, so a `lib.V` query fails `segment_match` (dot-separated). The Go backend + normalises the import path, or matches on `/` as well (D6 keeps `:::`). +- **`of`, not `parameters[]` (A9).** Go backends resolve formals by `of`; `formal_in:0` is the + receiver for a method. +- **Models** `cldk/models/go/`: `extra="ignore"` (PR #434's choice, matching TS/Java since #386) + mirror of Go v2 (module/type/field/callable/body node/edges/external symbols incl. `receiver` and + `body`, `degraded`), mirroring how `cldk/models/python` and `java` model L3/L4. - **Facade** `GoAnalysis` via `CLDK.go(...)` and the legacy shim; `GoAnalysisBackend` ABC with `N="Go"`, `P="GO_"`. - **Surface:** the decided CLAUDE.md SDK facade (type-centric: `get_types`/`get_type`, @@ -696,6 +938,12 @@ library pass-through and library sink (`database/sql`, `os/exec`, `fmt`); generi `testdata/generics`: instantiation → `Origin()` mapping, generic interface dispatch); an HTTP-handler → SQL-sink end-to-end taint fixture with a sanitizer variant (the "done" fixture). +**Added by the 2026-10-25 re-check (A23):** a Go 1.23+ module (range-over-func — pins the D8 gap — +and range over int; `testdata/dataflow` declares `go 1.21`, #28); a method value (A8); a directly +called literal with captures (A6); a deferred closure that writes a named result (A7); a value +struct holding a slice (D1 gap 2); a forced L3-skipped callable (A16); an interface call whose +concrete target is a library type (A19); one fixture per SSA wrapper kind (§4.8 unwrap list). + **Analyzer gates** (each level gated before advancing — `codeanalyzer-backend` HARD-GATE): CFG gate, dominance gate (exact CDG), PDG-slice gate (exact set), DFG gates, L4 gates (no dangling; arity; summary exists for known transitive flow; `points-to` ⊇ adds-only; ports carry dependence), @@ -713,25 +961,30 @@ fixture, `taint()` finds the flow from a handler's request parameter to `databas `query` through an interface call; a callable sanitizer cuts it; an unrelated pair is `exhausted`; identical answers from the JSON and Neo4j backends. -**Real-world evaluation** (non-gating metrics + crash/determinism/time gates): the prior eval set — -cobra (small), dae, openbao, cockroach (scale) — plus listmonk and memos (HTTP → SQL; memos through -the `store.Driver` interface) and nats-server (concurrency / CFG stress) [user, 2026-10-23]. dae and -cockroach need their code-generation step (bpf2go, protobuf) before analysis. Record per app: wall time and peak RSS at `-a 3`/`-a 4`, body/edge counts, % call -sites with resolved targets (go/types vs VTA), % externals with stand-ins, determinism. Gate: all -four complete `-a 4` without crash and deterministically; no hard time budget, but timings are -reported and a regression budget is set after the first measurement. +**Real-world evaluation** (non-gating metrics + crash/determinism/time gates; amended, A23): the six +apps — cobra (small), dae, openbao, listmonk and memos (HTTP → SQL; memos through the +`store.Driver` interface), nats-server (concurrency / CFG stress) — plus **Kubernetes as the scale +benchmark**, run as two modules: the root module `kubernetes` and `k8s-api` +(`docs/design/notes/l3-dataflow-metrics.md` § Corpus). cockroach is no longer a benchmark. dae +needs its code-generation step (bpf2go) before analysis (`scripts/eval_l3.sh --prepare`). Record per +app: wall time and peak RSS at `-a 3`/`-a 4`, body/edge counts, % call sites with resolved targets +(go/types vs VTA), % externals with stand-ins, `:` formal volume (A5), determinism. Gate: +the six apps and both Kubernetes modules complete `-a 4` without crash and deterministically +(`-j 1`/`-j N` and two runs on Kubernetes, not just fixtures); the D3 memory budget is reported +against (§4.13); no hard time budget, but timings are reported and a regression budget is set after +the first measurement. Results go in `l3-dataflow-metrics.md` § L4. ## 9. Phasing and release | Phase | Repo | Content | Release | -> **Phases 0 and 1 ship together in one PR / one release** [user, 2026-10-23]; they stay separate rows for traceability. +> **Phases 0 and 1 ship together in one PR / one release** [user, 2026-10-23]; they stay separate rows for traceability. **Both have shipped, in PR #16.** |---|---|---|---| -| 0 | analyzer | id grammar `can:///go/…`; **L2 external calls (§4.14)**; L1 co-positioned call keys; `is_deferred`; `typeKind` alias/defined; `GO_DDG` `_k`; call-site position index (perf) | analyzer next **breaking** version (wheel is at 0.1.0 with no local tags — builder confirms the released version and bumps the major, or the minor if still 0.x) | -| 1 | analyzer | L3: body statements, CFG (+defer chain, panic), CDG, syntactic DDG, CLI `-a 3`/`--graph-field-depth`, Neo4j CFG/CDG/DDG, gates | minor | -| 2 | analyzer | L4: SSA+VTA, `degraded` marker (§4.15), external stand-ins (`external_symbols`), vertices, param edges, summaries, port wiring, type-based alias delta, Neo4j L4 overlay, `--emit neo4j` at L4, real-app eval | minor | -| 3 | python-sdk | Go v2 models, facade, local + Neo4j backends, L1–L4 accessors, taint e2e | minor; pins the analyzer only after Phase 2 is released | +| 0 | analyzer | **shipped (PR #16).** id grammar `can:///go/…`; **L2 external calls (§4.14)**; L1 co-positioned call keys; `is_deferred`; `typeKind` alias/defined; `GO_DDG` `_k`; call-site position index (perf) | analyzer next **breaking** version (wheel is at 0.1.0 with no local tags — builder confirms the released version and bumps the major, or the minor if still 0.x) | +| 1 | analyzer | **shipped (PR #16).** L3: body statements, CFG (+defer chain, panic), CDG, syntactic DDG, CLI `-a 3`/`--graph-field-depth`, Neo4j CFG/CDG/DDG, gates | minor | +| 2 | analyzer | **Prerequisites, in order (A24):** (1) #26 shared call/closure walker (incl. `//line` spans), first — returns `(*ast.CallExpr, key)`, `(*ast.FuncLit, closure name)` and call → host maps, replacing the index-paired L1/L3 walks; (2) #17 closure call resolution, after #26 — closures carry a declaring `Path` and `call_graph` gains edges from closures; (3) the #22 L3 fixes (D1, §4.5), then the #28 satellite-`parent` tests and a Go 1.23+ fixture — before L4 port wiring (changing L3 after L4 would re-baseline every L4 gate); (4) #27 streaming JSON and AST release — before the real-app L4 eval, can run in parallel with the L4 core; (5) schema and CLI scaffolding (A14, A18, A19, A17 declarations, a `-a 1..4` monotonicity harness) — the first L4 task. Dependencies: #26 → #17 → (VTA, summaries); #22 fixes → port wiring and gates; scaffolding → all of L4; #27 → the Kubernetes eval. **Then L4:** SSA spike (D3 budget) and build (D2), SSA → L3 join, VTA, `degraded` marker (§4.15), external stand-ins (`external_symbols` `receiver`/`body`), vertices, param edges, summaries, port wiring, type-based alias delta, Neo4j L4 overlay, `--emit neo4j` at L4 (per D4), real-app eval | minor | +| 3 | python-sdk | Go v2 models (first the §7 prerequisites: `GoType.kind`, L3 fields), facade, local + Neo4j backends, L1–L4 accessors, taint e2e | minor; pins the analyzer only after Phase 2 is released | Tracking: `codellm-devkit/.github` parent issue + one sub-issue per PR (Phase 0+1 as one) (filed just-in-time), per `issue-and-pr-tracking.md`. Each phase gets its own implementation plan. @@ -758,12 +1011,19 @@ dispatch + type-based may-alias). - Library callback modelling (`sort.Slice`, `http.HandleFunc`). - Promote `callable.source_file` to the keystone (existing deferral). - Struct-tag modelling (existing deferral). +- **Incremental L4 / incremental Neo4j analysis — CLDK roadmap** (D9): re-analysis after a PR merges + without a whole-program run. +- Range-over-func modelling (D8): `for … range f` as a call site of `f`, wiring data reaching the + iterator's `yield` arguments to the loop variables. +- App-wide L4 purge-and-rewrite on a lazy Bolt push (D5 option (c)). +- Loading dependencies from export data instead of source (drop `NeedDeps` syntax) — a large RSS + cut at every level, but it changes the L1/L2 load path (D3). ## 12. Risks | Risk | Mitigation | |---|---| -| SSA/VTA cost on cockroach-scale | SSA only at `-a 4`; measure in Phase 2 eval; `--strict` / degrade path | +| SSA/VTA cost at Kubernetes scale | SSA only at `-a 4`, project packages only (D2); early SSA+VTA spike sets the D3 budget; #27 streaming; `--strict` / degrade path | | False refutations from a missed construct | over-approximation posture; "ports carry dependence" probe; per-construct fixtures; unresolved-call intra-caller pass-through | | Over-approximation noise (defer chain, pass-through, CDG in summaries) | acceptable by anti-goal; library models file narrows hot spots | | Breaking id change disrupts users | ship alone in Phase 0, before SDK pin; no released SDK consumes Go output | diff --git a/docs/design/specs/v2-schema-structure.md b/docs/design/specs/v2-schema-structure.md index aa7b45a..0b8cc71 100644 --- a/docs/design/specs/v2-schema-structure.md +++ b/docs/design/specs/v2-schema-structure.md @@ -12,7 +12,12 @@ the JSON tags from that file. ├─ schema_version # "2.0.0" ├─ language # "go" ├─ max_level # 1 (symbol table) | 2 (+ call graph) | 3 (+ cfg/cdg/ddg) +│ # | 4 (+ SDG; L4, Phase 2) ├─ k_limit # L3+ only: access-path bound (--graph-field-depth, default 3) +├─ degraded[] # L4, Phase 2; optional: absent when nothing degraded +│ └─ { code, detail?, packages[]?, callables[]? } +│ # code: no_ssa | no_vta | partial_ssa | partial_types | +│ # unmapped_ssa | l3_skipped | range_func (open vocabulary) ├─ analyzer │ ├─ name # "codeanalyzer-go" │ └─ version @@ -45,8 +50,12 @@ the JSON tags from that file. │ └─ │ ├─ src # can:// callable id │ ├─ dst # can:// callable id, or an @external id - │ ├─ prov[] # optional: e.g. ["go/types"] + │ ├─ prov[] # optional: e.g. ["go/types"]; L4 adds ["vta"] (merged) │ └─ weight # optional + ├─ param_in[] # L4, Phase 2; optional: { src, dst, var } — global ids, + │ # actual_in → formal_in; var = the callee formal's `of` + ├─ param_out[] # L4, Phase 2; optional: { src, dst, var } — global ids, + │ # formal_out → actual_out; sorted (src, dst) └─ external_symbols # L2+; optional: { "<@external id>": external } └─ # library callee: `can:///@external//` ├─ id @@ -57,7 +66,11 @@ the JSON tags from that file. │ # library interface method (e.g. "Writer.Write") ├─ parameters[] # { name, type, is_variadic? } (no span: not in-project); │ # receiver excluded; unnamed params are "_" - └─ return_type # optional + ├─ return_type # optional + ├─ receiver # L4, Phase 2; optional: { name, type } of a method's + │ # receiver (gives formal_in:0's `of`; "_recv" if unnamed) + └─ body # L4, Phase 2; optional: stand-in formal vertices only, + # { "@formal_in:" | "@formal_out[:]": vertex } ``` ## `` node (recursive) @@ -91,13 +104,24 @@ The same shape whether it sits under a type's `callables` (methods), a module's │ │ ├─ callee # null | can:// id (key always present) │ │ ├─ is_goroutine # optional: present only when true │ │ └─ is_deferred # optional: present only when true -│ └─ # L3+ -│ ├─ kind # statement | branch | loop | switch | return | entry | exit -│ ├─ id # @ (@entry → @entry) -│ └─ span { start, end, bytes } # the statement; absent on entry / exit +│ ├─ # L3+ +│ │ ├─ kind # statement | branch | loop | switch | return | entry | exit +│ │ ├─ id # @ (@entry → @entry) +│ │ └─ span { start, end, bytes } # the statement; absent on entry / exit +│ └─ # L4, Phase 2: SDG port, span-less +│ ├─ kind # formal_in | formal_out | actual_in | actual_out +│ ├─ id # @ +│ ├─ of # L4: the bound name — param / receiver name, "_", +│ │ # "_recv", "", ":", +│ │ # ":", ":::" +│ └─ parent # actual_* only: the call key; for a non-call closure +│ # capture site, the key of the node containing the literal ├─ cfg[] # L3+, optional: { src, dst, kind } — local body keys ├─ cdg[] # L3+, optional: { src, dst } -└─ ddg[] # L3+, optional: { src, dst, var, prov } (prov = ["ssa"]) +├─ ddg[] # L3+, optional: { src, dst, var, prov } (prov = ["ssa"]; +│ # L4 adds ["reaching-defs"] port wiring, ["points-to"]) +└─ summary[] # L4, Phase 2; optional: { src, dst } — local keys, + # /actual_in: → /actual_out[:] ``` ### L3 body nodes and edges @@ -125,9 +149,15 @@ The same shape whether it sits under a type's `callables` (methods), a module's `prov` is always `["ssa"]` at L3. Sorted by `(src, dst, var)`. It is **not refutation-complete**: writes to captured variables inside closures and writes through reference-typed fields of value-typed struct arguments/receivers are not modelled at L3 - (no edge ≠ no flow); L4 closes them. A deferred call's receiver/argument paths are keyed on - the variable at exit, so a reassignment after the `defer` can add a spurious edge or kill an - earlier def. See `v2-l3-l4-dataflow.md` §4.5 "L3 precision". + (no edge ≠ no flow). The 2026-10-25 re-check decided to close both at L3, plus the deferred-operand + reassignment miss (D1, Phase 2 prerequisite). A deferred call's receiver/argument paths are keyed + on the variable at exit, so a reassignment after the `defer` can add a spurious edge, and (until + the D1 fix lands) kill an earlier def. See `v2-l3-l4-dataflow.md` §4.5 "L3 precision". +- **L4 vertex keys (L4, Phase 2).** Formals `@formal_in:`, `@formal_out` / `@formal_out:`; + actuals `/actual_in:`, `/actual_out` / `…/actual_out:`. `formal_in` and + `actual_in` are always indexed; `formal_out`/`actual_out` are bare iff the group has exactly one + vertex. The SDK reads `of`, never an index into `parameters[]` (`formal_in:0` is a method's + receiver). See `v2-l3-l4-dataflow.md` §4.6–§4.7. - **Spans.** A statement node's span is its AST node's range, and a clause head ends at its colon. For a callable with `source_file` (a method declared apart from its receiver type, or a closure inside one), body spans index `symbol_table[source_file].source`, as the @@ -154,6 +184,11 @@ span - **L3+ only:** `k_limit`; body node `parent`; non-call body nodes; a callable's `cfg` / `cdg` / `ddg`. These are omitted for a callable with no body. Each is a pure addition: the `-a 2` document is a subset of the `-a 3` one. +- **L4 only (Phase 2):** `degraded` (and only when something degraded), `formal_*` / `actual_*` + vertices and their `of`, a callable's `summary`, `param_in` / `param_out`, + `external_symbols[].receiver` / `.body`, `ddg` edges with `prov` `reaching-defs` / `points-to`, + `call_graph` `prov` `vta`. Bodiless interface methods and stand-ins get formals too. Each is a + pure addition: `-a 3` ⊆ `-a 4`. - **`omitempty` — emitted only when non-empty:** `content_hash`, `imports[].alias`, `base_types`, `interfaces`, `fields`, `source_file`, `return_type`, `error_channel`, `metrics`, a callable's nested `callables`, `parameters[].is_variadic`, edge `prov` / From 5d77600a5d294d9313e856adf9e9554f67690a2c Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Sun, 25 Oct 2026 15:20:54 -0400 Subject: [PATCH 02/65] fix: share one call/closure walker; real positions under //line (#26) --- docs/design/specs/v2-schema-structure.md | 4 + internal/astwalk/walk.go | 73 ++++++++++ internal/astwalk/walk_test.go | 127 ++++++++++++++++++ internal/dataflow/bodynodes.go | 79 ++++++----- internal/dataflow/bodynodes_test.go | 66 +++++++-- internal/dataflow/cfg_test.go | 32 ++--- internal/dataflow/ir.go | 5 +- internal/dataflow/run.go | 32 +++-- internal/dataflow/run_test.go | 97 +++++++++---- internal/schema/schema.go | 7 + internal/semantic_analysis/call_graph.go | 4 +- internal/semantic_analysis/callsite_index.go | 17 +-- internal/syntactic_analysis/l3units.go | 52 ++++--- internal/syntactic_analysis/symbol_table.go | 123 ++++++++--------- .../syntactic_analysis/v2emit/gate_l2_test.go | 13 +- .../syntactic_analysis/v2emit/gate_l3_test.go | 30 +++++ .../syntactic_analysis/v2emit/gate_test.go | 67 +++++++++ 17 files changed, 606 insertions(+), 222 deletions(-) create mode 100644 internal/astwalk/walk.go create mode 100644 internal/astwalk/walk_test.go diff --git a/docs/design/specs/v2-schema-structure.md b/docs/design/specs/v2-schema-structure.md index 0b8cc71..bbe6283 100644 --- a/docs/design/specs/v2-schema-structure.md +++ b/docs/design/specs/v2-schema-structure.md @@ -173,6 +173,10 @@ span └─ bytes [from, to] # UTF-8 byte offsets into the owning module.source ``` +`start` / `end` are real file positions (`token.FileSet.PositionFor(p, false)`): `//line` +directives are never applied, so in generated code that carries them (goyacc, templ) every +span, and every body key derived from one, still indexes `module.source`. + ## Presence rules - **Always present:** every `id` (body nodes included, from L1 on — parity with diff --git a/internal/astwalk/walk.go b/internal/astwalk/walk.go new file mode 100644 index 0000000..7a23c5e --- /dev/null +++ b/internal/astwalk/walk.go @@ -0,0 +1,73 @@ +// Package astwalk is the one call-site / closure walker of a callable body, +// shared by L1 (the symbol-table builder records call sites and names +// closures with it) and L3/L4 (the dataflow body builder pairs its call nodes +// with those sites). One walk, so the two levels can never drift apart. It +// imports only the standard library, so any package may use it. +package astwalk + +import ( + "go/ast" + "strconv" +) + +// Call is one call expression of a body, with the go / defer statement flag +// of the statement whose own call it is. +type Call struct { + Expr *ast.CallExpr + Goroutine bool // the call of a `go` statement + Deferred bool // the call of a `defer` statement +} + +// Calls lists body's call expressions in L1 recording order: pre-order — an +// outer call before the calls nested in it, a `go` / `defer` statement's call +// before the calls in its function value and arguments — and no function +// literal bodies (each literal is its own callable; see Literals). A nil body +// has no calls. +func Calls(body *ast.BlockStmt) []Call { + if body == nil { + return nil + } + var out []Call + goroutine := map[*ast.CallExpr]bool{} + deferred := map[*ast.CallExpr]bool{} + ast.Inspect(body, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.GoStmt: + goroutine[n.Call] = true + case *ast.DeferStmt: + deferred[n.Call] = true + case *ast.FuncLit: + return false + case *ast.CallExpr: + out = append(out, Call{Expr: n, Goroutine: goroutine[n], Deferred: deferred[n]}) + } + return true + }) + return out +} + +// Literal is one outermost function literal of a body and its closure name. +type Literal struct { + Lit *ast.FuncLit + Name string // closure_1, closure_2, … +} + +// Literals lists the outermost function literals of body in ast.Inspect +// order, named closure_1, closure_2, …. A literal nested in another is not +// listed: it is the enclosing literal's own (Literals(lit.Body)). A nil body +// has none. +func Literals(body *ast.BlockStmt) []Literal { + if body == nil { + return nil + } + var out []Literal + ast.Inspect(body, func(n ast.Node) bool { + lit, ok := n.(*ast.FuncLit) + if !ok { + return true + } + out = append(out, Literal{Lit: lit, Name: "closure_" + strconv.Itoa(len(out)+1)}) + return false + }) + return out +} diff --git a/internal/astwalk/walk_test.go b/internal/astwalk/walk_test.go new file mode 100644 index 0000000..5ca854a --- /dev/null +++ b/internal/astwalk/walk_test.go @@ -0,0 +1,127 @@ +package astwalk_test + +import ( + "go/ast" + "go/parser" + "go/token" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" +) + +// parseFlow parses testdata/dataflow/flow/flow.go and returns its functions by +// name. +func parseFlow(t *testing.T) (*token.FileSet, map[string]*ast.FuncDecl) { + t.Helper() + _, thisFile, _, _ := runtime.Caller(0) + path := filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "dataflow", "flow", "flow.go") + fset := token.NewFileSet() + f, err := parser.ParseFile(fset, path, nil, parser.SkipObjectResolution) + if err != nil { + t.Fatal(err) + } + decls := map[string]*ast.FuncDecl{} + for _, d := range f.Decls { + if fd, ok := d.(*ast.FuncDecl); ok && fd.Recv == nil { + decls[fd.Name.Name] = fd + } + } + return fset, decls +} + +// describe renders a call as its source text plus its go/defer flags. +func describe(fset *token.FileSet, src []byte, c astwalk.Call) string { + tf := fset.File(c.Expr.Pos()) + s := string(src[tf.Offset(c.Expr.Pos()):tf.Offset(c.Expr.End())]) + if c.Goroutine { + s += " [go]" + } + if c.Deferred { + s += " [defer]" + } + return s +} + +// TestWalk_CallsOrderMatchesL1: Calls lists a body's calls in L1 recording +// order — pre-order (an outer call before the calls nested in it, a go/defer +// statement's call before the calls in its operands), no function literal +// bodies — with the go/defer flag on the statement's own call only. +func TestWalk_CallsOrderMatchesL1(t *testing.T) { + fset, decls := parseFlow(t) + _, thisFile, _, _ := runtime.Caller(0) + src := mustRead(t, filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "dataflow", "flow", "flow.go")) + for fn, want := range map[string][]string{ + "GoArgs": {"f(g(x)) [go]", "g(x)", "func(y int) { _ = y }(h()) [go]", "h()"}, + "Chain": {"s.A().B()", "s.A()"}, + "Nested": {"f(g(y))", "g(y)"}, + "EarlyReturnDefer": {"cleanup() [defer]", "work(g(a)) [defer]", "g(a)", "cleanup()"}, + "Closure": {"h()"}, + } { + d, ok := decls[fn] + if !ok { + t.Fatalf("flow.go has no %s", fn) + } + var got []string + for _, c := range astwalk.Calls(d.Body) { + got = append(got, describe(fset, src, c)) + } + if strings.Join(got, " | ") != strings.Join(want, " | ") { + t.Errorf("%s: Calls = %q, want %q", fn, got, want) + } + } + if got := astwalk.Calls(nil); len(got) != 0 { + t.Errorf("Calls(nil) = %v, want none", got) + } +} + +const literalsSrc = `package p + +func outer(g func(func())) { + a := func() { + _ = func() {} // nested: a's own literal, not outer's + } + g(func() {}) + defer func() {}() + _ = a +} +` + +// TestWalk_LiteralsNaming: Literals returns only the outermost function +// literals of a body, in ast.Inspect order, named closure_1, closure_2, …; +// a literal nested in another is the inner literal's own closure_1. +func TestWalk_LiteralsNaming(t *testing.T) { + fset := token.NewFileSet() + f, err := parser.ParseFile(fset, "p.go", literalsSrc, 0) + if err != nil { + t.Fatal(err) + } + body := f.Decls[0].(*ast.FuncDecl).Body + lits := astwalk.Literals(body) + var got []string + for _, l := range lits { + got = append(got, l.Name+"@"+strings.TrimPrefix(fset.Position(l.Lit.Pos()).String(), "p.go:")) + } + if want := "closure_1@4:7,closure_2@7:4,closure_3@8:8"; strings.Join(got, ",") != want { + t.Errorf("Literals = %v, want %s", got, want) + } + inner := astwalk.Literals(lits[0].Lit.Body) + if len(inner) != 1 || inner[0].Name != "closure_1" || fset.Position(inner[0].Lit.Pos()).Line != 5 { + t.Errorf("nested Literals = %+v, want one closure_1 on line 5", inner) + } + if got := astwalk.Literals(nil); len(got) != 0 { + t.Errorf("Literals(nil) = %v, want none", got) + } +} + +func mustRead(t *testing.T, path string) []byte { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + return data +} diff --git a/internal/dataflow/bodynodes.go b/internal/dataflow/bodynodes.go index cdaafca..c690be0 100644 --- a/internal/dataflow/bodynodes.go +++ b/internal/dataflow/bodynodes.go @@ -6,6 +6,7 @@ import ( "sort" "strconv" + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" ) @@ -34,12 +35,12 @@ import ( // statement gets one only as a label target. Function literal bodies are // skipped: closures are their own callables. // -// calls holds the L1 call-node keys in L1 recording order (v2.CallBodyKeys -// over the callable's call sites). They are paired with the body's call -// expressions by recording index (ruling R5), never by position: under a -// //line directive the recorded line:col does not name the call's token.Pos. -// The caller checks len(calls) == RecordedCallCount(body); surplus keys are -// ignored and calls past the end of the list get no call node. +// calls holds the L1 call nodes: each call-node key (v2.CallBodyKeys over the +// callable's call sites) with the raw CallExpr.Lparen of its site. They are +// paired with the body's call expressions by Lparen — never by index or by +// line:col — so their order is irrelevant. A call no entry names gets no call +// node; entries naming no call of the body are ignored. Run skips a unit +// with an unpaired call rather than build it. // // Keys: call nodes take their L1 key. Statement nodes take // v2.LocalID(line, col) of their anchor; a statement starting with a call @@ -56,7 +57,7 @@ import ( // nodes folded into a node to that node's key: a bare or `go` call's // *ast.CallExpr, an if/for/switch init, a for post, a type switch guard, a comm // clause's statement, and a *ast.LabeledStmt (to its inner statement's node). -func BuildNodes(fset *token.FileSet, body *ast.BlockStmt, calls []string) ([]Node, map[ast.Node]string) { +func BuildNodes(fset *token.FileSet, body *ast.BlockStmt, calls []UnitCall) ([]Node, map[ast.Node]string) { if body == nil { return []Node{}, map[ast.Node]string{} } @@ -66,8 +67,8 @@ func BuildNodes(fset *token.FileSet, body *ast.BlockStmt, calls []string) ([]Nod taken: make(map[string]bool, len(calls)), keyOf: map[ast.Node]string{}, } - for _, k := range calls { - b.taken[k] = true + for _, c := range calls { + b.taken[c.Key] = true } b.stmtList(body.List) @@ -75,7 +76,7 @@ func BuildNodes(fset *token.FileSet, body *ast.BlockStmt, calls []string) ([]Nod out := make([]Node, 0, len(b.nodes)+2) out = append(out, Node{Key: EntryKey, Kind: KindEntry}) for _, n := range b.nodes { - p, e := fset.Position(n.Pos), fset.Position(n.End) + p, e := fset.PositionFor(n.Pos, false), fset.PositionFor(n.End, false) n.StartLC, n.EndLC = [2]int{p.Line, p.Column}, [2]int{e.Line, e.Column} out = append(out, n) } @@ -83,43 +84,39 @@ func BuildNodes(fset *token.FileSet, body *ast.BlockStmt, calls []string) ([]Nod return out, b.keyOf } -// recordedCalls lists body's call expressions in the L1 recording order of -// buildCallSites (internal/syntactic_analysis): pre-order — an outer call -// before the calls nested in it, a `go` / `defer` statement's call before the -// calls in its function value and arguments — and no function literal bodies. -// The two walks must stay identical: keys pair with calls by index (R5). -func recordedCalls(body *ast.BlockStmt) []*ast.CallExpr { - var out []*ast.CallExpr - ast.Inspect(body, func(n ast.Node) bool { - switch n := n.(type) { - case *ast.FuncLit: - return false - case *ast.CallExpr: - out = append(out, n) - } - return true - }) +// RecordedCallCount is the number of L1 call sites body records +// (astwalk.Calls): a cross-check on the length of a unit's calls. +func RecordedCallCount(body *ast.BlockStmt) int { + return len(astwalk.Calls(body)) +} + +// assignCallKeys pairs each recorded call of body with the L1 key whose +// Lparen is the call's. +func assignCallKeys(body *ast.BlockStmt, calls []UnitCall) map[*ast.CallExpr]string { + out, _ := pairCalls(body, calls) return out } -// RecordedCallCount is the number of L1 call sites body records: the length -// the calls argument of BuildNodes must have. -func RecordedCallCount(body *ast.BlockStmt) int { - if body == nil { - return 0 - } - return len(recordedCalls(body)) +// unpairedCalls returns the number of recorded calls of body and how many of +// them no entry of calls names by Lparen. +func unpairedCalls(body *ast.BlockStmt, calls []UnitCall) (recorded, unpaired int) { + paired, recorded := pairCalls(body, calls) + return recorded, recorded - len(paired) } -// assignCallKeys pairs the i-th recorded call with the i-th L1 key. -func assignCallKeys(body *ast.BlockStmt, calls []string) map[*ast.CallExpr]string { - out := map[*ast.CallExpr]string{} - for i, call := range recordedCalls(body) { - if i < len(calls) { - out[call] = calls[i] +func pairCalls(body *ast.BlockStmt, calls []UnitCall) (map[*ast.CallExpr]string, int) { + byLparen := make(map[token.Pos]string, len(calls)) + for _, c := range calls { + byLparen[c.Lparen] = c.Key + } + recorded := astwalk.Calls(body) + out := make(map[*ast.CallExpr]string, len(recorded)) + for _, c := range recorded { + if k, ok := byLparen[c.Expr.Lparen]; ok { + out[c.Expr] = k } } - return out + return out, len(recorded) } type builder struct { @@ -270,7 +267,7 @@ func (b *builder) folded(k string, s ast.Stmt) { // position's LocalID, or the next free "/n" suffix when an L1 call (or an // earlier node) already holds it. func (b *builder) add(kind NodeKind, n ast.Node, pos, end token.Pos) string { - p := b.fset.Position(pos) + p := b.fset.PositionFor(pos, false) base := v2.LocalID(p.Line, p.Column) key := base for i := 2; b.taken[key]; i++ { diff --git a/internal/dataflow/bodynodes_test.go b/internal/dataflow/bodynodes_test.go index e84afb4..256a5d1 100644 --- a/internal/dataflow/bodynodes_test.go +++ b/internal/dataflow/bodynodes_test.go @@ -5,6 +5,7 @@ import ( "go/token" "go/types" "path/filepath" + "reflect" "runtime" "sync" "testing" @@ -63,17 +64,21 @@ func loadFlow(t *testing.T) *flowFixture { } // callKeys builds the L1 call-node keys exactly as the driver does: keys from -// v2.CallBodyKeys over the call sites' [line, col], in recording order. -// BuildNodes pairs them with the AST calls by index (ruling R5), never by -// rebuilding a token.Pos. -func (fx *flowFixture) callKeys(t *testing.T, fn string) []string { +// v2.CallBodyKeys over the call sites' [line, col], in recording order, each +// with its site's Lparen. BuildNodes pairs them with the AST calls by Lparen. +func (fx *flowFixture) callKeys(t *testing.T, fn string) []dataflow.UnitCall { t.Helper() sites := fx.sites[fn] pos := make([][2]int, len(sites)) for i, cs := range sites { pos[i] = [2]int{cs.StartLine, cs.StartColumn} } - return v2.CallBodyKeys(pos) + keys := v2.CallBodyKeys(pos) + out := make([]dataflow.UnitCall, len(sites)) + for i, cs := range sites { + out[i] = dataflow.UnitCall{Key: keys[i], Lparen: cs.Lparen} + } + return out } func (fx *flowFixture) decl(t *testing.T, fn string) *ast.FuncDecl { @@ -222,8 +227,8 @@ func TestBodyNodes_KindsPerConstruct(t *testing.T) { func TestBodyNodes_BareCallSharesKey(t *testing.T) { fx := loadFlow(t) l1 := map[string]bool{} - for _, k := range fx.callKeys(t, "EarlyReturnDefer") { - l1[k] = true + for _, c := range fx.callKeys(t, "EarlyReturnDefer") { + l1[c.Key] = true } _, byKey, keyOf := build(t, "EarlyReturnDefer") n := wantKind(t, "EarlyReturnDefer", byKey, "131:2", dataflow.KindCall) @@ -353,9 +358,9 @@ func TestBodyNodes_CallKeysMatchL1(t *testing.T) { fx := loadFlow(t) for fn := range fx.decls { _, byKey, _ := build(t, fn) - for _, k := range fx.callKeys(t, fn) { - if n, ok := byKey[k]; !ok || n.Kind != dataflow.KindCall { - t.Errorf("%s: L1 call key %s has no call node", fn, k) + for _, c := range fx.callKeys(t, fn) { + if n, ok := byKey[c.Key]; !ok || n.Kind != dataflow.KindCall { + t.Errorf("%s: L1 call key %s has no call node", fn, c.Key) } } } @@ -370,3 +375,44 @@ func TestBodyNodes_BodylessCallableEmitsNothing(t *testing.T) { t.Errorf("nil body: want empty (non-nil) slice and map") } } + +// TestBuildNodes_PairsByLparenNotIndex: BuildNodes pairs each L1 call key +// with the call whose Lparen it carries, so the order of the calls slice is +// irrelevant: a permuted slice keys every call exactly as the recording-order +// slice does — co-positioned chained calls (s.A().B() at l:c, s.A() at +// l:c/2) included. +func TestBuildNodes_PairsByLparenNotIndex(t *testing.T) { + fx := loadFlow(t) + for _, fn := range []string{"EarlyReturnDefer", "GoArgs", "Chain", "Nested", "Spawn"} { + body := fx.decl(t, fn).Body + calls := fx.callKeys(t, fn) + if len(calls) < 2 { + t.Fatalf("%s: %d calls, want ≥ 2 to permute", fn, len(calls)) + } + rev := make([]dataflow.UnitCall, len(calls)) + for i, c := range calls { + rev[len(calls)-1-i] = c + } + nodes, keyOf := dataflow.BuildNodes(fx.fset, body, calls) + nodes2, keyOf2 := dataflow.BuildNodes(fx.fset, body, rev) + if !reflect.DeepEqual(nodes, nodes2) || !reflect.DeepEqual(keyOf, keyOf2) { + t.Errorf("%s: permuted calls build different nodes", fn) + } + want := map[token.Pos]string{} + for _, c := range calls { + want[c.Lparen] = c.Key + } + n := 0 + for node, key := range keyOf2 { + if call, ok := node.(*ast.CallExpr); ok { + n++ + if want[call.Lparen] != key { + t.Errorf("%s: call at %v keyed %s, want %s", fn, fx.fset.Position(call.Pos()), key, want[call.Lparen]) + } + } + } + if n != len(calls) { + t.Errorf("%s: %d call expressions keyed, want %d", fn, n, len(calls)) + } + } +} diff --git a/internal/dataflow/cfg_test.go b/internal/dataflow/cfg_test.go index 90d3bd3..9ae2c9a 100644 --- a/internal/dataflow/cfg_test.go +++ b/internal/dataflow/cfg_test.go @@ -15,6 +15,7 @@ import ( "golang.org/x/tools/go/cfg" "golang.org/x/tools/go/packages" + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" ) @@ -768,28 +769,23 @@ func gateOne(t *testing.T, name string, nodes []dataflow.Node, es []dataflow.Cfg return chain } -// gateCallKeys lists the key of every call in body in the L1 recording order -// (pre-order, an outer call before its nested calls; no function literal -// bodies), as -// v2.CallBodyKeys assigns them from each call's (line-directive adjusted) -// position. -func gateCallKeys(fset *token.FileSet, body *ast.BlockStmt) []string { - var calls []*ast.CallExpr - ast.Inspect(body, func(n ast.Node) bool { - switch n := n.(type) { - case *ast.FuncLit: - return false - case *ast.CallExpr: - calls = append(calls, n) - } - return true - }) +// gateCallKeys lists the L1 call node of every call in body in the L1 +// recording order (astwalk.Calls), each keyed as v2.CallBodyKeys assigns it +// from the call's real (not //line-adjusted) position and carrying its +// Lparen. +func gateCallKeys(fset *token.FileSet, body *ast.BlockStmt) []dataflow.UnitCall { + calls := astwalk.Calls(body) pos := make([][2]int, len(calls)) for i, c := range calls { - p := fset.Position(c.Pos()) + p := fset.PositionFor(c.Expr.Pos(), false) pos[i] = [2]int{p.Line, p.Column} } - return v2.CallBodyKeys(pos) + keys := v2.CallBodyKeys(pos) + out := make([]dataflow.UnitCall, len(calls)) + for i, c := range calls { + out[i] = dataflow.UnitCall{Key: keys[i], Lparen: c.Expr.Lparen} + } + return out } // ---- canPanic rules (spec §4.3.2) ---- diff --git a/internal/dataflow/ir.go b/internal/dataflow/ir.go index 41b2fa2..f0de084 100644 --- a/internal/dataflow/ir.go +++ b/internal/dataflow/ir.go @@ -39,8 +39,9 @@ const ( // - Key is the local key: "l:c" (v2.LocalID of the node's anchor position), // "l:c/2", … for co-positioned nodes, or EntryKey / ExitKey. // - Pos/End is the span of Stmt; NoPos for synthetic nodes. StartLC/EndLC -// are their [line, col] (fset.Position, so //line directives apply, as -// they do to the L1 call-site positions); zero for synthetic nodes. +// are their real [line, col] (fset.PositionFor(p, false): //line +// directives are not applied, as they are not to the L1 call-site +// positions); zero for synthetic nodes. // - Stmt is the AST node the body node stands for: // a statement (*ast.ExprStmt for a bare call, *ast.GoStmt for a `go` call), // a clause head (*ast.CaseClause / *ast.CommClause), diff --git a/internal/dataflow/run.go b/internal/dataflow/run.go index fb3e01e..9d5be5f 100644 --- a/internal/dataflow/run.go +++ b/internal/dataflow/run.go @@ -24,14 +24,22 @@ type L3Key struct{ Path, Signature string } // // - Fn is the *ast.FuncDecl or *ast.FuncLit; a closure is its own unit. // - Info is the type info of Fn's package (nil degrades: no def/use). -// - Calls is the callable's L1 call-node keys in L1 recording order -// (v2.CallBodyKeys over its call sites). BuildNodes pairs them with the -// body's calls by index (ruling R5); Run checks the counts first. +// - Calls is the callable's L1 call nodes: each L1 call-node key +// (v2.CallBodyKeys over its call sites) with its site's CallExpr.Lparen. +// BuildNodes pairs them with the body's calls by Lparen, so their order +// does not matter. type Unit struct { Key L3Key Fn ast.Node Info *types.Info - Calls []string + Calls []UnitCall +} + +// UnitCall is one L1 call node of a unit: its key and the raw Lparen of the +// call expression it was recorded from. +type UnitCall struct { + Key string + Lparen token.Pos } // Run builds the L3 overlay of every unit: the pruned body nodes, CFG, CDG @@ -42,9 +50,11 @@ type Unit struct { // // A unit that panics in any stage is skipped the same way (runOne). // -// A unit whose L1 call-key count differs from its body's recorded call count -// gets no result and a warning on stderr: pairing by index would mis-key its -// call nodes, so its body stays at L1 rather than being silently wrong. +// A unit with a body call that no L1 call key names (by Lparen) gets no +// result and a warning on stderr: a call node without its L1 key would be +// silently wrong, so its body stays at L1. A count mismatch between the L1 +// keys and the body's recorded calls (RecordedCallCount) is a cross-check +// only: it warns, and the unit is still built when every call is paired. func Run(fset *token.FileSet, units []Unit, k, jobs int) map[L3Key]Result { if jobs < 1 { jobs = runtime.GOMAXPROCS(0) @@ -92,8 +102,12 @@ func runOne(fset *token.FileSet, u Unit, k int) (res *Result) { if body == nil { return nil } - if n := RecordedCallCount(body); n != len(u.Calls) { - utils.Warn("L3 skipped for %s (%s): %d L1 call keys for %d calls in the body", u.Key.Signature, u.Key.Path, len(u.Calls), n) + recorded, missing := unpairedCalls(body, u.Calls) + if recorded != len(u.Calls) { + utils.Warn("L3 for %s (%s): %d L1 call keys for %d calls in the body", u.Key.Signature, u.Key.Path, len(u.Calls), recorded) + } + if missing > 0 { + utils.Warn("L3 skipped for %s (%s): %d calls in the body have no L1 call key", u.Key.Signature, u.Key.Path, missing) return nil } nodes, keyOf := BuildNodes(fset, body, u.Calls) diff --git a/internal/dataflow/run_test.go b/internal/dataflow/run_test.go index 0e723c8..79428c4 100644 --- a/internal/dataflow/run_test.go +++ b/internal/dataflow/run_test.go @@ -68,9 +68,9 @@ func TestRun_JobsDeterminism(t *testing.T) { } // lineDirectiveSrc maps its body to a different (virtual) file and line with -// a //line directive. L1 records call positions through fset.Position, which -// honours the directive, so the call keys name virtual lines; pairing them -// with the AST calls by recording index must still work (ruling R5). +// a //line directive. Positions are real file positions (PositionFor(p, +// false)), so the call keys and spans name the real lines 7–11 of p.go, not +// the virtual 900+, and calls pair with their keys by Lparen (#26). const lineDirectiveSrc = `package p func f(x int) int { return x } @@ -114,20 +114,22 @@ func funcDecl(t *testing.T, f *ast.File, name string) *ast.FuncDecl { return nil } -// TestRun_LineDirective: the call keys of a body under a //line directive -// land on the call nodes, and spans carry the directive-adjusted positions -// the keys name. +// TestRun_LineDirective: under a //line directive the call keys and spans +// are real positions, not the directive's virtual ones, and every call key +// lands on its call node. func TestRun_LineDirective(t *testing.T) { fset, f, info := parseCheck(t, lineDirectiveSrc) g := funcDecl(t, f, "g") - keys := gateCallKeys(fset, g.Body) - // Virtual positions; the file has only 12 real lines, so rebuilding a - // token.Pos from them (File.LineStart(900)) would panic. - if strings.Join(keys, ",") != "900:7,900:9,902:3" { - t.Fatalf("call keys = %v, want the virtual 900:7, 900:9, 902:3", keys) + calls := gateCallKeys(fset, g.Body) + var keys []string + for _, c := range calls { + keys = append(keys, c.Key) + } + if strings.Join(keys, ",") != "7:7,7:9,9:3" { + t.Fatalf("call keys = %v, want the real 7:7, 7:9, 9:3", keys) } key := dataflow.L3Key{Path: "p.go", Signature: "p.g"} - out := dataflow.Run(fset, []dataflow.Unit{{Key: key, Fn: g, Info: info, Calls: keys}}, 3, 1) + out := dataflow.Run(fset, []dataflow.Unit{{Key: key, Fn: g, Info: info, Calls: calls}}, 3, 1) res, ok := out[key] if !ok { t.Fatal("no result for g") @@ -141,29 +143,68 @@ func TestRun_LineDirective(t *testing.T) { t.Errorf("call key %s has no call node", k) } } - if n := byKey["901:2"]; n.Kind != dataflow.KindBranch || n.StartLC != [2]int{901, 2} || n.EndLC != [2]int{903, 3} { - t.Errorf("if node = %+v, want branch spanning 901:2–903:3", n) + if n := byKey["8:2"]; n.Kind != dataflow.KindBranch || n.StartLC != [2]int{8, 2} || n.EndLC != [2]int{10, 3} { + t.Errorf("if node = %+v, want branch spanning 8:2–10:3", n) } } -// TestRun_CallCountMismatchSkips: a unit whose L1 key list does not match -// its body's calls gets no result (never a silently mis-keyed one); other -// units are unaffected. -func TestRun_CallCountMismatchSkips(t *testing.T) { +// TestRun_UnpairedCallSkips: a unit with a body call that no L1 key names +// (by Lparen) gets no result (never a silently mis-keyed one); other units +// are unaffected. +func TestRun_UnpairedCallSkips(t *testing.T) { fset, f, info := parseCheck(t, lineDirectiveSrc) g, fn := funcDecl(t, f, "g"), funcDecl(t, f, "f") bad := dataflow.L3Key{Path: "p.go", Signature: "p.g"} good := dataflow.L3Key{Path: "p.go", Signature: "p.f"} - out := dataflow.Run(fset, []dataflow.Unit{ - {Key: bad, Fn: g, Info: info, Calls: []string{"900:7"}}, // 1 key, 3 calls - {Key: good, Fn: fn, Info: info, Calls: nil}, - }, 3, 2) + var out map[dataflow.L3Key]dataflow.Result + stderr := captureStderr(t, func() { + out = dataflow.Run(fset, []dataflow.Unit{ + {Key: bad, Fn: g, Info: info, Calls: gateCallKeys(fset, g.Body)[:1]}, // 1 key, 3 calls + {Key: good, Fn: fn, Info: info, Calls: nil}, + }, 3, 2) + }) if _, ok := out[bad]; ok { - t.Error("mismatched unit produced a result") + t.Error("unit with unpaired calls produced a result") } if r, ok := out[good]; !ok || len(r.Cfg) == 0 { t.Errorf("good unit result = %+v, %v", r, ok) } + if !strings.Contains(stderr, "L3 skipped for p.g (p.go): 2 calls in the body have no L1 call key") { + t.Errorf("want the unpaired-call warning; got:\n%s", stderr) + } +} + +// TestRun_CallCountMismatchWarnsOnly: the L1-key count is a cross-check, not +// the pairing: a surplus key (here one naming no call of the body) warns, and +// the unit is still built with every call keyed by its Lparen. +func TestRun_CallCountMismatchWarnsOnly(t *testing.T) { + fset, f, info := parseCheck(t, lineDirectiveSrc) + g := funcDecl(t, f, "g") + key := dataflow.L3Key{Path: "p.go", Signature: "p.g"} + calls := append(gateCallKeys(fset, g.Body), dataflow.UnitCall{Key: "99:1", Lparen: token.NoPos}) + var out map[dataflow.L3Key]dataflow.Result + stderr := captureStderr(t, func() { + out = dataflow.Run(fset, []dataflow.Unit{{Key: key, Fn: g, Info: info, Calls: calls}}, 3, 1) + }) + if !strings.Contains(stderr, "4 L1 call keys for 3 calls in the body") { + t.Errorf("want the count cross-check warning; got:\n%s", stderr) + } + res, ok := out[key] + if !ok { + t.Fatal("count mismatch alone skipped the unit") + } + byKey := map[string]dataflow.Node{} + for _, n := range res.Nodes { + byKey[n.Key] = n + } + for _, c := range calls[:3] { + if n, ok := byKey[c.Key]; !ok || n.Kind != dataflow.KindCall { + t.Errorf("call key %s has no call node", c.Key) + } + } + if _, ok := byKey["99:1"]; ok { + t.Error("surplus key produced a node") + } } // TestRun_ResultContract: on the flow fixture every result has @entry and @@ -326,9 +367,9 @@ func captureStderr(t *testing.T, fn func()) string { return <-done } -// TestRun_FlowFixtureHasNoCallCountMismatch: L1's call walk (buildCallSites) -// and L3's (recordedCalls) agree on every flow-fixture body — including calls -// in a `go` statement's arguments — so no unit is skipped with a warning. +// TestRun_FlowFixtureHasNoCallCountMismatch: every L1 call site of every +// flow-fixture body pairs with its call by Lparen — including calls in a `go` +// statement's arguments — so no unit warns or is skipped. func TestRun_FlowFixtureHasNoCallCountMismatch(t *testing.T) { _, thisFile, _, _ := runtime.Caller(0) dir := filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "dataflow") @@ -340,8 +381,8 @@ func TestRun_FlowFixtureHasNoCallCountMismatch(t *testing.T) { units := b.L3Units(st) var out map[dataflow.L3Key]dataflow.Result stderr := captureStderr(t, func() { out = dataflow.Run(b.Fset(), units, 3, 4) }) - if strings.Contains(stderr, "L1 call keys for") { - t.Errorf("call-count mismatch warnings:\n%s", stderr) + if strings.Contains(stderr, "L1 call keys for") || strings.Contains(stderr, "have no L1 call key") { + t.Errorf("call pairing warnings:\n%s", stderr) } if len(out) != len(units) { t.Errorf("%d results for %d units", len(out), len(units)) diff --git a/internal/schema/schema.go b/internal/schema/schema.go index 0161769..7c2d377 100644 --- a/internal/schema/schema.go +++ b/internal/schema/schema.go @@ -5,6 +5,8 @@ // transformation. Design decisions are recorded in .claude/SCHEMA_DECISIONS.md. package schema +import "go/token" + // ─── Leaf models ───────────────────────────────────────────────────────────── // GoImport represents a single import declaration in a Go source file. @@ -92,6 +94,11 @@ type GoCallsite struct { StartColumn int `json:"start_column"` EndLine int `json:"end_line"` EndColumn int `json:"end_column"` + // Lparen is the call's raw CallExpr.Lparen in the builder's FileSet: the + // in-memory join key from an L1 site to its *ast.CallExpr (L3 body nodes, + // and go/ssa CallCommon.Pos() at L4). Never serialized; meaningless once + // the FileSet is gone. + Lparen token.Pos `json:"-"` } // ─── Callable ───────────────────────────────────────────────────────────────── diff --git a/internal/semantic_analysis/call_graph.go b/internal/semantic_analysis/call_graph.go index 8279c6c..2911cdf 100644 --- a/internal/semantic_analysis/call_graph.go +++ b/internal/semantic_analysis/call_graph.go @@ -289,8 +289,8 @@ func (cg *CallGraphBuilder) recordExternal(fn *types.Func, fullSig string) { // resolveCallSite resolves a single call site to its callee function by looking // it up in the package's call-site index under the key L1 recorded it with: -// relPath (the file declaring the enclosing callable) and the site's -// directive-adjusted start and end line:col. Returns nil when the site cannot be +// relPath (the file declaring the enclosing callable) and the site's real +// (not //line-adjusted) start and end line:col. Returns nil when the site cannot be // resolved (graceful fallback). func (cg *CallGraphBuilder) resolveCallSite(pkg *packages.Package, relPath string, site *schema.GoCallsite) *types.Func { if pkg.TypesInfo == nil { diff --git a/internal/semantic_analysis/callsite_index.go b/internal/semantic_analysis/callsite_index.go index d4ec07e..243e5fc 100644 --- a/internal/semantic_analysis/callsite_index.go +++ b/internal/semantic_analysis/callsite_index.go @@ -12,12 +12,9 @@ import ( // callKey identifies a call expression exactly as L1 records its site: the // project-relative path of the file that contains it (the token.File name, the // same path as the enclosing callable's Path) and its start and end line:col -// from fset.Position, which applies //line directives. Start alone is -// ambiguous: a chained s.A().B() starts where its inner s.A() does. -// -// The key is never turned back into a token.Pos: under a //line directive the -// recorded line names a virtual line (often past the file's real line count), -// so rebuilding a position from it would be wrong or panic. +// as real file positions (fset.PositionFor(p, false): //line directives are +// not applied, as L1 does not apply them). Start alone is ambiguous: a chained +// s.A().B() starts where its inner s.A() does. type callKey struct { path string startLine, startCol, endLine, endColumn int @@ -26,9 +23,9 @@ type callKey struct { // callSiteIndex maps a call expression's callKey to the call. type callSiteIndex map[callKey]*ast.CallExpr -// newCallSiteIndex walks each file of pkg once and indexes every call. When -// two calls of one file share a key (only possible when //line directives map -// both to the same virtual range), the first in source order wins. +// newCallSiteIndex walks each file of pkg once and indexes every call. Real +// positions make the key unique per file (two calls never share both a start +// and an end); should one repeat, the first in source order wins. func newCallSiteIndex(projectDir string, fset *token.FileSet, pkg *packages.Package) callSiteIndex { idx := callSiteIndex{} for _, f := range pkg.Syntax { @@ -42,7 +39,7 @@ func newCallSiteIndex(projectDir string, fset *token.FileSet, pkg *packages.Pack if !ok { return true } - start, end := fset.Position(call.Pos()), fset.Position(call.End()) + start, end := fset.PositionFor(call.Pos(), false), fset.PositionFor(call.End(), false) k := callKey{rel, start.Line, start.Column, end.Line, end.Column} if _, dup := idx[k]; !dup { idx[k] = call diff --git a/internal/syntactic_analysis/l3units.go b/internal/syntactic_analysis/l3units.go index 3cac08d..07772d3 100644 --- a/internal/syntactic_analysis/l3units.go +++ b/internal/syntactic_analysis/l3units.go @@ -1,11 +1,11 @@ package syntactic_analysis import ( - "fmt" "go/ast" "go/types" "sort" + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" @@ -22,14 +22,13 @@ import ( // which the symbol table overwrote, from taking the survivor's key. // Declarations the symbol table dropped (cgo wrappers, methods whose // receiver type was not found) get no unit. -// - A FuncLit is matched to its L1 closure by the traversal that named it: -// buildInnerCallables numbers the outermost literals of a body -// closure_1, closure_2, … in ast.Inspect order and recurses per literal, -// so walking the same way pairs each literal with InnerCallables[name]. -// Its start line is checked too. Its key's Path is the enclosing -// callable's declaring file. +// - A FuncLit is matched to its L1 closure by name: buildInnerCallables +// names the outermost literals of a body with astwalk.Literals and +// recurses per literal, so walking the same way pairs each literal with +// InnerCallables[name]. Its start line is checked too. Its key's Path is +// the enclosing callable's declaring file. // - Calls is v2.CallBodyKeys over the callable's L1 call sites, the exact -// keys the L1 emitter writes. +// keys the L1 emitter writes, each with its site's Lparen. // // A mismatch (which would mean the walk drifted from L1's) is logged and the // callable is left at L1. @@ -73,7 +72,7 @@ func (b *SymbolTableBuilder) L3Units(symbolTable map[string]schema.GoFile) []dat Signature: b.declSignature(pkg, fd), } c, ok := byKey[key] - if !ok || seen[key] || c.StartLine != b.fset.Position(fd.Pos()).Line { + if !ok || seen[key] || c.StartLine != b.position(fd.Pos()).Line { continue } seen[key] = true @@ -92,26 +91,18 @@ func (b *SymbolTableBuilder) L3Units(symbolTable map[string]schema.GoFile) []dat return units } -// closureUnits appends a unit for every function literal of body, mirroring -// buildInnerCallables' naming walk. +// closureUnits appends a unit for every function literal of body, walking +// astwalk.Literals as buildInnerCallables does. func (b *SymbolTableBuilder) closureUnits(units []dataflow.Unit, path string, body *ast.BlockStmt, outer schema.GoCallable, info *types.Info) []dataflow.Unit { - n := 0 - ast.Inspect(body, func(node ast.Node) bool { - lit, ok := node.(*ast.FuncLit) - if !ok { - return true + for _, l := range astwalk.Literals(body) { + ic, ok := outer.InnerCallables[l.Name] + if !ok || ic.StartLine != b.position(l.Lit.Pos()).Line { + utils.Warn("L3 skipped for %s.%s (%s): no matching L1 closure", outer.Signature, l.Name, path) + continue } - n++ - name := fmt.Sprintf("closure_%d", n) - ic, ok := outer.InnerCallables[name] - if !ok || ic.StartLine != b.fset.Position(lit.Pos()).Line { - utils.Warn("L3 skipped for %s.%s (%s): no matching L1 closure", outer.Signature, name, path) - return false - } - units = append(units, unit(dataflow.L3Key{Path: path, Signature: ic.Signature}, lit, info, ic)) - units = b.closureUnits(units, path, lit.Body, ic, info) - return false - }) + units = append(units, unit(dataflow.L3Key{Path: path, Signature: ic.Signature}, l.Lit, info, ic)) + units = b.closureUnits(units, path, l.Lit.Body, ic, info) + } return units } @@ -120,5 +111,10 @@ func unit(key dataflow.L3Key, fn ast.Node, info *types.Info, c schema.GoCallable for i, cs := range c.CallSites { pos[i] = [2]int{cs.StartLine, cs.StartColumn} } - return dataflow.Unit{Key: key, Fn: fn, Info: info, Calls: v2.CallBodyKeys(pos)} + keys := v2.CallBodyKeys(pos) + calls := make([]dataflow.UnitCall, len(c.CallSites)) + for i, cs := range c.CallSites { + calls[i] = dataflow.UnitCall{Key: keys[i], Lparen: cs.Lparen} + } + return dataflow.Unit{Key: key, Fn: fn, Info: info, Calls: calls} } diff --git a/internal/syntactic_analysis/symbol_table.go b/internal/syntactic_analysis/symbol_table.go index a0e2b35..f6f80dd 100644 --- a/internal/syntactic_analysis/symbol_table.go +++ b/internal/syntactic_analysis/symbol_table.go @@ -12,6 +12,7 @@ import ( "golang.org/x/tools/go/packages" + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" "github.com/codellm-devkit/codeanalyzer-go/internal/utils" ) @@ -37,6 +38,16 @@ func NewSymbolTableBuilder(projectDir string) *SymbolTableBuilder { } } +// position is p's real file position: //line directives are NOT applied +// (token.FileSet.PositionFor(p, false)). Every line:col the builder records — +// call sites, callable / type / field / import spans — indexes the file that +// holds the node, the file module.source is read from, so spans slice to the +// node's text even in generated code (goyacc, templ) carrying //line +// directives. +func (b *SymbolTableBuilder) position(p token.Pos) token.Position { + return b.fset.PositionFor(p, false) +} + // Build loads all packages under projectDir (or only targetFiles if non-empty), // walks each file, and returns the symbol table keyed by relative file path. func (b *SymbolTableBuilder) Build(targetFiles []string, skipTests bool) (map[string]schema.GoFile, error) { @@ -268,8 +279,8 @@ func (b *SymbolTableBuilder) buildImports(astFile *ast.File) []schema.GoImport { if imp.Name != nil { alias = imp.Name.Name } - pos := b.fset.Position(imp.Pos()) - end := b.fset.Position(imp.End()) + pos := b.position(imp.Pos()) + end := b.position(imp.End()) imports = append(imports, schema.GoImport{ Module: path, Alias: alias, @@ -286,8 +297,8 @@ func (b *SymbolTableBuilder) buildFileComments(astFile *ast.File) []schema.GoCom var comments []schema.GoComment for _, cg := range astFile.Comments { for _, c := range cg.List { - pos := b.fset.Position(c.Pos()) - end := b.fset.Position(c.End()) + pos := b.position(c.Pos()) + end := b.position(c.End()) comments = append(comments, schema.GoComment{ Content: c.Text, StartLine: pos.Line, @@ -307,8 +318,8 @@ func (b *SymbolTableBuilder) docComments(doc *ast.CommentGroup) []schema.GoComme } var comments []schema.GoComment for _, c := range doc.List { - pos := b.fset.Position(c.Pos()) - end := b.fset.Position(c.End()) + pos := b.position(c.Pos()) + end := b.position(c.End()) comments = append(comments, schema.GoComment{ Content: c.Text, StartLine: pos.Line, @@ -362,8 +373,8 @@ func (b *SymbolTableBuilder) buildType( typSig = pkg.Types.Path() + "." + typeName } - pos := b.fset.Position(spec.Pos()) - end := b.fset.Position(spec.End()) + pos := b.position(spec.Pos()) + end := b.position(spec.End()) gt := &schema.GoType{ Name: typeName, @@ -437,8 +448,8 @@ func (b *SymbolTableBuilder) buildStructFields(pkg *packages.Package, st *ast.St isEmbedded := len(field.Names) == 0 if isEmbedded { - pos := b.fset.Position(field.Pos()) - end := b.fset.Position(field.End()) + pos := b.position(field.Pos()) + end := b.position(field.End()) fields = append(fields, schema.GoField{ Name: typStr, Type: typStr, @@ -451,8 +462,8 @@ func (b *SymbolTableBuilder) buildStructFields(pkg *packages.Package, st *ast.St continue } for _, name := range field.Names { - pos := b.fset.Position(name.Pos()) - end := b.fset.Position(field.End()) + pos := b.position(name.Pos()) + end := b.position(field.End()) fields = append(fields, schema.GoField{ Name: name.Name, Type: typStr, @@ -491,8 +502,8 @@ func (b *SymbolTableBuilder) collectInterfaceMethods(pkg *packages.Package, it * continue } for _, name := range method.Names { - pos := b.fset.Position(name.Pos()) - end := b.fset.Position(method.End()) + pos := b.position(name.Pos()) + end := b.position(method.End()) var retType string var retTypes []string @@ -544,8 +555,8 @@ func (b *SymbolTableBuilder) buildCallable( } isExported := unicode.IsUpper(rune(name[0])) - pos := b.fset.Position(decl.Pos()) - end := b.fset.Position(decl.End()) + pos := b.position(decl.Pos()) + end := b.position(decl.End()) sig := b.declSignature(pkg, decl) @@ -561,7 +572,7 @@ func (b *SymbolTableBuilder) buildCallable( retType, retTypes := b.buildReturnTypes(pkg, decl.Type) bodyStart := pos.Line if decl.Body != nil { - bodyStart = b.fset.Position(decl.Body.Pos()).Line + bodyStart = b.position(decl.Body.Pos()).Line } callable := &schema.GoCallable{ @@ -626,8 +637,8 @@ func (b *SymbolTableBuilder) buildParams(pkg *packages.Package, ft *ast.FuncType if _, ok := field.Type.(*ast.Ellipsis); ok { isVariadic = true } - pos := b.fset.Position(field.Pos()) - end := b.fset.Position(field.End()) + pos := b.position(field.Pos()) + end := b.position(field.End()) if len(field.Names) == 0 { params = append(params, schema.GoParameter{ Name: "_", Type: typStr, IsVariadic: isVariadic, @@ -684,42 +695,23 @@ func (b *SymbolTableBuilder) joinReturnTypes(parts []string) string { // ─── Call sites ─────────────────────────────────────────────────────────────── -// buildCallSites walks the function body and records every call expression. -// callee_signature is left nil here — the resolver backfills it in the call-graph stage. +// buildCallSites records every call expression of the function body in the +// order astwalk.Calls lists them (the order L3 pairs its call nodes with by +// Lparen). callee_signature is left nil here — the resolver backfills it in +// the call-graph stage. func (b *SymbolTableBuilder) buildCallSites(pkg *packages.Package, body *ast.BlockStmt) []schema.GoCallsite { var sites []schema.GoCallsite - // goroutine / deferred hold the call of each `go` / `defer` statement seen - // so far. Pre-order traversal visits the statement before its call, so the - // CallExpr case can tag it, and continues into the call so calls in its - // function value and arguments (evaluated by the launching goroutine, at - // the statement) are recorded as ordinary sites. The L3 walk - // (dataflow.recordedCalls) must record the same calls in the same order. - goroutine := map[*ast.CallExpr]bool{} - deferred := map[*ast.CallExpr]bool{} - ast.Inspect(body, func(n ast.Node) bool { - switch node := n.(type) { - case *ast.GoStmt: - goroutine[node.Call] = true - case *ast.DeferStmt: - deferred[node.Call] = true - case *ast.FuncLit: - // Closure bodies are handled by buildInnerCallables; don't double-count. - return false - case *ast.CallExpr: - site := b.callExprToSite(pkg, node, goroutine[node]) - if site != nil { - site.IsDeferred = deferred[node] - sites = append(sites, *site) - } - } - return true - }) + for _, c := range astwalk.Calls(body) { + site := b.callExprToSite(pkg, c.Expr, c.Goroutine) + site.IsDeferred = c.Deferred + sites = append(sites, *site) + } return sites } func (b *SymbolTableBuilder) callExprToSite(pkg *packages.Package, call *ast.CallExpr, isGoroutine bool) *schema.GoCallsite { - pos := b.fset.Position(call.Pos()) - end := b.fset.Position(call.End()) + pos := b.position(call.Pos()) + end := b.position(call.End()) var methodName, receiverExpr, receiverType string isConstructor := false @@ -771,6 +763,7 @@ func (b *SymbolTableBuilder) callExprToSite(pkg *packages.Package, call *ast.Cal StartColumn: pos.Column, EndLine: end.Line, EndColumn: end.Column, + Lparen: call.Lparen, } } @@ -784,7 +777,7 @@ func (b *SymbolTableBuilder) buildLocalVars(pkg *packages.Package, body *ast.Blo if node.Tok.String() == ":=" { for i, lhs := range node.Lhs { if ident, ok := lhs.(*ast.Ident); ok { - pos := b.fset.Position(ident.Pos()) + pos := b.position(ident.Pos()) typStr := "" if pkg.TypesInfo != nil { if obj := pkg.TypesInfo.ObjectOf(ident); obj != nil { @@ -807,7 +800,7 @@ func (b *SymbolTableBuilder) buildLocalVars(pkg *packages.Package, body *ast.Blo for _, spec := range gen.Specs { if vs, ok := spec.(*ast.ValueSpec); ok { for i, name := range vs.Names { - pos := b.fset.Position(name.Pos()) + pos := b.position(name.Pos()) typStr := "" if vs.Type != nil { typStr = b.typeString(pkg, vs.Type) @@ -834,22 +827,17 @@ func (b *SymbolTableBuilder) buildLocalVars(pkg *packages.Package, body *ast.Blo return vars } -// buildInnerCallables walks body and collects each FuncLit as a named closure. -// Only the top level is captured; nested closures appear in the closure's own -// InnerCallables (populated when buildCallSites recurses into lit.Body). +// buildInnerCallables collects the outermost function literals of body as +// named closures (astwalk.Literals: closure_1, closure_2, … in ast.Inspect +// order). A nested literal is its enclosing closure's own inner callable, +// built by the recursion on lit.Body. func (b *SymbolTableBuilder) buildInnerCallables(pkg *packages.Package, outerSig string, body *ast.BlockStmt) map[string]schema.GoCallable { inner := map[string]schema.GoCallable{} - n := 0 - ast.Inspect(body, func(node ast.Node) bool { - lit, ok := node.(*ast.FuncLit) - if !ok { - return true - } - n++ - name := fmt.Sprintf("closure_%d", n) + for _, l := range astwalk.Literals(body) { + lit, name := l.Lit, l.Name sig := outerSig + "." + name - pos := b.fset.Position(lit.Pos()) - endPos := b.fset.Position(lit.End()) + pos := b.position(lit.Pos()) + endPos := b.position(lit.End()) _, retTypes := b.buildReturnTypes(pkg, lit.Type) retType := b.joinReturnTypes(retTypes) @@ -872,8 +860,7 @@ func (b *SymbolTableBuilder) buildInnerCallables(pkg *packages.Package, outerSig ic.InnerCallables = b.buildInnerCallables(pkg, sig, lit.Body) } inner[name] = ic - return false // don't recurse; nested closures are handled above - }) + } return inner } @@ -895,7 +882,7 @@ func (b *SymbolTableBuilder) buildPackageVars(pkg *packages.Package, astFile *as continue } for i, name := range vs.Names { - pos := b.fset.Position(name.Pos()) + pos := b.position(name.Pos()) typStr := "" if vs.Type != nil { typStr = b.typeString(pkg, vs.Type) @@ -1012,7 +999,7 @@ func (b *SymbolTableBuilder) nodeSource(node ast.Node) string { if tf == nil || b.fset.File(node.End()) != tf { return "" } - pos := b.fset.Position(node.Pos()) + pos := b.position(node.Pos()) if pos.Filename == "" { return "" } diff --git a/internal/syntactic_analysis/v2emit/gate_l2_test.go b/internal/syntactic_analysis/v2emit/gate_l2_test.go index 5e46baf..3beb0ae 100644 --- a/internal/syntactic_analysis/v2emit/gate_l2_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l2_test.go @@ -490,10 +490,11 @@ func TestGateL2_VendoredCalleeIsExternal(t *testing.T) { } } -// TestGateL2_LineDirectiveCallsResolve (I1): L1 records call sites through -// fset.Position, which applies //line directives, so a site under a directive -// names a virtual line:col (here line 900 of a 17-line file). The resolver -// must find it by that same key and backfill its callee and edge. +// TestGateL2_LineDirectiveCallsResolve (I1, #26): L1 records call sites at +// their real file positions, never the //line-adjusted ones, so a site under +// a directive keeps its real line:col (here 13:2, which the directive maps +// to virtual.y:900). The resolver must find it by that same key and backfill +// its callee and edge. func TestGateL2_LineDirectiveCallsResolve(t *testing.T) { out := buildDataflowL2Fresh(t) fn := functionByName(t, mustModule(t, out, "linedir/linedir.go"), "Caller") @@ -511,8 +512,8 @@ func TestGateL2_LineDirectiveCallsResolve(t *testing.T) { } callees[*n.Callee] = true } - if _, virtual := fn.Body["900:2"]; !virtual { - t.Fatalf("no call node at the virtual 900:2; body %v", keys(fn.Body)) + if _, real := fn.Body["13:2"]; !real { + t.Fatalf("no call node at the real 13:2; body %v", keys(fn.Body)) } edges := map[string]bool{} for _, e := range out.Application.CallGraph { diff --git a/internal/syntactic_analysis/v2emit/gate_l3_test.go b/internal/syntactic_analysis/v2emit/gate_l3_test.go index 8f6a976..1209fe4 100644 --- a/internal/syntactic_analysis/v2emit/gate_l3_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l3_test.go @@ -464,3 +464,33 @@ func TestGateL3_GoStatementArgumentCalls(t *testing.T) { } } } + +// TestGateL3_LineDirectiveRealPositions (#26): under //line directives the +// L3 body of linedir.Caller is built (its calls pair with their L1 keys by +// Lparen), its keys are real line:col, and every non-synthetic node's span +// slices module.source to its text. +func TestGateL3_LineDirectiveRealPositions(t *testing.T) { + fx := buildL3(t, "dataflow") + mod := mustModule(t, fx.l3, "linedir/linedir.go") + c := functionByName(t, mod, "Caller") + want := map[string]string{ + "@entry": "", "@exit": "", + "13:2": "os.Getenv(helper())", + "13:12": "helper()", + "15:2": "helper()", + } + if got, w := strings.Join(sortedKeys(c.Body), ","), strings.Join(sortedKeys(want), ","); got != w { + t.Fatalf("Caller body keys = %s, want %s", got, w) + } + for k, text := range want { + if text == "" { + continue + } + if got := c.Body[k].Span.Slice(mod.Source); got != text { + t.Errorf("Caller %s span slices to %q, want %q", k, got, text) + } + } + if len(c.Cfg) == 0 { + t.Error("Caller: empty cfg") + } +} diff --git a/internal/syntactic_analysis/v2emit/gate_test.go b/internal/syntactic_analysis/v2emit/gate_test.go index 82a6626..cc37d35 100644 --- a/internal/syntactic_analysis/v2emit/gate_test.go +++ b/internal/syntactic_analysis/v2emit/gate_test.go @@ -620,3 +620,70 @@ func anyEdgeFromTo(edges []v2.Edge, src, dst string) bool { } return false } + +// TestGate_LineDirectiveSpansSliceToText (#26): positions are real file +// positions (token.FileSet.PositionFor(p, false)), never //line-adjusted, so +// under a directive every span still indexes module.source. In +// testdata/dataflow/linedir every call node's span slices to text that +// starts with its callee expression, every callable's to its declaration, +// and the call keys name the calls' real line:col. +func TestGate_LineDirectiveSpansSliceToText(t *testing.T) { + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata", "dataflow")) + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}}, "dataflow", dir, 1, "gate-test", 0, nil) + const rel = "linedir/linedir.go" + mod := mustModule(t, out, rel) + calls := 0 + for sig, fn := range st[rel].Functions { + c := mod.Functions[sig] + text := c.Span.Slice(mod.Source) + if c.Span.Bytes[0] >= c.Span.Bytes[1] || !strings.HasPrefix(text, "func "+fn.Name+"(") { + t.Errorf("callable %s: span %v slices to %q, want func %s(…", sig, c.Span, text, fn.Name) + } + keys := v2.CallBodyKeys(siteStarts(fn.CallSites)) + for i, cs := range fn.CallSites { + n, ok := c.Body[keys[i]] + if !ok { + t.Errorf("%s: no call node %s; body %v", sig, keys[i], keys) + continue + } + calls++ + callee := cs.MethodName + if cs.ReceiverExpr != "" { + callee = cs.ReceiverExpr + "." + callee + } + if got := n.Span.Slice(mod.Source); n.Span.Bytes[0] >= n.Span.Bytes[1] || !strings.HasPrefix(got, callee+"(") { + t.Errorf("%s call %s: span %v slices to %q, want %s(…", sig, keys[i], n.Span, got, callee) + } + } + } + caller := functionByName(t, mod, "Caller") + if got := strings.Join(sortedKeys(caller.Body), ","); got != "13:12,13:2,15:2" { + t.Errorf("Caller call keys = %s, want the real 13:12,13:2,15:2", got) + } + if calls != 3 { + t.Errorf("checked %d call nodes, want 3", calls) + } +} + +func siteStarts(sites []schema.GoCallsite) [][2]int { + pos := make([][2]int, len(sites)) + for i, cs := range sites { + pos[i] = [2]int{cs.StartLine, cs.StartColumn} + } + return pos +} + +func sortedKeys[V any](m map[string]V) []string { + ks := keys(m) + sort.Strings(ks) + return ks +} From 7aa2b1a169da8cd3f6a41b5bc48b433469f0ab26 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Sun, 25 Oct 2026 15:35:08 -0400 Subject: [PATCH 03/65] fix(l2): resolve call sites inside closures (#17) --- internal/core/analyzer_test.go | 17 ++- internal/semantic_analysis/call_graph.go | 11 ++ internal/semantic_analysis/call_graph_test.go | 39 +++++ internal/semantic_analysis/chi_golden_test.go | 41 ++++- internal/syntactic_analysis/symbol_table.go | 11 +- internal/syntactic_analysis/v2emit/emit.go | 4 +- .../syntactic_analysis/v2emit/gate_l2_test.go | 140 ++++++++++++++++++ .../syntactic_analysis/v2emit/sigindex.go | 8 +- 8 files changed, 259 insertions(+), 12 deletions(-) diff --git a/internal/core/analyzer_test.go b/internal/core/analyzer_test.go index 3cc70a5..9b08ce1 100644 --- a/internal/core/analyzer_test.go +++ b/internal/core/analyzer_test.go @@ -294,15 +294,24 @@ func TestCaching_EagerForcesRebuild(t *testing.T) { // ── Helpers ─────────────────────────────────────────────────────────────────── +// allSignatures collects every callable signature, closures included (a closure +// is a call-graph edge source). func allSignatures(app *schema.GoApplication) map[string]bool { sigs := map[string]bool{} + var add func(sig string, c schema.GoCallable) + add = func(sig string, c schema.GoCallable) { + sigs[sig] = true + for _, ic := range c.InnerCallables { + add(ic.Signature, ic) + } + } for _, f := range app.SymbolTable { - for sig := range f.Functions { - sigs[sig] = true + for sig, c := range f.Functions { + add(sig, c) } for _, t := range f.Types { - for sig := range t.Methods { - sigs[sig] = true + for sig, c := range t.Methods { + add(sig, c) } } } diff --git a/internal/semantic_analysis/call_graph.go b/internal/semantic_analysis/call_graph.go index 2911cdf..b71a214 100644 --- a/internal/semantic_analysis/call_graph.go +++ b/internal/semantic_analysis/call_graph.go @@ -121,6 +121,8 @@ func buildKnownSigs(symbolTable map[string]schema.GoFile) map[string]bool { } // resolveCallable backfills callee_signature on each call site and produces edges. +// It then does the same for the callable's closures (InnerCallables) in sorted +// key order, recursively, after the callable's own sites. // // An in-project callee (in knownSigs) gets an edge. A library callee (a package // outside the project) is recorded in cg.externals, its site is tagged with the @@ -181,6 +183,15 @@ func (cg *CallGraphBuilder) resolveCallable( }) } + // Closures: their own sites, then theirs in turn. Each carries its enclosing + // callable's declaring file as Path, so the site index and the edge dedup + // treat it like any other callable. + for _, name := range sortedKeys(callable.InnerCallables) { + inner, innerEdges := cg.resolveCallable(pkg, callable.InnerCallables[name], seen, knownSigs) + callable.InnerCallables[name] = inner + edges = append(edges, innerEdges...) + } + return callable, edges } diff --git a/internal/semantic_analysis/call_graph_test.go b/internal/semantic_analysis/call_graph_test.go index 84cb74f..2357407 100644 --- a/internal/semantic_analysis/call_graph_test.go +++ b/internal/semantic_analysis/call_graph_test.go @@ -161,3 +161,42 @@ func firstDiff(a, b []string) string { } return fmt.Sprintf("length differs (%d vs %d)", len(a), len(b)) } + +// TestCallGraph_ClosureSitesResolve: call sites inside function literals are +// backfilled and produce edges whose source is the closure signature and whose +// SourcePath is the declaring file (#17). +func TestCallGraph_ClosureSitesResolve(t *testing.T) { + dir, err := filepath.Abs("../../testdata/dataflow") + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + + const outer = "example.com/dataflow/flow.Closure" + lit, ok := st["flow/flow.go"].Functions[outer].InnerCallables["closure_1"] + if !ok { + t.Fatal("Closure has no closure_1") + } + if lit.Path != "flow/flow.go" { + t.Errorf("closure Path = %q, want flow/flow.go", lit.Path) + } + if len(lit.CallSites) != 1 || lit.CallSites[0].CalleeSignature == nil || + *lit.CallSites[0].CalleeSignature != "example.com/dataflow/flow.f" { + t.Fatalf("closure call site not resolved: %+v", lit.CallSites) + } + for _, e := range edges { + if e.Source == outer+".closure_1" && e.Target == "example.com/dataflow/flow.f" { + if e.SourcePath != "flow/flow.go" { + t.Errorf("edge SourcePath = %q, want flow/flow.go", e.SourcePath) + } + return + } + } + t.Error("no edge from the closure to flow.f") +} diff --git a/internal/semantic_analysis/chi_golden_test.go b/internal/semantic_analysis/chi_golden_test.go index 1d09efb..3bb6e36 100644 --- a/internal/semantic_analysis/chi_golden_test.go +++ b/internal/semantic_analysis/chi_golden_test.go @@ -11,7 +11,9 @@ package semantic_analysis_test // - Throttle -> ThrottleWithOpts (throttle.go:33) is gained for the same reason; // - seven edges from the inner call of a chained call (e.g. // mx.NotFoundHandler().ServeHTTP, Chain(...).HandlerFunc) are gained, since -// the outer call no longer shadows the co-positioned inner one. +// the outer call no longer shadows the co-positioned inner one; +// - thirty-seven edges whose source is a closure (.closure_N) are gained: +// call sites inside function literals are resolved too (#17). var chiGoldenEdges = []string{ "github.com/go-chi/chi/v5.Context.RoutePattern -> github.com/go-chi/chi/v5.replaceWildcards", "github.com/go-chi/chi/v5.Middlewares.Handler -> github.com/go-chi/chi/v5.chain", @@ -33,14 +35,18 @@ var chiGoldenEdges = []string{ "github.com/go-chi/chi/v5.Mux.MethodNotAllowed -> github.com/go-chi/chi/v5.Chain", "github.com/go-chi/chi/v5.Mux.MethodNotAllowed -> github.com/go-chi/chi/v5.Middlewares.HandlerFunc", "github.com/go-chi/chi/v5.Mux.MethodNotAllowed -> github.com/go-chi/chi/v5.Mux.updateSubRoutes", + "github.com/go-chi/chi/v5.Mux.MethodNotAllowed.closure_1 -> github.com/go-chi/chi/v5.Mux.MethodNotAllowed", "github.com/go-chi/chi/v5.Mux.MethodNotAllowedHandler -> github.com/go-chi/chi/v5.methodNotAllowedHandler", "github.com/go-chi/chi/v5.Mux.Mount -> github.com/go-chi/chi/v5.Mux.MethodNotAllowed", "github.com/go-chi/chi/v5.Mux.Mount -> github.com/go-chi/chi/v5.Mux.NotFound", "github.com/go-chi/chi/v5.Mux.Mount -> github.com/go-chi/chi/v5.Mux.handle", "github.com/go-chi/chi/v5.Mux.Mount -> github.com/go-chi/chi/v5.node.findPattern", + "github.com/go-chi/chi/v5.Mux.Mount.closure_1 -> github.com/go-chi/chi/v5.Mux.nextRoutePath", + "github.com/go-chi/chi/v5.Mux.Mount.closure_1 -> github.com/go-chi/chi/v5.RouteContext", "github.com/go-chi/chi/v5.Mux.NotFound -> github.com/go-chi/chi/v5.Chain", "github.com/go-chi/chi/v5.Mux.NotFound -> github.com/go-chi/chi/v5.Middlewares.HandlerFunc", "github.com/go-chi/chi/v5.Mux.NotFound -> github.com/go-chi/chi/v5.Mux.updateSubRoutes", + "github.com/go-chi/chi/v5.Mux.NotFound.closure_1 -> github.com/go-chi/chi/v5.Mux.NotFound", "github.com/go-chi/chi/v5.Mux.Options -> github.com/go-chi/chi/v5.Mux.handle", "github.com/go-chi/chi/v5.Mux.Patch -> github.com/go-chi/chi/v5.Mux.handle", "github.com/go-chi/chi/v5.Mux.Post -> github.com/go-chi/chi/v5.Mux.handle", @@ -62,6 +68,7 @@ var chiGoldenEdges = []string{ "github.com/go-chi/chi/v5.Mux.routeHTTP -> github.com/go-chi/chi/v5.node.FindRoute", "github.com/go-chi/chi/v5.Mux.updateRouteHandler -> github.com/go-chi/chi/v5.chain", "github.com/go-chi/chi/v5.Mux.updateSubRoutes -> github.com/go-chi/chi/v5.node.routes", + "github.com/go-chi/chi/v5.NewMux.closure_1 -> github.com/go-chi/chi/v5.NewRouteContext", "github.com/go-chi/chi/v5.NewRouter -> github.com/go-chi/chi/v5.NewMux", "github.com/go-chi/chi/v5.URLParam -> github.com/go-chi/chi/v5.Context.URLParam", "github.com/go-chi/chi/v5.URLParam -> github.com/go-chi/chi/v5.RouteContext", @@ -94,14 +101,29 @@ var chiGoldenEdges = []string{ "github.com/go-chi/chi/v5.walk -> github.com/go-chi/chi/v5.Routes.Middlewares", "github.com/go-chi/chi/v5.walk -> github.com/go-chi/chi/v5.Routes.Routes", "github.com/go-chi/chi/v5.walk -> github.com/go-chi/chi/v5.walk", + "github.com/go-chi/chi/v5/middleware.BasicAuth.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.basicAuthFailed", + "github.com/go-chi/chi/v5/middleware.CleanPath.closure_1 -> github.com/go-chi/chi/v5.RouteContext", + "github.com/go-chi/chi/v5/middleware.ClientIPFromHeader.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.parseHeaderAddr", + "github.com/go-chi/chi/v5/middleware.ClientIPFromXFF.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.walkXFF", + "github.com/go-chi/chi/v5/middleware.ClientIPFromXFF.closure_1.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.inAnyPrefix", + "github.com/go-chi/chi/v5/middleware.ClientIPFromXFF.closure_1.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.parseHeaderAddr", + "github.com/go-chi/chi/v5/middleware.ClientIPFromXFFTrustedProxies.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.parseHeaderAddr", + "github.com/go-chi/chi/v5/middleware.ClientIPFromXFFTrustedProxies.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.walkXFF", "github.com/go-chi/chi/v5/middleware.Compress -> github.com/go-chi/chi/v5/middleware.NewCompressor", + "github.com/go-chi/chi/v5/middleware.Compressor.Handler.closure_1 -> github.com/go-chi/chi/v5/middleware.Compressor.selectEncoder", + "github.com/go-chi/chi/v5/middleware.Compressor.Handler.closure_1 -> github.com/go-chi/chi/v5/middleware.compressResponseWriter.Close", "github.com/go-chi/chi/v5/middleware.Compressor.selectEncoder -> github.com/go-chi/chi/v5/middleware.ioResetterWriter.Reset", "github.com/go-chi/chi/v5/middleware.Compressor.selectEncoder -> github.com/go-chi/chi/v5/middleware.matchAcceptEncoding", + "github.com/go-chi/chi/v5/middleware.ContentCharset.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.contentEncoding", "github.com/go-chi/chi/v5/middleware.DefaultLogFormatter.NewLogEntry -> github.com/go-chi/chi/v5/middleware.GetClientIP", "github.com/go-chi/chi/v5/middleware.DefaultLogFormatter.NewLogEntry -> github.com/go-chi/chi/v5/middleware.GetReqID", "github.com/go-chi/chi/v5/middleware.DefaultLogFormatter.NewLogEntry -> github.com/go-chi/chi/v5/middleware.cW", "github.com/go-chi/chi/v5/middleware.GetClientIP -> github.com/go-chi/chi/v5/middleware.GetClientIPAddr", + "github.com/go-chi/chi/v5/middleware.GetHead.closure_1 -> github.com/go-chi/chi/v5.NewRouteContext", + "github.com/go-chi/chi/v5/middleware.GetHead.closure_1 -> github.com/go-chi/chi/v5.RouteContext", + "github.com/go-chi/chi/v5/middleware.GetHead.closure_1 -> github.com/go-chi/chi/v5.Routes.Match", "github.com/go-chi/chi/v5/middleware.HeaderRoute.IsMatch -> github.com/go-chi/chi/v5/middleware.Pattern.Match", + "github.com/go-chi/chi/v5/middleware.HeaderRouter.Handler.closure_1 -> github.com/go-chi/chi/v5/middleware.HeaderRoute.IsMatch", "github.com/go-chi/chi/v5/middleware.HeaderRouter.Route -> github.com/go-chi/chi/v5/middleware.NewPattern", "github.com/go-chi/chi/v5/middleware.HeaderRouter.RouteAny -> github.com/go-chi/chi/v5/middleware.NewPattern", "github.com/go-chi/chi/v5/middleware.NewCompressor -> github.com/go-chi/chi/v5/middleware.Compressor.SetEncoder", @@ -111,8 +133,25 @@ var chiGoldenEdges = []string{ "github.com/go-chi/chi/v5/middleware.Profiler -> github.com/go-chi/chi/v5.Mux.HandleFunc", "github.com/go-chi/chi/v5/middleware.Profiler -> github.com/go-chi/chi/v5.Mux.Use", "github.com/go-chi/chi/v5/middleware.Profiler -> github.com/go-chi/chi/v5.NewRouter", + "github.com/go-chi/chi/v5/middleware.RealIP.closure_1 -> github.com/go-chi/chi/v5/middleware.realIP", + "github.com/go-chi/chi/v5/middleware.Recoverer.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.GetLogEntry", + "github.com/go-chi/chi/v5/middleware.Recoverer.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.LogEntry.Panic", + "github.com/go-chi/chi/v5/middleware.Recoverer.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.PrintPrettyStack", + "github.com/go-chi/chi/v5/middleware.RedirectSlashes.closure_1 -> github.com/go-chi/chi/v5.RouteContext", + "github.com/go-chi/chi/v5/middleware.RequestLogger.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.LogFormatter.NewLogEntry", + "github.com/go-chi/chi/v5/middleware.RequestLogger.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.NewWrapResponseWriter", + "github.com/go-chi/chi/v5/middleware.RequestLogger.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.WithLogEntry", + "github.com/go-chi/chi/v5/middleware.RequestLogger.closure_1.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.LogEntry.Write", + "github.com/go-chi/chi/v5/middleware.RequestLogger.closure_1.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.WrapResponseWriter.BytesWritten", + "github.com/go-chi/chi/v5/middleware.RequestLogger.closure_1.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.WrapResponseWriter.Status", + "github.com/go-chi/chi/v5/middleware.StripSlashes.closure_1 -> github.com/go-chi/chi/v5.RouteContext", + "github.com/go-chi/chi/v5/middleware.SupressNotFound.closure_1.closure_1 -> github.com/go-chi/chi/v5.Mux.NotFoundHandler", + "github.com/go-chi/chi/v5/middleware.SupressNotFound.closure_1.closure_1 -> github.com/go-chi/chi/v5.RouteContext", + "github.com/go-chi/chi/v5/middleware.SupressNotFound.closure_1.closure_1 -> github.com/go-chi/chi/v5.Routes.Match", "github.com/go-chi/chi/v5/middleware.Throttle -> github.com/go-chi/chi/v5/middleware.ThrottleWithOpts", "github.com/go-chi/chi/v5/middleware.ThrottleBacklog -> github.com/go-chi/chi/v5/middleware.ThrottleWithOpts", + "github.com/go-chi/chi/v5/middleware.ThrottleWithOpts.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.throttler.setRetryAfterHeaderIfNeeded", + "github.com/go-chi/chi/v5/middleware.URLFormat.closure_1 -> github.com/go-chi/chi/v5.RouteContext", "github.com/go-chi/chi/v5/middleware.basicWriter.Write -> github.com/go-chi/chi/v5/middleware.basicWriter.maybeWriteHeader", "github.com/go-chi/chi/v5/middleware.basicWriter.maybeWriteHeader -> github.com/go-chi/chi/v5/middleware.basicWriter.WriteHeader", "github.com/go-chi/chi/v5/middleware.compressResponseWriter.Close -> github.com/go-chi/chi/v5/middleware.compressResponseWriter.writer", diff --git a/internal/syntactic_analysis/symbol_table.go b/internal/syntactic_analysis/symbol_table.go index f6f80dd..f0b9c18 100644 --- a/internal/syntactic_analysis/symbol_table.go +++ b/internal/syntactic_analysis/symbol_table.go @@ -600,7 +600,7 @@ func (b *SymbolTableBuilder) buildCallable( callable.Code = b.nodeSource(decl) callable.CallSites = b.buildCallSites(pkg, decl.Body) callable.LocalVariables = b.buildLocalVars(pkg, decl.Body) - callable.InnerCallables = b.buildInnerCallables(pkg, sig, decl.Body) + callable.InnerCallables = b.buildInnerCallables(pkg, sig, callable.Path, decl.Body) } return callable @@ -830,8 +830,10 @@ func (b *SymbolTableBuilder) buildLocalVars(pkg *packages.Package, body *ast.Blo // buildInnerCallables collects the outermost function literals of body as // named closures (astwalk.Literals: closure_1, closure_2, … in ast.Inspect // order). A nested literal is its enclosing closure's own inner callable, -// built by the recursion on lit.Body. -func (b *SymbolTableBuilder) buildInnerCallables(pkg *packages.Package, outerSig string, body *ast.BlockStmt) map[string]schema.GoCallable { +// built by the recursion on lit.Body. Every closure carries the enclosing +// callable's declaring file as its Path (Go has no cross-file closures), so +// the resolver keys its call sites and edges like any other callable's. +func (b *SymbolTableBuilder) buildInnerCallables(pkg *packages.Package, outerSig, path string, body *ast.BlockStmt) map[string]schema.GoCallable { inner := map[string]schema.GoCallable{} for _, l := range astwalk.Literals(body) { lit, name := l.Lit, l.Name @@ -843,6 +845,7 @@ func (b *SymbolTableBuilder) buildInnerCallables(pkg *packages.Package, outerSig ic := schema.GoCallable{ Name: name, + Path: path, Signature: sig, Parameters: b.buildParams(pkg, lit.Type), ReturnType: retType, @@ -857,7 +860,7 @@ func (b *SymbolTableBuilder) buildInnerCallables(pkg *packages.Package, outerSig ic.Code = b.nodeSource(lit) ic.CallSites = b.buildCallSites(pkg, lit.Body) ic.LocalVariables = b.buildLocalVars(pkg, lit.Body) - ic.InnerCallables = b.buildInnerCallables(pkg, sig, lit.Body) + ic.InnerCallables = b.buildInnerCallables(pkg, sig, path, lit.Body) } inner[name] = ic } diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 888f8b6..90faf73 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -141,8 +141,8 @@ func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceC // module its outermost enclosing callable nests under). declRel and declLI are // the file the callable's positions are relative to and its lineIndex: the // nesting file for a top-level callable, the enclosing callable's declaring -// file for a closure (Go has no cross-file closures; v1 closures carry no -// Path). A callable's own c.Path, when set, is its declaring file. +// file for a closure (Go has no cross-file closures; a v1 closure's Path is +// that same file). A callable's own c.Path, when set, is its declaring file. // // When the declaring file differs from the containing module — a method whose // receiver type is in another file, and every closure (nested ones included) diff --git a/internal/syntactic_analysis/v2emit/gate_l2_test.go b/internal/syntactic_analysis/v2emit/gate_l2_test.go index 3beb0ae..16abf16 100644 --- a/internal/syntactic_analysis/v2emit/gate_l2_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l2_test.go @@ -562,3 +562,143 @@ func TestGateL2_ExternalUnnamedParamsArePositional(t *testing.T) { } } } + +// emitDirL2 builds dir through the real builder + resolver and emits v2 at L2. +func emitDirL2(t *testing.T, dir, app string) *v2.Analysis { + t.Helper() + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + return Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, app, dir, 2, "gate-test", 0, nil) +} + +// assertNoDanglingEdges fails when an edge endpoint is neither a callable id +// nor an external_symbols key. +func assertNoDanglingEdges(t *testing.T, out *v2.Analysis) { + t.Helper() + ids := allCallableIDs(out) + for id := range out.Application.ExternalSymbols { + ids[id] = true + } + for i, e := range out.Application.CallGraph { + if !ids[e.Src] || !ids[e.Dst] { + t.Errorf("edge[%d] dangles: %s -> %s", i, e.Src, e.Dst) + } + } +} + +// TestGateL2_ClosureCallSitesResolve: a call inside a function literal resolves +// like any other (#17). flow.Closure's literal calls f(x) at 217:10. +func TestGateL2_ClosureCallSitesResolve(t *testing.T) { + out := buildDataflowL2Fresh(t) + mod := mustModule(t, out, "flow/flow.go") + closure := functionByName(t, mod, "Closure") + if len(closure.Callables) != 1 { + t.Fatalf("Closure has %d closures, want 1", len(closure.Callables)) + } + var lit v2.Callable + for _, c := range closure.Callables { + lit = c + } + wantCallee := functionByName(t, mod, "f").ID + + n, ok := lit.Body["217:10"] + if !ok { + t.Fatalf("closure body has no 217:10 call; keys=%v", keys(lit.Body)) + } + if n.Callee == nil || *n.Callee != wantCallee { + t.Errorf("closure call 217:10 callee = %v, want %s", n.Callee, wantCallee) + } + found := false + for _, e := range out.Application.CallGraph { + if e.Src == lit.ID && e.Dst == wantCallee { + found = true + if len(e.Prov) != 1 || e.Prov[0] != "go/types" { + t.Errorf("closure edge prov = %v, want [go/types]", e.Prov) + } + } + } + if !found { + t.Errorf("call_graph has no edge %s -> %s", lit.ID, wantCallee) + } + assertNoDanglingEdges(t, out) +} + +// TestGateL2_ChiClosureCallsResolve: on chi, closure call nodes resolve and no +// edge dangles. +func TestGateL2_ChiClosureCallsResolve(t *testing.T) { + out := buildChiL2Fresh(t, chiDir(t)) + resolved := 0 + var walk func(c v2.Callable, inClosure bool) + walk = func(c v2.Callable, inClosure bool) { + if inClosure { + for _, n := range c.Body { + if n.Kind == "call" && n.Callee != nil { + resolved++ + } + } + } + for _, in := range c.Callables { + walk(in, true) + } + } + for _, mod := range out.Application.SymbolTable { + for _, fn := range mod.Functions { + walk(fn, false) + } + for _, ty := range mod.Types { + for _, m := range ty.Callables { + walk(m, false) + } + } + } + if resolved == 0 { + t.Error("no closure call node has a callee on chi") + } + t.Logf("chi closure call nodes with a callee: %d", resolved) + assertNoDanglingEdges(t, out) +} + +// TestGateL2_CrossFileMethodClosureEdge: a closure inside a method declared in a +// different file than its receiver type nests under the type's module but is +// declared in the method's file. Its edge must resolve to the closure's id. +func TestGateL2_CrossFileMethodClosureEdge(t *testing.T) { + dir := t.TempDir() + for name, src := range map[string]string{ + "go.mod": "module example.com/p\n\ngo 1.21\n", + "a.go": "package p\n\ntype T struct{}\n", + "b.go": "package p\n\nfunc helper() {}\n\nfunc (t *T) Run() {\n\tfn := func() {\n\t\tinner := func() { helper() }\n\t\tinner()\n\t\thelper()\n\t}\n\tfn()\n}\n", + } { + if err := os.WriteFile(filepath.Join(dir, name), []byte(src), 0o644); err != nil { + t.Fatal(err) + } + } + out := emitDirL2(t, dir, "p") + run := mustModule(t, out, "a.go").Types["T"].Callables["example.com/p.T.Run"] + if len(run.Callables) != 1 { + t.Fatalf("Run has %d closures, want 1", len(run.Callables)) + } + helper := functionByName(t, mustModule(t, out, "b.go"), "helper").ID + var fn v2.Callable + for _, c := range run.Callables { + fn = c + } + var inner v2.Callable + for _, c := range fn.Callables { + inner = c + } + has := map[[2]string]bool{} + for _, e := range out.Application.CallGraph { + has[[2]string{e.Src, e.Dst}] = true + } + for _, c := range []v2.Callable{fn, inner} { + if c.ID == "" || !has[[2]string{c.ID, helper}] { + t.Errorf("missing edge %s -> %s", c.ID, helper) + } + } + assertNoDanglingEdges(t, out) +} diff --git a/internal/syntactic_analysis/v2emit/sigindex.go b/internal/syntactic_analysis/v2emit/sigindex.go index 992af19..00ed238 100644 --- a/internal/syntactic_analysis/v2emit/sigindex.go +++ b/internal/syntactic_analysis/v2emit/sigindex.go @@ -80,8 +80,14 @@ func (idx sigIndex) addCallable(parentID, relPath string, c schema.GoCallable) { } // Precise entry: a callable's declaring file plus its signature is unique - // even for init, because at most one init exists per file. + // even for init, because at most one init exists per file. Call-graph + // edges carry the declaring file (c.Path), which differs from relPath for + // a method declared apart from its receiver type, and for the closures in + // one; index both so either resolves. idx.bySigPath[sigPathKey{sig: c.Signature, path: relPath}] = callID + if c.Path != "" && c.Path != relPath { + idx.bySigPath[sigPathKey{sig: c.Signature, path: c.Path}] = callID + } for _, inner := range c.InnerCallables { idx.addCallable(callID, relPath, inner) From 02a034ec1e0f2ea4fe80dd7a7ab24b6c6321bbc6 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Sun, 25 Oct 2026 16:46:37 -0400 Subject: [PATCH 04/65] feat(l3): model closure capture writes and deferred-operand writes (#22) --- internal/dataflow/defuse.go | 185 ++++++++++++++++++++++++++-- internal/dataflow/defuse_calls.go | 53 ++++++++ internal/dataflow/defuse_entry.go | 79 ++++++++++++ internal/dataflow/defuse_test.go | 68 +++++++++- internal/dataflow/export_test.go | 12 ++ internal/dataflow/reachdefs.go | 15 ++- internal/dataflow/reachdefs_test.go | 78 +++++++++--- internal/dataflow/run.go | 6 +- testdata/dataflow/flow/flow.go | 74 ++++++++++- 9 files changed, 532 insertions(+), 38 deletions(-) diff --git a/internal/dataflow/defuse.go b/internal/dataflow/defuse.go index 4162c57..00f5a39 100644 --- a/internal/dataflow/defuse.go +++ b/internal/dataflow/defuse.go @@ -18,11 +18,18 @@ import ( // // A strong def never has a wildcard step (any write through an index, a // dereference or past k is weak), so a kill never removes more than the -// written path covers. The `:` pseudo path (nil Base) appears only as -// a registration's weak def and its deferred call node's use. Each list is -// deduplicated and sorted by (Spell, base position). +// written path covers. The `:` pseudo path (nil Base) appears as a +// registration's weak def, its deferred call node's use, and the weak def of +// a node writing one of the deferred call's operand bases. +// +// Shallow is the subset of Defs that kills only its own path, not the +// extensions below it: `s = append(s, …)` rebinds s, but the new s keeps the +// old elements, so it kills s and not s[*] or s.* (#28). +// +// Each list is deduplicated and sorted by (Spell, base position). type DefUse struct { Defs, WeakDefs, Uses []Path + Shallow []Path } // DeferPrefix starts the pseudo path a defer registration weak-defines and its @@ -35,6 +42,10 @@ const DeferPrefix = ":" // BuildNodes' map, used to find the deferred call node's key from a defer // registration (keyOf[DeferStmt.Call]). info must be non-nil. // +// The capture-write and deferred-operand rules below need facts about the +// whole callable, which DefUseOf derives from fn and keyOf on every call; +// DefUses computes them once for a node set and returns the same result. +// // Rules by node: // // - @entry defines a base path for every object in entryBases; @exit has @@ -60,7 +71,8 @@ const DeferPrefix = ":" // nested in them) and weak-defines the pseudo path ":", // = keyOf[DeferStmt.Call], the deferred call node's Key. The // deferred call node (Deferred) uses that pseudo path, every named result -// of fn and, for `defer func(){…}()`, what the closure captures; it +// of fn and, for `defer func(){…}()`, what the closure captures (and +// weak-defines what it writes of them: capture writes, below); it // weak-defines what the call may write through (call rule), uses those // same paths (it reads receiver and argument state at exit, not at // registration) and never strong-defines anything (R11: chain @@ -89,8 +101,56 @@ const DeferPrefix = ":" // nothing except append(dst, …), copy(dst, …), delete(m, …) and clear(x), // which weak-define dst[*], m[*], x[*] (copy(dst[i:], …) and append(dst[:0], // …) give dst[*] too: append may write into dst's backing array); append's -// result also flows to the assignment's LHS. +// result also flows to the assignment's LHS. `s = append(s, …)` (the LHS is +// append's destination path, sliceBase stripped) strong-defines s as +// Shallow: the new s keeps the old elements, so an earlier `s[0] = v` still +// reaches a later read of s[0] (#28). +// +// Callable-level rules (spec §4.5 "L3 precision", D1). They only add edges: +// +// - Capture writes (gap 1). A function literal may run at any later point +// (called in place, stored, passed to once.Do, deferred, launched), and +// its writes to the variables it captures are writes of this callable. +// Let W be the union of CaptureWrites over the outermost literals of fn's +// body (a literal nested in another belongs to the outer one's unit, and +// CaptureWrites already includes what it writes). The host of a literal +// (the node whose DefUse is computed from an expression containing it: +// for `defer func(){…}()` the deferred call node, for `go func(){…}()` +// the go node) weak-defines each path of CaptureWrites(lit). Every strong +// def in fn whose base is a base of W is weak instead (@entry excepted: +// nothing reaches it to kill). This is flow-insensitive per captured +// variable on purpose: `h := func(){ x = 1 }; x = 2; h()` must not let +// x = 2 kill the closure's write. +// - Deferred operands (gap 3). For each `defer` in fn's body (not in a +// nested literal), its operand bases are the receiver's base of a method +// value and each argument's base (sliceBase stripped, & stripped); a +// function literal's captures do not count (the deferred call node +// already uses them at exit). Every non-@entry, non-deferred node that +// defines (strongly or weakly) a path on such a base also weak-defines +// that defer's ":", which nothing kills, so `defer +// fill(b); b[0] = s; b = o` keeps b[0] = s → fill. A def before the +// registration gets it too: harmless, an over-approximation. func DefUseOf(n Node, info *types.Info, k int, entryBases []types.Object, fn ast.Node, keyOf map[ast.Node]string) DefUse { + return defUseOf(n, info, k, entryBases, fn, keyOf, newFacts(fn, info, k, keyOf)) +} + +// DefUses is DefUseOf of every node of nodes, keyed by Key, with the +// callable-level facts computed once. +func DefUses(nodes []Node, info *types.Info, k int, entryBases []types.Object, fn ast.Node, keyOf map[ast.Node]string) map[string]DefUse { + return defUses(nodes, info, k, entryBases, fn, keyOf, newFacts(fn, info, k, keyOf)) +} + +func defUses(nodes []Node, info *types.Info, k int, entryBases []types.Object, fn ast.Node, keyOf map[ast.Node]string, f *facts) map[string]DefUse { + out := make(map[string]DefUse, len(nodes)) + for _, n := range nodes { + out[n.Key] = defUseOf(n, info, k, entryBases, fn, keyOf, f) + } + return out +} + +// defUseOf is DefUseOf with the callable-level facts given; nil facts apply +// none of the callable-level rules (the pre-D1 sets, for the gates). +func defUseOf(n Node, info *types.Info, k int, entryBases []types.Object, fn ast.Node, keyOf map[ast.Node]string, f *facts) DefUse { c := &collector{info: info, k: k, seen: map[pathKind]bool{}} switch { case n.Kind == KindEntry: @@ -103,12 +163,14 @@ func DefUseOf(n Node, info *types.Info, k int, entryBases []types.Object, fn ast if !ok { break } + c.facts = f c.add(kindUse, Path{Spell: DeferPrefix + n.Key}) for _, r := range namedResults(fn, info) { c.add(kindUse, Path{Base: r, Spell: r.Name()}) } - if lit, ok := ast.Unparen(call.Fun).(*ast.FuncLit); ok { - c.captures(lit) + lit, _ := ast.Unparen(call.Fun).(*ast.FuncLit) + if lit != nil { + c.captureUses(lit) } c.callWrites(call) // The call runs at exit and reads, then, the state behind what it may @@ -117,17 +179,43 @@ func DefUseOf(n Node, info *types.Info, k int, entryBases []types.Object, fn ast for _, p := range c.du.WeakDefs { c.add(kindUse, p) } + if lit != nil { + // The closure's own writes (gap 1): it already uses its captures. + c.captureWrites(lit) + } default: c.fn = fn c.keyOf = keyOf + c.facts = f c.stmt(n.Stmt) + c.deferOperandWrites() } sortPaths(c.du.Defs) sortPaths(c.du.WeakDefs) sortPaths(c.du.Uses) + sortPaths(c.du.Shallow) return c.du } +// deferOperandWrites weak-defines ":" of every defer whose +// operand base this node defines. +func (c *collector) deferOperandWrites() { + if c.facts == nil || len(c.facts.deferOps) == 0 { + return + } + var spells []string + for _, list := range [][]Path{c.du.Defs, c.du.WeakDefs} { + for _, p := range list { + if p.Base != nil { + spells = append(spells, c.facts.deferOps[p.Base]...) + } + } + } + for _, s := range spells { + c.add(kindWeak, Path{Spell: s}) + } +} + type pathKind struct { p Path kind int @@ -144,22 +232,27 @@ type collector struct { k int fn ast.Node keyOf map[ast.Node]string + facts *facts // callable-level facts; nil applies no callable-level rule du DefUse seen map[pathKind]bool + shallow map[Path]bool weakOnly bool // record every def as weak (a range with `=`) } // fork returns an empty collector with c's settings. func (c *collector) fork() *collector { - return &collector{info: c.info, k: c.k, fn: c.fn, keyOf: c.keyOf, seen: map[pathKind]bool{}} + return &collector{info: c.info, k: c.k, fn: c.fn, keyOf: c.keyOf, facts: c.facts, seen: map[pathKind]bool{}} } // mergeLoop adds a's facts to c, keeping a strong def strong only when other -// strong-defines the same path too. +// strong-defines the same path too (Shallow when either side's is). func (c *collector) mergeLoop(a, other *collector) { for _, p := range a.du.Defs { if other.seen[pathKind{p, kindDef}] { c.add(kindDef, p) + if a.shallow[p] || other.shallow[p] { + c.markShallow(p) + } } else { c.add(kindWeak, p) } @@ -190,6 +283,9 @@ func sortPaths(ps []Path) { } func (c *collector) add(kind int, p Path) { + if kind == kindDef && c.facts != nil && p.Base != nil && c.facts.weakBases[p.Base] { + kind = kindWeak // a base a literal of this callable may write (D1 gap 1) + } if c.seen[pathKind{p, kind}] { return } @@ -204,6 +300,20 @@ func (c *collector) add(kind int, p Path) { } } +// markShallow records strong def p as Shallow; a no-op unless p is in Defs. +func (c *collector) markShallow(p Path) { + if !c.seen[pathKind{p, kindDef}] { + return + } + if c.shallow == nil { + c.shallow = map[Path]bool{} + } + if !c.shallow[p] { + c.shallow[p] = true + c.du.Shallow = append(c.du.Shallow, p) + } +} + func (c *collector) addUse(b pathB) { c.add(kindUse, b.limit(c.k)) } func (c *collector) addWeak(b pathB) { c.add(kindWeak, b.limit(c.k)) } @@ -215,8 +325,11 @@ func (c *collector) stmt(s ast.Node) { for _, r := range s.Rhs { c.use(r) } - for _, l := range s.Lhs { + for i, l := range s.Lhs { c.assignTo(l, compound) + if !compound && len(s.Lhs) == len(s.Rhs) { + c.appendOnto(l, s.Rhs[i]) + } } case *ast.IncDecStmt: c.assignTo(s.X, true) @@ -474,13 +587,63 @@ func (c *collector) recv(ch ast.Expr) { c.indexReads(ch) } -// captures uses every variable function literal lit captures. +// appendOnto marks the strong def of lhs Shallow when rhs is append(lhs, …) +// (or append(lhs[:j], …)): the new slice keeps lhs's elements, so the def +// kills lhs but not lhs[*] (#28). callWrites already weak-defines lhs[*]. +func (c *collector) appendOnto(lhs, rhs ast.Expr) { + if c.facts == nil { + return + } + call, ok := ast.Unparen(rhs).(*ast.CallExpr) + if !ok || len(call.Args) == 0 { + return + } + if name, ok := builtinName(call.Fun, c.info); !ok || name != "append" { + return + } + l, ok := pathOf(lhs, c.info) + if !ok || l.base == nil { + return + } + dst, _ := sliceBase(call.Args[0]) + d, ok := pathOf(dst, c.info) + if !ok { + return + } + if p := l.limit(c.k); p == d.limit(c.k) { + c.markShallow(p) + } +} + +// captures uses every variable function literal lit captures and, with the +// callable-level facts, weak-defines what lit may write of them (D1 gap 1). func (c *collector) captures(lit *ast.FuncLit) { + c.captureUses(lit) + c.captureWrites(lit) +} + +// captureUses uses every variable function literal lit captures. +func (c *collector) captureUses(lit *ast.FuncLit) { for _, v := range freeVars(lit, c.info) { c.add(kindUse, Path{Base: v, Spell: v.Name()}) } } +// captureWrites weak-defines CaptureWrites(lit), with the callable-level +// facts only. +func (c *collector) captureWrites(lit *ast.FuncLit) { + if c.facts == nil { + return + } + ws, ok := c.facts.capW[lit] + if !ok { + ws = CaptureWrites(lit, c.info, c.k) + } + for _, p := range ws { + c.add(kindWeak, p) + } +} + // isQualified reports whether e is a qualified identifier pkg.Name. func isQualified(e *ast.SelectorExpr, info *types.Info) bool { x, ok := e.X.(*ast.Ident) diff --git a/internal/dataflow/defuse_calls.go b/internal/dataflow/defuse_calls.go index 2e223fa..c6ff332 100644 --- a/internal/dataflow/defuse_calls.go +++ b/internal/dataflow/defuse_calls.go @@ -4,6 +4,7 @@ import ( "go/ast" "go/token" "go/types" + "sort" ) // The call rule of DefUseOf (spec §4.5), split into what a call reads and @@ -159,3 +160,55 @@ func refTyped(t types.Type) bool { } return false } + +// deferOperandBases maps each operand base of every `defer` in body (not in a +// nested function literal) to the defers' ":" pseudo paths, +// sorted: the base of a method value's receiver and of each argument path +// (sliceBase and & stripped; a non-path argument has none). A function +// literal's captures are not operands: the deferred call node already uses +// them at exit. A defer keyOf does not name is skipped. +func deferOperandBases(body *ast.BlockStmt, info *types.Info, keyOf map[ast.Node]string) map[types.Object][]string { + out := map[types.Object][]string{} + add := func(e ast.Expr, spell string) { + e, _ = sliceBase(e) + e = ast.Unparen(e) + if u, ok := e.(*ast.UnaryExpr); ok && u.Op == token.AND { + e = u.X + } + b, ok := pathOf(e, info) + if !ok || b.base == nil { + return + } + for _, s := range out[b.base] { + if s == spell { + return + } + } + out[b.base] = append(out[b.base], spell) + } + ast.Inspect(body, func(n ast.Node) bool { + switch d := n.(type) { + case *ast.FuncLit: + return false + case *ast.DeferStmt: + key, ok := keyOf[d.Call] + if !ok { + return true + } + spell := DeferPrefix + key + if sel, ok := ast.Unparen(d.Call.Fun).(*ast.SelectorExpr); ok { + if s := info.Selections[sel]; s != nil && s.Kind() == types.MethodVal { + add(sel.X, spell) + } + } + for _, a := range d.Call.Args { + add(a, spell) + } + } + return true + }) + for _, ss := range out { + sort.Strings(ss) + } + return out +} diff --git a/internal/dataflow/defuse_entry.go b/internal/dataflow/defuse_entry.go index a52543d..30d1c64 100644 --- a/internal/dataflow/defuse_entry.go +++ b/internal/dataflow/defuse_entry.go @@ -4,6 +4,8 @@ import ( "go/ast" "go/types" "sort" + + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" ) // What @entry defines (EntryBases) and the callable-level bases DefUseOf @@ -134,3 +136,80 @@ func freeVars(lit *ast.FuncLit, info *types.Info) []types.Object { }) return out } + +// facts are the callable-level inputs of DefUseOf's D1 rules (spec §4.5 "L3 +// precision"), computed once per callable from its body. +type facts struct { + // capW is CaptureWrites of each outermost function literal of the body. + capW map[*ast.FuncLit][]Path + // weakBases is the base of every path in W, the union of capW: a strong + // def on one of them is weak (gap 1). + weakBases map[types.Object]bool + // deferOps maps an operand base of a registered defer to the ":" + // pseudo paths of the defers it is an operand of, sorted (gap 3). + deferOps map[types.Object][]string +} + +// newFacts computes fn's callable-level facts; nil for a nil info or a +// callable without a body. +func newFacts(fn ast.Node, info *types.Info, k int, keyOf map[ast.Node]string) *facts { + body := funcBody(fn) + if info == nil || body == nil { + return nil + } + f := &facts{capW: map[*ast.FuncLit][]Path{}, weakBases: map[types.Object]bool{}} + for _, l := range astwalk.Literals(body) { + ws := CaptureWrites(l.Lit, info, k) + f.capW[l.Lit] = ws + for _, p := range ws { + f.weakBases[p.Base] = true + } + } + f.deferOps = deferOperandBases(body, info, keyOf) + return f +} + +// CaptureWrites returns the paths function literal lit, or any literal nested +// in it, may write (strong or weak defs, by DefUseOf's def rules, k-limited), +// restricted to bases that are free variables of lit (freeVars: a variable +// declared inside lit, or at package scope, is not a capture). Sorted like a +// DefUse list (by spelling, then base position). +// +// A write is collected wherever it sits in lit's body: in a nested block, a +// loop or branch body, a deferred call (whose writes the deferred call node +// performs) or a nested literal's body. +func CaptureWrites(lit *ast.FuncLit, info *types.Info, k int) []Path { + if lit == nil || info == nil { + return nil + } + free := map[types.Object]bool{} + for _, v := range freeVars(lit, info) { + free[v] = true + } + if len(free) == 0 { + return nil + } + c := &collector{info: info, k: k, seen: map[pathKind]bool{}} + ast.Inspect(lit.Body, func(n ast.Node) bool { + switch s := n.(type) { + case *ast.DeferStmt: + c.stmt(s) + c.callWrites(s.Call) + case ast.Stmt: + c.stmt(s) + } + return true + }) + seen := map[Path]bool{} + var out []Path + for _, list := range [][]Path{c.du.Defs, c.du.WeakDefs} { + for _, p := range list { + if p.Base != nil && free[p.Base] && !seen[p] { + seen[p] = true + out = append(out, p) + } + } + } + sortPaths(out) + return out +} diff --git a/internal/dataflow/defuse_test.go b/internal/dataflow/defuse_test.go index d23d0ed..2976770 100644 --- a/internal/dataflow/defuse_test.go +++ b/internal/dataflow/defuse_test.go @@ -267,12 +267,13 @@ func TestDefUse_DeferRegistrationCarriesArgs(t *testing.T) { wantDU(t, "mu.Unlock() (deferred)", call, nil, ss("mu.*"), ss(d, "err", "mu.*", "n")) // defer func(){…}(): nothing is evaluated at registration but the func - // value; the deferred call uses what the closure captures. + // value; the deferred call uses what the closure captures and (D1 gap 1) + // weak-defines what it writes of them: err = nil. reg, _ = fx.du(t, "Defers", "defer func() {", 0) call, ck = fx.du(t, "Defers", "defer func() {", len("defer ")) d = ":" + ck wantDU(t, "defer func(){…}", reg, nil, ss(d), nil) - wantDU(t, "func(){…}() (deferred)", call, nil, nil, ss(d, "err", "n", "x")) + wantDU(t, "func(){…}() (deferred)", call, nil, ss("err"), ss(d, "err", "n", "x")) reg, _ = fx.du(t, "Defers", "defer mut2(s, x)", 0) call, ck = fx.du(t, "Defers", "defer mut2(s, x)", len("defer ")) @@ -419,3 +420,66 @@ func TestDefUse_SubsliceArgs(t *testing.T) { wantDU(t, "clear(m)", flowDU(t, "ClearAll", "507:2"), nil, ss("m[*]"), ss("m")) wantDU(t, "clear(s[:n])", flowDU(t, "ClearAll", "508:2"), nil, ss("s[*]"), ss("n", "s")) } + +// firstLit is the first function literal in fn's body (outermost first). +func firstLit(t *testing.T, fn string) *ast.FuncLit { + t.Helper() + var lit *ast.FuncLit + ast.Inspect(loadFlow(t).decl(t, fn).Body, func(n ast.Node) bool { + if f, ok := n.(*ast.FuncLit); ok && lit == nil { + lit = f + } + return lit == nil + }) + if lit == nil { + t.Fatalf("%s: no function literal", fn) + } + return lit +} + +// TestDefUse_CaptureWrites (D1 gap 1): CaptureWrites is what a literal or a +// literal nested in it may write of the literal's free variables; the host +// weak-defines it, and a strong def of a captured-and-written base is weak. +func TestDefUse_CaptureWrites(t *testing.T) { + fx := loadFlow(t) + // CaptureNested's f: the nested literal's x = y (y is f's own), and the + // fill(buf) write through buf; z is only read. + wantSet(t, "CaptureWrites(CaptureNested f)", dataflow.CaptureWrites(firstLit(t, "CaptureNested"), fx.info, kTest), "buf.*", "x") + wantSet(t, "CaptureWrites(Closure h)", dataflow.CaptureWrites(firstLit(t, "Closure"), fx.info, kTest)) + + wantDU(t, "CaptureNested 613:2", flowDU(t, "CaptureNested", "613:2"), ss("f"), ss("buf.*", "x"), ss("buf", "x", "z")) + // x, z := 0, 0: x is written by the literal, so its def is weak; z stays. + wantDU(t, "CaptureNested 612:2", flowDU(t, "CaptureNested", "612:2"), ss("z"), ss("x"), nil) + wantDU(t, "CaptureWrite 564:2", flowDU(t, "CaptureWrite", "564:2"), nil, ss("x"), ss("x")) + wantDU(t, "CaptureThenReassign 573:2", flowDU(t, "CaptureThenReassign", "573:2"), nil, ss("x"), nil) + wantDU(t, "OnceDo 588:2", flowDU(t, "OnceDo", "588:2"), nil, ss("cfg", "once.*"), ss("cfg", "once")) + // The deferred closure call node is the host, not the registration: + // err = wrapErr(err) writes err, and err.* through the interface argument. + wantDU(t, "DeferWrap 557:2", flowDU(t, "DeferWrap", "557:2"), nil, ss(":557:8"), nil) + wantDU(t, "DeferWrap 557:8", flowDU(t, "DeferWrap", "557:8"), nil, ss("err", "err.*"), ss(":557:8", "err")) + wantDU(t, "DeferWrap 555:2", flowDU(t, "DeferWrap", "555:2"), nil, ss("err"), nil) + // go func(){ x = <-ch }(): the go node is the host. + wantDU(t, "CaptureGo 627:5", flowDU(t, "CaptureGo", "627:5"), nil, ss("x"), ss("ch", "x")) + wantDU(t, "CaptureGo 626:2", flowDU(t, "CaptureGo", "626:2"), nil, ss("x"), nil) +} + +// TestDefUse_AppendOntoIsShallow (#28): s = append(s, n) strong-defines s +// Shallow (it kills s, not s[*]) besides append's s[*] weak def. +func TestDefUse_AppendOntoIsShallow(t *testing.T) { + du := flowDU(t, "AppendKeepsElems", "604:2") + wantDU(t, "AppendKeepsElems 604:2", du, ss("s"), ss("s[*]"), ss("n", "s")) + wantSet(t, "AppendKeepsElems 604:2 Shallow", du.Shallow, "s") + // out := append(dst[:0], x): the LHS is not the destination. + wantSet(t, "AppendReuse 541:2 Shallow", flowDU(t, "AppendReuse", "541:2").Shallow) +} + +// TestDefUse_DeferOperandWrites (D1 gap 3): a node writing a base the +// registered defer fill(b) takes as an operand also weak-defines its pseudo +// path; the registration and the deferred call node are unchanged. +func TestDefUse_DeferOperandWrites(t *testing.T) { + wantDU(t, "DeferReassign 596:2", flowDU(t, "DeferReassign", "596:2"), nil, ss(":595:8", "b[*]"), ss("b", "s")) + wantDU(t, "DeferReassign 597:2", flowDU(t, "DeferReassign", "597:2"), ss("b"), ss(":595:8"), ss("o")) + wantDU(t, "DeferReassign 595:2", flowDU(t, "DeferReassign", "595:2"), nil, ss(":595:8"), ss("b")) + wantDU(t, "DeferReassign 595:8", flowDU(t, "DeferReassign", "595:8"), nil, ss("b.*"), ss(":595:8", "b.*")) + wantDU(t, "DeferReassign @entry", flowDU(t, "DeferReassign", dataflow.EntryKey), ss("b", "o", "s"), nil, nil) +} diff --git a/internal/dataflow/export_test.go b/internal/dataflow/export_test.go index 0b9aab0..82c38e7 100644 --- a/internal/dataflow/export_test.go +++ b/internal/dataflow/export_test.go @@ -1,8 +1,20 @@ package dataflow +import ( + "go/ast" + "go/types" +) + // CanPanic exposes canPanic to the external test package. var CanPanic = canPanic // PostDominators exposes the immediate post-dominator map to the external // test package. var PostDominators = postDominators + +// DefUsesPreD1 is DefUses without the callable-level D1 rules (capture +// writes, deferred-operand writes, append's Shallow def): the L3 def/use sets +// before #22, for the gate that the D1 rules only add ddg edges. +func DefUsesPreD1(nodes []Node, info *types.Info, k int, entryBases []types.Object, fn ast.Node, keyOf map[ast.Node]string) map[string]DefUse { + return defUses(nodes, info, k, entryBases, fn, keyOf, nil) +} diff --git a/internal/dataflow/reachdefs.go b/internal/dataflow/reachdefs.go index 97dd592..6609930 100644 --- a/internal/dataflow/reachdefs.go +++ b/internal/dataflow/reachdefs.go @@ -16,9 +16,12 @@ import ( // - GEN is every def and weak def of the node. // - KILL is every def d, anywhere in the body, with p.Covers(d.path) for a // strong def p of the node (`x = …` kills x, x.f, x[*]; `x.f = …` kills -// x.f… but not x). Weak defs kill nothing. Strong defs carry no wildcard -// (DefUseOf), so a kill never over-reaches. A deferred call node only -// weak-defines (R11), so nothing is killed through the exit chain. +// x.f… but not x); for a Shallow strong def p, only the defs of p itself +// (`s = append(s, …)` kills s, not s[*]: #28). Weak defs kill nothing. +// Strong defs carry no wildcard (DefUseOf), so a kill never over-reaches. +// A deferred call node only weak-defines (R11), so nothing is killed +// through the exit chain. GEN is applied after KILL, so a node's own weak +// def (the s[*] of `s = append(s, …)`) survives its own strong def. // // A node's uses read IN — the defs reaching it before its own defs apply — // so `y = y + 1` in a loop links to itself across the back edge, never @@ -69,9 +72,13 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { for _, i := range genOf[v] { gen[v][i/64] |= 1 << (i % 64) } + shallow := map[Path]bool{} + for _, p := range du[k].Shallow { + shallow[p] = true + } for _, p := range du[k].Defs { for i, d := range defs { - if p.Covers(d.path) { + if p == d.path || !shallow[p] && p.Covers(d.path) { kill[v][i/64] |= 1 << (i % 64) } } diff --git a/internal/dataflow/reachdefs_test.go b/internal/dataflow/reachdefs_test.go index 2868f0b..41449f2 100644 --- a/internal/dataflow/reachdefs_test.go +++ b/internal/dataflow/reachdefs_test.go @@ -15,15 +15,10 @@ import ( func dd(src, dst, v string) dataflow.DdgEdge { return dataflow.DdgEdge{Src: src, Dst: dst, Var: v} } -// duMap is DefUseOf for every node of the pruned set, keyed by node Key — -// BuildDDG's input, computed the way the analyzer computes it. +// duMap is DefUses of the pruned set, keyed by node Key — BuildDDG's input, +// computed the way the analyzer computes it. func duMap(nodes []dataflow.Node, info *types.Info, fn ast.Node, keyOf map[ast.Node]string) map[string]dataflow.DefUse { - bases := dataflow.EntryBases(fn, info) - out := map[string]dataflow.DefUse{} - for _, n := range nodes { - out[n.Key] = dataflow.DefUseOf(n, info, kTest, bases, fn, keyOf) - } - return out + return dataflow.DefUses(nodes, info, kTest, dataflow.EntryBases(fn, info), fn, keyOf) } // ddgOf builds fn's pruned nodes, CFG, CDG and DDG from the flow fixture with @@ -340,12 +335,13 @@ func TestSlice_BackwardExact(t *testing.T) { // TestDDG_CorpusGate runs BuildDDG over every function and function literal // body in testdata/dataflow and testdata/chi: no panic, the output contract // (checkDDG: endpoints are pruned non-satellite nodes, sorted, deduped), two -// builds deep-equal, and the result independent of input order (nodes, CFG -// edges and every DefUse list reversed). +// builds deep-equal, the result independent of input order (nodes, CFG +// edges and every DefUse list reversed), and a superset of the DDG of the +// pre-D1 def/use sets (the D1 rules only add edges). func TestDDG_CorpusGate(t *testing.T) { _, thisFile, _, _ := runtime.Caller(0) root := filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata") - total, edges := 0, 0 + total, edges, oldEdges := 0, 0, 0 for _, dir := range []string{"dataflow", "chi"} { fset, cases := loadDiffCases(t, filepath.Join(root, dir)) for _, c := range cases { @@ -357,6 +353,14 @@ func TestDDG_CorpusGate(t *testing.T) { checkDDG(t, c.name, kept, got) edges += len(got) + // DefUseOf (callable facts derived per call) agrees with DefUses. + bases := dataflow.EntryBases(c.fn, c.info) + for _, n := range kept { + if one := dataflow.DefUseOf(n, c.info, kTest, bases, c.fn, keyOf); !reflect.DeepEqual(one, du[n.Key]) { + t.Errorf("%s %s: DefUseOf %+v != DefUses %+v", c.name, n.Key, one, du[n.Key]) + } + } + if again := dataflow.BuildDDG(kept, es, duMap(kept, c.info, c.fn, keyOf)); !reflect.DeepEqual(got, again) { t.Errorf("%s: two builds differ", c.name) } @@ -366,17 +370,31 @@ func TestDDG_CorpusGate(t *testing.T) { reverse(re) rdu := map[string]dataflow.DefUse{} for k, v := range du { - rdu[k] = dataflow.DefUse{Defs: reversed(v.Defs), WeakDefs: reversed(v.WeakDefs), Uses: reversed(v.Uses)} + rdu[k] = dataflow.DefUse{Defs: reversed(v.Defs), WeakDefs: reversed(v.WeakDefs), Uses: reversed(v.Uses), Shallow: reversed(v.Shallow)} } if perm := dataflow.BuildDDG(rn, re, rdu); !reflect.DeepEqual(got, perm) { t.Errorf("%s: DDG depends on input order", c.name) } + + // The D1 rules (#22, #28) only add edges: every edge of the + // pre-D1 def/use sets is still there. + have := map[dataflow.DdgEdge]bool{} + for _, g := range got { + have[g] = true + } + old := dataflow.BuildDDG(kept, es, dataflow.DefUsesPreD1(kept, c.info, kTest, dataflow.EntryBases(c.fn, c.info), c.fn, keyOf)) + for _, o := range old { + if !have[o] { + t.Errorf("%s: D1 rules dropped pre-D1 ddg %s→%s on %q", c.name, o.Src, o.Dst, o.Var) + } + } + oldEdges += len(old) } } if total < 100 || edges < 1000 { t.Errorf("ddg gate covered only %d bodies / %d edges", total, edges) } - t.Logf("ddg gate: %d bodies, %d edges", total, edges) + t.Logf("ddg gate: %d bodies, %d edges (%d pre-D1, all kept)", total, edges, oldEdges) } func reverse[T any](s []T) { @@ -435,11 +453,43 @@ func TestDDG_AppendWritesDstBackingArray(t *testing.T) { // chain is return 558:2 → 557:8 (err = wrapErr(err)) → 556:8 (observe(err)) // → @exit. A deferred call node never strong-defines, so err = wrapErr(nil) // (555:2) reaches the return, the rewriting closure and, through it, the -// observer at the end of the chain. +// observer at the end of the chain. The rewriting closure weak-defines its +// captured err (D1 gap 1), so its write reaches the observer too. func TestDDG_DeferredNamedResultWriteNeverKills(t *testing.T) { hasDDG(t, "DeferWrap", dd("555:2", "558:2", "err"), // the bare return reads the named result dd("555:2", "557:8", "err"), // the rewriting closure reads it at exit dd("555:2", "556:8", "err"), // and the def survives past it + dd("557:8", "556:8", "err"), // the rewrite's capture write (D1 gap 1) ) } + +// TestDDG_ClosureCaptureWrites (D1 gap 1): the node hosting a function literal +// weak-defines what the literal may write of its captures, and every strong +// def of a captured-and-written variable in the callable is weak, so the +// closure's write reaches the later read. +func TestDDG_ClosureCaptureWrites(t *testing.T) { + // func(){ x = 1 }() at 564:2 → return x. + hasDDG(t, "CaptureWrite", dd("564:2", "565:2", "x")) + // h := func(){ x = 1 } at 572:2; x = 2 (573:2) does not kill it. + hasDDG(t, "CaptureThenReassign", dd("572:2", "575:2", "x")) + // once.Do(func(){ cfg = load() }) at 588:2 → return cfg. + hasDDG(t, "OnceDo", dd("588:2", "589:2", "cfg")) + // f := func(){ func(){ x = y }() } at 613:2: the nested literal's write. + hasDDG(t, "CaptureNested", dd("613:2", "620:2", "x")) + // go func(){ x = <-ch }() at 627:5: the go node hosts the literal. + hasDDG(t, "CaptureGo", dd("627:5", "628:2", "x")) +} + +// TestDDG_DeferOperandAfterReassign (D1 gap 3): in `defer fill(b); b[0] = s; +// b = o` the element write also weak-defines the defer's pseudo path, which +// b = o does not kill, so it reaches the deferred fill (595:8). +func TestDDG_DeferOperandAfterReassign(t *testing.T) { + hasDDG(t, "DeferReassign", dd("596:2", "595:8", ":595:8")) +} + +// TestDDG_AppendKeepsElementDefs (#28): s = append(s, n) may keep s's backing +// array, so it does not kill the earlier s[0] = 5. +func TestDDG_AppendKeepsElementDefs(t *testing.T) { + hasDDG(t, "AppendKeepsElems", dd("603:2", "605:2", "s[*]")) +} diff --git a/internal/dataflow/run.go b/internal/dataflow/run.go index 9d5be5f..bb5ea1c 100644 --- a/internal/dataflow/run.go +++ b/internal/dataflow/run.go @@ -112,11 +112,7 @@ func runOne(fset *token.FileSet, u Unit, k int) (res *Result) { } nodes, keyOf := BuildNodes(fset, body, u.Calls) kept, cfg := BuildCFG(nodes, keyOf, body, u.Info) - bases := EntryBases(u.Fn, u.Info) - du := make(map[string]DefUse, len(kept)) - for _, n := range kept { - du[n.Key] = DefUseOf(n, u.Info, k, bases, u.Fn, keyOf) - } + du := DefUses(kept, u.Info, k, EntryBases(u.Fn, u.Info), u.Fn, keyOf) return &Result{ Nodes: kept, Cfg: cfg, diff --git a/testdata/dataflow/flow/flow.go b/testdata/dataflow/flow/flow.go index 5c392f8..531c77b 100644 --- a/testdata/dataflow/flow/flow.go +++ b/testdata/dataflow/flow/flow.go @@ -549,11 +549,81 @@ func observe(err error) { _ = err } // DeferWrap has a deferred closure that rewrites the named result err (R11). // The chain runs the rewrite, then the observer, then @exit. Deferred nodes -// never strong-define, and L3 records no def for the closure's write to its -// captured err, so the earlier def of err still reaches the observer. +// never strong-define; the rewrite weak-defines its captured err (D1), so the +// earlier def of err and the rewrite both reach the observer. func DeferWrap() (err error) { err = wrapErr(nil) defer func() { observe(err) }() defer func() { err = wrapErr(err) }() return } + +// CaptureWrite calls a literal that writes the captured x (D1 gap 1). +func CaptureWrite() int { + x := 0 + func() { x = 1 }() + return x +} + +// CaptureThenReassign stores a literal that writes x, rebinds x, then calls +// the literal: x = 2 must not kill the closure's write. +func CaptureThenReassign() int { + x := 0 + h := func() { x = 1 } + x = 2 + h() + return x +} + +type onceT struct{} + +func (o *onceT) Do(f func()) { f() } + +func load() int { return 1 } + +// OnceDo writes cfg through a literal passed to a method. +func OnceDo() int { + var once onceT + var cfg int + once.Do(func() { cfg = load() }) + return cfg +} + +// DeferReassign writes an element of b after registering fill(b), then +// rebinds b (D1 gap 3): the element write still reaches the deferred fill. +func DeferReassign(b, o []byte, s byte) { + defer fill(b) + b[0] = s + b = o +} + +// AppendKeepsElems writes an element, then appends in place (#28): append may +// keep the backing array, so the s[0] def survives s = append(s, n). +func AppendKeepsElems(s []int, n int) int { + s[0] = 5 + s = append(s, n) + return s[0] +} + +// CaptureNested writes x from a literal nested in the stored one, and buf's +// elements through a call. y is the outer literal's own variable and z is +// only read: neither is a capture write. +func CaptureNested(buf []byte) int { + x, z := 0, 0 + f := func() { + y := 0 + func() { x = y; y = 1 }() + fill(buf) + _ = z + } + f() + return x +} + +// CaptureGo launches a literal that writes the captured x: the go node is +// its host. +func CaptureGo(ch chan int) int { + x := 0 + go func() { x = <-ch }() + return x +} From 9d5286c3d9528a76c0fcd06832288b5b98d5c6e3 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Sun, 25 Oct 2026 17:26:36 -0400 Subject: [PATCH 05/65] feat(l3): writes through reference fields of value structs; bucketed kills --- internal/dataflow/bodynodes.go | 27 ++++ internal/dataflow/bodynodes_test.go | 59 +++++++++ internal/dataflow/cfg_test.go | 64 +++++++++ internal/dataflow/defuse.go | 18 ++- internal/dataflow/defuse_calls.go | 71 ++++++++-- internal/dataflow/defuse_test.go | 131 ++++++++++++++++++- internal/dataflow/export_test.go | 7 +- internal/dataflow/reachdefs.go | 34 ++++- internal/dataflow/reachdefs_test.go | 196 ++++++++++++++++++++++++++++ testdata/dataflow/flow/flow.go | 71 ++++++++++ testdata/rangefunc/go.mod | 3 + testdata/rangefunc/r.go | 23 ++++ 12 files changed, 679 insertions(+), 25 deletions(-) create mode 100644 testdata/rangefunc/go.mod create mode 100644 testdata/rangefunc/r.go diff --git a/internal/dataflow/bodynodes.go b/internal/dataflow/bodynodes.go index c690be0..a2d0f14 100644 --- a/internal/dataflow/bodynodes.go +++ b/internal/dataflow/bodynodes.go @@ -53,6 +53,9 @@ import ( // synthetic nodes have NoPos and a nil Stmt. A nil body yields an empty slice // and an empty map (no entry/exit: a bodyless callable has no flow). // +// Spans: a node's End is clampEnd'ed to Pos's file (a malformed file's +// recovered nodes can end past it). +// // The returned map keys every node's Stmt to its key. It also maps the AST // nodes folded into a node to that node's key: a bare or `go` call's // *ast.CallExpr, an if/for/switch init, a for post, a type switch guard, a comm @@ -76,6 +79,7 @@ func BuildNodes(fset *token.FileSet, body *ast.BlockStmt, calls []UnitCall) ([]N out := make([]Node, 0, len(b.nodes)+2) out = append(out, Node{Key: EntryKey, Kind: KindEntry}) for _, n := range b.nodes { + n.End = clampEnd(fset, n.Pos, n.End) p, e := fset.PositionFor(n.Pos, false), fset.PositionFor(n.End, false) n.StartLC, n.EndLC = [2]int{p.Line, p.Column}, [2]int{e.Line, e.Column} out = append(out, n) @@ -84,6 +88,29 @@ func BuildNodes(fset *token.FileSet, body *ast.BlockStmt, calls []UnitCall) ([]N return out, b.keyOf } +// clampEnd keeps a node's End inside the file its Pos is in (#28). The +// parser recovers from a syntax error with nodes whose End is one past the +// end of their file: in a shared FileSet that position has no file (the last +// file) or is the next file's base. Such an End is clamped to the end of +// Pos's file: the recovered node really runs to there (recovery swallowed +// the rest of the file), so the span keeps all of its text and stays in its +// own file. An End before Pos in the same file (no known source of one) is +// clamped to Pos, an empty span: no text is invented. A Pos with no file is +// left as it is. +func clampEnd(fset *token.FileSet, pos, end token.Pos) token.Pos { + tf := fset.File(pos) + if tf == nil { + return end + } + if fset.File(end) != tf { + return tf.Pos(tf.Size()) + } + if end < pos { + return pos + } + return end +} + // RecordedCallCount is the number of L1 call sites body records // (astwalk.Calls): a cross-check on the length of a unit's calls. func RecordedCallCount(body *ast.BlockStmt) int { diff --git a/internal/dataflow/bodynodes_test.go b/internal/dataflow/bodynodes_test.go index 256a5d1..fe6cf44 100644 --- a/internal/dataflow/bodynodes_test.go +++ b/internal/dataflow/bodynodes_test.go @@ -2,6 +2,7 @@ package dataflow_test import ( "go/ast" + "go/parser" "go/token" "go/types" "path/filepath" @@ -416,3 +417,61 @@ func TestBuildNodes_PairsByLparenNotIndex(t *testing.T) { } } } + +// TestBuildNodes_EndClampedToFile (#28): in testdata/malformed the parser +// recovers from each syntax error with nodes whose End lies one past the end +// of the file: in the shared FileSet it has no file (the last file) or is the +// next file's base. BuildNodes clamps such an End to the end of Pos's file +// (and an End before Pos to Pos), so every node's [StartLC, EndLC] is a +// non-inverted range inside the node's own file. +func TestBuildNodes_EndClampedToFile(t *testing.T) { + _, thisFile, _, _ := runtime.Caller(0) + dir := filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "malformed") + fset := token.NewFileSet() + // Parse in name order into one FileSet, as go/packages does, so a bad + // End of one file is the next file's base. + var files []*ast.File + for _, name := range []string{"case.go", "doc.go", "if.go", "range.go", "recv.go"} { + f, _ := parser.ParseFile(fset, filepath.Join(dir, name), nil, parser.SkipObjectResolution) + if f == nil { + t.Fatalf("parse %s: no file", name) + } + files = append(files, f) + } + bad, checked := 0, 0 + for _, f := range files { + tf := fset.File(f.Pos()) + for _, d := range f.Decls { + fd, ok := d.(*ast.FuncDecl) + if !ok || fd.Body == nil { + continue + } + if fset.File(fd.End()) != tf { + bad++ + } + nodes, _ := dataflow.BuildNodes(fset, fd.Body, gateCallKeys(fset, fd.Body)) + for _, n := range nodes { + if n.Pos == token.NoPos { + continue + } + checked++ + p := fset.PositionFor(n.Pos, false) + if fset.File(n.End) != tf && n.End != tf.Pos(tf.Size()) { + t.Errorf("%s %s: End %v is outside Pos's file", p.Filename, n.Key, n.End) + } + if n.End < n.Pos { + t.Errorf("%s %s: End %v before Pos %v", p.Filename, n.Key, n.End, n.Pos) + } + if n.EndLC[0] < n.StartLC[0] || n.EndLC[0] == n.StartLC[0] && n.EndLC[1] < n.StartLC[1] { + t.Errorf("%s %s: EndLC %v before StartLC %v", p.Filename, n.Key, n.EndLC, n.StartLC) + } + if n.EndLC[0] > tf.LineCount() || n.EndLC[0] == 0 { + t.Errorf("%s %s: EndLC %v outside the file's %d lines", p.Filename, n.Key, n.EndLC, tf.LineCount()) + } + } + } + } + if bad < 4 || checked == 0 { + t.Errorf("malformed fixture: %d functions with a bad End, %d nodes checked; want ≥ 4 and > 0", bad, checked) + } +} diff --git a/internal/dataflow/cfg_test.go b/internal/dataflow/cfg_test.go index 9ae2c9a..f575e2c 100644 --- a/internal/dataflow/cfg_test.go +++ b/internal/dataflow/cfg_test.go @@ -9,6 +9,7 @@ import ( "path/filepath" "reflect" "runtime" + "sort" "strings" "testing" @@ -1297,3 +1298,66 @@ func TestCFG_DifferentialCatchesMissingEdge(t *testing.T) { t.Errorf("differential missed the cut edges; got %v", errs) } } + +// TestCFG_RangeIntAndRangeFunc (#28): exact CFGs for the Go 1.23 range forms +// in testdata/rangefunc (a go 1.23 module: testdata/dataflow is go 1.21). A +// range over an integer is a plain loop header; a range over an iterator +// function calls the iterator on every trip, so its header may panic as well +// (canPanic): an exception edge to @exit. Both agree with go/cfg. +func TestCFG_RangeIntAndRangeFunc(t *testing.T) { + _, thisFile, _, _ := runtime.Caller(0) + fset, cases := loadDiffCases(t, filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "rangefunc")) + want := map[string][]dataflow.CfgEdge{ + "example.com/rangefunc.RangeInt": { + e(entry, "8:2", "fallthrough"), + e("8:2", "9:2", "fallthrough"), + e("9:2", "10:3", "true"), + e("9:2", "12:2", "false"), + e("10:3", "9:2", "loop_back"), + e("12:2", exit, "return"), + }, + "example.com/rangefunc.RangeFunc": { + e(entry, "18:2", "fallthrough"), + e("18:2", "19:2", "fallthrough"), + e("19:2", "20:3", "true"), + e("19:2", "22:2", "false"), + e("19:2", exit, "exception"), + e("20:3", "19:2", "loop_back"), + e("22:2", exit, "return"), + }, + } + seen := 0 + for _, c := range cases { + w, ok := want[c.name] + if !ok { + continue + } + seen++ + nodes, keyOf := dataflow.BuildNodes(fset, c.body, gateCallKeys(fset, c.body)) + kept, got := dataflow.BuildCFG(nodes, keyOf, c.body, c.info) + checkContract(t, c.name, kept, got) + sortCfg(w) // BuildCFG orders by (src, dst, kind) as strings + if !reflect.DeepEqual(got, w) { + t.Errorf("%s:\n got %v\nwant %v", c.name, got, w) + } + if diffOne(t.Errorf, c.name, c.body, c.info, keyOf, got) == 0 { + t.Errorf("%s: differential checked no pairs", c.name) + } + } + if seen != len(want) { + t.Errorf("rangefunc: found %d of %d functions", seen, len(want)) + } +} + +func sortCfg(es []dataflow.CfgEdge) { + sort.Slice(es, func(i, j int) bool { + a, b := es[i], es[j] + if a.Src != b.Src { + return a.Src < b.Src + } + if a.Dst != b.Dst { + return a.Dst < b.Dst + } + return a.Kind < b.Kind + }) +} diff --git a/internal/dataflow/defuse.go b/internal/dataflow/defuse.go index 00f5a39..6bfd0da 100644 --- a/internal/dataflow/defuse.go +++ b/internal/dataflow/defuse.go @@ -95,9 +95,12 @@ const DeferPrefix = ":" // weak-defines a.* for each argument of pointer, slice, map, chan, func or // interface type (`&x` gives x.*; a slice expression of a path gives its // base's elements: buf[:n], arr[:] and (*pa)[i:] give buf[*], arr[*], pa[*]) -// and recv.* for a receiver that is -// reference-typed (the same kinds) or bound to a pointer-receiver method. A value-typed struct -// argument is not weak-defined. Conversions write nothing; builtins write +// and recv.* for a receiver that is reference-typed (the same kinds) or bound +// to a pointer-receiver method. A value struct or array argument or receiver +// that may hold a reference (MayHoldRef: a field or element, transitively, of +// a reference kind) weak-defines a.* too, the whole base, never per field +// (D1 gap 2); one that holds no reference is copied and not weak-defined. +// Conversions write nothing; builtins write // nothing except append(dst, …), copy(dst, …), delete(m, …) and clear(x), // which weak-define dst[*], m[*], x[*] (copy(dst[i:], …) and append(dst[:0], // …) give dst[*] too: append may write into dst's backing array); append's @@ -149,9 +152,11 @@ func defUses(nodes []Node, info *types.Info, k int, entryBases []types.Object, f } // defUseOf is DefUseOf with the callable-level facts given; nil facts apply -// none of the callable-level rules (the pre-D1 sets, for the gates). +// none of the D1 rules (the pre-D1 sets, for the gates): no callable-level +// rule, and the call rule writes through reference kinds only (refTyped, not +// MayHoldRef). func defUseOf(n Node, info *types.Info, k int, entryBases []types.Object, fn ast.Node, keyOf map[ast.Node]string, f *facts) DefUse { - c := &collector{info: info, k: k, seen: map[pathKind]bool{}} + c := &collector{info: info, k: k, seen: map[pathKind]bool{}, preD1: f == nil} switch { case n.Kind == KindEntry: for _, b := range entryBases { @@ -237,11 +242,12 @@ type collector struct { seen map[pathKind]bool shallow map[Path]bool weakOnly bool // record every def as weak (a range with `=`) + preD1 bool // call rule over reference kinds only (the pre-D1 gate sets) } // fork returns an empty collector with c's settings. func (c *collector) fork() *collector { - return &collector{info: c.info, k: c.k, fn: c.fn, keyOf: c.keyOf, facts: c.facts, seen: map[pathKind]bool{}} + return &collector{info: c.info, k: c.k, fn: c.fn, keyOf: c.keyOf, facts: c.facts, seen: map[pathKind]bool{}, preD1: c.preD1} } // mergeLoop adds a's facts to c, keeping a strong def strong only when other diff --git a/internal/dataflow/defuse_calls.go b/internal/dataflow/defuse_calls.go index c6ff332..abffbe2 100644 --- a/internal/dataflow/defuse_calls.go +++ b/internal/dataflow/defuse_calls.go @@ -31,12 +31,13 @@ func (c *collector) callReads(call *ast.CallExpr, captures bool) { } } -// callWrites collects the weak defs of call itself: a.* for each -// reference-typed argument (&x gives x.*), recv.* for a reference-typed -// receiver or a pointer-receiver method, and the element writes of append, -// copy, delete and clear. append(dst, …) may write into dst's backing array -// (spare capacity, or dst[:0]), so it weak-defines dst[*] besides its result -// flowing to the assignment. Conversions and other builtins write nothing. +// callWrites collects the weak defs of call itself: a.* for each argument +// that may hold a reference (MayHoldRef: &x gives x.*), recv.* for a +// receiver that may hold one or is bound to a pointer-receiver method, and +// the element writes of append, copy, delete and clear. append(dst, …) may +// write into dst's backing array (spare capacity, or dst[:0]), so it +// weak-defines dst[*] besides its result flowing to the assignment. +// Conversions and other builtins write nothing. func (c *collector) callWrites(call *ast.CallExpr) { if isType(call.Fun, c.info) { return @@ -55,18 +56,27 @@ func (c *collector) callWrites(call *ast.CallExpr) { } if sel, ok := ast.Unparen(call.Fun).(*ast.SelectorExpr); ok { if s := c.info.Selections[sel]; s != nil && s.Kind() == types.MethodVal { - if ptrRecv(s.Obj()) || refTyped(c.info.TypeOf(sel.X)) { + if ptrRecv(s.Obj()) || c.holdsRef(c.info.TypeOf(sel.X)) { c.writeThrough(sel.X) } } } for _, a := range call.Args { - if refTyped(c.info.TypeOf(a)) { + if c.holdsRef(c.info.TypeOf(a)) { c.writeThrough(a) } } } +// holdsRef is the call rule's write-through predicate: MayHoldRef, or the +// reference kinds alone (refTyped) for the pre-D1 gate sets. +func (c *collector) holdsRef(t types.Type) bool { + if c.preD1 { + return refTyped(t) + } + return MayHoldRef(t) +} + // writeThrough weak-defines a.* (x.* for &x) when a is a path, and base[*] // when a slices a path (buf[:n], arr[:], (*p)[i:]): the callee writes the // elements the slice shares with its base. @@ -150,9 +160,9 @@ func isPointer(t types.Type) bool { return ok } -// refTyped reports whether a value of type t lets a callee write state the -// caller sees: pointer, slice, map, chan, func or interface (a type -// parameter's underlying type is its constraint interface, so it counts). +// refTyped reports whether t is itself a reference kind: pointer, slice, +// map, chan, func or interface (a type parameter's underlying type is its +// constraint interface, so it counts). func refTyped(t types.Type) bool { switch underlying(t).(type) { case *types.Pointer, *types.Slice, *types.Map, *types.Chan, *types.Signature, *types.Interface: @@ -161,6 +171,45 @@ func refTyped(t types.Type) bool { return false } +// MayHoldRef reports whether a value of type t, passed by value, lets a +// callee write state the caller sees (spec §4.5 Calls, D1 gap 2): t is a +// reference kind (refTyped), or a struct or array any of whose fields or +// elements, transitively through named types, may hold one (a value +// cfg{items []int} shares items' backing array with its copy). A struct or +// array that holds no reference is copied whole and is false. Cycle-safe: a +// named type already on the walk is not re-entered (a value-only cycle, +// possible only in a package with type errors, holds nothing new). The call +// rule and the L4 by-ref formal_out rule share it. +func MayHoldRef(t types.Type) bool { + return mayHoldRef(t, map[*types.Named]bool{}) +} + +func mayHoldRef(t types.Type, seen map[*types.Named]bool) bool { + if t == nil { + return false + } + if n, ok := types.Unalias(t).(*types.Named); ok { + if seen[n] { + return false + } + seen[n] = true + } + if refTyped(t) { + return true + } + switch u := t.Underlying().(type) { + case *types.Struct: + for i := 0; i < u.NumFields(); i++ { + if mayHoldRef(u.Field(i).Type(), seen) { + return true + } + } + case *types.Array: + return mayHoldRef(u.Elem(), seen) + } + return false +} + // deferOperandBases maps each operand base of every `defer` in body (not in a // nested function literal) to the defers' ":" pseudo paths, // sorted: the base of a method value's receiver and of each argument path diff --git a/internal/dataflow/defuse_test.go b/internal/dataflow/defuse_test.go index 2976770..1fcb072 100644 --- a/internal/dataflow/defuse_test.go +++ b/internal/dataflow/defuse_test.go @@ -163,13 +163,15 @@ func TestDefUse_FieldDefIsPrefixDef(t *testing.T) { func TestDefUse_CallWeakDefsRefArgs(t *testing.T) { fx := loadDU(t) du, _ := fx.du(t, "Calls", "mut(&req", 0) - wantDU(t, "mut(...)", du, nil, ss("m.*", "p.*", "req.*", "s.*"), ss("m", "p", "req", "s", "t")) + // t is a value T, whose field s is a slice: it may hold a reference, so + // the callee may write through it (D1 gap 2). + wantDU(t, "mut(...)", du, nil, ss("m.*", "p.*", "req.*", "s.*", "t.*"), ss("m", "p", "req", "s", "t")) du, _ = fx.du(t, "Calls", "t.Ptr()", 0) wantDU(t, "t.Ptr()", du, nil, ss("t.*"), ss("t")) du, _ = fx.du(t, "Calls", "p.Ptr()", 0) wantDU(t, "p.Ptr()", du, nil, ss("p.*"), ss("p")) du, _ = fx.du(t, "Calls", "t.Val()", 0) - wantDU(t, "t.Val()", du, nil, nil, ss("t")) + wantDU(t, "t.Val()", du, nil, ss("t.*"), ss("t")) // value receiver holding T.s (D1 gap 2) du, _ = fx.du(t, "Calls", "n := len(s)", 0) wantDU(t, "n := len(s)", du, ss("n"), nil, ss("s")) du, _ = fx.du(t, "Calls", "s = append(s, n)", 0) @@ -393,7 +395,7 @@ func TestDefUse_ReferenceTypedReceiverIsWeakDefined(t *testing.T) { du, _ = fx.du(t, "Recv", "h.Set(", 0) wantDU(t, "h.Set(...)", du, nil, ss("h.*"), ss("h")) du, _ = fx.du(t, "Recv", "\tt.Val()\n}\n\nfunc IfShadow", 1) - wantDU(t, "t.Val() value struct", du, nil, nil, ss("t")) + wantDU(t, "t.Val() value struct", du, nil, ss("t.*"), ss("t")) // T.s is a slice (D1 gap 2) } func TestDefUse_IfInitShadows(t *testing.T) { @@ -483,3 +485,126 @@ func TestDefUse_DeferOperandWrites(t *testing.T) { wantDU(t, "DeferReassign 595:8", flowDU(t, "DeferReassign", "595:8"), nil, ss("b.*"), ss(":595:8", "b.*")) wantDU(t, "DeferReassign @entry", flowDU(t, "DeferReassign", dataflow.EntryKey), ss("b", "o", "s"), nil, nil) } + +// TestMayHoldRef (D1 gap 2): the reference kinds, and structs and arrays +// whose fields or elements transitively may hold one, through named types, +// cycle-safe; a reference-free struct or array, a string and a basic do not. +func TestMayHoldRef(t *testing.T) { + i := types.Typ[types.Int] + sl := types.NewSlice(i) + st := func(ts ...types.Type) *types.Struct { + var fs []*types.Var + for n, ft := range ts { + fs = append(fs, types.NewField(0, nil, "f"+string(rune('a'+n)), ft, false)) + } + return types.NewStruct(fs, nil) + } + named := func(name string, u types.Type) *types.Named { + n := types.NewNamed(types.NewTypeName(0, nil, name, nil), nil, nil) + if u != nil { + n.SetUnderlying(u) + } + return n + } + empty := types.NewInterfaceType(nil, nil) + tp := types.NewTypeParam(types.NewTypeName(0, nil, "P", nil), empty) + + // A cycle through value types only (invalid Go, but go/types builds it in + // a package with type errors): A{b B}, B{a A, n int}. + a, b := named("A", nil), named("B", nil) + a.SetUnderlying(st(b)) + b.SetUnderlying(st(a, i)) + // The same cycle with a slice two levels down: C{d D}, D{c C, s []int}. + c, d := named("C", nil), named("D", nil) + c.SetUnderlying(st(d)) + d.SetUnderlying(st(c, sl)) + + for _, tc := range []struct { + name string + t types.Type + want bool + }{ + {"pointer", types.NewPointer(i), true}, + {"slice", sl, true}, + {"map", types.NewMap(i, i), true}, + {"chan", types.NewChan(types.SendRecv, i), true}, + {"func", types.NewSignatureType(nil, nil, nil, nil, nil, false), true}, + {"interface", empty, true}, + {"type parameter", tp, true}, + {"int", i, false}, + {"string", types.Typ[types.String], false}, + {"nil", nil, false}, + {"plain struct", st(i, types.Typ[types.String]), false}, + {"struct with slice", st(i, sl), true}, + {"named struct with slice", named("Cfg", st(sl)), true}, + {"struct nesting a struct with a map", st(st(i, types.NewMap(i, i))), true}, + {"array of int", types.NewArray(i, 2), false}, + {"array of slices", types.NewArray(sl, 2), true}, + {"array of structs with a pointer", types.NewArray(st(types.NewPointer(i)), 3), true}, + {"struct of arrays of int", st(types.NewArray(i, 4)), false}, + {"value cycle, no reference", a, false}, + {"value cycle, reference below", c, true}, + } { + if got := dataflow.MayHoldRef(tc.t); got != tc.want { + t.Errorf("MayHoldRef(%s) = %v, want %v", tc.name, got, tc.want) + } + } +} + +// TestDefUse_ValueStructMayHoldRef (D1 gap 2): a value argument or receiver +// that may hold a reference weak-defines a.* (whole base, never per field); +// a reference-free one is only read. +func TestDefUse_ValueStructMayHoldRef(t *testing.T) { + wantDU(t, "FillCfg 651:2", flowDU(t, "FillCfg", "651:2"), nil, ss("c.*"), ss("c")) + wantDU(t, "ValueRefs 659:2", flowDU(t, "ValueRefs", "659:2"), nil, ss("c.*"), ss("c")) + wantDU(t, "ValueRefs 660:2", flowDU(t, "ValueRefs", "660:2"), nil, nil, ss("p")) + wantDU(t, "ValueRefs 661:2", flowDU(t, "ValueRefs", "661:2"), nil, ss("a.*", "n.*"), ss("a", "b", "n")) + // A value receiver holding no reference is a copy: read only. + wantDU(t, "PlainRecv 698:2", flowDU(t, "PlainRecv", "698:2"), nil, nil, ss("p")) +} + +// TestDefUse_SatelliteParents (#28): a call in a folded for init, cond or +// post, a case list, a select comm clause (send value, receive operand) or a +// range key or expression is a satellite of the header or clause head that +// expression folds into, and carries an empty DefUse. +func TestDefUse_SatelliteParents(t *testing.T) { + _, byKey, _ := build(t, "SatParents") + want := map[string]string{ + "674:11": "674:2", "674:21": "674:2", "674:31": "674:2", // for init, cond, post + "678:7": "678:2", "678:13": "678:2", // case f(1), g(2): + "682:13": "682:2", // case ch <- f(s): + "683:14": "683:2", // case v := <-chanOf(ch): + "687:9": "687:2", "687:26": "687:2", // for xs[f(2)], x = range ids(xs) + } + calls := 0 + for k, n := range byKey { + if n.Kind != dataflow.KindCall { + continue + } + calls++ + p, ok := want[k] + if !ok { + t.Errorf("SatParents: unexpected call node %s (parent %q)", k, n.Parent) + continue + } + if n.Parent != p { + t.Errorf("SatParents: call %s Parent = %q, want %q", k, n.Parent, p) + } + wantDU(t, "SatParents "+k, flowDU(t, "SatParents", k), nil, nil, nil) + } + if calls != len(want) { + t.Errorf("SatParents: %d call nodes, want %d", calls, len(want)) + } + // The hosts carry their satellites' flow. + wantKind(t, "SatParents", byKey, "674:2", dataflow.KindLoop) + wantKind(t, "SatParents", byKey, "678:2", dataflow.KindStatement) + wantKind(t, "SatParents", byKey, "682:2", dataflow.KindStatement) + wantKind(t, "SatParents", byKey, "683:2", dataflow.KindStatement) + wantKind(t, "SatParents", byKey, "687:2", dataflow.KindLoop) + wantDU(t, "SatParents 682:2", flowDU(t, "SatParents", "682:2"), nil, ss("ch[*]"), ss("ch", "s")) + // <-chanOf(ch) receives from a call, not a path: the call reads ch and + // writes through it. + wantDU(t, "SatParents 683:2", flowDU(t, "SatParents", "683:2"), ss("v"), ss("ch.*"), ss("ch")) + // `=` range: weak key and value; ids(xs) writes through xs. + wantDU(t, "SatParents 687:2", flowDU(t, "SatParents", "687:2"), nil, ss("x", "xs.*", "xs[*]"), ss("xs")) +} diff --git a/internal/dataflow/export_test.go b/internal/dataflow/export_test.go index 82c38e7..f6568bf 100644 --- a/internal/dataflow/export_test.go +++ b/internal/dataflow/export_test.go @@ -12,9 +12,10 @@ var CanPanic = canPanic // test package. var PostDominators = postDominators -// DefUsesPreD1 is DefUses without the callable-level D1 rules (capture -// writes, deferred-operand writes, append's Shallow def): the L3 def/use sets -// before #22, for the gate that the D1 rules only add ddg edges. +// DefUsesPreD1 is DefUses without the D1 rules (capture writes, +// deferred-operand writes, append's Shallow def, writes through value +// structs and arrays that may hold a reference): the L3 def/use sets before +// #22, for the gate that the D1 rules only add ddg edges. func DefUsesPreD1(nodes []Node, info *types.Info, k int, entryBases []types.Object, fn ast.Node, keyOf map[ast.Node]string) map[string]DefUse { return defUses(nodes, info, k, entryBases, fn, keyOf, nil) } diff --git a/internal/dataflow/reachdefs.go b/internal/dataflow/reachdefs.go index 6609930..2116631 100644 --- a/internal/dataflow/reachdefs.go +++ b/internal/dataflow/reachdefs.go @@ -1,6 +1,7 @@ package dataflow import ( + "go/types" "math/bits" "sort" ) @@ -22,6 +23,9 @@ import ( // A deferred call node only weak-defines (R11), so nothing is killed // through the exit chain. GEN is applied after KILL, so a node's own weak // def (the s[*] of `s = append(s, …)`) survives its own strong def. +// Covers needs equal bases (and, for a nil base, equal spelling heads), +// so p is tested only against the defs of its kill bucket (#27 item 1): +// O(defs per base) per strong def, not O(defs in the body). // // A node's uses read IN — the defs reaching it before its own defs apply — // so `y = y + 1` in a loop links to itself across the back edge, never @@ -63,6 +67,16 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { } } + // Kill index (#27 item 1): p.Covers(d) and p == d both need the same + // Base, so a strong def is tested only against the defs of its bucket. + // A nil-base path (a pseudo path, an unresolved identifier) is bucketed + // by its spelling head: equal heads are necessary for Covers. + buckets := map[killBucket][]int{} + for i, d := range defs { + b := bucketOf(d.path) + buckets[b] = append(buckets[b], i) + } + words := (len(defs) + 63) / 64 newSet := func() []uint64 { return make([]uint64, words) } gen := make([][]uint64, n) @@ -77,8 +91,8 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { shallow[p] = true } for _, p := range du[k].Defs { - for i, d := range defs { - if p == d.path || !shallow[p] && p.Covers(d.path) { + for _, i := range buckets[bucketOf(p)] { + if d := defs[i]; p == d.path || !shallow[p] && p.Covers(d.path) { kill[v][i/64] |= 1 << (i % 64) } } @@ -158,3 +172,19 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { }) return res } + +// killBucket is the kill index key of a path: its Base, or for a nil Base +// its spelling head (splitSpell's base spelling; a pseudo path is all head). +// Two paths in different buckets never cover one another. +type killBucket struct { + base types.Object + head string +} + +func bucketOf(p Path) killBucket { + if p.Base != nil { + return killBucket{base: p.Base} + } + head, _ := splitSpell(p.Spell) + return killBucket{head: head} +} diff --git a/internal/dataflow/reachdefs_test.go b/internal/dataflow/reachdefs_test.go index 41449f2..bdaec0b 100644 --- a/internal/dataflow/reachdefs_test.go +++ b/internal/dataflow/reachdefs_test.go @@ -493,3 +493,199 @@ func TestDDG_DeferOperandAfterReassign(t *testing.T) { func TestDDG_AppendKeepsElementDefs(t *testing.T) { hasDDG(t, "AppendKeepsElems", dd("603:2", "605:2", "s[*]")) } + +// TestDDG_ValueStructWithSliceIsWrittenThrough (D1 gap 2): fillCfg(c) with a +// value struct c whose field items is a slice may write c.items[*]; the call +// weak-defines c.*, which reaches the later read of c.items[0]. +func TestDDG_ValueStructWithSliceIsWrittenThrough(t *testing.T) { + wantDDG(t, "FillCfg", + dd(entry, "651:2", "c"), + dd(entry, "652:2", "c.items[*]"), + dd("651:2", "652:2", "c.items[*]"), + ) + // A value receiver over a slice field (c.reset()) and value arguments + // that may hold a reference (n nests one, a is an array of slices) are + // written through; p (ints only) and b ([2]int) are not. + hasDDG(t, "ValueRefs", + dd("659:2", "662:2", "c.items[*]"), + dd("661:2", "662:2", "n.in.items[*]"), + dd("661:2", "662:2", "a[*][*]"), + ) + _, _, got := ddgOf(t, "ValueRefs") + for _, g := range got { + if g.Src != entry && (g.Var == "p.a" || g.Var == "b[*]") { + t.Errorf("ValueRefs: a reference-free value operand is written through: %v", g) + } + } +} + +// flatDDG is a reference reaching-definitions DDG with the kill rule of +// BuildDDG tested against every definition of the body (no bucketing), on +// plain map sets: the oracle TestDDG_KillBucketsMatchFlat compares against. +func flatDDG(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]dataflow.DefUse) []dataflow.DdgEdge { + type def struct { + node string + path dataflow.Path + } + isNode := map[string]bool{} + for _, n := range nodes { + if n.Parent == "" { + isNode[n.Key] = true + } + } + pred := map[string][]string{} + for _, e := range cfg { + if isNode[e.Src] && isNode[e.Dst] { + pred[e.Dst] = append(pred[e.Dst], e.Src) + } + } + var all []def + for k := range isNode { + d := du[k] + for _, list := range [][]dataflow.Path{d.Defs, d.WeakDefs} { + for _, p := range list { + all = append(all, def{k, p}) + } + } + } + killed := func(k string, d def) bool { + shallow := map[dataflow.Path]bool{} + for _, p := range du[k].Shallow { + shallow[p] = true + } + for _, p := range du[k].Defs { + if p == d.path || !shallow[p] && p.Covers(d.path) { + return true + } + } + return false + } + in, out := map[string]map[def]bool{}, map[string]map[def]bool{} + for k := range isNode { + in[k], out[k] = map[def]bool{}, map[def]bool{} + } + for changed := true; changed; { + changed = false + for k := range isNode { + for _, p := range pred[k] { + for d := range out[p] { + in[k][d] = true + } + } + next := map[def]bool{} + for d := range in[k] { + if !killed(k, d) { + next[d] = true + } + } + for _, d := range all { + if d.node == k { + next[d] = true + } + } + if len(next) != len(out[k]) { + changed = true + } else { + for d := range next { + if !out[k][d] { + changed = true + break + } + } + } + out[k] = next + } + } + edges := map[dataflow.DdgEdge]bool{} + for k := range isNode { + for d := range in[k] { + for _, u := range du[k].Uses { + if d.path.Covers(u) || u.Covers(d.path) { + edges[dataflow.DdgEdge{Src: d.node, Dst: k, Var: u.Spell}] = true + } + } + } + } + res := []dataflow.DdgEdge{} + for e := range edges { + res = append(res, e) + } + sort.Slice(res, func(i, j int) bool { return ddgLess(res[i], res[j]) }) + return res +} + +// TestDDG_KillBucketsMatchFlat (#27 item 1): BuildDDG tests kills only within +// the bucket of a def's base (nil-base paths by spelling head); it equals the +// flat reference over the whole flow and chi corpus and over a hand-built +// line mixing shadowed bases, unresolved (nil-base) paths sharing a spelling +// with a real base, and pseudo paths. +func TestDDG_KillBucketsMatchFlat(t *testing.T) { + _, thisFile, _, _ := runtime.Caller(0) + root := filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata") + total := 0 + for _, dir := range []string{"dataflow", "chi"} { + fset, cases := loadDiffCases(t, filepath.Join(root, dir)) + for _, c := range cases { + nodes, keyOf := dataflow.BuildNodes(fset, c.body, gateCallKeys(fset, c.body)) + kept, es := dataflow.BuildCFG(nodes, keyOf, c.body, c.info) + du := duMap(kept, c.info, c.fn, keyOf) + got, want := dataflow.BuildDDG(kept, es, du), flatDDG(kept, es, du) + if !reflect.DeepEqual(got, want) { + t.Errorf("%s: bucketed DDG (%d edges) != flat (%d edges)", c.name, len(got), len(want)) + } + total += len(got) + } + } + if total < 1000 { + t.Errorf("kill-bucket gate covered only %d edges", total) + } + + // Hand-built: x (outer), x (shadowing inner, same spelling), the + // unresolved x and u (nil base), and a : pseudo path. + xo := types.NewVar(token.NoPos, nil, "x", types.Typ[types.Int]) + xi := types.NewVar(token.NoPos, nil, "x", types.Typ[types.Int]) + P := func(b types.Object, s ...string) []dataflow.Path { + var out []dataflow.Path + for _, sp := range s { + out = append(out, dataflow.Path{Base: b, Spell: sp}) + } + return out + } + cat := func(ps ...[]dataflow.Path) []dataflow.Path { + var out []dataflow.Path + for _, p := range ps { + out = append(out, p...) + } + return out + } + keys := []string{entry, "1:1", "1:2", "1:3", "1:4", "1:5", "1:6", exit} + var nodes []dataflow.Node + var cfg []dataflow.CfgEdge + for i, k := range keys { + nodes = append(nodes, dataflow.Node{Key: k}) + if i > 0 { + cfg = append(cfg, dataflow.CfgEdge{Src: keys[i-1], Dst: k, Kind: "fallthrough"}) + } + } + cfg = append(cfg, dataflow.CfgEdge{Src: "1:5", Dst: "1:2", Kind: "loop_back"}) + du := map[string]dataflow.DefUse{ + entry: {Defs: cat(P(xo, "x"), P(xi, "x.f"))}, + "1:1": {Defs: P(nil, "x", "u.f"), WeakDefs: P(nil, ":9:9", "u.*")}, + "1:2": {Defs: P(xi, "x"), Uses: cat(P(xo, "x"), P(nil, "x", "u"))}, + "1:3": {Defs: P(nil, "u.f"), Uses: cat(P(xi, "x.f"), P(nil, "u.f.g"))}, + "1:4": {Defs: cat(P(xo, "x"), P(nil, ":9:9", "u")), Shallow: P(xo, "x"), WeakDefs: P(xo, "x[*]")}, + "1:5": {Uses: cat(P(xo, "x[*]"), P(nil, "x", "u", ":9:9"), P(xi, "x"))}, + "1:6": {Defs: P(nil, "u"), Uses: cat(P(xo, "x"), P(nil, "u.f", ":9:9"))}, + } + got, want := dataflow.BuildDDG(nodes, cfg, du), flatDDG(nodes, cfg, du) + if !reflect.DeepEqual(got, want) { + t.Errorf("hand-built kills:\n got %v\nwant %v", got, want) + } + // The unresolved u (1:4) kills u.f of 1:1 and 1:3 by spelling head: only + // 1:4's own u reaches 1:6's u.f. + for _, g := range got { + if g.Dst == "1:6" && g.Var == "u.f" && g.Src != "1:4" { + t.Errorf("hand-built kills: %v survives the unresolved u = … at 1:4", g) + } + } +} diff --git a/testdata/dataflow/flow/flow.go b/testdata/dataflow/flow/flow.go index 531c77b..1f60e61 100644 --- a/testdata/dataflow/flow/flow.go +++ b/testdata/dataflow/flow/flow.go @@ -627,3 +627,74 @@ func CaptureGo(ch chan int) int { go func() { x = <-ch }() return x } + +type cfg struct{ items []int } + +func fillCfg(c cfg) { c.items[0] = 1 } + +func (c cfg) reset() { c.items[0] = 0 } + +type plain struct{ a, b int } + +func usePlain(p plain) { _ = p } + +type nest struct { + in cfg + arr [2][]int +} + +func useNest(n nest, a [2][]int, b [2]int) { _, _, _ = n, a, b } + +// FillCfg passes a value struct holding a slice (D1 gap 2): the callee may +// write c.items[*], so the call weak-defines c.*, which reaches the read. +func FillCfg(c cfg) int { + fillCfg(c) + return c.items[0] +} + +// ValueRefs passes value operands that may hold a reference (a value +// receiver over a slice field, a struct nesting one, an array of slices) and +// two that cannot (a struct and an array of ints). +func ValueRefs(c cfg, p plain, n nest, a [2][]int, b [2]int) int { + c.reset() + usePlain(p) + useNest(n, a, b) + return c.items[0] + p.a + n.in.items[0] + a[0][0] + b[0] +} + +func chanOf(ch chan int) chan int { return ch } + +func ids(xs []int) []int { return xs } + +// SatParents puts calls in a folded for init and post, a case list, select +// comm clauses and a range key and expression (#28): each call is a +// satellite of the header or clause head its expression folds into. +func SatParents(n int, ch chan int, xs []int) int { + s := 0 + for i := f(0); i < g(n); i = f(i) + 1 { + s += i + } + switch n { + case f(1), g(2): + s++ + } + select { + case ch <- f(s): + case v := <-chanOf(ch): + s += v + } + var x int + for xs[f(2)], x = range ids(xs) { + s += x + } + return s +} + +func (p plain) sum() int { return p.a + p.b } + +// PlainRecv calls a value-receiver method on a struct that holds no +// reference: a copy, so nothing is written through it. +func PlainRecv(p plain) int { + p.sum() + return p.a +} diff --git a/testdata/rangefunc/go.mod b/testdata/rangefunc/go.mod new file mode 100644 index 0000000..af1ce4b --- /dev/null +++ b/testdata/rangefunc/go.mod @@ -0,0 +1,3 @@ +module example.com/rangefunc + +go 1.23 diff --git a/testdata/rangefunc/r.go b/testdata/rangefunc/r.go new file mode 100644 index 0000000..8b5993f --- /dev/null +++ b/testdata/rangefunc/r.go @@ -0,0 +1,23 @@ +// Package rangefunc holds the Go 1.23 range forms (#28): a range over an +// integer and a range over an iterator function. testdata/dataflow declares +// go 1.21, which rejects both. +package rangefunc + +// RangeInt ranges over an integer: the header neither calls nor panics. +func RangeInt() int { + s := 0 + for i := range 10 { + s += i + } + return s +} + +// RangeFunc ranges over an iterator function: the header calls seq, so it +// may panic. +func RangeFunc(seq func(func(int) bool)) int { + s := 0 + for v := range seq { + s += v + } + return s +} From e8f219b3e915b89baedd346635c3a57d4d3fc212 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Sun, 25 Oct 2026 18:03:07 -0400 Subject: [PATCH 06/65] perf: stream analysis.json and release ASTs after use (#27) --- cmd/codeanalyzer/main.go | 11 +- internal/core/analyzer.go | 87 ++++++--- internal/core/analyzer_test.go | 47 +++++ internal/dataflow/run.go | 11 ++ internal/dataflow/run_test.go | 50 +++++ .../v2emit/determinism_test.go | 64 +++++++ internal/syntactic_analysis/v2emit/emit.go | 175 +++++++++++++++--- 7 files changed, 390 insertions(+), 55 deletions(-) diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index 296c5ba..7657296 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -177,16 +177,11 @@ via CLDK(language="go").analysis(project_path=...).`, return nil } - // When no --output dir is given, write JSON to cobra's output + // When no --output dir is given, stream JSON to cobra's output // writer so tests can capture it via cmd.SetOut. Both paths render - // through core so the v1/v2 schema selection stays in one place. + // through core so the schema selection stays in one place. if outputDir == "" { - data, err := core.RenderJSON(app, analyzer.L3(), opts) - if err != nil { - return err - } - _, err = cmd.OutOrStdout().Write(data) - return err + return core.StreamJSON(cmd.OutOrStdout(), app, analyzer.L3(), opts) } return core.WriteOutput(app, analyzer.L3(), opts) }, diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index 2329528..655de2e 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -13,8 +13,10 @@ package core import ( + "bufio" "encoding/json" "fmt" + "io" "os" "os/exec" "path/filepath" @@ -114,6 +116,13 @@ func (a *Analyzer) Analyze() (*schema.GoApplication, error) { if a.opts.Level >= options.LevelDataflow { a.l3 = dataflow.Run(builder.Fset(), builder.L3Units(symbolTable), KLimit(a.opts), a.opts.Jobs) utils.Info("dataflow: %d callables", len(a.l3)) + // Run was the last reader of Node.Stmt (the emitters read keys, spans + // and edges only), so drop the AST references now: the parsed + // packages are then unreachable once Analyze returns, before the + // cache write and the emission. (#27) + for _, r := range a.l3 { + r.Release() + } } return a.finalizeAndCache(app) @@ -176,44 +185,78 @@ func (a *Analyzer) saveCache(app *schema.GoApplication) error { return os.WriteFile(cachePath, data, 0o644) } +// checkFormat rejects every output format but json (the default when empty) +// with an explicit error rather than silently falling back. +func checkFormat(opts options.AnalysisOptions) error { + switch opts.Format { + case "", "json": + return nil + case "msgpack": + return fmt.Errorf("msgpack output is not yet implemented; use --format json") + default: + return fmt.Errorf("unsupported output format %q; supported: json", opts.Format) + } +} + // RenderJSON serializes the analysis to JSON as the canonical v2 containment -// tree via the v2emit emitter. This is the single serialization point, so both -// the stdout and file output paths agree. Only "json" is supported; other -// formats return an explicit error rather than silently falling back. +// tree via the v2emit emitter, as one []byte. It is the reference the +// streaming StreamJSON / WriteOutput paths are byte-identical to; the CLI +// streams instead, so the document never exists in memory at once. // // l3 is the Analyzer's L3() side table (nil below level 3). At level 3 the // envelope carries k_limit = KLimit(opts) whether or not l3 is set. func RenderJSON(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, opts options.AnalysisOptions) ([]byte, error) { - format := opts.Format - if format == "" { - format = "json" + if err := checkFormat(opts); err != nil { + return nil, err } - switch format { - case "json": - // only supported format - case "msgpack": - return nil, fmt.Errorf("msgpack output is not yet implemented; use --format json") - default: - return nil, fmt.Errorf("unsupported output format %q; supported: json", format) - } - payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, KLimit(opts), l3) return json.Marshal(payload) } -// WriteOutput serializes the analysis (in the selected schema) and writes it to -// outputDir/analysis.json, or to stdout when outputDir is empty. +// StreamJSON writes the bytes RenderJSON returns to w, emitting and encoding +// one module at a time (v2emit.EmitTo) through a buffered writer. +func StreamJSON(w io.Writer, app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, opts options.AnalysisOptions) error { + if err := checkFormat(opts); err != nil { + return err + } + bw := bufio.NewWriterSize(w, 1<<20) + if err := v2emit.EmitTo(bw, app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, KLimit(opts), l3); err != nil { + return err + } + return bw.Flush() +} + +// WriteOutput streams the analysis to outputDir/analysis.json, or to stdout +// when outputDir is empty. The file is written beside its destination and +// renamed into place, so a failed run never leaves a truncated analysis.json. func WriteOutput(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, opts options.AnalysisOptions) error { - data, err := RenderJSON(app, l3, opts) - if err != nil { + if err := checkFormat(opts); err != nil { return err } if opts.OutputDir == "" { - _, err = os.Stdout.Write(data) - return err + return StreamJSON(os.Stdout, app, l3, opts) } if err := utils.EnsureDir(opts.OutputDir); err != nil { return err } - return os.WriteFile(filepath.Join(opts.OutputDir, "analysis.json"), data, 0o644) + dst := filepath.Join(opts.OutputDir, "analysis.json") + f, err := os.CreateTemp(opts.OutputDir, ".analysis.json.*") + if err != nil { + return err + } + tmp := f.Name() + err = StreamJSON(f, app, l3, opts) + if cerr := f.Close(); err == nil { + err = cerr + } + if err == nil { + err = os.Chmod(tmp, 0o644) + } + if err == nil { + err = os.Rename(tmp, dst) + } + if err != nil { + _ = os.Remove(tmp) + } + return err } diff --git a/internal/core/analyzer_test.go b/internal/core/analyzer_test.go index 9b08ce1..f3940af 100644 --- a/internal/core/analyzer_test.go +++ b/internal/core/analyzer_test.go @@ -1,6 +1,7 @@ package core_test import ( + "bytes" "encoding/json" "os" "path/filepath" @@ -183,6 +184,52 @@ func TestWriteOutput_UnknownFormatErrors(t *testing.T) { } } +// TestWriteOutput_StreamsRenderJSONBytes: the streamed file and writer paths +// write exactly RenderJSON's bytes at -a 3, and Analyze has already released +// every L3 node's AST reference (#27). +func TestWriteOutput_StreamsRenderJSONBytes(t *testing.T) { + opts := options.AnalysisOptions{ + InputPath: filepath.Join(testdataDir(), "dataflow"), AppName: "dataflow", Level: options.LevelDataflow, + SkipTests: true, CacheDir: t.TempDir(), OutputDir: t.TempDir(), + } + a := core.New(opts) + app, err := a.Analyze() + if err != nil { + t.Fatal(err) + } + for key, r := range a.L3() { + for _, n := range r.Nodes { + if n.Stmt != nil { + t.Fatalf("%v: node %s still holds its AST after Analyze", key, n.Key) + } + } + } + want, err := core.RenderJSON(app, a.L3(), opts) + if err != nil { + t.Fatal(err) + } + var buf bytes.Buffer + if err := core.StreamJSON(&buf, app, a.L3(), opts); err != nil { + t.Fatal(err) + } + if !bytes.Equal(buf.Bytes(), want) { + t.Error("StreamJSON bytes differ from RenderJSON") + } + if err := core.WriteOutput(app, a.L3(), opts); err != nil { + t.Fatal(err) + } + got, err := os.ReadFile(filepath.Join(opts.OutputDir, "analysis.json")) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(got, want) { + t.Error("analysis.json differs from RenderJSON") + } + if ents, _ := os.ReadDir(opts.OutputDir); len(ents) != 1 { + t.Errorf("output dir holds %d entries, want only analysis.json", len(ents)) + } +} + // ── Caching tests ───────────────────────────────────────────────────────────── // These tests must run their own analysis to exercise the caching machinery. diff --git a/internal/dataflow/run.go b/internal/dataflow/run.go index bb5ea1c..b79113a 100644 --- a/internal/dataflow/run.go +++ b/internal/dataflow/run.go @@ -130,3 +130,14 @@ func funcBody(fn ast.Node) *ast.BlockStmt { } return nil } + +// Release drops the result's references into the AST (each Node.Stmt), so a +// finished run no longer keeps the parsed packages reachable. Every other +// field (keys, spans, edges) stays: they are all the emitter reads. Call it +// once the last consumer of Stmt is done; Run's own stages are the only one +// today. +func (r Result) Release() { + for i := range r.Nodes { + r.Nodes[i].Stmt = nil + } +} diff --git a/internal/dataflow/run_test.go b/internal/dataflow/run_test.go index 79428c4..92d6981 100644 --- a/internal/dataflow/run_test.go +++ b/internal/dataflow/run_test.go @@ -388,3 +388,53 @@ func TestRun_FlowFixtureHasNoCallCountMismatch(t *testing.T) { t.Errorf("%d results for %d units", len(out), len(units)) } } + +// TestResult_ReleaseDropsASTs: Release nils every Node.Stmt, and the -a 3 +// JSON emitted after it is byte-identical to the JSON emitted before it (no +// emitter reads Stmt). +func TestResult_ReleaseDropsASTs(t *testing.T) { + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "dataflow")) + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatal(err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + out := dataflow.Run(b.Fset(), b.L3Units(st), 3, 4) + emit := func() []byte { + data, err := json.Marshal(v2emit.Emit(app, "dataflow", dir, 3, "rel", 3, out)) + if err != nil { + t.Fatal(err) + } + return data + } + before := emit() + stmts := 0 + for _, r := range out { + for _, n := range r.Nodes { + if n.Stmt != nil { + stmts++ + } + } + r.Release() + } + if stmts == 0 { + t.Fatal("no node had a Stmt before Release") + } + for key, r := range out { + for _, n := range r.Nodes { + if n.Stmt != nil { + t.Errorf("%v: node %s keeps its Stmt after Release", key, n.Key) + } + } + } + if !bytes.Equal(before, emit()) { + t.Error("-a 3 JSON changed after Release") + } +} diff --git a/internal/syntactic_analysis/v2emit/determinism_test.go b/internal/syntactic_analysis/v2emit/determinism_test.go index b7d4d81..6d3a99b 100644 --- a/internal/syntactic_analysis/v2emit/determinism_test.go +++ b/internal/syntactic_analysis/v2emit/determinism_test.go @@ -14,9 +14,14 @@ package v2emit // fails here rather than only in an external CLI diff. import ( + "bytes" "encoding/json" + "fmt" + "path/filepath" + "runtime" "testing" + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" @@ -74,3 +79,62 @@ func mustMarshalCallGraph(t *testing.T, out *v2.Analysis) string { } return string(c.Application.CallGraph) } + +// TestEncode_ByteIdenticalToMarshal locks the streaming writers to +// json.Marshal byte for byte (#27): sorted map keys, omitempty / omitzero, +// the call-node-only callee, HTML escaping (module sources carry `<-` and +// `&`), and no trailing newline. Encode streams a built tree; EmitTo builds +// and streams one module at a time. Both must equal json.Marshal(Emit(...)). +func TestEncode_ByteIdenticalToMarshal(t *testing.T) { + for _, name := range []string{"chi", "dataflow", "generics"} { + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata", name)) + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("%s: symbol table build failed: %v", name, err) + } + apps := map[int]*schema.GoApplication{1: {SymbolTable: st, CallGraph: []schema.GoCallEdge{}}} + check := func(level int, app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result) { + t.Run(fmt.Sprintf("%s/a%d", name, level), func(t *testing.T) { + tree := Emit(app, name, dir, level, "enc-test", 3, l3) + want, err := json.Marshal(tree) + if err != nil { + t.Fatal(err) + } + var enc bytes.Buffer + if err := Encode(&enc, tree); err != nil { + t.Fatal(err) + } + if !bytes.Equal(enc.Bytes(), want) { + t.Errorf("Encode differs from json.Marshal at byte %d (len %d vs %d)", firstDiff(enc.Bytes(), want), enc.Len(), len(want)) + } + var lazy bytes.Buffer + if err := EmitTo(&lazy, app, name, dir, level, "enc-test", 3, l3); err != nil { + t.Fatal(err) + } + if !bytes.Equal(lazy.Bytes(), want) { + t.Errorf("EmitTo differs from json.Marshal(Emit) at byte %d (len %d vs %d)", firstDiff(lazy.Bytes(), want), lazy.Len(), len(want)) + } + }) + } + check(1, apps[1], nil) + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + check(2, app, nil) + check(3, app, dataflow.Run(b.Fset(), b.L3Units(st), 3, 4)) + } +} + +func firstDiff(a, b []byte) int { + for i := 0; i < len(a) && i < len(b); i++ { + if a[i] != b[i] { + return i + } + } + return min(len(a), len(b)) +} diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 90faf73..81ea931 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -1,6 +1,11 @@ package v2emit import ( + "bytes" + "encoding/json" + "fmt" + "io" + "iter" "os" "path/filepath" "sort" @@ -25,48 +30,168 @@ import ( // L3 side table, except span byte offsets, which are derived from source via // lineIndex. func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, analyzerVersion string, kLimit int, l3 map[dataflow.L3Key]dataflow.Result) *v2.Analysis { - appID := v2.AppID(appName) + e := newEmitter(app, appName, projectDir, maxLevel, l3) + out := e.envelope(app, maxLevel, analyzerVersion, kLimit) + out.Application.SymbolTable = make(map[string]v2.Module, len(app.SymbolTable)) + for _, relPath := range sortedFileKeys(app.SymbolTable) { + out.Application.SymbolTable[relPath] = e.module(relPath, app.SymbolTable[relPath]) + } + return out +} + +// EmitTo writes the analysis.json bytes of Emit's payload to w — the same bytes +// json.Marshal(Emit(...)) produces — without building the whole tree: each +// module is emitted, marshalled and dropped in symbol_table key order, so peak +// memory is the envelope plus the largest module, not the document. +func EmitTo(w io.Writer, app *schema.GoApplication, appName, projectDir string, maxLevel int, analyzerVersion string, kLimit int, l3 map[dataflow.L3Key]dataflow.Result) error { + e := newEmitter(app, appName, projectDir, maxLevel, l3) + out := e.envelope(app, maxLevel, analyzerVersion, kLimit) + return encode(w, out, func(yield func(string, v2.Module) bool) { + for _, relPath := range sortedFileKeys(app.SymbolTable) { + if !yield(relPath, e.module(relPath, app.SymbolTable[relPath])) { + return + } + } + }) +} + +// Encode writes a to w as the exact bytes json.Marshal(a) produces (no +// trailing newline), streaming symbol_table one module at a time so the whole +// document never exists as one []byte. +func Encode(w io.Writer, a *v2.Analysis) error { + st := a.Application.SymbolTable + if st == nil { // json.Marshal writes null; nothing to stream + data, err := json.Marshal(a) + if err != nil { + return err + } + _, err = w.Write(data) + return err + } + keys := make([]string, 0, len(st)) + for k := range st { + keys = append(keys, k) + } + sort.Strings(keys) + env := *a + env.Application.SymbolTable = nil + return encode(w, &env, func(yield func(string, v2.Module) bool) { + for _, k := range keys { + if !yield(k, st[k]) { + return + } + } + }) +} + +// symbolTableHole is the envelope's symbol_table as json.Marshal writes it +// when the map is empty. Every field before it is a number or a JSON string; +// inside a string every '"' is written as \", and the needle's second '"' +// follows 'e', so its first match is the field itself. +const symbolTableHole = `"symbol_table":{}` + +// encode writes env (its SymbolTable ignored) with symbol_table filled from +// modules, which must yield keys in sort.Strings order — the order +// json.Marshal writes a map[string] in. Keys are marshalled as JSON strings, +// which is how encoding/json writes a string map key (same escaping). +func encode(w io.Writer, env *v2.Analysis, modules iter.Seq2[string, v2.Module]) error { + shell := *env + shell.Application.SymbolTable = map[string]v2.Module{} + data, err := json.Marshal(&shell) + if err != nil { + return err + } + i := bytes.Index(data, []byte(symbolTableHole)) + if i < 0 { + return fmt.Errorf("v2emit: no %s in the envelope", symbolTableHole) + } + split := i + len(symbolTableHole) - 1 // at the closing '}' + if _, err := w.Write(data[:split]); err != nil { + return err + } + first := true + for k, m := range modules { + if err := writeEntry(w, k, m, first); err != nil { + return err + } + first = false + } + _, err = w.Write(data[split:]) + return err +} + +func writeEntry(w io.Writer, key string, m v2.Module, first bool) error { + k, err := json.Marshal(key) + if err != nil { + return err + } + v, err := json.Marshal(m) + if err != nil { + return err + } + buf := make([]byte, 0, len(k)+len(v)+2) + if !first { + buf = append(buf, ',') + } + buf = append(buf, k...) + buf = append(buf, ':') + if _, err := w.Write(buf); err != nil { + return err + } + _, err = w.Write(v) + return err +} - // The signature→can:// id index translates v1 identity (signature strings) - // into v2 node ids. It is what the L2 refinement uses to backfill body - // callees and to map call_graph edge endpoints onto real callable ids. +// emitter is the per-run state Emit and EmitTo share. +// +// The signature→can:// id index (idx) translates v1 identity (signature +// strings) into v2 node ids. It is what the L2 refinement uses to backfill +// body callees and to map call_graph edge endpoints onto real callable ids. +// +// srcs holds one lineIndex per source file, read once and shared. A method +// whose receiver type lives in a different file is nested under that type's +// module but must have its span computed against ITS OWN file (CLAUDE.md § +// Schema decisions → callable.source_file); the cache lets emitCallable reach +// any file's index without re-reading it per callable. +type emitter struct { + appID string + idx sigIndex + srcs *sourceCache + flow l3Overlay +} + +func newEmitter(app *schema.GoApplication, appName, projectDir string, maxLevel int, l3 map[dataflow.L3Key]dataflow.Result) *emitter { + appID := v2.AppID(appName) idx := buildSigIndex(appID, app.SymbolTable) idx.externals = externalIDs(appID, app.Externals) + return &emitter{appID: appID, idx: idx, srcs: newSourceCache(projectDir), flow: newL3Overlay(maxLevel, l3)} +} - // One lineIndex per source file, read once and shared. A method whose - // receiver type lives in a different file is nested under that type's module - // but must have its span computed against ITS OWN file (CLAUDE.md § Schema - // decisions → callable.source_file); the cache lets emitCallable reach any - // file's index without re-reading it per callable. - srcs := newSourceCache(projectDir) - flow := newL3Overlay(maxLevel, l3) - +// envelope is the payload without its symbol_table (nil): the manifest, the +// call graph and the external symbols. +func (e *emitter) envelope(app *schema.GoApplication, maxLevel int, analyzerVersion string, kLimit int) *v2.Analysis { out := &v2.Analysis{ SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: maxLevel, Analyzer: v2.AnalyzerTag{Name: "codeanalyzer-go", Version: analyzerVersion}, Application: v2.Application{ - ID: appID, - Kind: "application", - SymbolTable: make(map[string]v2.Module, len(app.SymbolTable)), - CallGraph: emitCallGraph(app.CallGraph, idx), + ID: e.appID, + Kind: "application", + CallGraph: emitCallGraph(app.CallGraph, e.idx), }, } - out.Application.ExternalSymbols = emitExternalSymbols(appID, app.Externals) - if flow.on { + out.Application.ExternalSymbols = emitExternalSymbols(e.appID, app.Externals) + if e.flow.on { out.KLimit = kLimit } - - // Deterministic module order does not affect the JSON map, but iterating - // sorted keeps any incidental ordering (and logs) stable across runs. - for _, relPath := range sortedFileKeys(app.SymbolTable) { - file := app.SymbolTable[relPath] - out.Application.SymbolTable[relPath] = emitModule(appID, relPath, file, idx, srcs, flow) - } return out } +func (e *emitter) module(relPath string, file schema.GoFile) v2.Module { + return emitModule(e.appID, relPath, file, e.idx, e.srcs, e.flow) +} + // emitModule builds a v2 module from a v1 GoFile, reading the file's source so // spans can slice off it. A file that cannot be read still emits its structure // (with empty source and zeroed byte spans) rather than dropping the module. From d0eb60f40006f29e891af8a816ee3a5c860c589b Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Sun, 25 Oct 2026 18:12:46 -0400 Subject: [PATCH 07/65] feat(l4): accept -a 4 and declare the L4 schema fields --- README.md | 5 +- cmd/codeanalyzer/main.go | 30 ++-- cmd/codeanalyzer/main_test.go | 162 ++++++++++++++++-- internal/neo4j/schema.go | 6 +- internal/neo4j/schema_test.go | 38 ++++ internal/options/options.go | 8 +- internal/schema/schema.go | 4 + internal/schema/v2/schema.go | 53 +++++- internal/semantic_analysis/call_graph.go | 10 ++ .../v2emit/determinism_test.go | 58 +++++++ internal/syntactic_analysis/v2emit/emit.go | 6 +- .../syntactic_analysis/v2emit/gate_l2_test.go | 20 +++ .../syntactic_analysis/v2emit/gate_l4_test.go | 123 +++++++++++++ 13 files changed, 484 insertions(+), 39 deletions(-) create mode 100644 internal/syntactic_analysis/v2emit/gate_l4_test.go diff --git a/README.md b/README.md index 7050aa0..f3326ac 100644 --- a/README.md +++ b/README.md @@ -162,13 +162,13 @@ Aliases: cango, codeanalyzer-go Flags: - -a, --analysis-level int Analysis level: 1=symbol table, 2=+call graph, 3=+intraprocedural cfg/cdg/ddg (default 1) + -a, --analysis-level int Analysis level: 1=symbol table, 2=+call graph, 3=+intraprocedural cfg/cdg/ddg, 4=+interprocedural SDG (default 1) --app-name string Application anchor name for can:// ids and Neo4j :Application (default: input dir name) -c, --cache-dir string Cache directory (default: ~/.cldk/go-cache) --eager Force clean rebuild (ignore cache) --emit string Output projection: json|neo4j|schema (default "json") -f, --format string Output format: json|msgpack (default "json") - --graph-field-depth int Field-path depth bound for L3 dataflow nodes (k limit; applies at -a 3 only) (default 3) + --graph-field-depth int Field-path depth bound for L3 dataflow nodes (k limit; applies at -a 3 and -a 4) (default 3) -h, --help help for cango -i, --input string Project root to analyze (required) -j, --jobs int Worker parallelism (default: CPU cores) @@ -178,6 +178,7 @@ Flags: --neo4j-user string Neo4j username (env NEO4J_USERNAME, default neo4j) -o, --output string Output directory for analysis.json (default: stdout) --skip-tests Skip *_test.go files (default true) + --strict Fail non-zero on any L4 degradation (default: degrade and record it) -t, --target-files strings Restrict analysis to specific files (incremental mode) -v, --verbose count Verbosity (repeat for more detail) --version Print version and exit diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index 7657296..382f8c1 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -40,6 +40,7 @@ func rootCmd() *cobra.Command { appName string level int fieldDepth int + strict bool targetFiles []string skipTests bool eager bool @@ -79,6 +80,9 @@ via CLDK(language="go").analysis(project_path=...).`, if err := validateAnalysisLevel(level); err != nil { return err } + if level == int(options.LevelSDG) && len(targetFiles) > 0 { + return fmt.Errorf("--target-files is not supported at analysis level 4 (whole-program); run without -t, or use -a 3") + } if err := validateFieldDepth(cmd, level, fieldDepth); err != nil { return err } @@ -91,8 +95,13 @@ via CLDK(language="go").analysis(project_path=...).`, } // Force full depth: the graph carries every implemented // level's facts in both projections. (Output is always the - // canonical v2 tree.) - level = int(options.LevelDataflow) + // canonical v2 tree.) L4 is whole-program, so a targeted run + // projects at L3 and says so. + level = int(options.LevelSDG) + if len(targetFiles) > 0 { + level = int(options.LevelDataflow) + utils.Warn("--target-files: projecting the graph at level 3 (level 4 is whole-program); L4 rows keep the last full run's facts until a full run") + } case options.EmitSchema: path, err := neo4j.EmitSchema(outputDir) if err != nil { @@ -136,6 +145,7 @@ via CLDK(language="go").analysis(project_path=...).`, Emit: options.EmitTarget(emit), AppName: appName, Level: options.AnalysisLevel(level), + Strict: strict, AnalyzerVersion: version, TargetFiles: targetFiles, SkipTests: skipTests, @@ -195,9 +205,10 @@ via CLDK(language="go").analysis(project_path=...).`, f.StringVar(&appName, "app-name", "", "Application anchor name for can:// ids and Neo4j :Application (default: input dir name)") f.IntVarP(&level, "analysis-level", "a", 1, - "Analysis level: 1=symbol table, 2=+call graph, 3=+intraprocedural cfg/cdg/ddg") + "Analysis level: 1=symbol table, 2=+call graph, 3=+intraprocedural cfg/cdg/ddg, 4=+interprocedural SDG") f.IntVar(&fieldDepth, "graph-field-depth", 3, - "Field-path depth bound for L3 dataflow nodes (k limit; applies at -a 3 only)") + "Field-path depth bound for L3 dataflow nodes (k limit; applies at -a 3 and -a 4)") + f.BoolVar(&strict, "strict", false, "Fail non-zero on any L4 degradation (default: degrade and record it)") f.StringSliceVarP(&targetFiles, "target-files", "t", nil, "Restrict analysis to specific files (incremental mode)") f.BoolVar(&skipTests, "skip-tests", true, "Skip *_test.go files") @@ -216,17 +227,12 @@ via CLDK(language="go").analysis(project_path=...).`, return cmd } -// validateAnalysisLevel accepts the implemented levels (1..3). Level 4 is -// reserved for a later release and gets its own message. +// validateAnalysisLevel accepts the implemented levels (1..4). func validateAnalysisLevel(level int) error { - switch { - case level >= 1 && level <= int(options.LevelDataflow): + if level >= 1 && level <= int(options.LevelSDG) { return nil - case level == 4: - return fmt.Errorf("analysis level 4 is not implemented yet") - default: - return fmt.Errorf("analysis level %d is not supported; use 1, 2 or 3", level) } + return fmt.Errorf("analysis level %d is not supported; use 1, 2, 3 or 4", level) } // validateFieldDepth checks --graph-field-depth. The flag only means something diff --git a/cmd/codeanalyzer/main_test.go b/cmd/codeanalyzer/main_test.go index cc35426..5c147a4 100644 --- a/cmd/codeanalyzer/main_test.go +++ b/cmd/codeanalyzer/main_test.go @@ -353,22 +353,6 @@ func TestRootCmd_Level3Accepted(t *testing.T) { } } -func TestRootCmd_Level4RejectedInThisRelease(t *testing.T) { - td := cliTestdataDir() - - _, _, err := runCmd( - "--input", filepath.Join(td, "greeter"), - "--analysis-level", "4", - "--cache-dir", t.TempDir(), - ) - if err == nil { - t.Fatal("expected error for -a 4, got nil") - } - if err.Error() != "analysis level 4 is not implemented yet" { - t.Errorf("error = %q, want %q", err.Error(), "analysis level 4 is not implemented yet") - } -} - func TestRootCmd_UnsupportedLevelRejected(t *testing.T) { td := cliTestdataDir() @@ -503,3 +487,149 @@ func TestRootCmd_MalformedSourceDoesNotPanic(t *testing.T) { }) } } + +// ── L4 scaffolding ─────────────────────────────────────────────────────────── + +const ( + errL4WithTargetFiles = "--target-files is not supported at analysis level 4 (whole-program); run without -t, or use -a 3" + warnNeo4jTargetFiles = "--target-files: projecting the graph at level 3 (level 4 is whole-program); L4 rows keep the last full run's facts until a full run" +) + +func TestCLI_Level4Accepted(t *testing.T) { + td := cliTestdataDir() + out, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-level", "4", + "--graph-field-depth", "2", + "--cache-dir", t.TempDir(), + ) + if err != nil { + t.Fatalf("-a 4 should succeed; got %v", err) + } + var v struct { + MaxLevel int `json:"max_level"` + KLimit int `json:"k_limit"` + } + if jsonErr := json.Unmarshal([]byte(out), &v); jsonErr != nil { + t.Fatalf("stdout is not valid JSON: %v", jsonErr) + } + if v.MaxLevel != 4 { + t.Errorf("max_level = %d, want 4", v.MaxLevel) + } + if v.KLimit != 2 { + t.Errorf("k_limit = %d, want 2 (--graph-field-depth is valid at -a 4)", v.KLimit) + } +} + +func TestCLI_Level4WithTargetFilesRejected(t *testing.T) { + td := cliTestdataDir() + _, _, err := runCmd( + "--input", filepath.Join(td, "multipackage"), + "--analysis-level", "4", + "--target-files", filepath.Join(td, "multipackage", "server", "server.go"), + "--cache-dir", t.TempDir(), + ) + if err == nil { + t.Fatal("expected an error for -a 4 -t, got nil") + } + if err.Error() != errL4WithTargetFiles { + t.Errorf("error = %q, want %q", err.Error(), errL4WithTargetFiles) + } +} + +func TestCLI_Level3WithTargetFilesStillWorks(t *testing.T) { + td := cliTestdataDir() + out, _, err := runCmd( + "--input", filepath.Join(td, "multipackage"), + "--analysis-level", "3", + "--target-files", filepath.Join(td, "multipackage", "server", "server.go"), + "--cache-dir", t.TempDir(), + ) + if err != nil { + t.Fatalf("-a 3 -t should succeed; got %v", err) + } + if !strings.Contains(out, `"server/server.go"`) { + t.Error("-a 3 -t: server/server.go should appear in output") + } +} + +func TestCLI_Neo4jWithTargetFilesFallsBackToL3(t *testing.T) { + td := cliTestdataDir() + out := t.TempDir() + var runErr error + stderr := captureStderr(t, func() { + _, _, runErr = runCmd( + "--input", filepath.Join(td, "multipackage"), + "--emit", "neo4j", + "--target-files", filepath.Join(td, "multipackage", "server", "server.go"), + "--output", out, + "--cache-dir", t.TempDir(), + ) + }) + if runErr != nil { + t.Fatalf("--emit neo4j -t should succeed; got %v", runErr) + } + if n := strings.Count(stderr, warnNeo4jTargetFiles); n != 1 { + t.Errorf("D9 warning appeared %d times, want exactly 1; stderr:\n%s", n, stderr) + } + data, err := os.ReadFile(filepath.Join(out, "graph.cypher")) + if err != nil { + t.Fatalf("graph.cypher not written: %v", err) + } + cy := string(data) + // Effective level 3: the :GoApplication node carries max_level 3 and the + // L3 overlay is present. + if !strings.Contains(cy, "max_level: 3") && !strings.Contains(cy, `"max_level":3`) && !strings.Contains(cy, "max_level = 3") { + t.Errorf("graph.cypher does not record max_level 3") + } + for _, rel := range []string{"GO_CFG_NEXT", "GO_DDG"} { + if !strings.Contains(cy, rel) { + t.Errorf("graph.cypher should carry the L3 %s relationship", rel) + } + } + // No L4 rows. Trivially true until the L4 projection lands (Task 16), + // where it becomes the real guard. + for _, rel := range []string{"GO_PARAM_IN", "GO_PARAM_OUT", "GO_SUMMARY"} { + if strings.Contains(cy, rel) { + t.Errorf("graph.cypher must carry no %s rows under -t", rel) + } + } +} + +func TestCLI_Neo4jWithoutTargetFilesWarnsNothing(t *testing.T) { + td := cliTestdataDir() + var runErr error + stderr := captureStderr(t, func() { + _, _, runErr = runCmd( + "--input", filepath.Join(td, "greeter"), + "--emit", "neo4j", + "--output", t.TempDir(), + "--cache-dir", t.TempDir(), + ) + }) + if runErr != nil { + t.Fatalf("--emit neo4j should succeed; got %v", runErr) + } + if strings.Contains(stderr, "projecting the graph at level 3") { + t.Errorf("the D9 warning must appear only under -t; stderr:\n%s", stderr) + } +} + +func TestCLI_StrictFlag(t *testing.T) { + td := cliTestdataDir() + if rootCmd().Flags().Lookup("strict") == nil { + t.Fatal("--strict flag is not declared") + } + out, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-level", "4", + "--strict", + "--cache-dir", t.TempDir(), + ) + if err != nil { + t.Fatalf("-a 4 --strict should succeed on a clean project; got %v", err) + } + if !json.Valid([]byte(out)) { + t.Error("stdout is not valid JSON") + } +} diff --git a/internal/neo4j/schema.go b/internal/neo4j/schema.go index 6fae733..f168408 100644 --- a/internal/neo4j/schema.go +++ b/internal/neo4j/schema.go @@ -184,7 +184,7 @@ type RelSpec struct { var Nodes = []NodeSpec{ { Label: LabelApplication, MergeLabel: LabelApplication, Key: "id", CanNode: true, - Props: []string{"schema_version", "language", "max_level", "analyzer_name", "analyzer_version"}, + Props: []string{"schema_version", "language", "max_level", "analyzer_name", "analyzer_version", "degraded_json"}, }, { Label: LabelModule, MergeLabel: LabelModule, Key: "id", CanNode: true, @@ -269,7 +269,7 @@ var Rels = []RelSpec{ {Type: RelDDG, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Props: []string{"var", "prov"}, Discriminant: "var|prov"}, // Reserved L4 overlay (zero rows until L4 lands). - {Type: RelParamIn, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Reserved: true}, - {Type: RelParamOut, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Reserved: true}, + {Type: RelParamIn, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Props: []string{"var"}, Reserved: true}, + {Type: RelParamOut, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Props: []string{"var"}, Reserved: true}, {Type: RelSummary, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Reserved: true}, } diff --git a/internal/neo4j/schema_test.go b/internal/neo4j/schema_test.go index 212cd61..746308c 100644 --- a/internal/neo4j/schema_test.go +++ b/internal/neo4j/schema_test.go @@ -252,3 +252,41 @@ func TestContract_L3OverlayNotReserved(t *testing.T) { } } } + +// TestContract_L4RelsCarryVar: the L4 param relationships declare the `var` +// property they will carry, and stay Reserved until the L4 projection lands; +// :GoApplication declares degraded_json for the degradation marker. +func TestContract_L4RelsCarryVar(t *testing.T) { + seen := map[string]bool{} + for _, r := range Rels { + switch r.Type { + case RelParamIn, RelParamOut: + seen[r.Type] = true + if len(r.Props) != 1 || r.Props[0] != "var" { + t.Errorf("%s Props = %v, want [var]", r.Type, r.Props) + } + if !r.Reserved { + t.Errorf("%s must stay Reserved until Task 16", r.Type) + } + case RelSummary: + if !r.Reserved { + t.Errorf("%s must stay Reserved until Task 16", r.Type) + } + } + } + if len(seen) != 2 { + t.Errorf("param relationships declared: %v", seen) + } + for _, n := range Nodes { + if n.Label != LabelApplication { + continue + } + found := false + for _, p := range n.Props { + found = found || p == "degraded_json" + } + if !found { + t.Errorf(":GoApplication props %v lack degraded_json", n.Props) + } + } +} diff --git a/internal/options/options.go b/internal/options/options.go index 1006679..d4e1ecc 100644 --- a/internal/options/options.go +++ b/internal/options/options.go @@ -11,6 +11,8 @@ const ( LevelCallGraph AnalysisLevel = 2 // LevelDataflow adds intraprocedural CFG/CDG/DDG on top of the call graph. LevelDataflow AnalysisLevel = 3 + // LevelSDG adds the interprocedural system dependence graph (L4) on top of L3. + LevelSDG AnalysisLevel = 4 ) // EmitTarget selects the analyzer's output projection (cli-contract.md § Neo4j). @@ -38,9 +40,11 @@ type AnalysisOptions struct { // AppName anchors application identity (can:// and the Neo4j // :Application node). Defaults to the input directory's base name. AppName string - // AnalysisLevel controls symbol-table-only (1), + call graph (2), or - // + intraprocedural dataflow (3). + // AnalysisLevel controls symbol-table-only (1), + call graph (2), + // + intraprocedural dataflow (3), or + the interprocedural SDG (4). Level AnalysisLevel + // Strict makes the run fail non-zero on any L4 degradation (--strict). + Strict bool // AnalyzerVersion is the analyzer's own version, recorded in the v2 // manifest's analyzer{} tag. Set by the CLI from its build-stamped value. AnalyzerVersion string diff --git a/internal/schema/schema.go b/internal/schema/schema.go index 7c2d377..68b6538 100644 --- a/internal/schema/schema.go +++ b/internal/schema/schema.go @@ -215,6 +215,10 @@ type GoExternalCallee struct { Name string `json:"name"` Params []GoParameter `json:"parameters"` ReturnType string `json:"return_type,omitempty"` + // Receiver is the method receiver (nil for a plain function): named by the + // receiver variable, or "_recv" when unnamed or "_"; typed by the origin's + // receiver type string. + Receiver *GoParameter `json:"receiver,omitempty"` } // GoEntrypoint marks a callable as a framework entry point. diff --git a/internal/schema/v2/schema.go b/internal/schema/v2/schema.go index 2293c09..81b5b0d 100644 --- a/internal/schema/v2/schema.go +++ b/internal/schema/v2/schema.go @@ -6,9 +6,11 @@ // The model is ONE additive containment tree // (application → module → type → callable → body) with typed edge overlays // laid over it (a Code Property Graph). This package declares the L1 tree, the -// L2 refinement/edge slots and the L3 body completion (statement nodes, body -// node id/parent, the callable's cfg/cdg/ddg). L4 fields (param_*) are not -// declared here — they arrive in their own train and reuse these ids/spans +// L2 refinement/edge slots, the L3 body completion (statement nodes, body +// node id/parent, the callable's cfg/cdg/ddg) and the L4 fields (the envelope's +// degraded marker, body node `of`, a callable's summary, the application's +// param_in/param_out, an external symbol's receiver and body). The L4 fields +// are all omitempty and absent below -a 4; they reuse the L1-L3 ids/spans // unchanged. // // Conventions from the keystone, held here: @@ -55,6 +57,19 @@ type Analysis struct { KLimit int `json:"k_limit,omitempty"` Analyzer AnalyzerTag `json:"analyzer"` Application Application `json:"application"` + // Degraded records the L4 degradation markers (spec §4.15); absent when + // the run degraded nothing and below -a 4. + Degraded []Degraded `json:"degraded,omitempty"` +} + +// Degraded is one degradation marker: an L4 step that fell back to a coarser +// result. Code is a closed vocabulary; Detail, Packages and Callables say what +// was affected. +type Degraded struct { + Code string `json:"code"` + Detail string `json:"detail,omitempty"` + Packages []string `json:"packages,omitempty"` + Callables []string `json:"callables,omitempty"` } // AnalyzerTag identifies the producing analyzer and version. @@ -78,6 +93,19 @@ type Application struct { // reaches, keyed by their `can:///@external//` id. // Absent when the analysis saw no library call. ExternalSymbols map[string]ExternalSymbol `json:"external_symbols,omitempty"` + // ParamIn and ParamOut are the L4 interprocedural parameter edges between + // global body-node ids. Absent below -a 4. + ParamIn []ParamEdge `json:"param_in,omitempty"` + ParamOut []ParamEdge `json:"param_out,omitempty"` +} + +// ParamEdge is an L4 interprocedural parameter edge (call-site actual → +// callee formal for param_in, callee formal/return → call-site result for +// param_out) on the access path Var. +type ParamEdge struct { + Src string `json:"src"` + Dst string `json:"dst"` + Var string `json:"var"` } // ExternalSymbol is a library callable that call-graph edges and call nodes @@ -93,6 +121,12 @@ type ExternalSymbol struct { Signature string `json:"signature"` Parameters []ExternalParameter `json:"parameters"` ReturnType string `json:"return_type,omitempty"` + // Receiver is the method receiver (absent on a plain function): the name + // is the receiver variable, or _recv when unnamed or `_`; the type is the + // origin's receiver type string. + Receiver *ExternalParameter `json:"receiver,omitempty"` + // Body holds the L4 stand-in body nodes of a library callee (spec §6). + Body map[string]BodyNode `json:"body,omitempty"` } // ExternalParameter is a library callee's formal parameter. It has no span: the @@ -197,6 +231,9 @@ type Callable struct { Cfg []CfgEdge `json:"cfg,omitempty"` Cdg []CdgEdge `json:"cdg,omitempty"` Ddg []DdgEdge `json:"ddg,omitempty"` + // Summary is the L4 summary edge list (formal-in → formal-out through the + // callable); endpoints are local body keys. Absent below -a 4. + Summary []SummaryEdge `json:"summary,omitempty"` // Callables are nested closures / function literals, each with its own // can:// id (replaces v1 InnerCallables). Callables map[string]Callable `json:"callables,omitempty"` @@ -238,6 +275,16 @@ type BodyNode struct { IsGoroutine bool `json:"is_goroutine,omitempty"` // IsDeferred is true when the call is preceded by `defer` (net-new vs v1). IsDeferred bool `json:"is_deferred,omitempty"` + // Of names the value an L4 vertex carries (, :, a + // parameter name, :, :::). Absent below + // -a 4 and on L1-L3 nodes. + Of string `json:"of,omitempty"` +} + +// SummaryEdge is an L4 summary edge between two local body keys of one callable. +type SummaryEdge struct { + Src string `json:"src"` + Dst string `json:"dst"` } // MarshalJSON writes the callee key on call nodes only: it is the call diff --git a/internal/semantic_analysis/call_graph.go b/internal/semantic_analysis/call_graph.go index b71a214..6696cfd 100644 --- a/internal/semantic_analysis/call_graph.go +++ b/internal/semantic_analysis/call_graph.go @@ -289,12 +289,22 @@ func (cg *CallGraphBuilder) recordExternal(fn *types.Func, fullSig string) { ret = "(" + strings.Join(rets, ", ") + ")" } + var receiver *schema.GoParameter + if recv := sig.Recv(); recv != nil { + name := recv.Name() + if name == "" || name == "_" { + name = "_recv" + } + receiver = &schema.GoParameter{Name: name, Type: types.TypeString(recv.Type(), nil)} + } + cg.externals[fullSig] = schema.GoExternalCallee{ ImportPath: importPath, Sig: strings.TrimPrefix(fullSig, importPath+"."), Name: fn.Name(), Params: params, ReturnType: ret, + Receiver: receiver, } } diff --git a/internal/syntactic_analysis/v2emit/determinism_test.go b/internal/syntactic_analysis/v2emit/determinism_test.go index 6d3a99b..012ce49 100644 --- a/internal/syntactic_analysis/v2emit/determinism_test.go +++ b/internal/syntactic_analysis/v2emit/determinism_test.go @@ -138,3 +138,61 @@ func firstDiff(a, b []byte) int { } return min(len(a), len(b)) } + +// TestEncode_L4EnvelopeFieldsByteIdentical: the L4 envelope and tree fields +// (degraded after application, param_in/out inside it, summary, of, receiver, +// external body) go through the streaming writer's envelope split unchanged. +func TestEncode_L4EnvelopeFieldsByteIdentical(t *testing.T) { + call := "x" + a := &v2.Analysis{ + SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: 4, KLimit: 3, + Analyzer: v2.AnalyzerTag{Name: "codeanalyzer-go", Version: "t"}, + Degraded: []v2.Degraded{{Code: "ssa_build", Detail: `has "symbol_table":{} & `, Packages: []string{"p"}, Callables: []string{"c"}}}, + Application: v2.Application{ + ID: "can://app", Kind: "application", + SymbolTable: map[string]v2.Module{ + "a.go": {ID: "m", Functions: map[string]v2.Callable{"f": { + ID: "f", Summary: []v2.SummaryEdge{{Src: "a", Dst: "b"}}, + Body: map[string]v2.BodyNode{ + "@fin": {Kind: "formal_in", Of: "p0"}, + "c1": {Kind: "call", Callee: &call}, + }, + }}}, + }, + ExternalSymbols: map[string]v2.ExternalSymbol{"e": { + ID: "e", Kind: "external", + Receiver: &v2.ExternalParameter{Name: "_recv", Type: "*strings.Builder"}, + Body: map[string]v2.BodyNode{"@entry": {Kind: "entry"}}, + }}, + ParamIn: []v2.ParamEdge{{Src: "s", Dst: "d", Var: "v"}}, + ParamOut: []v2.ParamEdge{{Src: "d", Dst: "s", Var: ""}}, + }, + } + want, err := json.Marshal(a) + if err != nil { + t.Fatal(err) + } + for _, k := range []string{`"degraded":`, `"param_in":`, `"param_out":`, `"summary":`, `"of":"p0"`, `"receiver":`} { + if !bytes.Contains(want, []byte(k)) { + t.Errorf("marshalled tree lacks %s", k) + } + } + var enc bytes.Buffer + if err := Encode(&enc, a); err != nil { + t.Fatal(err) + } + if !bytes.Equal(enc.Bytes(), want) { + t.Errorf("Encode differs from json.Marshal at byte %d", firstDiff(enc.Bytes(), want)) + } + // And absent when empty. + plain, _ := json.Marshal(&v2.Analysis{}) + for _, k := range []string{"degraded", "param_in", "param_out", "summary", `"of"`, "receiver"} { + if bytes.Contains(plain, []byte(k)) { + t.Errorf("empty analysis serialises %s", k) + } + } + n, _ := json.Marshal(v2.BodyNode{Kind: "statement"}) + if bytes.Contains(n, []byte(`"of"`)) { + t.Errorf("empty Of serialised: %s", n) + } +} diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 81ea931..8317d26 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -399,10 +399,14 @@ func emitExternalSymbols(appID string, externals map[string]schema.GoExternalCal for _, p := range ext.Params { params = append(params, v2.ExternalParameter{Name: p.Name, Type: p.Type, IsVariadic: p.IsVariadic}) } - out[id] = v2.ExternalSymbol{ + sym := v2.ExternalSymbol{ ID: id, Kind: "external", Name: ext.Name, Module: ext.ImportPath, Signature: ext.Sig, Parameters: params, ReturnType: ext.ReturnType, } + if ext.Receiver != nil { + sym.Receiver = &v2.ExternalParameter{Name: ext.Receiver.Name, Type: ext.Receiver.Type} + } + out[id] = sym } return out } diff --git a/internal/syntactic_analysis/v2emit/gate_l2_test.go b/internal/syntactic_analysis/v2emit/gate_l2_test.go index 16abf16..8fadd90 100644 --- a/internal/syntactic_analysis/v2emit/gate_l2_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l2_test.go @@ -337,6 +337,26 @@ func F() string { if len(sym.Parameters) != 1 || sym.Parameters[0].Name != "s" { t.Errorf("params = %+v, want exactly [s] (receiver excluded)", sym.Parameters) } + if sym.Receiver == nil || sym.Receiver.Name != "b" || sym.Receiver.Type != "*strings.Builder" { + t.Errorf("receiver = %+v, want {b *strings.Builder}", sym.Receiver) + } +} + +// TestGateL2_ExternalFunctionHasNoReceiver: only methods carry a receiver. +func TestGateL2_ExternalFunctionHasNoReceiver(t *testing.T) { + out := emitTemp(t, `package p + +import "strings" + +func F() string { return strings.ToUpper("x") } +`) + sym, ok := out.Application.ExternalSymbols[v2.ExternalID(out.Application.ID, "strings", "ToUpper")] + if !ok { + t.Fatalf("external_symbols missing strings.ToUpper") + } + if sym.Receiver != nil { + t.Errorf("function has receiver %+v", sym.Receiver) + } } // TestGateL2_ConversionIsNotACall locks that a type conversion is not minted as diff --git a/internal/syntactic_analysis/v2emit/gate_l4_test.go b/internal/syntactic_analysis/v2emit/gate_l4_test.go new file mode 100644 index 0000000..6408f1f --- /dev/null +++ b/internal/syntactic_analysis/v2emit/gate_l4_test.go @@ -0,0 +1,123 @@ +package v2emit + +// L4 conformance harness for the v2 emitter. +// +// At this stage -a 4 reuses the -a 3 pipeline unchanged: the tree equals -a 3 +// except for max_level. buildL4 runs that pipeline with maxLevel 4 and an +// empty sdg result; later tasks fill the L4 content in, and the monotonicity +// gate below keeps holding as they do (-a 3 ⊆ -a 4). + +import ( + "path/filepath" + "runtime" + "sync" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" +) + +type l4Fixture struct { + st map[string]schema.GoFile + l3, l4 *v2.Analysis + res map[dataflow.L3Key]dataflow.Result +} + +var ( + l4Mu sync.Mutex + l4Cache = map[string]*l4Fixture{} +) + +// buildL4 runs the -a 4 pipeline over testdata/ once and emits both the +// L3 and the L4 payload from the same resolved symbol table. +func buildL4(t *testing.T, name string) *l4Fixture { + t.Helper() + l4Mu.Lock() + defer l4Mu.Unlock() + if fx, ok := l4Cache[name]; ok { + return fx + } + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata", name)) + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + res := dataflow.Run(b.Fset(), b.L3Units(st), 3, 4) + fx := &l4Fixture{ + st: st, + l3: Emit(app, name, dir, 3, "gate-test", 3, res), + l4: Emit(app, name, dir, 4, "gate-test", 3, res), + res: res, + } + l4Cache[name] = fx + return fx +} + +// supersetL4 records where a (-a 3) is not contained in b (-a 4): every object +// key and value, and every list entry. Nothing may be added yet either, so the +// check is exact; later tasks widen it with the sanctioned L4 additions. +func supersetL4(path string, a, b any, errs *[]string) { + switch av := a.(type) { + case map[string]any: + bv, ok := b.(map[string]any) + if !ok { + *errs = append(*errs, path+": object changed type") + return + } + for k, x := range av { + y, ok := bv[k] + if !ok { + *errs = append(*errs, path+"."+k+" missing at L4") + continue + } + supersetL4(path+"."+k, x, y, errs) + } + case []any: + bv, ok := b.([]any) + if !ok || len(av) > len(bv) { + *errs = append(*errs, path+": list shrank or changed type") + return + } + for i := range av { + supersetL4(path+"[]", av[i], bv[i], errs) + } + default: + if a != b { + *errs = append(*errs, path+": value changed") + } + } +} + +// TestGateL4_SupersetOfL3 is the monotonicity gate: everything in the -a 3 JSON +// is present and equal in the -a 4 JSON. Only max_level, the level marker +// itself, may differ. +func TestGateL4_SupersetOfL3(t *testing.T) { + for _, name := range []string{"dataflow", "chi", "generics"} { + fx := buildL4(t, name) + a, b := toAny(t, fx.l3), toAny(t, fx.l4) + if b["max_level"] != float64(4) { + t.Errorf("%s: max_level = %v, want 4", name, b["max_level"]) + } + a["max_level"], b["max_level"] = nil, nil + var errs []string + supersetL4("$", a, b, &errs) + for i, e := range errs { + if i == 20 { + t.Errorf("%s: … %d more", name, len(errs)-20) + break + } + t.Errorf("%s: %s", name, e) + } + } +} From 4b2d0af479bac72a00e18bb2b2d9705f82bc406e Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Sun, 25 Oct 2026 19:09:15 -0400 Subject: [PATCH 08/65] feat(l4): build go/ssa over the project load --- internal/core/analyzer.go | 71 ++++++++++-- internal/core/sdg_test.go | 75 +++++++++++++ internal/sdg/ir.go | 101 +++++++++++++++++ internal/sdg/run.go | 57 ++++++++++ internal/sdg/ssa.go | 209 +++++++++++++++++++++++++++++++++++ internal/sdg/ssa_test.go | 225 ++++++++++++++++++++++++++++++++++++++ scripts/eval_l3.sh | 44 ++++++-- 7 files changed, 767 insertions(+), 15 deletions(-) create mode 100644 internal/core/sdg_test.go create mode 100644 internal/sdg/ir.go create mode 100644 internal/sdg/run.go create mode 100644 internal/sdg/ssa.go create mode 100644 internal/sdg/ssa_test.go diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index 655de2e..8ee1a7d 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -10,6 +10,7 @@ // 3. Resolver-based call graph (semantic_analysis) — if level >= 2 // 4. Pass pipeline (analysis/registry) // 5. Intraprocedural dataflow (dataflow) — if level >= 3 +// 6. System dependence graph (sdg) — if level >= 4 package core import ( @@ -20,11 +21,13 @@ import ( "os" "os/exec" "path/filepath" + "strings" "github.com/codellm-devkit/codeanalyzer-go/internal/analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/options" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" @@ -38,6 +41,8 @@ type Analyzer struct { // schema.GoApplication: schema must not depend on dataflow, and the // cached v1 model has no L3 facts. l3 map[dataflow.L3Key]dataflow.Result + // l4 is the Phase 6 result (nil below L4). Nothing emits it yet. + l4 *sdg.Result } // L3 returns the dataflow side table of the last Analyze run: one result per @@ -45,6 +50,9 @@ type Analyzer struct { // level 3. Pass it to RenderJSON / WriteOutput / v2emit.Emit. func (a *Analyzer) L3() map[dataflow.L3Key]dataflow.Result { return a.l3 } +// L4 returns the SDG result of the last Analyze run. Nil below level 4. +func (a *Analyzer) L4() *sdg.Result { return a.l4 } + // KLimit is the access-path bound used at L3: opts.FieldDepth, or the // --graph-field-depth default of 3 when a non-CLI caller left it 0. func KLimit(opts options.AnalysisOptions) int { @@ -113,21 +121,70 @@ func (a *Analyzer) Analyze() (*schema.GoApplication, error) { } // ── Phase 5: Intraprocedural dataflow (L3) ──────────────────────────────── + var units []dataflow.Unit if a.opts.Level >= options.LevelDataflow { - a.l3 = dataflow.Run(builder.Fset(), builder.L3Units(symbolTable), KLimit(a.opts), a.opts.Jobs) + units = builder.L3Units(symbolTable) + a.l3 = dataflow.Run(builder.Fset(), units, KLimit(a.opts), a.opts.Jobs) utils.Info("dataflow: %d callables", len(a.l3)) - // Run was the last reader of Node.Stmt (the emitters read keys, spans - // and edges only), so drop the AST references now: the parsed - // packages are then unreachable once Analyze returns, before the - // cache write and the emission. (#27) - for _, r := range a.l3 { - r.Release() + } + + // ── Phase 6: System dependence graph (L4) ───────────────────────────────── + if a.opts.Level >= options.LevelSDG { + res := sdg.Run(sdg.Input{ + Fset: builder.Fset(), + Pkgs: builder.Pkgs(), + ProjectDir: a.opts.InputPath, + SymbolTable: symbolTable, + Externals: app.Externals, + Units: units, + L3: a.l3, + K: KLimit(a.opts), + Jobs: a.opts.Jobs, + }) + a.l4 = &res + if err := a.reportDegraded(res.Degraded); err != nil { + return nil, err } } + // The last reader of Node.Stmt is done (dataflow.Run, or sdg.Run at L4, + // which binds call-site actuals from the AST; the emitters read keys, + // spans and edges only), so drop the AST references now: the parsed + // packages are then unreachable once Analyze returns, before the cache + // write and the emission. (#27) + for _, r := range a.l3 { + r.Release() + } + return a.finalizeAndCache(app) } +// reportDegraded logs one warning per L4 degradation, or, under --strict, +// fails with an error naming every degraded code. +func (a *Analyzer) reportDegraded(deg []sdg.Degraded) error { + if len(deg) == 0 { + return nil + } + if a.opts.Strict { + codes := make([]string, len(deg)) + for i, d := range deg { + codes[i] = d.Code + } + return fmt.Errorf("--strict: analysis level 4 degraded: %s", strings.Join(codes, ", ")) + } + for _, d := range deg { + msg := "L4 degraded: " + d.Code + if d.Detail != "" { + msg += " (" + d.Detail + ")" + } + if len(d.Packages) > 0 { + msg += ": " + strings.Join(d.Packages, ", ") + } + utils.Warn("%s", msg) + } + return nil +} + // materialize runs `go mod download` to ensure the module graph is available // for go/packages to resolve imports and types. Idempotent and cached. func (a *Analyzer) materialize() error { diff --git a/internal/core/sdg_test.go b/internal/core/sdg_test.go new file mode 100644 index 0000000..5147d23 --- /dev/null +++ b/internal/core/sdg_test.go @@ -0,0 +1,75 @@ +package core_test + +import ( + "path/filepath" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/core" + "github.com/codellm-devkit/codeanalyzer-go/internal/options" +) + +// TestAnalyze_Level4RunsSDG: -a 4 runs the L4 pass (SSA + VTA) after L3 and +// keeps its result; a clean project degrades nothing. Below -a 4 there is no +// L4 result. +func TestAnalyze_Level4RunsSDG(t *testing.T) { + opts := options.AnalysisOptions{ + InputPath: greeterDir(t), Level: options.LevelSDG, SkipTests: true, CacheDir: t.TempDir(), Strict: true, + } + a := core.New(opts) + if _, err := a.Analyze(); err != nil { + t.Fatal(err) + } + if a.L4() == nil { + t.Fatal("-a 4 produced no L4 result") + } + if len(a.L4().Degraded) != 0 { + t.Errorf("degraded = %+v, want none", a.L4().Degraded) + } + if len(a.L3()) == 0 { + t.Fatal("-a 4 produced no L3 results") + } + for k, r := range a.L3() { + for _, n := range r.Nodes { + if n.Stmt != nil { + t.Fatalf("%v: node %s keeps its Stmt after -a 4", k, n.Key) + } + } + } + + opts.Level = options.LevelDataflow + a3 := core.New(opts) + if _, err := a3.Analyze(); err != nil { + t.Fatal(err) + } + if a3.L4() != nil { + t.Error("-a 3 ran the L4 pass") + } +} + +// TestAnalyze_Level4StrictFailsOnDegraded: an ill-typed package gets no SSA. +// By default -a 4 degrades and still succeeds; with --strict Analyze fails +// and the error names the degraded codes. +func TestAnalyze_Level4StrictFailsOnDegraded(t *testing.T) { + opts := options.AnalysisOptions{ + InputPath: filepath.Join(testdataDir(), "typeerr"), Level: options.LevelSDG, SkipTests: true, CacheDir: t.TempDir(), + } + a := core.New(opts) + if _, err := a.Analyze(); err != nil { + t.Fatalf("non-strict -a 4 failed: %v", err) + } + if a.L4() == nil || len(a.L4().Degraded) == 0 { + t.Fatalf("L4 = %+v, want a degraded result", a.L4()) + } + + opts.Strict = true + _, err := core.New(opts).Analyze() + if err == nil { + t.Fatal("--strict -a 4 on a degraded project succeeded") + } + for _, code := range []string{"partial_ssa", "no_ssa"} { + if !strings.Contains(err.Error(), code) { + t.Errorf("error %q does not name %s", err, code) + } + } +} diff --git a/internal/sdg/ir.go b/internal/sdg/ir.go new file mode 100644 index 0000000..a4d1843 --- /dev/null +++ b/internal/sdg/ir.go @@ -0,0 +1,101 @@ +// Package sdg builds the L4 overlay (-a 4): the interprocedural system +// dependence graph over the L3 results. It builds go/ssa over the analyzer's +// go/packages load (bodies for project packages only), resolves dynamic +// calls with VTA, and derives formal/actual vertices, param edges and +// summaries. +// +// The IR here holds no can:// ids: vertices are keyed by (L3Key, local key) +// and cross-callable edges by L3Key pairs and external signatures. The v2 +// emitter mints the global ids. This package must not import v2emit or core. +package sdg + +import ( + "go/token" + + "golang.org/x/tools/go/packages" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" +) + +// Target is a call target: an in-project callable or a library stand-in. +type Target struct { + Callable dataflow.L3Key // in-project callable; interface methods use the interface's declaring file + External string // full callee signature (key of app.Externals) for a stand-in; else "" +} + +// VertexKind is an L4 port vertex kind. +type VertexKind string // "formal_in" | "formal_out" | "actual_in" | "actual_out" + +// Vertex is one L4 port vertex of a callable, keyed by its local key. +type Vertex struct { + Key, Of, Parent string + Kind VertexKind +} + +// Ref names a vertex of a target by its local key. +type Ref struct { + Owner Target + Key string +} + +// ParamEdge is a param_in / param_out edge between two targets' vertices. +type ParamEdge struct { + Src, Dst Ref + Var string +} + +// PortEdge is a callable-local ddg edge added at L4. +type PortEdge struct{ Src, Dst, Var, Prov string } // callable-local ddg addition; Prov "reaching-defs" | "points-to" + +// SummaryEdge is a call-site summary edge, actual_in to actual_out (local keys). +type SummaryEdge struct{ Src, Dst string } + +// CallableSDG is the L4 overlay of one callable. +type CallableSDG struct { + Vertices []Vertex + Ddg []PortEdge + Summary []SummaryEdge +} + +// Degraded is one degradation marker entry (spec §4.15). +type Degraded struct { + Code, Detail string + Packages []string + Callables []dataflow.L3Key +} + +// Result is the L4 overlay of a run. +type Result struct { + Callables map[dataflow.L3Key]CallableSDG + Externals map[string][]Vertex // stand-in formal vertices by full callee sig + NewExternals map[string]schema.GoExternalCallee // externals only VTA reached + ParamIn, ParamOut []ParamEdge + CallEdges []schema.GoCallEdge // prov ["vta"] + Degraded []Degraded +} + +// Input is what Run needs from the earlier phases. +type Input struct { + Fset *token.FileSet + Pkgs map[string]*packages.Package + ProjectDir string + SymbolTable map[string]schema.GoFile + Externals map[string]schema.GoExternalCallee + Units []dataflow.Unit + L3 map[dataflow.L3Key]dataflow.Result + K, Jobs int + Models Models // Task 15; zero value = defaults +} + +// Models overrides library stand-in summaries, keyed by the external id's +// / tail; each pair is a (src, dst) port edge. Loading and +// applying them comes later (ruling R-P4); the zero value means defaults. +type Models map[string][][2]string + +// Degraded codes (spec §4.15; an open vocabulary of plain strings). +const ( + CodeNoSSA = "no_ssa" + CodePartialSSA = "partial_ssa" + CodeNoVTA = "no_vta" +) diff --git a/internal/sdg/run.go b/internal/sdg/run.go new file mode 100644 index 0000000..6bfffa2 --- /dev/null +++ b/internal/sdg/run.go @@ -0,0 +1,57 @@ +package sdg + +import ( + "fmt" + "time" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/callgraph/vta" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// vtaCallGraph is vta.CallGraph (a test seam for the recovered-panic path). +var vtaCallGraph = vta.CallGraph + +// Run builds the L4 overlay of in. For now it builds SSA over the project +// packages and runs VTA once over the project function set, and returns only +// the degradation: no vertices or edges yet. +func Run(in Input) Result { + start := time.Now() + prog, pkgs, deg := buildSSA(in) + utils.Info("L4: SSA built %d project packages in %s", len(pkgs), time.Since(start).Round(time.Millisecond)) + if len(pkgs) == 0 { + return Result{Degraded: deg} + } + + start = time.Now() + funcs := projectFuncs(prog, pkgs) + cg, err := runVTA(funcs) + if err != nil { + utils.Warn("L4: VTA failed: %v", err) + deg = append(deg, Degraded{Code: CodeNoVTA, Detail: err.Error()}) + } else { + utils.Info("L4: VTA over %d functions: %d edges in %s", len(funcs), edgeCount(cg), time.Since(start).Round(time.Millisecond)) + } + return Result{Degraded: deg} +} + +// runVTA runs VTA with no initial graph (VTA then uses its internal lazy CHA +// over funcs), turning a panic into an error. +func runVTA(funcs map[*ssa.Function]bool) (cg *callgraph.Graph, err error) { + defer func() { + if r := recover(); r != nil { + cg, err = nil, fmt.Errorf("panic: %v", r) + } + }() + return vtaCallGraph(funcs, nil), nil +} + +func edgeCount(cg *callgraph.Graph) int { + n := 0 + for _, node := range cg.Nodes { + n += len(node.Out) + } + return n +} diff --git a/internal/sdg/ssa.go b/internal/sdg/ssa.go new file mode 100644 index 0000000..c8f2b9f --- /dev/null +++ b/internal/sdg/ssa.go @@ -0,0 +1,209 @@ +package sdg + +import ( + "fmt" + "go/types" + "sort" + "strings" + + "golang.org/x/tools/go/packages" + "golang.org/x/tools/go/ssa" + "golang.org/x/tools/go/ssa/ssautil" + + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// buildHook, when set, runs just before each package's Build (a test seam +// for the recovered-panic path). +var buildHook func(*ssa.Package) + +// buildSSA builds go/ssa over the project packages of in.Pkgs (spec §4.8, D2). +// +// ssautil.Packages gets only the project packages, so SSA bodies exist for +// them alone; every dependency (stdlib, modules, vendored code) is created +// from types, without bodies. The mode is InstantiateGenerics (VTA needs it) +// | BuildSerially. Each package is then built on its own, sorted by path, +// under recover: +// +// - A package that is ill-typed (pkg.IllTyped, which also covers a broken +// dependency) or that ssautil left nil, or whose Build panicked, gets no +// SSA: it is listed under one partial_ssa entry. +// - When no package built, no_ssa follows. +// +// It returns the built packages in path order. +func buildSSA(in Input) (*ssa.Program, []*ssa.Package, []Degraded) { + roots := projectPackages(in) + var failed []string + var ok []*packages.Package + for _, p := range roots { + if p.IllTyped { + failed = append(failed, p.PkgPath) + continue + } + ok = append(ok, p) + } + + var prog *ssa.Program + var built []*ssa.Package + if len(ok) > 0 { + var spkgs []*ssa.Package + prog, spkgs = ssautil.Packages(ok, ssa.InstantiateGenerics|ssa.BuildSerially) + for i, sp := range spkgs { + if sp == nil { + failed = append(failed, ok[i].PkgPath) + continue + } + if err := buildPackage(sp); err != nil { + utils.Warn("L4: SSA build of %s failed: %v", ok[i].PkgPath, err) + failed = append(failed, ok[i].PkgPath) + continue + } + built = append(built, sp) + } + } + + var deg []Degraded + if len(failed) > 0 { + sort.Strings(failed) + deg = append(deg, Degraded{ + Code: CodePartialSSA, + Detail: fmt.Sprintf("%d package(s) have no SSA (type errors or a recovered build panic)", len(failed)), + Packages: failed, + }) + } + if len(built) == 0 && len(roots) > 0 { + deg = append(deg, Degraded{Code: CodeNoSSA, Detail: "no project package could be built in SSA form"}) + } + return prog, built, deg +} + +// buildPackage builds one SSA package, turning a panic into an error. +func buildPackage(p *ssa.Package) (err error) { + defer func() { + if r := recover(); r != nil { + err = fmt.Errorf("panic: %v", r) + } + }() + if buildHook != nil { + buildHook(p) + } + p.Build() + return nil +} + +// projectPackages returns the packages of in.Pkgs that hold project code (a +// Go file under ProjectDir, not vendored), sorted by path. Generated test +// mains live in the build cache and so are never project packages. +func projectPackages(in Input) []*packages.Package { + var out []*packages.Package + for _, p := range in.Pkgs { + if isProject(in.ProjectDir, p) { + out = append(out, p) + } + } + sort.Slice(out, func(i, j int) bool { return out[i].PkgPath < out[j].PkgPath }) + return out +} + +func isProject(dir string, p *packages.Package) bool { + for _, f := range p.GoFiles { + if utils.IsWithin(dir, f) && !utils.IsVendored(utils.RelativePath(dir, f)) { + return true + } + } + return false +} + +// projectFuncs is the VTA function set (spec §4.8, VTA scope): the SSA +// functions of the built project packages — package-level functions and +// inits, every declared method (generic origins included), the method-set +// wrappers of their non-generic named types — and, transitively, their +// anonymous functions and every function their code refers to that is +// project code (generic instances of project functions, $bound / $thunk +// wrappers). Functions declared outside the project are left out, so VTA's +// cost stays bounded by the project's size. +func projectFuncs(prog *ssa.Program, pkgs []*ssa.Package) map[*ssa.Function]bool { + project := make(map[*types.Package]bool, len(pkgs)) + for _, p := range pkgs { + project[p.Pkg] = true + } + funcs := map[*ssa.Function]bool{} + var work []*ssa.Function + add := func(fn *ssa.Function) { + if fn != nil && !funcs[fn] && isProjectFunc(fn, project) { + funcs[fn] = true + work = append(work, fn) + } + } + for _, p := range pkgs { + names := make([]string, 0, len(p.Members)) + for name := range p.Members { + names = append(names, name) + } + sort.Strings(names) + for _, name := range names { + switch m := p.Members[name].(type) { + case *ssa.Function: + add(m) + case *ssa.Type: + named, ok := types.Unalias(m.Type()).(*types.Named) + if !ok { + continue + } + for i := 0; i < named.NumMethods(); i++ { + add(prog.FuncValue(named.Method(i))) + } + if named.TypeParams() == nil && !types.IsInterface(named) { + for _, t := range []types.Type{named, types.NewPointer(named)} { + mset := prog.MethodSets.MethodSet(t) + for i := 0; i < mset.Len(); i++ { + add(prog.MethodValue(mset.At(i))) + } + } + } + } + } + } + var buf [10]*ssa.Value + for len(work) > 0 { + fn := work[len(work)-1] + work = work[:len(work)-1] + for _, anon := range fn.AnonFuncs { + add(anon) + } + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + for _, op := range instr.Operands(buf[:0]) { + if g, ok := (*op).(*ssa.Function); ok { + add(g) + } + } + } + } + } + return funcs +} + +// isProjectFunc reports whether fn's code is project code: it belongs to a +// built project package, is an instance (or a closure of one) of a project +// generic, or is a synthetic wrapper ($bound, $thunk, promotion, pointer +// indirection) of a project method. +func isProjectFunc(fn *ssa.Function, project map[*types.Package]bool) bool { + for f := fn; f != nil; f = f.Parent() { + if f.Pkg != nil { + return project[f.Pkg.Pkg] + } + if o := f.Origin(); o != nil && o.Pkg != nil { + return project[o.Pkg.Pkg] + } + if f.Parent() == nil { + if strings.HasPrefix(f.Synthetic, "from type information") { + return false + } + if obj := f.Object(); obj != nil { + return project[obj.Pkg()] + } + } + } + return false +} diff --git a/internal/sdg/ssa_test.go b/internal/sdg/ssa_test.go new file mode 100644 index 0000000..664ee0b --- /dev/null +++ b/internal/sdg/ssa_test.go @@ -0,0 +1,225 @@ +package sdg + +import ( + "path/filepath" + "runtime" + "slices" + "strings" + "testing" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + "golang.org/x/tools/go/ssa/ssautil" + + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// loadInput loads testdata/ the way the analyzer does (symbol-table +// builder, tests skipped) and returns the sdg input over it. +func loadInput(t *testing.T, name string) Input { + t.Helper() + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", name)) + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + return Input{Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, K: 3, Jobs: 1} +} + +// homePath is the import path of the package fn's code comes from: its own +// package, its generic origin's, or (for a wrapper) the wrapped object's. +func homePath(fn *ssa.Function) string { + if fn.Pkg != nil { + return fn.Pkg.Pkg.Path() + } + if o := fn.Origin(); o != nil && o.Pkg != nil { + return o.Pkg.Pkg.Path() + } + if obj := fn.Object(); obj != nil && obj.Pkg() != nil { + return obj.Pkg().Path() + } + return "" +} + +// TestSSA_BuildsProjectPackagesOnly: bodies are built for the project's +// packages only. Every function with syntax lies in a project file, and no +// function declared outside the project (stdlib, the vendored vlib) has blocks. +func TestSSA_BuildsProjectPackagesOnly(t *testing.T) { + in := loadInput(t, "dataflow") + prog, pkgs, deg := buildSSA(in) + if len(deg) != 0 { + t.Fatalf("degraded = %+v, want none", deg) + } + var built []string + for _, p := range pkgs { + built = append(built, p.Pkg.Path()) + } + want := []string{ + "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/inits", + "example.com/dataflow/l1", "example.com/dataflow/linedir", "example.com/dataflow/vend", + "example.com/dataflow/xfile", + } + if !slices.Equal(built, want) { + t.Fatalf("built packages = %v, want %v", built, want) + } + + project := map[string]bool{} + for _, p := range want { + project[p] = true + } + withBody := 0 + for fn := range ssautil.AllFunctions(prog) { + if syn := fn.Syntax(); syn != nil { + file := in.Fset.File(syn.Pos()).Name() + if !utils.IsWithin(in.ProjectDir, file) || utils.IsVendored(utils.RelativePath(in.ProjectDir, file)) { + t.Errorf("%s has syntax outside the project: %s", fn, file) + } + } + // Wrappers ($bound, promotion, thunks) are synthesized with blocks + // for whatever they wrap; only declared code and instances count. + declared := fn.Synthetic == "" || strings.HasPrefix(fn.Synthetic, "instance of") + if declared && len(fn.Blocks) > 0 { + if !project[homePath(fn)] { + t.Errorf("%s (package %q) has blocks; only project bodies are built", fn, homePath(fn)) + } + withBody++ + } + } + if withBody == 0 { + t.Fatal("no project function was built") + } + if fn := prog.ImportedPackage("example.com/dataflow/flow").Func("IfElse"); fn == nil || len(fn.Blocks) == 0 { + t.Error("flow.IfElse has no SSA body") + } + osPkg := prog.ImportedPackage("os") + if osPkg == nil { + t.Fatal("dependency os has no SSA package") + } + if fn := osPkg.Func("Getenv"); fn == nil || len(fn.Blocks) != 0 { + t.Errorf("os.Getenv = %v, want a bodiless function from types", fn) + } + vlib := prog.ImportedPackage("example.com/vlib") + if vlib == nil { + t.Fatal("vendored dependency vlib has no SSA package") + } + if fn := vlib.Func("F"); fn == nil || len(fn.Blocks) != 0 { + t.Errorf("vlib.F = %v, want a bodiless function from types (vendored code is not the project)", fn) + } +} + +// TestSSA_PanicDegradesPackage: a panic while one package builds is +// recovered; that package is listed under partial_ssa and the rest still +// build. When every package panics, no_ssa is added. +func TestSSA_PanicDegradesPackage(t *testing.T) { + in := loadInput(t, "dataflow") + defer func() { buildHook = nil }() + + buildHook = func(p *ssa.Package) { + if p.Pkg.Path() == "example.com/dataflow/flow" { + panic("boom") + } + } + prog, pkgs, deg := buildSSA(in) + if len(deg) != 1 || deg[0].Code != "partial_ssa" || !slices.Equal(deg[0].Packages, []string{"example.com/dataflow/flow"}) { + t.Fatalf("degraded = %+v, want one partial_ssa naming example.com/dataflow/flow", deg) + } + for _, p := range pkgs { + if p.Pkg.Path() == "example.com/dataflow/flow" { + t.Error("the panicked package is still returned as built") + } + } + if len(pkgs) != 6 { + t.Errorf("built %d packages, want the other 6", len(pkgs)) + } + if fn := prog.ImportedPackage("example.com/dataflow/l1"); fn == nil || len(fn.Members) == 0 { + t.Error("l1 was not built after flow panicked") + } + + buildHook = func(*ssa.Package) { panic("boom") } + _, pkgs, deg = buildSSA(in) + if len(pkgs) != 0 { + t.Errorf("built %d packages, want 0", len(pkgs)) + } + var codes []string + for _, d := range deg { + codes = append(codes, d.Code) + } + if !slices.Equal(codes, []string{"partial_ssa", "no_ssa"}) { + t.Fatalf("codes = %v, want [partial_ssa no_ssa]", codes) + } + if len(deg[0].Packages) != 7 { + t.Errorf("partial_ssa lists %v, want all 7 packages", deg[0].Packages) + } + if res := Run(in); len(res.Degraded) != 2 || res.Degraded[1].Code != "no_ssa" { + t.Errorf("Run degraded = %+v, want partial_ssa + no_ssa and no VTA", res.Degraded) + } +} + +// TestSSA_IllTypedPackageDegrades: a package with type errors gets no SSA and +// is listed under partial_ssa; it is the only package, so no_ssa follows. +func TestSSA_IllTypedPackageDegrades(t *testing.T) { + in := loadInput(t, "typeerr") + _, pkgs, deg := buildSSA(in) + if len(pkgs) != 0 { + t.Errorf("built %d packages from an ill-typed project", len(pkgs)) + } + if len(deg) != 2 || deg[0].Code != "partial_ssa" || !slices.Equal(deg[0].Packages, []string{"example.com/typeerr"}) || deg[1].Code != "no_ssa" { + t.Fatalf("degraded = %+v, want partial_ssa [example.com/typeerr] then no_ssa", deg) + } +} + +// TestRun_CleanFixtureNotDegraded: SSA and VTA over the fixture degrade +// nothing. +func TestRun_CleanFixtureNotDegraded(t *testing.T) { + if res := Run(loadInput(t, "dataflow")); len(res.Degraded) != 0 { + t.Fatalf("degraded = %+v, want none", res.Degraded) + } +} + +// TestRun_VTAPanicDegrades: a VTA panic is recovered as no_vta. +func TestRun_VTAPanicDegrades(t *testing.T) { + in := loadInput(t, "dataflow") + defer func(f func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph) { vtaCallGraph = f }(vtaCallGraph) + vtaCallGraph = func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph { panic("vta boom") } + res := Run(in) + if len(res.Degraded) != 1 || res.Degraded[0].Code != "no_vta" { + t.Fatalf("degraded = %+v, want one no_vta", res.Degraded) + } +} + +// TestProjectFuncs_CoversClosuresAndMethods: the VTA function set holds the +// project's functions, their closures and methods, and nothing declared +// outside the project. +func TestProjectFuncs_CoversClosuresAndMethods(t *testing.T) { + in := loadInput(t, "dataflow") + prog, pkgs, _ := buildSSA(in) + funcs := projectFuncs(prog, pkgs) + project := map[string]bool{} + for _, p := range pkgs { + project[p.Pkg.Path()] = true + } + anon, methods := 0, 0 + for fn := range funcs { + if !project[homePath(fn)] { + t.Errorf("%s (package %q) is outside the project", fn, homePath(fn)) + } + if fn.Parent() != nil { + anon++ + } + if fn.Signature.Recv() != nil { + methods++ + } + } + if anon == 0 || methods == 0 { + t.Errorf("funcs has %d closures and %d methods; want both > 0", anon, methods) + } + if !funcs[prog.ImportedPackage("example.com/dataflow/flow").Func("IfElse")] { + t.Error("flow.IfElse is not in the VTA function set") + } +} diff --git a/scripts/eval_l3.sh b/scripts/eval_l3.sh index 0cb8fb7..cad7dd1 100755 --- a/scripts/eval_l3.sh +++ b/scripts/eval_l3.sh @@ -39,6 +39,7 @@ # 2. cango -a 3 -j 1 # 3. cango -a 3 -j N (N = CPU count) # 4. cango -a 3 -j N (second run) +# 5. cango -a 4 -j 1 -v (only when CANGO_LEVELS includes 4) # and compares the three -a 3 analysis.json files byte for byte. It records wall time, peak # RSS (macOS /usr/bin/time -l), module/callable/body-node/edge counts, external_symbols, the # call_graph split, the prepare-time type-check status, the number of type-error warnings @@ -59,6 +60,9 @@ # EVAL_JOBS N for the -j N runs (default: CPU count) # RUN_TIMEOUT per-run limit, seconds (default 3600); on expiry the run is killed, the app # is marked TIMEOUT and its remaining runs are skipped +# CANGO_LEVELS levels to run (default "2 3"). Levels 2 and 3 always run (the L3 baseline); +# adding 4 (e.g. "2 3 4") also runs step 5 and records its wall time, peak RSS, and the +# SSA build time, VTA time and VTA edge count from its -v log. # EVAL_PRUNE 1 = after the comparison, delete the warm-up and both -j N analysis.json files # (keeps a2j1 and a3j1, and every .err/.time). For very large apps. # Any other environment (GOOS, GOFLAGS, ...) is inherited by the analyzer unchanged. @@ -73,6 +77,9 @@ EVAL_GO=${EVAL_GO:-go1.27.3} DATE=$(date +%Y-%m-%d) EVAL_NOTE=${EVAL_NOTE:-$EVAL_OUT/l3-$DATE.md} RUN_TIMEOUT=${RUN_TIMEOUT:-3600} +CANGO_LEVELS=${CANGO_LEVELS:-"2 3"} +RUN_L4=0 +case " $CANGO_LEVELS " in *" 4 "*) RUN_L4=1 ;; esac EVAL_PRUNE=${EVAL_PRUNE:-0} JOBS=${EVAL_JOBS:-$( (sysctl -n hw.ncpu 2>/dev/null || nproc 2>/dev/null || echo 4) | head -1)} REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd) @@ -142,7 +149,7 @@ if [ "${1:-}" = "--clean" ]; then [ -d "$EVAL_OUT" ] || exit 0 case $EVAL_OUT in /|"$HOME"|"$EVAL_SRC") echo "refusing to clean $EVAL_OUT" >&2; exit 2 ;; esac for n in $(manifest_names) $(ls "$EVAL_OUT" 2>/dev/null | sed -n 's/-a3j1$//p'); do - for t in warm a2j1 a3j1 a3jN a3jN2; do + for t in warm a2j1 a3j1 a3jN a3jN2 a4j1; do rm -rf "$EVAL_OUT/$n-$t" "$EVAL_OUT/$n-$t.err" "$EVAL_OUT/$n-$t.time" "$EVAL_OUT/$n-$t.timeout" done rm -rf "$EVAL_OUT/cache-$n" @@ -321,15 +328,16 @@ JQ_COUNTS=' ([$a.call_graph[]|select(.dst|contains("/@external/")|not)]|length), ([$a.call_graph[]|select(.dst|contains("/@external/"))]|length) ] | @tsv' -# run_one NAME TAG LEVEL JOBS -> sets RC WALL RSS; output in $EVAL_OUT/NAME-TAG/analysis.json +# run_one NAME TAG LEVEL JOBS [FLAG...] -> sets RC WALL RSS; output in $EVAL_OUT/NAME-TAG/analysis.json run_one() { local name=$1 tag=$2 level=$3 jobs=$4 + shift 4 local dir=$EVAL_OUT/$name-$tag rm -rf "$dir"; mkdir -p "$dir" # shellcheck disable=SC2086 env $APP_ENV GOTOOLCHAIN="$RUN_GO" /usr/bin/time -l -o "$dir.time" \ "$CANGO" -i "$APP" --app-name "$name" -a "$level" -j "$jobs" \ - --eager -c "$EVAL_OUT/cache-$name" -o "$dir" >/dev/null 2>"$dir.err" & + --eager -c "$EVAL_OUT/cache-$name" -o "$dir" "$@" >/dev/null 2>"$dir.err" & local pid=$! waited=0 while kill -0 "$pid" 2>/dev/null; do if [ "$waited" -ge "$RUN_TIMEOUT" ]; then @@ -358,8 +366,13 @@ CANGO_DIRTY=$(printf '%s\n' "$BUILD_INFO" | awk -F'=' '/vcs.modified=/{print $2} echo echo "Analyzer: $("$CANGO" --version 2>&1 | head -1), built with ${CANGO_GO:-unknown} (runtime.Version), rev ${CANGO_REV:-unknown}; runs use GOTOOLCHAIN=$RUN_GO; host: $(uname -sm); j=$JOBS; all runs --eager." echo - echo "| app | status | type-check | wall a2 j1 (s) | wall a3 j1 (s) | wall a3 jN (s) | RSS a3 j1 (MB) | modules | callables | body nodes | cfg | cdg | ddg | ext symbols | cg in-project | cg external | -j identical | L3 skips | type errors | toolchain warn |" - echo "|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|" + if [ "$RUN_L4" -eq 1 ]; then + L4H=" wall a4 j1 (s) | RSS a4 j1 (MB) | RSS a4/a3 | SSA a4 | VTA a4 | VTA edges |"; L4S="---|---|---|---|---|---|" + else + L4H=""; L4S="" + fi + echo "| app | status | type-check | wall a2 j1 (s) | wall a3 j1 (s) | wall a3 jN (s) | RSS a3 j1 (MB) | modules | callables | body nodes | cfg | cdg | ddg | ext symbols | cg in-project | cg external | -j identical | L3 skips | type errors | toolchain warn |$L4H" + echo "|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|$L4S" } >"$EVAL_NOTE" FAILS="" @@ -394,6 +407,21 @@ for T in $TARGETS; do if [ "$RCN" -ne 124 ]; then run_one "$name" a3jN2 3 "$JOBS"; RCN2=$RC; fi fi + L4COLS="" + if [ "$RUN_L4" -eq 1 ]; then + W4=n/a; R4=n/a; RATIO=n/a; RC4=0 + if [ "$RC3" -ne 124 ] && [ "$RC2" -ne 124 ]; then + run_one "$name" a4j1 4 1 -v; W4=$WALL; R4=$RSS; RC4=$RC + if [ "$RC4" -ne 0 ]; then W4="$W4 (rc $RC4)"; fi + fi + case $R3$R4 in *[!0-9]*|"") ;; *) RATIO=$(awk -v a="$R4" -v b="$R3" 'BEGIN{printf "%.2f", a/b}') ;; esac + e4=$EVAL_OUT/$name-a4j1.err + SSA4=$(sed -n 's/.*L4: SSA built .* in \(.*\)$/\1/p' "$e4" 2>/dev/null | head -1) + VTA4=$(sed -n 's/.*L4: VTA over .* edges in \(.*\)$/\1/p' "$e4" 2>/dev/null | head -1) + VE4=$(sed -n 's/.*L4: VTA over .*: \([0-9]*\) edges in .*/\1/p' "$e4" 2>/dev/null | head -1) + L4COLS=" $W4 | $R4 | $RATIO | ${SSA4:-n/a} | ${VTA4:-n/a} | ${VE4:-n/a} |" + fi + ident=n/a if [ "$RC2" -eq 124 ] || [ "$RC3" -eq 124 ] || [ "$RCN" -eq 124 ] || [ "$RCN2" -eq 124 ]; then status="TIMEOUT(>${RUN_TIMEOUT}s; remaining runs skipped)" @@ -420,13 +448,13 @@ for T in $TARGETS; do IFS=$'\t' read -r c1 c2 c3 c4 c5 c6 c7 c8 c9 <>"$EVAL_NOTE" + "$ident" "$skips" "$terrs" "$twarn" "$L4COLS" >>"$EVAL_NOTE" if [ "$EVAL_PRUNE" = 1 ] && [ "$ident" = yes ]; then rm -f "$EVAL_OUT/$name-warm/analysis.json" "$EVAL_OUT/$name-a3jN/analysis.json" \ - "$EVAL_OUT/$name-a3jN2/analysis.json" + "$EVAL_OUT/$name-a3jN2/analysis.json" "$EVAL_OUT/$name-a4j1/analysis.json" fi done From 69aa99a01a0aac09aeab4df5af9da1e5e877e17e Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Sun, 25 Oct 2026 19:29:36 -0400 Subject: [PATCH 09/65] fix(l4): rebuild SSA without a package whose build panicked; recover VTA set construction --- internal/sdg/run.go | 19 +++++----- internal/sdg/ssa.go | 46 ++++++++++++++++++++---- internal/sdg/ssa_test.go | 77 ++++++++++++++++++++++++++++++++++++++++ testdata/ssapanic/a/a.go | 13 +++++++ testdata/ssapanic/b/b.go | 6 ++++ testdata/ssapanic/g/g.go | 5 +++ testdata/ssapanic/go.mod | 3 ++ testdata/ssapanic/w/w.go | 7 ++++ 8 files changed, 162 insertions(+), 14 deletions(-) create mode 100644 testdata/ssapanic/a/a.go create mode 100644 testdata/ssapanic/b/b.go create mode 100644 testdata/ssapanic/g/g.go create mode 100644 testdata/ssapanic/go.mod create mode 100644 testdata/ssapanic/w/w.go diff --git a/internal/sdg/run.go b/internal/sdg/run.go index 6bfffa2..8501141 100644 --- a/internal/sdg/run.go +++ b/internal/sdg/run.go @@ -26,26 +26,29 @@ func Run(in Input) Result { } start = time.Now() - funcs := projectFuncs(prog, pkgs) - cg, err := runVTA(funcs) + cg, nfuncs, err := runVTA(prog, pkgs) if err != nil { - utils.Warn("L4: VTA failed: %v", err) + utils.Debug("L4: VTA failed: %v", err) deg = append(deg, Degraded{Code: CodeNoVTA, Detail: err.Error()}) } else { - utils.Info("L4: VTA over %d functions: %d edges in %s", len(funcs), edgeCount(cg), time.Since(start).Round(time.Millisecond)) + utils.Info("L4: VTA over %d functions: %d edges in %s", nfuncs, edgeCount(cg), time.Since(start).Round(time.Millisecond)) } return Result{Degraded: deg} } -// runVTA runs VTA with no initial graph (VTA then uses its internal lazy CHA -// over funcs), turning a panic into an error. -func runVTA(funcs map[*ssa.Function]bool) (cg *callgraph.Graph, err error) { +// runVTA builds the project function set and runs VTA over it with no +// initial graph (VTA then uses its internal lazy CHA over that set), turning +// a panic in either into an error. Both stages build functions on demand +// (projectFuncs' MethodValue wrappers, VTA's method lookups), so both sit +// under the recover. +func runVTA(prog *ssa.Program, pkgs []*ssa.Package) (cg *callgraph.Graph, nfuncs int, err error) { defer func() { if r := recover(); r != nil { cg, err = nil, fmt.Errorf("panic: %v", r) } }() - return vtaCallGraph(funcs, nil), nil + funcs := projectFuncs(prog, pkgs) + return vtaCallGraph(funcs, nil), len(funcs), nil } func edgeCount(cg *callgraph.Graph) int { diff --git a/internal/sdg/ssa.go b/internal/sdg/ssa.go index c8f2b9f..adc261c 100644 --- a/internal/sdg/ssa.go +++ b/internal/sdg/ssa.go @@ -17,6 +17,20 @@ import ( // for the recovered-panic path). var buildHook func(*ssa.Package) +// callHook, when set, runs on the static callee of every call instruction SSA +// emits, inside the build (a test seam for panics mid-build: after a builder +// has created shared generic instances, or while a wrapper is built on demand). +var callHook func(*types.Func) + +// installCallHook wires callHook into prog through the builder's no-return +// predicate, which SSA consults on every static call it emits. Production +// runs never set callHook, so prog keeps its nil predicate. +func installCallHook(prog *ssa.Program) { + if hook := callHook; hook != nil { + prog.SetNoReturn(func(f *types.Func) bool { hook(f); return false }) + } +} + // buildSSA builds go/ssa over the project packages of in.Pkgs (spec §4.8, D2). // // ssautil.Packages gets only the project packages, so SSA bodies exist for @@ -26,9 +40,16 @@ var buildHook func(*ssa.Package) // under recover: // // - A package that is ill-typed (pkg.IllTyped, which also covers a broken -// dependency) or that ssautil left nil, or whose Build panicked, gets no -// SSA: it is listed under one partial_ssa entry. -// - When no package built, no_ssa follows. +// dependency) or that ssautil left nil gets no SSA. +// - A package whose Build panics gets no SSA either, and the whole program +// is discarded and rebuilt without it (it then enters from types, as a +// dependency). A builder that panics never marks the generic instances +// and wrappers it created as done, so any later build or method lookup +// sharing one would block forever — not a panic, and not recoverable. +// The rebuild repeats until a pass builds every remaining package, at +// most once per package. +// - Every package that got no SSA is listed under one partial_ssa entry; +// when no package built, no_ssa follows. // // It returns the built packages in path order. func buildSSA(in Input) (*ssa.Program, []*ssa.Package, []Degraded) { @@ -45,21 +66,34 @@ func buildSSA(in Input) (*ssa.Program, []*ssa.Package, []Degraded) { var prog *ssa.Program var built []*ssa.Package - if len(ok) > 0 { + for len(ok) > 0 { var spkgs []*ssa.Package prog, spkgs = ssautil.Packages(ok, ssa.InstantiateGenerics|ssa.BuildSerially) + installCallHook(prog) + built = built[:0] + var good []*packages.Package + panicked := -1 for i, sp := range spkgs { if sp == nil { failed = append(failed, ok[i].PkgPath) continue } if err := buildPackage(sp); err != nil { - utils.Warn("L4: SSA build of %s failed: %v", ok[i].PkgPath, err) + utils.Debug("L4: SSA build of %s failed: %v; rebuilding without it", ok[i].PkgPath, err) failed = append(failed, ok[i].PkgPath) - continue + panicked = i + break } + good = append(good, ok[i]) built = append(built, sp) } + if panicked < 0 { + break + } + // Drop this (possibly wedged) program; rebuild from the packages not + // yet known bad: the ones that built, and the ones not reached. + ok = append(good, ok[panicked+1:]...) + prog, built = nil, nil } var deg []Degraded diff --git a/internal/sdg/ssa_test.go b/internal/sdg/ssa_test.go index 664ee0b..4dfb34e 100644 --- a/internal/sdg/ssa_test.go +++ b/internal/sdg/ssa_test.go @@ -1,11 +1,13 @@ package sdg import ( + "go/types" "path/filepath" "runtime" "slices" "strings" "testing" + "time" "golang.org/x/tools/go/callgraph" "golang.org/x/tools/go/ssa" @@ -223,3 +225,78 @@ func TestProjectFuncs_CoversClosuresAndMethods(t *testing.T) { t.Error("flow.IfElse is not in the VTA function set") } } + +// panicOn returns a callHook that panics on a call to the function named name. +func panicOn(name string) func(*types.Func) { + return func(f *types.Func) { + if f.Name() == name { + panic("boom in " + name) + } + } +} + +// withTimeout runs f, failing the test if it does not return within d (a +// deadlocked SSA program blocks forever; recover cannot catch that). +func withTimeout(t *testing.T, d time.Duration, f func()) { + t.Helper() + done := make(chan struct{}) + go func() { defer close(done); f() }() + select { + case <-done: + case <-time.After(d): + t.Fatalf("did not finish within %s: the SSA program deadlocked", d) + } +} + +// TestSSA_MidBuildPanicDoesNotDeadlock: testdata/ssapanic packages a and b +// share the instance g.Map[int]. Building a creates it under a's builder, +// then panics on the call to Boom, so a's builder never marks that instance +// done. Package b, built next in the same program, would wait on it forever. +// buildSSA must instead drop the program and rebuild without a: the run +// finishes, partial_ssa names a alone, and b keeps its SSA body. +func TestSSA_MidBuildPanicDoesNotDeadlock(t *testing.T) { + in := loadInput(t, "ssapanic") + callHook = panicOn("Boom") + defer func() { callHook = nil }() + + var prog *ssa.Program + var pkgs []*ssa.Package + var deg []Degraded + withTimeout(t, 30*time.Second, func() { prog, pkgs, deg = buildSSA(in) }) + + if len(deg) != 1 || deg[0].Code != "partial_ssa" || !slices.Equal(deg[0].Packages, []string{"example.com/ssapanic/a"}) { + t.Fatalf("degraded = %+v, want one partial_ssa naming example.com/ssapanic/a", deg) + } + var built []string + for _, p := range pkgs { + built = append(built, p.Pkg.Path()) + } + if want := []string{"example.com/ssapanic/b", "example.com/ssapanic/g", "example.com/ssapanic/w"}; !slices.Equal(built, want) { + t.Fatalf("built = %v, want %v", built, want) + } + useB := prog.ImportedPackage("example.com/ssapanic/b").Func("UseB") + if useB == nil || len(useB.Blocks) == 0 { + t.Fatal("b.UseB has no SSA body after a's build panicked") + } + if a := prog.ImportedPackage("example.com/ssapanic/a"); a != nil { + t.Errorf("the rebuilt program still holds package a (%d members)", len(a.Members)) + } + if res := Run(in); len(res.Degraded) != 1 || res.Degraded[0].Code != "partial_ssa" { + t.Errorf("Run degraded = %+v, want only partial_ssa (VTA runs on the rebuilt program)", res.Degraded) + } +} + +// TestRun_FuncSetPanicDegradesToNoVTA: building the VTA function set builds +// method wrappers on demand ((*w.V).Never, which nothing calls, so the package +// build never made it). A panic there is recovered as no_vta. +func TestRun_FuncSetPanicDegradesToNoVTA(t *testing.T) { + in := loadInput(t, "ssapanic") + callHook = panicOn("Never") + defer func() { callHook = nil }() + + var res Result + withTimeout(t, 30*time.Second, func() { res = Run(in) }) + if len(res.Degraded) != 1 || res.Degraded[0].Code != "no_vta" { + t.Fatalf("degraded = %+v, want one no_vta", res.Degraded) + } +} diff --git a/testdata/ssapanic/a/a.go b/testdata/ssapanic/a/a.go new file mode 100644 index 0000000..837394b --- /dev/null +++ b/testdata/ssapanic/a/a.go @@ -0,0 +1,13 @@ +// Package a instantiates g.Map[int] and then calls Boom. A test hook panics +// on the call to Boom, after a's builder has created the shared instance. +package a + +import "example.com/ssapanic/g" + +func Boom() {} + +func UseA() int { + v := g.Map[int](1) + Boom() + return v +} diff --git a/testdata/ssapanic/b/b.go b/testdata/ssapanic/b/b.go new file mode 100644 index 0000000..e0611ea --- /dev/null +++ b/testdata/ssapanic/b/b.go @@ -0,0 +1,6 @@ +// Package b instantiates the same g.Map[int] as package a. +package b + +import "example.com/ssapanic/g" + +func UseB() int { return g.Map[int](2) } diff --git a/testdata/ssapanic/g/g.go b/testdata/ssapanic/g/g.go new file mode 100644 index 0000000..6d3c1cc --- /dev/null +++ b/testdata/ssapanic/g/g.go @@ -0,0 +1,5 @@ +// Package g declares a generic that packages a and b both instantiate at the +// same type argument, so they share one SSA instance. +package g + +func Map[T any](x T) T { return x } diff --git a/testdata/ssapanic/go.mod b/testdata/ssapanic/go.mod new file mode 100644 index 0000000..bc404d8 --- /dev/null +++ b/testdata/ssapanic/go.mod @@ -0,0 +1,3 @@ +module example.com/ssapanic + +go 1.21 diff --git a/testdata/ssapanic/w/w.go b/testdata/ssapanic/w/w.go new file mode 100644 index 0000000..efd57c8 --- /dev/null +++ b/testdata/ssapanic/w/w.go @@ -0,0 +1,7 @@ +// Package w has a value method that no code calls: its pointer wrapper +// (*V).Never is built only on demand, when the VTA function set is computed. +package w + +type V struct{} + +func (V) Never() {} From 845deab0add4f20fd9cf3806ad4b4f8b9bc1d4d7 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 00:30:10 -0400 Subject: [PATCH 10/65] feat(l4): join go/ssa to the L3 units --- internal/core/analyzer.go | 23 +- internal/core/sdg_test.go | 3 + internal/dataflow/bodynodes.go | 9 +- internal/dataflow/cfg_exits.go | 9 +- internal/dataflow/ir.go | 11 + internal/dataflow/run.go | 52 +- internal/dataflow/run_test.go | 56 +- internal/sdg/differential_test.go | 418 ++++++++++++ internal/sdg/ir.go | 9 +- internal/sdg/join.go | 439 ++++++++++++ internal/sdg/join_test.go | 713 ++++++++++++++++++++ internal/sdg/run.go | 61 +- internal/sdg/ssa_test.go | 43 +- internal/semantic_analysis/call_graph.go | 43 +- internal/syntactic_analysis/signature.go | 7 +- internal/syntactic_analysis/symbol_table.go | 6 +- testdata/cgo/c/c.go | 10 + testdata/cgo/c/d.go | 4 + testdata/cgo/go.mod | 3 + testdata/dataflow/generic/b.go | 13 + testdata/dataflow/xtest/a.go | 45 ++ testdata/dataflow/xtest/a_test.go | 19 + testdata/dataflow/xtest/b_test.go | 12 + testdata/dataflow/xtest/user/user.go | 9 + testdata/rangefunc/r.go | 11 + testdata/unmapped/go.mod | 3 + testdata/unmapped/u.go | 13 + 27 files changed, 1935 insertions(+), 109 deletions(-) create mode 100644 internal/sdg/differential_test.go create mode 100644 internal/sdg/join.go create mode 100644 internal/sdg/join_test.go create mode 100644 testdata/cgo/c/c.go create mode 100644 testdata/cgo/c/d.go create mode 100644 testdata/cgo/go.mod create mode 100644 testdata/dataflow/xtest/a.go create mode 100644 testdata/dataflow/xtest/a_test.go create mode 100644 testdata/dataflow/xtest/b_test.go create mode 100644 testdata/dataflow/xtest/user/user.go create mode 100644 testdata/unmapped/go.mod create mode 100644 testdata/unmapped/u.go diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index 8ee1a7d..80d5d2d 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -124,7 +124,13 @@ func (a *Analyzer) Analyze() (*schema.GoApplication, error) { var units []dataflow.Unit if a.opts.Level >= options.LevelDataflow { units = builder.L3Units(symbolTable) - a.l3 = dataflow.Run(builder.Fset(), units, KLimit(a.opts), a.opts.Jobs) + if a.opts.Level >= options.LevelSDG { + // L4 reads each result's def/use sets and call keys (the SSA + // join); Release drops them after sdg.Run. + a.l3 = dataflow.RunWithSideTables(builder.Fset(), units, KLimit(a.opts), a.opts.Jobs) + } else { + a.l3 = dataflow.Run(builder.Fset(), units, KLimit(a.opts), a.opts.Jobs) + } utils.Info("dataflow: %d callables", len(a.l3)) } @@ -147,14 +153,13 @@ func (a *Analyzer) Analyze() (*schema.GoApplication, error) { } } - // The last reader of Node.Stmt is done (dataflow.Run, or sdg.Run at L4, - // which binds call-site actuals from the AST; the emitters read keys, - // spans and edges only), so drop the AST references now: the parsed - // packages are then unreachable once Analyze returns, before the cache - // write and the emission. (#27) - for _, r := range a.l3 { - r.Release() - } + // The last reader of Node.Stmt and of the L4 side tables is done + // (dataflow.Run, or sdg.Run at L4, which joins SSA to them and binds + // call-site actuals from the AST; the emitters read keys, spans and + // edges only), so drop them now: the parsed packages are then + // unreachable once Analyze returns, before the cache write and the + // emission. (#27) + dataflow.ReleaseAll(a.l3) return a.finalizeAndCache(app) } diff --git a/internal/core/sdg_test.go b/internal/core/sdg_test.go index 5147d23..35446eb 100644 --- a/internal/core/sdg_test.go +++ b/internal/core/sdg_test.go @@ -35,6 +35,9 @@ func TestAnalyze_Level4RunsSDG(t *testing.T) { t.Fatalf("%v: node %s keeps its Stmt after -a 4", k, n.Key) } } + if r.DefUse != nil || r.CallKeys != nil { + t.Fatalf("%v keeps its L4 side tables after -a 4", k) + } } opts.Level = options.LevelDataflow diff --git a/internal/dataflow/bodynodes.go b/internal/dataflow/bodynodes.go index a2d0f14..a3738cc 100644 --- a/internal/dataflow/bodynodes.go +++ b/internal/dataflow/bodynodes.go @@ -124,13 +124,8 @@ func assignCallKeys(body *ast.BlockStmt, calls []UnitCall) map[*ast.CallExpr]str return out } -// unpairedCalls returns the number of recorded calls of body and how many of -// them no entry of calls names by Lparen. -func unpairedCalls(body *ast.BlockStmt, calls []UnitCall) (recorded, unpaired int) { - paired, recorded := pairCalls(body, calls) - return recorded, recorded - len(paired) -} - +// pairCalls pairs each recorded call of body with the L1 key whose Lparen is +// the call's, and returns the pairs and the number of recorded calls. func pairCalls(body *ast.BlockStmt, calls []UnitCall) (map[*ast.CallExpr]string, int) { byLparen := make(map[token.Pos]string, len(calls)) for _, c := range calls { diff --git a/internal/dataflow/cfg_exits.go b/internal/dataflow/cfg_exits.go index 4c7c39a..4e90b2d 100644 --- a/internal/dataflow/cfg_exits.go +++ b/internal/dataflow/cfg_exits.go @@ -319,7 +319,7 @@ func canPanic(n ast.Node, info *types.Info) bool { case *ast.ForStmt: parts = []ast.Node{s.Init, s.Cond, s.Post} case *ast.RangeStmt: - if rangesOverFunc(s.X, info) { + if RangesOverFunc(s.X, info) { return true // each iteration calls the iterator } parts = []ast.Node{s.Key, s.Value, s.X} @@ -493,10 +493,11 @@ func sliceToArray(call *ast.CallExpr, info *types.Info) bool { return ok } -// rangesOverFunc reports whether a range statement over x iterates a +// RangesOverFunc reports whether a range statement over x iterates a // function (range-over-func). Without info it cannot tell and says no; the -// header's own expressions are still inspected. -func rangesOverFunc(x ast.Expr, info *types.Info) bool { +// header's own expressions are still inspected. L4 lists the callables that +// hold one under degraded code range_func (D8). +func RangesOverFunc(x ast.Expr, info *types.Info) bool { if info == nil || x == nil { return false } diff --git a/internal/dataflow/ir.go b/internal/dataflow/ir.go index f0de084..b0895ff 100644 --- a/internal/dataflow/ir.go +++ b/internal/dataflow/ir.go @@ -76,9 +76,20 @@ type CdgEdge struct{ Src, Dst string } type DdgEdge struct{ Src, Dst, Var string } // Result is the L3 overlay of one callable body. +// +// DefUse and CallKeys are side tables for L4 (ruling R-P2), filled only by +// RunWithSideTables and dropped by Release; the emitters never read them: +// +// - DefUse is DefUses over Nodes, keyed by node key. +// - CallKeys maps the raw Lparen of every call the body pairs with an L1 +// call key to that key: the SSA join's call key (CallCommon.Pos() is the +// CallExpr's Lparen). type Result struct { Nodes []Node Cfg []CfgEdge Cdg []CdgEdge Ddg []DdgEdge + + DefUse map[string]DefUse + CallKeys map[token.Pos]string } diff --git a/internal/dataflow/run.go b/internal/dataflow/run.go index b79113a..d36f48b 100644 --- a/internal/dataflow/run.go +++ b/internal/dataflow/run.go @@ -56,6 +56,18 @@ type UnitCall struct { // keys and the body's recorded calls (RecordedCallCount) is a cross-check // only: it warns, and the unit is still built when every call is paired. func Run(fset *token.FileSet, units []Unit, k, jobs int) map[L3Key]Result { + return run(fset, units, k, jobs, false) +} + +// RunWithSideTables is Run that also keeps each result's DefUse and CallKeys +// (the -a 4 input). Below -a 4 nothing reads them, so Run drops them with the +// rest of each callable's working state instead of holding them all until +// Release. +func RunWithSideTables(fset *token.FileSet, units []Unit, k, jobs int) map[L3Key]Result { + return run(fset, units, k, jobs, true) +} + +func run(fset *token.FileSet, units []Unit, k, jobs int, keep bool) map[L3Key]Result { if jobs < 1 { jobs = runtime.GOMAXPROCS(0) } @@ -67,7 +79,7 @@ func Run(fset *token.FileSet, units []Unit, k, jobs int) map[L3Key]Result { go func() { defer wg.Done() for i := range next { - results[i] = runOne(fset, units[i], k) + results[i] = runOne(fset, units[i], k, keep) } }() } @@ -90,7 +102,7 @@ func Run(fset *token.FileSet, units []Unit, k, jobs int) map[L3Key]Result { // CFG, def-use or DDG builders) is recovered: that callable stays at L1 with a // warning (and, at -vv, the panic's stack), and the rest of the run — and the // L1/L2 output — is unaffected. -func runOne(fset *token.FileSet, u Unit, k int) (res *Result) { +func runOne(fset *token.FileSet, u Unit, k int, keep bool) (res *Result) { defer func() { if r := recover(); r != nil { utils.Warn("L3 skipped for %s (%s): %v", u.Key.Signature, u.Key.Path, r) @@ -102,7 +114,8 @@ func runOne(fset *token.FileSet, u Unit, k int) (res *Result) { if body == nil { return nil } - recorded, missing := unpairedCalls(body, u.Calls) + paired, recorded := pairCalls(body, u.Calls) + missing := recorded - len(paired) if recorded != len(u.Calls) { utils.Warn("L3 for %s (%s): %d L1 call keys for %d calls in the body", u.Key.Signature, u.Key.Path, len(u.Calls), recorded) } @@ -113,12 +126,20 @@ func runOne(fset *token.FileSet, u Unit, k int) (res *Result) { nodes, keyOf := BuildNodes(fset, body, u.Calls) kept, cfg := BuildCFG(nodes, keyOf, body, u.Info) du := DefUses(kept, u.Info, k, EntryBases(u.Fn, u.Info), u.Fn, keyOf) - return &Result{ + res = &Result{ Nodes: kept, Cfg: cfg, Cdg: BuildCDG(kept, cfg), Ddg: BuildDDG(kept, cfg, du), } + if keep { + res.DefUse = du + res.CallKeys = make(map[token.Pos]string, len(paired)) + for call, key := range paired { + res.CallKeys[call.Lparen] = key + } + } + return res } func funcBody(fn ast.Node) *ast.BlockStmt { @@ -131,13 +152,24 @@ func funcBody(fn ast.Node) *ast.BlockStmt { return nil } -// Release drops the result's references into the AST (each Node.Stmt), so a -// finished run no longer keeps the parsed packages reachable. Every other -// field (keys, spans, edges) stays: they are all the emitter reads. Call it -// once the last consumer of Stmt is done; Run's own stages are the only one -// today. -func (r Result) Release() { +// Release drops the result's references into the AST (each Node.Stmt) and +// its L4 side tables (DefUse, CallKeys), so a finished run no longer keeps +// the parsed packages reachable. Every other field (keys, spans, edges) +// stays: they are all the emitter reads. Call it once the last consumer is +// done: dataflow.Run below -a 4, sdg.Run at -a 4. A Result held in a map is +// a copy: release a map with ReleaseAll. +func (r *Result) Release() { for i := range r.Nodes { r.Nodes[i].Stmt = nil } + r.DefUse = nil + r.CallKeys = nil +} + +// ReleaseAll releases every result of m in place. +func ReleaseAll(m map[L3Key]Result) { + for k, r := range m { + r.Release() + m[k] = r + } } diff --git a/internal/dataflow/run_test.go b/internal/dataflow/run_test.go index 92d6981..b68e541 100644 --- a/internal/dataflow/run_test.go +++ b/internal/dataflow/run_test.go @@ -11,6 +11,7 @@ import ( "io" "os" "path/filepath" + "reflect" "runtime" "strings" "testing" @@ -422,8 +423,8 @@ func TestResult_ReleaseDropsASTs(t *testing.T) { stmts++ } } - r.Release() } + dataflow.ReleaseAll(out) if stmts == 0 { t.Fatal("no node had a Stmt before Release") } @@ -437,4 +438,57 @@ func TestResult_ReleaseDropsASTs(t *testing.T) { if !bytes.Equal(before, emit()) { t.Error("-a 3 JSON changed after Release") } + for key, r := range out { + if r.DefUse != nil || r.CallKeys != nil { + t.Errorf("%v: DefUse / CallKeys kept after Release", key) + } + } +} + +// TestRun_KeepsDefUseAndCallKeys: with side tables (-a 4), until Release, a +// result keeps its nodes' def/use sets (keyed by node key) and its call keys +// by Lparen, which the L4 SSA join and the go/ssa differential read (ruling +// R-P2). Plain Run (-a 3) keeps neither, and its graphs are the same. +func TestRun_KeepsDefUseAndCallKeys(t *testing.T) { + fset, f, info := parseCheck(t, lineDirectiveSrc) + g := funcDecl(t, f, "g") + calls := gateCallKeys(fset, g.Body) + key := dataflow.L3Key{Path: "p.go", Signature: "p.g"} + units := []dataflow.Unit{{Key: key, Fn: g, Info: info, Calls: calls}} + res, ok := dataflow.RunWithSideTables(fset, units, 3, 1)[key] + if !ok { + t.Fatal("no result for g") + } + plain := dataflow.Run(fset, units, 3, 1)[key] + if plain.DefUse != nil || plain.CallKeys != nil { + t.Error("Run kept the L4 side tables") + } + if !reflect.DeepEqual(plain.Cfg, res.Cfg) || !reflect.DeepEqual(plain.Ddg, res.Ddg) || !reflect.DeepEqual(plain.Cdg, res.Cdg) { + t.Error("side tables changed the graphs") + } + if len(res.CallKeys) != len(calls) { + t.Fatalf("CallKeys = %v, want one per call (%d)", res.CallKeys, len(calls)) + } + for _, c := range calls { + if res.CallKeys[c.Lparen] != c.Key { + t.Errorf("CallKeys[%v] = %q, want %q", c.Lparen, res.CallKeys[c.Lparen], c.Key) + } + } + if len(res.DefUse) != len(res.Nodes) { + t.Fatalf("DefUse has %d entries for %d nodes", len(res.DefUse), len(res.Nodes)) + } + for _, n := range res.Nodes { + if _, ok := res.DefUse[n.Key]; !ok { + t.Errorf("node %s has no DefUse", n.Key) + } + } + res.Release() + if res.DefUse != nil || res.CallKeys != nil { + t.Error("Release kept DefUse / CallKeys") + } + for _, n := range res.Nodes { + if n.Stmt != nil { + t.Errorf("node %s keeps its Stmt after Release", n.Key) + } + } } diff --git a/internal/sdg/differential_test.go b/internal/sdg/differential_test.go new file mode 100644 index 0000000..5d785ed --- /dev/null +++ b/internal/sdg/differential_test.go @@ -0,0 +1,418 @@ +package sdg + +import ( + "fmt" + "go/ast" + "go/token" + "os" + "sort" + "strings" + "testing" + + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// The go/ssa def-use differential (#22, spec §4.5 "L3 precision"): an +// independent answer key for the syntactic L3 ddg. It does not gate: L3 and +// SSA model different things (SSA is value flow over registers, L3 is +// access-path flow between statements), so mismatches are reported, and the +// test asserts only that the mapping layer works (ruling R-P6). + +// ssaPair is one SSA def-use pair of a callable mapped to its L3 body keys. +type ssaPair struct { + Unit dataflow.L3Key + Def, Use string // local body keys + What string // the SSA value, for the report +} + +// bodyIndex maps a raw position to the innermost CFG body node of one L3 +// result containing it. Satellite call nodes are skipped (their def/use +// belongs to the host statement, DefUseOf), and so are deferred call nodes: +// SSA evaluates a deferred call's function value and arguments at the +// registration, the `defer` statement node. A position in the signature +// (a parameter spilled to the heap) is @entry; one at the closing brace (the +// implicit return) is @exit. +type bodyIndex struct { + nodes []dataflow.Node + lbrace, rbrace token.Pos +} + +func newBodyIndex(r dataflow.Result, fn ast.Node) bodyIndex { + var ns []dataflow.Node + for _, n := range r.Nodes { + if n.Pos.IsValid() && n.Parent == "" && !n.Deferred { + ns = append(ns, n) + } + } + b := bodyIndex{nodes: ns} + switch fn := fn.(type) { + case *ast.FuncDecl: + b.lbrace, b.rbrace = fn.Body.Lbrace, fn.Body.Rbrace + case *ast.FuncLit: + b.lbrace, b.rbrace = fn.Body.Lbrace, fn.Body.Rbrace + } + return b +} + +func (b bodyIndex) at(pos token.Pos) (string, bool) { + if b.lbrace.IsValid() { + if pos < b.lbrace { + return dataflow.EntryKey, true + } + if pos >= b.rbrace { + return dataflow.ExitKey, true + } + } + best := -1 + for i, n := range b.nodes { + if n.Pos <= pos && pos < n.End { + if best < 0 || n.End-n.Pos < b.nodes[best].End-b.nodes[best].Pos { + best = i + } + } + } + if best < 0 { + return "", false + } + return b.nodes[best].Key, true +} + +// ssaPairs returns the def-use pairs of the SSA functions of funcs that join +// a unit by body, mapped to body keys: for each SSA value with a definition +// site (a Parameter or FreeVar: @entry; an instruction: the body node at its +// position) and each of its referrers, the referrer's body node. Phis and +// position-less value instructions (Extract, implicit conversions, +// MakeClosure) are looked through on both sides; a Store through a value is +// a write, not a use. Pairs within one node are dropped (L3 edges are +// between nodes). +func ssaPairs(j *join, funcs map[*ssa.Function]bool) (pairs []ssaPair, instrs, unmappedInstrs int) { + idx := map[dataflow.L3Key]bodyIndex{} + for fn := range funcs { + unit, kind := j.resolve(fn) + if kind != joinBody { + continue + } + r, ok := j.results[unit] + if !ok { + continue + } + bi, ok := idx[unit] + if !ok { + bi = newBodyIndex(r, j.units[unit].Fn) + idx[unit] = bi + } + keyOf := func(instr ssa.Instruction) (string, bool) { + p := instr.Pos() + if !p.IsValid() { + return "", false + } + instrs++ + k, ok := bi.at(p) + if !ok { + unmappedInstrs++ + } + return k, ok + } + seen := map[ssaPair]bool{} + add := func(def, use, what string) { + if def == use { + return + } + p := ssaPair{Unit: unit, Def: def, Use: use, What: what} + if !seen[p] { + seen[p] = true + pairs = append(pairs, p) + } + } + var values []ssa.Value + for _, p := range fn.Params { + values = append(values, p) + } + for _, fv := range fn.FreeVars { + values = append(values, fv) + } + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + if v, ok := instr.(ssa.Value); ok { + values = append(values, v) + } + } + } + for _, v := range values { + var def string + switch v := v.(type) { + case *ssa.Parameter, *ssa.FreeVar: + def = dataflow.EntryKey + case *ssa.Phi: + continue // looked through from its operands' side + case ssa.Instruction: + k, ok := keyOf(v) + if !ok { + continue // position-less: looked through from its operands + } + def = k + } + for _, use := range useSites(v, keyOf) { + add(def, use, v.Name()+" = "+v.String()) + } + } + } + sort.Slice(pairs, func(a, b int) bool { + x, y := pairs[a], pairs[b] + if x.Unit != y.Unit { + return x.Unit.Path+x.Unit.Signature < y.Unit.Path+y.Unit.Signature + } + if x.Def != y.Def { + return x.Def < y.Def + } + if x.Use != y.Use { + return x.Use < y.Use + } + return x.What < y.What + }) + return pairs, instrs, unmappedInstrs +} + +// useSites returns the body keys of v's referrers, looking through phis and +// position-less value instructions. +func useSites(v ssa.Value, keyOf func(ssa.Instruction) (string, bool)) []string { + var out []string + seen := map[ssa.Value]bool{} + var walk func(ssa.Value) + walk = func(v ssa.Value) { + if seen[v] { + return + } + seen[v] = true + refs := v.Referrers() + if refs == nil { + return + } + for _, r := range *refs { + if phi, ok := r.(*ssa.Phi); ok { + walk(phi) + continue + } + if st, ok := r.(*ssa.Store); ok && st.Addr == v && st.Val != v { + continue // a write through v, not a read of it + } + if writeOnlyAddr(r, v) { + continue // &v.f / &v[i] taken only to store through: a write + } + if k, ok := keyOf(r); ok { + out = append(out, k) + continue + } + if rv, ok := r.(ssa.Value); ok { + walk(rv) + } + } + } + walk(v) + return out +} + +// writeOnlyAddr reports whether r is a field or element address of v whose +// every referrer stores through it. +func writeOnlyAddr(r ssa.Instruction, v ssa.Value) bool { + var refs *[]ssa.Instruction + switch a := r.(type) { + case *ssa.FieldAddr: + if a.X != v { + return false + } + refs = a.Referrers() + case *ssa.IndexAddr: + if a.X != v { + return false + } + refs = a.Referrers() + default: + return false + } + if refs == nil || len(*refs) == 0 { + return false + } + for _, x := range *refs { + if st, ok := x.(*ssa.Store); !ok || st.Addr != r.(ssa.Value) { + return false + } + } + return true +} + +// Triage classes of an SSA pair with no direct L3 ddg edge. All but +// classGap are known differences of representation: the flow is in L3. +const ( + classTransitive = "transitive in L3 (SSA copy-propagates through copies, closure values and captured cells)" + classClause = "bound at a different node (switch tag / select comm / type-switch binding: header in one, clause in the other)" + classDefer = "deferred call: SSA evaluates at the registration, L3 reads at the deferred call node (two-node rule)" + classSlot = "SSA bookkeeping (anonymous result slot)" + classGap = "candidate gap: no L3 ddg path" +) + +// classify triages a mapped SSA pair p of result r that has no direct L3 +// ddg edge. +func classify(r dataflow.Result, p ssaPair) string { + succ := map[string][]string{} + for _, e := range r.Ddg { + succ[e.Src] = append(succ[e.Src], e.Dst) + } + reach := func(src, dst string) bool { + seen := map[string]bool{src: true} + for work := []string{src}; len(work) > 0; work = work[1:] { + for _, m := range succ[work[0]] { + if m == dst { + return true + } + if !seen[m] { + seen[m] = true + work = append(work, m) + } + } + } + return false + } + node := map[string]dataflow.Node{} + for _, n := range r.Nodes { + node[n.Key] = n + } + // header returns the innermost switch/select node holding node n. + header := func(n dataflow.Node) (dataflow.Node, bool) { + var best dataflow.Node + found := false + for _, h := range r.Nodes { + if h.Kind == dataflow.KindSwitch && h.Pos < n.Pos && n.End <= h.End && (!found || h.End-h.Pos < best.End-best.Pos) { + best, found = h, true + } + } + return best, found + } + clauses := func(h dataflow.Node) []dataflow.Node { + var out []dataflow.Node + for _, n := range r.Nodes { + switch n.Stmt.(type) { + case *ast.CaseClause, *ast.CommClause: + if hh, ok := header(n); ok && hh.Key == h.Key { + out = append(out, n) + } + } + } + return out + } + d, u := node[p.Def], node[p.Use] + switch s := u.Stmt.(type) { + case *ast.DeferStmt: + for _, n := range r.Nodes { + if n.Deferred && n.Pos == s.Call.Pos() && (p.Def == n.Key || reach(p.Def, n.Key)) { + return classDefer + } + } + case *ast.CaseClause, *ast.CommClause, *ast.SelectStmt: + h := u + if _, ok := u.Stmt.(*ast.SelectStmt); !ok { + h, _ = header(u) + } + if p.Def == h.Key || reach(p.Def, h.Key) { + return classClause + } + for _, c := range clauses(h) { + if p.Def == c.Key || reach(p.Def, c.Key) { + return classClause + } + } + } + // A value SSA binds at a header (a select's received tuple) or at a + // clause (a type switch's per-clause assertion) that L3 binds at the + // other. + if h, ok := header(d); ok && reach(h.Key, p.Use) { + return classClause + } + if d.Kind == dataflow.KindSwitch { + for _, c := range clauses(d) { + if reach(c.Key, p.Use) { + return classClause + } + } + } + if strings.Contains(p.What, " = local ") && strings.HasSuffix(p.What, " ()") { + return classSlot + } + if reach(p.Def, p.Use) { + return classTransitive + } + return classGap +} + +// TestSSADifferential_Report: on the fixtures, every project SSA function +// and call joins, and at least one SSA def-use pair maps to body keys; then +// it logs every mapped pair with no L3 ddg edge between the same two nodes +// (any variable), triaged (classify). Mismatches never fail the test. Set +// CANGO_SSA_DIFF_OUT= to also write the report there. +func TestSSADifferential_Report(t *testing.T) { + var report strings.Builder + for _, name := range []string{"dataflow", "chi"} { + in := loadInput(t, name) + _, j, funcs := joined(t, in) + // The mapping layer is complete: every function joins (but those of + // declarations the symbol table dropped, pinned by + // TestJoin_EveryProjectFunctionMaps) and no positioned call of a + // joined function fails to join. + dropped := droppedDecls(in) + for fn := range funcs { + if _, kind := j.resolve(fn); kind == joinNone && fn.Synthetic != synthPackageInit && !inDropped(fn, dropped) { + t.Errorf("%s: %s joins nothing", name, fn) + } + } + for _, b := range describeUnmapped(j, funcs) { + if strings.HasPrefix(b, "call ") { + t.Errorf("%s: unmapped %s", name, b) + } + } + pairs, instrs, unmapped := ssaPairs(j, funcs) + if len(pairs) == 0 { + t.Errorf("%s: no SSA def-use pair mapped to body keys", name) + } + if unmapped > 0 { + t.Errorf("%s: %d positioned SSA instructions of joined functions lie in no body node", name, unmapped) + } + has := map[[2]string]map[dataflow.L3Key]bool{} + for k, r := range in.L3 { + for _, e := range r.Ddg { + p := [2]string{e.Src, e.Dst} + if has[p] == nil { + has[p] = map[dataflow.L3Key]bool{} + } + has[p][k] = true + } + } + missing := 0 + classes := map[string]int{} + for _, p := range pairs { + if has[[2]string{p.Def, p.Use}][p.Unit] { + continue + } + missing++ + class := classify(in.L3[p.Unit], p) + classes[class]++ + fmt.Fprintf(&report, "%s\t%s\t%s\t%s -> %s\t%s\t%s\n", name, p.Unit.Path, p.Unit.Signature, p.Def, p.Use, p.What, class) + } + t.Logf("%s: %d SSA def-use pairs mapped (%d positioned instructions, %d outside every body node); %d with no L3 ddg edge", + name, len(pairs), instrs, unmapped, missing) + for _, c := range []string{classTransitive, classClause, classDefer, classSlot, classGap} { + if classes[c] > 0 { + t.Logf("%s: %4d %s", name, classes[c], c) + } + } + } + if report.Len() > 0 { + t.Logf("SSA def-use pairs with no L3 ddg edge (fixture, file, callable, def -> use, SSA value, class):\n%s", report.String()) + } + if out := os.Getenv("CANGO_SSA_DIFF_OUT"); out != "" { + if err := os.WriteFile(out, []byte(report.String()), 0o644); err != nil { + t.Error(err) + } + } +} diff --git a/internal/sdg/ir.go b/internal/sdg/ir.go index a4d1843..26b9ca4 100644 --- a/internal/sdg/ir.go +++ b/internal/sdg/ir.go @@ -95,7 +95,10 @@ type Models map[string][][2]string // Degraded codes (spec §4.15; an open vocabulary of plain strings). const ( - CodeNoSSA = "no_ssa" - CodePartialSSA = "partial_ssa" - CodeNoVTA = "no_vta" + CodeNoSSA = "no_ssa" + CodePartialSSA = "partial_ssa" + CodeNoVTA = "no_vta" + CodeUnmappedSSA = "unmapped_ssa" + CodeL3Skipped = "l3_skipped" + CodeRangeFunc = "range_func" ) diff --git a/internal/sdg/join.go b/internal/sdg/join.go new file mode 100644 index 0000000..91ffc06 --- /dev/null +++ b/internal/sdg/join.go @@ -0,0 +1,439 @@ +package sdg + +import ( + "fmt" + "go/ast" + "go/token" + "go/types" + "sort" + "strings" + + "golang.org/x/tools/go/packages" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// SSA Synthetic tags the join tells apart (x/tools v0.52 go/ssa). +const ( + synthPackageInit = "package initializer" + synthYield = "range-over-func yield" + synthInstWrapper = "instantiation wrapper" +) + +// joinKind is how an SSA function joined its callable. +type joinKind int + +const ( + joinNone joinKind = iota + joinBody // its code is the callable's body: by syntax, origin, or a folded yield + joinWrapper // a synthetic wrapper (or a bodiless stub) of the callable: by Object() +) + +// join maps go/ssa back to the L3 units (spec §4.8, SSA → L3 join). It keys +// only on AST pointer identity and raw Lparen positions, never on line:col or +// SSA names, so //line directives, SSA's anonymous-function numbering and +// build order cannot move a mapping. Its maps are read-only after newJoin, so +// it is safe for concurrent use, and it never builds SSA code (it may be +// used after a no_vta). +type join struct { + fset *token.FileSet + projectDir string + byFn map[ast.Node]dataflow.L3Key // Unit.Fn → unit + bySig map[dataflow.L3Key]bool // (declaring file, signature) of every unit and interface method + calls map[dataflow.L3Key]map[token.Pos]string // unit → call Lparen → L1/L3 call key + projPaths map[string]bool // import paths of the project packages + units map[dataflow.L3Key]dataflow.Unit // by key + results map[dataflow.L3Key]dataflow.Result // in.L3 + pkgs map[string]*packages.Package // in.Pkgs +} + +// newJoin indexes in's units. A unit's call keys are its L3 result's +// CallKeys; a unit with no result (l3_skipped), or one built without side +// tables, falls back to its L1 call keys (Unit.Calls), which the emitter +// writes either way. +func newJoin(in Input) *join { + j := &join{ + fset: in.Fset, + projectDir: in.ProjectDir, + byFn: make(map[ast.Node]dataflow.L3Key, len(in.Units)), + bySig: make(map[dataflow.L3Key]bool, len(in.Units)), + calls: make(map[dataflow.L3Key]map[token.Pos]string, len(in.Units)), + projPaths: map[string]bool{}, + units: make(map[dataflow.L3Key]dataflow.Unit, len(in.Units)), + results: in.L3, + pkgs: in.Pkgs, + } + for _, u := range in.Units { + j.byFn[u.Fn] = u.Key + j.bySig[u.Key] = true + j.units[u.Key] = u + if r, ok := in.L3[u.Key]; ok && r.CallKeys != nil { + j.calls[u.Key] = r.CallKeys + continue + } + m := make(map[token.Pos]string, len(u.Calls)) + for _, c := range u.Calls { + m[c.Lparen] = c.Key + } + j.calls[u.Key] = m + } + // Interface methods have no body and so no unit, but a $bound wrapper of + // one (a method value of an interface) still names it: index them by + // (declaring file, signature) too, the file being the interface's. + for path, f := range in.SymbolTable { + for _, t := range f.Types { + for _, c := range t.Methods { + if c.Path == "" { + j.bySig[dataflow.L3Key{Path: path, Signature: c.Signature}] = true + } + } + } + } + for _, p := range projectPackages(in) { + j.projPaths[p.PkgPath] = true + } + return j +} + +// fnUnit returns the L3 callable fn joins: +// +// 1. fn.Syntax() is a unit's Fn (a declared function or method, init#N, +// a function literal, a generic instance or instantiation wrapper — +// both keep their origin's syntax; a wrapper joins as a wrapper). +// 2. Else, for a top-level function, fn.Origin()'s syntax: an instance +// with no syntax of its own (go/ssa v0.52 always copies it, so this is +// a guard). Closures never call Origin: for a closure of an instance it +// may build a package. +// 3. Else, for a function with no syntax — a synthetic wrapper ($bound, +// $thunk, promoted or pointer wrapper) or a bodiless stub of a project +// function seen through another test variant of its package — the +// function fn.Object() names, by (declaring file, SignatureOf), the +// symbol table's own canonicalizer. An interface method's key (the +// interface's file) counts here too. +// 4. A range-over-func yield folds into fn.Parent(). +// +// The package initializer, closures of package-level variable initializers +// and "from type information" functions declared outside the project map to +// nothing. +func (j *join) fnUnit(fn *ssa.Function) (dataflow.L3Key, bool) { + k, kind := j.resolve(fn) + return k, kind != joinNone +} + +func (j *join) resolve(fn *ssa.Function) (dataflow.L3Key, joinKind) { + for fn != nil && fn.Synthetic == synthYield { + fn = fn.Parent() + } + if fn == nil { + return dataflow.L3Key{}, joinNone + } + if syn := fn.Syntax(); syn != nil { + k, ok := j.byFn[syn] + switch { + case !ok: + return dataflow.L3Key{}, joinNone + case strings.HasPrefix(fn.Synthetic, synthInstWrapper): + // It keeps its origin's syntax, but its body is a synthetic + // call of the origin. + return k, joinWrapper + } + return k, joinBody + } + if fn.Parent() == nil { + if o := fn.Origin(); o != nil && o != fn { + if k, ok := j.byFn[o.Syntax()]; ok { + return k, joinBody + } + } + } + if fn.Synthetic == synthPackageInit { + return dataflow.L3Key{}, joinNone + } + if obj, ok := fn.Object().(*types.Func); ok { + if k, ok := j.declKey(obj.Origin()); ok && j.bySig[k] { + return k, joinWrapper + } + } + return dataflow.L3Key{}, joinNone +} + +// declKey is obj's (declaring file, signature) when it is declared in a +// project file. +func (j *join) declKey(obj *types.Func) (dataflow.L3Key, bool) { + tf := j.fset.File(obj.Pos()) + if tf == nil || !utils.IsWithin(j.projectDir, tf.Name()) { + return dataflow.L3Key{}, false + } + rel := utils.RelativePath(j.projectDir, tf.Name()) + if utils.IsVendored(rel) { + return dataflow.L3Key{}, false + } + return dataflow.L3Key{Path: rel, Signature: syntactic_analysis.SignatureOf(obj)}, true +} + +// callSite returns the unit and L1/L3 call key of instr: its function must +// join by body (a wrapper's calls are synthetic) and its CallCommon.Pos() — +// the CallExpr's Lparen — must be one of the unit's call keys. A call with +// no position (one go/ssa synthesizes: the defer stack, a nil check, an +// init call, a range-over-func iterator call) maps to nothing. +func (j *join) callSite(instr ssa.CallInstruction) (unit dataflow.L3Key, callKey string, ok bool) { + pos := instr.Common().Pos() + if !pos.IsValid() { + return dataflow.L3Key{}, "", false + } + k, kind := j.resolve(instr.Parent()) + if kind != joinBody { + return dataflow.L3Key{}, "", false + } + key, ok := j.calls[k][pos] + if !ok { + return dataflow.L3Key{}, "", false + } + return k, key, true +} + +// externalOf returns the full callee signature (the app.Externals key) of a +// function fn.Object() names outside the project: a library function or +// method, a wrapper of one, or its "from type information" stub. The +// signature is L2's (semantic_analysis.CalleeSignatureOf), so L2 and L4 +// external ids agree. +func (j *join) externalOf(fn *ssa.Function) (sig string, ok bool) { + obj, ok := fn.Object().(*types.Func) + if !ok || obj.Pkg() == nil || !j.isLibrary(obj) { + return "", false + } + return semantic_analysis.CalleeSignatureOf(obj), true +} + +// isLibrary mirrors the L2 rule: declared in a file outside the project or +// vendored; without a file, in a package that is not a project package. +func (j *join) isLibrary(obj *types.Func) bool { + if tf := j.fset.File(obj.Pos()); tf != nil { + if !utils.IsWithin(j.projectDir, tf.Name()) { + return true + } + return utils.IsVendored(utils.RelativePath(j.projectDir, tf.Name())) + } + return !j.projPaths[obj.Pkg().Path()] +} + +// joinCoverage counts how the project functions and their call instructions +// joined. Only call instructions with a source position count: a call +// go/ssa synthesizes has none and no call node. +type joinCoverage struct { + Funcs, Unmapped, Wrappers, Inits int // Inits: package initializers (no callable by design) + Calls, UnmappedCalls, Synthetic int // Calls: positioned calls in body-joined functions + unmappedPkgs map[string]bool +} + +// coverage joins every function of funcs (the project functions) and its +// calls. A package initializer has no callable (L1 records none for +// package-level variable initializers) and its calls are not counted; a +// closure in such an initializer is unmapped. +func (j *join) coverage(funcs map[*ssa.Function]bool) joinCoverage { + c := joinCoverage{unmappedPkgs: map[string]bool{}} + for fn := range funcs { + if fn.Synthetic == synthPackageInit { + c.Inits++ + continue + } + c.Funcs++ + _, kind := j.resolve(fn) + switch kind { + case joinNone: + c.Unmapped++ + c.unmappedPkgs[homePath(fn)] = true + utils.Debug("L4: unmapped SSA function %s [%s]", fn, fn.Synthetic) + case joinWrapper: + c.Wrappers++ + } + forCalls(fn, func(call ssa.CallInstruction) { + switch { + case !call.Common().Pos().IsValid(): + c.Synthetic++ + case kind == joinWrapper: + c.Synthetic++ + default: + c.Calls++ + if _, _, ok := j.callSite(call); !ok { + c.UnmappedCalls++ + c.unmappedPkgs[homePath(fn)] = true + utils.Debug("L4: unmapped SSA call %s in %s at %s", call, fn, j.fset.PositionFor(call.Common().Pos(), false)) + } + } + }) + } + return c +} + +// degraded is the unmapped_ssa entry for c, or nil. A listed package with +// cgo files gets a note: cgo files are not in the symbol table yet (#29). +func (j *join) degraded(c joinCoverage) *Degraded { + if c.Unmapped == 0 && c.UnmappedCalls == 0 { + return nil + } + pkgs := sortedKeys(c.unmappedPkgs) + detail := fmt.Sprintf("%d SSA function(s) and %d call instruction(s) map to no callable or call node", c.Unmapped, c.UnmappedCalls) + for _, p := range pkgs { + if hasCgo(j.fset, j.pkgs[p]) { + detail += "; cgo files are not in the symbol table (#29)" + break + } + } + return &Degraded{Code: CodeUnmappedSSA, Detail: detail, Packages: pkgs} +} + +// hasCgo reports whether p compiles cgo-generated files: go/packages then +// parses the generated files (in the build cache) instead of the ones in +// GoFiles. The load does not request CompiledGoFiles, so the test is a +// Syntax file that GoFiles does not list. +func hasCgo(fset *token.FileSet, p *packages.Package) bool { + if p == nil { + return false + } + files := make(map[string]bool, len(p.GoFiles)) + for _, f := range p.GoFiles { + files[f] = true + } + for _, f := range p.Syntax { + if tf := fset.File(f.Pos()); tf != nil && !files[tf.Name()] { + return true + } + } + return false +} + +// forCalls calls f on every call instruction of fn's body. +func forCalls(fn *ssa.Function, f func(ssa.CallInstruction)) { + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + if call, ok := instr.(ssa.CallInstruction); ok { + f(call) + } + } + } +} + +// homePath is the import path of the package fn's code comes from: its own +// package, its generic origin's, its enclosing function's, or (for a +// wrapper) the wrapped object's. +func homePath(fn *ssa.Function) string { + for f := fn; f != nil; f = f.Parent() { + if f.Pkg != nil { + return f.Pkg.Pkg.Path() + } + if f.Parent() == nil { + if o := f.Origin(); o != nil && o.Pkg != nil { + return o.Pkg.Pkg.Path() + } + if obj := f.Object(); obj != nil && obj.Pkg() != nil { + return obj.Pkg().Path() + } + } + } + return "" +} + +// l3Skipped is the l3_skipped entry (spec §4.15, A16): the units with no L3 +// result (a recovered panic, or a body call with no L1 key), or nil. +func l3Skipped(in Input) *Degraded { + var keys []dataflow.L3Key + for _, u := range in.Units { + if _, ok := in.L3[u.Key]; !ok { + keys = append(keys, u.Key) + } + } + if len(keys) == 0 { + return nil + } + sortKeys(keys) + return &Degraded{ + Code: CodeL3Skipped, + Detail: fmt.Sprintf("%d callable(s) have no L3 result (a recovered panic or a call with no L1 key)", len(keys)), + Callables: keys, + } +} + +// rangeFunc is the range_func entry (spec §4.15, D8): the units whose own +// body (not a nested literal: that is its own unit) ranges over a function, +// and their packages, or nil. It reads the AST only, so it holds whether or +// not SSA was built. +func rangeFunc(in Input) *Degraded { + pkgOf := map[string]string{} + for _, p := range projectPackages(in) { + for _, f := range p.GoFiles { + pkgOf[f] = p.PkgPath + } + } + var keys []dataflow.L3Key + pkgs := map[string]bool{} + for _, u := range in.Units { + if !rangesOverFunc(u) { + continue + } + keys = append(keys, u.Key) + if tf := in.Fset.File(u.Fn.Pos()); tf != nil { + if p, ok := pkgOf[tf.Name()]; ok { + pkgs[p] = true + } + } + } + if len(keys) == 0 { + return nil + } + sortKeys(keys) + return &Degraded{ + Code: CodeRangeFunc, + Detail: "range-over-func is not modelled at L4 (D8): what the iterator passes to yield does not reach the loop variables", + Packages: sortedKeys(pkgs), + Callables: keys, + } +} + +func rangesOverFunc(u dataflow.Unit) bool { + var body *ast.BlockStmt + switch fn := u.Fn.(type) { + case *ast.FuncDecl: + body = fn.Body + case *ast.FuncLit: + body = fn.Body + } + if body == nil || u.Info == nil { + return false + } + found := false + ast.Inspect(body, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.FuncLit: + return false + case *ast.RangeStmt: + if dataflow.RangesOverFunc(n.X, u.Info) { + found = true + } + } + return !found + }) + return found +} + +func sortKeys(keys []dataflow.L3Key) { + sort.Slice(keys, func(i, j int) bool { + if keys[i].Path != keys[j].Path { + return keys[i].Path < keys[j].Path + } + return keys[i].Signature < keys[j].Signature + }) +} + +func sortedKeys(m map[string]bool) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + sort.Strings(out) + return out +} diff --git a/internal/sdg/join_test.go b/internal/sdg/join_test.go new file mode 100644 index 0000000..87eaf6c --- /dev/null +++ b/internal/sdg/join_test.go @@ -0,0 +1,713 @@ +package sdg + +import ( + "go/ast" + "go/types" + "path/filepath" + "slices" + "sort" + "strings" + "testing" + + "golang.org/x/tools/go/ssa" + "golang.org/x/tools/go/ssa/ssautil" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// joined builds SSA over in and the join, and returns them with the +// project-function set the join is measured on: every function of the +// program whose code is project code (projectFuncs' rule) — not only the VTA +// seeds, so wrappers built on demand and every test variant count too. +func joined(t *testing.T, in Input) (*ssa.Program, *join, map[*ssa.Function]bool) { + t.Helper() + prog, pkgs, deg := buildSSA(in) + for _, d := range deg { + t.Logf("degraded: %+v", d) + } + if len(pkgs) == 0 { + t.Fatal("no SSA package built") + } + j := newJoin(in) + project := map[string]bool{} + for _, p := range pkgs { + project[p.Pkg.Path()] = true + } + funcs := map[*ssa.Function]bool{} + for fn := range ssautil.AllFunctions(prog) { + if project[homePath(fn)] && !strings.HasPrefix(fn.Synthetic, "from type information") { + funcs[fn] = true + } + } + for fn := range projectFuncs(prog, pkgs) { + funcs[fn] = true + } + return prog, j, funcs +} + +// describeUnmapped lists the functions and calls of funcs that join nothing. +func describeUnmapped(j *join, funcs map[*ssa.Function]bool) []string { + var out []string + for fn := range funcs { + if fn.Synthetic == synthPackageInit { + continue + } + _, kind := j.resolve(fn) + if kind == joinNone { + out = append(out, "func "+fn.String()+" ["+fn.Synthetic+"]") + continue + } + if kind != joinBody { + continue + } + forCalls(fn, func(c ssa.CallInstruction) { + if c.Common().Pos().IsValid() { + if _, _, ok := j.callSite(c); !ok { + out = append(out, "call "+c.String()+" in "+fn.String()) + } + } + }) + } + sort.Strings(out) + return out +} + +// droppedDecls returns the bodies of the project FuncDecls that have no L3 +// unit: the declarations the symbol table dropped (cgo wrappers, #29, and +// methods whose receiver type it did not find — today also a method declared +// before its receiver type in the same file). Their SSA functions, and +// closures in them, cannot join: that is unmapped_ssa by design. +func droppedDecls(in Input) map[ast.Node]bool { + units := map[ast.Node]bool{} + for _, u := range in.Units { + units[u.Fn] = true + } + out := map[ast.Node]bool{} + for _, p := range projectPackages(in) { + for _, f := range p.Syntax { + tf := in.Fset.File(f.Pos()) + if tf == nil || !utils.IsWithin(in.ProjectDir, tf.Name()) { + continue + } + for _, d := range f.Decls { + if fd, ok := d.(*ast.FuncDecl); ok && fd.Body != nil && !units[fd] { + out[fd] = true + } + } + } + } + return out +} + +// inDropped reports whether fn's code is (inside) a dropped declaration. +func inDropped(fn *ssa.Function, dropped map[ast.Node]bool) bool { + for f := fn; f != nil; f = f.Parent() { + if dropped[f.Syntax()] { + return true + } + } + return false +} + +// TestJoin_EveryProjectFunctionMaps: on the fixtures every project SSA +// function joins a callable and every positioned call instruction in a +// body-joined function joins its call node, except the functions of +// declarations the symbol table dropped (chi has one: +// http2FancyWriter.Push is declared before its receiver type). Every unit is +// the image of at least one SSA function. +func TestJoin_EveryProjectFunctionMaps(t *testing.T) { + wantDropped := map[string][]string{ + "chi": {"(*github.com/go-chi/chi/v5/middleware.http2FancyWriter).Push"}, + } + for _, name := range []string{"dataflow", "chi", "generics", "rangefunc"} { + t.Run(name, func(t *testing.T) { + in := loadInput(t, name) + _, j, funcs := joined(t, in) + dropped := droppedDecls(in) + var lost []string + for fn := range funcs { + if _, kind := j.resolve(fn); kind == joinNone && fn.Synthetic != synthPackageInit { + if !inDropped(fn, dropped) { + t.Errorf("%s [%s] joins nothing and is not a dropped declaration", fn, fn.Synthetic) + } + lost = append(lost, fn.String()) + } + } + sort.Strings(lost) + if !slices.Equal(lost, wantDropped[name]) { + t.Errorf("unmapped functions = %v, want %v", lost, wantDropped[name]) + } + for _, b := range describeUnmapped(j, funcs) { + if strings.HasPrefix(b, "call ") { + t.Errorf("unmapped %s", b) + } + } + c := j.coverage(funcs) + if d := j.degraded(c); (d != nil) != (len(wantDropped[name]) > 0) { + t.Errorf("degraded = %+v", d) + } + hit := map[dataflow.L3Key]bool{} + for fn := range funcs { + if k, kind := j.resolve(fn); kind == joinBody { + hit[k] = true + } + } + for _, u := range in.Units { + if !hit[u.Key] { + t.Errorf("unit %v is the image of no SSA function", u.Key) + } + } + t.Logf("%s: %d funcs (%d by wrapper, %d unmapped, +%d package inits), %d positioned calls (%d unmapped, +%d synthetic), %d units", + name, c.Funcs, c.Wrappers, c.Unmapped, c.Inits, c.Calls, c.UnmappedCalls, c.Synthetic, len(in.Units)) + }) + } +} + +// fnByName returns the SSA function of package pkg named name (a member, or +// a method "T.M" of a named type, pointer receiver first). +func fnByName(t *testing.T, prog *ssa.Program, pkg, name string) *ssa.Function { + t.Helper() + p := prog.ImportedPackage(pkg) + if p == nil { + t.Fatalf("no SSA package %s", pkg) + } + if typ, meth, ok := strings.Cut(name, "."); ok { + T := p.Type(typ) + if T == nil { + t.Fatalf("%s has no type %s", pkg, typ) + } + for _, recv := range []bool{true, false} { + mset := prog.MethodSets.MethodSet(T.Type()) + if recv { + mset = prog.MethodSets.MethodSet(ptrTo(T.Type())) + } + if sel := mset.Lookup(p.Pkg, meth); sel != nil { + return prog.MethodValue(sel) + } + } + t.Fatalf("%s.%s has no method %s", pkg, typ, meth) + } + fn := p.Func(name) + if fn == nil { + t.Fatalf("%s has no func %s", pkg, name) + } + return fn +} + +// TestJoin_MultipleInitsAndLineDirectives (Review Focus 4): each init#N joins +// its own file's init by AST pointer, and the calls of each join that file's +// call keys; calls under a //line directive join by Lparen. +func TestJoin_MultipleInitsAndLineDirectives(t *testing.T) { + in := loadInput(t, "dataflow") + prog, j, _ := joined(t, in) + + const initSig = "example.com/dataflow/inits.init" + files := map[string]string{} // init#N → file + inits := prog.ImportedPackage("example.com/dataflow/inits") + for _, name := range []string{"init#1", "init#2"} { + fn := inits.Func(name) + if fn == nil { + t.Fatalf("no %s", name) + } + k, ok := j.fnUnit(fn) + if !ok || k.Signature != initSig { + t.Fatalf("%s joins %v, %v; want an %s unit", name, k, ok, initSig) + } + files[name] = k.Path + calls := 0 + forCalls(fn, func(c ssa.CallInstruction) { + if !c.Common().Pos().IsValid() { + return + } + u, key, ok := j.callSite(c) + if !ok || u != k { + t.Errorf("%s: call %s joins (%v, %q, %v), want its own unit %v", name, c, u, key, ok, k) + } + calls++ + }) + if calls != 2 { + t.Errorf("%s: %d positioned calls, want 2 (setup, os.Getenv)", name, calls) + } + } + if files["init#1"] == files["init#2"] { + t.Errorf("both inits join %s", files["init#1"]) + } + want := []string{filepath.Join("inits", "a.go"), filepath.Join("inits", "b.go")} + got := []string{files["init#1"], files["init#2"]} + sort.Strings(got) + if !slices.Equal(got, want) { + t.Errorf("init files = %v, want %v", got, want) + } + // The synthetic package initializer calls each init#N with no position. + if _, ok := j.fnUnit(inits.Func("init")); ok { + t.Error("the package initializer joins a callable") + } + + caller := fnByName(t, prog, "example.com/dataflow/linedir", "Caller") + k, ok := j.fnUnit(caller) + if !ok || k.Signature != "example.com/dataflow/linedir.Caller" { + t.Fatalf("linedir.Caller joins %v, %v", k, ok) + } + u := j.units[k] + wantKeys := map[string]bool{} + for _, c := range u.Calls { + wantKeys[c.Key] = true + } + gotKeys := map[string]bool{} + forCalls(caller, func(c ssa.CallInstruction) { + if _, key, ok := j.callSite(c); ok { + gotKeys[key] = true + } + }) + if len(wantKeys) != 3 || len(gotKeys) != len(wantKeys) { + t.Errorf("linedir.Caller call keys = %v, want every L1 key %v", gotKeys, wantKeys) + } + // The real line of os.Getenv(helper()) is 13: keys are real positions, + // and a virtual virtual.y:900 position would have joined nothing. + for key := range gotKeys { + if strings.HasPrefix(key, "900:") { + t.Errorf("call key %s is a //line virtual position", key) + } + } +} + +// TestJoin_ExternalTestPackage (Review Focus 5). With tests loaded, the +// external test package xtest_test and the test variant "xtest [xtest.test]" +// are SSA packages too; package user imports the plain xtest, so the program +// holds two SSA packages with path xtest, sharing a.go's syntax. Every +// project function of both still joins (variant-safe by AST pointer), the +// test files' functions join the test files' callables, and the plain and +// the variant Add join the same callable. With --skip-tests the test +// functions are absent and nothing dangles. +func TestJoin_ExternalTestPackage(t *testing.T) { + const xt = "example.com/dataflow/xtest" + aGo, testGo := filepath.Join("xtest", "a.go"), filepath.Join("xtest", "a_test.go") + for _, skip := range []bool{false, true} { + t.Run(map[bool]string{false: "tests", true: "skip-tests"}[skip], func(t *testing.T) { + in := loadInputOpts(t, "dataflow", skip) + prog, j, funcs := joined(t, in) + if bad := describeUnmapped(j, funcs); len(bad) > 0 { + t.Errorf("%d unmapped:\n %s", len(bad), strings.Join(bad, "\n ")) + } + + // Every SSA (*Counter).Add, of every variant: with tests, the + // test variant "xtest [xtest.test]" (a root, built from a.go's + // shared syntax) and the plain xtest that user imports (not a + // root: created from types, a bodiless stub). Both join the one + // callable: by syntax and by Object(). + wantAdd := dataflow.L3Key{Path: aGo, Signature: xt + ".Counter.Add"} + kinds := map[joinKind]int{} + for fn := range ssautil.AllFunctions(prog) { + if fn.Name() == "Add" && fn.Signature.Recv() != nil && homePath(fn) == xt && !strings.HasPrefix(fn.Synthetic, "wrapper") { + k, kind := j.resolve(fn) + if k != wantAdd { + t.Errorf("%s [%s] (pkg %p) joins %v, %v; want %v", fn, fn.Synthetic, fn.Pkg, k, kind, wantAdd) + } + kinds[kind]++ + } + } + if kinds[joinBody] != 1 { + t.Errorf("(*Counter).Add has %d SSA bodies, want 1 (kinds %v)", kinds[joinBody], kinds) + } + if !skip && kinds[joinWrapper] != 1 { + t.Errorf("tests: (*Counter).Add has %d types-only stubs, want 1 (kinds %v)", kinds[joinWrapper], kinds) + } + // user.Use calls the plain xtest's stubs; they still name the + // in-project callables, never an external. + use := fnByName(t, prog, xt+"/user", "Use") + forCalls(use, func(c ssa.CallInstruction) { + callee := c.Common().StaticCallee() + if callee == nil { + return + } + if _, ok := j.fnUnit(callee); !ok { + t.Errorf("user.Use callee %s [%s] joins nothing", callee, callee.Synthetic) + } + if sig, ok := j.externalOf(callee); ok { + t.Errorf("user.Use callee %s is external %s", callee, sig) + } + }) + + var testFns []dataflow.L3Key + for fn := range funcs { + if k, kind := j.resolve(fn); kind == joinBody && k.Path == testGo { + testFns = append(testFns, k) + } + } + sortKeys(testFns) + testFns = slices.Compact(testFns) + if skip { + if len(testFns) != 0 { + t.Errorf("test callables joined with --skip-tests: %v", testFns) + } + if p := prog.ImportedPackage(xt + "_test"); p != nil { + t.Errorf("xtest_test has an SSA package with --skip-tests") + } + return + } + want := []dataflow.L3Key{ + {Path: testGo, Signature: xt + "_test.TestAdd"}, + {Path: testGo, Signature: xt + "_test.TestAdd.closure_1"}, + {Path: testGo, Signature: xt + "_test.helper"}, + } + if !slices.Equal(testFns, want) { + t.Errorf("test callables joined = %v, want %v", testFns, want) + } + // helper's c.Add(2) joins its call key in a_test.go. + helper := fnByName(t, prog, xt+"_test", "helper") + n := 0 + forCalls(helper, func(c ssa.CallInstruction) { + if u, key, ok := j.callSite(c); ok && u == want[2] && key != "" { + n++ + } + }) + if n != 1 { + t.Errorf("helper: %d calls joined, want 1", n) + } + }) + } +} + +// TestJoin_WrappersUnwrap: each SSA wrapper kind (§4.8 unwrap list) joins +// the method it wraps: $bound (method value), $thunk (method expression), +// the pointer wrapper of a value method, and a promoted method of an +// embedding struct. Their calls are synthetic: none joins a call node. +func TestJoin_WrappersUnwrap(t *testing.T) { + in := loadInput(t, "dataflow") + prog, j, funcs := joined(t, in) + const xt = "example.com/dataflow/xtest" + add := dataflow.L3Key{Path: filepath.Join("xtest", "a.go"), Signature: xt + ".Counter.Add"} + get := dataflow.L3Key{Path: filepath.Join("xtest", "a.go"), Signature: xt + ".Counter.Get"} + + kinds := map[string]dataflow.L3Key{} + for fn := range funcs { + if homePath(fn) != xt || fn.Synthetic == "" { + continue + } + // Keyed by fn.String(): an interface method's wrappers share the + // concrete method's short names (Add$bound, Add$thunk). + if strings.HasPrefix(fn.Synthetic, "bound method wrapper") || strings.HasPrefix(fn.Synthetic, "thunk") { + kinds[fn.String()] = mustJoin(t, j, fn) + } + if fn.Syntax() == nil && fn.Synthetic != synthPackageInit { + if _, kind := j.resolve(fn); kind == joinWrapper { + forCalls(fn, func(c ssa.CallInstruction) { + if _, _, ok := j.callSite(c); ok { + t.Errorf("wrapper %s: call %s joined a call node", fn, c) + } + }) + } + } + } + for _, name := range []string{"(*" + xt + ".Counter).Add$bound", "(*" + xt + ".Counter).Add$thunk"} { + if k, ok := kinds[name]; !ok || k != add { + t.Errorf("%s joins %v (seen %v), want %v (wrappers seen: %v)", name, k, ok, add, kinds) + } + } + iface := dataflow.L3Key{Path: filepath.Join("xtest", "a.go"), Signature: xt + ".Adder.Add"} + var ifaceWrappers int + for fn := range funcs { + if obj := fn.Object(); obj != nil && fn.Syntax() == nil && obj.Pkg() != nil && obj.Pkg().Path() == xt && obj.Name() == "Add" && types.IsInterface(fn.Object().Type().(*types.Signature).Recv().Type()) { + if k := mustJoin(t, j, fn); k != iface { + t.Errorf("%s [%s] joins %v, want the interface method %v", fn, fn.Synthetic, k, iface) + } + ifaceWrappers++ + } + } + if ifaceWrappers != 2 { + t.Errorf("%d wrappers of the interface method Adder.Add, want 2 ($bound, $thunk)", ifaceWrappers) + } + if k := mustJoin(t, j, fnByName(t, prog, xt, "Counter.Get")); k != get { + t.Errorf("(*Counter).Get (pointer wrapper) joins %v, want %v", k, get) + } + if k := mustJoin(t, j, fnByName(t, prog, xt, "Outer.Add")); k != add { + t.Errorf("Outer.Add (promoted) joins %v, want %v", k, add) + } +} + +func mustJoin(t *testing.T, j *join, fn *ssa.Function) dataflow.L3Key { + t.Helper() + k, ok := j.fnUnit(fn) + if !ok { + t.Errorf("%s [%s] joins nothing", fn, fn.Synthetic) + } + return k +} + +// TestJoin_GenericInstancesJoinOrigin: with InstantiateGenerics, every +// instance of a generic function or method joins its one origin callable, +// and its calls join the origin's call keys. +func TestJoin_GenericInstancesJoinOrigin(t *testing.T) { + in := loadInput(t, "generics") + _, j, funcs := joined(t, in) + byOrigin := map[dataflow.L3Key]int{} + for fn := range funcs { + if !strings.HasPrefix(fn.Synthetic, "instance of") { + continue + } + k, kind := j.resolve(fn) + if kind != joinBody { + t.Errorf("instance %s joins %v, %v", fn, k, kind) + continue + } + byOrigin[k]++ + } + add := dataflow.L3Key{Path: filepath.Join("set", "set.go"), Signature: "example.com/generics/set.Set.Add"} + if byOrigin[add] == 0 { + t.Errorf("no instance of Set.Add joins %v (instances: %v)", add, byOrigin) + } + + // Twice[T] calls Ident[T] through an instantiation wrapper (its type + // arguments are parameterized): it joins Ident as a wrapper. + in = loadInput(t, "dataflow") + _, j, funcs = joined(t, in) + ident := dataflow.L3Key{Path: filepath.Join("generic", "b.go"), Signature: "example.com/dataflow/generic.Ident"} + wrappers := 0 + for fn := range funcs { + if strings.HasPrefix(fn.Synthetic, synthInstWrapper) { + k, kind := j.resolve(fn) + if k != ident || kind != joinWrapper { + t.Errorf("%s joins %v, %v; want %v as a wrapper", fn, k, kind, ident) + } + wrappers++ + } + } + if wrappers == 0 { + t.Error("no instantiation wrapper in testdata/dataflow/generic") + } +} + +// TestJoin_RangeFuncYieldFolds: a range-over-func yield function folds into +// its enclosing callable, and the call in the loop body joins that +// callable's call key. +func TestJoin_RangeFuncYieldFolds(t *testing.T) { + in := loadInput(t, "rangefunc") + prog, j, _ := joined(t, in) + outer := fnByName(t, prog, "example.com/rangefunc", "RangeFuncCall") + want, ok := j.fnUnit(outer) + if !ok { + t.Fatal("RangeFuncCall joins nothing") + } + var yield *ssa.Function + for _, a := range outer.AnonFuncs { + if a.Synthetic == synthYield { + yield = a + } + } + if yield == nil { + t.Fatal("RangeFuncCall has no yield function") + } + if k, ok := j.fnUnit(yield); !ok || k != want { + t.Errorf("yield joins %v, %v; want %v", k, ok, want) + } + n := 0 + forCalls(yield, func(c ssa.CallInstruction) { + if u, _, ok := j.callSite(c); ok && u == want { + n++ + } + }) + if n != 1 { + t.Errorf("yield: %d calls joined (sink(v)), want 1", n) + } +} + +// TestJoin_UnmappedDegrades: a function literal in a package-level variable +// initializer has no callable, so its SSA function (a child of the package +// initializer) is unmapped_ssa, counted, with its package listed. +func TestJoin_UnmappedDegrades(t *testing.T) { + in := loadInput(t, "unmapped") + _, j, funcs := joined(t, in) + c := j.coverage(funcs) + if c.Unmapped != 1 { + t.Errorf("coverage = %+v, want 1 unmapped function", c) + } + d := j.degraded(c) + if d == nil || d.Code != CodeUnmappedSSA || !slices.Equal(d.Packages, []string{"example.com/unmapped"}) { + t.Fatalf("degraded = %+v, want unmapped_ssa [example.com/unmapped]", d) + } + if strings.Contains(d.Detail, "#29") { + t.Errorf("detail %q names #29 for a package with no cgo", d.Detail) + } + res := Run(in) + if !hasCode(res.Degraded, CodeUnmappedSSA) { + t.Errorf("Run degraded = %+v, want unmapped_ssa", res.Degraded) + } +} + +// TestJoin_CgoNamesIssue29: a cgo package's functions have no callable +// (#29), so they are unmapped_ssa and the detail names the issue. +func TestJoin_CgoNamesIssue29(t *testing.T) { + in := loadInput(t, "cgo") + res := Run(in) + var d *Degraded + for i := range res.Degraded { + if res.Degraded[i].Code == CodeUnmappedSSA { + d = &res.Degraded[i] + } + } + if d == nil { + t.Skipf("no unmapped_ssa (cgo disabled?): %+v", res.Degraded) + } + if !strings.Contains(d.Detail, "#29") || !slices.Equal(d.Packages, []string{"example.com/cgo/c"}) { + t.Errorf("unmapped_ssa = %+v, want package example.com/cgo/c and a #29 note", d) + } +} + +func hasCode(deg []Degraded, code string) bool { + for _, d := range deg { + if d.Code == code { + return true + } + } + return false +} + +// TestSDG_L3SkippedDegrades (A16): a unit with no L3 result is listed by key +// under l3_skipped. +func TestSDG_L3SkippedDegrades(t *testing.T) { + in := loadInput(t, "dataflow") + if hasCode(Run(in).Degraded, CodeL3Skipped) { + t.Fatal("clean fixture has l3_skipped") + } + drop := []dataflow.L3Key{in.Units[3].Key, in.Units[0].Key} + l3 := map[dataflow.L3Key]dataflow.Result{} + for k, r := range in.L3 { + l3[k] = r + } + for _, k := range drop { + delete(l3, k) + } + in.L3 = l3 + res := Run(in) + var got *Degraded + for i := range res.Degraded { + if res.Degraded[i].Code == CodeL3Skipped { + got = &res.Degraded[i] + } + } + if got == nil { + t.Fatalf("degraded = %+v, want l3_skipped", res.Degraded) + } + want := []dataflow.L3Key{in.Units[0].Key, in.Units[3].Key} // units are sorted by key + if !slices.Equal(got.Callables, want) { + t.Errorf("l3_skipped callables = %v, want %v", got.Callables, want) + } + // A skipped unit still joins: its calls fall back to its L1 call keys. + if hasCode(res.Degraded, CodeUnmappedSSA) { + t.Errorf("dropping L3 results unmapped SSA: %+v", res.Degraded) + } +} + +// TestSDG_IllTypedPackageDegrades (Review Focus 1): testdata/typeerr does not +// type-check, so it gets no SSA: partial_ssa lists it (then no_ssa), nothing +// panics, and its L3 results are those of -a 3. +func TestSDG_IllTypedPackageDegrades(t *testing.T) { + in := loadInput(t, "typeerr") + plain := dataflow.Run(in.Fset, in.Units, 3, 1) + res := Run(in) + codes := codesOf(res.Degraded) + if !slices.Equal(codes, []string{CodePartialSSA, CodeNoSSA}) { + t.Fatalf("codes = %v, want [partial_ssa no_ssa]", codes) + } + if !slices.Equal(res.Degraded[0].Packages, []string{"example.com/typeerr"}) { + t.Errorf("partial_ssa packages = %v", res.Degraded[0].Packages) + } + if len(in.L3) == 0 || len(in.L3) != len(plain) { + t.Fatalf("L3 has %d results, -a 3 has %d", len(in.L3), len(plain)) + } + for k, r := range plain { + got := in.L3[k] + if !slices.Equal(got.Cfg, r.Cfg) || !slices.Equal(got.Cdg, r.Cdg) || !slices.Equal(got.Ddg, r.Ddg) || len(got.Nodes) != len(r.Nodes) { + t.Errorf("%v: L3 differs from -a 3", k) + } + } +} + +func codesOf(deg []Degraded) []string { + out := make([]string, len(deg)) + for i, d := range deg { + out[i] = d.Code + } + return out +} + +// TestSDG_RangeFuncDegrades (D8): a program that ranges over a function is +// degraded range_func, listing the callables (by key) and their package; +// the join itself is complete. +func TestSDG_RangeFuncDegrades(t *testing.T) { + in := loadInput(t, "rangefunc") + res := Run(in) + if codes := codesOf(res.Degraded); !slices.Equal(codes, []string{CodeRangeFunc}) { + t.Fatalf("codes = %v, want [range_func]", codes) + } + d := res.Degraded[0] + want := []dataflow.L3Key{ + {Path: "r.go", Signature: "example.com/rangefunc.RangeFunc"}, + {Path: "r.go", Signature: "example.com/rangefunc.RangeFuncCall"}, + } + if !slices.Equal(d.Callables, want) || !slices.Equal(d.Packages, []string{"example.com/rangefunc"}) { + t.Errorf("range_func = %+v, want callables %v in example.com/rangefunc", d, want) + } + if hasCode(Run(loadInput(t, "dataflow")).Degraded, CodeRangeFunc) { + t.Error("dataflow (go 1.21, no range-over-func) is range_func") + } +} + +// TestJoin_Deterministic: two independent loads and SSA builds give the same +// join of every project function, compared by (function string, key) — the +// join must not depend on SSA build or map order. +func TestJoin_Deterministic(t *testing.T) { + render := func() []string { + in := loadInput(t, "chi") + _, j, funcs := joined(t, in) + var out []string + for fn := range funcs { + k, kind := j.resolve(fn) + line := fn.String() + " → " + k.Path + " " + k.Signature + " " + string(rune('0'+kind)) + forCalls(fn, func(c ssa.CallInstruction) { + if _, key, ok := j.callSite(c); ok { + line += " " + key + } + }) + out = append(out, line) + } + sort.Strings(out) + return out + } + if a, b := render(), render(); !slices.Equal(a, b) { + t.Error("two runs join differently") + } +} + +// TestJoin_ExternalOf: a library callee's signature is L2's: the call of +// os.Getenv in inits is os.Getenv, (*strings.Builder) methods are +// strings.Builder., and a project function is not external. +func TestJoin_ExternalOf(t *testing.T) { + in := loadInput(t, "dataflow") + prog, j, _ := joined(t, in) + init1 := prog.ImportedPackage("example.com/dataflow/inits").Func("init#1") + var got []string + forCalls(init1, func(c ssa.CallInstruction) { + if callee := c.Common().StaticCallee(); callee != nil { + if sig, ok := j.externalOf(callee); ok { + got = append(got, sig) + } else if _, ok := j.fnUnit(callee); !ok { + t.Errorf("callee %s is neither external nor in-project", callee) + } + } + }) + if !slices.Equal(got, []string{"os.Getenv"}) { + t.Errorf("externals of init#1 = %v, want [os.Getenv]", got) + } + if fn := prog.ImportedPackage("example.com/vlib"); fn != nil { + if sig, ok := j.externalOf(fn.Func("F")); !ok || sig != "example.com/vlib.F" { + t.Errorf("vendored vlib.F: externalOf = %q, %v; want example.com/vlib.F", sig, ok) + } + } + if _, ok := j.externalOf(fnByName(t, prog, "example.com/dataflow/flow", "IfElse")); ok { + t.Error("flow.IfElse is external") + } +} + +func ptrTo(t types.Type) types.Type { return types.NewPointer(t) } diff --git a/internal/sdg/run.go b/internal/sdg/run.go index 8501141..8b7172a 100644 --- a/internal/sdg/run.go +++ b/internal/sdg/run.go @@ -15,40 +15,77 @@ import ( var vtaCallGraph = vta.CallGraph // Run builds the L4 overlay of in. For now it builds SSA over the project -// packages and runs VTA once over the project function set, and returns only -// the degradation: no vertices or edges yet. +// packages, joins it to the L3 units, and runs VTA once over the project +// function set, and returns only the degradation: no vertices or edges yet. +// +// Order: the join is computed before VTA, so it never reads a program a VTA +// panic may have wedged; after no_vta, later stages use the program for the +// join alone. The degradations are, in order: partial_ssa / no_ssa (SSA), +// l3_skipped and range_func (from the units, whether or not SSA built), +// unmapped_ssa (the join), no_vta. func Run(in Input) Result { start := time.Now() prog, pkgs, deg := buildSSA(in) utils.Info("L4: SSA built %d project packages in %s", len(pkgs), time.Since(start).Round(time.Millisecond)) + for _, d := range []*Degraded{l3Skipped(in), rangeFunc(in)} { + if d != nil { + deg = append(deg, *d) + } + } if len(pkgs) == 0 { return Result{Degraded: deg} } + // The project function set (VTA's seeds) is built under recover: it + // builds wrappers on demand, and a panic there may wedge the program, + // so neither the join nor VTA then reads it. + start = time.Now() + funcs, err := funcSet(prog, pkgs) + if err != nil { + utils.Debug("L4: VTA function set failed: %v", err) + return Result{Degraded: append(deg, Degraded{Code: CodeNoVTA, Detail: err.Error()})} + } + j := newJoin(in) + cov := j.coverage(funcs) + utils.Info("L4: SSA join: %d functions (%d by wrapper), %d calls; %d functions, %d calls unmapped in %s", + cov.Funcs, cov.Wrappers, cov.Calls, cov.Unmapped, cov.UnmappedCalls, time.Since(start).Round(time.Millisecond)) + if d := j.degraded(cov); d != nil { + deg = append(deg, *d) + } + start = time.Now() - cg, nfuncs, err := runVTA(prog, pkgs) + cg, err := runVTA(funcs) if err != nil { utils.Debug("L4: VTA failed: %v", err) deg = append(deg, Degraded{Code: CodeNoVTA, Detail: err.Error()}) } else { - utils.Info("L4: VTA over %d functions: %d edges in %s", nfuncs, edgeCount(cg), time.Since(start).Round(time.Millisecond)) + utils.Info("L4: VTA over %d functions: %d edges in %s", len(funcs), edgeCount(cg), time.Since(start).Round(time.Millisecond)) } return Result{Degraded: deg} } -// runVTA builds the project function set and runs VTA over it with no -// initial graph (VTA then uses its internal lazy CHA over that set), turning -// a panic in either into an error. Both stages build functions on demand -// (projectFuncs' MethodValue wrappers, VTA's method lookups), so both sit -// under the recover. -func runVTA(prog *ssa.Program, pkgs []*ssa.Package) (cg *callgraph.Graph, nfuncs int, err error) { +// funcSet is projectFuncs under recover: it builds functions on demand +// (MethodValue wrappers), so a builder panic is turned into an error. +func funcSet(prog *ssa.Program, pkgs []*ssa.Package) (funcs map[*ssa.Function]bool, err error) { + defer func() { + if r := recover(); r != nil { + funcs, err = nil, fmt.Errorf("panic: %v", r) + } + }() + return projectFuncs(prog, pkgs), nil +} + +// runVTA runs VTA over funcs with no initial graph (VTA then uses its +// internal lazy CHA over that set), turning a panic into an error. VTA builds +// functions on demand (method lookups), so it may wedge the program: after +// an error, later stages must not read the program beyond the join. +func runVTA(funcs map[*ssa.Function]bool) (cg *callgraph.Graph, err error) { defer func() { if r := recover(); r != nil { cg, err = nil, fmt.Errorf("panic: %v", r) } }() - funcs := projectFuncs(prog, pkgs) - return vtaCallGraph(funcs, nil), len(funcs), nil + return vtaCallGraph(funcs, nil), nil } func edgeCount(cg *callgraph.Graph) int { diff --git a/internal/sdg/ssa_test.go b/internal/sdg/ssa_test.go index 4dfb34e..eee6a16 100644 --- a/internal/sdg/ssa_test.go +++ b/internal/sdg/ssa_test.go @@ -13,13 +13,21 @@ import ( "golang.org/x/tools/go/ssa" "golang.org/x/tools/go/ssa/ssautil" + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/utils" ) -// loadInput loads testdata/ the way the analyzer does (symbol-table -// builder, tests skipped) and returns the sdg input over it. +// loadInput loads testdata/ the way the analyzer does at -a 4 +// (symbol-table builder, tests skipped, L3 with side tables) and returns the +// sdg input over it. func loadInput(t *testing.T, name string) Input { + t.Helper() + return loadInputOpts(t, name, true) +} + +// loadInputOpts is loadInput with the --skip-tests choice. +func loadInputOpts(t *testing.T, name string, skipTests bool) Input { t.Helper() _, thisFile, _, _ := runtime.Caller(0) dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", name)) @@ -27,26 +35,15 @@ func loadInput(t *testing.T, name string) Input { t.Fatal(err) } b := syntactic_analysis.NewSymbolTableBuilder(dir) - st, err := b.Build(nil, true) + st, err := b.Build(nil, skipTests) if err != nil { t.Fatalf("symbol table build failed: %v", err) } - return Input{Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, K: 3, Jobs: 1} -} - -// homePath is the import path of the package fn's code comes from: its own -// package, its generic origin's, or (for a wrapper) the wrapped object's. -func homePath(fn *ssa.Function) string { - if fn.Pkg != nil { - return fn.Pkg.Pkg.Path() - } - if o := fn.Origin(); o != nil && o.Pkg != nil { - return o.Pkg.Pkg.Path() - } - if obj := fn.Object(); obj != nil && obj.Pkg() != nil { - return obj.Pkg().Path() + units := b.L3Units(st) + return Input{ + Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Units: units, L3: dataflow.RunWithSideTables(b.Fset(), units, 3, 1), K: 3, Jobs: 1, } - return "" } // TestSSA_BuildsProjectPackagesOnly: bodies are built for the project's @@ -65,7 +62,7 @@ func TestSSA_BuildsProjectPackagesOnly(t *testing.T) { want := []string{ "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/inits", "example.com/dataflow/l1", "example.com/dataflow/linedir", "example.com/dataflow/vend", - "example.com/dataflow/xfile", + "example.com/dataflow/xfile", "example.com/dataflow/xtest", "example.com/dataflow/xtest/user", } if !slices.Equal(built, want) { t.Fatalf("built packages = %v, want %v", built, want) @@ -136,8 +133,8 @@ func TestSSA_PanicDegradesPackage(t *testing.T) { t.Error("the panicked package is still returned as built") } } - if len(pkgs) != 6 { - t.Errorf("built %d packages, want the other 6", len(pkgs)) + if len(pkgs) != 8 { + t.Errorf("built %d packages, want the other 8", len(pkgs)) } if fn := prog.ImportedPackage("example.com/dataflow/l1"); fn == nil || len(fn.Members) == 0 { t.Error("l1 was not built after flow panicked") @@ -155,8 +152,8 @@ func TestSSA_PanicDegradesPackage(t *testing.T) { if !slices.Equal(codes, []string{"partial_ssa", "no_ssa"}) { t.Fatalf("codes = %v, want [partial_ssa no_ssa]", codes) } - if len(deg[0].Packages) != 7 { - t.Errorf("partial_ssa lists %v, want all 7 packages", deg[0].Packages) + if len(deg[0].Packages) != 9 { + t.Errorf("partial_ssa lists %v, want all 9 packages", deg[0].Packages) } if res := Run(in); len(res.Degraded) != 2 || res.Degraded[1].Code != "no_ssa" { t.Errorf("Run degraded = %+v, want partial_ssa + no_ssa and no VTA", res.Degraded) diff --git a/internal/semantic_analysis/call_graph.go b/internal/semantic_analysis/call_graph.go index 6696cfd..73343ba 100644 --- a/internal/semantic_analysis/call_graph.go +++ b/internal/semantic_analysis/call_graph.go @@ -2,7 +2,7 @@ // // This stage uses the same golang.org/x/tools/go/packages load that built the // symbol table. For each recorded call site it resolves the callee to a -// *types.Func, derives its signature via signatureOf(), backfills +// *types.Func, derives its signature via SignatureOf(), backfills // callee_signature in place, and emits an identity-only GoCallEdge. // // Precision choice: declared-type dispatch (CHA-style). Pointer-receiver @@ -23,6 +23,7 @@ import ( "golang.org/x/tools/go/packages" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/utils" ) @@ -148,7 +149,7 @@ func (cg *CallGraphBuilder) resolveCallable( utils.Debug("unresolved call site: %s in %s", site.MethodName, callable.Signature) continue } - calleeSig := calleeSignatureOf(fn) + calleeSig := CalleeSignatureOf(fn) // Backfill the site whether the callee is in-project or a library. site.CalleeSignature = &calleeSig @@ -360,38 +361,12 @@ func (cg *CallGraphBuilder) resolveCallExpr(pkg *packages.Package, call *ast.Cal return nil } -// calleeSignatureOf is the call-site mirror of signatureOf — same canonicalization. -// Must produce byte-identical strings to signatureOf() in syntactic_analysis. -func calleeSignatureOf(obj types.Object) string { - if obj == nil { - return "" - } - pkg := obj.Pkg() - pkgPath := "" - if pkg != nil { - pkgPath = pkg.Path() - } - - switch o := obj.(type) { - case *types.Func: - sig := o.Type().(*types.Signature) - recv := sig.Recv() - if recv != nil { - recvType := recv.Type() - if ptr, ok := recvType.(*types.Pointer); ok { - recvType = ptr.Elem() - } - if named, ok := recvType.(*types.Named); ok { - typeName := named.Obj().Name() - return pkgPath + "." + typeName + "." + o.Name() - } - } - return pkgPath + "." + o.Name() - case *types.TypeName: - return pkgPath + "." + o.Name() - default: - return pkgPath + "." + obj.Name() - } +// CalleeSignatureOf is the signature of a resolved callee, the call-site side +// of the one canonicalizer (syntactic_analysis.SignatureOf), so caller-side +// and callee-side ids are identical by construction. The L4 SSA join uses it +// for external callees (ruling R-P5). +func CalleeSignatureOf(obj types.Object) string { + return syntactic_analysis.SignatureOf(obj) } // packageForFile returns the *packages.Package that contains the given relative file path. diff --git a/internal/syntactic_analysis/signature.go b/internal/syntactic_analysis/signature.go index cdfd71e..352f48c 100644 --- a/internal/syntactic_analysis/signature.go +++ b/internal/syntactic_analysis/signature.go @@ -6,17 +6,19 @@ import ( "go/types" ) -// signatureOf is the single canonicalizer for all signature strings in the analyzer. +// SignatureOf is the single canonicalizer for all signature strings in the analyzer. // It produces the edge id used in GoCallable.signature, GoType.signature, and every // call-graph edge source/target. All callers must use this function — never build // signatures ad-hoc. Caller-side and callee-side ids are then guaranteed identical. +// The call-graph builder (semantic_analysis.CalleeSignatureOf) and the L4 SSA join +// (internal/sdg) reach it through that one exported entry point. // // Format: // - Package function: "pkg/path.FuncName" // - Method: "pkg/path.TypeName.MethodName" // - Interface method: "pkg/path.InterfaceName.MethodName" // - Type: "pkg/path.TypeName" -func signatureOf(obj types.Object) string { +func SignatureOf(obj types.Object) string { if obj == nil { return "" } @@ -48,4 +50,3 @@ func signatureOf(obj types.Object) string { return fmt.Sprintf("%s.%s", pkgPath, obj.Name()) } } - diff --git a/internal/syntactic_analysis/symbol_table.go b/internal/syntactic_analysis/symbol_table.go index f0b9c18..becf00a 100644 --- a/internal/syntactic_analysis/symbol_table.go +++ b/internal/syntactic_analysis/symbol_table.go @@ -366,7 +366,7 @@ func (b *SymbolTableBuilder) buildType( var typSig string if pkg.TypesInfo != nil { if obj, ok := pkg.TypesInfo.Defs[spec.Name]; ok && obj != nil { - typSig = signatureOf(obj) + typSig = SignatureOf(obj) } } if typSig == "" { @@ -606,12 +606,12 @@ func (b *SymbolTableBuilder) buildCallable( return callable } -// declSignature is a FuncDecl's callable signature: signatureOf its object, +// declSignature is a FuncDecl's callable signature: SignatureOf its object, // or, without type info, [.].. func (b *SymbolTableBuilder) declSignature(pkg *packages.Package, decl *ast.FuncDecl) string { if pkg.TypesInfo != nil { if obj, ok := pkg.TypesInfo.Defs[decl.Name]; ok && obj != nil { - if sig := signatureOf(obj); sig != "" { + if sig := SignatureOf(obj); sig != "" { return sig } } diff --git a/testdata/cgo/c/c.go b/testdata/cgo/c/c.go new file mode 100644 index 0000000..91956f7 --- /dev/null +++ b/testdata/cgo/c/c.go @@ -0,0 +1,10 @@ +// Package c has a cgo file. go/packages parses the cgo-generated copy of it +// (in the build cache) instead, so its functions have no symbol-table +// callable (#29) and their SSA functions are unmapped_ssa. +package c + +// int twice(int x) { return 2 * x; } +import "C" + +// Twice calls C. +func Twice(x int) int { return int(C.twice(C.int(x))) } diff --git a/testdata/cgo/c/d.go b/testdata/cgo/c/d.go new file mode 100644 index 0000000..2e90fca --- /dev/null +++ b/testdata/cgo/c/d.go @@ -0,0 +1,4 @@ +package c + +// Plain is a plain Go function in the cgo package. +func Plain() int { return 1 } diff --git a/testdata/cgo/go.mod b/testdata/cgo/go.mod new file mode 100644 index 0000000..585ff94 --- /dev/null +++ b/testdata/cgo/go.mod @@ -0,0 +1,3 @@ +module example.com/cgo + +go 1.21 diff --git a/testdata/dataflow/generic/b.go b/testdata/dataflow/generic/b.go index 713f17b..6bbafac 100644 --- a/testdata/dataflow/generic/b.go +++ b/testdata/dataflow/generic/b.go @@ -7,3 +7,16 @@ type B struct{ s string } var pb atomic.Pointer[B] func StoreB() { pb.Store(nil) } + +// Ident is generic; Twice passes it its own type parameter, so SSA builds +// Twice's call through an instantiation wrapper of Ident (the type +// arguments are parameterized), which joins Ident via Origin(). +func Ident[T any](v T) T { return v } + +func Twice[T any](v T) T { + f := Ident[T] + return f(v) +} + +// UseTwice instantiates Twice. +func UseTwice() int { return Twice(1) } diff --git a/testdata/dataflow/xtest/a.go b/testdata/dataflow/xtest/a.go new file mode 100644 index 0000000..435d601 --- /dev/null +++ b/testdata/dataflow/xtest/a.go @@ -0,0 +1,45 @@ +// Package xtest is called from an external test package (a_test.go, package +// xtest_test) and from an in-package test (b_test.go). With +// --skip-tests=false the loader also yields the test variant +// "xtest [xtest.test]"; package user imports the plain xtest, so SSA holds +// both. The join must map each variant's functions to its one callable. +// +// It also holds one function per SSA wrapper kind the join unwraps: a method +// value ($bound), a method expression ($thunk), a promoted method and a +// pointer wrapper of a value method. +package xtest + +// Counter counts. +type Counter struct{ n int } + +// Add adds d through a closure. +func (c *Counter) Add(d int) int { + inc := func() { c.n += d } + inc() + return c.n +} + +// Get is a value method: (*Counter).Get is its pointer wrapper. +func (c Counter) Get() int { return c.n } + +// Outer promotes Counter's methods. +type Outer struct{ *Counter } + +// New returns a counter. +func New() *Counter { return &Counter{} } + +// Bound returns a method value: SSA makes the closure (*Counter).Add$bound. +func Bound(c *Counter) func(int) int { return c.Add } + +// Thunk returns a method expression: SSA makes (*Counter).Add$thunk. +func Thunk() func(*Counter, int) int { return (*Counter).Add } + +// Adder is an interface: a method value of it is a $bound wrapper of the +// interface method, and a method expression a $thunk of it. +type Adder interface{ Add(int) int } + +// IfaceBound returns a method value of an interface. +func IfaceBound(a Adder) func(int) int { return a.Add } + +// IfaceThunk returns a method expression of an interface. +func IfaceThunk() func(Adder, int) int { return Adder.Add } diff --git a/testdata/dataflow/xtest/a_test.go b/testdata/dataflow/xtest/a_test.go new file mode 100644 index 0000000..385d24c --- /dev/null +++ b/testdata/dataflow/xtest/a_test.go @@ -0,0 +1,19 @@ +package xtest_test + +import ( + "testing" + + "example.com/dataflow/xtest" +) + +func helper(c *xtest.Counter) int { return c.Add(2) } + +func TestAdd(t *testing.T) { + c := xtest.New() + check := func(n int) { + if n != 2 { + t.Fatal(n) + } + } + check(helper(c)) +} diff --git a/testdata/dataflow/xtest/b_test.go b/testdata/dataflow/xtest/b_test.go new file mode 100644 index 0000000..118cf73 --- /dev/null +++ b/testdata/dataflow/xtest/b_test.go @@ -0,0 +1,12 @@ +package xtest + +import "testing" + +// TestNew is an in-package test: it makes the loader yield the variant +// "example.com/dataflow/xtest [example.com/dataflow/xtest.test]", which has +// the plain package's path and shares its parsed a.go. +func TestNew(t *testing.T) { + if New().Add(1) != 1 { + t.Fatal("Add") + } +} diff --git a/testdata/dataflow/xtest/user/user.go b/testdata/dataflow/xtest/user/user.go new file mode 100644 index 0000000..e68f75c --- /dev/null +++ b/testdata/dataflow/xtest/user/user.go @@ -0,0 +1,9 @@ +// Package user imports xtest from non-test code. With --skip-tests=false the +// loader gives it the plain xtest while xtest_test imports the test variant +// "xtest [xtest.test]": SSA then holds two packages for one import path. +package user + +import "example.com/dataflow/xtest" + +// Use calls into xtest. +func Use() int { return xtest.New().Add(3) } diff --git a/testdata/rangefunc/r.go b/testdata/rangefunc/r.go index 8b5993f..eedc0b5 100644 --- a/testdata/rangefunc/r.go +++ b/testdata/rangefunc/r.go @@ -21,3 +21,14 @@ func RangeFunc(seq func(func(int) bool)) int { } return s } + +func sink(int) {} + +// RangeFuncCall calls sink in the body of a range over an iterator: in SSA +// the call sits in the synthetic yield function, which folds into +// RangeFuncCall. +func RangeFuncCall(seq func(func(int) bool)) { + for v := range seq { + sink(v) + } +} diff --git a/testdata/unmapped/go.mod b/testdata/unmapped/go.mod new file mode 100644 index 0000000..f9322a7 --- /dev/null +++ b/testdata/unmapped/go.mod @@ -0,0 +1,3 @@ +module example.com/unmapped + +go 1.21 diff --git a/testdata/unmapped/u.go b/testdata/unmapped/u.go new file mode 100644 index 0000000..8a8602c --- /dev/null +++ b/testdata/unmapped/u.go @@ -0,0 +1,13 @@ +// Package unmapped holds a function literal in a package-level variable +// initializer. The symbol table has no callable for it (L1 records callables +// of declarations only), so its SSA function, a child of the synthetic +// package initializer, maps to nothing: unmapped_ssa. +package unmapped + +func helper() int { return 1 } + +// Handler is a package-level function literal. +var Handler = func() int { return helper() } + +// Use is a declared function: it maps. +func Use() int { return Handler() } From 6f56ed882392da9bff6cf7a21f77cb155a915587 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 01:00:32 -0400 Subject: [PATCH 11/65] fix(l4): fnUnit joins only units; interface methods have their own accessor --- internal/sdg/join.go | 117 +++++++++++++++++++++++++++++------ internal/sdg/join_test.go | 65 +++++++++++++++++-- internal/sdg/run.go | 4 +- internal/sdg/ssa_test.go | 10 +-- testdata/dataflow/asm/asm.go | 10 +++ testdata/dataflow/asm/asm.s | 2 + 6 files changed, 178 insertions(+), 30 deletions(-) create mode 100644 testdata/dataflow/asm/asm.go create mode 100644 testdata/dataflow/asm/asm.s diff --git a/internal/sdg/join.go b/internal/sdg/join.go index 91ffc06..1051321 100644 --- a/internal/sdg/join.go +++ b/internal/sdg/join.go @@ -28,9 +28,11 @@ const ( type joinKind int const ( - joinNone joinKind = iota - joinBody // its code is the callable's body: by syntax, origin, or a folded yield - joinWrapper // a synthetic wrapper (or a bodiless stub) of the callable: by Object() + joinNone joinKind = iota + joinBody // its code is a unit's body: by syntax, origin, or a folded yield + joinWrapper // a synthetic wrapper (or a types-only stub) of a unit: by Object() + joinIface // an interface method ($bound / $thunk of it): a symbol-table key, no unit + joinBodiless // a declared function with no Go body (assembly, linkname), or a wrapper of one: a symbol-table key, no unit ) // join maps go/ssa back to the L3 units (spec §4.8, SSA → L3 join). It keys @@ -43,7 +45,8 @@ type join struct { fset *token.FileSet projectDir string byFn map[ast.Node]dataflow.L3Key // Unit.Fn → unit - bySig map[dataflow.L3Key]bool // (declaring file, signature) of every unit and interface method + iface map[dataflow.L3Key]bool // (interface's file, signature) of every interface method + bodiless map[dataflow.L3Key]bool // (declaring file, signature) of every bodiless declared callable calls map[dataflow.L3Key]map[token.Pos]string // unit → call Lparen → L1/L3 call key projPaths map[string]bool // import paths of the project packages units map[dataflow.L3Key]dataflow.Unit // by key @@ -60,7 +63,8 @@ func newJoin(in Input) *join { fset: in.Fset, projectDir: in.ProjectDir, byFn: make(map[ast.Node]dataflow.L3Key, len(in.Units)), - bySig: make(map[dataflow.L3Key]bool, len(in.Units)), + iface: map[dataflow.L3Key]bool{}, + bodiless: map[dataflow.L3Key]bool{}, calls: make(map[dataflow.L3Key]map[token.Pos]string, len(in.Units)), projPaths: map[string]bool{}, units: make(map[dataflow.L3Key]dataflow.Unit, len(in.Units)), @@ -69,7 +73,6 @@ func newJoin(in Input) *join { } for _, u := range in.Units { j.byFn[u.Fn] = u.Key - j.bySig[u.Key] = true j.units[u.Key] = u if r, ok := in.L3[u.Key]; ok && r.CallKeys != nil { j.calls[u.Key] = r.CallKeys @@ -81,25 +84,55 @@ func newJoin(in Input) *join { } j.calls[u.Key] = m } - // Interface methods have no body and so no unit, but a $bound wrapper of - // one (a method value of an interface) still names it: index them by - // (declaring file, signature) too, the file being the interface's. + // Callables with a symbol-table key but no unit (no body, so no L3): + // - interface methods, keyed by the interface's file (c.Path is ""); + // a $bound / $thunk of one names it; + // - declared functions and methods with no Go body (assembly or + // linkname implementations); only a FuncDecl with a nil body counts, + // so a declaration L3Units skipped despite its body (a second init in + // one file) never lands here. + declared := map[dataflow.L3Key]bool{} for path, f := range in.SymbolTable { + for _, c := range f.Functions { + declared[dataflow.L3Key{Path: c.Path, Signature: c.Signature}] = true + } for _, t := range f.Types { for _, c := range t.Methods { if c.Path == "" { - j.bySig[dataflow.L3Key{Path: path, Signature: c.Signature}] = true + j.iface[dataflow.L3Key{Path: path, Signature: c.Signature}] = true + } else { + declared[dataflow.L3Key{Path: c.Path, Signature: c.Signature}] = true } } } } for _, p := range projectPackages(in) { j.projPaths[p.PkgPath] = true + if p.TypesInfo == nil { + continue + } + for _, f := range p.Syntax { + for _, d := range f.Decls { + fd, ok := d.(*ast.FuncDecl) + if !ok || fd.Body != nil { + continue + } + if obj, ok := p.TypesInfo.Defs[fd.Name].(*types.Func); ok { + if k, ok := j.declKey(obj); ok && declared[k] { + j.bodiless[k] = true + } + } + } + } } return j } -// fnUnit returns the L3 callable fn joins: +// fnUnit returns the L3 unit fn joins. ok means a unit exists: j.units[k] +// holds it (its Fn and Info are set), though its L3 result may be absent +// (l3_skipped). A function that names a callable with no unit — an +// interface method (ifaceMethod) or a bodiless declaration (bodilessDecl) — +// is !ok here. The join: // // 1. fn.Syntax() is a unit's Fn (a declared function or method, init#N, // a function literal, a generic instance or instantiation wrapper — @@ -112,8 +145,7 @@ func newJoin(in Input) *join { // $thunk, promoted or pointer wrapper) or a bodiless stub of a project // function seen through another test variant of its package — the // function fn.Object() names, by (declaring file, SignatureOf), the -// symbol table's own canonicalizer. An interface method's key (the -// interface's file) counts here too. +// symbol table's own canonicalizer. // 4. A range-over-func yield folds into fn.Parent(). // // The package initializer, closures of package-level variable initializers @@ -121,7 +153,40 @@ func newJoin(in Input) *join { // nothing. func (j *join) fnUnit(fn *ssa.Function) (dataflow.L3Key, bool) { k, kind := j.resolve(fn) - return k, kind != joinNone + if kind != joinBody && kind != joinWrapper { + return dataflow.L3Key{}, false + } + if _, ok := j.units[k]; !ok { + return dataflow.L3Key{}, false + } + return k, true +} + +// ifaceMethod returns the symbol-table key of the interface method fn +// stands for — the interface's declaring file and the method's signature, +// as the symbol table keys bodiless interface methods (GoType.Methods with +// Path "") — for a $bound (method value) or $thunk (method expression) of an +// interface method. The key has no unit and no L3 result: fnUnit is !ok for +// fn. Later stages bind such a target to the interface method's formals and +// pass-through summary (spec §4.8, D7). +func (j *join) ifaceMethod(fn *ssa.Function) (dataflow.L3Key, bool) { + if k, kind := j.resolve(fn); kind == joinIface { + return k, true + } + return dataflow.L3Key{}, false +} + +// bodilessDecl returns the symbol-table key of the declared function or +// method fn stands for when its declaration has no Go body (implemented in +// assembly, or by //go:linkname): fn itself (a bodiless SSA function with +// that FuncDecl as syntax) or a wrapper of it. The key has no unit and no +// L3 result: fnUnit is !ok for fn. Like an interface method, it is a target +// with formals and a pass-through summary. +func (j *join) bodilessDecl(fn *ssa.Function) (dataflow.L3Key, bool) { + if k, kind := j.resolve(fn); kind == joinBodiless { + return k, true + } + return dataflow.L3Key{}, false } func (j *join) resolve(fn *ssa.Function) (dataflow.L3Key, joinKind) { @@ -135,13 +200,17 @@ func (j *join) resolve(fn *ssa.Function) (dataflow.L3Key, joinKind) { k, ok := j.byFn[syn] switch { case !ok: + if fd, isDecl := syn.(*ast.FuncDecl); isDecl && fd.Body == nil { + break // a bodiless declaration: by Object(), below + } return dataflow.L3Key{}, joinNone case strings.HasPrefix(fn.Synthetic, synthInstWrapper): // It keeps its origin's syntax, but its body is a synthetic // call of the origin. return k, joinWrapper + default: + return k, joinBody } - return k, joinBody } if fn.Parent() == nil { if o := fn.Origin(); o != nil && o != fn { @@ -154,8 +223,15 @@ func (j *join) resolve(fn *ssa.Function) (dataflow.L3Key, joinKind) { return dataflow.L3Key{}, joinNone } if obj, ok := fn.Object().(*types.Func); ok { - if k, ok := j.declKey(obj.Origin()); ok && j.bySig[k] { - return k, joinWrapper + if k, ok := j.declKey(obj.Origin()); ok { + switch { + case j.units[k].Fn != nil: + return k, joinWrapper + case j.iface[k]: + return k, joinIface + case j.bodiless[k]: + return k, joinBodiless + } } } return dataflow.L3Key{}, joinNone @@ -226,6 +302,7 @@ func (j *join) isLibrary(obj *types.Func) bool { // go/ssa synthesizes has none and no call node. type joinCoverage struct { Funcs, Unmapped, Wrappers, Inits int // Inits: package initializers (no callable by design) + NoUnit int // mapped to a callable with no unit: interface methods, bodiless declarations Calls, UnmappedCalls, Synthetic int // Calls: positioned calls in body-joined functions unmappedPkgs map[string]bool } @@ -250,13 +327,15 @@ func (j *join) coverage(funcs map[*ssa.Function]bool) joinCoverage { utils.Debug("L4: unmapped SSA function %s [%s]", fn, fn.Synthetic) case joinWrapper: c.Wrappers++ + case joinIface, joinBodiless: + c.NoUnit++ } forCalls(fn, func(call ssa.CallInstruction) { switch { case !call.Common().Pos().IsValid(): c.Synthetic++ - case kind == joinWrapper: - c.Synthetic++ + case kind == joinWrapper || kind == joinIface || kind == joinBodiless: + c.Synthetic++ // a wrapper's calls are synthetic default: c.Calls++ if _, _, ok := j.callSite(call); !ok { diff --git a/internal/sdg/join_test.go b/internal/sdg/join_test.go index 87eaf6c..d29289e 100644 --- a/internal/sdg/join_test.go +++ b/internal/sdg/join_test.go @@ -387,8 +387,8 @@ func TestJoin_WrappersUnwrap(t *testing.T) { } // Keyed by fn.String(): an interface method's wrappers share the // concrete method's short names (Add$bound, Add$thunk). - if strings.HasPrefix(fn.Synthetic, "bound method wrapper") || strings.HasPrefix(fn.Synthetic, "thunk") { - kinds[fn.String()] = mustJoin(t, j, fn) + if (strings.HasPrefix(fn.Synthetic, "bound method wrapper") || strings.HasPrefix(fn.Synthetic, "thunk")) && !strings.Contains(fn.String(), "Adder") { + kinds[fn.String()] = mustJoin(t, j, fn) // concrete-method wrappers; Adder's are checked below } if fn.Syntax() == nil && fn.Synthetic != synthPackageInit { if _, kind := j.resolve(fn); kind == joinWrapper { @@ -405,12 +405,23 @@ func TestJoin_WrappersUnwrap(t *testing.T) { t.Errorf("%s joins %v (seen %v), want %v (wrappers seen: %v)", name, k, ok, add, kinds) } } + // An interface method has a symbol-table key but no unit: its $bound and + // $thunk are !ok for fnUnit and name it through ifaceMethod. iface := dataflow.L3Key{Path: filepath.Join("xtest", "a.go"), Signature: xt + ".Adder.Add"} + if _, ok := j.units[iface]; ok { + t.Fatalf("interface method %v has a unit", iface) + } var ifaceWrappers int for fn := range funcs { if obj := fn.Object(); obj != nil && fn.Syntax() == nil && obj.Pkg() != nil && obj.Pkg().Path() == xt && obj.Name() == "Add" && types.IsInterface(fn.Object().Type().(*types.Signature).Recv().Type()) { - if k := mustJoin(t, j, fn); k != iface { - t.Errorf("%s [%s] joins %v, want the interface method %v", fn, fn.Synthetic, k, iface) + if k, ok := j.fnUnit(fn); ok { + t.Errorf("fnUnit(%s [%s]) = %v, true; want !ok (an interface method has no unit)", fn, fn.Synthetic, k) + } + if k, ok := j.ifaceMethod(fn); !ok || k != iface { + t.Errorf("ifaceMethod(%s [%s]) = %v, %v; want %v, true", fn, fn.Synthetic, k, ok, iface) + } + if _, ok := j.bodilessDecl(fn); ok { + t.Errorf("bodilessDecl(%s) is ok for an interface method", fn) } ifaceWrappers++ } @@ -418,6 +429,16 @@ func TestJoin_WrappersUnwrap(t *testing.T) { if ifaceWrappers != 2 { t.Errorf("%d wrappers of the interface method Adder.Add, want 2 ($bound, $thunk)", ifaceWrappers) } + // A concrete method and its wrappers still join their unit, and are not + // interface methods. + for _, fn := range []*ssa.Function{fnByName(t, prog, xt, "Counter.Add"), fnByName(t, prog, xt, "Outer.Add")} { + if k, ok := j.fnUnit(fn); !ok || k != add || j.units[k].Fn == nil { + t.Errorf("fnUnit(%s) = %v, %v; want the unit %v", fn, k, ok, add) + } + if _, ok := j.ifaceMethod(fn); ok { + t.Errorf("ifaceMethod(%s) is ok for a concrete method", fn) + } + } if k := mustJoin(t, j, fnByName(t, prog, xt, "Counter.Get")); k != get { t.Errorf("(*Counter).Get (pointer wrapper) joins %v, want %v", k, get) } @@ -426,6 +447,42 @@ func TestJoin_WrappersUnwrap(t *testing.T) { } } +// TestJoin_BodilessDeclHasNoUnit: a function declared without a Go body +// (testdata/dataflow/asm, implemented in assembly) is in the symbol table +// but has no unit. Its SSA function is !ok for fnUnit and names it through +// bodilessDecl; it is not unmapped, and its caller's call still joins. +func TestJoin_BodilessDeclHasNoUnit(t *testing.T) { + in := loadInput(t, "dataflow") + prog, j, funcs := joined(t, in) + const pkg = "example.com/dataflow/asm" + want := dataflow.L3Key{Path: filepath.Join("asm", "asm.go"), Signature: pkg + ".Add"} + if _, ok := j.units[want]; ok { + t.Fatalf("bodiless %v has a unit", want) + } + add := fnByName(t, prog, pkg, "Add") + if k, ok := j.fnUnit(add); ok { + t.Errorf("fnUnit(asm.Add) = %v, true; want !ok", k) + } + if k, ok := j.bodilessDecl(add); !ok || k != want { + t.Errorf("bodilessDecl(asm.Add) = %v, %v; want %v, true", k, ok, want) + } + if _, ok := j.ifaceMethod(add); ok { + t.Error("ifaceMethod(asm.Add) is ok") + } + if c := j.coverage(funcs); c.NoUnit == 0 { + t.Errorf("coverage %+v counts no function without a unit", c) + } + n := 0 + forCalls(fnByName(t, prog, pkg, "Use"), func(c ssa.CallInstruction) { + if _, _, ok := j.callSite(c); ok { + n++ + } + }) + if n != 1 { + t.Errorf("asm.Use: %d calls joined, want 1", n) + } +} + func mustJoin(t *testing.T, j *join, fn *ssa.Function) dataflow.L3Key { t.Helper() k, ok := j.fnUnit(fn) diff --git a/internal/sdg/run.go b/internal/sdg/run.go index 8b7172a..16c3bda 100644 --- a/internal/sdg/run.go +++ b/internal/sdg/run.go @@ -47,8 +47,8 @@ func Run(in Input) Result { } j := newJoin(in) cov := j.coverage(funcs) - utils.Info("L4: SSA join: %d functions (%d by wrapper), %d calls; %d functions, %d calls unmapped in %s", - cov.Funcs, cov.Wrappers, cov.Calls, cov.Unmapped, cov.UnmappedCalls, time.Since(start).Round(time.Millisecond)) + utils.Info("L4: SSA join: %d functions (%d by wrapper, %d without a unit), %d calls; %d functions, %d calls unmapped in %s", + cov.Funcs, cov.Wrappers, cov.NoUnit, cov.Calls, cov.Unmapped, cov.UnmappedCalls, time.Since(start).Round(time.Millisecond)) if d := j.degraded(cov); d != nil { deg = append(deg, *d) } diff --git a/internal/sdg/ssa_test.go b/internal/sdg/ssa_test.go index eee6a16..4707c7c 100644 --- a/internal/sdg/ssa_test.go +++ b/internal/sdg/ssa_test.go @@ -60,7 +60,7 @@ func TestSSA_BuildsProjectPackagesOnly(t *testing.T) { built = append(built, p.Pkg.Path()) } want := []string{ - "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/inits", + "example.com/dataflow/asm", "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/inits", "example.com/dataflow/l1", "example.com/dataflow/linedir", "example.com/dataflow/vend", "example.com/dataflow/xfile", "example.com/dataflow/xtest", "example.com/dataflow/xtest/user", } @@ -133,8 +133,8 @@ func TestSSA_PanicDegradesPackage(t *testing.T) { t.Error("the panicked package is still returned as built") } } - if len(pkgs) != 8 { - t.Errorf("built %d packages, want the other 8", len(pkgs)) + if len(pkgs) != 9 { + t.Errorf("built %d packages, want the other 9", len(pkgs)) } if fn := prog.ImportedPackage("example.com/dataflow/l1"); fn == nil || len(fn.Members) == 0 { t.Error("l1 was not built after flow panicked") @@ -152,8 +152,8 @@ func TestSSA_PanicDegradesPackage(t *testing.T) { if !slices.Equal(codes, []string{"partial_ssa", "no_ssa"}) { t.Fatalf("codes = %v, want [partial_ssa no_ssa]", codes) } - if len(deg[0].Packages) != 9 { - t.Errorf("partial_ssa lists %v, want all 9 packages", deg[0].Packages) + if len(deg[0].Packages) != 10 { + t.Errorf("partial_ssa lists %v, want all 10 packages", deg[0].Packages) } if res := Run(in); len(res.Degraded) != 2 || res.Degraded[1].Code != "no_ssa" { t.Errorf("Run degraded = %+v, want partial_ssa + no_ssa and no VTA", res.Degraded) diff --git a/testdata/dataflow/asm/asm.go b/testdata/dataflow/asm/asm.go new file mode 100644 index 0000000..4906011 --- /dev/null +++ b/testdata/dataflow/asm/asm.go @@ -0,0 +1,10 @@ +// Package asm declares a function with no Go body (implemented in assembly, +// asm.s). The symbol table records it as a callable, but it has no L3 unit: +// the SSA join maps it through bodilessDecl, not fnUnit. +package asm + +// Add is implemented in assembly. +func Add(a, b int) int + +// Use calls Add. +func Use() int { return Add(1, 2) } diff --git a/testdata/dataflow/asm/asm.s b/testdata/dataflow/asm/asm.s new file mode 100644 index 0000000..d17a030 --- /dev/null +++ b/testdata/dataflow/asm/asm.s @@ -0,0 +1,2 @@ +// Add is declared in asm.go; this file only lets the package build with a +// bodiless declaration. From c6f2bd477d32684e6d0f17dc1477aa2df606887d Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 01:47:10 -0400 Subject: [PATCH 12/65] fix(l1): keep methods declared above their receiver type --- internal/sdg/join_test.go | 4 +- internal/syntactic_analysis/symbol_table.go | 23 +++++-- .../syntactic_analysis/v2emit/gate_l3_test.go | 61 +++++++++++++++++++ testdata/dataflow/flow/flow.go | 5 ++ 4 files changed, 84 insertions(+), 9 deletions(-) diff --git a/internal/sdg/join_test.go b/internal/sdg/join_test.go index d29289e..9c3a7f4 100644 --- a/internal/sdg/join_test.go +++ b/internal/sdg/join_test.go @@ -117,9 +117,7 @@ func inDropped(fn *ssa.Function, dropped map[ast.Node]bool) bool { // http2FancyWriter.Push is declared before its receiver type). Every unit is // the image of at least one SSA function. func TestJoin_EveryProjectFunctionMaps(t *testing.T) { - wantDropped := map[string][]string{ - "chi": {"(*github.com/go-chi/chi/v5/middleware.http2FancyWriter).Push"}, - } + wantDropped := map[string][]string{} for _, name := range []string{"dataflow", "chi", "generics", "rangefunc"} { t.Run(name, func(t *testing.T) { in := loadInput(t, name) diff --git a/internal/syntactic_analysis/symbol_table.go b/internal/syntactic_analysis/symbol_table.go index becf00a..ada2bed 100644 --- a/internal/syntactic_analysis/symbol_table.go +++ b/internal/syntactic_analysis/symbol_table.go @@ -242,7 +242,14 @@ func (b *SymbolTableBuilder) buildGoFile( FileSize: fileSize, } - // Walk top-level declarations. + // Walk top-level declarations. A method may be declared above its receiver + // type, so methods are collected here and attached once every type of the + // file is known. + type pendingMethod struct { + typeName string + callable *schema.GoCallable + } + var methods []pendingMethod for _, decl := range astFile.Decls { switch d := decl.(type) { case *ast.GenDecl: @@ -253,18 +260,22 @@ func (b *SymbolTableBuilder) buildGoFile( continue } if d.Recv != nil { - // Method — attach to its type. if typeName := b.receiverTypeName(d.Recv); typeName != "" { - if gt, ok := gf.Types[typeName]; ok { - gt.Methods[callable.Signature] = *callable - gf.Types[typeName] = gt - } + methods = append(methods, pendingMethod{typeName, callable}) } } else { gf.Functions[callable.Signature] = *callable } } } + // Attach in source order. A receiver type in another file is left to + // reconcileCrossFileMethods. + for _, m := range methods { + if gt, ok := gf.Types[m.typeName]; ok { + gt.Methods[m.callable.Signature] = *m.callable + gf.Types[m.typeName] = gt + } + } return gf } diff --git a/internal/syntactic_analysis/v2emit/gate_l3_test.go b/internal/syntactic_analysis/v2emit/gate_l3_test.go index 1209fe4..e0d4b23 100644 --- a/internal/syntactic_analysis/v2emit/gate_l3_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l3_test.go @@ -494,3 +494,64 @@ func TestGateL3_LineDirectiveRealPositions(t *testing.T) { t.Error("Caller: empty cfg") } } + +// TestSymbolTable_MethodDeclaredAboveItsType: a method declared above its +// receiver type in the same file is kept. It has its call site at L1, the +// call to f resolves at -a 2 (callee id and call_graph edge), and at -a 3 it +// has body nodes and a ddg. +func TestSymbolTable_MethodDeclaredAboveItsType(t *testing.T) { + fx := buildL3(t, "dataflow") + const sig = "example.com/dataflow/flow.early.Bump" + const fID = "example.com/dataflow/flow.f" + + find := func(out *v2.Analysis) (v2.Callable, bool) { + for _, mod := range out.Application.SymbolTable { + for _, ty := range mod.Types { + for _, m := range ty.Callables { + if strings.HasSuffix(m.ID, "/"+sig) { + return m, true + } + } + } + } + return v2.Callable{}, false + } + + bump, ok := find(fx.l2) + if !ok { + t.Fatalf("no callable %s under type early at -a 2", sig) + } + var call *v2.BodyNode + for _, n := range bump.Body { + n := n + if n.Kind == "call" { + call = &n + } + } + if call == nil { + t.Fatalf("%s has no call site", sig) + } + if call.Callee == nil || !strings.HasSuffix(*call.Callee, "/"+fID) { + t.Fatalf("call to f not resolved at -a 2: callee = %v", call.Callee) + } + sawEdge := false + for _, e := range fx.l2.Application.CallGraph { + if e.Src == bump.ID && e.Dst == *call.Callee { + sawEdge = true + } + } + if !sawEdge { + t.Errorf("no call_graph edge %s -> %s", bump.ID, *call.Callee) + } + + b3, ok := find(fx.l3) + if !ok { + t.Fatalf("no callable %s at -a 3", sig) + } + if _, ok := b3.Body["@entry"]; !ok { + t.Errorf("%s has no @entry body node at -a 3 (keys %d)", sig, len(b3.Body)) + } + if len(b3.Ddg) == 0 { + t.Errorf("%s has no ddg at -a 3", sig) + } +} diff --git a/testdata/dataflow/flow/flow.go b/testdata/dataflow/flow/flow.go index 1f60e61..3c8f426 100644 --- a/testdata/dataflow/flow/flow.go +++ b/testdata/dataflow/flow/flow.go @@ -698,3 +698,8 @@ func PlainRecv(p plain) int { p.sum() return p.a } + +// Bump is a method declared above its receiver type (L1 used to drop it). +func (e *early) Bump() int { return f(e.n) } + +type early struct{ n int } From c91351cdeef32db0ee81b9e7ae621993574a12ad Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 04:16:28 -0400 Subject: [PATCH 13/65] feat(l4): VTA dispatch targets in call_graph --- internal/core/analyzer.go | 4 + internal/core/sdg_test.go | 31 ++ internal/schema/schema.go | 7 + internal/sdg/ir.go | 1 + internal/sdg/join_test.go | 3 +- internal/sdg/run.go | 65 ++-- internal/sdg/ssa.go | 77 ++++- internal/sdg/ssa_test.go | 20 +- internal/sdg/vta.go | 291 ++++++++++++++++ internal/sdg/vta_test.go | 327 ++++++++++++++++++ internal/semantic_analysis/call_graph.go | 22 +- internal/semantic_analysis/call_graph_test.go | 24 ++ internal/syntactic_analysis/v2emit/emit.go | 9 +- .../syntactic_analysis/v2emit/gate_l4_test.go | 130 ++++++- testdata/dataflow/initclosure/a.go | 19 + testdata/dataflow/initclosure/b.go | 5 + testdata/dataflow/sdg/sdg.go | 100 ++++++ 17 files changed, 1073 insertions(+), 62 deletions(-) create mode 100644 internal/sdg/vta.go create mode 100644 internal/sdg/vta_test.go create mode 100644 testdata/dataflow/initclosure/a.go create mode 100644 testdata/dataflow/initclosure/b.go create mode 100644 testdata/dataflow/sdg/sdg.go diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index 80d5d2d..414338f 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -142,6 +142,7 @@ func (a *Analyzer) Analyze() (*schema.GoApplication, error) { ProjectDir: a.opts.InputPath, SymbolTable: symbolTable, Externals: app.Externals, + CallGraph: app.CallGraph, Units: units, L3: a.l3, K: KLimit(a.opts), @@ -151,6 +152,9 @@ func (a *Analyzer) Analyze() (*schema.GoApplication, error) { if err := a.reportDegraded(res.Degraded); err != nil { return nil, err } + // VTA's dispatch edges and the externals only VTA reached (spec §4.8): + // additions to the L2 call graph, so -a 3 ⊆ -a 4. + res.ApplyTo(app) } // The last reader of Node.Stmt and of the L4 side tables is done diff --git a/internal/core/sdg_test.go b/internal/core/sdg_test.go index 35446eb..a8bccdc 100644 --- a/internal/core/sdg_test.go +++ b/internal/core/sdg_test.go @@ -76,3 +76,34 @@ func TestAnalyze_Level4StrictFailsOnDegraded(t *testing.T) { } } } + +// TestAnalyze_Level4MergesVTAEdges: at -a 4 the analyzer merges sdg.Run's +// VTA edges into the call graph and its VTA-only externals into +// app.Externals; -a 3 has neither. +func TestAnalyze_Level4MergesVTAEdges(t *testing.T) { + opts := options.AnalysisOptions{ + InputPath: filepath.Join(testdataDir(), "dataflow"), AppName: "dataflow", Level: options.LevelSDG, + SkipTests: true, CacheDir: t.TempDir(), + } + const src, dst = "example.com/dataflow/sdg.Buffered", "bytes.Buffer.Write" + vtaEdge := func(level options.AnalysisLevel) (vta, ext bool) { + opts.Level = level + app, err := core.New(opts).Analyze() + if err != nil { + t.Fatal(err) + } + for _, e := range app.CallGraph { + if e.Source == src && e.Target == dst { + vta = len(e.Provenance) == 1 && e.Provenance[0] == "vta" + } + } + _, ext = app.Externals[dst] + return vta, ext + } + if vta, ext := vtaEdge(options.LevelSDG); !vta || !ext { + t.Errorf("-a 4: VTA edge %s -> %s = %v, external = %v; want both", src, dst, vta, ext) + } + if vta, ext := vtaEdge(options.LevelDataflow); vta || ext { + t.Errorf("-a 3: VTA edge = %v, external = %v; want neither", vta, ext) + } +} diff --git a/internal/schema/schema.go b/internal/schema/schema.go index 68b6538..7c9e96d 100644 --- a/internal/schema/schema.go +++ b/internal/schema/schema.go @@ -193,6 +193,13 @@ type GoCallEdge struct { // whose signature is already unique (the common case). Internal-only; not // part of the v2 output schema. SourcePath string `json:"-"` + + // TargetPath is the relative file declaring the target callable, set only + // when the signature alone may not name it: a VTA target can be a closure + // of a package-level init (".init.closure_1" repeats once per file + // with an init). Empty for L2 edges (a static call never targets an init + // or a closure) and for an external target. Internal-only, like SourcePath. + TargetPath string `json:"-"` } // ─── Root object ────────────────────────────────────────────────────────────── diff --git a/internal/sdg/ir.go b/internal/sdg/ir.go index 26b9ca4..e17e4ad 100644 --- a/internal/sdg/ir.go +++ b/internal/sdg/ir.go @@ -82,6 +82,7 @@ type Input struct { ProjectDir string SymbolTable map[string]schema.GoFile Externals map[string]schema.GoExternalCallee + CallGraph []schema.GoCallEdge // L2 edges (app.CallGraph) Units []dataflow.Unit L3 map[dataflow.L3Key]dataflow.Result K, Jobs int diff --git a/internal/sdg/join_test.go b/internal/sdg/join_test.go index 9c3a7f4..073b910 100644 --- a/internal/sdg/join_test.go +++ b/internal/sdg/join_test.go @@ -40,7 +40,8 @@ func joined(t *testing.T, in Input) (*ssa.Program, *join, map[*ssa.Function]bool funcs[fn] = true } } - for fn := range projectFuncs(prog, pkgs) { + project2, _ := projectFuncs(prog, pkgs) + for fn := range project2 { funcs[fn] = true } return prog, j, funcs diff --git a/internal/sdg/run.go b/internal/sdg/run.go index 16c3bda..595df48 100644 --- a/internal/sdg/run.go +++ b/internal/sdg/run.go @@ -15,8 +15,9 @@ import ( var vtaCallGraph = vta.CallGraph // Run builds the L4 overlay of in. For now it builds SSA over the project -// packages, joins it to the L3 units, and runs VTA once over the project -// function set, and returns only the degradation: no vertices or edges yet. +// packages, joins it to the L3 units, resolves every call site's dispatch +// targets (L2's static callee ∪ VTA's callees) and returns the call-graph +// edges and externals only VTA found, and the degradation: no vertices yet. // // Order: the join is computed before VTA, so it never reads a program a VTA // panic may have wedged; after no_vta, later stages use the program for the @@ -24,55 +25,79 @@ var vtaCallGraph = vta.CallGraph // l3_skipped and range_func (from the units, whether or not SSA built), // unmapped_ssa (the join), no_vta. func Run(in Input) Result { + res, _ := run(in) + return res +} + +// run is Run that also returns the call-site dispatch. Every site keeps its +// static L2 targets whether or not SSA or VTA ran. +func run(in Input) (Result, *dispatch) { + d := newDispatch(in) start := time.Now() prog, pkgs, deg := buildSSA(in) utils.Info("L4: SSA built %d project packages in %s", len(pkgs), time.Since(start).Round(time.Millisecond)) - for _, d := range []*Degraded{l3Skipped(in), rangeFunc(in)} { - if d != nil { - deg = append(deg, *d) + for _, dg := range []*Degraded{l3Skipped(in), rangeFunc(in)} { + if dg != nil { + deg = append(deg, *dg) } } if len(pkgs) == 0 { - return Result{Degraded: deg} + return d.result(deg), d } - // The project function set (VTA's seeds) is built under recover: it - // builds wrappers on demand, and a panic there may wedge the program, - // so neither the join nor VTA then reads it. + // The VTA function set is built under recover: it builds wrappers on + // demand, and a panic there may wedge the program, so neither the join + // nor VTA then reads it. start = time.Now() - funcs, err := funcSet(prog, pkgs) + funcs, all, err := funcSet(prog, pkgs) if err != nil { utils.Debug("L4: VTA function set failed: %v", err) - return Result{Degraded: append(deg, Degraded{Code: CodeNoVTA, Detail: err.Error()})} + return d.result(append(deg, Degraded{Code: CodeNoVTA, Detail: err.Error()})), d } j := newJoin(in) cov := j.coverage(funcs) utils.Info("L4: SSA join: %d functions (%d by wrapper, %d without a unit), %d calls; %d functions, %d calls unmapped in %s", cov.Funcs, cov.Wrappers, cov.NoUnit, cov.Calls, cov.Unmapped, cov.UnmappedCalls, time.Since(start).Round(time.Millisecond)) - if d := j.degraded(cov); d != nil { - deg = append(deg, *d) + if dg := j.degraded(cov); dg != nil { + deg = append(deg, *dg) } + d.addInvokes(j, funcs) start = time.Now() - cg, err := runVTA(funcs) + cg, err := runVTA(all) if err != nil { utils.Debug("L4: VTA failed: %v", err) deg = append(deg, Degraded{Code: CodeNoVTA, Detail: err.Error()}) } else { - utils.Info("L4: VTA over %d functions: %d edges in %s", len(funcs), edgeCount(cg), time.Since(start).Round(time.Millisecond)) + d.addVTA(j, cg) + utils.Info("L4: VTA over %d functions (%d library targets): %d edges in %s", + len(all), len(all)-len(funcs), edgeCount(cg), time.Since(start).Round(time.Millisecond)) } - return Result{Degraded: deg} + res := d.result(deg) + utils.Info("L4: dispatch: %d call sites with targets; %d call_graph edges and %d externals added by VTA", + len(d.sites), len(res.CallEdges), len(res.NewExternals)) + return res, d } // funcSet is projectFuncs under recover: it builds functions on demand -// (MethodValue wrappers), so a builder panic is turned into an error. -func funcSet(prog *ssa.Program, pkgs []*ssa.Package) (funcs map[*ssa.Function]bool, err error) { +// (MethodValue wrappers), so a builder panic is turned into an error. It +// returns the project functions (the join is measured on them) and the VTA +// set: those plus the target-only library functions. +func funcSet(prog *ssa.Program, pkgs []*ssa.Package) (funcs, all map[*ssa.Function]bool, err error) { defer func() { if r := recover(); r != nil { - funcs, err = nil, fmt.Errorf("panic: %v", r) + funcs, all, err = nil, nil, fmt.Errorf("panic: %v", r) } }() - return projectFuncs(prog, pkgs), nil + funcs, targetOnly := projectFuncs(prog, pkgs) + all = make(map[*ssa.Function]bool, len(funcs)+len(targetOnly)) + for fn := range funcs { + all[fn] = true + } + for fn := range targetOnly { + all[fn] = true + } + return funcs, all, nil } // runVTA runs VTA over funcs with no initial graph (VTA then uses its diff --git a/internal/sdg/ssa.go b/internal/sdg/ssa.go index adc261c..b7efd15 100644 --- a/internal/sdg/ssa.go +++ b/internal/sdg/ssa.go @@ -9,6 +9,7 @@ import ( "golang.org/x/tools/go/packages" "golang.org/x/tools/go/ssa" "golang.org/x/tools/go/ssa/ssautil" + "golang.org/x/tools/go/types/typeutil" "github.com/codellm-devkit/codeanalyzer-go/internal/utils" ) @@ -148,20 +149,37 @@ func isProject(dir string, p *packages.Package) bool { return false } -// projectFuncs is the VTA function set (spec §4.8, VTA scope): the SSA -// functions of the built project packages — package-level functions and -// inits, every declared method (generic origins included), the method-set +// projectFuncs is the VTA function set (spec §4.8, VTA scope). funcs holds +// the SSA functions of the built project packages — package-level functions +// and inits, every declared method (generic origins included), the method-set // wrappers of their non-generic named types — and, transitively, their -// anonymous functions and every function their code refers to that is -// project code (generic instances of project functions, $bound / $thunk -// wrappers). Functions declared outside the project are left out, so VTA's -// cost stays bounded by the project's size. -func projectFuncs(prog *ssa.Program, pkgs []*ssa.Package) map[*ssa.Function]bool { +// anonymous functions and every function their code refers to that is project +// code (generic instances of project functions, $bound / $thunk wrappers). +// Its code is what VTA propagates types through, so VTA's cost stays bounded +// by the project's size; the SSA join is measured on it. +// +// targetOnly holds the functions declared outside the project that a dynamic +// call in project code may reach, which VTA can return only when they are in +// its set (it intersects the propagated types with a lazy CHA over the set): +// +// - a library function used as a value in project code (any operand but +// the static callee of a call: VTA resolves static calls without the set); +// - the methods (prog.MethodValue) of each type that a MakeInterface in +// project code converts to an interface — the dynamic types an interface +// call there may dispatch to — that are not project code: a library +// type's methods, or a project type's wrappers of promoted library +// methods. A project method found this way joins funcs. +// +// A library function has no SSA body (its package is created from types), so +// targetOnly adds no propagation work; a library promotion wrapper's body is +// a single synthetic call. +func projectFuncs(prog *ssa.Program, pkgs []*ssa.Package) (funcs, targetOnly map[*ssa.Function]bool) { project := make(map[*types.Package]bool, len(pkgs)) for _, p := range pkgs { project[p.Pkg] = true } - funcs := map[*ssa.Function]bool{} + funcs = map[*ssa.Function]bool{} + targetOnly = map[*ssa.Function]bool{} var work []*ssa.Function add := func(fn *ssa.Function) { if fn != nil && !funcs[fn] && isProjectFunc(fn, project) { @@ -169,6 +187,29 @@ func projectFuncs(prog *ssa.Program, pkgs []*ssa.Package) map[*ssa.Function]bool work = append(work, fn) } } + addTarget := func(fn *ssa.Function) { + switch { + case fn == nil || funcs[fn]: + case isProjectFunc(fn, project): + add(fn) + default: + targetOnly[fn] = true + } + } + var converted typeutil.Map // types already expanded by a MakeInterface + addMethods := func(t types.Type) { + if converted.At(t) != nil { + return + } + converted.Set(t, true) + if _, isParam := types.Unalias(t).(*types.TypeParam); isParam || types.IsInterface(t) { + return + } + mset := prog.MethodSets.MethodSet(t) + for i := 0; i < mset.Len(); i++ { + addTarget(prog.MethodValue(mset.At(i))) // nil for a generic method or type + } + } for _, p := range pkgs { names := make([]string, 0, len(p.Members)) for name := range p.Members { @@ -207,15 +248,27 @@ func projectFuncs(prog *ssa.Program, pkgs []*ssa.Package) map[*ssa.Function]bool } for _, b := range fn.Blocks { for _, instr := range b.Instrs { + var callee *ssa.Value + if call, ok := instr.(ssa.CallInstruction); ok && !call.Common().IsInvoke() { + callee = &call.Common().Value + } for _, op := range instr.Operands(buf[:0]) { - if g, ok := (*op).(*ssa.Function); ok { - add(g) + g, ok := (*op).(*ssa.Function) + switch { + case !ok: + case op == callee: + add(g) // a static callee: VTA needs it only as project code + default: + addTarget(g) } } + if mi, ok := instr.(*ssa.MakeInterface); ok { + addMethods(mi.X.Type()) + } } } } - return funcs + return funcs, targetOnly } // isProjectFunc reports whether fn's code is project code: it belongs to a diff --git a/internal/sdg/ssa_test.go b/internal/sdg/ssa_test.go index 4707c7c..7a1de53 100644 --- a/internal/sdg/ssa_test.go +++ b/internal/sdg/ssa_test.go @@ -14,12 +14,13 @@ import ( "golang.org/x/tools/go/ssa/ssautil" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/utils" ) // loadInput loads testdata/ the way the analyzer does at -a 4 -// (symbol-table builder, tests skipped, L3 with side tables) and returns the +// (symbol-table builder, tests skipped, the L2 call graph, L3 with side tables) and returns the // sdg input over it. func loadInput(t *testing.T, name string) Input { t.Helper() @@ -39,9 +40,12 @@ func loadInputOpts(t *testing.T, name string, skipTests bool) Input { if err != nil { t.Fatalf("symbol table build failed: %v", err) } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) units := b.L3Units(st) return Input{ Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Externals: cg.Externals(), CallGraph: edges, Units: units, L3: dataflow.RunWithSideTables(b.Fset(), units, 3, 1), K: 3, Jobs: 1, } } @@ -60,8 +64,8 @@ func TestSSA_BuildsProjectPackagesOnly(t *testing.T) { built = append(built, p.Pkg.Path()) } want := []string{ - "example.com/dataflow/asm", "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/inits", - "example.com/dataflow/l1", "example.com/dataflow/linedir", "example.com/dataflow/vend", + "example.com/dataflow/asm", "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/initclosure", "example.com/dataflow/inits", + "example.com/dataflow/l1", "example.com/dataflow/linedir", "example.com/dataflow/sdg", "example.com/dataflow/vend", "example.com/dataflow/xfile", "example.com/dataflow/xtest", "example.com/dataflow/xtest/user", } if !slices.Equal(built, want) { @@ -133,8 +137,8 @@ func TestSSA_PanicDegradesPackage(t *testing.T) { t.Error("the panicked package is still returned as built") } } - if len(pkgs) != 9 { - t.Errorf("built %d packages, want the other 9", len(pkgs)) + if len(pkgs) != 11 { + t.Errorf("built %d packages, want the other 11", len(pkgs)) } if fn := prog.ImportedPackage("example.com/dataflow/l1"); fn == nil || len(fn.Members) == 0 { t.Error("l1 was not built after flow panicked") @@ -152,8 +156,8 @@ func TestSSA_PanicDegradesPackage(t *testing.T) { if !slices.Equal(codes, []string{"partial_ssa", "no_ssa"}) { t.Fatalf("codes = %v, want [partial_ssa no_ssa]", codes) } - if len(deg[0].Packages) != 10 { - t.Errorf("partial_ssa lists %v, want all 10 packages", deg[0].Packages) + if len(deg[0].Packages) != 12 { + t.Errorf("partial_ssa lists %v, want all 12 packages", deg[0].Packages) } if res := Run(in); len(res.Degraded) != 2 || res.Degraded[1].Code != "no_ssa" { t.Errorf("Run degraded = %+v, want partial_ssa + no_ssa and no VTA", res.Degraded) @@ -198,7 +202,7 @@ func TestRun_VTAPanicDegrades(t *testing.T) { func TestProjectFuncs_CoversClosuresAndMethods(t *testing.T) { in := loadInput(t, "dataflow") prog, pkgs, _ := buildSSA(in) - funcs := projectFuncs(prog, pkgs) + funcs, _ := projectFuncs(prog, pkgs) project := map[string]bool{} for _, p := range pkgs { project[p.Pkg.Path()] = true diff --git a/internal/sdg/vta.go b/internal/sdg/vta.go new file mode 100644 index 0000000..4b24942 --- /dev/null +++ b/internal/sdg/vta.go @@ -0,0 +1,291 @@ +package sdg + +import ( + "cmp" + "go/types" + "slices" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" +) + +// siteKey names one call site: its unit and its L1/L3 call key. +type siteKey struct { + unit dataflow.L3Key + call string +} + +// l2Edge is the identity of an L2 call_graph edge: L2 dedups per (declaring +// file, source signature, target signature). +type l2Edge struct{ path, src, dst string } + +// dispatch holds every call site's targets (spec §4.8): the static callee L2 +// resolved ∪ the callees VTA found ∪, for an interface call, the interface +// method (D7). Targets are mapped to the L3 join's keys — a unit, an +// interface method or a bodiless declaration, or an external signature — and +// deduplicated after mapping, so a generic instance and its origin, or a +// wrapper and the method it wraps, are one target. A callee that maps to +// nothing (unmapped_ssa, already reported) is not a target. +type dispatch struct { + sites map[siteKey][]Target // each sorted by compareTarget, no duplicates + l2 map[l2Edge]bool + l2Ext map[string]schema.GoExternalCallee // in.Externals, read-only + newExt map[string]schema.GoExternalCallee // externals L2 never saw +} + +// compareTarget orders targets by Callable (Path, Signature), then External. +func compareTarget(a, b Target) int { + return cmp.Or( + cmp.Compare(a.Callable.Path, b.Callable.Path), + cmp.Compare(a.Callable.Signature, b.Callable.Signature), + cmp.Compare(a.External, b.External), + ) +} + +// newDispatch seeds every unit's call sites with their L2 static callee: the +// callee signature L2 backfilled into the symbol table, as an external when +// L2 tagged it with an import path, else as the symbol-table callable of that +// signature. It needs no SSA, so a run that degrades to no_ssa or no_vta +// keeps the static targets (go/types dispatch only). +func newDispatch(in Input) *dispatch { + d := &dispatch{ + sites: map[siteKey][]Target{}, + l2: make(map[l2Edge]bool, len(in.CallGraph)), + l2Ext: in.Externals, + newExt: map[string]schema.GoExternalCallee{}, + } + for _, e := range in.CallGraph { + d.l2[l2Edge{e.SourcePath, e.Source, e.Target}] = true + } + + callables := map[dataflow.L3Key]schema.GoCallable{} + bySig := map[string]dataflow.L3Key{} + keep := func(sig string, k dataflow.L3Key) { + // A signature names one callable, except init (never a callee). + if old, ok := bySig[sig]; !ok || k.Path < old.Path { + bySig[sig] = k + } + } + var walk func(c schema.GoCallable) + walk = func(c schema.GoCallable) { + callables[dataflow.L3Key{Path: c.Path, Signature: c.Signature}] = c + for _, ic := range c.InnerCallables { + walk(ic) + } + } + for path, f := range in.SymbolTable { + for _, c := range f.Functions { + keep(c.Signature, dataflow.L3Key{Path: c.Path, Signature: c.Signature}) + walk(c) + } + for _, t := range f.Types { + for _, c := range t.Methods { + if c.Path == "" { // an interface method, keyed by its interface's file + keep(c.Signature, dataflow.L3Key{Path: path, Signature: c.Signature}) + continue + } + keep(c.Signature, dataflow.L3Key{Path: c.Path, Signature: c.Signature}) + walk(c) + } + } + } + + for _, u := range in.Units { + c, ok := callables[u.Key] + if !ok || len(c.CallSites) != len(u.Calls) { // Unit.Calls is built index by index from CallSites + continue + } + for i, cs := range c.CallSites { + if cs.CalleeSignature == nil { + continue + } + sig := *cs.CalleeSignature + site := siteKey{u.Key, u.Calls[i].Key} + if cs.CalleeImportPath != "" { + d.add(site, Target{External: sig}) + } else if k, ok := bySig[sig]; ok { + d.add(site, Target{Callable: k}) + } + } + } + return d +} + +// targets returns the dispatch targets of the call site callKey of siteUnit, +// sorted by Target (Callable Path, Signature, then External): the static L2 +// callee ∪ the VTA callees, and for an interface call always the interface +// method (D7). The slice is shared: callers must not modify it. +func (d *dispatch) targets(siteUnit dataflow.L3Key, callKey string) []Target { + return d.sites[siteKey{siteUnit, callKey}] +} + +func (d *dispatch) add(site siteKey, t Target) { + ts := d.sites[site] + i, found := slices.BinarySearchFunc(ts, t, compareTarget) + if !found { + d.sites[site] = slices.Insert(ts, i, t) + } +} + +// addInvokes binds every interface call of a body-joined project function to +// its interface method (D7), whether or not L2 resolved it and whether or not +// VTA finds a concrete target. It reads instructions only, so it is safe +// before VTA. +func (d *dispatch) addInvokes(j *join, funcs map[*ssa.Function]bool) { + for fn := range funcs { + forCalls(fn, func(call ssa.CallInstruction) { + cc := call.Common() + if !cc.IsInvoke() { + return + } + unit, key, ok := j.callSite(call) + if !ok { + return + } + if t, ok := d.methodTarget(j, cc.Method); ok { + d.add(siteKey{unit, key}, t) + } + }) + } +} + +// methodTarget is the target of the interface method m: the symbol table's +// interface method for a project interface, else the external of a library +// one. A method of a universe type (error.Error) or of an interface the +// symbol table does not record (an interface literal) has no target. +func (d *dispatch) methodTarget(j *join, m *types.Func) (Target, bool) { + if m == nil || m.Pkg() == nil { + return Target{}, false + } + if j.isLibrary(m) { + sig := semantic_analysis.CalleeSignatureOf(m) + d.external(m, sig) + return Target{External: sig}, true + } + if k, ok := j.declKey(m.Origin()); ok && j.iface[k] { + return Target{Callable: k}, true + } + return Target{}, false +} + +// addVTA adds VTA's callees: every edge whose site joins a call node and +// whose callee maps to a target. +func (d *dispatch) addVTA(j *join, cg *callgraph.Graph) { + for _, node := range cg.Nodes { + for _, e := range node.Out { + if e.Site == nil || e.Callee == nil || e.Callee.Func == nil { + continue + } + unit, key, ok := j.callSite(e.Site) + if !ok { + continue + } + if t, ok := d.calleeTarget(j, e.Callee.Func); ok { + d.add(siteKey{unit, key}, t) + } + } + } +} + +// calleeTarget maps an SSA callee to its target: the unit it joins (by body +// or as a wrapper; an instance joins its origin), an interface method, a +// bodiless declaration, or a library function's external. An unmapped project +// function is no target. +func (d *dispatch) calleeTarget(j *join, fn *ssa.Function) (Target, bool) { + if k, ok := j.fnUnit(fn); ok { + return Target{Callable: k}, true + } + if k, ok := j.ifaceMethod(fn); ok { + return Target{Callable: k}, true + } + if k, ok := j.bodilessDecl(fn); ok { + return Target{Callable: k}, true + } + if sig, ok := j.externalOf(fn); ok { + d.external(fn.Object().(*types.Func), sig) + return Target{External: sig}, true + } + return Target{}, false +} + +// external records the library callee obj (full signature sig) when L2 did +// not, with L2's own conversion, so its entry is the one L2 would have made. +func (d *dispatch) external(obj *types.Func, sig string) { + if _, ok := d.l2Ext[sig]; ok { + return + } + if _, ok := d.newExt[sig]; ok { + return + } + d.newExt[sig] = semantic_analysis.ExternalCallee(obj, sig) +} + +// result is the Result of the dispatch: one call_graph edge per (caller, +// target) pair that L2 lacks, prov ["vta"], weight 1 — Source the caller's +// signature and SourcePath its declaring file, as L2 has them (closures +// included), Target the callee signature (the full external signature for an +// external) and TargetPath the callee's declaring file (empty for an +// external), since a closure of an init has a per-file signature — sorted by +// (SourcePath, Source, Target, TargetPath); and the externals only VTA +// reached. +func (d *dispatch) result(deg []Degraded) Result { + type edgeID struct{ srcPath, src, dst, dstPath string } + seen := map[edgeID]bool{} + var ids []edgeID + for site, ts := range d.sites { + for _, t := range ts { + id := edgeID{site.unit.Path, site.unit.Signature, t.External, ""} + if t.External == "" { + id.dst, id.dstPath = t.Callable.Signature, t.Callable.Path + } + if d.l2[l2Edge{id.srcPath, id.src, id.dst}] || seen[id] { + continue + } + seen[id] = true + ids = append(ids, id) + } + } + slices.SortFunc(ids, func(a, b edgeID) int { + return cmp.Or(cmp.Compare(a.srcPath, b.srcPath), cmp.Compare(a.src, b.src), + cmp.Compare(a.dst, b.dst), cmp.Compare(a.dstPath, b.dstPath)) + }) + res := Result{Degraded: deg} + for _, id := range ids { + res.CallEdges = append(res.CallEdges, schema.GoCallEdge{ + Source: id.src, + Target: id.dst, + Type: "CALL_DEP", + Weight: 1, + Provenance: []string{"vta"}, + Tags: map[string]string{}, + SourcePath: id.srcPath, + TargetPath: id.dstPath, + }) + } + if len(d.newExt) > 0 { + res.NewExternals = d.newExt + } + return res +} + +// ApplyTo merges the L4 call-graph additions into app (spec §4.8): the VTA +// edges into app.CallGraph (semantic_analysis.MergeEdges: union prov, sum +// weight per edge identity) and the VTA-only externals into app.Externals, so +// every new edge's target resolves. Both are additions: -a 3 ⊆ -a 4. +func (r *Result) ApplyTo(app *schema.GoApplication) { + if len(r.CallEdges) > 0 { + app.CallGraph = semantic_analysis.MergeEdges(app.CallGraph, r.CallEdges) + } + if len(r.NewExternals) > 0 && app.Externals == nil { + app.Externals = make(map[string]schema.GoExternalCallee, len(r.NewExternals)) + } + for sig, ext := range r.NewExternals { + if _, ok := app.Externals[sig]; !ok { + app.Externals[sig] = ext + } + } +} diff --git a/internal/sdg/vta_test.go b/internal/sdg/vta_test.go new file mode 100644 index 0000000..ddec8e4 --- /dev/null +++ b/internal/sdg/vta_test.go @@ -0,0 +1,327 @@ +package sdg + +import ( + "cmp" + "path/filepath" + "slices" + "testing" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" +) + +const sdgPkg = "example.com/dataflow/sdg" + +var sdgFile = filepath.Join("sdg", "sdg.go") + +// sdgKey is the L3Key of a callable declared in testdata/dataflow/sdg/sdg.go. +func sdgKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: sdgFile, Signature: sdgPkg + "." + name} +} + +func sdgTarget(name string) Target { return Target{Callable: sdgKey(name)} } + +// runDispatch runs L4 over testdata/dataflow and returns its result and the +// call-site dispatch. +func runDispatch(t *testing.T) (Input, Result, *dispatch) { + t.Helper() + in := loadInput(t, "dataflow") + res, d := run(in) + if len(res.Degraded) != 0 { + t.Fatalf("degraded = %+v, want none", res.Degraded) + } + return in, res, d +} + +// hasEdge reports whether edges holds src → dst (src declared in sdg.go) +// with prov ["vta"] and weight 1. +func hasVTAEdge(edges []schema.GoCallEdge, src, dst string) bool { + for _, e := range edges { + if e.Source == src && e.Target == dst && e.SourcePath == sdgFile { + return slices.Equal(e.Provenance, []string{"vta"}) && e.Weight == 1 + } + } + return false +} + +func hasEdgeTo(edges []schema.GoCallEdge, src, dst string) bool { + for _, e := range edges { + if e.Source == src && e.Target == dst { + return true + } + } + return false +} + +// TestVTA_InterfaceTargets: UseStore's s.Get(k) dispatches to the interface +// method (D7) and to both implementations; memStore's pointer wrapper (from +// UseStore(&memStore{})) maps to memStore.Get. Only the concrete targets are +// new call_graph edges: L2 already has UseStore → Store.Get. +func TestVTA_InterfaceTargets(t *testing.T) { + _, res, d := runDispatch(t) + got := d.targets(sdgKey("UseStore"), "34:9") + want := []Target{sdgTarget("Store.Get"), sdgTarget("memStore.Get"), sdgTarget("sqlStore.Get")} + if !slices.Equal(got, want) { + t.Fatalf("targets(UseStore, 34:9) = %v, want %v", got, want) + } + src := sdgPkg + ".UseStore" + for _, dst := range []string{sdgPkg + ".memStore.Get", sdgPkg + ".sqlStore.Get"} { + if !hasVTAEdge(res.CallEdges, src, dst) { + t.Errorf("CallEdges lack %s → %s (prov [vta], weight 1): %+v", src, dst, res.CallEdges) + } + } + if hasEdgeTo(res.CallEdges, src, sdgPkg+".Store.Get") { + t.Errorf("CallEdges repeat the L2 edge %s → Store.Get", src) + } +} + +// TestVTA_FuncValueTargets: CallFuncValue's f(1) has no static callee; VTA +// resolves it to double, the function PassDouble passes. +func TestVTA_FuncValueTargets(t *testing.T) { + _, res, d := runDispatch(t) + got := d.targets(sdgKey("CallFuncValue"), "51:9") + if want := []Target{sdgTarget("double")}; !slices.Equal(got, want) { + t.Fatalf("targets(CallFuncValue, 51:9) = %v, want %v", got, want) + } + if !hasVTAEdge(res.CallEdges, sdgPkg+".CallFuncValue", sdgPkg+".double") { + t.Errorf("CallEdges lack CallFuncValue → double: %+v", res.CallEdges) + } + // A static call keeps its one L2 target. + got = d.targets(sdgKey("PassDouble"), "56:9") + if want := []Target{sdgTarget("CallFuncValue")}; !slices.Equal(got, want) { + t.Errorf("targets(PassDouble, 56:9) = %v, want %v", got, want) + } +} + +// TestVTA_PromotedAndWrapperTargetsMap (Review Focus 3): VTA returns the +// synthetic wrappers — Outer's promotion wrapper and *Outer's +// promotion-and-indirection wrapper for o.M(), *memStore's pointer wrapper for +// s.Get(k) — and each maps to the declared method, not to nothing. +func TestVTA_PromotedAndWrapperTargetsMap(t *testing.T) { + in := loadInput(t, "dataflow") + prog, pkgs, _ := buildSSA(in) + _, all, err := funcSet(prog, pkgs) + if err != nil { + t.Fatal(err) + } + cg, err := runVTA(all) + if err != nil { + t.Fatal(err) + } + j := newJoin(in) + raw := func(unit dataflow.L3Key, key string) []*ssa.Function { + var out []*ssa.Function + for _, node := range cg.Nodes { + for _, e := range node.Out { + if u, k, ok := j.callSite(e.Site); ok && u == unit && k == key { + out = append(out, e.Callee.Func) + } + } + } + return out + } + for _, c := range []struct { + unit dataflow.L3Key + key string + wrappers int + want dataflow.L3Key + }{ + {sdgKey("CallPromoted"), "80:9", 2, sdgKey("Inner.M")}, + {sdgKey("UseStore"), "34:9", 1, sdgKey("memStore.Get")}, + } { + n := 0 + for _, fn := range raw(c.unit, c.key) { + if fn.Synthetic == "" { + continue + } + n++ + if k, ok := j.fnUnit(fn); !ok || k != c.want { + t.Errorf("wrapper %s [%s] maps to %v, %v; want %v", fn, fn.Synthetic, k, ok, c.want) + } + } + if n != c.wrappers { + t.Errorf("%v %s: %d wrapper callees, want %d (callees %v)", c.unit, c.key, n, c.wrappers, raw(c.unit, c.key)) + } + } + + _, res, d := runDispatch(t) + got := d.targets(sdgKey("CallPromoted"), "80:9") + if want := []Target{sdgTarget("Inner.M"), sdgTarget("Mer.M")}; !slices.Equal(got, want) { + t.Fatalf("targets(CallPromoted, 80:9) = %v, want %v", got, want) + } + if !hasVTAEdge(res.CallEdges, sdgPkg+".CallPromoted", sdgPkg+".Inner.M") { + t.Errorf("CallEdges lack CallPromoted → Inner.M: %+v", res.CallEdges) + } +} + +// TestVTA_LibraryTargetBecomesExternal (A19): Buffered's w.Write(p) reaches +// (*bytes.Buffer).Write only through VTA. The target is the external +// bytes.Buffer.Write, recorded in NewExternals with its receiver, beside the +// interface method io.Writer.Write (D7), which L2 already recorded. +func TestVTA_LibraryTargetBecomesExternal(t *testing.T) { + in, res, d := runDispatch(t) + got := d.targets(sdgKey("Buffered"), "93:2") + if want := []Target{{External: "bytes.Buffer.Write"}, {External: "io.Writer.Write"}}; !slices.Equal(got, want) { + t.Fatalf("targets(Buffered, 93:2) = %v, want %v", got, want) + } + if _, ok := in.Externals["bytes.Buffer.Write"]; ok { + t.Fatal("L2 already records bytes.Buffer.Write; the fixture no longer pins a VTA-only external") + } + if _, ok := in.Externals["io.Writer.Write"]; !ok { + t.Fatal("L2 does not record io.Writer.Write") + } + ext, ok := res.NewExternals["bytes.Buffer.Write"] + if !ok { + t.Fatalf("NewExternals lack bytes.Buffer.Write: %v", res.NewExternals) + } + if ext.ImportPath != "bytes" || ext.Sig != "Buffer.Write" || ext.Name != "Write" || ext.ReturnType != "(int, error)" { + t.Errorf("bytes.Buffer.Write = %+v", ext) + } + if len(ext.Params) != 1 || ext.Params[0].Name != "p" || ext.Params[0].Type != "[]byte" { + t.Errorf("bytes.Buffer.Write params = %+v, want [p []byte]", ext.Params) + } + if ext.Receiver == nil || ext.Receiver.Name != "b" || ext.Receiver.Type != "*bytes.Buffer" { + t.Errorf("bytes.Buffer.Write receiver = %+v, want b *bytes.Buffer", ext.Receiver) + } + if _, ok := res.NewExternals["io.Writer.Write"]; ok { + t.Error("NewExternals repeat the L2 external io.Writer.Write") + } + for sig := range res.NewExternals { + if _, ok := in.Externals[sig]; ok { + t.Errorf("NewExternals repeat the L2 external %s", sig) + } + } + if !hasVTAEdge(res.CallEdges, sdgPkg+".Buffered", "bytes.Buffer.Write") { + t.Errorf("CallEdges lack Buffered → bytes.Buffer.Write: %+v", res.CallEdges) + } +} + +// TestVTA_EdgesNewSortedAndResolvable: every VTA edge is new (not an L2 +// (caller, callee) pair), the list is sorted and duplicate-free, its source +// is a unit, and its target is a symbol-table callable or an external +// (L2's or a new one). +func TestVTA_EdgesNewSortedAndResolvable(t *testing.T) { + in, res, _ := runDispatch(t) + if len(res.CallEdges) == 0 { + t.Fatal("no VTA edges") + } + type key struct{ path, src, dst, dstPath string } + l2 := map[key]bool{} + for _, e := range in.CallGraph { + l2[key{e.SourcePath, e.Source, e.Target, ""}] = true + } + units := map[dataflow.L3Key]bool{} + for _, u := range in.Units { + units[u.Key] = true + } + sigs := map[string]bool{} // every callable the emitter indexes, closures included + var walk func(c schema.GoCallable) + walk = func(c schema.GoCallable) { + sigs[c.Signature] = true + for _, ic := range c.InnerCallables { + walk(ic) + } + } + for _, f := range in.SymbolTable { + for _, c := range f.Functions { + walk(c) + } + for _, ty := range f.Types { + for _, c := range ty.Methods { + walk(c) + } + } + } + for i, e := range res.CallEdges { + k := key{e.SourcePath, e.Source, e.Target, e.TargetPath} + if l2[key{k.path, k.src, k.dst, ""}] { + t.Errorf("edge %+v is already in L2", e) + } + if i > 0 { + p := res.CallEdges[i-1] + pk := key{p.SourcePath, p.Source, p.Target, p.TargetPath} + if c := cmp.Or(cmp.Compare(pk.path, k.path), cmp.Compare(pk.src, k.src), cmp.Compare(pk.dst, k.dst), cmp.Compare(pk.dstPath, k.dstPath)); c >= 0 { + t.Errorf("edges %+v, %+v out of order or duplicated", p, e) + } + } + if !units[dataflow.L3Key{Path: e.SourcePath, Signature: e.Source}] { + t.Errorf("edge source %s (%s) is not a unit", e.Source, e.SourcePath) + } + _, l2ext := in.Externals[e.Target] + _, newExt := res.NewExternals[e.Target] + if !sigs[e.Target] && !l2ext && !newExt { + t.Errorf("edge target %s resolves to nothing", e.Target) + } + } +} + +// TestVTA_Deterministic: two runs give the same targets, edges and externals. +func TestVTA_Deterministic(t *testing.T) { + _, a, da := runDispatch(t) + _, b, db := runDispatch(t) + if !slices.EqualFunc(a.CallEdges, b.CallEdges, func(x, y schema.GoCallEdge) bool { + return x.Source == y.Source && x.Target == y.Target && x.SourcePath == y.SourcePath && x.TargetPath == y.TargetPath && + slices.Equal(x.Provenance, y.Provenance) && x.Weight == y.Weight + }) { + t.Error("CallEdges differ between runs") + } + if len(da.sites) != len(db.sites) { + t.Fatalf("%d vs %d sites", len(da.sites), len(db.sites)) + } + for s, ts := range da.sites { + if !slices.Equal(ts, db.sites[s]) { + t.Errorf("site %v: %v vs %v", s, ts, db.sites[s]) + } + } + if len(a.NewExternals) != len(b.NewExternals) { + t.Errorf("%d vs %d new externals", len(a.NewExternals), len(b.NewExternals)) + } +} + +// TestVTA_NoVTAKeepsStaticTargets: when VTA fails, every site keeps its L2 +// target (the interface method for an interface call) and no edge is added. +func TestVTA_NoVTAKeepsStaticTargets(t *testing.T) { + in := loadInput(t, "dataflow") + defer func(f func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph) { vtaCallGraph = f }(vtaCallGraph) + vtaCallGraph = func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph { panic("vta boom") } + res, d := run(in) + if len(res.CallEdges) != 0 || len(res.NewExternals) != 0 { + t.Errorf("no_vta run added %d edges, %d externals", len(res.CallEdges), len(res.NewExternals)) + } + if got, want := d.targets(sdgKey("UseStore"), "34:9"), []Target{sdgTarget("Store.Get")}; !slices.Equal(got, want) { + t.Errorf("targets(UseStore, 34:9) = %v, want %v", got, want) + } + if got, want := d.targets(sdgKey("Buffered"), "93:2"), []Target{{External: "io.Writer.Write"}}; !slices.Equal(got, want) { + t.Errorf("targets(Buffered, 93:2) = %v, want %v", got, want) + } + if got := d.targets(sdgKey("CallFuncValue"), "51:9"); len(got) != 0 { + t.Errorf("targets(CallFuncValue, 51:9) = %v, want none", got) + } +} + +// TestVTA_InitClosureTargetsKeepTheirFile: Run's h() reaches the closure of +// each file's init. Both share the signature initclosure.init.closure_1, so +// each target, and each edge's TargetPath, names its own declaring file. +func TestVTA_InitClosureTargetsKeepTheirFile(t *testing.T) { + _, res, d := runDispatch(t) + const pkg = "example.com/dataflow/initclosure" + a, b := filepath.Join("initclosure", "a.go"), filepath.Join("initclosure", "b.go") + sig := pkg + ".init.closure_1" + got := d.targets(dataflow.L3Key{Path: a, Signature: pkg + ".Run"}, "16:8") + want := []Target{{Callable: dataflow.L3Key{Path: a, Signature: sig}}, {Callable: dataflow.L3Key{Path: b, Signature: sig}}} + if !slices.Equal(got, want) { + t.Fatalf("targets(Run, 16:8) = %v, want %v", got, want) + } + files := map[string]bool{} + for _, e := range res.CallEdges { + if e.Source == pkg+".Run" && e.Target == sig { + files[e.TargetPath] = true + } + } + if !files[a] || !files[b] || len(files) != 2 { + t.Errorf("Run → %s edges have TargetPaths %v, want %s and %s", sig, files, a, b) + } +} diff --git a/internal/semantic_analysis/call_graph.go b/internal/semantic_analysis/call_graph.go index 73343ba..7619193 100644 --- a/internal/semantic_analysis/call_graph.go +++ b/internal/semantic_analysis/call_graph.go @@ -247,8 +247,6 @@ func (cg *CallGraphBuilder) modulePath() string { } // recordExternal registers a library callee, keyed by its full signature. -// Parameter names and types come from the callee's *types.Signature; unnamed -// parameters become "_". The receiver is not a parameter. func (cg *CallGraphBuilder) recordExternal(fn *types.Func, fullSig string) { if _, ok := cg.externals[fullSig]; ok { return @@ -256,6 +254,15 @@ func (cg *CallGraphBuilder) recordExternal(fn *types.Func, fullSig string) { if cg.externals == nil { cg.externals = map[string]schema.GoExternalCallee{} } + cg.externals[fullSig] = ExternalCallee(fn, fullSig) +} + +// ExternalCallee is the external_symbols entry of the library callee fn, whose +// full signature (CalleeSignatureOf) is fullSig. L2 and the L4 VTA pass both +// build their entries with it, so an external has one shape whichever found it. +// Parameter names and types come from the callee's *types.Signature; unnamed +// parameters become "_". The receiver is not a parameter. +func ExternalCallee(fn *types.Func, fullSig string) schema.GoExternalCallee { importPath := fn.Pkg().Path() // Record the declared signature, not the instance the call site selected: a // method on a generic library type (atomic.Pointer[A].Store) must yield one @@ -299,7 +306,7 @@ func (cg *CallGraphBuilder) recordExternal(fn *types.Func, fullSig string) { receiver = &schema.GoParameter{Name: name, Type: types.TypeString(recv.Type(), nil)} } - cg.externals[fullSig] = schema.GoExternalCallee{ + return schema.GoExternalCallee{ ImportPath: importPath, Sig: strings.TrimPrefix(fullSig, importPath+"."), Name: fn.Name(), @@ -382,14 +389,17 @@ func (cg *CallGraphBuilder) packageForFile(relPath string) *packages.Package { } // MergeEdges merges two edge lists, unioning provenance and accumulating weight -// for duplicate (source, target) pairs. Mirrors Python's merge_edges(). +// for duplicate edges. Mirrors Python's merge_edges(), with the edge identity +// (SourcePath, Source, Target, TargetPath). A signature is not unique (every +// file's func init shares ".init", and so do its closures), so the +// declaring files keep two inits' edges apart. func MergeEdges(primary, secondary []schema.GoCallEdge) []schema.GoCallEdge { - type key struct{ src, tgt string } + type key struct{ path, src, tgt, tgtPath string } index := map[key]int{} result := make([]schema.GoCallEdge, 0, len(primary)+len(secondary)) add := func(e schema.GoCallEdge) { - k := key{e.Source, e.Target} + k := key{e.SourcePath, e.Source, e.Target, e.TargetPath} if idx, exists := index[k]; exists { // Merge provenance (union) and accumulate weight. provSet := map[string]bool{} diff --git a/internal/semantic_analysis/call_graph_test.go b/internal/semantic_analysis/call_graph_test.go index 2357407..8aae710 100644 --- a/internal/semantic_analysis/call_graph_test.go +++ b/internal/semantic_analysis/call_graph_test.go @@ -120,6 +120,30 @@ func TestMergeEdges_OrderPreserved(t *testing.T) { } } +// TestMergeEdges_KeepsDeclaringFilesApart: two inits (one signature, two +// files) calling one callee are two edges, and so are two VTA edges to the +// same-signature closures of two files' inits; the same file pair merges. +func TestMergeEdges_KeepsDeclaringFilesApart(t *testing.T) { + at := func(e schema.GoCallEdge, src, dst string) schema.GoCallEdge { + e.SourcePath, e.TargetPath = src, dst + return e + } + primary := []schema.GoCallEdge{at(edge("p.init", "p.setup", 1, "go/types"), "a.go", "")} + secondary := []schema.GoCallEdge{ + at(edge("p.init", "p.setup", 1, "go/types"), "b.go", ""), + at(edge("p.Run", "p.init.closure_1", 1, "vta"), "a.go", "a.go"), + at(edge("p.Run", "p.init.closure_1", 1, "vta"), "a.go", "b.go"), + at(edge("p.init", "p.setup", 1, "vta"), "a.go", ""), + } + result := semantic_analysis.MergeEdges(primary, secondary) + if len(result) != 4 { + t.Fatalf("got %d edges, want 4: %+v", len(result), result) + } + if result[0].Weight != 2 || len(result[0].Provenance) != 2 { + t.Errorf("a.go init edge = %+v, want weight 2, prov [go/types vta]", result[0]) + } +} + // ── Call-site index ─────────────────────────────────────────────────────────── // TestCallGraph_IndexMatchesWalk locks the chi edge set (see chiGoldenEdges for the delta vs the walk) after diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 8317d26..322b20e 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -449,10 +449,15 @@ func emitCallGraph(edges []schema.GoCallEdge, idx sigIndex) []v2.Edge { // Resolve the source by (signature, declaring file): a package-level // init signature repeats once per file in a package, so the file is // what pins the edge to the init that actually contains the call site. - // The target stays signature-only — a call target is never an init - // (Go forbids calling init), and every other signature is unique. + // The target is signature-only unless the edge carries its declaring + // file: a static call target is never an init (Go forbids calling + // init), but a VTA target may be a closure of one, whose signature + // repeats per file. src, srcOK := idx.idForWithPath(e.Source, e.SourcePath) dst, dstOK := idx.resolve(e.Target) + if e.TargetPath != "" { + dst, dstOK = idx.idForWithPath(e.Target, e.TargetPath) + } if !srcOK || !dstOK { continue } diff --git a/internal/syntactic_analysis/v2emit/gate_l4_test.go b/internal/syntactic_analysis/v2emit/gate_l4_test.go index 6408f1f..93f591c 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_test.go @@ -2,20 +2,24 @@ package v2emit // L4 conformance harness for the v2 emitter. // -// At this stage -a 4 reuses the -a 3 pipeline unchanged: the tree equals -a 3 -// except for max_level. buildL4 runs that pipeline with maxLevel 4 and an -// empty sdg result; later tasks fill the L4 content in, and the monotonicity -// gate below keeps holding as they do (-a 3 ⊆ -a 4). +// buildL4 runs the -a 4 pipeline: the -a 3 one, then sdg.Run, whose VTA +// call_graph edges and VTA-only externals are merged into the application +// before the L4 payload is emitted (as core does). Later tasks fill in more L4 +// content, and the monotonicity gate below keeps holding as they do +// (-a 3 ⊆ -a 4). import ( + "encoding/json" "path/filepath" "runtime" + "slices" "sync" "testing" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" ) @@ -53,21 +57,43 @@ func buildL4(t *testing.T, name string) *l4Fixture { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} - res := dataflow.Run(b.Fset(), b.L3Units(st), 3, 4) - fx := &l4Fixture{ - st: st, - l3: Emit(app, name, dir, 3, "gate-test", 3, res), - l4: Emit(app, name, dir, 4, "gate-test", 3, res), - res: res, - } + units := b.L3Units(st) + res := dataflow.RunWithSideTables(b.Fset(), units, 3, 4) + fx := &l4Fixture{st: st, l3: Emit(app, name, dir, 3, "gate-test", 3, res), res: res} + l4 := sdg.Run(sdg.Input{ + Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Externals: app.Externals, CallGraph: app.CallGraph, + Units: units, L3: res, K: 3, Jobs: 4, + }) + l4.ApplyTo(app) + fx.l4 = Emit(app, name, dir, 4, "gate-test", 3, res) l4Cache[name] = fx return fx } // supersetL4 records where a (-a 3) is not contained in b (-a 4): every object -// key and value, and every list entry. Nothing may be added yet either, so the -// check is exact; later tasks widen it with the sanctioned L4 additions. +// key and value, and every list entry, by position — except call_graph, a +// sorted edge list that VTA adds edges into, where every -a 3 edge must occur +// unchanged somewhere in the -a 4 list. func supersetL4(path string, a, b any, errs *[]string) { + if path == "$.application.call_graph" { + av, aok := a.([]any) + bv, bok := b.([]any) + if !aok || !bok { + *errs = append(*errs, path+": changed type") + return + } + have := map[string]bool{} + for _, e := range bv { + have[jsonKey(e)] = true + } + for _, e := range av { + if !have[jsonKey(e)] { + *errs = append(*errs, path+": edge "+jsonKey(e)+" missing or changed at L4") + } + } + return + } switch av := a.(type) { case map[string]any: bv, ok := b.(map[string]any) @@ -121,3 +147,81 @@ func TestGateL4_SupersetOfL3(t *testing.T) { } } } + +func jsonKey(v any) string { + data, _ := json.Marshal(v) + return string(data) +} + +// TestGateL4_CallGraphHasVTAProv: at -a 4 the call graph gains VTA's dispatch +// edges (spec §4.8), prov ["vta"], weight 1, each one new: the concrete +// implementations of an interface call, the function passed to a func-value +// call, a promoted method, and a library method only VTA reaches, whose +// external (with its receiver) joins external_symbols. Every endpoint +// resolves, and the L2 edges are unchanged (the superset gate). +func TestGateL4_CallGraphHasVTAProv(t *testing.T) { + fx := buildL4(t, "dataflow") + app := fx.l4.Application + mod := v2.ModuleID(app.ID, "sdg/sdg.go") + fn := func(sig string) string { + return v2.CallableID(mod, "example.com/dataflow/sdg."+sig) + } + method := func(typ, m string) string { + return v2.CallableID(v2.TypeID(mod, "example.com/dataflow/sdg."+typ), "example.com/dataflow/sdg."+typ+"."+m) + } + bufWrite := v2.ExternalID(app.ID, "bytes", "Buffer.Write") + want := [][2]string{ + {fn("UseStore"), method("memStore", "Get")}, + {fn("UseStore"), method("sqlStore", "Get")}, + {fn("CallFuncValue"), fn("double")}, + {fn("CallPromoted"), method("Inner", "M")}, + {fn("Buffered"), bufWrite}, + } + l3 := map[[2]string]bool{} + for _, e := range fx.l3.Application.CallGraph { + l3[[2]string{e.Src, e.Dst}] = true + } + vta := map[[2]string]v2.Edge{} + for _, e := range app.CallGraph { + if slices.Contains(e.Prov, "vta") { + k := [2]string{e.Src, e.Dst} + if !slices.Equal(e.Prov, []string{"vta"}) || e.Weight != 1 || l3[k] { + t.Errorf("VTA edge %+v: want prov [vta], weight 1, not an L3 edge", e) + } + vta[k] = e + } + } + for _, k := range want { + if _, ok := vta[k]; !ok { + t.Errorf("no VTA edge %s -> %s", k[0], k[1]) + } + } + if l3[[2]string{fn("UseStore"), method("Store", "Get")}] == false { + t.Errorf("L2 edge UseStore -> Store.Get missing") + } + if _, ok := fx.l3.Application.ExternalSymbols[bufWrite]; ok { + t.Fatalf("-a 3 already has %s; the fixture no longer pins a VTA-only external", bufWrite) + } + ext, ok := app.ExternalSymbols[bufWrite] + if !ok { + t.Fatalf("external_symbols lack %s", bufWrite) + } + if ext.Receiver == nil || ext.Receiver.Name != "b" || ext.Receiver.Type != "*bytes.Buffer" || ext.Signature != "Buffer.Write" { + t.Errorf("%s = %+v (receiver %+v)", bufWrite, ext, ext.Receiver) + } + // Each file's init closure is its own target: the edges keep both. + initClosure := func(file string) string { + m := v2.ModuleID(app.ID, "initclosure/"+file) + init := v2.CallableID(m, "example.com/dataflow/initclosure.init") + return v2.CallableID(init, "example.com/dataflow/initclosure.init.closure_1") + } + run := v2.CallableID(v2.ModuleID(app.ID, "initclosure/a.go"), "example.com/dataflow/initclosure.Run") + for _, file := range []string{"a.go", "b.go"} { + if _, ok := vta[[2]string{run, initClosure(file)}]; !ok { + t.Errorf("no VTA edge Run -> %s's init closure", file) + } + } + for _, name := range []string{"dataflow", "chi", "generics"} { + assertNoDanglingEdges(t, buildL4(t, name).l4) + } +} diff --git a/testdata/dataflow/initclosure/a.go b/testdata/dataflow/initclosure/a.go new file mode 100644 index 0000000..377e27e --- /dev/null +++ b/testdata/dataflow/initclosure/a.go @@ -0,0 +1,19 @@ +// Package initclosure has a func init() with a closure in more than one file. +// The closures share the signature example.com/dataflow/initclosure.init.closure_1, +// so a call edge to one must carry its declaring file. +package initclosure + +var hooks []func() int + +func init() { + hooks = append(hooks, func() int { return 1 }) +} + +// Run calls every hook through a function value. +func Run() int { + n := 0 + for _, h := range hooks { + n += h() + } + return n +} diff --git a/testdata/dataflow/initclosure/b.go b/testdata/dataflow/initclosure/b.go new file mode 100644 index 0000000..be0e13b --- /dev/null +++ b/testdata/dataflow/initclosure/b.go @@ -0,0 +1,5 @@ +package initclosure + +func init() { + hooks = append(hooks, func() int { return 2 }) +} diff --git a/testdata/dataflow/sdg/sdg.go b/testdata/dataflow/sdg/sdg.go new file mode 100644 index 0000000..4d41053 --- /dev/null +++ b/testdata/dataflow/sdg/sdg.go @@ -0,0 +1,100 @@ +// Package sdg is the L4 fixture for the dataflow train: one named function per +// interprocedural construct. Tests pin keys by line:col, so keep each +// statement on its own line and append new functions at the end. +package sdg + +import ( + "bytes" + "io" +) + +// Store is an interface with two implementations. +type Store interface { + Get(k string) string +} + +type memStore struct { + m map[string]string +} + +func (s memStore) Get(k string) string { + return s.m[k] +} + +type sqlStore struct { + dsn string +} + +func (s *sqlStore) Get(k string) string { + return s.dsn + k +} + +// UseStore dispatches s.Get through the interface. +func UseStore(s Store, k string) string { + return s.Get(k) +} + +// Stores passes both implementations to UseStore, memStore also by pointer +// (its pointer wrapper maps to memStore.Get). +func Stores() { + UseStore(memStore{}, "a") + UseStore(&memStore{}, "b") + UseStore(&sqlStore{}, "c") +} + +func double(x int) int { + return x * 2 +} + +// CallFuncValue calls f through a function value. +func CallFuncValue(f func(int) int) int { + return f(1) +} + +// PassDouble passes double to CallFuncValue. +func PassDouble() int { + return CallFuncValue(double) +} + +// Inner's method M is promoted to Outer. +type Inner struct { + n int +} + +func (i Inner) M() int { + return i.n +} + +// Outer embeds Inner. +type Outer struct { + Inner +} + +// Mer is satisfied by Outer and *Outer through the promoted M. +type Mer interface { + M() int +} + +// CallPromoted calls M through the interface. +func CallPromoted(o Mer) int { + return o.M() +} + +// PassOuter passes an Outer and an *Outer: the targets are the promotion +// wrapper and the promotion-and-pointer wrapper, both of Inner.M. +func PassOuter() int { + a := CallPromoted(Outer{}) + b := CallPromoted(&Outer{}) + return a + b +} + +// Buffered writes through an io.Writer. +func Buffered(w io.Writer, p []byte) { + w.Write(p) +} + +// PassBuffer passes a *bytes.Buffer: (*bytes.Buffer).Write is a target only +// VTA finds. +func PassBuffer() { + Buffered(&bytes.Buffer{}, nil) +} From cd38f260f4bb75dc22a965cfd27c4e92c8ffc823 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 07:34:49 -0400 Subject: [PATCH 14/65] feat(l4): global and mutation footprints over call-graph SCCs --- internal/dataflow/accesspath.go | 7 + internal/dataflow/defuse_calls.go | 15 +- internal/sdg/footprint.go | 783 ++++++++++++++++++++++++++++++ internal/sdg/footprint_test.go | 222 +++++++++ internal/sdg/scc.go | 85 ++++ testdata/dataflow/sdg/globals.go | 30 ++ testdata/dataflow/sdg/sdg.go | 41 ++ 7 files changed, 1182 insertions(+), 1 deletion(-) create mode 100644 internal/sdg/footprint.go create mode 100644 internal/sdg/footprint_test.go create mode 100644 internal/sdg/scc.go create mode 100644 testdata/dataflow/sdg/globals.go diff --git a/internal/dataflow/accesspath.go b/internal/dataflow/accesspath.go index c860636..90f2907 100644 --- a/internal/dataflow/accesspath.go +++ b/internal/dataflow/accesspath.go @@ -271,6 +271,13 @@ func baseSpell(v types.Object) string { return v.Name() } +// BaseSpell is baseSpell for package sdg: the L4 global formals and +// footprints spell a package-level variable as its L3 ddg base. +func BaseSpell(v types.Object) string { return baseSpell(v) } + +// IsPkgVar is isPkgVar for package sdg (the L4 global footprints). +func IsPkgVar(obj types.Object) bool { return isPkgVar(obj) } + // isPkgVar reports whether obj is a package-level variable. It checks // membership of the package scope by identity, which holds for packages // type-checked from source and imported from export data alike. diff --git a/internal/dataflow/defuse_calls.go b/internal/dataflow/defuse_calls.go index abffbe2..9f8ac28 100644 --- a/internal/dataflow/defuse_calls.go +++ b/internal/dataflow/defuse_calls.go @@ -180,8 +180,21 @@ func refTyped(t types.Type) bool { // named type already on the walk is not re-entered (a value-only cycle, // possible only in a package with type errors, holds nothing new). The call // rule and the L4 by-ref formal_out rule share it. +// +// Only a struct or array needs the walk and its cycle set: every other type +// is decided by refTyped alone, with no allocation. func MayHoldRef(t types.Type) bool { - return mayHoldRef(t, map[*types.Named]bool{}) + if t == nil { + return false + } + if refTyped(t) { + return true + } + switch t.Underlying().(type) { + case *types.Struct, *types.Array: + return mayHoldRef(t, map[*types.Named]bool{}) + } + return false } func mayHoldRef(t types.Type, seen map[*types.Named]bool) bool { diff --git a/internal/sdg/footprint.go b/internal/sdg/footprint.go new file mode 100644 index 0000000..bf741cc --- /dev/null +++ b/internal/sdg/footprint.go @@ -0,0 +1,783 @@ +package sdg + +import ( + "fmt" + "go/ast" + "go/token" + "go/types" + "slices" + "strings" + + "golang.org/x/tools/go/packages" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// Footprint is what a call target touches beyond its formals' values (spec +// §4.6, §4.9): the package-level variables it reads and writes, itself or +// through any callee, spelled "pkgpath::Name" (the L3 ddg base spelling), and +// the params and receiver it may write through (by-ref outs), by base name, +// or by the D6 `of` (`_`, `_recv`) for an unnamed one. Each list is sorted +// and deduplicated. Lists may be shared between footprints: callers must not +// modify them. +type Footprint struct { + GlobalReads, GlobalWrites []string + Mutated []string +} + +// globals is a callable's global read and write sets (sorted spellings). +type globals struct{ reads, writes []string } + +// footprints computes the footprint of every unit and of every dispatch +// target (spec §4.9). It reads the L3 side tables, the dispatch and go/types +// only, never the SSA program, so it is safe after a no_vta. +// +// - A unit with an L3 result: its direct globals are the package-level +// variable bases of its DefUse (uses are reads; strong and weak defs are +// writes; @entry, which defines every global the body touches, is not a +// write) and every global it takes the address of (&v, or a pointer +// method value v.M): a write through that address is not a def of v at +// L3. Globals the filter drops (globalFilter, A5) are removed. Its +// footprint is propagated bottom-up over the call graph (callGraph), so +// it holds every global any callee touches. +// - Mutated: a receiver or param p whose type may hold a reference +// (dataflow.MayHoldRef, A10) and that some node other than @entry +// defines a path through (p.f, p.*, p[*]: an assignment, a call-rule weak +// def where p is a by-ref argument or receiver, a capture write). It is +// not propagated: the L3 call rule already weak-defines p.* at a call +// passing p, so the caller's own DefUse records it. +// - A stand-in, an interface method, a bodiless declaration and a unit +// with no L3 result (or no type info): no globals, and Mutated = every +// by-ref formal (byRefFormals). +func footprints(in Input, d *dispatch) map[Target]Footprint { + nodes, callees := callGraph(in, d) + gf := newGlobalFilter(in) + formals := newFormalIndex(in, d) + + direct := map[Target]globals{} + mutated := map[Target][]string{} + for _, u := range in.Units { + if !analysed(in, u) { + gf.mentions(u) // no DefUse: every global it names may be written + continue + } + t := Target{Callable: u.Key} + r := in.L3[u.Key] + direct[t] = gf.direct(u, r) + mutated[t] = mutatedFormals(u, r) + } + gf.initializers() + for t, g := range direct { + direct[t] = globals{reads: gf.keep(g.reads), writes: gf.keep(g.writes)} + } + + prop := propagate(nodes, callees, direct) + out := make(map[Target]Footprint, len(nodes)) + for _, t := range nodes { + if _, ok := direct[t]; ok { + g := prop[t] + out[t] = Footprint{GlobalReads: g.reads, GlobalWrites: g.writes, Mutated: mutated[t]} + continue + } + out[t] = Footprint{Mutated: byRefFormals(formals.of(t))} + } + return out +} + +// analysed reports whether unit u has the L3 facts footprints read: an L3 +// result with its DefUse side table, and type info. +func analysed(in Input, u dataflow.Unit) bool { + r, ok := in.L3[u.Key] + return ok && r.DefUse != nil && u.Info != nil +} + +// callGraph returns the call graph footprints propagate over (spec §4.9): +// every unit and every dispatch target, sorted by compareTarget, and each +// unit's callees, the union of its call sites' dispatch targets (L2's static +// callee ∪ VTA ∪ the interface method, D7), sorted. A call node the CFG +// pruned keeps its targets (an over-approximation). A unit footprints cannot +// analyse (analysed) has no callees: its footprint carries no globals. +func callGraph(in Input, d *dispatch) ([]Target, map[Target][]Target) { + callees := map[Target][]Target{} + nodeSet := map[Target]bool{} + ok := map[dataflow.L3Key]bool{} + for _, u := range in.Units { + nodeSet[Target{Callable: u.Key}] = true + ok[u.Key] = analysed(in, u) + } + for site, ts := range d.sites { + for _, t := range ts { + nodeSet[t] = true + } + if !ok[site.unit] { + continue + } + u := Target{Callable: site.unit} + callees[u] = append(callees[u], ts...) + } + for u, ts := range callees { + slices.SortFunc(ts, compareTarget) + callees[u] = slices.Compact(ts) + } + nodes := make([]Target, 0, len(nodeSet)) + for t := range nodeSet { + nodes = append(nodes, t) + } + slices.SortFunc(nodes, compareTarget) + return nodes, callees +} + +// propagate closes the direct global sets over callees, bottom-up over the +// SCCs (callees first): a target's set is the union of the direct sets of +// every target it reaches, itself included. That is the least fixpoint of +// "a target's set ⊇ its own ∪ each callee's"; every member of an SCC reaches +// every other, so the fixpoint gives all members one set, the union of the +// members' direct sets and of the sets of the SCCs they call (computed +// earlier), and one pass per SCC reaches it. When the union adds nothing to +// one contribution, that contribution's slice is shared rather than copied. +func propagate(nodes []Target, callees map[Target][]Target, direct map[Target]globals) map[Target]globals { + comps := sccs(nodes, func(t Target) []Target { return callees[t] }) + compOf := make(map[Target]int, len(nodes)) + out := make(map[Target]globals, len(nodes)) + for ci, comp := range comps { + for _, m := range comp { + compOf[m] = ci + } + var reads, writes [][]string + for _, m := range comp { + if g, ok := direct[m]; ok { + reads, writes = append(reads, g.reads), append(writes, g.writes) + } + for _, c := range callees[m] { + if compOf[c] != ci { + g := out[c] + reads, writes = append(reads, g.reads), append(writes, g.writes) + } + } + } + g := globals{reads: unionSorted(reads), writes: unionSorted(writes)} + for _, m := range comp { + out[m] = g + } + } + return out +} + +// unionSorted is the sorted union of the sorted, deduplicated lists. It +// returns the largest list itself when the union is no larger, and nil when +// it is empty. +func unionSorted(lists [][]string) []string { + var largest []string + n := 0 + for _, l := range lists { + if len(l) > len(largest) { + largest = l + } + n += len(l) + } + if n == len(largest) { + if len(largest) == 0 { + return nil + } + return largest + } + all := make([]string, 0, n) + for _, l := range lists { + all = append(all, l...) + } + slices.Sort(all) + all = slices.Compact(all) + if len(all) == len(largest) { + return largest + } + return slices.Clip(all) +} + +// mutatedFormals is the Mutated set of unit u from its L3 result r (A10): +// the receiver and params that may hold a reference and that a node other +// than @entry defines a path through, by name, sorted. +func mutatedFormals(u dataflow.Unit, r dataflow.Result) []string { + byRef := map[types.Object]bool{} + for _, obj := range formalObjs(u.Fn, u.Info) { + if dataflow.MayHoldRef(obj.Type()) { + byRef[obj] = true + } + } + if len(byRef) == 0 { + return nil + } + hit := map[string]bool{} + for key, du := range r.DefUse { + if key == dataflow.EntryKey { + continue + } + for _, list := range [][]dataflow.Path{du.Defs, du.WeakDefs} { + for _, p := range list { + // A path with a step after its base: the base's own spelling is + // its name (a param is never a package-level variable). + if byRef[p.Base] && len(p.Spell) > len(p.Base.Name()) { + hit[p.Base.Name()] = true + } + } + } + } + return sortedKeys(hit) +} + +// formalObjs returns the receiver and parameter objects of fn (a +// *ast.FuncDecl or *ast.FuncLit), in declaration order; "_" and unnamed ones +// have none. +func formalObjs(fn ast.Node, info *types.Info) []types.Object { + var lists []*ast.FieldList + switch fn := fn.(type) { + case *ast.FuncDecl: + lists = []*ast.FieldList{fn.Recv, fn.Type.Params} + case *ast.FuncLit: + lists = []*ast.FieldList{fn.Type.Params} + } + var out []types.Object + for _, fl := range lists { + if fl == nil { + continue + } + for _, f := range fl.List { + for _, name := range f.Names { + if obj := info.Defs[name]; obj != nil && name.Name != "_" { + out = append(out, obj) + } + } + } + } + return out +} + +// globalFilter is the A5 global filter (spec §4.6): a package-level +// variable is kept iff it is declared outside the project (os::Args), or +// some project code writes it — a strong or weak def in any unit (init and +// closures included), &v or a pointer method value, a write in a unit L3 +// could not analyse, or a write in a package-level initializer — or its +// initializer contains a call. +type globalFilter struct { + fset *token.FileSet + pkgs []*packages.Package + project map[string]bool // import paths of the project packages + kept map[string]bool // project globals that pass, by spelling + spellings map[types.Object]string +} + +func newGlobalFilter(in Input) *globalFilter { + f := &globalFilter{ + fset: in.Fset, + pkgs: projectPackages(in), + project: map[string]bool{}, + kept: map[string]bool{}, + spellings: map[types.Object]string{}, + } + for _, p := range f.pkgs { + f.project[p.PkgPath] = true + } + return f +} + +// spell returns the spelling of obj when it is a package-level variable. +func (f *globalFilter) spell(obj types.Object) (string, bool) { + if s, ok := f.spellings[obj]; ok { + return s, s != "" + } + s := "" + if dataflow.IsPkgVar(obj) { + s = dataflow.BaseSpell(obj) + } + f.spellings[obj] = s + return s, s != "" +} + +// write records a write of obj when it is a project global. +func (f *globalFilter) write(obj types.Object) { + if s, ok := f.spell(obj); ok && f.project[obj.Pkg().Path()] { + f.kept[s] = true + } +} + +// keep returns the globals of list (sorted spellings) the filter keeps. +func (f *globalFilter) keep(list []string) []string { + var out []string + for _, s := range list { + pkg, _, _ := strings.Cut(s, "::") + if !f.project[pkg] || f.kept[s] { + out = append(out, s) + } + } + return out +} + +// direct returns the unfiltered direct globals of unit u (see footprints) +// and records its writes of project globals. +func (f *globalFilter) direct(u dataflow.Unit, r dataflow.Result) globals { + reads, writes := map[string]bool{}, map[string]bool{} + read := func(obj types.Object) { + if s, ok := f.spell(obj); ok { + reads[s] = true + } + } + write := func(obj types.Object) { + if s, ok := f.spell(obj); ok { + writes[s] = true + f.write(obj) + } + } + for key, du := range r.DefUse { + if key == dataflow.EntryKey { + continue + } + for _, p := range du.Uses { + read(p.Base) + } + for _, list := range [][]dataflow.Path{du.Defs, du.WeakDefs} { + for _, p := range list { + write(p.Base) + } + } + } + addressTaken(u.Fn, u.Info, write) + return globals{reads: sortedKeys(reads), writes: sortedKeys(writes)} +} + +// addressTaken calls f on the base of every &x and of every pointer method +// value x.M (a call x.M() included) whose receiver x is not a pointer: the +// address may be written through. It walks fn's body, not its function +// literals: they are their own units. +func addressTaken(fn ast.Node, info *types.Info, f func(types.Object)) { + var body *ast.BlockStmt + switch fn := fn.(type) { + case *ast.FuncDecl: + body = fn.Body + case *ast.FuncLit: + body = fn.Body + } + if body == nil { + return + } + base := func(e ast.Expr) { + if p, ok := dataflow.PathOf(e, info, 1); ok && p.Base != nil { + f(p.Base) + } + } + ast.Inspect(body, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.FuncLit: + return false + case *ast.UnaryExpr: + if n.Op == token.AND { + base(n.X) + } + case *ast.SelectorExpr: + sel := info.Selections[n] + if sel == nil || sel.Kind() != types.MethodVal { + break + } + if m, ok := sel.Obj().(*types.Func); ok { + if recv := m.Signature().Recv(); recv != nil && isPtr(recv.Type()) && !isPtr(info.TypeOf(n.X)) { + base(n.X) + } + } + } + return true + }) +} + +func isPtr(t types.Type) bool { + if t == nil { + return false + } + _, ok := t.Underlying().(*types.Pointer) + return ok +} + +// mentions records every project global unit u (nested literals included) +// names as written: with no DefUse, any of them may be. +func (f *globalFilter) mentions(u dataflow.Unit) { + if u.Info == nil { + // No type info: nothing to resolve a name against. Keep every + // global the unit's package declares. + f.keepPackageOf(u) + return + } + ast.Inspect(u.Fn, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + f.write(u.Info.Uses[id]) + } + return true + }) +} + +// keepPackageOf keeps every package-level variable of the project package +// that declares u. +func (f *globalFilter) keepPackageOf(u dataflow.Unit) { + tf := f.fset.File(u.Fn.Pos()) + if tf == nil { + return + } + for _, p := range f.pkgs { + if !slices.Contains(p.GoFiles, tf.Name()) || p.Types == nil { + continue + } + scope := p.Types.Scope() + for _, name := range scope.Names() { + f.write(scope.Lookup(name)) + } + } +} + +// initializers applies the filter's rules to the package-level variable +// declarations of every project package: a variable whose initializer +// contains a call (not a conversion or a builtin, and not in a function +// literal's body: that runs only when the literal is called) is kept; so is +// every project global an initializer may write — one named inside a call's +// operands, under &, or in a function literal (L1 has no callable for a +// package-level literal, so no DefUse says what it writes). +func (f *globalFilter) initializers() { + for _, p := range f.pkgs { + info := p.TypesInfo + if info == nil { + continue + } + for _, file := range p.Syntax { + for _, decl := range file.Decls { + gd, ok := decl.(*ast.GenDecl) + if !ok || gd.Tok != token.VAR { + continue + } + for _, spec := range gd.Specs { + vs := spec.(*ast.ValueSpec) + for _, v := range vs.Values { + if hasCall(v, info) { + for _, name := range vs.Names { + f.write(info.Defs[name]) + } + } + f.initWrites(v, info, false) + } + } + } + } + } +} + +// hasCall reports whether e calls a function outside a function literal's +// body: a conversion or a builtin call does not count, but its operands are +// searched. +func hasCall(e ast.Expr, info *types.Info) bool { + found := false + ast.Inspect(e, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.FuncLit: + return false + case *ast.CallExpr: + if tv, ok := info.Types[n.Fun]; !ok || !(tv.IsType() || tv.IsBuiltin()) { + found = true + } + } + return !found + }) + return found +} + +// initWrites records as written every project global n names in a hot +// position: inside a call (its function value and operands), under &, or in +// a function literal. +func (f *globalFilter) initWrites(n ast.Node, info *types.Info, hot bool) { + ast.Inspect(n, func(m ast.Node) bool { + switch m := m.(type) { + case *ast.Ident: + if hot { + f.write(info.Uses[m]) + } + case *ast.CallExpr: + if !hot { + f.initWrites(m, info, true) + return false + } + case *ast.FuncLit: + f.initWrites(m.Body, info, true) + return false + case *ast.UnaryExpr: + if m.Op == token.AND && !hot { + f.initWrites(m.X, info, true) + return false + } + } + return true + }) +} + +// formal is one receiver or declared parameter of a call target: its D6 +// `of` name and its type (nil when only the type's string is known). +type formal struct { + name string + typ types.Type + typStr string + recv bool + variadic bool +} + +// byRefFormals returns the names of the formals that are by-ref (the shared +// predicate dataflow.MayHoldRef on a known type, else refTypeString), +// sorted. +func byRefFormals(fs []formal) []string { + var out []string + for _, f := range fs { + if f.typ != nil && dataflow.MayHoldRef(f.typ) || f.typ == nil && (f.variadic || refTypeString(f.typStr)) { + out = append(out, f.name) + } + } + slices.Sort(out) + return out +} + +// refTypeString is the by-ref predicate on a type known only by its string +// (spec §4.6, the conservative fallback): false only for a basic type, or an +// array of one; a named type that cannot be resolved may hold a reference. +func refTypeString(s string) bool { + for strings.HasPrefix(s, "[") { + i := strings.IndexByte(s, ']') + if i <= 1 { // "[]T": a slice + return true + } + s = s[i+1:] + } + if b, ok := types.Universe.Lookup(s).(*types.TypeName); ok { + if _, basic := b.Type().(*types.Basic); basic { + return false + } + } + return true +} + +// formalIndex finds the formals of call targets, with their go/types types +// wherever the load has them: +// +// - a unit: its Fn's signature (types.Info); +// - an interface method or a bodiless declaration (no unit): the +// *types.Func a project package's scope declares under that key; +// - an external: the names from its GoExternalCallee (already D6), the +// types from the *types.Func its import path's package declares, found +// through the load's import graph (every external is reached from +// project code, so its package is loaded, from source or export data). +// +// When no type is found, the symbol table's or the external's type strings +// are used with refTypeString. +type formalIndex struct { + in Input + units map[dataflow.L3Key]dataflow.Unit + decls map[dataflow.L3Key]*types.Func // project funcs and methods by (declaring file, signature) + strs map[dataflow.L3Key]schema.GoCallable + ext map[string]schema.GoExternalCallee + pkgByPath map[string]*types.Package +} + +func newFormalIndex(in Input, d *dispatch) *formalIndex { + x := &formalIndex{ + in: in, + units: make(map[dataflow.L3Key]dataflow.Unit, len(in.Units)), + decls: map[dataflow.L3Key]*types.Func{}, + strs: map[dataflow.L3Key]schema.GoCallable{}, + ext: map[string]schema.GoExternalCallee{}, + pkgByPath: map[string]*types.Package{}, + } + for _, u := range in.Units { + x.units[u.Key] = u + } + for sig, e := range in.Externals { + x.ext[sig] = e + } + for sig, e := range d.newExt { + x.ext[sig] = e + } + var walk func(path string, c schema.GoCallable) + walk = func(path string, c schema.GoCallable) { + x.strs[dataflow.L3Key{Path: path, Signature: c.Signature}] = c + for _, ic := range c.InnerCallables { + walk(ic.Path, ic) + } + } + for path, f := range in.SymbolTable { + for _, c := range f.Functions { + walk(c.Path, c) + } + for _, t := range f.Types { + for _, c := range t.Methods { + if c.Path == "" { // an interface method, keyed by its interface's file + walk(path, c) + continue + } + walk(c.Path, c) + } + } + } + j := &join{fset: in.Fset, projectDir: in.ProjectDir} // declKey reads only these two + for _, p := range projectPackages(in) { + if p.Types == nil { + continue + } + scope := p.Types.Scope() + for _, name := range scope.Names() { + switch obj := scope.Lookup(name).(type) { + case *types.Func: + x.addDecl(j, obj) + case *types.TypeName: + named, ok := obj.Type().(*types.Named) + if !ok { + continue + } + for i := 0; i < named.NumMethods(); i++ { + x.addDecl(j, named.Method(i)) + } + if it, ok := named.Underlying().(*types.Interface); ok { + for i := 0; i < it.NumExplicitMethods(); i++ { + x.addDecl(j, it.ExplicitMethod(i)) + } + } + } + } + } + var visit func(p *packages.Package) + visit = func(p *packages.Package) { + if p == nil || p.Types == nil || x.pkgByPath[p.PkgPath] != nil { + return + } + x.pkgByPath[p.PkgPath] = p.Types + for _, q := range p.Imports { + visit(q) + } + } + for _, p := range in.Pkgs { + visit(p) + } + return x +} + +func (x *formalIndex) addDecl(j *join, fn *types.Func) { + if k, ok := j.declKey(fn); ok { + x.decls[k] = fn + } +} + +// of returns t's receiver (first) and declared params. +func (x *formalIndex) of(t Target) []formal { + if t.External != "" { + return x.external(t.External) + } + if u, ok := x.units[t.Callable]; ok && u.Info != nil { + if sig := unitSignature(u); sig != nil { + return sigFormals(sig) + } + } + if fn, ok := x.decls[t.Callable]; ok { + return sigFormals(fn.Signature()) + } + c, ok := x.strs[t.Callable] + if !ok { + utils.Debug("L4: no formals for %v", t) + return nil + } + var out []formal + if c.ReceiverType != "" { + out = append(out, formal{name: recvName(c.ReceiverName), typStr: c.ReceiverType, recv: true}) + } + for i, p := range c.Parameters { + out = append(out, formal{name: paramName(p.Name, i), typStr: p.Type, variadic: p.IsVariadic}) + } + return out +} + +// external returns the formals of the external with full signature sig. +func (x *formalIndex) external(sig string) []formal { + e, ok := x.ext[sig] + if !ok { + utils.Debug("L4: no external %s", sig) + return nil + } + var out []formal + if e.Receiver != nil { + out = append(out, formal{name: e.Receiver.Name, typStr: e.Receiver.Type, recv: true}) + } + for _, p := range e.Params { + out = append(out, formal{name: p.Name, typStr: p.Type, variadic: p.IsVariadic}) + } + if fn := x.libraryFunc(e); fn != nil { + if typed := sigFormals(fn.Signature()); len(typed) == len(out) { + for i := range out { + out[i].typ = typed[i].typ + } + } + } + return out +} + +// libraryFunc looks up e's *types.Func: a function of its package's scope, +// or a method (Sig "Type.Method") of a type it declares. +func (x *formalIndex) libraryFunc(e schema.GoExternalCallee) *types.Func { + pkg := x.pkgByPath[e.ImportPath] + if pkg == nil { + return nil + } + typ, method, isMethod := strings.Cut(e.Sig, ".") + if !isMethod { + fn, _ := pkg.Scope().Lookup(e.Sig).(*types.Func) + return fn + } + tn, ok := pkg.Scope().Lookup(typ).(*types.TypeName) + if !ok { + return nil + } + obj, _, _ := types.LookupFieldOrMethod(tn.Type(), true, pkg, method) + fn, _ := obj.(*types.Func) + return fn +} + +// unitSignature is the signature of u's Fn. +func unitSignature(u dataflow.Unit) *types.Signature { + switch fn := u.Fn.(type) { + case *ast.FuncDecl: + if f, ok := u.Info.Defs[fn.Name].(*types.Func); ok { + return f.Signature() + } + case *ast.FuncLit: + sig, _ := u.Info.TypeOf(fn).(*types.Signature) + return sig + } + return nil +} + +// sigFormals returns sig's receiver (an interface method's receiver is its +// interface) and params, named by D6. +func sigFormals(sig *types.Signature) []formal { + var out []formal + if r := sig.Recv(); r != nil { + out = append(out, formal{name: recvName(r.Name()), typ: r.Type(), recv: true}) + } + for i := 0; i < sig.Params().Len(); i++ { + p := sig.Params().At(i) + out = append(out, formal{name: paramName(p.Name(), i), typ: p.Type(), variadic: sig.Variadic() && i == sig.Params().Len()-1}) + } + return out +} + +// recvName and paramName are the D6 `of` of a receiver and of declared +// param i (receiver excluded). +func recvName(name string) string { + if name == "" || name == "_" { + return "_recv" + } + return name +} + +func paramName(name string, i int) string { + if name == "" || name == "_" { + return fmt.Sprintf("_%d", i) + } + return name +} diff --git a/internal/sdg/footprint_test.go b/internal/sdg/footprint_test.go new file mode 100644 index 0000000..b47edf1 --- /dev/null +++ b/internal/sdg/footprint_test.go @@ -0,0 +1,222 @@ +package sdg + +import ( + "math/rand" + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +const configSpell = sdgPkg + "::Config" + +// globalsKey is the L3Key of a callable declared in +// testdata/dataflow/sdg/globals.go. +func globalsKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("sdg", "globals.go"), Signature: sdgPkg + "." + name} +} + +func runFootprints(t *testing.T) (Input, *dispatch, map[Target]Footprint) { + t.Helper() + in, _, d := runDispatch(t) + return in, d, footprints(in, d) +} + +func fpOf(t *testing.T, fps map[Target]Footprint, tg Target) Footprint { + t.Helper() + fp, ok := fps[tg] + if !ok { + t.Fatalf("no footprint for %v", tg) + } + return fp +} + +// TestFootprint_TransitiveGlobalReads: useConfig never names Config, but it +// calls readConfig, which reads it; initConfig writes it. +func TestFootprint_TransitiveGlobalReads(t *testing.T) { + _, _, fps := runFootprints(t) + want := []string{configSpell} + if got := fpOf(t, fps, sdgTarget("useConfig")); !slices.Equal(got.GlobalReads, want) || len(got.GlobalWrites) != 0 { + t.Errorf("useConfig = %+v, want reads %v, no writes", got, want) + } + if got := fpOf(t, fps, sdgTarget("initConfig")); !slices.Equal(got.GlobalWrites, want) || len(got.GlobalReads) != 0 { + t.Errorf("initConfig = %+v, want writes %v, no reads", got, want) + } +} + +// TestFootprint_GlobalFilter (A5): constOnly (literal initializer, never +// written) is dropped everywhere; os::Args (a library variable) is kept, and +// so are home (its initializer calls os.Getenv) and counter (written only +// through &counter). +func TestFootprint_GlobalFilter(t *testing.T) { + _, _, fps := runFootprints(t) + for tg, fp := range fps { + for _, g := range slices.Concat(fp.GlobalReads, fp.GlobalWrites) { + if strings.HasSuffix(g, "::constOnly") { + t.Errorf("%v has filtered global %s", tg, g) + } + } + } + if got := fpOf(t, fps, sdgTarget("readConst")); len(got.GlobalReads) != 0 { + t.Errorf("readConst reads %v, want none", got.GlobalReads) + } + for _, c := range []struct { + fn string + want string + }{ + {"readArgs", "os::Args"}, + {"readHome", sdgPkg + "::home"}, + {"readCounter", sdgPkg + "::counter"}, + } { + if got := fpOf(t, fps, Target{Callable: globalsKey(c.fn)}); !slices.Equal(got.GlobalReads, []string{c.want}) { + t.Errorf("%s reads %v, want [%s]", c.fn, got.GlobalReads, c.want) + } + } +} + +// TestFootprint_MutualRecursionFixpoint: readConfig and useConfig are one +// SCC and agree on its footprint. On a synthetic graph where the fact sits on +// the SCC member sorted last, only a fixpoint gives it to the first one (and +// to a caller of the SCC). +func TestFootprint_MutualRecursionFixpoint(t *testing.T) { + in, d, fps := runFootprints(t) + nodes, callees := callGraph(in, d) + comps := sccs(nodes, func(tg Target) []Target { return callees[tg] }) + want := []Target{sdgTarget("readConfig"), sdgTarget("useConfig")} + if !slices.ContainsFunc(comps, func(c []Target) bool { return slices.Equal(c, want) }) { + t.Errorf("no SCC %v", want) + } + for _, tg := range want { + if got := fpOf(t, fps, tg); !slices.Equal(got.GlobalReads, []string{configSpell}) { + t.Errorf("%v reads %v, want [%s]", tg, got.GlobalReads, configSpell) + } + } + + a, b, c := sdgTarget("a"), sdgTarget("b"), sdgTarget("c") + syn := map[Target][]Target{a: {b}, b: {a}, c: {a}} + direct := map[Target]globals{b: {reads: []string{"p::G"}, writes: []string{"p::H"}}} + got := propagate([]Target{c, b, a}, syn, direct) + for _, tg := range []Target{a, b, c} { + if g := got[tg]; !slices.Equal(g.reads, []string{"p::G"}) || !slices.Equal(g.writes, []string{"p::H"}) { + t.Errorf("%v = %+v, want reads [p::G], writes [p::H]", tg, g) + } + } +} + +// TestFootprint_Mutated (A10): a callable mutates a param or receiver it +// writes through, by the shared by-ref predicate; a stand-in, an interface +// method and a bodiless declaration mutate every by-ref formal. +func TestFootprint_Mutated(t *testing.T) { + _, _, fps := runFootprints(t) + for _, c := range []struct { + tg Target + want []string + }{ + {sdgTarget("mutate"), []string{"p", "s"}}, // n is a copy holding no reference; q is never written + {sdgTarget("sqlStore.setDSN"), []string{"s"}}, + {sdgTarget("memStore.Get"), nil}, // reads through s only + {sdgTarget("Store.Get"), []string{"_recv"}}, // interface method: the receiver is an interface + {Target{External: "io.Writer.Write"}, []string{"_recv", "p"}}, + {Target{External: "bytes.Buffer.Write"}, []string{"b", "p"}}, + {Target{Callable: dataflow.L3Key{Path: filepath.Join("asm", "asm.go"), Signature: "example.com/dataflow/asm.Add"}}, nil}, + } { + if got := fpOf(t, fps, c.tg); !slices.Equal(got.Mutated, c.want) { + t.Errorf("%v mutates %v, want %v", c.tg, got.Mutated, c.want) + } + } +} + +// TestFootprint_L3SkippedMutatesEveryByRefFormal: with its L3 result gone, +// mutate has no direct globals and mutates every by-ref formal (p, s and q; +// n holds no reference). +func TestFootprint_L3SkippedMutatesEveryByRefFormal(t *testing.T) { + in, _, d := runDispatch(t) + l3 := make(map[dataflow.L3Key]dataflow.Result, len(in.L3)) + for k, r := range in.L3 { + if k != sdgKey("mutate") { + l3[k] = r + } + } + in.L3 = l3 + got := fpOf(t, footprints(in, d), sdgTarget("mutate")) + if want := []string{"p", "q", "s"}; !slices.Equal(got.Mutated, want) || len(got.GlobalReads)+len(got.GlobalWrites) != 0 { + t.Errorf("l3-skipped mutate = %+v, want mutated %v, no globals", got, want) + } +} + +// TestSDG_GenericInstancesShareOneCallable (Review Focus 2): the join maps a +// generic instance to its origin, so testdata/dataflow/generic has one +// footprint per origin callable and none per instance. +func TestSDG_GenericInstancesShareOneCallable(t *testing.T) { + in, _, fps := runFootprints(t) + units := map[dataflow.L3Key]bool{} + for _, u := range in.Units { + if strings.HasPrefix(u.Key.Path, "generic"+string(filepath.Separator)) { + units[u.Key] = true + } + } + got := map[dataflow.L3Key]bool{} + for tg := range fps { + if strings.HasPrefix(tg.Callable.Path, "generic"+string(filepath.Separator)) { + got[tg.Callable] = true + } + } + if len(units) == 0 || len(got) != len(units) { + t.Fatalf("generic footprint keys = %v, want the %d units %v", got, len(units), units) + } + for k := range got { + if !units[k] || strings.Contains(k.Signature, "[") { + t.Errorf("footprint key %v is not an origin unit", k) + } + } + twice := dataflow.L3Key{Path: filepath.Join("generic", "b.go"), Signature: "example.com/dataflow/generic.Twice"} + if !units[twice] { + t.Fatalf("no unit %v", twice) + } + storeA := fpOf(t, fps, Target{Callable: dataflow.L3Key{Path: filepath.Join("generic", "a.go"), Signature: "example.com/dataflow/generic.StoreA"}}) + if want := []string{"example.com/dataflow/generic::pa"}; !slices.Equal(storeA.GlobalWrites, want) { + t.Errorf("StoreA writes %v, want %v (pa.Store has a pointer receiver)", storeA.GlobalWrites, want) + } +} + +// TestSCC_Deterministic: shuffling the node order and every successor list +// gives the same SCCs, callees first, each sorted. +func TestSCC_Deterministic(t *testing.T) { + n := func(s string) Target { return Target{External: s} } + edges := map[Target][]Target{ + n("a"): {n("b")}, + n("b"): {n("c"), n("e")}, + n("c"): {n("a"), n("d")}, + n("d"): {}, + n("e"): {n("f")}, + n("f"): {n("e"), n("g")}, + n("g"): {n("g")}, + n("h"): {n("a"), n("x")}, // x is reachable only: not in nodes + } + var nodes []Target + for k := range edges { + nodes = append(nodes, k) + } + slices.SortFunc(nodes, compareTarget) + want := sccs(nodes, func(tg Target) []Target { return edges[tg] }) + wantStr := [][]Target{{n("d")}, {n("g")}, {n("e"), n("f")}, {n("a"), n("b"), n("c")}, {n("x")}, {n("h")}} + if !slices.EqualFunc(want, wantStr, slices.Equal) { + t.Fatalf("sccs = %v, want %v", want, wantStr) + } + rng := rand.New(rand.NewSource(1)) + for i := 0; i < 50; i++ { + shuf := slices.Clone(nodes) + rng.Shuffle(len(shuf), func(i, j int) { shuf[i], shuf[j] = shuf[j], shuf[i] }) + se := map[Target][]Target{} + for k, v := range edges { + v = slices.Clone(v) + rng.Shuffle(len(v), func(i, j int) { v[i], v[j] = v[j], v[i] }) + se[k] = v + } + if got := sccs(shuf, func(tg Target) []Target { return se[tg] }); !slices.EqualFunc(got, want, slices.Equal) { + t.Fatalf("shuffle %d: sccs = %v, want %v", i, got, want) + } + } +} diff --git a/internal/sdg/scc.go b/internal/sdg/scc.go new file mode 100644 index 0000000..8b63133 --- /dev/null +++ b/internal/sdg/scc.go @@ -0,0 +1,85 @@ +package sdg + +import "slices" + +// sccs returns the strongly connected components of the graph nodes and succ +// span (spec §4.9), callees first: a component comes after every component it +// reaches. Nodes only succ reaches are included. Each component is sorted by +// compareTarget. +// +// It is Tarjan's algorithm with an explicit stack, so a deep call chain +// cannot overflow the goroutine stack. The roots are visited in +// compareTarget order and each successor list is sorted and deduplicated +// first, so the result depends on neither the order of nodes nor the order +// succ returns. +func sccs(nodes []Target, succ func(Target) []Target) [][]Target { + roots := slices.Clone(nodes) + slices.SortFunc(roots, compareTarget) + roots = slices.Compact(roots) + + type vertex struct { + index, low int + onStack bool + } + type frame struct { + v Target + succ []Target + next int + } + state := make(map[Target]*vertex, len(roots)) + var stack []Target // Tarjan's stack + var calls []frame // the explicit DFS stack + var out [][]Target + + visit := func(v Target) { + n := len(state) + state[v] = &vertex{index: n, low: n, onStack: true} + stack = append(stack, v) + s := slices.Clone(succ(v)) + slices.SortFunc(s, compareTarget) + calls = append(calls, frame{v: v, succ: slices.Compact(s)}) + } + for _, r := range roots { + if state[r] != nil { + continue + } + visit(r) + for len(calls) > 0 { + top := len(calls) - 1 + f := &calls[top] + sv := state[f.v] + if f.next < len(f.succ) { + w := f.succ[f.next] + f.next++ + switch sw := state[w]; { + case sw == nil: + visit(w) + case sw.onStack: + sv.low = min(sv.low, sw.index) + } + continue + } + v := f.v + calls = calls[:top] + if top > 0 { + p := state[calls[top-1].v] + p.low = min(p.low, sv.low) + } + if sv.low != sv.index { + continue + } + i := len(stack) - 1 + for stack[i] != v { + i-- + } + comp := slices.Clone(stack[i:]) + stack = stack[:i] + for _, w := range comp { + state[w].onStack = false + } + slices.SortFunc(comp, compareTarget) + out = append(out, comp) + } + } + return out +} diff --git a/testdata/dataflow/sdg/globals.go b/testdata/dataflow/sdg/globals.go new file mode 100644 index 0000000..3f85e02 --- /dev/null +++ b/testdata/dataflow/sdg/globals.go @@ -0,0 +1,30 @@ +package sdg + +// The globals of the L4 footprint fixture that need the os import; they live +// apart from sdg.go so that its line:col keys stay put. + +import "os" + +// home's initializer calls a function: the global filter keeps it. +var home = os.Getenv("HOME") + +// counter is written only through its address: the global filter keeps it. +var counter int + +// readArgs reads os.Args, a non-project package variable. +func readArgs() []string { + return os.Args +} + +func readHome() string { + return home +} + +func bump() { + p := &counter + *p = *p + 1 +} + +func readCounter() int { + return counter +} diff --git a/testdata/dataflow/sdg/sdg.go b/testdata/dataflow/sdg/sdg.go index 4d41053..25236d8 100644 --- a/testdata/dataflow/sdg/sdg.go +++ b/testdata/dataflow/sdg/sdg.go @@ -98,3 +98,44 @@ func Buffered(w io.Writer, p []byte) { func PassBuffer() { Buffered(&bytes.Buffer{}, nil) } + +// Config is written only in initConfig and read only in readConfig, which +// recurses through useConfig: useConfig reads it transitively. +var Config string + +// constOnly is initialized from a literal and never written: the global +// filter drops it. +var constOnly = 3 + +func initConfig() { + Config = "on" +} + +func readConfig() string { + if Config == "" { + return useConfig() + } + return Config +} + +func useConfig() string { + return readConfig() +} + +func readConst() int { + return constOnly +} + +// mutate writes through p (a pointer) and s (a value holding a map); n is a +// value copy and q is never written through. +func mutate(p *Inner, s memStore, n Inner, q *Inner) { + p.n = 1 + s.m["k"] = "v" + n.n = 2 + _ = q +} + +// setDSN writes through its pointer receiver. +func (s *sqlStore) setDSN(d string) { + s.dsn = d +} From bc6da6ce17ddf507c1f896a95e4ebd7ed6f43b13 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 08:04:40 -0400 Subject: [PATCH 15/65] fix(l4): footprints over-approximate aliases, skipped units, literals passed to libraries --- internal/sdg/footprint.go | 253 ++++++++++++++++++++++++------- internal/sdg/footprint_test.go | 93 +++++++++++- internal/sdg/vta.go | 4 +- testdata/dataflow/sdg/globals.go | 48 ++++++ testdata/dataflow/sdg/sdg.go | 26 ++++ 5 files changed, 363 insertions(+), 61 deletions(-) diff --git a/internal/sdg/footprint.go b/internal/sdg/footprint.go index bf741cc..5697bf3 100644 --- a/internal/sdg/footprint.go +++ b/internal/sdg/footprint.go @@ -10,6 +10,7 @@ import ( "golang.org/x/tools/go/packages" + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" "github.com/codellm-devkit/codeanalyzer-go/internal/utils" @@ -34,7 +35,7 @@ type globals struct{ reads, writes []string } // target (spec §4.9). It reads the L3 side tables, the dispatch and go/types // only, never the SSA program, so it is safe after a no_vta. // -// - A unit with an L3 result: its direct globals are the package-level +// - A unit with an L3 result (analysed): its direct globals are the package-level // variable bases of its DefUse (uses are reads; strong and weak defs are // writes; @entry, which defines every global the body touches, is not a // write) and every global it takes the address of (&v, or a pointer @@ -42,15 +43,22 @@ type globals struct{ reads, writes []string } // L3. Globals the filter drops (globalFilter, A5) are removed. Its // footprint is propagated bottom-up over the call graph (callGraph), so // it holds every global any callee touches. -// - Mutated: a receiver or param p whose type may hold a reference -// (dataflow.MayHoldRef, A10) and that some node other than @entry -// defines a path through (p.f, p.*, p[*]: an assignment, a call-rule weak -// def where p is a by-ref argument or receiver, a capture write). It is -// not propagated: the L3 call rule already weak-defines p.* at a call -// passing p, so the caller's own DefUse records it. -// - A stand-in, an interface method, a bodiless declaration and a unit -// with no L3 result (or no type info): no globals, and Mutated = every -// by-ref formal (byRefFormals). +// - A unit footprints cannot analyse (an L3 result dropped, l3_skipped; +// no type info): its direct globals are syntactic — every package-level +// variable its Fn names, nested literals included, as both read and +// written (mentions) — and it keeps its callees (callGraph). +// - Mutated (A10, fix I1): for an analysed unit, a receiver or param p +// whose type may hold a reference (dataflow.MayHoldRef) and that some +// node other than @entry uses or defines. A use is enough: a write +// through a local alias (`q := p; q.x = v`, a range variable over p) or +// passing the alias on is no def of p at L3, and no alias analysis +// tells them apart. A by-ref formal never used stays out. It is not +// propagated: a caller passing p by ref uses p, and a closure writing +// through a captured p makes its host use p (captureUses), so the +// encloser's own DefUse records it. +// - A stand-in, an interface method, a bodiless declaration and an +// unanalysed unit: Mutated = every by-ref formal (byRefFormals); only +// the unanalysed unit has globals. func footprints(in Input, d *dispatch) map[Target]Footprint { nodes, callees := callGraph(in, d) gf := newGlobalFilter(in) @@ -59,11 +67,12 @@ func footprints(in Input, d *dispatch) map[Target]Footprint { direct := map[Target]globals{} mutated := map[Target][]string{} for _, u := range in.Units { + t := Target{Callable: u.Key} if !analysed(in, u) { - gf.mentions(u) // no DefUse: every global it names may be written + direct[t] = gf.mentions(u) // no DefUse: every global it names may be read or written + mutated[t] = byRefFormals(formals.of(t)) continue } - t := Target{Callable: u.Key} r := in.L3[u.Key] direct[t] = gf.direct(u, r) mutated[t] = mutatedFormals(u, r) @@ -94,29 +103,65 @@ func analysed(in Input, u dataflow.Unit) bool { } // callGraph returns the call graph footprints propagate over (spec §4.9): -// every unit and every dispatch target, sorted by compareTarget, and each -// unit's callees, the union of its call sites' dispatch targets (L2's static -// callee ∪ VTA ∪ the interface method, D7), sorted. A call node the CFG -// pruned keeps its targets (an over-approximation). A unit footprints cannot -// analyse (analysed) has no callees: its footprint carries no globals. +// every unit and every target, sorted by compareTarget, and each unit's +// callees, sorted: +// +// - the union of its call sites' dispatch targets (L2's static callee ∪ +// VTA ∪ the interface method, D7). A call node the CFG pruned keeps its +// targets (an over-approximation); +// - for an unanalysed unit dispatch seeded no site for (fix I2), its L2 +// call_graph edges (in.CallGraph), each target mapped as newDispatch +// maps a static callee; +// - every function literal of its body (outermost: a nested one is its +// enclosing literal's) not called at its own position (fix I3): one +// handed to library code (once.Do, errgroup.Go, http.HandleFunc, +// sort.Slice) or stored may run during the call, and no call edge +// reaches it. This mirrors the L3 capture-write rule. A literal called +// in place (`func(){…}()`, deferred or launched) has its call edge. func callGraph(in Input, d *dispatch) ([]Target, map[Target][]Target) { callees := map[Target][]Target{} nodeSet := map[Target]bool{} - ok := map[dataflow.L3Key]bool{} + byFn := make(map[ast.Node]dataflow.L3Key, len(in.Units)) for _, u := range in.Units { nodeSet[Target{Callable: u.Key}] = true - ok[u.Key] = analysed(in, u) + byFn[u.Fn] = u.Key } + seeded := map[dataflow.L3Key]bool{} for site, ts := range d.sites { for _, t := range ts { nodeSet[t] = true } - if !ok[site.unit] { - continue - } + seeded[site.unit] = true u := Target{Callable: site.unit} callees[u] = append(callees[u], ts...) } + l2 := map[dataflow.L3Key][]string{} + for _, e := range in.CallGraph { + k := dataflow.L3Key{Path: e.SourcePath, Signature: e.Source} + l2[k] = append(l2[k], e.Target) + } + for _, u := range in.Units { + src := Target{Callable: u.Key} + if !seeded[u.Key] && !analysed(in, u) { + for _, sig := range l2[u.Key] { + t, ok := Target{External: sig}, true + if _, ext := in.Externals[sig]; !ext { + var k dataflow.L3Key + k, ok = d.bySig[sig] + t = Target{Callable: k} + } + if ok { + nodeSet[t] = true + callees[src] = append(callees[src], t) + } + } + } + for _, lit := range uncalledLiterals(u.Fn) { + if k, ok := byFn[lit]; ok { + callees[src] = append(callees[src], Target{Callable: k}) + } + } + } for u, ts := range callees { slices.SortFunc(ts, compareTarget) callees[u] = slices.Compact(ts) @@ -129,6 +174,43 @@ func callGraph(in Input, d *dispatch) ([]Target, map[Target][]Target) { return nodes, callees } +// uncalledLiterals returns the outermost function literals of fn's body +// that are not the function of a call (`func(){…}()`, also deferred or +// launched), in source order. +func uncalledLiterals(fn ast.Node) []*ast.FuncLit { + body := fnBody(fn) + if body == nil { + return nil + } + called := map[*ast.FuncLit]bool{} + ast.Inspect(body, func(n ast.Node) bool { + if call, ok := n.(*ast.CallExpr); ok { + if lit, ok := ast.Unparen(call.Fun).(*ast.FuncLit); ok { + called[lit] = true + } + } + return true + }) + var out []*ast.FuncLit + for _, l := range astwalk.Literals(body) { + if !called[l.Lit] { + out = append(out, l.Lit) + } + } + return out +} + +// fnBody is the body of a *ast.FuncDecl or *ast.FuncLit (nil without one). +func fnBody(fn ast.Node) *ast.BlockStmt { + switch fn := fn.(type) { + case *ast.FuncDecl: + return fn.Body + case *ast.FuncLit: + return fn.Body + } + return nil +} + // propagate closes the direct global sets over callees, bottom-up over the // SCCs (callees first): a target's set is the union of the direct sets of // every target it reaches, itself included. That is the least fixpoint of @@ -195,9 +277,9 @@ func unionSorted(lists [][]string) []string { return slices.Clip(all) } -// mutatedFormals is the Mutated set of unit u from its L3 result r (A10): -// the receiver and params that may hold a reference and that a node other -// than @entry defines a path through, by name, sorted. +// mutatedFormals is the Mutated set of analysed unit u from its L3 result r +// (A10, fix I1): the receiver and params that may hold a reference and that +// a node other than @entry uses or defines, by name, sorted. func mutatedFormals(u dataflow.Unit, r dataflow.Result) []string { byRef := map[types.Object]bool{} for _, obj := range formalObjs(u.Fn, u.Info) { @@ -213,11 +295,9 @@ func mutatedFormals(u dataflow.Unit, r dataflow.Result) []string { if key == dataflow.EntryKey { continue } - for _, list := range [][]dataflow.Path{du.Defs, du.WeakDefs} { + for _, list := range [][]dataflow.Path{du.Uses, du.Defs, du.WeakDefs} { for _, p := range list { - // A path with a step after its base: the base's own spelling is - // its name (a param is never a package-level variable). - if byRef[p.Base] && len(p.Spell) > len(p.Base.Name()) { + if byRef[p.Base] { hit[p.Base.Name()] = true } } @@ -350,13 +430,7 @@ func (f *globalFilter) direct(u dataflow.Unit, r dataflow.Result) globals { // address may be written through. It walks fn's body, not its function // literals: they are their own units. func addressTaken(fn ast.Node, info *types.Info, f func(types.Object)) { - var body *ast.BlockStmt - switch fn := fn.(type) { - case *ast.FuncDecl: - body = fn.Body - case *ast.FuncLit: - body = fn.Body - } + body := fnBody(fn) if body == nil { return } @@ -396,26 +470,36 @@ func isPtr(t types.Type) bool { return ok } -// mentions records every project global unit u (nested literals included) -// names as written: with no DefUse, any of them may be. -func (f *globalFilter) mentions(u dataflow.Unit) { +// mentions returns the syntactic direct globals of unanalysed unit u (fix +// I2): every package-level variable its Fn names, nested literals included, +// as both read and written (with no DefUse, any of them may be either), and +// records the writes. With no type info nothing resolves a name, so it is +// every package-level variable of u's package. +func (f *globalFilter) mentions(u dataflow.Unit) globals { + set := map[string]bool{} + note := func(obj types.Object) { + if s, ok := f.spell(obj); ok { + set[s] = true + f.write(obj) + } + } if u.Info == nil { - // No type info: nothing to resolve a name against. Keep every - // global the unit's package declares. - f.keepPackageOf(u) - return + f.packageVars(u, note) + } else { + ast.Inspect(u.Fn, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + note(u.Info.Uses[id]) + } + return true + }) } - ast.Inspect(u.Fn, func(n ast.Node) bool { - if id, ok := n.(*ast.Ident); ok { - f.write(u.Info.Uses[id]) - } - return true - }) + all := sortedKeys(set) + return globals{reads: all, writes: all} } -// keepPackageOf keeps every package-level variable of the project package -// that declares u. -func (f *globalFilter) keepPackageOf(u dataflow.Unit) { +// packageVars calls note on every object of the scope of the project +// package that declares u. +func (f *globalFilter) packageVars(u dataflow.Unit, note func(types.Object)) { tf := f.fset.File(u.Fn.Pos()) if tf == nil { return @@ -426,7 +510,7 @@ func (f *globalFilter) keepPackageOf(u dataflow.Unit) { } scope := p.Types.Scope() for _, name := range scope.Names() { - f.write(scope.Lookup(name)) + note(scope.Lookup(name)) } } } @@ -437,8 +521,13 @@ func (f *globalFilter) keepPackageOf(u dataflow.Unit) { // literal's body: that runs only when the literal is called) is kept; so is // every project global an initializer may write — one named inside a call's // operands, under &, or in a function literal (L1 has no callable for a -// package-level literal, so no DefUse says what it writes). +// package-level literal, so no DefUse says what it writes). Last, a chained +// initializer (fix M4): a variable whose initializer names a library +// package-level variable or a kept one is kept, to a fixpoint (`var argv = +// os.Args`; `var cfgDir = home + "/x"` with home kept). func (f *globalFilter) initializers() { + type chain struct{ names, named []types.Object } + var chains []chain for _, p := range f.pkgs { info := p.TypesInfo if info == nil { @@ -452,18 +541,70 @@ func (f *globalFilter) initializers() { } for _, spec := range gd.Specs { vs := spec.(*ast.ValueSpec) + c := chain{} + for _, name := range vs.Names { + c.names = append(c.names, info.Defs[name]) + } for _, v := range vs.Values { if hasCall(v, info) { - for _, name := range vs.Names { - f.write(info.Defs[name]) + for _, obj := range c.names { + f.write(obj) } } f.initWrites(v, info, false) + c.named = append(c.named, namedVars(v, info)...) + } + if len(c.named) > 0 { + chains = append(chains, c) } } } } } + for changed := true; changed; { + changed = false + for _, c := range chains { + if !slices.ContainsFunc(c.named, f.carries) || !slices.ContainsFunc(c.names, f.dropped) { + continue + } + for _, obj := range c.names { + f.write(obj) + } + changed = true + } + } +} + +// carries reports whether package-level variable obj passes the filter as +// it stands: a library variable, or a kept project one. +func (f *globalFilter) carries(obj types.Object) bool { + s, ok := f.spell(obj) + return ok && (!f.project[obj.Pkg().Path()] || f.kept[s]) +} + +// dropped reports whether obj is a project package-level variable the +// filter does not keep (yet). +func (f *globalFilter) dropped(obj types.Object) bool { + s, ok := f.spell(obj) + return ok && f.project[obj.Pkg().Path()] && !f.kept[s] +} + +// namedVars returns the package-level variables e names outside a function +// literal's body (a literal's body runs only when it is called). +func namedVars(e ast.Expr, info *types.Info) []types.Object { + var out []types.Object + ast.Inspect(e, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.FuncLit: + return false + case *ast.Ident: + if obj := info.Uses[n]; dataflow.IsPkgVar(obj) { + out = append(out, obj) + } + } + return true + }) + return out } // hasCall reports whether e calls a function outside a function literal's diff --git a/internal/sdg/footprint_test.go b/internal/sdg/footprint_test.go index b47edf1..5fcaa1b 100644 --- a/internal/sdg/footprint_test.go +++ b/internal/sdg/footprint_test.go @@ -105,8 +105,8 @@ func TestFootprint_MutualRecursionFixpoint(t *testing.T) { } } -// TestFootprint_Mutated (A10): a callable mutates a param or receiver it -// writes through, by the shared by-ref predicate; a stand-in, an interface +// TestFootprint_Mutated (A10, fix I1): a callable mutates a by-ref param or +// receiver (the shared predicate) it uses or writes; a stand-in, an interface // method and a bodiless declaration mutate every by-ref formal. func TestFootprint_Mutated(t *testing.T) { _, _, fps := runFootprints(t) @@ -114,9 +114,9 @@ func TestFootprint_Mutated(t *testing.T) { tg Target want []string }{ - {sdgTarget("mutate"), []string{"p", "s"}}, // n is a copy holding no reference; q is never written + {sdgTarget("mutate"), []string{"p", "q", "s"}}, // n is a copy holding no reference; q is used (`_ = q`) {sdgTarget("sqlStore.setDSN"), []string{"s"}}, - {sdgTarget("memStore.Get"), nil}, // reads through s only + {sdgTarget("memStore.Get"), []string{"s"}}, // s holds a map and is used: a use is enough (I1) {sdgTarget("Store.Get"), []string{"_recv"}}, // interface method: the receiver is an interface {Target{External: "io.Writer.Write"}, []string{"_recv", "p"}}, {Target{External: "bytes.Buffer.Write"}, []string{"b", "p"}}, @@ -220,3 +220,88 @@ func TestSCC_Deterministic(t *testing.T) { } } } + +// TestFootprint_MutatedThroughAlias (fix I1): a by-ref formal used at any +// node other than @entry is mutated — a write through a local alias (a range +// variable, q := p) or passing the alias on is not a def of the formal at L3. +// One never used is not. +func TestFootprint_MutatedThroughAlias(t *testing.T) { + _, _, fps := runFootprints(t) + for _, c := range []struct { + fn string + want []string + }{ + {"aliasRange", []string{"s"}}, + {"aliasAssign", []string{"p"}}, + {"aliasCall", []string{"p"}}, + {"unusedRef", nil}, + } { + if got := fpOf(t, fps, sdgTarget(c.fn)); !slices.Equal(got.Mutated, c.want) { + t.Errorf("%s mutates %v, want %v", c.fn, got.Mutated, c.want) + } + } +} + +// TestFootprint_L3SkippedKeepsGlobalsAndCallees (fix I2): with skipX's L3 +// result gone, skipX still has the global it names (g1, as read and written) +// and its callee skipY's (g2), and so does its caller. +func TestFootprint_L3SkippedKeepsGlobalsAndCallees(t *testing.T) { + in, _, d := runDispatch(t) + skipped := globalsKey("skipX") + l3 := make(map[dataflow.L3Key]dataflow.Result, len(in.L3)) + for k, r := range in.L3 { + if k != skipped { + l3[k] = r + } + } + in.L3 = l3 + fps := footprints(in, d) + g1, g2 := sdgPkg+"::g1", sdgPkg+"::g2" + for _, k := range []dataflow.L3Key{skipped, globalsKey("callsSkipX")} { + got := fpOf(t, fps, Target{Callable: k}) + if !slices.Equal(got.GlobalReads, []string{g1, g2}) || !slices.Contains(got.GlobalWrites, g1) { + t.Errorf("%s = %+v, want reads [%s %s] and writes ⊇ [%s]", k.Signature, got, g1, g2, g1) + } + } + + // With no dispatch sites either (a call-count mismatch leaves + // newDispatch nothing for it), the L2 call graph still gives its callees. + for site := range d.sites { + if site.unit == skipped { + delete(d.sites, site) + } + } + got := fpOf(t, footprints(in, d), Target{Callable: globalsKey("callsSkipX")}) + if !slices.Contains(got.GlobalReads, g2) { + t.Errorf("callsSkipX reads %v without skipX's dispatch sites, want ⊇ [%s]", got.GlobalReads, g2) + } +} + +// TestFootprint_LiteralPassedToLibrary (fix I3): Get hands a literal to +// once.Do, which never calls it, so no call edge reaches the literal; the +// encloser → literal edge still gives Get and its caller the literal's write +// of inst. +func TestFootprint_LiteralPassedToLibrary(t *testing.T) { + _, _, fps := runFootprints(t) + inst := sdgPkg + "::inst" + for _, fn := range []string{"Get", "callsGet"} { + if got := fpOf(t, fps, Target{Callable: globalsKey(fn)}); !slices.Contains(got.GlobalWrites, inst) { + t.Errorf("%s writes %v, want ⊇ [%s]", fn, got.GlobalWrites, inst) + } + } +} + +// TestFootprint_ChainedInitializers (fix M4): a global initialized from a +// library variable (argv = os.Args) or from a kept global (cfgDir = home + +// "/x") is kept. +func TestFootprint_ChainedInitializers(t *testing.T) { + _, _, fps := runFootprints(t) + for _, c := range []struct{ fn, want string }{ + {"readArgv", sdgPkg + "::argv"}, + {"readCfgDir", sdgPkg + "::cfgDir"}, + } { + if got := fpOf(t, fps, Target{Callable: globalsKey(c.fn)}); !slices.Equal(got.GlobalReads, []string{c.want}) { + t.Errorf("%s reads %v, want [%s]", c.fn, got.GlobalReads, c.want) + } + } +} diff --git a/internal/sdg/vta.go b/internal/sdg/vta.go index 4b24942..1e4006a 100644 --- a/internal/sdg/vta.go +++ b/internal/sdg/vta.go @@ -31,7 +31,8 @@ type l2Edge struct{ path, src, dst string } // wrapper and the method it wraps, are one target. A callee that maps to // nothing (unmapped_ssa, already reported) is not a target. type dispatch struct { - sites map[siteKey][]Target // each sorted by compareTarget, no duplicates + sites map[siteKey][]Target // each sorted by compareTarget, no duplicates + bySig map[string]dataflow.L3Key // symbol-table callable of each signature (init excepted) l2 map[l2Edge]bool l2Ext map[string]schema.GoExternalCallee // in.Externals, read-only newExt map[string]schema.GoExternalCallee // externals L2 never saw @@ -94,6 +95,7 @@ func newDispatch(in Input) *dispatch { } } + d.bySig = bySig for _, u := range in.Units { c, ok := callables[u.Key] if !ok || len(c.CallSites) != len(u.Calls) { // Unit.Calls is built index by index from CallSites diff --git a/testdata/dataflow/sdg/globals.go b/testdata/dataflow/sdg/globals.go index 3f85e02..ec4dc53 100644 --- a/testdata/dataflow/sdg/globals.go +++ b/testdata/dataflow/sdg/globals.go @@ -28,3 +28,51 @@ func bump() { func readCounter() int { return counter } + +// g1 and g2 are read by skipX and skipY; tests force skipX to have no L3 +// result. +var g1 = os.Getenv("G1") + +var g2 = os.Getenv("G2") + +func skipX() string { + return g1 + skipY() +} + +func skipY() string { + return g2 +} + +func callsSkipX() string { + return skipX() +} + +// lazyOnce stands for library code that runs a callback later: Do keeps f +// and never calls it, so no call edge reaches Get's literal. +type lazyOnce struct{ f func() } + +func (o *lazyOnce) Do(f func()) { o.f = f } + +var once lazyOnce + +var inst *Inner + +func build() *Inner { return &Inner{} } + +// Get writes inst only inside the literal it hands to once.Do. +func Get() *Inner { + once.Do(func() { inst = build() }) + return inst +} + +func callsGet() *Inner { return Get() } + +// argv is initialized from a library variable, cfgDir from a kept global: +// the chained initializer rule keeps both. +var argv = os.Args + +var cfgDir = home + "/x" + +func readArgv() []string { return argv } + +func readCfgDir() string { return cfgDir } diff --git a/testdata/dataflow/sdg/sdg.go b/testdata/dataflow/sdg/sdg.go index 25236d8..3bfe1db 100644 --- a/testdata/dataflow/sdg/sdg.go +++ b/testdata/dataflow/sdg/sdg.go @@ -139,3 +139,29 @@ func mutate(p *Inner, s memStore, n Inner, q *Inner) { func (s *sqlStore) setDSN(d string) { s.dsn = d } + +// aliasRange writes through s's elements via the loop variable, an alias. +func aliasRange(s []*Inner, v int) { + for _, it := range s { + it.n = v + } +} + +// aliasAssign writes through p via the local alias q. +func aliasAssign(p *Inner, v int) { + q := p + q.n = v +} + +// aliasCall passes p's alias q to a callee. +func aliasCall(p *Inner) { + q := p + takeInner(q) +} + +func takeInner(p *Inner) {} + +// unusedRef never uses its by-ref param p. +func unusedRef(p *Inner, v int) int { + return v +} From 38750c578ed0b18863170c30e496dd6727739f25 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 08:14:23 -0400 Subject: [PATCH 16/65] fix(l4): every function literal is a callee of its encloser; honour TargetPath --- internal/sdg/footprint.go | 67 ++++++++++++-------------------- internal/sdg/footprint_test.go | 48 +++++++++++++++++++++++ testdata/dataflow/sdg/globals.go | 10 +++++ 3 files changed, 82 insertions(+), 43 deletions(-) diff --git a/internal/sdg/footprint.go b/internal/sdg/footprint.go index 5697bf3..a91a75f 100644 --- a/internal/sdg/footprint.go +++ b/internal/sdg/footprint.go @@ -110,14 +110,18 @@ func analysed(in Input, u dataflow.Unit) bool { // VTA ∪ the interface method, D7). A call node the CFG pruned keeps its // targets (an over-approximation); // - for an unanalysed unit dispatch seeded no site for (fix I2), its L2 -// call_graph edges (in.CallGraph), each target mapped as newDispatch -// maps a static callee; +// call_graph edges (in.CallGraph): the callable of the edge's +// TargetPath when it has one (a repeated signature, such as one init +// closure per file, names the right file), else the target mapped as +// newDispatch maps a static callee; // - every function literal of its body (outermost: a nested one is its -// enclosing literal's) not called at its own position (fix I3): one -// handed to library code (once.Do, errgroup.Go, http.HandleFunc, -// sort.Slice) or stored may run during the call, and no call edge -// reaches it. This mirrors the L3 capture-write rule. A literal called -// in place (`func(){…}()`, deferred or launched) has its call edge. +// enclosing literal's; fixes I3, N1). One handed to library code +// (once.Do, errgroup.Go, http.HandleFunc, sort.Slice) or stored may run +// during the call with no call edge to it. One called in place +// (`func(){…}()`, deferred or launched) has a call edge only from VTA +// (L2 resolves no FuncLit callee), so without the literal edge a +// no_vta run would lose its globals. This mirrors the L3 capture-write +// rule; Compact drops the duplicate when VTA's edge is there too. func callGraph(in Input, d *dispatch) ([]Target, map[Target][]Target) { callees := map[Target][]Target{} nodeSet := map[Target]bool{} @@ -135,20 +139,23 @@ func callGraph(in Input, d *dispatch) ([]Target, map[Target][]Target) { u := Target{Callable: site.unit} callees[u] = append(callees[u], ts...) } - l2 := map[dataflow.L3Key][]string{} + l2 := map[dataflow.L3Key][]schema.GoCallEdge{} for _, e := range in.CallGraph { k := dataflow.L3Key{Path: e.SourcePath, Signature: e.Source} - l2[k] = append(l2[k], e.Target) + l2[k] = append(l2[k], e) } for _, u := range in.Units { src := Target{Callable: u.Key} if !seeded[u.Key] && !analysed(in, u) { - for _, sig := range l2[u.Key] { - t, ok := Target{External: sig}, true - if _, ext := in.Externals[sig]; !ext { - var k dataflow.L3Key - k, ok = d.bySig[sig] - t = Target{Callable: k} + for _, e := range l2[u.Key] { + t, ok := Target{Callable: dataflow.L3Key{Path: e.TargetPath, Signature: e.Target}}, true + if e.TargetPath == "" { + t = Target{External: e.Target} + if _, ext := in.Externals[e.Target]; !ext { + var k dataflow.L3Key + k, ok = d.bySig[e.Target] + t = Target{Callable: k} + } } if ok { nodeSet[t] = true @@ -156,8 +163,8 @@ func callGraph(in Input, d *dispatch) ([]Target, map[Target][]Target) { } } } - for _, lit := range uncalledLiterals(u.Fn) { - if k, ok := byFn[lit]; ok { + for _, lit := range astwalk.Literals(fnBody(u.Fn)) { + if k, ok := byFn[lit.Lit]; ok { callees[src] = append(callees[src], Target{Callable: k}) } } @@ -174,32 +181,6 @@ func callGraph(in Input, d *dispatch) ([]Target, map[Target][]Target) { return nodes, callees } -// uncalledLiterals returns the outermost function literals of fn's body -// that are not the function of a call (`func(){…}()`, also deferred or -// launched), in source order. -func uncalledLiterals(fn ast.Node) []*ast.FuncLit { - body := fnBody(fn) - if body == nil { - return nil - } - called := map[*ast.FuncLit]bool{} - ast.Inspect(body, func(n ast.Node) bool { - if call, ok := n.(*ast.CallExpr); ok { - if lit, ok := ast.Unparen(call.Fun).(*ast.FuncLit); ok { - called[lit] = true - } - } - return true - }) - var out []*ast.FuncLit - for _, l := range astwalk.Literals(body) { - if !called[l.Lit] { - out = append(out, l.Lit) - } - } - return out -} - // fnBody is the body of a *ast.FuncDecl or *ast.FuncLit (nil without one). func fnBody(fn ast.Node) *ast.BlockStmt { switch fn := fn.(type) { diff --git a/internal/sdg/footprint_test.go b/internal/sdg/footprint_test.go index 5fcaa1b..fe80df7 100644 --- a/internal/sdg/footprint_test.go +++ b/internal/sdg/footprint_test.go @@ -2,12 +2,16 @@ package sdg import ( "math/rand" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" "path/filepath" "slices" "strings" "testing" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" ) const configSpell = sdgPkg + "::Config" @@ -305,3 +309,47 @@ func TestFootprint_ChainedInitializers(t *testing.T) { } } } + +// TestFootprint_CalledLiteralsWithoutVTA (fix N1): with VTA failed, only VTA +// would give a call edge to a literal called in place (L2 resolves no +// FuncLit callee), so the encloser → literal edge must cover every literal: +// calledLits writes litG (`func(){…}()`) and litH (`defer func(){…}()`). +func TestFootprint_CalledLiteralsWithoutVTA(t *testing.T) { + in := loadInput(t, "dataflow") + defer func(f func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph) { vtaCallGraph = f }(vtaCallGraph) + vtaCallGraph = func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph { panic("vta boom") } + _, d := run(in) + got := fpOf(t, footprints(in, d), Target{Callable: globalsKey("calledLits")}) + if want := []string{sdgPkg + "::litG", sdgPkg + "::litH"}; !slices.Equal(got.GlobalWrites, want) { + t.Errorf("calledLits writes %v under no_vta, want %v", got.GlobalWrites, want) + } +} + +// TestFootprint_L2FallbackHonoursTargetPath (fix N2): an L2 edge with a +// TargetPath maps to that file's callable, not to the lowest-path twin of a +// repeated signature (both files' init closures are +// initclosure.init.closure_1). The edge is synthetic, from an L3-skipped +// unit with no dispatch sites. +func TestFootprint_L2FallbackHonoursTargetPath(t *testing.T) { + in, _, d := runDispatch(t) + const pkg = "example.com/dataflow/initclosure" + src := globalsKey("callsSkipX") + b := dataflow.L3Key{Path: filepath.Join("initclosure", "b.go"), Signature: pkg + ".init.closure_1"} + l3 := make(map[dataflow.L3Key]dataflow.Result, len(in.L3)) + for k, r := range in.L3 { + if k != src { + l3[k] = r + } + } + in.L3 = l3 + for site := range d.sites { + if site.unit == src { + delete(d.sites, site) + } + } + in.CallGraph = []schema.GoCallEdge{{Source: src.Signature, SourcePath: src.Path, Target: b.Signature, TargetPath: b.Path}} + _, callees := callGraph(in, d) + if got, want := callees[Target{Callable: src}], []Target{{Callable: b}}; !slices.Equal(got, want) { + t.Errorf("callees(%s) = %v, want %v", src.Signature, got, want) + } +} diff --git a/testdata/dataflow/sdg/globals.go b/testdata/dataflow/sdg/globals.go index ec4dc53..6fce392 100644 --- a/testdata/dataflow/sdg/globals.go +++ b/testdata/dataflow/sdg/globals.go @@ -76,3 +76,13 @@ var cfgDir = home + "/x" func readArgv() []string { return argv } func readCfgDir() string { return cfgDir } + +// litG and litH are written only by literals called where they are written. +var litG, litH int + +// calledLits writes litG and litH only inside literals it calls in place +// (one deferred): with no VTA, no call edge reaches them. +func calledLits() { + defer func() { litH = 2 }() + func() { litG = 1 }() +} From fb4e907b85f484ea9b1d513e895ec6bed927cdb4 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 10:45:11 -0400 Subject: [PATCH 17/65] feat(l4): formal and actual vertices --- internal/dataflow/bodynodes.go | 19 +- internal/dataflow/defuse_entry.go | 4 + internal/sdg/ir.go | 10 +- internal/sdg/vertices.go | 701 ++++++++++++++++++++++++++++++ internal/sdg/vertices_test.go | 382 ++++++++++++++++ internal/sdg/vta.go | 15 +- testdata/dataflow/sdg/sdg.go | 121 ++++++ 7 files changed, 1242 insertions(+), 10 deletions(-) create mode 100644 internal/sdg/vertices.go create mode 100644 internal/sdg/vertices_test.go diff --git a/internal/dataflow/bodynodes.go b/internal/dataflow/bodynodes.go index a3738cc..89f36d2 100644 --- a/internal/dataflow/bodynodes.go +++ b/internal/dataflow/bodynodes.go @@ -290,14 +290,23 @@ func (b *builder) folded(k string, s ast.Stmt) { // earlier node) already holds it. func (b *builder) add(kind NodeKind, n ast.Node, pos, end token.Pos) string { p := b.fset.PositionFor(pos, false) - base := v2.LocalID(p.Line, p.Column) + key := NextFreeKey(v2.LocalID(p.Line, p.Column), b.taken) + b.nodes = append(b.nodes, Node{Key: key, Kind: kind, Pos: pos, End: end, Stmt: n}) + b.keyOf[n] = key + return key +} + +// NextFreeKey returns base, or base + "/n" for the least n ≥ 2 that taken +// does not hold, and marks it taken. It is the one rule for keying a node at +// a position an L1 call (or an earlier node) may already hold: body nodes +// (BuildNodes) and the L4 capture sites both use it, so neither ever moves +// an L1 call key. +func NextFreeKey(base string, taken map[string]bool) string { key := base - for i := 2; b.taken[key]; i++ { + for i := 2; taken[key]; i++ { key = base + "/" + strconv.Itoa(i) } - b.taken[key] = true - b.nodes = append(b.nodes, Node{Key: key, Kind: kind, Pos: pos, End: end, Stmt: n}) - b.keyOf[n] = key + taken[key] = true return key } diff --git a/internal/dataflow/defuse_entry.go b/internal/dataflow/defuse_entry.go index 30d1c64..9da5bd6 100644 --- a/internal/dataflow/defuse_entry.go +++ b/internal/dataflow/defuse_entry.go @@ -109,6 +109,10 @@ func EntryBases(fn ast.Node, info *types.Info) []types.Object { return append(out, globals...) } +// FreeVars is freeVars, exported for the L4 capture formals (spec §4.6): +// one capture list, in one order, for L3's @entry and L4's ports. +func FreeVars(lit *ast.FuncLit, info *types.Info) []types.Object { return freeVars(lit, info) } + // freeVars returns the variables function literal lit captures: every // variable its body (nested literals included) uses that is declared outside // lit and not at package scope, by position, then name. diff --git a/internal/sdg/ir.go b/internal/sdg/ir.go index e17e4ad..147a3c4 100644 --- a/internal/sdg/ir.go +++ b/internal/sdg/ir.go @@ -25,7 +25,15 @@ type Target struct { } // VertexKind is an L4 port vertex kind. -type VertexKind string // "formal_in" | "formal_out" | "actual_in" | "actual_out" +type VertexKind string + +// The L4 port vertex kinds (shared vocabulary). +const ( + KindFormalIn VertexKind = "formal_in" + KindFormalOut VertexKind = "formal_out" + KindActualIn VertexKind = "actual_in" + KindActualOut VertexKind = "actual_out" +) // Vertex is one L4 port vertex of a callable, keyed by its local key. type Vertex struct { diff --git a/internal/sdg/vertices.go b/internal/sdg/vertices.go new file mode 100644 index 0000000..9a05056 --- /dev/null +++ b/internal/sdg/vertices.go @@ -0,0 +1,701 @@ +package sdg + +import ( + "go/ast" + "go/token" + "go/types" + "slices" + "strconv" + "strings" + + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// The `of` markers of the L4 vertices (spec §4.6, D6): Python's spelling, +// which the SDK's resolve_value and value_candidate read. +const ( + ofReturn = "" + ofCapture = ":" + ofGlobal = ":" +) + +// ports builds the L4 port vertices (spec §4.6): the formals of every call +// target and the actuals of every resolved call site and capture site. It +// reads the L3 results, the dispatch, the footprints and go/types, never the +// SSA program, so it is safe after a no_vta. Formal sets are cached: it is +// not safe for concurrent use. +type ports struct { + in Input + d *dispatch + fps map[Target]Footprint + fx *formalIndex + units map[dataflow.L3Key]dataflow.Unit + byFn map[ast.Node]dataflow.L3Key + sets map[Target]*formalSet + calls map[dataflow.L3Key]map[string]*ast.CallExpr // unit → call key → call + kept map[dataflow.L3Key]map[string]bool // unit → its kept L3 node keys +} + +func newPorts(in Input, d *dispatch) *ports { + p := &ports{ + in: in, + d: d, + fps: footprints(in, d), + fx: newFormalIndex(in, d), + units: make(map[dataflow.L3Key]dataflow.Unit, len(in.Units)), + byFn: make(map[ast.Node]dataflow.L3Key, len(in.Units)), + sets: map[Target]*formalSet{}, + calls: map[dataflow.L3Key]map[string]*ast.CallExpr{}, + kept: map[dataflow.L3Key]map[string]bool{}, + } + for _, u := range in.Units { + p.units[u.Key] = u + p.byFn[u.Fn] = u.Key + } + return p +} + +// formalSet is the formals of one target, with how a call site binds them: +// by position (receiver, declared params, results) or by `of` (captures, +// globals). +type formalSet struct { + in, out []Vertex + recv bool // inPos[0] is the receiver + params int // declared params + inPos []string // formal_in key of the receiver (when recv), then of each declared param + posOf []string // the `of` of each inPos entry + outPos []string // formal_out key of each inPos entry that is a by-ref out, else "" + results []string // formal_out key of each result + captures []string // `of` of each capture, in formal order + capOut []string // `of` of each written capture + globIn []string // `of` of each global formal_in, sorted + globOut []string // `of` of each global formal_out, sorted + inByOf map[string]string + outByOf map[string]string +} + +// formals returns t's formal_in and formal_out vertices (spec §4.6, A9, D6), +// in index order: +// +// - formal_in (`@formal_in:`, always indexed): the receiver (its name, +// `_recv` when unnamed or `_`), the declared params (`_` when unnamed +// or `_`, i the declared index), the captures of a closure +// (`:`, dataflow.FreeVars order), then the globals the +// footprint reads or writes (`:::`, sorted); +// - formal_out (`@formal_out` when there is exactly one, else +// `@formal_out:`): the results by position (`` for one, +// `:` for several, never the user's names), the by-ref +// params the footprint mutates in declaration order (receiver first, +// `of` = the formal's), the captures the closure writes +// (dataflow.CaptureWrites), then the global writes. +// +// t may be any target: a unit, an interface method or bodiless declaration +// (its *types.Func), or an external (its GoExternalCallee, typed through the +// load). The slices are shared: callers must not modify them. +func (p *ports) formals(t Target) (in, out []Vertex) { + fs := p.formalSet(t) + return fs.in, fs.out +} + +func (p *ports) formalSet(t Target) *formalSet { + if fs, ok := p.sets[t]; ok { + return fs + } + fs := &formalSet{inByOf: map[string]string{}, outByOf: map[string]string{}} + p.sets[t] = fs + formals := p.fx.of(t) + fp, ok := p.fps[t] + if !ok { + fp = Footprint{Mutated: byRefFormals(formals)} + } + mutated := map[string]bool{} + for _, m := range fp.Mutated { + mutated[m] = true + } + var ins, outs []string // `of`s + var outFrom []int // per out: the inPos entry it mirrors, or -1 + for i, f := range formals { + if i == 0 && f.recv { + fs.recv = true + } else { + fs.params++ + } + ins = append(ins, f.name) + fs.posOf = append(fs.posOf, f.name) + } + nRes := p.resultCount(t) + for i := range nRes { + outs = append(outs, resultOf(i, nRes)) + outFrom = append(outFrom, -1) + } + for i, f := range formals { + if mutated[f.name] { + outs = append(outs, f.name) + outFrom = append(outFrom, i) + } + } + if u, ok := p.units[t.Callable]; ok && t.External == "" && u.Info != nil { + if lit, ok := u.Fn.(*ast.FuncLit); ok { + fs.captures, fs.capOut = captureOfs(lit, u.Info, p.in.K) + } + } + ins = append(ins, fs.captures...) + outs = append(outs, fs.capOut...) + for range fs.capOut { + outFrom = append(outFrom, -1) + } + for _, g := range unionSorted([][]string{fp.GlobalReads, fp.GlobalWrites}) { + fs.globIn = append(fs.globIn, ofGlobal+g) + } + for _, g := range fp.GlobalWrites { + fs.globOut = append(fs.globOut, ofGlobal+g) + outFrom = append(outFrom, -1) + } + ins = append(ins, fs.globIn...) + outs = append(outs, fs.globOut...) + + fs.outPos = make([]string, len(fs.posOf)) + for i, of := range ins { + key := "@formal_in:" + strconv.Itoa(i) + fs.in = append(fs.in, Vertex{Key: key, Of: of, Kind: KindFormalIn}) + if i < len(fs.posOf) { + fs.inPos = append(fs.inPos, key) + } else { + fs.inByOf[of] = key + } + } + for i, of := range outs { + key := "@formal_out" + if len(outs) > 1 { + key += ":" + strconv.Itoa(i) + } + fs.out = append(fs.out, Vertex{Key: key, Of: of, Kind: KindFormalOut}) + switch { + case i < nRes: + fs.results = append(fs.results, key) + case outFrom[i] >= 0: + fs.outPos[outFrom[i]] = key + default: + fs.outByOf[of] = key + } + } + return fs +} + +// resultOf is the D6 `of` of result i of n. +func resultOf(i, n int) string { + if n == 1 { + return ofReturn + } + return ofReturn + ":" + strconv.Itoa(i) +} + +// captureOfs returns the `of` of every variable lit captures (dataflow.FreeVars +// order) and of those it writes (dataflow.CaptureWrites bases), in that order. +// A capture's name is unique: every variable a literal names from outside it +// resolves in the one scope chain the literal sits in. +func captureOfs(lit *ast.FuncLit, info *types.Info, k int) (all, written []string) { + w := map[types.Object]bool{} + for _, path := range dataflow.CaptureWrites(lit, info, k) { + w[path.Base] = true + } + for _, v := range dataflow.FreeVars(lit, info) { + all = append(all, ofCapture+v.Name()) + if w[v] { + written = append(written, ofCapture+v.Name()) + } + } + return all, written +} + +// resultCount is the number of results of t: from its go/types signature +// where the load has one, else from the symbol table's or the external's +// type strings. +func (p *ports) resultCount(t Target) int { + x := p.fx + if t.External != "" { + e, ok := x.ext[t.External] + if !ok { + return 0 + } + if fn := x.libraryFunc(e); fn != nil { + return fn.Signature().Results().Len() + } + return countResults(e.ReturnType) + } + if u, ok := x.units[t.Callable]; ok && u.Info != nil { + if sig := unitSignature(u); sig != nil { + return sig.Results().Len() + } + } + if fn, ok := x.decls[t.Callable]; ok { + return fn.Signature().Results().Len() + } + return len(x.strs[t.Callable].ReturnTypes) +} + +// countResults counts the results of a GoExternalCallee.ReturnType: "" is +// none, "(A, B)" (types.TypeString's tuple) is the top-level commas plus one, +// anything else is one. +func countResults(s string) int { + if s == "" { + return 0 + } + if !strings.HasPrefix(s, "(") || !strings.HasSuffix(s, ")") { + return 1 + } + n, depth := 1, 0 + for _, r := range s[1 : len(s)-1] { + switch r { + case '(', '[', '{': + depth++ + case ')', ']', '}': + depth-- + case ',': + if depth == 0 { + n++ + } + } + } + return n +} + +// slotRecv is the call-side slot of the receiver: x of a method call x.M(…), +// or the function-value operand f of f(…) for a bound-method target (A8). +// Slots 0, 1, … are the call's function-type params. +const slotRecv = -1 + +// portBind binds one actual of a call or capture site to one target's +// formal: the param_in / param_out endpoints. +type portBind struct { + Actual string + Target Target + Formal string +} + +// site is the actuals of one call or capture site, and how each binds to +// each target's formals (in: actual_in → formal_in; out: formal_out → +// actual_out). +type site struct { + In, Out []Vertex + BindIn, BindOut []portBind +} + +// actuals returns the actual_in and actual_out vertices of call site +// callKey of unit (spec §4.6, A8, A11), whose dispatch targets are ts +// (dispatch.targets, or siteTargets). Keys are `/actual_in:` +// and `/actual_out` (exactly one) or `/actual_out:`; +// Parent is callKey. A call with no target, or whose node the CFG pruned, +// has none. +func (p *ports) actuals(unit dataflow.L3Key, callKey string, ts []Target) (in, out []Vertex) { + s := p.callSite(unit, callKey, ts) + return s.In, s.Out +} + +// callSite is actuals with the bindings. The actuals are the union over ts: +// +// - by position: the receiver slot (x of x.M(…); a method expression's +// first argument; for a bound-method target, the function-value +// operand), then the call's function-type params in order. A variadic +// call's excess values collapse onto its last param, and f(g()) has one +// value per g result. Each slot's `of` is that of the formal it binds +// in the static callee (L2's), else in the first target (ts order) that +// binds it. +// - then, at a directly called literal (`func(…){…}(…)`, deferred or +// launched), the literal's captures after the declared args; +// - then the globals by `of`: the static callee's (sorted), then any +// other target's extras in target order. +// +// actual_out: the results by position, then the by-ref slots (a slot whose +// bound formal some target mutates), then the written captures of a +// directly called literal, then the global writes (by `of`, as for in). +func (p *ports) callSite(unit dataflow.L3Key, callKey string, ts []Target) site { + if len(ts) == 0 || p.pruned(unit, callKey) { + return site{} + } + u := p.units[unit] + call := p.callExpr(unit, callKey) + if call == nil { + return site{} + } + static, hasStatic := p.d.static[siteKey{unit, callKey}] + if !hasStatic || !slices.Contains(ts, static) { + static, hasStatic = Target{}, false + } + order := ts + if hasStatic { + order = append([]Target{static}, slices.DeleteFunc(slices.Clone(ts), func(t Target) bool { return t == static })...) + } + + shape := callShapeOf(call, u.Info) + type slotBind struct { + t Target + pos int // index into the target's inPos + } + binds := map[int][]slotBind{} + for _, t := range order { + fs := p.formalSet(t) + for slot, pos := range shape.bind(fs) { + binds[slot] = append(binds[slot], slotBind{t, pos}) + } + } + slots := make([]int, 0, len(binds)) + for slot := range binds { + slots = append(slots, slot) + } + slices.Sort(slots) + + var s site + addIn := func(of string) string { + key := callKey + "/actual_in:" + strconv.Itoa(len(s.In)) + s.In = append(s.In, Vertex{Key: key, Of: of, Parent: callKey, Kind: KindActualIn}) + return key + } + slotOf := map[int]string{} + for _, slot := range slots { + b := binds[slot][0] + fs := p.formalSet(b.t) + slotOf[slot] = fs.posOf[b.pos] + key := addIn(slotOf[slot]) + for _, b := range binds[slot] { + s.BindIn = append(s.BindIn, portBind{key, b.t, p.formalSet(b.t).inPos[b.pos]}) + } + } + + // A directly called literal binds its captures here (A6). + var lit Target + litOK := false + if fl, ok := ast.Unparen(call.Fun).(*ast.FuncLit); ok { + if k, ok := p.byFn[fl]; ok && slices.Contains(ts, Target{Callable: k}) { + lit, litOK = Target{Callable: k}, true + } + } + if litOK { + fs := p.formalSet(lit) + for _, of := range fs.captures { + s.BindIn = append(s.BindIn, portBind{addIn(of), lit, fs.inByOf[of]}) + } + } + for _, of := range unionByOf(order, p, func(fs *formalSet) []string { return fs.globIn }) { + key := addIn(of) + for _, t := range order { + if f, ok := p.formalSet(t).inByOf[of]; ok { + s.BindIn = append(s.BindIn, portBind{key, t, f}) + } + } + } + + // actual_out: collect (of, bindings), then key once the count is known. + type pending struct { + of string + binds []portBind // Actual filled in below + } + var outs []pending + nRes := shape.results + if nRes < 0 { + nRes = len(p.formalSet(order[0]).results) + } + for i := range nRes { + o := pending{of: resultOf(i, nRes)} + for _, t := range order { + if fs := p.formalSet(t); i < len(fs.results) { + o.binds = append(o.binds, portBind{Target: t, Formal: fs.results[i]}) + } + } + outs = append(outs, o) + } + for _, slot := range slots { + o := pending{of: slotOf[slot]} + for _, b := range binds[slot] { + if f := p.formalSet(b.t).outPos[b.pos]; f != "" { + o.binds = append(o.binds, portBind{Target: b.t, Formal: f}) + } + } + if len(o.binds) > 0 { + outs = append(outs, o) + } + } + if litOK { + fs := p.formalSet(lit) + for _, of := range fs.capOut { + outs = append(outs, pending{of: of, binds: []portBind{{Target: lit, Formal: fs.outByOf[of]}}}) + } + } + for _, of := range unionByOf(order, p, func(fs *formalSet) []string { return fs.globOut }) { + o := pending{of: of} + for _, t := range order { + if f, ok := p.formalSet(t).outByOf[of]; ok { + o.binds = append(o.binds, portBind{Target: t, Formal: f}) + } + } + outs = append(outs, o) + } + for i, o := range outs { + key := callKey + "/actual_out" + if len(outs) > 1 { + key += ":" + strconv.Itoa(i) + } + s.Out = append(s.Out, Vertex{Key: key, Of: o.of, Parent: callKey, Kind: KindActualOut}) + for _, b := range o.binds { + b.Actual = key + s.BindOut = append(s.BindOut, b) + } + } + return s +} + +// unionByOf is the union of list(formalSet(t)) over order: the first +// target's list, then each later target's `of`s not seen yet, in order. +func unionByOf(order []Target, p *ports, list func(*formalSet) []string) []string { + var out []string + seen := map[string]bool{} + for _, t := range order { + for _, of := range list(p.formalSet(t)) { + if !seen[of] { + seen[of] = true + out = append(out, of) + } + } + } + return out +} + +// callShape is what a call expression says about how its values bind: is +// it a method call (x.M(…), the receiver is x), how many params its +// function type has, how many values it passes (f(g()) passes each g +// result), and how many results it has (-1 when its type is unknown). +type callShape struct { + method bool + nParams int + values int + results int +} + +func callShapeOf(call *ast.CallExpr, info *types.Info) callShape { + s := callShape{values: len(call.Args), results: -1} + var sig *types.Signature + if info != nil { + if len(call.Args) == 1 { + if tup, ok := info.TypeOf(call.Args[0]).(*types.Tuple); ok { + s.values = tup.Len() + } + } + if sel, ok := ast.Unparen(call.Fun).(*ast.SelectorExpr); ok { + if x := info.Selections[sel]; x != nil && x.Kind() == types.MethodVal { + s.method = true + } + } + if t := info.TypeOf(call.Fun); t != nil { + sig, _ = t.Underlying().(*types.Signature) + } + } + if sig == nil { + s.nParams = s.values + return s + } + s.nParams = sig.Params().Len() + s.results = sig.Results().Len() + return s +} + +// bind maps the call's slots to fs's positional formals (indices into +// fs.inPos): +// +// - no receiver: call param j binds declared param j; +// - a method call x.M(…): x (slotRecv) binds the receiver, param j +// declared param j; +// - a call with one more param than fs declares — a method expression +// T.M(s, x), or its thunk reached through a function value: param 0 +// binds the receiver, param j declared param j-1; +// - otherwise a bound method value (f := s.M; f(x), A8): the +// function-value operand (slotRecv) binds the receiver, param j declared +// param j. +// +// Values bind in order, one actual per bound param. Only a variadic call +// passes more values than it has params: the excess collapse onto the last +// param (weak, as Java), which value nParams-1 already binds, so they add +// no actual. A param no value reaches (a variadic call with no excess) +// binds nothing. +func (s callShape) bind(fs *formalSet) map[int]int { + out := map[int]int{} + base, off := 0, 0 // declared param d is fs.inPos[base+d] and binds call param d+off + if fs.recv { + base = 1 + if !s.method && s.nParams == fs.params+1 { + off = 1 + out[0] = 0 // a method expression always passes its receiver + } else { + out[slotRecv] = 0 + } + } + for j := range min(s.values, s.nParams) { + if d := j - off; d >= 0 && d < fs.params { + out[j] = base + d + } + } + return out +} + +// pruned reports whether unit's call node callKey is not among its kept L3 +// nodes. A unit with no L3 result (l3_skipped) has no CFG to prune by: its +// L1 call nodes all stand. +func (p *ports) pruned(unit dataflow.L3Key, callKey string) bool { + if _, ok := p.in.L3[unit]; !ok { + return false + } + return !p.keptKeys(unit)[callKey] +} + +// keptKeys is the set of unit's kept L3 node keys (empty with no result). +func (p *ports) keptKeys(unit dataflow.L3Key) map[string]bool { + keep, ok := p.kept[unit] + if !ok { + keep = map[string]bool{} + for _, n := range p.in.L3[unit].Nodes { + keep[n.Key] = true + } + p.kept[unit] = keep + } + return keep +} + +// callExpr returns the call expression of unit's call key, paired by Lparen +// as BuildNodes pairs it. +func (p *ports) callExpr(unit dataflow.L3Key, callKey string) *ast.CallExpr { + m, ok := p.calls[unit] + if !ok { + u := p.units[unit] + byLparen := make(map[token.Pos]string, len(u.Calls)) + for _, c := range u.Calls { + byLparen[c.Lparen] = c.Key + } + m = map[string]*ast.CallExpr{} + for _, c := range astwalk.Calls(fnBody(u.Fn)) { + if k, ok := byLparen[c.Expr.Lparen]; ok { + m[k] = c.Expr + } + } + p.calls[unit] = m + } + return m[callKey] +} + +// siteTargets is the dispatch targets of unit's call callKey, plus, for a +// literal called in place, that literal's callable: L2 resolves no FuncLit +// callee, so without VTA (no_vta) only the syntax names it, and its +// captures must still bind at the call. +func (p *ports) siteTargets(unit dataflow.L3Key, callKey string) []Target { + ts := p.d.targets(unit, callKey) + if call := p.callExpr(unit, callKey); call != nil { + if fl, ok := ast.Unparen(call.Fun).(*ast.FuncLit); ok { + if k, ok := p.byFn[fl]; ok { + t := Target{Callable: k} + if i, found := slices.BinarySearchFunc(ts, t, compareTarget); !found { + ts = slices.Insert(slices.Clone(ts), i, t) + } + } + } + } + return ts +} + +// captureSites returns the capture-site actuals of unit u (A6): for every +// outermost function literal of its body that is not called in place and +// captures a variable, `/actual_in:` per capture and +// `/actual_out[:]` per written capture (`of` +// `:`), with Parent the host: the innermost CFG node holding +// the literal (hostOf). is the literal's line:col, taking the next +// free "/n" among taken (dataflow.NextFreeKey, as BodyNodes keys a +// statement), so it never equals a body key. A literal whose host the CFG +// pruned is dead and has none; so has every literal of a unit with no L3 +// result (no CFG to host it). +func (p *ports) captureSites(u dataflow.Unit, taken map[string]bool) []Vertex { + _, ok := p.in.L3[u.Key] + body := fnBody(u.Fn) + if !ok || body == nil || u.Info == nil { + return nil + } + direct := map[*ast.FuncLit]bool{} + for _, c := range astwalk.Calls(body) { + if fl, ok := ast.Unparen(c.Expr.Fun).(*ast.FuncLit); ok { + direct[fl] = true + } + } + var all []dataflow.Node // every body node, pruned ones included; built once + kept := p.keptKeys(u.Key) + var out []Vertex + for _, l := range astwalk.Literals(body) { + if direct[l.Lit] { + continue + } + caps, written := captureOfs(l.Lit, u.Info, p.in.K) + if len(caps) == 0 { + continue + } + if all == nil { + all, _ = dataflow.BuildNodes(p.in.Fset, body, u.Calls) + } + host, ok := hostOf(all, l.Lit) + if !ok || !kept[host] { + continue + } + pos := p.in.Fset.PositionFor(l.Lit.Pos(), false) + key := dataflow.NextFreeKey(v2.LocalID(pos.Line, pos.Column), taken) + for i, of := range caps { + out = append(out, Vertex{Key: key + "/actual_in:" + strconv.Itoa(i), Of: of, Parent: host, Kind: KindActualIn}) + } + for i, of := range written { + k := key + "/actual_out" + if len(written) > 1 { + k += ":" + strconv.Itoa(i) + } + out = append(out, Vertex{Key: k, Of: of, Parent: host, Kind: KindActualOut}) + } + } + return out +} + +// hostOf is the key of the innermost CFG node of nodes (BuildNodes' output, +// before pruning) whose span holds lit. A satellite call node is no CFG +// node; nor is a deferred call node the host of a literal in its operands: +// they are evaluated at the defer statement, whose DefUse uses the +// literal's captures. +func hostOf(nodes []dataflow.Node, lit *ast.FuncLit) (string, bool) { + best, found := dataflow.Node{}, false + for _, n := range nodes { + if n.Parent != "" || n.Deferred || !n.Pos.IsValid() || n.Pos > lit.Pos() || lit.End() > n.End { + continue + } + if !found || n.End-n.Pos < best.End-best.Pos { + best, found = n, true + } + } + return best.Key, found +} + +// vertices returns every L4 vertex of unit k, sorted by key: its formals, +// the actuals of each resolved, unpruned call site (siteTargets), and its +// capture sites. +func (p *ports) vertices(k dataflow.L3Key) []Vertex { + u, ok := p.units[k] + if !ok { + return nil + } + in, out := p.formals(Target{Callable: k}) + vs := slices.Concat(in, out) + taken := make(map[string]bool, len(u.Calls)) + for _, c := range u.Calls { + taken[c.Key] = true + ai, ao := p.actuals(k, c.Key, p.siteTargets(k, c.Key)) + vs = append(vs, ai...) + vs = append(vs, ao...) + } + for key := range p.keptKeys(k) { + taken[key] = true + } + vs = append(vs, p.captureSites(u, taken)...) + slices.SortFunc(vs, func(a, b Vertex) int { return strings.Compare(a.Key, b.Key) }) + return vs +} diff --git a/internal/sdg/vertices_test.go b/internal/sdg/vertices_test.go new file mode 100644 index 0000000..864d48c --- /dev/null +++ b/internal/sdg/vertices_test.go @@ -0,0 +1,382 @@ +package sdg + +import ( + "fmt" + "path/filepath" + "slices" + "strings" + "testing" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// runPorts runs L4 over testdata/dataflow and returns its vertex builder. +func runPorts(t *testing.T) (Input, *dispatch, *ports) { + t.Helper() + in, _, d := runDispatch(t) + return in, d, newPorts(in, d) +} + +func fin(i, of string) Vertex { return Vertex{Key: "@formal_in:" + i, Of: of, Kind: KindFormalIn} } +func fout(i, of string) Vertex { return Vertex{Key: "@formal_out" + i, Of: of, Kind: KindFormalOut} } +func ain(call, i, of, parent string) Vertex { + return Vertex{Key: call + "/actual_in:" + i, Of: of, Parent: parent, Kind: KindActualIn} +} +func aout(call, i, of, parent string) Vertex { + return Vertex{Key: call + "/actual_out" + i, Of: of, Parent: parent, Kind: KindActualOut} +} + +func checkVertices(t *testing.T, what string, got, want []Vertex) { + t.Helper() + if !slices.Equal(got, want) { + t.Errorf("%s:\n got %+v\n want %+v", what, got, want) + } +} + +func checkFormals(t *testing.T, p *ports, tg Target, wantIn, wantOut []Vertex) { + t.Helper() + in, out := p.formals(tg) + checkVertices(t, fmt.Sprint("formal_in of ", tg), in, wantIn) + checkVertices(t, fmt.Sprint("formal_out of ", tg), out, wantOut) +} + +func checkActuals(t *testing.T, p *ports, d *dispatch, unit dataflow.L3Key, call string, wantIn, wantOut []Vertex) { + t.Helper() + in, out := p.actuals(unit, call, d.targets(unit, call)) + checkVertices(t, "actual_in of "+unit.Signature+"@"+call, in, wantIn) + checkVertices(t, "actual_out of "+unit.Signature+"@"+call, out, wantOut) +} + +// TestVertices_FormalOrderAndOf: formal_in is receiver, declared params +// (unnamed or _ as _, i the declared index), captures, globals; formal_out +// is results by position (:, never user names), by-ref params in +// declaration order (receiver first), written captures, global writes. +func TestVertices_FormalOrderAndOf(t *testing.T) { + _, _, p := runPorts(t) + checkFormals(t, p, sdgTarget("multiT.Multi"), + []Vertex{fin("0", "s"), fin("1", "a"), fin("2", "_1")}, + []Vertex{fout(":0", ":0"), fout(":1", ":1")}) + // Mutated is p, q, s: declaration order puts s before q. + checkFormals(t, p, sdgTarget("mutate"), + []Vertex{fin("0", "p"), fin("1", "s"), fin("2", "n"), fin("3", "q")}, + []Vertex{fout(":0", "p"), fout(":1", "s"), fout(":2", "q")}) + checkFormals(t, p, sdgTarget("sqlStore.setDSN"), + []Vertex{fin("0", "s"), fin("1", "d")}, + []Vertex{fout("", "s")}) + // A transitively read global is a formal_in after the declared params. + checkFormals(t, p, sdgTarget("useConfig"), + []Vertex{fin("0", ":"+configSpell)}, + []Vertex{fout("", "")}) + checkFormals(t, p, sdgTarget("initConfig"), + []Vertex{fin("0", ":"+configSpell)}, + []Vertex{fout("", ":"+configSpell)}) + // A closure: captures after its params; the written capture is an out. + checkFormals(t, p, sdgTarget("CaptureSite.closure_1"), + []Vertex{fin("0", ":x")}, + []Vertex{fout(":0", ""), fout(":1", ":x")}) + checkFormals(t, p, sdgTarget("DirectLit.closure_1"), + []Vertex{fin("0", "a"), fin("1", ":y")}, + []Vertex{fout("", ":y")}) + // An interface method (no unit) and a library interface method: the + // unnamed receiver is _recv. + checkFormals(t, p, sdgTarget("Store.Get"), + []Vertex{fin("0", "_recv"), fin("1", "k")}, + []Vertex{fout(":0", ""), fout(":1", "_recv")}) + checkFormals(t, p, Target{External: "io.Writer.Write"}, + []Vertex{fin("0", "_recv"), fin("1", "p")}, + []Vertex{fout(":0", ":0"), fout(":1", ":1"), fout(":2", "_recv"), fout(":3", "p")}) +} + +// TestVertices_SingleOutIsBare: a callable with exactly one formal_out, and a +// call site with exactly one actual_out, use the bare key. A named result is +// , never its name. +func TestVertices_SingleOutIsBare(t *testing.T) { + _, d, p := runPorts(t) + checkFormals(t, p, sdgTarget("Named"), nil, []Vertex{fout("", "")}) + checkActuals(t, p, d, sdgKey("useConfig"), "122:9", + []Vertex{ain("122:9", "0", ":"+configSpell, "122:9")}, + []Vertex{aout("122:9", "", "", "122:9")}) +} + +// TestVertices_VariadicCollapses: excess actuals of a variadic call collapse +// onto the last formal (one actual_in), each result of f(g()) is its own +// actual, and both results of pair() collapse onto Variadic's xs. +func TestVertices_VariadicCollapses(t *testing.T) { + _, d, p := runPorts(t) + k := sdgKey("CallVariadic") + // xs is a used slice: a by-ref out after the result. + checkActuals(t, p, d, k, "195:7", + []Vertex{ain("195:7", "0", "xs", "195:7")}, + []Vertex{aout("195:7", ":0", "", "195:7"), aout("195:7", ":1", "xs", "195:7")}) + checkActuals(t, p, d, k, "196:7", + []Vertex{ain("196:7", "0", "xs", "196:7")}, + []Vertex{aout("196:7", ":0", "", "196:7"), aout("196:7", ":1", "xs", "196:7")}) + checkActuals(t, p, d, k, "197:7", + []Vertex{ain("197:7", "0", "a", "197:7"), ain("197:7", "1", "b", "197:7")}, + []Vertex{aout("197:7", "", "", "197:7")}) + // pair() has no params: no actual_in; two results. + checkActuals(t, p, d, k, "196:16", nil, + []Vertex{aout("196:16", ":0", ":0", "196:16"), aout("196:16", ":1", ":1", "196:16")}) +} + +// TestVertices_MethodValueBindsReceiver (A8): a method call binds its +// receiver at actual_in:0; a bound method value's function-value operand +// binds it too; a method expression's first argument binds it. The actual's +// of is the static callee's formal (the interface method's _recv, not an +// implementation's name), or the first target's without one; a function +// value that may be a plain function or a bound method keeps the receiver +// slot at 0. +func TestVertices_MethodValueBindsReceiver(t *testing.T) { + _, d, p := runPorts(t) + recvOut := []Vertex{aout("204:9", ":0", "", "204:9"), aout("204:9", ":1", "s", "204:9")} + checkActuals(t, p, d, sdgKey("MethodValue"), "204:9", + []Vertex{ain("204:9", "0", "s", "204:9"), ain("204:9", "1", "k", "204:9")}, recvOut) + checkActuals(t, p, d, sdgKey("MethodExpr"), "209:9", + []Vertex{ain("209:9", "0", "s", "209:9"), ain("209:9", "1", "k", "209:9")}, + []Vertex{aout("209:9", ":0", "", "209:9"), aout("209:9", ":1", "s", "209:9")}) + checkActuals(t, p, d, sdgKey("UseStore"), "34:9", + []Vertex{ain("34:9", "0", "_recv", "34:9"), ain("34:9", "1", "k", "34:9")}, + []Vertex{aout("34:9", ":0", "", "34:9"), aout("34:9", ":1", "_recv", "34:9")}) + // Targets bytes.Buffer.Write (sorted first) and io.Writer.Write (static). + checkActuals(t, p, d, sdgKey("Buffered"), "93:2", + []Vertex{ain("93:2", "0", "_recv", "93:2"), ain("93:2", "1", "p", "93:2")}, + []Vertex{aout("93:2", ":0", ":0", "93:2"), aout("93:2", ":1", ":1", "93:2"), + aout("93:2", ":2", "_recv", "93:2"), aout("93:2", ":3", "p", "93:2")}) + // A variadic method expression passing only the receiver: v binds the + // receiver and nothing reaches xs (no actual for it, in or out). + checkActuals(t, p, d, sdgKey("MethodExprVariadic"), "287:9", + []Vertex{ain("287:9", "0", "_recv", "287:9")}, + []Vertex{aout("287:9", "", "", "287:9")}) + // sqlStore.Get (offset 1) and upper (offset 0): f binds 0 for the method. + checkActuals(t, p, d, sdgKey("MixedValue"), "241:9", + []Vertex{ain("241:9", "0", "s", "241:9"), ain("241:9", "1", "k", "241:9")}, + []Vertex{aout("241:9", ":0", "", "241:9"), aout("241:9", ":1", "s", "241:9")}) +} + +// TestVertices_BindsEachTargetFormal: a site binds each actual to the +// formal of every target that has it, by position (receiver, params, +// results, by-ref outs) or by `of` (captures, globals) — the param_in / +// param_out endpoints. An actual a target lacks binds nothing there. +func TestVertices_BindsEachTargetFormal(t *testing.T) { + _, d, p := runPorts(t) + k := sdgKey("MixedValue") + s := p.callSite(k, "241:9", d.targets(k, "241:9")) + get, up := sdgTarget("sqlStore.Get"), sdgTarget("upper") + wantIn := []portBind{ + {"241:9/actual_in:0", get, "@formal_in:0"}, + {"241:9/actual_in:1", get, "@formal_in:1"}, + {"241:9/actual_in:1", up, "@formal_in:0"}, + } + if !slices.Equal(s.BindIn, wantIn) { + t.Errorf("MixedValue BindIn = %v, want %v", s.BindIn, wantIn) + } + // sqlStore.Get's outs are and s (by-ref); upper's only . + wantOut := []portBind{ + {"241:9/actual_out:0", get, "@formal_out:0"}, + {"241:9/actual_out:0", up, "@formal_out"}, + {"241:9/actual_out:1", get, "@formal_out:1"}, + } + if !slices.Equal(s.BindOut, wantOut) { + t.Errorf("MixedValue BindOut = %v, want %v", s.BindOut, wantOut) + } + // A transitively read global binds by `of`; a directly called literal's + // capture binds to the literal's capture formal. + k = sdgKey("useConfig") + s = p.callSite(k, "122:9", d.targets(k, "122:9")) + if want := []portBind{{"122:9/actual_in:0", sdgTarget("readConfig"), "@formal_in:0"}}; !slices.Equal(s.BindIn, want) { + t.Errorf("useConfig BindIn = %v, want %v", s.BindIn, want) + } + k = sdgKey("DirectLit") + s = p.callSite(k, "226:2", d.targets(k, "226:2")) + lit := sdgTarget("DirectLit.closure_1") + if want := []portBind{{"226:2/actual_in:0", lit, "@formal_in:0"}, {"226:2/actual_in:1", lit, "@formal_in:1"}}; !slices.Equal(s.BindIn, want) { + t.Errorf("DirectLit BindIn = %v, want %v", s.BindIn, want) + } + if want := []portBind{{"226:2/actual_out", lit, "@formal_out"}}; !slices.Equal(s.BindOut, want) { + t.Errorf("DirectLit BindOut = %v, want %v", s.BindOut, want) + } +} + +// TestVertices_NoVTABindsDirectLiteral: L2 resolves no FuncLit callee, so +// without VTA only the syntax names a literal called in place; its declared +// args and captures still bind at the call. +func TestVertices_NoVTABindsDirectLiteral(t *testing.T) { + in := loadInput(t, "dataflow") + defer func(f func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph) { vtaCallGraph = f }(vtaCallGraph) + vtaCallGraph = func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph { panic("vta boom") } + _, d := run(in) + p := newPorts(in, d) + k := sdgKey("DirectLit") + if ts := d.targets(k, "226:2"); len(ts) != 0 { + t.Fatalf("no_vta targets(DirectLit, 226:2) = %v: the test no longer covers the gap", ts) + } + in1, out1 := p.actuals(k, "226:2", p.siteTargets(k, "226:2")) + checkVertices(t, "no_vta actual_in", in1, []Vertex{ain("226:2", "0", "a", "226:2"), ain("226:2", "1", ":y", "226:2")}) + checkVertices(t, "no_vta actual_out", out1, []Vertex{aout("226:2", "", ":y", "226:2")}) +} + +// TestVertices_DirectlyCalledLiteralBindsAtCall (A6): a literal called in +// place binds its captures at that call node after the declared args, and +// has no capture site; so does a deferred one and one whose call is a +// satellite whose statement shares its position. +func TestVertices_DirectlyCalledLiteralBindsAtCall(t *testing.T) { + _, d, p := runPorts(t) + checkActuals(t, p, d, sdgKey("DirectLit"), "226:2", + []Vertex{ain("226:2", "0", "a", "226:2"), ain("226:2", "1", ":y", "226:2")}, + []Vertex{aout("226:2", "", ":y", "226:2")}) + checkActuals(t, p, d, sdgKey("ChainLit"), "259:2", + []Vertex{ain("259:2", "0", ":x", "259:2")}, + []Vertex{aout("259:2", "", "", "259:2")}) + flowKey := func(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("flow", "flow.go"), Signature: "example.com/dataflow/flow." + name} + } + // defer func() { err = wrapErr(err) }() at 557:8: the deferred call node. + checkActuals(t, p, d, flowKey("DeferWrap"), "557:8", + []Vertex{ain("557:8", "0", ":err", "557:8")}, + []Vertex{aout("557:8", "", ":err", "557:8")}) + for _, k := range []dataflow.L3Key{sdgKey("DirectLit"), sdgKey("ChainLit"), flowKey("DeferWrap")} { + for _, v := range p.vertices(k) { + if v.Kind == KindActualIn || v.Kind == KindActualOut { + if _, ok := d.sites[siteKey{k, strings.SplitN(v.Key, "/actual", 2)[0]}]; !ok { + t.Errorf("%s: %s is a capture site, want only call-node actuals", k.Signature, v.Key) + } + } + } + } +} + +// TestVertices_CaptureSite (A6): a literal not called in place gets +// capture-site actuals keyed at its line:col, parented on the innermost CFG +// node holding it: the statement storing it, or the call it is passed to. +func TestVertices_CaptureSite(t *testing.T) { + _, _, p := runPorts(t) + got := capturesOf(p.vertices(sdgKey("CaptureSite"))) + checkVertices(t, "CaptureSite capture site", got, []Vertex{ + ain("215:7", "0", ":x", "215:2"), + aout("215:7", "", ":x", "215:2"), + }) + got = capturesOf(p.vertices(dataflow.L3Key{Path: filepath.Join("flow", "flow.go"), Signature: "example.com/dataflow/flow.OnceDo"})) + checkVertices(t, "OnceDo capture site", got, []Vertex{ + ain("588:10", "0", ":cfg", "588:2"), + aout("588:10", "", ":cfg", "588:2"), + }) + // Passed to a satellite call: the assignment, not the call, hosts it; + // the literal only reads v, so it has no actual_out. + got = capturesOf(p.vertices(sdgKey("CaptureArg"))) + checkVertices(t, "CaptureArg capture site", got, []Vertex{ain("267:13", "0", ":v", "267:2")}) + // A literal after a return is dead: its host is pruned. + if got := capturesOf(p.vertices(sdgKey("DeadLit"))); len(got) != 0 { + t.Errorf("DeadLit capture site = %v, want none", got) + } +} + +// capturesOf keeps the vertices of vs that are not at a call key (l:c or +// l:c/n followed by /actual_*) of a call: the capture-site actuals, whose +// prefix is a literal's key, told apart by their : of. +func capturesOf(vs []Vertex) []Vertex { + var out []Vertex + for _, v := range vs { + if (v.Kind == KindActualIn || v.Kind == KindActualOut) && strings.HasPrefix(v.Of, ":") && v.Parent != strings.SplitN(v.Key, "/actual", 2)[0] { + out = append(out, v) + } + } + return out +} + +// TestVertices_PrunedAndUnresolvedCallsHaveNone: a call node the CFG pruned +// and a call with no target get no vertices. +func TestVertices_PrunedAndUnresolvedCallsHaveNone(t *testing.T) { + _, d, p := runPorts(t) + if ts := d.targets(sdgKey("PrunedCall"), "247:9"); len(ts) == 0 { + t.Fatalf("PrunedCall@247:9 has no targets: the fixture no longer tests pruning") + } + for _, c := range []struct { + k dataflow.L3Key + call string + }{{sdgKey("PrunedCall"), "247:9"}, {sdgKey("callUnknown"), "252:9"}} { + in, out := p.actuals(c.k, c.call, d.targets(c.k, c.call)) + if len(in)+len(out) != 0 { + t.Errorf("%s@%s: actuals %v %v, want none", c.k.Signature, c.call, in, out) + } + for _, v := range p.vertices(c.k) { + if strings.HasPrefix(v.Key, c.call+"/") { + t.Errorf("%s has vertex %s", c.k.Signature, v.Key) + } + } + } +} + +// TestVertices_CaptureSiteKeyNeverCollides: for every callable of sdg and +// flow, vertex keys are unique, sorted, and never a body key (the kept nodes +// and every L1 call key); the key allocator skips taken keys like +// BodyNodes.add. +func TestVertices_CaptureSiteKeyNeverCollides(t *testing.T) { + in, _, p := runPorts(t) + n := 0 + for _, u := range in.Units { + dir := strings.SplitN(u.Key.Path, string(filepath.Separator), 2)[0] + if dir != "sdg" && dir != "flow" { + continue + } + body := map[string]bool{} + for _, nd := range in.L3[u.Key].Nodes { + body[nd.Key] = true + } + for _, c := range u.Calls { + body[c.Key] = true + } + vs := p.vertices(u.Key) + seen := map[string]bool{} + for i, v := range vs { + if body[v.Key] || seen[v.Key] { + t.Errorf("%s: vertex key %s collides", u.Key.Signature, v.Key) + } + seen[v.Key] = true + if i > 0 && vs[i-1].Key >= v.Key { + t.Errorf("%s: vertices not sorted at %s", u.Key.Signature, v.Key) + } + n++ + } + } + if n == 0 { + t.Fatal("no vertices on sdg and flow") + } + // The capture site takes the next free key the way BodyNodes does. + u := unitOf(t, in, sdgKey("CaptureSite")) + got := p.captureSites(u, map[string]bool{"215:7": true}) + if len(got) == 0 || !strings.HasPrefix(got[0].Key, "215:7/2/") { + t.Errorf("capture site with 215:7 taken = %v, want keyed 215:7/2", got) + } + taken := map[string]bool{"259:2": true, "259:2/2": true} + if got := dataflow.NextFreeKey("259:2", taken); got != "259:2/3" || !taken[got] { + t.Errorf("dataflow.NextFreeKey(259:2) = %s, want 259:2/3 (and taken)", got) + } + if got := dataflow.NextFreeKey("7:1", taken); got != "7:1" { + t.Errorf("dataflow.NextFreeKey(7:1) = %s, want 7:1", got) + } +} + +func unitOf(t *testing.T, in Input, k dataflow.L3Key) dataflow.Unit { + t.Helper() + for _, u := range in.Units { + if u.Key == k { + return u + } + } + t.Fatalf("no unit %v", k) + return dataflow.Unit{} +} + +// TestVertices_Deterministic: two builders over two loads agree. +func TestVertices_Deterministic(t *testing.T) { + in1, _, p1 := runPorts(t) + _, _, p2 := runPorts(t) + for _, u := range in1.Units { + if a, b := p1.vertices(u.Key), p2.vertices(u.Key); !slices.Equal(a, b) { + t.Errorf("%v: vertices differ across runs:\n%v\n%v", u.Key, a, b) + } + } +} diff --git a/internal/sdg/vta.go b/internal/sdg/vta.go index 1e4006a..d9480b7 100644 --- a/internal/sdg/vta.go +++ b/internal/sdg/vta.go @@ -32,6 +32,7 @@ type l2Edge struct{ path, src, dst string } // nothing (unmapped_ssa, already reported) is not a target. type dispatch struct { sites map[siteKey][]Target // each sorted by compareTarget, no duplicates + static map[siteKey]Target // the L2 static callee of a site, when it maps to a target bySig map[string]dataflow.L3Key // symbol-table callable of each signature (init excepted) l2 map[l2Edge]bool l2Ext map[string]schema.GoExternalCallee // in.Externals, read-only @@ -55,6 +56,7 @@ func compareTarget(a, b Target) int { func newDispatch(in Input) *dispatch { d := &dispatch{ sites: map[siteKey][]Target{}, + static: map[siteKey]Target{}, l2: make(map[l2Edge]bool, len(in.CallGraph)), l2Ext: in.Externals, newExt: map[string]schema.GoExternalCallee{}, @@ -107,11 +109,16 @@ func newDispatch(in Input) *dispatch { } sig := *cs.CalleeSignature site := siteKey{u.Key, u.Calls[i].Key} - if cs.CalleeImportPath != "" { - d.add(site, Target{External: sig}) - } else if k, ok := bySig[sig]; ok { - d.add(site, Target{Callable: k}) + t := Target{External: sig} + if cs.CalleeImportPath == "" { + k, ok := bySig[sig] + if !ok { + continue + } + t = Target{Callable: k} } + d.add(site, t) + d.static[site] = t } } return d diff --git a/testdata/dataflow/sdg/sdg.go b/testdata/dataflow/sdg/sdg.go index 3bfe1db..f68efee 100644 --- a/testdata/dataflow/sdg/sdg.go +++ b/testdata/dataflow/sdg/sdg.go @@ -165,3 +165,124 @@ func takeInner(p *Inner) {} func unusedRef(p *Inner, v int) int { return v } + +// multiT is a value receiver type that holds no reference. +type multiT struct{ n int } + +// Multi has a receiver, a named and a blank param, and two results. +func (s multiT) Multi(a int, _ string) (int, error) { + return a + s.n, nil +} + +// Named has one named result: its formal_out is , never n. +func Named() (n int) { + n = 1 + return +} + +// Variadic collapses excess actuals onto xs. +func Variadic(xs ...int) int { + return len(xs) +} + +func pair() (int, int) { return 1, 2 } + +func takeTwo(a, b int) int { return a + b } + +// CallVariadic passes three values, then pair's two results, to Variadic, +// and pair's results to takeTwo. +func CallVariadic() int { + a := Variadic(1, 2, 3) + b := Variadic(pair()) + c := takeTwo(pair()) + return a + b + c +} + +// MethodValue calls s.Get through a bound method value. +func MethodValue(s *sqlStore, k string) string { + f := s.Get + return f(k) +} + +// MethodExpr calls Get as a method expression: s binds the receiver. +func MethodExpr(s *sqlStore, k string) string { + return (*sqlStore).Get(s, k) +} + +// CaptureSite stores a literal that reads and writes the captured x. +func CaptureSite(v int) func() int { + x := v + f := func() int { + x++ + return x + } + return f +} + +// DirectLit calls a literal in place: a binds v, and the capture y binds at +// the call. +func DirectLit(v int) int { + y := 0 + func(a int) { + y = a + }(v) + return y +} + +func upper(t string) string { return t } + +// MixedValue calls a function value that is either a plain function or a +// bound method: only the method binds the function value as its receiver. +func MixedValue(s *sqlStore, b bool) string { + f := upper + if b { + f = s.Get + } + return f("k") +} + +// PrunedCall's second return is unreachable: its call node is pruned. +func PrunedCall(v int) int { + return v + return double(v) +} + +// callUnknown calls a function value no caller supplies: unresolved. +func callUnknown(f func(int) int) int { + return f(1) +} + +// ChainLit starts a statement with a called literal whose result is +// written: the call holds the literal's line:col, the statement its /2. +func ChainLit(v int) { + x := 0 + func() *Inner { _ = x; return &Inner{} }().n = v +} + +func apply(f func() int) int { return f() } + +// CaptureArg passes a capturing literal to a call nested in an assignment: +// the assignment, not the satellite call, hosts the capture site. +func CaptureArg(v int) int { + n := apply(func() int { return v }) + return n +} + +// DeadLit stores a capturing literal after a return, inside a kept branch: +// the literal is dead and has no capture site. +func DeadLit(v int) { + if v > 0 { + return + _ = func() int { return v } + } +} + +type sumT struct{} + +func (sumT) Sum(xs ...int) int { return len(xs) } + +// MethodExprVariadic passes only the receiver to a variadic method +// expression: nothing reaches xs. +func MethodExprVariadic(v sumT) int { + return sumT.Sum(v) +} From 396e563a1ce4cf2ff0583bd74c96f39e08147f3f Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 12:11:01 -0400 Subject: [PATCH 18/65] feat(l4): port wiring ddg edges --- internal/dataflow/accesspath.go | 8 + internal/dataflow/reachdefs.go | 13 +- internal/sdg/vertices.go | 114 ++++++-- internal/sdg/wiring.go | 442 ++++++++++++++++++++++++++++++++ internal/sdg/wiring_test.go | 357 ++++++++++++++++++++++++++ testdata/dataflow/sdg/sdg.go | 15 ++ 6 files changed, 928 insertions(+), 21 deletions(-) create mode 100644 internal/sdg/wiring.go create mode 100644 internal/sdg/wiring_test.go diff --git a/internal/dataflow/accesspath.go b/internal/dataflow/accesspath.go index 90f2907..307d190 100644 --- a/internal/dataflow/accesspath.go +++ b/internal/dataflow/accesspath.go @@ -271,6 +271,14 @@ func baseSpell(v types.Object) string { return v.Name() } +// SpellBase returns the base spelling of an emitted path spelling: "x" of +// "x.f[*]", "pkgpath::Name" of "pkgpath::Name.F", a pseudo-base whole. The +// L4 port wiring maps a ddg edge's var to the formal of its base with it. +func SpellBase(s string) string { + head, _ := splitSpell(s) + return head +} + // BaseSpell is baseSpell for package sdg: the L4 global formals and // footprints spell a package-level variable as its L3 ddg base. func BaseSpell(v types.Object) string { return baseSpell(v) } diff --git a/internal/dataflow/reachdefs.go b/internal/dataflow/reachdefs.go index 2116631..bca8680 100644 --- a/internal/dataflow/reachdefs.go +++ b/internal/dataflow/reachdefs.go @@ -72,8 +72,10 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { // A nil-base path (a pseudo path, an unresolved identifier) is bucketed // by its spelling head: equal heads are necessary for Covers. buckets := map[killBucket][]int{} + defBucket := make([]killBucket, len(defs)) for i, d := range defs { b := bucketOf(d.path) + defBucket[i] = b buckets[b] = append(buckets[b], i) } @@ -136,18 +138,27 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { } } + // Use matching is bucketed like the kills: interference (Covers either + // way) needs the same bucket, so a reaching def is tested only against + // the uses of its bucket — O(defs + matches) per node rather than + // O(defs × uses), which the L4 re-solve's global uses at hosts need. edges := map[DdgEdge]bool{} for v, k := range g.keys { uses := du[k].Uses if len(uses) == 0 { continue } + useBuckets := make(map[killBucket][]Path, len(uses)) + for _, u := range uses { + b := bucketOf(u) + useBuckets[b] = append(useBuckets[b], u) + } for w, x := range in[v] { for x != 0 { b := bits.TrailingZeros64(x) x &^= 1 << b d := defs[w*64+b] - for _, u := range uses { + for _, u := range useBuckets[defBucket[w*64+b]] { if d.path.Covers(u) || u.Covers(d.path) { edges[DdgEdge{Src: g.keys[d.node], Dst: k, Var: u.Spell}] = true } diff --git a/internal/sdg/vertices.go b/internal/sdg/vertices.go index 9a05056..c9723a9 100644 --- a/internal/sdg/vertices.go +++ b/internal/sdg/vertices.go @@ -609,12 +609,48 @@ func (p *ports) siteTargets(unit dataflow.L3Key, callKey string) []Target { // the literal (hostOf). is the literal's line:col, taking the next // free "/n" among taken (dataflow.NextFreeKey, as BodyNodes keys a // statement), so it never equals a body key. A literal whose host the CFG -// pruned is dead and has none; so has every literal of a unit with no L3 -// result (no CFG to host it). +// pruned is dead and has none. +// +// A unit with no L3 result (l3_skipped) has no CFG to host a literal: its +// host is the innermost L1 call node whose call holds the literal (`once.Do( +// func(){…})`), else @entry (ruling, Task 12 review). Its port wiring is +// pass-through (wire). func (p *ports) captureSites(u dataflow.Unit, taken map[string]bool) []Vertex { - _, ok := p.in.L3[u.Key] + var out []Vertex + for _, cs := range p.capSites(u, taken) { + out = append(out, cs.vertices()...) + } + return out +} + +// capSite is one capture site: its key, its host, and each capture (in +// actual_in order) with its actual_in key and, when the literal writes it, +// its actual_out key. +type capSite struct { + key, host string + caps []types.Object + ofs []string + in, out []string // per capture; out[i] == "" when not written +} + +func (cs capSite) vertices() []Vertex { + var vs []Vertex + for i, of := range cs.ofs { + vs = append(vs, Vertex{Key: cs.in[i], Of: of, Parent: cs.host, Kind: KindActualIn}) + } + for i, of := range cs.ofs { + if cs.out[i] != "" { + vs = append(vs, Vertex{Key: cs.out[i], Of: of, Parent: cs.host, Kind: KindActualOut}) + } + } + return vs +} + +// capSites is captureSites with each site's capture objects. +func (p *ports) capSites(u dataflow.Unit, taken map[string]bool) []capSite { + _, analysed := p.in.L3[u.Key] body := fnBody(u.Fn) - if !ok || body == nil || u.Info == nil { + if body == nil || u.Info == nil { return nil } direct := map[*ast.FuncLit]bool{} @@ -625,38 +661,76 @@ func (p *ports) captureSites(u dataflow.Unit, taken map[string]bool) []Vertex { } var all []dataflow.Node // every body node, pruned ones included; built once kept := p.keptKeys(u.Key) - var out []Vertex + var out []capSite for _, l := range astwalk.Literals(body) { if direct[l.Lit] { continue } - caps, written := captureOfs(l.Lit, u.Info, p.in.K) + caps := dataflow.FreeVars(l.Lit, u.Info) if len(caps) == 0 { continue } - if all == nil { - all, _ = dataflow.BuildNodes(p.in.Fset, body, u.Calls) + var host string + if analysed { + if all == nil { + all, _ = dataflow.BuildNodes(p.in.Fset, body, u.Calls) + } + h, ok := hostOf(all, l.Lit) + if !ok || !kept[h] { + continue + } + host = h + } else { + host = p.skippedHost(u, l.Lit) } - host, ok := hostOf(all, l.Lit) - if !ok || !kept[host] { - continue + written := map[types.Object]bool{} + for _, path := range dataflow.CaptureWrites(l.Lit, u.Info, p.in.K) { + written[path.Base] = true + } + nOut := 0 + for _, v := range caps { + if written[v] { + nOut++ + } } pos := p.in.Fset.PositionFor(l.Lit.Pos(), false) - key := dataflow.NextFreeKey(v2.LocalID(pos.Line, pos.Column), taken) - for i, of := range caps { - out = append(out, Vertex{Key: key + "/actual_in:" + strconv.Itoa(i), Of: of, Parent: host, Kind: KindActualIn}) - } - for i, of := range written { - k := key + "/actual_out" - if len(written) > 1 { - k += ":" + strconv.Itoa(i) + cs := capSite{key: dataflow.NextFreeKey(v2.LocalID(pos.Line, pos.Column), taken), host: host, caps: caps} + j := 0 + for i, v := range caps { + cs.ofs = append(cs.ofs, ofCapture+v.Name()) + cs.in = append(cs.in, cs.key+"/actual_in:"+strconv.Itoa(i)) + o := "" + if written[v] { + o = cs.key + "/actual_out" + if nOut > 1 { + o += ":" + strconv.Itoa(j) + } + j++ } - out = append(out, Vertex{Key: k, Of: of, Parent: host, Kind: KindActualOut}) + cs.out = append(cs.out, o) } + out = append(out, cs) } return out } +// skippedHost is the host of a capture site in a unit with no L3 result: +// the innermost of its L1 call nodes whose call expression holds lit, else +// @entry. +func (p *ports) skippedHost(u dataflow.Unit, lit *ast.FuncLit) string { + host, best := dataflow.EntryKey, token.Pos(-1) + for _, c := range u.Calls { + call := p.callExpr(u.Key, c.Key) + if call == nil || call.Pos() > lit.Pos() || lit.End() > call.End() { + continue + } + if span := call.End() - call.Pos(); best < 0 || span < best { + host, best = c.Key, span + } + } + return host +} + // hostOf is the key of the innermost CFG node of nodes (BuildNodes' output, // before pruning) whose span holds lit. A satellite call node is no CFG // node; nor is a deferred call node the host of a literal in its operands: diff --git a/internal/sdg/wiring.go b/internal/sdg/wiring.go new file mode 100644 index 0000000..8755572 --- /dev/null +++ b/internal/sdg/wiring.go @@ -0,0 +1,442 @@ +package sdg + +import ( + "cmp" + "go/ast" + "go/types" + "slices" + "strings" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// ProvReachingDefs is the prov of every port-wiring ddg edge (spec §4.7, +// Python's label; L3's own edges are "ssa"). +const ProvReachingDefs = "reaching-defs" + +// wire returns the port wiring of unit k (spec §4.7, A4, A7, A11): the +// callable-local ddg edges, prov "reaching-defs", that connect its L4 +// vertices to its body. They come from re-solving reaching definitions over +// an augmented DefUse (resolve) and from the vertices themselves: +// +// - each L3 edge @entry → n on a base with a formal_in is mirrored as +// @formal_in: → n (same var); the L3 edge stays and is not repeated; +// - each re-solved edge d → @exit on an output base b is emitted as +// d → the formal_out of b (var its `of`), never into @exit; +// - each other re-solved edge not in L3 (from the A4 uses and weak defs) +// is emitted as is; +// - a new edge from @entry on a base with a formal_in starts at that +// formal_in instead (Python's formal_for remap); @entry defines a named +// result's zero value and keeps those edges; +// - every return node → each formal_out of an unnamed (or `_`) result +// (var `` / `:`); a named result reaches its +// formal_out through the @exit rule; +// - at each resolved call site (A11): host → each actual_in, and each +// actual_out → host (var the actual's `of`), host = the call node's +// Parent, else the call node (a bare, `go` or deferred call); +// - at each capture site (A6, A7): every node defining a captured base → +// its actual_in; its actual_out → every node using the base, and → the +// encloser's own formal_out for the base when it is an output base (a +// named result, a by-ref param, or a capture the encloser writes). +// +// An edge's var is the `of` of the vertex it touches, the formal_out's when +// it ends at one; a body → body edge keeps the L3 path spelling. +// +// A unit with no L3 result (l3_skipped) has no body nodes to wire: it is +// wired pass-through (wireSkipped). +// +// The edges are sorted by (Src, Dst, Var) and unique; none repeats an L3 +// edge. +func (p *ports) wire(k dataflow.L3Key) []PortEdge { + u, ok := p.units[k] + if !ok { + return nil + } + w := p.newWiring(u) + if w == nil { + return p.wireSkipped(u) + } + l3 := make(map[dataflow.DdgEdge]bool, len(w.r.Ddg)) + for _, e := range w.r.Ddg { + l3[e] = true + } + out := map[PortEdge]bool{} + // add drops an L3 edge. No rule below yields one (an edge touching a + // vertex is never L3's, and a body → body edge is emitted only when it is + // new), so this only guards the "no PortEdge repeats L3" contract. + add := func(src, dst, v string) { + if !l3[dataflow.DdgEdge{Src: src, Dst: dst, Var: v}] { + out[PortEdge{Src: src, Dst: dst, Var: v, Prov: ProvReachingDefs}] = true + } + } + + for _, e := range dataflow.BuildDDG(w.r.Nodes, w.r.Cfg, w.du) { + b := dataflow.SpellBase(e.Var) + switch { + case e.Dst == dataflow.ExitKey: + fo, ok := w.exitOut[e.Var] + if !ok { + continue + } + src := e.Src + if fi, ok := w.inOf(b); ok && src == dataflow.EntryKey { + src = fi + } + add(src, fo.Key, fo.Of) + case l3[e]: + if fi, ok := w.inOf(b); ok && e.Src == dataflow.EntryKey { + add(fi, e.Dst, e.Var) + } + default: + src := e.Src + if fi, ok := w.inOf(b); ok && src == dataflow.EntryKey { + src = fi + } + add(src, e.Dst, e.Var) + } + } + + for i, fo := range w.fs.results { + if w.resObjs[i] != nil { + continue + } + of := resultOf(i, len(w.fs.results)) + for _, n := range w.r.Nodes { + if n.Kind == dataflow.KindReturn { + add(n.Key, fo, of) + } + } + } + + for _, s := range w.sites { + for _, v := range s.site.In { + add(s.host, v.Key, v.Of) + } + for _, v := range s.site.Out { + add(v.Key, s.host, v.Of) + } + } + + for _, cs := range p.capSites(u, w.taken) { + for i, v := range cs.caps { + of := cs.ofs[i] + for _, n := range w.r.Nodes { + d := w.r.DefUse[n.Key] + if hasBase(d.Defs, v) || hasBase(d.WeakDefs, v) { + src := n.Key + if fi, ok := w.inByObj[v]; ok && src == dataflow.EntryKey { + src = fi + } + add(src, cs.in[i], of) + } + } + if cs.out[i] == "" { + continue + } + for _, n := range w.r.Nodes { + if n.Key != dataflow.ExitKey && hasBase(w.r.DefUse[n.Key].Uses, v) { + add(cs.out[i], n.Key, of) + } + } + if fo, ok := w.outByObj[v]; ok { + add(cs.out[i], fo.Key, fo.Of) + } + } + } + return sortedPortEdges(out) +} + +// resolve re-solves unit k's reaching definitions over the augmented DefUse +// (newWiring): L3's ddg plus the A4 and @exit edges. nil without an L3 +// result. +func (p *ports) resolve(k dataflow.L3Key) []dataflow.DdgEdge { + u, ok := p.units[k] + if !ok { + return nil + } + w := p.newWiring(u) + if w == nil { + return nil + } + return dataflow.BuildDDG(w.r.Nodes, w.r.Cfg, w.du) +} + +// wiring is one analysed unit's port-wiring inputs. +type wiring struct { + r dataflow.Result + fs *formalSet + du map[string]dataflow.DefUse // du': L3's DefUse, augmented + exitOut map[string]Vertex // @exit use spelling → its formal_out + inByBase map[string]string // base spelling → formal_in key + inByObj map[types.Object]string // base object → formal_in key + outByObj map[types.Object]Vertex // output base object → its formal_out + resObjs []types.Object // per result position: its object, nil if unnamed or `_` + sites []hostedSite + taken map[string]bool // every L1 call key and kept node key (capture-site keys) +} + +type hostedSite struct { + host string + site site +} + +func (w *wiring) inOf(base string) (string, bool) { + k, ok := w.inByBase[base] + return k, ok +} + +// newWiring builds the augmented DefUse du' of analysed unit u (nil without +// an L3 DefUse): a copy of L3's, plus +// +// 1. (A4) at each host of a resolved, kept call site, the union over its +// targets of their footprints' GlobalReads as uses and GlobalWrites as +// weak defs (weak defs kill nothing, so every L3 edge survives); +// 2. at @entry, a def of every global formal_in base, so that a global +// only a callee reads still has a def (its formal_in) at the host; +// 3. at @exit, a use of every output base: the named results, the by-ref +// params in Footprint.Mutated (the receiver too), the captures the +// closure writes, and the footprint's GlobalWrites. +// +// A global is the unit's own object for it when its DefUse names one, else +// a nil-base path spelled "pkgpath::Name" (the spelling of an unresolved +// package member at L3), consistent within the unit. +func (p *ports) newWiring(u dataflow.Unit) *wiring { + r, ok := p.in.L3[u.Key] + if !ok || r.DefUse == nil || u.Info == nil { + return nil + } + t := Target{Callable: u.Key} + fs := p.formalSet(t) + fp := p.fps[t] + w := &wiring{ + r: r, + fs: fs, + du: make(map[string]dataflow.DefUse, len(r.DefUse)+1), + exitOut: map[string]Vertex{}, + inByBase: map[string]string{}, + inByObj: map[types.Object]string{}, + outByObj: map[types.Object]Vertex{}, + taken: make(map[string]bool, len(u.Calls)+len(r.Nodes)), + } + for key, d := range r.DefUse { + w.du[key] = d + } + globals := map[string]types.Object{} + for _, d := range r.DefUse { + for _, list := range [][]dataflow.Path{d.Defs, d.WeakDefs, d.Uses} { + for _, path := range list { + if dataflow.IsPkgVar(path.Base) { + globals[dataflow.BaseSpell(path.Base)] = path.Base + } + } + } + } + global := func(g string) dataflow.Path { return dataflow.Path{Base: globals[g], Spell: g} } + outKey := make(map[string]string, len(fs.out)) + outs := make(map[string]Vertex, len(fs.out)) + for _, v := range fs.out { + outKey[v.Of] = v.Key + outs[v.Key] = v + } + outVertex := func(key string) Vertex { return outs[key] } + + // formal_in by base: declared formals by name, captures, globals. + objs := formalObjs(u.Fn, u.Info) + byName := map[string]types.Object{} + for _, o := range objs { + byName[o.Name()] = o + } + for i, of := range fs.posOf { + w.inByBase[of] = fs.inPos[i] + if o, ok := byName[of]; ok { + w.inByObj[o] = fs.inPos[i] + } + } + var caps []types.Object + if lit, ok := u.Fn.(*ast.FuncLit); ok { + caps = dataflow.FreeVars(lit, u.Info) + } + for _, v := range caps { + if k, ok := fs.inByOf[ofCapture+v.Name()]; ok { + w.inByBase[v.Name()] = k + w.inByObj[v] = k + } + } + for _, of := range fs.globIn { + w.inByBase[strings.TrimPrefix(of, ofGlobal)] = fs.inByOf[of] + } + + // @exit uses: the output bases, each mapped to its formal_out. + var exitUses []dataflow.Path + exitUse := func(path dataflow.Path, key string) { + exitUses = append(exitUses, path) + w.exitOut[path.Spell] = outVertex(key) + if path.Base != nil { + w.outByObj[path.Base] = outVertex(key) + } + } + w.resObjs = resultObjs(u.Fn, u.Info) + for i, o := range w.resObjs { + if o != nil && i < len(fs.results) { + exitUse(dataflow.Path{Base: o, Spell: o.Name()}, fs.results[i]) + } + } + for i, of := range fs.posOf { + if key := fs.outPos[i]; key != "" { + if o, ok := byName[of]; ok { + exitUse(dataflow.Path{Base: o, Spell: o.Name()}, key) + } + } + } + for _, v := range caps { + if key, ok := fs.outByOf[ofCapture+v.Name()]; ok { + exitUse(dataflow.Path{Base: v, Spell: v.Name()}, key) + } + } + for _, g := range fp.GlobalWrites { + if key, ok := outKey[ofGlobal+g]; ok { + exitUse(global(g), key) + } + } + d := w.du[dataflow.ExitKey] + d.Uses = append(slices.Clip(d.Uses), exitUses...) + w.du[dataflow.ExitKey] = d + + var entryDefs []dataflow.Path + for _, of := range fs.globIn { + entryDefs = append(entryDefs, global(strings.TrimPrefix(of, ofGlobal))) + } + d = w.du[dataflow.EntryKey] + d.Defs = append(slices.Clip(d.Defs), entryDefs...) + w.du[dataflow.EntryKey] = d + + // Call sites: the hosts, their A4 facts and their actuals. + nodes := make(map[string]dataflow.Node, len(r.Nodes)) + for _, n := range r.Nodes { + nodes[n.Key] = n + w.taken[n.Key] = true + } + for _, c := range u.Calls { + w.taken[c.Key] = true + } + for _, c := range u.Calls { + n, ok := nodes[c.Key] + if !ok { + continue // pruned: no vertices, no wiring + } + host := c.Key + if n.Parent != "" { + host = n.Parent + } + if _, ok := nodes[host]; !ok { + continue + } + ts := p.siteTargets(u.Key, c.Key) + if len(ts) == 0 { + continue + } + var reads, writes [][]string + for _, t := range ts { + f := p.fps[t] + reads, writes = append(reads, f.GlobalReads), append(writes, f.GlobalWrites) + } + if rs, ws := unionSorted(reads), unionSorted(writes); len(rs)+len(ws) > 0 { + d := w.du[host] + uses, weak := slices.Clip(d.Uses), slices.Clip(d.WeakDefs) + for _, g := range rs { + uses = append(uses, global(g)) + } + for _, g := range ws { + weak = append(weak, global(g)) + } + d.Uses, d.WeakDefs = uses, weak + w.du[host] = d + } + if s := p.callSite(u.Key, c.Key, ts); len(s.In)+len(s.Out) > 0 { + w.sites = append(w.sites, hostedSite{host: host, site: s}) + } + } + return w +} + +// wireSkipped is the pass-through wiring of a unit with no L3 result +// (ruling, Task 12 review; l3_skipped is a degraded code): with no body +// nodes, every formal_in → every actual_in of its call and capture sites and +// → every formal_out, and every actual_out → every formal_out. Each edge's +// var is its destination vertex's `of`. +func (p *ports) wireSkipped(u dataflow.Unit) []PortEdge { + fin, fout := p.formals(Target{Callable: u.Key}) + var ins, outs []Vertex + taken := make(map[string]bool, len(u.Calls)) + for _, c := range u.Calls { + taken[c.Key] = true + s := p.callSite(u.Key, c.Key, p.siteTargets(u.Key, c.Key)) + ins, outs = append(ins, s.In...), append(outs, s.Out...) + } + for _, v := range p.captureSites(u, taken) { + if v.Kind == KindActualIn { + ins = append(ins, v) + } else { + outs = append(outs, v) + } + } + out := map[PortEdge]bool{} + for _, f := range fin { + for _, a := range slices.Concat(ins, fout) { + out[PortEdge{Src: f.Key, Dst: a.Key, Var: a.Of, Prov: ProvReachingDefs}] = true + } + } + for _, a := range outs { + for _, f := range fout { + out[PortEdge{Src: a.Key, Dst: f.Key, Var: f.Of, Prov: ProvReachingDefs}] = true + } + } + return sortedPortEdges(out) +} + +// resultObjs returns, per result position of fn, its named result object, +// nil for an unnamed or `_` one. +func resultObjs(fn ast.Node, info *types.Info) []types.Object { + var ft *ast.FuncType + switch fn := fn.(type) { + case *ast.FuncDecl: + ft = fn.Type + case *ast.FuncLit: + ft = fn.Type + } + if ft == nil || ft.Results == nil { + return nil + } + var out []types.Object + for _, f := range ft.Results.List { + if len(f.Names) == 0 { + out = append(out, nil) + continue + } + for _, name := range f.Names { + var o types.Object + if name.Name != "_" { + o = info.Defs[name] + } + out = append(out, o) + } + } + return out +} + +func hasBase(list []dataflow.Path, v types.Object) bool { + return slices.ContainsFunc(list, func(p dataflow.Path) bool { return p.Base == v }) +} + +func comparePortEdge(a, b PortEdge) int { + return cmp.Or(strings.Compare(a.Src, b.Src), strings.Compare(a.Dst, b.Dst), strings.Compare(a.Var, b.Var), strings.Compare(a.Prov, b.Prov)) +} + +func sortedPortEdges(set map[PortEdge]bool) []PortEdge { + out := make([]PortEdge, 0, len(set)) + for e := range set { + out = append(out, e) + } + slices.SortFunc(out, comparePortEdge) + return out +} diff --git a/internal/sdg/wiring_test.go b/internal/sdg/wiring_test.go new file mode 100644 index 0000000..03e8ab4 --- /dev/null +++ b/internal/sdg/wiring_test.go @@ -0,0 +1,357 @@ +package sdg + +import ( + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +func pe(src, dst, v string) PortEdge { + return PortEdge{Src: src, Dst: dst, Var: v, Prov: ProvReachingDefs} +} + +func flowKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("flow", "flow.go"), Signature: "example.com/dataflow/flow." + name} +} + +// fixtureUnits returns the units of the sdg and flow fixture packages. +func fixtureUnits(in Input) []dataflow.Unit { + var out []dataflow.Unit + for _, u := range in.Units { + dir := strings.SplitN(u.Key.Path, string(filepath.Separator), 2)[0] + if dir == "sdg" || dir == "flow" { + out = append(out, u) + } + } + return out +} + +func checkHas(t *testing.T, what string, got []PortEdge, want ...PortEdge) { + t.Helper() + for _, e := range want { + if !slices.Contains(got, e) { + t.Errorf("%s lacks %+v; got %+v", what, e, got) + } + } +} + +// reaches reports whether dst is reachable from src over edges. +func reaches(edges []PortEdge, src, dst string) bool { + seen := map[string]bool{src: true} + work := []string{src} + for len(work) > 0 { + n := work[0] + work = work[1:] + for _, e := range edges { + if e.Src == n && !seen[e.Dst] { + if e.Dst == dst { + return true + } + seen[e.Dst] = true + work = append(work, e.Dst) + } + } + } + return false +} + +// TestWiring_FormalInReachesUses: each @entry → n edge on a formal's base is +// mirrored as @formal_in: → n with the edge's var (the @entry edge stays +// L3's). UseStore's whole wiring is pinned: the return statement hosts the +// satellite s.Get(k), defines the result and writes through s (by-ref out). +func TestWiring_FormalInReachesUses(t *testing.T) { + _, _, p := runPorts(t) + got := p.wire(sdgKey("UseStore")) + want := []PortEdge{ + pe("34:2", "34:9/actual_in:0", "_recv"), + pe("34:2", "34:9/actual_in:1", "k"), + pe("34:2", "@formal_out:0", ""), + pe("34:2", "@formal_out:1", "s"), + pe("34:9/actual_out:0", "34:2", ""), + pe("34:9/actual_out:1", "34:2", "_recv"), + pe("@formal_in:0", "34:2", "s"), + pe("@formal_in:0", "@formal_out:1", "s"), + pe("@formal_in:1", "34:2", "k"), + } + if !slices.Equal(got, want) { + t.Errorf("wire(UseStore):\n got %+v\n want %+v", got, want) + } + // A field use keeps the edge's spelling. + checkHas(t, "wire(multiT.Multi)", p.wire(sdgKey("multiT.Multi")), + pe("@formal_in:0", "174:2", "s.n"), pe("@formal_in:1", "174:2", "a")) + // A closure's capture formal mirrors its @entry def of the capture. + checkHas(t, "wire(DeferWrap.closure_2)", p.wire(flowKey("DeferWrap.closure_2")), + pe("@formal_in:0", "557:17", "err")) +} + +// TestWiring_ReturnReachesFormalOut: with unnamed results, every return node +// reaches each result's formal_out (var its `of`); a named result reaches it +// through its defs reaching @exit, and the bare return adds nothing. +func TestWiring_ReturnReachesFormalOut(t *testing.T) { + _, _, p := runPorts(t) + checkHas(t, "wire(multiT.Multi)", p.wire(sdgKey("multiT.Multi")), + pe("174:2", "@formal_out:0", ":0"), pe("174:2", "@formal_out:1", ":1")) + got := p.wire(sdgKey("Named")) + checkHas(t, "wire(Named)", got, pe("179:2", "@formal_out", "")) + for _, e := range got { + if e.Src == "180:2" { + t.Errorf("wire(Named) has %+v: a bare return defines nothing", e) + } + } + // A by-ref param's writes reach its formal_out. + checkHas(t, "wire(mutate)", p.wire(sdgKey("mutate")), + pe("132:2", "@formal_out:0", "p"), pe("133:2", "@formal_out:1", "s")) +} + +// TestWiring_NamedResultViaDeferredClosure (A7): DeferWrap's deferred +// `func() { err = wrapErr(err) }()` hosts its own actuals: the capture's +// actual_out reaches the deferred call node, whose weak def of err reaches +// @formal_out. A stored literal writing a named result or a by-ref param +// wires its capture actual_out straight to the encloser's formal_out. +func TestWiring_NamedResultViaDeferredClosure(t *testing.T) { + _, _, p := runPorts(t) + got := p.wire(flowKey("DeferWrap")) + checkHas(t, "wire(DeferWrap)", got, + pe("557:8", "557:8/actual_in:0", ":err"), + pe("557:8/actual_out", "557:8", ":err"), + pe("557:8", "@formal_out", "")) + if !reaches(got, "557:8/actual_out", "@formal_out") { + t.Errorf("wire(DeferWrap): 557:8/actual_out does not reach @formal_out: %+v", got) + } + checkHas(t, "wire(CaptureNamed)", p.wire(sdgKey("CaptureNamed")), + pe("293:2", "293:7/actual_in:0", ":err"), + pe("293:7/actual_out", "293:2", ":err"), + pe("293:7/actual_out", "295:2", ":err"), + pe("293:7/actual_out", "@formal_out", ""), + pe("293:2", "@formal_out", "")) + checkHas(t, "wire(CaptureParam)", p.wire(sdgKey("CaptureParam")), + pe("@formal_in:0", "301:7/actual_in:0", ":p"), + pe("301:7/actual_out", "301:2", ":p"), + pe("301:7/actual_out", "@formal_out", "p"), + pe("301:2", "@formal_out", "p")) + // A capture site passed to a call (OnceDo): its defs and uses. + checkHas(t, "wire(OnceDo)", p.wire(flowKey("OnceDo")), + pe("587:2", "588:10/actual_in:0", ":cfg"), + pe("588:10/actual_out", "589:2", ":cfg")) +} + +// TestWiring_TransitiveGlobalAtHost (A4): useConfig reads Config only +// through readConfig. The host uses Config, which @formal_in(Config) defines, +// and feeds the call's actual_in for it. callsGet's callee writes inst: the +// actual_out feeds the host, whose weak def of inst reaches its formal_out. +func TestWiring_TransitiveGlobalAtHost(t *testing.T) { + _, _, p := runPorts(t) + got := p.wire(sdgKey("useConfig")) + checkHas(t, "wire(useConfig)", got, + pe("@formal_in:0", "122:2", configSpell), + pe("122:2", "122:9/actual_in:0", ":"+configSpell)) + if !reaches(got, "@formal_in:0", "122:9/actual_in:0") { + t.Errorf("wire(useConfig): @formal_in:0 does not reach 122:9/actual_in:0") + } + + k := globalsKey("callsGet") + inst := ":" + sdgPkg + "::inst" + var aout, fout string + for _, v := range p.vertices(k) { + switch { + case v.Kind == KindActualOut && v.Of == inst: + aout = v.Key + case v.Kind == KindFormalOut && v.Of == inst: + fout = v.Key + } + } + if aout == "" || fout == "" { + t.Fatalf("callsGet has no inst actual_out / formal_out: %+v", p.vertices(k)) + } + host := strings.SplitN(aout, "/actual_out", 2)[0] + if n := nodeOf(p.in.L3[k], host); n.Parent != "" { + host = n.Parent + } + got = p.wire(k) + checkHas(t, "wire(callsGet)", got, pe(aout, host, inst), pe(host, fout, inst)) +} + +func nodeOf(r dataflow.Result, key string) dataflow.Node { + for _, n := range r.Nodes { + if n.Key == key { + return n + } + } + return dataflow.Node{} +} + +// TestWiring_L3EdgesUntouched: the augmented re-solve keeps every L3 ddg +// edge (weak defs kill nothing), no port edge repeats an L3 edge, none ends +// at @exit, every prov is reaching-defs, every endpoint is a kept body node +// or a vertex of the callable, and the output is sorted and unique. +func TestWiring_L3EdgesUntouched(t *testing.T) { + in, _, p := runPorts(t) + n := 0 + for _, u := range in.Units { + r, ok := in.L3[u.Key] + if !ok { + continue + } + re := p.resolve(u.Key) + l3 := map[dataflow.DdgEdge]bool{} + for _, e := range r.Ddg { + l3[e] = true + if !slices.Contains(re, e) { + t.Errorf("%s: re-solve lost L3 edge %+v", u.Key.Signature, e) + } + } + ends := map[string]bool{} + for _, nd := range r.Nodes { + ends[nd.Key] = true + } + for _, v := range p.vertices(u.Key) { + ends[v.Key] = true + } + got := p.wire(u.Key) + for i, e := range got { + n++ + if l3[dataflow.DdgEdge{Src: e.Src, Dst: e.Dst, Var: e.Var}] { + t.Errorf("%s: port edge %+v repeats an L3 edge", u.Key.Signature, e) + } + if e.Dst == dataflow.ExitKey || e.Prov != ProvReachingDefs || !ends[e.Src] || !ends[e.Dst] { + t.Errorf("%s: bad port edge %+v", u.Key.Signature, e) + } + if i > 0 && comparePortEdge(got[i-1], e) >= 0 { + t.Errorf("%s: port edges not sorted/unique at %+v", u.Key.Signature, e) + } + } + } + if n == 0 { + t.Fatal("no port edges on testdata/dataflow") + } +} + +// TestWiring_PortsCarryDependence mirrors the SDK's _ports_carry_dependence +// probe per callable: every sdg/flow callable with a formal_in has a port +// edge leaving one, unless none of its formal bases is used (no node but +// @entry uses its spelling, and no call site's targets read it, A4). +func TestWiring_PortsCarryDependence(t *testing.T) { + in, d, p := runPorts(t) + checked := 0 + for _, u := range fixtureUnits(in) { + if fails, ok := portsCarry(in, d, p, u); ok { + checked++ + } else if fails { + t.Errorf("%s: no port edge leaves a formal_in, though a formal base is used: %+v", u.Key.Signature, p.wire(u.Key)) + } + } + if checked < 20 { + t.Fatalf("only %d callables with a used formal_in: the gate covers too little", checked) + } +} + +// portsCarry checks u: ok when a port edge leaves one of its formal_ins; +// fails when none does though some formal base is used. Both false: exempt. +func portsCarry(in Input, d *dispatch, p *ports, u dataflow.Unit) (fails, ok bool) { + fin, _ := p.formals(Target{Callable: u.Key}) + if len(fin) == 0 { + return false, false + } + for _, e := range p.wire(u.Key) { + if strings.HasPrefix(e.Src, "@formal_in:") { + return false, true + } + } + return formalUsed(in, d, p, u, fin), false +} + +// formalUsed reports whether some formal_in base of u is used: a node other +// than @entry uses a path with its spelling, or (a global) a resolved call +// site's target reads it. A unit with no L3 result counts as using them all. +func formalUsed(in Input, d *dispatch, p *ports, u dataflow.Unit, fin []Vertex) bool { + r, ok := in.L3[u.Key] + if !ok { + return true + } + bases := map[string]bool{} + for _, v := range fin { + b := strings.TrimPrefix(strings.TrimPrefix(v.Of, ofCapture), ofGlobal) + bases[b] = true + } + for key, du := range r.DefUse { + if key == dataflow.EntryKey { + continue + } + for _, use := range du.Uses { + if bases[dataflow.SpellBase(use.Spell)] { + return true + } + } + } + for _, c := range u.Calls { + if p.pruned(u.Key, c.Key) { + continue + } + for _, t := range p.siteTargets(u.Key, c.Key) { + for _, g := range p.fps[t].GlobalReads { + if bases[g] { + return true + } + } + } + } + return false +} + +// TestWiring_L3SkippedPassThrough (ruling, Task 12 review): a unit with no +// L3 result gets capture sites hosted by the L1 call holding the literal, +// else @entry, and pass-through wiring: every formal_in feeds every actual_in, +// every actual_out every formal_out, and every formal_in every formal_out. +func TestWiring_L3SkippedPassThrough(t *testing.T) { + in, d, _ := runPorts(t) + skip := map[dataflow.L3Key]bool{sdgKey("CaptureSite"): true, sdgKey("CaptureArg"): true, flowKey("OnceDo"): true} + l3 := make(map[dataflow.L3Key]dataflow.Result, len(in.L3)) + for k, r := range in.L3 { + if !skip[k] { + l3[k] = r + } + } + in.L3 = l3 + p := newPorts(in, d) + + checkVertices(t, "skipped CaptureSite capture site", capturesOf(p.vertices(sdgKey("CaptureSite"))), []Vertex{ + ain("215:7", "0", ":x", "@entry"), + aout("215:7", "", ":x", "@entry"), + }) + checkVertices(t, "skipped CaptureArg capture site", capturesOf(p.vertices(sdgKey("CaptureArg"))), []Vertex{ + ain("267:13", "0", ":v", "267:7"), + }) + checkVertices(t, "skipped OnceDo capture site", capturesOf(p.vertices(flowKey("OnceDo"))), []Vertex{ + ain("588:10", "0", ":cfg", "588:2"), + aout("588:10", "", ":cfg", "588:2"), + }) + + got := p.wire(sdgKey("CaptureSite")) + want := []PortEdge{ + pe("215:7/actual_out", "@formal_out", ""), + pe("@formal_in:0", "215:7/actual_in:0", ":x"), + pe("@formal_in:0", "@formal_out", ""), + } + if !slices.Equal(got, want) { + t.Errorf("skipped wire(CaptureSite):\n got %+v\n want %+v", got, want) + } + checkHas(t, "skipped wire(CaptureArg)", p.wire(sdgKey("CaptureArg")), + pe("@formal_in:0", "267:13/actual_in:0", ":v"), + pe("@formal_in:0", "267:7/actual_in:0", "f"), + pe("267:7/actual_out:0", "@formal_out", ""), + pe("267:7/actual_out:1", "@formal_out", "")) +} + +// TestWiring_Deterministic: two builders over two loads agree. +func TestWiring_Deterministic(t *testing.T) { + in1, _, p1 := runPorts(t) + _, _, p2 := runPorts(t) + for _, u := range in1.Units { + if a, b := p1.wire(u.Key), p2.wire(u.Key); !slices.Equal(a, b) { + t.Errorf("%v: port edges differ across runs:\n%v\n%v", u.Key, a, b) + } + } +} diff --git a/testdata/dataflow/sdg/sdg.go b/testdata/dataflow/sdg/sdg.go index f68efee..242c116 100644 --- a/testdata/dataflow/sdg/sdg.go +++ b/testdata/dataflow/sdg/sdg.go @@ -286,3 +286,18 @@ func (sumT) Sum(xs ...int) int { return len(xs) } func MethodExprVariadic(v sumT) int { return sumT.Sum(v) } + +// CaptureNamed stores a literal that rewrites the named result err: the +// capture site's actual_out reaches err's formal_out (A7). +func CaptureNamed() (err error) { + h := func() { err = nil } + h() + return +} + +// CaptureParam stores a literal that writes through the by-ref param p: the +// capture site's actual_out reaches p's formal_out (A7). +func CaptureParam(p *Inner) { + f := func() { p.n = 1 } + f() +} From 723fe48ce6c91ba3ffd9f1c7eed96c1e68f0fc89 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 13:44:36 -0400 Subject: [PATCH 19/65] feat(l4): summaries and parameter edges --- internal/sdg/params.go | 136 +++++++++ internal/sdg/run.go | 35 ++- internal/sdg/summary.go | 577 +++++++++++++++++++++++++++++++++++ internal/sdg/summary_test.go | 454 +++++++++++++++++++++++++++ internal/sdg/vertices.go | 7 +- internal/sdg/wiring.go | 2 +- testdata/dataflow/sdg/sdg.go | 46 +++ 7 files changed, 1248 insertions(+), 9 deletions(-) create mode 100644 internal/sdg/params.go create mode 100644 internal/sdg/summary.go create mode 100644 internal/sdg/summary_test.go diff --git a/internal/sdg/params.go b/internal/sdg/params.go new file mode 100644 index 0000000..f1f7370 --- /dev/null +++ b/internal/sdg/params.go @@ -0,0 +1,136 @@ +package sdg + +import ( + "cmp" + "slices" + "strings" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// params returns the param_in and param_out edges of unit k (spec §4.7), +// one set per target: +// +// - at each resolved call site with vertices: param_in from each actual_in +// to each target's formal_in it binds, and param_out from each target's +// formal_out to the actual_out it binds; +// - at each capture site (A6): param_in from each capture's actual_in to +// the closure's `:` formal_in, and param_out from the closure's +// `:` formal_out to the site's actual_out. +// +// Var is the target formal's `of`. An actual whose `of` a target lacks (a +// global another target touches) binds nothing in it, so it gets no edge. +func (b *sdgBuilder) params(k dataflow.L3Key) (in, out []ParamEdge) { + u, ok := b.p.units[k] + if !ok { + return nil, nil + } + caller := Target{Callable: k} + taken := make(map[string]bool, len(u.Calls)) + for _, c := range u.Calls { + taken[c.Key] = true + s := b.p.callSite(k, c.Key, b.p.siteTargets(k, c.Key)) + for _, pb := range s.BindIn { + fi := b.formalInfo(pb.Target) + if j, ok := fi.inPos[pb.Formal]; ok { + in = append(in, ParamEdge{Src: Ref{caller, pb.Actual}, Dst: Ref{pb.Target, pb.Formal}, Var: fi.inOf[j]}) + } + } + for _, pb := range s.BindOut { + fi := b.formalInfo(pb.Target) + if j, ok := fi.outPos[pb.Formal]; ok { + out = append(out, ParamEdge{Src: Ref{pb.Target, pb.Formal}, Dst: Ref{caller, pb.Actual}, Var: fi.outOf[j]}) + } + } + } + for key := range b.p.keptKeys(k) { + taken[key] = true + } + for _, cs := range b.p.capSites(u, taken) { + lk, ok := b.p.byFn[cs.lit] + if !ok { + continue + } + lit := Target{Callable: lk} + fs := b.p.formalSet(lit) + for i, of := range cs.ofs { + if f, ok := fs.inByOf[of]; ok { + in = append(in, ParamEdge{Src: Ref{caller, cs.in[i]}, Dst: Ref{lit, f}, Var: of}) + } + if cs.out[i] == "" { + continue + } + if f, ok := fs.outByOf[of]; ok { + out = append(out, ParamEdge{Src: Ref{lit, f}, Dst: Ref{caller, cs.out[i]}, Var: of}) + } + } + } + return in, out +} + +// compareParamEdge orders param edges by (Src owner, Src key, Dst owner, +// Dst key, Var), owners by compareTarget: stable before ids are minted. +func compareParamEdge(a, b ParamEdge) int { + return cmp.Or( + compareTarget(a.Src.Owner, b.Src.Owner), strings.Compare(a.Src.Key, b.Src.Key), + compareTarget(a.Dst.Owner, b.Dst.Owner), strings.Compare(a.Dst.Key, b.Dst.Key), + strings.Compare(a.Var, b.Var), + ) +} + +func sortParamEdges(es []ParamEdge) []ParamEdge { + slices.SortFunc(es, compareParamEdge) + return slices.Compact(es) +} + +// fill computes the flows and fills res's L4 overlay, serially: +// +// - Callables[k] for every unit: its vertices, port edges and call-site +// summary edges; +// - ParamIn / ParamOut over every unit's call and capture sites, sorted +// by compareParamEdge; +// - the formal vertices (sorted by key) of every target with no unit that +// some param edge reaches: Externals[sig] for a stand-in, and +// Callables[k] (vertices only) for an interface method or a bodiless +// declaration (spec §4.8), so no param edge dangles. +func (b *sdgBuilder) fill(res *Result) { + b.computeFlows() + res.Callables = make(map[dataflow.L3Key]CallableSDG, len(b.in.Units)) + var ins, outs []ParamEdge + for _, u := range b.in.Units { + k := u.Key + res.Callables[k] = CallableSDG{ + Vertices: b.p.vertices(k), + Ddg: b.portEdges(k), + Summary: b.summaryEdges(k), + } + pi, po := b.params(k) + ins, outs = append(ins, pi...), append(outs, po...) + } + res.ParamIn, res.ParamOut = sortParamEdges(ins), sortParamEdges(outs) + + // The targets with no unit some param edge reaches: their formals. + opaque := map[Target]bool{} + for _, e := range res.ParamIn { + opaque[e.Dst.Owner] = true + } + for _, e := range res.ParamOut { + opaque[e.Src.Owner] = true + } + for t := range opaque { + if _, unit := b.p.units[t.Callable]; unit && t.External == "" { + continue + } + in, out := b.p.formals(t) + vs := slices.Concat(in, out) + slices.SortFunc(vs, func(a, b Vertex) int { return strings.Compare(a.Key, b.Key) }) + if t.External == "" { + res.Callables[t.Callable] = CallableSDG{Vertices: vs} + continue + } + if res.Externals == nil { + res.Externals = map[string][]Vertex{} + } + res.Externals[t.External] = vs + } +} diff --git a/internal/sdg/run.go b/internal/sdg/run.go index 595df48..3f8d2e2 100644 --- a/internal/sdg/run.go +++ b/internal/sdg/run.go @@ -14,10 +14,12 @@ import ( // vtaCallGraph is vta.CallGraph (a test seam for the recovered-panic path). var vtaCallGraph = vta.CallGraph -// Run builds the L4 overlay of in. For now it builds SSA over the project -// packages, joins it to the L3 units, resolves every call site's dispatch -// targets (L2's static callee ∪ VTA's callees) and returns the call-graph -// edges and externals only VTA found, and the degradation: no vertices yet. +// Run builds the L4 overlay of in. It builds SSA over the project packages, +// joins it to the L3 units, resolves every call site's dispatch targets +// (L2's static callee ∪ VTA's callees), and returns the call-graph edges and +// externals only VTA found, the degradation, and the SDG: every unit's +// vertices, port edges and summaries, the param edges, and the stand-in +// formals. // // Order: the join is computed before VTA, so it never reads a program a VTA // panic may have wedged; after no_vta, later stages use the program for the @@ -30,8 +32,16 @@ func Run(in Input) Result { } // run is Run that also returns the call-site dispatch. Every site keeps its -// static L2 targets whether or not SSA or VTA ran. +// static L2 targets whether or not SSA or VTA ran, so the SDG is built on +// every path. func run(in Input) (Result, *dispatch) { + res, d := dispatchRun(in) + overlay(in, d, &res) + return res, d +} + +// dispatchRun is run up to the dispatch: SSA, the join and VTA. +func dispatchRun(in Input) (Result, *dispatch) { d := newDispatch(in) start := time.Now() prog, pkgs, deg := buildSSA(in) @@ -79,6 +89,21 @@ func run(in Input) (Result, *dispatch) { return res, d } +// overlay fills res's vertices, port edges, summaries and param edges +// (summary.go, params.go), serially. It reads the dispatch, the L3 results +// and go/types, never the SSA program, so it runs after a no_vta too (the +// dispatch then holds the L2 static targets). +func overlay(in Input, d *dispatch, res *Result) { + start := time.Now() + newSDGBuilder(in, d).fill(res) + var nv, nd, ns int + for _, c := range res.Callables { + nv, nd, ns = nv+len(c.Vertices), nd+len(c.Ddg), ns+len(c.Summary) + } + utils.Info("L4: %d vertices, %d port edges, %d summary edges, %d param_in, %d param_out, %d stand-ins in %s", + nv, nd, ns, len(res.ParamIn), len(res.ParamOut), len(res.Externals), time.Since(start).Round(time.Millisecond)) +} + // funcSet is projectFuncs under recover: it builds functions on demand // (MethodValue wrappers), so a builder panic is turned into an error. It // returns the project functions (the join is measured on them) and the VTA diff --git a/internal/sdg/summary.go b/internal/sdg/summary.go new file mode 100644 index 0000000..33f736c --- /dev/null +++ b/internal/sdg/summary.go @@ -0,0 +1,577 @@ +package sdg + +import ( + "math/bits" + "slices" + "strings" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// sdgBuilder computes the flow summaries (spec §4.9) and the L4 overlay of +// a run over one ports. Like ports, it is not safe for concurrent use. +type sdgBuilder struct { + in Input + d *dispatch + p *ports + flows map[Target]*flowSet + info map[Target]*formalInfo + sites map[dataflow.L3Key][]siteFlow + graph map[dataflow.L3Key]*unitGraph + wired map[dataflow.L3Key][]PortEdge + done bool +} + +func newSDGBuilder(in Input, d *dispatch) *sdgBuilder { + return &sdgBuilder{ + in: in, + d: d, + p: newPorts(in, d), + flows: map[Target]*flowSet{}, + info: map[Target]*formalInfo{}, + sites: map[dataflow.L3Key][]siteFlow{}, + graph: map[dataflow.L3Key]*unitGraph{}, + wired: map[dataflow.L3Key][]PortEdge{}, + } +} + +// flowSet is a target's flows: bit (i, j) is set iff formal_in i may flow +// to formal_out j. +type flowSet struct { + nIn, nOut, words int + bits []uint64 +} + +func newFlowSet(nIn, nOut int) *flowSet { + w := (nOut + 63) / 64 + return &flowSet{nIn: nIn, nOut: nOut, words: w, bits: make([]uint64, nIn*w)} +} + +func (f *flowSet) set(i, j int) { f.bits[i*f.words+j/64] |= 1 << (j % 64) } + +func (f *flowSet) has(i, j int) bool { + return f != nil && i < f.nIn && j < f.nOut && f.bits[i*f.words+j/64]&(1<<(j%64)) != 0 +} + +func (f *flowSet) count() int { + n := 0 + if f != nil { + for _, w := range f.bits { + n += bits.OnesCount64(w) + } + } + return n +} + +// equal reports whether f and g hold the same flows (nil holds none). +func (f *flowSet) equal(g *flowSet) bool { + if f.count() == 0 || g.count() == 0 { + return f.count() == g.count() + } + return f.nIn == g.nIn && f.nOut == g.nOut && slices.Equal(f.bits, g.bits) +} + +func (f *flowSet) pairs() [][2]int { + var out [][2]int + if f == nil { + return nil + } + for i := range f.nIn { + for j := range f.nOut { + if f.has(i, j) { + out = append(out, [2]int{i, j}) + } + } + } + return out +} + +// formalInfo is a target's formals by index: their keys' positions and +// their `of`s. +type formalInfo struct { + inPos, outPos map[string]int + inOf, outOf []string +} + +func (b *sdgBuilder) formalInfo(t Target) *formalInfo { + if fi, ok := b.info[t]; ok { + return fi + } + in, out := b.p.formals(t) + fi := &formalInfo{inPos: make(map[string]int, len(in)), outPos: make(map[string]int, len(out))} + for i, v := range in { + fi.inPos[v.Key] = i + fi.inOf = append(fi.inOf, v.Of) + } + for j, v := range out { + fi.outPos[v.Key] = j + fi.outOf = append(fi.outOf, v.Of) + } + b.info[t] = fi + return fi +} + +// siteFlow is one resolved call site of a unit: its actuals and, per +// target, which actual binds which of that target's formals (by index). +type siteFlow struct { + in, out []Vertex + binds []siteBind +} + +type siteBind struct { + t Target + ins, outs []bindPos +} + +// bindPos binds the actual at index actual (into siteFlow.in or .out) to a +// target's formal at index formal. +type bindPos struct{ actual, formal int } + +// sitesOf returns the resolved call sites of unit k that have vertices (the +// sites vertices() gives actuals), in u.Calls order. Capture sites are not +// calls (spec §4.6) and are not among them. +func (b *sdgBuilder) sitesOf(k dataflow.L3Key) []siteFlow { + if s, ok := b.sites[k]; ok { + return s + } + var out []siteFlow + for _, c := range b.p.units[k].Calls { + s := b.p.callSite(k, c.Key, b.p.siteTargets(k, c.Key)) + if len(s.In)+len(s.Out) == 0 { + continue + } + sf := siteFlow{in: s.In, out: s.Out} + inIdx := make(map[string]int, len(s.In)) + for i, v := range s.In { + inIdx[v.Key] = i + } + outIdx := make(map[string]int, len(s.Out)) + for i, v := range s.Out { + outIdx[v.Key] = i + } + byT := map[Target]*siteBind{} + var order []Target + bind := func(t Target) *siteBind { + sb, ok := byT[t] + if !ok { + sb = &siteBind{t: t} + byT[t] = sb + order = append(order, t) + } + return sb + } + for _, pb := range s.BindIn { + if j, ok := b.formalInfo(pb.Target).inPos[pb.Formal]; ok { + sb := bind(pb.Target) + sb.ins = append(sb.ins, bindPos{inIdx[pb.Actual], j}) + } + } + for _, pb := range s.BindOut { + if j, ok := b.formalInfo(pb.Target).outPos[pb.Formal]; ok { + sb := bind(pb.Target) + sb.outs = append(sb.outs, bindPos{outIdx[pb.Actual], j}) + } + } + slices.SortFunc(order, compareTarget) + for _, t := range order { + sf.binds = append(sf.binds, *byT[t]) + } + out = append(out, sf) + } + b.sites[k] = out + return out +} + +// matrix is the site's summary under the current flows: bit i*len(out)+j is +// set iff some target has a flow from the formal actual_in i binds to the +// formal actual_out j binds. +func (b *sdgBuilder) matrix(s siteFlow) []uint64 { + n := len(s.in) * len(s.out) + m := make([]uint64, (n+63)/64) + for _, sb := range s.binds { + f := b.flows[sb.t] + if f.count() == 0 { + continue + } + for _, ip := range sb.ins { + for _, op := range sb.outs { + if f.has(ip.formal, op.formal) { + bit := ip.actual*len(s.out) + op.actual + m[bit/64] |= 1 << (bit % 64) + } + } + } + } + return m +} + +// summaryEdges is the summary edges of every call site of unit k under the +// current flows, sorted by (Src, Dst). +func (b *sdgBuilder) summaryEdges(k dataflow.L3Key) []SummaryEdge { + var out []SummaryEdge + for _, s := range b.sitesOf(k) { + m := b.matrix(s) + for i, ai := range s.in { + for j, ao := range s.out { + if bit := i*len(s.out) + j; m[bit/64]&(1<<(bit%64)) != 0 { + out = append(out, SummaryEdge{Src: ai.Key, Dst: ao.Key}) + } + } + } + } + slices.SortFunc(out, compareSummaryEdge) + return slices.Compact(out) +} + +func compareSummaryEdge(a, b SummaryEdge) int { + if c := strings.Compare(a.Src, b.Src); c != 0 { + return c + } + return strings.Compare(a.Dst, b.Dst) +} + +// unitGraph is an analysed unit's flow graph without its summary edges: L3 +// ddg ∪ cdg ∪ the port wiring, over node indices. +type unitGraph struct { + idx map[string]int + succ [][]int32 + ins, outs []int // node of each formal_in / formal_out, by formal index +} + +func (b *sdgBuilder) portEdges(k dataflow.L3Key) []PortEdge { + if e, ok := b.wired[k]; ok { + return e + } + e := b.p.wire(k) + b.wired[k] = e + return e +} + +func (b *sdgBuilder) graphOf(k dataflow.L3Key) *unitGraph { + if g, ok := b.graph[k]; ok { + return g + } + g := &unitGraph{idx: map[string]int{}} + node := func(key string) int { + i, ok := g.idx[key] + if !ok { + i = len(g.succ) + g.idx[key] = i + g.succ = append(g.succ, nil) + } + return i + } + edge := func(src, dst string) { + s := node(src) + g.succ[s] = append(g.succ[s], int32(node(dst))) + } + in, out := b.p.formals(Target{Callable: k}) + for _, v := range in { + g.ins = append(g.ins, node(v.Key)) + } + for _, v := range out { + g.outs = append(g.outs, node(v.Key)) + } + r := b.in.L3[k] + for _, e := range r.Ddg { + edge(e.Src, e.Dst) + } + for _, e := range r.Cdg { + edge(e.Src, e.Dst) + } + for _, e := range b.portEdges(k) { + edge(e.Src, e.Dst) + } + for _, s := range b.sitesOf(k) { + for _, v := range slices.Concat(s.in, s.out) { + node(v.Key) + } + } + for i, s := range g.succ { + slices.Sort(s) + g.succ[i] = slices.Compact(s) + } + b.graph[k] = g + return g +} + +// opaque reports whether t's flows are not computed from a body: a stand-in, +// an interface method, a bodiless declaration, or a unit without the L3 +// facts the port wiring needs (l3_skipped, A16). +func (b *sdgBuilder) opaque(t Target) bool { + if t.External != "" { + return true + } + u, ok := b.p.units[t.Callable] + return !ok || !analysed(b.in, u) +} + +// opaqueFlows is the flows of an opaque target: every formal_in → every +// formal_out (spec §6, §4.8 D7, A16), unless Models has an entry for the +// stand-in (keyed /; flows [in-of, out-of]; an empty list +// is no flow). An interface method's flows are all-to-all whatever its VTA +// targets are, so the D7 union over them adds nothing. +func (b *sdgBuilder) opaqueFlows(t Target) *flowSet { + fi := b.formalInfo(t) + f := newFlowSet(len(fi.inOf), len(fi.outOf)) + if model, ok := b.model(t); ok { + for _, pair := range model { + i, j := slices.Index(fi.inOf, pair[0]), slices.Index(fi.outOf, pair[1]) + if i < 0 || j < 0 { + utils.Debug("L4: model of %s names no formal: %v", t.External, pair) + continue + } + f.set(i, j) + } + return f + } + for i := range f.nIn { + for j := range f.nOut { + f.set(i, j) + } + } + return f +} + +func (b *sdgBuilder) model(t Target) ([][2]string, bool) { + if t.External == "" || len(b.in.Models) == 0 { + return nil, false + } + e, ok := b.p.fx.ext[t.External] + if !ok { + return nil, false + } + m, ok := b.in.Models[e.ImportPath+"/"+e.Sig] + return m, ok +} + +// bodyFlows is the flows of analysed unit k under its callees' current +// flows: forward reachability from each @formal_in over the unit graph plus +// the summary edges at its call sites; (i, j) iff @formal_out j is reached +// from @formal_in i. Reachability from all formal_ins is solved at once: the +// graph's SCCs reachable from a formal_in are condensed (one source bitset +// per SCC) and propagated once, in topological order. +func (b *sdgBuilder) bodyFlows(k dataflow.L3Key) *flowSet { + g := b.graphOf(k) + f := newFlowSet(len(g.ins), len(g.outs)) + if len(g.ins) == 0 || len(g.outs) == 0 { + return f + } + extra := map[int][]int32{} + for _, s := range b.sitesOf(k) { + m := b.matrix(s) + for i, ai := range s.in { + src := g.idx[ai.Key] + for j, ao := range s.out { + if bit := i*len(s.out) + j; m[bit/64]&(1<<(bit%64)) != 0 { + extra[src] = append(extra[src], int32(g.idx[ao.Key])) + } + } + } + } + succ := func(n int) ([]int32, []int32) { return g.succ[n], extra[n] } + comp, order := condense(len(g.succ), g.ins, succ) + w := (len(g.ins) + 63) / 64 + reach := make([]uint64, len(order)*w) + for i, n := range g.ins { + c := comp[n] + reach[int(c)*w+i/64] |= 1 << (i % 64) + } + // order is sinks first: walk it backwards, so every SCC is complete + // before it is pushed to its successors. + for ci := len(order) - 1; ci >= 0; ci-- { + src := reach[ci*w : (ci+1)*w] + for _, n := range order[ci] { + a, x := succ(n) + for _, list := range [2][]int32{a, x} { + for _, m := range list { + cm := int(comp[m]) + if cm == ci { + continue + } + dst := reach[cm*w : (cm+1)*w] + for z := range dst { + dst[z] |= src[z] + } + } + } + } + } + for j, n := range g.outs { + c := comp[n] + if c < 0 { + continue + } + row := reach[int(c)*w : (int(c)+1)*w] + for i := range g.ins { + if row[i/64]&(1<<(i%64)) != 0 { + f.set(i, j) + } + } + } + return f +} + +// condense returns the SCCs of the graph of n nodes and succ reachable from +// roots (iterative Tarjan): comp[v] is v's SCC index (-1 when unreached), +// and order the SCCs' members, sinks first (an SCC comes after every SCC it +// reaches). +func condense(n int, roots []int, succ func(int) ([]int32, []int32)) ([]int32, [][]int) { + comp := make([]int32, n) + index := make([]int32, n) + low := make([]int32, n) + for i := range comp { + comp[i], index[i] = -1, -1 + } + onStack := make([]bool, n) + var stack []int + var order [][]int + type frame struct{ v, next int } + var calls []frame + next := int32(0) + edge := func(v, i int) (int, bool) { + a, x := succ(v) + if i < len(a) { + return int(a[i]), true + } + if i -= len(a); i < len(x) { + return int(x[i]), true + } + return 0, false + } + visit := func(v int) { + index[v], low[v] = next, next + next++ + stack = append(stack, v) + onStack[v] = true + calls = append(calls, frame{v: v}) + } + for _, r := range roots { + if index[r] >= 0 { + continue + } + visit(r) + for len(calls) > 0 { + top := len(calls) - 1 + fr := &calls[top] + v := fr.v + if w, ok := edge(v, fr.next); ok { + fr.next++ + switch { + case index[w] < 0: + visit(w) + case onStack[w]: + low[v] = min(low[v], index[w]) + } + continue + } + calls = calls[:top] + if top > 0 { + p := calls[top-1].v + low[p] = min(low[p], low[v]) + } + if low[v] != index[v] { + continue + } + i := len(stack) - 1 + for stack[i] != v { + i-- + } + members := slices.Clone(stack[i:]) + stack = stack[:i] + c := int32(len(order)) + for _, m := range members { + onStack[m] = false + comp[m] = c + } + order = append(order, members) + } + } + return comp, order +} + +// solve recomputes t's flows and reports whether they changed. +func (b *sdgBuilder) solve(t Target) bool { + var f *flowSet + if b.opaque(t) { + if _, ok := b.flows[t]; ok { + return false // constant + } + f = b.opaqueFlows(t) + } else { + f = b.bodyFlows(t.Callable) + } + old, had := b.flows[t] + b.flows[t] = f + return !had && f.count() > 0 || had && !old.equal(f) +} + +// summarize computes the flows of every unit and call target (spec §4.9): +// (i, j) pairs, formal_in index i → formal_out index j, sorted. Callees come +// first over the call graph's SCCs (callGraph, sccs); a recursive SCC +// iterates to its least fixpoint. +func (b *sdgBuilder) summarize() map[Target][][2]int { + b.computeFlows() + out := make(map[Target][][2]int, len(b.flows)) + for t, f := range b.flows { + out[t] = f.pairs() + } + return out +} + +func (b *sdgBuilder) computeFlows() { + if b.done { + return + } + b.done = true + nodes, callees := callGraph(b.in, b.d) + comps := sccs(nodes, func(t Target) []Target { return callees[t] }) + solveSCCs(comps, callees, b.solve) +} + +// solveSCCs solves comps in order (callees first, as sccs returns them). A +// non-recursive singleton is solved once. A recursive SCC (several members, +// or one calling itself) is iterated to a fixpoint with a worklist: every +// member once, then each member whose callee in the SCC changed. solve +// recomputes a target's flows from its callees' current ones and reports a +// change; it must be monotone, so over finite flow sets this terminates. +func solveSCCs(comps [][]Target, callees map[Target][]Target, solve func(Target) bool) { + for _, comp := range comps { + if len(comp) == 1 && !slices.Contains(callees[comp[0]], comp[0]) { + solve(comp[0]) + continue + } + in := make(map[Target]bool, len(comp)) + for _, m := range comp { + in[m] = true + } + callers := map[Target][]Target{} + for _, m := range comp { + for _, c := range callees[m] { + if in[c] { + callers[c] = append(callers[c], m) + } + } + } + work := slices.Clone(comp) + queued := make(map[Target]bool, len(comp)) + for _, m := range comp { + queued[m] = true + } + for len(work) > 0 { + m := work[0] + work = work[1:] + queued[m] = false + if !solve(m) { + continue + } + for _, c := range callers[m] { + if !queued[c] { + queued[c] = true + work = append(work, c) + } + } + } + } +} diff --git a/internal/sdg/summary_test.go b/internal/sdg/summary_test.go new file mode 100644 index 0000000..159f0ab --- /dev/null +++ b/internal/sdg/summary_test.go @@ -0,0 +1,454 @@ +package sdg + +import ( + "cmp" + "slices" + "strings" + "testing" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +func se(src, dst string) SummaryEdge { return SummaryEdge{Src: src, Dst: dst} } + +func checkSummary(t *testing.T, what string, got, want []SummaryEdge) { + t.Helper() + if !slices.Equal(got, want) { + t.Errorf("%s:\n got %+v\n want %+v", what, got, want) + } +} + +// siteSummaryOf returns the summary edges of res's unit k whose source is an +// actual of call (or capture) key call. +func siteSummaryOf(res Result, k dataflow.L3Key, call string) []SummaryEdge { + var out []SummaryEdge + for _, e := range res.Callables[k].Summary { + if strings.HasPrefix(e.Src, call+"/") { + out = append(out, e) + } + } + return out +} + +func checkFlows(t *testing.T, flows map[Target][][2]int, tg Target, want [][2]int) { + t.Helper() + got, ok := flows[tg] + if !ok { + t.Errorf("no flows for %v", tg) + return + } + if !slices.Equal(got, want) { + t.Errorf("flows of %v = %v, want %v", tg, got, want) + } +} + +// TestSummary_TransitiveFlow: a(x) → b(x) → c(x) returns x, so c and b each +// have the flow x → , and a's call of b has the summary edge +// actual_in:0 → actual_out. +func TestSummary_TransitiveFlow(t *testing.T) { + _, res, _ := runDispatch(t) + b := newSDGBuilder(runPortsInput(t)) + flows := b.summarize() + for _, name := range []string{"a", "b", "c"} { + checkFlows(t, flows, sdgTarget(name), [][2]int{{0, 0}}) + } + checkSummary(t, "summary of a", res.Callables[sdgKey("a")].Summary, + []SummaryEdge{se("308:9/actual_in:0", "308:9/actual_out")}) + checkSummary(t, "summary of b", res.Callables[sdgKey("b")].Summary, + []SummaryEdge{se("312:9/actual_in:0", "312:9/actual_out")}) + if s := res.Callables[sdgKey("c")].Summary; len(s) != 0 { + t.Errorf("c calls nothing; summary = %+v", s) + } +} + +// TestSummary_NoFlowNoEdge: ignore(x, y) returns y, so its only flow is y → +// , and its call site has no edge from actual_in:0. +func TestSummary_NoFlowNoEdge(t *testing.T) { + _, res, _ := runDispatch(t) + flows := newSDGBuilder(runPortsInput(t)).summarize() + checkFlows(t, flows, sdgTarget("ignore"), [][2]int{{1, 0}}) + checkSummary(t, "summary of callIgnore", res.Callables[sdgKey("callIgnore")].Summary, + []SummaryEdge{se("326:9/actual_in:1", "326:9/actual_out")}) +} + +// TestSummary_RecursiveFixpoint: dropX and dropY form one recursive SCC. +// Its fixpoint is the least one: dropX's x flows nowhere, so dropY's call +// of dropX has no edge from actual_in:0. +func TestSummary_RecursiveFixpoint(t *testing.T) { + _, res, _ := runDispatch(t) + flows := newSDGBuilder(runPortsInput(t)).summarize() + checkFlows(t, flows, sdgTarget("dropX"), [][2]int{{1, 0}}) + checkFlows(t, flows, sdgTarget("dropY"), [][2]int{{0, 0}}) + checkSummary(t, "summary of dropY", res.Callables[sdgKey("dropY")].Summary, + []SummaryEdge{se("339:9/actual_in:1", "339:9/actual_out")}) + checkSummary(t, "summary of dropX", res.Callables[sdgKey("dropX")].Summary, + []SummaryEdge{se("335:9/actual_in:0", "335:9/actual_out")}) +} + +// TestSummary_FixpointIterates: the SCC driver iterates a recursive SCC to +// its least fixpoint. In this synthetic graph p → q → p, p's flow exists only +// once q's does and q's only through r (outside the SCC) and p, so a single +// pass in member order (p first) misses p's flow. +func TestSummary_FixpointIterates(t *testing.T) { + p, q, r := Target{External: "p"}, Target{External: "q"}, Target{External: "r"} + callees := map[Target][]Target{p: {q}, q: {p, r}} + comps := sccs([]Target{p, q, r}, func(t Target) []Target { return callees[t] }) + flows := map[Target][][2]int{} + solves := 0 + solve := func(t Target) bool { + solves++ + var f [][2]int + switch t { + case r: + f = [][2]int{{0, 0}} + case q: // q passes r's flow on + f = flows[r] + case p: // p passes q's flow on + f = flows[q] + } + old, had := flows[t] + flows[t] = f + return !had && len(f) > 0 || had && !slices.Equal(old, f) + } + solveSCCs(comps, callees, solve) + for _, tg := range []Target{p, q, r} { + checkFlows(t, flows, tg, [][2]int{{0, 0}}) + } + if solves < 4 { + t.Errorf("solved %d times: the recursive SCC was not iterated", solves) + } +} + +// TestSummary_NoBodyTargetsAllToAll: an interface method, a stand-in and an +// l3_skipped unit get every formal_in → every formal_out; a Models entry +// overrides a stand-in, and an empty one cuts every flow. +func TestSummary_NoBodyTargetsAllToAll(t *testing.T) { + in, d, _ := runPorts(t) + flows := newSDGBuilder(in, d).summarize() + all := func(nIn, nOut int) [][2]int { + var out [][2]int + for i := range nIn { + for j := range nOut { + out = append(out, [2]int{i, j}) + } + } + return out + } + // Store.Get: formal_in _recv, k; formal_out , _recv. + checkFlows(t, flows, sdgTarget("Store.Get"), all(2, 2)) + write := Target{External: "io.Writer.Write"} + if _, ok := flows[write]; !ok { + t.Fatalf("no flows for %v; targets of Buffered: %v", write, d.targets(sdgKey("Buffered"), "94:2")) + } + checkFlows(t, flows, write, all(2, 4)) + + in.Models = Models{"io/Writer.Write": {{"p", ":0"}}} + flows = newSDGBuilder(in, d).summarize() + checkFlows(t, flows, write, [][2]int{{1, 0}}) + in.Models = Models{"io/Writer.Write": {}} + flows = newSDGBuilder(in, d).summarize() + checkFlows(t, flows, write, nil) + + skip := map[dataflow.L3Key]bool{sdgKey("ignore"): true} + l3 := make(map[dataflow.L3Key]dataflow.Result, len(in.L3)) + for k, r := range in.L3 { + if !skip[k] { + l3[k] = r + } + } + in.L3, in.Models = l3, nil + flows = newSDGBuilder(in, d).summarize() + checkFlows(t, flows, sdgTarget("ignore"), all(2, 1)) +} + +// TestParams_PerTargetWithVar: UseStore's s.Get(k) has one param_in set per +// target (the interface method and both implementations), each var that +// target's formal name, and one param_out set per target. +func TestParams_PerTargetWithVar(t *testing.T) { + _, res, _ := runDispatch(t) + use := sdgTarget("UseStore") + iface, mem, sql := sdgTarget("Store.Get"), sdgTarget("memStore.Get"), sdgTarget("sqlStore.Get") + ref := func(o Target, k string) Ref { return Ref{Owner: o, Key: k} } + var gotIn, gotOut []ParamEdge + for _, e := range res.ParamIn { + if e.Src.Owner == use { + gotIn = append(gotIn, e) + } + } + for _, e := range res.ParamOut { + if e.Dst.Owner == use { + gotOut = append(gotOut, e) + } + } + wantIn := []ParamEdge{ + {ref(use, "34:9/actual_in:0"), ref(iface, "@formal_in:0"), "_recv"}, + {ref(use, "34:9/actual_in:0"), ref(mem, "@formal_in:0"), "s"}, + {ref(use, "34:9/actual_in:0"), ref(sql, "@formal_in:0"), "s"}, + {ref(use, "34:9/actual_in:1"), ref(iface, "@formal_in:1"), "k"}, + {ref(use, "34:9/actual_in:1"), ref(mem, "@formal_in:1"), "k"}, + {ref(use, "34:9/actual_in:1"), ref(sql, "@formal_in:1"), "k"}, + } + slices.SortFunc(wantIn, compareParamEdge) + if !slices.Equal(gotIn, wantIn) { + t.Errorf("param_in of UseStore:\n got %+v\n want %+v", gotIn, wantIn) + } + wantOut := []ParamEdge{ + {ref(iface, "@formal_out:0"), ref(use, "34:9/actual_out:0"), ""}, + {ref(iface, "@formal_out:1"), ref(use, "34:9/actual_out:1"), "_recv"}, + {ref(mem, "@formal_out:0"), ref(use, "34:9/actual_out:0"), ""}, + {ref(mem, "@formal_out:1"), ref(use, "34:9/actual_out:1"), "s"}, + {ref(sql, "@formal_out:0"), ref(use, "34:9/actual_out:0"), ""}, + {ref(sql, "@formal_out:1"), ref(use, "34:9/actual_out:1"), "s"}, + } + slices.SortFunc(wantOut, compareParamEdge) + if !slices.Equal(gotOut, wantOut) { + t.Errorf("param_out of UseStore:\n got %+v\n want %+v", gotOut, wantOut) + } + // The interface method passes everything through (D7), so the site's + // summary is all-to-all. + checkSummary(t, "summary of UseStore", res.Callables[sdgKey("UseStore")].Summary, []SummaryEdge{ + se("34:9/actual_in:0", "34:9/actual_out:0"), se("34:9/actual_in:0", "34:9/actual_out:1"), + se("34:9/actual_in:1", "34:9/actual_out:0"), se("34:9/actual_in:1", "34:9/actual_out:1"), + }) +} + +// TestParams_CaptureSite: a stored literal's capture site gets param_in and +// param_out to the closure's capture formals, and no summary edge. +func TestParams_CaptureSite(t *testing.T) { + _, res, _ := runDispatch(t) + host, lit := sdgTarget("CaptureSite"), sdgTarget("CaptureSite.closure_1") + want := []ParamEdge{ + {Ref{host, "215:7/actual_in:0"}, Ref{lit, "@formal_in:0"}, ":x"}, + } + var got []ParamEdge + for _, e := range res.ParamIn { + if e.Src.Owner == host { + got = append(got, e) + } + } + if !slices.Equal(got, want) { + t.Errorf("param_in of CaptureSite:\n got %+v\n want %+v", got, want) + } + wantOut := []ParamEdge{ + {Ref{lit, "@formal_out:1"}, Ref{host, "215:7/actual_out"}, ":x"}, + } + got = nil + for _, e := range res.ParamOut { + if e.Dst.Owner == host { + got = append(got, e) + } + } + if !slices.Equal(got, wantOut) { + t.Errorf("param_out of CaptureSite:\n got %+v\n want %+v", got, wantOut) + } + if s := siteSummaryOf(res, sdgKey("CaptureSite"), "215:7"); len(s) != 0 { + t.Errorf("capture site has summary edges %+v", s) + } +} + +// vertexSet is the vertex keys of every owner in res. +func vertexSet(res Result) map[Target]map[string]bool { + out := map[Target]map[string]bool{} + add := func(t Target, vs []Vertex) { + m := map[string]bool{} + for _, v := range vs { + m[v.Key] = true + } + out[t] = m + } + for k, c := range res.Callables { + add(Target{Callable: k}, c.Vertices) + } + for sig, vs := range res.Externals { + add(Target{External: sig}, vs) + } + return out +} + +// TestParams_NoDanglingEndpoints: every param_in / param_out endpoint and +// every summary edge endpoint names a vertex of its owner; the lists are +// sorted and unique. +func TestParams_NoDanglingEndpoints(t *testing.T) { + check := func(t *testing.T, res Result) { + t.Helper() + if len(res.ParamIn) == 0 || len(res.ParamOut) == 0 { + t.Fatalf("param_in %d, param_out %d: want both", len(res.ParamIn), len(res.ParamOut)) + } + vs := vertexSet(res) + for _, list := range [][]ParamEdge{res.ParamIn, res.ParamOut} { + for _, e := range list { + for _, r := range []Ref{e.Src, e.Dst} { + if !vs[r.Owner][r.Key] { + t.Errorf("edge %+v: %v has no vertex %s", e, r.Owner, r.Key) + } + } + if e.Var == "" { + t.Errorf("edge %+v has no var", e) + } + } + if !slices.IsSortedFunc(list, compareParamEdge) { + t.Error("param edges are not sorted") + } + if len(slices.CompactFunc(slices.Clone(list), func(a, b ParamEdge) bool { return a == b })) != len(list) { + t.Error("param edges repeat") + } + } + for k, c := range res.Callables { + own := vs[Target{Callable: k}] + for _, e := range c.Summary { + if !own[e.Src] || !own[e.Dst] { + t.Errorf("%v: summary edge %+v dangles", k, e) + } + } + if !slices.IsSortedFunc(c.Summary, compareSummaryEdge) { + t.Errorf("%v: summary not sorted", k) + } + } + } + t.Run("vta", func(t *testing.T) { + _, res, _ := runDispatch(t) + check(t, res) + }) + t.Run("no_vta", func(t *testing.T) { + in := loadInput(t, "dataflow") + defer func(f func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph) { vtaCallGraph = f }(vtaCallGraph) + vtaCallGraph = func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph { panic("vta boom") } + res := Run(in) + if !hasCode(res.Degraded, CodeNoVTA) { + t.Fatalf("degraded = %+v, want no_vta", res.Degraded) + } + check(t, res) + }) +} + +// TestSDG_RunFillsCallablesAndExternals: Run gives every unit its vertices, +// port edges and summary, every stand-in target its formals, and an +// interface method its formals alone. +func TestSDG_RunFillsCallablesAndExternals(t *testing.T) { + in, res, _ := runDispatch(t) + for _, u := range in.Units { + if _, ok := res.Callables[u.Key]; !ok { + t.Errorf("no CallableSDG for unit %v", u.Key) + } + } + got := res.Callables[sdgKey("UseStore")] + if len(got.Vertices) == 0 || len(got.Ddg) == 0 { + t.Errorf("UseStore: %d vertices, %d port edges", len(got.Vertices), len(got.Ddg)) + } + iface := res.Callables[sdgKey("Store.Get")] + checkVertices(t, "interface method Store.Get", iface.Vertices, + []Vertex{fin("0", "_recv"), fin("1", "k"), fout(":0", ""), fout(":1", "_recv")}) + if len(iface.Ddg)+len(iface.Summary) != 0 { + t.Errorf("Store.Get has edges: %+v", iface) + } + vs, ok := res.Externals["io.Writer.Write"] + if !ok { + t.Fatalf("Externals lack io.Writer.Write: %v", keysOf(res.Externals)) + } + // p ([]byte) and the receiver are by-ref outs. + want := []Vertex{fin("0", "_recv"), fin("1", "p"), fout(":0", ":0"), fout(":1", ":1"), fout(":2", "_recv"), fout(":3", "p")} + slices.SortFunc(want, func(a, b Vertex) int { return cmp.Compare(a.Key, b.Key) }) + checkVertices(t, "stand-in io.Writer.Write", vs, want) +} + +func keysOf[V any](m map[string]V) []string { + var out []string + for k := range m { + out = append(out, k) + } + slices.Sort(out) + return out +} + +// TestSDG_Deterministic: two runs over two loads give equal L4 overlays. +func TestSDG_Deterministic(t *testing.T) { + _, a, _ := runDispatch(t) + _, b, _ := runDispatch(t) + if !slices.Equal(a.ParamIn, b.ParamIn) || !slices.Equal(a.ParamOut, b.ParamOut) { + t.Error("param edges differ across runs") + } + for k, c := range a.Callables { + d := b.Callables[k] + if !slices.Equal(c.Vertices, d.Vertices) || !slices.Equal(c.Ddg, d.Ddg) || !slices.Equal(c.Summary, d.Summary) { + t.Errorf("%v differs across runs", k) + } + } +} + +// runPortsInput is runPorts' input and dispatch. +func runPortsInput(t *testing.T) (Input, *dispatch) { + t.Helper() + in, d, _ := runPorts(t) + return in, d +} + +// TestSummary_ImplicitFlowViaCdg: implicit's x reaches no return over the +// ddg, only over the cdg (the returns depend on the if), so the flow x → +// needs the cdg in the graph (spec §4.9, Python precedent). +func TestSummary_ImplicitFlowViaCdg(t *testing.T) { + flows := newSDGBuilder(runPortsInput(t)).summarize() + checkFlows(t, flows, sdgTarget("implicit"), [][2]int{{0, 0}}) +} + +// TestSummary_CondensedReachMatchesBFS: bodyFlows (one pass over the +// condensed graph) equals a plain BFS from each formal_in over the same +// edges, on every analysed fixture unit, under the final callee flows. +func TestSummary_CondensedReachMatchesBFS(t *testing.T) { + b := newSDGBuilder(runPortsInput(t)) + b.computeFlows() + n := 0 + for _, u := range b.in.Units { + k := u.Key + if b.opaque(Target{Callable: k}) { + continue + } + g := b.graphOf(k) + succ := map[int][]int{} + for v, s := range g.succ { + for _, m := range s { + succ[v] = append(succ[v], int(m)) + } + } + for _, s := range b.sitesOf(k) { + m := b.matrix(s) + for i, ai := range s.in { + for j, ao := range s.out { + if bit := i*len(s.out) + j; m[bit/64]&(1<<(bit%64)) != 0 { + succ[g.idx[ai.Key]] = append(succ[g.idx[ai.Key]], g.idx[ao.Key]) + } + } + } + } + var want [][2]int + for i, src := range g.ins { + seen := map[int]bool{src: true} + work := []int{src} + for len(work) > 0 { + v := work[0] + work = work[1:] + for _, m := range succ[v] { + if !seen[m] { + seen[m] = true + work = append(work, m) + } + } + } + for j, o := range g.outs { + if seen[o] { + want = append(want, [2]int{i, j}) + } + } + } + slices.SortFunc(want, func(a, b [2]int) int { return cmp.Or(cmp.Compare(a[0], b[0]), cmp.Compare(a[1], b[1])) }) + if got := b.bodyFlows(k).pairs(); !slices.Equal(got, want) { + t.Errorf("%v: bodyFlows %v, BFS %v", k, got, want) + } + n++ + } + if n < 50 { + t.Errorf("checked %d units", n) + } +} diff --git a/internal/sdg/vertices.go b/internal/sdg/vertices.go index c9723a9..a8f869a 100644 --- a/internal/sdg/vertices.go +++ b/internal/sdg/vertices.go @@ -623,11 +623,12 @@ func (p *ports) captureSites(u dataflow.Unit, taken map[string]bool) []Vertex { return out } -// capSite is one capture site: its key, its host, and each capture (in -// actual_in order) with its actual_in key and, when the literal writes it, +// capSite is one capture site: its key, its host, its literal, and each +// capture (in actual_in order) with its actual_in key and, when the literal writes it, // its actual_out key. type capSite struct { key, host string + lit *ast.FuncLit caps []types.Object ofs []string in, out []string // per capture; out[i] == "" when not written @@ -694,7 +695,7 @@ func (p *ports) capSites(u dataflow.Unit, taken map[string]bool) []capSite { } } pos := p.in.Fset.PositionFor(l.Lit.Pos(), false) - cs := capSite{key: dataflow.NextFreeKey(v2.LocalID(pos.Line, pos.Column), taken), host: host, caps: caps} + cs := capSite{key: dataflow.NextFreeKey(v2.LocalID(pos.Line, pos.Column), taken), host: host, lit: l.Lit, caps: caps} j := 0 for i, v := range caps { cs.ofs = append(cs.ofs, ofCapture+v.Name()) diff --git a/internal/sdg/wiring.go b/internal/sdg/wiring.go index 8755572..4d95c0d 100644 --- a/internal/sdg/wiring.go +++ b/internal/sdg/wiring.go @@ -97,7 +97,7 @@ func (p *ports) wire(k dataflow.L3Key) []PortEdge { } for i, fo := range w.fs.results { - if w.resObjs[i] != nil { + if i < len(w.resObjs) && w.resObjs[i] != nil { continue } of := resultOf(i, len(w.fs.results)) diff --git a/testdata/dataflow/sdg/sdg.go b/testdata/dataflow/sdg/sdg.go index 242c116..0e98e69 100644 --- a/testdata/dataflow/sdg/sdg.go +++ b/testdata/dataflow/sdg/sdg.go @@ -301,3 +301,49 @@ func CaptureParam(p *Inner) { f := func() { p.n = 1 } f() } + +// a passes x to b, which passes it to c, which returns it: a's call of b +// has a summary edge from its actual_in:0 to its actual_out. +func a(x int) int { + return b(x) +} + +func b(x int) int { + return c(x) +} + +func c(x int) int { + return x +} + +// ignore never uses x: no summary edge leaves the actual_in:0 of a call +// of it. +func ignore(x, y int) int { + return y +} + +func callIgnore(v, w int) int { + return ignore(v, w) +} + +// dropX and dropY are mutually recursive and dropX never uses x: no summary +// edge leaves the actual_in:0 of dropY's call of dropX. +func dropX(x, y int) int { + if y == 0 { + return y + } + return dropY(y - 1) +} + +func dropY(y int) int { + return dropX(0, y) +} + +// implicit's result depends on x only through control: x reaches +// over the cdg. +func implicit(x int) int { + if x > 0 { + return 1 + } + return 0 +} From 34ecbdef24f59e0eede917db66c73bb3322ceeda Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 16:02:47 -0400 Subject: [PATCH 20/65] feat(l4): library stand-ins and --library-models --- README.md | 1 + cmd/codeanalyzer/main.go | 7 + cmd/codeanalyzer/main_test.go | 73 ++++++++ docs/design/specs/v2-l3-l4-dataflow.md | 22 ++- internal/core/analyzer.go | 8 + internal/core/sdg_test.go | 48 ++++++ internal/options/options.go | 3 + internal/sdg/ir.go | 10 +- internal/sdg/models.go | 137 +++++++++++++++ internal/sdg/models_test.go | 222 +++++++++++++++++++++++++ internal/sdg/params.go | 59 ++++--- internal/sdg/summary.go | 67 ++++---- internal/sdg/summary_test.go | 50 ++++++ testdata/dataflow/sdg/globals.go | 13 ++ testdata/dataflow/sdg/stdlib.go | 34 ++++ 15 files changed, 689 insertions(+), 65 deletions(-) create mode 100644 internal/sdg/models.go create mode 100644 internal/sdg/models_test.go create mode 100644 testdata/dataflow/sdg/stdlib.go diff --git a/README.md b/README.md index f3326ac..4110360 100644 --- a/README.md +++ b/README.md @@ -172,6 +172,7 @@ Flags: -h, --help help for cango -i, --input string Project root to analyze (required) -j, --jobs int Worker parallelism (default: CPU cores) + --library-models string JSON file of library stand-in flows overriding the all-to-all default (-a 4 and --emit neo4j only) --neo4j-database string Neo4j database (env NEO4J_DATABASE, optional) --neo4j-password string Neo4j password (env NEO4J_PASSWORD, default neo4j) --neo4j-uri string Live Bolt push target (env NEO4J_URI); omit to write graph.cypher diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index 382f8c1..8c755c1 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -41,6 +41,7 @@ func rootCmd() *cobra.Command { level int fieldDepth int strict bool + libraryModels string targetFiles []string skipTests bool eager bool @@ -86,6 +87,9 @@ via CLDK(language="go").analysis(project_path=...).`, if err := validateFieldDepth(cmd, level, fieldDepth); err != nil { return err } + if cmd.Flags().Changed("library-models") && level < int(options.LevelSDG) { + return fmt.Errorf("--library-models applies only at analysis level 4") + } case options.EmitNeo4j: if cmd.Flags().Changed("analysis-level") { return fmt.Errorf("--analysis-level does not apply to --emit neo4j; the graph is always projected at full depth") @@ -146,6 +150,7 @@ via CLDK(language="go").analysis(project_path=...).`, AppName: appName, Level: options.AnalysisLevel(level), Strict: strict, + LibraryModels: libraryModels, AnalyzerVersion: version, TargetFiles: targetFiles, SkipTests: skipTests, @@ -209,6 +214,8 @@ via CLDK(language="go").analysis(project_path=...).`, f.IntVar(&fieldDepth, "graph-field-depth", 3, "Field-path depth bound for L3 dataflow nodes (k limit; applies at -a 3 and -a 4)") f.BoolVar(&strict, "strict", false, "Fail non-zero on any L4 degradation (default: degrade and record it)") + f.StringVar(&libraryModels, "library-models", "", + "JSON file of library stand-in flows overriding the all-to-all default (-a 4 and --emit neo4j only)") f.StringSliceVarP(&targetFiles, "target-files", "t", nil, "Restrict analysis to specific files (incremental mode)") f.BoolVar(&skipTests, "skip-tests", true, "Skip *_test.go files") diff --git a/cmd/codeanalyzer/main_test.go b/cmd/codeanalyzer/main_test.go index 5c147a4..8e9d324 100644 --- a/cmd/codeanalyzer/main_test.go +++ b/cmd/codeanalyzer/main_test.go @@ -633,3 +633,76 @@ func TestCLI_StrictFlag(t *testing.T) { t.Error("stdout is not valid JSON") } } + +// ── --library-models ───────────────────────────────────────────────────────── + +const errLibraryModelsLevel = "--library-models applies only at analysis level 4" + +func writeModelsFile(t *testing.T, body string) string { + t.Helper() + path := filepath.Join(t.TempDir(), "models.json") + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + return path +} + +func TestCLI_LibraryModelsBelowLevel4Rejected(t *testing.T) { + td := cliTestdataDir() + models := writeModelsFile(t, `{"strconv/Atoi": []}`) + for _, level := range []string{"1", "2", "3"} { + _, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-level", level, + "--library-models", models, + "--cache-dir", t.TempDir(), + ) + if err == nil || err.Error() != errLibraryModelsLevel { + t.Errorf("-a %s --library-models: error = %v, want %q", level, err, errLibraryModelsLevel) + } + } +} + +func TestCLI_LibraryModelsAtLevel4AndNeo4j(t *testing.T) { + td := cliTestdataDir() + models := writeModelsFile(t, `{"fmt/Sprintf": [["format", ""]]}`) + out, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-level", "4", + "--library-models", models, + "--cache-dir", t.TempDir(), + ) + if err != nil { + t.Fatalf("-a 4 --library-models should succeed; got %v", err) + } + if !json.Valid([]byte(out)) { + t.Error("stdout is not valid JSON") + } + if _, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--emit", "neo4j", + "--library-models", models, + "--output", t.TempDir(), + "--cache-dir", t.TempDir(), + ); err != nil { + t.Fatalf("--emit neo4j --library-models should succeed; got %v", err) + } +} + +func TestCLI_LibraryModelsBadFileErrors(t *testing.T) { + td := cliTestdataDir() + for name, path := range map[string]string{ + "missing": filepath.Join(t.TempDir(), "absent.json"), + "malformed": writeModelsFile(t, `{"fmt/Sprintf": [`), + } { + _, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-level", "4", + "--library-models", path, + "--cache-dir", t.TempDir(), + ) + if err == nil || !strings.Contains(err.Error(), path) { + t.Errorf("%s: error = %v, want one naming %s", name, err, path) + } + } +} diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index a2ce5bf..9ed1ee3 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -604,6 +604,14 @@ Context-insensitive. Per callable compute: predicate, §4.6), propagated up the SCC DAG and injected at hosts as uses / weak defs (A4). - No hammock regions (neither sibling built them; recorded as a known simplification). - Stand-ins (§6) contribute their modelled summary (default all-inputs → all-outputs). +- **Global → global summary edges are not emitted [user, 2026-10-26].** A `summary` edge whose + `actual_in` and `actual_out` both have a `:` `of` is omitted; every edge with at least + one non-global end is kept, and the flows (and so the global formals, actuals and param edges) + are computed in full. Under the coarse host wiring (A15) a summary edge only shortcuts + host → `actual_in` → … → `actual_out` → host, and sources and sinks address `formal_in`, so a + global → global summary adds no reachability for an HRB-unaware walk. Measured on nats-server: + 97.7% of 101M summary edges were global → global (10.8 GB peak RSS against D3's 1.5×). They + return with the HRB two-phase walk (§11). - **Unanalysed callables (amended, A16).** A callable whose L3 result was dropped (a contained panic or a call-count mismatch, `run.go`) has no `ddg`, so taint entering through `param_in` would stop at its `formal_in`. Such callables get formals and a **pass-through summary** (like an @@ -859,8 +867,17 @@ addition now; the cross-language SDK taint revisit is deferred (§11). present"); not made worse by stand-ins. Recorded as a known over-approximation. - **Models file (optional overrides):** `--library-models ` (builder may also embed a small default set), mapping an external id/signature to an explicit flow list, e.g. - `strconv.Atoi: [] ` (no flow), `strings.TrimSpace: [[0,""]]`. Starts empty or + `{"strconv/Atoi": [], "strings/TrimSpace": [["s",""]]}`. Starts empty or near-empty; no catalogue of sources/sinks — that stays the caller's (SDK rule). + **Format (amended, Task 15):** a JSON object; key = `/`, the external id's + tail (`strconv/Atoi`, `database/sql/DB.Query`); value = a list of `[in, out]` pairs of `of` + strings (`"s"`, `"_0"`, `"_recv"`, `""`, `":1"`, a by-ref param's name). + `[]` = no flow; a missing key = the all-to-all default. Valid only at `-a 4` and with + `--emit neo4j` (`-a 1..3`: error `--library-models applies only at analysis level 4`); a + missing, unreadable or malformed file is an error naming the path. An entry naming an `of` + the external lacks is a warning and is ignored for that external, which keeps the all-to-all + default (a typo never refutes a flow); an entry for an external no call site targets is ignored + (Debug). - **Not modelled:** callbacks invoked by library code (`sort.Slice(xs, less)`, `http.HandleFunc(path, h)`): the function-value argument flows in, but the library calling it back is not an edge. Taint *starting* in a handler is unaffected (sources are the handler's own @@ -1007,7 +1024,8 @@ dispatch + type-based may-alias). multi-node selectors in `commons`); promote `external_symbols.body` stand-ins into the keystone if adopted. - Andersen points-to over `go/ssa` replacing the type-based oracle (`prov` unchanged). -- Context-sensitive (HRB two-phase) taint traversal in the SDK. +- Context-sensitive (HRB two-phase) taint traversal in the SDK. With it, reinstate the global → + global `summary` edges omitted at L4 (§4.9, [user, 2026-10-26]). - Library callback modelling (`sort.Slice`, `http.HandleFunc`). - Promote `callable.source_file` to the keystone (existing deferral). - Struct-tag modelling (existing deferral). diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index 414338f..b8001f9 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -77,6 +77,13 @@ func (a *Analyzer) Analyze() (*schema.GoApplication, error) { a.opts.InputPath = abs utils.Info("analyzing project: %s", a.opts.InputPath) + // The L4 library models are read first, so a bad file fails the run + // before any analysis (also under --emit neo4j -t, which runs at L3). + models, err := sdg.LoadModels(a.opts.LibraryModels) + if err != nil { + return nil, err + } + // ── Phase 1: Project materialization ────────────────────────────────────── if err := a.materialize(); err != nil { // Degrade gracefully — log but don't abort. Partial types are better than nothing. @@ -147,6 +154,7 @@ func (a *Analyzer) Analyze() (*schema.GoApplication, error) { L3: a.l3, K: KLimit(a.opts), Jobs: a.opts.Jobs, + Models: models, }) a.l4 = &res if err := a.reportDegraded(res.Degraded); err != nil { diff --git a/internal/core/sdg_test.go b/internal/core/sdg_test.go index a8bccdc..e69d61b 100644 --- a/internal/core/sdg_test.go +++ b/internal/core/sdg_test.go @@ -1,6 +1,7 @@ package core_test import ( + "os" "path/filepath" "strings" "testing" @@ -107,3 +108,50 @@ func TestAnalyze_Level4MergesVTAEdges(t *testing.T) { t.Errorf("-a 3: VTA edge = %v, external = %v; want neither", vta, ext) } } + +// TestAnalyze_LibraryModels: --library-models is loaded into the L4 input. By +// default Greet's call of fmt.Sprintf has summary edges; the model +// fmt/Sprintf: [] (no flow) leaves it none. A missing or malformed file +// fails the run, naming the path. +func TestAnalyze_LibraryModels(t *testing.T) { + summaries := func(t *testing.T, models string) int { + t.Helper() + a := core.New(options.AnalysisOptions{ + InputPath: greeterDir(t), Level: options.LevelSDG, SkipTests: true, CacheDir: t.TempDir(), LibraryModels: models, + }) + if _, err := a.Analyze(); err != nil { + t.Fatal(err) + } + n := 0 + for k, c := range a.L4().Callables { + if strings.HasSuffix(k.Signature, "Greeter.Greet") { + n += len(c.Summary) + } + } + return n + } + if n := summaries(t, ""); n == 0 { + t.Fatal("Greet has no summary edges by default") + } + dir := t.TempDir() + good := filepath.Join(dir, "models.json") + if err := os.WriteFile(good, []byte(`{"fmt/Sprintf": []}`), 0o644); err != nil { + t.Fatal(err) + } + if n := summaries(t, good); n != 0 { + t.Errorf("fmt/Sprintf modelled with no flow: Greet has %d summary edges", n) + } + + bad := filepath.Join(dir, "bad.json") + if err := os.WriteFile(bad, []byte(`{"fmt/Sprintf": [`), 0o644); err != nil { + t.Fatal(err) + } + for _, path := range []string{filepath.Join(dir, "absent.json"), bad} { + _, err := core.New(options.AnalysisOptions{ + InputPath: greeterDir(t), Level: options.LevelSDG, SkipTests: true, CacheDir: t.TempDir(), LibraryModels: path, + }).Analyze() + if err == nil || !strings.Contains(err.Error(), path) { + t.Errorf("%s: Analyze error = %v, want one naming the path", path, err) + } + } +} diff --git a/internal/options/options.go b/internal/options/options.go index d4e1ecc..01a1af8 100644 --- a/internal/options/options.go +++ b/internal/options/options.go @@ -45,6 +45,9 @@ type AnalysisOptions struct { Level AnalysisLevel // Strict makes the run fail non-zero on any L4 degradation (--strict). Strict bool + // LibraryModels is the --library-models JSON file overriding library + // stand-in summaries at L4 (sdg.LoadModels). Empty = the defaults. + LibraryModels string // AnalyzerVersion is the analyzer's own version, recorded in the v2 // manifest's analyzer{} tag. Set by the CLI from its build-stamped value. AnalyzerVersion string diff --git a/internal/sdg/ir.go b/internal/sdg/ir.go index 147a3c4..d0d9bae 100644 --- a/internal/sdg/ir.go +++ b/internal/sdg/ir.go @@ -94,12 +94,14 @@ type Input struct { Units []dataflow.Unit L3 map[dataflow.L3Key]dataflow.Result K, Jobs int - Models Models // Task 15; zero value = defaults + Models Models // --library-models (LoadModels); nil = the defaults } -// Models overrides library stand-in summaries, keyed by the external id's -// / tail; each pair is a (src, dst) port edge. Loading and -// applying them comes later (ruling R-P4); the zero value means defaults. +// Models overrides library stand-in summaries (spec §6), keyed by the +// external id's / tail (`strconv/Atoi`): each pair is a +// flow [in, out] from the stand-in's formal_in of `of` in to its formal_out +// of `of` out. An empty list is no flow; an external with no entry keeps +// every input → every output. type Models map[string][][2]string // Degraded codes (spec §4.15; an open vocabulary of plain strings). diff --git a/internal/sdg/models.go b/internal/sdg/models.go new file mode 100644 index 0000000..f103232 --- /dev/null +++ b/internal/sdg/models.go @@ -0,0 +1,137 @@ +package sdg + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "slices" + + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// warnf logs a warning (a test seam over utils.Warn). +var warnf = utils.Warn + +// LoadModels reads a --library-models file (spec §6): a JSON object whose +// keys are the external id's / tail (`strconv/Atoi`, +// `database/sql/DB.Query`) and whose values are lists of [in, out] pairs of +// `of` strings (`"s"`, `"_0"`, `"_recv"`, `""`, `":1"`, a +// by-ref param's name). An empty list is no flow. An empty path gives +// Models{}. +// +// Only the shape is checked here: which `of`s an external has is known once +// its formals are, so a pair naming one it lacks is reported by the summary +// pass (opaqueFlows), which then keeps that external's default. +func LoadModels(path string) (Models, error) { + if path == "" { + return Models{}, nil + } + data, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("--library-models: %w", err) + } + m, err := parseModels(data) + if err != nil { + return nil, fmt.Errorf("--library-models %s: %w", path, err) + } + return m, nil +} + +func parseModels(data []byte) (Models, error) { + var raw map[string]json.RawMessage + dec := json.NewDecoder(bytes.NewReader(data)) + if err := dec.Decode(&raw); err != nil { + return nil, err + } + if _, err := dec.Token(); !errors.Is(err, io.EOF) { + return nil, errors.New("data after the top-level object") + } + if raw == nil { + return nil, errors.New("want a JSON object, got null") + } + out := make(Models, len(raw)) + for key, v := range raw { + var pairs [][]string + if err := json.Unmarshal(v, &pairs); err != nil { + return nil, fmt.Errorf("%s: want a list of [in, out] pairs: %w", key, err) + } + if pairs == nil { + return nil, fmt.Errorf("%s: want a list of [in, out] pairs, got null", key) + } + flows := make([][2]string, 0, len(pairs)) + for _, p := range pairs { + if len(p) != 2 { + return nil, fmt.Errorf("%s: pair %q: want [in, out]", key, p) + } + flows = append(flows, [2]string{p[0], p[1]}) + } + out[key] = flows + } + return out, nil +} + +// modelKey is the Models key of external t: /. +func (b *sdgBuilder) modelKey(t Target) (string, bool) { + e, ok := b.p.fx.ext[t.External] + if !ok { + return "", false + } + return e.ImportPath + "/" + e.Sig, true +} + +// model returns the Models entry of stand-in t, if any. +func (b *sdgBuilder) model(t Target) ([][2]string, string, bool) { + if t.External == "" || len(b.in.Models) == 0 { + return nil, "", false + } + key, ok := b.modelKey(t) + if !ok { + return nil, "", false + } + m, ok := b.in.Models[key] + return m, key, ok +} + +// modelFlows is stand-in t's flows under its model, or false when it has +// none or the model names an `of` t lacks: the caller then keeps the +// all-to-all default (a typo must never refute a flow). The unknown `of`s +// are one warning. +func (b *sdgBuilder) modelFlows(t Target, fi *formalInfo) (*flowSet, bool) { + model, key, ok := b.model(t) + if !ok { + return nil, false + } + b.modelled[key] = true + f := newFlowSet(len(fi.inOf), len(fi.outOf)) + var unknown []string + for _, pair := range model { + i, j := slices.Index(fi.inOf, pair[0]), slices.Index(fi.outOf, pair[1]) + if i < 0 { + unknown = append(unknown, "in "+pair[0]) + } + if j < 0 { + unknown = append(unknown, "out "+pair[1]) + } + if i >= 0 && j >= 0 { + f.set(i, j) + } + } + if len(unknown) > 0 { + warnf("--library-models: %s has no formal %v (formal_in %v, formal_out %v); keeping its default, every input to every output", + key, unknown, fi.inOf, fi.outOf) + return nil, false + } + return f, true +} + +// unusedModels logs, at Debug, every Models entry no targeted external has. +func (b *sdgBuilder) unusedModels() { + for key := range b.in.Models { + if !b.modelled[key] { + utils.Debug("L4: --library-models: no call site targets %s; entry ignored", key) + } + } +} diff --git a/internal/sdg/models_test.go b/internal/sdg/models_test.go new file mode 100644 index 0000000..d16b81b --- /dev/null +++ b/internal/sdg/models_test.go @@ -0,0 +1,222 @@ +package sdg + +import ( + "cmp" + "fmt" + "os" + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// stdlibKey is the L3Key of a callable declared in +// testdata/dataflow/sdg/stdlib.go. +func stdlibKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("sdg", "stdlib.go"), Signature: sdgPkg + "." + name} +} + +func byKey(vs []Vertex) []Vertex { + vs = slices.Clone(vs) + slices.SortFunc(vs, func(a, b Vertex) int { return cmp.Compare(a.Key, b.Key) }) + return vs +} + +// TestStandIn_VerticesFromSignature: fmt.Sprintf's stand-in has formal_in +// format and a (its declared params, named from go/types) and formal_out +// ; a ([]any, a slice) is also a by-ref out (the shared predicate). +func TestStandIn_VerticesFromSignature(t *testing.T) { + _, res, _ := runDispatch(t) + vs, ok := res.Externals["fmt.Sprintf"] + if !ok { + t.Fatalf("Externals lack fmt.Sprintf: %v", keysOf(res.Externals)) + } + checkVertices(t, "stand-in fmt.Sprintf", vs, + byKey([]Vertex{fin("0", "format"), fin("1", "a"), fout(":0", ""), fout(":1", "a")})) +} + +// TestStandIn_ByRefOut: json.Unmarshal(data []byte, v any) may write through +// v, so v is a formal_out after the result (data, a slice, is one too). +func TestStandIn_ByRefOut(t *testing.T) { + _, res, _ := runDispatch(t) + vs, ok := res.Externals["encoding/json.Unmarshal"] + if !ok { + t.Fatalf("Externals lack encoding/json.Unmarshal: %v", keysOf(res.Externals)) + } + checkVertices(t, "stand-in encoding/json.Unmarshal", vs, byKey([]Vertex{ + fin("0", "data"), fin("1", "v"), + fout(":0", ""), fout(":1", "data"), fout(":2", "v"), + })) +} + +// TestStandIn_EveryTargetedExternal: Externals holds exactly the externals +// some call site targets, each with the formals ports gives it; one with no +// formals at all (runtime.Gosched) is listed with an empty vertex list. +func TestStandIn_EveryTargetedExternal(t *testing.T) { + in, res, d := runDispatch(t) + p := newPorts(in, d) + want := map[string]bool{} + for _, ts := range d.sites { + for _, tg := range ts { + if tg.External != "" { + want[tg.External] = true + } + } + } + if got := keysOf(res.Externals); !slices.Equal(got, keysOf(want)) { + t.Errorf("Externals = %v, want %v", got, keysOf(want)) + } + for sig, vs := range res.Externals { + fi, fo := p.formals(Target{External: sig}) + checkVertices(t, "stand-in "+sig, vs, byKey(slices.Concat(fi, fo))) + } + vs, ok := res.Externals["runtime.Gosched"] + if !ok || vs == nil || len(vs) != 0 { + t.Errorf("runtime.Gosched: listed %v, vertices %#v; want an empty, non-nil list", ok, vs) + } +} + +// fillWith fills the L4 overlay of the dataflow fixture under models, and +// returns it with the warnings it logged. +func fillWith(t *testing.T, models Models) (Result, []string) { + t.Helper() + in, d := runPortsInput(t) + in.Models = models + var warns []string + defer func(f func(string, ...any)) { warnf = f }(warnf) + warnf = func(format string, args ...any) { warns = append(warns, fmt.Sprintf(format, args...)) } + var res Result + newSDGBuilder(in, d).fill(&res) + return res, warns +} + +const atoiSite = "27:10" + +// TestModels_OverrideCutsFlow: the model strconv/Atoi: [] (no flow) leaves +// atoi's call of it with no summary edge; by default it has one per result. +func TestModels_OverrideCutsFlow(t *testing.T) { + res, _ := fillWith(t, nil) + checkSummary(t, "default summary of atoi", siteSummaryOf(res, stdlibKey("atoi"), atoiSite), []SummaryEdge{ + se(atoiSite+"/actual_in:0", atoiSite+"/actual_out:0"), se(atoiSite+"/actual_in:0", atoiSite+"/actual_out:1"), + }) + res, warns := fillWith(t, Models{"strconv/Atoi": {}}) + if s := siteSummaryOf(res, stdlibKey("atoi"), atoiSite); len(s) != 0 { + t.Errorf("strconv/Atoi modelled with no flow; summary = %+v", s) + } + if len(warns) != 0 { + t.Errorf("warnings: %v", warns) + } +} + +// TestModels_PairsByOf: a model's pairs name formals by `of`: s → :0 +// keeps that one edge and drops the error result's. +func TestModels_PairsByOf(t *testing.T) { + res, warns := fillWith(t, Models{"strconv/Atoi": {{"s", ":0"}}}) + checkSummary(t, "summary of atoi", siteSummaryOf(res, stdlibKey("atoi"), atoiSite), + []SummaryEdge{se(atoiSite+"/actual_in:0", atoiSite+"/actual_out:0")}) + if len(warns) != 0 { + t.Errorf("warnings: %v", warns) + } +} + +// TestModels_UnknownOfFallsBackToAllToAll: a pair naming an `of` the +// external lacks is a warning, and the external keeps its all-to-all +// default: a typo never refutes a flow. +func TestModels_UnknownOfFallsBackToAllToAll(t *testing.T) { + for _, c := range []struct { + pair [2]string + unknown string + }{{[2]string{"str", ":0"}, "in str"}, {[2]string{"s", ""}, "out "}} { + pair := c.pair + res, warns := fillWith(t, Models{"strconv/Atoi": {{"s", ":1"}, pair}}) + checkSummary(t, fmt.Sprint("summary of atoi under ", pair), siteSummaryOf(res, stdlibKey("atoi"), atoiSite), []SummaryEdge{ + se(atoiSite+"/actual_in:0", atoiSite+"/actual_out:0"), se(atoiSite+"/actual_in:0", atoiSite+"/actual_out:1"), + }) + if len(warns) != 1 || !strings.Contains(warns[0], "strconv/Atoi") || !strings.Contains(warns[0], c.unknown) { + t.Errorf("pair %v: warnings %q, want one naming strconv/Atoi and the unknown of", pair, warns) + } + } +} + +// TestModels_UncalledEntryIgnored: an entry for an external no site targets +// is no warning and changes nothing. +func TestModels_UncalledEntryIgnored(t *testing.T) { + res, warns := fillWith(t, Models{"example.com/nosuch/F": {}}) + if len(warns) != 0 { + t.Errorf("warnings: %v", warns) + } + if s := siteSummaryOf(res, stdlibKey("atoi"), atoiSite); len(s) != 2 { + t.Errorf("summary of atoi = %+v, want the default", s) + } +} + +func writeModels(t *testing.T, body string) string { + t.Helper() + path := filepath.Join(t.TempDir(), "models.json") + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + return path +} + +// TestModels_Load: a models file maps / to [in, out] +// pairs; [] is no flow; an empty path is no models. +func TestModels_Load(t *testing.T) { + m, err := LoadModels(writeModels(t, `{ + "strconv/Atoi": [], + "strings/TrimSpace": [["s", ""]], + "database/sql/DB.Query": [["query", ":0"], ["args", ":0"]] + }`)) + if err != nil { + t.Fatal(err) + } + want := Models{ + "strconv/Atoi": {}, + "strings/TrimSpace": {{"s", ""}}, + "database/sql/DB.Query": {{"query", ":0"}, {"args", ":0"}}, + } + if len(m) != len(want) { + t.Fatalf("models = %v, want %v", m, want) + } + for k, v := range want { + got, ok := m[k] + if !ok || !slices.Equal(got, v) || got == nil { + t.Errorf("models[%s] = %#v, want %#v", k, got, v) + } + } + m, err = LoadModels("") + if err != nil || m == nil || len(m) != 0 { + t.Errorf(`LoadModels("") = %v, %v; want Models{}, nil`, m, err) + } +} + +// TestModels_BadFileErrors: a missing file, malformed JSON and a value that +// is not a list of [in, out] string pairs are errors naming the path. +func TestModels_BadFileErrors(t *testing.T) { + missing := filepath.Join(t.TempDir(), "absent.json") + paths := map[string]string{"missing": missing} + for name, body := range map[string]string{ + "malformed": `{"strconv/Atoi": [`, + "not object": `[]`, + "null": `null`, + "null value": `{"strconv/Atoi": null}`, + "short pair": `{"strconv/Atoi": [["s"]]}`, + "long pair": `{"strconv/Atoi": [["s", ":0", "x"]]}`, + "number": `{"strconv/Atoi": [[0, ":0"]]}`, + "trailing": `{} {}`, + } { + paths[name] = writeModels(t, body) + } + for name, path := range paths { + m, err := LoadModels(path) + if err == nil { + t.Errorf("%s: LoadModels = %v, want an error", name, m) + continue + } + if !strings.Contains(err.Error(), path) { + t.Errorf("%s: error %q does not name %s", name, err, path) + } + } +} diff --git a/internal/sdg/params.go b/internal/sdg/params.go index f1f7370..b2ca3f7 100644 --- a/internal/sdg/params.go +++ b/internal/sdg/params.go @@ -89,10 +89,12 @@ func sortParamEdges(es []ParamEdge) []ParamEdge { // summary edges; // - ParamIn / ParamOut over every unit's call and capture sites, sorted // by compareParamEdge; -// - the formal vertices (sorted by key) of every target with no unit that -// some param edge reaches: Externals[sig] for a stand-in, and -// Callables[k] (vertices only) for an interface method or a bodiless -// declaration (spec §4.8), so no param edge dangles. +// - Externals[sig]: the formal vertices (sorted by key) of every stand-in +// some call site targets (spec §6), an empty list for one with no +// formals (runtime.Gosched); +// - Callables[k] (vertices only) for every interface method or bodiless +// declaration some param edge reaches (spec §4.8), so no param edge +// dangles. func (b *sdgBuilder) fill(res *Result) { b.computeFlows() res.Callables = make(map[dataflow.L3Key]CallableSDG, len(b.in.Units)) @@ -109,28 +111,41 @@ func (b *sdgBuilder) fill(res *Result) { } res.ParamIn, res.ParamOut = sortParamEdges(ins), sortParamEdges(outs) - // The targets with no unit some param edge reaches: their formals. - opaque := map[Target]bool{} - for _, e := range res.ParamIn { - opaque[e.Dst.Owner] = true - } - for _, e := range res.ParamOut { - opaque[e.Src.Owner] = true - } - for t := range opaque { - if _, unit := b.p.units[t.Callable]; unit && t.External == "" { - continue - } + formals := func(t Target) []Vertex { in, out := b.p.formals(t) vs := slices.Concat(in, out) + if vs == nil { + vs = []Vertex{} + } slices.SortFunc(vs, func(a, b Vertex) int { return strings.Compare(a.Key, b.Key) }) - if t.External == "" { - res.Callables[t.Callable] = CallableSDG{Vertices: vs} - continue + return vs + } + for _, ts := range b.d.sites { + for _, t := range ts { + if t.External == "" { + continue + } + if res.Externals == nil { + res.Externals = map[string][]Vertex{} + } + if _, done := res.Externals[t.External]; !done { + res.Externals[t.External] = formals(t) + } } - if res.Externals == nil { - res.Externals = map[string][]Vertex{} + } + + // The in-project targets with no unit some param edge reaches. + for _, list := range [][]ParamEdge{res.ParamIn, res.ParamOut} { + for _, e := range list { + for _, t := range []Target{e.Src.Owner, e.Dst.Owner} { + if _, unit := b.p.units[t.Callable]; unit || t.External != "" { + continue + } + if _, done := res.Callables[t.Callable]; !done { + res.Callables[t.Callable] = CallableSDG{Vertices: formals(t)} + } + } } - res.Externals[t.External] = vs } + b.unusedModels() } diff --git a/internal/sdg/summary.go b/internal/sdg/summary.go index 33f736c..0c8d306 100644 --- a/internal/sdg/summary.go +++ b/internal/sdg/summary.go @@ -6,7 +6,6 @@ import ( "strings" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" - "github.com/codellm-devkit/codeanalyzer-go/internal/utils" ) // sdgBuilder computes the flow summaries (spec §4.9) and the L4 overlay of @@ -20,19 +19,22 @@ type sdgBuilder struct { sites map[dataflow.L3Key][]siteFlow graph map[dataflow.L3Key]*unitGraph wired map[dataflow.L3Key][]PortEdge - done bool + // modelled is the Models keys some targeted stand-in has. + modelled map[string]bool + done bool } func newSDGBuilder(in Input, d *dispatch) *sdgBuilder { return &sdgBuilder{ - in: in, - d: d, - p: newPorts(in, d), - flows: map[Target]*flowSet{}, - info: map[Target]*formalInfo{}, - sites: map[dataflow.L3Key][]siteFlow{}, - graph: map[dataflow.L3Key]*unitGraph{}, - wired: map[dataflow.L3Key][]PortEdge{}, + in: in, + d: d, + p: newPorts(in, d), + flows: map[Target]*flowSet{}, + info: map[Target]*formalInfo{}, + sites: map[dataflow.L3Key][]siteFlow{}, + graph: map[dataflow.L3Key]*unitGraph{}, + wired: map[dataflow.L3Key][]PortEdge{}, + modelled: map[string]bool{}, } } @@ -207,13 +209,25 @@ func (b *sdgBuilder) matrix(s siteFlow) []uint64 { } // summaryEdges is the summary edges of every call site of unit k under the -// current flows, sorted by (Src, Dst). +// current flows, sorted by (Src, Dst), except those from a global actual_in +// to a global actual_out (`of` `:` at both ends; ruling [user, +// 2026-10-26], spec §4.9). Under the coarse host wiring (A15) a summary edge +// only shortcuts host → actual_in → … → actual_out → host, and sources and +// sinks address formal_in, so a global → global summary adds no +// reachability for an HRB-unaware walk; it is omitted to bound the output +// (on nats-server 97.7% of all summary edges were global → global). They +// return with the deferred HRB two-phase walk (spec §11). The flows keep +// every pair: only the emitted list is filtered. func (b *sdgBuilder) summaryEdges(k dataflow.L3Key) []SummaryEdge { var out []SummaryEdge for _, s := range b.sitesOf(k) { m := b.matrix(s) for i, ai := range s.in { + inGlobal := strings.HasPrefix(ai.Of, ofGlobal) for j, ao := range s.out { + if inGlobal && strings.HasPrefix(ao.Of, ofGlobal) { + continue + } if bit := i*len(s.out) + j; m[bit/64]&(1<<(bit%64)) != 0 { out = append(out, SummaryEdge{Src: ai.Key, Dst: ao.Key}) } @@ -308,24 +322,15 @@ func (b *sdgBuilder) opaque(t Target) bool { } // opaqueFlows is the flows of an opaque target: every formal_in → every -// formal_out (spec §6, §4.8 D7, A16), unless Models has an entry for the -// stand-in (keyed /; flows [in-of, out-of]; an empty list -// is no flow). An interface method's flows are all-to-all whatever its VTA -// targets are, so the D7 union over them adds nothing. +// formal_out (spec §6, §4.8 D7, A16), unless it is a stand-in with a valid +// Models entry (modelFlows). An interface method's flows are all-to-all +// whatever its VTA targets are, so the D7 union over them adds nothing. func (b *sdgBuilder) opaqueFlows(t Target) *flowSet { fi := b.formalInfo(t) - f := newFlowSet(len(fi.inOf), len(fi.outOf)) - if model, ok := b.model(t); ok { - for _, pair := range model { - i, j := slices.Index(fi.inOf, pair[0]), slices.Index(fi.outOf, pair[1]) - if i < 0 || j < 0 { - utils.Debug("L4: model of %s names no formal: %v", t.External, pair) - continue - } - f.set(i, j) - } + if f, ok := b.modelFlows(t, fi); ok { return f } + f := newFlowSet(len(fi.inOf), len(fi.outOf)) for i := range f.nIn { for j := range f.nOut { f.set(i, j) @@ -334,18 +339,6 @@ func (b *sdgBuilder) opaqueFlows(t Target) *flowSet { return f } -func (b *sdgBuilder) model(t Target) ([][2]string, bool) { - if t.External == "" || len(b.in.Models) == 0 { - return nil, false - } - e, ok := b.p.fx.ext[t.External] - if !ok { - return nil, false - } - m, ok := b.in.Models[e.ImportPath+"/"+e.Sig] - return m, ok -} - // bodyFlows is the flows of analysed unit k under its callees' current // flows: forward reachability from each @formal_in over the unit graph plus // the summary edges at its call sites; (i, j) iff @formal_out j is reached diff --git a/internal/sdg/summary_test.go b/internal/sdg/summary_test.go index 159f0ab..a1f69c5 100644 --- a/internal/sdg/summary_test.go +++ b/internal/sdg/summary_test.go @@ -452,3 +452,53 @@ func TestSummary_CondensedReachMatchesBFS(t *testing.T) { t.Errorf("checked %d units", n) } } + +// TestSummary_OmitsGlobalToGlobal (ruling [user, 2026-10-26], spec §4.9): a +// call site whose callee reads and writes globals gets no summary edge from +// a global actual_in to a global actual_out, but the callee's flows still +// hold those pairs, and every other summary edge stands. +func TestSummary_OmitsGlobalToGlobal(t *testing.T) { + in, res, _ := runDispatch(t) + b := newSDGBuilder(runPortsInput(t)) + flows := b.summarize() + + // appendConfig(v): formal_in v, Config; formal_out , Config. + // Its flows keep Config → Config. + checkFlows(t, flows, Target{Callable: globalsKey("appendConfig")}, [][2]int{{0, 0}, {0, 1}, {1, 0}, {1, 1}}) + checkSummary(t, "summary of useAppend", res.Callables[globalsKey("useAppend")].Summary, []SummaryEdge{ + se("100:9/actual_in:0", "100:9/actual_out:0"), se("100:9/actual_in:0", "100:9/actual_out:1"), + se("100:9/actual_in:1", "100:9/actual_out:0"), + }) + // Get reads and writes inst and once (formal_in inst, once; formal_out + // , inst, once): inst → inst and once → once stay in its flows. + get := Target{Callable: globalsKey("Get")} + fl := flows[get] + for _, p := range [][2]int{{0, 1}, {1, 2}} { + if !slices.Contains(fl, p) { + t.Errorf("flows of Get = %v lack the global → global pair %v", fl, p) + } + } + checkSummary(t, "summary of callsGet", res.Callables[globalsKey("callsGet")].Summary, []SummaryEdge{ + se("68:33/actual_in:0", "68:33/actual_out:0"), se("68:33/actual_in:1", "68:33/actual_out:0"), + }) + // useConfig and readConfig only read Config: their global → + // edges are unchanged. + checkSummary(t, "summary of useConfig", res.Callables[sdgKey("useConfig")].Summary, + []SummaryEdge{se("122:9/actual_in:0", "122:9/actual_out")}) + checkSummary(t, "summary of readConfig", res.Callables[sdgKey("readConfig")].Summary, + []SummaryEdge{se("116:10/actual_in:0", "116:10/actual_out")}) + + // No unit anywhere keeps a global → global summary edge. + for _, u := range in.Units { + c := res.Callables[u.Key] + of := map[string]string{} + for _, v := range c.Vertices { + of[v.Key] = v.Of + } + for _, e := range c.Summary { + if strings.HasPrefix(of[e.Src], ofGlobal) && strings.HasPrefix(of[e.Dst], ofGlobal) { + t.Errorf("%v: global → global summary edge %+v", u.Key, e) + } + } + } +} diff --git a/testdata/dataflow/sdg/globals.go b/testdata/dataflow/sdg/globals.go index 6fce392..e4905ca 100644 --- a/testdata/dataflow/sdg/globals.go +++ b/testdata/dataflow/sdg/globals.go @@ -86,3 +86,16 @@ func calledLits() { defer func() { litH = 2 }() func() { litG = 1 }() } + +// appendConfig reads and writes Config (a weak write: Config flows to +// itself) and takes v: a call of it has the global → global summary +// Config → Config, which is not emitted, beside v → Config, v → +// and Config → , which are. +func appendConfig(v string) string { + Config += v + return Config +} + +func useAppend(v string) string { + return appendConfig(v) +} diff --git a/testdata/dataflow/sdg/stdlib.go b/testdata/dataflow/sdg/stdlib.go new file mode 100644 index 0000000..89d9ebe --- /dev/null +++ b/testdata/dataflow/sdg/stdlib.go @@ -0,0 +1,34 @@ +package sdg + +// The library calls of the L4 stand-in fixture; they live apart from sdg.go +// so that its line:col keys stay put. + +import ( + "encoding/json" + "fmt" + "runtime" + "strconv" +) + +// format calls fmt.Sprintf: its stand-in has formal_ins format and a. +func format(a int) string { + return fmt.Sprintf("%d", a) +} + +// decode calls json.Unmarshal, whose v (any) is a by-ref out. +func decode(data []byte) map[string]int { + var m map[string]int + _ = json.Unmarshal(data, &m) + return m +} + +// atoi calls strconv.Atoi: s flows to both results unless a model says not. +func atoi(s string) int { + n, _ := strconv.Atoi(s) + return n +} + +// yield calls runtime.Gosched, an external with no formals at all. +func yield() { + runtime.Gosched() +} From e340fdd268887193fd44d973aac0d8c6da4c9a6a Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 17:13:38 -0400 Subject: [PATCH 21/65] perf(l4): release SDG caches early, intern port names, lower GC target during L4 --- cmd/codeanalyzer/main.go | 6 +++- cmd/codeanalyzer/main_test.go | 29 +++++++++++++++ docs/design/specs/v2-l3-l4-dataflow.md | 42 ++++++++++++++-------- internal/sdg/memlevers_test.go | 49 ++++++++++++++++++++++++++ internal/sdg/models.go | 12 +++++-- internal/sdg/models_test.go | 44 ++++++++++++++++++++--- internal/sdg/params.go | 20 ++++++++++- internal/sdg/run.go | 25 +++++++++++++ internal/sdg/summary.go | 42 +++++++++++++++++++++- internal/sdg/vertices.go | 29 +++++++++++---- 10 files changed, 269 insertions(+), 29 deletions(-) create mode 100644 internal/sdg/memlevers_test.go diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index 8c755c1..5e3dc55 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -104,7 +104,11 @@ via CLDK(language="go").analysis(project_path=...).`, level = int(options.LevelSDG) if len(targetFiles) > 0 { level = int(options.LevelDataflow) - utils.Warn("--target-files: projecting the graph at level 3 (level 4 is whole-program); L4 rows keep the last full run's facts until a full run") + msg := "--target-files: projecting the graph at level 3 (level 4 is whole-program); L4 rows keep the last full run's facts until a full run" + if cmd.Flags().Changed("library-models") { + msg += "; --library-models is ignored at level 3" + } + utils.Warn("%s", msg) } case options.EmitSchema: path, err := neo4j.EmitSchema(outputDir) diff --git a/cmd/codeanalyzer/main_test.go b/cmd/codeanalyzer/main_test.go index 8e9d324..73702f2 100644 --- a/cmd/codeanalyzer/main_test.go +++ b/cmd/codeanalyzer/main_test.go @@ -689,6 +689,35 @@ func TestCLI_LibraryModelsAtLevel4AndNeo4j(t *testing.T) { } } +// TestCLI_Neo4jTargetFilesWithLibraryModelsWarnsOnce: --emit neo4j -t runs at +// L3, so --library-models is unused there; the D9 warning says so on its one +// stderr line. +func TestCLI_Neo4jTargetFilesWithLibraryModelsWarnsOnce(t *testing.T) { + td := cliTestdataDir() + models := writeModelsFile(t, `{"fmt/Sprintf": []}`) + var runErr error + stderr := captureStderr(t, func() { + _, _, runErr = runCmd( + "--input", filepath.Join(td, "multipackage"), + "--emit", "neo4j", + "--target-files", filepath.Join(td, "multipackage", "server", "server.go"), + "--library-models", models, + "--output", t.TempDir(), + "--cache-dir", t.TempDir(), + ) + }) + if runErr != nil { + t.Fatalf("--emit neo4j -t --library-models should succeed; got %v", runErr) + } + want := "[codeanalyzer-go WARN] " + warnNeo4jTargetFiles + "; --library-models is ignored at level 3\n" + if n := strings.Count(stderr, want); n != 1 { + t.Errorf("want exactly one line %q; stderr:\n%s", want, stderr) + } + if n := strings.Count(stderr, "projecting the graph at level 3"); n != 1 { + t.Errorf("D9 warning appeared %d times, want 1; stderr:\n%s", n, stderr) + } +} + func TestCLI_LibraryModelsBadFileErrors(t *testing.T) { td := cliTestdataDir() for name, path := range map[string]string{ diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index 9ed1ee3..843b3da 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -610,8 +610,8 @@ Context-insensitive. Per callable compute: are computed in full. Under the coarse host wiring (A15) a summary edge only shortcuts host → `actual_in` → … → `actual_out` → host, and sources and sinks address `formal_in`, so a global → global summary adds no reachability for an HRB-unaware walk. Measured on nats-server: - 97.7% of 101M summary edges were global → global (10.8 GB peak RSS against D3's 1.5×). They - return with the HRB two-phase walk (§11). + 97.7% of 101M summary edges were global → global; the cut took `sdg.Run`'s peak RSS from + 11.1 GB to ~4.4 GB (2.34M summary edges left). They return with the HRB two-phase walk (§11). - **Unanalysed callables (amended, A16).** A callable whose L3 result was dropped (a contained panic or a call-count mismatch, `run.go`) has no `ddg`, so taint entering through `param_in` would stop at its `formal_in`. Such callables get formals and a **pass-through summary** (like an @@ -711,17 +711,25 @@ Fill the reserved overlay, no contract renames: No hard node/time caps (neither sibling has them; a cap silently drops edges ⇒ false refutation). Termination comes from k-limiting and finite summary domains. -**Memory (amended, A20; D3) [user: budget].** Today L3 nodes keep their AST (`ir.go` `Stmt`), the +**Memory (amended, A20; D3) [user, 2026-10-26: measured, not gated].** Today L3 nodes keep their AST (`ir.go` `Stmt`), the whole envelope is marshalled into one `[]byte` (`core/analyzer.go`), and most RSS is `go/packages` type info (Kubernetes root: 7.1 GB at `-a 3`). L4 adds SSA bodies for project packages, VTA, summaries and roughly 1.5–2.5× the body nodes (the ports). - **Prerequisites:** #27 streaming JSON output and AST release (before the real-app L4 eval). - L4 needs the ASTs to bind actuals, so each callable's L4 vertices are built **before** its AST references are dropped. The SSA program is shared and freed after L4. -- **Budget (D3):** `-a 4` peak RSS ≤ **1.5× `-a 3`** on the Kubernetes root, measured by an early - SSA+VTA spike task. If the spike blows it, record it in the ledger and keep going. Loading - dependencies from export data (dropping `NeedDeps` syntax) is a separate follow-up, not this - train (§11). +- **Memory is measured, not gated (D3, amended) [user, 2026-10-26].** L4 is whole-program + analysis and needs more memory than L3; the former "≤ 1.5× `-a 3`" budget is dropped. The + real-app eval (§8) reports, per app, the absolute `-a 4` peak RSS and its ratio to `-a 3`. + Loading dependencies from export data (dropping `NeedDeps` syntax) is a separate follow-up, not + this train (§11). +- **Output-neutral levers (Task 15b).** `sdg.Run` releases each unit's flow graph and call-site + bindings once its call-graph SCC is solved and its summary edges are produced, and the cached + port wiring once `fill` has copied it into the result; it interns the formal keys and `of` + strings (every `:` spelling) so identical spellings share one string; and it lowers + the process GC target to 50 while it runs (`debug.SetGCPercent`, restored on return, never + raised above the caller's), so L1–L3 and emission keep the default. None changes a fact: the + result is deeply equal with them off (`TestSDG_OutputUnchangedByMemoryLevers`). ### 4.14 L2 external calls (Phase 0, shipped in PR #16) [user: fix it; details are calls] @@ -801,8 +809,9 @@ amendments are folded into the sections above. Decisions: `go/ssa` def-use differential is built with the L4 SSA join and does not gate (§4.5). - **D2 [call]:** SSA comes from `ssautil.Packages` over the same load, built per package with `ssa.InstantiateGenerics|ssa.BuildSerially` under `recover` (§4.8). -- **D3 [user]:** `-a 4` peak RSS ≤ 1.5× `-a 3` on the Kubernetes root, measured by the spike; if - blown, record it in the ledger and keep going; export-data loading is a separate follow-up (§4.13). +- **D3 [user; amended user, 2026-10-26]:** memory is measured and reported per app (absolute + `-a 4` peak RSS and its ratio to `-a 3`), not gated: L4 is whole-program. (Was: `-a 4` peak + RSS ≤ 1.5× `-a 3` on the Kubernetes root.) Export-data loading is a separate follow-up (§4.13). - **D4 [user]:** `--emit neo4j` runs at max implemented depth = L4; `-a`/`--graph-field-depth` with it stay an error, as in Python and Java v2 (§4.12). - **D5 [call]:** a lazy Bolt push upserts every L4 row on every push; removing stale L4 rows needs @@ -876,8 +885,13 @@ addition now; the cross-language SDK taint revisit is deferred (§11). `--emit neo4j` (`-a 1..3`: error `--library-models applies only at analysis level 4`); a missing, unreadable or malformed file is an error naming the path. An entry naming an `of` the external lacks is a warning and is ignored for that external, which keeps the all-to-all - default (a typo never refutes a flow); an entry for an external no call site targets is ignored - (Debug). + default (a typo never refutes a flow); an entry for an external no call site targets (or a + misspelt key, `strconv.Atoi`) is ignored with one warning per key, in sorted key order: + `--library-models: no call site targets ; entry ignored (keys are /, e.g. strconv/Atoi)`. + A method of a generic library type is keyed by its declared (origin) signature + (`sync/atomic/Pointer.Store`), so one entry covers every instantiation. Under + `--emit neo4j -t` (L3, D9) the file is still read and checked but unused, and the D9 warning + ends `; --library-models is ignored at level 3`. - **Not modelled:** callbacks invoked by library code (`sort.Slice(xs, less)`, `http.HandleFunc(path, h)`): the function-value argument flows in, but the library calling it back is not an edge. Taint *starting* in a handler is unaffected (sources are the handler's own @@ -987,8 +1001,8 @@ needs its code-generation step (bpf2go) before analysis (`scripts/eval_l3.sh --p app: wall time and peak RSS at `-a 3`/`-a 4`, body/edge counts, % call sites with resolved targets (go/types vs VTA), % externals with stand-ins, `:` formal volume (A5), determinism. Gate: the six apps and both Kubernetes modules complete `-a 4` without crash and deterministically -(`-j 1`/`-j N` and two runs on Kubernetes, not just fixtures); the D3 memory budget is reported -against (§4.13); no hard time budget, but timings are reported and a regression budget is set after +(`-j 1`/`-j N` and two runs on Kubernetes, not just fixtures); memory is reported per app as +absolute `-a 4` peak RSS and its ratio to `-a 3`, not gated (D3, §4.13); no hard time budget, but timings are reported and a regression budget is set after the first measurement. Results go in `l3-dataflow-metrics.md` § L4. ## 9. Phasing and release @@ -1041,7 +1055,7 @@ dispatch + type-based may-alias). | Risk | Mitigation | |---|---| -| SSA/VTA cost at Kubernetes scale | SSA only at `-a 4`, project packages only (D2); early SSA+VTA spike sets the D3 budget; #27 streaming; `--strict` / degrade path | +| SSA/VTA cost at Kubernetes scale | SSA only at `-a 4`, project packages only (D2); memory measured and reported per app (D3); #27 streaming; `--strict` / degrade path | | False refutations from a missed construct | over-approximation posture; "ports carry dependence" probe; per-construct fixtures; unresolved-call intra-caller pass-through | | Over-approximation noise (defer chain, pass-through, CDG in summaries) | acceptable by anti-goal; library models file narrows hot spots | | Breaking id change disrupts users | ship alone in Phase 0, before SDK pin; no released SDK consumes Go output | diff --git a/internal/sdg/memlevers_test.go b/internal/sdg/memlevers_test.go new file mode 100644 index 0000000..6b599e4 --- /dev/null +++ b/internal/sdg/memlevers_test.go @@ -0,0 +1,49 @@ +package sdg + +import ( + "reflect" + "runtime/debug" + "testing" +) + +// TestSDG_OutputUnchangedByMemoryLevers: the memory levers (early cache +// release, interned port names, the L4 GC target) change memory, never +// facts: Run's result on dataflow and chi is deeply equal with them off +// (memLevers = false) and on. +func TestSDG_OutputUnchangedByMemoryLevers(t *testing.T) { + for _, name := range []string{"dataflow", "chi"} { + t.Run(name, func(t *testing.T) { + defer func(v bool) { memLevers = v }(memLevers) + memLevers = false + off := Run(loadInput(t, name)) + memLevers = true + on := Run(loadInput(t, name)) + if len(on.Callables) == 0 || len(on.ParamIn) == 0 { + t.Fatalf("empty overlay: %d callables, %d param_in", len(on.Callables), len(on.ParamIn)) + } + if !reflect.DeepEqual(off, on) { + for k, c := range off.Callables { + if !reflect.DeepEqual(c, on.Callables[k]) { + t.Errorf("callable %v differs:\n off %+v\n on %+v", k, c, on.Callables[k]) + break + } + } + t.Fatalf("Run(%s) differs with the memory levers on", name) + } + }) + } +} + +// TestRun_RestoresGCPercent: Run lowers the GC target only while it runs: +// the caller's setting is back on return, and a lower one is never raised. +func TestRun_RestoresGCPercent(t *testing.T) { + in := loadInput(t, "greeter") + for _, pct := range []int{200, 20} { + prev := debug.SetGCPercent(pct) + Run(in) + got := debug.SetGCPercent(prev) + if got != pct { + t.Errorf("GC percent after Run = %d, want the caller's %d", got, pct) + } + } +} diff --git a/internal/sdg/models.go b/internal/sdg/models.go index f103232..75b28f1 100644 --- a/internal/sdg/models.go +++ b/internal/sdg/models.go @@ -127,11 +127,19 @@ func (b *sdgBuilder) modelFlows(t Target, fi *formalInfo) (*flowSet, bool) { return f, true } -// unusedModels logs, at Debug, every Models entry no targeted external has. +// unusedModels warns, once per key and in sorted key order, of every Models +// entry no targeted external has: an uncalled external, or a misspelt key +// (`strconv.Atoi` for `strconv/Atoi`), which would otherwise silently keep +// the default. func (b *sdgBuilder) unusedModels() { + keys := make([]string, 0, len(b.in.Models)) for key := range b.in.Models { if !b.modelled[key] { - utils.Debug("L4: --library-models: no call site targets %s; entry ignored", key) + keys = append(keys, key) } } + slices.Sort(keys) + for _, key := range keys { + warnf("--library-models: no call site targets %s; entry ignored (keys are /, e.g. strconv/Atoi)", key) + } } diff --git a/internal/sdg/models_test.go b/internal/sdg/models_test.go index d16b81b..9872e52 100644 --- a/internal/sdg/models_test.go +++ b/internal/sdg/models_test.go @@ -141,17 +141,53 @@ func TestModels_UnknownOfFallsBackToAllToAll(t *testing.T) { } // TestModels_UncalledEntryIgnored: an entry for an external no site targets -// is no warning and changes nothing. +// (a misspelt key too: strconv.Atoi for strconv/Atoi) changes nothing and +// is one warning per key, in sorted key order. func TestModels_UncalledEntryIgnored(t *testing.T) { - res, warns := fillWith(t, Models{"example.com/nosuch/F": {}}) - if len(warns) != 0 { - t.Errorf("warnings: %v", warns) + res, warns := fillWith(t, Models{"strconv.Atoi": {}, "example.com/nosuch/F": {}, "strconv/Atoi": {{"s", ":0"}, {"s", ":1"}}}) + want := []string{ + "--library-models: no call site targets example.com/nosuch/F; entry ignored (keys are /, e.g. strconv/Atoi)", + "--library-models: no call site targets strconv.Atoi; entry ignored (keys are /, e.g. strconv/Atoi)", + } + if !slices.Equal(warns, want) { + t.Errorf("warnings:\n got %q\n want %q", warns, want) } if s := siteSummaryOf(res, stdlibKey("atoi"), atoiSite); len(s) != 2 { t.Errorf("summary of atoi = %+v, want the default", s) } } +// TestModels_GenericExternalKey: a method of a generic library type is +// keyed by its origin signature (sync/atomic/Pointer.Store, never an +// instance's type arguments), so one entry covers every instantiation: +// StoreA (atomic.Pointer[A]) and StoreB (atomic.Pointer[B]) both lose +// their summary edges under `[]`. +func TestModels_GenericExternalKey(t *testing.T) { + storeA := dataflow.L3Key{Path: filepath.Join("generic", "a.go"), Signature: "example.com/dataflow/generic.StoreA"} + storeB := dataflow.L3Key{Path: filepath.Join("generic", "b.go"), Signature: "example.com/dataflow/generic.StoreB"} + in, d := runPortsInput(t) + b := newSDGBuilder(in, d) + key, ok := b.modelKey(Target{External: "sync/atomic.Pointer.Store"}) + if want := "sync/atomic/Pointer.Store"; !ok || key != want { + t.Fatalf("modelKey(sync/atomic.Pointer.Store) = %q, %v; want %q", key, ok, want) + } + res, _ := fillWith(t, nil) + for _, k := range []dataflow.L3Key{storeA, storeB} { + if s := res.Callables[k].Summary; len(s) != 4 { + t.Errorf("default summary of %s = %+v, want 4 edges (x, val → x, val)", k.Signature, s) + } + } + res, warns := fillWith(t, Models{"sync/atomic/Pointer.Store": {}}) + for _, k := range []dataflow.L3Key{storeA, storeB} { + if s := res.Callables[k].Summary; len(s) != 0 { + t.Errorf("sync/atomic/Pointer.Store modelled with no flow; summary of %s = %+v", k.Signature, s) + } + } + if len(warns) != 0 { + t.Errorf("warnings: %v", warns) + } +} + func writeModels(t *testing.T, body string) string { t.Helper() path := filepath.Join(t.TempDir(), "models.json") diff --git a/internal/sdg/params.go b/internal/sdg/params.go index b2ca3f7..1d602ab 100644 --- a/internal/sdg/params.go +++ b/internal/sdg/params.go @@ -78,6 +78,19 @@ func compareParamEdge(a, b ParamEdge) int { ) } +// releaseUnit drops what fill cached for unit k once its overlay is in the +// Result: the wiring (the Result shares its slice), its summary edges and +// call sites, and the ports' call and kept-node indexes of k (read only by +// k's own vertices, wiring and params). +func (b *sdgBuilder) releaseUnit(k dataflow.L3Key) { + delete(b.wired, k) + delete(b.summ, k) + delete(b.sites, k) + delete(b.graph, k) + delete(b.p.calls, k) + delete(b.p.kept, k) +} + func sortParamEdges(es []ParamEdge) []ParamEdge { slices.SortFunc(es, compareParamEdge) return slices.Compact(es) @@ -104,10 +117,15 @@ func (b *sdgBuilder) fill(res *Result) { res.Callables[k] = CallableSDG{ Vertices: b.p.vertices(k), Ddg: b.portEdges(k), - Summary: b.summaryEdges(k), + Summary: b.summary(k), } pi, po := b.params(k) ins, outs = append(ins, pi...), append(outs, po...) + if memLevers { + // The Result now holds unit k's overlay; nothing below reads + // its caches again. + b.releaseUnit(k) + } } res.ParamIn, res.ParamOut = sortParamEdges(ins), sortParamEdges(outs) diff --git a/internal/sdg/run.go b/internal/sdg/run.go index 3f8d2e2..70bf54a 100644 --- a/internal/sdg/run.go +++ b/internal/sdg/run.go @@ -2,6 +2,7 @@ package sdg import ( "fmt" + "runtime/debug" "time" "golang.org/x/tools/go/callgraph" @@ -14,6 +15,17 @@ import ( // vtaCallGraph is vta.CallGraph (a test seam for the recovered-panic path). var vtaCallGraph = vta.CallGraph +// memLevers enables the output-neutral memory levers of Run (spec §4.13): +// early release of the builder's per-unit caches, interned port names, and +// the lower GC target. A test seam: TestSDG_OutputUnchangedByMemoryLevers +// turns it off to check the result is the same. +var memLevers = true + +// l4GCPercent is the GC target during Run. L4's live heap grows by the +// whole-program overlay, and GOGC=100 headroom doubles the transient +// garbage on top of it; 50 trades some GC time for that headroom. +const l4GCPercent = 50 + // Run builds the L4 overlay of in. It builds SSA over the project packages, // joins it to the L3 units, resolves every call site's dispatch targets // (L2's static callee ∪ VTA's callees), and returns the call-graph edges and @@ -26,7 +38,20 @@ var vtaCallGraph = vta.CallGraph // join alone. The degradations are, in order: partial_ssa / no_ssa (SSA), // l3_skipped and range_func (from the units, whether or not SSA built), // unmapped_ssa (the join), no_vta. +// +// Run lowers the process-wide GC target to l4GCPercent while it runs and +// restores the previous one on return (debug.SetGCPercent is global: a +// caller running other work concurrently shares the lower target). A target +// already at or below it, or GC off (GOGC=off), is kept. L1–L3 and the +// emission run at the caller's setting. func Run(in Input) Result { + if memLevers { + prev := debug.SetGCPercent(l4GCPercent) + if prev < l4GCPercent { + debug.SetGCPercent(prev) + } + defer debug.SetGCPercent(prev) + } res, _ := run(in) return res } diff --git a/internal/sdg/summary.go b/internal/sdg/summary.go index 0c8d306..28bd297 100644 --- a/internal/sdg/summary.go +++ b/internal/sdg/summary.go @@ -19,6 +19,9 @@ type sdgBuilder struct { sites map[dataflow.L3Key][]siteFlow graph map[dataflow.L3Key]*unitGraph wired map[dataflow.L3Key][]PortEdge + // summ is each unit's summary edges, produced as soon as its SCC is + // solved (releaseSCC); fill moves them into the Result. + summ map[dataflow.L3Key][]SummaryEdge // modelled is the Models keys some targeted stand-in has. modelled map[string]bool done bool @@ -34,6 +37,7 @@ func newSDGBuilder(in Input, d *dispatch) *sdgBuilder { sites: map[dataflow.L3Key][]siteFlow{}, graph: map[dataflow.L3Key]*unitGraph{}, wired: map[dataflow.L3Key][]PortEdge{}, + summ: map[dataflow.L3Key][]SummaryEdge{}, modelled: map[string]bool{}, } } @@ -253,6 +257,15 @@ type unitGraph struct { ins, outs []int // node of each formal_in / formal_out, by formal index } +// summary is unit k's summary edges: the ones releaseSCC produced, else +// summaryEdges under the current flows. +func (b *sdgBuilder) summary(k dataflow.L3Key) []SummaryEdge { + if s, ok := b.summ[k]; ok { + return s + } + return b.summaryEdges(k) +} + func (b *sdgBuilder) portEdges(k dataflow.L3Key) []PortEdge { if e, ok := b.wired[k]; ok { return e @@ -520,7 +533,34 @@ func (b *sdgBuilder) computeFlows() { b.done = true nodes, callees := callGraph(b.in, b.d) comps := sccs(nodes, func(t Target) []Target { return callees[t] }) - solveSCCs(comps, callees, b.solve) + if !memLevers { + solveSCCs(comps, callees, b.solve) + return + } + for _, comp := range comps { + solveSCCs([][]Target{comp}, callees, b.solve) + b.releaseSCC(comp) + } +} + +// releaseSCC produces the summary edges of every unit of the solved SCC comp +// and drops its unit graphs and call sites (a memory lever, spec §4.13). A +// unit's call-site targets are all its callees (callGraph), so once its SCC +// is solved every flow its summaries read is final: no SCC solved later +// changes them, and no other unit reads its graph or sites. +func (b *sdgBuilder) releaseSCC(comp []Target) { + for _, t := range comp { + if t.External != "" { + continue + } + k := t.Callable + if _, unit := b.p.units[k]; !unit { + continue + } + b.summ[k] = b.summaryEdges(k) + delete(b.graph, k) + delete(b.sites, k) + } } // solveSCCs solves comps in order (callees first, as sccs returns them). A diff --git a/internal/sdg/vertices.go b/internal/sdg/vertices.go index a8f869a..7721a91 100644 --- a/internal/sdg/vertices.go +++ b/internal/sdg/vertices.go @@ -36,6 +36,7 @@ type ports struct { sets map[Target]*formalSet calls map[dataflow.L3Key]map[string]*ast.CallExpr // unit → call key → call kept map[dataflow.L3Key]map[string]bool // unit → its kept L3 node keys + names map[string]string // interned formal keys and `of`s (intern) } func newPorts(in Input, d *dispatch) *ports { @@ -49,6 +50,7 @@ func newPorts(in Input, d *dispatch) *ports { sets: map[Target]*formalSet{}, calls: map[dataflow.L3Key]map[string]*ast.CallExpr{}, kept: map[dataflow.L3Key]map[string]bool{}, + names: map[string]string{}, } for _, u := range in.Units { p.units[u.Key] = u @@ -57,6 +59,21 @@ func newPorts(in Input, d *dispatch) *ports { return p } +// intern returns the one shared copy of s (a memory lever, spec §4.13): +// every target in a large SCC has the same few hundred `:` formals, +// and every target the same `@formal_in:` keys, so building each anew +// would hold one copy per vertex. +func (p *ports) intern(s string) string { + if !memLevers { + return s + } + if c, ok := p.names[s]; ok { + return c + } + p.names[s] = s + return s +} + // formalSet is the formals of one target, with how a call site binds them: // by position (receiver, declared params, results) or by `of` (captures, // globals). @@ -127,7 +144,7 @@ func (p *ports) formalSet(t Target) *formalSet { } nRes := p.resultCount(t) for i := range nRes { - outs = append(outs, resultOf(i, nRes)) + outs = append(outs, p.intern(resultOf(i, nRes))) outFrom = append(outFrom, -1) } for i, f := range formals { @@ -147,10 +164,10 @@ func (p *ports) formalSet(t Target) *formalSet { outFrom = append(outFrom, -1) } for _, g := range unionSorted([][]string{fp.GlobalReads, fp.GlobalWrites}) { - fs.globIn = append(fs.globIn, ofGlobal+g) + fs.globIn = append(fs.globIn, p.intern(ofGlobal+g)) } for _, g := range fp.GlobalWrites { - fs.globOut = append(fs.globOut, ofGlobal+g) + fs.globOut = append(fs.globOut, p.intern(ofGlobal+g)) outFrom = append(outFrom, -1) } ins = append(ins, fs.globIn...) @@ -158,7 +175,7 @@ func (p *ports) formalSet(t Target) *formalSet { fs.outPos = make([]string, len(fs.posOf)) for i, of := range ins { - key := "@formal_in:" + strconv.Itoa(i) + key := p.intern("@formal_in:" + strconv.Itoa(i)) fs.in = append(fs.in, Vertex{Key: key, Of: of, Kind: KindFormalIn}) if i < len(fs.posOf) { fs.inPos = append(fs.inPos, key) @@ -169,7 +186,7 @@ func (p *ports) formalSet(t Target) *formalSet { for i, of := range outs { key := "@formal_out" if len(outs) > 1 { - key += ":" + strconv.Itoa(i) + key = p.intern(key + ":" + strconv.Itoa(i)) } fs.out = append(fs.out, Vertex{Key: key, Of: of, Kind: KindFormalOut}) switch { @@ -398,7 +415,7 @@ func (p *ports) callSite(unit dataflow.L3Key, callKey string, ts []Target) site nRes = len(p.formalSet(order[0]).results) } for i := range nRes { - o := pending{of: resultOf(i, nRes)} + o := pending{of: p.intern(resultOf(i, nRes))} for _, t := range order { if fs := p.formalSet(t); i < len(fs.results) { o.binds = append(o.binds, portBind{Target: t, Formal: fs.results[i]}) From 3c671f0ce36f6a883ccb38c03a42cf9fecfb4fb8 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 19:24:15 -0400 Subject: [PATCH 22/65] feat(l4): emit the SDG in JSON and Neo4j --- cmd/codeanalyzer/main.go | 6 +- cmd/codeanalyzer/main_test.go | 54 +- internal/core/analyzer.go | 22 +- internal/core/analyzer_test.go | 38 +- internal/core/dataflow_test.go | 2 +- internal/core/testsetup_test.go | 2 +- internal/dataflow/run_test.go | 4 +- internal/neo4j/bolt.go | 64 +- internal/neo4j/bolt_integration_test.go | 77 +++ internal/neo4j/bolt_session_test.go | 171 +++++ internal/neo4j/bolt_test.go | 5 +- internal/neo4j/project.go | 72 +- internal/neo4j/project_test.go | 228 ++++++- internal/neo4j/schema.go | 29 +- internal/neo4j/schema_test.go | 53 +- .../v2emit/determinism_freshbuild_test.go | 2 +- .../v2emit/determinism_test.go | 8 +- internal/syntactic_analysis/v2emit/emit.go | 149 +++- internal/syntactic_analysis/v2emit/emit_l4.go | 320 +++++++++ .../syntactic_analysis/v2emit/emit_l4_test.go | 31 + .../syntactic_analysis/v2emit/emit_test.go | 10 +- .../v2emit/gate_l2_inits_test.go | 2 +- .../syntactic_analysis/v2emit/gate_l2_test.go | 18 +- .../syntactic_analysis/v2emit/gate_l3_test.go | 4 +- .../syntactic_analysis/v2emit/gate_l4_test.go | 634 +++++++++++++++++- .../syntactic_analysis/v2emit/gate_test.go | 10 +- testdata/storedlit/go.mod | 3 + testdata/storedlit/storedlit.go | 31 + 28 files changed, 1899 insertions(+), 150 deletions(-) create mode 100644 internal/syntactic_analysis/v2emit/emit_l4.go create mode 100644 internal/syntactic_analysis/v2emit/emit_l4_test.go create mode 100644 testdata/storedlit/go.mod create mode 100644 testdata/storedlit/storedlit.go diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index 5e3dc55..9e0a941 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -179,7 +179,7 @@ via CLDK(language="go").analysis(project_path=...).`, // JSON path emits) and project it. --neo4j-uri set → live Bolt push; // absent → a graph.cypher snapshot in the output dir. if emitTarget == options.EmitNeo4j { - payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, core.KLimit(opts), analyzer.L3()) + payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, core.KLimit(opts), analyzer.L3(), analyzer.L4()) boltCfg := neo4j.BoltConfig{ URI: opts.Neo4jURI, User: opts.Neo4jUser, @@ -200,9 +200,9 @@ via CLDK(language="go").analysis(project_path=...).`, // writer so tests can capture it via cmd.SetOut. Both paths render // through core so the schema selection stays in one place. if outputDir == "" { - return core.StreamJSON(cmd.OutOrStdout(), app, analyzer.L3(), opts) + return core.StreamJSON(cmd.OutOrStdout(), app, analyzer.L3(), analyzer.L4(), opts) } - return core.WriteOutput(app, analyzer.L3(), opts) + return core.WriteOutput(app, analyzer.L3(), analyzer.L4(), opts) }, } diff --git a/cmd/codeanalyzer/main_test.go b/cmd/codeanalyzer/main_test.go index 73702f2..7da9999 100644 --- a/cmd/codeanalyzer/main_test.go +++ b/cmd/codeanalyzer/main_test.go @@ -587,13 +587,57 @@ func TestCLI_Neo4jWithTargetFilesFallsBackToL3(t *testing.T) { t.Errorf("graph.cypher should carry the L3 %s relationship", rel) } } - // No L4 rows. Trivially true until the L4 projection lands (Task 16), - // where it becomes the real guard. - for _, rel := range []string{"GO_PARAM_IN", "GO_PARAM_OUT", "GO_SUMMARY"} { - if strings.Contains(cy, rel) { - t.Errorf("graph.cypher must carry no %s rows under -t", rel) + // No L4 rows (R-P8): no MERGE of a GO_PARAM_IN / GO_PARAM_OUT / + // GO_SUMMARY relationship, no port GO_DDG row, no vertex body node. The + // same fixture without -t does carry each of them, so the guard is real. + for rel, n := range l4CypherRows(cy) { + if n != 0 { + t.Errorf("graph.cypher under -t carries %d %s rows", n, rel) } } + full := t.TempDir() + if _, _, err := runCmd( + "--input", filepath.Join(td, "multipackage"), + "--emit", "neo4j", + "--output", full, + "--cache-dir", t.TempDir(), + ); err != nil { + t.Fatalf("--emit neo4j should succeed; got %v", err) + } + data, err = os.ReadFile(filepath.Join(full, "graph.cypher")) + if err != nil { + t.Fatal(err) + } + for rel, n := range l4CypherRows(string(data)) { + if n == 0 { + t.Errorf("graph.cypher without -t carries no %s rows", rel) + } + } +} + +// l4CypherRows counts the L4 rows of a graph.cypher script: the rows of +// each UNWIND batch that MERGEs a GO_PARAM_IN / GO_PARAM_OUT / GO_SUMMARY +// relationship, the port GO_DDG rows (prov reaching-defs), and the vertex +// :GoBodyNode rows (kind formal_in, formal_out, actual_in, actual_out). +func l4CypherRows(cy string) map[string]int { + n := map[string]int{"GO_PARAM_IN": 0, "GO_PARAM_OUT": 0, "GO_SUMMARY": 0, "GO_DDG reaching-defs": 0, "vertex": 0} + for _, stmt := range strings.Split(cy, ";\n") { + rows := strings.Count(stmt, "\n {") + for _, rel := range []string{"GO_PARAM_IN", "GO_PARAM_OUT", "GO_SUMMARY"} { + if strings.Contains(stmt, "MERGE (a)-[r:"+rel+"]->(b)") { + n[rel] += rows + } + } + if strings.Contains(stmt, "-[r:GO_DDG {_k: row.k}]->") { + n["GO_DDG reaching-defs"] += strings.Count(stmt, "|reaching-defs'") + } + if strings.Contains(stmt, "MERGE (n:GoBodyNode") { + for _, k := range []string{"formal_in", "formal_out", "actual_in", "actual_out"} { + n["vertex"] += strings.Count(stmt, "kind: '"+k+"'") + } + } + } + return n } func TestCLI_Neo4jWithoutTargetFilesWarnsNothing(t *testing.T) { diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index b8001f9..f1f4cdd 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -41,7 +41,8 @@ type Analyzer struct { // schema.GoApplication: schema must not depend on dataflow, and the // cached v1 model has no L3 facts. l3 map[dataflow.L3Key]dataflow.Result - // l4 is the Phase 6 result (nil below L4). Nothing emits it yet. + // l4 is the Phase 6 result (nil below L4). Pass it to RenderJSON / + // WriteOutput / v2emit.Emit with L3(). l4 *sdg.Result } @@ -277,24 +278,25 @@ func checkFormat(opts options.AnalysisOptions) error { // streaming StreamJSON / WriteOutput paths are byte-identical to; the CLI // streams instead, so the document never exists in memory at once. // -// l3 is the Analyzer's L3() side table (nil below level 3). At level 3 the -// envelope carries k_limit = KLimit(opts) whether or not l3 is set. -func RenderJSON(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, opts options.AnalysisOptions) ([]byte, error) { +// l3 is the Analyzer's L3() side table (nil below level 3) and l4 its L4() +// result (nil below level 4). At level 3 the envelope carries k_limit = +// KLimit(opts) whether or not l3 is set. +func RenderJSON(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, l4 *sdg.Result, opts options.AnalysisOptions) ([]byte, error) { if err := checkFormat(opts); err != nil { return nil, err } - payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, KLimit(opts), l3) + payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, KLimit(opts), l3, l4) return json.Marshal(payload) } // StreamJSON writes the bytes RenderJSON returns to w, emitting and encoding // one module at a time (v2emit.EmitTo) through a buffered writer. -func StreamJSON(w io.Writer, app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, opts options.AnalysisOptions) error { +func StreamJSON(w io.Writer, app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, l4 *sdg.Result, opts options.AnalysisOptions) error { if err := checkFormat(opts); err != nil { return err } bw := bufio.NewWriterSize(w, 1<<20) - if err := v2emit.EmitTo(bw, app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, KLimit(opts), l3); err != nil { + if err := v2emit.EmitTo(bw, app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, KLimit(opts), l3, l4); err != nil { return err } return bw.Flush() @@ -303,12 +305,12 @@ func StreamJSON(w io.Writer, app *schema.GoApplication, l3 map[dataflow.L3Key]da // WriteOutput streams the analysis to outputDir/analysis.json, or to stdout // when outputDir is empty. The file is written beside its destination and // renamed into place, so a failed run never leaves a truncated analysis.json. -func WriteOutput(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, opts options.AnalysisOptions) error { +func WriteOutput(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, l4 *sdg.Result, opts options.AnalysisOptions) error { if err := checkFormat(opts); err != nil { return err } if opts.OutputDir == "" { - return StreamJSON(os.Stdout, app, l3, opts) + return StreamJSON(os.Stdout, app, l3, l4, opts) } if err := utils.EnsureDir(opts.OutputDir); err != nil { return err @@ -319,7 +321,7 @@ func WriteOutput(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Resul return err } tmp := f.Name() - err = StreamJSON(f, app, l3, opts) + err = StreamJSON(f, app, l3, l4, opts) if cerr := f.Close(); err == nil { err = cerr } diff --git a/internal/core/analyzer_test.go b/internal/core/analyzer_test.go index f3940af..a7198dd 100644 --- a/internal/core/analyzer_test.go +++ b/internal/core/analyzer_test.go @@ -3,6 +3,7 @@ package core_test import ( "bytes" "encoding/json" + "fmt" "os" "path/filepath" "testing" @@ -140,7 +141,7 @@ func TestCallGraph_CallSitesBackfilled(t *testing.T) { func TestWriteOutput_ValidJSON(t *testing.T) { outDir := t.TempDir() - if err := core.WriteOutput(sharedGreeterL2, nil, options.AnalysisOptions{OutputDir: outDir, Format: "json"}); err != nil { + if err := core.WriteOutput(sharedGreeterL2, nil, nil, options.AnalysisOptions{OutputDir: outDir, Format: "json"}); err != nil { t.Fatalf("WriteOutput: %v", err) } data, err := os.ReadFile(filepath.Join(outDir, "analysis.json")) @@ -162,7 +163,7 @@ func TestWriteOutput_ValidJSON(t *testing.T) { func TestWriteOutput_EmptyFormatDefaultsToJSON(t *testing.T) { outDir := t.TempDir() - if err := core.WriteOutput(sharedGreeterL1, nil, options.AnalysisOptions{OutputDir: outDir, Format: ""}); err != nil { + if err := core.WriteOutput(sharedGreeterL1, nil, nil, options.AnalysisOptions{OutputDir: outDir, Format: ""}); err != nil { t.Fatalf("WriteOutput with empty format: %v", err) } if _, err := os.Stat(filepath.Join(outDir, "analysis.json")); err != nil { @@ -172,24 +173,32 @@ func TestWriteOutput_EmptyFormatDefaultsToJSON(t *testing.T) { func TestWriteOutput_MsgpackNotImplemented(t *testing.T) { outDir := t.TempDir() - if err := core.WriteOutput(sharedGreeterL1, nil, options.AnalysisOptions{OutputDir: outDir, Format: "msgpack"}); err == nil { + if err := core.WriteOutput(sharedGreeterL1, nil, nil, options.AnalysisOptions{OutputDir: outDir, Format: "msgpack"}); err == nil { t.Fatal("expected error for --format msgpack, got nil") } } func TestWriteOutput_UnknownFormatErrors(t *testing.T) { outDir := t.TempDir() - if err := core.WriteOutput(sharedGreeterL1, nil, options.AnalysisOptions{OutputDir: outDir, Format: "csv"}); err == nil { + if err := core.WriteOutput(sharedGreeterL1, nil, nil, options.AnalysisOptions{OutputDir: outDir, Format: "csv"}); err == nil { t.Fatal("expected error for unknown format, got nil") } } // TestWriteOutput_StreamsRenderJSONBytes: the streamed file and writer paths -// write exactly RenderJSON's bytes at -a 3, and Analyze has already released -// every L3 node's AST reference (#27). +// write exactly RenderJSON's bytes at -a 3 and -a 4 (the L4 param edges and +// degraded marker included), and Analyze has already released every L3 +// node's AST reference (#27). func TestWriteOutput_StreamsRenderJSONBytes(t *testing.T) { + for _, level := range []options.AnalysisLevel{options.LevelDataflow, options.LevelSDG} { + t.Run(fmt.Sprintf("a%d", level), func(t *testing.T) { streamsRenderJSONBytes(t, "dataflow", level) }) + } + t.Run("a4-degraded", func(t *testing.T) { streamsRenderJSONBytes(t, "rangefunc", options.LevelSDG) }) +} + +func streamsRenderJSONBytes(t *testing.T, fixture string, level options.AnalysisLevel) { opts := options.AnalysisOptions{ - InputPath: filepath.Join(testdataDir(), "dataflow"), AppName: "dataflow", Level: options.LevelDataflow, + InputPath: filepath.Join(testdataDir(), fixture), AppName: fixture, Level: level, SkipTests: true, CacheDir: t.TempDir(), OutputDir: t.TempDir(), } a := core.New(opts) @@ -204,18 +213,27 @@ func TestWriteOutput_StreamsRenderJSONBytes(t *testing.T) { } } } - want, err := core.RenderJSON(app, a.L3(), opts) + want, err := core.RenderJSON(app, a.L3(), a.L4(), opts) if err != nil { t.Fatal(err) } + if level == options.LevelSDG { + key := `"param_in":[` + if fixture == "rangefunc" { + key = `"degraded":[` + } + if !bytes.Contains(want, []byte(key)) { + t.Fatalf("-a 4 JSON of %s has no %s", fixture, key) + } + } var buf bytes.Buffer - if err := core.StreamJSON(&buf, app, a.L3(), opts); err != nil { + if err := core.StreamJSON(&buf, app, a.L3(), a.L4(), opts); err != nil { t.Fatal(err) } if !bytes.Equal(buf.Bytes(), want) { t.Error("StreamJSON bytes differ from RenderJSON") } - if err := core.WriteOutput(app, a.L3(), opts); err != nil { + if err := core.WriteOutput(app, a.L3(), a.L4(), opts); err != nil { t.Fatal(err) } got, err := os.ReadFile(filepath.Join(opts.OutputDir, "analysis.json")) diff --git a/internal/core/dataflow_test.go b/internal/core/dataflow_test.go index 362bf18..222d7e9 100644 --- a/internal/core/dataflow_test.go +++ b/internal/core/dataflow_test.go @@ -23,7 +23,7 @@ func TestAnalyze_Level3RunsDataflow(t *testing.T) { if len(a.L3()) == 0 { t.Fatal("-a 3 produced no dataflow results") } - data, err := core.RenderJSON(app, a.L3(), opts) + data, err := core.RenderJSON(app, a.L3(), a.L4(), opts) if err != nil { t.Fatal(err) } diff --git a/internal/core/testsetup_test.go b/internal/core/testsetup_test.go index db04684..f68d89b 100644 --- a/internal/core/testsetup_test.go +++ b/internal/core/testsetup_test.go @@ -76,7 +76,7 @@ func runTestMain(m *testing.M) int { fmt.Fprintf(os.Stderr, "testsetup %s: Analyze: %v\n", f.name, err) return 1 } - if err := core.WriteOutput(app, nil, opts); err != nil { + if err := core.WriteOutput(app, nil, nil, opts); err != nil { fmt.Fprintf(os.Stderr, "testsetup %s: WriteOutput: %v\n", f.name, err) return 1 } diff --git a/internal/dataflow/run_test.go b/internal/dataflow/run_test.go index b68e541..619963f 100644 --- a/internal/dataflow/run_test.go +++ b/internal/dataflow/run_test.go @@ -50,7 +50,7 @@ func TestRun_JobsDeterminism(t *testing.T) { if len(l3) != len(units) { t.Errorf("-j %d: %d results for %d units", jobs, len(l3), len(units)) } - data, err := json.Marshal(v2emit.Emit(app, "chi", dir, 3, "det", 3, l3)) + data, err := json.Marshal(v2emit.Emit(app, "chi", dir, 3, "det", 3, l3, nil)) if err != nil { t.Fatal(err) } @@ -409,7 +409,7 @@ func TestResult_ReleaseDropsASTs(t *testing.T) { app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} out := dataflow.Run(b.Fset(), b.L3Units(st), 3, 4) emit := func() []byte { - data, err := json.Marshal(v2emit.Emit(app, "dataflow", dir, 3, "rel", 3, out)) + data, err := json.Marshal(v2emit.Emit(app, "dataflow", dir, 3, "rel", 3, out, nil)) if err != nil { t.Fatal(err) } diff --git a/internal/neo4j/bolt.go b/internal/neo4j/bolt.go index 6dc6c08..4e48123 100644 --- a/internal/neo4j/bolt.go +++ b/internal/neo4j/bolt.go @@ -20,13 +20,24 @@ import ( // app's id; a mismatch forces a full re-UPSERT (never a delete — that is // --eager's job); // 3. partition nodes by owning module (NodeRow.module); shared nodes (externals, -// neutral repo nodes) have none and are MERGE-only, always upserted; +// neutral repo nodes) have none and are MERGE-only, always upserted; L4 +// vertex nodes (a callable's or a stand-in's formal/actual vertices) are +// their own group; // 4. --eager? scoped purge of everything under the app prefix, then rebuild; // 5. diff each module's content_hash against the DB → the changed set; // 6. per changed module: on --eager, purge its owned edges + stale decls, then // upsert its nodes; WITHOUT --eager the push only adds/updates; -// 7. upsert edges owned by a changed module (or shared); -// 8. on a full --eager run, orphan-prune modules whose source vanished. +// 7. upsert every L4 vertex node, changed module or not (D5, below); +// 8. upsert every edge (an unchanged edge's MERGE is a no-op); +// 9. on a full --eager run, orphan-prune modules whose source vanished. +// +// D5 (spec §4.11): L4 facts are not module-local — editing a callee changes +// its callers' summary edges, actual vertices and even formal_in indices — +// so the content_hash diff gates the L1–L3 rows only. A lazy push upserts +// every L4 row on every push: the vertex nodes (step 7), the stand-in +// bodies among them, and the GO_PARAM_IN / GO_PARAM_OUT / GO_SUMMARY rows +// and the port GO_DDG rows (prov ≠ ssa) with every other edge (step 8). It +// is additive: a stale L4 row stays until an --eager push. // // A push NEVER deletes by default (#171): steps 4/6/8 run under --eager only. // Every destructive statement is scoped on the can:/// id prefix (#173), @@ -95,9 +106,9 @@ func (w *boltWriter) run() error { } } - // Partition nodes by owning module; shared nodes (module == "") are always - // upserted. - byModule, shared := partitionNodes(w.rows.Nodes) + // Partition nodes by owning module; shared nodes (module == "") and L4 + // vertex nodes are always upserted. + byModule, shared, l4 := partitionNodes(w.rows.Nodes) // Diff content_hash per module unless a mismatch/eager forces everything. changed := map[string]bool{} @@ -129,12 +140,15 @@ func (w *boltWriter) run() error { return err } } + // L4 vertex nodes, after the module purges so an --eager purge of a + // changed module cannot delete them (D5). + if err := w.upsertNodes(l4); err != nil { + return err + } - // Upsert edges owned by a changed module (owner = source node's module) or - // shared. Determining ownership precisely needs the source node's module; we - // conservatively upsert every edge whose source id is under a changed module - // prefix, plus all shared/app-level edges. A MERGE-upsert of an unchanged - // edge is a harmless no-op. + // Upsert every edge: the L4 rows must be (D5), and for the L1–L3 rows + // the precise owner (the source node's module) buys nothing — a + // MERGE-upsert of an unchanged edge is a harmless no-op. if err := w.upsertEdges(w.rows.Edges); err != nil { return err } @@ -418,16 +432,34 @@ func propsToAny(p Props) map[string]any { // ─── partitioning & hashing ─────────────────────────────────────────────────── -func partitionNodes(nodes []NodeRow) (byModule map[string][]NodeRow, shared []NodeRow) { +// partitionNodes splits nodes into the module-owned L1–L3 nodes (by owning +// module), the shared nodes (no module), and the L4 vertex nodes, owned by +// a module or by a stand-in, which a push always upserts (D5). +func partitionNodes(nodes []NodeRow) (byModule map[string][]NodeRow, shared, l4 []NodeRow) { byModule = map[string][]NodeRow{} for _, n := range nodes { - if n.Module == "" { + switch { + case isL4Node(n): + l4 = append(l4, n) + case n.Module == "": shared = append(shared, n) - continue + default: + byModule[n.Module] = append(byModule[n.Module], n) } - byModule[n.Module] = append(byModule[n.Module], n) } - return byModule, shared + return byModule, shared, l4 +} + +// isL4Node reports an L4 vertex: a :GoBodyNode of a formal / actual kind. +func isL4Node(n NodeRow) bool { + if n.Labels[0] != LabelBodyNode { + return false + } + switch n.Props["kind"] { + case "formal_in", "formal_out", "actual_in", "actual_out": + return true + } + return false } // moduleHash returns the content_hash the projector put on the :GoModule node in diff --git a/internal/neo4j/bolt_integration_test.go b/internal/neo4j/bolt_integration_test.go index 72ac8b7..debc6d3 100644 --- a/internal/neo4j/bolt_integration_test.go +++ b/internal/neo4j/bolt_integration_test.go @@ -16,6 +16,7 @@ package neo4j import ( "context" + "fmt" "os" "testing" @@ -128,3 +129,79 @@ func countSubgraph(t *testing.T, ctx context.Context, cfg BoltConfig, appID stri } return nodes, edges } + +// TestIntegration_LazyPushRestoresL4Rows (D5): after an eager push of the +// -a 4 dataflow graph, delete its L4 rows (every GO_PARAM_IN / GO_PARAM_OUT +// / GO_SUMMARY relationship and every vertex body node); a lazy push of the +// same rows — every module unchanged, so its L1–L3 rows are skipped — puts +// every one of them back. +func TestIntegration_LazyPushRestoresL4Rows(t *testing.T) { + ctx := context.Background() + cfg := itConfig(t) + a := buildDataflowL4(t) + rows := Project(a) + seed := cfg + seed.Eager, seed.FullRun = true, true + if err := WriteBolt(ctx, rows, a.Application.ID, seed); err != nil { + t.Fatalf("seed: %v", err) + } + want := l4Counts(t, ctx, cfg, a.Application.ID) + for k, n := range want { + if n == 0 { + t.Fatalf("seeded graph has no %s", k) + } + } + if want["GO_PARAM_IN"] != countEdges(rows, RelParamIn) || want["GO_SUMMARY"] != countEdges(rows, RelSummary) { + t.Errorf("seeded L4 counts %v differ from the projection", want) + } + runIT(t, ctx, cfg, "MATCH (n:GoBodyNode) WHERE n.id STARTS WITH $p AND n.kind IN ['formal_in','formal_out','actual_in','actual_out'] DETACH DELETE n", + map[string]any{"p": a.Application.ID + "/"}) + if got := l4Counts(t, ctx, cfg, a.Application.ID); got["vertex"] != 0 || got["GO_PARAM_IN"] != 0 { + t.Fatalf("L4 rows not deleted: %v", got) + } + if err := WriteBolt(ctx, rows, a.Application.ID, cfg); err != nil { + t.Fatalf("lazy push: %v", err) + } + if got := l4Counts(t, ctx, cfg, a.Application.ID); fmt.Sprint(got) != fmt.Sprint(want) { + t.Errorf("lazy push restored %v, want %v", got, want) + } +} + +func runIT(t *testing.T, ctx context.Context, cfg BoltConfig, q string, params map[string]any) { + t.Helper() + drv, err := neo4j.NewDriverWithContext(cfg.URI, neo4j.BasicAuth(cfg.User, cfg.Password, "")) + if err != nil { + t.Fatalf("driver: %v", err) + } + defer drv.Close(ctx) + if _, err := neo4j.ExecuteQuery(ctx, drv, q, params, neo4j.EagerResultTransformer, neo4j.ExecuteQueryWithDatabase(cfg.Database)); err != nil { + t.Fatalf("%s: %v", q, err) + } +} + +// l4Counts counts this app's L4 rows: vertex body nodes and the three L4 +// relationship types, plus the port GO_DDG rows. +func l4Counts(t *testing.T, ctx context.Context, cfg BoltConfig, appID string) map[string]int { + t.Helper() + drv, err := neo4j.NewDriverWithContext(cfg.URI, neo4j.BasicAuth(cfg.User, cfg.Password, "")) + if err != nil { + t.Fatalf("driver: %v", err) + } + defer drv.Close(ctx) + p := map[string]any{"p": appID + "/"} + out := map[string]int{} + for k, q := range map[string]string{ + "vertex": "MATCH (n:GoBodyNode) WHERE n.id STARTS WITH $p AND n.kind IN ['formal_in','formal_out','actual_in','actual_out'] RETURN count(n)", + "GO_PARAM_IN": "MATCH (n:GoBodyNode)-[r:GO_PARAM_IN]->() WHERE n.id STARTS WITH $p RETURN count(r)", + "GO_PARAM_OUT": "MATCH (n:GoBodyNode)-[r:GO_PARAM_OUT]->() WHERE n.id STARTS WITH $p RETURN count(r)", + "GO_SUMMARY": "MATCH (n:GoBodyNode)-[r:GO_SUMMARY]->() WHERE n.id STARTS WITH $p RETURN count(r)", + "GO_DDG port": "MATCH (n:GoBodyNode)-[r:GO_DDG]->() WHERE n.id STARTS WITH $p AND r.prov <> ['ssa'] RETURN count(r)", + } { + res, err := neo4j.ExecuteQuery(ctx, drv, q, p, neo4j.EagerResultTransformer, neo4j.ExecuteQueryWithDatabase(cfg.Database)) + if err != nil { + t.Fatalf("%s: %v", q, err) + } + out[k] = int(res.Records[0].Values[0].(int64)) + } + return out +} diff --git a/internal/neo4j/bolt_session_test.go b/internal/neo4j/bolt_session_test.go index ce13d83..e5d0137 100644 --- a/internal/neo4j/bolt_session_test.go +++ b/internal/neo4j/bolt_session_test.go @@ -72,3 +72,174 @@ func TestEagerPurge_OnlyOnFullRun(t *testing.T) { } } } + +// fakeDB is a DB-free SessionWithContext for the push algorithm: reads +// answer the schema-version gate with SchemaVersion and the content_hash +// diff with hashes[module id] (an unchanged module when it equals the local +// hash); every write statement is recorded with its parameters. +type fakeDB struct { + neo4j.SessionWithContext + hashes map[string]string + writes []fakeStmt +} + +type fakeStmt struct { + q string + params map[string]any +} + +type fakeTx struct { + neo4j.ManagedTransaction + db *fakeDB +} + +func (s *fakeDB) Run(_ context.Context, q string, params map[string]any, _ ...func(*neo4j.TransactionConfig)) (neo4j.ResultWithContext, error) { + s.writes = append(s.writes, fakeStmt{q, params}) + return consumedResult{}, nil +} + +func (s *fakeDB) ExecuteWrite(ctx context.Context, work neo4j.ManagedTransactionWork, _ ...func(*neo4j.TransactionConfig)) (any, error) { + return work(fakeTx{db: s}) +} + +func (s *fakeDB) ExecuteRead(ctx context.Context, work neo4j.ManagedTransactionWork, _ ...func(*neo4j.TransactionConfig)) (any, error) { + return work(fakeTx{db: s}) +} + +func (tx fakeTx) Run(_ context.Context, q string, params map[string]any) (neo4j.ResultWithContext, error) { + switch { + case strings.Contains(q, "a.schema_version"): + return &fakeResult{rec: &neo4j.Record{Keys: []string{"v"}, Values: []any{SchemaVersion}}}, nil + case strings.Contains(q, "m.content_hash"): + h, ok := tx.db.hashes[params["id"].(string)] + if !ok { + return &fakeResult{}, nil + } + return &fakeResult{rec: &neo4j.Record{Keys: []string{"h"}, Values: []any{h}}}, nil + } + tx.db.writes = append(tx.db.writes, fakeStmt{q, params}) + return &fakeResult{}, nil +} + +type fakeResult struct { + neo4j.ResultWithContext + rec *neo4j.Record + done bool +} + +func (r *fakeResult) Next(context.Context) bool { + if r.rec == nil || r.done { + return false + } + r.done = true + return true +} +func (r *fakeResult) Record() *neo4j.Record { return r.rec } +func (r *fakeResult) Err() error { return nil } + +// pushedIDs returns the node keys (row.k of node MERGEs) and the edge +// (type, a, b) triples a push wrote. +func (s *fakeDB) pushed() (nodes map[string]bool, edges map[string]int) { + nodes, edges = map[string]bool{}, map[string]int{} + for _, st := range s.writes { + rows, _ := st.params["rows"].([]map[string]any) + switch { + case strings.Contains(st.q, "MERGE (n:"): + for _, r := range rows { + nodes[r["k"].(string)] = true + } + case strings.Contains(st.q, "MERGE (a)-[r:"): + typ := st.q[strings.Index(st.q, "[r:")+3:] + typ = typ[:strings.IndexAny(typ, " ]")] + edges[typ] += len(rows) + } + } + return nodes, edges +} + +// TestBolt_LazyPushUpsertsL4RowsOfUnchangedModules (D5): on a lazy push +// where every module's content_hash matches the DB, the L1–L3 nodes of the +// (unchanged) modules are skipped, but every L4 row is upserted — each param +// vertex of a callable, each stand-in body node, and every GO_PARAM_IN / +// GO_PARAM_OUT / GO_SUMMARY row and port GO_DDG row (prov ≠ ssa). On an +// eager push nothing is skipped. +func TestBolt_LazyPushUpsertsL4RowsOfUnchangedModules(t *testing.T) { + a := buildDataflowL4(t) + rows := Project(a) + byModule, _, _ := partitionNodes(rows.Nodes) + hashes := map[string]string{} + for m, ns := range byModule { + hashes[m] = moduleHash(ns) + if hashes[m] == "" { + t.Fatalf("module %s has no content_hash", m) + } + } + db := &fakeDB{hashes: hashes} + w := &boltWriter{ctx: context.Background(), session: db, rows: rows, appID: a.Application.ID} + if err := w.run(); err != nil { + t.Fatal(err) + } + nodes, edges := db.pushed() + + var vertices, l3Skipped int + for _, n := range rows.Nodes { + if n.Labels[0] != LabelBodyNode { + continue + } + kind, _ := n.Props["kind"].(string) + switch { + case isVertex(kind): + vertices++ + if !nodes[n.Value] { + t.Errorf("L4 vertex %s of an unchanged module not upserted", n.Value) + } + case n.Module != "": + if nodes[n.Value] { + t.Errorf("L3 body node %s of an unchanged module upserted on a lazy push", n.Value) + } + l3Skipped++ + } + } + if vertices == 0 || l3Skipped == 0 { + t.Fatalf("fixture: %d vertices, %d L3 body nodes", vertices, l3Skipped) + } + standIns := 0 + for _, ext := range a.Application.ExternalSymbols { + for _, n := range ext.Body { + standIns++ + if !nodes[n.ID] { + t.Errorf("stand-in vertex %s not upserted", n.ID) + } + } + } + if standIns == 0 { + t.Fatal("fixture has no stand-in body") + } + portRows := 0 + for _, e := range rows.Edges { + if p, _ := e.Props["prov"].([]string); e.Type == RelDDG && strings.Join(p, ",") != "ssa" { + portRows++ + } + } + for _, rel := range []string{RelParamIn, RelParamOut, RelSummary, RelDDG} { + if want := countEdges(rows, rel); want == 0 || edges[rel] != want { + t.Errorf("%s: pushed %d rows, projected %d", rel, edges[rel], want) + } + } + if portRows == 0 { + t.Error("fixture has no port GO_DDG row") + } + + // Eager: every node of every module is upserted. + db = &fakeDB{hashes: hashes} + w = &boltWriter{ctx: context.Background(), session: db, rows: rows, appID: a.Application.ID, cfg: BoltConfig{Eager: true, FullRun: true}} + if err := w.run(); err != nil { + t.Fatal(err) + } + nodes, _ = db.pushed() + for _, n := range rows.Nodes { + if !nodes[n.Value] { + t.Fatalf("eager push skipped %s", n.Value) + } + } +} diff --git a/internal/neo4j/bolt_test.go b/internal/neo4j/bolt_test.go index bb3723a..cd6972f 100644 --- a/internal/neo4j/bolt_test.go +++ b/internal/neo4j/bolt_test.go @@ -19,13 +19,16 @@ func TestPartitionNodes_SharedHaveNoModule(t *testing.T) { b.Node([]string{SymbolLabel, LabelExternal, MarkerLabel}, "id", "can://app/@external/fmt", nil, "") // shared rows := b.Finish() - byModule, shared := partitionNodes(rows.Nodes) + byModule, shared, l4 := partitionNodes(rows.Nodes) if len(byModule["x.go"]) != 2 { t.Errorf("module x.go should own 2 nodes, got %d", len(byModule["x.go"])) } if len(shared) != 1 { t.Errorf("expected 1 shared (no-module) node, got %d", len(shared)) } + if len(l4) != 0 { + t.Errorf("expected no L4 node, got %d", len(l4)) + } } func TestModuleHash_ReadsModuleNode(t *testing.T) { diff --git a/internal/neo4j/project.go b/internal/neo4j/project.go index 8c74346..9584338 100644 --- a/internal/neo4j/project.go +++ b/internal/neo4j/project.go @@ -18,9 +18,9 @@ import ( // every id the app owns, which is what makes prefix scoping exact. // // Walk order (spine): application → per module (types → callables → fields → -// body call nodes) → call_graph → imports → neutral artifact/package layer → -// reserved L3/L4 overlay (zero rows at the current max level). Every endpoint -// resolves to a declared node or an id-keyed :GoExternal ghost — nothing dangles. +// body nodes and the L3/L4 intra-callable overlay) → call_graph → imports → +// stand-in bodies → param edges. Every endpoint resolves to a declared node or +// an id-keyed :GoExternal ghost — nothing dangles. // Project is the pure (tree) → GraphRows function. appID is analysis.Application.ID // (the can:// root); it is read from the envelope, never recomposed. @@ -37,6 +37,8 @@ func Project(a *v2.Analysis) GraphRows { p.projectModule(mod) } p.projectCallGraph() + p.projectStandIns() + p.projectParamEdges() return p.b.Finish() } @@ -59,11 +61,25 @@ func (p *projector) projectApplication() { "max_level": a.MaxLevel, "analyzer_name": a.Analyzer.Name, "analyzer_version": a.Analyzer.Version, + "degraded_json": degradedJSON(a.Degraded), }, "", // the application root belongs to no single module ) } +// degradedJSON is the envelope's degraded[] (spec §4.15) as a JSON string, +// nil (pruned) when nothing degraded. +func degradedJSON(d []v2.Degraded) Value { + if len(d) == 0 { + return nil + } + data, err := json.Marshal(d) + if err != nil { + return nil + } + return string(data) +} + func (p *projector) appRef() NodeRef { return NodeRef{Label: LabelApplication, KeyProp: "id", Value: p.analysis.Application.ID} } @@ -174,6 +190,9 @@ func (p *projector) projectCallable(parentRef NodeRef, module string, c v2.Calla return cRef } +// projectBodyNode emits one body node of the callable (or, for a stand-in, +// the :GoExternal) at callableRef and its GO_HAS_BODY_NODE. module is the +// owning module, "" for a stand-in's vertices (shared, like the ghost). func (p *projector) projectBodyNode(callableRef NodeRef, module, callableID, local string, bn v2.BodyNode) { // GlobalOrdinal, not callableID+"@"+local: synthetic keys ("@entry") already // carry the "@", and the JSON body node id uses the same rule. @@ -195,8 +214,8 @@ func (p *projector) projectBodyNode(callableRef NodeRef, module, callableID, loc "is_deferred": boolOrNil(bn.IsDeferred), "span_json": bodySpanJSON(bn.Span), "call_node": callNode, - // "var" (JSON `of`) is L4: the v2 BodyNode carries no such field yet, - // and a nil prop is pruned, so L3 emits none. + // var is JSON `of`: set on L4 vertices only (pruned when empty). + "var": emptyToNil(bn.Of), }, module, ) @@ -213,10 +232,11 @@ func (p *projector) projectBodyNode(callableRef NodeRef, module, callableID, loc } } -// projectFlow emits a callable's L3 intra-callable overlay. JSON edge endpoints -// are LOCAL body keys; the graph addresses :GoBodyNode by its global id, so each -// endpoint goes through v2.GlobalOrdinal. Parallel edges survive via _k: -// GO_CFG_NEXT per kind, GO_DDG per (var, prov). +// projectFlow emits a callable's intra-callable overlay: L3's cfg / cdg / ddg +// and L4's port ddg edges (prov reaching-defs / points-to) and summary. JSON +// edge endpoints are LOCAL body keys; the graph addresses :GoBodyNode by its +// global id, so each endpoint goes through v2.GlobalOrdinal. Parallel edges +// survive via _k: GO_CFG_NEXT per kind, GO_DDG per (var, prov). func (p *projector) projectFlow(c v2.Callable) { ref := func(local string) NodeRef { return NodeRef{Label: LabelBodyNode, KeyProp: "id", Value: v2.GlobalOrdinal(c.ID, local)} @@ -234,6 +254,40 @@ func (p *projector) projectFlow(c v2.Callable) { Props{"var": e.Var, "prov": stringsOrNil(prov)}, e.Var+"|"+strings.Join(prov, ",")) } + for _, e := range c.Summary { + p.b.Edge(RelSummary, ref(e.Src), ref(e.Dst), nil, "") + } +} + +// ─── L4: stand-ins and param edges ─────────────────────────────────────────── + +// projectStandIns emits each library stand-in's body (its formal vertices, +// spec §6) as :GoBodyNode rows owned by the external's :GoExternal ghost — +// the same id-keyed node call_graph endpoints use — via GO_HAS_BODY_NODE. +// Stand-ins are app-scoped: their rows belong to no module. +func (p *projector) projectStandIns() { + for id, ext := range p.analysis.Application.ExternalSymbols { + if len(ext.Body) == 0 { + continue + } + ghost := p.callEndpoint(id) + for local, bn := range ext.Body { + p.projectBodyNode(ghost, "", id, local, bn) + } + } +} + +// projectParamEdges emits GO_PARAM_IN / GO_PARAM_OUT {var}, one row per JSON +// param_in / param_out edge. Endpoints are body node ids already (the +// emitter minted them), addressed as :GoBodyNode. +func (p *projector) projectParamEdges() { + ref := func(id string) NodeRef { return NodeRef{Label: LabelBodyNode, KeyProp: "id", Value: id} } + for _, e := range p.analysis.Application.ParamIn { + p.b.Edge(RelParamIn, ref(e.Src), ref(e.Dst), Props{"var": e.Var}, "") + } + for _, e := range p.analysis.Application.ParamOut { + p.b.Edge(RelParamOut, ref(e.Src), ref(e.Dst), Props{"var": e.Var}, "") + } } // ─── imports ────────────────────────────────────────────────────────────────── diff --git a/internal/neo4j/project_test.go b/internal/neo4j/project_test.go index a2d545e..d9d16a6 100644 --- a/internal/neo4j/project_test.go +++ b/internal/neo4j/project_test.go @@ -1,14 +1,17 @@ package neo4j import ( + "encoding/json" "path/filepath" "runtime" "strings" + "sync" "testing" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" @@ -36,7 +39,7 @@ func buildMultipackageL2(t *testing.T) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} - return v2emit.Emit(app, "multipackage", dir, 2, "neo4j-gate", 0, nil) + return v2emit.Emit(app, "multipackage", dir, 2, "neo4j-gate", 0, nil, nil) } func TestProject_NoDanglingEndpoints(t *testing.T) { @@ -216,7 +219,7 @@ func buildDataflowL3(t *testing.T) *v2.Analysis { edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} res := dataflow.Run(b.Fset(), b.L3Units(st), 3, 4) - return v2emit.Emit(app, "dataflow", dir, 3, "neo4j-gate", 3, res) + return v2emit.Emit(app, "dataflow", dir, 3, "neo4j-gate", 3, res, nil) } // walkCallables visits every callable (methods and nested closures included). @@ -498,3 +501,224 @@ func TestProject_CallNodeResolvesWithinOwner(t *testing.T) { t.Errorf("only %d call_node props checked", checked) } } + +// ─── L4 overlay ────────────────────────────────────────────────────────────── + +var ( + l4Once sync.Once + l4Analysis *v2.Analysis +) + +// buildDataflowL4 builds testdata/dataflow through the real -a 4 wiring +// (symbol table, resolver, dataflow, sdg.Run merged into the application) +// and emits the v2 L4 analysis. Built once: the projection never mutates it. +func buildDataflowL4(t *testing.T) *v2.Analysis { + t.Helper() + l4Once.Do(func() { + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "dataflow")) + if err != nil { + t.Fatalf("resolving fixture dir: %v", err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + units := b.L3Units(st) + res := dataflow.RunWithSideTables(b.Fset(), units, 3, 4) + l4 := sdg.Run(sdg.Input{ + Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Externals: app.Externals, CallGraph: app.CallGraph, Units: units, L3: res, K: 3, Jobs: 4, + }) + l4.ApplyTo(app) + l4Analysis = v2emit.Emit(app, "dataflow", dir, 4, "neo4j-gate", 3, res, &l4) + }) + if l4Analysis == nil { + t.Fatal("L4 fixture build failed") + } + return l4Analysis +} + +func isVertex(kind string) bool { + switch kind { + case "formal_in", "formal_out", "actual_in", "actual_out": + return true + } + return false +} + +// TestProject_L4CountsMatchJSON is the cross-projection gate at -a 4: per +// kind, the graph carries exactly the JSON's facts — GO_PARAM_IN / OUT per +// param edge, GO_SUMMARY per summary edge, GO_DDG per ddg edge (port edges +// included), one :GoBodyNode and one GO_HAS_BODY_NODE per body node of a +// callable or a stand-in. +func TestProject_L4CountsMatchJSON(t *testing.T) { + a := buildDataflowL4(t) + rows := Project(a) + var ddg, port, summary, body, vertices int + walkCallables(a, func(c v2.Callable) { + ddg += len(c.Ddg) + for _, e := range c.Ddg { + if strings.Join(e.Prov, ",") != "ssa" { + port++ + } + } + summary += len(c.Summary) + body += len(c.Body) + for _, n := range c.Body { + if isVertex(n.Kind) { + vertices++ + } + } + }) + standIns := 0 + for _, ext := range a.Application.ExternalSymbols { + standIns += len(ext.Body) + } + if port == 0 || summary == 0 || vertices == 0 || standIns == 0 || len(a.Application.ParamIn) == 0 || len(a.Application.ParamOut) == 0 { + t.Fatalf("fixture has no L4 content: port=%d summary=%d vertices=%d stand-ins=%d param_in=%d param_out=%d", + port, summary, vertices, standIns, len(a.Application.ParamIn), len(a.Application.ParamOut)) + } + portRows := 0 + for _, e := range rows.Edges { + if e.Type == RelDDG { + if p, _ := e.Props["prov"].([]string); strings.Join(p, ",") != "ssa" { + portRows++ + if e.Key != e.Props["var"].(string)+"|"+strings.Join(p, ",") { + t.Errorf("port GO_DDG _k = %q", e.Key) + } + } + } + } + nodes, vertexRows := 0, 0 + for _, n := range rows.Nodes { + if n.Labels[0] == LabelBodyNode { + nodes++ + if k, _ := n.Props["kind"].(string); isVertex(k) { + vertexRows++ + } + } + } + for _, c := range []struct { + name string + got, want int + }{ + {RelParamIn, countEdges(rows, RelParamIn), len(a.Application.ParamIn)}, + {RelParamOut, countEdges(rows, RelParamOut), len(a.Application.ParamOut)}, + {RelSummary, countEdges(rows, RelSummary), summary}, + {RelDDG, countEdges(rows, RelDDG), ddg}, + {"GO_DDG (port)", portRows, port}, + {RelHasBodyNode, countEdges(rows, RelHasBodyNode), body + standIns}, + {":GoBodyNode", nodes, body + standIns}, + {":GoBodyNode (vertex)", vertexRows, vertices + standIns}, + } { + if c.got != c.want { + t.Errorf("%s rows = %d, JSON = %d", c.name, c.got, c.want) + } + } +} + +// TestProject_L4VertexProps: a vertex projects as a :GoBodyNode with var = +// its `of` and call_node = its JSON parent (the local key); a stand-in's +// vertices hang off its :GoExternal ghost by GO_HAS_BODY_NODE; param edges +// carry var and join two :GoBodyNode ids. +func TestProject_L4VertexProps(t *testing.T) { + a := buildDataflowL4(t) + rows := Project(a) + byID := map[string]NodeRow{} + for _, n := range rows.Nodes { + byID[n.Labels[0]+"\x00"+n.Value] = n + } + walkCallables(a, func(c v2.Callable) { + for _, n := range c.Body { + if !isVertex(n.Kind) { + continue + } + row, ok := byID[LabelBodyNode+"\x00"+n.ID] + if !ok { + t.Errorf("vertex %s not projected", n.ID) + continue + } + if row.Props["var"] != n.Of { + t.Errorf("%s var = %v, want %q", n.ID, row.Props["var"], n.Of) + } + if cn, has := row.Props["call_node"]; (n.Parent != "") != has || (has && cn != n.Parent) { + t.Errorf("%s call_node = %v, want parent %q", n.ID, cn, n.Parent) + } + if _, has := row.Props["span_json"]; has { + t.Errorf("%s carries span_json", n.ID) + } + } + }) + owner := map[string]NodeRef{} + for _, e := range rows.Edges { + if e.Type == RelHasBodyNode { + owner[e.To.Value] = e.From + } + } + checked := 0 + for id, ext := range a.Application.ExternalSymbols { + if _, ok := byID[SymbolLabel+"\x00"+id]; len(ext.Body) > 0 && !ok { + t.Errorf("stand-in %s has no :GoExternal node", id) + } + for _, n := range ext.Body { + if from := owner[n.ID]; from.Value != id || from.Label != SymbolLabel { + t.Errorf("stand-in vertex %s: GO_HAS_BODY_NODE from %+v, want %s", n.ID, from, id) + } + if row := byID[LabelBodyNode+"\x00"+n.ID]; row.Props["var"] != n.Of || row.Module != "" { + t.Errorf("stand-in vertex %s row = %+v", n.ID, row) + } + checked++ + } + } + if checked == 0 { + t.Fatal("no stand-in vertex checked") + } + for _, e := range rows.Edges { + if e.Type != RelParamIn && e.Type != RelParamOut { + continue + } + if e.From.Label != LabelBodyNode || e.To.Label != LabelBodyNode || e.Props["var"] == nil { + t.Errorf("%s row %+v", e.Type, e) + } + } +} + +// TestProject_DegradedJSONOnApplication: the envelope's degraded[] rides +// :GoApplication as the JSON string prop degraded_json, absent when the +// run degraded nothing. +func TestProject_DegradedJSONOnApplication(t *testing.T) { + a := overlayFixture() + a.MaxLevel = 4 + a.Degraded = []v2.Degraded{ + {Code: "no_vta", Detail: "panic: x"}, + {Code: "range_func", Packages: []string{"example.com/p"}, Callables: []string{"can://go/x/a.go/f()"}}, + } + want, err := json.Marshal(a.Degraded) + if err != nil { + t.Fatal(err) + } + app := func(rows GraphRows) NodeRow { + for _, n := range rows.Nodes { + if n.Labels[0] == LabelApplication { + return n + } + } + t.Fatal("no :GoApplication row") + return NodeRow{} + } + if got := app(Project(a)).Props["degraded_json"]; got != string(want) { + t.Errorf("degraded_json = %v, want %s", got, want) + } + a.Degraded = nil + if got, has := app(Project(a)).Props["degraded_json"]; has { + t.Errorf("clean run carries degraded_json %v", got) + } + if _, has := app(Project(buildDataflowL4(t))).Props["degraded_json"]; has { + t.Errorf("dataflow (clean) carries degraded_json") + } +} diff --git a/internal/neo4j/schema.go b/internal/neo4j/schema.go index f168408..928e1b4 100644 --- a/internal/neo4j/schema.go +++ b/internal/neo4j/schema.go @@ -125,8 +125,9 @@ const ( RelCDG = "GO_CDG" RelDDG = "GO_DDG" - // Reserved L4 overlay — declared NOW, emits ZERO rows until L4 lands - // (CLAUDE.md). The projector fills these with no schema-contract change. + // L4 overlay (LIVE at -a 4): interprocedural edges between :GoBodyNode + // vertex ids (GO_PARAM_IN/OUT carry var) and call-site summary edges. + // Declared since L1 and filled with no schema-contract change. RelParamIn = "GO_PARAM_IN" RelParamOut = "GO_PARAM_OUT" RelSummary = "GO_SUMMARY" @@ -175,8 +176,8 @@ type RelSpec struct { // present in the graph: it is emitted via the RowBuilder's edge-only-when- // resolved path and kept at finish() only if the target was emitted. Deferred bool - // Reserved marks the L4 overlay edges: declared now, emits zero rows at - // the current max level (L3). + // Reserved marks an edge declared ahead of the level that fills it: it + // emits zero rows. None is Reserved today (the L4 overlay is live). Reserved bool } @@ -207,11 +208,13 @@ var Nodes = []NodeSpec{ }, { Label: LabelBodyNode, MergeLabel: LabelBodyNode, Key: "id", CanNode: true, - // kind "call" at L1, statement|branch|loop|switch|return|entry|exit at L3; - // is_goroutine/is_deferred pruned when false. call_node = the enclosing - // statement's LOCAL key (JSON `parent` verbatim, resolved against the - // owning callable; Python parity); var = JSON `of` (L4 — absent at L3). - // span_json absent on the synthetic entry/exit nodes. + // kind "call" at L1, statement|branch|loop|switch|return|entry|exit at L3, + // formal_in|formal_out|actual_in|actual_out at L4; is_goroutine / + // is_deferred pruned when false. call_node = the enclosing statement's + // (an actual vertex's: its call's) LOCAL key — JSON `parent` verbatim, + // resolved against the owning callable or stand-in; Python parity. + // var = JSON `of` (L4 vertices only). span_json absent on the + // synthetic entry/exit nodes and on vertices. Props: []string{"kind", "is_goroutine", "is_deferred", "span_json", "call_node", "var"}, }, { @@ -268,8 +271,8 @@ var Rels = []RelSpec{ {Type: RelCDG, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}}, {Type: RelDDG, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Props: []string{"var", "prov"}, Discriminant: "var|prov"}, - // Reserved L4 overlay (zero rows until L4 lands). - {Type: RelParamIn, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Props: []string{"var"}, Reserved: true}, - {Type: RelParamOut, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Props: []string{"var"}, Reserved: true}, - {Type: RelSummary, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Reserved: true}, + // L4 overlay (live at -a 4). + {Type: RelParamIn, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Props: []string{"var"}}, + {Type: RelParamOut, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Props: []string{"var"}}, + {Type: RelSummary, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}}, } diff --git a/internal/neo4j/schema_test.go b/internal/neo4j/schema_test.go index 746308c..4069038 100644 --- a/internal/neo4j/schema_test.go +++ b/internal/neo4j/schema_test.go @@ -138,6 +138,46 @@ func TestSchemaDocument_RoundTrips(t *testing.T) { func TestConformance_ProjectorEmitsOnlyDeclared(t *testing.T) { t.Run("L2", func(t *testing.T) { checkConformance(t, Project(buildMultipackageL2(t))) }) t.Run("L3", func(t *testing.T) { checkConformance(t, Project(buildDataflowL3(t))) }) + t.Run("L4", func(t *testing.T) { checkConformance(t, Project(buildDataflowL4(t))) }) +} + +// unproducedRels are the declared relationships whose facts the Go analyzer +// does not produce yet, so no fixture can emit a row of them (declared +// contract, empty projection — CLAUDE.md § Repository-artifact layer): +// the neutral repository layer and its two GO_ resolution claims, and the +// inheritance pair (base_types[] are type strings, not type ids, so the +// deferred GO_EMBEDS never resolves; interfaces[] is not computed). +var unproducedRels = map[string]bool{ + RelHasArtifact: true, RelDeclaresDependency: true, RelLocks: true, RelDefinesConfig: true, + RelProvides: true, RelUnresolvedImport: true, + RelEmbeds: true, RelSatisfies: true, +} + +// TestConformance_EveryDeclaredRelEmitted: the -a 4 projection of the +// dataflow fixture emits at least one row of every declared relationship +// the analyzer produces facts for — the L4 overlay (GO_PARAM_IN, +// GO_PARAM_OUT, GO_SUMMARY) included — and none of a Reserved one. +func TestConformance_EveryDeclaredRelEmitted(t *testing.T) { + rows := Project(buildDataflowL4(t)) + for _, r := range Rels { + n := countEdges(rows, r.Type) + switch { + case r.Reserved && n > 0: + t.Errorf("%s is Reserved but has %d rows", r.Type, n) + case !r.Reserved && !unproducedRels[r.Type] && n == 0: + t.Errorf("%s is declared but the fixture emits no row of it", r.Type) + } + } +} + +// TestContract_L4OverlayNotReserved: the L4 overlay is live — no declared +// relationship is Reserved any more. +func TestContract_L4OverlayNotReserved(t *testing.T) { + for _, r := range Rels { + if r.Reserved { + t.Errorf("%s is still Reserved", r.Type) + } + } } func checkConformance(t *testing.T, rows GraphRows) { @@ -218,7 +258,7 @@ func checkConformance(t *testing.T, rows GraphRows) { func TestContract_L3OverlayNotReserved(t *testing.T) { want := map[string]bool{ RelCFGNext: false, RelCDG: false, RelDDG: false, - RelParamIn: true, RelParamOut: true, RelSummary: true, + RelParamIn: false, RelParamOut: false, RelSummary: false, } for _, r := range Rels { if res, ok := want[r.Type]; ok { @@ -254,8 +294,8 @@ func TestContract_L3OverlayNotReserved(t *testing.T) { } // TestContract_L4RelsCarryVar: the L4 param relationships declare the `var` -// property they will carry, and stay Reserved until the L4 projection lands; -// :GoApplication declares degraded_json for the degradation marker. +// property they carry; :GoApplication declares degraded_json for the +// degradation marker. func TestContract_L4RelsCarryVar(t *testing.T) { seen := map[string]bool{} for _, r := range Rels { @@ -265,13 +305,6 @@ func TestContract_L4RelsCarryVar(t *testing.T) { if len(r.Props) != 1 || r.Props[0] != "var" { t.Errorf("%s Props = %v, want [var]", r.Type, r.Props) } - if !r.Reserved { - t.Errorf("%s must stay Reserved until Task 16", r.Type) - } - case RelSummary: - if !r.Reserved { - t.Errorf("%s must stay Reserved until Task 16", r.Type) - } } } if len(seen) != 2 { diff --git a/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go b/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go index f178a3e..f89fb6a 100644 --- a/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go +++ b/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go @@ -64,7 +64,7 @@ func buildChiL2Fresh(t *testing.T, dir string) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} - return Emit(app, "chi", dir, 2, "gate-test", 0, nil) + return Emit(app, "chi", dir, 2, "gate-test", 0, nil, nil) } // TestDeterminism_FreshBuildsByteIdentical locks that independent from-disk diff --git a/internal/syntactic_analysis/v2emit/determinism_test.go b/internal/syntactic_analysis/v2emit/determinism_test.go index 012ce49..e87e9cf 100644 --- a/internal/syntactic_analysis/v2emit/determinism_test.go +++ b/internal/syntactic_analysis/v2emit/determinism_test.go @@ -46,7 +46,7 @@ func TestDeterminism_CallGraphStableAcrossEmits(t *testing.T) { // edges slice without a stable sort, repeated marshals still match (same // slice), so to actually exercise order-independence we marshal a shuffled // copy of the edge slice too. - first := mustMarshalCallGraph(t, Emit(app, "multipackage", dir, 2, "det-test", 0, nil)) + first := mustMarshalCallGraph(t, Emit(app, "multipackage", dir, 2, "det-test", 0, nil, nil)) // Reverse the input edge order and re-emit: a correct emitter sorts, so the // serialized call_graph must be unchanged. @@ -55,7 +55,7 @@ func TestDeterminism_CallGraphStableAcrossEmits(t *testing.T) { rev[len(edges)-1-i] = e } appRev := &schema.GoApplication{SymbolTable: st, CallGraph: rev} - second := mustMarshalCallGraph(t, Emit(appRev, "multipackage", dir, 2, "det-test", 0, nil)) + second := mustMarshalCallGraph(t, Emit(appRev, "multipackage", dir, 2, "det-test", 0, nil, nil)) if first != second { t.Errorf("call_graph not order-independent: emitCallGraph must sort edges to a total order\nfirst=%s\nsecond=%s", first, second) @@ -100,7 +100,7 @@ func TestEncode_ByteIdenticalToMarshal(t *testing.T) { apps := map[int]*schema.GoApplication{1: {SymbolTable: st, CallGraph: []schema.GoCallEdge{}}} check := func(level int, app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result) { t.Run(fmt.Sprintf("%s/a%d", name, level), func(t *testing.T) { - tree := Emit(app, name, dir, level, "enc-test", 3, l3) + tree := Emit(app, name, dir, level, "enc-test", 3, l3, nil) want, err := json.Marshal(tree) if err != nil { t.Fatal(err) @@ -113,7 +113,7 @@ func TestEncode_ByteIdenticalToMarshal(t *testing.T) { t.Errorf("Encode differs from json.Marshal at byte %d (len %d vs %d)", firstDiff(enc.Bytes(), want), enc.Len(), len(want)) } var lazy bytes.Buffer - if err := EmitTo(&lazy, app, name, dir, level, "enc-test", 3, l3); err != nil { + if err := EmitTo(&lazy, app, name, dir, level, "enc-test", 3, l3, nil); err != nil { t.Fatal(err) } if !bytes.Equal(lazy.Bytes(), want) { diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 322b20e..8ff0b1e 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -6,14 +6,17 @@ import ( "fmt" "io" "iter" + "maps" "os" "path/filepath" + "slices" "sort" "strings" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" ) // Emit transforms the v1 GoApplication into the v2 Analysis payload. appName is @@ -24,27 +27,35 @@ import ( // // kLimit is the L3 access-path bound (--graph-field-depth), written as the // envelope's k_limit at maxLevel >= 3 only. l3 is dataflow.Run's side table, -// nil below L3; see emit_l3.go for how it merges into each callable. +// nil below L3; see emit_l3.go for how it merges into each callable. l4 is +// sdg.Run's result, nil below L4 (and ignored there); see emit_l4.go for how +// it merges into the tree and mints every L4 id. // // This is a pure re-serialization: every fact comes from the v1 model and the -// L3 side table, except span byte offsets, which are derived from source via -// lineIndex. -func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, analyzerVersion string, kLimit int, l3 map[dataflow.L3Key]dataflow.Result) *v2.Analysis { - e := newEmitter(app, appName, projectDir, maxLevel, l3) +// L3/L4 side tables, except span byte offsets, which are derived from source +// via lineIndex. +func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, analyzerVersion string, kLimit int, l3 map[dataflow.L3Key]dataflow.Result, l4 *sdg.Result) *v2.Analysis { + e := newEmitter(app, appName, projectDir, maxLevel, l3, l4) out := e.envelope(app, maxLevel, analyzerVersion, kLimit) out.Application.SymbolTable = make(map[string]v2.Module, len(app.SymbolTable)) for _, relPath := range sortedFileKeys(app.SymbolTable) { out.Application.SymbolTable[relPath] = e.module(relPath, app.SymbolTable[relPath]) } + late := e.sdg.late() + out.Application.ParamIn = collectEdges(late.paramIn) + out.Application.ParamOut = collectEdges(late.paramOut) + out.Degraded = late.degraded return out } // EmitTo writes the analysis.json bytes of Emit's payload to w — the same bytes // json.Marshal(Emit(...)) produces — without building the whole tree: each // module is emitted, marshalled and dropped in symbol_table key order, so peak -// memory is the envelope plus the largest module, not the document. -func EmitTo(w io.Writer, app *schema.GoApplication, appName, projectDir string, maxLevel int, analyzerVersion string, kLimit int, l3 map[dataflow.L3Key]dataflow.Result) error { - e := newEmitter(app, appName, projectDir, maxLevel, l3) +// memory is the envelope plus the largest module, not the document. The L4 +// param edges follow the modules (they need every callable's id) and are +// minted and written one at a time. +func EmitTo(w io.Writer, app *schema.GoApplication, appName, projectDir string, maxLevel int, analyzerVersion string, kLimit int, l3 map[dataflow.L3Key]dataflow.Result, l4 *sdg.Result) error { + e := newEmitter(app, appName, projectDir, maxLevel, l3, l4) out := e.envelope(app, maxLevel, analyzerVersion, kLimit) return encode(w, out, func(yield func(string, v2.Module) bool) { for _, relPath := range sortedFileKeys(app.SymbolTable) { @@ -52,7 +63,19 @@ func EmitTo(w io.Writer, app *schema.GoApplication, appName, projectDir string, return } } - }) + }, e.sdg.late) +} + +// collectEdges is seq as a slice, nil when empty (omitempty). +func collectEdges(seq iter.Seq[v2.ParamEdge]) []v2.ParamEdge { + if seq == nil { + return nil + } + var out []v2.ParamEdge + for e := range seq { + out = append(out, e) + } + return out } // Encode writes a to w as the exact bytes json.Marshal(a) produces (no @@ -81,6 +104,15 @@ func Encode(w io.Writer, a *v2.Analysis) error { return } } + }, func() lateFields { + lf := lateFields{degraded: a.Degraded} + if len(a.Application.ParamIn) > 0 { + lf.paramIn = slices.Values(a.Application.ParamIn) + } + if len(a.Application.ParamOut) > 0 { + lf.paramOut = slices.Values(a.Application.ParamOut) + } + return lf }) } @@ -90,13 +122,18 @@ func Encode(w io.Writer, a *v2.Analysis) error { // follows 'e', so its first match is the field itself. const symbolTableHole = `"symbol_table":{}` -// encode writes env (its SymbolTable ignored) with symbol_table filled from -// modules, which must yield keys in sort.Strings order — the order -// json.Marshal writes a map[string] in. Keys are marshalled as JSON strings, -// which is how encoding/json writes a string map key (same escaping). -func encode(w io.Writer, env *v2.Analysis, modules iter.Seq2[string, v2.Module]) error { +// encode writes env (its SymbolTable, ParamIn, ParamOut and Degraded +// ignored) with symbol_table filled from modules, which must yield keys in +// sort.Strings order — the order json.Marshal writes a map[string] in — and +// then the fields late returns, called once the modules are written: +// param_in and param_out close the application (its last two fields) and +// degraded closes the envelope (its last field), as json.Marshal orders +// them. Keys are marshalled as JSON strings, which is how encoding/json +// writes a string map key (same escaping). +func encode(w io.Writer, env *v2.Analysis, modules iter.Seq2[string, v2.Module], late func() lateFields) error { shell := *env shell.Application.SymbolTable = map[string]v2.Module{} + shell.Application.ParamIn, shell.Application.ParamOut, shell.Degraded = nil, nil, nil data, err := json.Marshal(&shell) if err != nil { return err @@ -105,6 +142,11 @@ func encode(w io.Writer, env *v2.Analysis, modules iter.Seq2[string, v2.Module]) if i < 0 { return fmt.Errorf("v2emit: no %s in the envelope", symbolTableHole) } + // With param_in, param_out and degraded omitted, the shell ends with the + // application's closing '}' and then the envelope's. + if !bytes.HasSuffix(data, []byte("}}")) { + return fmt.Errorf("v2emit: the envelope does not end with the application") + } split := i + len(symbolTableHole) - 1 // at the closing '}' if _, err := w.Write(data[:split]); err != nil { return err @@ -116,7 +158,32 @@ func encode(w io.Writer, env *v2.Analysis, modules iter.Seq2[string, v2.Module]) } first = false } - _, err = w.Write(data[split:]) + if _, err := w.Write(data[split : len(data)-2]); err != nil { + return err + } + lf := late() + if err := writeParamList(w, "param_in", lf.paramIn); err != nil { + return err + } + if err := writeParamList(w, "param_out", lf.paramOut); err != nil { + return err + } + if _, err := io.WriteString(w, "}"); err != nil { + return err + } + if len(lf.degraded) > 0 { + deg, err := json.Marshal(lf.degraded) + if err != nil { + return err + } + if _, err := io.WriteString(w, `,"degraded":`); err != nil { + return err + } + if _, err := w.Write(deg); err != nil { + return err + } + } + _, err = io.WriteString(w, "}") return err } @@ -158,13 +225,17 @@ type emitter struct { idx sigIndex srcs *sourceCache flow l3Overlay + sdg *l4Overlay // nil below L4 } -func newEmitter(app *schema.GoApplication, appName, projectDir string, maxLevel int, l3 map[dataflow.L3Key]dataflow.Result) *emitter { +func newEmitter(app *schema.GoApplication, appName, projectDir string, maxLevel int, l3 map[dataflow.L3Key]dataflow.Result, l4 *sdg.Result) *emitter { appID := v2.AppID(appName) idx := buildSigIndex(appID, app.SymbolTable) idx.externals = externalIDs(appID, app.Externals) - return &emitter{appID: appID, idx: idx, srcs: newSourceCache(projectDir), flow: newL3Overlay(maxLevel, l3)} + return &emitter{ + appID: appID, idx: idx, srcs: newSourceCache(projectDir), + flow: newL3Overlay(maxLevel, l3), sdg: newL4Overlay(maxLevel, l4, idx), + } } // envelope is the payload without its symbol_table (nil): the manifest, the @@ -181,7 +252,7 @@ func (e *emitter) envelope(app *schema.GoApplication, maxLevel int, analyzerVers CallGraph: emitCallGraph(app.CallGraph, e.idx), }, } - out.Application.ExternalSymbols = emitExternalSymbols(e.appID, app.Externals) + out.Application.ExternalSymbols = emitExternalSymbols(e.appID, app.Externals, e.sdg.standIns()) if e.flow.on { out.KLimit = kLimit } @@ -189,7 +260,7 @@ func (e *emitter) envelope(app *schema.GoApplication, maxLevel int, analyzerVers } func (e *emitter) module(relPath string, file schema.GoFile) v2.Module { - return emitModule(e.appID, relPath, file, e.idx, e.srcs, e.flow) + return emitModule(e.appID, relPath, file, e.idx, e.srcs, e.flow, e.sdg) } // emitModule builds a v2 module from a v1 GoFile, reading the file's source so @@ -199,7 +270,7 @@ func (e *emitter) module(relPath string, file schema.GoFile) v2.Module { // modRel is this module's own file path (a symbol_table key). It is threaded to // callables so a method declared in another file can be detected and given a // span against its own source plus a source_file pointer. -func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *sourceCache, flow l3Overlay) v2.Module { +func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *sourceCache, flow l3Overlay, l4 *l4Overlay) v2.Module { li := srcs.index(modRel) source := li.source modID := v2.ModuleID(appID, modRel) @@ -219,10 +290,10 @@ func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *so } for name, t := range file.Types { - mod.Types[name] = emitType(modID, modRel, t, idx, srcs, flow) + mod.Types[name] = emitType(modID, modRel, t, idx, srcs, flow, l4) } for sig, fn := range file.Functions { - mod.Functions[sig] = emitCallable(modID, modRel, modRel, li, fn, idx, srcs, flow) + mod.Functions[sig] = emitCallable(modID, modRel, modRel, li, fn, idx, srcs, flow, l4) } return mod } @@ -231,7 +302,7 @@ func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *so // kind and splitting embedding (base_types) from computed satisfaction. Methods // resolve into the type's callables{}. modRel is the module the type lives in; // its methods may be declared in other files (handled in emitCallable). -func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceCache, flow l3Overlay) v2.Type { +func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceCache, flow l3Overlay, l4 *l4Overlay) v2.Type { li := srcs.index(modRel) typeID := v2.TypeID(modID, t.Signature) out := v2.Type{ @@ -253,7 +324,7 @@ func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceC } } for sig, m := range t.Methods { - out.Callables[sig] = emitCallable(typeID, modRel, modRel, li, m, idx, srcs, flow) + out.Callables[sig] = emitCallable(typeID, modRel, modRel, li, m, idx, srcs, flow, l4) } return out } @@ -280,7 +351,10 @@ func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceC // signature), completes its body and adds cfg/cdg/ddg (emit_l3.go). Its spans // use the same declaring-file lineIndex, so a cross-file callable's statement // spans index its declaring file's source, like its own span. -func emitCallable(parentID, modRel, declRel string, declLI *lineIndex, c schema.GoCallable, idx sigIndex, srcs *sourceCache, flow l3Overlay) v2.Callable { +// +// At L4 the same key finds the callable's SDG overlay (emit_l4.go), applied +// after the L3 one, and records the callable's id for the param edges. +func emitCallable(parentID, modRel, declRel string, declLI *lineIndex, c schema.GoCallable, idx sigIndex, srcs *sourceCache, flow l3Overlay, l4 *l4Overlay) v2.Callable { callID := v2.CallableID(parentID, c.Signature) // Resolve the file this callable's positions are relative to. @@ -315,10 +389,12 @@ func emitCallable(parentID, modRel, declRel string, declLI *lineIndex, c schema. // Closures nest in the same containing module and are declared in the // same file as their enclosing callable. for sig, inner := range c.InnerCallables { - out.Callables[sig] = emitCallable(callID, modRel, declRel, li, inner, idx, srcs, flow) + out.Callables[sig] = emitCallable(callID, modRel, declRel, li, inner, idx, srcs, flow, l4) } } - flow.apply(&out, li, dataflow.L3Key{Path: declRel, Signature: c.Signature}) + key := dataflow.L3Key{Path: declRel, Signature: c.Signature} + flow.apply(&out, li, key) + l4.apply(&out, key) return out } @@ -387,13 +463,20 @@ func externalIDs(appID string, externals map[string]schema.GoExternalCallee) map } // emitExternalSymbols builds application.external_symbols, keyed by @external -// id. Nil (omitted from JSON) when no library callee was recorded. -func emitExternalSymbols(appID string, externals map[string]schema.GoExternalCallee) map[string]v2.ExternalSymbol { +// id. Nil (omitted from JSON) when no library callee was recorded. At L4 +// standIns (by full callee signature) gives each stand-in its body: its +// formal vertices, id v2.GlobalOrdinal(external id, key). +func emitExternalSymbols(appID string, externals map[string]schema.GoExternalCallee, standIns map[string][]sdg.Vertex) map[string]v2.ExternalSymbol { + for _, sig := range slices.Sorted(maps.Keys(standIns)) { + if _, ok := externals[sig]; !ok { + warnf("L4 emit: stand-in %s is not an external symbol; its body is dropped", sig) + } + } if len(externals) == 0 { return nil } out := make(map[string]v2.ExternalSymbol, len(externals)) - for _, ext := range externals { + for fullSig, ext := range externals { id := v2.ExternalID(appID, ext.ImportPath, ext.Sig) params := make([]v2.ExternalParameter, 0, len(ext.Params)) for _, p := range ext.Params { @@ -406,6 +489,12 @@ func emitExternalSymbols(appID string, externals map[string]schema.GoExternalCal if ext.Receiver != nil { sym.Receiver = &v2.ExternalParameter{Name: ext.Receiver.Name, Type: ext.Receiver.Type} } + if vs := standIns[fullSig]; len(vs) > 0 { + sym.Body = make(map[string]v2.BodyNode, len(vs)) + for _, v := range vs { + sym.Body[v.Key] = vertexNode(id, v) + } + } out[id] = sym } return out diff --git a/internal/syntactic_analysis/v2emit/emit_l4.go b/internal/syntactic_analysis/v2emit/emit_l4.go new file mode 100644 index 0000000..9908d57 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/emit_l4.go @@ -0,0 +1,320 @@ +package v2emit + +import ( + "cmp" + "encoding/json" + "fmt" + "io" + "iter" + "slices" + "strings" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// warnf reports an L4 fact the emitter cannot place (a test seam). +var warnf = utils.Warn + +// l4Overlay merges sdg.Run's result into the emitted tree (spec §4.6, §4.7, +// §4.15). It is nil below -a 4, and then the tree is exactly the L3 one. +// +// The sdg IR holds no ids; this overlay mints every one of them: +// - a callable's vertices land in its body under their local keys, with +// id v2.GlobalOrdinal(callable id, key), kind, of and parent (actuals +// only); no span and, not being call nodes, no callee key; +// - its port edges join its ddg with prov [Prov]; +// - its summary edges are its summary; +// - a stand-in's formals land in its external_symbols entry's body, with +// id v2.GlobalOrdinal(external id, key) — @formal_in:; +// - param_in / param_out endpoints and the degraded callables resolve +// through ids, the sdg.Target → id map filled in the same walk that +// applies the L3 overlay (externals are known up front, from their +// import path and signature). +// +// A Ref or degraded callable that names no emitted node is an upstream bug: +// it is dropped with a warning, never emitted dangling. +type l4Overlay struct { + res *sdg.Result + ids map[sdg.Target]string + // unresolved records the targets already warned about. + unresolved map[sdg.Target]bool +} + +func newL4Overlay(maxLevel int, res *sdg.Result, idx sigIndex) *l4Overlay { + if maxLevel < 4 || res == nil { + return nil + } + o := &l4Overlay{ + res: res, + ids: make(map[sdg.Target]string, len(res.Callables)+len(idx.externals)), + unresolved: map[sdg.Target]bool{}, + } + for sig, id := range idx.externals { + o.ids[sdg.Target{External: sig}] = id + } + return o +} + +// apply records c's id under its target key (the callable's declaring file +// and signature, the key the L3 overlay uses) and adds its L4 overlay: the +// vertices to its body, the port edges to its ddg (re-sorted by (src, dst, +// var, prov)), and its summary. +func (o *l4Overlay) apply(c *v2.Callable, key dataflow.L3Key) { + if o == nil { + return + } + t := sdg.Target{Callable: key} + if prev, dup := o.ids[t]; dup && prev != c.ID { + warnf("L4 emit: %s and %s share the target key %s %s", prev, c.ID, key.Path, key.Signature) + } + o.ids[t] = c.ID + r, ok := o.res.Callables[key] + if !ok { + return + } + for _, v := range r.Vertices { + if _, clash := c.Body[v.Key]; clash { + warnf("L4 emit: vertex %s of %s collides with a body node; dropped", v.Key, c.ID) + continue + } + c.Body[v.Key] = vertexNode(c.ID, v) + } + if len(r.Ddg) > 0 { + ddg := slices.Grow(c.Ddg, len(r.Ddg)) + for _, e := range r.Ddg { + ddg = append(ddg, v2.DdgEdge{Src: e.Src, Dst: e.Dst, Var: e.Var, Prov: []string{e.Prov}}) + } + slices.SortStableFunc(ddg, compareDdg) + c.Ddg = ddg + } + if len(r.Summary) > 0 { + c.Summary = make([]v2.SummaryEdge, len(r.Summary)) + for i, e := range r.Summary { + c.Summary[i] = v2.SummaryEdge{Src: e.Src, Dst: e.Dst} + } + slices.SortFunc(c.Summary, func(a, b v2.SummaryEdge) int { + return cmp.Or(strings.Compare(a.Src, b.Src), strings.Compare(a.Dst, b.Dst)) + }) + } +} + +// vertexNode is the body node of vertex v of the callable or stand-in owner. +func vertexNode(owner string, v sdg.Vertex) v2.BodyNode { + return v2.BodyNode{Kind: string(v.Kind), ID: v2.GlobalOrdinal(owner, v.Key), Parent: v.Parent, Of: v.Of} +} + +// compareDdg orders a callable's ddg by (src, dst, var, prov). L3's own +// list is sorted by (src, dst, var) with prov ["ssa"] throughout, so the +// L3 edges keep their relative order. +func compareDdg(a, b v2.DdgEdge) int { + return cmp.Or( + strings.Compare(a.Src, b.Src), strings.Compare(a.Dst, b.Dst), strings.Compare(a.Var, b.Var), + slices.Compare(a.Prov, b.Prov), + ) +} + +// standIns returns the stand-in formals by full callee signature (nil +// below -a 4). +func (o *l4Overlay) standIns() map[string][]sdg.Vertex { + if o == nil { + return nil + } + return o.res.Externals +} + +// late returns the fields written after the symbol table — param_in, +// param_out and degraded — which need every callable's id, so they are +// computed once the module walk has filled ids. +func (o *l4Overlay) late() lateFields { + if o == nil { + return lateFields{} + } + return lateFields{ + paramIn: o.paramEdges("param_in", o.res.ParamIn), + paramOut: o.paramEdges("param_out", o.res.ParamOut), + degraded: o.degraded(), + } +} + +// mintedEdge is a param edge with its owners' ids resolved: its minted +// endpoints are GlobalOrdinal(src, e.Src.Key) and GlobalOrdinal(dst, +// e.Dst.Key), built only when the edge is written. +type mintedEdge struct { + src, dst string + e *sdg.ParamEdge +} + +// paramEdges returns list's edges with minted endpoints, sorted by (src, +// dst, var) over the minted ids, duplicates dropped. The sort works on the +// owner ids and local keys (compareOrdinal compares the joined ids without +// building them), so the whole list is never minted at once: on the +// streaming path each edge's ids exist only while it is written. +func (o *l4Overlay) paramEdges(list string, edges []sdg.ParamEdge) iter.Seq[v2.ParamEdge] { + ms := make([]mintedEdge, 0, len(edges)) + for i := range edges { + e := &edges[i] + src, okS := o.resolve(list, e.Src.Owner) + dst, okD := o.resolve(list, e.Dst.Owner) + if okS && okD { + ms = append(ms, mintedEdge{src: src, dst: dst, e: e}) + } + } + compare := func(a, b mintedEdge) int { + return cmp.Or( + compareOrdinal(a.src, a.e.Src.Key, b.src, b.e.Src.Key), + compareOrdinal(a.dst, a.e.Dst.Key, b.dst, b.e.Dst.Key), + strings.Compare(a.e.Var, b.e.Var), + ) + } + slices.SortFunc(ms, compare) + return func(yield func(v2.ParamEdge) bool) { + for i, m := range ms { + if i > 0 && compare(ms[i-1], m) == 0 { + continue + } + if !yield(v2.ParamEdge{ + Src: v2.GlobalOrdinal(m.src, m.e.Src.Key), + Dst: v2.GlobalOrdinal(m.dst, m.e.Dst.Key), + Var: m.e.Var, + }) { + return + } + } + } +} + +// resolve is t's id, warning once per target that names no emitted node. +func (o *l4Overlay) resolve(what string, t sdg.Target) (string, bool) { + if id, ok := o.ids[t]; ok { + return id, true + } + if !o.unresolved[t] { + o.unresolved[t] = true + warnf("L4 emit: %s names %s, which is not in the tree; its edges are dropped", what, describeTarget(t)) + } + return "", false +} + +func describeTarget(t sdg.Target) string { + if t.External != "" { + return "external " + t.External + } + return fmt.Sprintf("callable %s %s", t.Callable.Path, t.Callable.Signature) +} + +// degraded mints the degradation markers: each callable as its id (sorted, +// unique), entries sorted by (code, detail); nil when nothing degraded. +func (o *l4Overlay) degraded() []v2.Degraded { + if len(o.res.Degraded) == 0 { + return nil + } + out := make([]v2.Degraded, 0, len(o.res.Degraded)) + for _, d := range o.res.Degraded { + v := v2.Degraded{Code: d.Code, Detail: d.Detail, Packages: nonEmpty(d.Packages)} + for _, k := range d.Callables { + if id, ok := o.resolve("degraded "+d.Code, sdg.Target{Callable: k}); ok { + v.Callables = append(v.Callables, id) + } + } + slices.Sort(v.Callables) + v.Callables = slices.Compact(v.Callables) + out = append(out, v) + } + slices.SortStableFunc(out, func(a, b v2.Degraded) int { + return cmp.Or(strings.Compare(a.Code, b.Code), strings.Compare(a.Detail, b.Detail)) + }) + return out +} + +// compareOrdinal compares GlobalOrdinal(o1, k1) with GlobalOrdinal(o2, k2) +// byte-wise without building either. +func compareOrdinal(o1, k1, o2, k2 string) int { + if o1 == o2 { + return compareJoined([3]string{"", ordinalSep(k1), k1}, [3]string{"", ordinalSep(k2), k2}) + } + if n := min(len(o1), len(o2)); o1[:n] != o2[:n] { + return strings.Compare(o1, o2) + } + // One owner id is a prefix of the other (a callable and its closure). + return compareJoined([3]string{o1, ordinalSep(k1), k1}, [3]string{o2, ordinalSep(k2), k2}) +} + +// ordinalSep is what v2.GlobalOrdinal puts between a callable id and k. +func ordinalSep(k string) string { + if strings.HasPrefix(k, "@") { + return "" + } + return "@" +} + +// compareJoined compares the concatenations of a's and b's segments. +func compareJoined(a, b [3]string) int { + i, j, ai, bj := 0, 0, 0, 0 + for { + for i < len(a) && ai == len(a[i]) { + i, ai = i+1, 0 + } + for j < len(b) && bj == len(b[j]) { + j, bj = j+1, 0 + } + switch { + case i == len(a) && j == len(b): + return 0 + case i == len(a): + return -1 + case j == len(b): + return 1 + } + if ca, cb := a[i][ai], b[j][bj]; ca != cb { + if ca < cb { + return -1 + } + return 1 + } + ai, bj = ai+1, bj+1 + } +} + +// lateFields are the payload fields that follow the symbol table and need +// the whole tree's ids: the application's param_in / param_out (its last +// two fields) and the envelope's degraded (its last field). +type lateFields struct { + paramIn, paramOut iter.Seq[v2.ParamEdge] + degraded []v2.Degraded +} + +// writeParamList writes `,"":[e,e,…]` with each edge as json.Marshal +// writes it, or nothing when seq is empty (omitempty). +func writeParamList(w io.Writer, key string, seq iter.Seq[v2.ParamEdge]) error { + if seq == nil { + return nil + } + var err error + first := true + for e := range seq { + data, merr := json.Marshal(e) + if merr != nil { + err = merr + break + } + lead := "," + if first { + lead = `,"` + key + `":[` + first = false + } + if _, err = io.WriteString(w, lead); err != nil { + break + } + if _, err = w.Write(data); err != nil { + break + } + } + if err != nil || first { + return err + } + _, err = io.WriteString(w, "]") + return err +} diff --git a/internal/syntactic_analysis/v2emit/emit_l4_test.go b/internal/syntactic_analysis/v2emit/emit_l4_test.go new file mode 100644 index 0000000..d166479 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/emit_l4_test.go @@ -0,0 +1,31 @@ +package v2emit + +import ( + "math/rand/v2" + "strings" + "testing" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// TestCompareOrdinal_MatchesMintedIds: compareOrdinal orders (owner, key) +// pairs exactly as their minted ids v2.GlobalOrdinal(owner, key) compare, +// including an owner that is a prefix of the other (a callable and its +// closure) and the "@"-tag keys, whose ids carry no extra "@". +func TestCompareOrdinal_MatchesMintedIds(t *testing.T) { + owners := []string{ + "can://a/go/f.go/p.F", "can://a/go/f.go/p.F/p.F.closure_1", "can://a/go/f.go/p.F2", + "can://a/go/f.go/p.F@", "can://a/@external/fmt/Sprintf", "can://a/@external/fmt", + "can://a/go/f.go/T/p.T.M", "", + } + keys := []string{"@formal_in:0", "@formal_in:10", "@formal_out", "@entry", "12:3/actual_in:0", "12:3/2/actual_out:1", "9:1", "", "@", "/x"} + r := rand.New(rand.NewPCG(1, 2)) + pick := func(s []string) string { return s[r.IntN(len(s))] } + for range 20000 { + o1, k1, o2, k2 := pick(owners), pick(keys), pick(owners), pick(keys) + want := strings.Compare(v2.GlobalOrdinal(o1, k1), v2.GlobalOrdinal(o2, k2)) + if got := compareOrdinal(o1, k1, o2, k2); got != want { + t.Fatalf("compareOrdinal(%q, %q, %q, %q) = %d, want %d", o1, k1, o2, k2, got, want) + } + } +} diff --git a/internal/syntactic_analysis/v2emit/emit_test.go b/internal/syntactic_analysis/v2emit/emit_test.go index c5ae286..87fe4e2 100644 --- a/internal/syntactic_analysis/v2emit/emit_test.go +++ b/internal/syntactic_analysis/v2emit/emit_test.go @@ -37,7 +37,7 @@ func buildGreeter(t *testing.T) (*schema.GoApplication, string) { func TestEmit_Envelope(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1, "test", 0, nil) + out := Emit(app, "greeter", dir, 1, "test", 0, nil, nil) if out.SchemaVersion != "2.0.0" { t.Errorf("schema_version = %q, want 2.0.0", out.SchemaVersion) @@ -61,7 +61,7 @@ func TestEmit_Envelope(t *testing.T) { func TestEmit_ModuleSourceAndIDs(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1, "test", 0, nil) + out := Emit(app, "greeter", dir, 1, "test", 0, nil, nil) mod, ok := out.Application.SymbolTable["main.go"] if !ok { @@ -84,7 +84,7 @@ func TestEmit_ModuleSourceAndIDs(t *testing.T) { func TestEmit_CallableBodyAndSpanSlice(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1, "test", 0, nil) + out := Emit(app, "greeter", dir, 1, "test", 0, nil, nil) mod := out.Application.SymbolTable["main.go"] var main v2.Callable @@ -123,7 +123,7 @@ func TestEmit_CallableBodyAndSpanSlice(t *testing.T) { func TestEmit_CalleeSerializesAsNull(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1, "test", 0, nil) + out := Emit(app, "greeter", dir, 1, "test", 0, nil, nil) data, err := json.Marshal(out) if err != nil { t.Fatalf("marshal failed: %v", err) @@ -144,7 +144,7 @@ func TestEmit_CalleeSerializesAsNull(t *testing.T) { func TestEmit_TypeKindAndMethods(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1, "test", 0, nil) + out := Emit(app, "greeter", dir, 1, "test", 0, nil, nil) // Find the Greeter type wherever it lives; assert struct kind + methods. var found bool diff --git a/internal/syntactic_analysis/v2emit/gate_l2_inits_test.go b/internal/syntactic_analysis/v2emit/gate_l2_inits_test.go index 529329c..287cd81 100644 --- a/internal/syntactic_analysis/v2emit/gate_l2_inits_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l2_inits_test.go @@ -37,7 +37,7 @@ func buildDataflowL2Fresh(t *testing.T) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} - return Emit(app, "dataflow", dir, 2, "gate-test", 0, nil) + return Emit(app, "dataflow", dir, 2, "gate-test", 0, nil, nil) } // TestGateL2_EveryInitKeepsItsEdges: each file's init has its own edge to the diff --git a/internal/syntactic_analysis/v2emit/gate_l2_test.go b/internal/syntactic_analysis/v2emit/gate_l2_test.go index 8fadd90..347d3db 100644 --- a/internal/syntactic_analysis/v2emit/gate_l2_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l2_test.go @@ -42,7 +42,7 @@ func buildMultipackageL2(t *testing.T) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) // mutates st in place: backfills callee_signature app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} - return Emit(app, "multipackage", dir, 2, "gate-test", 0, nil) + return Emit(app, "multipackage", dir, 2, "gate-test", 0, nil, nil) } // allCallableIDs collects every callable/type id in the emitted tree. @@ -189,12 +189,12 @@ func TestGateL2_SupersetOfL1(t *testing.T) { t.Fatalf("symbol table build failed: %v", err) } // L1 snapshot BEFORE the resolver mutates call sites. - l1 := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: nil}, "multipackage", dir, 1, "gate-test", 0, nil) + l1 := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: nil}, "multipackage", dir, 1, "gate-test", 0, nil, nil) // Now run the resolver (mutates st) and emit L2. cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) - l2 := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges}, "multipackage", dir, 2, "gate-test", 0, nil) + l2 := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges}, "multipackage", dir, 2, "gate-test", 0, nil, nil) if len(l1.Application.SymbolTable) != len(l2.Application.SymbolTable) { t.Fatalf("module count changed L1->L2: %d vs %d", @@ -410,7 +410,7 @@ func emitTemp(t *testing.T, src string) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} - return Emit(app, "p", dir, 2, "gate-test", 0, nil) + return Emit(app, "p", dir, 2, "gate-test", 0, nil, nil) } // TestGateL2_ExternalParamsHaveNoSpan locks that external parameters serialize @@ -442,7 +442,7 @@ func TestGateL2_GenericLibraryMethodUsesDeclaredSignature(t *testing.T) { } cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) - out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "dataflow", dir, 2, "gate-test", 0, nil) + out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "dataflow", dir, 2, "gate-test", 0, nil, nil) ext := v2.ExternalID(out.Application.ID, "sync/atomic", "Pointer.Store") sym, ok := out.Application.ExternalSymbols[ext] if !ok { @@ -466,7 +466,7 @@ func TestGateL2_IncrementalModeKeepsProjectCalleesInternal(t *testing.T) { } cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) - out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "multipackage", dir, 2, "gate-test", 0, nil) + out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "multipackage", dir, 2, "gate-test", 0, nil, nil) const projectModule = "example.com/multipackage" for id, sym := range out.Application.ExternalSymbols { @@ -492,7 +492,7 @@ func TestGateL2_VendoredCalleeIsExternal(t *testing.T) { } cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) - out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "dataflow", dir, 2, "gate-test", 0, nil) + out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "dataflow", dir, 2, "gate-test", 0, nil, nil) ext := v2.ExternalID(out.Application.ID, "example.com/vlib", "F") sym, ok := out.Application.ExternalSymbols[ext] @@ -564,7 +564,7 @@ func TestGateL2_ExternalUnnamedParamsArePositional(t *testing.T) { } cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) - out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "dataflow", dir, 2, "gate-test", 0, nil) + out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "dataflow", dir, 2, "gate-test", 0, nil, nil) for sig, want := range map[string][]string{"G": {"_0", "_1"}, "H": {"_0", "n"}} { ext := v2.ExternalID(out.Application.ID, "example.com/vlib", sig) @@ -593,7 +593,7 @@ func emitDirL2(t *testing.T, dir, app string) *v2.Analysis { } cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) - return Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, app, dir, 2, "gate-test", 0, nil) + return Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, app, dir, 2, "gate-test", 0, nil, nil) } // assertNoDanglingEdges fails when an edge endpoint is neither a callable id diff --git a/internal/syntactic_analysis/v2emit/gate_l3_test.go b/internal/syntactic_analysis/v2emit/gate_l3_test.go index e0d4b23..eaf31f6 100644 --- a/internal/syntactic_analysis/v2emit/gate_l3_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l3_test.go @@ -69,8 +69,8 @@ func buildL3(t *testing.T, name string) *l3Fixture { res := dataflow.Run(b.Fset(), b.L3Units(st), 3, 4) fx := &l3Fixture{ st: st, - l2: Emit(app, name, dir, 2, "gate-test", 3, nil), - l3: Emit(app, name, dir, 3, "gate-test", 3, res), + l2: Emit(app, name, dir, 2, "gate-test", 3, nil, nil), + l3: Emit(app, name, dir, 3, "gate-test", 3, res, nil), res: res, } l3Cache[name] = fx diff --git a/internal/syntactic_analysis/v2emit/gate_l4_test.go b/internal/syntactic_analysis/v2emit/gate_l4_test.go index 93f591c..191effd 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_test.go @@ -4,15 +4,18 @@ package v2emit // // buildL4 runs the -a 4 pipeline: the -a 3 one, then sdg.Run, whose VTA // call_graph edges and VTA-only externals are merged into the application -// before the L4 payload is emitted (as core does). Later tasks fill in more L4 -// content, and the monotonicity gate below keeps holding as they do -// (-a 3 ⊆ -a 4). +// before the L4 payload is emitted (as core does), with the SDG overlay +// (vertices, port ddg, summaries, param edges, stand-in bodies, degraded). +// The monotonicity gate below holds over it (-a 3 ⊆ -a 4). import ( + "bytes" "encoding/json" + "fmt" "path/filepath" "runtime" "slices" + "strings" "sync" "testing" @@ -28,6 +31,8 @@ type l4Fixture struct { st map[string]schema.GoFile l3, l4 *v2.Analysis res map[dataflow.L3Key]dataflow.Result + sdg *sdg.Result + dir string } var ( @@ -59,24 +64,26 @@ func buildL4(t *testing.T, name string) *l4Fixture { app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} units := b.L3Units(st) res := dataflow.RunWithSideTables(b.Fset(), units, 3, 4) - fx := &l4Fixture{st: st, l3: Emit(app, name, dir, 3, "gate-test", 3, res), res: res} + fx := &l4Fixture{st: st, l3: Emit(app, name, dir, 3, "gate-test", 3, res, nil), res: res, dir: dir} l4 := sdg.Run(sdg.Input{ Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, Externals: app.Externals, CallGraph: app.CallGraph, Units: units, L3: res, K: 3, Jobs: 4, }) l4.ApplyTo(app) - fx.l4 = Emit(app, name, dir, 4, "gate-test", 3, res) + fx.sdg = &l4 + fx.l4 = Emit(app, name, dir, 4, "gate-test", 3, res, &l4) l4Cache[name] = fx return fx } // supersetL4 records where a (-a 3) is not contained in b (-a 4): every object -// key and value, and every list entry, by position — except call_graph, a -// sorted edge list that VTA adds edges into, where every -a 3 edge must occur -// unchanged somewhere in the -a 4 list. +// key and value, and every list entry, by position — except the sorted edge +// lists L4 adds entries into (call_graph gets VTA edges, a callable's ddg +// gets port edges), where every -a 3 edge must occur unchanged somewhere in +// the -a 4 list. func supersetL4(path string, a, b any, errs *[]string) { - if path == "$.application.call_graph" { + if path == "$.application.call_graph" || strings.HasSuffix(path, ".ddg") { av, aok := a.([]any) bv, bok := b.([]any) if !aok || !bok { @@ -129,7 +136,7 @@ func supersetL4(path string, a, b any, errs *[]string) { // is present and equal in the -a 4 JSON. Only max_level, the level marker // itself, may differ. func TestGateL4_SupersetOfL3(t *testing.T) { - for _, name := range []string{"dataflow", "chi", "generics"} { + for _, name := range l4Fixtures { fx := buildL4(t, name) a, b := toAny(t, fx.l3), toAny(t, fx.l4) if b["max_level"] != float64(4) { @@ -225,3 +232,610 @@ func TestGateL4_CallGraphHasVTAProv(t *testing.T) { assertNoDanglingEdges(t, buildL4(t, name).l4) } } + +// ─── the L4 overlay in the tree (Task 16) ──────────────────────────────────── + +// l4Fixtures are the fixtures the L4 tree gates run over. +var l4Fixtures = []string{"dataflow", "chi", "generics", "storedlit"} + +// isVertexKind reports an L4 port vertex kind. +func isVertexKind(k string) bool { + switch sdg.VertexKind(k) { + case sdg.KindFormalIn, sdg.KindFormalOut, sdg.KindActualIn, sdg.KindActualOut: + return true + } + return false +} + +// l4Callables is every callable of the tree (methods and closures included). +func l4Callables(a *v2.Analysis) []v2.Callable { + var out []v2.Callable + var walk func(c v2.Callable) + walk = func(c v2.Callable) { + out = append(out, c) + for _, in := range c.Callables { + walk(in) + } + } + for _, m := range a.Application.SymbolTable { + for _, ty := range m.Types { + for _, c := range ty.Callables { + walk(c) + } + } + for _, f := range m.Functions { + walk(f) + } + } + return out +} + +// bodyIndex maps every body node id of the tree — callables' and stand-ins' +// — to the node and the id of the callable or external that owns it. +type ownedNode struct { + owner string + key string + node v2.BodyNode +} + +func bodyIndex(a *v2.Analysis) map[string]ownedNode { + idx := map[string]ownedNode{} + for _, c := range l4Callables(a) { + for k, n := range c.Body { + idx[n.ID] = ownedNode{c.ID, k, n} + } + } + for id, ext := range a.Application.ExternalSymbols { + for k, n := range ext.Body { + idx[n.ID] = ownedNode{id, k, n} + } + } + return idx +} + +// TestGateL4_NoDanglingEndpoints: every L4 edge endpoint is a node of the +// tree — ddg and summary endpoints are body keys of their callable, param +// edge endpoints are body node ids (a callable's or a stand-in's) — every +// degraded callable is a callable id, and nothing the SDG computed is lost on +// the way: the tree carries every vertex, port edge, summary edge, param +// edge and stand-in formal of the sdg.Result. +func TestGateL4_NoDanglingEndpoints(t *testing.T) { + for _, name := range l4Fixtures { + fx := buildL4(t, name) + a := fx.l4 + idx := bodyIndex(a) + ids := allCallableIDs(a) + var vertices, port, summary int + for _, c := range l4Callables(a) { + for k, n := range c.Body { + if isVertexKind(n.Kind) { + vertices++ + } + if n.Parent != "" { + if _, ok := c.Body[n.Parent]; !ok { + t.Errorf("%s: %s: body %s parent %q is not a body key", name, c.ID, k, n.Parent) + } + } + } + for _, e := range c.Ddg { + for _, k := range []string{e.Src, e.Dst} { + if _, ok := c.Body[k]; !ok { + t.Errorf("%s: %s: ddg endpoint %q is not a body key", name, c.ID, k) + } + } + if !slices.Equal(e.Prov, []string{"ssa"}) { + port++ + } + } + for _, e := range c.Summary { + for _, k := range []string{e.Src, e.Dst} { + if n, ok := c.Body[k]; !ok || !isVertexKind(n.Kind) { + t.Errorf("%s: %s: summary endpoint %q is not a vertex", name, c.ID, k) + } + } + } + summary += len(c.Summary) + } + for list, edges := range map[string][]v2.ParamEdge{"param_in": a.Application.ParamIn, "param_out": a.Application.ParamOut} { + for _, e := range edges { + for _, id := range []string{e.Src, e.Dst} { + if _, ok := idx[id]; !ok { + t.Errorf("%s: %s endpoint %s is not a body node", name, list, id) + } + } + } + } + for _, d := range a.Degraded { + for _, id := range d.Callables { + if !ids[id] { + t.Errorf("%s: degraded %s names %s, not a callable id", name, d.Code, id) + } + } + } + + var wantV, wantP, wantS, wantX int + for _, c := range fx.sdg.Callables { + wantV, wantP, wantS = wantV+len(c.Vertices), wantP+len(c.Ddg), wantS+len(c.Summary) + } + gotX := 0 + for _, ext := range a.Application.ExternalSymbols { + gotX += len(ext.Body) + } + for _, vs := range fx.sdg.Externals { + wantX += len(vs) + } + for _, c := range []struct { + what string + got, want int + }{ + {"vertices", vertices, wantV}, + {"port ddg edges", port, wantP}, + {"summary edges", summary, wantS}, + {"param_in", len(a.Application.ParamIn), len(fx.sdg.ParamIn)}, + {"param_out", len(a.Application.ParamOut), len(fx.sdg.ParamOut)}, + {"stand-in formals", gotX, wantX}, + } { + if c.got != c.want { + t.Errorf("%s: tree has %d %s, sdg.Result %d", name, c.got, c.what, c.want) + } + } + if name != "storedlit" && (wantV == 0 || len(fx.sdg.ParamIn) == 0 || wantS == 0) { + t.Errorf("%s: empty overlay (vertices %d, param_in %d, summary %d)", name, wantV, len(fx.sdg.ParamIn), wantS) + } + t.Logf("%s: %d vertices, %d port ddg, %d summary, %d param_in, %d param_out, %d stand-in formals", + name, vertices, port, summary, len(a.Application.ParamIn), len(a.Application.ParamOut), gotX) + } +} + +// TestGateL4_VertexIdsAreGlobalOrdinals: every vertex — a callable's or a +// stand-in's — has id v2.GlobalOrdinal(owner id, key), a kind, an `of`, no +// span and no callee key; only actual vertices carry a parent. A stand-in +// vertex id is @formal_in: (one "@" after the external id). +func TestGateL4_VertexIdsAreGlobalOrdinals(t *testing.T) { + for _, name := range l4Fixtures { + fx := buildL4(t, name) + for id, on := range bodyIndex(fx.l4) { + n := on.node + if want := v2.GlobalOrdinal(on.owner, on.key); id != want { + t.Errorf("%s: body id %s, want %s", name, id, want) + } + if !isVertexKind(n.Kind) { + if n.Of != "" { + t.Errorf("%s: non-vertex %s carries of %q", name, id, n.Of) + } + continue + } + if n.Of == "" || n.Span != (v2.Span{}) || n.Callee != nil { + t.Errorf("%s: vertex %s = %+v, want an of, no span, no callee", name, id, n) + } + actual := n.Kind == string(sdg.KindActualIn) || n.Kind == string(sdg.KindActualOut) + if actual != (n.Parent != "") { + t.Errorf("%s: vertex %s kind %s parent %q", name, id, n.Kind, n.Parent) + } + data, err := json.Marshal(n) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(data), `"callee"`) || strings.Contains(string(data), `"span"`) { + t.Errorf("%s: vertex %s serialises %s", name, id, data) + } + } + for id, ext := range fx.l4.Application.ExternalSymbols { + for k := range ext.Body { + if !strings.HasPrefix(k, "@formal_") || strings.Count(ext.Body[k].ID[len(id):], "@") != 1 { + t.Errorf("%s: stand-in %s body key %s id %s", name, id, k, ext.Body[k].ID) + } + } + } + } +} + +// TestGateL4_ArityMatchesFormals: every param_in runs from an actual_in of +// the caller to a formal_in that exists in the target's body (a callable or +// a stand-in), and every param_out from such a formal_out to an actual_out; +// var is the target formal's `of` on both. +func TestGateL4_ArityMatchesFormals(t *testing.T) { + for _, name := range l4Fixtures { + fx := buildL4(t, name) + idx := bodyIndex(fx.l4) + check := func(list string, e v2.ParamEdge, actual, formal string, formalEnd string) { + src, dst := idx[e.Src], idx[e.Dst] + act, form := src, dst + if formalEnd == "src" { + act, form = dst, src + } + if act.node.Kind != actual || form.node.Kind != formal { + t.Errorf("%s: %s %s -> %s joins %s to %s", name, list, e.Src, e.Dst, src.node.Kind, dst.node.Kind) + } + if e.Var != form.node.Of { + t.Errorf("%s: %s %s -> %s var %q, target formal of %q", name, list, e.Src, e.Dst, e.Var, form.node.Of) + } + } + for _, e := range fx.l4.Application.ParamIn { + check("param_in", e, string(sdg.KindActualIn), string(sdg.KindFormalIn), "dst") + } + for _, e := range fx.l4.Application.ParamOut { + check("param_out", e, string(sdg.KindActualOut), string(sdg.KindFormalOut), "src") + } + } +} + +// TestGateL4_ParamEdgesSorted: param_in and param_out are sorted by +// (src, dst, var) over the minted ids and hold no duplicate; each +// callable's ddg is sorted by (src, dst, var, prov) and its summary by +// (src, dst). +func TestGateL4_ParamEdgesSorted(t *testing.T) { + for _, name := range l4Fixtures { + fx := buildL4(t, name) + for list, edges := range map[string][]v2.ParamEdge{"param_in": fx.l4.Application.ParamIn, "param_out": fx.l4.Application.ParamOut} { + for i := 1; i < len(edges); i++ { + a, b := edges[i-1], edges[i] + if cmpStrings(a.Src, b.Src, a.Dst, b.Dst, a.Var, b.Var) >= 0 { + t.Errorf("%s: %s[%d..%d] out of order: %+v, %+v", name, list, i-1, i, a, b) + } + } + } + for _, c := range l4Callables(fx.l4) { + for i := 1; i < len(c.Ddg); i++ { + a, b := c.Ddg[i-1], c.Ddg[i] + if cmpStrings(a.Src, b.Src, a.Dst, b.Dst, a.Var, b.Var, strings.Join(a.Prov, ","), strings.Join(b.Prov, ",")) >= 0 { + t.Errorf("%s: %s ddg[%d..%d] out of order: %+v, %+v", name, c.ID, i-1, i, a, b) + } + } + for i := 1; i < len(c.Summary); i++ { + a, b := c.Summary[i-1], c.Summary[i] + if cmpStrings(a.Src, b.Src, a.Dst, b.Dst) >= 0 { + t.Errorf("%s: %s summary[%d..%d] out of order", name, c.ID, i-1, i) + } + } + } + } +} + +// cmpStrings compares the pairs (xs[0], xs[1]), (xs[2], xs[3]), … in turn. +func cmpStrings(xs ...string) int { + for i := 0; i+1 < len(xs); i += 2 { + if c := strings.Compare(xs[i], xs[i+1]); c != 0 { + return c + } + } + return 0 +} + +// TestGateL4_UseStoreEdges pins the emitted L4 of sdg.UseStore (its +// interface call s.Get at 34:9 binds Store.Get, memStore.Get and +// sqlStore.Get): its vertices, its all-to-all summary, and its param edges +// with the minted ids. +func TestGateL4_UseStoreEdges(t *testing.T) { + fx := buildL4(t, "dataflow") + app := fx.l4.Application + mod := v2.ModuleID(app.ID, "sdg/sdg.go") + useStore := v2.CallableID(mod, "example.com/dataflow/sdg.UseStore") + method := func(typ, m string) string { + return v2.CallableID(v2.TypeID(mod, "example.com/dataflow/sdg."+typ), "example.com/dataflow/sdg."+typ+"."+m) + } + var c v2.Callable + for _, x := range l4Callables(fx.l4) { + if x.ID == useStore { + c = x + } + } + for k, of := range map[string]string{ + "@formal_in:0": "s", "@formal_in:1": "k", "@formal_out:0": "", "@formal_out:1": "s", + "34:9/actual_in:0": "_recv", "34:9/actual_in:1": "k", + "34:9/actual_out:0": "", "34:9/actual_out:1": "_recv", + } { + n, ok := c.Body[k] + if !ok || n.Of != of || n.ID != v2.GlobalOrdinal(useStore, k) { + t.Errorf("UseStore body %s = %+v, want of %q", k, n, of) + } + } + if len(c.Summary) != 4 { + t.Errorf("UseStore summary = %v, want all-to-all over 2×2", c.Summary) + } + in := map[[3]string]bool{} + for _, e := range app.ParamIn { + in[[3]string{e.Src, e.Dst, e.Var}] = true + } + out := map[[3]string]bool{} + for _, e := range app.ParamOut { + out[[3]string{e.Src, e.Dst, e.Var}] = true + } + site := func(k string) string { return v2.GlobalOrdinal(useStore, "34:9/"+k) } + for _, e := range [][3]string{ + {site("actual_in:0"), method("Store", "Get") + "@formal_in:0", "_recv"}, + {site("actual_in:0"), method("memStore", "Get") + "@formal_in:0", "s"}, + {site("actual_in:0"), method("sqlStore", "Get") + "@formal_in:0", "s"}, + {site("actual_in:1"), method("sqlStore", "Get") + "@formal_in:1", "k"}, + } { + if !in[e] { + t.Errorf("no param_in %v", e) + } + } + for _, e := range [][3]string{ + {method("Store", "Get") + "@formal_out:0", site("actual_out:0"), ""}, + {method("memStore", "Get") + "@formal_out:1", site("actual_out:1"), "s"}, + } { + if !out[e] { + t.Errorf("no param_out %v", e) + } + } + stores := v2.CallableID(mod, "example.com/dataflow/sdg.Stores") + if !in[[3]string{v2.GlobalOrdinal(stores, "40:2/actual_in:1"), useStore + "@formal_in:1", "k"}] { + t.Errorf("no param_in Stores@40:2/actual_in:1 -> UseStore@formal_in:1") + } +} + +// TestGateL4_StandInBody: a library callee the project calls carries its +// stand-in formals in external_symbols[].body (spec §6), and a param edge +// reaches them. +func TestGateL4_StandInBody(t *testing.T) { + fx := buildL4(t, "dataflow") + app := fx.l4.Application + id := v2.ExternalID(app.ID, "fmt", "Sprintf") + ext, ok := app.ExternalSymbols[id] + if !ok { + t.Fatalf("no external %s", id) + } + // a ...any is a by-ref out (an interface slice), so formal_out is indexed. + for k, of := range map[string]string{"@formal_in:0": "format", "@formal_in:1": "a", "@formal_out:0": "", "@formal_out:1": "a"} { + n, ok := ext.Body[k] + if !ok || n.Of != of || n.ID != id+k { + t.Errorf("%s body %s = %+v, want of %q", id, k, n, of) + } + } + reached := false + for _, e := range app.ParamIn { + reached = reached || e.Dst == id+"@formal_in:0" + } + if !reached { + t.Errorf("no param_in reaches %s@formal_in:0", id) + } + if _, ok := fx.l3.Application.ExternalSymbols[id]; !ok { + t.Fatalf("-a 3 lacks %s", id) + } + if fx.l3.Application.ExternalSymbols[id].Body != nil { + t.Errorf("-a 3 carries a stand-in body") + } +} + +// TestGateL4_DegradedMintsCallables: a degraded run's markers land on the +// envelope with each callable as its can:// id, sorted, entries by code. +func TestGateL4_DegradedMintsCallables(t *testing.T) { + fx := buildL4(t, "rangefunc") + deg := fx.l4.Degraded + if len(deg) == 0 { + t.Fatal("rangefunc: no degraded marker") + } + ids := allCallableIDs(fx.l4) + found := false + for i, d := range deg { + if i > 0 && cmpStrings(deg[i-1].Code, d.Code, deg[i-1].Detail, d.Detail) > 0 { + t.Errorf("degraded not sorted by code: %v", deg) + } + if !slices.IsSorted(d.Callables) { + t.Errorf("%s callables not sorted: %v", d.Code, d.Callables) + } + for _, id := range d.Callables { + if !ids[id] { + t.Errorf("%s names %s, not a callable id", d.Code, id) + } + } + if d.Code == sdg.CodeRangeFunc { + found = true + want := v2.CallableID(v2.ModuleID(fx.l4.Application.ID, "r.go"), "example.com/rangefunc.RangeFunc") + if !slices.Contains(d.Callables, want) { + t.Errorf("range_func callables %v lack %s", d.Callables, want) + } + } + } + if !found { + t.Errorf("degraded %+v has no range_func", deg) + } + if fx.l3.Degraded != nil { + t.Errorf("-a 3 carries degraded") + } + if len(buildL4(t, "dataflow").l4.Degraded) != 0 { + t.Errorf("dataflow degraded = %+v, want none", buildL4(t, "dataflow").l4.Degraded) + } +} + +// TestGateL4_StoredLiteralGlobalReachable (Task 13 review): storedlit's +// Publish writes the global sink only inside a literal it stores, and Fire +// calls that literal through a function value nothing resolves — so inside +// Publish, sink's formal_out is fed only by its formal_in (the identity +// edge). The flow v → sink → readSink → must still be reachable in +// the emitted graph (ddg ∪ cdg ∪ summary ∪ param_in ∪ param_out, the SDK's +// walk): the caller's host statement Publish(v) weak-defines sink, whose +// later use feeds readSink's global formal_in. +func TestGateL4_StoredLiteralGlobalReachable(t *testing.T) { + fx := buildL4(t, "storedlit") + a := fx.l4 + succ := map[string][]string{} + for _, c := range l4Callables(a) { + g := func(k string) string { return v2.GlobalOrdinal(c.ID, k) } + for _, e := range c.Ddg { + succ[g(e.Src)] = append(succ[g(e.Src)], g(e.Dst)) + } + for _, e := range c.Cdg { + succ[g(e.Src)] = append(succ[g(e.Src)], g(e.Dst)) + } + for _, e := range c.Summary { + succ[g(e.Src)] = append(succ[g(e.Src)], g(e.Dst)) + } + } + for _, e := range slices.Concat(a.Application.ParamIn, a.Application.ParamOut) { + succ[e.Src] = append(succ[e.Src], e.Dst) + } + reach := func(from string) map[string]bool { + seen := map[string]bool{from: true} + for work := []string{from}; len(work) > 0; { + n := work[len(work)-1] + work = work[:len(work)-1] + for _, m := range succ[n] { + if !seen[m] { + seen[m] = true + work = append(work, m) + } + } + } + return seen + } + mod := v2.ModuleID(a.Application.ID, "storedlit.go") + fn := func(name string) string { return v2.CallableID(mod, "example.com/storedlit."+name) } + idx := bodyIndex(a) + formal := func(owner, kind, of string) string { + for id, on := range idx { + if on.owner == owner && on.node.Kind == kind && on.node.Of == of { + return id + } + } + t.Fatalf("%s has no %s of %q", owner, kind, of) + return "" + } + src := formal(fn("PublishThenRead"), "formal_in", "v") + r := reach(src) + for _, dst := range []string{ + formal(fn("readSink"), "formal_in", ":example.com/storedlit::sink"), + formal(fn("PublishThenRead"), "formal_out", ""), + } { + if !r[dst] { + t.Errorf("%s does not reach %s", src, dst) + } + } + // The gap the caller covers: over Publish's own edges (ddg ∪ cdg ∪ + // summary), v does not reach sink's formal_out — sink's formal_out is fed + // only by its formal_in, the identity edge, as the literal's write has no + // param_out. Update this when the gap closes. + var pub v2.Callable + for _, c := range l4Callables(a) { + if c.ID == fn("Publish") { + pub = c + } + } + local := map[string][]string{} + for _, e := range pub.Ddg { + local[e.Src] = append(local[e.Src], e.Dst) + } + for _, e := range pub.Cdg { + local[e.Src] = append(local[e.Src], e.Dst) + } + for _, e := range pub.Summary { + local[e.Src] = append(local[e.Src], e.Dst) + } + keyOf := func(id string) string { return idx[id].key } + seen := map[string]bool{keyOf(formal(fn("Publish"), "formal_in", "v")): true} + for work := []string{keyOf(formal(fn("Publish"), "formal_in", "v"))}; len(work) > 0; { + n := work[len(work)-1] + work = work[:len(work)-1] + for _, m := range local[n] { + if !seen[m] { + seen[m] = true + work = append(work, m) + } + } + } + sinkOut := keyOf(formal(fn("Publish"), "formal_out", ":example.com/storedlit::sink")) + if seen[sinkOut] { + t.Errorf("Publish: v now reaches %s intra-procedurally; the stored-literal gap closed, update this test", sinkOut) + } + var feeders []string + for _, e := range pub.Ddg { + if e.Dst == sinkOut { + feeders = append(feeders, e.Src) + } + } + if !slices.Equal(feeders, []string{keyOf(formal(fn("Publish"), "formal_in", ":example.com/storedlit::sink"))}) { + t.Errorf("Publish: %s fed by %v, want only sink's formal_in (identity edge)", sinkOut, feeders) + } +} + +// buildL4Jobs runs the -a 4 pipeline over dir from scratch with -j jobs and +// returns EmitTo's bytes, checking they equal json.Marshal(Emit(...)). +func buildL4Jobs(t *testing.T, dir, name string, jobs int) []byte { + t.Helper() + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + units := b.L3Units(st) + res := dataflow.RunWithSideTables(b.Fset(), units, 3, jobs) + l4 := sdg.Run(sdg.Input{ + Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Externals: app.Externals, CallGraph: app.CallGraph, + Units: units, L3: res, K: 3, Jobs: jobs, + }) + l4.ApplyTo(app) + dataflow.ReleaseAll(res) + var buf bytes.Buffer + if err := EmitTo(&buf, app, name, dir, 4, "det", 3, res, &l4); err != nil { + t.Fatal(err) + } + want, err := json.Marshal(Emit(app, name, dir, 4, "det", 3, res, &l4)) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(buf.Bytes(), want) { + t.Fatalf("%s: EmitTo differs from json.Marshal(Emit) at byte %d (len %d vs %d)", name, firstDiff(buf.Bytes(), want), buf.Len(), len(want)) + } + return buf.Bytes() +} + +// TestGateL4_DeterministicAcrossJobs: the -a 4 JSON is byte-identical +// between -j 1 and -j 8 and across fresh runs, on sdg, flow and chi; the +// streaming writer matches json.Marshal of the built tree. +func TestGateL4_DeterministicAcrossJobs(t *testing.T) { + _, thisFile, _, _ := runtime.Caller(0) + td := filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata") + for name, rel := range map[string]string{ + "sdg": filepath.Join("dataflow", "sdg"), + "flow": filepath.Join("dataflow", "flow"), + "chi": "chi", + } { + dir, err := filepath.Abs(filepath.Join(td, rel)) + if err != nil { + t.Fatal(err) + } + first := buildL4Jobs(t, dir, name, 1) + if !bytes.Contains(first, []byte(`"param_in":[`)) { + t.Errorf("%s: -a 4 JSON has no param_in", name) + } + for i, jobs := range []int{8, 1, 8} { + if got := buildL4Jobs(t, dir, name, jobs); !bytes.Equal(got, first) { + t.Errorf("%s: run %d (-j %d) differs from -j 1 at byte %d", name, i+2, jobs, firstDiff(got, first)) + } + } + } +} + +// TestEmitL4_UnresolvedRefIsDroppedAndWarned: a param edge whose endpoint +// names no emitted callable is a bug upstream; the emitter never writes it +// with a dangling endpoint (it warns and drops it). +func TestEmitL4_UnresolvedRefIsDroppedAndWarned(t *testing.T) { + fx := buildL4(t, "dataflow") + res := *fx.sdg + ghost := sdg.Target{Callable: dataflow.L3Key{Path: "nowhere.go", Signature: "example.com/nowhere.F"}} + res.ParamIn = append(slices.Clone(res.ParamIn), sdg.ParamEdge{ + Src: sdg.Ref{Owner: ghost, Key: "1:1/actual_in:0"}, + Dst: sdg.Ref{Owner: ghost, Key: "@formal_in:0"}, + Var: "x", + }) + var warned []string + prev := warnf + warnf = func(format string, args ...any) { warned = append(warned, fmt.Sprintf(format, args...)) } + defer func() { warnf = prev }() + app := &schema.GoApplication{SymbolTable: fx.st, Externals: map[string]schema.GoExternalCallee{}} + out := Emit(app, "dataflow", fx.dir, 4, "t", 3, fx.res, &res) + for _, e := range out.Application.ParamIn { + if strings.Contains(e.Src, "nowhere") || strings.Contains(e.Dst, "nowhere") { + t.Errorf("dangling param_in emitted: %+v", e) + } + } + if len(warned) == 0 || !strings.Contains(strings.Join(warned, "\n"), "nowhere.go") { + t.Errorf("no warning names the unresolved ref; warned %q", warned) + } +} diff --git a/internal/syntactic_analysis/v2emit/gate_test.go b/internal/syntactic_analysis/v2emit/gate_test.go index cc37d35..4679a60 100644 --- a/internal/syntactic_analysis/v2emit/gate_test.go +++ b/internal/syntactic_analysis/v2emit/gate_test.go @@ -58,7 +58,7 @@ func buildMultipackageV2(t *testing.T) *v2.Analysis { t.Fatalf("symbol table build failed: %v", err) } app := &schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}} - return Emit(app, "multipackage", dir, 1, "gate-test", 0, nil) + return Emit(app, "multipackage", dir, 1, "gate-test", 0, nil, nil) } // mustModule fetches a module by relative path or fails. @@ -322,7 +322,7 @@ func TestGateL1_SupersetOfV1(t *testing.T) { t.Fatalf("symbol table build failed: %v", err) } app := &schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}} - out := Emit(app, "multipackage", dir, 1, "gate-test", 0, nil) + out := Emit(app, "multipackage", dir, 1, "gate-test", 0, nil, nil) // Every v1 file, type, method and function reappears in v2. for rel, v1file := range st { @@ -387,7 +387,7 @@ func buildDataflowL1V2(t *testing.T) *v2.Analysis { t.Fatalf("symbol table build failed: %v", err) } app := &schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}} - return Emit(app, "dataflow", dir, 1, "gate-test", 0, nil) + return Emit(app, "dataflow", dir, 1, "gate-test", 0, nil, nil) } func TestGateL1_TypeKindsAliasAndDefined(t *testing.T) { @@ -478,7 +478,7 @@ func buildDataflowL2V2(t *testing.T) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges} - return Emit(app, "dataflow", dir, 2, "gate-test", 0, nil) + return Emit(app, "dataflow", dir, 2, "gate-test", 0, nil, nil) } // TestGateL2_CoPositionedCallsResolveSeparately: in s.A().B() the outer call @@ -638,7 +638,7 @@ func TestGate_LineDirectiveSpansSliceToText(t *testing.T) { if err != nil { t.Fatalf("symbol table build failed: %v", err) } - out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}}, "dataflow", dir, 1, "gate-test", 0, nil) + out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}}, "dataflow", dir, 1, "gate-test", 0, nil, nil) const rel = "linedir/linedir.go" mod := mustModule(t, out, rel) calls := 0 diff --git a/testdata/storedlit/go.mod b/testdata/storedlit/go.mod new file mode 100644 index 0000000..e0929f2 --- /dev/null +++ b/testdata/storedlit/go.mod @@ -0,0 +1,3 @@ +module example.com/storedlit + +go 1.21 diff --git a/testdata/storedlit/storedlit.go b/testdata/storedlit/storedlit.go new file mode 100644 index 0000000..cacae73 --- /dev/null +++ b/testdata/storedlit/storedlit.go @@ -0,0 +1,31 @@ +// Package storedlit is the L4 fixture for a global written only inside a +// stored function literal that runs later through a function value no +// analysis resolves (Task 13 review): inside Publish, v reaches sink's +// formal_out only through the caller's host, never through the literal. +package storedlit + +import "sync/atomic" + +// sink is written only by the literal Publish stores. +var sink string + +var hook atomic.Value + +// Publish stores a literal that writes v into sink. +func Publish(v string) { + hook.Store(func() { sink = v }) +} + +// Fire calls the stored literal through an unresolved function value. +func Fire() { + hook.Load().(func())() +} + +func readSink() string { return sink } + +// PublishThenRead returns what the stored literal wrote. +func PublishThenRead(v string) string { + Publish(v) + Fire() + return readSink() +} From 57ff5a4cc08bf88b7110a01845481c2a5b71dc43 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 20:41:06 -0400 Subject: [PATCH 23/65] feat(l4): type-based may-alias points-to edges --- internal/dataflow/accesspath.go | 9 +- internal/dataflow/reachdefs.go | 111 +++++++- internal/dataflow/reachdefs_test.go | 147 +++++++++-- internal/sdg/alias.go | 108 ++++++++ internal/sdg/alias_test.go | 239 ++++++++++++++++++ internal/sdg/run.go | 11 +- internal/sdg/wiring.go | 85 +++++-- internal/sdg/wiring_test.go | 7 +- .../syntactic_analysis/v2emit/gate_l4_test.go | 25 ++ testdata/dataflow/sdg/sdg.go | 34 +++ 10 files changed, 728 insertions(+), 48 deletions(-) create mode 100644 internal/sdg/alias.go create mode 100644 internal/sdg/alias_test.go diff --git a/internal/dataflow/accesspath.go b/internal/dataflow/accesspath.go index 307d190..9415792 100644 --- a/internal/dataflow/accesspath.go +++ b/internal/dataflow/accesspath.go @@ -43,7 +43,14 @@ func (p Path) Covers(q Path) bool { } ph, ps := splitSpell(p.Spell) qh, qs := splitSpell(q.Spell) - if ph != qh || len(ps) > len(qs) { + return ph == qh && stepsCover(ps, qs) +} + +// stepsCover is Covers on step lists alone, bases ignored: ps is a prefix of +// qs, a final wildcard of ps matching any non-empty remainder. The alias +// rule (BuildDDGAlias) compares the suffixes of two different bases with it. +func stepsCover(ps, qs []string) bool { + if len(ps) > len(qs) { return false } for i, s := range ps { diff --git a/internal/dataflow/reachdefs.go b/internal/dataflow/reachdefs.go index bca8680..64d663c 100644 --- a/internal/dataflow/reachdefs.go +++ b/internal/dataflow/reachdefs.go @@ -43,6 +43,52 @@ import ( // cfg or the DefUse lists. The output is deduplicated, sorted by (Src, Dst, // Var) and non-nil. Its prov (["ssa"] at L3) is added at emission. func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { + ddg, _ := solveDDG(nodes, cfg, du, nil) + return ddg +} + +// BuildDDGAlias is BuildDDG with a may-alias oracle (spec §4.10): the same +// reaching-definitions fixpoint, kills and use matching, plus one rule. A +// use u at n also links from a def d in IN(n) whose base differs from u's +// when +// +// - d's path has a non-empty suffix (a step after its base): a bare def +// `p = …` rebinds the variable and writes no memory another base can +// reach (ruling, Task 17), so only a write through a base aliases. Hence +// no alias edge leaves @entry, whose defs are all bare. The use side is +// unrestricted: a bare use (`return req`) reads the whole pointee; +// - the suffixes are prefix-compatible, in Covers' two-directional sense +// with the bases ignored (d.f covers u.f.g, u.* covers d.f); and +// - mayAlias(d.path, u) holds. +// +// Alias never kills: a strong def through one base kills only the defs of +// its own base (BuildDDG's KILL), so a def that may be overwritten through +// an alias still reaches. The edge is d.node → n with Var = u.Spell, as for +// BuildDDG. The output is BuildDDG's plus the alias edges, deduplicated, +// sorted by (Src, Dst, Var) and non-nil; a nil mayAlias yields BuildDDG. +func BuildDDGAlias(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias func(a, b Path) bool) []DdgEdge { + ddg, alias := solveDDG(nodes, cfg, du, mayAlias) + if len(alias) == 0 { + return ddg + } + out := append(ddg, alias...) + sortDdg(out) + return out +} + +// BuildDDGSplit is BuildDDGAlias split in two: BuildDDG's edges, and the +// edges only the alias rule yields (none of which is in the first list), +// each sorted by (Src, Dst, Var). One fixpoint serves both: the L4 port +// wiring (package sdg) maps the first to reaching-defs port edges and the +// second to its points-to delta. +func BuildDDGSplit(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias func(a, b Path) bool) (ddg, alias []DdgEdge) { + return solveDDG(nodes, cfg, du, mayAlias) +} + +// solveDDG is the fixpoint and use matching of BuildDDG and BuildDDGAlias: +// the BuildDDG edges, and (mayAlias non-nil) the alias-only edges, both +// sorted, the first non-nil. +func solveDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias func(a, b Path) bool) (ddg, alias []DdgEdge) { g := newPdomGraph(nodes, cfg) n := len(g.keys) @@ -167,10 +213,74 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { } } + if mayAlias != nil { + alias = aliasEdges(g.keys, du, in, func(i int) Path { return defs[i].path }, func(i int) string { return g.keys[defs[i].node] }, edges, mayAlias) + } + return sortedDdg(edges), alias +} + +// aliasEdges is BuildDDGAlias's alias rule over a solved fixpoint: for each +// node with uses, each reaching def with a suffix is matched against each +// use of a different base. Edges already in base are skipped. The result is +// sorted (nil when empty). +func aliasEdges(keys []string, du map[string]DefUse, in [][]uint64, pathOf func(int) Path, nodeOf func(int) string, base map[DdgEdge]bool, mayAlias func(a, b Path) bool) []DdgEdge { + type use struct { + path Path + steps []string + } + steps := map[int][]string{} // def index → its steps; absent: not yet split + edges := map[DdgEdge]bool{} + for v, k := range keys { + var uses []use + for _, u := range du[k].Uses { + _, s := splitSpell(u.Spell) + uses = append(uses, use{u, s}) + } + if len(uses) == 0 { + continue + } + for w, x := range in[v] { + for x != 0 { + b := bits.TrailingZeros64(x) + x &^= 1 << b + i := w*64 + b + ds, ok := steps[i] + if !ok { + _, ds = splitSpell(pathOf(i).Spell) + steps[i] = ds + } + if len(ds) == 0 { + continue // a bare def writes no aliased memory + } + d := pathOf(i) + for _, u := range uses { + if u.path.Base == d.Base || !(stepsCover(ds, u.steps) || stepsCover(u.steps, ds)) { + continue + } + e := DdgEdge{Src: nodeOf(i), Dst: k, Var: u.path.Spell} + if !base[e] && !edges[e] && mayAlias(d, u.path) { + edges[e] = true + } + } + } + } + } + if len(edges) == 0 { + return nil + } + return sortedDdg(edges) +} + +func sortedDdg(edges map[DdgEdge]bool) []DdgEdge { res := make([]DdgEdge, 0, len(edges)) for e := range edges { res = append(res, e) } + sortDdg(res) + return res +} + +func sortDdg(res []DdgEdge) { sort.Slice(res, func(i, j int) bool { a, b := res[i], res[j] if a.Src != b.Src { @@ -181,7 +291,6 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { } return a.Var < b.Var }) - return res } // killBucket is the kill index key of a path: its Base, or for a nil Base diff --git a/internal/dataflow/reachdefs_test.go b/internal/dataflow/reachdefs_test.go index bdaec0b..cbc28ec 100644 --- a/internal/dataflow/reachdefs_test.go +++ b/internal/dataflow/reachdefs_test.go @@ -7,6 +7,7 @@ import ( "path/filepath" "reflect" "runtime" + "slices" "sort" "testing" @@ -519,14 +520,17 @@ func TestDDG_ValueStructWithSliceIsWrittenThrough(t *testing.T) { } } -// flatDDG is a reference reaching-definitions DDG with the kill rule of -// BuildDDG tested against every definition of the body (no bucketing), on -// plain map sets: the oracle TestDDG_KillBucketsMatchFlat compares against. -func flatDDG(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]dataflow.DefUse) []dataflow.DdgEdge { - type def struct { - node string - path dataflow.Path - } +// flatDef is a definition of the flat reference: a node and a path. +type flatDef struct { + Node string + Path dataflow.Path +} + +// flatReaching is a reference reaching-definitions fixpoint with the kill +// rule of BuildDDG tested against every definition of the body (no +// bucketing), on plain map sets: IN of every CFG node. +func flatReaching(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]dataflow.DefUse) map[string]map[flatDef]bool { + type def = flatDef isNode := map[string]bool{} for _, n := range nodes { if n.Parent == "" { @@ -554,7 +558,7 @@ func flatDDG(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]datafl shallow[p] = true } for _, p := range du[k].Defs { - if p == d.path || !shallow[p] && p.Covers(d.path) { + if p == d.Path || !shallow[p] && p.Covers(d.Path) { return true } } @@ -579,7 +583,7 @@ func flatDDG(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]datafl } } for _, d := range all { - if d.node == k { + if d.Node == k { next[d] = true } } @@ -596,12 +600,41 @@ func flatDDG(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]datafl out[k] = next } } + return in +} + +// flatDDG is the reference DDG over flatReaching: each use links from each +// reaching def it interferes with. TestDDG_KillBucketsMatchFlat compares +// BuildDDG against it. +func flatDDG(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]dataflow.DefUse) []dataflow.DdgEdge { + return flatDDGAlias(nodes, cfg, du, nil) +} + +// anyBase is the broadest oracle: any two resolved bases may alias. The flat +// alias gate uses it so that the matching, not the oracle, is under test. +func anyBase(a, b dataflow.Path) bool { return a.Base != nil && b.Base != nil } + +// flatDDGAlias is BuildDDGAlias's reference, formulated independently: over +// flatReaching, a use u also links from a reaching def d of another base +// whose spelling has a suffix when d's suffix, rebased onto u's base, +// interferes with u (Covers either way) and mayAlias(d, u). nil mayAlias is +// flatDDG. +func flatDDGAlias(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]dataflow.DefUse, mayAlias func(a, b dataflow.Path) bool) []dataflow.DdgEdge { edges := map[dataflow.DdgEdge]bool{} - for k := range isNode { - for d := range in[k] { + for k, ins := range flatReaching(nodes, cfg, du) { + for d := range ins { + suffix := d.Path.Spell[len(dataflow.SpellBase(d.Path.Spell)):] for _, u := range du[k].Uses { - if d.path.Covers(u) || u.Covers(d.path) { - edges[dataflow.DdgEdge{Src: d.node, Dst: k, Var: u.Spell}] = true + if d.Path.Covers(u) || u.Covers(d.Path) { + edges[dataflow.DdgEdge{Src: d.Node, Dst: k, Var: u.Spell}] = true + continue + } + if mayAlias == nil || suffix == "" || u.Base == d.Path.Base { + continue + } + r := dataflow.Path{Base: u.Base, Spell: dataflow.SpellBase(u.Spell) + suffix} + if (r.Covers(u) || u.Covers(r)) && mayAlias(d.Path, u) { + edges[dataflow.DdgEdge{Src: d.Node, Dst: k, Var: u.Spell}] = true } } } @@ -622,7 +655,7 @@ func flatDDG(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]datafl func TestDDG_KillBucketsMatchFlat(t *testing.T) { _, thisFile, _, _ := runtime.Caller(0) root := filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata") - total := 0 + total, aliased := 0, 0 for _, dir := range []string{"dataflow", "chi"} { fset, cases := loadDiffCases(t, filepath.Join(root, dir)) for _, c := range cases { @@ -634,11 +667,19 @@ func TestDDG_KillBucketsMatchFlat(t *testing.T) { t.Errorf("%s: bucketed DDG (%d edges) != flat (%d edges)", c.name, len(got), len(want)) } total += len(got) + gotA, wantA := dataflow.BuildDDGAlias(kept, es, du, anyBase), flatDDGAlias(kept, es, du, anyBase) + if !reflect.DeepEqual(gotA, wantA) { + t.Errorf("%s: bucketed alias DDG (%d edges) != flat (%d edges)", c.name, len(gotA), len(wantA)) + } + aliased += len(gotA) - len(got) } } if total < 1000 { t.Errorf("kill-bucket gate covered only %d edges", total) } + if aliased < 100 { + t.Errorf("alias gate covered only %d alias edges", aliased) + } // Hand-built: x (outer), x (shadowing inner, same spelling), the // unresolved x and u (nil base), and a : pseudo path. @@ -681,6 +722,9 @@ func TestDDG_KillBucketsMatchFlat(t *testing.T) { if !reflect.DeepEqual(got, want) { t.Errorf("hand-built kills:\n got %v\nwant %v", got, want) } + if gotA, wantA := dataflow.BuildDDGAlias(nodes, cfg, du, anyBase), flatDDGAlias(nodes, cfg, du, anyBase); !reflect.DeepEqual(gotA, wantA) { + t.Errorf("hand-built alias kills:\n got %v\nwant %v", gotA, wantA) + } // The unresolved u (1:4) kills u.f of 1:1 and 1:3 by spelling head: only // 1:4's own u reaches 1:6's u.f. for _, g := range got { @@ -689,3 +733,76 @@ func TestDDG_KillBucketsMatchFlat(t *testing.T) { } } } + +// TestDDGAlias_Rule pins BuildDDGAlias on a hand-built line, with p and q +// pointers of one type (may alias) and v an int: +// +// @entry defines p, q, v +// 1:1 p.f = v +// 1:2 q.f = v (a strong def through q: kills q.f… only) +// 1:3 use p.f, q.f.g, q, p.g +// 1:4 q = nil (a bare def: rebinds q, writes no aliased memory) +// 1:5 use p.f +// +// 1:1's p.f reaches 1:2's bare use q (the pointer q.f = v reads). At 1:3, +// by alias: 1:2's q.f may have written p.f, yet 1:1's p.f still +// reaches by its own base (alias never kills); 1:1's p.f reaches q.f.g (its +// suffix covers) and the bare use q (a bare use reads the whole pointee); +// p.g's suffix is incompatible with q.f. At 1:5 nothing aliases: 1:4 killed +// q.f by q's own base, and its bare def of q aliases nothing, like @entry's. +func TestDDGAlias_Rule(t *testing.T) { + ptr := types.NewPointer(types.NewStruct(nil, nil)) + p := types.NewVar(token.NoPos, nil, "p", ptr) + q := types.NewVar(token.NoPos, nil, "q", ptr) + v := types.NewVar(token.NoPos, nil, "v", types.Typ[types.Int]) + P := func(b types.Object, s string) dataflow.Path { return dataflow.Path{Base: b, Spell: s} } + keys := []string{entry, "1:1", "1:2", "1:3", "1:4", "1:5", exit} + var nodes []dataflow.Node + var cfg []dataflow.CfgEdge + for i, k := range keys { + nodes = append(nodes, dataflow.Node{Key: k}) + if i > 0 { + cfg = append(cfg, dataflow.CfgEdge{Src: keys[i-1], Dst: k, Kind: "fallthrough"}) + } + } + du := map[string]dataflow.DefUse{ + entry: {Defs: []dataflow.Path{P(p, "p"), P(q, "q"), P(v, "v")}}, + "1:1": {Defs: []dataflow.Path{P(p, "p.f")}, Uses: []dataflow.Path{P(p, "p"), P(v, "v")}}, + "1:2": {Defs: []dataflow.Path{P(q, "q.f")}, Uses: []dataflow.Path{P(q, "q"), P(v, "v")}}, + "1:3": {Uses: []dataflow.Path{P(p, "p.f"), P(q, "q.f.g"), P(q, "q"), P(p, "p.g")}}, + "1:4": {Defs: []dataflow.Path{P(q, "q")}}, + "1:5": {Uses: []dataflow.Path{P(p, "p.f")}}, + } + mayAlias := func(a, b dataflow.Path) bool { return a.Base != v && b.Base != v && a.Base != nil && b.Base != nil } + base := dataflow.BuildDDG(nodes, cfg, du) + got := dataflow.BuildDDGAlias(nodes, cfg, du, mayAlias) + ddg, alias := dataflow.BuildDDGSplit(nodes, cfg, du, mayAlias) + if !reflect.DeepEqual(ddg, base) { + t.Errorf("split ddg != BuildDDG:\n got %v\nwant %v", ddg, base) + } + want := []dataflow.DdgEdge{ + dd("1:1", "1:2", "q"), // the bare use q at 1:2, as p covers p.f on its own base + dd("1:1", "1:3", "q"), + dd("1:1", "1:3", "q.f.g"), + dd("1:2", "1:3", "p.f"), + } + if !reflect.DeepEqual(alias, want) { + t.Errorf("alias edges:\n got %v\nwant %v", alias, want) + } + for _, e := range base { + if !slices.Contains(got, e) { + t.Errorf("BuildDDGAlias lost BuildDDG edge %v", e) + } + } + if len(got) != len(base)+len(alias) { + t.Errorf("BuildDDGAlias = %d edges, want BuildDDG %d + alias %d", len(got), len(base), len(alias)) + } + for _, e := range []dataflow.DdgEdge{dd("1:1", "1:3", "p.f"), dd("1:1", "1:5", "p.f")} { + if !slices.Contains(base, e) { + t.Errorf("%v: q.f = v killed p.f across bases: %v", e, base) + } + } + if nilA := dataflow.BuildDDGAlias(nodes, cfg, du, nil); !reflect.DeepEqual(nilA, base) { + t.Errorf("nil oracle: got %v, want BuildDDG %v", nilA, base) + } +} diff --git a/internal/sdg/alias.go b/internal/sdg/alias.go new file mode 100644 index 0000000..89d5d4f --- /dev/null +++ b/internal/sdg/alias.go @@ -0,0 +1,108 @@ +package sdg + +import ( + "go/types" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// ProvPointsTo is the prov of every may-alias ddg edge (spec §4.10): the L4 +// semantic delta over L3's "ssa" edges and the "reaching-defs" port wiring. +const ProvPointsTo = "points-to" + +// AliasOracle answers may-alias queries between two access paths of one +// callable (spec §4.10). The interface is frozen so that a points-to +// analysis (an Andersen solve over go/ssa) can replace the type-based MVP +// without a schema change; prov stays "points-to". +// +// The DDG solver (dataflow.BuildDDGAlias) consults it only for a def and a +// use with different bases and prefix-compatible suffixes, the def's suffix +// non-empty, so MayAlias decides on the bases alone. +type AliasOracle interface { + MayAlias(a, b dataflow.Path) bool +} + +// aliasOracle is the oracle the port wiring consults; nil turns the alias +// pass off (a test seam: TestAlias_AddsOnly compares Run off vs on). +var aliasOracle AliasOracle = typeOracle{} + +// typeOracle is the type-based may-alias MVP (spec §4.10). Two paths may +// alias iff +// +// - both have a base object (a nil base, a ":" pseudo path or an +// unresolved identifier, never aliases anything), and +// - they share the base, or both bases' types may hold a reference +// (dataflow.MayHoldRef, the predicate the by-ref rules share) and either +// the types are identical or one is an interface the other's type +// implements (types.Implements on T or *T). +// +// A value type that holds no reference never aliases: its copy is its own +// memory. A package-level variable is a base like any other, so a global +// pointer may alias a local one of its type; so may a closure's capture. +// A type parameter is compared by identity only, never through its +// constraint's interface. +type typeOracle struct{} + +func (typeOracle) MayAlias(a, b dataflow.Path) bool { + if a.Base == nil || b.Base == nil { + return false + } + if a.Base == b.Base { + return true + } + return typesMayAlias(a.Base.Type(), b.Base.Type()) +} + +func typesMayAlias(ta, tb types.Type) bool { + if !dataflow.MayHoldRef(ta) || !dataflow.MayHoldRef(tb) { + return false + } + if types.Identical(ta, tb) { + return true + } + return implements(ta, tb) || implements(tb, ta) +} + +// implements reports whether iface is an interface type (not a type +// parameter) that t or *t implements. +func implements(t, iface types.Type) bool { + if _, tp := iface.(*types.TypeParam); tp { + return false + } + it, ok := iface.Underlying().(*types.Interface) + if !ok { + return false + } + if types.Implements(t, it) { + return true + } + if _, ptr := t.Underlying().(*types.Pointer); ptr || types.IsInterface(t) { + return false + } + return types.Implements(types.NewPointer(t), it) +} + +// memoOracle memoizes o per base pair for one unit's alias solve: the +// solver asks about the same two bases once per reaching def and use, and +// types.Implements walks method sets. It lives only for one wire call. +// Only typeOracle, which decides on the bases alone, is memoized; any other +// oracle is asked every time. +func memoOracle(o AliasOracle) func(a, b dataflow.Path) bool { + if _, ok := o.(typeOracle); !ok { + return o.MayAlias + } + type pair struct{ a, b types.Object } + memo := map[pair]bool{} + return func(a, b dataflow.Path) bool { + if a.Base == nil || b.Base == nil { + return o.MayAlias(a, b) + } + k := pair{a.Base, b.Base} + v, ok := memo[k] + if !ok { + v = o.MayAlias(a, b) + memo[k] = v + } + return v + } +} diff --git a/internal/sdg/alias_test.go b/internal/sdg/alias_test.go new file mode 100644 index 0000000..893ca78 --- /dev/null +++ b/internal/sdg/alias_test.go @@ -0,0 +1,239 @@ +package sdg + +import ( + "go/token" + "go/types" + "reflect" + "slices" + "strings" + "testing" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +func pt(src, dst, v string) PortEdge { + return PortEdge{Src: src, Dst: dst, Var: v, Prov: ProvPointsTo} +} + +func pointsTo(es []PortEdge) []PortEdge { + var out []PortEdge + for _, e := range es { + if e.Prov == ProvPointsTo { + out = append(out, e) + } + } + return out +} + +// runAlias runs L4 over testdata/dataflow with the alias oracle o (nil: off). +func runAlias(t *testing.T, o AliasOracle) Result { + t.Helper() + defer func(v AliasOracle) { aliasOracle = v }(aliasOracle) + aliasOracle = o + return Run(loadInput(t, "dataflow")) +} + +// TestAlias_PointerCopyLinks: in AliasPtr (p := req; p.UserID = id; return +// req.UserID) the write through the copy p reaches the read through req only +// by may-alias, as a points-to edge 360:2 → 361:2 on req.UserID; the same +// write reaches req's by-ref formal_out (var its `of`, as wire maps an +// @exit edge). Those are AliasPtr's only points-to edges. +// +// AliasPtrWhole returns the bare req (ruling, Task 17: the use side is +// unrestricted): the write p.UserID is prefix-compatible with the whole read. +func TestAlias_PointerCopyLinks(t *testing.T) { + res := runAlias(t, typeOracle{}) + for _, c := range []struct { + fn string + want []PortEdge + }{ + {"AliasPtr", []PortEdge{pt("360:2", "361:2", "req.UserID"), pt("360:2", "@formal_out:1", "req")}}, + {"AliasPtrWhole", []PortEdge{pt("368:2", "369:2", "req"), pt("368:2", "@formal_out:1", "req")}}, + // aliasAssign (q := p; q.n = v): q.n = v reaches p's formal_out. + {"aliasAssign", []PortEdge{pt("153:2", "@formal_out", "p")}}, + } { + if got := pointsTo(res.Callables[sdgKey(c.fn)].Ddg); !slices.Equal(got, c.want) { + t.Errorf("%s points-to edges:\n got %+v\n want %+v", c.fn, got, c.want) + } + } +} + +// TestAlias_ValueCopiesNeverAlias: AliasValue copies req by value (no +// reference inside Req), so the write to the copy reaches nothing through +// req: no points-to edge. The oracle itself refuses value types, nil bases +// and pseudo paths, and accepts an interface its operand implements (T or +// *T), a global pointer with a local one, and identical reference types. +func TestAlias_ValueCopiesNeverAlias(t *testing.T) { + res := runAlias(t, typeOracle{}) + if got := pointsTo(res.Callables[sdgKey("AliasValue")].Ddg); len(got) != 0 { + t.Errorf("AliasValue has points-to edges %+v", got) + } + + pkg := types.NewPackage("example.com/p", "p") + v := func(name string, t types.Type) dataflow.Path { + return dataflow.Path{Base: types.NewVar(token.NoPos, pkg, name, t), Spell: name + ".f"} + } + st := types.NewStruct([]*types.Var{types.NewField(token.NoPos, pkg, "f", types.Typ[types.Int], false)}, nil) + named := types.NewNamed(types.NewTypeName(token.NoPos, pkg, "S", nil), st, nil) + m := types.NewFunc(token.NoPos, pkg, "M", types.NewSignatureType(types.NewVar(token.NoPos, pkg, "s", types.NewPointer(named)), nil, nil, nil, nil, false)) + named.AddMethod(m) + iface := types.NewInterfaceType([]*types.Func{types.NewFunc(token.NoPos, pkg, "M", types.NewSignatureType(nil, nil, nil, nil, nil, false))}, nil).Complete() + other := types.NewInterfaceType([]*types.Func{types.NewFunc(token.NoPos, pkg, "N", types.NewSignatureType(nil, nil, nil, nil, nil, false))}, nil).Complete() + gv := types.NewVar(token.NoPos, pkg, "G", types.NewPointer(named)) + pkg.Scope().Insert(gv) + global := dataflow.Path{Base: gv, Spell: "example.com/p::G.f"} + + o := typeOracle{} + for _, c := range []struct { + what string + a, b dataflow.Path + want bool + }{ + {"value structs", v("a", named), v("b", named), false}, + {"ints", v("a", types.Typ[types.Int]), v("b", types.Typ[types.Int]), false}, + {"identical pointers", v("a", types.NewPointer(named)), v("b", types.NewPointer(named)), true}, + {"pointer, value", v("a", types.NewPointer(named)), v("b", named), false}, + {"different pointers", v("a", types.NewPointer(named)), v("b", types.NewPointer(types.Typ[types.Int])), false}, + {"*S implements I", v("a", types.NewPointer(named)), v("b", iface), true}, + {"I, *S", v("a", iface), v("b", types.NewPointer(named)), true}, + {"S's *S implements I, but S is a value", v("a", named), v("b", iface), false}, + {"*S does not implement J", v("a", types.NewPointer(named)), v("b", other), false}, + {"global and local pointer", global, v("b", types.NewPointer(named)), true}, + {"nil base", dataflow.Path{Spell: "u.f"}, v("b", types.NewPointer(named)), false}, + {"pseudo path", dataflow.Path{Spell: ":9:9"}, v("b", types.NewPointer(named)), false}, + } { + if got := o.MayAlias(c.a, c.b); got != c.want { + t.Errorf("%s: MayAlias = %v, want %v", c.what, got, c.want) + } + } +} + +// TestAlias_AddsOnly: the alias pass only adds. With the oracle off vs on, +// Run's Result is identical except that (rulings, Task 17) the on run's +// callable ddg holds the points-to edges besides the off run's edges, which +// it keeps unchanged, and its summary edges are a superset (the flows read +// the points-to edges, spec §4.9). No points-to edge repeats the (src, dst, +// var) of an L3 or reaching-defs edge, every endpoint is a body node or a +// vertex of the callable, no summary edge is global → global (#30), and +// skipped units get none. +func TestAlias_AddsOnly(t *testing.T) { + in := loadInput(t, "dataflow") + defer func(v AliasOracle) { aliasOracle = v }(aliasOracle) + aliasOracle = nil + off := Run(in) + aliasOracle = typeOracle{} + on := Run(loadInput(t, "dataflow")) + + var nPT, nSumm int + for k, c := range on.Callables { + o := off.Callables[k] + var rest []PortEdge + for _, e := range c.Ddg { + if e.Prov != ProvPointsTo { + rest = append(rest, e) + } + } + if !slices.Equal(rest, o.Ddg) { + t.Errorf("%s: non-points-to ddg changed:\n off %+v\n on %+v", k.Signature, o.Ddg, rest) + } + for _, e := range o.Summary { + if !slices.Contains(c.Summary, e) { + t.Errorf("%s: summary edge %+v lost", k.Signature, e) + } + } + nSumm += len(c.Summary) - len(o.Summary) + if !reflect.DeepEqual(c.Vertices, o.Vertices) { + t.Errorf("%s: vertices changed", k.Signature) + } + of := map[string]string{} + for _, v := range c.Vertices { + of[v.Key] = v.Of + } + for _, e := range c.Summary { + if strings.HasPrefix(of[e.Src], ofGlobal) && strings.HasPrefix(of[e.Dst], ofGlobal) { + t.Errorf("%s: global → global summary %+v", k.Signature, e) + } + } + ends := map[string]bool{} + l3 := map[dataflow.DdgEdge]bool{} + if r, ok := in.L3[k]; ok { + for _, n := range r.Nodes { + ends[n.Key] = true + } + for _, e := range r.Ddg { + l3[e] = true + } + } + for _, v := range c.Vertices { + ends[v.Key] = true + } + rd := map[dataflow.DdgEdge]bool{} + for _, e := range rest { + rd[dataflow.DdgEdge{Src: e.Src, Dst: e.Dst, Var: e.Var}] = true + } + for _, e := range pointsTo(c.Ddg) { + nPT++ + t3 := dataflow.DdgEdge{Src: e.Src, Dst: e.Dst, Var: e.Var} + if l3[t3] || rd[t3] { + t.Errorf("%s: points-to %+v repeats an L3 or reaching-defs edge", k.Signature, e) + } + if !ends[e.Src] || !ends[e.Dst] || e.Dst == dataflow.ExitKey || e.Src == dataflow.EntryKey { + t.Errorf("%s: points-to %+v dangles or touches @entry/@exit", k.Signature, e) + } + } + for i := 1; i < len(c.Ddg); i++ { + if comparePortEdge(c.Ddg[i-1], c.Ddg[i]) >= 0 { + t.Errorf("%s: ddg not sorted/unique at %+v", k.Signature, c.Ddg[i]) + } + } + } + if nPT == 0 || nSumm == 0 { + t.Fatalf("alias pass added %d points-to and %d summary edges: the gate covers nothing", nPT, nSumm) + } + t.Logf("dataflow: %d points-to edges, %d summary edges added", nPT, nSumm) + + off.Callables, on.Callables = nil, nil + if !reflect.DeepEqual(off, on) { + t.Errorf("Run's result outside Callables differs with the alias pass on") + } +} + +// TestAlias_SummaryThroughAlias: aliasAssign's v reaches its p only through +// the alias q (q := p; q.n = v), so its caller CallAliasAssign gains the +// summary edge from v's actual_in to p's actual_out with the pass on, and +// lacks it off. +func TestAlias_SummaryThroughAlias(t *testing.T) { + want := SummaryEdge{Src: "382:2/actual_in:1", Dst: "382:2/actual_out"} + if got := runAlias(t, typeOracle{}).Callables[sdgKey("CallAliasAssign")].Summary; !slices.Contains(got, want) { + t.Errorf("CallAliasAssign summary %+v lacks %+v", got, want) + } + if got := runAlias(t, nil).Callables[sdgKey("CallAliasAssign")].Summary; slices.Contains(got, want) { + t.Errorf("CallAliasAssign summary %+v has %+v without the alias pass", got, want) + } +} + +// TestAlias_SkippedAndNoVTA: an l3_skipped unit gets no points-to edge (its +// pass-through wiring is already all-to-all), and under no_vta the pass +// still runs (it is intraprocedural; it reads no SSA). +func TestAlias_SkippedAndNoVTA(t *testing.T) { + in := loadInput(t, "dataflow") + delete(in.L3, sdgKey("AliasPtr")) + res := Run(in) + if got := pointsTo(res.Callables[sdgKey("AliasPtr")].Ddg); len(got) != 0 { + t.Errorf("skipped AliasPtr has points-to edges %+v", got) + } + + in = loadInput(t, "dataflow") + defer func(f func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph) { vtaCallGraph = f }(vtaCallGraph) + vtaCallGraph = func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph { panic("vta boom") } + res = Run(in) + if len(res.Degraded) == 0 || res.Degraded[len(res.Degraded)-1].Code != CodeNoVTA { + t.Fatalf("degraded = %+v, want no_vta", res.Degraded) + } + if got := pointsTo(res.Callables[sdgKey("AliasPtr")].Ddg); !slices.Contains(got, pt("360:2", "361:2", "req.UserID")) { + t.Errorf("no_vta AliasPtr points-to edges %+v lack 360:2 → 361:2", got) + } +} diff --git a/internal/sdg/run.go b/internal/sdg/run.go index 70bf54a..51be8f7 100644 --- a/internal/sdg/run.go +++ b/internal/sdg/run.go @@ -121,12 +121,17 @@ func dispatchRun(in Input) (Result, *dispatch) { func overlay(in Input, d *dispatch, res *Result) { start := time.Now() newSDGBuilder(in, d).fill(res) - var nv, nd, ns int + var nv, nd, np, ns int for _, c := range res.Callables { nv, nd, ns = nv+len(c.Vertices), nd+len(c.Ddg), ns+len(c.Summary) + for _, e := range c.Ddg { + if e.Prov == ProvPointsTo { + np++ + } + } } - utils.Info("L4: %d vertices, %d port edges, %d summary edges, %d param_in, %d param_out, %d stand-ins in %s", - nv, nd, ns, len(res.ParamIn), len(res.ParamOut), len(res.Externals), time.Since(start).Round(time.Millisecond)) + utils.Info("L4: %d vertices, %d port edges (%d points-to), %d summary edges, %d param_in, %d param_out, %d stand-ins in %s", + nv, nd, np, ns, len(res.ParamIn), len(res.ParamOut), len(res.Externals), time.Since(start).Round(time.Millisecond)) } // funcSet is projectFuncs under recover: it builds functions on demand diff --git a/internal/sdg/wiring.go b/internal/sdg/wiring.go index 4d95c0d..d0682a0 100644 --- a/internal/sdg/wiring.go +++ b/internal/sdg/wiring.go @@ -42,11 +42,18 @@ const ProvReachingDefs = "reaching-defs" // An edge's var is the `of` of the vertex it touches, the formal_out's when // it ends at one; a body → body edge keeps the L3 path spelling. // +// The same re-solve, with aliasOracle consulted (dataflow.BuildDDGSplit), +// yields the points-to delta (spec §4.10): each alias-only edge, mapped by +// the rules above (port), prov "points-to", unless its (src, dst, var) +// already is an L3 or reaching-defs edge. It never removes or changes one: +// alias adds no kills. A skipped unit gets none (its wiring is already +// all-to-all). +// // A unit with no L3 result (l3_skipped) has no body nodes to wire: it is // wired pass-through (wireSkipped). // -// The edges are sorted by (Src, Dst, Var) and unique; none repeats an L3 -// edge. +// The edges are sorted by (Src, Dst, Var, Prov) and unique; none repeats an +// L3 edge, and no (src, dst, var) has both provs. func (p *ports) wire(k dataflow.L3Key) []PortEdge { u, ok := p.units[k] if !ok { @@ -70,29 +77,14 @@ func (p *ports) wire(k dataflow.L3Key) []PortEdge { } } - for _, e := range dataflow.BuildDDG(w.r.Nodes, w.r.Cfg, w.du) { - b := dataflow.SpellBase(e.Var) - switch { - case e.Dst == dataflow.ExitKey: - fo, ok := w.exitOut[e.Var] - if !ok { - continue - } - src := e.Src - if fi, ok := w.inOf(b); ok && src == dataflow.EntryKey { - src = fi - } - add(src, fo.Key, fo.Of) - case l3[e]: - if fi, ok := w.inOf(b); ok && e.Src == dataflow.EntryKey { - add(fi, e.Dst, e.Var) - } - default: - src := e.Src - if fi, ok := w.inOf(b); ok && src == dataflow.EntryKey { - src = fi - } - add(src, e.Dst, e.Var) + var mayAlias func(a, b dataflow.Path) bool + if aliasOracle != nil { + mayAlias = memoOracle(aliasOracle) + } + resolved, alias := dataflow.BuildDDGSplit(w.r.Nodes, w.r.Cfg, w.du, mayAlias) + for _, e := range resolved { + if src, dst, v, ok := w.port(e, l3[e]); ok { + add(src, dst, v) } } @@ -143,9 +135,52 @@ func (p *ports) wire(k dataflow.L3Key) []PortEdge { } } } + + // The points-to delta (spec §4.10): the alias-only edges of the same + // re-solve, mapped onto the vertices like the re-solved edges, minus any + // (src, dst, var) an L3 or reaching-defs edge already has. + for _, e := range alias { + src, dst, v, ok := w.port(e, false) + if !ok || l3[dataflow.DdgEdge{Src: src, Dst: dst, Var: v}] { + continue + } + if !out[PortEdge{Src: src, Dst: dst, Var: v, Prov: ProvReachingDefs}] { + out[PortEdge{Src: src, Dst: dst, Var: v, Prov: ProvPointsTo}] = true + } + } return sortedPortEdges(out) } +// port maps a re-solved ddg edge e of the unit onto its port edge (wire's +// rules), ok false when e yields none. inL3 says e is an L3 edge. +// +// - e → @exit on an output base: e.Src → its formal_out, var the +// formal_out's `of`; on any other base, nothing; +// - an L3 edge: only its formal_in mirror, when it leaves @entry on a base +// with a formal_in; else nothing (the L3 edge stays L3's); +// - any other edge: as is, from the base's formal_in instead of @entry. +func (w *wiring) port(e dataflow.DdgEdge, inL3 bool) (src, dst, v string, ok bool) { + b := dataflow.SpellBase(e.Var) + src = e.Src + fi, hasIn := w.inOf(b) + fromEntry := hasIn && src == dataflow.EntryKey + if fromEntry { + src = fi + } + switch { + case e.Dst == dataflow.ExitKey: + fo, ok := w.exitOut[e.Var] + if !ok { + return "", "", "", false + } + return src, fo.Key, fo.Of, true + case inL3: + return src, e.Dst, e.Var, fromEntry + default: + return src, e.Dst, e.Var, true + } +} + // resolve re-solves unit k's reaching definitions over the augmented DefUse // (newWiring): L3's ddg plus the A4 and @exit edges. nil without an L3 // result. diff --git a/internal/sdg/wiring_test.go b/internal/sdg/wiring_test.go index 03e8ab4..0d45281 100644 --- a/internal/sdg/wiring_test.go +++ b/internal/sdg/wiring_test.go @@ -185,8 +185,9 @@ func nodeOf(r dataflow.Result, key string) dataflow.Node { // TestWiring_L3EdgesUntouched: the augmented re-solve keeps every L3 ddg // edge (weak defs kill nothing), no port edge repeats an L3 edge, none ends -// at @exit, every prov is reaching-defs, every endpoint is a kept body node -// or a vertex of the callable, and the output is sorted and unique. +// at @exit, every prov is reaching-defs or (the alias delta, Task 17) +// points-to, every endpoint is a kept body node or a vertex of the callable, +// and the output is sorted and unique. func TestWiring_L3EdgesUntouched(t *testing.T) { in, _, p := runPorts(t) n := 0 @@ -216,7 +217,7 @@ func TestWiring_L3EdgesUntouched(t *testing.T) { if l3[dataflow.DdgEdge{Src: e.Src, Dst: e.Dst, Var: e.Var}] { t.Errorf("%s: port edge %+v repeats an L3 edge", u.Key.Signature, e) } - if e.Dst == dataflow.ExitKey || e.Prov != ProvReachingDefs || !ends[e.Src] || !ends[e.Dst] { + if e.Dst == dataflow.ExitKey || e.Prov != ProvReachingDefs && e.Prov != ProvPointsTo || !ends[e.Src] || !ends[e.Dst] { t.Errorf("%s: bad port edge %+v", u.Key.Signature, e) } if i > 0 && comparePortEdge(got[i-1], e) >= 0 { diff --git a/internal/syntactic_analysis/v2emit/gate_l4_test.go b/internal/syntactic_analysis/v2emit/gate_l4_test.go index 191effd..f090ad5 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_test.go @@ -839,3 +839,28 @@ func TestEmitL4_UnresolvedRefIsDroppedAndWarned(t *testing.T) { t.Errorf("no warning names the unresolved ref; warned %q", warned) } } + +// TestGateL4_PointsToEdges (spec §4.10): AliasPtr's write through the copy +// p reaches the read through req, and req's formal_out, as ddg edges with +// prov ["points-to"], distinct from the reaching-defs edges on the same +// endpoints. +func TestGateL4_PointsToEdges(t *testing.T) { + fx := buildL4(t, "dataflow") + mod := v2.ModuleID(fx.l4.Application.ID, "sdg/sdg.go") + id := v2.CallableID(mod, "example.com/dataflow/sdg.AliasPtr") + var got [][3]string + for _, c := range l4Callables(fx.l4) { + if c.ID != id { + continue + } + for _, e := range c.Ddg { + if slices.Equal(e.Prov, []string{"points-to"}) { + got = append(got, [3]string{e.Src, e.Dst, e.Var}) + } + } + } + want := [][3]string{{"360:2", "361:2", "req.UserID"}, {"360:2", "@formal_out:1", "req"}} + if !slices.Equal(got, want) { + t.Errorf("AliasPtr points-to ddg = %v, want %v", got, want) + } +} diff --git a/testdata/dataflow/sdg/sdg.go b/testdata/dataflow/sdg/sdg.go index 0e98e69..78a26a4 100644 --- a/testdata/dataflow/sdg/sdg.go +++ b/testdata/dataflow/sdg/sdg.go @@ -347,3 +347,37 @@ func implicit(x int) int { } return 0 } + +// Req is a request whose field a pointer copy writes (Task 17, §4.10). +type Req struct { + UserID string +} + +// AliasPtr writes req.UserID through the pointer copy p: the write reaches +// the return's read of req.UserID only through the may-alias oracle. +func AliasPtr(req *Req, id string) string { + p := req + p.UserID = id + return req.UserID +} + +// AliasPtrWhole is AliasPtr returning the whole pointer: the bare use req +// is prefix-compatible with the aliased write p.UserID. +func AliasPtrWhole(req *Req, id string) *Req { + p := req + p.UserID = id + return req +} + +// AliasValue copies req by value: the write to the copy never reaches req. +func AliasValue(req Req, id string) string { + p := req + p.UserID = id + return req.UserID +} + +// CallAliasAssign calls aliasAssign, whose v reaches its p only through the +// alias q: the call has a summary edge from v's actual_in to p's actual_out. +func CallAliasAssign(p *Inner, v int) { + aliasAssign(p, v) +} From a2be9a45ce6ddeacdddd4daaec978cf2251859cf Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 21:41:13 -0400 Subject: [PATCH 24/65] fix(l4): may-alias compares pointee types, aliases interfaces and address-taken values --- docs/design/specs/v2-l3-l4-dataflow.md | 41 ++ internal/dataflow/reachdefs.go | 138 ++++--- internal/dataflow/reachdefs_test.go | 93 +++-- internal/sdg/alias.go | 439 ++++++++++++++++++---- internal/sdg/alias_test.go | 194 +++++++--- internal/sdg/testdata/wire_sdg_off.golden | 395 +++++++++++++++++++ internal/sdg/vertices.go | 3 + internal/sdg/wiring.go | 15 +- testdata/dataflow/sdg/sdg.go | 65 ++++ 9 files changed, 1207 insertions(+), 176 deletions(-) create mode 100644 internal/sdg/testdata/wire_sdg_off.golden diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index 843b3da..7819269 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -635,6 +635,47 @@ oracle"); `go/pointer` (deprecated). The oracle's API is frozen as an interface so an Andersen implementation can replace it later without schema change (`prov` stays `points-to`). +**As built (Task 17 and its fix round 1) [call].** `AliasOracle.MayAlias(d, u)` asks whether a write +to `d` may change memory a read of `u` observes. The oracle is built per callable +(`newAliasOracle(unit, root)`). The solver re-runs the same reaching-definitions fixpoint over the +port wiring's augmented `du'`. It never adds a kill across bases. It consults the oracle only for a +def and a use of different bases inside a **one-step offset window**: after at most one leading step +is dropped from each, the suffixes are prefix-compatible. +- **Anchors.** The oracle matches each path at offset 0 (its base) or offset 1 (its base and first + step). The anchored prefix is one of two things: + - a *reference*: the object it points to. A bare def of a reference rebinds it and writes + nothing that another base observes. + - an *address-taken value*: its own storage. +- **Compatible references.** + - Pointee types are compared structurally on the underlying types, with `types.Identical` on the + element types. This covers pointers, slices, maps (key and element), chans of either direction, + and identical signatures. So `type Bytes []byte` aliases `[]byte`, and `http.Header` aliases + `map[string][]string`. + - Any two interface (or type-parameter) types are compatible. + - An interface is compatible with a type that implements it. +- **Address-taken values.** A value x is address-taken when the callable's top-level declaration + (closures included) contains any of: + - `&x`, `&x.f…` or `&x[i]…`; + - `x[:]` or `x[i:j]` on an array value; + - a call or method value of a pointer-receiver method on x or `x.f…`. + + Package-level variables are always address-taken. +- **Address-taken matching.** An address-taken value of type T may alias: + - a `*T`; + - an interface that `*T` implements; + - for an array `[N]E`, a `[]E`. + + A field whose address is taken, or a field of reference type, matches at offset 1. So `p := &o.f; + p.g = v` reaches `o.f.g`, and `m := s.cache; m[k] = v` reaches `s.cache[k]`. +- **Never alias.** Two distinct value variables never share storage. Neither do value copies that are + not address-taken. +- **Known gaps (false negatives).** + - Offsets deeper than one step (`&x.f.g` against a `*G`). + - Two prefixes of one base that alias each other. + - An interface holding a value struct whose reference field is written through the interface. + + The Andersen upgrade closes them. + ### 4.11 Neo4j projection [call] Fill the reserved overlay, no contract renames: diff --git a/internal/dataflow/reachdefs.go b/internal/dataflow/reachdefs.go index 64d663c..7744c87 100644 --- a/internal/dataflow/reachdefs.go +++ b/internal/dataflow/reachdefs.go @@ -50,16 +50,16 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { // BuildDDGAlias is BuildDDG with a may-alias oracle (spec §4.10): the same // reaching-definitions fixpoint, kills and use matching, plus one rule. A // use u at n also links from a def d in IN(n) whose base differs from u's -// when +// when mayAlias(d, u) holds: a write to d may change the memory a read of u +// observes. The oracle decides on the whole paths (which of their prefixes +// may share memory, and whether d writes through a reference or only +// rebinds it), so it is asymmetric: d is the write, u the read. // -// - d's path has a non-empty suffix (a step after its base): a bare def -// `p = …` rebinds the variable and writes no memory another base can -// reach (ruling, Task 17), so only a write through a base aliases. Hence -// no alias edge leaves @entry, whose defs are all bare. The use side is -// unrestricted: a bare use (`return req`) reads the whole pointee; -// - the suffixes are prefix-compatible, in Covers' two-directional sense -// with the bases ignored (d.f covers u.f.g, u.* covers d.f); and -// - mayAlias(d.path, u) holds. +// The solver consults it only within the one-step offset window: for some +// i, j ∈ {0, 1}, d's steps after its first i and u's after its first j are +// prefix-compatible, in Covers' two-directional sense with the bases +// ignored (d.f covers u.f.g; u.* covers d.f; p.g against x.f.g at j = 1). +// An oracle accepting a pair outside the window is not asked about it. // // Alias never kills: a strong def through one base kills only the defs of // its own base (BuildDDG's KILL), so a def that may be overwritten through @@ -71,24 +71,38 @@ func BuildDDGAlias(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias f if len(alias) == 0 { return ddg } - out := append(ddg, alias...) - sortDdg(out) - return out + set := make(map[DdgEdge]bool, len(ddg)+len(alias)) + for _, e := range ddg { + set[e] = true + } + for _, e := range alias { + set[e.DdgEdge] = true + } + return sortedDdg(set) +} + +// AliasEdge is an edge only BuildDDGAlias's alias rule yields. Def is the +// base spelling of the def it links from when the edge leaves @entry (which +// defines several bases; the L4 port wiring starts the edge at that base's +// formal_in), else "". +type AliasEdge struct { + DdgEdge + Def string } -// BuildDDGSplit is BuildDDGAlias split in two: BuildDDG's edges, and the -// edges only the alias rule yields (none of which is in the first list), -// each sorted by (Src, Dst, Var). One fixpoint serves both: the L4 port -// wiring (package sdg) maps the first to reaching-defs port edges and the -// second to its points-to delta. -func BuildDDGSplit(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias func(a, b Path) bool) (ddg, alias []DdgEdge) { +// BuildDDGSplit is BuildDDGAlias split in two: BuildDDG's edges, sorted by +// (Src, Dst, Var), and the edges only the alias rule yields, none of whose +// DdgEdge is in the first list, sorted by (Src, Dst, Var, Def). One fixpoint +// serves both: the L4 port wiring (package sdg) maps the first to +// reaching-defs port edges and the second to its points-to delta. +func BuildDDGSplit(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias func(a, b Path) bool) (ddg []DdgEdge, alias []AliasEdge) { return solveDDG(nodes, cfg, du, mayAlias) } // solveDDG is the fixpoint and use matching of BuildDDG and BuildDDGAlias: // the BuildDDG edges, and (mayAlias non-nil) the alias-only edges, both // sorted, the first non-nil. -func solveDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias func(a, b Path) bool) (ddg, alias []DdgEdge) { +func solveDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias func(a, b Path) bool) (ddg []DdgEdge, alias []AliasEdge) { g := newPdomGraph(nodes, cfg) n := len(g.keys) @@ -219,17 +233,18 @@ func solveDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias func(a return sortedDdg(edges), alias } -// aliasEdges is BuildDDGAlias's alias rule over a solved fixpoint: for each -// node with uses, each reaching def with a suffix is matched against each -// use of a different base. Edges already in base are skipped. The result is -// sorted (nil when empty). -func aliasEdges(keys []string, du map[string]DefUse, in [][]uint64, pathOf func(int) Path, nodeOf func(int) string, base map[DdgEdge]bool, mayAlias func(a, b Path) bool) []DdgEdge { +// aliasEdges is BuildDDGAlias's alias rule over a solved fixpoint: at each +// node with uses, each reaching def is matched against each use of a +// different base within the offset window (inWindow), and the oracle +// decides. Edges already in base are skipped. The result is sorted (nil +// when empty). +func aliasEdges(keys []string, du map[string]DefUse, in [][]uint64, pathOf func(int) Path, nodeOf func(int) string, base map[DdgEdge]bool, mayAlias func(a, b Path) bool) []AliasEdge { type use struct { path Path steps []string } steps := map[int][]string{} // def index → its steps; absent: not yet split - edges := map[DdgEdge]bool{} + edges := map[AliasEdge]bool{} for v, k := range keys { var uses []use for _, u := range du[k].Uses { @@ -249,16 +264,17 @@ func aliasEdges(keys []string, du map[string]DefUse, in [][]uint64, pathOf func( _, ds = splitSpell(pathOf(i).Spell) steps[i] = ds } - if len(ds) == 0 { - continue // a bare def writes no aliased memory - } d := pathOf(i) + src := nodeOf(i) + def := "" + if src == EntryKey { + def = SpellBase(d.Spell) + } for _, u := range uses { - if u.path.Base == d.Base || !(stepsCover(ds, u.steps) || stepsCover(u.steps, ds)) { + if u.path.Base == d.Base || !inWindow(ds, u.steps) || !mayAlias(d, u.path) { continue } - e := DdgEdge{Src: nodeOf(i), Dst: k, Var: u.path.Spell} - if !base[e] && !edges[e] && mayAlias(d, u.path) { + if e := (AliasEdge{DdgEdge{Src: src, Dst: k, Var: u.path.Spell}, def}); !base[e.DdgEdge] { edges[e] = true } } @@ -268,7 +284,54 @@ func aliasEdges(keys []string, du map[string]DefUse, in [][]uint64, pathOf func( if len(edges) == 0 { return nil } - return sortedDdg(edges) + res := make([]AliasEdge, 0, len(edges)) + for e := range edges { + res = append(res, e) + } + sort.Slice(res, func(i, j int) bool { + a, b := res[i], res[j] + if a.DdgEdge != b.DdgEdge { + return ddgLess(a.DdgEdge, b.DdgEdge) + } + return a.Def < b.Def + }) + return res +} + +// inWindow reports whether two step lists are prefix-compatible after +// dropping at most one leading step from each (BuildDDGAlias's window). +func inWindow(ds, us []string) bool { + for i := 0; i <= min(1, len(ds)); i++ { + for j := 0; j <= min(1, len(us)); j++ { + if StepsCompatible(ds[i:], us[j:]) { + return true + } + } + } + return false +} + +// StepsCompatible reports whether two step lists (the steps of two paths +// after their bases, or after a common offset) are prefix-compatible: one +// covers the other (Covers with the bases ignored). The alias oracle +// compares the steps below two may-alias prefixes with it. +func StepsCompatible(a, b []string) bool { return stepsCover(a, b) || stepsCover(b, a) } + +// PathSteps returns the steps of a path spelling after its base ("x.f[*]" +// → [".f", "[*]"]); a pseudo path has none. +func PathSteps(spell string) []string { + _, s := splitSpell(spell) + return s +} + +func ddgLess(a, b DdgEdge) bool { + if a.Src != b.Src { + return a.Src < b.Src + } + if a.Dst != b.Dst { + return a.Dst < b.Dst + } + return a.Var < b.Var } func sortedDdg(edges map[DdgEdge]bool) []DdgEdge { @@ -281,16 +344,7 @@ func sortedDdg(edges map[DdgEdge]bool) []DdgEdge { } func sortDdg(res []DdgEdge) { - sort.Slice(res, func(i, j int) bool { - a, b := res[i], res[j] - if a.Src != b.Src { - return a.Src < b.Src - } - if a.Dst != b.Dst { - return a.Dst < b.Dst - } - return a.Var < b.Var - }) + sort.Slice(res, func(i, j int) bool { return ddgLess(res[i], res[j]) }) } // killBucket is the kill index key of a path: its Base, or for a nil Base diff --git a/internal/dataflow/reachdefs_test.go b/internal/dataflow/reachdefs_test.go index cbc28ec..a8fd9f9 100644 --- a/internal/dataflow/reachdefs_test.go +++ b/internal/dataflow/reachdefs_test.go @@ -616,24 +616,22 @@ func anyBase(a, b dataflow.Path) bool { return a.Base != nil && b.Base != nil } // flatDDGAlias is BuildDDGAlias's reference, formulated independently: over // flatReaching, a use u also links from a reaching def d of another base -// whose spelling has a suffix when d's suffix, rebased onto u's base, -// interferes with u (Covers either way) and mayAlias(d, u). nil mayAlias is -// flatDDG. +// when mayAlias(d, u) and, dropping at most one leading step from each, d's +// remaining steps rebased onto u's base interfere with u's remaining steps +// rebased likewise (Covers either way). nil mayAlias is flatDDG. func flatDDGAlias(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]dataflow.DefUse, mayAlias func(a, b dataflow.Path) bool) []dataflow.DdgEdge { edges := map[dataflow.DdgEdge]bool{} for k, ins := range flatReaching(nodes, cfg, du) { for d := range ins { - suffix := d.Path.Spell[len(dataflow.SpellBase(d.Path.Spell)):] for _, u := range du[k].Uses { if d.Path.Covers(u) || u.Covers(d.Path) { edges[dataflow.DdgEdge{Src: d.Node, Dst: k, Var: u.Spell}] = true continue } - if mayAlias == nil || suffix == "" || u.Base == d.Path.Base { + if mayAlias == nil || d.Path.Base == nil || u.Base == d.Path.Base { continue } - r := dataflow.Path{Base: u.Base, Spell: dataflow.SpellBase(u.Spell) + suffix} - if (r.Covers(u) || u.Covers(r)) && mayAlias(d.Path, u) { + if flatWindow(d.Path, u) && mayAlias(d.Path, u) { edges[dataflow.DdgEdge{Src: d.Node, Dst: k, Var: u.Spell}] = true } } @@ -647,6 +645,30 @@ func flatDDGAlias(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]d return res } +// flatWindow is the one-step offset window on spellings: both suffixes, +// each with up to one leading step dropped, rebased onto u's base. +func flatWindow(d, u dataflow.Path) bool { + tails := func(p dataflow.Path) []string { + suf := p.Spell[len(dataflow.SpellBase(p.Spell)):] + out := []string{suf} + if steps := dataflow.PathSteps(p.Spell); len(steps) > 0 { + out = append(out, suf[len(steps[0]):]) + } + return out + } + head := dataflow.SpellBase(u.Spell) + for _, ds := range tails(d) { + for _, us := range tails(u) { + a := dataflow.Path{Base: u.Base, Spell: head + ds} + b := dataflow.Path{Base: u.Base, Spell: head + us} + if a.Covers(b) || b.Covers(a) { + return true + } + } + } + return false +} + // TestDDG_KillBucketsMatchFlat (#27 item 1): BuildDDG tests kills only within // the bucket of a def's base (nil-base paths by spelling head); it equals the // flat reference over the whole flow and chi corpus and over a hand-built @@ -735,21 +757,26 @@ func TestDDG_KillBucketsMatchFlat(t *testing.T) { } // TestDDGAlias_Rule pins BuildDDGAlias on a hand-built line, with p and q -// pointers of one type (may alias) and v an int: +// pointers of one type and v an int, under an oracle that, like the L4 +// typeOracle, lets only a write through a pointer (a def with a step) alias: // // @entry defines p, q, v // 1:1 p.f = v -// 1:2 q.f = v (a strong def through q: kills q.f… only) -// 1:3 use p.f, q.f.g, q, p.g -// 1:4 q = nil (a bare def: rebinds q, writes no aliased memory) +// 1:2 q.f = v; q.g.h = v (strong defs through q: kill q.f…, q.g.h… only) +// 1:3 use p.f, q.f.g, q, p.g, p.h.f.k, p.k.m +// 1:4 q = nil (a bare def: rebinds q, writes no aliased memory) // 1:5 use p.f // -// 1:1's p.f reaches 1:2's bare use q (the pointer q.f = v reads). At 1:3, -// by alias: 1:2's q.f may have written p.f, yet 1:1's p.f still -// reaches by its own base (alias never kills); 1:1's p.f reaches q.f.g (its -// suffix covers) and the bare use q (a bare use reads the whole pointee); -// p.g's suffix is incompatible with q.f. At 1:5 nothing aliases: 1:4 killed -// q.f by q's own base, and its bare def of q aliases nothing, like @entry's. +// 1:1's p.f reaches 1:2's bare use q. At 1:3, by alias: 1:2's q.f may have +// written p.f, yet 1:1's p.f still reaches by its own base (alias never +// kills); 1:1's p.f reaches q.f.g (its suffix covers) and the bare use q. +// The one-step offset window admits p.g and p.k.m against q.f (one leading +// step dropped from each leaves [] against [] or [.m]: q.f's storage may be +// what p.k points to), and p.h.f.k against q.f (.h dropped); the oracle +// decides them (yes). p.k.m against q.g.h is outside the window ([.h] +// against [.m] at best): the oracle is never asked about that pair. At 1:5 nothing aliases: 1:4 +// killed q's defs by its own base, and its bare def of q aliases nothing, +// like @entry's. func TestDDGAlias_Rule(t *testing.T) { ptr := types.NewPointer(types.NewStruct(nil, nil)) p := types.NewVar(token.NoPos, nil, "p", ptr) @@ -768,26 +795,46 @@ func TestDDGAlias_Rule(t *testing.T) { du := map[string]dataflow.DefUse{ entry: {Defs: []dataflow.Path{P(p, "p"), P(q, "q"), P(v, "v")}}, "1:1": {Defs: []dataflow.Path{P(p, "p.f")}, Uses: []dataflow.Path{P(p, "p"), P(v, "v")}}, - "1:2": {Defs: []dataflow.Path{P(q, "q.f")}, Uses: []dataflow.Path{P(q, "q"), P(v, "v")}}, - "1:3": {Uses: []dataflow.Path{P(p, "p.f"), P(q, "q.f.g"), P(q, "q"), P(p, "p.g")}}, + "1:2": {Defs: []dataflow.Path{P(q, "q.f"), P(q, "q.g.h")}, Uses: []dataflow.Path{P(q, "q"), P(v, "v")}}, + "1:3": {Uses: []dataflow.Path{P(p, "p.f"), P(q, "q.f.g"), P(q, "q"), P(p, "p.g"), P(p, "p.h.f.k"), P(p, "p.k.m")}}, "1:4": {Defs: []dataflow.Path{P(q, "q")}}, "1:5": {Uses: []dataflow.Path{P(p, "p.f")}}, } - mayAlias := func(a, b dataflow.Path) bool { return a.Base != v && b.Base != v && a.Base != nil && b.Base != nil } + asked := map[[2]string]bool{} + mayAlias := func(a, b dataflow.Path) bool { + asked[[2]string{a.Spell, b.Spell}] = true + return a.Base != v && b.Base != v && a.Base != nil && b.Base != nil && len(dataflow.PathSteps(a.Spell)) > 0 + } base := dataflow.BuildDDG(nodes, cfg, du) got := dataflow.BuildDDGAlias(nodes, cfg, du, mayAlias) ddg, alias := dataflow.BuildDDGSplit(nodes, cfg, du, mayAlias) if !reflect.DeepEqual(ddg, base) { t.Errorf("split ddg != BuildDDG:\n got %v\nwant %v", ddg, base) } + var plain []dataflow.DdgEdge + for _, e := range alias { + if e.Def != "" { + t.Errorf("%v: Def set off @entry", e) + } + plain = append(plain, e.DdgEdge) + } want := []dataflow.DdgEdge{ - dd("1:1", "1:2", "q"), // the bare use q at 1:2, as p covers p.f on its own base + dd("1:1", "1:2", "q"), // the bare use q at 1:2 reads the object p.f is in dd("1:1", "1:3", "q"), dd("1:1", "1:3", "q.f.g"), dd("1:2", "1:3", "p.f"), + dd("1:2", "1:3", "p.g"), + dd("1:2", "1:3", "p.h.f.k"), + dd("1:2", "1:3", "p.k.m"), + } + if !reflect.DeepEqual(plain, want) { + t.Errorf("alias edges:\n got %v\nwant %v", plain, want) + } + if asked[[2]string{"q.g.h", "p.k.m"}] { + t.Errorf("oracle asked about q.g.h, p.k.m outside the window") } - if !reflect.DeepEqual(alias, want) { - t.Errorf("alias edges:\n got %v\nwant %v", alias, want) + if !asked[[2]string{"q.f", "p.k.m"}] { + t.Errorf("oracle not asked about q.f, p.k.m inside the window") } for _, e := range base { if !slices.Contains(got, e) { diff --git a/internal/sdg/alias.go b/internal/sdg/alias.go index 89d5d4f..c7fe5cb 100644 --- a/internal/sdg/alias.go +++ b/internal/sdg/alias.go @@ -1,6 +1,8 @@ package sdg import ( + "go/ast" + "go/token" "go/types" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" @@ -11,98 +13,409 @@ import ( const ProvPointsTo = "points-to" // AliasOracle answers may-alias queries between two access paths of one -// callable (spec §4.10). The interface is frozen so that a points-to -// analysis (an Andersen solve over go/ssa) can replace the type-based MVP -// without a schema change; prov stays "points-to". +// callable (spec §4.10): MayAlias(d, u) reports whether a write to d may +// change memory a read of u observes, d and u having different bases. The +// interface is frozen so that a points-to analysis (an Andersen solve over +// go/ssa) can replace the type-based MVP without a schema change; prov stays +// "points-to". // -// The DDG solver (dataflow.BuildDDGAlias) consults it only for a def and a -// use with different bases and prefix-compatible suffixes, the def's suffix -// non-empty, so MayAlias decides on the bases alone. +// The DDG solver (dataflow.BuildDDGAlias) consults it only within its +// one-step offset window, so an oracle need not accept pairs outside it. type AliasOracle interface { MayAlias(a, b dataflow.Path) bool } -// aliasOracle is the oracle the port wiring consults; nil turns the alias -// pass off (a test seam: TestAlias_AddsOnly compares Run off vs on). -var aliasOracle AliasOracle = typeOracle{} +// newAliasOracle builds the oracle of unit u; root is the top-level +// declaration enclosing u.Fn (u.Fn itself for a declaration, the outermost +// enclosing function for a closure), which typeOracle scans for taken +// addresses. nil turns the alias pass off. A test seam: TestAlias_AddsOnly +// compares Run with it off and on. +var newAliasOracle func(u dataflow.Unit, root ast.Node) AliasOracle = newTypeOracle -// typeOracle is the type-based may-alias MVP (spec §4.10). Two paths may -// alias iff +// typeOracle is the type-based may-alias MVP of one callable (spec §4.10, +// rulings of Task 17 and its fix round 1). // -// - both have a base object (a nil base, a ":" pseudo path or an -// unresolved identifier, never aliases anything), and -// - they share the base, or both bases' types may hold a reference -// (dataflow.MayHoldRef, the predicate the by-ref rules share) and either -// the types are identical or one is an interface the other's type -// implements (types.Implements on T or *T). +// A path is matched through its anchors: a prefix of it, at offset 0 (the +// base) or 1 (the base and its first step), that names one object, with the +// steps after the prefix (rem) addressing memory inside that object. An +// anchor is // -// A value type that holds no reference never aliases: its copy is its own -// memory. A package-level variable is a base like any other, so a global -// pointer may alias a local one of its type; so may a closure's capture. -// A type parameter is compared by identity only, never through its -// constraint's interface. -type typeOracle struct{} - -func (typeOracle) MayAlias(a, b dataflow.Path) bool { - if a.Base == nil || b.Base == nil { - return false +// - a reference anchor: the prefix has a reference type (pointer, slice, +// map, chan, func, interface or type parameter); the object is what it +// points to. A write reaches the object only through a step after the +// prefix: a bare def `p = …` (or `x.f = …` at offset 1) rebinds the +// reference and writes nothing another base observes; +// - a storage anchor: the prefix is an addressable value whose address may +// be taken (taken below); the object is its own storage, so a bare def +// `x = …` overwrites it. +// +// At offset 1 the first step is a field (".f") or element ("[*]") whose +// declared type decides the anchor: a reference-typed field or element is a +// reference anchor (it may share its target with another base: `m := +// s.cache; m[k] = v` reaches `s.cache[k]`); a field or element whose address +// is taken is a storage anchor. A path that ends at the base, or whose first +// step is the ".*" wildcard, stands for every first step, so every field and +// element of the base's type is tried. +// +// MayAlias(d, u) holds iff a writing anchor of d and an anchor of u may be +// the same object and their rems are prefix-compatible +// (dataflow.StepsCompatible). Two anchors may be the same object iff +// +// - both are references with compatible types (I1, I2): any two interface +// (or type-parameter) types; an interface and a type that implements it +// (a type parameter: that satisfies its constraint); else, on the +// underlying types, pointers with identical elements, slices with +// identical elements, maps with identical keys and elements, chans with +// identical elements whatever their directions, identical signatures; +// - a reference and a storage of type T (I3): a pointer to T, an +// interface *T implements, or (T an array [N]E) a slice []E; +// - never two storages: distinct variables never share storage. +// +// Taken addresses are scanned over root, closures included (an &x on a +// captured x may sit in the encloser): &x, &x.f…, &x[i]… and x[:], x[i:j] +// on an array value make x and (x, its first step) taken; so does a call or +// method value of a pointer-receiver method on an addressable value x or +// x.f… (the callee may keep the pointer). A package-level variable, and +// each of its fields and elements, is always taken: &G may sit in any +// function. +// +// A nil base (a ":" pseudo path, an unresolved identifier) never +// aliases. Globals are bases like any other; so are a closure's captures. +// +// Known gaps, all false negatives: offsets deeper than one step (&x.f.g +// against a *G; q := x.f.g against x.f.g.h); two paths of one base whose +// prefixes alias (x.f and x.g pointing to one object: the solver only asks +// about different bases); and an interface whose dynamic value is a value +// struct holding a reference (i.* written through the interface against the +// struct's reference field). +type typeOracle struct { + taken map[types.Object]bool + takenSub map[subKey]bool + anchors map[dataflow.Path][]anchor + memo map[[2]dataflow.Path]bool +} + +type subKey struct { + base types.Object + step string +} + +type anchor struct { + ref bool // a reference anchor; else a storage anchor + t types.Type // the reference's type, or the storage's + rem []string // the steps after the prefix (nil: every one) + write bool // a def of the path writes inside the object +} + +func newTypeOracle(u dataflow.Unit, root ast.Node) AliasOracle { + o := &typeOracle{ + taken: map[types.Object]bool{}, + takenSub: map[subKey]bool{}, + anchors: map[dataflow.Path][]anchor{}, + memo: map[[2]dataflow.Path]bool{}, + } + if root != nil && u.Info != nil { + o.scan(root, u.Info) } - if a.Base == b.Base { + return o +} + +// scan records the addresses root takes (typeOracle's doc). +func (o *typeOracle) scan(root ast.Node, info *types.Info) { + ast.Inspect(root, func(n ast.Node) bool { + switch e := n.(type) { + case *ast.UnaryExpr: + if e.Op == token.AND { + o.take(e.X, info) + } + case *ast.SliceExpr: + if isArray(info.TypeOf(e.X)) { + o.take(e.X, info) + } + case *ast.SelectorExpr: + if ptrMethodOnValue(e, info) { + o.take(e.X, info) + } + } return true + }) +} + +// take records x (an addressed operand) and its base's first step. +func (o *typeOracle) take(x ast.Expr, info *types.Info) { + p, ok := dataflow.PathOf(x, info, 2) + if !ok || p.Base == nil { + return + } + o.taken[p.Base] = true + if s := dataflow.PathSteps(p.Spell); len(s) > 0 && s[0] != ".*" { + o.takenSub[subKey{p.Base, s[0]}] = true } - return typesMayAlias(a.Base.Type(), b.Base.Type()) } -func typesMayAlias(ta, tb types.Type) bool { - if !dataflow.MayHoldRef(ta) || !dataflow.MayHoldRef(tb) { +// ptrMethodOnValue reports whether sel is a pointer-receiver method (call or +// value) selected on a non-pointer, non-interface operand: Go takes its +// address implicitly. +func ptrMethodOnValue(sel *ast.SelectorExpr, info *types.Info) bool { + s := info.Selections[sel] + if s == nil || s.Kind() != types.MethodVal || s.Indirect() { return false } - if types.Identical(ta, tb) { - return true + sig, ok := s.Obj().Type().(*types.Signature) + if !ok || sig.Recv() == nil { + return false + } + if _, ptr := sig.Recv().Type().Underlying().(*types.Pointer); !ptr { + return false + } + t := info.TypeOf(sel.X) + if t == nil || types.IsInterface(t) { + return false } - return implements(ta, tb) || implements(tb, ta) + _, ptr := t.Underlying().(*types.Pointer) + return !ptr } -// implements reports whether iface is an interface type (not a type -// parameter) that t or *t implements. -func implements(t, iface types.Type) bool { - if _, tp := iface.(*types.TypeParam); tp { +func isArray(t types.Type) bool { + if t == nil { return false } - it, ok := iface.Underlying().(*types.Interface) - if !ok { + _, ok := t.Underlying().(*types.Array) + return ok +} + +func (o *typeOracle) MayAlias(d, u dataflow.Path) bool { + if d.Base == nil || u.Base == nil { return false } - if types.Implements(t, it) { + if d.Base == u.Base { return true } - if _, ptr := t.Underlying().(*types.Pointer); ptr || types.IsInterface(t) { - return false + k := [2]dataflow.Path{d, u} + if v, ok := o.memo[k]; ok { + return v } - return types.Implements(types.NewPointer(t), it) + v := o.mayAlias(d, u) + o.memo[k] = v + return v } -// memoOracle memoizes o per base pair for one unit's alias solve: the -// solver asks about the same two bases once per reaching def and use, and -// types.Implements walks method sets. It lives only for one wire call. -// Only typeOracle, which decides on the bases alone, is memoized; any other -// oracle is asked every time. -func memoOracle(o AliasOracle) func(a, b dataflow.Path) bool { - if _, ok := o.(typeOracle); !ok { - return o.MayAlias +func (o *typeOracle) mayAlias(d, u dataflow.Path) bool { + ua := o.anchorsOf(u) + for _, a := range o.anchorsOf(d) { + if !a.write { + continue + } + for _, b := range ua { + if sameObject(a, b) && dataflow.StepsCompatible(a.rem, b.rem) { + return true + } + } + } + return false +} + +// anchorsOf returns p's anchors (typeOracle's doc), memoized. +func (o *typeOracle) anchorsOf(p dataflow.Path) []anchor { + if as, ok := o.anchors[p]; ok { + return as } - type pair struct{ a, b types.Object } - memo := map[pair]bool{} - return func(a, b dataflow.Path) bool { - if a.Base == nil || b.Base == nil { - return o.MayAlias(a, b) + var as []anchor + bt := p.Base.Type() + steps := dataflow.PathSteps(p.Spell) + global := dataflow.IsPkgVar(p.Base) + baseRef := refType(bt) + switch { + case baseRef: + as = append(as, anchor{ref: true, t: bt, rem: steps, write: len(steps) > 0}) + case global || o.taken[p.Base]: + as = append(as, anchor{t: bt, rem: steps, write: true}) + } + var firsts []string + var rem []string + if len(steps) > 0 && steps[0] != ".*" { + firsts, rem = steps[:1], steps[1:] + } else { + firsts = childSteps(bt) // every first step; rem nil + } + for _, s := range firsts { + ft := childType(bt, s) + if ft == nil { + continue } - k := pair{a.Base, b.Base} - v, ok := memo[k] - if !ok { - v = o.MayAlias(a, b) - memo[k] = v + if refType(ft) { + // A write through the field's reference needs a step past it: one + // more listed step, or a wildcard standing for it. + w := len(steps) >= 2 || len(steps) == 1 && steps[0] == ".*" + as = append(as, anchor{ref: true, t: ft, rem: rem, write: w}) + } + if global || o.takenSub[subKey{p.Base, s}] { + // The field's own storage: written by any def reaching it — not by + // a bare def of a pointer base, which only rebinds the pointer. + w := len(steps) > 0 || !baseRef + as = append(as, anchor{t: ft, rem: rem, write: w}) + } + } + o.anchors[p] = as + return as +} + +// childSteps returns every first step a path on a base of type t may take: +// its struct fields (through a pointer) by name, and "[*]" for a container. +func childSteps(t types.Type) []string { + t = deref(t) + var out []string + switch u := t.Underlying().(type) { + case *types.Struct: + for i := 0; i < u.NumFields(); i++ { + out = append(out, "."+u.Field(i).Name()) + } + case *types.Slice, *types.Array, *types.Map, *types.Chan: + out = append(out, "[*]") + } + return out +} + +// childType is the declared type of step s below a base of type t (a field +// through a pointer, an element of a slice, array, array pointer, map or +// chan), nil when there is none. +func childType(t types.Type, s string) types.Type { + t = deref(t) + switch u := t.Underlying().(type) { + case *types.Struct: + for i := 0; i < u.NumFields(); i++ { + if "."+u.Field(i).Name() == s { + return u.Field(i).Type() + } + } + case *types.Slice: + if s == "[*]" { + return u.Elem() + } + case *types.Array: + if s == "[*]" { + return u.Elem() + } + case *types.Map: + if s == "[*]" { + return u.Elem() + } + case *types.Chan: + if s == "[*]" { + return u.Elem() + } + } + return nil +} + +func deref(t types.Type) types.Type { + if p, ok := t.Underlying().(*types.Pointer); ok { + return p.Elem() + } + return t +} + +// refType reports whether t is a reference kind: pointer, slice, map, chan, +// func, interface or type parameter. +func refType(t types.Type) bool { + switch t.Underlying().(type) { + case *types.Pointer, *types.Slice, *types.Map, *types.Chan, *types.Signature, *types.Interface: + return true + } + return false +} + +func sameObject(a, b anchor) bool { + switch { + case a.ref && b.ref: + return refsMayAlias(a.t, b.t) + case a.ref: + return refToStorage(a.t, b.t) + case b.ref: + return refToStorage(b.t, a.t) + } + return false +} + +// refsMayAlias reports whether two references of types a and b may point to +// one object (I1, I2: typeOracle's doc). +func refsMayAlias(a, b types.Type) bool { + ia, ib := types.IsInterface(a), types.IsInterface(b) + switch { + case ia && ib: + return true + case ia: + return implementsIface(b, a) + case ib: + return implementsIface(a, b) + } + switch ua := a.Underlying().(type) { + case *types.Pointer: + ub, ok := b.Underlying().(*types.Pointer) + return ok && types.Identical(ua.Elem(), ub.Elem()) + case *types.Slice: + ub, ok := b.Underlying().(*types.Slice) + return ok && types.Identical(ua.Elem(), ub.Elem()) + case *types.Map: + ub, ok := b.Underlying().(*types.Map) + return ok && types.Identical(ua.Key(), ub.Key()) && types.Identical(ua.Elem(), ub.Elem()) + case *types.Chan: + ub, ok := b.Underlying().(*types.Chan) + return ok && types.Identical(ua.Elem(), ub.Elem()) + case *types.Signature: + return types.Identical(ua, b.Underlying()) + } + return false +} + +// refToStorage reports whether a reference of type r may point to the +// storage of a value of type t (I3: typeOracle's doc). +func refToStorage(r, t types.Type) bool { + if types.IsInterface(r) { + return implementsIface(types.NewPointer(t), r) + } + switch ur := r.Underlying().(type) { + case *types.Pointer: + return types.Identical(ur.Elem(), t) + case *types.Slice: + at, ok := t.Underlying().(*types.Array) + return ok && types.Identical(ur.Elem(), at.Elem()) + } + return false +} + +// implementsIface reports whether a value of type t may be held by a value +// of interface type iface: t implements it, or (iface a type parameter) +// satisfies its constraint. +func implementsIface(t, iface types.Type) bool { + if tp, ok := iface.(*types.TypeParam); ok { + c, ok := tp.Constraint().Underlying().(*types.Interface) + return ok && types.Satisfies(t, c) + } + it, ok := iface.Underlying().(*types.Interface) + return ok && types.Implements(t, it) +} + +// rootOf returns the top-level declaration enclosing unit u's function: u.Fn +// for a declaration, else the outermost unit function of u's file whose +// extent contains u.Fn (a package-level literal is its own root). +func (p *ports) rootOf(u dataflow.Unit) ast.Node { + if _, ok := u.Fn.(*ast.FuncDecl); ok || u.Fn == nil { + return u.Fn + } + if p.fnsByPath == nil { + p.fnsByPath = map[string][]ast.Node{} + for _, v := range p.in.Units { + if v.Fn != nil { + p.fnsByPath[v.Key.Path] = append(p.fnsByPath[v.Key.Path], v.Fn) + } + } + } + root := u.Fn + for _, fn := range p.fnsByPath[u.Key.Path] { + if fn.Pos() <= root.Pos() && root.End() <= fn.End() && fn.Pos() < root.Pos() { + root = fn } - return v } + return root } diff --git a/internal/sdg/alias_test.go b/internal/sdg/alias_test.go index 893ca78..a8b34c6 100644 --- a/internal/sdg/alias_test.go +++ b/internal/sdg/alias_test.go @@ -1,8 +1,13 @@ package sdg import ( + "flag" + "fmt" + "go/ast" "go/token" "go/types" + "os" + "path/filepath" "reflect" "slices" "strings" @@ -14,6 +19,8 @@ import ( "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" ) +var update = flag.Bool("update", false, "rewrite golden files") + func pt(src, dst, v string) PortEdge { return PortEdge{Src: src, Dst: dst, Var: v, Prov: ProvPointsTo} } @@ -28,11 +35,13 @@ func pointsTo(es []PortEdge) []PortEdge { return out } -// runAlias runs L4 over testdata/dataflow with the alias oracle o (nil: off). -func runAlias(t *testing.T, o AliasOracle) Result { +// runAlias runs L4 over testdata/dataflow with the alias pass on or off. +func runAlias(t *testing.T, on bool) Result { t.Helper() - defer func(v AliasOracle) { aliasOracle = v }(aliasOracle) - aliasOracle = o + defer func(v func(dataflow.Unit, ast.Node) AliasOracle) { newAliasOracle = v }(newAliasOracle) + if !on { + newAliasOracle = nil + } return Run(loadInput(t, "dataflow")) } @@ -44,8 +53,10 @@ func runAlias(t *testing.T, o AliasOracle) Result { // // AliasPtrWhole returns the bare req (ruling, Task 17: the use side is // unrestricted): the write p.UserID is prefix-compatible with the whole read. +// The fix-round-1 fixtures (address-taken values, reference fields copied +// out, unrelated interfaces) are pinned whole too. func TestAlias_PointerCopyLinks(t *testing.T) { - res := runAlias(t, typeOracle{}) + res := runAlias(t, true) for _, c := range []struct { fn string want []PortEdge @@ -54,6 +65,17 @@ func TestAlias_PointerCopyLinks(t *testing.T) { {"AliasPtrWhole", []PortEdge{pt("368:2", "369:2", "req"), pt("368:2", "@formal_out:1", "req")}}, // aliasAssign (q := p; q.n = v): q.n = v reaches p's formal_out. {"aliasAssign", []PortEdge{pt("153:2", "@formal_out", "p")}}, + // Fix round 1, I3: address-taken values. The value's own def + // (`var req Req`, storage) reaches the use p of `p.UserID = id`: p + // points into it. + {"AliasPtrValue", []PortEdge{pt("388:2", "390:2", "p"), pt("390:2", "391:2", "req.UserID")}}, + {"AliasArraySlice", []PortEdge{pt("396:2", "398:2", "s"), pt("398:2", "399:2", "arr[*]")}}, + {"AliasFieldAddr", []PortEdge{pt("405:2", "407:2", "p"), pt("407:2", "408:2", "o.Inner.n")}}, + // Option A: reference-typed fields copied out (one-step offset). + {"AliasRefField", []PortEdge{pt("424:2", "425:2", "s.m[*]"), pt("424:2", "@formal_out:1", "s")}}, + {"AliasPtrField", []PortEdge{pt("436:2", "437:2", "r.ptr.n"), pt("436:2", "@formal_out:1", "r")}}, + // I2: an io.Writer and an io.Reader holding one *bytes.Buffer. + {"AliasIfaces", []PortEdge{pt("445:2", "446:2", "r"), pt("445:2", "@formal_out:1", "f"), pt("446:2", "@formal_out:1", "f")}}, } { if got := pointsTo(res.Callables[sdgKey(c.fn)].Ddg); !slices.Equal(got, c.want) { t.Errorf("%s points-to edges:\n got %+v\n want %+v", c.fn, got, c.want) @@ -62,51 +84,101 @@ func TestAlias_PointerCopyLinks(t *testing.T) { } // TestAlias_ValueCopiesNeverAlias: AliasValue copies req by value (no -// reference inside Req), so the write to the copy reaches nothing through -// req: no points-to edge. The oracle itself refuses value types, nil bases -// and pseudo paths, and accepts an interface its operand implements (T or -// *T), a global pointer with a local one, and identical reference types. +// reference inside Req) and AliasValueCopies copies s twice, no address +// taken: no points-to edge. The oracle table pins the type rules (typeOracle's +// doc; fix round 1, I1–I3). func TestAlias_ValueCopiesNeverAlias(t *testing.T) { - res := runAlias(t, typeOracle{}) - if got := pointsTo(res.Callables[sdgKey("AliasValue")].Ddg); len(got) != 0 { - t.Errorf("AliasValue has points-to edges %+v", got) + res := runAlias(t, true) + for _, fn := range []string{"AliasValue", "AliasValueCopies"} { + if got := pointsTo(res.Callables[sdgKey(fn)].Ddg); len(got) != 0 { + t.Errorf("%s has points-to edges %+v", fn, got) + } } pkg := types.NewPackage("example.com/p", "p") - v := func(name string, t types.Type) dataflow.Path { - return dataflow.Path{Base: types.NewVar(token.NoPos, pkg, name, t), Spell: name + ".f"} - } - st := types.NewStruct([]*types.Var{types.NewField(token.NoPos, pkg, "f", types.Typ[types.Int], false)}, nil) - named := types.NewNamed(types.NewTypeName(token.NoPos, pkg, "S", nil), st, nil) - m := types.NewFunc(token.NoPos, pkg, "M", types.NewSignatureType(types.NewVar(token.NoPos, pkg, "s", types.NewPointer(named)), nil, nil, nil, nil, false)) - named.AddMethod(m) - iface := types.NewInterfaceType([]*types.Func{types.NewFunc(token.NoPos, pkg, "M", types.NewSignatureType(nil, nil, nil, nil, nil, false))}, nil).Complete() - other := types.NewInterfaceType([]*types.Func{types.NewFunc(token.NoPos, pkg, "N", types.NewSignatureType(nil, nil, nil, nil, nil, false))}, nil).Complete() - gv := types.NewVar(token.NoPos, pkg, "G", types.NewPointer(named)) + newNamed := func(name string, u types.Type) *types.Named { + return types.NewNamed(types.NewTypeName(token.NoPos, pkg, name, nil), u, nil) + } + field := func(name string, t types.Type) *types.Var { return types.NewField(token.NoPos, pkg, name, t, false) } + S := newNamed("S", types.NewStruct([]*types.Var{field("f", types.Typ[types.Int])}, nil)) + S.AddMethod(types.NewFunc(token.NoPos, pkg, "M", types.NewSignatureType(types.NewVar(token.NoPos, pkg, "s", types.NewPointer(S)), nil, nil, nil, nil, false))) + method := func(name string) *types.Interface { + return types.NewInterfaceType([]*types.Func{types.NewFunc(token.NoPos, pkg, name, types.NewSignatureType(nil, nil, nil, nil, nil, false))}, nil).Complete() + } + I, J := newNamed("I", method("M")), newNamed("J", method("N")) + Inner := newNamed("Inner", types.NewStruct([]*types.Var{field("n", types.Typ[types.Int])}, nil)) + Outer := newNamed("Outer", types.NewStruct([]*types.Var{types.NewField(token.NoPos, pkg, "Inner", Inner, true)}, nil)) + bytesT := types.NewSlice(types.Typ[types.Byte]) + Bytes := newNamed("Bytes", bytesT) + header := types.NewMap(types.Typ[types.String], types.NewSlice(types.Typ[types.String])) + Header := newNamed("Header", header) + ch := func(d types.ChanDir) types.Type { return types.NewChan(d, types.Typ[types.Int]) } + fn := types.NewSignatureType(nil, nil, nil, types.NewTuple(types.NewVar(token.NoPos, pkg, "", types.Typ[types.Int])), nil, false) + arr := types.NewArray(types.Typ[types.Int], 4) + + o := newTypeOracle(dataflow.Unit{}, nil).(*typeOracle) + vars := map[string]*types.Var{} + P := func(name string, t types.Type, spell string) dataflow.Path { + v, ok := vars[name] + if !ok { + v = types.NewVar(token.NoPos, pkg, name, t) + vars[name] = v + } + return dataflow.Path{Base: v, Spell: spell} + } + gv := types.NewVar(token.NoPos, pkg, "G", types.NewPointer(S)) pkg.Scope().Insert(gv) - global := dataflow.Path{Base: gv, Spell: "example.com/p::G.f"} + gs := types.NewVar(token.NoPos, pkg, "GS", S) + pkg.Scope().Insert(gs) + taken := P("x", S, "x.f") + o.taken[taken.Base] = true + tarr := P("arr", arr, "arr[*]") + o.taken[tarr.Base] = true + o.takenSub[subKey{tarr.Base, "[*]"}] = true + to := P("o", Outer, "o.Inner.n") + o.taken[to.Base] = true + o.takenSub[subKey{to.Base, ".Inner"}] = true - o := typeOracle{} for _, c := range []struct { what string - a, b dataflow.Path + d, u dataflow.Path want bool }{ - {"value structs", v("a", named), v("b", named), false}, - {"ints", v("a", types.Typ[types.Int]), v("b", types.Typ[types.Int]), false}, - {"identical pointers", v("a", types.NewPointer(named)), v("b", types.NewPointer(named)), true}, - {"pointer, value", v("a", types.NewPointer(named)), v("b", named), false}, - {"different pointers", v("a", types.NewPointer(named)), v("b", types.NewPointer(types.Typ[types.Int])), false}, - {"*S implements I", v("a", types.NewPointer(named)), v("b", iface), true}, - {"I, *S", v("a", iface), v("b", types.NewPointer(named)), true}, - {"S's *S implements I, but S is a value", v("a", named), v("b", iface), false}, - {"*S does not implement J", v("a", types.NewPointer(named)), v("b", other), false}, - {"global and local pointer", global, v("b", types.NewPointer(named)), true}, - {"nil base", dataflow.Path{Spell: "u.f"}, v("b", types.NewPointer(named)), false}, - {"pseudo path", dataflow.Path{Spell: ":9:9"}, v("b", types.NewPointer(named)), false}, + {"value structs", P("a1", S, "a1.f"), P("b1", S, "b1.f"), false}, + {"ints", P("a2", types.Typ[types.Int], "a2"), P("b2", types.Typ[types.Int], "b2"), false}, + {"identical pointers", P("a3", types.NewPointer(S), "a3.f"), P("b3", types.NewPointer(S), "b3.f"), true}, + {"bare def of a pointer rebinds it", P("a3", nil, "a3"), P("b3", nil, "b3.f"), false}, + {"pointer, untaken value", P("a3", nil, "a3.f"), P("b4", S, "b4.f"), false}, + {"different pointees", P("a3", nil, "a3.f"), P("b5", types.NewPointer(types.Typ[types.Int]), "b5.*"), false}, + {"*S implements I", P("a3", nil, "a3.f"), P("i1", I, "i1.*"), true}, + {"I, *S", P("i1", nil, "i1.*"), P("a3", nil, "a3.f"), true}, + {"value S, I (only *S implements)", P("a1", nil, "a1.f"), P("i1", nil, "i1.*"), false}, + {"*S does not implement J", P("a3", nil, "a3.f"), P("j1", J, "j1.*"), false}, + {"I2: two unrelated interfaces", P("i1", nil, "i1.*"), P("j1", nil, "j1.*"), true}, + {"I1: Bytes vs []byte", P("s1", Bytes, "s1[*]"), P("s2", bytesT, "s2[*]"), true}, + {"I1: Header vs map[string][]string", P("h1", Header, "h1[*]"), P("h2", header, "h2[*]"), true}, + {"I1: chan int vs <-chan int", P("c1", ch(types.SendRecv), "c1[*]"), P("c2", ch(types.RecvOnly), "c2[*]"), true}, + {"I1: chan<- int vs chan int", P("c3", ch(types.SendOnly), "c3[*]"), P("c1", nil, "c1[*]"), true}, + {"I1: []int vs []string", P("s3", types.NewSlice(types.Typ[types.Int]), "s3[*]"), P("s4", types.NewSlice(types.Typ[types.String]), "s4[*]"), false}, + {"I1: identical funcs", P("f1", fn, "f1.*"), P("f2", fn, "f2.*"), true}, + {"global and local pointer", dataflow.Path{Base: gv, Spell: "example.com/p::G.f"}, P("a3", nil, "a3.f"), true}, + {"I3: *S write, taken S read", P("a3", nil, "a3.f"), taken, true}, + {"I3: taken S write, *S read", P("x", nil, "x.f"), P("a3", nil, "a3.f"), true}, + {"I3: taken S bare def, *S read", P("x", nil, "x"), P("a3", nil, "a3.f"), true}, + {"I3: global value is always taken", P("a3", nil, "a3.f"), dataflow.Path{Base: gs, Spell: "example.com/p::GS.f"}, true}, + {"I3: two taken values never share storage", P("x", nil, "x.f"), dataflow.Path{Base: gs, Spell: "example.com/p::GS.f"}, false}, + {"I3: sliced array vs []int", P("s3", nil, "s3[*]"), tarr, true}, + {"I3: &o.Inner offset, p.n vs o.Inner.n", P("p1", types.NewPointer(Inner), "p1.n"), to, true}, + {"I3: &o.Inner offset, reverse", to, P("p1", nil, "p1.n"), true}, + {"I3: *Inner vs untaken Outer field", P("p1", nil, "p1.n"), P("o2", Outer, "o2.Inner.n"), false}, + {"ref field: m vs s.m", P("m1", Header, "m1[*]"), P("hs", newNamed("HS", types.NewStruct([]*types.Var{field("m", header)}, nil)), "hs.m[*]"), true}, + {"ref field: bare read of the struct", P("m1", nil, "m1[*]"), P("hs", nil, "hs"), true}, + {"ref field: field rebind writes no target", P("hs", nil, "hs.m"), P("m1", nil, "m1[*]"), false}, + {"nil base", dataflow.Path{Spell: "u.f"}, P("a3", nil, "a3.f"), false}, + {"pseudo path", dataflow.Path{Spell: ":9:9"}, P("a3", nil, "a3.f"), false}, } { - if got := o.MayAlias(c.a, c.b); got != c.want { - t.Errorf("%s: MayAlias = %v, want %v", c.what, got, c.want) + if got := o.MayAlias(c.d, c.u); got != c.want { + t.Errorf("%s: MayAlias(%s, %s) = %v, want %v", c.what, c.d.Spell, c.u.Spell, got, c.want) } } } @@ -121,10 +193,10 @@ func TestAlias_ValueCopiesNeverAlias(t *testing.T) { // skipped units get none. func TestAlias_AddsOnly(t *testing.T) { in := loadInput(t, "dataflow") - defer func(v AliasOracle) { aliasOracle = v }(aliasOracle) - aliasOracle = nil + defer func(v func(dataflow.Unit, ast.Node) AliasOracle) { newAliasOracle = v }(newAliasOracle) + newAliasOracle = nil off := Run(in) - aliasOracle = typeOracle{} + newAliasOracle = newTypeOracle on := Run(loadInput(t, "dataflow")) var nPT, nSumm int @@ -207,10 +279,10 @@ func TestAlias_AddsOnly(t *testing.T) { // lacks it off. func TestAlias_SummaryThroughAlias(t *testing.T) { want := SummaryEdge{Src: "382:2/actual_in:1", Dst: "382:2/actual_out"} - if got := runAlias(t, typeOracle{}).Callables[sdgKey("CallAliasAssign")].Summary; !slices.Contains(got, want) { + if got := runAlias(t, true).Callables[sdgKey("CallAliasAssign")].Summary; !slices.Contains(got, want) { t.Errorf("CallAliasAssign summary %+v lacks %+v", got, want) } - if got := runAlias(t, nil).Callables[sdgKey("CallAliasAssign")].Summary; slices.Contains(got, want) { + if got := runAlias(t, false).Callables[sdgKey("CallAliasAssign")].Summary; slices.Contains(got, want) { t.Errorf("CallAliasAssign summary %+v has %+v without the alias pass", got, want) } } @@ -237,3 +309,37 @@ func TestAlias_SkippedAndNoVTA(t *testing.T) { t.Errorf("no_vta AliasPtr points-to edges %+v lack 360:2 → 361:2", got) } } + +// TestWiring_GoldenWithAliasOff (M5) pins wire's output for every sdg +// fixture unit with the alias pass off, byte for byte, against +// testdata/wire_sdg_off.golden, which was generated by the pre-refactor +// wiring (7ec77e7) over this fixture: the port() refactor of the wiring is +// output-neutral. Regenerate with -update only for an intended change. +func TestWiring_GoldenWithAliasOff(t *testing.T) { + defer func(v func(dataflow.Unit, ast.Node) AliasOracle) { newAliasOracle = v }(newAliasOracle) + newAliasOracle = nil + in, _, p := runPorts(t) + var b strings.Builder + for _, u := range in.Units { + if u.Key.Path != sdgFile { + continue + } + fmt.Fprintf(&b, "%s\n", u.Key.Signature) + for _, e := range p.wire(u.Key) { + fmt.Fprintf(&b, "\t%s -> %s %s %s\n", e.Src, e.Dst, e.Var, e.Prov) + } + } + golden := filepath.Join("testdata", "wire_sdg_off.golden") + if *update { + if err := os.WriteFile(golden, []byte(b.String()), 0o644); err != nil { + t.Fatal(err) + } + } + want, err := os.ReadFile(golden) + if err != nil { + t.Fatal(err) + } + if b.String() != string(want) { + t.Errorf("wire output (alias off) differs from %s; diff with -update", golden) + } +} diff --git a/internal/sdg/testdata/wire_sdg_off.golden b/internal/sdg/testdata/wire_sdg_off.golden new file mode 100644 index 0000000..dbcc13f --- /dev/null +++ b/internal/sdg/testdata/wire_sdg_off.golden @@ -0,0 +1,395 @@ +example.com/dataflow/sdg.AliasArraySlice + 399:2 -> @formal_out reaching-defs + @formal_in:0 -> 398:2 v reaching-defs +example.com/dataflow/sdg.AliasFieldAddr + 408:2 -> @formal_out reaching-defs + @formal_in:0 -> 407:2 v reaching-defs +example.com/dataflow/sdg.AliasIfaces + 445:2 -> 445:2/actual_in:0 _recv reaching-defs + 445:2 -> 445:2/actual_in:1 p reaching-defs + 445:2 -> @formal_out:2 p reaching-defs + 445:2/actual_out:0 -> 445:2 :0 reaching-defs + 445:2/actual_out:1 -> 445:2 :1 reaching-defs + 445:2/actual_out:2 -> 445:2 _recv reaching-defs + 445:2/actual_out:3 -> 445:2 p reaching-defs + 446:10/actual_out:0 -> 446:2 :0 reaching-defs + 446:10/actual_out:1 -> 446:2 :1 reaching-defs + 446:10/actual_out:2 -> 446:2 _recv reaching-defs + 446:10/actual_out:3 -> 446:2 p reaching-defs + 446:2 -> 446:10/actual_in:0 _recv reaching-defs + 446:2 -> 446:10/actual_in:1 p reaching-defs + 446:2 -> @formal_out:2 p reaching-defs + 447:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 443:2 f reaching-defs + @formal_in:0 -> 444:2 f reaching-defs + @formal_in:0 -> @formal_out:1 f reaching-defs + @formal_in:1 -> 445:2 p reaching-defs + @formal_in:1 -> 446:2 p reaching-defs + @formal_in:1 -> @formal_out:2 p reaching-defs +example.com/dataflow/sdg.AliasPtr + 361:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 359:2 req reaching-defs + @formal_in:0 -> 361:2 req.UserID reaching-defs + @formal_in:0 -> @formal_out:1 req reaching-defs + @formal_in:1 -> 360:2 id reaching-defs +example.com/dataflow/sdg.AliasPtrField + 437:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 435:2 r.ptr reaching-defs + @formal_in:0 -> 437:2 r.ptr.n reaching-defs + @formal_in:0 -> @formal_out:1 r reaching-defs + @formal_in:1 -> 436:2 v reaching-defs +example.com/dataflow/sdg.AliasPtrValue + 391:2 -> @formal_out reaching-defs + @formal_in:0 -> 390:2 id reaching-defs +example.com/dataflow/sdg.AliasPtrWhole + 369:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 367:2 req reaching-defs + @formal_in:0 -> 369:2 req reaching-defs + @formal_in:0 -> @formal_out:1 req reaching-defs + @formal_in:1 -> 368:2 id reaching-defs +example.com/dataflow/sdg.AliasRefField + 425:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 423:2 s.m reaching-defs + @formal_in:0 -> 425:2 s.m[*] reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 424:2 k reaching-defs + @formal_in:1 -> 425:2 k reaching-defs + @formal_in:2 -> 424:2 v reaching-defs +example.com/dataflow/sdg.AliasValue + 376:2 -> @formal_out reaching-defs + @formal_in:0 -> 374:2 req reaching-defs + @formal_in:0 -> 376:2 req.UserID reaching-defs + @formal_in:1 -> 375:2 id reaching-defs +example.com/dataflow/sdg.AliasValueCopies + 417:2 -> @formal_out reaching-defs + @formal_in:0 -> 414:2 s reaching-defs + @formal_in:0 -> 415:2 s reaching-defs + @formal_in:1 -> 416:2 v reaching-defs +example.com/dataflow/sdg.Buffered + 93:2 -> 93:2/actual_in:0 _recv reaching-defs + 93:2 -> 93:2/actual_in:1 p reaching-defs + 93:2 -> @formal_out:0 w reaching-defs + 93:2 -> @formal_out:1 p reaching-defs + 93:2/actual_out:0 -> 93:2 :0 reaching-defs + 93:2/actual_out:1 -> 93:2 :1 reaching-defs + 93:2/actual_out:2 -> 93:2 _recv reaching-defs + 93:2/actual_out:3 -> 93:2 p reaching-defs + @formal_in:0 -> 93:2 w reaching-defs + @formal_in:0 -> @formal_out:0 w reaching-defs + @formal_in:1 -> 93:2 p reaching-defs + @formal_in:1 -> @formal_out:1 p reaching-defs +example.com/dataflow/sdg.CallAliasAssign + 382:2 -> 382:2/actual_in:0 p reaching-defs + 382:2 -> 382:2/actual_in:1 v reaching-defs + 382:2 -> @formal_out p reaching-defs + 382:2/actual_out -> 382:2 p reaching-defs + @formal_in:0 -> 382:2 p reaching-defs + @formal_in:0 -> @formal_out p reaching-defs + @formal_in:1 -> 382:2 v reaching-defs +example.com/dataflow/sdg.CallFuncValue + 51:2 -> 51:9/actual_in:0 x reaching-defs + 51:2 -> @formal_out:0 reaching-defs + 51:9/actual_out -> 51:2 reaching-defs + @formal_in:0 -> 51:2 f reaching-defs + @formal_in:0 -> @formal_out:1 f reaching-defs +example.com/dataflow/sdg.CallPromoted + 80:2 -> 80:9/actual_in:0 _recv reaching-defs + 80:2 -> @formal_out:0 reaching-defs + 80:2 -> @formal_out:1 o reaching-defs + 80:9/actual_out:0 -> 80:2 reaching-defs + 80:9/actual_out:1 -> 80:2 _recv reaching-defs + @formal_in:0 -> 80:2 o reaching-defs + @formal_in:0 -> @formal_out:1 o reaching-defs +example.com/dataflow/sdg.CallVariadic + 195:2 -> 195:7/actual_in:0 xs reaching-defs + 195:7/actual_out:0 -> 195:2 reaching-defs + 195:7/actual_out:1 -> 195:2 xs reaching-defs + 196:16/actual_out:0 -> 196:2 :0 reaching-defs + 196:16/actual_out:1 -> 196:2 :1 reaching-defs + 196:2 -> 196:7/actual_in:0 xs reaching-defs + 196:7/actual_out:0 -> 196:2 reaching-defs + 196:7/actual_out:1 -> 196:2 xs reaching-defs + 197:15/actual_out:0 -> 197:2 :0 reaching-defs + 197:15/actual_out:1 -> 197:2 :1 reaching-defs + 197:2 -> 197:7/actual_in:0 a reaching-defs + 197:2 -> 197:7/actual_in:1 b reaching-defs + 197:7/actual_out -> 197:2 reaching-defs + 198:2 -> @formal_out reaching-defs +example.com/dataflow/sdg.CaptureArg + 267:2 -> 267:7/actual_in:0 f reaching-defs + 267:7/actual_out:0 -> 267:2 reaching-defs + 267:7/actual_out:1 -> 267:2 f reaching-defs + 268:2 -> @formal_out reaching-defs + @formal_in:0 -> 267:13/actual_in:0 :v reaching-defs + @formal_in:0 -> 267:2 v reaching-defs +example.com/dataflow/sdg.CaptureArg.closure_1 + 267:26 -> @formal_out reaching-defs + @formal_in:0 -> 267:26 v reaching-defs +example.com/dataflow/sdg.CaptureNamed + 293:2 -> 293:7/actual_in:0 :err reaching-defs + 293:2 -> @formal_out reaching-defs + 293:7/actual_out -> 293:2 :err reaching-defs + 293:7/actual_out -> 295:2 :err reaching-defs + 293:7/actual_out -> @formal_out reaching-defs + @entry -> 293:7/actual_in:0 :err reaching-defs + @entry -> @formal_out reaching-defs +example.com/dataflow/sdg.CaptureNamed.closure_1 + 293:16 -> @formal_out :err reaching-defs +example.com/dataflow/sdg.CaptureParam + 301:2 -> 301:7/actual_in:0 :p reaching-defs + 301:2 -> @formal_out p reaching-defs + 301:7/actual_out -> 301:2 :p reaching-defs + 301:7/actual_out -> @formal_out p reaching-defs + @formal_in:0 -> 301:2 p reaching-defs + @formal_in:0 -> 301:7/actual_in:0 :p reaching-defs + @formal_in:0 -> @formal_out p reaching-defs +example.com/dataflow/sdg.CaptureParam.closure_1 + 301:16 -> @formal_out :p reaching-defs + @formal_in:0 -> 301:16 p reaching-defs + @formal_in:0 -> @formal_out :p reaching-defs +example.com/dataflow/sdg.CaptureSite + 214:2 -> 215:7/actual_in:0 :x reaching-defs + 215:2 -> 215:7/actual_in:0 :x reaching-defs + 215:7/actual_out -> 215:2 :x reaching-defs + 219:2 -> @formal_out reaching-defs + @formal_in:0 -> 214:2 v reaching-defs +example.com/dataflow/sdg.CaptureSite.closure_1 + 216:3 -> @formal_out:1 :x reaching-defs + 217:3 -> @formal_out:0 reaching-defs + @formal_in:0 -> 216:3 x reaching-defs +example.com/dataflow/sdg.ChainLit + 259:2/2 -> 259:2/actual_in:0 :x reaching-defs + 259:2/actual_out -> 259:2/2 reaching-defs + @formal_in:0 -> 259:2/2 v reaching-defs +example.com/dataflow/sdg.ChainLit.closure_1 + 259:25 -> @formal_out reaching-defs + @formal_in:0 -> 259:18 x reaching-defs +example.com/dataflow/sdg.DeadLit + @formal_in:0 -> 274:2 v reaching-defs +example.com/dataflow/sdg.DeadLit.closure_1 + 276:20 -> @formal_out reaching-defs + @formal_in:0 -> 276:20 v reaching-defs +example.com/dataflow/sdg.DirectLit + 226:2 -> 226:2/actual_in:0 a reaching-defs + 226:2 -> 226:2/actual_in:1 :y reaching-defs + 226:2/actual_out -> 226:2 :y reaching-defs + 229:2 -> @formal_out reaching-defs + @formal_in:0 -> 226:2 v reaching-defs +example.com/dataflow/sdg.DirectLit.closure_1 + 227:3 -> @formal_out :y reaching-defs + @formal_in:0 -> 227:3 a reaching-defs +example.com/dataflow/sdg.Inner.M + 65:2 -> @formal_out reaching-defs + @formal_in:0 -> 65:2 i.n reaching-defs +example.com/dataflow/sdg.MethodExpr + 209:2 -> 209:9/actual_in:0 s reaching-defs + 209:2 -> 209:9/actual_in:1 k reaching-defs + 209:2 -> @formal_out:0 reaching-defs + 209:2 -> @formal_out:1 s reaching-defs + 209:9/actual_out:0 -> 209:2 reaching-defs + 209:9/actual_out:1 -> 209:2 s reaching-defs + @formal_in:0 -> 209:2 s reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 209:2 k reaching-defs +example.com/dataflow/sdg.MethodExprVariadic + 287:2 -> 287:9/actual_in:0 _recv reaching-defs + 287:2 -> @formal_out reaching-defs + 287:9/actual_out -> 287:2 reaching-defs + @formal_in:0 -> 287:2 v reaching-defs +example.com/dataflow/sdg.MethodValue + 204:2 -> 204:9/actual_in:0 s reaching-defs + 204:2 -> 204:9/actual_in:1 k reaching-defs + 204:2 -> @formal_out:0 reaching-defs + 204:9/actual_out:0 -> 204:2 reaching-defs + 204:9/actual_out:1 -> 204:2 s reaching-defs + @formal_in:0 -> 203:2 s reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 204:2 k reaching-defs +example.com/dataflow/sdg.MixedValue + 241:2 -> 241:9/actual_in:0 s reaching-defs + 241:2 -> 241:9/actual_in:1 k reaching-defs + 241:2 -> @formal_out:0 reaching-defs + 241:9/actual_out:0 -> 241:2 reaching-defs + 241:9/actual_out:1 -> 241:2 s reaching-defs + @formal_in:0 -> 239:3 s reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 238:2 b reaching-defs +example.com/dataflow/sdg.Named + 179:2 -> @formal_out reaching-defs +example.com/dataflow/sdg.PassBuffer + 99:2 -> 99:2/actual_in:0 w reaching-defs + 99:2 -> 99:2/actual_in:1 p reaching-defs + 99:2/actual_out:0 -> 99:2 w reaching-defs + 99:2/actual_out:1 -> 99:2 p reaching-defs +example.com/dataflow/sdg.PassDouble + 56:2 -> 56:9/actual_in:0 f reaching-defs + 56:2 -> @formal_out reaching-defs + 56:9/actual_out:0 -> 56:2 reaching-defs + 56:9/actual_out:1 -> 56:2 f reaching-defs +example.com/dataflow/sdg.PassOuter + 86:2 -> 86:7/actual_in:0 o reaching-defs + 86:7/actual_out:0 -> 86:2 reaching-defs + 86:7/actual_out:1 -> 86:2 o reaching-defs + 87:2 -> 87:7/actual_in:0 o reaching-defs + 87:7/actual_out:0 -> 87:2 reaching-defs + 87:7/actual_out:1 -> 87:2 o reaching-defs + 88:2 -> @formal_out reaching-defs +example.com/dataflow/sdg.PrunedCall + 246:2 -> @formal_out reaching-defs + @formal_in:0 -> 246:2 v reaching-defs +example.com/dataflow/sdg.Stores + 40:2 -> 40:2/actual_in:0 s reaching-defs + 40:2 -> 40:2/actual_in:1 k reaching-defs + 40:2/actual_out:0 -> 40:2 reaching-defs + 40:2/actual_out:1 -> 40:2 s reaching-defs + 41:2 -> 41:2/actual_in:0 s reaching-defs + 41:2 -> 41:2/actual_in:1 k reaching-defs + 41:2/actual_out:0 -> 41:2 reaching-defs + 41:2/actual_out:1 -> 41:2 s reaching-defs + 42:2 -> 42:2/actual_in:0 s reaching-defs + 42:2 -> 42:2/actual_in:1 k reaching-defs + 42:2/actual_out:0 -> 42:2 reaching-defs + 42:2/actual_out:1 -> 42:2 s reaching-defs +example.com/dataflow/sdg.UseStore + 34:2 -> 34:9/actual_in:0 _recv reaching-defs + 34:2 -> 34:9/actual_in:1 k reaching-defs + 34:2 -> @formal_out:0 reaching-defs + 34:2 -> @formal_out:1 s reaching-defs + 34:9/actual_out:0 -> 34:2 reaching-defs + 34:9/actual_out:1 -> 34:2 _recv reaching-defs + @formal_in:0 -> 34:2 s reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 34:2 k reaching-defs +example.com/dataflow/sdg.Variadic + 185:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 185:2 xs reaching-defs + @formal_in:0 -> @formal_out:1 xs reaching-defs +example.com/dataflow/sdg.a + 308:2 -> 308:9/actual_in:0 x reaching-defs + 308:2 -> @formal_out reaching-defs + 308:9/actual_out -> 308:2 reaching-defs + @formal_in:0 -> 308:2 x reaching-defs +example.com/dataflow/sdg.aliasAssign + @formal_in:0 -> 152:2 p reaching-defs + @formal_in:0 -> @formal_out p reaching-defs + @formal_in:1 -> 153:2 v reaching-defs +example.com/dataflow/sdg.aliasCall + 159:2 -> 159:2/actual_in:0 p reaching-defs + @formal_in:0 -> 158:2 p reaching-defs + @formal_in:0 -> @formal_out p reaching-defs +example.com/dataflow/sdg.aliasRange + @formal_in:0 -> 145:2 s reaching-defs + @formal_in:0 -> @formal_out s reaching-defs + @formal_in:1 -> 146:3 v reaching-defs +example.com/dataflow/sdg.apply + 262:32 -> @formal_out:0 reaching-defs + 262:39/actual_out -> 262:32 reaching-defs + @formal_in:0 -> 262:32 f reaching-defs + @formal_in:0 -> @formal_out:1 f reaching-defs +example.com/dataflow/sdg.b + 312:2 -> 312:9/actual_in:0 x reaching-defs + 312:2 -> @formal_out reaching-defs + 312:9/actual_out -> 312:2 reaching-defs + @formal_in:0 -> 312:2 x reaching-defs +example.com/dataflow/sdg.c + 316:2 -> @formal_out reaching-defs + @formal_in:0 -> 316:2 x reaching-defs +example.com/dataflow/sdg.callIgnore + 326:2 -> 326:9/actual_in:0 x reaching-defs + 326:2 -> 326:9/actual_in:1 y reaching-defs + 326:2 -> @formal_out reaching-defs + 326:9/actual_out -> 326:2 reaching-defs + @formal_in:0 -> 326:2 v reaching-defs + @formal_in:1 -> 326:2 w reaching-defs +example.com/dataflow/sdg.callUnknown + 252:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 252:2 f reaching-defs + @formal_in:0 -> @formal_out:1 f reaching-defs +example.com/dataflow/sdg.double + 46:2 -> @formal_out reaching-defs + @formal_in:0 -> 46:2 x reaching-defs +example.com/dataflow/sdg.dropX + 333:3 -> @formal_out reaching-defs + 335:2 -> 335:9/actual_in:0 y reaching-defs + 335:2 -> @formal_out reaching-defs + 335:9/actual_out -> 335:2 reaching-defs + @formal_in:1 -> 332:2 y reaching-defs + @formal_in:1 -> 333:3 y reaching-defs + @formal_in:1 -> 335:2 y reaching-defs +example.com/dataflow/sdg.dropY + 339:2 -> 339:9/actual_in:0 x reaching-defs + 339:2 -> 339:9/actual_in:1 y reaching-defs + 339:2 -> @formal_out reaching-defs + 339:9/actual_out -> 339:2 reaching-defs + @formal_in:0 -> 339:2 y reaching-defs +example.com/dataflow/sdg.ignore + 322:2 -> @formal_out reaching-defs + @formal_in:1 -> 322:2 y reaching-defs +example.com/dataflow/sdg.implicit + 346:3 -> @formal_out reaching-defs + 348:2 -> @formal_out reaching-defs + @formal_in:0 -> 345:2 x reaching-defs +example.com/dataflow/sdg.initConfig + 111:2 -> @formal_out :example.com/dataflow/sdg::Config reaching-defs +example.com/dataflow/sdg.memStore.Get + 21:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 21:2 s.m[*] reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 21:2 k reaching-defs +example.com/dataflow/sdg.multiT.Multi + 174:2 -> @formal_out:0 :0 reaching-defs + 174:2 -> @formal_out:1 :1 reaching-defs + @formal_in:0 -> 174:2 s.n reaching-defs + @formal_in:1 -> 174:2 a reaching-defs +example.com/dataflow/sdg.mutate + 132:2 -> @formal_out:0 p reaching-defs + 133:2 -> @formal_out:1 s reaching-defs + @formal_in:0 -> 132:2 p reaching-defs + @formal_in:0 -> @formal_out:0 p reaching-defs + @formal_in:1 -> 133:2 s.m reaching-defs + @formal_in:1 -> @formal_out:1 s reaching-defs + @formal_in:3 -> 135:2 q reaching-defs + @formal_in:3 -> @formal_out:2 q reaching-defs +example.com/dataflow/sdg.pair + 188:26 -> @formal_out:0 :0 reaching-defs + 188:26 -> @formal_out:1 :1 reaching-defs +example.com/dataflow/sdg.readConfig + 116:10/actual_out -> 116:3 reaching-defs + 116:3 -> 116:10/actual_in:0 :example.com/dataflow/sdg::Config reaching-defs + 116:3 -> @formal_out reaching-defs + 118:2 -> @formal_out reaching-defs + @formal_in:0 -> 115:2 example.com/dataflow/sdg::Config reaching-defs + @formal_in:0 -> 116:3 example.com/dataflow/sdg::Config reaching-defs + @formal_in:0 -> 118:2 example.com/dataflow/sdg::Config reaching-defs +example.com/dataflow/sdg.readConst + 126:2 -> @formal_out reaching-defs +example.com/dataflow/sdg.sqlStore.Get + 29:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 29:2 s.dsn reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 29:2 k reaching-defs +example.com/dataflow/sdg.sqlStore.setDSN + 140:2 -> @formal_out s reaching-defs + @formal_in:0 -> 140:2 s reaching-defs + @formal_in:0 -> @formal_out s reaching-defs + @formal_in:1 -> 140:2 d reaching-defs +example.com/dataflow/sdg.sumT.Sum + 282:34 -> @formal_out:0 reaching-defs + @formal_in:1 -> 282:34 xs reaching-defs + @formal_in:1 -> @formal_out:1 xs reaching-defs +example.com/dataflow/sdg.takeInner +example.com/dataflow/sdg.takeTwo + 190:30 -> @formal_out reaching-defs + @formal_in:0 -> 190:30 a reaching-defs + @formal_in:1 -> 190:30 b reaching-defs +example.com/dataflow/sdg.unusedRef + 166:2 -> @formal_out reaching-defs + @formal_in:1 -> 166:2 v reaching-defs +example.com/dataflow/sdg.upper + 232:31 -> @formal_out reaching-defs + @formal_in:0 -> 232:31 t reaching-defs +example.com/dataflow/sdg.useConfig + 122:2 -> 122:9/actual_in:0 :example.com/dataflow/sdg::Config reaching-defs + 122:2 -> @formal_out reaching-defs + 122:9/actual_out -> 122:2 reaching-defs + @formal_in:0 -> 122:2 example.com/dataflow/sdg::Config reaching-defs diff --git a/internal/sdg/vertices.go b/internal/sdg/vertices.go index 7721a91..12584bd 100644 --- a/internal/sdg/vertices.go +++ b/internal/sdg/vertices.go @@ -37,6 +37,9 @@ type ports struct { calls map[dataflow.L3Key]map[string]*ast.CallExpr // unit → call key → call kept map[dataflow.L3Key]map[string]bool // unit → its kept L3 node keys names map[string]string // interned formal keys and `of`s (intern) + // fnsByPath is every unit function by file (rootOf, the alias oracle's + // address-taken scan root); built on first use. + fnsByPath map[string][]ast.Node } func newPorts(in Input, d *dispatch) *ports { diff --git a/internal/sdg/wiring.go b/internal/sdg/wiring.go index d0682a0..87a7c84 100644 --- a/internal/sdg/wiring.go +++ b/internal/sdg/wiring.go @@ -78,8 +78,8 @@ func (p *ports) wire(k dataflow.L3Key) []PortEdge { } var mayAlias func(a, b dataflow.Path) bool - if aliasOracle != nil { - mayAlias = memoOracle(aliasOracle) + if newAliasOracle != nil { + mayAlias = newAliasOracle(u, p.rootOf(u)).MayAlias } resolved, alias := dataflow.BuildDDGSplit(w.r.Nodes, w.r.Cfg, w.du, mayAlias) for _, e := range resolved { @@ -140,7 +140,7 @@ func (p *ports) wire(k dataflow.L3Key) []PortEdge { // re-solve, mapped onto the vertices like the re-solved edges, minus any // (src, dst, var) an L3 or reaching-defs edge already has. for _, e := range alias { - src, dst, v, ok := w.port(e, false) + src, dst, v, ok := w.portFrom(e.DdgEdge, e.Def, false) if !ok || l3[dataflow.DdgEdge{Src: src, Dst: dst, Var: v}] { continue } @@ -160,7 +160,14 @@ func (p *ports) wire(k dataflow.L3Key) []PortEdge { // with a formal_in; else nothing (the L3 edge stays L3's); // - any other edge: as is, from the base's formal_in instead of @entry. func (w *wiring) port(e dataflow.DdgEdge, inL3 bool) (src, dst, v string, ok bool) { - b := dataflow.SpellBase(e.Var) + return w.portFrom(e, dataflow.SpellBase(e.Var), inL3) +} + +// portFrom is port with the base whose formal_in an edge from @entry starts +// at given: the use's base for a re-solved edge (def and use share it), the +// def's for an alias edge (AliasEdge.Def: @entry defines a value base whose +// address is taken, and the use reads it through another base). +func (w *wiring) portFrom(e dataflow.DdgEdge, b string, inL3 bool) (src, dst, v string, ok bool) { src = e.Src fi, hasIn := w.inOf(b) fromEntry := hasIn && src == dataflow.EntryKey diff --git a/testdata/dataflow/sdg/sdg.go b/testdata/dataflow/sdg/sdg.go index 78a26a4..a1cc2a9 100644 --- a/testdata/dataflow/sdg/sdg.go +++ b/testdata/dataflow/sdg/sdg.go @@ -381,3 +381,68 @@ func AliasValue(req Req, id string) string { func CallAliasAssign(p *Inner, v int) { aliasAssign(p, v) } + +// AliasPtrValue takes req's address (spec §4.10's motivating case): the +// write through p reaches the read of the value req. +func AliasPtrValue(id string) string { + var req Req + p := &req + p.UserID = id + return req.UserID +} + +// AliasArraySlice writes the array arr through its slice s. +func AliasArraySlice(v int) int { + var arr [4]int + s := arr[:] + s[0] = v + return arr[0] +} + +// AliasFieldAddr writes o's embedded Inner through its address p: a +// one-step offset (p.n is o.Inner.n). +func AliasFieldAddr(v int) int { + var o Outer + p := &o.Inner + p.n = v + return o.Inner.n +} + +// AliasValueCopies copies s twice by value, no address taken: the write to +// s1 never reaches s2 (Inner holds no reference). +func AliasValueCopies(s Inner, v int) int { + s1 := s + s2 := s + s1.n = v + return s2.n +} + +// AliasRefField copies the map field s.m out: the write through m reaches +// the read through s.m (a reference-typed field, one-step offset). +func AliasRefField(s memStore, k, v string) string { + m := s.m + m[k] = v + return s.m[k] +} + +type ptrHolder struct { + ptr *Inner +} + +// AliasPtrField copies the pointer field r.ptr out: the write through p +// reaches the read through r.ptr. +func AliasPtrField(r ptrHolder, v int) int { + p := r.ptr + p.n = v + return r.ptr.n +} + +// AliasIfaces holds one value in two unrelated interfaces: the write +// through w may change what r reads. +func AliasIfaces(f *bytes.Buffer, p []byte) int { + var r io.Reader = f + var w io.Writer = f + w.Write(p) + n, _ := r.Read(p) + return n +} From c74c11b680a06a3519248fb167b7582813b7fcfd Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Mon, 26 Oct 2026 22:15:24 -0400 Subject: [PATCH 25/65] fix(l4): may-alias keeps distinct-def entry edges, both anchors at offset 0, array-pointer slices --- internal/dataflow/reachdefs.go | 13 +++++-- internal/dataflow/reachdefs_test.go | 24 +++++++++++++ internal/sdg/alias.go | 39 +++++++++++++++----- internal/sdg/alias_test.go | 17 +++++++++ internal/sdg/testdata/wire_sdg_off.golden | 34 ++++++++++++++++++ testdata/dataflow/sdg/sdg.go | 44 +++++++++++++++++++++++ 6 files changed, 160 insertions(+), 11 deletions(-) diff --git a/internal/dataflow/reachdefs.go b/internal/dataflow/reachdefs.go index 7744c87..ac65bc1 100644 --- a/internal/dataflow/reachdefs.go +++ b/internal/dataflow/reachdefs.go @@ -91,8 +91,11 @@ type AliasEdge struct { } // BuildDDGSplit is BuildDDGAlias split in two: BuildDDG's edges, sorted by -// (Src, Dst, Var), and the edges only the alias rule yields, none of whose -// DdgEdge is in the first list, sorted by (Src, Dst, Var, Def). One fixpoint +// (Src, Dst, Var), and the edges the alias rule yields, sorted by (Src, Dst, +// Var, Def). An alias edge's DdgEdge is in the first list only when it +// leaves @entry from a def whose base is not the use's (Def ≠ the var's +// base): it then stands for another def than the base edge on that triple +// (fix round 2, N1). One fixpoint // serves both: the L4 port wiring (package sdg) maps the first to // reaching-defs port edges and the second to its points-to delta. func BuildDDGSplit(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias func(a, b Path) bool) (ddg []DdgEdge, alias []AliasEdge) { @@ -274,7 +277,11 @@ func aliasEdges(keys []string, du map[string]DefUse, in [][]uint64, pathOf func( if u.path.Base == d.Base || !inWindow(ds, u.steps) || !mayAlias(d, u.path) { continue } - if e := (AliasEdge{DdgEdge{Src: src, Dst: k, Var: u.path.Spell}, def}); !base[e.DdgEdge] { + // An edge from @entry whose def base is not the use's maps + // to another formal_in than the base edge on its triple: + // it is kept, and the wiring dedupes after mapping. + e := AliasEdge{DdgEdge{Src: src, Dst: k, Var: u.path.Spell}, def} + if !base[e.DdgEdge] || def != "" && def != SpellBase(u.path.Spell) { edges[e] = true } } diff --git a/internal/dataflow/reachdefs_test.go b/internal/dataflow/reachdefs_test.go index a8fd9f9..10211b7 100644 --- a/internal/dataflow/reachdefs_test.go +++ b/internal/dataflow/reachdefs_test.go @@ -853,3 +853,27 @@ func TestDDGAlias_Rule(t *testing.T) { t.Errorf("nil oracle: got %v, want BuildDDG %v", nilA, base) } } + +// TestDDGAlias_EntryDefKept (fix round 2, N1): @entry defines both r (a +// value whose address may be taken) and q (a pointer); 1:1 reads q.f. The +// base edge @entry → 1:1 (q.f) is q's own; the alias edge from r's def on +// the same triple is a different def (another formal_in at L4), so it is +// kept with Def "r". An oracle aliasing q with itself adds nothing. +func TestDDGAlias_EntryDefKept(t *testing.T) { + r := types.NewVar(token.NoPos, nil, "r", types.NewStruct(nil, nil)) + q := types.NewVar(token.NoPos, nil, "q", types.NewPointer(r.Type())) + nodes := []dataflow.Node{{Key: entry}, {Key: "1:1"}, {Key: exit}} + cfg := []dataflow.CfgEdge{{Src: entry, Dst: "1:1", Kind: "fallthrough"}, {Src: "1:1", Dst: exit, Kind: "fallthrough"}} + du := map[string]dataflow.DefUse{ + entry: {Defs: []dataflow.Path{{Base: r, Spell: "r"}, {Base: q, Spell: "q"}}}, + "1:1": {Uses: []dataflow.Path{{Base: q, Spell: "q.f"}}}, + } + ddg, alias := dataflow.BuildDDGSplit(nodes, cfg, du, func(a, b dataflow.Path) bool { return true }) + if want := []dataflow.DdgEdge{dd(entry, "1:1", "q.f")}; !reflect.DeepEqual(ddg, want) { + t.Errorf("ddg = %v, want %v", ddg, want) + } + want := []dataflow.AliasEdge{{DdgEdge: dd(entry, "1:1", "q.f"), Def: "r"}} + if !reflect.DeepEqual(alias, want) { + t.Errorf("alias = %+v, want %+v", alias, want) + } +} diff --git a/internal/sdg/alias.go b/internal/sdg/alias.go index c7fe5cb..b905363 100644 --- a/internal/sdg/alias.go +++ b/internal/sdg/alias.go @@ -45,9 +45,10 @@ var newAliasOracle func(u dataflow.Unit, root ast.Node) AliasOracle = newTypeOra // points to. A write reaches the object only through a step after the // prefix: a bare def `p = …` (or `x.f = …` at offset 1) rebinds the // reference and writes nothing another base observes; -// - a storage anchor: the prefix is an addressable value whose address may -// be taken (taken below); the object is its own storage, so a bare def -// `x = …` overwrites it. +// - a storage anchor: the prefix is an addressable variable whose address +// may be taken (taken below); the object is its own storage, so a bare +// def `x = …` overwrites it. A reference-typed variable whose address is +// taken has both anchors. // // At offset 1 the first step is a field (".f") or element ("[*]") whose // declared type decides the anchor: a reference-typed field or element is a @@ -65,8 +66,9 @@ var newAliasOracle func(u dataflow.Unit, root ast.Node) AliasOracle = newTypeOra // (or type-parameter) types; an interface and a type that implements it // (a type parameter: that satisfies its constraint); else, on the // underlying types, pointers with identical elements, slices with -// identical elements, maps with identical keys and elements, chans with -// identical elements whatever their directions, identical signatures; +// identical elements, a pointer to an array [N]E and a slice []E, maps +// with identical keys and elements, chans with identical elements +// whatever their directions, identical signatures; // - a reference and a storage of type T (I3): a pointer to T, an // interface *T implements, or (T an array [N]E) a slice []E; // - never two storages: distinct variables never share storage. @@ -225,10 +227,13 @@ func (o *typeOracle) anchorsOf(p dataflow.Path) []anchor { steps := dataflow.PathSteps(p.Spell) global := dataflow.IsPkgVar(p.Base) baseRef := refType(bt) - switch { - case baseRef: + // A reference-typed base whose address is taken has both anchors: what + // it points to, and its own storage (`pp := &p; *pp = b` rewrites p, so + // p.n then reads b.n: fix round 2, N2). + if baseRef { as = append(as, anchor{ref: true, t: bt, rem: steps, write: len(steps) > 0}) - case global || o.taken[p.Base]: + } + if global || o.taken[p.Base] { as = append(as, anchor{t: bt, rem: steps, write: true}) } var firsts []string @@ -349,6 +354,9 @@ func refsMayAlias(a, b types.Type) bool { case ib: return implementsIface(a, b) } + if arrayPtrSlice(a, b) || arrayPtrSlice(b, a) { + return true + } switch ua := a.Underlying().(type) { case *types.Pointer: ub, ok := b.Underlying().(*types.Pointer) @@ -368,6 +376,21 @@ func refsMayAlias(a, b types.Type) bool { return false } +// arrayPtrSlice reports whether a is a pointer to an array [N]E and b a +// slice []E: pa[:] shares the array (fix round 2, O1). +func arrayPtrSlice(a, b types.Type) bool { + pa, ok := a.Underlying().(*types.Pointer) + if !ok { + return false + } + arr, ok := pa.Elem().Underlying().(*types.Array) + if !ok { + return false + } + sl, ok := b.Underlying().(*types.Slice) + return ok && types.Identical(arr.Elem(), sl.Elem()) +} + // refToStorage reports whether a reference of type r may point to the // storage of a value of type t (I3: typeOracle's doc). func refToStorage(r, t types.Type) bool { diff --git a/internal/sdg/alias_test.go b/internal/sdg/alias_test.go index a8b34c6..fb40112 100644 --- a/internal/sdg/alias_test.go +++ b/internal/sdg/alias_test.go @@ -76,6 +76,18 @@ func TestAlias_PointerCopyLinks(t *testing.T) { {"AliasPtrField", []PortEdge{pt("436:2", "437:2", "r.ptr.n"), pt("436:2", "@formal_out:1", "r")}}, // I2: an io.Writer and an io.Reader holding one *bytes.Buffer. {"AliasIfaces", []PortEdge{pt("445:2", "446:2", "r"), pt("445:2", "@formal_out:1", "f"), pt("446:2", "@formal_out:1", "f")}}, + // Fix round 2, N1: alias edges from @entry whose def is another base + // than the use's start at the def's formal_in, though @entry → the + // use is also the use base's own edge: req (a value param whose + // address is taken) and the value global GReq (&GReq in takeGReq) + // reach q.UserID. + {"AliasEntryValue", []PortEdge{pt("@formal_in:0", "457:2", "q.UserID"), pt("@formal_in:0", "@formal_out:1", "q")}}, + {"AliasEntryGlobal", []PortEdge{pt("@formal_in:1", "469:2", "q.UserID"), pt("@formal_in:1", "@formal_out:1", "q")}}, + // N2: an address-taken reference has its storage anchor too. + {"AliasPtrPtr", []PortEdge{pt("474:2", "476:2", "pp"), pt("476:2", "477:2", "p.n")}}, + {"AliasSlicePtr", []PortEdge{pt("483:2", "484:2", "s[*]"), pt("483:2", "@formal_out:1", "s"), pt("@formal_in:0", "483:2", "ps")}}, + // O1: a slice of an array pointer aliases it. + {"AliasArrPtr", []PortEdge{pt("490:2", "491:2", "pa[*]"), pt("490:2", "@formal_out:1", "pa")}}, } { if got := pointsTo(res.Callables[sdgKey(c.fn)].Ddg); !slices.Equal(got, c.want) { t.Errorf("%s points-to edges:\n got %+v\n want %+v", c.fn, got, c.want) @@ -135,6 +147,7 @@ func TestAlias_ValueCopiesNeverAlias(t *testing.T) { tarr := P("arr", arr, "arr[*]") o.taken[tarr.Base] = true o.takenSub[subKey{tarr.Base, "[*]"}] = true + o.taken[P("tp", types.NewPointer(S), "tp").Base] = true to := P("o", Outer, "o.Inner.n") o.taken[to.Base] = true o.takenSub[subKey{to.Base, ".Inner"}] = true @@ -161,6 +174,10 @@ func TestAlias_ValueCopiesNeverAlias(t *testing.T) { {"I1: chan<- int vs chan int", P("c3", ch(types.SendOnly), "c3[*]"), P("c1", nil, "c1[*]"), true}, {"I1: []int vs []string", P("s3", types.NewSlice(types.Typ[types.Int]), "s3[*]"), P("s4", types.NewSlice(types.Typ[types.String]), "s4[*]"), false}, {"I1: identical funcs", P("f1", fn, "f1.*"), P("f2", fn, "f2.*"), true}, + {"O1: *[4]int vs []int", P("pa", types.NewPointer(arr), "pa[*]"), P("s3", nil, "s3[*]"), true}, + {"O1: []int vs *[4]int", P("s3", nil, "s3[*]"), P("pa", nil, "pa[*]"), true}, + {"O1: *[4]int vs []string", P("pa", nil, "pa[*]"), P("s4", nil, "s4[*]"), false}, + {"N2: taken pointer's storage, bare def", P("tp", types.NewPointer(S), "tp"), P("pp", types.NewPointer(types.NewPointer(S)), "pp.*"), true}, {"global and local pointer", dataflow.Path{Base: gv, Spell: "example.com/p::G.f"}, P("a3", nil, "a3.f"), true}, {"I3: *S write, taken S read", P("a3", nil, "a3.f"), taken, true}, {"I3: taken S write, *S read", P("x", nil, "x.f"), P("a3", nil, "a3.f"), true}, diff --git a/internal/sdg/testdata/wire_sdg_off.golden b/internal/sdg/testdata/wire_sdg_off.golden index dbcc13f..181191d 100644 --- a/internal/sdg/testdata/wire_sdg_off.golden +++ b/internal/sdg/testdata/wire_sdg_off.golden @@ -1,6 +1,23 @@ +example.com/dataflow/sdg.AliasArrPtr + 491:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 489:2 pa reaching-defs + @formal_in:0 -> 491:2 pa[*] reaching-defs + @formal_in:0 -> @formal_out:1 pa reaching-defs example.com/dataflow/sdg.AliasArraySlice 399:2 -> @formal_out reaching-defs @formal_in:0 -> 398:2 v reaching-defs +example.com/dataflow/sdg.AliasEntryGlobal + 469:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 469:2 q.UserID reaching-defs + @formal_in:0 -> @formal_out:1 q reaching-defs + @formal_in:1 -> 468:2 example.com/dataflow/sdg::GReq.UserID reaching-defs +example.com/dataflow/sdg.AliasEntryValue + 455:3 -> @formal_out:1 q reaching-defs + 457:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 455:3 req reaching-defs + @formal_in:1 -> 457:2 q.UserID reaching-defs + @formal_in:1 -> @formal_out:1 q reaching-defs + @formal_in:2 -> 454:2 c reaching-defs example.com/dataflow/sdg.AliasFieldAddr 408:2 -> @formal_out reaching-defs @formal_in:0 -> 407:2 v reaching-defs @@ -38,6 +55,12 @@ example.com/dataflow/sdg.AliasPtrField @formal_in:0 -> 437:2 r.ptr.n reaching-defs @formal_in:0 -> @formal_out:1 r reaching-defs @formal_in:1 -> 436:2 v reaching-defs +example.com/dataflow/sdg.AliasPtrPtr + 477:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 474:2 a reaching-defs + @formal_in:0 -> @formal_out:1 a reaching-defs + @formal_in:1 -> 476:2 b reaching-defs + @formal_in:1 -> @formal_out:2 b reaching-defs example.com/dataflow/sdg.AliasPtrValue 391:2 -> @formal_out reaching-defs @formal_in:0 -> 390:2 id reaching-defs @@ -55,6 +78,13 @@ example.com/dataflow/sdg.AliasRefField @formal_in:1 -> 424:2 k reaching-defs @formal_in:1 -> 425:2 k reaching-defs @formal_in:2 -> 424:2 v reaching-defs +example.com/dataflow/sdg.AliasSlicePtr + 484:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 482:2 s reaching-defs + @formal_in:0 -> 484:2 s[*] reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 483:2 b reaching-defs + @formal_in:1 -> @formal_out:2 b reaching-defs example.com/dataflow/sdg.AliasValue 376:2 -> @formal_out reaching-defs @formal_in:0 -> 374:2 req reaching-defs @@ -377,6 +407,10 @@ example.com/dataflow/sdg.sumT.Sum 282:34 -> @formal_out:0 reaching-defs @formal_in:1 -> 282:34 xs reaching-defs @formal_in:1 -> @formal_out:1 xs reaching-defs +example.com/dataflow/sdg.takeGReq + 463:24 -> @formal_out:0 reaching-defs + @formal_in:0 -> 463:24 example.com/dataflow/sdg::GReq reaching-defs + @formal_in:0 -> @formal_out:1 :example.com/dataflow/sdg::GReq reaching-defs example.com/dataflow/sdg.takeInner example.com/dataflow/sdg.takeTwo 190:30 -> @formal_out reaching-defs diff --git a/testdata/dataflow/sdg/sdg.go b/testdata/dataflow/sdg/sdg.go index a1cc2a9..c5cd33c 100644 --- a/testdata/dataflow/sdg/sdg.go +++ b/testdata/dataflow/sdg/sdg.go @@ -446,3 +446,47 @@ func AliasIfaces(f *bytes.Buffer, p []byte) int { n, _ := r.Read(p) return n } + +// AliasEntryValue: q may point to the value param req, so req's input +// reaches the read q.UserID through q (an alias edge from @entry whose def +// is req, not q). +func AliasEntryValue(req Req, q *Req, c bool) string { + if c { + q = &req + } + return q.UserID +} + +// GReq is a value global whose address is taken (takeGReq). +var GReq Req + +func takeGReq() *Req { return &GReq } + +// AliasEntryGlobal reads GReq and q, which may point to GReq: GReq's input +// reaches the read q.UserID. +func AliasEntryGlobal(q *Req) string { + s := GReq.UserID + return s + q.UserID +} + +// AliasPtrPtr rewrites p through its address pp: p.n then reads b.n. +func AliasPtrPtr(a, b *Inner) int { + p := a + pp := &p + *pp = b + return p.n +} + +// AliasSlicePtr rewrites the slice s through its address ps. +func AliasSlicePtr(s, b []int) int { + ps := &s + *ps = b + return s[0] +} + +// AliasArrPtr writes the array pa points to through its slice s. +func AliasArrPtr(pa *[4]int) int { + s := pa[:] + s[0] = 1 + return pa[0] +} From 7ca9c570e45dfa8ca2886b5299751107b63d1710 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 10:46:21 -0400 Subject: [PATCH 26/65] fix(l4): give a reference-holding capture its formal_out A closure that writes through a local alias of a captured pointer (`q := p; q.n = x`) writes no path of p, so it had no capture formal_out for p and the flow x -> p was refuted. Apply the shared by-ref predicate (MayHoldRef) to captures, as to params. Found by metamorphic relation MR5. --- docs/design/specs/v2-l3-l4-dataflow.md | 6 ++-- internal/sdg/summary_test.go | 23 +++++++++++++++ internal/sdg/testdata/wire_sdg_off.golden | 12 ++++++++ internal/sdg/vertices.go | 34 ++++++++++++++++------- testdata/dataflow/sdg/sdg.go | 9 ++++++ 5 files changed, 72 insertions(+), 12 deletions(-) diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index 7819269..5b50950 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -363,7 +363,7 @@ Per callable (formals), all with `kind` + `of`, no span: | `@formal_in:` | `:::` | each package-level variable the callable (transitively, via summary) reads | | `@formal_out` / `@formal_out:` | `` / `:` | **one per result position** — `` for a single result; `:` for each position of a multi-result, **never the user's result names** (D6) | | `@formal_out:` | parameter name | each parameter (incl. the receiver, pointer or value) satisfying the **shared by-ref predicate** the callable may write through (by-ref out) | -| `@formal_out:` | `:` | each captured variable a closure writes (strong or weak def) | +| `@formal_out:` | `:` | each captured variable a closure writes (strong or weak def), or that satisfies the by-ref predicate (a write through a local alias, `q := p; q.n = v`, is no def of `p`) | | `@formal_out:` | `:::` | each package-level variable written (transitively, via summary) | **Indexing and order (amended, A9) [call].** Python's `IdentityMap` rule verbatim @@ -420,7 +420,9 @@ miss later assignments (false refutation). Rule: - The closure callable gets `@formal_in:` `of:":"` per captured variable and `@formal_out:` `of:":"` per captured variable it **writes** (strong or weak - def). + def) **or that satisfies the shared by-ref predicate** (amended 2026-10-27, from metamorphic + relation MR5): as for a by-ref param, a write through a local alias (`q := p; q.n = v`) is no + def of the capture at L3, so a reference-holding capture always gets its `formal_out`. - In the enclosing callable, the `FuncLit` has a **capture site** (amended, A6 — a call's key is `call.Pos()` = `Fun.Pos()` = `FuncLit.Pos()`, so keying the site at the `func` keyword collides with the call node of `func(x){…}(v)`, `defer func(){…}()`, `go func(){…}()`): diff --git a/internal/sdg/summary_test.go b/internal/sdg/summary_test.go index a1f69c5..cb32a97 100644 --- a/internal/sdg/summary_test.go +++ b/internal/sdg/summary_test.go @@ -502,3 +502,26 @@ func TestSummary_OmitsGlobalToGlobal(t *testing.T) { } } } + +// TestSummary_CaptureWrittenThroughAlias: a literal that writes through a +// local alias of a captured pointer (`q := p; q.n = x`) writes no path of p, +// yet the write reaches p's pointee. p may hold a reference, so the literal +// has a capture formal_out for p (the shared by-ref predicate, A10, as a +// param in mutatedFormals): the closure has the flow x → p, and its capture +// site has an actual_out for p that reaches CaptureAlias's formal_out of p. +// Found by the metamorphic relation MR5 (introduce an alias), which turned +// CaptureParam's flow from x into a false "no flow". +func TestSummary_CaptureWrittenThroughAlias(t *testing.T) { + in, _, p := runPorts(t) + lit := Target{Callable: dataflow.L3Key{Path: sdgFile, Signature: sdgPkg + ".CaptureAlias.closure_1"}} + ins, outs := p.formals(lit) + if len(ins) != 2 || len(outs) != 1 || outs[0].Of != ":p" { + t.Fatalf("CaptureAlias.closure_1 formals = %+v / %+v, want 2 in, one out :p", ins, outs) + } + b := newSDGBuilder(in, p.d) + flows := b.summarize() + // closure_1: formal_in 0 = :p, 1 = :x; formal_out 0 = :p. + checkFlows(t, flows, lit, [][2]int{{0, 0}, {1, 0}}) + got := p.wire(sdgKey("CaptureAlias")) + checkHas(t, "wire(CaptureAlias)", got, pe("499:7/actual_out", "@formal_out", "p")) +} diff --git a/internal/sdg/testdata/wire_sdg_off.golden b/internal/sdg/testdata/wire_sdg_off.golden index 181191d..1852fd5 100644 --- a/internal/sdg/testdata/wire_sdg_off.golden +++ b/internal/sdg/testdata/wire_sdg_off.golden @@ -145,6 +145,18 @@ example.com/dataflow/sdg.CallVariadic 197:2 -> 197:7/actual_in:1 b reaching-defs 197:7/actual_out -> 197:2 reaching-defs 198:2 -> @formal_out reaching-defs +example.com/dataflow/sdg.CaptureAlias + 499:7/actual_out -> 499:2 :p reaching-defs + 499:7/actual_out -> @formal_out p reaching-defs + @formal_in:0 -> 499:2 p reaching-defs + @formal_in:0 -> 499:7/actual_in:0 :p reaching-defs + @formal_in:0 -> @formal_out p reaching-defs + @formal_in:1 -> 499:2 x reaching-defs + @formal_in:1 -> 499:7/actual_in:1 :x reaching-defs +example.com/dataflow/sdg.CaptureAlias.closure_1 + @formal_in:0 -> 499:16 p reaching-defs + @formal_in:0 -> @formal_out :p reaching-defs + @formal_in:1 -> 499:24 x reaching-defs example.com/dataflow/sdg.CaptureArg 267:2 -> 267:7/actual_in:0 f reaching-defs 267:7/actual_out:0 -> 267:2 reaching-defs diff --git a/internal/sdg/vertices.go b/internal/sdg/vertices.go index 12584bd..cb0fc62 100644 --- a/internal/sdg/vertices.go +++ b/internal/sdg/vertices.go @@ -213,14 +213,11 @@ func resultOf(i, n int) string { } // captureOfs returns the `of` of every variable lit captures (dataflow.FreeVars -// order) and of those it writes (dataflow.CaptureWrites bases), in that order. -// A capture's name is unique: every variable a literal names from outside it +// order) and of those it writes (writtenCaptures), in that order. A +// capture's name is unique: every variable a literal names from outside it // resolves in the one scope chain the literal sits in. func captureOfs(lit *ast.FuncLit, info *types.Info, k int) (all, written []string) { - w := map[types.Object]bool{} - for _, path := range dataflow.CaptureWrites(lit, info, k) { - w[path.Base] = true - } + w := writtenCaptures(lit, info, k) for _, v := range dataflow.FreeVars(lit, info) { all = append(all, ofCapture+v.Name()) if w[v] { @@ -230,6 +227,26 @@ func captureOfs(lit *ast.FuncLit, info *types.Info, k int) (all, written []strin return all, written } +// writtenCaptures is the set of captures of lit that get a capture +// formal_out (spec §4.6): the bases lit writes (dataflow.CaptureWrites), and, +// by the shared by-ref predicate (A10, fix I1, as for a param in +// mutatedFormals), every capture whose type may hold a reference +// (dataflow.MayHoldRef). A capture is used by definition, and a write +// through a local alias (`q := p; q.n = v`) or passing it on is no def of +// the capture at L3, so a use is enough. +func writtenCaptures(lit *ast.FuncLit, info *types.Info, k int) map[types.Object]bool { + w := map[types.Object]bool{} + for _, path := range dataflow.CaptureWrites(lit, info, k) { + w[path.Base] = true + } + for _, v := range dataflow.FreeVars(lit, info) { + if dataflow.MayHoldRef(v.Type()) { + w[v] = true + } + } + return w +} + // resultCount is the number of results of t: from its go/types signature // where the load has one, else from the symbol table's or the external's // type strings. @@ -704,10 +721,7 @@ func (p *ports) capSites(u dataflow.Unit, taken map[string]bool) []capSite { } else { host = p.skippedHost(u, l.Lit) } - written := map[types.Object]bool{} - for _, path := range dataflow.CaptureWrites(l.Lit, u.Info, p.in.K) { - written[path.Base] = true - } + written := writtenCaptures(l.Lit, u.Info, p.in.K) nOut := 0 for _, v := range caps { if written[v] { diff --git a/testdata/dataflow/sdg/sdg.go b/testdata/dataflow/sdg/sdg.go index c5cd33c..e02ec69 100644 --- a/testdata/dataflow/sdg/sdg.go +++ b/testdata/dataflow/sdg/sdg.go @@ -490,3 +490,12 @@ func AliasArrPtr(pa *[4]int) int { s[0] = 1 return pa[0] } + +// CaptureAlias stores a literal that writes through a local alias of the +// captured pointer p: the write is no def of p, but p may hold a reference, +// so the literal has a capture formal_out for p (the by-ref predicate) and +// x reaches p's formal_out (found by metamorphic relation MR5). +func CaptureAlias(p *Inner, x int) { + f := func() { q := p; q.n = x } + f() +} From 7c29d4469b7cec66f7e73784637767bee0c66d8e Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 11:05:15 -0400 Subject: [PATCH 27/65] test(l4): metamorphic relations for L3/L4 on fixtures and real apps --- docs/design/specs/v2-l3-l4-dataflow.md | 22 + internal/metamorphic/compare.go | 1262 ++++++++++++++++++ internal/metamorphic/mr_test.go | 797 +++++++++++ internal/metamorphic/realapp_test.go | 82 ++ internal/metamorphic/run.go | 100 ++ internal/metamorphic/transform.go | 1429 ++++++++++++++++++++ internal/metamorphic/transform_delta.go | 1458 +++++++++++++++++++++ internal/sdg/testdata/wire_sdg_off.golden | 8 + scripts/metamorphic.sh | 47 + testdata/dataflow/sdg/sdg.go | 11 + 10 files changed, 5216 insertions(+) create mode 100644 internal/metamorphic/compare.go create mode 100644 internal/metamorphic/mr_test.go create mode 100644 internal/metamorphic/realapp_test.go create mode 100644 internal/metamorphic/run.go create mode 100644 internal/metamorphic/transform.go create mode 100644 internal/metamorphic/transform_delta.go create mode 100755 scripts/metamorphic.sh diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index 5b50950..9a59232 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -1025,6 +1025,28 @@ determinism (`-j 1` vs `-j N`, two runs), monotonicity (`-a 1..4` superset via a Python's `test_v2_superset.py`), cross-projection (Neo4j vs JSON), Neo4j conformance, SDK model validation of every level's output. +**Metamorphic relations** (`internal/metamorphic`; Task 17b, amended 2026-10-27). A relation +transforms a copy of a module, analyzes it in-process at `-a 4`, and compares the result with +the original's through a position map. Callables are keyed by signature (plus the declaring file +for the per-file `init`), not by module id. There are three oracles: **exact** (the normalized +views are equal), **delta** (the view changes by exactly the edges each site declares, nothing +else changes, and the expected CDG is a reference post-dominator CDG of the expected CFG), and +**reach** (one-way: every `formal_in` → `actual_in`/`formal_out` flow of a sample of callables +survives over ddg ∪ cdg ∪ summary ∪ param_in ∪ param_out). MR1 adds blank lines and comments +(exact). MR2 reverses the top-level declarations (exact). MR3 moves a function and a method to +a new file (exact). MR4 alpha-renames locals (exact modulo spelling). MR5 introduces an alias +(reach). MR6 extracts a helper (reach). MR7 runs `--target-files` at `-a 3` against a full run +(exact on that file). MR8 wraps a statement in `if c {}`. MR9 inserts a kill. MR10 inserts a +copy. MR11 swaps two independent statements. MR12 adds an early `if c { return }`. MR8–MR12 +use the delta oracle, on local scalars in callables with an L3 result and no defer or goto. +MR13 replaces a sink argument with a constant: it is a precision relation, so the flow into +that `actual_in` must be gone and nothing else may be added. MR14 routes a call through a +one-implementation interface, a closure or `defer` (reach, plus the summary edges for the +interface form). MR15 extracts a helper and inlines it back (exact, byte-identical source). +Loops, `select`/`switch`/`fallthrough`, panic edges and the defer chain are out of scope for +the delta relations. The fixtures (`testdata/dataflow` flow and sdg, chi) run in `go test`; +the eval apps run behind the `metamorphic` build tag (`scripts/metamorphic.sh`). + **SDK gates:** slice gate (exact set on a `can://…@line:col` criterion); taint gate (known source→sink found; sanitizer variant reported cut, not dropped; contiguous witness); a pair with no flow lands in `exhausted`; context-sensitivity only as far as summaries support; JSON and Neo4j diff --git a/internal/metamorphic/compare.go b/internal/metamorphic/compare.go new file mode 100644 index 0000000..44fe959 --- /dev/null +++ b/internal/metamorphic/compare.go @@ -0,0 +1,1262 @@ +package metamorphic + +// The normalized view of an analysis and the three oracles over it. +// +// A View keys callables by Identity: the signature, which stays the same when +// a declaration moves between files (MR3), plus the declaring file for the +// per-file `init` and its closures. It does not key them by can:// id, which +// names the module. Body keys are local ("l:c[/n][/actual_*]", "@entry", …). +// Edges are strings "kind|src|dst|label[|prov]" over local keys. The +// application edges (call_graph, param_in, param_out) are over Node strings +// "identity|key". +// +// Map carries a view through a transform's position map into the +// transformed module's coordinates, so the oracles compare like with like: +// +// - Exact: the mapped original equals the transformed view. +// - Delta: the mapped original, with each expected Delta applied, equals +// the transformed view. So after − before and before − after are exactly +// the expected additions and removals, and nothing outside the delta's +// callable changed. +// - Reach: every sink reachable from a sampled source in the original is +// still reachable, through the mapped nodes, in the transformed view. The +// check is one-way: over-approximation may add flows. + +import ( + "encoding/json" + "fmt" + "io" + "maps" + "math/bits" + "math/rand/v2" + "regexp" + "slices" + "sort" + "strconv" + "strings" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// View is the normalized analysis. +type View struct { + Calls map[string]*CV + // App holds "cg|src|dst|prov" over identities, and "pin|src|dst|var" / + // "pout|src|dst|var" over Node strings. + App map[string]bool + + callers map[string]bool // lazily, by called +} + +// CV is one callable (or a library stand-in, Kind "external"). +type CV struct { + Ident string + File string // declaring file (source_file, else the module) + Line int // the declaration's first line + Kind string + L3 bool // has a cfg (an L3 result) + // Body maps each body key to "kind|parent|of|flags|callee". + Body map[string]string + Edges map[string]bool +} + +func (c *CV) clone() *CV { + n := *c + n.Body, n.Edges = maps.Clone(c.Body), maps.Clone(c.Edges) + return &n +} + +// Node is the application-scope name of body key key of callable ident. +func Node(ident, key string) string { return ident + "|" + key } + +func splitNode(n string) (string, string) { + i := strings.LastIndexByte(n, '|') + return n[:i], n[i+1:] +} + +// externalIdent is the Identity of an @external id: "@ext:" + its tail. +func externalIdent(id string) (string, bool) { + i := strings.Index(id, "/@external/") + if i < 0 { + return "", false + } + return "@ext:" + id[i+len("/@external/"):], true +} + +// NodeAttr is the body attribute string of a node of kind with no parent, +// `of`, flags or callee. +func NodeAttr(kind string) string { return kind + "||||" } + +// Collect normalizes an analysis.json stream, decoding one module at a +// time. Its application fields come in json.Marshal order: symbol_table +// before call_graph, external_symbols, param_in and param_out, so every +// callable id is known before an edge names it. +func Collect(r io.Reader) (*View, error) { + dec := json.NewDecoder(r) + v := &View{Calls: map[string]*CV{}, App: map[string]bool{}} + ids := map[string]string{} + callee := map[*CV]map[string]string{} + delim := func(want json.Delim) error { + t, err := dec.Token() + if err != nil { + return err + } + if d, ok := t.(json.Delim); !ok || d != want { + return fmt.Errorf("collect: got %v, want %v", t, want) + } + return nil + } + skip := func() error { + var raw json.RawMessage + return dec.Decode(&raw) + } + if err := delim('{'); err != nil { + return nil, err + } + for dec.More() { + t, err := dec.Token() + if err != nil { + return nil, err + } + if t != "application" { + if err := skip(); err != nil { + return nil, err + } + continue + } + if err := delim('{'); err != nil { + return nil, err + } + for dec.More() { + k, err := dec.Token() + if err != nil { + return nil, err + } + switch k { + case "symbol_table": + if err := delim('{'); err != nil { + return nil, err + } + for dec.More() { + rel, err := dec.Token() + if err != nil { + return nil, err + } + var m v2.Module + if err := dec.Decode(&m); err != nil { + return nil, err + } + v.addModule(rel.(string), &m, ids, callee) + } + if err := delim('}'); err != nil { + return nil, err + } + case "call_graph": + var es []v2.Edge + if err := dec.Decode(&es); err != nil { + return nil, err + } + for _, e := range es { + v.App["cg|"+resolveID(ids, e.Src)+"|"+resolveID(ids, e.Dst)+"|"+strings.Join(e.Prov, ",")] = true + } + case "external_symbols": + var ext map[string]v2.ExternalSymbol + if err := dec.Decode(&ext); err != nil { + return nil, err + } + for id, e := range ext { + ident, _ := externalIdent(id) + c := &CV{Ident: ident, Kind: "external", Body: map[string]string{}, Edges: map[string]bool{}} + for key, n := range e.Body { + c.Body[key] = n.Kind + "||" + n.Of + "||" + } + v.Calls[ident] = c + } + case "param_in", "param_out": + tag := "pin" + if k == "param_out" { + tag = "pout" + } + if err := delim('['); err != nil { + return nil, err + } + for dec.More() { + var e v2.ParamEdge + if err := dec.Decode(&e); err != nil { + return nil, err + } + v.App[tag+"|"+globalNode(ids, e.Src)+"|"+globalNode(ids, e.Dst)+"|"+e.Var] = true + } + if err := delim(']'); err != nil { + return nil, err + } + default: + if err := skip(); err != nil { + return nil, err + } + } + } + if err := delim('}'); err != nil { + return nil, err + } + } + for c, m := range callee { + for key, id := range m { + c.Body[key] += resolveID(ids, id) + } + } + return v, nil +} + +func resolveID(ids map[string]string, id string) string { + if ident, ok := ids[id]; ok { + return ident + } + if ident, ok := externalIdent(id); ok { + return ident + } + return "?" + id +} + +// globalNode is the Node of a global body-node id (v2.GlobalOrdinal). +func globalNode(ids map[string]string, gid string) string { + i := strings.LastIndexByte(gid, '@') + if i < 0 { + return "?" + gid + } + owner, local := gid[:i], gid[i+1:] + if local == "" || local[0] < '0' || local[0] > '9' { + local = "@" + local + } + return Node(resolveID(ids, owner), local) +} + +func (v *View) addModule(rel string, m *v2.Module, ids map[string]string, callee map[*CV]map[string]string) { + var walk func(c *v2.Callable, file string, topInit bool) + walk = func(c *v2.Callable, file string, topInit bool) { + if c.SourceFile != "" { + file = c.SourceFile + } + ident := c.Signature + if topInit { + ident = identityOf(c.Signature, file) + } + cv := &CV{Ident: ident, File: file, Line: c.Span.Start[0], Kind: c.Kind, L3: len(c.Cfg) > 0, + Body: make(map[string]string, len(c.Body)), Edges: map[string]bool{}} + calls := map[string]string{} + for key, n := range c.Body { + flags := "" + if n.IsGoroutine { + flags += "g" + } + if n.IsDeferred { + flags += "d" + } + cv.Body[key] = n.Kind + "|" + n.Parent + "|" + n.Of + "|" + flags + "|" + if n.Callee != nil { + calls[key] = *n.Callee + } + } + for _, e := range c.Cfg { + cv.Edges["cfg|"+e.Src+"|"+e.Dst+"|"+e.Kind] = true + } + for _, e := range c.Cdg { + cv.Edges["cdg|"+e.Src+"|"+e.Dst+"|"] = true + } + for _, e := range c.Ddg { + cv.Edges["ddg|"+e.Src+"|"+e.Dst+"|"+e.Var+"|"+strings.Join(e.Prov, ",")] = true + } + for _, e := range c.Summary { + cv.Edges["sum|"+e.Src+"|"+e.Dst+"|"] = true + } + if prev, dup := v.Calls[ident]; dup { + // Two callables share an identity: keep both visible, so the + // comparer reports the collision instead of hiding one. + ident += "#dup:" + prev.File + ":" + strconv.Itoa(prev.Line) + "/" + file + ":" + strconv.Itoa(cv.Line) + cv.Ident = ident + } + ids[c.ID] = ident + v.Calls[ident] = cv + callee[cv] = calls + for _, k := range slices.Sorted(maps.Keys(c.Callables)) { + inner := c.Callables[k] + walk(&inner, file, topInit) + } + } + for _, k := range slices.Sorted(maps.Keys(m.Functions)) { + fn := m.Functions[k] + walk(&fn, rel, strings.HasSuffix(fn.Signature, ".init")) + } + for _, tn := range slices.Sorted(maps.Keys(m.Types)) { + t := m.Types[tn] + for _, k := range slices.Sorted(maps.Keys(t.Callables)) { + c := t.Callables[k] + walk(&c, rel, false) + } + } +} + +// called reports whether some param_in edge targets callable ident. +func (v *View) called(ident string) bool { + if v.callers == nil { + v.callers = map[string]bool{} + for e := range v.App { + if f := strings.Split(e, "|"); f[0] == "pin" { + v.callers[f[3]] = true + } + } + } + return v.callers[ident] +} + +// ─── names (MR4) ────────────────────────────────────────────────────────────── + +// Fix rewrites a transformed view's spellings back to the original's. +type Fix func(string) string + +// StripSuffix is the Fix undoing a rename that appended suffix. +func StripSuffix(suffix string) Fix { + return func(s string) string { return strings.ReplaceAll(s, suffix, "") } +} + +// Apply returns v with fix applied to every `of` and every var label. +func (v *View) Apply(fix Fix) *View { + out := &View{Calls: make(map[string]*CV, len(v.Calls)), App: make(map[string]bool, len(v.App))} + for id, c := range v.Calls { + n := *c + n.Body, n.Edges = make(map[string]string, len(c.Body)), make(map[string]bool, len(c.Edges)) + for k, b := range c.Body { + f := strings.Split(b, "|") + f[2] = fix(f[2]) + n.Body[k] = strings.Join(f, "|") + } + for e := range c.Edges { + f := strings.Split(e, "|") + if f[0] == "ddg" { + f[3] = fix(f[3]) + } + n.Edges[strings.Join(f, "|")] = true + } + out.Calls[id] = &n + } + for e := range v.App { + if strings.HasPrefix(e, "pin|") || strings.HasPrefix(e, "pout|") { + i := strings.LastIndexByte(e, '|') + e = e[:i+1] + fix(e[i+1:]) + } + out.App[e] = true + } + return out +} + +// ─── mapping ────────────────────────────────────────────────────────────────── + +// Mapper maps an old source position to a new one. +type Mapper interface { + Map(Pos) (Pos, bool) +} + +// identity maps every position to itself. +type identity struct{} + +func (identity) Map(p Pos) (Pos, bool) { return p, true } + +// Compose is the map a, then b. +func Compose(a, b Mapper) Mapper { return composed{a, b} } + +type composed struct{ a, b Mapper } + +func (c composed) Map(p Pos) (Pos, bool) { + q, ok := c.a.Map(p) + if !ok { + return Pos{}, false + } + return c.b.Map(q) +} + +var posKey = regexp.MustCompile(`^([0-9]+):([0-9]+)(.*)$`) +var deferVar = regexp.MustCompile(`:([0-9]+):([0-9]+)`) + +// mapper carries the keys of one view through pm. +type mapper struct { + pm Mapper + unmapped []string +} + +// key maps local key k of a callable declared in file. It returns the new +// key and the new file. +func (m *mapper) key(file, k string) (string, string, bool) { + g := posKey.FindStringSubmatch(k) + if g == nil { + return k, file, true + } + l, _ := strconv.Atoi(g[1]) + c, _ := strconv.Atoi(g[2]) + p, ok := m.pm.Map(Pos{File: file, Line: l, Col: c}) + if !ok { + return k, file, false + } + return v2.LocalID(p.Line, p.Col) + g[3], p.File, true +} + +func (m *mapper) must(ident, file, k string) string { + nk, _, ok := m.key(file, k) + if !ok { + m.unmapped = append(m.unmapped, ident+" "+k) + return "?" + k + } + return nk +} + +func (m *mapper) label(file, s string) string { + return deferVar.ReplaceAllStringFunc(s, func(x string) string { + g := deferVar.FindStringSubmatch(x) + nk, _, ok := m.key(file, g[1]+":"+g[2]) + if !ok { + return x + } + return ":" + nk + }) +} + +func (m *mapper) callable(id string, c *CV) *CV { + n := &CV{Ident: c.Ident, File: c.File, Line: c.Line, Kind: c.Kind, L3: c.L3, + Body: make(map[string]string, len(c.Body)), Edges: make(map[string]bool, len(c.Edges))} + if c.File != "" { + if p, ok := m.pm.Map(Pos{File: c.File, Line: c.Line, Col: 1}); ok { + n.File, n.Line = p.File, p.Line + } else { + m.unmapped = append(m.unmapped, id+" declaration") + } + } + for k, b := range c.Body { + nk := m.must(id, c.File, k) + f := strings.Split(b, "|") + if f[1] != "" { + f[1] = m.must(id, c.File, f[1]) + } + n.Body[nk] = strings.Join(f, "|") + } + for e := range c.Edges { + f := strings.Split(e, "|") + f[1], f[2] = m.must(id, c.File, f[1]), m.must(id, c.File, f[2]) + if f[0] == "ddg" { + f[3] = m.label(c.File, f[3]) + } + n.Edges[strings.Join(f, "|")] = true + } + return n +} + +// Map returns v in the coordinates pm maps into, and the keys that did not +// map (sorted). A nil pm maps every key to itself. +func (v *View) Map(pm Mapper) (*View, []string) { + if pm == nil { + pm = identity{} + } + m := &mapper{pm: pm} + out := &View{Calls: make(map[string]*CV, len(v.Calls)), App: make(map[string]bool, len(v.App))} + for id, c := range v.Calls { + out.Calls[id] = m.callable(id, c) + } + node := func(nd string) string { + ident, k := splitNode(nd) + c, ok := v.Calls[ident] + if !ok { + return nd + } + return Node(ident, m.must(ident, c.File, k)) + } + for e := range v.App { + f := strings.Split(e, "|") + if f[0] == "pin" || f[0] == "pout" { + // tag | src ident | src key | dst ident | dst key | var + out.App[f[0]+"|"+node(f[1]+"|"+f[2])+"|"+node(f[3]+"|"+f[4])+"|"+f[5]] = true + continue + } + out.App[e] = true + } + sort.Strings(m.unmapped) + return out, m.unmapped +} + +// ─── exact ──────────────────────────────────────────────────────────────────── + +// Scope limits a comparison. Only, when set, compares just those callables +// and no application edges. Skip leaves those callables out, together with +// every application edge that touches them. KeysOnly compares body keys but +// not their attributes. +type Scope struct { + Only map[string]bool + Skip map[string]bool + KeysOnly bool +} + +func (s Scope) in(ident string) bool { + if s.Only != nil && !s.Only[ident] { + return false + } + return !s.Skip[ident] +} + +func (s Scope) appIn(e string) bool { + if s.Only != nil { + return false + } + f := strings.Split(e, "|") + for _, x := range f[1:] { + if s.Skip[x] { + return false + } + } + return true +} + +// Diff lists where want and got differ within scope, at most limit lines +// (all when limit <= 0). +func Diff(want, got *View, s Scope, limit int) []string { + var out []string + add := func(format string, a ...any) { out = append(out, fmt.Sprintf(format, a...)) } + for _, id := range unionKeys(want.Calls, got.Calls) { + if !s.in(id) { + continue + } + w, g := want.Calls[id], got.Calls[id] + switch { + case g == nil: + add("%s: callable missing", id) + continue + case w == nil: + add("%s: callable added (%s:%d)", id, g.File, g.Line) + continue + } + if w.Kind != g.Kind || w.File != g.File || w.Line != g.Line { + add("%s: %s %s:%d -> %s %s:%d", id, w.Kind, w.File, w.Line, g.Kind, g.File, g.Line) + } + for _, k := range unionKeys(w.Body, g.Body) { + wb, wok := w.Body[k] + gb, gok := g.Body[k] + switch { + case !gok: + add("%s: body node %s (%s) missing", id, k, wb) + case !wok: + add("%s: body node %s (%s) added", id, k, gb) + case wb != gb && !s.KeysOnly: + add("%s: body node %s: %s -> %s", id, k, wb, gb) + } + } + for _, e := range unionKeys(w.Edges, g.Edges) { + switch { + case !g.Edges[e]: + add("%s: edge %s missing", id, e) + case !w.Edges[e]: + add("%s: edge %s added", id, e) + } + } + } + for _, e := range unionKeys(want.App, got.App) { + if !s.appIn(e) { + continue + } + switch { + case !got.App[e]: + add("app: %s missing", e) + case !want.App[e]: + add("app: %s added", e) + } + } + if limit > 0 && len(out) > limit { + n := len(out) + out = append(out[:limit:limit], fmt.Sprintf("… and %d more", n-limit)) + } + return out +} + +func unionKeys[V any](a, b map[string]V) []string { + set := make(map[string]bool, len(a)+len(b)) + for k := range a { + set[k] = true + } + for k := range b { + set[k] = true + } + return slices.Sorted(maps.Keys(set)) +} + +// ─── delta ──────────────────────────────────────────────────────────────────── + +// Ref names a body node in a delta: a key of the original (Pre), mapped +// through the position map, or a mark the transform set in its new text. +type Ref struct { + Pre string + Mark string +} + +// Pre and Mark build Refs. +func Pre(key string) Ref { return Ref{Pre: key} } +func Mark(name string) Ref { return Ref{Mark: name} } + +func (r Ref) String() string { + if r.Mark != "" { + return "mark:" + r.Mark + } + return "pre:" + r.Pre +} + +// DEdge is one expected edge change. Kind is cfg, cdg, ddg or sum. Label is +// the cfg kind or the ddg var (empty otherwise), and Prov is the ddg +// provenance ("ssa" when empty). +type DEdge struct { + Kind string + Src, Dst Ref + Label string + Prov string +} + +// Edge constructors, so a relation reads like the table. +func CFG(src, dst Ref, kind string) DEdge { return DEdge{Kind: "cfg", Src: src, Dst: dst, Label: kind} } +func CDG(src, dst Ref) DEdge { return DEdge{Kind: "cdg", Src: src, Dst: dst} } +func DDG(src, dst Ref, v string) DEdge { return DEdge{Kind: "ddg", Src: src, Dst: dst, Label: v} } +func Port(src, dst Ref, v string) DEdge { + return DEdge{Kind: "ddg", Src: src, Dst: dst, Label: v, Prov: "reaching-defs"} +} + +// Delta is the expected change of one callable (an Identity of the +// original). Add and Remove are the edges added and removed, and Nodes the +// body nodes added (mark → kind). With CDGFromCFG the expected CDG is not +// listed: it is the reference CDG of the expected CFG (RefCDG). With AtMost +// the relation is a precision one: any subset of Remove may go, and nothing +// may be added. +type Delta struct { + Site string + Callable string + Add []DEdge + Remove []DEdge + Nodes map[string]string + CDGFromCFG bool + AtMost bool +} + +func (r Ref) resolve(m *mapper, file string, marks map[string]Pos) (string, error) { + if r.Mark != "" { + p, ok := marks[r.Mark] + if !ok { + return "", fmt.Errorf("no mark %s", r.Mark) + } + return v2.LocalID(p.Line, p.Col), nil + } + k, _, ok := m.key(file, r.Pre) + if !ok { + return "", fmt.Errorf("pre key %s does not map", r.Pre) + } + return k, nil +} + +func (e DEdge) str(m *mapper, file string, marks map[string]Pos) (string, error) { + s, err := e.Src.resolve(m, file, marks) + if err != nil { + return "", err + } + d, err := e.Dst.resolve(m, file, marks) + if err != nil { + return "", err + } + out := e.Kind + "|" + s + "|" + d + "|" + e.Label + if e.Kind == "ddg" { + out += "|" + cmpOr(e.Prov, "ssa") + } + return out, nil +} + +func cmpOr(s, def string) string { + if s == "" { + return def + } + return s +} + +// Expect applies ds to mapped (pre mapped through pm) and returns the view +// the transformed module must have. A removed edge the original lacks, and +// an added one it already has, are errors of the relation (its delta is +// wrong), as is a declared CDG that disagrees with RefCDG of the expected +// CFG. +func Expect(pre, mapped *View, pm Mapper, marks map[string]Pos, ds []Delta) (*View, error) { + if pm == nil { + pm = identity{} + } + out := &View{Calls: maps.Clone(mapped.Calls), App: mapped.App} + m := &mapper{pm: pm} + for _, d := range ds { + c0, c := pre.Calls[d.Callable], mapped.Calls[d.Callable] + if c0 == nil || c == nil { + return nil, fmt.Errorf("delta %s: no callable %s", d.Site, d.Callable) + } + if d.AtMost { + continue // CheckAtMost + } + n := c.clone() + for _, e := range d.Remove { + s, err := e.str(m, c0.File, marks) + if err != nil { + return nil, fmt.Errorf("delta %s: %w", d.Site, err) + } + if !n.Edges[s] { + return nil, fmt.Errorf("delta %s: removed edge %s is not in the original", d.Site, s) + } + delete(n.Edges, s) + } + for _, e := range d.Add { + s, err := e.str(m, c0.File, marks) + if err != nil { + return nil, fmt.Errorf("delta %s: %w", d.Site, err) + } + if n.Edges[s] { + return nil, fmt.Errorf("delta %s: added edge %s is already there", d.Site, s) + } + n.Edges[s] = true + } + for mark, kind := range d.Nodes { + k, err := Mark(mark).resolve(m, c0.File, marks) + if err != nil { + return nil, fmt.Errorf("delta %s: %w", d.Site, err) + } + n.Body[k] = NodeAttr(kind) + } + ref := RefCDG(n) + if d.CDGFromCFG { + for e := range n.Edges { + if strings.HasPrefix(e, "cdg|") { + delete(n.Edges, e) + } + } + for e := range ref { + n.Edges[e] = true + } + } else if diff := cdgDiff(n, ref); diff != "" { + return nil, fmt.Errorf("delta %s: declared CDG disagrees with the reference CDG of the expected CFG: %s", d.Site, diff) + } + out.Calls[d.Callable] = n + } + return out, nil +} + +func cdgDiff(c *CV, ref map[string]bool) string { + var out []string + for e := range c.Edges { + if strings.HasPrefix(e, "cdg|") && !ref[e] { + out = append(out, "-"+e) + } + } + for e := range ref { + if !c.Edges[e] { + out = append(out, "+"+e) + } + } + sort.Strings(out) + return strings.Join(out, " ") +} + +// CheckAtMost checks the AtMost deltas of ds: in each callable got is +// mapped minus some subset of Remove, with nothing added. +func CheckAtMost(pre, mapped, got *View, pm Mapper, marks map[string]Pos, ds []Delta) []string { + if pm == nil { + pm = identity{} + } + var out []string + m := &mapper{pm: pm} + for _, d := range ds { + if !d.AtMost { + continue + } + c0, w, g := pre.Calls[d.Callable], mapped.Calls[d.Callable], got.Calls[d.Callable] + if c0 == nil || w == nil || g == nil { + out = append(out, d.Site+": callable missing") + continue + } + allowed := map[string]bool{} + for _, e := range d.Remove { + if s, err := e.str(m, c0.File, marks); err == nil { + allowed[s] = true + } + } + for _, e := range unionKeys(w.Edges, g.Edges) { + switch { + case !g.Edges[e] && !allowed[e]: + out = append(out, fmt.Sprintf("%s: %s removed", d.Site, e)) + case !w.Edges[e]: + out = append(out, fmt.Sprintf("%s: %s added", d.Site, e)) + } + } + } + return out +} + +// ─── reference CDG ──────────────────────────────────────────────────────────── + +// RefCDG is the control-dependence graph of c's cfg edges, computed +// independently of the analyzer: post-dominator sets by fixed-point +// iteration, then x is control dependent on a when some edge a→b has x +// post-dominating b (reflexively) but not strictly post-dominating a +// (Ferrante–Ottenstein–Warren); self-loops are dropped, and every CFG node +// but @entry/@exit with no controller is anchored at @entry (the analyzer's +// documented convention, spec §4.4). +func RefCDG(c *CV) map[string]bool { + succ := map[string][]string{} + set := map[string]bool{"@exit": true} + for e := range c.Edges { + if !strings.HasPrefix(e, "cfg|") { + continue + } + f := strings.Split(e, "|") + succ[f[1]] = append(succ[f[1]], f[2]) + set[f[1]], set[f[2]] = true, true + } + nodes := slices.Sorted(maps.Keys(set)) + idx := make(map[string]int, len(nodes)) + for i, n := range nodes { + idx[n] = i + } + words := (len(nodes) + 63) / 64 + full := make([]uint64, words) + for i := range nodes { + full[i/64] |= 1 << (i % 64) + } + exit := idx["@exit"] + pdom := make([][]uint64, len(nodes)) + for i := range nodes { + pdom[i] = slices.Clone(full) + } + pdom[exit] = make([]uint64, words) + pdom[exit][exit/64] |= 1 << (exit % 64) + for changed := true; changed; { + changed = false + for i, n := range nodes { + if i == exit { + continue + } + nw := slices.Clone(full) + if len(succ[n]) == 0 { + nw = make([]uint64, words) // reaches no exit + } + for _, s := range succ[n] { + for w := range nw { + nw[w] &= pdom[idx[s]][w] + } + } + nw[i/64] |= 1 << (i % 64) + if !slices.Equal(nw, pdom[i]) { + pdom[i], changed = nw, true + } + } + } + has := func(set []uint64, j int) bool { return set[j/64]&(1<<(j%64)) != 0 } + out := map[string]bool{} + controlled := map[string]bool{} + for _, a := range nodes { + ia := idx[a] + for _, b := range succ[a] { + ib := idx[b] + for w, word := range pdom[ib] { + for word != 0 { + j := w*64 + bits.TrailingZeros64(word) + word &= word - 1 + if j == ia || (has(pdom[ia], j) && j != ia) { + continue + } + if nodes[j] == "@exit" { + continue + } + out["cdg|"+a+"|"+nodes[j]+"|"] = true + controlled[nodes[j]] = true + } + } + } + } + for _, n := range nodes { + if n != "@entry" && n != "@exit" && !controlled[n] { + out["cdg|@entry|"+n+"|"] = true + } + } + return out +} + +// ─── reachability ───────────────────────────────────────────────────────────── + +// Graph is the application-scope dependence graph over Node strings: ddg ∪ +// summary ∪ param_in ∪ param_out, plus cdg unless dataOnly. +type Graph map[string][]string + +// Graph builds v's dependence graph. +func (v *View) Graph(dataOnly bool) Graph { + g := Graph{} + for id, c := range v.Calls { + for e := range c.Edges { + f := strings.SplitN(e, "|", 4) + if f[0] == "cfg" || (dataOnly && f[0] == "cdg") { + continue + } + s := Node(id, f[1]) + g[s] = append(g[s], Node(id, f[2])) + } + } + for e := range v.App { + f := strings.Split(e, "|") + if f[0] != "pin" && f[0] != "pout" { + continue + } + s := f[1] + "|" + f[2] + g[s] = append(g[s], f[3]+"|"+f[4]) + } + return g +} + +// Reach is the set of nodes reachable from src, src included. +func (g Graph) Reach(src string) map[string]bool { + seen := map[string]bool{src: true} + stack := []string{src} + for len(stack) > 0 { + n := stack[len(stack)-1] + stack = stack[:len(stack)-1] + for _, d := range g[n] { + if !seen[d] { + seen[d] = true + stack = append(stack, d) + } + } + } + return seen +} + +var vertexTail = regexp.MustCompile(`^(.*)/(actual_in|actual_out)(:[0-9]+)?$`) + +// locator finds the counterpart in post of a node of pre. It tries the +// mapped key in the same callable first. Failing that, it tries any +// callable of post that holds the mapped position in the mapped file, since +// a statement can move into a new callable (MR6, MR14). A call-site vertex +// (/actual_*:i) matches by its call key, kind and `of`, not by index. +type locator struct { + pre, post *View + m *mapper + byFileKey map[string][]string + // vertex indexes each call-site vertex of post by callable, call key, + // kind and `of` (the least key when several match). + vertex map[[4]string]string + memo map[string]string // locate's results ("" = none) +} + +func newLocator(pre, post *View, pm Mapper) *locator { + if pm == nil { + pm = identity{} + } + l := &locator{pre: pre, post: post, m: &mapper{pm: pm}, byFileKey: map[string][]string{}, vertex: map[[4]string]string{}, memo: map[string]string{}} + for id, c := range post.Calls { + for k, b := range c.Body { + if h := vertexTail.FindStringSubmatch(k); h != nil { + vk := [4]string{id, h[1], h[2], strings.Split(b, "|")[2]} + if prev, ok := l.vertex[vk]; !ok || k < prev { + l.vertex[vk] = k + } + continue + } + if posKey.MatchString(k) { + l.byFileKey[c.File+" "+k] = append(l.byFileKey[c.File+" "+k], id) + } + } + } + for _, ids := range l.byFileKey { + slices.Sort(ids) + } + return l +} + +func (l *locator) locate(node string) (string, bool) { + if r, ok := l.memo[node]; ok { + return r, r != "" + } + r, ok := l.find(node) + l.memo[node] = r + return r, ok +} + +func (l *locator) find(node string) (string, bool) { + ident, key := splitNode(node) + c0 := l.pre.Calls[ident] + if c0 == nil { + return "", false + } + nk, nf, ok := l.m.key(c0.File, key) + if !ok { + return "", false + } + g := vertexTail.FindStringSubmatch(nk) + cands := []string{ident} + if posKey.MatchString(nk) { + base := nk + if g != nil { + base = g[1] + } + cands = append(cands, l.byFileKey[nf+" "+base]...) + } + of := "" + if b, ok := c0.Body[key]; ok { + of = strings.Split(b, "|")[2] + } + for _, id := range cands { + c := l.post.Calls[id] + if c == nil { + continue + } + if g == nil { + if _, ok := c.Body[nk]; ok { + return Node(id, nk), true + } + continue + } + if k, ok := l.vertex[[4]string{id, g[1], g[2], of}]; ok { + return Node(id, k), true + } + } + return "", false +} + +// ReachResult is the outcome of the reachability oracle. Lost and +// Unlocated hold at most maxReport entries each; NLost and NUnlocated count +// them all. +type ReachResult struct { + Callables, Sources, Pairs int + Lost []string // "source -> sink", no longer reachable + Unlocated []string // sources or sinks with no counterpart + NLost, NUnlocated int +} + +const maxReport = 200 + +func (r *ReachResult) lost(s string) { + r.NLost++ + if len(r.Lost) < maxReport { + r.Lost = append(r.Lost, s) + } +} + +func (r *ReachResult) unlocated(s string) { + r.NUnlocated++ + if len(r.Unlocated) < maxReport { + r.Unlocated = append(r.Unlocated, s) + } +} + +// igraph is a Graph over interned node indices. +type igraph struct { + idx map[string]int32 + name []string + succ [][]int32 +} + +func intern(g Graph) *igraph { + ig := &igraph{idx: make(map[string]int32, len(g))} + id := func(n string) int32 { + i, ok := ig.idx[n] + if !ok { + i = int32(len(ig.name)) + ig.idx[n] = i + ig.name = append(ig.name, n) + ig.succ = append(ig.succ, nil) + } + return i + } + for _, n := range slices.Sorted(maps.Keys(g)) { + s := id(n) + for _, d := range g[n] { + ig.succ[s] = append(ig.succ[s], id(d)) + } + } + return ig +} + +// reach marks in seen (len = node count, cleared by the caller) every node +// reachable from src, and returns them in visit order. +func (ig *igraph) reach(src int32, seen []bool, out []int32) []int32 { + out = append(out[:0], src) + seen[src] = true + for i := 0; i < len(out); i++ { + for _, d := range ig.succ[out[i]] { + if !seen[d] { + seen[d] = true + out = append(out, d) + } + } + } + return out +} + +// CheckReach samples up to n callables of pre with an L3 result (those in +// prefer first, then the others, shuffled with seed). For every @formal_in +// of each, it checks that every sink reachable in pre is reachable from the +// source's counterpart in post. A sink is a formal_out vertex or the +// actual_in of a call node. The actual_in of a capture site is not a sink: +// it feeds only its closure's formal_in (param_in; a capture site has no +// summary edge, spec §4.6), so whatever a flow into it reaches downstream is +// checked at that sink. +// +// cut lists pre nodes whose out-edges the pre graph leaves out (a relation +// that legitimately removes flows through them, MR14). +func CheckReach(pre, post *View, pm Mapper, prefer []string, n int, seed uint64, cut ...string) ReachResult { + gPre, gPost := pre.Graph(false), post.Graph(false) + for _, c := range cut { + delete(gPre, c) + } + iPre, iPost := intern(gPre), intern(gPost) + seenPre, seenPost := make([]bool, len(iPre.name)), make([]bool, len(iPost.name)) + var bufPre, bufPost []int32 + // sinkPost[i] is the post counterpart of pre node i (-1 none, -2 not a + // sink, -3 not yet computed). + sinkPost := make([]int32, len(iPre.name)) + for i := range sinkPost { + sinkPost[i] = -3 + } + l := newLocator(pre, post, pm) + var res ReachResult + for _, id := range SampleCallables(pre, prefer, n, seed) { + res.Callables++ + c := pre.Calls[id] + for _, k := range slices.Sorted(maps.Keys(c.Body)) { + if !strings.HasPrefix(k, "@formal_in:") { + continue + } + src := Node(id, k) + psrc, ok := l.locate(src) + if !ok { + res.unlocated("source " + src) + continue + } + res.Sources++ + si, ok := iPre.idx[src] + if !ok { + continue // no out-edge: reaches nothing + } + bufPost = bufPost[:0] + if pi, ok := iPost.idx[psrc]; ok { + bufPost = iPost.reach(pi, seenPost, bufPost) + } + bufPre = iPre.reach(si, seenPre, bufPre) + var lost []string + for _, ni := range bufPre { + if ni == si { + continue + } + if sinkPost[ni] == -3 { + sinkPost[ni] = -2 + if sink := iPre.name[ni]; pre.isSink(sink) { + sinkPost[ni] = -1 + if ps, ok := l.locate(sink); ok { + if pi, ok := iPost.idx[ps]; ok { + sinkPost[ni] = pi + } else { + sinkPost[ni] = int32(len(iPost.name)) // located, no edges: unreachable + } + } + } + } + switch p := sinkPost[ni]; { + case p == -2: + continue + case p == -1: + res.Pairs++ + res.unlocated("sink " + iPre.name[ni]) + default: + res.Pairs++ + if int(p) >= len(seenPost) || !seenPost[p] { + lost = append(lost, iPre.name[ni]) + } + } + } + for _, ni := range bufPre { + seenPre[ni] = false + } + for _, ni := range bufPost { + seenPost[ni] = false + } + slices.Sort(lost) + for _, sink := range lost { + res.lost(src + " -> " + sink) + } + } + } + return res +} + +func (v *View) isSink(node string) bool { + id, k := splitNode(node) + if strings.HasPrefix(k, "@formal_out") { + return true + } + g := vertexTail.FindStringSubmatch(k) + if g == nil || g[2] != "actual_in" { + return false + } + c := v.Calls[id] + return c != nil && strings.HasPrefix(c.Body[g[1]], "call|") +} + +// SampleCallables picks up to n callables of v with an L3 result: those in +// prefer first, then the rest, each group shuffled with seed. n <= 0 picks +// all of them. +func SampleCallables(v *View, prefer []string, n int, seed uint64) []string { + r := rand.New(rand.NewPCG(seed, 2)) + taken := map[string]bool{} + pick := func(ids []string) []string { + var out []string + for _, id := range ids { + if c := v.Calls[id]; c != nil && c.L3 && !taken[id] { + taken[id] = true + out = append(out, id) + } + } + slices.Sort(out) + r.Shuffle(len(out), func(i, j int) { out[i], out[j] = out[j], out[i] }) + return out + } + out := pick(prefer) + if n <= 0 || len(out) < n { + out = append(out, pick(slices.Collect(maps.Keys(v.Calls)))...) + } + if n > 0 && len(out) > n { + out = out[:n] + } + slices.Sort(out) + return out +} + +// ReachesFrom reports whether any node of srcs reaches dst over g. +func (g Graph) ReachesFrom(srcs []string, dst string) bool { + for _, s := range srcs { + if g.Reach(s)[dst] { + return true + } + } + return false +} + +// Witness is a shortest path src → dst over g (nil when none). +func (g Graph) Witness(src, dst string) []string { + prev := map[string]string{src: ""} + q := []string{src} + for len(q) > 0 { + n := q[0] + q = q[1:] + if n == dst { + var p []string + for x := dst; x != ""; x = prev[x] { + p = append([]string{x}, p...) + } + return p + } + for _, d := range g[n] { + if _, ok := prev[d]; !ok { + prev[d] = n + q = append(q, d) + } + } + } + return nil +} diff --git a/internal/metamorphic/mr_test.go b/internal/metamorphic/mr_test.go new file mode 100644 index 0000000..f50eb3c --- /dev/null +++ b/internal/metamorphic/mr_test.go @@ -0,0 +1,797 @@ +package metamorphic + +// The metamorphic relations on the fixtures (testdata/dataflow's flow and +// sdg packages, and chi), run by go test ./... The same harness runs them on +// the prepared eval apps behind the `metamorphic` build tag +// (realapp_test.go, scripts/metamorphic.sh). +// +// Each relation transforms a copy of the module, analyzes it, and checks +// one of three oracles against the original's analysis: +// +// - exact: the analysis is unchanged up to the position map (MR1-MR4, +// MR7, MR15); +// - delta: it changes by exactly the delta each site declares (MR8-MR12), +// or by at most a declared removal (MR13, a precision relation); +// - reach: every flow of the original survives (MR5, MR6, MR14). +// +// Every relation logs one MRSTAT line (sites, skips, pairs checked), which +// scripts/metamorphic.sh collects. + +import ( + "fmt" + "go/ast" + "maps" + "os" + "path/filepath" + "reflect" + "runtime" + "slices" + "strings" + "sync" + "testing" + "time" +) + +// seed is the fixed seed of every sample and site choice. +const seed = 1 + +// target is one module the relations run on. +type target struct { + name string + dir string + // pkgDirs restricts the transformed files to these package directories + // (all when empty). + pkgDirs []string + // nFiles and nCallables bound the samples (all when 0). + nFiles, nCallables int + + once sync.Once + ws *Workspace + pre *analyzed // -a 4 + pre3 *analyzed // -a 3 + err error + mu sync.Mutex +} + +// analyzed is one analyzed module. +type analyzed struct { + view *View + errPkgs map[string]bool // packages with load errors + dur time.Duration +} + +func analyze(dir string, level int, targets []string) (*analyzed, error) { + start := time.Now() + r, err := Analyze(dir, level, targets) + if err != nil { + return nil, err + } + a := &analyzed{view: r.View, errPkgs: map[string]bool{}, dur: time.Since(start)} + for _, e := range r.Errors { + a.errPkgs[e[:strings.Index(e, ": ")]] = true + } + return a, nil +} + +func testdata(name string) string { + _, file, _, _ := runtime.Caller(0) + return filepath.Join(filepath.Dir(file), "..", "..", "testdata", name) +} + +var fixtures = []*target{ + {name: "dataflow", dir: testdata("dataflow"), pkgDirs: []string{"flow", "sdg"}}, + {name: "chi", dir: testdata("chi"), nFiles: 12, nCallables: 60}, +} + +// load loads the target's workspace and its original analysis once. +func (tg *target) load(t *testing.T) { + t.Helper() + tg.once.Do(func() { + tg.ws, tg.err = LoadWorkspace(tg.dir) + if tg.err == nil { + tg.pre, tg.err = analyze(tg.ws.Dir, 4, nil) + } + }) + if tg.err != nil { + t.Fatalf("%s: %v", tg.name, tg.err) + } +} + +func (tg *target) level3(t *testing.T) *analyzed { + tg.mu.Lock() + defer tg.mu.Unlock() + if tg.pre3 == nil { + a, err := analyze(tg.ws.Dir, 3, nil) + if err != nil { + t.Fatal(err) + } + tg.pre3 = a + } + return tg.pre3 +} + +// files is the sample of eligible files the relations transform. +func (tg *target) files() []string { + return Sample(tg.ws.Eligible(tg.pkgDirs...), tg.nFiles, seed) +} + +// relation is one metamorphic relation. +type relation struct { + name string + t Transform + oracle string // exact | delta | reach + fix Fix // spelling fix for the transformed view (MR4) +} + +// outcome is what one relation found on one target. +type outcome struct { + sites int + skipped map[string]int + violations []string + pairs int + summaries int // MR14a summary edges checked + dur time.Duration +} + +func (o *outcome) fail(format string, a ...any) { + o.violations = append(o.violations, fmt.Sprintf(format, a...)) +} + +// logStat writes the relation's MRSTAT line. +func logStat(t *testing.T, mr, tg string, o *outcome) { + t.Helper() + var sk []string + for _, k := range slices.Sorted(maps.Keys(o.skipped)) { + sk = append(sk, fmt.Sprintf("%s=%d", k, o.skipped[k])) + } + t.Logf("MRSTAT %s %s sites=%d pairs=%d summaries=%d violations=%d time=%s skipped=[%s]", + mr, tg, o.sites, o.pairs, o.summaries, len(o.violations), o.dur.Round(time.Millisecond), strings.Join(sk, "; ")) +} + +// transformAndAnalyze applies r to tg and analyzes the result at -a 4. A +// transformed module that type-checks worse than the original is a test +// bug and fails the test. +func transformAndAnalyze(t *testing.T, tg *target, tr Transform, ws *Workspace, files []string, pre *View) (*Result, *analyzed) { + t.Helper() + res, err := ws.Apply(tr, files, filepath.Join(t.TempDir(), "m"), seed, pre) + if err != nil { + t.Fatalf("%s on %s: %v", tr.Name, tg.name, err) + } + if len(res.Files) == 0 { + return res, nil + } + post, err := analyze(res.Dir, 4, nil) + if err != nil { + t.Fatalf("%s on %s: analyze: %v", tr.Name, tg.name, err) + } + checkTypes(t, tg, tr.Name, res, post) + return res, post +} + +// checkTypes fails when the transformed module has a package with load +// errors the original had none in. +func checkTypes(t *testing.T, tg *target, mr string, res *Result, post *analyzed) { + t.Helper() + for p := range post.errPkgs { + if !tg.pre.errPkgs[p] { + r, _ := Analyze(res.Dir, 1, nil) + var msgs []string + for _, e := range r.Errors { + if strings.HasPrefix(e, p+": ") { + msgs = append(msgs, e) + } + } + t.Fatalf("%s on %s: the transformed module does not type-check (a transform bug): %s (kept at %s)", + mr, tg.name, strings.Join(msgs, "; "), keep(t, res.Dir)) + } + } +} + +// keep copies a failing transformed module out of the test's temp dir. +func keep(t *testing.T, dir string) string { + dst, err := os.MkdirTemp("", "mr-fail-*") + if err != nil { + return dir + } + if err := copyTree(dir, dst); err != nil { + return dir + } + return dst +} + +// runRelation runs r on tg and reports its violations. +func runRelation(t *testing.T, tg *target, r relation) *outcome { + t.Helper() + tg.load(t) + start := time.Now() + o := &outcome{skipped: map[string]int{}} + defer func() { o.dur = time.Since(start) }() + pre := tg.pre.view + res, post := transformAndAnalyze(t, tg, r.t, tg.ws, tg.files(), pre) + maps.Copy(o.skipped, res.Skipped) + o.sites = len(res.Sites) + if post == nil { + return o + } + got := post.view + if r.fix != nil { + got = got.Apply(r.fix) + } + mapped, unmapped := pre.Map(res.Map) + if r.oracle != "reach" && len(unmapped) > 0 { + o.fail("keys of the original do not map: %s", strings.Join(head(unmapped, 10), ", ")) + return o + } + switch r.oracle { + case "exact": + for _, d := range Diff(mapped, got, Scope{}, 40) { + o.fail("%s", d) + } + case "delta": + checkDelta(o, pre, mapped, got, res) + case "reach": + checkReach(o, tg, pre, got, res) + } + if len(o.violations) > 0 { + t.Logf("%s on %s: transformed module kept at %s", r.name, tg.name, keep(t, res.Dir)) + } + return o +} + +func head(s []string, n int) []string { + if len(s) > n { + return append(slices.Clone(s[:n]), fmt.Sprintf("… and %d more", len(s)-n)) + } + return s +} + +// checkDelta is the delta oracle: the transformed view equals the original +// with every site's delta applied (MR8-MR12), and for a precision site +// (MR13) the site's checks hold. +func checkDelta(o *outcome, pre, mapped, got *View, res *Result) { + var ds []Delta + atMost := map[string]bool{} + for _, s := range res.Sites { + if s.Delta != nil { + ds = append(ds, *s.Delta) + if s.Delta.AtMost { + atMost[s.Delta.Callable] = true + } + } + } + want, err := Expect(pre, mapped, res.Map, res.Marks, ds) + if err != nil { + o.fail("relation: %v", err) + return + } + if len(atMost) == 0 { + for _, d := range Diff(want, got, Scope{}, 40) { + o.fail("%s", d) + } + return + } + // A precision relation: nothing added anywhere; outside the sites only + // summary edges may go; at a site, its declared removals and summary + // edges. + for _, d := range Diff(want, got, Scope{Skip: atMost}, 0) { + if strings.Contains(d, "|") && strings.HasSuffix(d, " missing") && strings.Contains(d, ": edge sum|") { + continue + } + o.fail("%s", d) + } + for _, d := range CheckAtMost(pre, mapped, got, res.Map, res.Marks, ds) { + if strings.Contains(d, ": sum|") && strings.HasSuffix(d, " removed") { + continue + } + o.fail("%s", d) + } + g := got.Graph(true) + l := newLocator(pre, got, res.Map) + for _, s := range res.Sites { + if len(s.NoFlow) == 0 { + continue + } + c := got.Calls[s.Callable] + var srcs []string + for _, k := range slices.Sorted(maps.Keys(c.Body)) { + if strings.HasPrefix(k, "@formal_in:") && !strings.HasPrefix(strings.Split(c.Body[k], "|")[2], ":") { + srcs = append(srcs, Node(s.Callable, k)) + } + } + for _, ref := range s.NoFlow { + dst, ok := l.locate(Node(s.Callable, ref.Pre)) + if !ok { + o.fail("%s: no counterpart of %s", s.Desc, ref.Pre) + continue + } + if g.ReachesFrom(srcs, dst) { + o.fail("%s: a formal_in still reaches %s (a constant argument cannot carry a flow)", s.Desc, dst) + } + } + } +} + +// checkReach is the reachability oracle, plus MR14's summary checks. +func checkReach(o *outcome, tg *target, pre, got *View, res *Result) { + var cut []string + for _, s := range res.Sites { + cut = append(cut, s.Cut...) + } + rr := CheckReach(pre, got, res.Map, res.Touched, tg.nCallables, seed, cut...) + o.pairs = rr.Pairs + gPre := pre.Graph(false) + for i, l := range head(rr.Lost, 40) { + msg := "lost flow " + l + if p := strings.SplitN(l, " -> ", 2); i < 5 && len(p) == 2 { + msg += "\n\t\twitness in the original: " + strings.Join(gPre.Witness(p[0], p[1]), "\n\t\t ") + } + o.fail("%s", msg) + } + for _, u := range head(rr.Unlocated, 20) { + o.fail("no counterpart: %s", u) + } + if rr.NLost > len(rr.Lost) || rr.NUnlocated > len(rr.Unlocated) { + o.fail("%d lost flows and %d unlocated nodes in all", rr.NLost, rr.NUnlocated) + } + l := newLocator(pre, got, res.Map) + for _, s := range res.Sites { + for _, sc := range s.Summary { + o.summaries++ + src, ok1 := l.locate(Node(s.Callable, sc.Src.Pre)) + dst, ok2 := l.locate(Node(s.Callable, sc.Dst.Pre)) + if !ok1 || !ok2 { + o.fail("%s: summary %s: no counterpart", s.Desc, sc.Pre) + continue + } + id, sk := splitNode(src) + _, dk := splitNode(dst) + if c := got.Calls[id]; c == nil || !c.Edges["sum|"+sk+"|"+dk+"|"] { + o.fail("%s: summary %s has no counterpart %s -> %s", s.Desc, sc.Pre, sk, dk) + } + } + } +} + +// ─── the relations ──────────────────────────────────────────────────────────── + +var relations = []relation{ + {name: "MR1", t: MR1, oracle: "exact"}, + {name: "MR2", t: MR2, oracle: "exact"}, + {name: "MR3", t: MR3, oracle: "exact"}, + {name: "MR4", t: MR4, oracle: "exact", fix: StripSuffix(Suffix4)}, + {name: "MR5", t: MR5, oracle: "reach"}, + {name: "MR6", t: MR6, oracle: "reach"}, + {name: "MR8", t: MR8, oracle: "delta"}, + {name: "MR9", t: MR9, oracle: "delta"}, + {name: "MR10", t: MR10, oracle: "delta"}, + {name: "MR11", t: MR11, oracle: "delta"}, + {name: "MR12", t: MR12, oracle: "delta"}, + {name: "MR13", t: MR13, oracle: "delta"}, + {name: "MR14", t: MR14, oracle: "reach"}, +} + +// knownSkips names the relation/target pairs skipped for a filed issue. +var knownSkips = map[string]string{} + +func runOn(t *testing.T, targets []*target, r relation, requireSites bool) { + sites, ran := 0, 0 + var mu sync.Mutex + t.Run(r.name, func(t *testing.T) { + for _, tg := range targets { + t.Run(tg.name, func(t *testing.T) { + if why, ok := knownSkips[r.name+"/"+tg.name]; ok { + t.Skip(why) + } + t.Parallel() + o := runRelation(t, tg, r) + mu.Lock() + sites += o.sites + ran++ + mu.Unlock() + logStat(t, r.name, tg.name, o) + for _, v := range o.violations { + t.Errorf("%s on %s: %s", r.name, tg.name, v) + } + }) + } + }) + if requireSites && ran == len(targets) && sites == 0 { + t.Errorf("%s exercised no site on the fixtures", r.name) + } +} + +func TestMR_Fixtures(t *testing.T) { + for _, r := range relations { + runOn(t, fixtures, r, true) + } + t.Run("MR7", func(t *testing.T) { + for _, tg := range fixtures { + t.Run(tg.name, func(t *testing.T) { + t.Parallel() + o := runMR7(t, tg) + logStat(t, "MR7", tg.name, o) + for _, v := range o.violations { + t.Errorf("MR7 on %s: %s", tg.name, v) + } + }) + } + }) + t.Run("MR15", func(t *testing.T) { + for _, tg := range fixtures { + t.Run(tg.name, func(t *testing.T) { + t.Parallel() + o := runMR15(t, tg) + logStat(t, "MR15", tg.name, o) + for _, v := range o.violations { + t.Errorf("MR15 on %s: %s", tg.name, v) + } + }) + } + }) +} + +// runMR7 is MR7: --target-files at -a 3 on one sampled file equals the +// full -a 3 run on that file's callables. +func runMR7(t *testing.T, tg *target) *outcome { + tg.load(t) + start := time.Now() + o := &outcome{skipped: map[string]int{}} + defer func() { o.dur = time.Since(start) }() + full := tg.level3(t) + for _, rel := range Sample(tg.files(), 2, seed) { + part, err := analyze(tg.ws.Dir, 3, []string{filepath.Join(tg.ws.Dir, rel)}) + if err != nil { + t.Fatal(err) + } + only := map[string]bool{} + for id, c := range full.view.Calls { + if c.File == rel { + only[id] = true + } + } + o.sites++ + for _, d := range Diff(full.view, part.view, Scope{Only: only}, 40) { + o.fail("%s: %s", rel, d) + } + } + return o +} + +// runMR15 is MR15: MR6 then Inline6 gives back the original source, byte +// for byte, and the original analysis exactly. +func runMR15(t *testing.T, tg *target) *outcome { + tg.load(t) + start := time.Now() + o := &outcome{skipped: map[string]int{}} + defer func() { o.dur = time.Since(start) }() + files := tg.files() + ext, err := tg.ws.Apply(MR6, files, filepath.Join(t.TempDir(), "x"), seed, nil) + if err != nil { + t.Fatal(err) + } + maps.Copy(o.skipped, ext.Skipped) + if len(ext.Files) == 0 { + return o + } + ws2, err := LoadWorkspace(ext.Dir) + if err != nil { + t.Fatal(err) + } + inl, err := ws2.Apply(Inline6, ext.Files, filepath.Join(t.TempDir(), "i"), seed, nil) + if err != nil { + t.Fatal(err) + } + o.sites = len(inl.Sites) + for _, rel := range ext.Files { + a, _ := os.ReadFile(filepath.Join(tg.ws.Dir, rel)) + b, _ := os.ReadFile(filepath.Join(inl.Dir, rel)) + if string(a) != string(b) { + o.fail("%s: inlining the MR6 helpers does not give back the original source", rel) + } + } + post, err := analyze(inl.Dir, 4, nil) + if err != nil { + t.Fatal(err) + } + mapped, unmapped := tg.pre.view.Map(Compose(ext.Map, inl.Map)) + if len(unmapped) > 0 { + o.fail("keys do not map: %s", strings.Join(head(unmapped, 10), ", ")) + } + for _, d := range Diff(mapped, post.view, Scope{}, 40) { + o.fail("%s", d) + } + return o +} + +// ─── harness tests (Step 1) ─────────────────────────────────────────────────── + +// TestPosMap_RoundTrip: every transform's PosMap maps each old statement and +// call it keeps to a node of the same kind, in the transformed module (which +// type-checks: LoadWorkspace would report its errors). +func TestPosMap_RoundTrip(t *testing.T) { + tg := fixtures[0] + tg.load(t) + for _, tr := range []Transform{MR1, MR2, MR3, MR4, MR5, MR6, MR8, MR9, MR10, MR11, MR12, MR13, MR14} { + t.Run(tr.Name, func(t *testing.T) { + t.Parallel() + res, err := tg.ws.Apply(tr, tg.files(), filepath.Join(t.TempDir(), "m"), seed, tg.pre.view) + if err != nil { + t.Fatal(err) + } + if len(res.Files) == 0 { + t.Fatalf("%s changed no file", tr.Name) + } + ws2, err := LoadWorkspace(res.Dir) + if err != nil { + t.Fatal(err) + } + for _, f := range ws2.Files { + if f.Skip == "package has errors" { + t.Fatalf("%s: %s does not type-check: %v", tr.Name, f.Rel, f.Pkg.Errors) + } + } + checked := 0 + for _, rel := range res.Files { + old := tg.ws.Files[rel] + ast.Inspect(old.AST, func(n ast.Node) bool { + switch n.(type) { + case ast.Stmt, *ast.CallExpr: + default: + return true + } + if _, isBlock := n.(*ast.BlockStmt); isBlock { + return true + } + p, ok := res.Map.Map(old.pos(n.Pos())) + if !ok { + return true // deleted or replaced + } + nf := ws2.Files[p.File] + if nf == nil { + t.Errorf("%s: %v maps to %s, not a file", tr.Name, old.pos(n.Pos()), p.File) + return true + } + want := reflect.TypeOf(n) + if !nodeAt(nf, p, want) && !kindChanged(tr, n, want) { + t.Errorf("%s: %s %v maps to %v, which holds no %v", tr.Name, want, old.pos(n.Pos()), p, want) + } + checked++ + return true + }) + } + if checked == 0 { + t.Errorf("%s: nothing checked", tr.Name) + } + }) + } +} + +// kindChanged reports whether tr changes the kind of statement n on +// purpose: MR14 defers a call statement, so the expression statement +// becomes a defer statement. +func kindChanged(tr Transform, n ast.Node, want reflect.Type) bool { + return tr.Name == "MR14" && want == reflect.TypeOf(&ast.ExprStmt{}) +} + +// nodeAt reports whether f has a node of type want starting at p. +func nodeAt(f *SrcFile, p Pos, want reflect.Type) bool { + off, ok := f.lines.offset(p.Line, p.Col) + if !ok { + return false + } + found := false + ast.Inspect(f.AST, func(n ast.Node) bool { + if n == nil || found { + return false + } + if f.off(n.Pos()) == off && reflect.TypeOf(n) == want { + found = true + } + return f.off(n.Pos()) <= off && off < f.off(n.End())+1 + }) + return found +} + +// TestCompare_DetectsChange: each oracle flags a hand-made change. An +// untransformed copy compares equal; removing one ddg edge fails the exact +// oracle; a delta naming that removal passes; and removing a param_in edge +// that carried a flow fails the reachability oracle. +func TestCompare_DetectsChange(t *testing.T) { + tg := fixtures[0] + tg.load(t) + pre := tg.pre.view + cp, err := analyze(copyOf(t, tg.ws.Dir), 4, nil) + if err != nil { + t.Fatal(err) + } + if d := Diff(pre, cp.view, Scope{}, 10); len(d) > 0 { + t.Fatalf("an untransformed copy differs: %v", d) + } + + id, edge := firstEdge(pre, "ddg") + got := cloneView(pre) + delete(got.Calls[id].Edges, edge) + d := Diff(pre, got, Scope{}, 0) + if len(d) != 1 || !strings.Contains(d[0], edge) || !strings.HasSuffix(d[0], "missing") { + t.Fatalf("exact oracle on a one-edge removal: %v", d) + } + f := strings.Split(edge, "|") + want, err := Expect(pre, pre, nil, nil, []Delta{{Site: "test", Callable: id, + Remove: []DEdge{{Kind: "ddg", Src: Pre(f[1]), Dst: Pre(f[2]), Label: f[3], Prov: f[4]}}}}) + if err != nil { + t.Fatal(err) + } + if d := Diff(want, got, Scope{}, 0); len(d) > 0 { + t.Errorf("delta oracle with the removal declared: %v", d) + } + if d := Diff(want, pre, Scope{}, 0); len(d) != 1 { + t.Errorf("delta oracle with the removal declared, edge kept: %v", d) + } + if _, err := Expect(pre, got, nil, nil, []Delta{{Site: "test", Callable: id, + Remove: []DEdge{{Kind: "ddg", Src: Pre(f[1]), Dst: Pre(f[2]), Label: f[3], Prov: f[4]}}}}); err == nil { + t.Error("a delta removing an edge the original lacks is not reported") + } + + // Reachability: drop every param_in edge; some flow is lost. + cut := cloneView(pre) + for e := range cut.App { + if strings.HasPrefix(e, "pin|") { + delete(cut.App, e) + } + } + if rr := CheckReach(pre, cut, nil, nil, 0, seed); len(rr.Lost) == 0 || rr.Pairs == 0 { + t.Errorf("reachability oracle without param_in: %d pairs, none lost", rr.Pairs) + } + if rr := CheckReach(pre, pre, nil, nil, 0, seed); len(rr.Lost) > 0 || len(rr.Unlocated) > 0 { + t.Errorf("reachability oracle on itself: lost %v, unlocated %v", head(rr.Lost, 5), head(rr.Unlocated, 5)) + } +} + +// TestRefCDG: the analyzer's CDG equals the reference CDG of its CFG on +// every fixture callable (the delta relations derive expected CDGs with it). +func TestRefCDG(t *testing.T) { + for _, tg := range fixtures { + tg.load(t) + n := 0 + for _, id := range slices.Sorted(maps.Keys(tg.pre.view.Calls)) { + c := tg.pre.view.Calls[id] + if !c.L3 { + continue + } + n++ + if d := cdgDiff(c, RefCDG(c)); d != "" { + t.Errorf("%s %s: cdg differs from the reference: %s", tg.name, id, d) + } + } + if n < 50 { + t.Errorf("%s: only %d callables", tg.name, n) + } + } +} + +func firstEdge(v *View, kind string) (string, string) { + for _, id := range slices.Sorted(maps.Keys(v.Calls)) { + for _, e := range slices.Sorted(maps.Keys(v.Calls[id].Edges)) { + if strings.HasPrefix(e, kind+"|") { + return id, e + } + } + } + return "", "" +} + +func cloneView(v *View) *View { + out := &View{Calls: map[string]*CV{}, App: maps.Clone(v.App)} + for id, c := range v.Calls { + out.Calls[id] = c.clone() + } + return out +} + +func copyOf(t *testing.T, dir string) string { + dst := filepath.Join(t.TempDir(), "copy") + if err := copyTree(dir, dst); err != nil { + t.Fatal(err) + } + return dst +} + +// ─── one relation per oracle, on one named fixture site ─────────────────────── +// +// The table above runs every relation on whole samples. These three pin one +// site each, so a relation's shape reads end to end. + +// TestMR2_ReverseDeclarations_Exact: reversing sdg/stdlib.go's declarations +// leaves the -a 4 analysis exactly as it was, once every key is moved to its +// statement's new line. +func TestMR2_ReverseDeclarations_Exact(t *testing.T) { + tg := fixtures[0] + tg.load(t) + res, post := transformAndAnalyze(t, tg, MR2, tg.ws, []string{"sdg/stdlib.go"}, tg.pre.view) + mapped, unmapped := tg.pre.view.Map(res.Map) + if len(unmapped) > 0 { + t.Fatalf("keys do not map: %v", unmapped) + } + if d := Diff(mapped, post.view, Scope{}, 20); len(d) > 0 { + t.Errorf("MR2 changed the analysis:\n%s", strings.Join(d, "\n")) + } +} + +// TestMR9_InsertKill_Delta: in ScalarSink, inserting `x = 0` before +// `sinkInt(x)` moves every def edge of `x := a + 1` on x to the kill and +// splices the kill into the CFG; the CDG is the reference CDG of that CFG, +// and nothing else in the application changes. +func TestMR9_InsertKill_Delta(t *testing.T) { + tg := fixtures[0] + tg.load(t) + res, post := transformAndAnalyze(t, tg, MR9, tg.ws, []string{"sdg/sdg.go"}, tg.pre.view) + var site *Site + for i, s := range res.Sites { + if strings.HasSuffix(s.Callable, ".ScalarSink") { + site = &res.Sites[i] + } + } + if site == nil { + t.Fatal("MR9 chose no site in ScalarSink") + } + var k Ref // the kill, a node the transform adds (its one mark) + for m := range site.Delta.Nodes { + k = Mark(m) + } + want := Delta{Site: site.Desc, Callable: site.Callable, CDGFromCFG: true, + Nodes: site.Delta.Nodes, + Remove: []DEdge{ + DDG(Pre("509:2"), Pre("510:2"), "x"), + DDG(Pre("509:2"), Pre("511:2"), "x"), + CFG(Pre("509:2"), Pre("510:2"), "fallthrough"), + }, + Add: []DEdge{ + DDG(k, Pre("510:2"), "x"), + DDG(k, Pre("511:2"), "x"), + CFG(Pre("509:2"), k, "fallthrough"), + CFG(k, Pre("510:2"), "fallthrough"), + }, + } + mapped, _ := tg.pre.view.Map(res.Map) + var ds []Delta + for _, s := range res.Sites { + if s.Callable == want.Callable { + ds = append(ds, want) + } else { + ds = append(ds, *s.Delta) + } + } + exp, err := Expect(tg.pre.view, mapped, res.Map, res.Marks, ds) + if err != nil { + t.Fatal(err) + } + if d := Diff(exp, post.view, Scope{}, 20); len(d) > 0 { + t.Errorf("MR9 delta:\n%s", strings.Join(d, "\n")) + } +} + +// TestMR5_IntroduceAlias_Reach: rewriting `q.n = v` in aliasAssign as +// `mr5q := q; mr5q.n = v` (an alias of an alias) keeps every flow from the +// formal_ins of aliasAssign and of its caller CallAliasAssign: v still +// reaches the formal_out of p. +func TestMR5_IntroduceAlias_Reach(t *testing.T) { + tg := fixtures[0] + tg.load(t) + res, post := transformAndAnalyze(t, tg, MR5, tg.ws, []string{"sdg/sdg.go"}, tg.pre.view) + f := "example.com/dataflow/sdg.aliasAssign" + if !slices.Contains(res.Touched, f) { + t.Fatalf("MR5 did not rewrite %s", f) + } + rr := CheckReach(tg.pre.view, post.view, res.Map, []string{f, "example.com/dataflow/sdg.CallAliasAssign"}, 2, seed) + if rr.Pairs == 0 { + t.Fatal("no flow checked") + } + if len(rr.Lost) > 0 || len(rr.Unlocated) > 0 { + t.Errorf("MR5: lost %v, unlocated %v", rr.Lost, rr.Unlocated) + } + // The flow the alias carries: v (formal_in 1) → p's formal_out. + g := post.view.Graph(true) + if !g.Reach(Node(f, "@formal_in:1"))[Node(f, "@formal_out")] { + t.Error("after MR5, v no longer reaches aliasAssign's formal_out for p") + } +} diff --git a/internal/metamorphic/realapp_test.go b/internal/metamorphic/realapp_test.go new file mode 100644 index 0000000..a5f43b9 --- /dev/null +++ b/internal/metamorphic/realapp_test.go @@ -0,0 +1,82 @@ +//go:build metamorphic + +package metamorphic + +// The metamorphic relations on the prepared eval apps (scripts/eval_l3.sh +// --prepare memos nats-server), run by scripts/metamorphic.sh: +// +// go test -tags metamorphic -run TestMR_RealApps ./internal/metamorphic/ +// +// Each app is $EVAL_SRC/ (default /tmp/eval). A missing app is +// skipped. MR_APPS (space-separated) picks the apps, MR_RELATIONS the +// relations (all when unset). Each relation samples 20 files and 50 +// callables with seed 1. + +import ( + "os" + "path/filepath" + "slices" + "strings" + "testing" +) + +func realTargets(t *testing.T) []*target { + src := os.Getenv("EVAL_SRC") + if src == "" { + src = "/tmp/eval" + } + names := strings.Fields(os.Getenv("MR_APPS")) + if len(names) == 0 { + names = []string{"memos", "nats-server"} + } + var out []*target + for _, n := range names { + dir := filepath.Join(src, n) + if _, err := os.Stat(filepath.Join(dir, "go.mod")); err != nil { + t.Logf("%s: not prepared at %s; skipped", n, dir) + continue + } + out = append(out, &target{name: n, dir: dir, nFiles: 20, nCallables: 50}) + } + return out +} + +func TestMR_RealApps(t *testing.T) { + tgs := realTargets(t) + if len(tgs) == 0 { + t.Skip("no prepared eval app under $EVAL_SRC") + } + want := strings.Fields(os.Getenv("MR_RELATIONS")) + pick := func(name string) bool { return len(want) == 0 || slices.Contains(want, name) } + for _, tg := range tgs { + t.Run(tg.name, func(t *testing.T) { + tg.load(t) + t.Logf("MRLOAD %s files=%d eligible=%d skipped=%v analysis=%s", + tg.name, len(tg.ws.Files), len(tg.ws.Eligible()), tg.ws.SkipCounts(), tg.pre.dur.Round(1e6)) + for _, r := range relations { + if !pick(r.name) { + continue + } + t.Run(r.name, func(t *testing.T) { + if why, ok := knownSkips[r.name+"/"+tg.name]; ok { + t.Skip(why) + } + o := runRelation(t, tg, r) + logStat(t, r.name, tg.name, o) + for _, v := range o.violations { + t.Errorf("%s on %s: %s", r.name, tg.name, v) + } + }) + } + if pick("MR15") { + t.Run("MR15", func(t *testing.T) { + o := runMR15(t, tg) + logStat(t, "MR15", tg.name, o) + for _, v := range o.violations { + t.Errorf("MR15 on %s: %s", tg.name, v) + } + }) + } + }) + } +} diff --git a/internal/metamorphic/run.go b/internal/metamorphic/run.go new file mode 100644 index 0000000..076629b --- /dev/null +++ b/internal/metamorphic/run.go @@ -0,0 +1,100 @@ +// Package metamorphic holds the metamorphic relations (MRs) over the L3/L4 +// analysis: source transforms that must leave the analysis unchanged up to a +// known renaming (the exact oracle), or must keep every flow the original had +// (the one-way reachability oracle). Fixtures only catch an invariance or +// soundness bug when someone writes that exact case; the relations check it +// on any module, real apps included (spec §8, Metamorphic relations). +package metamorphic + +import ( + "fmt" + "io" + "path/filepath" + "sort" + + "github.com/codellm-devkit/codeanalyzer-go/internal/analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" +) + +// AppName is the --app-name every run uses, so both sides of a relation +// mint the same can:// prefix (external ids included). +const AppName = "mr" + +// Run is one in-process analysis: its normalized view and the load errors +// of the project packages (from the analyzer's own go/packages load). +type Run struct { + View *View + Errors []string +} + +// Analyze runs the analyzer pipeline over dir at level (3 or 4) the way +// core.Analyzer.Analyze does — symbol table, call graph, pass pipeline, +// dataflow, and at level 4 sdg.Run with its call-graph additions — minus +// `go mod download` (the inputs are already materialized), and emits the v2 +// payload. targets is --target-files (absolute paths); nil is a full run. +// Test files are skipped (the CLI default). +func Analyze(dir string, level int, targets []string) (*Run, error) { + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(targets, true) + if err != nil { + return nil, fmt.Errorf("symbol table: %w", err) + } + var errs []string + for _, p := range b.Pkgs() { + for _, e := range p.Errors { + errs = append(errs, p.PkgPath+": "+e.Msg) + } + } + sort.Strings(errs) + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + app := &schema.GoApplication{SymbolTable: st, CallGraph: cg.Build(st), Externals: cg.Externals(), Entrypoints: map[string][]schema.GoEntrypoint{}} + app.Externals = cg.Externals() + if err := analysis.RunPipeline(app, analysis.AnalysisContext{ProjectDir: dir}); err != nil { + return nil, err + } + units := b.L3Units(st) + const k, jobs = 3, 4 + var res map[dataflow.L3Key]dataflow.Result + var l4 *sdg.Result + if level >= 4 { + res = dataflow.RunWithSideTables(b.Fset(), units, k, jobs) + r := sdg.Run(sdg.Input{ + Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Externals: app.Externals, CallGraph: app.CallGraph, + Units: units, L3: res, K: k, Jobs: jobs, + }) + r.ApplyTo(app) + l4 = &r + } else { + res = dataflow.Run(b.Fset(), units, k, jobs) + } + dataflow.ReleaseAll(res) + // The payload is streamed (EmitTo) into the view one module at a time, + // so a large app's tree never exists in memory at once. + pr, pw := io.Pipe() + go func() { + pw.CloseWithError(v2emit.EmitTo(pw, app, AppName, dir, level, "metamorphic", k, res, l4)) + }() + v, err := Collect(pr) + pr.CloseWithError(err) + if err != nil { + return nil, err + } + return &Run{View: v, Errors: errs}, nil +} + +// realDir is dir with symlinks resolved: go/packages reports resolved file +// names, and the analyzer keys files relative to the input root, so a root +// under a symlink (macOS /var → /private/var) would put every file outside it. +func realDir(dir string) (string, error) { + abs, err := filepath.Abs(dir) + if err != nil { + return "", err + } + return filepath.EvalSymlinks(abs) +} diff --git a/internal/metamorphic/transform.go b/internal/metamorphic/transform.go new file mode 100644 index 0000000..23fb208 --- /dev/null +++ b/internal/metamorphic/transform.go @@ -0,0 +1,1429 @@ +package metamorphic + +// Source transforms. Each one rewrites some files of a module copy and +// returns a PosMap from every old position it kept to its new position, the +// marks it set in its new text, and its sites (what the relations check). +// +// The transforms find their sites with go/ast and go/types. The workspace is +// loaded with the analyzer's own go/packages mode. They never re-print a +// whole file: go/printer would reformat it and move free-floating comments, +// and with them every position in the file. Instead a new file is spliced +// from byte ranges of the old one, copied verbatim, and new text. That keeps +// the PosMap exact: a copied byte maps by its offset in the copy, and the +// start of a replaced range (an anchor) maps to the start of its +// replacement. + +import ( + "bytes" + "errors" + "fmt" + "go/ast" + "go/token" + "go/types" + "io/fs" + "math/rand/v2" + "os" + "path/filepath" + "slices" + "sort" + "strconv" + "strings" + + "golang.org/x/tools/go/packages" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// ─── positions ──────────────────────────────────────────────────────────────── + +// Pos is a source position: a file relative to the module root, a 1-based +// line and a 1-based byte column (the go/token convention body keys use). +type Pos struct { + File string + Line, Col int +} + +// Key is p's body key, "line:col". +func (p Pos) Key() string { return v2.LocalID(p.Line, p.Col) } + +// lineTable holds the offset of each line start of a file. +type lineTable []int + +func newLineTable(src []byte) lineTable { + t := lineTable{0} + for i, c := range src { + if c == '\n' { + t = append(t, i+1) + } + } + return t +} + +func (t lineTable) offset(line, col int) (int, bool) { + if line < 1 || line > len(t) || col < 1 { + return 0, false + } + return t[line-1] + col - 1, true +} + +func (t lineTable) pos(off int) (int, int) { + i := sort.Search(len(t), func(i int) bool { return t[i] > off }) - 1 + return i + 1, off - t[i] + 1 +} + +// seg maps old bytes [a, z) of a file to newFile at offset at (a copy), or, +// for an anchor, the single old offset a to at. +type seg struct { + a, z int + newFile string + at int +} + +// PosMap maps old positions to new ones. A file no transform rewrote maps to +// itself. In a rewritten file a position maps when its byte was copied or it +// is an anchor, and is unmapped (deleted or replaced text) otherwise. Every +// old byte is copied at most once, so the copies of a file are disjoint. +type PosMap struct { + old map[string]lineTable + new map[string]lineTable + copies map[string][]seg // by old file, sorted by a + anchors map[string]map[int]seg +} + +func newPosMap() *PosMap { + return &PosMap{old: map[string]lineTable{}, new: map[string]lineTable{}, + copies: map[string][]seg{}, anchors: map[string]map[int]seg{}} +} + +// Map returns p's new position. +func (m *PosMap) Map(p Pos) (Pos, bool) { + lt, ok := m.old[p.File] + if !ok { + return p, true + } + off, ok := lt.offset(p.Line, p.Col) + if !ok { + return Pos{}, false + } + s, ok := m.anchors[p.File][off] + if !ok { + cs := m.copies[p.File] + i := sort.Search(len(cs), func(i int) bool { return cs[i].a > off }) - 1 + if i < 0 || off >= cs[i].z { + return Pos{}, false + } + s = cs[i] + } + l, c := m.new[s.newFile].pos(s.at + off - s.a) + return Pos{File: s.newFile, Line: l, Col: c}, true +} + +// ─── workspace ──────────────────────────────────────────────────────────────── + +// Workspace is a module loaded for transformation: every project file with +// its syntax, types and source. +type Workspace struct { + Dir string + Fset *token.FileSet + Files map[string]*SrcFile // by path relative to Dir +} + +// SrcFile is one project file. +type SrcFile struct { + Rel string + Pkg *packages.Package + AST *ast.File + Src []byte + TF *token.File + lines lineTable + // Skip is why the file is not eligible for transformation ("" when it is). + Skip string +} + +func (f *SrcFile) off(p token.Pos) int { return f.TF.Offset(p) } + +// pos is the old position of p. +func (f *SrcFile) pos(p token.Pos) Pos { + l, c := f.lines.pos(f.off(p)) + return Pos{File: f.Rel, Line: l, Col: c} +} + +// key is the body key of p. +func (f *SrcFile) key(p token.Pos) string { return f.pos(p).Key() } + +// LoadWorkspace loads the module at dir with the analyzer's go/packages mode +// (test files excluded, as at the CLI default). +func LoadWorkspace(dir string) (*Workspace, error) { + dir, err := realDir(dir) + if err != nil { + return nil, err + } + fset := token.NewFileSet() + cfg := &packages.Config{ + Mode: packages.NeedName | packages.NeedFiles | packages.NeedSyntax | packages.NeedTypes | + packages.NeedTypesInfo | packages.NeedImports | packages.NeedDeps, + Dir: dir, Fset: fset, + } + pkgs, err := packages.Load(cfg, "./...") + if err != nil { + return nil, err + } + w := &Workspace{Dir: dir, Fset: fset, Files: map[string]*SrcFile{}} + for _, p := range pkgs { + for _, af := range p.Syntax { + tf := fset.File(af.Pos()) + if tf == nil || !utils.IsWithin(dir, tf.Name()) { + continue + } + rel := filepath.ToSlash(utils.RelativePath(dir, tf.Name())) + if utils.IsTestFile(rel) || utils.IsVendored(rel) { + continue + } + src, err := os.ReadFile(tf.Name()) + if err != nil { + return nil, err + } + f := &SrcFile{Rel: rel, Pkg: p, AST: af, Src: src, TF: tf, lines: newLineTable(src)} + f.Skip = skipReason(f) + w.Files[rel] = f + } + } + return w, nil +} + +// skipReason is why f is not transformed: cgo, //line directives and +// generated code (spans follow other rules there), type errors in its +// package (the transforms need its types), or a source that does not match +// its parse. +func skipReason(f *SrcFile) string { + switch { + case len(f.Pkg.Errors) > 0: + return "package has errors" + case f.TF.Size() != len(f.Src): + return "source differs from its parse" + case ast.IsGenerated(f.AST): + return "generated" + case f.Pkg.TypesInfo == nil: + return "no types" + } + for _, s := range f.AST.Imports { + if s.Path.Value == `"C"` { + return "cgo" + } + } + for _, cg := range f.AST.Comments { + for _, c := range cg.List { + if strings.HasPrefix(c.Text, "//line ") || strings.HasPrefix(c.Text, "/*line ") { + return "//line directive" + } + } + } + return "" +} + +// Eligible returns the transformable files, sorted, restricted to the +// package directories in dirs when dirs is non-empty. +func (w *Workspace) Eligible(dirs ...string) []string { + var out []string + for rel, f := range w.Files { + if f.Skip != "" { + continue + } + if len(dirs) > 0 && !slices.Contains(dirs, filepath.ToSlash(filepath.Dir(rel))) { + continue + } + out = append(out, rel) + } + sort.Strings(out) + return out +} + +// SkipCounts counts the ineligible files by reason. +func (w *Workspace) SkipCounts() map[string]int { + out := map[string]int{} + for _, f := range w.Files { + if f.Skip != "" { + out[f.Skip]++ + } + } + return out +} + +// Sample returns n of files chosen with seed (all when n <= 0 or n >= len). +func Sample(files []string, n int, seed uint64) []string { + out := slices.Clone(files) + sort.Strings(out) + if n <= 0 || n >= len(out) { + return out + } + r := rand.New(rand.NewPCG(seed, 0)) + r.Shuffle(len(out), func(i, j int) { out[i], out[j] = out[j], out[i] }) + out = out[:n] + sort.Strings(out) + return out +} + +// ─── transform driver ───────────────────────────────────────────────────────── + +// Transform is one source transform, applied file by file. edit returns +// nil, or errSkip, to leave a file unchanged. +type Transform struct { + Name string + edit func(c *tctx, f *SrcFile) (*fileEdit, error) +} + +// piece is a run of a new file's text: a literal, a copy of old bytes +// [a, z) of f, an anchor (old offset a of f maps to here; writes nothing) +// or a mark (records here under its name; writes nothing). +type piece struct { + lit string + f *SrcFile + a, z int + anchor bool + mark string +} + +func lit(s string) piece { return piece{lit: s} } +func cp(f *SrcFile, a, z int) piece { return piece{f: f, a: a, z: z} } +func anchorAt(f *SrcFile, a int) piece { return piece{f: f, a: a, z: a, anchor: true} } +func mark(name string) piece { return piece{mark: name} } + +type fileEdit struct { + pieces []piece // the old file's new text + created []newFile // files added next to it + touched []string // Identity of every callable changed + sites []Site + // skips counts the sites declined, by reason. + skips map[string]int +} + +func (e *fileEdit) skip(reason string) { + if e.skips == nil { + e.skips = map[string]int{} + } + e.skips[reason]++ +} + +type newFile struct { + rel string + pieces []piece +} + +// Site is one place a transform changed, with what its relation expects. +type Site struct { + Desc string // file:line and what was done + Callable string // Identity, in the original + // Delta is the expected change (delta and precision relations). + Delta *Delta + // NoFlow lists vertices (refs into Callable) that no @formal_in of + // Callable may reach over data edges after the transform (MR13). + NoFlow []Ref + // Reloc maps a call key of Callable in the original to its call node + // after the transform, in RelocTo (MR14a: the call moved into a method). + Reloc map[string]Ref + RelocTo string + // Summary lists summary edges that must exist after the transform in + // Callable, each the counterpart of one in the original (MR14a). + Summary []SumCheck + // Cut lists the Nodes of the original whose out-edges the reachability + // oracle leaves out of the original's graph (MR14b, see there). + Cut []string +} + +// SumCheck is one expected summary edge. +type SumCheck struct { + Pre string // the original edge, for the message + Src, Dst Ref +} + +// tctx is the per-run state of a transform. +type tctx struct { + w *Workspace + pre *View // the original's analysis (nil when the transform needs none) + rng *rand.Rand + n int // a counter for fresh names and marks +} + +func (c *tctx) fresh(prefix string) string { + c.n++ + return prefix + strconv.Itoa(c.n) +} + +// Result is a transformed module. +type Result struct { + Dir string // the transformed copy's root + Map *PosMap // old → new positions + Marks map[string]Pos + Files []string // rewritten + Created []string + // Touched is the Identity of every callable a transform changed, sorted. + Touched []string + Sites []Site + // Skipped counts, per reason, the files and sites left unchanged. + Skipped map[string]int +} + +// errSkip is returned by an edit that leaves its file unchanged. +type errSkip string + +func (e errSkip) Error() string { return string(e) } + +// Apply copies the module to dst, applies t to files with seed and returns +// the result. pre is the original's analysis, which the delta transforms +// read to choose and describe their sites. +func (w *Workspace) Apply(t Transform, files []string, dst string, seed uint64, pre *View) (*Result, error) { + if err := copyTree(w.Dir, dst); err != nil { + return nil, err + } + dst, err := realDir(dst) + if err != nil { + return nil, err + } + c := &tctx{w: w, pre: pre, rng: rand.New(rand.NewPCG(seed, 1))} + res := &Result{Dir: dst, Skipped: map[string]int{}, Map: newPosMap(), Marks: map[string]Pos{}} + touched := map[string]bool{} + for _, rel := range files { + f := w.Files[rel] + if f == nil || f.Skip != "" { + return nil, fmt.Errorf("%s: not an eligible file", rel) + } + fe, err := t.edit(c, f) + var skip errSkip + if errors.As(err, &skip) { + res.Skipped["file: "+string(skip)]++ + if fe != nil { + for r, n := range fe.skips { + res.Skipped["site: "+r] += n + } + } + continue + } + if err != nil { + return nil, fmt.Errorf("%s %s: %w", t.Name, rel, err) + } + if fe == nil || fe.pieces == nil { + res.Skipped["file: no site"]++ + continue + } + for r, n := range fe.skips { + res.Skipped["site: "+r] += n + } + if err := res.write(dst, rel, fe.pieces); err != nil { + return nil, err + } + res.Files = append(res.Files, rel) + for _, nf := range fe.created { + if _, err := os.Stat(filepath.Join(dst, nf.rel)); err == nil { + return nil, fmt.Errorf("%s: %s exists", t.Name, nf.rel) + } + if err := res.write(dst, nf.rel, nf.pieces); err != nil { + return nil, err + } + res.Created = append(res.Created, nf.rel) + } + for _, id := range fe.touched { + touched[id] = true + } + res.Sites = append(res.Sites, fe.sites...) + } + res.Touched = slices.Sorted(func(yield func(string) bool) { + for id := range touched { + if !yield(id) { + return + } + } + }) + for _, s := range res.Map.copies { + sort.Slice(s, func(i, j int) bool { return s[i].a < s[j].a }) + } + return res, nil +} + +// write assembles pieces into dst/rel and records their segments and marks. +func (r *Result) write(dst, rel string, ps []piece) error { + var buf bytes.Buffer + marks := map[string]int{} + for _, p := range ps { + switch { + case p.mark != "": + if _, dup := r.Marks[p.mark]; dup { + return fmt.Errorf("%s: duplicate mark %s", rel, p.mark) + } + marks[p.mark] = buf.Len() + continue + case p.f == nil: + buf.WriteString(p.lit) + continue + case p.a < 0 || p.z > len(p.f.Src) || p.a > p.z: + return fmt.Errorf("%s: bad piece [%d,%d) of %s", rel, p.a, p.z, p.f.Rel) + } + old := p.f.Rel + if _, ok := r.Map.old[old]; !ok { + r.Map.old[old] = p.f.lines + r.Map.anchors[old] = map[int]seg{} + } + s := seg{a: p.a, z: p.z, newFile: rel, at: buf.Len()} + if p.anchor { + r.Map.anchors[old][p.a] = s + continue + } + if p.z > p.a { + r.Map.copies[old] = append(r.Map.copies[old], s) + buf.Write(p.f.Src[p.a:p.z]) + } + } + lt := newLineTable(buf.Bytes()) + r.Map.new[rel] = lt + for name, off := range marks { + l, c := lt.pos(off) + r.Marks[name] = Pos{File: rel, Line: l, Col: c} + } + return os.WriteFile(filepath.Join(dst, rel), buf.Bytes(), 0o644) +} + +// copyTree copies the regular files under src to dst, skipping VCS and +// JavaScript dependency trees. +func copyTree(src, dst string) error { + return filepath.WalkDir(src, func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + rel, _ := filepath.Rel(src, p) + if d.IsDir() { + if n := d.Name(); rel != "." && (n == ".git" || n == "node_modules") { + return filepath.SkipDir + } + return os.MkdirAll(filepath.Join(dst, rel), 0o755) + } + if !d.Type().IsRegular() { + return nil + } + data, err := os.ReadFile(p) + if err != nil { + return err + } + return os.WriteFile(filepath.Join(dst, rel), data, 0o644) + }) +} + +// edit is one change to a file: old bytes [a, z) replaced by out. With +// anchor, old offset a maps to the start of the replacement. +type edit struct { + a, z int + out []piece + anchor bool +} + +func insert(at int, out ...piece) edit { return edit{a: at, z: at, out: out} } + +// applyEdits is f's text with edits applied (they must not overlap; an +// insertion at an offset comes before a replacement starting there). +func applyEdits(f *SrcFile, edits []edit) ([]piece, error) { + sort.SliceStable(edits, func(i, j int) bool { + if edits[i].a != edits[j].a { + return edits[i].a < edits[j].a + } + return edits[i].z < edits[j].z + }) + var ps []piece + cur := 0 + for _, e := range edits { + if e.a < cur { + return nil, fmt.Errorf("overlapping edits at offset %d", e.a) + } + if e.a > cur { + ps = append(ps, cp(f, cur, e.a)) + } + if e.anchor { + ps = append(ps, anchorAt(f, e.a)) + } + ps = append(ps, e.out...) + cur = e.z + } + if cur < len(f.Src) { + ps = append(ps, cp(f, cur, len(f.Src))) + } + return ps, nil +} + +// ─── shared helpers ─────────────────────────────────────────────────────────── + +func identityOf(sig, file string) string { return sig + "#" + file } + +func lineStart(src []byte, off int) int { + return bytes.LastIndexByte(src[:off], '\n') + 1 +} + +// lineEnd is the offset just past the '\n' ending the line holding off +// (len(src) on the last line). +func lineEnd(src []byte, off int) int { + if i := bytes.IndexByte(src[off:], '\n'); i >= 0 { + return off + i + 1 + } + return len(src) +} + +// blankTail reports whether src[off:] up to the end of its line is blank or +// a line comment. +func blankTail(src []byte, off int) bool { + rest := strings.TrimLeft(string(src[off:lineEnd(src, off)]), " \t\r\n") + return rest == "" || strings.HasPrefix(rest, "//") +} + +func blankHead(src []byte, off int) bool { + return strings.TrimLeft(string(src[lineStart(src, off):off]), " \t") == "" +} + +func declDoc(d ast.Decl) *ast.CommentGroup { + switch d := d.(type) { + case *ast.FuncDecl: + return d.Doc + case *ast.GenDecl: + return d.Doc + } + return nil +} + +// chunk is the whole lines of decl i of f with its doc comment: [a, z). ok +// is false when another declaration shares one of those lines. +func chunk(f *SrcFile, i int) (a, z int, ok bool) { + d := f.AST.Decls[i] + start := d.Pos() + if doc := declDoc(d); doc != nil { + start = doc.Pos() + } + so, eo := f.off(start), f.off(d.End()) + if !blankHead(f.Src, so) || !blankTail(f.Src, eo) { + return 0, 0, false + } + a, z = lineStart(f.Src, so), lineEnd(f.Src, eo) + if i > 0 && f.off(f.AST.Decls[i-1].End()) > a { + return 0, 0, false + } + if i+1 < len(f.AST.Decls) { + n := f.AST.Decls[i+1] + ns := n.Pos() + if doc := declDoc(n); doc != nil { + ns = doc.Pos() + } + if f.off(ns) < z { + return 0, 0, false + } + } + return a, z, true +} + +func isImport(d ast.Decl) bool { + g, ok := d.(*ast.GenDecl) + return ok && g.Tok == token.IMPORT +} + +// funcIdentity is the comparer's Identity of FuncDecl fd of f. +func funcIdentity(f *SrcFile, fd *ast.FuncDecl) string { + obj := f.Pkg.TypesInfo.Defs[fd.Name] + if obj == nil { + return "" + } + sig := syntactic_analysis.SignatureOf(obj) + if fd.Recv == nil && fd.Name.Name == "init" { + return identityOf(sig, f.Rel) + } + return sig +} + +// localVar is obj as a function-local variable: a parameter, result, +// receiver or local, not a field or a package-level variable. +func localVar(f *SrcFile, obj types.Object) (*types.Var, bool) { + v, ok := obj.(*types.Var) + if !ok || v.IsField() || v.Pkg() != f.Pkg.Types || v.Parent() == nil || v.Parent() == v.Pkg().Scope() { + return nil, false + } + return v, true +} + +func contains[T ast.Node](n ast.Node, pred func(T) bool) bool { + found := false + ast.Inspect(n, func(n ast.Node) bool { + if x, ok := n.(T); ok && pred(x) { + found = true + } + return !found + }) + return found +} + +func hasGoto(n ast.Node) bool { + return contains(n, func(b *ast.BranchStmt) bool { return b.Tok == token.GOTO }) +} + +// usesPrefix reports whether any identifier of f's package starts with prefix. +func usesPrefix(f *SrcFile, prefix string) bool { + for _, af := range f.Pkg.Syntax { + if contains(af, func(id *ast.Ident) bool { return strings.HasPrefix(id.Name, prefix) }) { + return true + } + } + return false +} + +func funcDecls(f *SrcFile) []*ast.FuncDecl { + var out []*ast.FuncDecl + for _, d := range f.AST.Decls { + if fd, ok := d.(*ast.FuncDecl); ok { + out = append(out, fd) + } + } + return out +} + +// stmtLists calls fn with every statement list of body outside function +// literals: blocks, and case and comm clause bodies. +func stmtLists(body *ast.BlockStmt, fn func([]ast.Stmt)) { + ast.Inspect(body, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.FuncLit: + return false + case *ast.BlockStmt: + fn(n.List) + case *ast.CaseClause: + fn(n.Body) + case *ast.CommClause: + fn(n.Body) + } + return true + }) +} + +// ─── MR1: blank lines and comments ──────────────────────────────────────────── + +// MR1 inserts a comment and blank lines at the top of each file and before +// every top-level declaration. +var MR1 = Transform{Name: "MR1", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + edits := []edit{insert(0, lit("// mr1: inserted at the top\n\n"))} + prevEnd := 0 + n := 0 + for _, d := range f.AST.Decls { + start := d.Pos() + if doc := declDoc(d); doc != nil { + start = doc.Pos() + } + at := lineStart(f.Src, f.off(start)) + if at >= prevEnd && at > 0 { + edits = append(edits, insert(at, lit("\n// mr1: inserted between declarations\n\n\n"))) + n++ + } + prevEnd = f.off(d.End()) + } + ps, err := applyEdits(f, edits) + return &fileEdit{pieces: ps, sites: []Site{{Desc: fmt.Sprintf("%s: %d insertions", f.Rel, n+1)}}}, err +}} + +// ─── MR2: reverse the top-level declarations ────────────────────────────────── + +// MR2 reverses the order of a file's top-level declarations after its +// imports, so methods land above their types. Each declaration moves with its +// doc comment and the comments before it. +var MR2 = Transform{Name: "MR2", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + decls := f.AST.Decls + first := 0 + for first < len(decls) && isImport(decls[first]) { + first++ + } + if len(decls)-first < 2 { + return nil, errSkip("fewer than 2 declarations") + } + header := lineEnd(f.Src, f.off(f.AST.Name.End())) + if first > 0 { + header = lineEnd(f.Src, f.off(decls[first-1].End())) + } + // Chunk i runs from the end of decl i-1's last line to the end of decl + // i's last line: it holds the comments before decl i. + var bounds []int + for i := first; i < len(decls); i++ { + if _, _, ok := chunk(f, i); !ok { + return nil, errSkip("declarations share a line") + } + bounds = append(bounds, lineEnd(f.Src, f.off(decls[i].End()))) + } + bounds[len(bounds)-1] = len(f.Src) + ps := []piece{cp(f, 0, header)} + for i := len(bounds) - 1; i >= 0; i-- { + a := header + if i > 0 { + a = bounds[i-1] + } + ps = append(ps, cp(f, a, bounds[i])) + if !bytes.HasSuffix(f.Src[a:bounds[i]], []byte("\n")) { + ps = append(ps, lit("\n")) + } + } + var touched []string + for _, fd := range funcDecls(f) { + touched = append(touched, funcIdentity(f, fd)) + } + return &fileEdit{pieces: ps, touched: touched, + sites: []Site{{Desc: fmt.Sprintf("%s: %d declarations reversed", f.Rel, len(bounds))}}}, nil +}} + +// ─── MR3: move a function and a method to a new file ────────────────────────── + +// MR3 moves one top-level function and one method of a file (never an init) +// into a new file of the same package, which imports what they use. An +// import of the old file that only they used becomes a blank import, so the +// old file keeps its lines and still type-checks. +var MR3 = Transform{Name: "MR3", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + info := f.Pkg.TypesInfo + for _, s := range f.AST.Imports { + if s.Name != nil && s.Name.Name == "." { + return nil, errSkip("dot import") + } + } + var fns, meths []int + for i, d := range f.AST.Decls { + fd, ok := d.(*ast.FuncDecl) + if !ok || fd.Name.Name == "init" || fd.Name.Name == "_" { + continue + } + if _, _, ok := chunk(f, i); !ok { + continue + } + if fd.Recv == nil { + fns = append(fns, i) + } else { + meths = append(meths, i) + } + } + var move []int + if len(fns) > 0 { + move = append(move, fns[c.rng.IntN(len(fns))]) + } + if len(meths) > 0 { + move = append(move, meths[c.rng.IntN(len(meths))]) + } + if len(move) == 0 { + return nil, errSkip("nothing to move") + } + slices.Sort(move) + + pkgUses := func(n ast.Node) map[*types.PkgName]bool { + out := map[*types.PkgName]bool{} + ast.Inspect(n, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + if pn, ok := info.Uses[id].(*types.PkgName); ok { + out[pn] = true + } + } + return true + }) + return out + } + moved, kept := map[*types.PkgName]bool{}, map[*types.PkgName]bool{} + for i, d := range f.AST.Decls { + dst := kept + if slices.Contains(move, i) { + dst = moved + } + for pn := range pkgUses(d) { + dst[pn] = true + } + } + + var edits []edit + var imports []string + for _, s := range f.AST.Imports { + pn := importName(info, s) + if pn == nil || pn.Name() == "_" { + continue + } + if moved[pn] { + imports = append(imports, pn.Name()+" "+s.Path.Value) + } + if !kept[pn] { + edits = append(edits, edit{a: f.off(s.Pos()), z: f.off(s.End()), out: []piece{lit("_ " + s.Path.Value)}}) + } + } + created := []piece{lit("package " + f.AST.Name.Name + "\n\n")} + if len(imports) > 0 { + sort.Strings(imports) + created = append(created, lit("import (\n\t"+strings.Join(imports, "\n\t")+"\n)\n")) + } + base := strings.TrimSuffix(filepath.Base(f.Rel), ".go") + "_mr3.go" + rel := filepath.ToSlash(filepath.Join(filepath.Dir(f.Rel), base)) + var touched []string + var sites []Site + for _, i := range move { + a, z, _ := chunk(f, i) + edits = append(edits, edit{a: a, z: z}) + created = append(created, lit("\n"), cp(f, a, z)) + if !bytes.HasSuffix(f.Src[a:z], []byte("\n")) { + created = append(created, lit("\n")) + } + fd := f.AST.Decls[i].(*ast.FuncDecl) + id := funcIdentity(f, fd) + touched = append(touched, id) + sites = append(sites, Site{Desc: fmt.Sprintf("%s:%d: %s moved to %s", f.Rel, f.pos(fd.Pos()).Line, fd.Name.Name, rel), Callable: id}) + } + ps, err := applyEdits(f, edits) + if err != nil { + return nil, err + } + return &fileEdit{pieces: ps, touched: touched, sites: sites, created: []newFile{{rel: rel, pieces: created}}}, nil +}} + +// importName is the package name import spec s declares in its file. +func importName(info *types.Info, s *ast.ImportSpec) *types.PkgName { + if s.Name != nil { + pn, _ := info.Defs[s.Name].(*types.PkgName) + return pn + } + pn, _ := info.Implicits[s].(*types.PkgName) + return pn +} + +// ─── MR4: alpha-rename locals ───────────────────────────────────────────────── + +// Suffix4 is the suffix MR4 appends to every local variable and parameter +// name. A file whose package already has an identifier containing it is +// skipped, so the comparer can undo the rename by stripping it. +const Suffix4 = "_mr4" + +// MR4 renames every local variable, parameter, named result and receiver of +// the file's functions (closures included) to name+Suffix4. +var MR4 = Transform{Name: "MR4", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + for _, af := range f.Pkg.Syntax { + if contains(af, func(id *ast.Ident) bool { return strings.Contains(id.Name, Suffix4) }) { + return nil, errSkip("name clash") + } + } + info := f.Pkg.TypesInfo + var edits []edit + var touched []string + n := 0 + rename := func(id *ast.Ident) { + edits = append(edits, edit{a: f.off(id.Pos()), z: f.off(id.End()), anchor: true, out: []piece{lit(id.Name + Suffix4)}}) + n++ + } + for _, fd := range funcDecls(f) { + before := n + ast.Inspect(fd, func(nd ast.Node) bool { + switch x := nd.(type) { + case *ast.TypeSwitchStmt: + // `switch v := x.(type)`: v has no object of its own; each + // clause has an implicit one, which its uses resolve to. + if as, ok := x.Assign.(*ast.AssignStmt); ok { + if id, ok := as.Lhs[0].(*ast.Ident); ok && id.Name != "_" { + rename(id) + } + } + case *ast.Ident: + if x.Name == "_" { + return true + } + obj := info.Defs[x] + if obj == nil { + obj = info.Uses[x] + } + if _, ok := localVar(f, obj); ok { + rename(x) + } + } + return true + }) + if n > before { + touched = append(touched, funcIdentity(f, fd)) + } + } + if len(edits) == 0 { + return nil, errSkip("no locals") + } + ps, err := applyEdits(f, edits) + return &fileEdit{pieces: ps, touched: touched, + sites: []Site{{Desc: fmt.Sprintf("%s: %d identifiers renamed", f.Rel, n)}}}, err +}} + +// ─── MR5: introduce an alias ────────────────────────────────────────────────── + +// MR5 rewrites each `x.f = e` of the file, where x is a local and f a direct +// field of it, as `q := x; q.f = e` when x is a pointer to a struct, or as +// `q := &x; q.f = e` when x is a struct value. q is fresh. The write now goes +// through an alias of x. Other bases are left alone: a value copy does not +// alias. +var MR5 = Transform{Name: "MR5", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + if usesPrefix(f, "mr5q") { + return nil, errSkip("name clash") + } + info := f.Pkg.TypesInfo + fe := &fileEdit{} + var edits []edit + for _, fd := range funcDecls(f) { + if fd.Body == nil || hasGoto(fd.Body) { + continue + } + before := len(fe.sites) + stmtListsAll(fd.Body, func(list []ast.Stmt) { + for _, s := range list { + as, ok := s.(*ast.AssignStmt) + if !ok || as.Tok != token.ASSIGN || len(as.Lhs) != 1 || len(as.Rhs) != 1 { + continue + } + sel, ok := as.Lhs[0].(*ast.SelectorExpr) + if !ok { + continue + } + x, ok := sel.X.(*ast.Ident) + if !ok { + continue + } + v, ok := localVar(f, info.Uses[x]) + if !ok { + continue + } + if s := info.Selections[sel]; s == nil || s.Kind() != types.FieldVal || len(s.Index()) != 1 { + continue + } + var rhs string + switch t := v.Type().(type) { + case *types.Pointer: + if _, ok := t.Elem().Underlying().(*types.Struct); !ok { + fe.skip("MR5: not a struct pointer") + continue + } + rhs = x.Name + case *types.Named: + if _, ok := t.Underlying().(*types.Struct); !ok { + fe.skip("MR5: not a struct") + continue + } + rhs = "&" + x.Name + default: + fe.skip("MR5: not a struct or a pointer to one") + continue + } + q := c.fresh("mr5q") + at := f.off(as.Pos()) + edits = append(edits, + insert(at, lit(q+" := "+rhs+"; ")), + edit{a: at, z: f.off(x.End()), anchor: true, out: []piece{lit(q)}}) + fe.sites = append(fe.sites, Site{Desc: fmt.Sprintf("%s:%d: %s via %s", f.Rel, f.lines.lineOf(at), x.Name, q), Callable: funcIdentity(f, fd)}) + } + }) + if len(fe.sites) > before { + fe.touched = append(fe.touched, funcIdentity(f, fd)) + } + } + if len(edits) == 0 { + return fe, errSkip("no field write") + } + ps, err := applyEdits(f, edits) + fe.pieces = ps + return fe, err +}} + +// stmtListsAll is stmtLists including function literal bodies. +func stmtListsAll(body *ast.BlockStmt, fn func([]ast.Stmt)) { + ast.Inspect(body, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.BlockStmt: + fn(n.List) + case *ast.CaseClause: + fn(n.Body) + case *ast.CommClause: + fn(n.Body) + } + return true + }) +} + +// ─── MR6: extract a helper ──────────────────────────────────────────────────── + +// Prefix6 starts every MR6 helper's name. +const Prefix6 = "mr6h_" + +// MR6 extracts, in each function of the file, one straight-line run of 2-4 +// statements into a new top-level helper at the end of the file. The helper +// takes every outer local the run names and returns them (in-out, so a +// write to a copy is assigned back), then the locals the run declares that +// the function uses after it: +// +// var n1 T1; o1, o2, n1 = mr6h_f_1(o1, o2) +// +// A run qualifies when: +// - its statements are simple (assignment, expression, ++/--, send, var +// declaration) and hold no function literal or recover(); +// - it names no outer local that a closure captures or that is +// address-taken anywhere in the function (copy-in/copy-out would then +// change what the program does); +// - it names no function-local type or constant; +// - every type in the helper's signature can be spelled in the file. +// +// Generic functions and functions with a goto are skipped. +var MR6 = Transform{Name: "MR6", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + if usesPrefix(f, Prefix6) { + return nil, errSkip("name clash") + } + fe := &fileEdit{} + var edits []edit + var helpers []piece + for _, fd := range funcDecls(f) { + if fd.Body == nil || fd.Type.TypeParams != nil || hasGoto(fd.Body) || recvGeneric(fd) { + continue + } + runs := mr6Runs(f, fd) + if len(runs) == 0 { + continue + } + r := runs[c.rng.IntN(len(runs))] + name := c.fresh(Prefix6 + fd.Name.Name + "_") + var params, results, args, outs []string + for _, v := range r.outer { + params = append(params, v.v.Name()+" "+v.t) + results = append(results, v.t) + args = append(args, v.v.Name()) + outs = append(outs, v.v.Name()) + } + var decl string + for _, v := range r.fresh { + decl += "var " + v.v.Name() + " " + v.t + "; " + results = append(results, v.t) + outs = append(outs, v.v.Name()) + } + call := name + "(" + strings.Join(args, ", ") + ")" + if len(outs) > 0 { + call = decl + strings.Join(outs, ", ") + " = " + call + } + edits = append(edits, edit{a: r.a, z: r.z, out: []piece{lit(call)}}) + head := "\n\n// " + name + " is extracted by MR6.\nfunc " + name + "(" + strings.Join(params, ", ") + ")" + if len(results) > 0 { + head += " (" + strings.Join(results, ", ") + ")" + } + helpers = append(helpers, lit(head+" {\n\t"), cp(f, r.a, r.z)) + if len(outs) > 0 { + helpers = append(helpers, lit("\n\treturn "+strings.Join(outs, ", "))) + } + helpers = append(helpers, lit("\n}\n")) + id := funcIdentity(f, fd) + fe.touched = append(fe.touched, id) + fe.sites = append(fe.sites, Site{Desc: fmt.Sprintf("%s:%d: %d statements into %s", f.Rel, f.lines.lineOf(r.a), r.n, name), Callable: id}) + } + if len(edits) == 0 { + return fe, errSkip("no run") + } + edits = append(edits, insert(len(f.Src), helpers...)) + ps, err := applyEdits(f, edits) + fe.pieces = ps + return fe, err +}} + +func (t lineTable) lineOf(off int) int { + l, _ := t.pos(off) + return l +} + +func recvGeneric(fd *ast.FuncDecl) bool { + if fd.Recv == nil || len(fd.Recv.List) == 0 { + return false + } + t := fd.Recv.List[0].Type + if s, ok := t.(*ast.StarExpr); ok { + t = s.X + } + switch t.(type) { + case *ast.IndexExpr, *ast.IndexListExpr: + return true + } + return false +} + +type runVar struct { + v *types.Var + t string // the type, spelled in the file +} + +type run struct { + a, z int + n int + outer []runVar // outer locals named, by first use + fresh []runVar // locals declared in the run and used after it +} + +// mr6Runs returns the qualifying runs of fd (see MR6). +func mr6Runs(f *SrcFile, fd *ast.FuncDecl) []run { + info := f.Pkg.TypesInfo + pinned := pinnedVars(f, fd.Body) + var out []run + stmtLists(fd.Body, func(list []ast.Stmt) { + for i := range list { + for n := 2; n <= 4 && i+n <= len(list); n++ { + w := list[i : i+n] + if slices.ContainsFunc(w, func(s ast.Stmt) bool { return !simpleStmt(info, s) }) { + break + } + if r, ok := makeRun(f, fd, w, pinned); ok { + out = append(out, r) + } + } + } + }) + return out +} + +func simpleStmt(info *types.Info, s ast.Stmt) bool { + switch s := s.(type) { + case *ast.AssignStmt, *ast.ExprStmt, *ast.IncDecStmt, *ast.SendStmt: + case *ast.DeclStmt: + if g, ok := s.Decl.(*ast.GenDecl); !ok || g.Tok != token.VAR { + return false + } + default: + return false + } + return !contains(s, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.FuncLit: + return true + case *ast.Ident: + b, isB := info.Uses[n].(*types.Builtin) + return isB && b.Name() == "recover" + } + return false + }) +} + +// pinnedVars are the locals of body whose identity matters beyond their +// value: captured by a function literal, or address-taken (&x, an array +// sliced, a pointer-receiver method called on an addressable value). +func pinnedVars(f *SrcFile, body ast.Node) map[*types.Var]bool { + info := f.Pkg.TypesInfo + out := map[*types.Var]bool{} + base := func(e ast.Expr) *types.Var { + for { + switch x := e.(type) { + case *ast.ParenExpr: + e = x.X + case *ast.SelectorExpr: + if s := info.Selections[x]; s != nil && s.Indirect() { + return nil // through a pointer: not the variable itself + } + e = x.X + case *ast.IndexExpr: + if _, isArr := info.TypeOf(x.X).Underlying().(*types.Array); !isArr { + return nil + } + e = x.X + case *ast.Ident: + v, _ := localVar(f, info.Uses[x]) + return v + default: + return nil + } + } + } + ast.Inspect(body, func(n ast.Node) bool { + switch x := n.(type) { + case *ast.FuncLit: + ast.Inspect(x.Body, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + if v, ok := localVar(f, info.Uses[id]); ok && (v.Pos() < x.Pos() || v.Pos() >= x.End()) { + out[v] = true + } + } + return true + }) + case *ast.UnaryExpr: + if x.Op == token.AND { + if v := base(x.X); v != nil { + out[v] = true + } + } + case *ast.SliceExpr: + if t := info.TypeOf(x.X); t != nil { + if _, isArr := t.Underlying().(*types.Array); isArr { + if v := base(x.X); v != nil { + out[v] = true + } + } + } + case *ast.SelectorExpr: + if s := info.Selections[x]; s != nil && s.Kind() == types.MethodVal { + if sig, ok := s.Obj().Type().(*types.Signature); ok && sig.Recv() != nil { + if _, ptrRecv := sig.Recv().Type().(*types.Pointer); ptrRecv { + if v := base(x.X); v != nil { + out[v] = true + } + } + } + } + } + return true + }) + return out +} + +func makeRun(f *SrcFile, fd *ast.FuncDecl, w []ast.Stmt, pinned map[*types.Var]bool) (run, bool) { + info := f.Pkg.TypesInfo + r := run{a: f.off(w[0].Pos()), z: f.off(w[len(w)-1].End()), n: len(w)} + lo, hi := w[0].Pos(), w[len(w)-1].End() + seen := map[*types.Var]bool{} + var fresh []*types.Var + ok := true + for _, s := range w { + ast.Inspect(s, func(n ast.Node) bool { + id, isID := n.(*ast.Ident) + if !isID || !ok { + return ok + } + obj := info.Uses[id] + if obj == nil { + obj = info.Defs[id] + } + if obj == nil { + return true + } + // A function-local type, constant or function name would not be + // in scope in the helper. + if _, isVar := obj.(*types.Var); !isVar { + if _, isPkg := obj.(*types.PkgName); !isPkg && obj.Pkg() == f.Pkg.Types && + obj.Parent() != nil && obj.Parent() != f.Pkg.Types.Scope() { + ok = false + } + return true + } + v, local := localVar(f, obj) + if !local || seen[v] { + return true + } + seen[v] = true + if v.Pos() >= lo && v.Pos() < hi { + fresh = append(fresh, v) + return true + } + if pinned[v] { + ok = false + return false + } + t, spelled := spell(f, v.Type()) + if !spelled { + ok = false + return false + } + r.outer = append(r.outer, runVar{v, t}) + return true + }) + } + if !ok || len(r.outer) == 0 { + return run{}, false + } + names := map[string]bool{} + for _, v := range r.outer { + if names[v.v.Name()] { + return run{}, false + } + names[v.v.Name()] = true + } + for _, v := range fresh { + if names[v.Name()] { + return run{}, false + } + if !usedAfter(info, fd, v, hi) { + continue + } + t, spelled := spell(f, v.Type()) + if !spelled { + return run{}, false + } + r.fresh = append(r.fresh, runVar{v, t}) + } + return r, true +} + +func usedAfter(info *types.Info, fd *ast.FuncDecl, v *types.Var, after token.Pos) bool { + return contains(fd.Body, func(id *ast.Ident) bool { return id.Pos() >= after && info.Uses[id] == v }) +} + +// spell is t written in f, package-qualified by f's import names. It fails +// for a type f cannot name: an unexported or function-local named type, a +// type parameter, or a package f does not import. +func spell(f *SrcFile, t types.Type) (string, bool) { + names := map[string]string{} + for _, s := range f.AST.Imports { + if pn := importName(f.Pkg.TypesInfo, s); pn != nil && pn.Name() != "_" && pn.Name() != "." { + names[pn.Imported().Path()] = pn.Name() + } + } + ok := true + named := func(o *types.TypeName) { + if o.Pkg() == nil { + return + } + if o.Parent() != o.Pkg().Scope() || (o.Pkg() != f.Pkg.Types && (!o.Exported() || names[o.Pkg().Path()] == "")) { + ok = false + } + } + var check func(t types.Type, depth int) + check = func(t types.Type, depth int) { + if !ok || depth > 20 { + ok = false + return + } + switch t := t.(type) { + case *types.Named: + named(t.Obj()) + if args := t.TypeArgs(); args != nil { + for i := range args.Len() { + check(args.At(i), depth+1) + } + } + case *types.Alias: + named(t.Obj()) + case *types.TypeParam, *types.Tuple: + ok = false + case *types.Pointer: + check(t.Elem(), depth+1) + case *types.Slice: + check(t.Elem(), depth+1) + case *types.Array: + check(t.Elem(), depth+1) + case *types.Chan: + check(t.Elem(), depth+1) + case *types.Map: + check(t.Key(), depth+1) + check(t.Elem(), depth+1) + case *types.Signature: + for _, tup := range []*types.Tuple{t.Params(), t.Results()} { + for i := range tup.Len() { + check(tup.At(i).Type(), depth+1) + } + } + case *types.Struct: + for i := range t.NumFields() { + if !t.Field(i).Exported() && t.Field(i).Pkg() != f.Pkg.Types { + ok = false + } + check(t.Field(i).Type(), depth+1) + } + case *types.Interface: + for i := range t.NumExplicitMethods() { + check(t.ExplicitMethod(i).Type(), depth+1) + } + for i := range t.NumEmbeddeds() { + check(t.EmbeddedType(i), depth+1) + } + case *types.Basic: + if t.Info()&types.IsUntyped != 0 || t.Kind() == types.UnsafePointer { + ok = false + } + } + } + check(t, 0) + if !ok { + return "", false + } + return types.TypeString(t, func(p *types.Package) string { + if p == f.Pkg.Types { + return "" + } + return names[p.Path()] + }), true +} diff --git a/internal/metamorphic/transform_delta.go b/internal/metamorphic/transform_delta.go new file mode 100644 index 0000000..2e34b0f --- /dev/null +++ b/internal/metamorphic/transform_delta.go @@ -0,0 +1,1458 @@ +package metamorphic + +// The expected-delta transforms (MR8–MR12), the precision transform (MR13), +// the call-routing transform (MR14) and extract-then-inline (MR15). +// +// The delta transforms work only where the analysis is exact (brief +// amendment 2026-10-27). Their sites are statements over local scalar +// variables (basic types, never captured or address-taken) built from +// identifiers, literals and operators that cannot panic. The callable must +// have an L3 result and no defer, goto or label. Loops, select / switch / +// fallthrough, panic edges and the defer chain are out of scope: a site's +// own CFG edges are never exception or loop_back edges. Each delta is +// computed from the original's own edges (the pre view), so the relation +// states what changes, not what the edges are. +// +// Each callable gets at most one site, so deltas never interact. Sites are +// chosen with the transform's seeded generator. + +import ( + "fmt" + "go/ast" + "go/token" + "go/types" + "maps" + "slices" + "strconv" + "strings" +) + +// ─── pre-view queries ───────────────────────────────────────────────────────── + +type edgeRec struct{ kind, src, dst, label, prov string } + +func (e edgeRec) dedge() DEdge { + return DEdge{Kind: e.kind, Src: Pre(e.src), Dst: Pre(e.dst), Label: e.label, Prov: e.prov} +} + +func edgesOf(cv *CV, kind string) []edgeRec { + var out []edgeRec + for _, e := range slices.Sorted(maps.Keys(cv.Edges)) { + f := strings.Split(e, "|") + if f[0] != kind { + continue + } + r := edgeRec{kind: f[0], src: f[1], dst: f[2], label: f[3]} + if len(f) > 4 { + r.prov = f[4] + } + out = append(out, r) + } + return out +} + +func filter(es []edgeRec, pred func(edgeRec) bool) []edgeRec { + var out []edgeRec + for _, e := range es { + if pred(e) { + out = append(out, e) + } + } + return out +} + +func into(cv *CV, kind, dst string) []edgeRec { + return filter(edgesOf(cv, kind), func(e edgeRec) bool { return e.dst == dst }) +} + +func outOf(cv *CV, kind, src string) []edgeRec { + return filter(edgesOf(cv, kind), func(e edgeRec) bool { return e.src == src }) +} + +// cfgReach is the set of CFG nodes reachable from src in cv. +func cfgReach(cv *CV, src string) map[string]bool { + succ := map[string][]string{} + for _, e := range edgesOf(cv, "cfg") { + succ[e.src] = append(succ[e.src], e.dst) + } + seen := map[string]bool{src: true} + stack := []string{src} + for len(stack) > 0 { + n := stack[len(stack)-1] + stack = stack[:len(stack)-1] + for _, d := range succ[n] { + if !seen[d] { + seen[d] = true + stack = append(stack, d) + } + } + } + return seen +} + +// onCycle reports whether CFG node k reaches itself. +func onCycle(cv *CV, k string) bool { + for _, e := range outOf(cv, "cfg", k) { + if cfgReach(cv, e.dst)[k] { + return true + } + } + return false +} + +// sameCDG reports whether cv's cdg equals the reference CDG of its cfg. +func sameCDG(cv *CV) bool { + return cdgDiff(cv, RefCDG(cv)) == "" +} + +// ─── eligibility ────────────────────────────────────────────────────────────── + +// scope is the per-callable state of a delta transform. +type scope struct { + f *SrcFile + fd *ast.FuncDecl + id string + cv *CV + pre *View + pinned map[*types.Var]bool +} + +// deltaScope returns fd's scope, or why it is not eligible. +func deltaScope(c *tctx, f *SrcFile, fd *ast.FuncDecl) (*scope, string) { + if fd.Body == nil { + return nil, "no body" + } + if fd.Type.TypeParams != nil || recvGeneric(fd) { + return nil, "generic" + } + if contains(fd.Body, func(n ast.Node) bool { + switch n.(type) { + case *ast.DeferStmt, *ast.LabeledStmt: + return true + } + return false + }) || hasGoto(fd.Body) { + return nil, "defer, goto or label" + } + id := funcIdentity(f, fd) + cv := c.pre.Calls[id] + if cv == nil || !cv.L3 { + return nil, "no L3 result" + } + if !sameCDG(cv) { + // Its CDG is not the reference CDG of its CFG: a finding the test + // reports separately (TestRefCDG), not a site. + return nil, "cdg differs from the reference" + } + return &scope{f: f, fd: fd, id: id, cv: cv, pre: c.pre, pinned: pinnedVars(f, fd.Body)}, "" +} + +// scalar is obj as a local scalar of s: a function-local variable of basic +// type, neither captured nor address-taken. +func (s *scope) scalar(obj types.Object) (*types.Var, bool) { + v, ok := localVar(s.f, obj) + if !ok || s.pinned[v] { + return nil, false + } + b, ok := v.Type().Underlying().(*types.Basic) + if !ok || b.Info()&(types.IsNumeric|types.IsString|types.IsBoolean) == 0 { + return nil, false + } + return v, true +} + +func (s *scope) obj(id *ast.Ident) types.Object { + info := s.f.Pkg.TypesInfo + if o := info.Defs[id]; o != nil { + return o + } + return info.Uses[id] +} + +// safeExpr reports whether e cannot panic and names only local scalars and +// constants: identifiers, literals, parentheses and operators other than +// division, remainder and shifts. +func (s *scope) safeExpr(e ast.Expr) bool { + switch e := e.(type) { + case *ast.BasicLit: + return true + case *ast.Ident: + switch o := s.f.Pkg.TypesInfo.Uses[e].(type) { + case *types.Const, *types.Nil: + return true + case *types.Var: + _, ok := s.scalar(o) + return ok + } + return false + case *ast.ParenExpr: + return s.safeExpr(e.X) + case *ast.UnaryExpr: + return (e.Op == token.SUB || e.Op == token.ADD || e.Op == token.NOT || e.Op == token.XOR) && s.safeExpr(e.X) + case *ast.BinaryExpr: + switch e.Op { + case token.QUO, token.REM, token.SHL, token.SHR: + return false + } + return s.safeExpr(e.X) && s.safeExpr(e.Y) + } + return false +} + +// facts are a safe statement's defined and used variables, by object. +type facts struct { + defs, uses []*types.Var + names map[string]bool // every identifier it names + define bool // a := statement +} + +func addVar(list []*types.Var, v *types.Var) []*types.Var { + if slices.Contains(list, v) { + return list + } + return append(list, v) +} + +// safeStmt returns st's facts when st is a safe simple statement: an +// assignment (=, :=, or an operator assignment that cannot panic) of safe +// expressions to local scalars, or ++/-- of one. +func (s *scope) safeStmt(st ast.Stmt) (*facts, bool) { + fs := &facts{names: map[string]bool{}} + ast.Inspect(st, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + fs.names[id.Name] = true + } + return true + }) + usesOf := func(e ast.Expr) { + ast.Inspect(e, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + if v, ok := s.scalar(s.f.Pkg.TypesInfo.Uses[id]); ok { + fs.uses = addVar(fs.uses, v) + } + } + return true + }) + } + lhs := func(e ast.Expr, use bool) bool { + id, ok := e.(*ast.Ident) + if !ok || id.Name == "_" { + return false + } + v, ok := s.scalar(s.obj(id)) + if !ok { + return false + } + fs.defs = addVar(fs.defs, v) + if use { + fs.uses = addVar(fs.uses, v) + } + return true + } + switch st := st.(type) { + case *ast.AssignStmt: + switch st.Tok { + case token.ASSIGN, token.DEFINE: + case token.ADD_ASSIGN, token.SUB_ASSIGN, token.MUL_ASSIGN, token.AND_ASSIGN, token.OR_ASSIGN, + token.XOR_ASSIGN, token.AND_NOT_ASSIGN: + default: + return nil, false + } + if len(st.Lhs) != len(st.Rhs) { + return nil, false + } + op := st.Tok != token.ASSIGN && st.Tok != token.DEFINE + for _, r := range st.Rhs { + if !s.safeExpr(r) { + return nil, false + } + usesOf(r) + } + for _, l := range st.Lhs { + if !lhs(l, op) { + return nil, false + } + } + fs.define = st.Tok == token.DEFINE + case *ast.IncDecStmt: + if !lhs(st.X, true) { + return nil, false + } + default: + return nil, false + } + return fs, true +} + +// safeReturn reports whether st is a return of safe expressions. +func (s *scope) safeReturn(st ast.Stmt) bool { + r, ok := st.(*ast.ReturnStmt) + if !ok { + return false + } + for _, e := range r.Results { + if !s.safeExpr(e) { + return false + } + } + return true +} + +// resultNamed reports whether name is a named result of s's function. +func (s *scope) resultNamed(name string) bool { + if s.fd.Type.Results == nil { + return false + } + for _, f := range s.fd.Type.Results.List { + for _, n := range f.Names { + if n.Name == name { + return true + } + } + } + return false +} + +// key is the body key of statement st, which must be a CFG node of s.cv. +func (s *scope) key(st ast.Stmt) (string, bool) { + k := s.f.key(st.Pos()) + b, ok := s.cv.Body[k] + if !ok || strings.Split(b, "|")[1] != "" { + return "", false + } + return k, true +} + +// only reports whether k's only CFG out-edge is a fallthrough to a node +// other than @exit, and returns it. +func (s *scope) only(k string) (edgeRec, bool) { + out := outOf(s.cv, "cfg", k) + if len(out) != 1 || out[0].label != "fallthrough" || out[0].dst == "@exit" { + return edgeRec{}, false + } + return out[0], true +} + +func zeroLit(t types.Type) string { + b := t.Underlying().(*types.Basic) + switch { + case b.Info()&types.IsString != 0: + return `""` + case b.Info()&types.IsBoolean != 0: + return "false" + } + return "0" +} + +// deltaTransform runs pick on every eligible function of the file, at most +// one site each. +func deltaTransform(name string, pick func(c *tctx, s *scope, fe *fileEdit) []edit) Transform { + return Transform{Name: name, edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + if c.pre == nil { + return nil, fmt.Errorf("%s needs the original's analysis", name) + } + if usesPrefix(f, strings.ToLower(name)) { + return nil, errSkip(name + ": name clash") + } + fe := &fileEdit{} + var edits []edit + for _, fd := range funcDecls(f) { + s, why := deltaScope(c, f, fd) + if s == nil { + fe.skip(name + ": callable: " + why) + continue + } + es := pick(c, s, fe) + if len(es) == 0 { + fe.skip(name + ": callable: no eligible site") + continue + } + edits = append(edits, es...) + fe.touched = append(fe.touched, s.id) + } + if len(edits) == 0 { + return fe, errSkip(name + ": no eligible site") + } + ps, err := applyEdits(f, edits) + fe.pieces = ps + return fe, err + }} +} + +// choose returns one of n candidates with c's generator. +func (c *tctx) choose(n int) int { return c.rng.IntN(n) } + +func desc(s *scope, st ast.Node, what string) string { + return fmt.Sprintf("%s:%d: %s: %s", s.f.Rel, s.f.pos(st.Pos()).Line, s.fd.Name.Name, what) +} + +// statements is every statement list of s's body outside function literals, +// with each statement. +func (s *scope) lists() [][]ast.Stmt { + var out [][]ast.Stmt + stmtLists(s.fd.Body, func(l []ast.Stmt) { out = append(out, l) }) + return out +} + +// ─── delta builder ──────────────────────────────────────────────────────────── + +// dbuild accumulates a Delta over a callable's pre view. Its DDG helpers +// keep the L4 port mirror of an @entry edge in step: an L3 edge @entry → n +// on a base that has a formal_in is mirrored as @formal_in: → n on the +// same var, prov reaching-defs (spec §4.7 port wiring). +type dbuild struct { + d *Delta + cv *CV + called bool +} + +func newDelta(s *scope, site string, nodes map[string]string) *dbuild { + return &dbuild{d: &Delta{Site: site, Callable: s.id, Nodes: nodes}, cv: s.cv, called: s.pre.called(s.id)} +} + +// formalIn is the @formal_in key whose `of` is name ("" when none). +func (b *dbuild) formalIn(name string) string { + for k, attr := range b.cv.Body { + if strings.HasPrefix(k, "@formal_in:") && strings.Split(attr, "|")[2] == name { + return k + } + } + return "" +} + +func (b *dbuild) rm(e edgeRec) { + b.d.Remove = append(b.d.Remove, e.dedge()) + if e.kind == "ddg" && e.src == "@entry" { + if fi := b.formalIn(e.label); fi != "" && b.cv.Edges["ddg|"+fi+"|"+e.dst+"|"+e.label+"|reaching-defs"] { + b.d.Remove = append(b.d.Remove, Port(Pre(fi), Pre(e.dst), e.label)) + } + } +} + +func (b *dbuild) add(e DEdge) { + for _, a := range b.d.Add { + if a == e { + return + } + } + b.d.Add = append(b.d.Add, e) + if e.Kind == "ddg" && e.Src.Pre == "@entry" { + if fi := b.formalIn(e.Label); fi != "" { + b.add(Port(Pre(fi), e.Dst, e.Label)) + } + } +} + +// ssaOn are the ssa ddg edges of es on var v; ok is false when es holds a +// non-ssa edge on v other than an @formal_in port mirror (a site the delta +// rules do not describe). +func ssaOn(es []edgeRec, v string) ([]edgeRec, bool) { + var out []edgeRec + for _, e := range es { + if e.label != v { + continue + } + switch { + case e.prov == "ssa": + out = append(out, e) + case e.prov == "reaching-defs" && strings.HasPrefix(e.src, "@formal_in:"): + default: + return nil, false + } + } + return out, true +} + +// sym is e with its refs spelled symbolically: a pre key as itself, a mark +// as "mark:". +func (e DEdge) sym() string { + ref := func(r Ref) string { + if r.Mark != "" { + return "mark:" + r.Mark + } + return r.Pre + } + out := e.Kind + "|" + ref(e.Src) + "|" + ref(e.Dst) + "|" + e.Label + if e.Kind == "ddg" { + out += "|" + cmpOr(e.Prov, "ssa") + } + return out +} + +// localFlows is the set of (formal_in, formal_out) pairs of a callable's +// edges, the flows its summary is computed from: reachability over its ddg, +// cdg and summary edges (sdg graphOf). +func localFlows(edges map[string]bool) map[[2]string]bool { + succ := map[string][]string{} + var srcs []string + for e := range edges { + f := strings.Split(e, "|") + if f[0] == "cfg" { + continue + } + succ[f[1]] = append(succ[f[1]], f[2]) + if strings.HasPrefix(f[1], "@formal_in:") { + srcs = append(srcs, f[1]) + } + } + out := map[[2]string]bool{} + for _, s := range srcs { + for n := range (Graph(succ)).Reach(s) { + if strings.HasPrefix(n, "@formal_out") { + out[[2]string{s, n}] = true + } + } + } + return out +} + +// preserves reports whether the delta leaves the callable's flows as they +// are. A delta that changes them would change the summary edges of every +// caller, outside the callable, which the delta oracle does not predict. +// +// The predicted edges name new nodes by mark ("mark:"); its CDG is +// derived from its CFG when the delta says so. +func (b *dbuild) preserves() bool { + post := maps.Clone(b.cv.Edges) + for _, e := range b.d.Remove { + delete(post, e.sym()) + } + for _, e := range b.d.Add { + post[e.sym()] = true + } + if b.d.CDGFromCFG { + for e := range post { + if strings.HasPrefix(e, "cdg|") { + delete(post, e) + } + } + for e := range RefCDG(&CV{Edges: post}) { + post[e] = true + } + } + return maps.Equal(localFlows(b.cv.Edges), localFlows(post)) +} + +// done records the delta as a site of fe, or the reason it is not one. +// +// A callable no in-project call site targets (no param_in edge into it) may +// change its flows: no caller's summary edges depend on them. +func (b *dbuild) done(fe *fileEdit, name string) bool { + if b.called && !b.preserves() { + fe.skip(name + ": site: changes the callable's flows") + return false + } + fe.sites = append(fe.sites, Site{Desc: b.d.Site, Callable: b.d.Callable, Delta: b.d}) + return true +} + +// ─── MR8: wrap a statement in `if c { S }` ──────────────────────────────────── + +// MR8 wraps a statement S in `if c := true; c { S }`. S is a safe +// assignment that reads every variable it writes (x++, x += e, x = x + e), +// so every def reaching S is visible as one of its ddg in-edges, and it +// falls through to its successor Y. Expected delta: +// - CFG: each X→S becomes X→B (same kind); B→S true; B→Y false. +// - CDG: derived from the expected CFG (the reference CDG). +// - DDG: S is now conditional, so each def D reaching S on x also reaches +// each use U that S's def of x reached: D→U on x, unless already there +// (with its @formal_in port mirror when D is @entry). The guard c is +// defined by the header's init and used by its condition, both folded +// into the header node B (spec §4.2). Within a node uses precede defs, +// so c adds no edge in straight-line code; when S lies on a CFG cycle +// (inside a loop body) the init's def reaches the condition around the +// loop, and c adds the self-loop B→B (a sound over-approximation of the +// header folding). +var MR8 = deltaTransform("MR8", func(c *tctx, s *scope, fe *fileEdit) []edit { + type cand struct { + st ast.Stmt + k string + y edgeRec + fs *facts + } + var cands []cand + for _, list := range s.lists() { + for _, st := range list { + fs, ok := s.safeStmt(st) + if !ok || fs.define || slices.ContainsFunc(fs.defs, func(v *types.Var) bool { return !slices.Contains(fs.uses, v) }) { + continue + } + k, ok := s.key(st) + if !ok { + continue + } + y, ok := s.only(k) + if !ok { + continue + } + cands = append(cands, cand{st, k, y, fs}) + } + } + for len(cands) > 0 { + i := c.choose(len(cands)) + x := cands[i] + cands = slices.Delete(cands, i, i+1) + b, g := c.fresh("mr8b"), c.fresh("mr8c") + db := newDelta(s, desc(s, x.st, "wrap in if"), map[string]string{b: "branch"}) + db.d.CDGFromCFG = true + for _, e := range into(s.cv, "cfg", x.k) { + db.rm(e) + db.add(CFG(Pre(e.src), Mark(b), e.label)) + } + db.add(CFG(Mark(b), Pre(x.k), "true")) + db.add(CFG(Mark(b), Pre(x.y.dst), "false")) + if onCycle(s.cv, x.k) { + db.add(DDG(Mark(b), Mark(b), g)) + } + ok := true + for _, v := range x.fs.defs { + ins, ok1 := ssaOn(into(s.cv, "ddg", x.k), v.Name()) + outs, ok2 := ssaOn(outOf(s.cv, "ddg", x.k), v.Name()) + if !ok1 || !ok2 { + ok = false + break + } + for _, in := range ins { + for _, o := range outs { + if !s.cv.Edges["ddg|"+in.src+"|"+o.dst+"|"+v.Name()+"|ssa"] { + db.add(DDG(Pre(in.src), Pre(o.dst), v.Name())) + } + } + } + } + if !ok { + fe.skip("MR8: site: non-ssa ddg on the variable") + continue + } + if !db.done(fe, "MR8") { + continue + } + a, z := s.f.off(x.st.Pos()), s.f.off(x.st.End()) + return []edit{ + insert(a, mark(b), lit("if "+g+" := true; "+g+" { ")), + insert(z, lit(" }")), + } + } + return nil +}) + +// ─── MR9: insert a kill ─────────────────────────────────────────────────────── + +// MR9 inserts the kill `x = k` (k the zero literal of x's type) between a +// safe statement D that defines local scalar x and the next statement U, +// which D falls through to and which reads x (D→U on x). Every path from D +// passes the kill, so expected delta: +// - DDG: each D→W on x becomes K→W on x. +// - CFG: D→U becomes D→K, K→U (fallthrough). +// - CDG: derived (K gets D's controllers). +var MR9 = deltaTransform("MR9", func(c *tctx, s *scope, fe *fileEdit) []edit { + type cand struct { + d, u ast.Stmt + dk, uk string + v *types.Var + } + var cands []cand + for _, list := range s.lists() { + for i := 0; i+1 < len(list); i++ { + fs, ok := s.safeStmt(list[i]) + if !ok { + continue + } + dk, ok1 := s.key(list[i]) + uk, ok2 := s.key(list[i+1]) + if !ok1 || !ok2 { + continue + } + if y, ok := s.only(dk); !ok || y.dst != uk { + continue + } + for _, v := range fs.defs { + if s.cv.Edges["ddg|"+dk+"|"+uk+"|"+v.Name()+"|ssa"] { + cands = append(cands, cand{list[i], list[i+1], dk, uk, v}) + } + } + } + } + for len(cands) > 0 { + i := c.choose(len(cands)) + x := cands[i] + cands = slices.Delete(cands, i, i+1) + k := c.fresh("mr9k") + name := x.v.Name() + db := newDelta(s, desc(s, x.u, "kill "+name+" before"), map[string]string{k: "statement"}) + db.d.CDGFromCFG = true + // Every def edge of D on x moves to the kill: the ssa edges to its + // uses, and the reaching-defs port edge to the formal_out of a + // named result x (the @exit rule, spec §4.7), whose var is the + // formal_out's `of`. + outs := filter(outOf(s.cv, "ddg", x.dk), func(e edgeRec) bool { + return e.label == name || e.prov == "reaching-defs" && strings.HasPrefix(e.dst, "@formal_out") && s.resultNamed(name) + }) + if slices.ContainsFunc(outs, func(e edgeRec) bool { return e.prov != "ssa" && !strings.HasPrefix(e.dst, "@formal_out") }) { + fe.skip("MR9: site: non-ssa ddg on the variable") + continue + } + for _, e := range outs { + db.rm(e) + db.add(DEdge{Kind: "ddg", Src: Mark(k), Dst: Pre(e.dst), Label: e.label, Prov: e.prov}) + } + db.rm(edgeRec{kind: "cfg", src: x.dk, dst: x.uk, label: "fallthrough"}) + db.add(CFG(Pre(x.dk), Mark(k), "fallthrough")) + db.add(CFG(Mark(k), Pre(x.uk), "fallthrough")) + if !db.done(fe, "MR9") { + continue + } + return []edit{insert(s.f.off(x.u.Pos()), mark(k), lit(name+" = "+zeroLit(x.v.Type())+"; "))} + } + return nil +}) + +// ─── MR10: insert a copy ────────────────────────────────────────────────────── + +// MR10 inserts `y := x` before a statement U and rewrites U's one read of +// local scalar x to y. U is a safe statement or a safe return that names x +// only there. Expected delta: +// - DDG: each D→U on x becomes D→C on x (with its port mirror when D is +// @entry); C→U on y. +// - CFG: each X→U becomes X→C (same kind); C→U fallthrough. +// - CDG: derived (C gets U's controllers). +var MR10 = deltaTransform("MR10", func(c *tctx, s *scope, fe *fileEdit) []edit { + if usesPrefix(s.f, "mr10y") { + return nil + } + type cand struct { + u ast.Stmt + uk string + id *ast.Ident + v *types.Var + } + var cands []cand + info := s.f.Pkg.TypesInfo + for _, list := range s.lists() { + for _, st := range list { + fs, ok := s.safeStmt(st) + if !ok && !s.safeReturn(st) { + continue + } + uk, ok := s.key(st) + if !ok { + continue + } + byVar := map[*types.Var][]*ast.Ident{} + ast.Inspect(st, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + if v, ok := s.scalar(s.obj(id)); ok { + byVar[v] = append(byVar[v], id) + } + } + return true + }) + for _, v := range slices.SortedFunc(maps.Keys(byVar), func(a, b *types.Var) int { return int(a.Pos() - b.Pos()) }) { + ids := byVar[v] + if len(ids) != 1 || info.Uses[ids[0]] != v { + continue // named twice, or only written + } + if fs != nil && slices.Contains(fs.defs, v) { + continue // an operator assignment reads and writes x + } + cands = append(cands, cand{st, uk, ids[0], v}) + } + } + } + for len(cands) > 0 { + i := c.choose(len(cands)) + x := cands[i] + cands = slices.Delete(cands, i, i+1) + name := x.v.Name() + ins, ok := ssaOn(into(s.cv, "ddg", x.uk), name) + if !ok || len(ins) == 0 { + fe.skip("MR10: site: no ssa def of the variable reaches it") + continue + } + cm, y := c.fresh("mr10c"), c.fresh("mr10y") + db := newDelta(s, desc(s, x.u, "copy "+name+" to "+y), map[string]string{cm: "statement"}) + db.d.CDGFromCFG = true + for _, e := range ins { + db.rm(e) + db.add(DDG(Pre(e.src), Mark(cm), name)) + } + db.add(DDG(Mark(cm), Pre(x.uk), y)) + for _, e := range into(s.cv, "cfg", x.uk) { + db.rm(e) + db.add(CFG(Pre(e.src), Mark(cm), e.label)) + } + db.add(CFG(Mark(cm), Pre(x.uk), "fallthrough")) + if !db.done(fe, "MR10") { + continue + } + return []edit{ + insert(s.f.off(x.u.Pos()), mark(cm), lit(y+" := "+name+"; ")), + {a: s.f.off(x.id.Pos()), z: s.f.off(x.id.End()), anchor: true, out: []piece{lit(y)}}, + } + } + return nil +}) + +// ─── MR11: swap two independent statements ──────────────────────────────────── + +// MR11 swaps adjacent safe statements S1; S2 that share no variable by name +// (neither writes a name the other mentions, so a := cannot shadow). Their +// text moves, and their keys map to the new positions. Expected delta: +// - DDG: none (after mapping). +// - CFG: each X→S1 becomes X→S2; S1→S2 becomes S2→S1; S2's out-edges +// leave S1 instead (same kinds). +// - CDG: derived. It equals the original's, since S1 and S2 sit in one +// basic block. +var MR11 = deltaTransform("MR11", func(c *tctx, s *scope, fe *fileEdit) []edit { + type cand struct { + s1, s2 ast.Stmt + k1, k2 string + } + var cands []cand + for _, list := range s.lists() { + for i := 0; i+1 < len(list); i++ { + f1, ok1 := s.safeStmt(list[i]) + f2, ok2 := s.safeStmt(list[i+1]) + if !ok1 || !ok2 { + continue + } + if slices.ContainsFunc(f1.defs, func(v *types.Var) bool { return f2.names[v.Name()] }) || + slices.ContainsFunc(f2.defs, func(v *types.Var) bool { return f1.names[v.Name()] }) { + continue + } + k1, ok1 := s.key(list[i]) + k2, ok2 := s.key(list[i+1]) + if !ok1 || !ok2 { + continue + } + if y, ok := s.only(k1); !ok || y.dst != k2 { + continue + } + if _, ok := s.only(k2); !ok { + continue + } + cands = append(cands, cand{list[i], list[i+1], k1, k2}) + } + } + if len(cands) == 0 { + return nil + } + x := cands[c.choose(len(cands))] + db := newDelta(s, desc(s, x.s1, "swap with the next statement"), nil) + db.d.CDGFromCFG = true + for _, e := range into(s.cv, "cfg", x.k1) { + db.rm(e) + db.add(CFG(Pre(e.src), Pre(x.k2), e.label)) + } + db.rm(edgeRec{kind: "cfg", src: x.k1, dst: x.k2, label: "fallthrough"}) + db.add(CFG(Pre(x.k2), Pre(x.k1), "fallthrough")) + for _, e := range outOf(s.cv, "cfg", x.k2) { + db.rm(e) + db.add(CFG(Pre(x.k1), Pre(e.dst), e.label)) + } + if !db.done(fe, "MR11") { + return nil + } + f := s.f + a1, z1 := f.off(x.s1.Pos()), f.off(x.s1.End()) + a2, z2 := f.off(x.s2.Pos()), f.off(x.s2.End()) + return []edit{{a: a1, z: z2, out: []piece{cp(f, a2, z2), cp(f, z1, a2), cp(f, a1, z1)}}} +}) + +// ─── MR12: an early return ──────────────────────────────────────────────────── + +// MR12 inserts `if c := true; c { return }` before a safe +// statement S of the function's top-level statement list. The function's +// results are unnamed, so the return names no variable. Expected delta: +// - CFG: each X→S becomes X→B (same kind); B→R true; R→@exit return; +// B→S false. +// - CDG: derived. Every node after S that @entry controlled is now +// controlled by B. +// - DDG: R → each @formal_out of a result (reaching-defs), the return's +// port wiring. S is not in a loop, so the guard c adds no edge (see +// MR8). +var MR12 = deltaTransform("MR12", func(c *tctx, s *scope, fe *fileEdit) []edit { + for k, attr := range s.cv.Body { + if strings.HasPrefix(k, "@formal_out") && !strings.HasPrefix(strings.Split(attr, "|")[2], "") { + return nil // a by-ref, capture or global out: an early exit adds def → exit paths + } + } + res := s.fd.Type.Results + var zeros []string + if res != nil { + for _, fl := range res.List { + if len(fl.Names) > 0 { + return nil + } + z, ok := zeroValue(s.f, s.f.Pkg.TypesInfo.TypeOf(fl.Type)) + if !ok { + return nil + } + zeros = append(zeros, z) + } + } + type cand struct { + st ast.Stmt + k string + } + var cands []cand + for _, st := range s.fd.Body.List { + if _, ok := s.safeStmt(st); !ok && !s.safeReturn(st) { + continue + } + if k, ok := s.key(st); ok { + cands = append(cands, cand{st, k}) + } + } + for len(cands) > 0 { + i := c.choose(len(cands)) + x := cands[i] + cands = slices.Delete(cands, i, i+1) + b, r, g := c.fresh("mr12b"), c.fresh("mr12r"), c.fresh("mr12c") + db := newDelta(s, desc(s, x.st, "early return before"), map[string]string{b: "branch", r: "return"}) + db.d.CDGFromCFG = true + for _, e := range into(s.cv, "cfg", x.k) { + db.rm(e) + db.add(CFG(Pre(e.src), Mark(b), e.label)) + } + db.add(CFG(Mark(b), Mark(r), "true")) + db.add(CFG(Mark(r), Pre("@exit"), "return")) + db.add(CFG(Mark(b), Pre(x.k), "false")) + for _, k := range slices.Sorted(maps.Keys(s.cv.Body)) { + if of := strings.Split(s.cv.Body[k], "|")[2]; strings.HasPrefix(k, "@formal_out") && strings.HasPrefix(of, "") { + db.add(Port(Mark(r), Pre(k), of)) + } + } + if !db.done(fe, "MR12") { + continue + } + ret := "return" + if len(zeros) > 0 { + ret += " " + strings.Join(zeros, ", ") + } + return []edit{insert(s.f.off(x.st.Pos()), mark(b), lit("if "+g+" := true; "+g+" { "), mark(r), lit(ret+" }; "))} + } + return nil +}) + +// zeroValue spells t's zero value in f, using no variable. +func zeroValue(f *SrcFile, t types.Type) (string, bool) { + if t == nil { + return "", false + } + if _, isTP := t.(*types.TypeParam); isTP { + return "", false + } + switch u := t.Underlying().(type) { + case *types.Basic: + switch { + case u.Info()&types.IsString != 0: + return `""`, true + case u.Info()&types.IsBoolean != 0: + return "false", true + case u.Info()&types.IsNumeric != 0: + return "0", true + case u.Kind() == types.UnsafePointer: + return "nil", true + } + return "", false + case *types.Pointer, *types.Slice, *types.Map, *types.Chan, *types.Signature, *types.Interface: + return "nil", true + case *types.Struct, *types.Array: + s, ok := spell(f, t) + if !ok { + return "", false + } + return s + "{}", true + } + return "", false +} + +// ─── MR13: a constant sink argument ─────────────────────────────────────────── + +// MR13 replaces the argument x of a call statement `g(…, x, …)` with the +// zero literal of x's type. x is a local scalar that the callable also names +// elsewhere, and every other argument is a constant. g is a plain function +// (no receiver, not variadic) whose formal_outs are only its results, and +// that touches no global, so the argument's value can only come from x. +// Before the change, some non-global @formal_in of the caller reaches the +// argument's actual_in over data edges. +// +// Expected (precision): afterwards no non-global @formal_in of the caller +// reaches that actual_in over data edges (ddg ∪ summary ∪ param_in ∪ +// param_out). So "no flow" can happen. +// Nothing may be added anywhere, and nothing removed but the caller's ddg +// edges into the call on x and summary edges anywhere (they follow from the +// flows that went). +var MR13 = Transform{Name: "MR13", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + if c.pre == nil { + return nil, fmt.Errorf("MR13 needs the original's analysis") + } + info := f.Pkg.TypesInfo + fe := &fileEdit{} + var edits []edit + g := c.pre.Graph(true) + for _, fd := range funcDecls(f) { + if fd.Body == nil { + continue + } + id := funcIdentity(f, fd) + cv := c.pre.Calls[id] + if cv == nil || !cv.L3 { + fe.skip("MR13: callable: no L3 result") + continue + } + s := &scope{f: f, fd: fd, id: id, cv: cv, pre: c.pre, pinned: pinnedVars(f, fd.Body)} + uses := map[*types.Var]int{} + ast.Inspect(fd.Body, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + if v, ok := info.Uses[id].(*types.Var); ok { + uses[v]++ + } + } + return true + }) + var srcs []string + for k, b := range cv.Body { + if strings.HasPrefix(k, "@formal_in:") && !strings.HasPrefix(strings.Split(b, "|")[2], ":") { + srcs = append(srcs, Node(id, k)) + } + } + slices.Sort(srcs) + var cands []mr13Cand + for _, list := range s.lists() { + for _, st := range list { + x, ok := mr13Site(s, st, uses) + if !ok { + continue + } + if !g.ReachesFrom(srcs, Node(id, x.vk)) { + fe.skip("MR13: site: no flow to the argument before") + continue + } + cands = append(cands, x) + } + } + if len(cands) == 0 { + fe.skip("MR13: callable: no eligible site") + continue + } + x := cands[c.choose(len(cands))] + d := &Delta{Site: desc(s, x.arg, "constant argument for "+x.v.Name()), Callable: id, AtMost: true} + for _, e := range filter(into(cv, "ddg", x.ck), func(e edgeRec) bool { return e.label == x.v.Name() }) { + d.Remove = append(d.Remove, e.dedge()) + } + fe.sites = append(fe.sites, Site{Desc: d.Site, Callable: id, Delta: d, NoFlow: []Ref{Pre(x.vk)}}) + fe.touched = append(fe.touched, id) + edits = append(edits, edit{a: f.off(x.arg.Pos()), z: f.off(x.arg.End()), anchor: true, out: []piece{lit(zeroLit(x.v.Type()))}}) + } + if len(edits) == 0 { + return fe, errSkip("MR13: no eligible site") + } + ps, err := applyEdits(f, edits) + fe.pieces = ps + return fe, err +}} + +type mr13Cand struct { + arg *ast.Ident + v *types.Var + ck, vk string +} + +// mr13Site is st as an MR13 site (see MR13). +func mr13Site(s *scope, st ast.Stmt, uses map[*types.Var]int) (mr13Cand, bool) { + info := s.f.Pkg.TypesInfo + es, ok := st.(*ast.ExprStmt) + if !ok { + return mr13Cand{}, false + } + call, ok := es.X.(*ast.CallExpr) + if !ok || call.Ellipsis.IsValid() { + return mr13Cand{}, false + } + fn, ok := ast.Unparen(call.Fun).(*ast.Ident) + if !ok { + return mr13Cand{}, false + } + callee, ok := info.Uses[fn].(*types.Func) + if !ok { + return mr13Cand{}, false + } + sig := callee.Type().(*types.Signature) + if sig.Variadic() || sig.Recv() != nil || sig.TypeParams() != nil || sig.Params().Len() != len(call.Args) { + return mr13Cand{}, false + } + ck := s.f.key(call.Pos()) + cb, ok := s.cv.Body[ck] + if !ok { + return mr13Cand{}, false + } + ccv := s.pre.Calls[strings.Split(cb, "|")[4]] + if ccv == nil { + return mr13Cand{}, false + } + for _, b := range ccv.Body { + f := strings.Split(b, "|") + if strings.HasPrefix(f[2], ":") || f[0] == "formal_out" && !strings.HasPrefix(f[2], "") { + return mr13Cand{}, false + } + } + var x mr13Cand + for i, a := range call.Args { + if tv, ok := info.Types[a]; ok && tv.Value != nil { + continue + } + id, ok := a.(*ast.Ident) + if !ok || x.arg != nil { + return mr13Cand{}, false + } + v, ok := s.scalar(info.Uses[id]) + // A local must keep a use, or it no longer compiles; a parameter + // need not. + if !ok || uses[v] < 2 && !isParam(s.fd, v) { + return mr13Cand{}, false + } + pname := sig.Params().At(i).Name() + if pname == "" || pname == "_" { + pname = "_" + strconv.Itoa(i) + } + x = mr13Cand{arg: id, v: v, ck: ck} + for _, k := range slices.Sorted(maps.Keys(s.cv.Body)) { + if strings.HasPrefix(k, ck+"/actual_in:") && strings.Split(s.cv.Body[k], "|")[2] == pname { + x.vk = k + } + } + } + return x, x.arg != nil && x.vk != "" +} + +// isParam reports whether v is a parameter, receiver or result of fd. +func isParam(fd *ast.FuncDecl, v *types.Var) bool { + for _, fl := range []*ast.FieldList{fd.Recv, fd.Type.Params, fd.Type.Results} { + if fl == nil { + continue + } + for _, f := range fl.List { + for _, n := range f.Names { + if n.Pos() == v.Pos() { + return true + } + } + } + } + return false +} + +// ─── MR14: route a call ─────────────────────────────────────────────────────── + +// MR14 routes one direct call statement `f(args)` per function, cycling +// through three forms: +// +// (a) through a single-implementation interface: +// mr14iN(mr14tN{}).f(args), where mr14tN.f calls f with the same +// parameter names; +// (b) through an in-place closure: func() { f(args) }(), in a function +// with no other literal (a new one would renumber closure_N); +// (c) deferred: defer f(args), only when the call is the function's last +// top-level statement, so it runs where it did (at exit, after every +// other statement). +// +// Oracle: one-way reachability. For (a), every summary edge of the +// original call site must have its counterpart (same `of` at both ends) at +// the new call site. +// +// Two flows the original has are not semantic and are tightened away +// (found on the fixtures, ruled false positives of the relation): +// - (c) anywhere but last: a deferred call runs at exit, so what it +// writes can no longer reach the statements after it +// (`defer readFull(r, arr[:]); return arr[0]` returns before +// readFull runs). Restricted to the last statement. +// - (b): a discarded result reaches the rest of the caller only by the +// coarse wiring (its actual_out → the call node, its host, spec §4.7) +// and then the call node's control dependence, which models that the +// call may panic, an exception edge (out of scope). Inside the +// closure that path ends at the closure. The original's graph leaves +// out the out-edges of the routed call's result actual_outs (Site.Cut). +var MR14 = Transform{Name: "MR14", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + if c.pre == nil { + return nil, fmt.Errorf("MR14 needs the original's analysis") + } + if usesPrefix(f, "mr14") { + return nil, errSkip("name clash") + } + info := f.Pkg.TypesInfo + fe := &fileEdit{} + var edits []edit + var tail []piece + for _, fd := range funcDecls(f) { + if fd.Body == nil { + continue + } + id := funcIdentity(f, fd) + cv := c.pre.Calls[id] + if cv == nil || !cv.L3 { + fe.skip("MR14: callable: no L3 result") + continue + } + type cand struct { + call *ast.CallExpr + callee *types.Func + ck string + ifaceOK bool + last bool + } + var cands []cand + last := map[ast.Stmt]bool{} + if n := len(fd.Body.List); n > 0 { + last[fd.Body.List[n-1]] = true + } + stmtLists(fd.Body, func(list []ast.Stmt) { + for _, st := range list { + es, ok := st.(*ast.ExprStmt) + if !ok { + continue + } + call, ok := es.X.(*ast.CallExpr) + if !ok { + continue + } + var callee *types.Func + switch fn := ast.Unparen(call.Fun).(type) { + case *ast.Ident: + callee, _ = info.Uses[fn].(*types.Func) + case *ast.SelectorExpr: + callee, _ = info.Uses[fn.Sel].(*types.Func) + } + if callee == nil { + continue + } + ck := f.key(call.Pos()) + if _, ok := cv.Body[ck+"/actual_in:0"]; !ok { + continue // no resolved callee with vertices + } + cands = append(cands, cand{call, callee, ck, ifaceForm(f, call, callee), last[st]}) + } + }) + if len(cands) == 0 { + fe.skip("MR14: callable: no direct call statement") + continue + } + x := cands[c.choose(len(cands))] + // A new literal would renumber the function's later closures + // (closure_N follows source order), so (b) needs a function + // without literals. + noLit := !contains(fd.Body, func(*ast.FuncLit) bool { return true }) + forms := []int{} + if x.ifaceOK { + forms = append(forms, 0) + } + if noLit { + forms = append(forms, 1) + } + if x.last { + forms = append(forms, 2) + } + if len(forms) == 0 { + fe.skip("MR14: site: no applicable form") + continue + } + form := forms[c.n%len(forms)] + a, z := f.off(x.call.Pos()), f.off(x.call.End()) + site := Site{Callable: id} + switch form { + case 0: + n := c.fresh("") + iface, impl := "mr14i"+n, "mr14t"+n + sig := x.callee.Type().(*types.Signature) + var params, names []string + for i := range sig.Params().Len() { + p := sig.Params().At(i) + t, _ := spell(f, p.Type()) + params = append(params, p.Name()+" "+t) + names = append(names, p.Name()) + } + var results []string + for i := range sig.Results().Len() { + t, _ := spell(f, sig.Results().At(i).Type()) + results = append(results, t) + } + msig := "(" + strings.Join(params, ", ") + ")" + if len(results) > 0 { + msig += " (" + strings.Join(results, ", ") + ")" + } + body := x.callee.Name() + "(" + strings.Join(names, ", ") + ")" + if len(results) > 0 { + body = "return " + body + } + m := x.callee.Name() + tail = append(tail, lit("\n\n// "+iface+" routes "+m+" through an interface (MR14).\ntype "+iface+" interface{ "+m+msig+" }\n\ntype "+impl+" struct{}\n\nfunc ("+impl+") "+m+msig+" { "+body+" }\n")) + // Only the function part is replaced: the arguments keep their + // text and positions. + edits = append(edits, edit{a: a, z: f.off(x.call.Lparen), anchor: true, out: []piece{lit(iface + "(" + impl + "{})." + m)}}) + site.Desc = desc(&scope{f: f, fd: fd}, x.call, "call "+m+" through "+iface) + for _, e := range edgesOf(cv, "sum") { + if strings.HasPrefix(e.src, x.ck+"/") && strings.HasPrefix(e.dst, x.ck+"/") { + site.Summary = append(site.Summary, SumCheck{Pre: e.src + " -> " + e.dst, Src: Pre(e.src), Dst: Pre(e.dst)}) + } + } + case 1: + edits = append(edits, insert(a, lit("func() { ")), insert(z, lit(" }()"))) + site.Desc = desc(&scope{f: f, fd: fd}, x.call, "call through a closure") + for k, b := range cv.Body { + if strings.HasPrefix(k, x.ck+"/actual_out") && strings.HasPrefix(strings.Split(b, "|")[2], "") { + site.Cut = append(site.Cut, Node(id, k)) + } + } + case 2: + edits = append(edits, insert(a, lit("defer "))) + site.Desc = desc(&scope{f: f, fd: fd}, x.call, "defer the call") + } + c.n++ + fe.sites = append(fe.sites, site) + fe.touched = append(fe.touched, id) + } + if len(edits) == 0 { + return fe, errSkip("MR14: no eligible site") + } + if len(tail) > 0 { + edits = append(edits, insert(len(f.Src), tail...)) + } + ps, err := applyEdits(f, edits) + fe.pieces = ps + return fe, err +}} + +// ifaceForm reports whether call can be routed through an interface: f is a +// non-generic, non-variadic function of the file's package, called by its +// bare name, with named, distinct parameters whose types (and results') the +// file can spell. +func ifaceForm(f *SrcFile, call *ast.CallExpr, callee *types.Func) bool { + if _, ok := ast.Unparen(call.Fun).(*ast.Ident); !ok || callee.Pkg() != f.Pkg.Types { + return false + } + sig := callee.Type().(*types.Signature) + if sig.Recv() != nil || sig.Variadic() || sig.TypeParams() != nil || call.Ellipsis.IsValid() { + return false + } + seen := map[string]bool{} + for i := range sig.Params().Len() { + p := sig.Params().At(i) + if p.Name() == "" || p.Name() == "_" || seen[p.Name()] || p.Name() == callee.Name() { + return false + } + seen[p.Name()] = true + if _, ok := spell(f, p.Type()); !ok { + return false + } + } + for i := range sig.Results().Len() { + if _, ok := spell(f, sig.Results().At(i).Type()); !ok { + return false + } + } + return sig.Params().Len() > 0 +} + +// ─── MR15: inline the MR6 helpers back ──────────────────────────────────────── + +// Inline6 undoes MR6 on its output: each `[var …; ]outs = mr6h_…(args)` +// statement becomes the helper's statements again, and the helpers are +// deleted. MR15 is MR6 then Inline6, which must give back the original +// source byte for byte, and the original analysis exactly. +var Inline6 = Transform{Name: "Inline6", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + helpers := map[string]*ast.FuncDecl{} + for _, fd := range funcDecls(f) { + if strings.HasPrefix(fd.Name.Name, Prefix6) { + helpers[fd.Name.Name] = fd + } + } + if len(helpers) == 0 { + return nil, errSkip("no MR6 helper") + } + fe := &fileEdit{} + var edits []edit + for _, fd := range funcDecls(f) { + if helpers[fd.Name.Name] != nil || fd.Body == nil { + continue + } + stmtLists(fd.Body, func(list []ast.Stmt) { + for i, st := range list { + name := helperCalled(st) + h := helpers[name] + if h == nil { + continue + } + // MR6 writes one `var` per fresh local: the call's results + // beyond its arguments. + start := i - helperFresh(st) + if start < 0 || slices.ContainsFunc(list[start:i], func(d ast.Stmt) bool { return !isVarDecl(d) }) { + return + } + body := h.Body.List + if len(body) > 0 { + if _, ok := body[len(body)-1].(*ast.ReturnStmt); ok { + body = body[:len(body)-1] + } + } + if len(body) == 0 { + continue + } + edits = append(edits, edit{a: f.off(list[start].Pos()), z: f.off(st.End()), + out: []piece{cp(f, f.off(body[0].Pos()), f.off(body[len(body)-1].End()))}}) + fe.sites = append(fe.sites, Site{Desc: f.Rel + ": inline " + name}) + } + }) + } + for _, h := range helpers { + a := f.off(h.Doc.Pos()) - 2 + z := f.off(h.End()) + 1 + if a < 0 || string(f.Src[a:a+2]) != "\n\n" || z > len(f.Src) || f.Src[z-1] != '\n' { + return nil, fmt.Errorf("%s: helper %s is not as MR6 writes it", f.Rel, h.Name.Name) + } + edits = append(edits, edit{a: a, z: z}) + } + ps, err := applyEdits(f, edits) + fe.pieces = ps + return fe, err +}} + +// helperFresh is the number of fresh locals an MR6 call statement assigns. +func helperFresh(st ast.Stmt) int { + as, ok := st.(*ast.AssignStmt) + if !ok { + return 0 + } + return len(as.Lhs) - len(as.Rhs[0].(*ast.CallExpr).Args) +} + +func isVarDecl(st ast.Stmt) bool { + d, ok := st.(*ast.DeclStmt) + if !ok { + return false + } + g, ok := d.Decl.(*ast.GenDecl) + return ok && g.Tok == token.VAR +} + +// helperCalled is the MR6 helper st calls, or "". +func helperCalled(st ast.Stmt) string { + var call ast.Expr + switch st := st.(type) { + case *ast.AssignStmt: + if len(st.Rhs) == 1 { + call = st.Rhs[0] + } + case *ast.ExprStmt: + call = st.X + } + ce, ok := call.(*ast.CallExpr) + if !ok { + return "" + } + id, ok := ce.Fun.(*ast.Ident) + if !ok || !strings.HasPrefix(id.Name, Prefix6) { + return "" + } + return id.Name +} diff --git a/internal/sdg/testdata/wire_sdg_off.golden b/internal/sdg/testdata/wire_sdg_off.golden index 1852fd5..071d7d3 100644 --- a/internal/sdg/testdata/wire_sdg_off.golden +++ b/internal/sdg/testdata/wire_sdg_off.golden @@ -279,6 +279,11 @@ example.com/dataflow/sdg.PassOuter example.com/dataflow/sdg.PrunedCall 246:2 -> @formal_out reaching-defs @formal_in:0 -> 246:2 v reaching-defs +example.com/dataflow/sdg.ScalarSink + 510:2 -> 510:2/actual_in:0 v reaching-defs + 510:2/actual_out -> 510:2 reaching-defs + 511:2 -> @formal_out reaching-defs + @formal_in:0 -> 509:2 a reaching-defs example.com/dataflow/sdg.Stores 40:2 -> 40:2/actual_in:0 s reaching-defs 40:2 -> 40:2/actual_in:1 k reaching-defs @@ -405,6 +410,9 @@ example.com/dataflow/sdg.readConfig @formal_in:0 -> 118:2 example.com/dataflow/sdg::Config reaching-defs example.com/dataflow/sdg.readConst 126:2 -> @formal_out reaching-defs +example.com/dataflow/sdg.sinkInt + 504:27 -> @formal_out reaching-defs + @formal_in:0 -> 504:27 v reaching-defs example.com/dataflow/sdg.sqlStore.Get 29:2 -> @formal_out:0 reaching-defs @formal_in:0 -> 29:2 s.dsn reaching-defs diff --git a/scripts/metamorphic.sh b/scripts/metamorphic.sh new file mode 100755 index 0000000..f3defe3 --- /dev/null +++ b/scripts/metamorphic.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# metamorphic.sh - run the L3/L4 metamorphic relations on prepared eval apps. +# +# Usage: +# scripts/metamorphic.sh APP [APP...] e.g. scripts/metamorphic.sh memos nats-server +# +# Each APP is a clone under $EVAL_SRC, prepared with scripts/eval_l3.sh --prepare APP. A missing +# app is skipped by the test (and reported). The relations transform a copy of the app in a temp +# dir and analyze it in-process; the app itself is never modified. Each relation samples 20 files +# and 50 callables with seed 1 (internal/metamorphic/realapp_test.go). +# +# Environment: +# EVAL_SRC clones (default /tmp/eval) +# EVAL_GO Go toolchain for the test run (default go1.27.3; eval_l3.sh's default) +# MR_RELATIONS relations to run, space-separated (default: all of MR1-MR6, MR8-MR15) +# MR_TIMEOUT go test -timeout (default 120m) +# MR_LOG log file (default $EVAL_OUT/metamorphic-.log) +# +# Prints one MRSTAT line per relation and app (sites, reachability pairs, violations, time, +# skips) and exits non-zero on any violation. +set -u + +EVAL_SRC=${EVAL_SRC:-/tmp/eval} +EVAL_OUT=${EVAL_OUT:-/tmp/eval-out} +EVAL_GO=${EVAL_GO:-go1.27.3} +MR_TIMEOUT=${MR_TIMEOUT:-120m} +MR_LOG=${MR_LOG:-$EVAL_OUT/metamorphic-$(date +%Y-%m-%d).log} +REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd) + +[ $# -gt 0 ] || { sed -n '2,9p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2; } +for a in "$@"; do + [ -f "$EVAL_SRC/$a/go.mod" ] || echo "warning: $a is not prepared at $EVAL_SRC/$a (skipped)" >&2 +done +mkdir -p "$(dirname "$MR_LOG")" + +start=$(date +%s) +(cd "$REPO_ROOT" && EVAL_SRC=$EVAL_SRC MR_APPS="$*" MR_RELATIONS="${MR_RELATIONS:-}" \ + GOTOOLCHAIN=${GOTOOLCHAIN:-$EVAL_GO} \ + go test -tags metamorphic -count=1 -timeout "$MR_TIMEOUT" -v -run 'TestMR_RealApps' ./internal/metamorphic/) \ + >"$MR_LOG" 2>&1 +rc=$? +end=$(date +%s) + +grep -E 'MRLOAD|MRSTAT' "$MR_LOG" | sed 's/^ *[a-z_]*\.go:[0-9]*: //' +grep -E '^\s+(realapp|mr)_test\.go:[0-9]+: MR[0-9]+ on ' "$MR_LOG" | sed 's/^ *//' | head -200 +echo "log: $MR_LOG ($((end - start)) s, go test rc $rc)" +exit $rc diff --git a/testdata/dataflow/sdg/sdg.go b/testdata/dataflow/sdg/sdg.go index e02ec69..9bddab0 100644 --- a/testdata/dataflow/sdg/sdg.go +++ b/testdata/dataflow/sdg/sdg.go @@ -499,3 +499,14 @@ func CaptureAlias(p *Inner, x int) { f := func() { q := p; q.n = x } f() } + +// sinkInt is a sink with no by-ref or global output. +func sinkInt(v int) int { return v } + +// ScalarSink passes a local scalar computed from a parameter to sinkInt (the +// metamorphic relations' constant-argument and call-routing sites). +func ScalarSink(a int) int { + x := a + 1 + sinkInt(x) + return x +} From 40b03ea28fec6dc08103f37058db63529f26afcd Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 12:11:47 -0400 Subject: [PATCH 28/65] fix(l3): a closure's host writes through every reference-holding capture A literal writing through a local alias of a captured pointer (`q := p; q.n = x`) writes no path of p, so at -a 3 its host did not weak-define p.* and a later read of p.n had no edge from it. Add x.* for every capture x that may hold a reference, mirroring 078616f at L4. --- docs/design/specs/v2-l3-l4-dataflow.md | 5 ++- internal/dataflow/defuse.go | 15 +++++---- internal/dataflow/defuse_entry.go | 37 +++++++++++++++++++++-- internal/dataflow/defuse_test.go | 12 +++++--- internal/dataflow/reachdefs_test.go | 4 +++ internal/sdg/testdata/wire_sdg_off.golden | 2 ++ testdata/dataflow/flow/flow.go | 13 ++++++++ 7 files changed, 74 insertions(+), 14 deletions(-) diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index 9a59232..ffcb755 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -326,7 +326,10 @@ port wiring (§9 Phase 2 prerequisites), and L4 still adds capture ports on top: - **Gap 1 — writes to captured variables inside closures.** `func(){ x = secret }()` or `once.Do(func(){ cfg = load() })` writes `x` / `cfg` in the closure's own body. **L3 fix:** the node containing the literal weak-defines each captured variable the literal (or a literal nested - in it) may write; the write-set is computed syntactically per literal. In sequential code a + in it) may write; the write-set is computed syntactically per literal, and (amended 2026-10-27, + from metamorphic relation MR5) also holds `x.*` for every captured `x` that may hold a reference + (`MayHoldRef`): a write through a local alias, `q := p; q.n = v`, writes no path of `p`. In + sequential code a closure cannot run before it exists, so this is complete apart from goroutine interleaving (a non-goal). Expect more `ddg` edges at capture sites (measured on Kubernetes). **L4** then adds a traceable source through capture ports (§4.6 closures rule). diff --git a/internal/dataflow/defuse.go b/internal/dataflow/defuse.go index 6bfd0da..87452ff 100644 --- a/internal/dataflow/defuse.go +++ b/internal/dataflow/defuse.go @@ -114,13 +114,16 @@ const DeferPrefix = ":" // - Capture writes (gap 1). A function literal may run at any later point // (called in place, stored, passed to once.Do, deferred, launched), and // its writes to the variables it captures are writes of this callable. -// Let W be the union of CaptureWrites over the outermost literals of fn's +// Let W be the union of hostWrites over the outermost literals of fn's // body (a literal nested in another belongs to the outer one's unit, and // CaptureWrites already includes what it writes). The host of a literal // (the node whose DefUse is computed from an expression containing it: // for `defer func(){…}()` the deferred call node, for `go func(){…}()` -// the go node) weak-defines each path of CaptureWrites(lit). Every strong -// def in fn whose base is a base of W is weak instead (@entry excepted: +// the go node) weak-defines each path of CaptureWrites(lit), and x.* for +// every capture x that may hold a reference (hostWrites: a write +// through a local alias, `q := p; q.n = v`, is no path of p). W is the +// union of those. Every strong def in fn whose base is a base of W is +// weak instead (@entry excepted: // nothing reaches it to kill). This is flow-insensitive per captured // variable on purpose: `h := func(){ x = 1 }; x = 2; h()` must not let // x = 2 kill the closure's write. @@ -635,15 +638,15 @@ func (c *collector) captureUses(lit *ast.FuncLit) { } } -// captureWrites weak-defines CaptureWrites(lit), with the callable-level -// facts only. +// captureWrites weak-defines hostWrites(lit), with the callable-level facts +// only. func (c *collector) captureWrites(lit *ast.FuncLit) { if c.facts == nil { return } ws, ok := c.facts.capW[lit] if !ok { - ws = CaptureWrites(lit, c.info, c.k) + ws = hostWrites(lit, c.info, c.k) } for _, p := range ws { c.add(kindWeak, p) diff --git a/internal/dataflow/defuse_entry.go b/internal/dataflow/defuse_entry.go index 9da5bd6..36b77a4 100644 --- a/internal/dataflow/defuse_entry.go +++ b/internal/dataflow/defuse_entry.go @@ -3,6 +3,7 @@ package dataflow import ( "go/ast" "go/types" + "slices" "sort" "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" @@ -144,7 +145,7 @@ func freeVars(lit *ast.FuncLit, info *types.Info) []types.Object { // facts are the callable-level inputs of DefUseOf's D1 rules (spec §4.5 "L3 // precision"), computed once per callable from its body. type facts struct { - // capW is CaptureWrites of each outermost function literal of the body. + // capW is hostWrites of each outermost function literal of the body. capW map[*ast.FuncLit][]Path // weakBases is the base of every path in W, the union of capW: a strong // def on one of them is weak (gap 1). @@ -163,7 +164,7 @@ func newFacts(fn ast.Node, info *types.Info, k int, keyOf map[ast.Node]string) * } f := &facts{capW: map[*ast.FuncLit][]Path{}, weakBases: map[types.Object]bool{}} for _, l := range astwalk.Literals(body) { - ws := CaptureWrites(l.Lit, info, k) + ws := hostWrites(l.Lit, info, k) f.capW[l.Lit] = ws for _, p := range ws { f.weakBases[p.Base] = true @@ -173,6 +174,38 @@ func newFacts(fn ast.Node, info *types.Info, k int, keyOf map[ast.Node]string) * return f } +// hostWrites is what the host of function literal lit weak-defines (D1 gap +// 1): CaptureWrites(lit), plus x.* for every capture x whose type may hold a +// reference (MayHoldRef). The literal may write through x without writing a +// path of x (`q := p; q.n = v` defines q.n, never p.*), exactly as a callee +// may write through a reference argument (the call rule), so a reference- +// holding capture is written through by the shared by-ref predicate, as at +// L4 (sdg writtenCaptures). +func hostWrites(lit *ast.FuncLit, info *types.Info, k int) []Path { + ws := CaptureWrites(lit, info, k) + seen := map[Path]bool{} + for _, p := range ws { + seen[p] = true + } + var add []Path + for _, v := range freeVars(lit, info) { + if !MayHoldRef(v.Type()) { + continue + } + p := pathB{base: v, head: baseSpell(v)}.with(".*").limit(k) + if !seen[p] { + seen[p] = true + add = append(add, p) + } + } + if len(add) == 0 { + return ws + } + out := append(slices.Clip(ws), add...) + sortPaths(out) + return out +} + // CaptureWrites returns the paths function literal lit, or any literal nested // in it, may write (strong or weak defs, by DefUseOf's def rules, k-limited), // restricted to bases that are free variables of lit (freeVars: a variable diff --git a/internal/dataflow/defuse_test.go b/internal/dataflow/defuse_test.go index 1fcb072..2618924 100644 --- a/internal/dataflow/defuse_test.go +++ b/internal/dataflow/defuse_test.go @@ -270,12 +270,13 @@ func TestDefUse_DeferRegistrationCarriesArgs(t *testing.T) { // defer func(){…}(): nothing is evaluated at registration but the func // value; the deferred call uses what the closure captures and (D1 gap 1) - // weak-defines what it writes of them: err = nil. + // weak-defines what it writes of them: err = nil, and err.* (an + // interface may hold a reference: hostWrites). reg, _ = fx.du(t, "Defers", "defer func() {", 0) call, ck = fx.du(t, "Defers", "defer func() {", len("defer ")) d = ":" + ck wantDU(t, "defer func(){…}", reg, nil, ss(d), nil) - wantDU(t, "func(){…}() (deferred)", call, nil, ss("err"), ss(d, "err", "n", "x")) + wantDU(t, "func(){…}() (deferred)", call, nil, ss("err", "err.*"), ss(d, "err", "n", "x")) reg, _ = fx.du(t, "Defers", "defer mut2(s, x)", 0) call, ck = fx.du(t, "Defers", "defer mut2(s, x)", len("defer ")) @@ -332,7 +333,7 @@ func TestDefUse_EntryDefinesParamsReceiverNamedResults(t *testing.T) { t.Errorf("EntryBases(closure) = %v, want %v", names, want) } h, _ := fx.du(t, "Entry", "h := func", 0) - wantDU(t, "h := func…", h, ss("h"), nil, ss("a", "r")) + wantDU(t, "h := func…", h, ss("h"), ss("r.*"), ss("a", "r")) // A flow-fixture function with no package vars: params only. wantDU(t, "Bump @entry", flowDU(t, "Bump", dataflow.EntryKey), ss("n"), nil, nil) @@ -460,8 +461,9 @@ func TestDefUse_CaptureWrites(t *testing.T) { wantDU(t, "DeferWrap 557:2", flowDU(t, "DeferWrap", "557:2"), nil, ss(":557:8"), nil) wantDU(t, "DeferWrap 557:8", flowDU(t, "DeferWrap", "557:8"), nil, ss("err", "err.*"), ss(":557:8", "err")) wantDU(t, "DeferWrap 555:2", flowDU(t, "DeferWrap", "555:2"), nil, ss("err"), nil) - // go func(){ x = <-ch }(): the go node is the host. - wantDU(t, "CaptureGo 627:5", flowDU(t, "CaptureGo", "627:5"), nil, ss("x"), ss("ch", "x")) + // go func(){ x = <-ch }(): the go node is the host; ch, a channel, may + // hold a reference, so it weak-defines ch.* too (hostWrites). + wantDU(t, "CaptureGo 627:5", flowDU(t, "CaptureGo", "627:5"), nil, ss("ch.*", "x"), ss("ch", "x")) wantDU(t, "CaptureGo 626:2", flowDU(t, "CaptureGo", "626:2"), nil, ss("x"), nil) } diff --git a/internal/dataflow/reachdefs_test.go b/internal/dataflow/reachdefs_test.go index 10211b7..9f2de2b 100644 --- a/internal/dataflow/reachdefs_test.go +++ b/internal/dataflow/reachdefs_test.go @@ -480,6 +480,10 @@ func TestDDG_ClosureCaptureWrites(t *testing.T) { hasDDG(t, "CaptureNested", dd("613:2", "620:2", "x")) // go func(){ x = <-ch }() at 627:5: the go node hosts the literal. hasDDG(t, "CaptureGo", dd("627:5", "628:2", "x")) + // func(){ q := p; q.n = 1 }() at 716:2 writes through p by an alias: + // no path of p, but p may hold a reference, so the host weak-defines + // p.*, which reaches return p.n (found by metamorphic relation MR5). + hasDDG(t, "CaptureAliasWrite", dd("716:2", "717:2", "p.n")) } // TestDDG_DeferOperandAfterReassign (D1 gap 3): in `defer fill(b); b[0] = s; diff --git a/internal/sdg/testdata/wire_sdg_off.golden b/internal/sdg/testdata/wire_sdg_off.golden index 071d7d3..5f9d5c3 100644 --- a/internal/sdg/testdata/wire_sdg_off.golden +++ b/internal/sdg/testdata/wire_sdg_off.golden @@ -146,6 +146,8 @@ example.com/dataflow/sdg.CallVariadic 197:7/actual_out -> 197:2 reaching-defs 198:2 -> @formal_out reaching-defs example.com/dataflow/sdg.CaptureAlias + 499:2 -> 499:7/actual_in:0 :p reaching-defs + 499:2 -> @formal_out p reaching-defs 499:7/actual_out -> 499:2 :p reaching-defs 499:7/actual_out -> @formal_out p reaching-defs @formal_in:0 -> 499:2 p reaching-defs diff --git a/testdata/dataflow/flow/flow.go b/testdata/dataflow/flow/flow.go index 3c8f426..2795b3c 100644 --- a/testdata/dataflow/flow/flow.go +++ b/testdata/dataflow/flow/flow.go @@ -703,3 +703,16 @@ func PlainRecv(p plain) int { func (e *early) Bump() int { return f(e.n) } type early struct{ n int } + +// aliasT has a scalar field n. +type aliasT struct{ n int } + +// CaptureAliasWrite writes through a local alias of the captured pointer p +// inside a literal called in place: the literal writes no path of p, but p +// may hold a reference, so its host weak-defines p.* and the write reaches +// the later read of p.n (found by metamorphic relation MR5). +func CaptureAliasWrite() int { + p := &aliasT{} + func() { q := p; q.n = 1 }() + return p.n +} From 3331abea79da9c1a718e44c1c10b2ce712bddec2 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 12:41:19 -0400 Subject: [PATCH 29/65] test(l4): metamorphic harness race, CDG mismatch as violation, tighter MR13 --- internal/metamorphic/compare.go | 53 +++++++++++++++++++++---- internal/metamorphic/mr_test.go | 28 ++++++++++--- internal/metamorphic/transform.go | 9 +++++ internal/metamorphic/transform_delta.go | 19 +++++---- 4 files changed, 89 insertions(+), 20 deletions(-) diff --git a/internal/metamorphic/compare.go b/internal/metamorphic/compare.go index 44fe959..d4e930e 100644 --- a/internal/metamorphic/compare.go +++ b/internal/metamorphic/compare.go @@ -34,6 +34,7 @@ import ( "sort" "strconv" "strings" + "sync" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" ) @@ -45,7 +46,11 @@ type View struct { // "pout|src|dst|var" over Node strings. App map[string]bool - callers map[string]bool // lazily, by called + // callerOnce guards callerOf, which is derived from App on first use + // (called, Callers). A View is shared by parallel relations, so its + // lazily built state must be built once. + callerOnce sync.Once + callerOf map[string][]string // callee identity → caller identities, by param_in } // CV is one callable (or a library stand-in, Kind "external"). @@ -296,17 +301,51 @@ func (v *View) addModule(rel string, m *v2.Module, ids map[string]string, callee } } +// callerIndex builds callerOf once. +func (v *View) callerIndex() map[string][]string { + v.callerOnce.Do(func() { + set := map[string]map[string]bool{} + for e := range v.App { + f := strings.Split(e, "|") // pin | src ident | src key | dst ident | dst key | var + if f[0] != "pin" { + continue + } + if set[f[3]] == nil { + set[f[3]] = map[string]bool{} + } + set[f[3]][f[1]] = true + } + v.callerOf = make(map[string][]string, len(set)) + for callee, cs := range set { + v.callerOf[callee] = slices.Sorted(maps.Keys(cs)) + } + }) + return v.callerOf +} + // called reports whether some param_in edge targets callable ident. func (v *View) called(ident string) bool { - if v.callers == nil { - v.callers = map[string]bool{} - for e := range v.App { - if f := strings.Split(e, "|"); f[0] == "pin" { - v.callers[f[3]] = true + return len(v.callerIndex()[ident]) > 0 +} + +// Callers is ident and every callable that reaches it over param_in edges +// (its transitive callers), the callables whose summary edges a change of +// ident's flows may change. +func (v *View) Callers(ident string) map[string]bool { + idx := v.callerIndex() + out := map[string]bool{ident: true} + stack := []string{ident} + for len(stack) > 0 { + n := stack[len(stack)-1] + stack = stack[:len(stack)-1] + for _, c := range idx[n] { + if !out[c] { + out[c] = true + stack = append(stack, c) } } } - return v.callers[ident] + return out } // ─── names (MR4) ────────────────────────────────────────────────────────────── diff --git a/internal/metamorphic/mr_test.go b/internal/metamorphic/mr_test.go index f50eb3c..12592f6 100644 --- a/internal/metamorphic/mr_test.go +++ b/internal/metamorphic/mr_test.go @@ -210,6 +210,9 @@ func runRelation(t *testing.T, tg *target, r relation) *outcome { res, post := transformAndAnalyze(t, tg, r.t, tg.ws, tg.files(), pre) maps.Copy(o.skipped, res.Skipped) o.sites = len(res.Sites) + for _, v := range res.Violations { + o.fail("%s", v) + } if post == nil { return o } @@ -270,16 +273,25 @@ func checkDelta(o *outcome, pre, mapped, got *View, res *Result) { } return } - // A precision relation: nothing added anywhere; outside the sites only - // summary edges may go; at a site, its declared removals and summary - // edges. + // A precision relation: nothing is added anywhere. At a site, its + // declared removals may go (CheckAtMost). A summary edge may go only in + // a site callable or one of its transitive callers (over param_in): + // the constant cuts a flow of the site callable, which changes its own + // summary edges and those of every caller whose summary is derived from + // its flows, and nothing else. + sumOK := map[string]bool{} + for id := range atMost { + maps.Copy(sumOK, pre.Callers(id)) + } for _, d := range Diff(want, got, Scope{Skip: atMost}, 0) { - if strings.Contains(d, "|") && strings.HasSuffix(d, " missing") && strings.Contains(d, ": edge sum|") { + if id, rest, ok := strings.Cut(d, ": edge sum|"); ok && strings.HasSuffix(rest, " missing") && sumOK[id] { continue } o.fail("%s", d) } for _, d := range CheckAtMost(pre, mapped, got, res.Map, res.Marks, ds) { + // A site callable is its own caller set's root, so its summary + // edges may go too. if strings.Contains(d, ": sum|") && strings.HasSuffix(d, " removed") { continue } @@ -319,6 +331,9 @@ func checkReach(o *outcome, tg *target, pre, got *View, res *Result) { } rr := CheckReach(pre, got, res.Map, res.Touched, tg.nCallables, seed, cut...) o.pairs = rr.Pairs + if len(res.Sites) > 0 && rr.Pairs == 0 { + o.fail("%d sites but no source-sink pair checked: the oracle checked nothing", len(res.Sites)) + } gPre := pre.Graph(false) for i, l := range head(rr.Lost, 40) { msg := "lost flow " + l @@ -752,7 +767,10 @@ func TestMR9_InsertKill_Delta(t *testing.T) { CFG(k, Pre("510:2"), "fallthrough"), }, } - mapped, _ := tg.pre.view.Map(res.Map) + mapped, unmapped := tg.pre.view.Map(res.Map) + if len(unmapped) > 0 { + t.Fatalf("keys do not map: %v", unmapped) + } var ds []Delta for _, s := range res.Sites { if s.Callable == want.Callable { diff --git a/internal/metamorphic/transform.go b/internal/metamorphic/transform.go index 23fb208..30ab5be 100644 --- a/internal/metamorphic/transform.go +++ b/internal/metamorphic/transform.go @@ -297,6 +297,9 @@ type fileEdit struct { sites []Site // skips counts the sites declined, by reason. skips map[string]int + // violations are analyzer faults the transform found while choosing + // sites (a CDG that is not the reference CDG of its CFG). + violations []string } func (e *fileEdit) skip(reason string) { @@ -363,6 +366,9 @@ type Result struct { Sites []Site // Skipped counts, per reason, the files and sites left unchanged. Skipped map[string]int + // Violations are analyzer faults found while choosing sites; the + // relation reports each one. + Violations []string } // errSkip is returned by an edit that leaves its file unchanged. @@ -390,6 +396,9 @@ func (w *Workspace) Apply(t Transform, files []string, dst string, seed uint64, return nil, fmt.Errorf("%s: not an eligible file", rel) } fe, err := t.edit(c, f) + if fe != nil { + res.Violations = append(res.Violations, fe.violations...) + } var skip errSkip if errors.As(err, &skip) { res.Skipped["file: "+string(skip)]++ diff --git a/internal/metamorphic/transform_delta.go b/internal/metamorphic/transform_delta.go index 2e34b0f..c888f2a 100644 --- a/internal/metamorphic/transform_delta.go +++ b/internal/metamorphic/transform_delta.go @@ -100,10 +100,9 @@ func onCycle(cv *CV, k string) bool { return false } -// sameCDG reports whether cv's cdg equals the reference CDG of its cfg. -func sameCDG(cv *CV) bool { - return cdgDiff(cv, RefCDG(cv)) == "" -} +// cdgMismatch prefixes the deltaScope reason for a callable whose CDG is +// not the reference CDG of its CFG. +const cdgMismatch = "cdg differs from the reference: " // ─── eligibility ────────────────────────────────────────────────────────────── @@ -139,10 +138,10 @@ func deltaScope(c *tctx, f *SrcFile, fd *ast.FuncDecl) (*scope, string) { if cv == nil || !cv.L3 { return nil, "no L3 result" } - if !sameCDG(cv) { - // Its CDG is not the reference CDG of its CFG: a finding the test - // reports separately (TestRefCDG), not a site. - return nil, "cdg differs from the reference" + if d := cdgDiff(cv, RefCDG(cv)); d != "" { + // Its CDG is not the reference CDG of its CFG: an analyzer CDG bug, + // reported as a violation (deltaTransform), never as a skip. + return nil, cdgMismatch + d } return &scope{f: f, fd: fd, id: id, cv: cv, pre: c.pre, pinned: pinnedVars(f, fd.Body)}, "" } @@ -358,6 +357,10 @@ func deltaTransform(name string, pick func(c *tctx, s *scope, fe *fileEdit) []ed var edits []edit for _, fd := range funcDecls(f) { s, why := deltaScope(c, f, fd) + if strings.HasPrefix(why, cdgMismatch) { + fe.violations = append(fe.violations, fmt.Sprintf("%s: %s %s", f.Rel, funcIdentity(f, fd), why)) + continue + } if s == nil { fe.skip(name + ": callable: " + why) continue From 30d74590f074a5a6cbc45db9adbe084b6b95d76a Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 13:54:16 -0400 Subject: [PATCH 30/65] test(l4): metamorphic examples module and memos-default real-app run --- docs/design/notes/l3-dataflow-metrics.md | 3 + internal/metamorphic/mr_test.go | 299 ++++++++++++++++++---- internal/metamorphic/realapp_test.go | 20 +- scripts/metamorphic.sh | 31 ++- testdata/metamorphic/conventions.go | 51 ++++ testdata/metamorphic/doc.go | 12 + testdata/metamorphic/go.mod | 3 + testdata/metamorphic/mr01_04_layout.go | 20 ++ testdata/metamorphic/mr05_alias.go | 16 ++ testdata/metamorphic/mr06_extract.go | 9 + testdata/metamorphic/mr08_wrap.go | 27 ++ testdata/metamorphic/mr09_kill.go | 22 ++ testdata/metamorphic/mr10_copy.go | 14 + testdata/metamorphic/mr11_swap.go | 8 + testdata/metamorphic/mr12_early_return.go | 14 + testdata/metamorphic/mr13_const_sink.go | 25 ++ testdata/metamorphic/mr14_route.go | 28 ++ 17 files changed, 544 insertions(+), 58 deletions(-) create mode 100644 testdata/metamorphic/conventions.go create mode 100644 testdata/metamorphic/doc.go create mode 100644 testdata/metamorphic/go.mod create mode 100644 testdata/metamorphic/mr01_04_layout.go create mode 100644 testdata/metamorphic/mr05_alias.go create mode 100644 testdata/metamorphic/mr06_extract.go create mode 100644 testdata/metamorphic/mr08_wrap.go create mode 100644 testdata/metamorphic/mr09_kill.go create mode 100644 testdata/metamorphic/mr10_copy.go create mode 100644 testdata/metamorphic/mr11_swap.go create mode 100644 testdata/metamorphic/mr12_early_return.go create mode 100644 testdata/metamorphic/mr13_const_sink.go create mode 100644 testdata/metamorphic/mr14_route.go diff --git a/docs/design/notes/l3-dataflow-metrics.md b/docs/design/notes/l3-dataflow-metrics.md index eb619f2..61f8a12 100644 --- a/docs/design/notes/l3-dataflow-metrics.md +++ b/docs/design/notes/l3-dataflow-metrics.md @@ -112,6 +112,9 @@ python3 -I scripts/l3_metrics.py cobra=/tmp/eval-out/cobra-a3j1/analysis.json \ - **What `eval_l3.sh` does.** It runs a warm-up, `-a 2 -j 1`, `-a 3 -j 1` and two `-a 3 -j N`, all `--eager`, with each app's manifest env (dae: `GOOS=linux`). It records wall time and peak RSS from `/usr/bin/time -l`, then compares the three `-a 3` outputs byte for byte. +- **Metamorphic relations (`scripts/metamorphic.sh`).** Run the default, `scripts/metamorphic.sh` + (memos, about 2 min), for any L3/L4 change; run `scripts/metamorphic.sh --all` (memos and + nats-server, about 25 min) before a release and for the Task 18 eval. - **What `scripts/l3_metrics.py` does.** It produces every count, ratio, distribution and check in this note. It has no dependencies and loads one file at a time; the 318 MB Kubernetes root file takes about 10 s. The per-family protobuf breakdown and the before/after edge diffs were diff --git a/internal/metamorphic/mr_test.go b/internal/metamorphic/mr_test.go index 12592f6..0a56f64 100644 --- a/internal/metamorphic/mr_test.go +++ b/internal/metamorphic/mr_test.go @@ -78,9 +78,14 @@ func testdata(name string) string { return filepath.Join(filepath.Dir(file), "..", "..", "testdata", name) } +// examples is testdata/metamorphic: small functions written to be eligible +// sites of every relation, and the convention pins (conventions.go). +var examples = &target{name: "examples", dir: testdata("metamorphic")} + var fixtures = []*target{ {name: "dataflow", dir: testdata("dataflow"), pkgDirs: []string{"flow", "sdg"}}, {name: "chi", dir: testdata("chi"), nFiles: 12, nCallables: 60}, + examples, } // load loads the target's workspace and its original analysis once. @@ -678,7 +683,11 @@ func TestRefCDG(t *testing.T) { t.Errorf("%s %s: cdg differs from the reference: %s", tg.name, id, d) } } - if n < 50 { + floor := 50 + if tg == examples { + floor = 30 // a small module of focused examples + } + if n < floor { t.Errorf("%s: only %d callables", tg.name, n) } } @@ -711,18 +720,63 @@ func copyOf(t *testing.T, dir string) string { return dst } -// ─── one relation per oracle, on one named fixture site ─────────────────────── +// ─── one relation per oracle, on one named example ──────────────────────────── // // The table above runs every relation on whole samples. These three pin one -// site each, so a relation's shape reads end to end. +// site each on testdata/metamorphic, so a relation's shape reads end to end. +// Keys are found by statement text (stmtKey), never by line number. + +const exPkg = "example.com/metamorphic." -// TestMR2_ReverseDeclarations_Exact: reversing sdg/stdlib.go's declarations -// leaves the -a 4 analysis exactly as it was, once every key is moved to its -// statement's new line. +// stmtKey is the body key of the statement of examples file rel that starts +// with text, in function fn (the first such statement). +func stmtKey(t *testing.T, rel, fn, text string) string { + t.Helper() + f := examples.ws.Files[rel] + if f == nil { + t.Fatalf("no file %s", rel) + } + for _, fd := range funcDecls(f) { + if fd.Name.Name != fn { + continue + } + var key string + ast.Inspect(fd.Body, func(n ast.Node) bool { + if st, ok := n.(ast.Stmt); ok && key == "" { + if _, block := st.(*ast.BlockStmt); !block && strings.HasPrefix(string(f.Src[f.off(st.Pos()):f.off(st.End())]), text) { + key = f.key(st.Pos()) + } + } + return key == "" + }) + if key != "" { + return key + } + } + t.Fatalf("%s: no statement %q in %s", rel, text, fn) + return "" +} + +// siteIn is the site of res in callable id. +func siteIn(t *testing.T, res *Result, id string) Site { + t.Helper() + for _, s := range res.Sites { + if s.Callable == id { + return s + } + } + t.Fatalf("no site in %s", id) + return Site{} +} + +// TestMR2_ReverseDeclarations_Exact: reversing mr01_04_layout.go's +// declarations (the method Scale goes back below its type) leaves the -a 4 +// analysis exactly as it was, once every key moves to its statement's new +// line. func TestMR2_ReverseDeclarations_Exact(t *testing.T) { - tg := fixtures[0] + tg := examples tg.load(t) - res, post := transformAndAnalyze(t, tg, MR2, tg.ws, []string{"sdg/stdlib.go"}, tg.pre.view) + res, post := transformAndAnalyze(t, tg, MR2, tg.ws, []string{"mr01_04_layout.go"}, tg.pre.view) mapped, unmapped := tg.pre.view.Map(res.Map) if len(unmapped) > 0 { t.Fatalf("keys do not map: %v", unmapped) @@ -732,39 +786,33 @@ func TestMR2_ReverseDeclarations_Exact(t *testing.T) { } } -// TestMR9_InsertKill_Delta: in ScalarSink, inserting `x = 0` before -// `sinkInt(x)` moves every def edge of `x := a + 1` on x to the kill and -// splices the kill into the CFG; the CDG is the reference CDG of that CFG, -// and nothing else in the application changes. +// TestMR9_InsertKill_Delta: in KillOneUse, inserting `x = 0` before +// `sinkInt(x)` moves both def edges of `x := a + 1` on x (to the call and +// to the return) to the kill and splices the kill into the CFG; the CDG is +// the reference CDG of that CFG, and nothing else in the module changes. func TestMR9_InsertKill_Delta(t *testing.T) { - tg := fixtures[0] + tg := examples tg.load(t) - res, post := transformAndAnalyze(t, tg, MR9, tg.ws, []string{"sdg/sdg.go"}, tg.pre.view) - var site *Site - for i, s := range res.Sites { - if strings.HasSuffix(s.Callable, ".ScalarSink") { - site = &res.Sites[i] - } - } - if site == nil { - t.Fatal("MR9 chose no site in ScalarSink") - } - var k Ref // the kill, a node the transform adds (its one mark) + res, post := transformAndAnalyze(t, tg, MR9, tg.ws, []string{"mr09_kill.go"}, tg.pre.view) + site := siteIn(t, res, exPkg+"KillOneUse") + var k Ref // the kill, the node the transform adds (its one mark) for m := range site.Delta.Nodes { k = Mark(m) } - want := Delta{Site: site.Desc, Callable: site.Callable, CDGFromCFG: true, - Nodes: site.Delta.Nodes, + d := stmtKey(t, "mr09_kill.go", "KillOneUse", "x := a + 1") + u := stmtKey(t, "mr09_kill.go", "KillOneUse", "sinkInt(x)") + r := stmtKey(t, "mr09_kill.go", "KillOneUse", "return x") + want := Delta{Site: site.Desc, Callable: site.Callable, CDGFromCFG: true, Nodes: site.Delta.Nodes, Remove: []DEdge{ - DDG(Pre("509:2"), Pre("510:2"), "x"), - DDG(Pre("509:2"), Pre("511:2"), "x"), - CFG(Pre("509:2"), Pre("510:2"), "fallthrough"), + DDG(Pre(d), Pre(u), "x"), + DDG(Pre(d), Pre(r), "x"), + CFG(Pre(d), Pre(u), "fallthrough"), }, Add: []DEdge{ - DDG(k, Pre("510:2"), "x"), - DDG(k, Pre("511:2"), "x"), - CFG(Pre("509:2"), k, "fallthrough"), - CFG(k, Pre("510:2"), "fallthrough"), + DDG(k, Pre(u), "x"), + DDG(k, Pre(r), "x"), + CFG(Pre(d), k, "fallthrough"), + CFG(k, Pre(u), "fallthrough"), }, } mapped, unmapped := tg.pre.view.Map(res.Map) @@ -783,24 +831,23 @@ func TestMR9_InsertKill_Delta(t *testing.T) { if err != nil { t.Fatal(err) } - if d := Diff(exp, post.view, Scope{}, 20); len(d) > 0 { - t.Errorf("MR9 delta:\n%s", strings.Join(d, "\n")) + if diff := Diff(exp, post.view, Scope{}, 20); len(diff) > 0 { + t.Errorf("MR9 delta:\n%s", strings.Join(diff, "\n")) } } -// TestMR5_IntroduceAlias_Reach: rewriting `q.n = v` in aliasAssign as -// `mr5q := q; mr5q.n = v` (an alias of an alias) keeps every flow from the -// formal_ins of aliasAssign and of its caller CallAliasAssign: v still -// reaches the formal_out of p. +// TestMR5_IntroduceAlias_Reach: rewriting `p.n = v` in SetPtr as +// `q := p; q.n = v` keeps every flow from SetPtr's formal_ins: v still +// reaches the formal_out of p, now through the alias oracle. func TestMR5_IntroduceAlias_Reach(t *testing.T) { - tg := fixtures[0] + tg := examples tg.load(t) - res, post := transformAndAnalyze(t, tg, MR5, tg.ws, []string{"sdg/sdg.go"}, tg.pre.view) - f := "example.com/dataflow/sdg.aliasAssign" + res, post := transformAndAnalyze(t, tg, MR5, tg.ws, []string{"mr05_alias.go"}, tg.pre.view) + f := exPkg + "SetPtr" if !slices.Contains(res.Touched, f) { t.Fatalf("MR5 did not rewrite %s", f) } - rr := CheckReach(tg.pre.view, post.view, res.Map, []string{f, "example.com/dataflow/sdg.CallAliasAssign"}, 2, seed) + rr := CheckReach(tg.pre.view, post.view, res.Map, []string{f}, 1, seed) if rr.Pairs == 0 { t.Fatal("no flow checked") } @@ -808,8 +855,168 @@ func TestMR5_IntroduceAlias_Reach(t *testing.T) { t.Errorf("MR5: lost %v, unlocated %v", rr.Lost, rr.Unlocated) } // The flow the alias carries: v (formal_in 1) → p's formal_out. - g := post.view.Graph(true) - if !g.Reach(Node(f, "@formal_in:1"))[Node(f, "@formal_out")] { - t.Error("after MR5, v no longer reaches aliasAssign's formal_out for p") + if !post.view.Graph(true).Reach(Node(f, "@formal_in:1"))[Node(f, "@formal_out")] { + t.Error("after MR5, v no longer reaches SetPtr's formal_out for p") + } +} + +// ─── the examples module ────────────────────────────────────────────────────── + +// TestExamples_EveryRelationHasASite: every relation MR1-MR15 transforms at +// least one site of testdata/metamorphic (a relation whose sites all +// vanished would pass vacuously). +func TestExamples_EveryRelationHasASite(t *testing.T) { + tg := examples + tg.load(t) + for _, r := range relations { + res, err := tg.ws.Apply(r.t, tg.files(), filepath.Join(t.TempDir(), r.name), seed, tg.pre.view) + if err != nil { + t.Fatal(err) + } + t.Logf("%s: %d sites", r.name, len(res.Sites)) + if len(res.Sites) == 0 { + t.Errorf("%s has no site on the examples module (skipped: %v)", r.name, res.Skipped) + } + } + if fs := Sample(tg.files(), 2, seed); len(fs) == 0 { + t.Error("MR7 has no file on the examples module") + } + ext, err := tg.ws.Apply(MR6, tg.files(), filepath.Join(t.TempDir(), "mr15"), seed, nil) + if err != nil { + t.Fatal(err) + } + if len(ext.Files) == 0 { + t.Error("MR15 has no site on the examples module") + } +} + +// ─── convention pins (testdata/metamorphic/conventions.go) ──────────────────── +// +// Task 17b encoded four analyzer conventions as oracle rules. Each test pins +// the analyzer fact its rule rests on, so a change that invalidates the rule +// fails here first, naming it, instead of silently weakening a relation. + +// TestConvention_CaptureSiteActualInIsNotASink pins rule (i), the +// reachability oracle's sink set: a capture site's actual_in has exactly +// one out-edge, the param_in to its closure's capture formal_in (no ddg, +// no summary), so it is not a sink. A call node's actual_in is one. +func TestConvention_CaptureSiteActualInIsNotASink(t *testing.T) { + const rule = "rule (i), capture-site actual_ins are not reachability sinks (CheckReach, View.isSink)" + tg := examples + tg.load(t) + v := tg.pre.view + id := exPkg + "ConvCaptureSite" + site := stmtKey(t, "conventions.go", "ConvCaptureSite", "f := func()") // the literal's host + litKey := "" + for k, b := range v.Calls[id].Body { + if strings.HasSuffix(k, "/actual_in:0") && strings.Split(b, "|")[1] == site { + litKey = strings.TrimSuffix(k, "/actual_in:0") + } + } + if litKey == "" { + t.Fatalf("%s: ConvCaptureSite has no capture site hosted at %s", rule, site) + } + ain := Node(id, litKey+"/actual_in:0") + g := v.Graph(false) + succ := g[ain] + if len(succ) != 1 || !strings.HasPrefix(succ[0], id+".closure_1|@formal_in:") { + t.Errorf("%s: %s's out-edges are %v, want exactly the param_in to closure_1's capture formal_in", rule, ain, succ) + } + if v.isSink(ain) { + t.Errorf("%s: %s is treated as a sink", rule, ain) + } + call := Node(exPkg+"KillOneUse", stmtKey(t, "mr09_kill.go", "KillOneUse", "sinkInt(x)")+"/actual_in:0") + if !v.isSink(call) { + t.Errorf("%s: the call actual_in %s is not a sink", rule, call) + } +} + +// TestConvention_GuardSelfLoopOnlyInLoops pins rule (ii), MR8's guard edge: +// the header `if c := true; c` folds init and condition into one node, so +// the guard c adds the ddg self-loop B→B on c exactly when the wrapped +// statement is on a CFG cycle (here, in a loop body), and no edge in +// straight-line code. +func TestConvention_GuardSelfLoopOnlyInLoops(t *testing.T) { + const rule = "rule (ii), MR8's guard self-loop B→B on c iff the wrapped statement is in a loop" + tg := examples + tg.load(t) + res, post := transformAndAnalyze(t, tg, MR8, tg.ws, []string{"conventions.go"}, tg.pre.view) + for fn, wantLoop := range map[string]bool{"ConvGuardInLoop": true, "ConvGuardStraight": false} { + id := exPkg + fn + s := siteIn(t, res, id) + var b string + for m := range s.Delta.Nodes { + b = res.Marks[m].Key() + } + var guard []string + for e := range post.view.Calls[id].Edges { + if f := strings.Split(e, "|"); f[0] == "ddg" && strings.HasPrefix(f[3], "mr8c") { + guard = append(guard, e) + } + } + switch { + case wantLoop && (len(guard) != 1 || !strings.HasPrefix(guard[0], "ddg|"+b+"|"+b+"|")): + t.Errorf("%s: %s: guard edges %v, want the one self-loop on %s", rule, fn, guard, b) + case !wantLoop && len(guard) != 0: + t.Errorf("%s: %s: guard edges %v, want none", rule, fn, guard) + } + } +} + +// TestConvention_DiscardedResultReachesOnlyViaHost pins rule (iii), MR14(b)'s +// cut: a discarded result's actual_out has the one out-edge actual_out → +// call node (the coarse host wiring, spec §4.7), and the call node reaches +// the rest of the callable only by control dependence (it may panic, an +// exception edge), so the closure form may lose that path. +func TestConvention_DiscardedResultReachesOnlyViaHost(t *testing.T) { + const rule = "rule (iii), MR14(b) cuts a discarded actual_out (reaches on only via host cdg)" + tg := examples + tg.load(t) + c := tg.pre.view.Calls[exPkg+"ConvDiscardedResult"] + call := stmtKey(t, "conventions.go", "ConvDiscardedResult", "p.sum()") + ret := stmtKey(t, "conventions.go", "ConvDiscardedResult", "return p.a") + out := edgesOf(c, "ddg") + from := filter(out, func(e edgeRec) bool { return e.src == call+"/actual_out" }) + if len(from) != 1 || from[0].dst != call || from[0].label != "" { + t.Errorf("%s: %s/actual_out edges %v, want only → the call node on ", rule, call, from) + } + if !c.Edges["cdg|"+call+"|"+ret+"|"] { + t.Errorf("%s: no cdg %s → %s (the call may panic)", rule, call, ret) + } + if len(filter(out, func(e edgeRec) bool { return e.src == call && e.dst == ret })) != 0 { + t.Errorf("%s: the call node has a ddg edge to the return: the result is not discarded", rule) + } +} + +// TestConvention_NamedResultPortMovesToKill pins rule (iv), MR9 on a named +// result: the def `n = a + 1` reaches @formal_out through the @exit port +// edge (reaching-defs, var ), and the kill MR9 inserts before the +// return takes it over. +func TestConvention_NamedResultPortMovesToKill(t *testing.T) { + const rule = "rule (iv), MR9's kill takes over a named result's D → @formal_out port edge" + tg := examples + tg.load(t) + id := exPkg + "ConvNamedKill" + d := stmtKey(t, "conventions.go", "ConvNamedKill", "n = a + 1") + port := "ddg|" + d + "|@formal_out||reaching-defs" + if !tg.pre.view.Calls[id].Edges[port] { + t.Fatalf("%s: no port edge %s before the kill", rule, port) + } + res, post := transformAndAnalyze(t, tg, MR9, tg.ws, []string{"conventions.go"}, tg.pre.view) + s := siteIn(t, res, id) + var k string + for m := range s.Delta.Nodes { + k = res.Marks[m].Key() + } + nd, _, ok := (&mapper{pm: res.Map}).key("conventions.go", d) + if !ok { + t.Fatalf("%s: %s does not map", rule, d) + } + edges := post.view.Calls[id].Edges + if edges["ddg|"+nd+"|@formal_out||reaching-defs"] { + t.Errorf("%s: the def %s still reaches @formal_out past the kill", rule, nd) + } + if !edges["ddg|"+k+"|@formal_out||reaching-defs"] { + t.Errorf("%s: the kill %s has no port edge to @formal_out", rule, k) } } diff --git a/internal/metamorphic/realapp_test.go b/internal/metamorphic/realapp_test.go index a5f43b9..b12d95d 100644 --- a/internal/metamorphic/realapp_test.go +++ b/internal/metamorphic/realapp_test.go @@ -9,13 +9,14 @@ package metamorphic // // Each app is $EVAL_SRC/ (default /tmp/eval). A missing app is // skipped. MR_APPS (space-separated) picks the apps, MR_RELATIONS the -// relations (all when unset). Each relation samples 20 files and 50 -// callables with seed 1. +// relations (all when unset). Each relation samples MR_SAMPLE_FILES files +// (default 20) and MR_SAMPLE_CALLABLES callables (default 50) with seed 1. import ( "os" "path/filepath" "slices" + "strconv" "strings" "testing" ) @@ -36,11 +37,24 @@ func realTargets(t *testing.T) []*target { t.Logf("%s: not prepared at %s; skipped", n, dir) continue } - out = append(out, &target{name: n, dir: dir, nFiles: 20, nCallables: 50}) + out = append(out, &target{name: n, dir: dir, nFiles: envInt(t, "MR_SAMPLE_FILES", 20), nCallables: envInt(t, "MR_SAMPLE_CALLABLES", 50)}) } return out } +// envInt is the positive integer in env var name, or def when unset. +func envInt(t *testing.T, name string, def int) int { + v := os.Getenv(name) + if v == "" { + return def + } + n, err := strconv.Atoi(v) + if err != nil || n <= 0 { + t.Fatalf("%s=%q: want a positive integer", name, v) + } + return n +} + func TestMR_RealApps(t *testing.T) { tgs := realTargets(t) if len(tgs) == 0 { diff --git a/scripts/metamorphic.sh b/scripts/metamorphic.sh index f3defe3..06a9cce 100755 --- a/scripts/metamorphic.sh +++ b/scripts/metamorphic.sh @@ -2,19 +2,25 @@ # metamorphic.sh - run the L3/L4 metamorphic relations on prepared eval apps. # # Usage: -# scripts/metamorphic.sh APP [APP...] e.g. scripts/metamorphic.sh memos nats-server +# scripts/metamorphic.sh the default set: memos (~2 min); run it for any L3/L4 change +# scripts/metamorphic.sh --all every app below: memos nats-server (~25 min; before a release) +# scripts/metamorphic.sh APP [APP...] the named apps, e.g. scripts/metamorphic.sh nats-server # # Each APP is a clone under $EVAL_SRC, prepared with scripts/eval_l3.sh --prepare APP. A missing # app is skipped by the test (and reported). The relations transform a copy of the app in a temp -# dir and analyze it in-process; the app itself is never modified. Each relation samples 20 files -# and 50 callables with seed 1 (internal/metamorphic/realapp_test.go). +# dir and analyze it in-process; the app itself is never modified. Each relation samples +# MR_SAMPLE_FILES files and MR_SAMPLE_CALLABLES callables with seed 1 +# (internal/metamorphic/realapp_test.go). # # Environment: -# EVAL_SRC clones (default /tmp/eval) -# EVAL_GO Go toolchain for the test run (default go1.27.3; eval_l3.sh's default) -# MR_RELATIONS relations to run, space-separated (default: all of MR1-MR6, MR8-MR15) -# MR_TIMEOUT go test -timeout (default 120m) -# MR_LOG log file (default $EVAL_OUT/metamorphic-.log) +# EVAL_SRC clones (default /tmp/eval) +# EVAL_GO Go toolchain for the test run (default go1.27.3; eval_l3.sh's default) +# MR_RELATIONS relations to run, space-separated (default: all of MR1-MR6, MR8-MR15) +# MR_SAMPLE_FILES files transformed per relation (default 20) +# MR_SAMPLE_CALLABLES callables sampled by the reachability oracle (default 50; 20 gives a +# quicker nats-server run) +# MR_TIMEOUT go test -timeout (default 120m) +# MR_LOG log file (default $EVAL_OUT/metamorphic-.log) # # Prints one MRSTAT line per relation and app (sites, reachability pairs, violations, time, # skips) and exits non-zero on any violation. @@ -27,7 +33,13 @@ MR_TIMEOUT=${MR_TIMEOUT:-120m} MR_LOG=${MR_LOG:-$EVAL_OUT/metamorphic-$(date +%Y-%m-%d).log} REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd) -[ $# -gt 0 ] || { sed -n '2,9p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 2; } +DEFAULT_APPS="memos" +ALL_APPS="memos nats-server" +case "${1:-}" in + -h|--help) sed -n '2,12p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 0 ;; + --all) set -- $ALL_APPS ;; + "") set -- $DEFAULT_APPS ;; +esac for a in "$@"; do [ -f "$EVAL_SRC/$a/go.mod" ] || echo "warning: $a is not prepared at $EVAL_SRC/$a (skipped)" >&2 done @@ -35,6 +47,7 @@ mkdir -p "$(dirname "$MR_LOG")" start=$(date +%s) (cd "$REPO_ROOT" && EVAL_SRC=$EVAL_SRC MR_APPS="$*" MR_RELATIONS="${MR_RELATIONS:-}" \ + MR_SAMPLE_FILES="${MR_SAMPLE_FILES:-20}" MR_SAMPLE_CALLABLES="${MR_SAMPLE_CALLABLES:-50}" \ GOTOOLCHAIN=${GOTOOLCHAIN:-$EVAL_GO} \ go test -tags metamorphic -count=1 -timeout "$MR_TIMEOUT" -v -run 'TestMR_RealApps' ./internal/metamorphic/) \ >"$MR_LOG" 2>&1 diff --git a/testdata/metamorphic/conventions.go b/testdata/metamorphic/conventions.go new file mode 100644 index 0000000..376df60 --- /dev/null +++ b/testdata/metamorphic/conventions.go @@ -0,0 +1,51 @@ +package examples + +// The oracle conventions internal/metamorphic encodes; each is pinned by a +// test there (TestConvention_*), which names the rule when it breaks. + +// ConvCaptureSite (i): the literal's capture site actual_in for p feeds only +// the closure's formal_in (param_in), never a summary or ddg edge: it is not +// a reachability sink. +func ConvCaptureSite(p *pt, v int) { + f := func() { p.n = v } + f() +} + +// ConvGuardInLoop (ii): MR8 wrapping `s += i` in a loop body gives the guard +// the self-loop B→B on c. +func ConvGuardInLoop(n int) int { + s := 0 + for i := 0; i < n; i++ { + s += i + s++ + } + return s +} + +// ConvGuardStraight (ii): MR8 wrapping `x += a` outside a loop gives no +// guard edge. +func ConvGuardStraight(a int) int { + x := a + x += a + return x +} + +// ConvDiscardedResult (iii): p.sum()'s discarded result reaches the return +// only through actual_out → call node (coarse wiring) and the call node's +// control dependence (it may panic): MR14(b) cuts that actual_out. +func ConvDiscardedResult(p pt2) int { + p.sum() + return p.a +} + +// pt2 has a value method. +type pt2 struct{ a, b int } + +func (p pt2) sum() int { return p.a + p.b } + +// ConvNamedKill (iv): `n = a + 1` reaches @formal_out through the @exit +// port edge; MR9's kill before the return takes it over. +func ConvNamedKill(a int) (n int) { + n = a + 1 + return +} diff --git a/testdata/metamorphic/doc.go b/testdata/metamorphic/doc.go new file mode 100644 index 0000000..afc2ca1 --- /dev/null +++ b/testdata/metamorphic/doc.go @@ -0,0 +1,12 @@ +// Package examples holds small functions written to be eligible sites of the +// metamorphic relations (internal/metamorphic): each one names the relation +// it targets and the effect the relation expects. conventions.go pins the +// oracle conventions those relations rely on. +package examples + +// sinkInt is the sink the call sites pass values to: it has no by-ref or +// global output, so a flow into its argument ends at the call. +func sinkInt(v int) int { return v } + +// sink2 is a two-argument sink. +func sink2(v, w int) int { return v + w } diff --git a/testdata/metamorphic/go.mod b/testdata/metamorphic/go.mod new file mode 100644 index 0000000..ec07201 --- /dev/null +++ b/testdata/metamorphic/go.mod @@ -0,0 +1,3 @@ +module example.com/metamorphic + +go 1.21 diff --git a/testdata/metamorphic/mr01_04_layout.go b/testdata/metamorphic/mr01_04_layout.go new file mode 100644 index 0000000..3183f53 --- /dev/null +++ b/testdata/metamorphic/mr01_04_layout.go @@ -0,0 +1,20 @@ +package examples + +// MR1-MR4, MR7, MR15 (layout, rename, target-files): a type with a method +// declared above it, a function, and locals to rename. Exact oracle: the +// analysis is unchanged up to positions (and spelling for MR4). + +// Scale is declared above its receiver type (MR2 reverses it back below). +func (c *counter) Scale(k int) int { + m := c.n * k + return m +} + +// counter is a struct with one field. +type counter struct{ n int } + +// Sum adds two values through locals (MR3 may move it; MR4 renames a, b, s). +func Sum(a, b int) int { + s := a + b + return s +} diff --git a/testdata/metamorphic/mr05_alias.go b/testdata/metamorphic/mr05_alias.go new file mode 100644 index 0000000..4870b2d --- /dev/null +++ b/testdata/metamorphic/mr05_alias.go @@ -0,0 +1,16 @@ +package examples + +// pt is the struct MR5's alias writes go through. +type pt struct{ n int } + +// SetPtr: MR5 rewrites p.n = v as `q := p; q.n = v`; v still reaches p's formal_out. +func SetPtr(p *pt, v int) { + p.n = v +} + +// SetValue: MR5 rewrites s.n = v as `q := &s; q.n = v`; v still reaches the return. +func SetValue(v int) int { + var s pt + s.n = v + return s.n +} diff --git a/testdata/metamorphic/mr06_extract.go b/testdata/metamorphic/mr06_extract.go new file mode 100644 index 0000000..0ce3078 --- /dev/null +++ b/testdata/metamorphic/mr06_extract.go @@ -0,0 +1,9 @@ +package examples + +// Extract: MR6 moves `x := a + 1; y := b * 2` into a helper taking a, b and +// returning x, y; a and b still reach the return through param_in/param_out. +func Extract(a, b int) int { + x := a + 1 + y := b * 2 + return x + y +} diff --git a/testdata/metamorphic/mr08_wrap.go b/testdata/metamorphic/mr08_wrap.go new file mode 100644 index 0000000..8aa4a70 --- /dev/null +++ b/testdata/metamorphic/mr08_wrap.go @@ -0,0 +1,27 @@ +package examples + +// WrapStraight: MR8 wraps `x += a` in `if c { … }`; the def before it now also +// reaches the return (x += a is conditional). +func WrapStraight(a int) int { + x := a + x += a + return x +} + +// WrapInLoop: MR8 wraps `s += i` (a loop-body statement); the guard's def +// reaches its own condition around the loop (B→B on c, see conventions.go). +func WrapInLoop(n int) int { + s := 0 + for i := 0; i < n; i++ { + s += i + s++ + } + return s +} + +// WrapBeforeReturn: MR8 wraps `y++`, the statement right before the return. +func WrapBeforeReturn(a int) int { + y := a * 2 + y++ + return y +} diff --git a/testdata/metamorphic/mr09_kill.go b/testdata/metamorphic/mr09_kill.go new file mode 100644 index 0000000..eb34712 --- /dev/null +++ b/testdata/metamorphic/mr09_kill.go @@ -0,0 +1,22 @@ +package examples + +// KillOneUse: MR9 inserts `x = 0` before sinkInt(x); x := a + 1 no longer +// reaches the call or the return, the kill does. +func KillOneUse(a int) int { + x := a + 1 + sinkInt(x) + return x +} + +// KillTwoUses: MR9 inserts `x = 0` before the return, which reads x twice. +func KillTwoUses(a int) int { + x := a + 1 + return x * x +} + +// KillNamed: MR9 inserts `n = 0` before the bare return; the kill takes over +// n's port edge into @formal_out (see conventions.go). +func KillNamed(a int) (n int) { + n = a + 1 + return +} diff --git a/testdata/metamorphic/mr10_copy.go b/testdata/metamorphic/mr10_copy.go new file mode 100644 index 0000000..b9f63ce --- /dev/null +++ b/testdata/metamorphic/mr10_copy.go @@ -0,0 +1,14 @@ +package examples + +// CopyInReturn: MR10 inserts `y := x` before the return and returns y*2; +// x's def now reaches the copy, the copy the return. +func CopyInReturn(a int) int { + x := a + 1 + return x * 2 +} + +// CopyInAssign: MR10 copies one of the reads of the assignment's right side. +func CopyInAssign(a, b int) int { + z := a - b + return z +} diff --git a/testdata/metamorphic/mr11_swap.go b/testdata/metamorphic/mr11_swap.go new file mode 100644 index 0000000..ac0d9ef --- /dev/null +++ b/testdata/metamorphic/mr11_swap.go @@ -0,0 +1,8 @@ +package examples + +// Swap: MR11 swaps the two independent definitions; ddg and cdg are unchanged. +func Swap(a, b int) int { + x := a + 1 + y := b + 2 + return x + y +} diff --git a/testdata/metamorphic/mr12_early_return.go b/testdata/metamorphic/mr12_early_return.go new file mode 100644 index 0000000..e3110c1 --- /dev/null +++ b/testdata/metamorphic/mr12_early_return.go @@ -0,0 +1,14 @@ +package examples + +// EarlyReturn: MR12 inserts `if c { return 0 }` before a statement; the +// statements after it become control dependent on the new branch. +func EarlyReturn(a int) int { + x := a + 1 + return x +} + +// EarlyReturnTwo: MR12 on a two-result function returns `0, ""` early. +func EarlyReturnTwo(a int) (int, string) { + x := a * 3 + return x, "ok" +} diff --git a/testdata/metamorphic/mr13_const_sink.go b/testdata/metamorphic/mr13_const_sink.go new file mode 100644 index 0000000..b734c78 --- /dev/null +++ b/testdata/metamorphic/mr13_const_sink.go @@ -0,0 +1,25 @@ +package examples + +// ConstScalar: MR13 replaces x in sinkInt(x) with 0; no formal_in reaches +// the call's actual_in any more. +func ConstScalar(a int) int { + x := a + 1 + sinkInt(x) + return x +} + +// ConstTwoArgs: MR13 replaces x in sink2(x, 3) with 0. +func ConstTwoArgs(a int) int { + x := a - 1 + sink2(x, 3) + return x +} + +// ConstInLoop: MR13 replaces x in the loop's sinkInt(x) with 0. +func ConstInLoop(a, n int) int { + x := a * 2 + for i := 0; i < n; i++ { + sinkInt(x) + } + return x +} diff --git a/testdata/metamorphic/mr14_route.go b/testdata/metamorphic/mr14_route.go new file mode 100644 index 0000000..8f8baea --- /dev/null +++ b/testdata/metamorphic/mr14_route.go @@ -0,0 +1,28 @@ +package examples + +import "strconv" + +// RouteIface: MR14(a) routes sinkInt(x) through a one-implementation +// interface (the literal rules out the closure form, and the call is not +// last, so not defer); the call's summary edge has its counterpart. +func RouteIface(a int) int { + x := a + 1 + _ = func() {} + sinkInt(x) + return x +} + +// RouteClosure: MR14(b) routes strconv.Itoa(x) through func(){ … }() (a +// library call: no interface form; not last: no defer). +func RouteClosure(a int) int { + x := a + 1 + strconv.Itoa(x) + return x +} + +// RouteDefer: MR14(c) defers strconv.Itoa(x), the last statement. +func RouteDefer(a int) { + x := a + 1 + _ = func() {} + strconv.Itoa(x) +} From c227081baadf447ada745e57c89ee2507d2ff6fd Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 14:08:32 -0400 Subject: [PATCH 31/65] test(l4): metamorphic script rejects unknown flags; unique statement lookup; stronger example checks --- internal/metamorphic/mr_test.go | 205 +++++++++++++++++++++++++++---- scripts/metamorphic.sh | 8 +- scripts/metamorphic_args_test.sh | 23 ++++ 3 files changed, 211 insertions(+), 25 deletions(-) create mode 100755 scripts/metamorphic_args_test.sh diff --git a/internal/metamorphic/mr_test.go b/internal/metamorphic/mr_test.go index 0a56f64..386f0dd 100644 --- a/internal/metamorphic/mr_test.go +++ b/internal/metamorphic/mr_test.go @@ -728,33 +728,54 @@ func copyOf(t *testing.T, dir string) string { const exPkg = "example.com/metamorphic." -// stmtKey is the body key of the statement of examples file rel that starts -// with text, in function fn (the first such statement). +// stmtKey is the body key of the one statement of function fn in examples +// file rel whose text is text, or starts with text followed by a token +// boundary (so "x := a + 1" never matches "x := a + 10"). Zero or several +// matches fail the test. func stmtKey(t *testing.T, rel, fn, text string) string { t.Helper() f := examples.ws.Files[rel] if f == nil { t.Fatalf("no file %s", rel) } + var keys []string for _, fd := range funcDecls(f) { if fd.Name.Name != fn { continue } - var key string ast.Inspect(fd.Body, func(n ast.Node) bool { - if st, ok := n.(ast.Stmt); ok && key == "" { - if _, block := st.(*ast.BlockStmt); !block && strings.HasPrefix(string(f.Src[f.off(st.Pos()):f.off(st.End())]), text) { - key = f.key(st.Pos()) - } + st, ok := n.(ast.Stmt) + if !ok { + return true + } + if _, block := st.(*ast.BlockStmt); block { + return true + } + if src := string(f.Src[f.off(st.Pos()):f.off(st.End())]); textMatch(src, text) { + keys = append(keys, f.key(st.Pos())) } - return key == "" + return true }) - if key != "" { - return key - } } - t.Fatalf("%s: no statement %q in %s", rel, text, fn) - return "" + if len(keys) != 1 { + t.Fatalf("%s: %d statements %q in %s (%v), want exactly one", rel, len(keys), text, fn, keys) + } + return keys[0] +} + +// textMatch reports whether src is text, or text then a token boundary: a +// character that cannot continue text's last token. +func textMatch(src, text string) bool { + if !strings.HasPrefix(src, text) { + return false + } + if len(src) == len(text) { + return true + } + word := func(c byte) bool { + return c == '_' || c >= '0' && c <= '9' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' + } + return !(word(text[len(text)-1]) && word(src[len(text)])) } // siteIn is the site of res in callable id. @@ -863,11 +884,22 @@ func TestMR5_IntroduceAlias_Reach(t *testing.T) { // ─── the examples module ────────────────────────────────────────────────────── // TestExamples_EveryRelationHasASite: every relation MR1-MR15 transforms at -// least one site of testdata/metamorphic (a relation whose sites all -// vanished would pass vacuously). +// least one site of testdata/metamorphic, and the relations with a file of +// their own land on the functions written for them: MR13 on mr13_*, each +// MR14 form (a interface, b closure, c defer) on mr14_*, MR7 exact on a +// named file, MR15 byte-identical on the MR6 example. func TestExamples_EveryRelationHasASite(t *testing.T) { tg := examples tg.load(t) + sitesIn := func(res *Result, file string) []Site { + var out []Site + for _, s := range res.Sites { + if strings.HasPrefix(s.Desc, file+":") { + out = append(out, s) + } + } + return out + } for _, r := range relations { res, err := tg.ws.Apply(r.t, tg.files(), filepath.Join(t.TempDir(), r.name), seed, tg.pre.view) if err != nil { @@ -877,16 +909,76 @@ func TestExamples_EveryRelationHasASite(t *testing.T) { if len(res.Sites) == 0 { t.Errorf("%s has no site on the examples module (skipped: %v)", r.name, res.Skipped) } + switch r.name { + case "MR13": + want := []string{"ConstScalar", "ConstTwoArgs", "ConstInLoop"} + for _, fn := range want { + if !slices.ContainsFunc(sitesIn(res, "mr13_const_sink.go"), func(s Site) bool { return s.Callable == exPkg+fn }) { + t.Errorf("MR13 has no site in %s (mr13_const_sink.go)", fn) + } + } + case "MR14": + forms := map[string]string{ // the function written for each form → its Desc + "RouteIface": "call sinkInt through mr14i", + "RouteClosure": "call through a closure", + "RouteDefer": "defer the call", + } + for fn, form := range forms { + if !slices.ContainsFunc(sitesIn(res, "mr14_route.go"), func(s Site) bool { + return s.Callable == exPkg+fn && strings.Contains(s.Desc, form) + }) { + t.Errorf("MR14 form %q has no site in %s (mr14_route.go)", form, fn) + } + } + } + } + + // MR7: --target-files on mr09_kill.go at -a 3 is exact on its callables. + full := tg.level3(t) + const rel7 = "mr09_kill.go" + part, err := analyze(tg.ws.Dir, 3, []string{filepath.Join(tg.ws.Dir, rel7)}) + if err != nil { + t.Fatal(err) + } + only := map[string]bool{} + for id, c := range full.view.Calls { + if c.File == rel7 { + only[id] = true + } + } + if len(only) < 3 { + t.Fatalf("MR7: only %d callables in %s", len(only), rel7) + } + if d := Diff(full.view, part.view, Scope{Only: only}, 20); len(d) > 0 { + t.Errorf("MR7 on %s:\n%s", rel7, strings.Join(d, "\n")) } - if fs := Sample(tg.files(), 2, seed); len(fs) == 0 { - t.Error("MR7 has no file on the examples module") + + // MR15: MR6 extracts a helper from Extract, and Inline6 gives + // mr06_extract.go back byte for byte. + const rel15 = "mr06_extract.go" + ext, err := tg.ws.Apply(MR6, []string{rel15}, filepath.Join(t.TempDir(), "mr15x"), seed, nil) + if err != nil { + t.Fatal(err) + } + if !slices.ContainsFunc(ext.Sites, func(s Site) bool { return s.Callable == exPkg+"Extract" }) { + t.Fatalf("MR15: MR6 extracted nothing from Extract: %v", ext.Skipped) } - ext, err := tg.ws.Apply(MR6, tg.files(), filepath.Join(t.TempDir(), "mr15"), seed, nil) + mid, _ := os.ReadFile(filepath.Join(ext.Dir, rel15)) + if !strings.Contains(string(mid), "func "+Prefix6) { + t.Fatalf("MR15: %s has no MR6 helper after the extraction", rel15) + } + ws2, err := LoadWorkspace(ext.Dir) if err != nil { t.Fatal(err) } - if len(ext.Files) == 0 { - t.Error("MR15 has no site on the examples module") + inl, err := ws2.Apply(Inline6, []string{rel15}, filepath.Join(t.TempDir(), "mr15i"), seed, nil) + if err != nil { + t.Fatal(err) + } + orig, _ := os.ReadFile(filepath.Join(tg.ws.Dir, rel15)) + back, _ := os.ReadFile(filepath.Join(inl.Dir, rel15)) + if string(orig) != string(back) { + t.Errorf("MR15: inlining does not restore %s byte for byte:\n--- original\n%s\n--- inlined\n%s", rel15, orig, back) } } @@ -954,15 +1046,58 @@ func TestConvention_GuardSelfLoopOnlyInLoops(t *testing.T) { guard = append(guard, e) } } + // The guard exists: B is a branch node, its source text is the + // `if := true; {` header MR8 wrote, and B → the wrapped + // statement is its true edge. Its variable is read from that text, + // so the no-edge check below cannot pass for a renamed guard. + if a := post.view.Calls[id].Body[b]; !strings.HasPrefix(a, "branch|") { + t.Fatalf("%s: %s: node %s is %q, want the guard branch", rule, fn, b, a) + } + g := guardVar(t, res, "conventions.go", b) + var on []string + for e := range post.view.Calls[id].Edges { + if f := strings.Split(e, "|"); f[0] == "ddg" && f[3] == g { + on = append(on, e) + } + } + if len(guard) != len(on) { + t.Errorf("%s: %s: guard edges %v are not the edges on %s, %v", rule, fn, guard, g, on) + } switch { - case wantLoop && (len(guard) != 1 || !strings.HasPrefix(guard[0], "ddg|"+b+"|"+b+"|")): - t.Errorf("%s: %s: guard edges %v, want the one self-loop on %s", rule, fn, guard, b) - case !wantLoop && len(guard) != 0: - t.Errorf("%s: %s: guard edges %v, want none", rule, fn, guard) + case wantLoop && (len(on) != 1 || on[0] != "ddg|"+b+"|"+b+"|"+g+"|ssa"): + t.Errorf("%s: %s: edges on %s are %v, want the one self-loop on %s", rule, fn, g, on, b) + case !wantLoop && len(on) != 0: + t.Errorf("%s: %s: edges on %s are %v, want none", rule, fn, g, on) } } } +// guardVar is the variable of the `if := true; {` header MR8 wrote +// at body key b of file rel in the transformed module res. +func guardVar(t *testing.T, res *Result, rel, b string) string { + t.Helper() + var p Pos + for _, m := range res.Marks { + if m.File == rel && m.Key() == b { + p = m + } + } + src, err := os.ReadFile(filepath.Join(res.Dir, p.File)) + if err != nil { + t.Fatal(err) + } + off, ok := newLineTable(src).offset(p.Line, p.Col) + if !ok { + t.Fatalf("no position %v", p) + } + head := string(src[off:]) + var c string + if _, err := fmt.Sscanf(head, "if %s := true;", &c); err != nil || !strings.HasPrefix(head, "if "+c+" := true; "+c+" {") { + t.Fatalf("no guard header at %v: %.40q", p, head) + } + return c +} + // TestConvention_DiscardedResultReachesOnlyViaHost pins rule (iii), MR14(b)'s // cut: a discarded result's actual_out has the one out-edge actual_out → // call node (the coarse host wiring, spec §4.7), and the call node reaches @@ -1020,3 +1155,25 @@ func TestConvention_NamedResultPortMovesToKill(t *testing.T) { t.Errorf("%s: the kill %s has no port edge to @formal_out", rule, k) } } + +// TestStmtKey_TokenBoundary: stmtKey's text match needs the whole text or a +// token boundary after it, so one statement prefix never matches a longer +// literal or identifier. +func TestStmtKey_TokenBoundary(t *testing.T) { + for _, c := range []struct { + src, text string + want bool + }{ + {"x := a + 1", "x := a + 1", true}, + {"x := a + 10", "x := a + 1", false}, + {"x := a + 1 // c", "x := a + 1", true}, + {"sinkInt(x)", "sinkInt(x)", true}, + {"sinkInt(x)", "sinkInt(", true}, + {"return xy", "return x", false}, + {"return x * x", "return x", true}, + } { + if got := textMatch(c.src, c.text); got != c.want { + t.Errorf("textMatch(%q, %q) = %v, want %v", c.src, c.text, got, c.want) + } + } +} diff --git a/scripts/metamorphic.sh b/scripts/metamorphic.sh index 06a9cce..ff4962d 100755 --- a/scripts/metamorphic.sh +++ b/scripts/metamorphic.sh @@ -35,8 +35,14 @@ REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd) DEFAULT_APPS="memos" ALL_APPS="memos nats-server" +for a in "$@"; do + case $a in + -h|--help) sed -n '2,12p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 0 ;; + --all) [ $# -eq 1 ] || { echo "--all takes no other arguments" >&2; exit 2; } ;; + -*) echo "unknown flag: $a" >&2; exit 2 ;; + esac +done case "${1:-}" in - -h|--help) sed -n '2,12p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 0 ;; --all) set -- $ALL_APPS ;; "") set -- $DEFAULT_APPS ;; esac diff --git a/scripts/metamorphic_args_test.sh b/scripts/metamorphic_args_test.sh new file mode 100755 index 0000000..f25885e --- /dev/null +++ b/scripts/metamorphic_args_test.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# metamorphic_args_test.sh - check scripts/metamorphic.sh's argument handling (no analysis runs). +# Usage: scripts/metamorphic_args_test.sh (exits non-zero on a failed check) +set -u +S=$(cd "$(dirname "$0")" && pwd)/metamorphic.sh +fail=0 +check() { # want-rc want-stderr-substring args... + local want=$1 msg=$2; shift 2 + local err rc + err=$("$S" "$@" 2>&1 >/dev/null); rc=$? + if [ "$rc" != "$want" ] || [[ "$err" != *"$msg"* ]]; then + echo "FAIL: metamorphic.sh $* -> rc $rc, stderr: $err (want rc $want, '$msg')"; fail=1 + else + echo "ok: metamorphic.sh $* -> rc $rc" + fi +} +check 2 "unknown flag: --al" --al +check 2 "unknown flag: -x" -x memos +check 2 "unknown flag: --memos" memos --memos +check 2 "--all takes no other arguments" --all memos +check 2 "--all takes no other arguments" memos --all +check 0 "Usage:" --help +exit $fail From 104d7e515c21df4ae85780b343911f1eb8577db0 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 14:26:37 -0400 Subject: [PATCH 32/65] fix(neo4j): force a full push when the stored level is lower A lazy push diffed content_hash only, so an L4 push onto a graph first pushed at L3 skipped unchanged modules and their edges' MATCH dropped rows silently. A stored max_level below the push's now forces a full upsert, like a schema_version mismatch. --- internal/neo4j/bolt.go | 61 ++++++++++++++++++------- internal/neo4j/bolt_session_test.go | 69 +++++++++++++++++++++++++++-- 2 files changed, 111 insertions(+), 19 deletions(-) diff --git a/internal/neo4j/bolt.go b/internal/neo4j/bolt.go index 4e48123..fbf080e 100644 --- a/internal/neo4j/bolt.go +++ b/internal/neo4j/bolt.go @@ -16,8 +16,9 @@ import ( // // Algorithm: // 1. ensure constraints + indexes; -// 2. schema-version gate: read :GoApplication.schema_version scoped to this -// app's id; a mismatch forces a full re-UPSERT (never a delete — that is +// 2. stored-application gate: read :GoApplication.schema_version and +// max_level scoped to this app's id; a version mismatch, or a stored level +// below this push's, forces a full re-UPSERT (never a delete — that is // --eager's job); // 3. partition nodes by owning module (NodeRow.module); shared nodes (externals, // neutral repo nodes) have none and are MERGE-only, always upserted; L4 @@ -90,11 +91,12 @@ func (w *boltWriter) run() error { return err } - // Schema-version gate: a producer/consumer mismatch forces a full re-upsert - // (it never deletes). We detect it but the upsert path below is already a - // full MERGE of every current node, so a mismatch simply means we don't - // trust the per-module content_hash diff and upsert everything. - forceFull, err := w.schemaVersionMismatch() + // Stored-application gate: a producer/consumer version mismatch, or a + // graph pushed at a lower level, forces a full re-upsert (it never + // deletes). The upsert path below is already a full MERGE of every + // current node, so it simply means we don't trust the per-module + // content_hash diff and upsert everything. + forceFull, err := w.storedAppStale() if err != nil { return err } @@ -183,14 +185,26 @@ func (w *boltWriter) ensureDDL() error { }) } -// schemaVersionMismatch reads the stored :GoApplication.schema_version for THIS -// app and compares to the producer's SchemaVersion. A mismatch (or absence) -// returns true → the caller upserts everything rather than trusting the diff. -func (w *boltWriter) schemaVersionMismatch() (bool, error) { +// storedAppStale reads the stored :GoApplication.schema_version and +// max_level for THIS app and reports whether the content_hash diff cannot be +// trusted, so the caller upserts everything: +// +// - a stored schema_version different from the producer's SchemaVersion; +// - a stored max_level below this push's. A module's content_hash covers +// its source, not the level it was analysed at, so an unchanged module +// of a graph pushed at L3 still lacks what an L4 push adds to it — its L4 +// vertices are upserted (D5), but rows the diff skips stay at L3, and an +// edge row whose endpoint is missing is dropped by its MATCH silently. +// +// No stored application (first push) forces nothing beyond the normal +// upsert, nor does a stored level at or above this push's. +func (w *boltWriter) storedAppStale() (bool, error) { var stored string + var storedLevel int64 + haveLevel := false err := w.read(func(tx neo4j.ManagedTransaction) error { res, err := tx.Run(w.ctx, - "MATCH (a:GoApplication {id: $id}) RETURN a.schema_version AS v", + "MATCH (a:GoApplication {id: $id}) RETURN a.schema_version AS v, a.max_level AS l", map[string]any{"id": w.appID}) if err != nil { return err @@ -199,15 +213,32 @@ func (w *boltWriter) schemaVersionMismatch() (bool, error) { if v, ok := res.Record().Get("v"); ok && v != nil { stored, _ = v.(string) } + if l, ok := res.Record().Get("l"); ok && l != nil { + storedLevel, haveLevel = l.(int64) + } } return res.Err() }) if err != nil { return false, err } - // No stored version (first push) is not a "mismatch" that forces anything - // beyond the normal upsert; a DIFFERENT stored version does. - return stored != "" && stored != SchemaVersion, nil + if stored != "" && stored != SchemaVersion { + return true, nil + } + return haveLevel && storedLevel < int64(w.pushLevel()), nil +} + +// pushLevel is the max_level of the application row this push carries (0 +// when it carries none). +func (w *boltWriter) pushLevel() int { + for _, n := range w.rows.Nodes { + if n.Labels[0] == LabelApplication && n.Value == w.appID { + if l, ok := n.Props["max_level"].(int); ok { + return l + } + } + } + return 0 } // ─── destructive steps (--eager only) ───────────────────────────────────────── diff --git a/internal/neo4j/bolt_session_test.go b/internal/neo4j/bolt_session_test.go index e5d0137..a367ffa 100644 --- a/internal/neo4j/bolt_session_test.go +++ b/internal/neo4j/bolt_session_test.go @@ -74,12 +74,14 @@ func TestEagerPurge_OnlyOnFullRun(t *testing.T) { } // fakeDB is a DB-free SessionWithContext for the push algorithm: reads -// answer the schema-version gate with SchemaVersion and the content_hash -// diff with hashes[module id] (an unchanged module when it equals the local -// hash); every write statement is recorded with its parameters. +// answer the stored-application gate with SchemaVersion and level (when +// non-zero; else no stored level) and the content_hash diff with +// hashes[module id] (an unchanged module when it equals the local hash); +// every write statement is recorded with its parameters. type fakeDB struct { neo4j.SessionWithContext hashes map[string]string + level int64 writes []fakeStmt } @@ -109,7 +111,11 @@ func (s *fakeDB) ExecuteRead(ctx context.Context, work neo4j.ManagedTransactionW func (tx fakeTx) Run(_ context.Context, q string, params map[string]any) (neo4j.ResultWithContext, error) { switch { case strings.Contains(q, "a.schema_version"): - return &fakeResult{rec: &neo4j.Record{Keys: []string{"v"}, Values: []any{SchemaVersion}}}, nil + var lvl any + if tx.db.level != 0 { + lvl = tx.db.level + } + return &fakeResult{rec: &neo4j.Record{Keys: []string{"v", "l"}, Values: []any{SchemaVersion, lvl}}}, nil case strings.Contains(q, "m.content_hash"): h, ok := tx.db.hashes[params["id"].(string)] if !ok { @@ -243,3 +249,58 @@ func TestBolt_LazyPushUpsertsL4RowsOfUnchangedModules(t *testing.T) { } } } + +// TestBolt_LazyPushForcesFullOnLowerStoredLevel: a graph first pushed at a +// lower level (an L3 push) has no L4 rows, and a lazy L4 push must not skip +// an unchanged module's L3 nodes: they may be missing L4 vertices' parents, +// and an edge MATCH on a missing endpoint drops the row silently. A stored +// max_level below the current one forces every module's upsert; an equal +// (or higher) stored level keeps the content_hash diff. +func TestBolt_LazyPushForcesFullOnLowerStoredLevel(t *testing.T) { + a := buildDataflowL4(t) + if a.MaxLevel != 4 { + t.Fatalf("fixture max_level = %d, want 4", a.MaxLevel) + } + rows := Project(a) + byModule, _, _ := partitionNodes(rows.Nodes) + hashes := map[string]string{} + for m, ns := range byModule { + hashes[m] = moduleHash(ns) + } + push := func(stored int64) map[string]bool { + db := &fakeDB{hashes: hashes, level: stored} + w := &boltWriter{ctx: context.Background(), session: db, rows: rows, appID: a.Application.ID} + if err := w.run(); err != nil { + t.Fatal(err) + } + nodes, _ := db.pushed() + return nodes + } + moduleNodes := 0 + for _, ns := range byModule { + moduleNodes += len(ns) + } + if moduleNodes == 0 { + t.Fatal("fixture has no module-owned nodes") + } + for _, stored := range []int64{1, 3} { + nodes := push(stored) + for m, ns := range byModule { + for _, n := range ns { + if !nodes[n.Value] { + t.Fatalf("stored level %d: module %s node %s not upserted", stored, m, n.Value) + } + } + } + } + for _, stored := range []int64{4, 5} { + nodes := push(stored) + for _, ns := range byModule { + for _, n := range ns { + if nodes[n.Value] { + t.Fatalf("stored level %d: unchanged module node %s upserted", stored, n.Value) + } + } + } + } +} From b0092f5f3c2fab55fcff13cab7bd61c53445d6c8 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 14:27:51 -0400 Subject: [PATCH 33/65] fix(l4): deterministic id for duplicate callable keys Two callables sharing an L3Key routed their L4 edges to whichever was emitted last, over map order. Keep the lexically smaller id. --- internal/syntactic_analysis/v2emit/emit_l4.go | 10 ++++++-- .../syntactic_analysis/v2emit/emit_l4_test.go | 23 +++++++++++++++++++ 2 files changed, 31 insertions(+), 2 deletions(-) diff --git a/internal/syntactic_analysis/v2emit/emit_l4.go b/internal/syntactic_analysis/v2emit/emit_l4.go index 9908d57..feb3df3 100644 --- a/internal/syntactic_analysis/v2emit/emit_l4.go +++ b/internal/syntactic_analysis/v2emit/emit_l4.go @@ -66,11 +66,17 @@ func (o *l4Overlay) apply(c *v2.Callable, key dataflow.L3Key) { if o == nil { return } + // Two callables sharing a target key route its param edges and degraded + // entries to the lexically smaller id: emission order walks maps, so + // last-write-wins would not be deterministic. t := sdg.Target{Callable: key} - if prev, dup := o.ids[t]; dup && prev != c.ID { + prev, dup := o.ids[t] + if dup && prev != c.ID { warnf("L4 emit: %s and %s share the target key %s %s", prev, c.ID, key.Path, key.Signature) } - o.ids[t] = c.ID + if !dup || c.ID < prev { + o.ids[t] = c.ID + } r, ok := o.res.Callables[key] if !ok { return diff --git a/internal/syntactic_analysis/v2emit/emit_l4_test.go b/internal/syntactic_analysis/v2emit/emit_l4_test.go index d166479..4cbd0c5 100644 --- a/internal/syntactic_analysis/v2emit/emit_l4_test.go +++ b/internal/syntactic_analysis/v2emit/emit_l4_test.go @@ -5,7 +5,9 @@ import ( "strings" "testing" + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" ) // TestCompareOrdinal_MatchesMintedIds: compareOrdinal orders (owner, key) @@ -29,3 +31,24 @@ func TestCompareOrdinal_MatchesMintedIds(t *testing.T) { } } } + +// TestL4Overlay_DuplicateKeyKeepsSmallerID: two callables sharing a target +// key (declaring file and signature) route its param edges and degraded +// entries to the lexically smaller id, whichever is applied first — +// emission walks maps, so last-write-wins would be nondeterministic. +func TestL4Overlay_DuplicateKeyKeepsSmallerID(t *testing.T) { + key := dataflow.L3Key{Path: "a.go", Signature: "example.com/a.F"} + small, big := "can://app/go/a.go/example.com/a.F", "can://app/go/b.go/example.com/a.F" + prev := warnf + warnf = func(string, ...any) {} + defer func() { warnf = prev }() + for _, order := range [][]string{{small, big}, {big, small}} { + o := newL4Overlay(4, &sdg.Result{}, sigIndex{}) + for _, id := range order { + o.apply(&v2.Callable{ID: id, Body: map[string]v2.BodyNode{}}, key) + } + if got := o.ids[sdg.Target{Callable: key}]; got != small { + t.Errorf("apply order %v: target routes to %s, want %s", order, got, small) + } + } +} From 7b5b35d0158e976ee020e785120d8bcf8e27df56 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 14:29:50 -0400 Subject: [PATCH 34/65] test(l4): no-dangling gates cover the L4 projection and degraded callables --- internal/neo4j/project_test.go | 39 +++++++++++++------ .../syntactic_analysis/v2emit/gate_l4_test.go | 25 +++++++++--- 2 files changed, 47 insertions(+), 17 deletions(-) diff --git a/internal/neo4j/project_test.go b/internal/neo4j/project_test.go index d9d16a6..e0bb16e 100644 --- a/internal/neo4j/project_test.go +++ b/internal/neo4j/project_test.go @@ -43,21 +43,36 @@ func buildMultipackageL2(t *testing.T) *v2.Analysis { } func TestProject_NoDanglingEndpoints(t *testing.T) { - rows := Project(buildMultipackageL2(t)) + for _, c := range []struct { + name string + a func(*testing.T) *v2.Analysis + }{ + {"multipackage L2", buildMultipackageL2}, + {"dataflow L4", buildDataflowL4}, + } { + rows := Project(c.a(t)) - // Every (label, value) that was emitted as a node. - emitted := make(map[string]bool, len(rows.Nodes)) - for _, n := range rows.Nodes { - for _, l := range n.Labels { - emitted[l+"\x00"+n.Value] = true + // Every (label, value) that was emitted as a node. + emitted := make(map[string]bool, len(rows.Nodes)) + for _, n := range rows.Nodes { + for _, l := range n.Labels { + emitted[l+"\x00"+n.Value] = true + } } - } - for _, e := range rows.Edges { - if !emitted[e.From.Label+"\x00"+e.From.Value] { - t.Errorf("%s edge FROM a non-emitted node: %s %s", e.Type, e.From.Label, e.From.Value) + for _, e := range rows.Edges { + if !emitted[e.From.Label+"\x00"+e.From.Value] { + t.Errorf("%s: %s edge FROM a non-emitted node: %s %s", c.name, e.Type, e.From.Label, e.From.Value) + } + if !emitted[e.To.Label+"\x00"+e.To.Value] { + t.Errorf("%s: %s edge TO a non-emitted node: %s %s", c.name, e.Type, e.To.Label, e.To.Value) + } } - if !emitted[e.To.Label+"\x00"+e.To.Value] { - t.Errorf("%s edge TO a non-emitted node: %s %s", e.Type, e.To.Label, e.To.Value) + if c.name == "dataflow L4" { + for _, rel := range []string{RelParamIn, RelParamOut, RelSummary} { + if countEdges(rows, rel) == 0 { + t.Errorf("%s: no %s row checked", c.name, rel) + } + } } } } diff --git a/internal/syntactic_analysis/v2emit/gate_l4_test.go b/internal/syntactic_analysis/v2emit/gate_l4_test.go index f090ad5..8b2bab1 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_test.go @@ -300,7 +300,9 @@ func bodyIndex(a *v2.Analysis) map[string]ownedNode { // the way: the tree carries every vertex, port edge, summary edge, param // edge and stand-in formal of the sdg.Result. func TestGateL4_NoDanglingEndpoints(t *testing.T) { - for _, name := range l4Fixtures { + // rangefunc: the one fixture with degraded callables, so a dropped one + // fails the degraded rows of the table below. + for _, name := range append(slices.Clone(l4Fixtures), "rangefunc") { fx := buildL4(t, name) a := fx.l4 idx := bodyIndex(a) @@ -353,7 +355,15 @@ func TestGateL4_NoDanglingEndpoints(t *testing.T) { } } - var wantV, wantP, wantS, wantX int + var wantV, wantP, wantS, wantX, wantD, gotD int + for _, d := range fx.sdg.Degraded { + ks := slices.Clone(d.Callables) + slices.SortFunc(ks, func(a, b dataflow.L3Key) int { return cmpStrings(a.Path, b.Path, a.Signature, b.Signature) }) + wantD += len(slices.Compact(ks)) + } + for _, d := range a.Degraded { + gotD += len(d.Callables) + } for _, c := range fx.sdg.Callables { wantV, wantP, wantS = wantV+len(c.Vertices), wantP+len(c.Ddg), wantS+len(c.Summary) } @@ -374,16 +384,21 @@ func TestGateL4_NoDanglingEndpoints(t *testing.T) { {"param_in", len(a.Application.ParamIn), len(fx.sdg.ParamIn)}, {"param_out", len(a.Application.ParamOut), len(fx.sdg.ParamOut)}, {"stand-in formals", gotX, wantX}, + {"degraded entries", len(a.Degraded), len(fx.sdg.Degraded)}, + {"degraded callables", gotD, wantD}, } { if c.got != c.want { t.Errorf("%s: tree has %d %s, sdg.Result %d", name, c.got, c.what, c.want) } } - if name != "storedlit" && (wantV == 0 || len(fx.sdg.ParamIn) == 0 || wantS == 0) { + if name != "storedlit" && name != "rangefunc" && (wantV == 0 || len(fx.sdg.ParamIn) == 0 || wantS == 0) { t.Errorf("%s: empty overlay (vertices %d, param_in %d, summary %d)", name, wantV, len(fx.sdg.ParamIn), wantS) } - t.Logf("%s: %d vertices, %d port ddg, %d summary, %d param_in, %d param_out, %d stand-in formals", - name, vertices, port, summary, len(a.Application.ParamIn), len(a.Application.ParamOut), gotX) + if name == "rangefunc" && wantD == 0 { + t.Errorf("rangefunc: no degraded callable") + } + t.Logf("%s: %d vertices, %d port ddg, %d summary, %d param_in, %d param_out, %d stand-in formals, %d degraded callables", + name, vertices, port, summary, len(a.Application.ParamIn), len(a.Application.ParamOut), gotX, gotD) } } From c5f14ea59938e2b9a30c5dbbf13e2797789f1a40 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 14:32:08 -0400 Subject: [PATCH 35/65] fix(eval): a failed -a 4 run fails the app and the script --- scripts/eval_l3.sh | 9 ++++++++- scripts/eval_l3_test.sh | 37 +++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-) create mode 100755 scripts/eval_l3_test.sh diff --git a/scripts/eval_l3.sh b/scripts/eval_l3.sh index cad7dd1..761cf74 100755 --- a/scripts/eval_l3.sh +++ b/scripts/eval_l3.sh @@ -40,7 +40,8 @@ # 3. cango -a 3 -j N (N = CPU count) # 4. cango -a 3 -j N (second run) # 5. cango -a 4 -j 1 -v (only when CANGO_LEVELS includes 4) -# and compares the three -a 3 analysis.json files byte for byte. It records wall time, peak +# and compares the three -a 3 analysis.json files byte for byte. A run that exits non-zero or +# times out (the -a 4 run included) fails the app, and the script then exits 1. It records wall time, peak # RSS (macOS /usr/bin/time -l), module/callable/body-node/edge counts, external_symbols, the # call_graph split, the prepare-time type-check status, the number of type-error warnings # ("package ..." stderr lines), the analyzer's too-old-toolchain warning, and the number of @@ -437,6 +438,12 @@ for T in $TARGETS; do ident=NO; status="FAIL(-j differs)"; FAILS="$FAILS $name" fi fi + # A failed or timed-out -a 4 run fails the app too (the -a 3 checks above stand). + if [ "$RUN_L4" -eq 1 ] && [ "$RC4" -ne 0 ]; then + if [ "$RC4" -eq 124 ]; then s4="TIMEOUT(a4 >${RUN_TIMEOUT}s)"; else s4="FAIL(rc a4=$RC4)"; fi + if [ "$status" = ok ]; then status=$s4; else status="$status $s4"; fi + case " $FAILS " in *" $name "*) ;; *) FAILS="$FAILS $name" ;; esac + fi skips=$(grep -c 'L3 skipped' "$EVAL_OUT/$name-a3j1.err" 2>/dev/null || true) terrs=$(grep -c 'WARN\] package ' "$EVAL_OUT/$name-a3j1.err" 2>/dev/null || true) diff --git a/scripts/eval_l3_test.sh b/scripts/eval_l3_test.sh new file mode 100755 index 0000000..6be8a94 --- /dev/null +++ b/scripts/eval_l3_test.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# eval_l3_test.sh - check scripts/eval_l3.sh's exit status against a stub analyzer (no analysis runs). +# Usage: scripts/eval_l3_test.sh (exits non-zero on a failed check; needs /usr/bin/time -l, i.e. macOS) +set -u +S=$(cd "$(dirname "$0")" && pwd)/eval_l3.sh +T=$(mktemp -d) +trap 'rm -rf "$T"' EXIT +mkdir -p "$T/app" +# The stub writes a fixed analysis.json to -o and exits $FAIL_LEVEL_RC when -a is $FAIL_LEVEL. +cat >"$T/cango" <<'STUB' +#!/usr/bin/env bash +[ "${1:-}" = "--version" ] && { echo "stub"; exit 0; } +out="" level="" +while [ $# -gt 0 ]; do + case $1 in -o) out=$2; shift ;; -a) level=$2; shift ;; esac + shift +done +[ -n "$out" ] && echo '{"application":{"symbol_table":{},"call_graph":[]}}' >"$out/analysis.json" +[ "$level" = "${FAIL_LEVEL:-}" ] && exit 3 +exit 0 +STUB +chmod +x "$T/cango" +fail=0 +check() { # want-rc want-note-substring env... + local want=$1 msg=$2; shift 2 + local rc + env CANGO="$T/cango" EVAL_OUT="$T/out" EVAL_NOTE="$T/note.md" EVAL_JOBS=2 "$@" "$S" "$T/app" >/dev/null 2>"$T/err"; rc=$? + if [ "$rc" != "$want" ] || ! grep -q -- "$msg" "$T/note.md"; then + echo "FAIL: $* eval_l3.sh -> rc $rc (want $want, note containing '$msg')"; sed 's/^/ /' "$T/err" "$T/note.md"; fail=1 + else + echo "ok: $* eval_l3.sh -> rc $rc" + fi +} +check 0 "| app | ok |" CANGO_LEVELS="2 3 4" +check 1 "FAIL(rc a4=3)" CANGO_LEVELS="2 3 4" FAIL_LEVEL=4 +check 1 "FAIL(rc a2=0 j1=3" CANGO_LEVELS="2 3" FAIL_LEVEL=3 +exit $fail From 587a3df01018486d1257a44fc9836f7bfde407ff Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 14:32:29 -0400 Subject: [PATCH 36/65] ci: run the metamorphic script argument checks --- .github/workflows/ci.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 380ebc1..3016800 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -24,6 +24,9 @@ jobs: - name: Test run: go test ./... -count=1 + - name: Metamorphic script arguments + run: scripts/metamorphic_args_test.sh + # The release binary is built with the latest Go (release.yml), so test on # it too; the job above tests the go.mod minimum. test-stable: From 0607ef0ab2cfaa71a4f567f60707cdd266354194 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 14:35:01 -0400 Subject: [PATCH 37/65] test(l4): a released SDG cache rebuilt before fill ends fails Count per-unit cache builds through a test seam; with the memory levers on each graph, call-site list and wiring is built once and fill leaves the caches empty. --- internal/sdg/memlevers_test.go | 42 ++++++++++++++++++++++++++++++++++ internal/sdg/summary.go | 16 +++++++++++++ 2 files changed, 58 insertions(+) diff --git a/internal/sdg/memlevers_test.go b/internal/sdg/memlevers_test.go index 6b599e4..a94d6ad 100644 --- a/internal/sdg/memlevers_test.go +++ b/internal/sdg/memlevers_test.go @@ -4,6 +4,8 @@ import ( "reflect" "runtime/debug" "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" ) // TestSDG_OutputUnchangedByMemoryLevers: the memory levers (early cache @@ -47,3 +49,43 @@ func TestRun_RestoresGCPercent(t *testing.T) { } } } + +// TestSDG_ReleasedCachesAreNotRebuilt (Task 15b): with the memory levers on, +// a unit's flow graph, call sites and wiring are each built at most once — +// a cache released before its last reader is rebuilt silently, which costs +// time and memory but changes no fact, so only a miss count catches it — +// and fill leaves the per-unit caches empty. +func TestSDG_ReleasedCachesAreNotRebuilt(t *testing.T) { + for _, name := range []string{"dataflow", "chi"} { + t.Run(name, func(t *testing.T) { + defer func(v bool) { memLevers = v }(memLevers) + memLevers = true + builds := map[string]map[dataflow.L3Key]int{} + defer func(f func(string, dataflow.L3Key)) { onCacheMiss = f }(onCacheMiss) + onCacheMiss = func(cache string, k dataflow.L3Key) { + if builds[cache] == nil { + builds[cache] = map[dataflow.L3Key]int{} + } + builds[cache][k]++ + } + in := loadInput(t, name) + res, d := dispatchRun(in) + b := newSDGBuilder(in, d) + b.fill(&res) + for _, cache := range []string{"graph", "sites", "wired"} { + if len(builds[cache]) == 0 { + t.Errorf("no %s cache built", cache) + } + for k, n := range builds[cache] { + if n > 1 { + t.Errorf("%s of %s built %d times: released before its last reader", cache, k.Signature, n) + } + } + } + if len(b.graph)+len(b.sites)+len(b.wired)+len(b.summ) != 0 { + t.Errorf("after fill: %d graphs, %d sites, %d wirings, %d summaries still cached", + len(b.graph), len(b.sites), len(b.wired), len(b.summ)) + } + }) + } +} diff --git a/internal/sdg/summary.go b/internal/sdg/summary.go index 28bd297..5b1c2c7 100644 --- a/internal/sdg/summary.go +++ b/internal/sdg/summary.go @@ -27,6 +27,13 @@ type sdgBuilder struct { done bool } +// onCacheMiss, when set, is called each time the builder builds unit k's +// entry of a per-unit cache ("graph", "sites" or "wired"). A test seam: +// TestSDG_ReleasedCachesAreNotRebuilt checks that, with the memory levers +// on, no entry is built twice — a cache released before its last reader +// (releaseSCC, releaseUnit) is rebuilt silently, costing time, not facts. +var onCacheMiss func(cache string, k dataflow.L3Key) + func newSDGBuilder(in Input, d *dispatch) *sdgBuilder { return &sdgBuilder{ in: in, @@ -141,6 +148,9 @@ func (b *sdgBuilder) sitesOf(k dataflow.L3Key) []siteFlow { if s, ok := b.sites[k]; ok { return s } + if onCacheMiss != nil { + onCacheMiss("sites", k) + } var out []siteFlow for _, c := range b.p.units[k].Calls { s := b.p.callSite(k, c.Key, b.p.siteTargets(k, c.Key)) @@ -270,6 +280,9 @@ func (b *sdgBuilder) portEdges(k dataflow.L3Key) []PortEdge { if e, ok := b.wired[k]; ok { return e } + if onCacheMiss != nil { + onCacheMiss("wired", k) + } e := b.p.wire(k) b.wired[k] = e return e @@ -279,6 +292,9 @@ func (b *sdgBuilder) graphOf(k dataflow.L3Key) *unitGraph { if g, ok := b.graph[k]; ok { return g } + if onCacheMiss != nil { + onCacheMiss("graph", k) + } g := &unitGraph{idx: map[string]int{}} node := func(key string) int { i, ok := g.idx[key] From 21126206d7b7e7a1db3e124b0cafafb791c37fea Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 14:49:38 -0400 Subject: [PATCH 38/65] test(neo4j): live lazy L4 push over a lower-level graph equals an eager one --- internal/neo4j/bolt_integration_test.go | 45 +++++++++++++++++++++++++ internal/neo4j/project_test.go | 11 +++++- 2 files changed, 55 insertions(+), 1 deletion(-) diff --git a/internal/neo4j/bolt_integration_test.go b/internal/neo4j/bolt_integration_test.go index debc6d3..f5b2c12 100644 --- a/internal/neo4j/bolt_integration_test.go +++ b/internal/neo4j/bolt_integration_test.go @@ -21,6 +21,8 @@ import ( "testing" "github.com/neo4j/neo4j-go-driver/v5/neo4j" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" ) func itConfig(t *testing.T) BoltConfig { @@ -205,3 +207,46 @@ func l4Counts(t *testing.T, ctx context.Context, cfg BoltConfig, appID string) m } return out } + +// TestIntegration_LazyL4PushOverLowerLevelGraph: a graph first pushed at a +// lower level (L2, L3), then pushed lazily at L4 with every module's source +// unchanged, ends equal to an eager L4 push. The stored max_level below 4 +// forces the full upsert; without it the content_hash diff skips the +// unchanged modules, and an L2 graph lacks the L3 statement nodes the L4 +// port and cfg/ddg rows end at, so their MATCH drops them silently. +func TestIntegration_LazyL4PushOverLowerLevelGraph(t *testing.T) { + ctx := context.Background() + cfg := itConfig(t) + eager := cfg + eager.Eager, eager.FullRun = true, true + l4 := buildDataflowL4(t) + id := l4.Application.ID + rows4 := Project(l4) + if err := WriteBolt(ctx, rows4, id, eager); err != nil { + t.Fatalf("eager L4: %v", err) + } + wantN, wantE := countSubgraph(t, ctx, cfg, id) + want := l4Counts(t, ctx, cfg, id) + + for _, seed := range []*v2.Analysis{buildDataflowAt(t, 2), buildDataflowL3(t)} { + if seed.Application.ID != id { + t.Fatalf("app ids differ: %s, %s", seed.Application.ID, id) + } + if err := WriteBolt(ctx, Project(seed), id, eager); err != nil { + t.Fatalf("eager L%d: %v", seed.MaxLevel, err) + } + if got := l4Counts(t, ctx, cfg, id); got["GO_PARAM_IN"] != 0 { + t.Fatalf("L%d graph has L4 rows: %v", seed.MaxLevel, got) + } + if err := WriteBolt(ctx, rows4, id, cfg); err != nil { + t.Fatalf("lazy L4: %v", err) + } + gotN, gotE := countSubgraph(t, ctx, cfg, id) + if got := l4Counts(t, ctx, cfg, id); fmt.Sprint(got) != fmt.Sprint(want) { + t.Errorf("lazy L4 over L%d: L4 rows %v, want %v", seed.MaxLevel, got, want) + } + if gotN != wantN || gotE != wantE { + t.Errorf("lazy L4 over L%d: %d nodes, %d edges; eager L4 %d, %d", seed.MaxLevel, gotN, gotE, wantN, wantE) + } + } +} diff --git a/internal/neo4j/project_test.go b/internal/neo4j/project_test.go index e0bb16e..cebe5f9 100644 --- a/internal/neo4j/project_test.go +++ b/internal/neo4j/project_test.go @@ -219,6 +219,12 @@ func TestProject_ExternalGhostCarriesSignature(t *testing.T) { // buildDataflowL3 builds testdata/dataflow through the real -a 3 wiring (symbol // table, resolver, dataflow driver) and emits the v2 L3 analysis. func buildDataflowL3(t *testing.T) *v2.Analysis { + t.Helper() + return buildDataflowAt(t, 3) +} + +// buildDataflowAt is buildDataflowL3 at level 2 or 3 (no dataflow below 3). +func buildDataflowAt(t *testing.T, level int) *v2.Analysis { t.Helper() _, thisFile, _, _ := runtime.Caller(0) dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "dataflow")) @@ -233,8 +239,11 @@ func buildDataflowL3(t *testing.T) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + if level < 3 { + return v2emit.Emit(app, "dataflow", dir, level, "neo4j-gate", 3, nil, nil) + } res := dataflow.Run(b.Fset(), b.L3Units(st), 3, 4) - return v2emit.Emit(app, "dataflow", dir, 3, "neo4j-gate", 3, res, nil) + return v2emit.Emit(app, "dataflow", dir, level, "neo4j-gate", 3, res, nil) } // walkCallables visits every callable (methods and nested closures included). From 0724e200c0967cbbb8d113e4010bf019b0eaf06b Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 15:16:17 -0400 Subject: [PATCH 39/65] fix(neo4j): force a full push when the stored level is not an integer --- internal/neo4j/bolt.go | 9 ++++--- internal/neo4j/bolt_session_test.go | 37 ++++++++++++++++++++++++++++- 2 files changed, 42 insertions(+), 4 deletions(-) diff --git a/internal/neo4j/bolt.go b/internal/neo4j/bolt.go index fbf080e..9bf4fb1 100644 --- a/internal/neo4j/bolt.go +++ b/internal/neo4j/bolt.go @@ -194,14 +194,16 @@ func (w *boltWriter) ensureDDL() error { // its source, not the level it was analysed at, so an unchanged module // of a graph pushed at L3 still lacks what an L4 push adds to it — its L4 // vertices are upserted (D5), but rows the diff skips stay at L3, and an -// edge row whose endpoint is missing is dropped by its MATCH silently. +// edge row whose endpoint is missing is dropped by its MATCH silently; +// - a stored max_level present but not an integer: it cannot be compared, +// so the push fails safe. // // No stored application (first push) forces nothing beyond the normal // upsert, nor does a stored level at or above this push's. func (w *boltWriter) storedAppStale() (bool, error) { var stored string var storedLevel int64 - haveLevel := false + haveLevel, badLevel := false, false err := w.read(func(tx neo4j.ManagedTransaction) error { res, err := tx.Run(w.ctx, "MATCH (a:GoApplication {id: $id}) RETURN a.schema_version AS v, a.max_level AS l", @@ -215,6 +217,7 @@ func (w *boltWriter) storedAppStale() (bool, error) { } if l, ok := res.Record().Get("l"); ok && l != nil { storedLevel, haveLevel = l.(int64) + badLevel = !haveLevel } } return res.Err() @@ -222,7 +225,7 @@ func (w *boltWriter) storedAppStale() (bool, error) { if err != nil { return false, err } - if stored != "" && stored != SchemaVersion { + if stored != "" && stored != SchemaVersion || badLevel { return true, nil } return haveLevel && storedLevel < int64(w.pushLevel()), nil diff --git a/internal/neo4j/bolt_session_test.go b/internal/neo4j/bolt_session_test.go index a367ffa..d73f0bb 100644 --- a/internal/neo4j/bolt_session_test.go +++ b/internal/neo4j/bolt_session_test.go @@ -82,7 +82,10 @@ type fakeDB struct { neo4j.SessionWithContext hashes map[string]string level int64 - writes []fakeStmt + // rawLevel, when non-nil, is the stored max_level returned verbatim + // (any type), overriding level. + rawLevel any + writes []fakeStmt } type fakeStmt struct { @@ -115,6 +118,9 @@ func (tx fakeTx) Run(_ context.Context, q string, params map[string]any) (neo4j. if tx.db.level != 0 { lvl = tx.db.level } + if tx.db.rawLevel != nil { + lvl = tx.db.rawLevel + } return &fakeResult{rec: &neo4j.Record{Keys: []string{"v", "l"}, Values: []any{SchemaVersion, lvl}}}, nil case strings.Contains(q, "m.content_hash"): h, ok := tx.db.hashes[params["id"].(string)] @@ -304,3 +310,32 @@ func TestBolt_LazyPushForcesFullOnLowerStoredLevel(t *testing.T) { } } } + +// TestBolt_LazyPushForcesFullOnUnreadableStoredLevel: a stored max_level +// that is present but not an integer (a float, a string — a graph written by +// another tool, or hand-edited) cannot be compared with this push's level, +// so the lazy push fails safe and upserts every module, as for a lower one. +func TestBolt_LazyPushForcesFullOnUnreadableStoredLevel(t *testing.T) { + a := buildDataflowL4(t) + rows := Project(a) + byModule, _, _ := partitionNodes(rows.Nodes) + hashes := map[string]string{} + for m, ns := range byModule { + hashes[m] = moduleHash(ns) + } + for _, raw := range []any{float64(4), "4", true} { + db := &fakeDB{hashes: hashes, rawLevel: raw} + w := &boltWriter{ctx: context.Background(), session: db, rows: rows, appID: a.Application.ID} + if err := w.run(); err != nil { + t.Fatal(err) + } + nodes, _ := db.pushed() + for m, ns := range byModule { + for _, n := range ns { + if !nodes[n.Value] { + t.Fatalf("stored max_level %T(%v): module %s node %s not upserted", raw, raw, m, n.Value) + } + } + } + } +} From b3ba3dc722bfb3802c8f10ab8f81772de7748c80 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 15:16:17 -0400 Subject: [PATCH 40/65] test(l4): count summary recomputes; assert the duplicate-key warning --- internal/sdg/memlevers_test.go | 28 ++++++++++++++++++- internal/sdg/summary.go | 6 +++- .../syntactic_analysis/v2emit/emit_l4_test.go | 12 ++++++-- 3 files changed, 42 insertions(+), 4 deletions(-) diff --git a/internal/sdg/memlevers_test.go b/internal/sdg/memlevers_test.go index a94d6ad..165f00d 100644 --- a/internal/sdg/memlevers_test.go +++ b/internal/sdg/memlevers_test.go @@ -51,7 +51,9 @@ func TestRun_RestoresGCPercent(t *testing.T) { } // TestSDG_ReleasedCachesAreNotRebuilt (Task 15b): with the memory levers on, -// a unit's flow graph, call sites and wiring are each built at most once — +// a unit's flow graph, call sites and wiring are each built at most once, +// and fill never recomputes summary edges releaseSCC produced ("summ" +// misses: none) — // a cache released before its last reader is rebuilt silently, which costs // time and memory but changes no fact, so only a miss count catches it — // and fill leaves the per-unit caches empty. @@ -82,6 +84,9 @@ func TestSDG_ReleasedCachesAreNotRebuilt(t *testing.T) { } } } + if n := len(builds["summ"]); n != 0 { + t.Errorf("fill recomputed the summary edges of %d units: releaseSCC's were released before fill read them", n) + } if len(b.graph)+len(b.sites)+len(b.wired)+len(b.summ) != 0 { t.Errorf("after fill: %d graphs, %d sites, %d wirings, %d summaries still cached", len(b.graph), len(b.sites), len(b.wired), len(b.summ)) @@ -89,3 +94,24 @@ func TestSDG_ReleasedCachesAreNotRebuilt(t *testing.T) { }) } } + +// TestSDG_SummaryMissHookFires: the "summ" miss TestSDG_ReleasedCachesAreNotRebuilt +// counts is live — with the memory levers off, releaseSCC never runs, so +// fill computes every unit's summary edges through the miss path. +func TestSDG_SummaryMissHookFires(t *testing.T) { + defer func(v bool) { memLevers = v }(memLevers) + memLevers = false + misses := 0 + defer func(f func(string, dataflow.L3Key)) { onCacheMiss = f }(onCacheMiss) + onCacheMiss = func(cache string, _ dataflow.L3Key) { + if cache == "summ" { + misses++ + } + } + in := loadInput(t, "dataflow") + res, d := dispatchRun(in) + newSDGBuilder(in, d).fill(&res) + if misses != len(in.Units) { + t.Errorf("summ misses with the levers off = %d, want one per unit (%d)", misses, len(in.Units)) + } +} diff --git a/internal/sdg/summary.go b/internal/sdg/summary.go index 5b1c2c7..61d93ea 100644 --- a/internal/sdg/summary.go +++ b/internal/sdg/summary.go @@ -28,7 +28,8 @@ type sdgBuilder struct { } // onCacheMiss, when set, is called each time the builder builds unit k's -// entry of a per-unit cache ("graph", "sites" or "wired"). A test seam: +// entry of a per-unit cache ("graph", "sites" or "wired"), or recomputes +// its summary edges because releaseSCC's ("summ") are gone. A test seam: // TestSDG_ReleasedCachesAreNotRebuilt checks that, with the memory levers // on, no entry is built twice — a cache released before its last reader // (releaseSCC, releaseUnit) is rebuilt silently, costing time, not facts. @@ -273,6 +274,9 @@ func (b *sdgBuilder) summary(k dataflow.L3Key) []SummaryEdge { if s, ok := b.summ[k]; ok { return s } + if onCacheMiss != nil { + onCacheMiss("summ", k) + } return b.summaryEdges(k) } diff --git a/internal/syntactic_analysis/v2emit/emit_l4_test.go b/internal/syntactic_analysis/v2emit/emit_l4_test.go index 4cbd0c5..cdf0d91 100644 --- a/internal/syntactic_analysis/v2emit/emit_l4_test.go +++ b/internal/syntactic_analysis/v2emit/emit_l4_test.go @@ -1,6 +1,7 @@ package v2emit import ( + "fmt" "math/rand/v2" "strings" "testing" @@ -35,14 +36,17 @@ func TestCompareOrdinal_MatchesMintedIds(t *testing.T) { // TestL4Overlay_DuplicateKeyKeepsSmallerID: two callables sharing a target // key (declaring file and signature) route its param edges and degraded // entries to the lexically smaller id, whichever is applied first — -// emission walks maps, so last-write-wins would be nondeterministic. +// emission walks maps, so last-write-wins would be nondeterministic — and +// the clash is warned once, naming both ids. func TestL4Overlay_DuplicateKeyKeepsSmallerID(t *testing.T) { key := dataflow.L3Key{Path: "a.go", Signature: "example.com/a.F"} small, big := "can://app/go/a.go/example.com/a.F", "can://app/go/b.go/example.com/a.F" + var warned []string prev := warnf - warnf = func(string, ...any) {} + warnf = func(format string, args ...any) { warned = append(warned, fmt.Sprintf(format, args...)) } defer func() { warnf = prev }() for _, order := range [][]string{{small, big}, {big, small}} { + warned = nil o := newL4Overlay(4, &sdg.Result{}, sigIndex{}) for _, id := range order { o.apply(&v2.Callable{ID: id, Body: map[string]v2.BodyNode{}}, key) @@ -50,5 +54,9 @@ func TestL4Overlay_DuplicateKeyKeepsSmallerID(t *testing.T) { if got := o.ids[sdg.Target{Callable: key}]; got != small { t.Errorf("apply order %v: target routes to %s, want %s", order, got, small) } + if len(warned) != 1 || !strings.Contains(warned[0], small) || !strings.Contains(warned[0], big) || + !strings.Contains(warned[0], "share the target key") { + t.Errorf("apply order %v: warnings %q, want one naming both ids", order, warned) + } } } From 1fac5b00b0ce838c72a0ecab4fdebad2e7b74c87 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 15:33:04 -0400 Subject: [PATCH 41/65] test(l4): HTTP-to-SQL done fixture --- internal/sdg/ssa_test.go | 10 +- .../v2emit/gate_l4_done_test.go | 161 ++++++++++++++++++ .../syntactic_analysis/v2emit/gate_l4_test.go | 8 +- testdata/dataflow/httpsql/httpsql.go | 77 +++++++++ 4 files changed, 248 insertions(+), 8 deletions(-) create mode 100644 internal/syntactic_analysis/v2emit/gate_l4_done_test.go create mode 100644 testdata/dataflow/httpsql/httpsql.go diff --git a/internal/sdg/ssa_test.go b/internal/sdg/ssa_test.go index 7a1de53..8aaacc9 100644 --- a/internal/sdg/ssa_test.go +++ b/internal/sdg/ssa_test.go @@ -64,7 +64,7 @@ func TestSSA_BuildsProjectPackagesOnly(t *testing.T) { built = append(built, p.Pkg.Path()) } want := []string{ - "example.com/dataflow/asm", "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/initclosure", "example.com/dataflow/inits", + "example.com/dataflow/asm", "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/httpsql", "example.com/dataflow/initclosure", "example.com/dataflow/inits", "example.com/dataflow/l1", "example.com/dataflow/linedir", "example.com/dataflow/sdg", "example.com/dataflow/vend", "example.com/dataflow/xfile", "example.com/dataflow/xtest", "example.com/dataflow/xtest/user", } @@ -137,8 +137,8 @@ func TestSSA_PanicDegradesPackage(t *testing.T) { t.Error("the panicked package is still returned as built") } } - if len(pkgs) != 11 { - t.Errorf("built %d packages, want the other 11", len(pkgs)) + if len(pkgs) != 12 { + t.Errorf("built %d packages, want the other 12", len(pkgs)) } if fn := prog.ImportedPackage("example.com/dataflow/l1"); fn == nil || len(fn.Members) == 0 { t.Error("l1 was not built after flow panicked") @@ -156,8 +156,8 @@ func TestSSA_PanicDegradesPackage(t *testing.T) { if !slices.Equal(codes, []string{"partial_ssa", "no_ssa"}) { t.Fatalf("codes = %v, want [partial_ssa no_ssa]", codes) } - if len(deg[0].Packages) != 12 { - t.Errorf("partial_ssa lists %v, want all 12 packages", deg[0].Packages) + if len(deg[0].Packages) != 13 { + t.Errorf("partial_ssa lists %v, want all 13 packages", deg[0].Packages) } if res := Run(in); len(res.Degraded) != 2 || res.Degraded[1].Code != "no_ssa" { t.Errorf("Run degraded = %+v, want partial_ssa + no_ssa and no VTA", res.Degraded) diff --git a/internal/syntactic_analysis/v2emit/gate_l4_done_test.go b/internal/syntactic_analysis/v2emit/gate_l4_done_test.go new file mode 100644 index 0000000..4d9066f --- /dev/null +++ b/internal/syntactic_analysis/v2emit/gate_l4_done_test.go @@ -0,0 +1,161 @@ +package v2emit + +// The L4 "done" gate (spec §8): on the HTTP → SQL fixture +// testdata/dataflow/httpsql, the emitted JSON graph carries a flow from an +// HTTP handler's request parameter to database/sql's DB.Query query +// argument through an interface call, and a handler that never queries +// reaches nothing there. + +import ( + "slices" + "testing" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// sdgGraph is the emitted JSON's dependence graph over global body node ids: +// every callable's ddg, cdg and summary edges (local keys minted with +// v2.GlobalOrdinal) and the application's param_in and param_out — the +// edge families the SDK's slice and taint walks follow. Each edge records +// its family, so a path can be checked for the hops it takes. +type sdgGraph struct { + succ map[string][]sdgHop +} + +type sdgHop struct { + to, family string +} + +func newSDGGraph(a *v2.Analysis) *sdgGraph { + g := &sdgGraph{succ: map[string][]sdgHop{}} + add := func(src, dst, family string) { g.succ[src] = append(g.succ[src], sdgHop{dst, family}) } + for _, c := range l4Callables(a) { + id := func(k string) string { return v2.GlobalOrdinal(c.ID, k) } + for _, e := range c.Ddg { + add(id(e.Src), id(e.Dst), "ddg") + } + for _, e := range c.Cdg { + add(id(e.Src), id(e.Dst), "cdg") + } + for _, e := range c.Summary { + add(id(e.Src), id(e.Dst), "summary") + } + } + for _, e := range a.Application.ParamIn { + add(e.Src, e.Dst, "param_in") + } + for _, e := range a.Application.ParamOut { + add(e.Src, e.Dst, "param_out") + } + return g +} + +// path is a shortest path from src to dst by BFS, as the hops taken (nil if +// dst is unreachable). Successors are visited in edge order, so the path is +// deterministic. +func (g *sdgGraph) path(src, dst string) []sdgHop { + type pred struct { + from string + hop sdgHop + } + prev := map[string]pred{src: {}} + for q := []string{src}; len(q) > 0; q = q[1:] { + n := q[0] + if n == dst { + var out []sdgHop + for n != src { + p := prev[n] + out = append(out, p.hop) + n = p.from + } + slices.Reverse(out) + return out + } + for _, h := range g.succ[n] { + if _, seen := prev[h.to]; !seen { + prev[h.to] = pred{n, h} + q = append(q, h.to) + } + } + } + return nil +} + +// TestGateL4_HTTPToSQLPath: Handler's closure reads r.URL.Query().Get("id") +// and calls repo.Find(id) on the Repo interface; VTA binds sqlRepo.Find, +// which passes "…" + id to db.Query. A BFS over the emitted graph from the +// closure's r formal_in reaches the stand-in DB.Query's formal_in of +// "query", and the path enters sqlRepo.Find through a param_in. Health +// never queries: from its r, the sink is unreachable. +func TestGateL4_HTTPToSQLPath(t *testing.T) { + fx := buildL4(t, "dataflow/httpsql") + a := fx.l4 + if len(a.Degraded) != 0 { + t.Fatalf("httpsql degraded: %+v", a.Degraded) + } + app := a.Application.ID + if app != "can://httpsql" { + t.Fatalf("application id %s, want can://httpsql", app) + } + idx := bodyIndex(a) + vertex := func(owner, kind, of string) string { + t.Helper() + var found []string + for id, on := range idx { + if on.owner == owner && on.node.Kind == kind && on.node.Of == of { + found = append(found, id) + } + } + if len(found) != 1 { + t.Fatalf("%s has %d %s vertices of %q, want 1", owner, len(found), kind, of) + } + return found[0] + } + mod := v2.ModuleID(app, "httpsql.go") + pkg := "example.com/dataflow/httpsql." + fn := func(name string) string { return v2.CallableID(mod, pkg+name) } + closure := func(name string) string { return v2.CallableID(fn(name), pkg+name+".closure_1") } + method := func(typ, m string) string { return v2.CallableID(v2.TypeID(mod, pkg+typ), pkg+typ+"."+m) } + + sink := vertex(v2.ExternalID(app, "database/sql", "DB.Query"), "formal_in", "query") + if sink != "can://httpsql/@external/database/sql/DB.Query@formal_in:1" { + t.Fatalf("sink vertex %s", sink) + } + find := method("sqlRepo", "Find") + g := newSDGGraph(a) + + src := vertex(closure("Handler"), "formal_in", "r") + p := g.path(src, sink) + if p == nil { + t.Fatalf("no path from Handler's r (%s) to %s", src, sink) + } + intoFind := slices.ContainsFunc(p, func(h sdgHop) bool { + return h.family == "param_in" && idx[h.to].owner == find + }) + if !intoFind { + t.Errorf("path %v does not enter %s through a param_in", p, find) + } + // The interface call binds sqlRepo.Find by VTA. + vta := false + for _, e := range a.Application.CallGraph { + vta = vta || e.Src == closure("Handler") && e.Dst == find && slices.Contains(e.Prov, "vta") + } + if !vta { + t.Errorf("no VTA call_graph edge %s -> %s", closure("Handler"), find) + } + for i, h := range p { + t.Logf("hop %d: %s -> %s", i, h.family, h.to) + } + // The shortest path may take the receiver (the coarse host wiring feeds + // every actual_in of repo.Find(id)); the flow of id itself must exist + // too: r → Find's id formal_in → the query actual_in. + findID := vertex(find, "formal_in", "id") + if g.path(src, findID) == nil || g.path(findID, sink) == nil { + t.Errorf("no path %s → %s → %s", src, findID, sink) + } + + // A pair with no flow: Health never touches the database. + if p := g.path(vertex(fn("Health"), "formal_in", "r"), sink); p != nil { + t.Errorf("Health's r reaches %s: %v", sink, p) + } +} diff --git a/internal/syntactic_analysis/v2emit/gate_l4_test.go b/internal/syntactic_analysis/v2emit/gate_l4_test.go index 8b2bab1..f530575 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_test.go @@ -41,7 +41,8 @@ var ( ) // buildL4 runs the -a 4 pipeline over testdata/ once and emits both the -// L3 and the L4 payload from the same resolved symbol table. +// L3 and the L4 payload from the same resolved symbol table. The app name is +// name's last element (cango's default: the input directory's base name). func buildL4(t *testing.T, name string) *l4Fixture { t.Helper() l4Mu.Lock() @@ -64,7 +65,8 @@ func buildL4(t *testing.T, name string) *l4Fixture { app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} units := b.L3Units(st) res := dataflow.RunWithSideTables(b.Fset(), units, 3, 4) - fx := &l4Fixture{st: st, l3: Emit(app, name, dir, 3, "gate-test", 3, res, nil), res: res, dir: dir} + appName := filepath.Base(dir) + fx := &l4Fixture{st: st, l3: Emit(app, appName, dir, 3, "gate-test", 3, res, nil), res: res, dir: dir} l4 := sdg.Run(sdg.Input{ Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, Externals: app.Externals, CallGraph: app.CallGraph, @@ -72,7 +74,7 @@ func buildL4(t *testing.T, name string) *l4Fixture { }) l4.ApplyTo(app) fx.sdg = &l4 - fx.l4 = Emit(app, name, dir, 4, "gate-test", 3, res, &l4) + fx.l4 = Emit(app, appName, dir, 4, "gate-test", 3, res, &l4) l4Cache[name] = fx return fx } diff --git a/testdata/dataflow/httpsql/httpsql.go b/testdata/dataflow/httpsql/httpsql.go new file mode 100644 index 0000000..5235618 --- /dev/null +++ b/testdata/dataflow/httpsql/httpsql.go @@ -0,0 +1,77 @@ +// Package httpsql is the L4 "done" fixture (spec §8): an HTTP handler's +// request reaches database/sql's DB.Query through an interface call, and a +// second handler takes the same route through strconv.Atoi first. +package httpsql + +import ( + "database/sql" + "net/http" + "strconv" +) + +// Repo is the storage interface the handlers call; sqlRepo is its one +// implementation, so the interface calls bind sqlRepo's methods by VTA. +type Repo interface { + Find(id string) (*sql.Rows, error) + SanitizedFind(id string) (*sql.Rows, error) +} + +type sqlRepo struct{ db *sql.DB } + +// NewRepo returns the database-backed Repo. +func NewRepo(db *sql.DB) Repo { return &sqlRepo{db: db} } + +// Find builds the query from id: the sink. +func (s *sqlRepo) Find(id string) (*sql.Rows, error) { + return s.db.Query("SELECT name FROM users WHERE id = " + id) +} + +// SanitizedFind parses id as a number first and queries with the number. +func (s *sqlRepo) SanitizedFind(id string) (*sql.Rows, error) { + n, err := strconv.Atoi(id) + if err != nil { + return nil, err + } + return s.db.Query("SELECT name FROM users WHERE id = " + strconv.Itoa(n)) +} + +// Handler reads the id query parameter and looks it up. +func Handler(repo Repo) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + id := r.URL.Query().Get("id") + rows, err := repo.Find(id) + if err != nil { + http.Error(w, "lookup failed", http.StatusInternalServerError) + return + } + rows.Close() + } +} + +// SanitizedHandler is Handler through SanitizedFind. +func SanitizedHandler(repo Repo) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + id := r.URL.Query().Get("id") + rows, err := repo.SanitizedFind(id) + if err != nil { + http.Error(w, "lookup failed", http.StatusInternalServerError) + return + } + rows.Close() + } +} + +// Health answers without touching the database. +func Health(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) +} + +// Routes wires the handlers to a database-backed Repo: without it no +// concrete type reaches the handlers' repo, and VTA binds repo.Find to +// nothing but the interface method. +func Routes(mux *http.ServeMux, db *sql.DB) { + repo := NewRepo(db) + mux.Handle("/user", Handler(repo)) + mux.Handle("/user/sanitized", SanitizedHandler(repo)) + mux.HandleFunc("/health", Health) +} From a2e7ea059c7f4fd337500a1ae4b4cef59120b8cc Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 15:35:31 -0400 Subject: [PATCH 42/65] feat(eval): -a 4 in the evaluation CANGO_LEVELS="2 3 4" runs the -a 4 triple (-j 1, -j N, rerun) with a byte-identity check and records L4 counts, times, bytes and degraded codes; l3_metrics.py adds an l4 section. --- scripts/eval_l3.sh | 87 ++++++++++++++++++++++------- scripts/eval_l3_test.sh | 16 ++++++ scripts/l3_metrics.py | 120 +++++++++++++++++++++++++++++++++++++--- 3 files changed, 195 insertions(+), 28 deletions(-) diff --git a/scripts/eval_l3.sh b/scripts/eval_l3.sh index 761cf74..84a9ee2 100755 --- a/scripts/eval_l3.sh +++ b/scripts/eval_l3.sh @@ -39,9 +39,12 @@ # 2. cango -a 3 -j 1 # 3. cango -a 3 -j N (N = CPU count) # 4. cango -a 3 -j N (second run) -# 5. cango -a 4 -j 1 -v (only when CANGO_LEVELS includes 4) -# and compares the three -a 3 analysis.json files byte for byte. A run that exits non-zero or -# times out (the -a 4 run included) fails the app, and the script then exits 1. It records wall time, peak +# 5. cango -a 4 -j 1 -v (only when CANGO_LEVELS includes 4; steps 5-7) +# 6. cango -a 4 -j N +# 7. cango -a 4 -j N (second run) +# and compares the three -a 3 analysis.json files byte for byte, and the three -a 4 ones. A run that +# exits non-zero or times out (the -a 4 runs included), or a byte difference at either level, fails +# the app, and the script then exits 1. It records wall time, peak # RSS (macOS /usr/bin/time -l), module/callable/body-node/edge counts, external_symbols, the # call_graph split, the prepare-time type-check status, the number of type-error warnings # ("package ..." stderr lines), the analyzer's too-old-toolchain warning, and the number of @@ -62,10 +65,15 @@ # RUN_TIMEOUT per-run limit, seconds (default 3600); on expiry the run is killed, the app # is marked TIMEOUT and its remaining runs are skipped # CANGO_LEVELS levels to run (default "2 3"). Levels 2 and 3 always run (the L3 baseline); -# adding 4 (e.g. "2 3 4") also runs step 5 and records its wall time, peak RSS, and the -# SSA build time, VTA time and VTA edge count from its -v log. -# EVAL_PRUNE 1 = after the comparison, delete the warm-up and both -j N analysis.json files -# (keeps a2j1 and a3j1, and every .err/.time). For very large apps. +# adding 4 (e.g. "2 3 4") also runs steps 5-7 and records, from the -a 4 -j 1 run: +# wall time, peak RSS (the whole process: L1-L3, SSA, the L4 overlay and the +# emission) and its ratio to -a 3, user CPU, analysis.json bytes, the SSA build, VTA +# and L4-overlay times, and from its -v log the L4 counts (vertices, param_in, +# param_out, summary, port ddg, points-to ddg, stand-ins, call sites with dispatch +# targets, unmapped SSA functions/calls) and the degraded codes; plus the -a 4 -j N +# wall time and the -a 4 byte-identity check. +# EVAL_PRUNE 1 = after the comparison, delete the warm-up, both -j N and every -a 4 +# analysis.json file (keeps a2j1 and a3j1, and every .err/.time). For very large apps. # Any other environment (GOOS, GOFLAGS, ...) is inherited by the analyzer unchanged. # # Requires: git, go, /usr/bin/time -l (BSD/macOS), cmp. jq is used for counts; without it the @@ -150,7 +158,7 @@ if [ "${1:-}" = "--clean" ]; then [ -d "$EVAL_OUT" ] || exit 0 case $EVAL_OUT in /|"$HOME"|"$EVAL_SRC") echo "refusing to clean $EVAL_OUT" >&2; exit 2 ;; esac for n in $(manifest_names) $(ls "$EVAL_OUT" 2>/dev/null | sed -n 's/-a3j1$//p'); do - for t in warm a2j1 a3j1 a3jN a3jN2 a4j1; do + for t in warm a2j1 a3j1 a3jN a3jN2 a4j1 a4jN a4jN2; do rm -rf "$EVAL_OUT/$n-$t" "$EVAL_OUT/$n-$t.err" "$EVAL_OUT/$n-$t.time" "$EVAL_OUT/$n-$t.timeout" done rm -rf "$EVAL_OUT/cache-$n" @@ -352,9 +360,30 @@ run_one() { RC=$? if [ -e "$dir.timeout" ]; then RC=124; fi WALL=$(awk '/ real /{print $1}' "$dir.time") + USER=$(awk '/ real /{print $3}' "$dir.time") RSS=$(awk '/maximum resident set size/{printf "%d", $1/1048576}' "$dir.time") + BYTES=$(wc -c <"$dir/analysis.json" 2>/dev/null | tr -d ' ') [ -n "$WALL" ] || WALL=n/a + [ -n "$USER" ] || USER=n/a [ -n "$RSS" ] || RSS=n/a + [ -n "$BYTES" ] || BYTES=n/a +} + +# l4_log ERRFILE -> the L4 columns of the note from an -a 4 -v stderr log: SSA, VTA and overlay +# times, then vertices | param_in | param_out | summary | port ddg | points-to | stand-ins | +# call sites with targets | unmapped functions/calls | degraded codes (sorted, comma-joined). +l4_log() { + local e=$1 ssa vta ov counts sites unm codes + ssa=$(sed -n 's/.*L4: SSA built .* in \(.*\)$/\1/p' "$e" 2>/dev/null | head -1) + vta=$(sed -n 's/.*L4: VTA over .* edges in \(.*\)$/\1/p' "$e" 2>/dev/null | head -1) + ov=$(sed -n 's/.*L4: [0-9]* vertices, .* stand-ins in \(.*\)$/\1/p' "$e" 2>/dev/null | head -1) + counts=$(sed -n 's/.*L4: \([0-9]*\) vertices, \([0-9]*\) port edges (\([0-9]*\) points-to), \([0-9]*\) summary edges, \([0-9]*\) param_in, \([0-9]*\) param_out, \([0-9]*\) stand-ins in .*/\1 | \5 | \6 | \4 | \2 | \3 | \7/p' "$e" 2>/dev/null | head -1) + sites=$(sed -n 's/.*L4: dispatch: \([0-9]*\) call sites with targets.*/\1/p' "$e" 2>/dev/null | head -1) + unm=$(sed -n 's/.*L4: SSA join: .*; \([0-9]*\) functions, \([0-9]*\) calls unmapped in .*/\1\/\2/p' "$e" 2>/dev/null | head -1) + codes=$(sed -n 's/.*L4 degraded: \([a-z_]*\) .*/\1/p' "$e" 2>/dev/null | sort -u | paste -sd, -) + [ -n "$counts" ] || counts="n/a | n/a | n/a | n/a | n/a | n/a | n/a" + printf '%s | %s | %s | %s | %s | %s | %s' "${ssa:-n/a}" "${vta:-n/a}" "${ov:-n/a}" "$counts" \ + "${sites:-n/a}" "${unm:-n/a}" "${codes:-none}" } BUILD_INFO=$(go version -m "$CANGO" 2>/dev/null) @@ -368,7 +397,8 @@ CANGO_DIRTY=$(printf '%s\n' "$BUILD_INFO" | awk -F'=' '/vcs.modified=/{print $2} echo "Analyzer: $("$CANGO" --version 2>&1 | head -1), built with ${CANGO_GO:-unknown} (runtime.Version), rev ${CANGO_REV:-unknown}; runs use GOTOOLCHAIN=$RUN_GO; host: $(uname -sm); j=$JOBS; all runs --eager." echo if [ "$RUN_L4" -eq 1 ]; then - L4H=" wall a4 j1 (s) | RSS a4 j1 (MB) | RSS a4/a3 | SSA a4 | VTA a4 | VTA edges |"; L4S="---|---|---|---|---|---|" + L4H=" wall a4 j1 (s) | wall a4 jN (s) | user CPU a4 j1 (s) | RSS a4 j1 (MB) | RSS a4/a3 | JSON a3 (B) | JSON a4 (B) | SSA | VTA | L4 overlay | vertices | param_in | param_out | summary | port ddg | points-to | stand-ins | sites with targets | unmapped fn/calls | degraded | a4 -j identical |" + L4S="---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|" else L4H=""; L4S="" fi @@ -397,12 +427,12 @@ for T in $TARGETS; do status=ok rm -f "$EVAL_OUT/$name"-*.timeout - W3=n/a; WN=n/a; R3=n/a; RC3=0; RCN=0; RCN2=0 + W3=n/a; WN=n/a; R3=n/a; B3=n/a; RC3=0; RCN=0; RCN2=0 # Warm-up (discarded): populates the Go build/module cache and the OS file cache so the # timed -a 2 and -a 3 runs are comparable. run_one "$name" warm 2 1; RCW=$RC run_one "$name" a2j1 2 1; W2=$WALL; RC2=$RC - if [ "$RCW" -ne 124 ] && [ "$RC" -ne 124 ]; then run_one "$name" a3j1 3 1; W3=$WALL; R3=$RSS; RC3=$RC; fi + if [ "$RCW" -ne 124 ] && [ "$RC" -ne 124 ]; then run_one "$name" a3j1 3 1; W3=$WALL; R3=$RSS; RC3=$RC; B3=$BYTES; fi if [ "$RC3" -ne 124 ] && [ "$RC2" -ne 124 ]; then run_one "$name" a3jN 3 "$JOBS"; WN=$WALL; RCN=$RC if [ "$RCN" -ne 124 ]; then run_one "$name" a3jN2 3 "$JOBS"; RCN2=$RC; fi @@ -410,17 +440,25 @@ for T in $TARGETS; do L4COLS="" if [ "$RUN_L4" -eq 1 ]; then - W4=n/a; R4=n/a; RATIO=n/a; RC4=0 + W4=n/a; WN4=n/a; U4=n/a; R4=n/a; B4=n/a; RATIO=n/a; RC4=0; RCN4=0; RCN42=0; ident4=n/a if [ "$RC3" -ne 124 ] && [ "$RC2" -ne 124 ]; then - run_one "$name" a4j1 4 1 -v; W4=$WALL; R4=$RSS; RC4=$RC - if [ "$RC4" -ne 0 ]; then W4="$W4 (rc $RC4)"; fi + run_one "$name" a4j1 4 1 -v; W4=$WALL; U4=$USER; R4=$RSS; B4=$BYTES; RC4=$RC + if [ "$RC4" -ne 124 ]; then run_one "$name" a4jN 4 "$JOBS"; WN4=$WALL; RCN4=$RC; fi + if [ "$RC4" -ne 124 ] && [ "$RCN4" -ne 124 ]; then run_one "$name" a4jN2 4 "$JOBS"; RCN42=$RC; fi + # One status for the triple: the first non-zero rc (124 = timeout). + for r in $RC4 $RCN4 $RCN42; do [ "$RC4" -eq 0 ] && RC4=$r; done + if [ "$RC4" -ne 0 ]; then W4="$W4 (rc $RC4)" + else + a4=$EVAL_OUT/$name-a4j1/analysis.json + if cmp -s "$a4" "$EVAL_OUT/$name-a4jN/analysis.json" && cmp -s "$a4" "$EVAL_OUT/$name-a4jN2/analysis.json"; then + ident4=yes + else + ident4=NO + fi + fi fi case $R3$R4 in *[!0-9]*|"") ;; *) RATIO=$(awk -v a="$R4" -v b="$R3" 'BEGIN{printf "%.2f", a/b}') ;; esac - e4=$EVAL_OUT/$name-a4j1.err - SSA4=$(sed -n 's/.*L4: SSA built .* in \(.*\)$/\1/p' "$e4" 2>/dev/null | head -1) - VTA4=$(sed -n 's/.*L4: VTA over .* edges in \(.*\)$/\1/p' "$e4" 2>/dev/null | head -1) - VE4=$(sed -n 's/.*L4: VTA over .*: \([0-9]*\) edges in .*/\1/p' "$e4" 2>/dev/null | head -1) - L4COLS=" $W4 | $R4 | $RATIO | ${SSA4:-n/a} | ${VTA4:-n/a} | ${VE4:-n/a} |" + L4COLS=" $W4 | $WN4 | $U4 | $R4 | $RATIO | $B3 | $B4 | $(l4_log "$EVAL_OUT/$name-a4j1.err") | $ident4 |" fi ident=n/a @@ -438,9 +476,15 @@ for T in $TARGETS; do ident=NO; status="FAIL(-j differs)"; FAILS="$FAILS $name" fi fi - # A failed or timed-out -a 4 run fails the app too (the -a 3 checks above stand). + # A failed or timed-out -a 4 run, or -a 4 outputs that differ, fail the app too (the -a 3 + # checks above stand). + s4="" if [ "$RUN_L4" -eq 1 ] && [ "$RC4" -ne 0 ]; then if [ "$RC4" -eq 124 ]; then s4="TIMEOUT(a4 >${RUN_TIMEOUT}s)"; else s4="FAIL(rc a4=$RC4)"; fi + elif [ "$RUN_L4" -eq 1 ] && [ "$ident4" = NO ]; then + s4="FAIL(a4 -j differs)" + fi + if [ -n "$s4" ]; then if [ "$status" = ok ]; then status=$s4; else status="$status $s4"; fi case " $FAILS " in *" $name "*) ;; *) FAILS="$FAILS $name" ;; esac fi @@ -461,7 +505,8 @@ EOC if [ "$EVAL_PRUNE" = 1 ] && [ "$ident" = yes ]; then rm -f "$EVAL_OUT/$name-warm/analysis.json" "$EVAL_OUT/$name-a3jN/analysis.json" \ - "$EVAL_OUT/$name-a3jN2/analysis.json" "$EVAL_OUT/$name-a4j1/analysis.json" + "$EVAL_OUT/$name-a3jN2/analysis.json" "$EVAL_OUT/$name-a4j1/analysis.json" \ + "$EVAL_OUT/$name-a4jN/analysis.json" "$EVAL_OUT/$name-a4jN2/analysis.json" fi done diff --git a/scripts/eval_l3_test.sh b/scripts/eval_l3_test.sh index 6be8a94..2f35ade 100755 --- a/scripts/eval_l3_test.sh +++ b/scripts/eval_l3_test.sh @@ -16,6 +16,18 @@ while [ $# -gt 0 ]; do shift done [ -n "$out" ] && echo '{"application":{"symbol_table":{},"call_graph":[]}}' >"$out/analysis.json" +# NONDET_LEVEL: a run at that level appends a per-run nonce (a nondeterministic analyzer). +[ "$level" = "${NONDET_LEVEL:-}" ] && echo "$$-$RANDOM" >>"$out/analysis.json" +# At -a 4 with -v, print the L4 log lines eval_l3.sh parses. +if [ "$level" = 4 ]; then + echo "[codeanalyzer-go] L4: SSA built 2 project packages in 5ms" >&2 + echo "[codeanalyzer-go] L4: SSA join: 9 functions (1 by wrapper, 0 without a unit), 20 calls; 1 functions, 3 calls unmapped in 1ms" >&2 + echo "[codeanalyzer-go] L4: VTA over 12 functions (3 library targets): 30 edges in 2ms" >&2 + echo "[codeanalyzer-go] L4: dispatch: 7 call sites with targets; 4 call_graph edges and 1 externals added by VTA" >&2 + echo "[codeanalyzer-go] L4: 40 vertices, 50 port edges (6 points-to), 8 summary edges, 11 param_in, 10 param_out, 5 stand-ins in 4ms" >&2 + echo "[codeanalyzer-go WARN] L4 degraded: unmapped_ssa (1 SSA function(s) and 3 call instruction(s) map to no callable or call node): example.com/a" >&2 + echo "[codeanalyzer-go WARN] L4 degraded: range_func (range-over-func is not modelled at L4 (D8)): example.com/b" >&2 +fi [ "$level" = "${FAIL_LEVEL:-}" ] && exit 3 exit 0 STUB @@ -32,6 +44,10 @@ check() { # want-rc want-note-substring env... fi } check 0 "| app | ok |" CANGO_LEVELS="2 3 4" +# The -a 4 triple: -j 1, -j N and a rerun, compared byte for byte; the L4 log counts and codes. +check 0 "| 40 | 11 | 10 | 8 | 50 | 6 | 5 | 7 | 1/3 | range_func,unmapped_ssa | yes |" CANGO_LEVELS="2 3 4" +check 1 "FAIL(a4 -j differs)" CANGO_LEVELS="2 3 4" NONDET_LEVEL=4 +check 1 "FAIL(-j differs)" CANGO_LEVELS="2 3" NONDET_LEVEL=3 check 1 "FAIL(rc a4=3)" CANGO_LEVELS="2 3 4" FAIL_LEVEL=4 check 1 "FAIL(rc a2=0 j1=3" CANGO_LEVELS="2 3" FAIL_LEVEL=3 exit $fail diff --git a/scripts/l3_metrics.py b/scripts/l3_metrics.py index 4394fe9..b5f3db5 100644 --- a/scripts/l3_metrics.py +++ b/scripts/l3_metrics.py @@ -8,7 +8,10 @@ switch / select / type-switch fan-out, defer-chain and exception-edge shape, the per-callable DDG distribution and its top callables, `var` classes (wildcard, `:` pseudo path), DDG volume by generated-file class and by callable name (e.g. protobuf `Marshal*` / `Unmarshal`), approximate bytes per section, and the internal-consistency -checks. The figures in docs/design/notes/l3-dataflow-metrics.md come from this script. It loads +checks. On an `-a 4` file it adds an `l4` object: vertices by kind and by `of` class, param_in / +param_out, summary edges (and how many are all-to-all at their call site), port / points-to ddg, +stand-ins and their formals, the share of call nodes with a VTA call_graph target, the degraded +codes, and the L4 consistency checks (no dangling endpoint, no global → global summary). The figures in docs/design/notes/l3-dataflow-metrics.md come from this script. It loads each file whole: a 1.1 GB analysis.json needs about 11 GB of RAM. """ import json @@ -20,6 +23,7 @@ CFG_KINDS = ["fallthrough", "true", "false", "switch_case", "loop_back", "exception", "return", "break", "continue", "defer_resume"] NODE_KINDS = ["call", "statement", "branch", "loop", "switch", "return", "entry", "exit"] +VERTEX_KINDS = ("formal_in", "formal_out", "actual_in", "actual_out") # L4 IF_ERR = re.compile(rb"^if\s+(?:[^{]*;\s*)?err\s*!=\s*nil\s*\{") @@ -102,8 +106,11 @@ def src(f): K["no_cfg"][c.get("kind") + (" with body" if c.get("body") else " no body")] += 1 continue C["with_cfg"] += 1 - body = c.get("body") or {} - cfg, cdg, ddg = c.get("cfg") or [], c.get("cdg") or [], c.get("ddg") or [] + # The L3 figures leave out the L4 overlay (port vertices, reaching-defs / points-to + # ddg), so they compare across -a 3 and -a 4; analyze_l4 counts the overlay. + body = {k: n for k, n in (c.get("body") or {}).items() if n.get("kind") not in VERTEX_KINDS} + cfg, cdg = c.get("cfg") or [], c.get("cdg") or [] + ddg = [e for e in c.get("ddg") or [] if e.get("prov") == ["ssa"]] if len(body) == 2: C["entry_exit_only_bodies"] += 1 @@ -234,19 +241,22 @@ def closure_of(start, adj): top.sort(key=lambda x: -x[0]) del top[5:] + l4 = analyze_l4(doc, st) if doc.get("max_level", 0) >= 4 else None ddg_per.sort() top.sort(key=lambda x: -x[0]) top_detail = [] for n, _, c in top[:5]: - vc = Counter(e["var"] for e in c["ddg"]) + l3ddg = [e for e in c["ddg"] if e.get("prov") == ["ssa"]] + vc = Counter(e["var"] for e in l3ddg) defs, uses = defaultdict(set), defaultdict(set) - for e in c["ddg"]: + for e in l3ddg: defs[e["var"]].add(e["src"]) uses[e["var"]].add(e["dst"]) top_detail.append({ - "signature": c.get("signature"), "ddg": n, "body": len(c["body"]), "cfg": len(c["cfg"]), + "signature": c.get("signature"), "ddg": n, "cfg": len(c["cfg"]), + "body": sum(1 for b in c["body"].values() if b.get("kind") not in VERTEX_KINDS), "cyclomatic": (c.get("metrics") or {}).get("cyclomatic"), - "node_kinds": dict(Counter(b.get("kind") for b in c["body"].values())), + "node_kinds": dict(Counter(b.get("kind") for b in c["body"].values() if b.get("kind") not in VERTEX_KINDS)), "distinct_vars": len(vc), "top_vars": [[v, k, len(defs[v]), len(uses[v])] for v, k in vc.most_common(4)]}) k1 = max(1, len(ddg_per) // 100) @@ -268,6 +278,102 @@ def closure_of(start, adj): "top1pct_share": round(sum(ddg_per[-k1:]) / max(1, C["ddg"]), 4)}, "ddg_top": top_detail, "check_failures": {k: v for k, v in chk.items() if v}, + **({"l4": l4} if l4 is not None else {}), + } + + + +def of_class(of): + for p in (":", ":", ""): + if of.startswith(p): + return p.rstrip(":") + return "_recv" if of == "_recv" else "param" + + +def analyze_l4(doc, st): + """The L4 overlay of an -a 4 analysis.json. + + Dispatch per call site comes from param_in: a site's targets are the owners of the formals + its actual_ins bind (`@formal_in:`); a target other than the call node's static + `callee` was added by VTA (an interface call's concrete methods, a func-value call's + functions). A site whose callee is not null but binds nothing has no vertices (a pruned call + node or an external with no formals).""" + a = doc["application"] + C, K, chk = Counter(), defaultdict(Counter), Counter() + ids, calls = set(), {} + for c, closure, mpath, sfile in callables(st): + body = c.get("body") or {} + sites = defaultdict(lambda: [0, 0]) # call key -> [#actual_in, #actual_out] + for k, n in body.items(): + ids.add(n.get("id")) + kd = n.get("kind") + if kd in VERTEX_KINDS: + K["vertices_by_kind"][kd] += 1 + K["vertex_of_class"][kd + " " + of_class(n.get("of", ""))] += 1 + if kd.startswith("actual"): + sites[n.get("parent")][kd == "actual_out"] += 1 + elif kd == "call": + calls[n["id"]] = n.get("callee") + per_site = Counter() + for e in c.get("summary") or []: + for x in (e["src"], e["dst"]): + chk["summary_endpoint_not_vertex"] += body.get(x, {}).get("kind") not in VERTEX_KINDS + if (body.get(e["src"], {}).get("of", "").startswith(":") + and body.get(e["dst"], {}).get("of", "").startswith(":")): + chk["summary_global_to_global"] += 1 + per_site[body.get(e["src"], {}).get("parent")] += 1 + C["summary"] += 1 + for site, (ni, no) in sites.items(): + if per_site[site]: + C["sites_with_summary"] += 1 + C["sites_with_summary_all_to_all"] += per_site[site] == ni * no + for e in c.get("ddg") or []: + prov = e.get("prov") or [] + if prov == ["reaching-defs"]: + C["port_ddg"] += 1 + elif prov == ["points-to"]: + C["points_to_ddg"] += 1 + C["vertices"] = sum(K["vertices_by_kind"].values()) + C["call_nodes"] = len(calls) + C["call_nodes_with_callee"] = sum(1 for v in calls.values() if v) + ext = a.get("external_symbols") or {} + C["external_symbols"] = len(ext) + for e in ext.values(): + b = e.get("body") or {} + C["stand_ins"] += bool(b) + C["stand_in_formals"] += len(b) + for n in b.values(): + ids.add(n.get("id")) + for e in a.get("call_graph") or []: + prov = e.get("prov") or [] + C["call_graph_vta"] += "vta" in prov + C["call_graph_vta_only"] += prov == ["vta"] + targets = defaultdict(set) + for lst in ("param_in", "param_out"): + es = a.get(lst) or [] + C[lst] = len(es) + for e in es: + chk[lst + "_endpoint_dangling"] += e["src"] not in ids or e["dst"] not in ids + if lst == "param_in" and "/actual_in:" in e["src"]: + targets[e["src"].rsplit("/actual_in:", 1)[0]].add(e["dst"].rsplit("@formal_in:", 1)[0]) + for site, ts in targets.items(): + if site not in calls: + C["capture_sites"] += 1 # a non-call capture site (a literal's position) + continue + C["call_sites_bound"] += 1 + extra = ts - {calls[site]} + C["call_sites_with_vta_target"] += bool(extra) + C["call_sites_multi_target"] += len(ts) > 1 + K["targets_per_bound_site"][min(len(ts), 10)] += 1 + deg = doc.get("degraded") or [] + return { + **dict(C), **{k: dict(v) for k, v in K.items()}, + "pct_call_nodes_with_vta_target": round(100 * C["call_sites_with_vta_target"] / max(1, C["call_nodes"]), 2), + "pct_bound_sites_with_vta_target": round(100 * C["call_sites_with_vta_target"] / max(1, C["call_sites_bound"]), 2), + "degraded": {d["code"]: {"packages": len(d.get("packages") or []), + "callables": len(d.get("callables") or []), + "detail": d.get("detail", "")} for d in deg}, + "check_failures": {k: v for k, v in chk.items() if v}, } From a98f6b788e72c0f0e1974ef9cb76f6d1ecfa30e1 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 16:01:08 -0400 Subject: [PATCH 43/65] docs: L4 evaluation, spec and README --- CLAUDE.md | 62 +++- README.md | 110 ++++++- docs/design/notes/l3-dataflow-metrics.md | 360 +++++++++++++++++++++-- docs/design/notes/l3-eval-2026-10.md | 38 ++- docs/design/roadmap.md | 34 ++- docs/design/specs/v2-l3-l4-dataflow.md | 94 ++++-- docs/design/specs/v2-schema-structure.md | 43 +-- internal/sdg/vta.go | 6 +- 8 files changed, 642 insertions(+), 105 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index b918fbf..e1aee78 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -95,10 +95,37 @@ committed spec is the full transcript: `docs/design/specs/v2-l1-emission.md`. - **`k_limit`** (envelope, L3+ only) = `--graph-field-depth` (default 3), the access-path bound of the syntactic `ddg` (`x.f.g.h` → `x.f.g.*`); `ddg.prov` is `["ssa"]` (parity). -### Deferred (not this train) -- L4 (SDG param edges) — reuses Group A ids/spans and the L3 vocabulary unchanged. Designed in - `docs/design/specs/v2-l3-l4-dataflow.md`. A Go-specific `select` cfg kind was not added - (`select` is a `switch` with `switch_case` edges). +### L4 SDG (`-a 4`; spec `docs/design/specs/v2-l3-l4-dataflow.md` §4.6–§4.16, §6) +Reuses Group A ids/spans and the L3 vocabulary unchanged; every L4 fact is an addition +(`-a 3` ⊆ `-a 4`, gated). Leaves: +- **Vertices** (body nodes, span-less, no `callee`): `@formal_in:`, `@formal_out[:]`, + `/actual_in:`, `/actual_out[:]` (Python's `IdentityMap` indexing: + `formal_in`/`actual_in` always indexed; an out is bare iff its group has one vertex). Actuals + carry `parent` = the call key (a non-call capture site: the literal's host). **`of`** (D6, + Python spelling, which the SDK's resolver reads): the param/receiver name; unnamed or `_` param + `_` (declared index, receiver excluded); unnamed or `_` receiver **`_recv`**; results + `` / **`:`** (never user result names; `:` is a Go leaf); + `:`; `:::`. The SDK reads `of`, never + `parameters[]` (`formal_in:0` is a method's receiver). +- **Edges:** application `param_in[]` / `param_out[]` `{src, dst, var}` (global ids, `var` = the + callee formal's `of`); callable `summary[] {src, dst}` (local keys, actual_in → actual_out); + port wiring in `ddg` with `prov: ["reaching-defs"]` (Python precedent; coarse host wiring: + host → every actual_in, every actual_out → host); may-alias `ddg` with `prov: ["points-to"]`. +- **Dispatch:** VTA over the project's functions (library targets in the set); `call_graph` + gains VTA-only edges `prov: ["vta"]`, weight 1 (an L2 edge is never changed). **D7:** an + interface call binds the interface method (pass-through summary) *and* every VTA target. +- **Stand-ins:** `external_symbols[].receiver {name, type}` and `.body` (formal vertices only, + ids `@formal_in:`); summary all-to-all unless `--library-models` overrides it + (keys `/`, values `[in, out]` `of` pairs). +- **`degraded[]`** envelope leaf `{code, detail?, packages[]?, callables[]?}`, absent when + nothing degraded; codes `no_ssa | no_vta | partial_ssa | partial_types | unmapped_ssa | + l3_skipped | range_func` (open vocabulary). `--strict` exits non-zero on any. +- **Rulings:** global → global summary edges are omitted [user, 2026-10-26; #30]; memory is + measured and reported per app, not gated (D3) [user, 2026-10-26]; range-over-func is not + modelled, degraded `range_func` (D8); `-a 4 -t` is an error and `--emit neo4j -t` projects + at L3 with a warning (D9). +- A Go-specific `select` cfg kind was not added (`select` is a `switch` with `switch_case` + edges). ## Neo4j-projection decisions @@ -185,11 +212,17 @@ shared neutral vocabulary verbatim; never rename a sibling's shared name). Spec: It carries `call_node` (JSON `parent` verbatim — the **local** key, see § Flattened properties) and `var` (JSON `of`; L4 — absent at L3). `GO_HAS_BODY_NODE` may originate at `:GoExternal` (L4 stand-ins). -### Reserved L4 overlay (declared now, emits nothing until L4 lands) -- `schema.go` still declares `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY` (`Reserved`) with - parity-correct names; the projector emits **zero rows** for them. When L4 lands they fill - with **no schema-contract change**. (`--emit neo4j` is "full-depth = max implemented" — - L3 today; the graph grows additively.) +### L4 overlay (live; `--emit neo4j` runs at L4, D4) +- `GO_PARAM_IN {var}`, `GO_PARAM_OUT {var}` and `GO_SUMMARY` are live (no longer `Reserved`; + same names as declared, no contract change). One row per JSON `param_in[]` / `param_out[]` / + `summary[]` entry; endpoints are `:GoBodyNode` global ids. Port and points-to ddg ride + `GO_DDG {var, prov}` (`_k = var|prov`). +- Vertices are `:GoBodyNode`s with `var` (= JSON `of`) and `call_node` (actuals); stand-in + vertices hang off their `:GoExternal` via `GO_HAS_BODY_NODE`. `:GoApplication.degraded_json` + carries the envelope's `degraded[]`. +- **Lazy push (D5):** every L4 row is upserted on every push (L4 facts are non-local); stale L4 + rows need `--eager`. A stored `max_level` below the push's (or unreadable) forces a full + upsert; a lower-level push leaves old L4 rows. `--emit neo4j -t` projects at L3 (D9). ### SDK facade (`GoAnalysisBackend`, on the `python-sdk` fork — frontend rung) - **Type-centric surface** (Go is the SDK's first class-less facade; no C precedent exists @@ -202,10 +235,13 @@ shared neutral vocabulary verbatim; never rename a sibling's shared name). Spec: getters + `get_source`. Slicing / CFG / CDG / DDG / reachability are **omitted** (they need L3/L4 the analyzer does not produce) per the ABC contract. (2026-10-24: L3 is now emitted at `-a 3` — per-callable `cfg`/`cdg`/`ddg` and `GO_CFG_NEXT`/`GO_CDG`/`GO_DDG` — - so L3 accessors are supportable now; interprocedural slicing and taint still need L4: - `param_in`/`param_out`/`summary` and the points-to `ddg`.) + so L3 accessors are supportable now. 2026-10-27: L4 is emitted at `-a 4` and by + `--emit neo4j` — `param_in`/`param_out`/`summary` and the points-to `ddg` — so + interprocedural slicing and taint are supportable too.) ### Deferred (not this train) -- SDK consumer slicing/taint surface and L3/L4 facade methods — arrive with the L3/L4 - analyzer train, reusing this graph vocabulary unchanged. +- SDK consumer slicing/taint surface and L3/L4 facade methods — the analyzer now emits L4; + the SDK side reuses this graph vocabulary unchanged. - Struct-tag modeling. (`defer_resume` landed with L3; see § L3 body and edges.) +- `GO_EMBEDS` / `GO_SATISFIES` resolution: `base_types[]` holds raw type strings and + `interfaces[]` is never computed (#31, pre-existing). diff --git a/README.md b/README.md index 4110360..0eca0eb 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,10 @@ The binary is fully self-contained: a single static executable with no runtime d - **Call graph** — a resolver-based call graph via `go/types`: each call site is resolved to its full import-path signature. Level 2 emits edges to project callees and to library callees (homed in `application.external_symbols` under `can:///@external//`). - Level 3 (`-a 3`) adds per-callable `cfg`, `cdg` and `ddg`. + Level 3 (`-a 3`) adds per-callable `cfg`, `cdg` and `ddg`. Level 4 (`-a 4`) adds the + interprocedural system dependence graph (SDG): parameter vertices, `param_in` / `param_out`, + call-site `summary` edges, VTA dispatch targets and library stand-ins — what the SDK's + slicing and `taint()` walk. - **Generics-aware** — Go 1.18+ generics (`Set[T]`, union-constraint interfaces, multi-type-param functions) are modeled in the unified type/callable schema. - **Self-contained binary** — a single static executable (`go build`, `CGO_ENABLED=0`); no runtime @@ -277,7 +280,7 @@ are build artifacts, not project source), so only your own declarations are emit | 1 | `-a 1` (default) | Symbol table only — types, functions, call sites | Implemented | | 2 | `-a 2` | Level 1 + resolver-based call graph via `go/types` | Implemented | | 3 | `-a 3` | Level 2 + per-callable statement nodes and `cfg` / `cdg` / `ddg` | Implemented | -| 4 | `-a 4` | Level 3 + interprocedural parameter edges | Not implemented (rejected) | +| 4 | `-a 4` | Level 3 + interprocedural SDG: VTA dispatch, parameter vertices, `param_in` / `param_out`, `summary`, port and `points-to` `ddg`, library stand-ins | Implemented | **Level 1** loads each package with `packages.NeedSyntax | NeedTypes | NeedTypesInfo` and walks the AST file by file. Call sites are recorded with `callee_signature = null` at this stage. @@ -291,6 +294,48 @@ recorded in the envelope as `k_limit`. It is a flag error below `-a 3` and with `--emit neo4j`. Callables are analyzed in parallel (`-j`), and the output is byte-identical for every `-j`. L3 only adds to the L2 output: every L2 fact stays as it is. +**Level 4** is whole-program. It builds `go/ssa` for the project packages over the same load and +runs VTA (`golang.org/x/tools/go/callgraph/vta`) over the project's functions, so interface and +function-value calls get their concrete targets: `call_graph` gains those edges with +`prov: ["vta"]` (an L2 edge is never changed). Every callable gets `formal_in` / `formal_out` +vertices (parameters, receiver, closure captures, the package variables it reads or writes, +results), and every resolved call site gets `actual_in` / `actual_out` vertices, wired into the +body's `ddg` (`prov: ["reaching-defs"]`). `param_in` / `param_out` (application scope) join each +call site to every target, including the interface method itself for an interface call, and a +bottom-up pass over the call graph's SCCs puts `summary` edges (`actual_in` → `actual_out`) at +each call site. Library callees get a **stand-in** in `external_symbols` (`body` holds only its +formal vertices; summary all inputs → all outputs unless `--library-models` overrides it). A +type-based may-alias oracle adds `ddg` edges with `prov: ["points-to"]`. `-a 3` output is a +subset of `-a 4` output, and `-j 1` and `-j N` output are byte-identical. + +- **Degradation.** When L4 cannot build all it needs, it still emits and records why in the + envelope's `degraded[]`. Each entry has a `code`, an optional `detail`, and optional `packages[]` + and `callables[]`. The codes are: + - `no_ssa` / `partial_ssa`: SSA failed, in whole or for some packages; + - `no_vta`: VTA failed; + - `partial_types`: some packages have type errors; + - `unmapped_ssa`: SSA functions or calls that map to no callable; + - `l3_skipped`: callables without an L3 result, which get a pass-through summary; + - `range_func`: range-over-func, which is not modelled at L4. + + It also logs one warning per entry. `--strict` makes any degradation a non-zero exit. +- **`--library-models `** overrides stand-in summaries: + `{"strconv/Atoi": [], "strings/TrimSpace": [["s", ""]]}`. A key is the external id's + `/` tail, and a value is a list of `[in, out]` pairs of `of` names (`[]` = no + flow). An entry naming an `of` the external lacks falls back to all-to-all, with a warning. A + key no call site targets is ignored, with a warning. The flag is valid only at `-a 4` and with + `--emit neo4j`. +- **`-t` / `--target-files` is not supported at `-a 4`** (whole-program): it is an error. Use + `-a 3 -t`, or run without `-t`. +- **Known limits** (see `docs/design/notes/l3-dataflow-metrics.md` § L4): + - library code calling back into the project (`sort.Slice`, `http.HandleFunc`) is not an + edge; + - range-over-func is not modelled (`range_func`); + - summaries are close to all-to-all, by design; + - `summary` edges whose two ends are both `:` vertices are omitted + (codeanalyzer-go#30). Reachability is unchanged, but a depth-bounded slice can stop earlier + than it would through that shortcut. + **Level 2** adds a resolver pass: for each call site, `go/types` resolves the callee to its full import-path signature (`pkgImportPath.TypeName.MethodName`). A project callee becomes an edge to its callable; a **library callee** (stdlib, a dependency, or a package vendored under `vendor/`) becomes an edge to an `@external` symbol, recorded once in `application.external_symbols`. A callee that is neither (an interface method of a project type, a conversion, a builtin, a call through a function value) gets no edge. Every resolved call node's `callee` is backfilled. ## Output schema @@ -369,9 +414,9 @@ node carries a `can:///…` `id`, a `kind`, and a `span`: ``` Key v2 schema properties: -- **Envelope** — `schema_version` (`"2.0.0"`), `language` (`"go"`), `max_level` (1, 2 or 3), - `k_limit` (L3 only: the `--graph-field-depth` value) and `analyzer{name,version}` wrap a - single `application` node. +- **Envelope** — `schema_version` (`"2.0.0"`), `language` (`"go"`), `max_level` (1–4), + `k_limit` (L3+: the `--graph-field-depth` value), `degraded[]` (L4, only when something + degraded) and `analyzer{name,version}` wrap a single `application` node. - **One containment tree** — `application → module → type → callable → body`, keyed as `symbol_table` (modules, by **project-relative file path**), `types`, `callables`, `body`. - **`id`** — every node carries a durable `can:///go///` id; @@ -406,6 +451,15 @@ Key v2 schema properties: A call nested in a statement carries `parent`, the statement's key. The callable gains `cfg[] {src, dst, kind}`, `cdg[] {src, dst}` and `ddg[] {src, dst, var, prov}`, whose endpoints are `body` keys. See `docs/design/specs/v2-schema-structure.md`. +- **L4 (`-a 4`)** — `body` gains span-less vertices `@formal_in:`, `@formal_out[:]`, + `/actual_in:`, `/actual_out[:]`, each with `of` (the bound name: + a parameter or receiver name, `_`, `_recv`, ``, `:`, + `:`, `:::`); actuals carry `parent` = the call key. + `ddg` gains `prov: ["reaching-defs"]` (port wiring) and `["points-to"]` edges; a callable gains + `summary[] {src, dst}` (local keys); the application gains `param_in[]` / `param_out[]` + `{src, dst, var}` (global body-node ids, `var` = the callee formal's `of`); an external gains + `receiver {name, type}` and a stand-in `body` of its formal vertices (ids + `@formal_in:`). ## Neo4j projection (`--emit neo4j`) @@ -441,10 +495,27 @@ What a re-emit replaces depends on the path: remain beside the new ones. Pass `--eager` when you push an L3 graph after editing source. The purge needs a full run: with `--target-files` it is skipped, because the push carries only the targeted packages. - -> **`--emit neo4j` is always full depth.** The graph is projected at the max implemented level (L3 -> today — symbol table, call graph, and every callable's body nodes with `GO_CFG_NEXT` / `GO_CDG` / -> `GO_DDG`) and canonical schema v2. Do **not** pass `-a`/`--analysis-level` +- **L4 rows on a lazy push (D5).** L4 facts are not local to a module: editing a callee changes + its callers' summaries and vertices. So a lazy push upserts **every** L4 row (vertices, + `GO_PARAM_IN` / `GO_PARAM_OUT`, `GO_SUMMARY`, port `GO_DDG`) on every push, not only the changed + modules'. It still adds and never removes: stale L4 rows stay until an `--eager` full run. +- **Level changes.** A lazy push whose level is above the stored `:GoApplication.max_level` (a graph + last pushed at L3, or with `-t`), or whose stored level is unreadable, upserts every module, so + no L4 row is dropped for a missing L3 endpoint. A push at a *lower* level onto a higher graph + diffs as usual and leaves the old L4 rows in place (additive). An **interrupted** push can leave + `:GoApplication` at the new `max_level` while some modules still hold older rows; the next lazy + push then trusts the diff. Recover with an `--eager` full run. +- **`--target-files` (D9).** L4 is whole-program, so `--emit neo4j -t` projects the targeted modules + at **L3** and writes no L4 rows; they keep the last full run's facts until the next full run. It + warns once on stderr: `--target-files: projecting the graph at level 3 (level 4 is + whole-program); L4 rows keep the last full run's facts until a full run` (plus `; + --library-models is ignored at level 3` when that flag is set). + +> **`--emit neo4j` is always full depth.** The graph is projected at the max implemented level (L4 +> — symbol table, call graph, every callable's body nodes with `GO_CFG_NEXT` / `GO_CDG` / +> `GO_DDG`, and the SDG: vertex `:GoBodyNode`s with `var`, `GO_PARAM_IN` / `GO_PARAM_OUT {var}`, +> `GO_SUMMARY`, stand-in vertices under their `:GoExternal`, and `degraded_json` on +> `:GoApplication`) and canonical schema v2. Do **not** pass `-a`/`--analysis-level` > with it — an explicit level is a flag error (`--analysis-level does not apply to --emit neo4j`), > because there is no shallower graph to ask for. @@ -578,8 +649,21 @@ check the following. as plain `line:col`. - Calls inside `go` statement arguments, such as `g()` in `go f(g())`, are now call sites. - Consumers that count call nodes will see more of them. +- **L4 (`-a 4`) is new.** It is additive: an `-a 3` consumer sees the same document plus new + keys (vertices, `of`, `summary`, `param_in` / `param_out`, `points-to` / `reaching-defs` `ddg`, + VTA `call_graph` edges, external `receiver` / `body`, `degraded`). Code that assumes every body + node has a `span`, or that every `ddg` edge has `prov: ["ssa"]`, must handle the L4 kinds. + `-a 4 -t` is an error (whole-program; D9). `--library-models` is new and valid only at L4. + `-a 4` takes several times the memory and output of `-a 3` (measured per app in + `docs/design/notes/l3-dataflow-metrics.md` § L4). - **Neo4j:** - - `--emit neo4j` projects L3, adding `GO_CFG_NEXT`, `GO_CDG` and `GO_DDG`. + - `--emit neo4j` projects **L4** (D4), adding `GO_CFG_NEXT`, `GO_CDG` and `GO_DDG`, and the SDG + rows `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY`, vertex `:GoBodyNode`s with `var`, and + `degraded_json`. `-a` with it is still an error. + - A lazy push upserts every L4 row on every push and removes none (D5); use `--eager` to drop + stale ones. `--emit neo4j -t` projects at L3 with a warning (D9). A push at a higher level + than the stored graph upserts everything; a lower-level push leaves the old L4 rows. See + [Neo4j projection](#neo4j-projection---emit-neo4j). - `:GoBodyNode.call_node` is the enclosing statement's local key (the JSON `parent`), not a global id. - `--eager` now works on Neo4j 5.x. @@ -595,10 +679,13 @@ check the following. | Type resolver | `golang.org/x/tools/go/packages` | Single API for both AST + full type resolution; handles modules natively | | Build/dep materialization | `go mod download` | Required before `packages.Load` so the module cache is warm; result cached by `go.sum` hash | | Packaging | Native binary (`go build`) | Zero-runtime-dep distribution; matches Rust/C++ analyzers | -| Analysis depth | Levels 1–3 | Symbol table, resolver call graph, intraprocedural cfg/cdg/ddg | +| Analysis depth | Levels 1–4 | Symbol table, resolver call graph, intraprocedural cfg/cdg/ddg, interprocedural SDG | | Call-graph dispatch | Declared-type resolution via `go/types.Selections` | CHA-equivalent; sufficient for cross-package reachability at Level 1 | | CFG (L3) | Hand-built from `go/ast` (`internal/dataflow`) | Adds what `go/cfg` does not model as edges: panic `exception` edges and the `defer` chain (`defer_resume`). A differential test checks reachability against `golang.org/x/tools/go/cfg` on every fixture body | | Def-use / DDG (L3) | Syntactic reaching definitions over k-limited access paths | `go/types` objects as bases (shadowing-safe); weak defs through reference arguments; `prov: ["ssa"]` for parity with the Java and Python L3 | +| SSA (L4) | `golang.org/x/tools/go/ssa` via `ssautil.Packages` over the same load, project packages only, built per package under `recover` | Bodies only for the project (dependencies from types); a panic degrades instead of aborting | +| Dispatch oracle (L4) | VTA (`go/callgraph/vta`) over the project's functions, library targets included | Concrete targets for interface and function-value calls; an interface call also binds the interface method (pass-through) | +| Alias oracle (L4) | Type-based may-alias (structural pointee compatibility, interface pairs, address-taken values, one-level field offset) | Additive `points-to` `ddg` edges; an interface so Andersen points-to can replace it without a schema change | ### Package structure @@ -612,6 +699,7 @@ codeanalyzer-go/ │ ├── syntactic_analysis/ # SymbolTableBuilder (packages.Load → AST walk) │ ├── semantic_analysis/ # CallGraphBuilder (go/types resolver) │ ├── dataflow/ # L3: body nodes, CFG, CDG, def-use, DDG; Run fans out per callable +│ ├── sdg/ # L4: SSA, VTA dispatch, footprints, vertices, port wiring, summaries, stand-ins, may-alias │ ├── analysis/ # Pluggable pass interface + registry (topo-ordered pipeline) │ ├── frameworks/ # BaseEntrypointFinder — extension seam for framework passes │ └── utils/ # DiscoverGoFiles, IsVendored, IsTestFile, logging diff --git a/docs/design/notes/l3-dataflow-metrics.md b/docs/design/notes/l3-dataflow-metrics.md index 61f8a12..9441430 100644 --- a/docs/design/notes/l3-dataflow-metrics.md +++ b/docs/design/notes/l3-dataflow-metrics.md @@ -31,8 +31,8 @@ degraded-input details. This note reuses its timings and extends its counts. rerun on the same trees where the earlier outputs were not kept (dae linux, both Kubernetes modules). -One analysis engine, so there is no engine comparison here. Interprocedural (L4) results will be -added under [§ L4](#l4-to-be-added). +One analysis engine, so there is no engine comparison here. Interprocedural (L4) results, measured +2026-10-27, are under [§ L4](#l4). ## Results at a glance @@ -102,8 +102,14 @@ scripts/eval_l3.sh cobra listmonk memos nats-server openbao dae kubernetes k8s-a # L3 metrics and consistency checks, one JSON object per app: python3 -I scripts/l3_metrics.py cobra=/tmp/eval-out/cobra-a3j1/analysis.json \ k8s-api=/tmp/eval-out/k8s-api-a3j1/analysis.json ... > metrics.jsonl + +# § L4: add the -a 4 triple (-j 1, -j N, rerun; byte-compared) and its counts: +CANGO_LEVELS="2 3 4" scripts/eval_l3.sh cobra listmonk memos nats-server openbao dae kubernetes k8s-api +python3 -I scripts/l3_metrics.py cobra=/tmp/eval-out/cobra-a4j1/analysis.json ... > metrics-a4.jsonl ``` +(`--all` also names cockroach, which is not prepared on this host, so the apps are listed.) + - **What `--prepare` does.** For each app in the script's manifest: a shallow, blob-filtered fetch of the pinned commit into `/tmp/eval/` (an existing clone is reused only at that commit), a tool check, the app's prep (dae: the `dae_bpf_headers` submodule and `make ebpf`, which needs @@ -789,26 +795,330 @@ in any total above. node against every def in the body. A reviewer probe of one 6,000-case switch-heavy function took 15 s. Nothing in this corpus comes close. -## L4 (to be added) - -L4 (interprocedural SDG) results will be added here, over the same corpus and pinned commits, with -the same reproduction harness (`scripts/eval_l3.sh --prepare`, then run). Planned measurements: - -- **The points-to delta.** `ddg` edges with `prov:["points-to"]` added on top of the L3 `ssa` - edges, which must be strictly additive (§4.10): count, share, per-app, and which access-path - shapes the type-based oracle links. -- **Parameter wiring.** `@formal_in` / `@formal_out` vertices and the per-call-site actual - vertices (§4.6), and `GO_PARAM_IN` / `GO_PARAM_OUT` edges (§4.7). These are counted per callable - and per resolved call site, against L2's `call_graph` edges. -- **Summary edges** (§4.9): count, SCC sizes, fixpoint iterations on recursive SCCs, and the - cost of the bottom-up pass. -- **Library stand-ins** (§6): the number of external stand-ins with modelled bodies, and how many - `formal_in`/`formal_out` vertices they contribute. -- **VTA dispatch** (§4.8): call edges added over the `go/types` call graph. -- **Cost and size.** Wall time, RSS and `analysis.json` growth of `-a 4` over `-a 3`, plus the - `-j` determinism check, with Kubernetes as the scale benchmark. -- **Consistency at scale.** Every param edge joins a real call site to a real formal, and every - summary edge spans a call site's actual-in to actual-out. -- **Effect on the L3 caveats above.** Whether points-to edges make the weak-def tail denser, and - whether the protobuf-marshaller triangle compounds across `MarshalToSizedBuffer(dAtA[:i])` - calls once they are wired interprocedurally. +## L4 + +L4 is the interprocedural system dependence graph (SDG). At `-a 4`, on top of L3, every callable +gets `formal_in` / `formal_out` vertices, and every resolved call site gets `actual_in` / +`actual_out` vertices. These are wired into the callable's `ddg` (`prov:["reaching-defs"]`). The +application carries `param_in` / `param_out`, every call site carries `summary` edges, VTA adds +dispatch targets to `call_graph` (`prov:["vta"]`), library callees get stand-in formals in +`external_symbols[].body`, and a type-based may-alias oracle adds `ddg` edges with +`prov:["points-to"]`. The semantics are in [`v2-l3-l4-dataflow.md`](../specs/v2-l3-l4-dataflow.md) +§4.6–§4.16 and §6. + +- **When and where:** measured 2026-10-27 on the same host, an Apple M3 Max (16 cores, 64 GB, + macOS 15). The analyzer was built by `scripts/eval_l3.sh` with **go1.27.3** (`runtime.Version()`), + and every run used `GOTOOLCHAIN=go1.27.3`. Every number in this section is go1.27.3 unless it + says otherwise. Absolute RSS differs by about 1 GB between go1.26.4 and go1.27.3 on nats-server + (Task 15b), so do not compare figures across toolchains. +- **Binary:** `feat/l4-sdg` at `c230592`. The `+dirty` marker covers uncommitted docs only; the + analyzer code is that commit's. It includes the Task 17 alias rounds (`04244db`, `18fca40`), + the capture fixes (`078616f`, `cd37383`) and the Task 18a fix-ups. +- **Protocol:** `CANGO_LEVELS="2 3 4" scripts/eval_l3.sh` over the same eight prepared trees and + pinned commits as the L3 sections. Each app runs the L3 sequence (§ How to reproduce) and then + the **`-a 4` triple**: `-a 4 -j 1 -v`, `-a 4 -j 16`, and a second `-a 4 -j 16`, all + `--eager`. The three `-a 4` files are compared byte for byte. Wall time, user CPU and peak RSS + come from `/usr/bin/time -l`. That peak is the **whole process**: load, L1–L3, SSA, VTA, the + L4 overlay and the streamed emission. L4 counts come from `scripts/l3_metrics.py` (its new `l4` + object) and the `-v` log. +- **Gate (spec §8):** every app completes `-a 4` with no crash, and its three outputs are + byte-identical. Memory is **measured and reported, not gated** (D3, user ruling 2026-10-26): + L4 is whole-program, and the former "≤ 1.5× `-a 3`" budget is dropped. The table reports the + absolute peak and its ratio to `-a 3`. + +### Results at a glance (L4) + +| | 6 apps | Kubernetes (`kubernetes` + `k8s-api`) | +|---|---|---| +| `-a 4` complete, no crash | 6/6 | 2/2 | +| `-j 1` = `-j 16` = `-j 16`, byte-identical | 6/6 | 2/2 | +| wall `-a 3` → `-a 4`, `-j 1` | 12.9 → 45.4 s | 13.5 → 31.5 s / 5.6 → 11.0 s | +| peak RSS `-a 4 -j 1` (× `-a 3`) | 290 MB (1.08×) to 5.9 GB (1.14×); nats-server 4.1 GB (3.64×) | **7.7 GB (1.05×)** / 2.4 GB (1.15×) | +| `analysis.json` `-a 3` → `-a 4` | 263 MB → 2.66 GB (10.1×; nats-server alone 1.78 GB) | 343 MB → 1.34 GB (3.9×) / 193 → 360 MB (1.9×) | +| vertices | 2.35M | 1.15M / 242k | +| `param_in` / `param_out` | 1.16M / 0.70M | 449k / 380k / 91k / 88k | +| `summary` | 3.55M | 1.26M / 206k | +| port `ddg` (`reaching-defs`) / `points-to` `ddg` | 4.69M / 653k | 1.84M / 373k / 439k / 39k | +| stand-ins (externals with a body) | 4,575 | 1,677 / 129 | +| bound call sites with a VTA-only target | 2.6–6.6% | 11.3% (11,294) / 0.4% | +| consistency checks (L3 and L4) | all PASS | all PASS | +| degraded codes | `range_func` (5 apps), `unmapped_ssa` (4) | `range_func`, `unmapped_ssa` / none | + +**Key takeaways.** + +1. **The scale gate passes.** The Kubernetes root completes `-a 4` in 31.5 s at `-j 1` (30.0 s at + `-j 16`) with a whole-process peak of 7.7 GB, only 1.05× its `-a 3` peak, and its three + outputs are byte-identical. The `go/packages` load still dominates RSS. L4 adds 18 s of wall + time: SSA takes 1.0 s, VTA 2.3 s and the overlay 10.3 s, and the rest is emitting the 1.34 GB + file. +2. **Memory is not the problem L4 has; output size is.** RSS grows by 5–40% on every app but + one. nats-server is the outlier at 3.6× (4.1 GB). It has one giant recursive SCC whose + global footprint (A4) gives 91% of its 1.58M vertices a `:` `of`. The JSON grows + 4–10× on the hand-written apps and 22.7× on nats-server (78 MB → 1.78 GB). Most of those + bytes are `:` vertices, their port wiring, and param edges carrying full global ids. +3. **Summaries are near all-to-all, as A15 predicts.** 74–94% of call sites with summary edges + have every `actual_in` → `actual_out` pair. At statement granularity the host already links + every use to every def (the L3 call rule), so a summary contributes the callee's global and + by-ref footprint, not intra-caller precision. +4. **VTA matters on a small share of call sites, but they are the ones taint needs.** Of the call + sites that bind a target, 2.6–6.6% gain one that only VTA found on the six apps, and 11.3% on + the Kubernetes root: interface calls (`store.Driver` in memos, `Repo` in the done fixture), + function values, and the D7 interface-method binding on top. `k8s-api`, generated marshallers + with no interfaces, has 0.4%. +5. **The points-to delta is moderate and strictly additive.** It is 7–19% of the port `ddg` on + the six apps (nats-server 9.6%) and 20% / 9% on Kubernetes. The superset gate holds: every + `-a 3` edge is in the `-a 4` file unchanged. +6. **Global port wiring is the volume driver.** On nats-server, 2.87M of 3.09M port `ddg` edges + (93%) carry a global `var`. On listmonk, one function, `cmd.initHTTPHandlers` (a route table + that registers about 300 handlers), holds 475k of the app's 523k port edges (91%). It is the + host of hundreds of calls, each with the same global actuals, and the coarse host wiring + multiplies them: every host use feeds every `actual_in`. +7. **Degradation is reported, never silent.** Five of the six apps and the Kubernetes root use + range-over-func somewhere (`range_func`, D8). Four apps and the root carry `unmapped_ssa`, + which comes almost entirely from closures in package-level `var` initializers. No app hit + `no_ssa`, `no_vta`, `partial_ssa`, `partial_types` or `l3_skipped`. + +### Per-app results (L4) + +Times are seconds, RSS is MB (2^20 bytes), and JSON is MB (10^6 bytes). Every row is +go1.27.3, `--eager`, one timed run per configuration. + +| app | wall `-a 3 -j 1` | wall `-a 4 -j 1` | wall `-a 4 -j 16` | user CPU `-a 4 -j 1` | RSS `-a 3 -j 1` | RSS `-a 4 -j 1` | RSS × | JSON `-a 3` | JSON `-a 4` | JSON × | SSA | VTA | overlay | `-a 4` identical | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| cobra | 0.45 | 0.54 | 0.52 | 1.13 | 268 | 290 | 1.08 | 2.1 | 6.7 | 3.2 | 9 ms | 5 ms | 74 ms | yes | +| listmonk | 0.93 | 2.37 | 2.37 | 5.84 | 989 | 1,384 | 1.40 | 10.6 | 103.3 | 9.8 | 33 ms | 31 ms | 1.04 s | yes | +| memos | 2.47 | 4.68 | 4.43 | 13.62 | 2,501 | 3,153 | 1.26 | 43.2 | 174.5 | 4.0 | 127 ms | 148 ms | 1.27 s | yes | +| nats-server | 2.31 | 23.48 | 23.27 | 42.70 | 1,128 | 4,103 | **3.64** | 78.2 | 1,775.2 | **22.7** | 224 ms | 183 ms | 14.76 s | yes | +| openbao | 5.01 | 10.77 | 10.27 | 30.13 | 5,173 | 5,893 | 1.14 | 101.6 | 473.4 | 4.7 | 292 ms | 403 ms | 3.59 s | yes | +| dae (linux) | 1.71 | 3.53 | 3.23 | 8.65 | 1,385 | 1,743 | 1.26 | 27.5 | 130.7 | 4.8 | 125 ms | 69 ms | 1.04 s | yes | +| **kubernetes** | 13.49 | **31.53** | 30.01 | 68.56 | 7,315 | **7,697** | 1.05 | 343.2 | 1,341.5 | 3.9 | 971 ms | 2.30 s | 10.31 s | **yes** | +| k8s-api | 5.55 | 10.98 | 9.44 | 24.27 | 2,044 | 2,356 | 1.15 | 192.8 | 360.2 | 1.9 | 517 ms | 291 ms | 3.44 s | yes | + +`-j 16` barely helps at `-a 4`, by at most 1.5 s: SSA, VTA and the summary pass run serially by +design (D2, `ssa.BuildSerially`; the SCC pass is serial for determinism), and emission is one +stream. + +| app | vertices | `formal_in` / `formal_out` | `actual_in` / `actual_out` | `:` share of vertices | `param_in` | `param_out` | `summary` | sites with summary: all-to-all | port `ddg` | `points-to` `ddg` | stand-ins (formals) | +|---|---|---|---|---|---|---|---|---|---|---|---| +| cobra | 7,491 | 945 / 965 | 2,593 / 2,988 | 21.1% | 2,635 | 3,038 | 6,050 | 1,232: 74.1% | 10,790 | 5,637 | 91 (337) | +| listmonk | 58,546 | 3,698 / 3,721 | 26,432 / 24,695 | 46.6% | 28,270 | 26,653 | 40,912 | 6,105: 80.3% | 522,668 | 20,767 | 506 (2,021) | +| memos | 171,002 | 26,770 / 27,174 | 55,933 / 61,125 | 28.3% | 59,004 | 64,762 | 216,629 | 19,147: 86.6% | 237,601 | 68,956 | 830 (3,248) | +| nats-server | 1,584,949 | 324,205 / 155,426 | 734,728 / 370,590 | **91.2%** | 862,759 | 431,317 | 2,576,921 | 29,878: 78.9% | 3,093,971 | 295,776 | 667 (2,503) | +| openbao | 402,746 | 74,643 / 50,842 | 154,802 / 122,459 | 50.0% | 164,146 | 132,226 | 550,412 | 36,933: 82.0% | 605,515 | 195,288 | 1,639 (6,816) | +| dae | 120,882 | 18,889 / 17,257 | 42,266 / 42,470 | 49.6% | 43,611 | 43,881 | 155,698 | 12,163: 82.7% | 215,620 | 67,138 | 842 (3,298) | +| **kubernetes** | 1,154,746 | 229,049 / 197,197 | 394,069 / 334,431 | 40.9% | 448,676 | 380,030 | 1,260,761 | 97,185: 76.0% | 1,840,380 | 373,490 | 1,677 (6,447) | +| k8s-api | 242,414 | 27,556 / 35,489 | 91,428 / 87,941 | 7.2% | 91,428 | 87,941 | 205,643 | 40,856: 93.7% | 439,414 | 38,963 | 129 (812) | + +Notes: +- The `-v` log's "port edges" count includes the `points-to` edges. The table splits them: + port = `reaching-defs`, plus `points-to`. +- A stand-in is an external with a `body`. Every external a call site targets has one, except + for the two to five per app whose formals are empty (`runtime.Gosched`). + +| app | call nodes | sites that bind a target | with a VTA-only target | with ≥ 2 targets | `call_graph` VTA edges | VTA function set (library targets) | VTA edges | sites with targets (log) | non-call capture sites | degraded | +|---|---|---|---|---|---|---|---|---|---|---| +| cobra | 1,474 | 1,240 | 32 (2.6%) | 14 | 45 | 411 (83) | 1,294 | 1,244 | 37 | none | +| listmonk | 7,057 | 6,126 | 220 (3.6%) | 63 | 272 | 1,809 (640) | 7,304 | 6,194 | 47 | `range_func` (1 pkg, 1 callable) | +| memos | 23,781 | 19,209 | 997 (5.2%) | 516 | 1,059 | 7,893 (1,443) | 21,417 | 19,436 | 380 | `range_func` (7, 10); `unmapped_ssa` (2 fn / 4 calls) | +| nats-server | 38,335 | 30,007 | 1,633 (5.4%) | 898 | 2,133 | 10,473 (718) | 39,668 | 30,679 | 587 | `range_func` (2, 27); `unmapped_ssa` (25 / 4) | +| openbao | 51,991 | 37,139 | 2,461 (6.6%) | 1,802 | 4,389 | 19,978 (3,181) | 58,876 | 38,377 | 822 | `range_func` (5, 6); `unmapped_ssa` (10 / 12) | +| dae | 15,354 | 12,218 | 628 (5.1%) | 228 | 888 | 4,498 (1,286) | 14,104 | 12,598 | 263 | `range_func` (5, 5); `unmapped_ssa` (40 / 125) | +| **kubernetes** | 193,481 | 99,757 | 11,294 (11.3%) | 4,652 | 15,897 | 94,601 (41,107) | 249,493 | 102,407 | 3,890 | `range_func` (26, 49); `unmapped_ssa` (3,720 / 39,700) | +| k8s-api | 87,808 | 40,856 | 172 (0.4%) | 0 | 172 | 26,527 (8,570) | 52,526 | 40,856 | 175 | none | + +- **"Sites that bind a target"** are call nodes with at least one `param_in` from an `actual_in`. + A call node binds nothing when its callee has no formals (`runtime.Gosched`), when the CFG + pruned it, or when it is unresolved: a builtin, a conversion, or a function value VTA did not + resolve. +- **"VTA-only"** means at least one bound target is not the call node's static `callee`. +- **"Sites with targets (log)"** is the analyzer's own dispatch count. It also counts sites whose + only target has no formals. +- **Degraded:** the `range_func` figures are (packages, callables) and the `unmapped_ssa` + figures are (functions / calls), from the `-v` line. + +### What changed since the earlier measurements + +The task reports measured nats-server at several points on the branch; those runs were go1.26.4, +`-j 8`. On the eval binary, go1.27.3 `-j 1`: +- **summary** is 2,576,921, against 2,547,371 after the Task 17 alias round (`04244db`) (+1.2%); +- **points-to** is 295,776, against 290,499 then and 292,707 after round 2 (`18fca40`); +- **param_out** is 431,317, against 431k at Task 14. + +The growth since `18fca40` comes from: +- the reference-holding capture `formal_out` (`078616f`, MR5); +- the L3 host weak-def of every reference-holding capture (`cd37383`). + +Both add edges only. The L3 `ddg` on nats-server went from 345,318 (§ Per-app counts) to 358,979 +(+4.0%). That includes `cd37383`'s capture weak defs (chi +0.26%) and the D1 L3 fixes since the +L3 sections were measured. Wall time and RSS in those reports were measured for one stage, not +end to end, on another toolchain. Compare the numbers here only with each other. + +**`unmapped_ssa`, re-measured.** The Kubernetes root has 3,720 unmapped functions and 39,700 +unmapped calls, against 3,754 / 39,728 at Task 9. Classified from `-vv`: +- 3,717 are closures in package-level `var` initializers (`init$N`; Ginkgo + `var _ = Describe(…, func(){…})`; 3,555 of them under `test/`); +- 3 are a second `func init()` in one file (`init#N`). + +The "method declared above its receiver type" cause Task 9 found is gone (fixed in `5c57a1a`). +None of these is a join bug. The other apps have the same cause in smaller numbers (memos 2, +dae 40, openbao 10, nats-server 25 functions). + +### Cost of the memory levers + +`sdg.Run` lowers the process GC target to 50 while it runs (Task 15b), and L1–L3 and the +emission keep the default. Here is what that costs, measured end to end against a local build +identical except for `l4GCPercent = 100` (not committed). Both builds are go1.27.3, `-a 4 -j 1 +--eager`, two runs each, and every output is byte-identical to the eval's. + +| app | GC target in L4 | wall (s) | user CPU (s) | sys (s) | peak RSS (MB) | +|---|---|---|---|---|---| +| nats-server | 50 (shipped) | 23.69 / 23.56 | 43.26 / 43.19 | 2.19 / 2.29 | 3,951 / 4,157 | +| nats-server | 100 | 23.40 / 23.00 | 34.80 / 34.61 | 1.87 / 2.16 | 4,463 / 4,563 | +| kubernetes | 50 (shipped) | 31.66 / 31.74 | 67.36 / 67.53 | 12.60 / 16.49 | 7,713 / 7,742 | +| kubernetes | 100 | 31.55 / 30.82 | 59.70 / 59.92 | 12.78 / 12.63 | 8,254 / 8,518 | + +- **CPU.** GOGC=50 costs **+24.6% user CPU** on nats-server (+8.5 s) and **+12.8%** on Kubernetes + (+7.6 s). +- **Wall time.** It costs +0.4 s on nats-server and +0.5 s on Kubernetes, about 1–2%. The extra + GC work runs on idle cores of a 16-core host. **On a CPU-starved host the extra CPU becomes wall + time**, up to the user-CPU delta. +- **Memory.** It saves 9–12% of peak RSS on nats-server (about 450 MB) and 7–9% on Kubernetes + (about 650 MB). +- The other levers (per-SCC cache release, interned port names) have no CPU cost to report. + Task 15b measured them: nats-server's live heap after fill went from 1.81 to 1.45 GB. + +**Where the end-to-end peak sits.** With the levers on, the `-a 4` peak is no longer in `sdg.Run`. +It is in the emission, which streams the envelope but builds each module's subtree whole before +writing it. The largest L4 modules are: + +| app | module | JSON | +|---|---|---| +| nats-server | `server/server.go` | 340 MB | +| nats-server | `server/jetstream.go` | 164 MB | +| nats-server | `server/client.go` | 131 MB | +| kubernetes | `test/integration/dra/dra.go` | 66 MB | +| listmonk | `cmd/handlers.go` | 58 MB | + +The application-scope `param_in` and `param_out` lists stream (nats-server: 368 MB and 180 MB of +JSON). The next lever, if a larger app needs one, is to stream each callable within a module +(spec §4.13, §11). It is not needed at this corpus's scale: Kubernetes is 1.05× its `-a 3` peak. + +### Consistency at scale (L4) + +`scripts/l3_metrics.py` runs its L4 checks on every callable and every edge of the eight `-a 4` +files. It applies the L3 checks of § Internal-consistency checks to the L3 part of the body only, +with vertices and port and points-to edges removed, so the L3 figures compare across levels. +Every check passes on every app: + +| check | result | +|---|---| +| every `param_in` / `param_out` endpoint is a body node id (a callable's or a stand-in's) | 0 dangling, 8/8 apps | +| every `summary` endpoint is a vertex of its callable | 0 failures | +| no `summary` edge has `:` vertices at both ends (#30) | 0 | +| L3 checks over the L3 part of every `-a 4` body (one `@entry`/`@exit`, reachability, endpoints, `prov == ["ssa"]` on the L3 edges) | all PASS | +| `-a 3` ⊆ `-a 4` | gated on the fixtures (`TestGateL4_SupersetOfL3`); on cobra, listmonk, memos and dae the L3 `body` / `cfg` / `cdg` / `ssa`-`ddg` counts of the `-a 4` file equal the `-a 3` file's exactly (counts, not an edge-by-edge diff; the other four apps were not compared) | + +The Go-side gates do the arity and kind checks per edge on the fixtures +(`TestGateL4_ArityMatchesFormals`, `TestGateL4_NoDanglingEndpoints`, +`TestGateL4_VertexIdsAreGlobalOrdinals`). The HTTP → SQL "done" fixture +(`testdata/dataflow/httpsql`, `TestGateL4_HTTPToSQLPath`) does a BFS over the emitted JSON graph. +The handler's `r` reaches `database/sql.DB.Query`'s `query` formal through a VTA-bound `param_in` +into `sqlRepo.Find`, and a handler that never queries does not reach it. + +**Metamorphic relations.** `scripts/metamorphic.sh` (memos, the default) passes on this HEAD with +0 violations in all 14 relations. It covered 20–32 sites for MR1–MR6, MR14 and MR15, and 33M +reachability pairs for MR5 and MR6. The run took about 2 min. The last `--all` run (memos and +nats-server, 0 violations, 23 min, 16.2 GB) is from Task 17b on `e160b17` and `4a6c885`. + +### Insights + +- **The coarse host wiring is cheap where calls are sparse, quadratic where they are dense.** A + statement that hosts n calls, each with g global actuals, gets about n·g port edges in each + direction, and a body that hosts many such statements multiplies that again. listmonk's + `initHTTPHandlers` (one function, 91% of the app's port edges) and nats-server's + `client.parse` (162k) are the extremes. Python's reaching-def-matched host wiring (spec §4.7, + optional) would cut this volume without losing a flow; it is not built. +- **Globals are most of L4 on server-shaped code.** nats-server's `Server` methods form one large + recursive SCC. Every package variable any member reads or writes becomes a formal of every + member (A4, transitive), so 91% of its vertices are `:` vertices. The cut of global → + global summaries (#30) already removed 98M summary edges there (spec §4.9). The vertices and + their port edges remain, and they are what sources and sinks can name. +- **VTA is cheap.** Under 2.3 s on Kubernetes, with a function set of 94,601, of which 41,107 are + target-only library functions. The SSA build is about 1 s. The dominant L4 compute cost is the + overlay: flow graphs, summaries and port wiring, where `graphOf`/`bodyFlows` and GC dominate + (Task 18a profile). The dominant wall-time cost on the large files is emission. +- **k8s-api is the counter-example.** Generated marshallers have few interface calls (0.4% VTA), + few globals (7% of vertices) and leaf-heavy call graphs. L4 adds only 1.9× bytes there, the + smallest growth in the corpus, because its L3 `ddg` already dominates the file. + +### Limitations (L4) + +These are known, by design or measured, and each is a source of noise (extra edges) or of a gap +(no edge does not mean no flow). Under the anti-goal, every gap is either closed by +over-approximation or reported as degraded. + +- **D7 noise: every interface call also binds the interface method.** + - Its pass-through summary is all-to-all, so an interface call with concrete VTA targets still + carries the all-to-all edges of the interface stand-in. + - Over-approximation by design: a call whose VTA set is empty still keeps its flow. + - Interfaces with no symbol-table node (the universe `error.Error()`, interface literals) get + no interface-method target. They keep only their VTA targets, and with none the L3 host + pass-through applies (spec §4.8). A sink cannot be named as `error.Error`. +- **D8: range-over-func is not modelled.** `for v := range seq` routes the loop body through a + synthetic `yield`, so what the iterator passes to `yield` does not reach `v` + interprocedurally. Every program that uses it is degraded `range_func`: 5 of the 6 apps and + the Kubernetes root. The SDK then refuses to certify "no flow" over that graph (spec §4.15). + That is safe, but it means `exhausted` is never returned on most real apps until D8 is + modelled. +- **Library callbacks are not edges.** In `sort.Slice(xs, less)` and `http.HandleFunc(path, h)`, + the function value flows into the stand-in, but the library calling it back is not modelled. + Taint that *starts* in a handler (its own parameters) is unaffected. +- **A15: summaries are near all-to-all** (74–94% of sites, table above). Under the coarse host + wiring, a summary edge adds no reachability for the SDK's context-insensitive walk. It is a + shortcut hop, and a precondition for a future HRB two-phase walk (spec §11). +- **#30: the global → global summary cut, and the depth-bounded slice caveat.** + - Summary edges with both ends `:` are omitted [user, 2026-10-26]. Measured: 97.7% of + 101M summary edges on nats-server. + - Reachability is unchanged. But python-sdk's slices and taint walks are depth-bounded + (`DEFAULT_DEPTH = 5` for slices). A dropped shortcut makes the remaining route longer (into + the callee's body and back), so a depth-bounded slice may stop short of a node it would + otherwise reach. + - They return with the HRB walk (codeanalyzer-go#30). +- **Option A alias volume.** + - The one-level offset also applies to reference-compatible field types (`m := s.cache; + p := r.ptr`), not only to `&x.f` (Task 17 ruling). That is most of the `points-to` volume: + 217k of 290k on nats-server at Task 17. + - `MayHoldRef` also admits value structs that hold a reference, so `s1.n` → `s2.n` between two + distinct value copies is a spurious edge. + - Both are sound over-approximations: 7–20% of port `ddg` here. +- **One-level alias offsets only.** + - Not caught: offsets deeper than one step (`&x.f.g` against a `*G`), two prefixes of one base + that alias each other, and an interface holding a value struct whose reference field is + written through the interface (spec §4.10). + - These are false negatives of the type-based oracle that an Andersen points-to over `go/ssa` + would close (§11). They are the one class of L4 gap that is neither over-approximated nor + degraded. +- **L3 capture weak-def breadth (`cd37383`).** + - At L3, a closure's host weak-defines every reference-holding captured base, read-only + captures included. That is broader than needed: a read-only capture could weak-define + `x.*` without downgrading the base. + - Sound. It costs about 0.26% L3 `ddg` on chi. +- **The stored literal reached through an unresolved function value (Task 13).** + - A global written only inside a literal that is stored and later called through a function + value nothing resolves has no `param_out` from that write. Inside the encloser, the global's + `formal_out` is fed only by its `formal_in` (the identity edge). + - The flow is still reachable at the caller, whose host statement weak-defines the global. + - `TestGateL4_StoredLiteralGlobalReachable` pins both the reachable flow and the + intra-procedural gap, so the gap closing fails the test on purpose. +- **#29: cgo.** cgo-generated wrapper functions have no L1 callable, so SSA functions and calls + inside them are `unmapped_ssa`. A flow through C is not modelled (non-goal). The fixture test + `TestJoin_CgoNamesIssue29` skips when cgo is off. +- **#31: `base_types` / `interfaces`.** + - This predates L4 and is not an L4 change, but L4 consumers meet it. `base_types[]` holds raw + type strings, not type ids, so `GO_EMBEDS` never resolves, and `interfaces[]` is never + computed (`GO_SATISFIES` is empty). + - VTA does not need either; it reads `go/types` directly. +- **Memory and size are reported, not bounded** (D3). The largest file in the corpus is 1.78 GB + (nats-server). Loading it whole in Python, as `scripts/l3_metrics.py` does, needs about + 10–20 GB. The SDK's JSON backend loads it whole too. For apps that size the Neo4j backend is + the practical one. diff --git a/docs/design/notes/l3-eval-2026-10.md b/docs/design/notes/l3-eval-2026-10.md index 4bddc41..753c2db 100644 --- a/docs/design/notes/l3-eval-2026-10.md +++ b/docs/design/notes/l3-eval-2026-10.md @@ -1,4 +1,4 @@ -# L3 real-app evaluation (2026-10-24) +# L3 / L4 real-app evaluation (2026-10-24, L4 2026-10-27) Open-source Go apps at pinned commits, analyzed with `cango -a 3` via `scripts/eval_l3.sh`. Host: Apple M3 Max, 16 cores, 64 GB, macOS. Analyzer built with go1.27.3 (see Finding 2). Every @@ -10,6 +10,42 @@ and "external" split `call_graph` edges by whether `dst` is an `@external` id. L3 volume, edge/node-kind breakdowns, cost, output size, the protobuf-marshaller measurement and consistency checks over these outputs: [`l3-dataflow-metrics.md`](l3-dataflow-metrics.md). +## L4 (`-a 4`) evaluation (2026-10-27) + +One `CANGO_LEVELS="2 3 4" scripts/eval_l3.sh` pass over the same eight prepared trees, with the +binary the script builds from `feat/l4-sdg` at `c230592`: +- `runtime.Version()` go1.27.3; runs use `GOTOOLCHAIN=go1.27.3`; +- the host as above, j = 16, every run `--eager`. + +After the L3 sequence, each app runs `-a 4 -j 1 -v`, `-a 4 -j 16`, and a second `-a 4 -j 16`, and +the three `-a 4` files are compared byte for byte. + +**Gate: passed.** All eight apps complete `-a 4` with exit 0 and no crash, and all three `-a 4` +outputs are byte-identical on every app. The `-a 3` checks above still pass on the same run. +Memory is reported, not gated (D3). Peak RSS is for the whole process. + +| app | status | wall `-a 3 -j 1` (s) | wall `-a 4 -j 1` (s) | wall `-a 4 -j 16` (s) | RSS `-a 3` (MB) | RSS `-a 4` (MB) | × | `analysis.json` `-a 4` (MB) | `-a 4` identical | degraded | +|---|---|---|---|---|---|---|---|---|---|---| +| cobra `adbc881` | ok | 0.45 | 0.54 | 0.52 | 268 | 290 | 1.08 | 6.7 | yes | none | +| listmonk `8216c4f` | ok | 0.93 | 2.37 | 2.37 | 989 | 1,384 | 1.40 | 103.3 | yes | `range_func` | +| memos `28719a9` | ok | 2.47 | 4.68 | 4.43 | 2,501 | 3,153 | 1.26 | 174.5 | yes | `range_func`, `unmapped_ssa` | +| nats-server `f8f6db8` | ok | 2.31 | 23.48 | 23.27 | 1,128 | 4,103 | 3.64 | 1,775.2 | yes | `range_func`, `unmapped_ssa` | +| openbao `1d58653` | ok | 5.01 | 10.77 | 10.27 | 5,173 | 5,893 | 1.14 | 473.4 | yes | `range_func`, `unmapped_ssa` | +| dae `e3fee8f` (linux) | ok | 1.71 | 3.53 | 3.23 | 1,385 | 1,743 | 1.26 | 130.7 | yes | `range_func`, `unmapped_ssa` | +| **kubernetes** `f78e722` | ok | 13.49 | **31.53** | 30.01 | 7,315 | **7,697** | 1.05 | 1,341.5 | **yes** | `range_func`, `unmapped_ssa` | +| k8s-api `f78e722` | ok | 5.55 | 10.98 | 9.44 | 2,044 | 2,356 | 1.15 | 360.2 | yes | none | + +- **Kubernetes is the scale gate.** The root module completes `-a 4` in 31.5 s at a 7.7 GB peak + (1.05× `-a 3`), deterministically. +- **nats-server is the memory and size outlier** (3.6× RSS, 1.78 GB of JSON). Its one large + recursive `Server` SCC turns most vertices into `:` formals and actuals. +- The L4 counts, the GOGC=50 CPU cost (+12.8% to +24.6% user CPU, about 1–2% wall on this host), + where the peak sits, the consistency checks and the limitations are in + [`l3-dataflow-metrics.md` § L4](l3-dataflow-metrics.md#l4). +- **Findings.** The eval found no crash and no nondeterminism. One script fix was needed: + `eval_l3.sh` learned the `-a 4` triple and its byte check (`c230592`). `--all` still includes + cockroach, which is not prepared on this host, so the run names the eight apps. + ## Corpus change: Kubernetes replaces cockroach as the scale benchmark - **Why.** cockroach is now under the proprietary CockroachDB Software License, so it cannot be a diff --git a/docs/design/roadmap.md b/docs/design/roadmap.md index aa00b94..42143df 100644 --- a/docs/design/roadmap.md +++ b/docs/design/roadmap.md @@ -10,7 +10,7 @@ A schema **major**: move the Go analyzer off the v1 keystone booleans, `signature` as id) onto the v2 canonical schema (additive tree, `can://` ids, `span` with byte offsets, `source` per module, typed edge lists). The SDK moves in lockstep as a coordinated major release. **Reach for this train: L1 + L2 to parity.** -L3 shipped in release A (below); L4 remains deferred (see Not now). +L3 shipped in release A (below); L4 is implemented on `feat/l4-sdg` (release B, below). Golden rule carried from `schema-migration.md`: keep everything that *computes* facts (parser, resolver, call-graph builder); replace only what *serializes* them. @@ -51,7 +51,7 @@ C (edge shape) ─┴─► L2 emission (call_graph rename @ application sc │ A/B/C ─► D (Neo4j relabel) ─► SDK v2 model remap ─► pin SDK→analyzer (once BOTH cut) │ - L3 (release A) ─► L4 (deferred) ┘ reuse A & C unchanged + L3 (release A) ─► L4 (release B) ┘ reuse A & C unchanged ``` ## Release trains @@ -70,15 +70,33 @@ A/B/C ─► D (Neo4j relabel) ─► SDK v2 model remap ─► pin SDK→analyz found two issues, both fixed before release: L2 call-graph dedup on repeated `.init` signatures (nondeterministic, dropped edges) and the Neo4j `--eager` purge failing on Neo4j 5.x. The release tag is the user's decision. -- **Later (deferred).** L4 (SDG `param_in`/`param_out`/`summary`, needs a points-to oracle). - Reuses group A & C vocabulary; no new schema-major coordination beyond what Train 1 - establishes. +- **Release B — L4 (`-a 4`, SDG).** Implemented on `feat/l4-sdg` + (spec `docs/design/specs/v2-l3-l4-dataflow.md` Phase 2). It adds: + - VTA dispatch over `go/ssa`; + - formal/actual vertices, `param_in`/`param_out`, call-site `summary` edges and port-wiring + `ddg`; + - library stand-ins with `--library-models`; + - a type-based may-alias oracle (`points-to` `ddg`); + - the `degraded` envelope marker and `--strict`; + - the Neo4j `GO_PARAM_IN`/`GO_PARAM_OUT`/`GO_SUMMARY` overlay, with `--emit neo4j` at L4. + + It reuses the group A and C vocabulary; every L4 fact is additive (`-a 3` ⊆ `-a 4`). The + real-app evaluation is in `docs/design/notes/l3-dataflow-metrics.md` § L4 and + `l3-eval-2026-10.md`. Kubernetes `-a 4` completes deterministically, and memory is reported + per app, not gated (D3). Follow-ups: + - #30: global → global summaries for an HRB walk; + - #31: `GO_EMBEDS`/`GO_SATISFIES`; + - range-over-func (D8); + - library callbacks; + - Andersen points-to; + - export-data loading. + + The python-sdk Go L3/L4 surface (spec §7) is the next rung. ## Not now (considered, deliberately excluded) -- **L4 dataflow (SDG)** — deferred; needs a points-to oracle and interprocedural - construction. (L3 is in release A.) -- **Points-to oracle** — L4 prerequisite; parked with L4. +- **Andersen points-to over `go/ssa`** — a precision upgrade replacing L4's type-based may-alias + oracle behind the same interface (`prov` stays `points-to`); not this train. - **`--materialize-expressions` / `--materialize-basic-blocks`** — optional body node kinds; out of scope for the parity migration. - **New framework detection** — orthogonal enrichment axis diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index ffcb755..4a422d2 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -1,7 +1,9 @@ # Spec: Go L3/L4 dataflow (CFG/CDG/DDG + SDG) with SDK taint (codeanalyzer-go) -**Status:** Phases 0–1 shipped (PR #16); L4 design amended by the 2026-10-25 re-check (§4.16) · -**Date:** 2026-10-23, amended 2026-10-25 · **Repos:** `codeanalyzer-go`, `python-sdk` · +**Status:** Phases 0–1 implemented (PR #16, open, not yet merged); Phase 2 (L4) implemented on +`feat/l4-sdg` and evaluated (`docs/design/notes/l3-dataflow-metrics.md` § L4); L4 design amended by +the 2026-10-25 re-check (§4.16) and the 2026-10-26/27 rulings · +**Date:** 2026-10-23, amended 2026-10-25, 2026-10-27 · **Repos:** `codeanalyzer-go`, `python-sdk` · **Supersedes:** the "L3/L4 deferred" entries in `docs/design/roadmap.md`. ## 1. Intent @@ -69,24 +71,25 @@ siblings is required too; and the id grammar has to change for polyglot apps any - **Golden rule** (from the v2 migration): keep everything that computes facts; add L3/L4 as new computation; the v2 emitter stays the single serialization point. - **Substrate:** `golang.org/x/tools v0.52.0` (`go.mod`, `go 1.26.0`) is already a dependency — - `go/cfg`, `go/ssa`, `go/ssa/ssautil`, `go/callgraph/vta`, `go/callgraph/cha` are available with - no new module. VTA's package doc still says "experimental" and carries a TODO on generic + `go/cfg`, `go/ssa`, `go/ssa/ssautil` and `go/callgraph/vta` are available with no new module. + VTA is seeded with no initial call graph (its internal lazy CHA, §4.8); `go/callgraph/cha` is + not used. VTA's package doc still says "experimental" and carries a TODO on generic instantiation wrappers, so every VTA call runs under `recover` (§4.8, §4.15). `x/tools/go/pointer` is deprecated: **do not adopt**. - Level flag semantics (`cli-contract.md`): `-a 3` implies `-a 2`; `-a 4` implies `-a 3`; an `-a 3` run must not pay L4 cost; `--emit neo4j` runs at max implemented depth (now 4) and rejects `-a` and `--graph-field-depth`. - python-sdk rules: the Iron Rule (no public accessor changes name/signature/return type); new - capability only as new methods; models are `extra="ignore"` mirrors of v2 (PR #434's choice, - matching TS/Java since python-sdk #386 — an additive analyzer release parses without an SDK - edit; §7). + capability only as new methods; models are `extra="ignore"` mirrors of v2 (PR #434's choice; + the TypeScript and Java wire models are `extra="ignore"` since python-sdk #386, per their + module docstrings on `main` — an additive analyzer release parses without an SDK edit; §7). ## 3. Current state (what exists) As of 2026-10-25 (code at `3504445`, branch `feat/l3-dataflow` = `feat/l4-sdg`): -- **Phases 0 and 1 shipped together in PR #16** (§9). L1/L2/L3 on v2, Neo4j projection live through - L3. +- **Phases 0 and 1 are implemented together on PR #16** (§9; open, not yet merged). L1/L2/L3 on + v2, Neo4j projection live through L3. - **Phase 0 (live):** `can:///go/…` ids (§4.1, `internal/schema/v2/ids.go`); L2 library calls with `application.external_symbols` (§4.14); co-positioned call keys `l:c/n`; `is_deferred`; `typeKind` emits `alias`/`defined`; `GO_DDG` `_k = var|prov`; the per-package call-site index @@ -105,8 +108,8 @@ As of 2026-10-25 (code at `3504445`, branch `feat/l3-dataflow` = `feat/l4-sdg`): no `--strict`; `--emit neo4j` forces `LevelDataflow` (L3); `v2.Analysis` has no `degraded` and the package doc says L4 fields are not declared; `ExternalSymbol` has no receiver or body; `GO_PARAM_IN`/`GO_PARAM_OUT`/`GO_SUMMARY` stay `Reserved`. -- **python-sdk:** Go v2 L1/L2 support is on PR #434 (branch `pr-434-v3`, unmerged); the SDK work - builds on it (§7). It cannot yet parse `-a 3` output (§7 prerequisites). `add-go-support` is +- **python-sdk:** Go v2 L1/L2 support is on PR #434 (branch `pr-434-v3`; merged into python-sdk + `main` on 2026-10-26, after this section was written); the SDK work builds on it (§7). It cannot yet parse `-a 3` output (§7 prerequisites). `add-go-support` is obsolete. ## 4. Decisions @@ -483,7 +486,7 @@ Go specifics: `isPkgVar`); without that they would be unnameable as sources. Constants and functions are never globals. This filter is sound under the over-approximation rule; `:` volume is measured in the eval (§8) and a size budget set from the first measurement. -- **Range-over-func (amended, A21; D8) [user].** `for v := range seq` has no `CallExpr` (no L1 call +- **Range-over-func (amended, A21; D8) [call].** `for v := range seq` has no `CallExpr` (no L1 call node), and SSA routes the loop body through a synthetic yield function, so data the iterator passes to `yield` (a `rows.Scan` result inside a DB iterator) never reaches `v` interprocedurally. **Not modelled this train:** a program that uses range-over-func gets degraded @@ -501,7 +504,8 @@ Go specifics: statement def-use): `@formal_in:k → first-use statements of that base` (replaces the `@entry` def for that base for port purposes — keep both, the `@entry` edge stays for L3 monotonicity); `return stmt → - @formal_out[:i]` (`var` ``/name), statement writing a by-ref param/global → + @formal_out[:i]` (`var` = the formal's `of`, `` / `:` — never the user's + result name, D6), statement writing a by-ref param/global → corresponding `@formal_out`; `host stmt → /actual_in:i` (`var` = arg spelling or `argN` if not an access path); `/actual_out[:i] → host stmt` (`var` = the LHS base it defines). **This wiring is what the SDK's `_ports_carry_dependence` probe checks; without it taint certifies false @@ -529,9 +533,12 @@ Must-have flow (3) needs concrete targets for interface and function-value calls built per D2 (amended, A12): `ssautil.Packages` over the **same** `go/packages` load, not `AllPackages`. -- **L2 refinement:** `call_graph` gains VTA-derived edges for interface/func-value call sites, `prov` - `["vta"]`, merged with existing `["go/types"]` edges via `MergeEdges` (union prov, sum weight). - This is additive (monotone). Body `callee` stays a single id (the static callee — the interface +- **L2 refinement (amended, Task 10):** `call_graph` gains VTA-derived edges for interface/func-value + call sites, `prov` `["vta"]`, weight 1 — **only** for a `(src, dst)` pair the L2 resolver did not + already find. An edge L2 found that VTA also finds is left exactly as L2 emitted it (its `prov` + is not unioned, its weight not summed): the `-a 3` `call_graph` is a byte-for-byte subset of the + `-a 4` one (the monotonicity gate). `Result.CallEdges` holds only the VTA-only pairs, so + `Result.ApplyTo`'s `MergeEdges` appends them and never unions into an L2 edge. Body `callee` stays a single id (the static callee — the interface method for an interface call) — the schema has one slot. Multiple concrete targets are carried by `call_graph` and, at L4, by one `param_in`/`param_out` set **per target** from the same `actual_in` vertices (the actual vertices are per call site, not per target). @@ -543,8 +550,15 @@ built per D2 (amended, A12): `ssautil.Packages` over the **same** `go/packages` `param_in`/`param_out` go to each concrete target **and** to the interface method; the call-site `summary` edges are the union over targets. **D7:** an interface call *always* binds to the interface method (pass-through) and every VTA target, even when every target is in-project — - per the anti-goal. Cost: every interface call (`io.Writer`, `error.Error()`) carries an - all-to-all summary. + per the anti-goal. Cost: every interface call (`io.Writer`) carries an all-to-all summary. +- **Interfaces with no symbol-table node (ruling, Task 10) [call].** `error.Error()` (the universe + `error`), methods of interface literals (`interface{ M() }`), and methods promoted through an + embedded interface whose origin is declared outside the symbol table have no callable id and no + external id, so such a call gets **no** interface-method target. It keeps its VTA targets; + when VTA finds none the call is unresolved, and the L3 host pass-through (§4.7: the call + statement uses its receiver and arguments and defines its LHS) keeps the intra-caller flow. No + false refutation; but a sink cannot be named as `error.Error`. A stand-in for the universe + `error.Error` is a follow-up (§11). - **VTA-only externals (amended, A19).** A VTA target outside the project that the `go/types` resolver never saw (e.g. `(*bytes.Buffer).Write` reached through `io.Writer`) is **added** to `external_symbols` at `-a 4`, with its stand-in body (§6). Monotone: `-a 3` ⊆ `-a 4`. @@ -617,6 +631,11 @@ Context-insensitive. Per callable compute: global → global summary adds no reachability for an HRB-unaware walk. Measured on nats-server: 97.7% of 101M summary edges were global → global; the cut took `sdg.Run`'s peak RSS from 11.1 GB to ~4.4 GB (2.34M summary edges left). They return with the HRB two-phase walk (§11). + Tracked as **codeanalyzer-go#30**. Caveat: the SDK's slices and taint walks are + **depth-bounded** (python-sdk `DEFAULT_DEPTH = 5` for slices); a dropped global → global + shortcut makes the remaining route longer (into the callee's body and back), so a depth-bounded + slice may stop short of a node it would have reached through the shortcut. Reachability with an + unbounded depth is unchanged. - **Unanalysed callables (amended, A16).** A callable whose L3 result was dropped (a contained panic or a call-count mismatch, `run.go`) has no `ddg`, so taint entering through `param_in` would stop at its `formal_in`. Such callables get formals and a **pass-through summary** (like an @@ -776,8 +795,14 @@ summaries and roughly 1.5–2.5× the body nodes (the ports). the process GC target to 50 while it runs (`debug.SetGCPercent`, restored on return, never raised above the caller's), so L1–L3 and emission keep the default. None changes a fact: the result is deeply equal with them off (`TestSDG_OutputUnchangedByMemoryLevers`). +- **Where the `-a 4` peak sits (measured, Task 16/18).** With the levers on, the end-to-end + peak is no longer in `sdg.Run` but in the emission: the streaming writer still builds one + module's subtree whole before writing it, and an L4 module of a large file is big + (nats-server's `server/server.go` module is 344 MB of JSON). The next lever, if one is needed, + is to stream per callable within a module; measured numbers per app are in + `l3-dataflow-metrics.md` § L4. -### 4.14 L2 external calls (Phase 0, shipped in PR #16) [user: fix it; details are calls] +### 4.14 L2 external calls (Phase 0, implemented on PR #16) [user: fix it; details are calls] Before Phase 0 the L2 resolver resolved library callees (`internal/semantic_analysis/call_graph.go`) and then dropped them: no `call_graph` edge, and the emitter left the call node's `callee` `null`. @@ -844,6 +869,11 @@ callables) but marks the envelope: returned. Same rule for `paths_between`-style refutations. Implemented in the Go backend (the diagnostic goes into the `blocked` ledger the walk already returns, which `taint_verdict` honours — shared `commons/` unchanged). +- **Go leaves (all additive, none in the shared keystone yet):** the `degraded` envelope field + itself; the entry's `callables[]` (`can://` ids, sorted; carried by `l3_skipped` and + `range_func`); the codes beyond the original four (`partial_types`, `unmapped_ssa`, + `l3_skipped`, `range_func`); and, in the vertex vocabulary it degrades over, the `_recv` `of` + for an unnamed or `_` receiver (D6) and `:` (§4.6). - Go-only leaf now; candidate for the keystone alongside the cross-language taint revisit (§11). ### 4.16 Decisions of the 2026-10-25 re-check @@ -867,7 +897,7 @@ amendments are folded into the sections above. Decisions: `:`; globals `:::` (§4.6). - **D7 [call]:** an interface call always binds to the interface method (pass-through summary) **and** to every VTA target (§4.8). -- **D8 [user]:** range-over-func is not modelled; a program that uses it gets degraded code +- **D8 [call]:** range-over-func is not modelled; a program that uses it gets degraded code `range_func`, and a fixture pins the gap (§4.6, §4.15). - **D9 [user]:** L4 is whole-program; `-a 1..3 -t` unchanged; `-a 4 -t` (JSON) is an error; `--emit neo4j -t` projects at L3 with a one-line stderr warning; exact texts and the full rule in @@ -959,7 +989,7 @@ branch `pr-434-v3`, unmerged as of 2026-10-25), fixing it first against the v2-o - `GoBodyNode` lacks `id`/`parent`/`of`/`is_deferred`; there is no `external_symbols`/ `param_in`/`summary`/`degraded`, and no `_ports_carry_dependence`. Add the L3 fields first (so `-a 3` output parses), then L4. - - `extra` policy is reconciled to **`ignore`** (§2 previously said `forbid`). + - `extra` policy is **`ignore`** (§2). - **Id splitting (A22).** External ids contain `/@external/` before the vertex `@`, so the Go backend must **never split a body-node id on its first `@`**; strip the known callable id prefix instead. @@ -1072,17 +1102,21 @@ the six apps and both Kubernetes modules complete `-a 4` without crash and deter (`-j 1`/`-j N` and two runs on Kubernetes, not just fixtures); memory is reported per app as absolute `-a 4` peak RSS and its ratio to `-a 3`, not gated (D3, §4.13); no hard time budget, but timings are reported and a regression budget is set after the first measurement. Results go in `l3-dataflow-metrics.md` § L4. +**Measured 2026-10-27 (go1.27.3, `c230592`):** the gate passed on all eight. Every app completes +`-a 4` and its `-j 1` / `-j 16` / rerun outputs are byte-identical. The Kubernetes root takes +31.5 s at a 7.7 GB peak (1.05× `-a 3`). nats-server is the outlier at 3.6× RSS and 1.78 GB of +JSON. The full results, insights and limitations are in `l3-dataflow-metrics.md` § L4. ## 9. Phasing and release | Phase | Repo | Content | Release | -> **Phases 0 and 1 ship together in one PR / one release** [user, 2026-10-23]; they stay separate rows for traceability. **Both have shipped, in PR #16.** +> **Phases 0 and 1 ship together in one PR / one release** [user, 2026-10-23]; they stay separate rows for traceability. **Both are implemented, on PR #16 (open, not yet merged).** |---|---|---|---| -| 0 | analyzer | **shipped (PR #16).** id grammar `can:///go/…`; **L2 external calls (§4.14)**; L1 co-positioned call keys; `is_deferred`; `typeKind` alias/defined; `GO_DDG` `_k`; call-site position index (perf) | analyzer next **breaking** version (wheel is at 0.1.0 with no local tags — builder confirms the released version and bumps the major, or the minor if still 0.x) | -| 1 | analyzer | **shipped (PR #16).** L3: body statements, CFG (+defer chain, panic), CDG, syntactic DDG, CLI `-a 3`/`--graph-field-depth`, Neo4j CFG/CDG/DDG, gates | minor | -| 2 | analyzer | **Prerequisites, in order (A24):** (1) #26 shared call/closure walker (incl. `//line` spans), first — returns `(*ast.CallExpr, key)`, `(*ast.FuncLit, closure name)` and call → host maps, replacing the index-paired L1/L3 walks; (2) #17 closure call resolution, after #26 — closures carry a declaring `Path` and `call_graph` gains edges from closures; (3) the #22 L3 fixes (D1, §4.5), then the #28 satellite-`parent` tests and a Go 1.23+ fixture — before L4 port wiring (changing L3 after L4 would re-baseline every L4 gate); (4) #27 streaming JSON and AST release — before the real-app L4 eval, can run in parallel with the L4 core; (5) schema and CLI scaffolding (A14, A18, A19, A17 declarations, a `-a 1..4` monotonicity harness) — the first L4 task. Dependencies: #26 → #17 → (VTA, summaries); #22 fixes → port wiring and gates; scaffolding → all of L4; #27 → the Kubernetes eval. **Then L4:** SSA spike (D3 budget) and build (D2), SSA → L3 join, VTA, `degraded` marker (§4.15), external stand-ins (`external_symbols` `receiver`/`body`), vertices, param edges, summaries, port wiring, type-based alias delta, Neo4j L4 overlay, `--emit neo4j` at L4 (per D4), real-app eval | minor | +| 0 | analyzer | **implemented (PR #16, open).** id grammar `can:///go/…`; **L2 external calls (§4.14)**; L1 co-positioned call keys; `is_deferred`; `typeKind` alias/defined; `GO_DDG` `_k`; call-site position index (perf) | analyzer next **breaking** version (wheel is at 0.1.0 with no local tags — builder confirms the released version and bumps the major, or the minor if still 0.x) | +| 1 | analyzer | **implemented (PR #16, open).** L3: body statements, CFG (+defer chain, panic), CDG, syntactic DDG, CLI `-a 3`/`--graph-field-depth`, Neo4j CFG/CDG/DDG, gates | minor | +| 2 | analyzer | **Prerequisites, in order (A24):** (1) #26 shared call/closure walker (incl. `//line` spans), first — returns `(*ast.CallExpr, key)`, `(*ast.FuncLit, closure name)` and call → host maps, replacing the index-paired L1/L3 walks; (2) #17 closure call resolution, after #26 — closures carry a declaring `Path` and `call_graph` gains edges from closures; (3) the #22 L3 fixes (D1, §4.5), then the #28 satellite-`parent` tests and a Go 1.23+ fixture — before L4 port wiring (changing L3 after L4 would re-baseline every L4 gate); (4) #27 streaming JSON and AST release — before the real-app L4 eval, can run in parallel with the L4 core; (5) schema and CLI scaffolding (A14, A18, A19, A17 declarations, a `-a 1..4` monotonicity harness) — the first L4 task. Dependencies: #26 → #17 → (VTA, summaries); #22 fixes → port wiring and gates; scaffolding → all of L4; #27 → the Kubernetes eval. **Then L4:** SSA spike (D3 measurement) and build (D2), SSA → L3 join, VTA, `degraded` marker (§4.15), external stand-ins (`external_symbols` `receiver`/`body`), vertices, param edges, summaries, port wiring, type-based alias delta, Neo4j L4 overlay, `--emit neo4j` at L4 (per D4), real-app eval | minor | | 3 | python-sdk | Go v2 models (first the §7 prerequisites: `GoType.kind`, L3 fields), facade, local + Neo4j backends, L1–L4 accessors, taint e2e | minor; pins the analyzer only after Phase 2 is released | Tracking: `codellm-devkit/.github` parent issue + one sub-issue per PR (Phase 0+1 as one) (filed just-in-time), @@ -1107,7 +1141,15 @@ dispatch + type-based may-alias). adopted. - Andersen points-to over `go/ssa` replacing the type-based oracle (`prov` unchanged). - Context-sensitive (HRB two-phase) taint traversal in the SDK. With it, reinstate the global → - global `summary` edges omitted at L4 (§4.9, [user, 2026-10-26]). + global `summary` edges omitted at L4 (§4.9, [user, 2026-10-26]; codeanalyzer-go#30 — also + the depth-bounded-slice caveat there). +- A stand-in for the universe `error.Error` (and interface literals), so an interface call on + them gets a D7 target (§4.8, Task 10 ruling). +- Stream per callable within a module during emission, if the emission-held module peak + (§4.13) matters on a larger app. +- `GO_EMBEDS` / `GO_SATISFIES` from `base_types[]` / `interfaces[]`: `base_types` holds raw type + strings, not type ids, and `interfaces[]` is never computed (codeanalyzer-go#31; pre-existing, + not an L4 change). - Library callback modelling (`sort.Slice`, `http.HandleFunc`). - Promote `callable.source_file` to the keystone (existing deferral). - Struct-tag modelling (existing deferral). diff --git a/docs/design/specs/v2-schema-structure.md b/docs/design/specs/v2-schema-structure.md index bbe6283..425bcc1 100644 --- a/docs/design/specs/v2-schema-structure.md +++ b/docs/design/specs/v2-schema-structure.md @@ -12,12 +12,13 @@ the JSON tags from that file. ├─ schema_version # "2.0.0" ├─ language # "go" ├─ max_level # 1 (symbol table) | 2 (+ call graph) | 3 (+ cfg/cdg/ddg) -│ # | 4 (+ SDG; L4, Phase 2) +│ # | 4 (+ SDG: vertices, param edges, summaries) ├─ k_limit # L3+ only: access-path bound (--graph-field-depth, default 3) -├─ degraded[] # L4, Phase 2; optional: absent when nothing degraded +├─ degraded[] # L4; optional: absent when nothing degraded │ └─ { code, detail?, packages[]?, callables[]? } │ # code: no_ssa | no_vta | partial_ssa | partial_types | -│ # unmapped_ssa | l3_skipped | range_func (open vocabulary) +│ # unmapped_ssa | l3_skipped | range_func (open vocabulary); +│ # callables[] = can:// ids (l3_skipped, range_func) ├─ analyzer │ ├─ name # "codeanalyzer-go" │ └─ version @@ -50,11 +51,12 @@ the JSON tags from that file. │ └─ │ ├─ src # can:// callable id │ ├─ dst # can:// callable id, or an @external id - │ ├─ prov[] # optional: e.g. ["go/types"]; L4 adds ["vta"] (merged) + │ ├─ prov[] # optional: e.g. ["go/types"]; L4 adds VTA-only edges, prov + │ │ # ["vta"], weight 1 (an L2 edge is never changed) │ └─ weight # optional - ├─ param_in[] # L4, Phase 2; optional: { src, dst, var } — global ids, + ├─ param_in[] # L4; optional: { src, dst, var } — global ids, │ # actual_in → formal_in; var = the callee formal's `of` - ├─ param_out[] # L4, Phase 2; optional: { src, dst, var } — global ids, + ├─ param_out[] # L4; optional: { src, dst, var } — global ids, │ # formal_out → actual_out; sorted (src, dst) └─ external_symbols # L2+; optional: { "<@external id>": external } └─ # library callee: `can:///@external//` @@ -67,9 +69,9 @@ the JSON tags from that file. ├─ parameters[] # { name, type, is_variadic? } (no span: not in-project); │ # receiver excluded; unnamed params are "_" ├─ return_type # optional - ├─ receiver # L4, Phase 2; optional: { name, type } of a method's + ├─ receiver # L4; optional: { name, type } of a method's │ # receiver (gives formal_in:0's `of`; "_recv" if unnamed) - └─ body # L4, Phase 2; optional: stand-in formal vertices only, + └─ body # L4; optional: stand-in formal vertices only, # { "@formal_in:" | "@formal_out[:]": vertex } ``` @@ -96,6 +98,8 @@ The same shape whether it sits under a type's `callables` (methods), a module's ├─ metrics # optional: { cyclomatic: N } ├─ callables # optional: nested closures (same shape) ├─ body # { "": body node }; key = "[/n]" | "@entry" | "@exit" +│ # | L4: "@formal_in:" | "@formal_out[:]" +│ # | "/actual_in:" | "/actual_out[:]" │ ├─ # L1+ │ │ ├─ kind # "call" │ │ ├─ id # L1+: @ @@ -108,7 +112,7 @@ The same shape whether it sits under a type's `callables` (methods), a module's │ │ ├─ kind # statement | branch | loop | switch | return | entry | exit │ │ ├─ id # @ (@entry → @entry) │ │ └─ span { start, end, bytes } # the statement; absent on entry / exit -│ └─ # L4, Phase 2: SDG port, span-less +│ └─ # L4: SDG port, span-less │ ├─ kind # formal_in | formal_out | actual_in | actual_out │ ├─ id # @ │ ├─ of # L4: the bound name — param / receiver name, "_", @@ -120,8 +124,9 @@ The same shape whether it sits under a type's `callables` (methods), a module's ├─ cdg[] # L3+, optional: { src, dst } ├─ ddg[] # L3+, optional: { src, dst, var, prov } (prov = ["ssa"]; │ # L4 adds ["reaching-defs"] port wiring, ["points-to"]) -└─ summary[] # L4, Phase 2; optional: { src, dst } — local keys, - # /actual_in: → /actual_out[:] +└─ summary[] # L4; optional: { src, dst } — local keys, + # /actual_in: → /actual_out[:]; + # no global → global edge (codeanalyzer-go#30) ``` ### L3 body nodes and edges @@ -147,13 +152,13 @@ The same shape whether it sits under a type's `callables` (methods), a module's - **`ddg`** comes from syntactic reaching definitions over k-limited access paths (`base(.field|[*])*`, past `k` → `.*`). `var` is the source spelling of the path, and `prov` is always `["ssa"]` at L3. Sorted by `(src, dst, var)`. It is **not - refutation-complete**: writes to captured variables inside closures and writes through - reference-typed fields of value-typed struct arguments/receivers are not modelled at L3 - (no edge ≠ no flow). The 2026-10-25 re-check decided to close both at L3, plus the deferred-operand - reassignment miss (D1, Phase 2 prerequisite). A deferred call's receiver/argument paths are keyed - on the variable at exit, so a reassignment after the `defer` can add a spurious edge, and (until - the D1 fix lands) kill an earlier def. See `v2-l3-l4-dataflow.md` §4.5 "L3 precision". -- **L4 vertex keys (L4, Phase 2).** Formals `@formal_in:`, `@formal_out` / `@formal_out:`; + refutation-complete** (no edge ≠ no flow): it is alias-free (two bases never interact; L4's + `points-to` edges add that). The D1 gaps of the 2026-10-25 re-check — closure capture writes, + writes through reference fields of value structs, and deferred-operand writes — are closed at + L3 (#22). A deferred call's receiver/argument paths are keyed on the variable at exit, so a + reassignment after the `defer` can add a spurious edge. See `v2-l3-l4-dataflow.md` §4.5 + "L3 precision". +- **L4 vertex keys (L4).** Formals `@formal_in:`, `@formal_out` / `@formal_out:`; actuals `/actual_in:`, `/actual_out` / `…/actual_out:`. `formal_in` and `actual_in` are always indexed; `formal_out`/`actual_out` are bare iff the group has exactly one vertex. The SDK reads `of`, never an index into `parameters[]` (`formal_in:0` is a method's @@ -188,7 +193,7 @@ span, and every body key derived from one, still indexes `module.source`. - **L3+ only:** `k_limit`; body node `parent`; non-call body nodes; a callable's `cfg` / `cdg` / `ddg`. These are omitted for a callable with no body. Each is a pure addition: the `-a 2` document is a subset of the `-a 3` one. -- **L4 only (Phase 2):** `degraded` (and only when something degraded), `formal_*` / `actual_*` +- **L4 only:** `degraded` (and only when something degraded), `formal_*` / `actual_*` vertices and their `of`, a callable's `summary`, `param_in` / `param_out`, `external_symbols[].receiver` / `.body`, `ddg` edges with `prov` `reaching-defs` / `points-to`, `call_graph` `prov` `vta`. Bodiless interface methods and stand-ins get formals too. Each is a diff --git a/internal/sdg/vta.go b/internal/sdg/vta.go index d9480b7..6fdbb26 100644 --- a/internal/sdg/vta.go +++ b/internal/sdg/vta.go @@ -282,9 +282,11 @@ func (d *dispatch) result(deg []Degraded) Result { } // ApplyTo merges the L4 call-graph additions into app (spec §4.8): the VTA -// edges into app.CallGraph (semantic_analysis.MergeEdges: union prov, sum -// weight per edge identity) and the VTA-only externals into app.Externals, so +// edges into app.CallGraph and the VTA-only externals into app.Externals, so // every new edge's target resolves. Both are additions: -a 3 ⊆ -a 4. +// CallEdges holds only pairs the L2 resolver did not find (result skips +// every d.l2 edge), so MergeEdges never meets an existing identity and +// appends: an L2 edge VTA also finds keeps its L2 prov and weight. func (r *Result) ApplyTo(app *schema.GoApplication) { if len(r.CallEdges) > 0 { app.CallGraph = semantic_analysis.MergeEdges(app.CallGraph, r.CallEdges) From 69d1204f7c9ac3e70f30949480aec00e8eb9ddb3 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 16:29:31 -0400 Subject: [PATCH 44/65] docs: L4 overlay live in the Neo4j spec; measured memory; eval keeps $USER --- README.md | 7 +++- .../issues/child-m1-schema-contract.md | 4 +- .../issues/parent-neo4j-projection.md | 3 +- docs/design/Neo4j_projection/roadmap.md | 7 ++-- .../specs/neo4j-projection.md | 8 ++-- docs/design/notes/l3-dataflow-metrics.md | 37 +++++++++++-------- docs/design/specs/v2-l3-l4-dataflow.md | 8 ++-- scripts/eval_l3.sh | 8 ++-- scripts/eval_l3_test.sh | 11 ++++++ 9 files changed, 59 insertions(+), 34 deletions(-) diff --git a/README.md b/README.md index 0eca0eb..a4e4615 100644 --- a/README.md +++ b/README.md @@ -654,8 +654,11 @@ check the following. VTA `call_graph` edges, external `receiver` / `body`, `degraded`). Code that assumes every body node has a `span`, or that every `ddg` edge has `prov: ["ssa"]`, must handle the L4 kinds. `-a 4 -t` is an error (whole-program; D9). `--library-models` is new and valid only at L4. - `-a 4` takes several times the memory and output of `-a 3` (measured per app in - `docs/design/notes/l3-dataflow-metrics.md` § L4). + **Measured cost** (2026-10-27, analyzer built with go1.27.3, eight real apps). On 7 of 8 + apps, the peak RSS of `-a 4` is 1.05–1.4× that of `-a 3`; Kubernetes is 1.05×, at 7.7 GB. + nats-server is the outlier at 3.6×, because its globals fan out through one recursive 822-callable + SCC. The JSON output is 1.9–23× that of `-a 3`. Per-app figures are in + `docs/design/notes/l3-dataflow-metrics.md` § L4. - **Neo4j:** - `--emit neo4j` projects **L4** (D4), adding `GO_CFG_NEXT`, `GO_CDG` and `GO_DDG`, and the SDG rows `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY`, vertex `:GoBodyNode`s with `var`, and diff --git a/docs/design/Neo4j_projection/issues/child-m1-schema-contract.md b/docs/design/Neo4j_projection/issues/child-m1-schema-contract.md index 3837bd5..c095bc4 100644 --- a/docs/design/Neo4j_projection/issues/child-m1-schema-contract.md +++ b/docs/design/Neo4j_projection/issues/child-m1-schema-contract.md @@ -28,7 +28,9 @@ JSON hot path) that is the declarative contract: `GO_HAS_METHOD`/`GO_HAS_FIELD`/`GO_HAS_BODY_NODE`), `GO_CALLS`/`GO_RESOLVES_TO`, `GO_EMBEDS`/`GO_SATISFIES`, `GO_IMPORTS`, the neutral repo edges, and `GO_PROVIDES`/ `GO_UNRESOLVED_IMPORT`. -- [ ] The **reserved L3/L4 overlay** vocabulary declared now (zero rows until the levels land): +- [ ] The **reserved L3/L4 overlay** vocabulary declared now (zero rows until the levels land; + *status 2026-10-27: both overlays are now live, `--emit neo4j` projects L4 and nothing is + `Reserved`*): `GO_HAS_BODY_NODE`/`GO_CFG_NEXT`(`_k=kind`)/`GO_CDG`/`GO_DDG`(`_k=var,prov`)/`GO_PARAM_IN`/ `GO_PARAM_OUT`/`GO_SUMMARY` + body-node statement kinds. - [ ] `_k` relationship discriminants on `GO_CFG_NEXT`, `GO_DDG`, `DECLARES_DEPENDENCY`. diff --git a/docs/design/Neo4j_projection/issues/parent-neo4j-projection.md b/docs/design/Neo4j_projection/issues/parent-neo4j-projection.md index d301b06..5613c48 100644 --- a/docs/design/Neo4j_projection/issues/parent-neo4j-projection.md +++ b/docs/design/Neo4j_projection/issues/parent-neo4j-projection.md @@ -43,7 +43,8 @@ Children (filed just-in-time on their repos, attached as sub-issues of this one) `N="Go"`/`P="GO_"`; one `:GoType` + `kind` property; `GO_EMBEDS` vs `GO_SATISFIES`; flattened `source_file`/`error_channel`/`is_goroutine`/`is_deferred`; narrower label set (no decorator/attribute/variable); neutral repo layer reused verbatim; full L3/L4 overlay -vocabulary declared now, zero rows until the levels land; type-centric SDK facade. +vocabulary declared now, zero rows until the levels land (*status 2026-10-27: the L3 and L4 +overlays are now live; `--emit neo4j` projects L4*); type-centric SDK facade. **Describe alternatives you've considered** diff --git a/docs/design/Neo4j_projection/roadmap.md b/docs/design/Neo4j_projection/roadmap.md index 3a209a0..740e1b9 100644 --- a/docs/design/Neo4j_projection/roadmap.md +++ b/docs/design/Neo4j_projection/roadmap.md @@ -192,9 +192,10 @@ and read-only Neo4j backends. - **`taint_flows` / slicing / reachability / source-sink** — the analyzer is a pure graph **provider**; labeled traversal is the SDK's job as Cypher queries, a later train. No analyzer surface for it. -- **L3/L4 overlay population** — the vocabulary is declared (decision L1) but emits zero rows - until the deferred L3/L4 analysis train lands. When it does, `project.go` fills the reserved - sections with **no `schema.go` change**. +- ~~**L3/L4 overlay population**~~ — done: the L3 train and then the L4 train (`feat/l4-sdg`) + filled the declared overlay in `project.go` with no contract change; `--emit neo4j` projects + at L4 and nothing in `schema.go` is `Reserved`. (The milestone text above — "zero rows at + L2" — describes the M1–M3 scope as built then.) - **Go-specific `cfg` edge kinds** (`defer_resume`, `select`) — recorded when L3 is designed. - **Struct-tag modeling** — Go has struct tags; v2 doesn't model them yet. Not forced in here. - **TS-SDK `GoAnalysis` encoding** — the facade vocabulary is designed once (M7 Python diff --git a/docs/design/Neo4j_projection/specs/neo4j-projection.md b/docs/design/Neo4j_projection/specs/neo4j-projection.md index b8e840a..bd1f350 100644 --- a/docs/design/Neo4j_projection/specs/neo4j-projection.md +++ b/docs/design/Neo4j_projection/specs/neo4j-projection.md @@ -80,7 +80,7 @@ or a verbatim reuse of shared vocabulary. "Ref" names the sibling precedent. | **CG1** | `call_graph` → **`GO_CALLS`** (weight, prov); `error_channel[]` → **string-array prop** on `:GoCallable` | `GO_CALLS` parity with `PY_CALLS` (endpoints = callable id or `:GoExternal` ghost; no dangling). Go errors are returned values, not thrown types — a value-shaped fact rides the node, not a `GO_RAISES` edge. | | **I1** | `GO_IMPORTS` (aggregated per target) → `:GoModule` (in-project) or `:GoExternal` ghost (stdlib/external, keyed by import path) | Parity with aggregated `PY_IMPORTS` + edge-only-when-resolved. Go imports are overwhelmingly external → ghost targets, never dangling, never minted as `:Package` (packages come from dependencies, not imports). | | **A1** | Declare neutral `:Artifact`/`:Package`/`:ConfigKey` + `HAS_ARTIFACT`/`DECLARES_DEPENDENCY`(`_k=kind`)/`LOCKS`/`DEFINES_CONFIG` **verbatim**; project only facts the analyzer produces | These are the cross-language **merge targets** — reused unchanged so a sibling over the same repo converges. go.mod deps → `:Package` `pkg:golang/…`; absent facts → **zero rows** (declared contract, empty projection). `GO_PROVIDES`/`GO_UNRESOLVED_IMPORT` stay `GO_`-prefixed (this analyzer's claims). | -| **L1** | Declare the **full L3/L4 overlay vocabulary now**; projector emits **zero rows** until the levels land | `GO_HAS_BODY_NODE`, `GO_CFG_NEXT`(`_k=kind`), `GO_CDG`, `GO_DDG`(`_k=var,prov`), `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY` + body-node statement kinds. `--emit neo4j` = "full-depth = max implemented" (**L3 today**: `GO_CFG_NEXT`, `GO_CDG`, `GO_DDG` and the L3 body nodes are live; only `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY` remain reserved for L4). The contract is declared once; L3 filled it with **no schema change**, and L4 will too. | +| **L1** | Declare the **full L3/L4 overlay vocabulary up front**; the projector emitted **zero rows** for a level until it landed | `GO_HAS_BODY_NODE`, `GO_CFG_NEXT`(`_k=kind`), `GO_CDG`, `GO_DDG`(`_k=var,prov`), `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY` + body-node statement kinds. `--emit neo4j` = "full-depth = max implemented" (**L4**, D4): the L3 overlay (`GO_CFG_NEXT`, `GO_CDG`, `GO_DDG`, L3 body nodes) and the L4 overlay (`GO_PARAM_IN {var}`, `GO_PARAM_OUT {var}`, `GO_SUMMARY`, vertex body nodes with `var`, stand-in vertices, `degraded_json`) are all live; nothing is `Reserved` (`schema.go`). The contract was declared once; L3 and L4 each filled it with **no schema change**. | | **K1** | `_k` discriminants: `GO_CFG_NEXT` per `kind`; `GO_DDG` per `(var,prov)`; `DECLARES_DEPENDENCY` per `kind` | Same mechanism as the siblings — parallel edges survive MERGE instead of collapsing. | | **V1** | `SCHEMA_VERSION = "2.0.0"`, stamped on `:GoApplication`; additive-MINOR suspended for the 2.0.0 line | Parity with the sibling versioning ruling; consumers gate on the analyzer-version floor. | @@ -109,7 +109,7 @@ Every `can://`-keyed node also carries `:GoCanNode`. - **Imports:** `GO_IMPORTS` (aggregated) - **Repository (neutral):** `HAS_ARTIFACT`, `DECLARES_DEPENDENCY` (`_k`), `LOCKS`, `DEFINES_CONFIG`; (`GO_`) `GO_PROVIDES`, `GO_UNRESOLVED_IMPORT` - **L3 overlay (live):** `GO_CFG_NEXT` (`kind`, `_k=kind`), `GO_CDG`, `GO_DDG` (`var`, `prov`, `_k=var|prov`) -- **Reserved L4 overlay (zero rows until L4 lands):** `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY` +- **L4 overlay (live):** `GO_PARAM_IN` (`var`), `GO_PARAM_OUT` (`var`), `GO_SUMMARY`; port and points-to `ddg` ride `GO_DDG`. One row per JSON `param_in[]` / `param_out[]` / `summary[]` entry; a lazy push upserts every L4 row (D5). ### CLI surface (parity with TS/Python) @@ -185,8 +185,8 @@ lockstep."** Concretely: - **Analyzer** ships the `internal/neo4j/` projection as part of the v2 line. `--emit neo4j` is independently landable behind the existing flag stubs; it produces a graph at - full-depth = L3 today (L3 landed additively, with no contract change) and grows additively - again when L4 arrives (no contract change — L1 reserve). + full-depth = L4, the max implemented level (L3 and then L4 landed additively, each with no + contract change — the L1 reserve). - **SDK** (`GoAnalysisBackend`) pins the analyzer version **only after** the analyzer v2 release is cut — same ordering constraint as the JSON wiring (the SDK's Go models can't parse v2 until then, and there is no Go wiring yet, so this is net-new, not a migration). diff --git a/docs/design/notes/l3-dataflow-metrics.md b/docs/design/notes/l3-dataflow-metrics.md index 9441430..d0302c3 100644 --- a/docs/design/notes/l3-dataflow-metrics.md +++ b/docs/design/notes/l3-dataflow-metrics.md @@ -809,8 +809,9 @@ dispatch targets to `call_graph` (`prov:["vta"]`), library callees get stand-in - **When and where:** measured 2026-10-27 on the same host, an Apple M3 Max (16 cores, 64 GB, macOS 15). The analyzer was built by `scripts/eval_l3.sh` with **go1.27.3** (`runtime.Version()`), and every run used `GOTOOLCHAIN=go1.27.3`. Every number in this section is go1.27.3 unless it - says otherwise. Absolute RSS differs by about 1 GB between go1.26.4 and go1.27.3 on nats-server - (Task 15b), so do not compare figures across toolchains. + says otherwise: figures quoted from earlier task reports carry their toolchain in parentheses. + Absolute RSS differs by about 1 GB between go1.26.4 and go1.27.3 on nats-server (Task 15b), so + do not compare figures across toolchains. - **Binary:** `feat/l4-sdg` at `c230592`. The `+dirty` marker covers uncommitted docs only; the analyzer code is that commit's. It includes the Task 17 alias rounds (`04244db`, `18fca40`), the capture fixes (`078616f`, `cd37383`) and the Task 18a fix-ups. @@ -938,23 +939,25 @@ Notes: ### What changed since the earlier measurements -The task reports measured nats-server at several points on the branch; those runs were go1.26.4, -`-j 8`. On the eval binary, go1.27.3 `-j 1`: -- **summary** is 2,576,921, against 2,547,371 after the Task 17 alias round (`04244db`) (+1.2%); -- **points-to** is 295,776, against 290,499 then and 292,707 after round 2 (`18fca40`); -- **param_out** is 431,317, against 431k at Task 14. +The task reports measured nats-server at several points on the branch, at `-j 8`, on the +toolchain given with each figure. On the eval binary, go1.27.3 `-j 1`: +- **summary** is 2,576,921, against 2,547,371 after the Task 17 alias round (`04244db`; + go1.26.4) (+1.2%); +- **points-to** is 295,776, against 290,499 then and 292,707 after round 2 (`18fca40`; both + go1.26.4); +- **param_out** is 431,317, against 431k at Task 14 (go1.27.3). The growth since `18fca40` comes from: - the reference-holding capture `formal_out` (`078616f`, MR5); - the L3 host weak-def of every reference-holding capture (`cd37383`). Both add edges only. The L3 `ddg` on nats-server went from 345,318 (§ Per-app counts) to 358,979 -(+4.0%). That includes `cd37383`'s capture weak defs (chi +0.26%) and the D1 L3 fixes since the +(+4.0%). That includes `cd37383`'s capture weak defs (chi +0.26%, go1.26.4) and the D1 L3 fixes since the L3 sections were measured. Wall time and RSS in those reports were measured for one stage, not end to end, on another toolchain. Compare the numbers here only with each other. **`unmapped_ssa`, re-measured.** The Kubernetes root has 3,720 unmapped functions and 39,700 -unmapped calls, against 3,754 / 39,728 at Task 9. Classified from `-vv`: +unmapped calls, against 3,754 / 39,728 at Task 9 (go1.27.3). Classified from `-vv`: - 3,717 are closures in package-level `var` initializers (`init$N`; Ginkgo `var _ = Describe(…, func(){…})`; 3,555 of them under `test/`); - 3 are a second `func init()` in one file (`init#N`). @@ -985,7 +988,8 @@ identical except for `l4GCPercent = 100` (not committed). Both builds are go1.27 - **Memory.** It saves 9–12% of peak RSS on nats-server (about 450 MB) and 7–9% on Kubernetes (about 650 MB). - The other levers (per-SCC cache release, interned port names) have no CPU cost to report. - Task 15b measured them: nats-server's live heap after fill went from 1.81 to 1.45 GB. + Task 15b measured them: nats-server's live heap after fill went from 1.81 to 1.45 GB + (go1.26.4). **Where the end-to-end peak sits.** With the levers on, the `-a 4` peak is no longer in `sdg.Run`. It is in the emission, which streams the envelope but builds each module's subtree whole before @@ -1028,7 +1032,8 @@ into `sqlRepo.Find`, and a handler that never queries does not reach it. **Metamorphic relations.** `scripts/metamorphic.sh` (memos, the default) passes on this HEAD with 0 violations in all 14 relations. It covered 20–32 sites for MR1–MR6, MR14 and MR15, and 33M reachability pairs for MR5 and MR6. The run took about 2 min. The last `--all` run (memos and -nats-server, 0 violations, 23 min, 16.2 GB) is from Task 17b on `e160b17` and `4a6c885`. +nats-server, 0 violations, 23 min, 16.2 GB; go1.27.3, the script's `EVAL_GO`) is from Task 17b +on `e160b17` and `4a6c885`. ### Insights @@ -1041,12 +1046,12 @@ nats-server, 0 violations, 23 min, 16.2 GB) is from Task 17b on `e160b17` and `4 - **Globals are most of L4 on server-shaped code.** nats-server's `Server` methods form one large recursive SCC. Every package variable any member reads or writes becomes a formal of every member (A4, transitive), so 91% of its vertices are `:` vertices. The cut of global → - global summaries (#30) already removed 98M summary edges there (spec §4.9). The vertices and + global summaries (#30) already removed 98M summary edges there (Task 15, go1.27.3; spec §4.9). The vertices and their port edges remain, and they are what sources and sinks can name. - **VTA is cheap.** Under 2.3 s on Kubernetes, with a function set of 94,601, of which 41,107 are target-only library functions. The SSA build is about 1 s. The dominant L4 compute cost is the overlay: flow graphs, summaries and port wiring, where `graphOf`/`bodyFlows` and GC dominate - (Task 18a profile). The dominant wall-time cost on the large files is emission. + (Task 18a profile, go1.26.4). The dominant wall-time cost on the large files is emission. - **k8s-api is the counter-example.** Generated marshallers have few interface calls (0.4% VTA), few globals (7% of vertices) and leaf-heavy call graphs. L4 adds only 1.9× bytes there, the smallest growth in the corpus, because its L3 `ddg` already dominates the file. @@ -1078,7 +1083,7 @@ over-approximation or reported as degraded. shortcut hop, and a precondition for a future HRB two-phase walk (spec §11). - **#30: the global → global summary cut, and the depth-bounded slice caveat.** - Summary edges with both ends `:` are omitted [user, 2026-10-26]. Measured: 97.7% of - 101M summary edges on nats-server. + 101M summary edges on nats-server (Task 14, go1.27.3). - Reachability is unchanged. But python-sdk's slices and taint walks are depth-bounded (`DEFAULT_DEPTH = 5` for slices). A dropped shortcut makes the remaining route longer (into the callee's body and back), so a depth-bounded slice may stop short of a node it would @@ -1087,7 +1092,7 @@ over-approximation or reported as degraded. - **Option A alias volume.** - The one-level offset also applies to reference-compatible field types (`m := s.cache; p := r.ptr`), not only to `&x.f` (Task 17 ruling). That is most of the `points-to` volume: - 217k of 290k on nats-server at Task 17. + 217k of 290k on nats-server at Task 17 (go1.26.4). - `MayHoldRef` also admits value structs that hold a reference, so `s1.n` → `s2.n` between two distinct value copies is a spurious edge. - Both are sound over-approximations: 7–20% of port `ddg` here. @@ -1102,7 +1107,7 @@ over-approximation or reported as degraded. - At L3, a closure's host weak-defines every reference-holding captured base, read-only captures included. That is broader than needed: a read-only capture could weak-define `x.*` without downgrading the base. - - Sound. It costs about 0.26% L3 `ddg` on chi. + - Sound. It costs about 0.26% L3 `ddg` on chi (go1.26.4). - **The stored literal reached through an unresolved function value (Task 13).** - A global written only inside a literal that is stored and later called through a function value nothing resolves has no `param_out` from that write. Inside the encloser, the global's diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index 4a422d2..0c19649 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -104,7 +104,8 @@ As of 2026-10-25 (code at `3504445`, branch `feat/l3-dataflow` = `feat/l4-sdg`): - **L2 dispatch** is still `go/types`: an interface-method call's callee is the bodiless interface method (`collectInterfaceMethods`); no concrete targets. Insufficient for must-have flow (3); L4 adds VTA (§4.8). -- **Not yet built (Phase 2):** `-a 4` is rejected by the CLI; no `LevelSDG` in `internal/options`; +- **Not yet built (Phase 2)** — *superseded: Phase 2 is now implemented; see the Status header. + Kept as the 2026-10-25 snapshot.* `-a 4` is rejected by the CLI; no `LevelSDG` in `internal/options`; no `--strict`; `--emit neo4j` forces `LevelDataflow` (L3); `v2.Analysis` has no `degraded` and the package doc says L4 fields are not declared; `ExternalSymbol` has no receiver or body; `GO_PARAM_IN`/`GO_PARAM_OUT`/`GO_SUMMARY` stay `Reserved`. @@ -726,7 +727,7 @@ Fill the reserved overlay, no contract renames: `body` vertices as `:GoBodyNode` with `GO_HAS_BODY_NODE` from the `:GoExternal`. - **`degraded_json`** (amended, A17) — a new `:GoApplication` property carrying the envelope's `degraded[]` (§4.15). -- Flip `Reserved:false`; conformance test must see every declared relationship emitted on the +- Flip `Reserved:false` (done: nothing in `schema.go` is `Reserved`); conformance test must see every declared relationship emitted on the fixture; cross-projection gate: Neo4j counts == JSON counts modulo `HAS_*`. - `--emit neo4j` now runs `-a 4` (D4, §4.12). - **Lazy Bolt push (amended, A17; D5) [call].** The lazy push diffs per module by `content_hash`, @@ -798,7 +799,8 @@ summaries and roughly 1.5–2.5× the body nodes (the ports). - **Where the `-a 4` peak sits (measured, Task 16/18).** With the levers on, the end-to-end peak is no longer in `sdg.Run` but in the emission: the streaming writer still builds one module's subtree whole before writing it, and an L4 module of a large file is big - (nats-server's `server/server.go` module is 344 MB of JSON). The next lever, if one is needed, + (nats-server's `server/server.go` module is 340 MB of JSON, measured in the 2026-10-27 eval, + `l3-dataflow-metrics.md` § L4). The next lever, if one is needed, is to stream per callable within a module; measured numbers per app are in `l3-dataflow-metrics.md` § L4. diff --git a/scripts/eval_l3.sh b/scripts/eval_l3.sh index 84a9ee2..0065c84 100755 --- a/scripts/eval_l3.sh +++ b/scripts/eval_l3.sh @@ -337,7 +337,7 @@ JQ_COUNTS=' ([$a.call_graph[]|select(.dst|contains("/@external/")|not)]|length), ([$a.call_graph[]|select(.dst|contains("/@external/"))]|length) ] | @tsv' -# run_one NAME TAG LEVEL JOBS [FLAG...] -> sets RC WALL RSS; output in $EVAL_OUT/NAME-TAG/analysis.json +# run_one NAME TAG LEVEL JOBS [FLAG...] -> sets RC WALL UCPU (user CPU) RSS BYTES; output in $EVAL_OUT/NAME-TAG/analysis.json run_one() { local name=$1 tag=$2 level=$3 jobs=$4 shift 4 @@ -360,11 +360,11 @@ run_one() { RC=$? if [ -e "$dir.timeout" ]; then RC=124; fi WALL=$(awk '/ real /{print $1}' "$dir.time") - USER=$(awk '/ real /{print $3}' "$dir.time") + UCPU=$(awk '/ real /{print $3}' "$dir.time") RSS=$(awk '/maximum resident set size/{printf "%d", $1/1048576}' "$dir.time") BYTES=$(wc -c <"$dir/analysis.json" 2>/dev/null | tr -d ' ') [ -n "$WALL" ] || WALL=n/a - [ -n "$USER" ] || USER=n/a + [ -n "$UCPU" ] || UCPU=n/a [ -n "$RSS" ] || RSS=n/a [ -n "$BYTES" ] || BYTES=n/a } @@ -442,7 +442,7 @@ for T in $TARGETS; do if [ "$RUN_L4" -eq 1 ]; then W4=n/a; WN4=n/a; U4=n/a; R4=n/a; B4=n/a; RATIO=n/a; RC4=0; RCN4=0; RCN42=0; ident4=n/a if [ "$RC3" -ne 124 ] && [ "$RC2" -ne 124 ]; then - run_one "$name" a4j1 4 1 -v; W4=$WALL; U4=$USER; R4=$RSS; B4=$BYTES; RC4=$RC + run_one "$name" a4j1 4 1 -v; W4=$WALL; U4=$UCPU; R4=$RSS; B4=$BYTES; RC4=$RC if [ "$RC4" -ne 124 ]; then run_one "$name" a4jN 4 "$JOBS"; WN4=$WALL; RCN4=$RC; fi if [ "$RC4" -ne 124 ] && [ "$RCN4" -ne 124 ]; then run_one "$name" a4jN2 4 "$JOBS"; RCN42=$RC; fi # One status for the triple: the first non-zero rc (124 = timeout). diff --git a/scripts/eval_l3_test.sh b/scripts/eval_l3_test.sh index 2f35ade..3f9107c 100755 --- a/scripts/eval_l3_test.sh +++ b/scripts/eval_l3_test.sh @@ -16,6 +16,8 @@ while [ $# -gt 0 ]; do shift done [ -n "$out" ] && echo '{"application":{"symbol_table":{},"call_graph":[]}}' >"$out/analysis.json" +# Record the environment's USER each run sees (eval_l3.sh must not clobber it). +[ -n "$out" ] && printf '%s\n' "${USER-}" >>"${USER_LOG:-/dev/null}" # NONDET_LEVEL: a run at that level appends a per-run nonce (a nondeterministic analyzer). [ "$level" = "${NONDET_LEVEL:-}" ] && echo "$$-$RANDOM" >>"$out/analysis.json" # At -a 4 with -v, print the L4 log lines eval_l3.sh parses. @@ -50,4 +52,13 @@ check 1 "FAIL(a4 -j differs)" CANGO_LEVELS="2 3 4" NONDET_LEVEL=4 check 1 "FAIL(-j differs)" CANGO_LEVELS="2 3" NONDET_LEVEL=3 check 1 "FAIL(rc a4=3)" CANGO_LEVELS="2 3 4" FAIL_LEVEL=4 check 1 "FAIL(rc a2=0 j1=3" CANGO_LEVELS="2 3" FAIL_LEVEL=3 +# $USER reaches every analyzer run unchanged (a script variable named USER would shadow it). +: >"$T/users" +env CANGO="$T/cango" EVAL_OUT="$T/out" EVAL_NOTE="$T/note.md" EVAL_JOBS=2 CANGO_LEVELS="2 3 4" \ + USER=evaluser USER_LOG="$T/users" "$S" "$T/app" >/dev/null 2>&1 +if [ "$(sort -u "$T/users")" = evaluser ] && [ "$(wc -l <"$T/users" | tr -d ' ')" -ge 8 ]; then + echo "ok: USER unchanged across all runs" +else + echo "FAIL: USER seen by the runs: $(sort -u "$T/users" | paste -sd, -)"; fail=1 +fi exit $fail From c9d1c0075c84185d566720c6a22787417b5889a5 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 16:29:31 -0400 Subject: [PATCH 45/65] test(l4): done gate checks a no-flow pair inside the DB plumbing --- .../v2emit/gate_l4_done_test.go | 17 +++++++++++++++++ testdata/dataflow/httpsql/httpsql.go | 3 ++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/internal/syntactic_analysis/v2emit/gate_l4_done_test.go b/internal/syntactic_analysis/v2emit/gate_l4_done_test.go index 4d9066f..6711b0b 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_done_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_done_test.go @@ -158,4 +158,21 @@ func TestGateL4_HTTPToSQLPath(t *testing.T) { if p := g.path(vertex(fn("Health"), "formal_in", "r"), sink); p != nil { t.Errorf("Health's r reaches %s: %v", sink, p) } + + // A no-flow pair inside the DB plumbing, so the gate catches + // over-connection and not only an uncalled callable: SanitizedHandler's + // request reaches the sink through SanitizedFind, but never Find's id. + // That fails if the repo.SanitizedFind site bound Find, if param edges + // were cross-wired between sites, or if a capture or summary leaked from + // one handler into the other. It depends on Routes registering Handler + // first: Handler(repo) has a by-ref out of the interface-typed repo, so + // the coarse host wiring carries Handler's flows forward into the later + // SanitizedHandler(repo), and not the other way round. + sanR := vertex(closure("SanitizedHandler"), "formal_in", "r") + if g.path(sanR, sink) == nil { + t.Errorf("SanitizedHandler's r does not reach %s (the pair below would be vacuous)", sink) + } + if p := g.path(sanR, findID); p != nil { + t.Errorf("SanitizedHandler's r reaches %s: %v", findID, p) + } } diff --git a/testdata/dataflow/httpsql/httpsql.go b/testdata/dataflow/httpsql/httpsql.go index 5235618..6efdac1 100644 --- a/testdata/dataflow/httpsql/httpsql.go +++ b/testdata/dataflow/httpsql/httpsql.go @@ -68,7 +68,8 @@ func Health(w http.ResponseWriter, r *http.Request) { // Routes wires the handlers to a database-backed Repo: without it no // concrete type reaches the handlers' repo, and VTA binds repo.Find to -// nothing but the interface method. +// nothing but the interface method. Keep Handler registered before +// SanitizedHandler: TestGateL4_HTTPToSQLPath's no-flow pair relies on it. func Routes(mux *http.ServeMux, db *sql.DB) { repo := NewRepo(db) mux.Handle("/user", Handler(repo)) From 6c7a0119ce4bc1711251e4753463cd1ec5959123 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 16:54:31 -0400 Subject: [PATCH 46/65] test(l4): done gate pins the sanitizer as a variable cut With strconv/Atoi: [] the Atoi summary edges go and the flow stays; a cut on n within SanitizedFind is bypassed by err (cdg, return) and the receiver (coarse host wiring), all correct dependence (codellm-devkit/.github#103). The no-flow pair moves to a separate entry, Audit, that shares no host variable. --- .../v2emit/gate_l4_done_test.go | 210 ++++++++++++++++-- testdata/dataflow/httpsql/httpsql.go | 21 +- 2 files changed, 207 insertions(+), 24 deletions(-) diff --git a/internal/syntactic_analysis/v2emit/gate_l4_done_test.go b/internal/syntactic_analysis/v2emit/gate_l4_done_test.go index 6711b0b..d3b4319 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_done_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_done_test.go @@ -3,14 +3,23 @@ package v2emit // The L4 "done" gate (spec §8): on the HTTP → SQL fixture // testdata/dataflow/httpsql, the emitted JSON graph carries a flow from an // HTTP handler's request parameter to database/sql's DB.Query query -// argument through an interface call, and a handler that never queries -// reaches nothing there. +// argument through an interface call, and unrelated pairs are unreachable. +// Sanitizers are SDK taint semantics (.github#103); the second test pins +// what L4 gives a sanitizer to work with. import ( + "path/filepath" + "runtime" "slices" + "strings" "testing" + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" ) // sdgGraph is the emitted JSON's dependence graph over global body node ids: @@ -24,28 +33,29 @@ type sdgGraph struct { type sdgHop struct { to, family string + v string // the edge's var (ddg, param_in, param_out); "" for cdg and summary } func newSDGGraph(a *v2.Analysis) *sdgGraph { g := &sdgGraph{succ: map[string][]sdgHop{}} - add := func(src, dst, family string) { g.succ[src] = append(g.succ[src], sdgHop{dst, family}) } + add := func(src, dst, family, v string) { g.succ[src] = append(g.succ[src], sdgHop{dst, family, v}) } for _, c := range l4Callables(a) { id := func(k string) string { return v2.GlobalOrdinal(c.ID, k) } for _, e := range c.Ddg { - add(id(e.Src), id(e.Dst), "ddg") + add(id(e.Src), id(e.Dst), "ddg", e.Var) } for _, e := range c.Cdg { - add(id(e.Src), id(e.Dst), "cdg") + add(id(e.Src), id(e.Dst), "cdg", "") } for _, e := range c.Summary { - add(id(e.Src), id(e.Dst), "summary") + add(id(e.Src), id(e.Dst), "summary", "") } } for _, e := range a.Application.ParamIn { - add(e.Src, e.Dst, "param_in") + add(e.Src, e.Dst, "param_in", e.Var) } for _, e := range a.Application.ParamOut { - add(e.Src, e.Dst, "param_out") + add(e.Src, e.Dst, "param_out", e.Var) } return g } @@ -54,6 +64,19 @@ func newSDGGraph(a *v2.Analysis) *sdgGraph { // dst is unreachable). Successors are visited in edge order, so the path is // deterministic. func (g *sdgGraph) path(src, dst string) []sdgHop { + return g.pathExcept(src, dst, nil) +} + +// underCallable is python-sdk's under_callable for one id: the callable +// itself, a body node of it (q@…), or a node of a callable nested in it +// (q/…). +func underCallable(id, q string) bool { + return id == q || strings.HasPrefix(id, q+"@") || strings.HasPrefix(id, q+"/") +} + +// pathExcept is path over the graph without the hops skip rejects (skip +// gets the hop's source node and the hop); a nil skip keeps every hop. +func (g *sdgGraph) pathExcept(src, dst string, skip func(from string, h sdgHop) bool) []sdgHop { type pred struct { from string hop sdgHop @@ -72,6 +95,9 @@ func (g *sdgGraph) path(src, dst string) []sdgHop { return out } for _, h := range g.succ[n] { + if skip != nil && skip(n, h) { + continue + } if _, seen := prev[h.to]; !seen { prev[h.to] = pred{n, h} q = append(q, h.to) @@ -160,19 +186,159 @@ func TestGateL4_HTTPToSQLPath(t *testing.T) { } // A no-flow pair inside the DB plumbing, so the gate catches - // over-connection and not only an uncalled callable: SanitizedHandler's - // request reaches the sink through SanitizedFind, but never Find's id. - // That fails if the repo.SanitizedFind site bound Find, if param edges - // were cross-wired between sites, or if a capture or summary leaked from - // one handler into the other. It depends on Routes registering Handler - // first: Handler(repo) has a by-ref out of the interface-typed repo, so - // the coarse host wiring carries Handler's flows forward into the later - // SanitizedHandler(repo), and not the other way round. - sanR := vertex(closure("SanitizedHandler"), "formal_in", "r") - if g.path(sanR, sink) == nil { - t.Errorf("SanitizedHandler's r does not reach %s (the pair below would be vacuous)", sink) - } - if p := g.path(sanR, findID); p != nil { - t.Errorf("SanitizedHandler's r reaches %s: %v", findID, p) + // over-connection and not only an uncalled callable. Audit is an entry + // point of its own: it builds its own Repo inline and calls only Count, so + // it shares no host variable and no in-project callee with the handlers. + // Its user reaches the sink (through Count), but never Find's id, and + // Handler's r never reaches Count's user. Either fails if an interface + // site bound the wrong method or param edges were cross-wired between + // sites. Reordering Routes' registrations or Audit's statements, or moving + // Audit in the file, leaves both pairs unreachable (checked by mutation). + count := method("sqlRepo", "Count") + auditUser := vertex(fn("Audit"), "formal_in", "user") + if g.path(auditUser, sink) == nil { + t.Errorf("Audit's user does not reach %s (the pairs below would be vacuous)", sink) + } + if p := g.path(auditUser, findID); p != nil { + t.Errorf("Audit's user reaches %s: %v", findID, p) + } + if p := g.path(src, vertex(count, "formal_in", "user")); p != nil { + t.Errorf("Handler's r reaches %s's user: %v", count, p) + } +} + +// buildHTTPSQLWithModels is buildL4 over testdata/dataflow/httpsql with +// --library-models models, uncached. +func buildHTTPSQLWithModels(t *testing.T, models sdg.Models) *v2.Analysis { + t.Helper() + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata", "dataflow", "httpsql")) + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + units := b.L3Units(st) + res := dataflow.RunWithSideTables(b.Fset(), units, 3, 4) + l4 := sdg.Run(sdg.Input{ + Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Externals: app.Externals, CallGraph: app.CallGraph, + Units: units, L3: res, K: 3, Jobs: 4, Models: models, + }) + l4.ApplyTo(app) + return Emit(app, "httpsql", dir, 4, "gate-test", 3, res, &l4) +} + +// TestGateL4_HTTPToSQLSanitizerIsSDKSemantics pins what L4 does — and +// does not do — about SanitizedFind's strconv.Atoi [user, 2026-10-27]. L4 is +// reaching-definitions data flow, not taint: `n, err := strconv.Atoi(id)` +// does make n (and err) depend on id, and that edge is correct. Treating a +// call as a sanitizer is the SDK taint analysis's job (callable or +// variable cuts, including the bypasses below), tracked in +// codellm-devkit/.github#103. +// +// - (a) The model strconv/Atoi: [] removes the Atoi call site's summary +// edges and nothing else that matters here: SanitizedHandler's r still +// reaches DB.Query's query, because the host statement links id → n. +// - (b) A variable cut on n within sqlRepo.SanitizedFind — the SDK's pair +// cut: drop the edges whose var is n and whose source node is under that +// callable (python-sdk commons/graphs.py:380-383) — does NOT sever the +// path either. Three bypasses remain, each correct data or control +// dependence: +// 1. err → cdg → Query: the same statement defines err; `if err != nil` +// controls the Query statement (implicit flow); +// 2. err → return → caller → receiver: `return nil, err` carries err to +// the caller, whose host feeds the receiver actual_in back into +// SanitizedFind's s, then s.db → Query; +// 3. caller host → receiver actual_in → s.db: the caller's host +// statement uses id, and the coarse host wiring feeds every host use +// into every actual_in, receiver included. +func TestGateL4_HTTPToSQLSanitizerIsSDKSemantics(t *testing.T) { + plain := buildL4(t, "dataflow/httpsql").l4 + modelled := buildHTTPSQLWithModels(t, sdg.Models{"strconv/Atoi": {}}) + app := modelled.Application.ID + mod := v2.ModuleID(app, "httpsql.go") + pkg := "example.com/dataflow/httpsql." + sanFind := v2.CallableID(v2.TypeID(mod, pkg+"sqlRepo"), pkg+"sqlRepo.SanitizedFind") + handler := v2.CallableID(v2.CallableID(mod, pkg+"SanitizedHandler"), pkg+"SanitizedHandler.closure_1") + sink := v2.ExternalID(app, "database/sql", "DB.Query") + "@formal_in:1" + r := v2.GlobalOrdinal(handler, "@formal_in:1") + id := v2.GlobalOrdinal(sanFind, "@formal_in:1") + recv := v2.GlobalOrdinal(sanFind, "@formal_in:0") + + callable := func(a *v2.Analysis, cid string) v2.Callable { + for _, c := range l4Callables(a) { + if c.ID == cid { + return c + } + } + t.Fatalf("no callable %s", cid) + return v2.Callable{} + } + // The Atoi call site: the actual_in whose param_in reaches strconv/Atoi. + atoi := v2.ExternalID(app, "strconv", "Atoi") + var atoiSite string + for _, e := range modelled.Application.ParamIn { + if strings.HasPrefix(e.Dst, atoi+"@") && strings.HasPrefix(e.Src, sanFind+"@") { + atoiSite = strings.TrimPrefix(e.Src, sanFind+"@") + atoiSite = atoiSite[:strings.Index(atoiSite, "/actual_in")] + } + } + if atoiSite == "" { + t.Fatal("no param_in from SanitizedFind to strconv/Atoi") + } + sitesSummary := func(a *v2.Analysis) int { + n := 0 + for _, e := range callable(a, sanFind).Summary { + if strings.HasPrefix(e.Src, atoiSite+"/") { + n++ + } + } + return n + } + + // (a) The model removes the Atoi summary edges; the flow stays. + if n := sitesSummary(plain); n == 0 { + t.Fatalf("without a model, the Atoi site %s has no summary edges (the model check would be vacuous)", atoiSite) + } + if n := sitesSummary(modelled); n != 0 { + t.Errorf("strconv/Atoi: [] leaves %d summary edges at %s", n, atoiSite) + } + g := newSDGGraph(modelled) + if g.path(r, sink) == nil { + t.Errorf("with strconv/Atoi: [], SanitizedHandler's r no longer reaches %s; L4 must keep the id → n data flow (.github#103)", sink) + } + + // (b) A variable cut on n within SanitizedFind does not sever it. + cutN := func(src string, h sdgHop) bool { return h.v == "n" && underCallable(src, sanFind) } + if g.pathExcept(r, sink, cutN) == nil { + t.Errorf("a cut on n within SanitizedFind severs %s → %s: a bypass below is gone; update this test and .github#103", r, sink) + } + // Bypass 1: err → cdg → Query, within SanitizedFind alone. + local := func(src string, h sdgHop) bool { + return cutN(src, h) || !underCallable(h.to, sanFind) && h.to != sink + } + if p := g.pathExcept(id, sink, local); p == nil || !slices.ContainsFunc(p, func(h sdgHop) bool { return h.family == "cdg" }) { + t.Errorf("bypass 1 (err → cdg → Query) gone: path %v", p) + } + // Bypass 2: err → return → the caller → the receiver, with cdg excluded. + noCDG := func(src string, h sdgHop) bool { return cutN(src, h) || h.family == "cdg" } + p2 := g.pathExcept(id, sink, noCDG) + if p2 == nil || !slices.ContainsFunc(p2, func(h sdgHop) bool { return h.family == "param_out" }) || + !slices.ContainsFunc(p2, func(h sdgHop) bool { return h.family == "param_in" && h.to == recv }) { + t.Errorf("bypass 2 (err → return → caller → receiver) gone: path %v", p2) + } + // Bypass 3: the caller's host feeds id into the receiver actual_in. + // r reaches the receiver without entering SanitizedFind's body first. + outside := func(_ string, h sdgHop) bool { return underCallable(h.to, sanFind) && h.to != recv } + p3 := g.pathExcept(r, recv, outside) + if p3 == nil || g.pathExcept(recv, sink, cutN) == nil { + t.Errorf("bypass 3 (caller host → receiver actual_in → s.db) gone: %v", p3) } } diff --git a/testdata/dataflow/httpsql/httpsql.go b/testdata/dataflow/httpsql/httpsql.go index 6efdac1..5f4055e 100644 --- a/testdata/dataflow/httpsql/httpsql.go +++ b/testdata/dataflow/httpsql/httpsql.go @@ -14,6 +14,7 @@ import ( type Repo interface { Find(id string) (*sql.Rows, error) SanitizedFind(id string) (*sql.Rows, error) + Count(user string) (*sql.Rows, error) } type sqlRepo struct{ db *sql.DB } @@ -35,6 +36,11 @@ func (s *sqlRepo) SanitizedFind(id string) (*sql.Rows, error) { return s.db.Query("SELECT name FROM users WHERE id = " + strconv.Itoa(n)) } +// Count counts a user's rows; only Audit calls it. +func (s *sqlRepo) Count(user string) (*sql.Rows, error) { + return s.db.Query("SELECT count(*) FROM events WHERE user = " + user) +} + // Handler reads the id query parameter and looks it up. func Handler(repo Repo) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { @@ -66,10 +72,21 @@ func Health(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) } +// Audit is an entry point apart from Routes: it builds its own Repo inline +// (no call), so it shares no host variable and no in-project callee with the +// handlers; its only in-project callee is sqlRepo.Count. +func Audit(db *sql.DB, user string) error { + var repo Repo = &sqlRepo{db: db} + rows, err := repo.Count(user) + if err != nil { + return err + } + return rows.Close() +} + // Routes wires the handlers to a database-backed Repo: without it no // concrete type reaches the handlers' repo, and VTA binds repo.Find to -// nothing but the interface method. Keep Handler registered before -// SanitizedHandler: TestGateL4_HTTPToSQLPath's no-flow pair relies on it. +// nothing but the interface method. func Routes(mux *http.ServeMux, db *sql.DB) { repo := NewRepo(db) mux.Handle("/user", Handler(repo)) From d90aadf5f6516a8ca7391bcf157f697ee882ac32 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 16:57:19 -0400 Subject: [PATCH 47/65] docs: library models change summaries, not statement-level reachability --- CLAUDE.md | 4 ++- README.md | 6 +++- docs/design/notes/l3-dataflow-metrics.md | 20 ++++++++++++- docs/design/specs/v2-l3-l4-dataflow.md | 36 ++++++++++++++++++------ 4 files changed, 55 insertions(+), 11 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index e1aee78..a1e5a28 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -116,7 +116,9 @@ Reuses Group A ids/spans and the L3 vocabulary unchanged; every L4 fact is an ad interface call binds the interface method (pass-through summary) *and* every VTA target. - **Stand-ins:** `external_symbols[].receiver {name, type}` and `.body` (formal vertices only, ids `@formal_in:`); summary all-to-all unless `--library-models` overrides it - (keys `/`, values `[in, out]` `of` pairs). + (keys `/`, values `[in, out]` `of` pairs). A model changes summaries, not + statement-level reachability; sanitizer semantics are the SDK's [user, 2026-10-27; + codellm-devkit/.github#103]. - **`degraded[]`** envelope leaf `{code, detail?, packages[]?, callables[]?}`, absent when nothing degraded; codes `no_ssa | no_vta | partial_ssa | partial_types | unmapped_ssa | l3_skipped | range_func` (open vocabulary). `--strict` exits non-zero on any. diff --git a/README.md b/README.md index a4e4615..4ab8cfd 100644 --- a/README.md +++ b/README.md @@ -322,7 +322,11 @@ subset of `-a 4` output, and `-j 1` and `-j N` output are byte-identical. - **`--library-models `** overrides stand-in summaries: `{"strconv/Atoi": [], "strings/TrimSpace": [["s", ""]]}`. A key is the external id's `/` tail, and a value is a list of `[in, out]` pairs of `of` names (`[]` = no - flow). An entry naming an `of` the external lacks falls back to all-to-all, with a warning. A + summary edges). A model changes summaries, **not statement-level reachability**: in + `n, err := strconv.Atoi(id)` the value assigned at the call site keeps its reaching-definition + edge from `id`, which is correct data flow. A model is therefore not a sanitizer. Cutting such a + flow is the SDK taint analysis's job ([codellm-devkit/.github#103](https://github.com/codellm-devkit/.github/issues/103)). A variable cut on `n` is also bypassed, by + implicit flow through `err` and by the coarse host wiring. An entry naming an `of` the external lacks falls back to all-to-all, with a warning. A key no call site targets is ignored, with a warning. The flag is valid only at `-a 4` and with `--emit neo4j`. - **`-t` / `--target-files` is not supported at `-a 4`** (whole-program): it is an error. Use diff --git a/docs/design/notes/l3-dataflow-metrics.md b/docs/design/notes/l3-dataflow-metrics.md index d0302c3..d47020a 100644 --- a/docs/design/notes/l3-dataflow-metrics.md +++ b/docs/design/notes/l3-dataflow-metrics.md @@ -1027,7 +1027,9 @@ The Go-side gates do the arity and kind checks per edge on the fixtures `TestGateL4_VertexIdsAreGlobalOrdinals`). The HTTP → SQL "done" fixture (`testdata/dataflow/httpsql`, `TestGateL4_HTTPToSQLPath`) does a BFS over the emitted JSON graph. The handler's `r` reaches `database/sql.DB.Query`'s `query` formal through a VTA-bound `param_in` -into `sqlRepo.Find`, and a handler that never queries does not reach it. +into `sqlRepo.Find`. A handler that never queries does not reach it, and neither does a separate +entry, `Audit`, reach `Find`'s `id`. `TestGateL4_HTTPToSQLSanitizerIsSDKSemantics` pins the +sanitizer variant (§ Limitations). **Metamorphic relations.** `scripts/metamorphic.sh` (memos, the default) passes on this HEAD with 0 violations in all 14 relations. It covered 20–32 sites for MR1–MR6, MR14 and MR15, and 33M @@ -1075,6 +1077,22 @@ over-approximation or reported as degraded. the Kubernetes root. The SDK then refuses to certify "no flow" over that graph (spec §4.15). That is safe, but it means `exhausted` is never returned on most real apps until D8 is modelled. +- **Library models and sanitizers change summaries, not statement-level reachability** + [user, 2026-10-27]. + - L4 is reaching-definitions data flow. In `n, err := strconv.Atoi(id)` the host statement + links `id` to `n` and `err` (the L3 call rule), and that is correct. + - `--library-models` `strconv/Atoi: []` removes only the Atoi call site's `summary` edges, so + the done fixture's sanitized path still reaches `DB.Query`. + - An SDK callable sanitizer cuts only nodes under the sanitizer's id, never the caller's host. + - A variable cut on `n` within `SanitizedFind` is bypassed three ways, each correct + dependence: + 1. `err` → `if err != nil` → the Query statement (cdg, implicit flow); + 2. `return nil, err` → the caller → the receiver `actual_in`; + 3. the caller's host feeding `id` into the receiver `actual_in` (coarse host wiring). + - Sanitizer semantics are the SDK taint analysis's job: [codellm-devkit/.github#103](https://github.com/codellm-devkit/.github/issues/103). + `TestGateL4_HTTPToSQLSanitizerIsSDKSemantics` pins each fact. + - The same statement-level call rule exists in codeanalyzer-python + (`access_paths.py:500-501`, `:350-356`), so this is cross-language. - **Library callbacks are not edges.** In `sort.Slice(xs, less)` and `http.HandleFunc(path, h)`, the function value flows into the stand-in, but the library calling it back is not modelled. Taint that *starts* in a handler (its own parameters) is unaffected. diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index 0c19649..f9f88c6 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -89,7 +89,7 @@ siblings is required too; and the id grammar has to change for polyglot apps any As of 2026-10-25 (code at `3504445`, branch `feat/l3-dataflow` = `feat/l4-sdg`): - **Phases 0 and 1 are implemented together on PR #16** (§9; open, not yet merged). L1/L2/L3 on - v2, Neo4j projection live through L3. + v2, Neo4j projection live through L3 (as of this snapshot; L4 is now live too, Status header). - **Phase 0 (live):** `can:///go/…` ids (§4.1, `internal/schema/v2/ids.go`); L2 library calls with `application.external_symbols` (§4.14); co-positioned call keys `l:c/n`; `is_deferred`; `typeKind` emits `alias`/`defined`; `GO_DDG` `_k = var|prov`; the per-package call-site index @@ -624,6 +624,16 @@ Context-insensitive. Per callable compute: predicate, §4.6), propagated up the SCC DAG and injected at hosts as uses / weak defs (A4). - No hammock regions (neither sibling built them; recorded as a known simplification). - Stand-ins (§6) contribute their modelled summary (default all-inputs → all-outputs). +- **Summaries are not sanitizers [user, 2026-10-27].** L4 is reaching-definitions data flow, not + taint. A modelled summary (`--library-models`), like an SDK callable sanitizer, changes the + call site's `summary` edges and the callee's flows, **not statement-level reachability**: the + value a call site assigns (`n, err := strconv.Atoi(id)`) keeps its reaching-definition edge + from `id` through the host statement (the L3 call rule, A15), and that edge is correct. A + variable cut (`("n", within=…)`) is bypassed too, by implicit flow (`err` → `if err != nil` → + the next statements, cdg), by results returned to the caller, and by the coarse host wiring + (the caller's host feeds every `actual_in`, receiver included). Deciding that a call + sanitizes is the SDK taint analysis's job: codellm-devkit/.github#103. The done fixture pins + all of this (`TestGateL4_HTTPToSQLSanitizerIsSDKSemantics`). - **Global → global summary edges are not emitted [user, 2026-10-26].** A `summary` edge whose `actual_in` and `actual_out` both have a `:` `of` is omitted; every edge with at least one non-global end is kept, and the flows (and so the global formals, actuals and param edges) @@ -959,7 +969,10 @@ addition now; the cross-language SDK taint revisit is deferred (§11). **Format (amended, Task 15):** a JSON object; key = `/`, the external id's tail (`strconv/Atoi`, `database/sql/DB.Query`); value = a list of `[in, out]` pairs of `of` strings (`"s"`, `"_0"`, `"_recv"`, `""`, `":1"`, a by-ref param's name). - `[]` = no flow; a missing key = the all-to-all default. Valid only at `-a 4` and with + `[]` = no summary edges at that external's call sites (not "no flow": the call site's + assigned values keep their statement-level reaching-definition edges, and cutting those is + SDK taint semantics, §4.9, codellm-devkit/.github#103); a missing key = the all-to-all + default. Valid only at `-a 4` and with `--emit neo4j` (`-a 1..3`: error `--library-models applies only at analysis level 4`); a missing, unreadable or malformed file is an error naming the path. An entry naming an `of` the external lacks is a warning and is ignored for that external, which keeps the all-to-all @@ -1045,7 +1058,8 @@ with ≥2 implementations; func-value call; `a→b→c` chain; mutual recursion; function read in another; multi-file flow; cross-file method; multi-result returns; variadic; library pass-through and library sink (`database/sql`, `os/exec`, `fmt`); generics (existing `testdata/generics`: instantiation → `Origin()` mapping, generic interface dispatch); an HTTP-handler → -SQL-sink end-to-end taint fixture with a sanitizer variant (the "done" fixture). +SQL-sink end-to-end taint fixture with a sanitizer variant (the "done" fixture; the sanitizer +variant pins what L4 gives a sanitizer, and its semantics are the SDK's, codellm-devkit/.github#103). **Added by the 2026-10-25 re-check (A23):** a Go 1.23+ module (range-over-func — pins the D8 gap — and range over int; `testdata/dataflow` declares `go 1.21`, #28); a method value (A8); a directly @@ -1083,14 +1097,18 @@ the delta relations. The fixtures (`testdata/dataflow` flow and sdg, chi) run in the eval apps run behind the `metamorphic` build tag (`scripts/metamorphic.sh`). **SDK gates:** slice gate (exact set on a `can://…@line:col` criterion); taint gate (known -source→sink found; sanitizer variant reported cut, not dropped; contiguous witness); a pair with no +source→sink found; sanitizer variant — pending codellm-devkit/.github#103, see "Done"; contiguous witness); a pair with no flow lands in `exhausted`; context-sensitivity only as far as summaries support; JSON and Neo4j backends agree; e2e level matrix. -**"Done"** (user-confirmed intent, designer-worded): through `CLDK.go(...)`, on the HTTP→SQL -fixture, `taint()` finds the flow from a handler's request parameter to `database/sql.DB.Query`'s -`query` through an interface call; a callable sanitizer cuts it; an unrelated pair is `exhausted`; -identical answers from the JSON and Neo4j backends. +**"Done"** (user-confirmed intent, designer-worded; amended [user, 2026-10-27]): through +`CLDK.go(...)`, on the HTTP→SQL fixture, `taint()` finds the flow from a handler's request +parameter to `database/sql.DB.Query`'s `query` through an interface call; an unrelated pair is +`exhausted`; identical answers from the JSON and Neo4j backends. Sanitizer handling (callable or +variable cuts) is SDK taint semantics, tracked in codellm-devkit/.github#103: L4's output is +correct reaching-definitions data flow, and neither a library model nor a variable cut on the +sanitized value severs the sanitized path (§4.9). The analyzer side is gated by +`TestGateL4_HTTPToSQLPath` and `TestGateL4_HTTPToSQLSanitizerIsSDKSemantics`. **Real-world evaluation** (non-gating metrics + crash/determinism/time gates; amended, A23): the six apps — cobra (small), dae, openbao, listmonk and memos (HTTP → SQL; memos through the @@ -1153,6 +1171,8 @@ dispatch + type-based may-alias). strings, not type ids, and `interfaces[]` is never computed (codeanalyzer-go#31; pre-existing, not an L4 change). - Library callback modelling (`sort.Slice`, `http.HandleFunc`). +- Sanitizer semantics in the SDK taint analysis (callable and variable cuts that account for the + host statement, implicit flow and coarse host wiring): codellm-devkit/.github#103. - Promote `callable.source_file` to the keystone (existing deferral). - Struct-tag modelling (existing deferral). - **Incremental L4 / incremental Neo4j analysis — CLDK roadmap** (D9): re-analysis after a PR merges From 4fb14b5c571bac71230f7e58bcec45c0e37b755d Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 18:40:53 -0400 Subject: [PATCH 48/65] fix(l4): follow VTA through synthetic wrappers A method value of an interface or a struct embedding one makes VTA return a $bound or promotion wrapper whose own call of the concrete method sits at a synthetic site no call node joins; the wrapper's VTA callees are now targets of the original site. --- internal/sdg/ssa_test.go | 22 ++++--- internal/sdg/vta.go | 55 +++++++++++++++- internal/sdg/wrap_test.go | 43 ++++++++++++ .../v2emit/gate_l4_wrap_test.go | 58 +++++++++++++++++ testdata/dataflow/wrap/wrap.go | 65 +++++++++++++++++++ 5 files changed, 231 insertions(+), 12 deletions(-) create mode 100644 internal/sdg/wrap_test.go create mode 100644 internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go create mode 100644 testdata/dataflow/wrap/wrap.go diff --git a/internal/sdg/ssa_test.go b/internal/sdg/ssa_test.go index 8aaacc9..5af0091 100644 --- a/internal/sdg/ssa_test.go +++ b/internal/sdg/ssa_test.go @@ -50,6 +50,14 @@ func loadInputOpts(t *testing.T, name string, skipTests bool) Input { } } +// dataflowPkgs are the project packages of testdata/dataflow (tests +// skipped), sorted. +var dataflowPkgs = []string{ + "example.com/dataflow/asm", "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/httpsql", "example.com/dataflow/initclosure", "example.com/dataflow/inits", + "example.com/dataflow/l1", "example.com/dataflow/linedir", "example.com/dataflow/sdg", "example.com/dataflow/vend", + "example.com/dataflow/wrap", "example.com/dataflow/xfile", "example.com/dataflow/xtest", "example.com/dataflow/xtest/user", +} + // TestSSA_BuildsProjectPackagesOnly: bodies are built for the project's // packages only. Every function with syntax lies in a project file, and no // function declared outside the project (stdlib, the vendored vlib) has blocks. @@ -63,11 +71,7 @@ func TestSSA_BuildsProjectPackagesOnly(t *testing.T) { for _, p := range pkgs { built = append(built, p.Pkg.Path()) } - want := []string{ - "example.com/dataflow/asm", "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/httpsql", "example.com/dataflow/initclosure", "example.com/dataflow/inits", - "example.com/dataflow/l1", "example.com/dataflow/linedir", "example.com/dataflow/sdg", "example.com/dataflow/vend", - "example.com/dataflow/xfile", "example.com/dataflow/xtest", "example.com/dataflow/xtest/user", - } + want := dataflowPkgs if !slices.Equal(built, want) { t.Fatalf("built packages = %v, want %v", built, want) } @@ -137,8 +141,8 @@ func TestSSA_PanicDegradesPackage(t *testing.T) { t.Error("the panicked package is still returned as built") } } - if len(pkgs) != 12 { - t.Errorf("built %d packages, want the other 12", len(pkgs)) + if len(pkgs) != len(dataflowPkgs)-1 { + t.Errorf("built %d packages, want the other %d", len(pkgs), len(dataflowPkgs)-1) } if fn := prog.ImportedPackage("example.com/dataflow/l1"); fn == nil || len(fn.Members) == 0 { t.Error("l1 was not built after flow panicked") @@ -156,8 +160,8 @@ func TestSSA_PanicDegradesPackage(t *testing.T) { if !slices.Equal(codes, []string{"partial_ssa", "no_ssa"}) { t.Fatalf("codes = %v, want [partial_ssa no_ssa]", codes) } - if len(deg[0].Packages) != 13 { - t.Errorf("partial_ssa lists %v, want all 13 packages", deg[0].Packages) + if !slices.Equal(deg[0].Packages, dataflowPkgs) { + t.Errorf("partial_ssa lists %v, want all %d packages", deg[0].Packages, len(dataflowPkgs)) } if res := Run(in); len(res.Degraded) != 2 || res.Degraded[1].Code != "no_ssa" { t.Errorf("Run degraded = %+v, want partial_ssa + no_ssa and no VTA", res.Degraded) diff --git a/internal/sdg/vta.go b/internal/sdg/vta.go index 6fdbb26..d09d441 100644 --- a/internal/sdg/vta.go +++ b/internal/sdg/vta.go @@ -182,7 +182,13 @@ func (d *dispatch) methodTarget(j *join, m *types.Func) (Target, bool) { } // addVTA adds VTA's callees: every edge whose site joins a call node and -// whose callee maps to a target. +// whose callee maps to a target. A callee that is a synthetic wrapper — a +// $bound or $thunk, a promotion, pointer or instantiation wrapper: SSA code +// with no syntax of its own — calls what it wraps at a synthetic call site, +// which joins no call node; for an interface method's wrapper (a method value +// of an interface, a struct embedding an interface) VTA's concrete callees +// sit only there. So the wrapper's own VTA callees, transitively through +// nested wrappers, are targets of the original site too. func (d *dispatch) addVTA(j *join, cg *callgraph.Graph) { for _, node := range cg.Nodes { for _, e := range node.Out { @@ -193,13 +199,56 @@ func (d *dispatch) addVTA(j *join, cg *callgraph.Graph) { if !ok { continue } - if t, ok := d.calleeTarget(j, e.Callee.Func); ok { - d.add(siteKey{unit, key}, t) + site := siteKey{unit, key} + for _, fn := range throughWrappers(cg, e.Callee.Func) { + if t, ok := d.calleeTarget(j, fn); ok { + d.add(site, t) + } } } } } +// throughWrappers returns fn and, when fn is a synthetic wrapper, the VTA +// callees of its calls, transitively through nested wrappers (each function +// once, so a cycle of wrappers ends). The order is the graph's; d.add sorts. +func throughWrappers(cg *callgraph.Graph, fn *ssa.Function) []*ssa.Function { + if !isWrapper(fn) { + return []*ssa.Function{fn} + } + seen := map[*ssa.Function]bool{fn: true} + out := []*ssa.Function{fn} + for work := []*ssa.Function{fn}; len(work) > 0; { + w := work[len(work)-1] + work = work[:len(work)-1] + node := cg.Nodes[w] + if node == nil { + continue + } + for _, e := range node.Out { + if e.Callee == nil || e.Callee.Func == nil || seen[e.Callee.Func] { + continue + } + g := e.Callee.Func + seen[g] = true + out = append(out, g) + if isWrapper(g) { + work = append(work, g) + } + } + } + return out +} + +// isWrapper reports whether fn is SSA-synthesized code with no syntax of its +// own: a $bound, $thunk, promotion or pointer wrapper. A range-over-func +// yield and an instance keep syntax (their code is the unit's own); an +// instantiation wrapper keeps its origin's syntax but is a synthetic call of +// the origin, which the join already maps. +func isWrapper(fn *ssa.Function) bool { + return fn.Synthetic != "" && fn.Syntax() == nil && fn.Synthetic != synthPackageInit +} + // calleeTarget maps an SSA callee to its target: the unit it joins (by body // or as a wrapper; an instance joins its origin), an interface method, a // bodiless declaration, or a library function's external. An unmapped project diff --git a/internal/sdg/wrap_test.go b/internal/sdg/wrap_test.go new file mode 100644 index 0000000..e272f2b --- /dev/null +++ b/internal/sdg/wrap_test.go @@ -0,0 +1,43 @@ +package sdg + +import ( + "path/filepath" + "slices" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +const wrapPkg = "example.com/dataflow/wrap" + +// wrapKey is the L3Key of a callable declared in testdata/dataflow/wrap/wrap.go. +func wrapKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("wrap", "wrap.go"), Signature: wrapPkg + "." + name} +} + +// TestVTA_TargetsBehindSyntheticWrappers (F1): VTA returns a synthetic +// wrapper for HandleBound's f(id) ((Store).Find$bound) and HandleDecorated's +// l.Find(id) (logged's promotion wrapper); the wrapper's own call of Find is +// a synthetic call site no call node joins. The wrapper's VTA targets are the +// original site's: each handler's site reaches sqlStore.Find, as the direct +// interface call does, and so does the global sqlStore.Find writes. +func TestVTA_TargetsBehindSyntheticWrappers(t *testing.T) { + in, _, d := runDispatch(t) + find := Target{Callable: wrapKey("sqlStore.Find")} + fps := footprints(in, d) + const global = wrapPkg + "::lastQuery" + for _, c := range []struct{ fn, site string }{ + {"HandleDirect", "33:9"}, + {"HandleBound", "39:9"}, + {"HandleDecorated", "50:9"}, + } { + got := d.targets(wrapKey(c.fn), c.site) + if !slices.Contains(got, find) { + t.Errorf("targets(%s, %s) = %v, want sqlStore.Find among them", c.fn, c.site, got) + } + fp := fps[Target{Callable: wrapKey(c.fn)}] + if !slices.Contains(fp.GlobalWrites, global) { + t.Errorf("%s footprint writes %v, want %s", c.fn, fp.GlobalWrites, global) + } + } +} diff --git a/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go b/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go new file mode 100644 index 0000000..e813ee4 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go @@ -0,0 +1,58 @@ +package v2emit + +import ( + "testing" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// l4Vertex is the one body node of owner with kind and of in a's tree. +func l4Vertex(t *testing.T, a *v2.Analysis, owner, kind, of string) string { + t.Helper() + var found []string + for id, on := range bodyIndex(a) { + if on.owner == owner && on.node.Kind == kind && on.node.Of == of { + found = append(found, id) + } + } + if len(found) != 1 { + t.Fatalf("%s has %d %s vertices of %q, want 1", owner, len(found), kind, of) + } + return found[0] +} + +// hasVertex reports whether owner has a body node of kind and of in a's tree. +func hasVertex(a *v2.Analysis, owner, kind, of string) bool { + for _, on := range bodyIndex(a) { + if on.owner == owner && on.node.Kind == kind && on.node.Of == of { + return true + } + } + return false +} + +// TestGateL4_FlowBehindSyntheticWrappers (F1): on testdata/dataflow/wrap, +// each handler's id reaches Exec's query formal — called directly through +// the interface, through a method value of it ((Store).Find$bound) and +// through a struct embedding it (a promotion wrapper) — and each handler's +// formal_out of the global sqlStore.Find writes exists. +func TestGateL4_FlowBehindSyntheticWrappers(t *testing.T) { + a := buildL4(t, "dataflow/wrap").l4 + if len(a.Degraded) != 0 { + t.Fatalf("wrap degraded: %+v", a.Degraded) + } + mod := v2.ModuleID(a.Application.ID, "wrap.go") + pkg := "example.com/dataflow/wrap." + fn := func(name string) string { return v2.CallableID(mod, pkg+name) } + sink := l4Vertex(t, a, fn("Exec"), "formal_in", "query") + g := newSDGGraph(a) + for _, h := range []string{"HandleDirect", "HandleBound", "HandleDecorated"} { + src := l4Vertex(t, a, fn(h), "formal_in", "id") + if p := g.path(src, sink); p == nil { + t.Errorf("%s's id does not reach Exec's query", h) + } + if !hasVertex(a, fn(h), "formal_out", ":example.com/dataflow/wrap::lastQuery") { + t.Errorf("%s has no formal_out of the global sqlStore.Find writes", h) + } + } +} diff --git a/testdata/dataflow/wrap/wrap.go b/testdata/dataflow/wrap/wrap.go new file mode 100644 index 0000000..d511c2f --- /dev/null +++ b/testdata/dataflow/wrap/wrap.go @@ -0,0 +1,65 @@ +// Package wrap is the L4 fixture for VTA targets behind synthetic wrappers +// (spec §4.8): an interface call reaches sqlStore.Find directly, through a +// method value of the interface (VTA returns the (Store).Find$bound wrapper) +// and through a struct embedding the interface (VTA returns logged's +// promotion wrapper). Each wrapper's own call of Find is a synthetic call +// site; its VTA targets are the original site's. Tests pin keys by line:col, +// so keep each statement on its own line and append new functions at the end. +package wrap + +// Store is the interface the handlers call. +type Store interface { + Find(id string) string +} + +// lastQuery is the global sqlStore.Find writes. +var lastQuery string + +type sqlStore struct{} + +// Find builds the query from id and runs it: the sink is Exec's query. +func (s *sqlStore) Find(id string) string { + lastQuery = id + return Exec("select " + id) +} + +// Exec is the sink. +func Exec(query string) string { + return query +} + +// HandleDirect calls Find through the interface. +func HandleDirect(s Store, id string) string { + return s.Find(id) +} + +// HandleBound calls Find through a method value of the interface. +func HandleBound(s Store, id string) string { + f := s.Find + return f(id) +} + +// logged decorates a Store by embedding it: Find is promoted. +type logged struct { + Store +} + +// HandleDecorated calls Find on a decorated Store, through the interface. +func HandleDecorated(s Store, id string) string { + var l Store = logged{s} + return l.Find(id) +} + +// MainDirect, MainBound and MainDecorated each pass a sqlStore to one +// handler: no host variable is shared between the handlers' call sites. +func MainDirect() { + HandleDirect(&sqlStore{}, "a") +} + +func MainBound() { + HandleBound(&sqlStore{}, "b") +} + +func MainDecorated() { + HandleDecorated(&sqlStore{}, "c") +} From 3752427d8e1d97e7cd1758a797f4402150b6ddb9 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 18:50:37 -0400 Subject: [PATCH 49/65] fix(l4): SSA for every package variant With tests loaded the PkgPath-keyed package map keeps only the test variant, so the plain package other packages import had no SSA bodies and VTA missed the types flowing through it. L4 now builds SSA over every loaded variant. --- internal/core/analyzer.go | 1 + internal/metamorphic/run.go | 2 +- internal/sdg/ir.go | 1 + internal/sdg/join_test.go | 16 ++-- internal/sdg/ssa.go | 32 ++++++-- internal/sdg/ssa_test.go | 4 +- internal/sdg/variant_test.go | 87 +++++++++++++++++++++ internal/syntactic_analysis/export.go | 6 ++ internal/syntactic_analysis/symbol_table.go | 8 +- testdata/dataflow/variant/p/p.go | 26 ++++++ testdata/dataflow/variant/p/p_test.go | 9 +++ testdata/dataflow/variant/u/u.go | 9 +++ 12 files changed, 182 insertions(+), 19 deletions(-) create mode 100644 internal/sdg/variant_test.go create mode 100644 testdata/dataflow/variant/p/p.go create mode 100644 testdata/dataflow/variant/p/p_test.go create mode 100644 testdata/dataflow/variant/u/u.go diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index f1f4cdd..e55dee4 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -147,6 +147,7 @@ func (a *Analyzer) Analyze() (*schema.GoApplication, error) { res := sdg.Run(sdg.Input{ Fset: builder.Fset(), Pkgs: builder.Pkgs(), + Variants: builder.Loaded(), ProjectDir: a.opts.InputPath, SymbolTable: symbolTable, Externals: app.Externals, diff --git a/internal/metamorphic/run.go b/internal/metamorphic/run.go index 076629b..8c96d98 100644 --- a/internal/metamorphic/run.go +++ b/internal/metamorphic/run.go @@ -64,7 +64,7 @@ func Analyze(dir string, level int, targets []string) (*Run, error) { if level >= 4 { res = dataflow.RunWithSideTables(b.Fset(), units, k, jobs) r := sdg.Run(sdg.Input{ - Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Fset: b.Fset(), Pkgs: b.Pkgs(), Variants: b.Loaded(), ProjectDir: dir, SymbolTable: st, Externals: app.Externals, CallGraph: app.CallGraph, Units: units, L3: res, K: k, Jobs: jobs, }) diff --git a/internal/sdg/ir.go b/internal/sdg/ir.go index d0d9bae..e32fd17 100644 --- a/internal/sdg/ir.go +++ b/internal/sdg/ir.go @@ -87,6 +87,7 @@ type Result struct { type Input struct { Fset *token.FileSet Pkgs map[string]*packages.Package + Variants []*packages.Package // every loaded package, test variants included (SymbolTableBuilder.Loaded); nil = Pkgs' values ProjectDir string SymbolTable map[string]schema.GoFile Externals map[string]schema.GoExternalCallee diff --git a/internal/sdg/join_test.go b/internal/sdg/join_test.go index 073b910..a056b8b 100644 --- a/internal/sdg/join_test.go +++ b/internal/sdg/join_test.go @@ -291,10 +291,10 @@ func TestJoin_ExternalTestPackage(t *testing.T) { } // Every SSA (*Counter).Add, of every variant: with tests, the - // test variant "xtest [xtest.test]" (a root, built from a.go's - // shared syntax) and the plain xtest that user imports (not a - // root: created from types, a bodiless stub). Both join the one - // callable: by syntax and by Object(). + // test variant "xtest [xtest.test]" and the plain xtest that + // user imports are both roots built from a.go's shared syntax + // (F2: the plain one's bodies carry the types user sees). Both + // join the one callable by syntax. wantAdd := dataflow.L3Key{Path: aGo, Signature: xt + ".Counter.Add"} kinds := map[joinKind]int{} for fn := range ssautil.AllFunctions(prog) { @@ -306,11 +306,9 @@ func TestJoin_ExternalTestPackage(t *testing.T) { kinds[kind]++ } } - if kinds[joinBody] != 1 { - t.Errorf("(*Counter).Add has %d SSA bodies, want 1 (kinds %v)", kinds[joinBody], kinds) - } - if !skip && kinds[joinWrapper] != 1 { - t.Errorf("tests: (*Counter).Add has %d types-only stubs, want 1 (kinds %v)", kinds[joinWrapper], kinds) + wantBodies := map[bool]int{true: 1, false: 2}[skip] + if kinds[joinBody] != wantBodies || len(kinds) != 1 { + t.Errorf("(*Counter).Add joins %v, want %d SSA bodies and nothing else", kinds, wantBodies) } // user.Use calls the plain xtest's stubs; they still name the // in-project callables, never an external. diff --git a/internal/sdg/ssa.go b/internal/sdg/ssa.go index b7efd15..3177453 100644 --- a/internal/sdg/ssa.go +++ b/internal/sdg/ssa.go @@ -3,6 +3,7 @@ package sdg import ( "fmt" "go/types" + "slices" "sort" "strings" @@ -32,7 +33,8 @@ func installCallHook(prog *ssa.Program) { } } -// buildSSA builds go/ssa over the project packages of in.Pkgs (spec §4.8, D2). +// buildSSA builds go/ssa over the project packages (projectPackages: every +// loaded variant) (spec §4.8, D2). // // ssautil.Packages gets only the project packages, so SSA bodies exist for // them alone; every dependency (stdlib, modules, vendored code) is created @@ -100,6 +102,7 @@ func buildSSA(in Input) (*ssa.Program, []*ssa.Package, []Degraded) { var deg []Degraded if len(failed) > 0 { sort.Strings(failed) + failed = slices.Compact(failed) // two variants of one path deg = append(deg, Degraded{ Code: CodePartialSSA, Detail: fmt.Sprintf("%d package(s) have no SSA (type errors or a recovered build panic)", len(failed)), @@ -126,17 +129,34 @@ func buildPackage(p *ssa.Package) (err error) { return nil } -// projectPackages returns the packages of in.Pkgs that hold project code (a -// Go file under ProjectDir, not vendored), sorted by path. Generated test -// mains live in the build cache and so are never project packages. +// projectPackages returns the loaded packages that hold project code (a Go +// file under ProjectDir, not vendored), sorted by (path, ID): every variant +// of in.Variants, else in.Pkgs. With tests loaded, a package path has two +// variants — the test variant "p [p.test]" (in.Pkgs keeps only it) and the +// plain p every other package imports — and both are project packages: SSA +// needs the plain p's bodies for VTA to see the types that flow through it +// (F2). The variants share their files' syntax, so both join the same +// units. Generated test mains live in the build cache and so are never +// project packages. func projectPackages(in Input) []*packages.Package { + all := in.Variants + if all == nil { + for _, p := range in.Pkgs { + all = append(all, p) + } + } var out []*packages.Package - for _, p := range in.Pkgs { + for _, p := range all { if isProject(in.ProjectDir, p) { out = append(out, p) } } - sort.Slice(out, func(i, j int) bool { return out[i].PkgPath < out[j].PkgPath }) + sort.Slice(out, func(i, j int) bool { + if out[i].PkgPath != out[j].PkgPath { + return out[i].PkgPath < out[j].PkgPath + } + return out[i].ID < out[j].ID + }) return out } diff --git a/internal/sdg/ssa_test.go b/internal/sdg/ssa_test.go index 5af0091..5144cdb 100644 --- a/internal/sdg/ssa_test.go +++ b/internal/sdg/ssa_test.go @@ -44,7 +44,7 @@ func loadInputOpts(t *testing.T, name string, skipTests bool) Input { edges := cg.Build(st) units := b.L3Units(st) return Input{ - Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Fset: b.Fset(), Pkgs: b.Pkgs(), Variants: b.Loaded(), ProjectDir: dir, SymbolTable: st, Externals: cg.Externals(), CallGraph: edges, Units: units, L3: dataflow.RunWithSideTables(b.Fset(), units, 3, 1), K: 3, Jobs: 1, } @@ -54,7 +54,7 @@ func loadInputOpts(t *testing.T, name string, skipTests bool) Input { // skipped), sorted. var dataflowPkgs = []string{ "example.com/dataflow/asm", "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/httpsql", "example.com/dataflow/initclosure", "example.com/dataflow/inits", - "example.com/dataflow/l1", "example.com/dataflow/linedir", "example.com/dataflow/sdg", "example.com/dataflow/vend", + "example.com/dataflow/l1", "example.com/dataflow/linedir", "example.com/dataflow/sdg", "example.com/dataflow/variant/p", "example.com/dataflow/variant/u", "example.com/dataflow/vend", "example.com/dataflow/wrap", "example.com/dataflow/xfile", "example.com/dataflow/xtest", "example.com/dataflow/xtest/user", } diff --git a/internal/sdg/variant_test.go b/internal/sdg/variant_test.go new file mode 100644 index 0000000..c835369 --- /dev/null +++ b/internal/sdg/variant_test.go @@ -0,0 +1,87 @@ +package sdg + +import ( + "path/filepath" + "reflect" + "slices" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +const variantPkg = "example.com/dataflow/variant/" + +var ( + variantUse = dataflow.L3Key{Path: filepath.Join("variant", "u", "u.go"), Signature: variantPkg + "u.Use"} + variantImpl = dataflow.L3Key{Path: filepath.Join("variant", "p", "p.go"), Signature: variantPkg + "p.impl.M"} +) + +// siteTargetsOf is the union of the dispatch targets of unit's call sites. +func siteTargetsOf(d *dispatch, unit dataflow.L3Key) []Target { + var out []Target + for site, ts := range d.sites { + if site.unit == unit { + out = append(out, ts...) + } + } + slices.SortFunc(out, compareTarget) + return slices.Compact(out) +} + +// TestSSA_EveryPackageVariant (F2): with tests loaded, go/packages yields +// the test variant "p [p.test]" beside the plain p that u imports. SSA +// must build both: u.Use's p.New().M(x) dispatches by VTA to impl.M only +// when the plain p, whose New u calls, has bodies. Use then writes the +// global impl.M writes, x reaches its , and nothing is degraded. +func TestSSA_EveryPackageVariant(t *testing.T) { + in := loadInputOpts(t, "dataflow", false) + res, d := run(in) + if len(res.Degraded) != 0 { + t.Fatalf("degraded = %+v, want none", res.Degraded) + } + impl := Target{Callable: variantImpl} + if ts := siteTargetsOf(d, variantUse); !slices.Contains(ts, impl) { + t.Errorf("Use's targets = %v, want impl.M among them", ts) + } + vta := false + for _, e := range res.CallEdges { + vta = vta || e.Source == variantUse.Signature && e.Target == variantImpl.Signature && slices.Equal(e.Provenance, []string{"vta"}) + } + if !vta { + t.Errorf("no VTA call_graph edge Use -> impl.M: %+v", res.CallEdges) + } + fp := footprints(in, d)[Target{Callable: variantUse}] + if !slices.Contains(fp.GlobalWrites, variantPkg+"p::Last") { + t.Errorf("Use writes %v, want %sp::Last", fp.GlobalWrites, variantPkg) + } + flows := newSDGBuilder(in, d).summarize() + if !slices.Contains(flows[Target{Callable: variantUse}], [2]int{0, 0}) { + t.Errorf("Use's flows = %v, want x → ([0 0])", flows[Target{Callable: variantUse}]) + } + // Every target and param edge appears once though two SSA variants of + // p join the same callables. + for site, ts := range d.sites { + if !slices.IsSortedFunc(ts, compareTarget) || len(slices.CompactFunc(slices.Clone(ts), func(a, b Target) bool { return a == b })) != len(ts) { + t.Errorf("site %v targets not sorted and unique: %v", site, ts) + } + } + if !slices.IsSortedFunc(res.ParamIn, compareParamEdge) || len(slices.Compact(slices.Clone(res.ParamIn))) != len(res.ParamIn) { + t.Error("param_in not sorted and unique") + } +} + +// TestSSA_VariantsNeutralWithoutTests (F2): with tests skipped there is one +// variant per path, so the L4 result is the same whether SSA is built from +// in.Variants or from in.Pkgs. +func TestSSA_VariantsNeutralWithoutTests(t *testing.T) { + in := loadInput(t, "dataflow") + if len(in.Variants) != len(in.Pkgs) { + t.Fatalf("%d variants, %d packages: want one variant per path with tests skipped", len(in.Variants), len(in.Pkgs)) + } + a := Run(in) + in.Variants = nil + b := Run(in) + if !reflect.DeepEqual(a, b) { + t.Error("L4 result differs between Variants and Pkgs with tests skipped") + } +} diff --git a/internal/syntactic_analysis/export.go b/internal/syntactic_analysis/export.go index fad49bb..a8d81c8 100644 --- a/internal/syntactic_analysis/export.go +++ b/internal/syntactic_analysis/export.go @@ -13,3 +13,9 @@ func (b *SymbolTableBuilder) Fset() *token.FileSet { return b.fset } // Pkgs returns the map of loaded packages keyed by package path. // Used by CallGraphBuilder for type-info lookups. func (b *SymbolTableBuilder) Pkgs() map[string]*packages.Package { return b.pkgs } + +// Loaded returns every loaded package with types, in load order. Unlike +// Pkgs it keeps every variant of a package path: with tests loaded, both +// the test variant "p [p.test]" and the plain p other packages import. +// L4 builds SSA over all of them (sdg.Input.Variants). +func (b *SymbolTableBuilder) Loaded() []*packages.Package { return b.loaded } diff --git a/internal/syntactic_analysis/symbol_table.go b/internal/syntactic_analysis/symbol_table.go index ada2bed..9ada56b 100644 --- a/internal/syntactic_analysis/symbol_table.go +++ b/internal/syntactic_analysis/symbol_table.go @@ -25,8 +25,13 @@ import ( type SymbolTableBuilder struct { projectDir string fset *token.FileSet - // pkgs is the flat list of loaded packages, keyed by package path. + // pkgs is the flat list of loaded packages, keyed by package path. With + // tests loaded, a package's test variant "p [p.test]" replaces the plain + // p here (L1–L3 read the variant, which holds every file of p). pkgs map[string]*packages.Package + // loaded is every loaded package with types, in load order: test + // variants and the plain packages they shadow in pkgs alike. + loaded []*packages.Package } // NewSymbolTableBuilder creates a builder for projectDir. @@ -97,6 +102,7 @@ func (b *SymbolTableBuilder) Build(targetFiles []string, skipTests bool) (map[st } if pkg.Types != nil { b.pkgs[pkg.PkgPath] = pkg + b.loaded = append(b.loaded, pkg) } } tooNew.warn() diff --git a/testdata/dataflow/variant/p/p.go b/testdata/dataflow/variant/p/p.go new file mode 100644 index 0000000..82a67bf --- /dev/null +++ b/testdata/dataflow/variant/p/p.go @@ -0,0 +1,26 @@ +// Package p is the L4 fixture for whole-program SSA with tests loaded +// (--skip-tests=false): p_test.go makes go/packages yield the test variant +// "p [p.test]" beside the plain p that package u imports. Only the plain p's +// impl reaches u's interface call, so SSA must build the plain p too. +package p + +// I is the interface u calls. +type I interface { + M(x string) string +} + +type impl struct{} + +// Last is the global impl.M writes. +var Last string + +// M records x and returns it. +func (impl) M(x string) string { + Last = x + return x +} + +// New returns the one implementation. +func New() I { + return impl{} +} diff --git a/testdata/dataflow/variant/p/p_test.go b/testdata/dataflow/variant/p/p_test.go new file mode 100644 index 0000000..4b65e52 --- /dev/null +++ b/testdata/dataflow/variant/p/p_test.go @@ -0,0 +1,9 @@ +package p + +import "testing" + +func TestM(t *testing.T) { + if New().M("a") != "a" { + t.Fatal("M") + } +} diff --git a/testdata/dataflow/variant/u/u.go b/testdata/dataflow/variant/u/u.go new file mode 100644 index 0000000..e306ca2 --- /dev/null +++ b/testdata/dataflow/variant/u/u.go @@ -0,0 +1,9 @@ +// Package u calls p's interface method on the value p.New returns. +package u + +import "example.com/dataflow/variant/p" + +// Use dispatches M through the interface p.New returns. +func Use(x string) string { + return p.New().M(x) +} From 4a2121d1126a326aa1d7ca9200f1196addef4dff Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 18:57:12 -0400 Subject: [PATCH 50/65] fix(l4): call_graph edges only for targets VTA produced A D7 interface-method target (or L2's static callee) at a site with no L2 edge was emitted with prov ["vta"]. It stays a dispatch target with its param edges; only VTA's own callees become call_graph edges. --- internal/sdg/vta.go | 30 +++++++++++++++++++++++++++++- internal/sdg/wrap_test.go | 23 +++++++++++++++++++++++ 2 files changed, 52 insertions(+), 1 deletion(-) diff --git a/internal/sdg/vta.go b/internal/sdg/vta.go index d09d441..c3f7356 100644 --- a/internal/sdg/vta.go +++ b/internal/sdg/vta.go @@ -33,12 +33,19 @@ type l2Edge struct{ path, src, dst string } type dispatch struct { sites map[siteKey][]Target // each sorted by compareTarget, no duplicates static map[siteKey]Target // the L2 static callee of a site, when it maps to a target + byVTA map[siteTarget]bool // the (site, target) pairs VTA produced: the call_graph edge candidates bySig map[string]dataflow.L3Key // symbol-table callable of each signature (init excepted) l2 map[l2Edge]bool l2Ext map[string]schema.GoExternalCallee // in.Externals, read-only newExt map[string]schema.GoExternalCallee // externals L2 never saw } +// siteTarget is one target of one call site. +type siteTarget struct { + site siteKey + t Target +} + // compareTarget orders targets by Callable (Path, Signature), then External. func compareTarget(a, b Target) int { return cmp.Or( @@ -57,6 +64,7 @@ func newDispatch(in Input) *dispatch { d := &dispatch{ sites: map[siteKey][]Target{}, static: map[siteKey]Target{}, + byVTA: map[siteTarget]bool{}, l2: make(map[l2Edge]bool, len(in.CallGraph)), l2Ext: in.Externals, newExt: map[string]schema.GoExternalCallee{}, @@ -203,6 +211,9 @@ func (d *dispatch) addVTA(j *join, cg *callgraph.Graph) { for _, fn := range throughWrappers(cg, e.Callee.Func) { if t, ok := d.calleeTarget(j, fn); ok { d.add(site, t) + if !isIfaceMethod(fn) { + d.byVTA[siteTarget{site, t}] = true + } } } } @@ -240,6 +251,18 @@ func throughWrappers(cg *callgraph.Graph, fn *ssa.Function) []*ssa.Function { return out } +// isIfaceMethod reports whether fn stands for an interface method (a $bound +// or $thunk of one): no concrete callee, so VTA found no edge to it — D7 +// binds the interface method itself (addInvokes). +func isIfaceMethod(fn *ssa.Function) bool { + obj, ok := fn.Object().(*types.Func) + if !ok { + return false + } + recv := obj.Type().(*types.Signature).Recv() + return recv != nil && types.IsInterface(recv.Type()) +} + // isWrapper reports whether fn is SSA-synthesized code with no syntax of its // own: a $bound, $thunk, promotion or pointer wrapper. A range-over-func // yield and an instance keep syntax (their code is the unit's own); an @@ -283,7 +306,9 @@ func (d *dispatch) external(obj *types.Func, sig string) { } // result is the Result of the dispatch: one call_graph edge per (caller, -// target) pair that L2 lacks, prov ["vta"], weight 1 — Source the caller's +// target) pair VTA produced (byVTA) that L2 lacks, prov ["vta"], weight 1 — +// a target bound otherwise (the D7 interface method, L2's static callee) is +// no edge, though it keeps its param edges — Source the caller's // signature and SourcePath its declaring file, as L2 has them (closures // included), Target the callee signature (the full external signature for an // external) and TargetPath the callee's declaring file (empty for an @@ -296,6 +321,9 @@ func (d *dispatch) result(deg []Degraded) Result { var ids []edgeID for site, ts := range d.sites { for _, t := range ts { + if !d.byVTA[siteTarget{site, t}] { + continue + } id := edgeID{site.unit.Path, site.unit.Signature, t.External, ""} if t.External == "" { id.dst, id.dstPath = t.Callable.Signature, t.Callable.Path diff --git a/internal/sdg/wrap_test.go b/internal/sdg/wrap_test.go index e272f2b..fa187e6 100644 --- a/internal/sdg/wrap_test.go +++ b/internal/sdg/wrap_test.go @@ -41,3 +41,26 @@ func TestVTA_TargetsBehindSyntheticWrappers(t *testing.T) { } } } + +// TestVTA_NoEdgeForTargetsVTADidNotFind (F7): a call_graph edge prov ["vta"] +// is one VTA produced. HandleBound's f(id) has no L2 callee; its targets +// include the interface method Store.Find (bound by D7, from the $bound +// wrapper VTA returned, not a VTA callee of the site's own) and sqlStore.Find +// (VTA, through the wrapper). Only the latter is an edge. The interface +// method stays a target: its param edges are unchanged. +func TestVTA_NoEdgeForTargetsVTADidNotFind(t *testing.T) { + _, res, d := runDispatch(t) + src := wrapKey("HandleBound") + if ts := d.targets(src, "39:9"); !slices.Contains(ts, Target{Callable: wrapKey("Store.Find")}) { + t.Fatalf("targets(HandleBound, 39:9) = %v, want Store.Find among them", ts) + } + var got []string + for _, e := range res.CallEdges { + if e.Source == src.Signature { + got = append(got, e.Target) + } + } + if want := []string{wrapPkg + ".sqlStore.Find"}; !slices.Equal(got, want) { + t.Errorf("VTA edges from HandleBound = %v, want %v", got, want) + } +} From 7d6faa129b54b0d8f1d7cb3d40df83171c8e1d01 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 19:00:11 -0400 Subject: [PATCH 51/65] refactor(l4): declKey is a free function formalIndex built a partially initialised join just to call it. --- internal/sdg/footprint.go | 11 +++++------ internal/sdg/join.go | 14 +++++++------- internal/sdg/vta.go | 2 +- 3 files changed, 13 insertions(+), 14 deletions(-) diff --git a/internal/sdg/footprint.go b/internal/sdg/footprint.go index a91a75f..efdd424 100644 --- a/internal/sdg/footprint.go +++ b/internal/sdg/footprint.go @@ -739,7 +739,6 @@ func newFormalIndex(in Input, d *dispatch) *formalIndex { } } } - j := &join{fset: in.Fset, projectDir: in.ProjectDir} // declKey reads only these two for _, p := range projectPackages(in) { if p.Types == nil { continue @@ -748,18 +747,18 @@ func newFormalIndex(in Input, d *dispatch) *formalIndex { for _, name := range scope.Names() { switch obj := scope.Lookup(name).(type) { case *types.Func: - x.addDecl(j, obj) + x.addDecl(obj) case *types.TypeName: named, ok := obj.Type().(*types.Named) if !ok { continue } for i := 0; i < named.NumMethods(); i++ { - x.addDecl(j, named.Method(i)) + x.addDecl(named.Method(i)) } if it, ok := named.Underlying().(*types.Interface); ok { for i := 0; i < it.NumExplicitMethods(); i++ { - x.addDecl(j, it.ExplicitMethod(i)) + x.addDecl(it.ExplicitMethod(i)) } } } @@ -781,8 +780,8 @@ func newFormalIndex(in Input, d *dispatch) *formalIndex { return x } -func (x *formalIndex) addDecl(j *join, fn *types.Func) { - if k, ok := j.declKey(fn); ok { +func (x *formalIndex) addDecl(fn *types.Func) { + if k, ok := declKey(x.in.Fset, x.in.ProjectDir, fn); ok { x.decls[k] = fn } } diff --git a/internal/sdg/join.go b/internal/sdg/join.go index 1051321..8f47eaf 100644 --- a/internal/sdg/join.go +++ b/internal/sdg/join.go @@ -118,7 +118,7 @@ func newJoin(in Input) *join { continue } if obj, ok := p.TypesInfo.Defs[fd.Name].(*types.Func); ok { - if k, ok := j.declKey(obj); ok && declared[k] { + if k, ok := declKey(j.fset, j.projectDir, obj); ok && declared[k] { j.bodiless[k] = true } } @@ -223,7 +223,7 @@ func (j *join) resolve(fn *ssa.Function) (dataflow.L3Key, joinKind) { return dataflow.L3Key{}, joinNone } if obj, ok := fn.Object().(*types.Func); ok { - if k, ok := j.declKey(obj.Origin()); ok { + if k, ok := declKey(j.fset, j.projectDir, obj.Origin()); ok { switch { case j.units[k].Fn != nil: return k, joinWrapper @@ -238,13 +238,13 @@ func (j *join) resolve(fn *ssa.Function) (dataflow.L3Key, joinKind) { } // declKey is obj's (declaring file, signature) when it is declared in a -// project file. -func (j *join) declKey(obj *types.Func) (dataflow.L3Key, bool) { - tf := j.fset.File(obj.Pos()) - if tf == nil || !utils.IsWithin(j.projectDir, tf.Name()) { +// project file of projectDir. +func declKey(fset *token.FileSet, projectDir string, obj *types.Func) (dataflow.L3Key, bool) { + tf := fset.File(obj.Pos()) + if tf == nil || !utils.IsWithin(projectDir, tf.Name()) { return dataflow.L3Key{}, false } - rel := utils.RelativePath(j.projectDir, tf.Name()) + rel := utils.RelativePath(projectDir, tf.Name()) if utils.IsVendored(rel) { return dataflow.L3Key{}, false } diff --git a/internal/sdg/vta.go b/internal/sdg/vta.go index c3f7356..ee0b0d0 100644 --- a/internal/sdg/vta.go +++ b/internal/sdg/vta.go @@ -183,7 +183,7 @@ func (d *dispatch) methodTarget(j *join, m *types.Func) (Target, bool) { d.external(m, sig) return Target{External: sig}, true } - if k, ok := j.declKey(m.Origin()); ok && j.iface[k] { + if k, ok := declKey(j.fset, j.projectDir, m.Origin()); ok && j.iface[k] { return Target{Callable: k}, true } return Target{}, false From 0a3ea120bb2ecbb8e3b268f63c9e679089c052be Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 19:16:48 -0400 Subject: [PATCH 52/65] fix(l4): CHA targets for interface calls on library-originated values MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit VTA sees no concrete type in library code, so ctx.Value(k).(Store).Find or a sync.Map round-trip kept only the D7 interface method. An invoke site whose receiver originates in a type assertion, a library call or a library load now also targets every project type implementing the interface. Also documents F1 and F7 in spec §4.8. --- README.md | 4 + docs/design/notes/l3-dataflow-metrics.md | 9 + docs/design/specs/v2-l3-l4-dataflow.md | 30 +++ internal/sdg/cha.go | 226 ++++++++++++++++++ internal/sdg/cha_test.go | 83 +++++++ internal/sdg/run.go | 1 + internal/sdg/ssa_test.go | 3 +- .../v2emit/gate_l4_wrap_test.go | 26 ++ testdata/dataflow/libstore/libstore.go | 96 ++++++++ 9 files changed, 477 insertions(+), 1 deletion(-) create mode 100644 internal/sdg/cha.go create mode 100644 internal/sdg/cha_test.go create mode 100644 testdata/dataflow/libstore/libstore.go diff --git a/README.md b/README.md index 4ab8cfd..0e949be 100644 --- a/README.md +++ b/README.md @@ -334,6 +334,10 @@ subset of `-a 4` output, and `-j 1` and `-j N` output are byte-identical. - **Known limits** (see `docs/design/notes/l3-dataflow-metrics.md` § L4): - library code calling back into the project (`sort.Slice`, `http.HandleFunc`) is not an edge; + - an interface call on a value that came out of library code (`ctx.Value(k).(Store)`, + `sync.Map.Load`) binds every project type implementing the interface (a targeted CHA + complement), but a value first stored in a project struct field or global and read back + later gets VTA's targets only, which miss the concrete type; - range-over-func is not modelled (`range_func`); - summaries are close to all-to-all, by design; - `summary` edges whose two ends are both `:` vertices are omitted diff --git a/docs/design/notes/l3-dataflow-metrics.md b/docs/design/notes/l3-dataflow-metrics.md index d47020a..317e785 100644 --- a/docs/design/notes/l3-dataflow-metrics.md +++ b/docs/design/notes/l3-dataflow-metrics.md @@ -1096,6 +1096,15 @@ over-approximation or reported as degraded. - **Library callbacks are not edges.** In `sort.Slice(xs, less)` and `http.HandleFunc(path, h)`, the function value flows into the stand-in, but the library calling it back is not modelled. Taint that *starts* in a handler (its own parameters) is unaffected. +- **Library-stored dynamic types (final review F3).** VTA sees no concrete type in library code. + An interface call whose receiver comes straight out of library code (`ctx.Value(k).(Store)`, + `sync.Map.Load`, a library call's result, a library field) gets a targeted CHA complement: + every project type implementing the interface. Measured (go1.26.4, `-j 8`): chi +54 targets at + 15 sites (the `w.(http.Flusher)` pattern), param_in 1,895 → 1,961 (+3.5%), param_out + 1,649 → 1,770 (+7.3%); nats-server +317 targets at 63 sites, param_in 862,765 → 863,075 + (+0.04%), param_out 431,327 → 431,854 (+0.12%). Remaining gap: a library-originated value + first stored in a project struct field or global, then read back and called, gets VTA's + targets only. - **A15: summaries are near all-to-all** (74–94% of sites, table above). Under the coarse host wiring, a summary edge adds no reachability for the SDK's context-insensitive walk. It is a shortcut hop, and a precondition for a future HRB two-phase walk (spec §11). diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index f9f88c6..4dd428f 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -552,6 +552,30 @@ built per D2 (amended, A12): `ssautil.Packages` over the **same** `go/packages` call-site `summary` edges are the union over targets. **D7:** an interface call *always* binds to the interface method (pass-through) and every VTA target, even when every target is in-project — per the anti-goal. Cost: every interface call (`io.Writer`) carries an all-to-all summary. + A target VTA did not produce — the D7 interface method, or a CHA-complement target (below) — is + a dispatch target with its param edges but **no `call_graph` edge** [final review F7]: `prov + ["vta"]` means VTA found it, and L2 already carries the static edge where it resolved. +- **Through synthetic wrappers [final review F1].** When VTA's callee at a project call site is a + synthetic wrapper with no syntax (`$bound`, `$thunk`, a promotion or pointer wrapper), the + wrapper's own call of what it wraps is a synthetic call site no call node joins. For an + interface method's wrapper — a method value `f := s.Find; f(id)`, or a struct embedding the + interface — VTA's concrete callees sit only there. The wrapper's VTA out-edges are followed + transitively (cycle-safe) and their joined callees are targets, and `call_graph` edges, of the + original site. +- **Targeted CHA complement [final review F3; controller ruling].** VTA propagates types through + project bodies only, so a value that passes through library storage — `ctx.Value(k).(Store)`, + `sync.Map.Load`, a library call's result — reaches an interface call with no concrete type VTA + sees, and only the D7 interface method (empty footprint) would remain. For an `invoke` site whose + receiver SSA value originates — through `Phi`, `Extract`, `ChangeInterface`, field reads and + `UnOp` loads — in a `TypeAssert`, the result of a call to a non-project function (or of a + library interface's method), or a load of a library global or through a library-typed + pointer or field, every **project** named type `T` or `*T` (non-generic) whose method set + implements the interface (`types.Implements`) adds the method its method set selects, mapped by + the join. A receiver that originates in project code (a `MakeInterface`, a parameter, a free + variable, a load of project storage) gets no CHA target: VTA covers it. Not a `call_graph` edge + (F7). **Remaining gap:** a library-originated value first stored in a project struct field or + global, then read back and called (the load is project storage, so the walk stops), is VTA's + alone and misses the concrete type — recorded in §6 beside library callbacks. - **Interfaces with no symbol-table node (ruling, Task 10) [call].** `error.Error()` (the universe `error`), methods of interface literals (`interface{ M() }`), and methods promoted through an embedded interface whose origin is declared outside the symbol table have no callable id and no @@ -987,6 +1011,12 @@ addition now; the cross-language SDK taint revisit is deferred (§11). `http.HandleFunc(path, h)`): the function-value argument flows in, but the library calling it back is not an edge. Taint *starting* in a handler is unaffected (sources are the handler's own params). Recorded limitation. +- **Not modelled:** a value that came out of library code (`ctx.Value(k)`, `sync.Map.Load`, a + library call's result), first stored in a project struct field or global and then read back + and called through an interface. The targeted CHA complement (§4.8) covers the call made on the + library-originated value directly; once the value is in project storage the receiver's origin + is a project load, and VTA, which sees no concrete type in library code, finds none. The call + keeps its D7 interface method (pass-through, empty footprint). Recorded limitation. - **SDK addressing:** `("query", "database/sql.DB.Query")` resolves through the Go backend's `resolve_callable` (domain = analysed callables ∪ stand-ins) then `resolve_value` over the stand-in body's `formal_in` `of`. Shared `commons/` code is untouched. diff --git a/internal/sdg/cha.go b/internal/sdg/cha.go new file mode 100644 index 0000000..86cd357 --- /dev/null +++ b/internal/sdg/cha.go @@ -0,0 +1,226 @@ +package sdg + +import ( + "go/ast" + "go/token" + "go/types" + + "golang.org/x/tools/go/ssa" + "golang.org/x/tools/go/types/typeutil" + + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" +) + +// cha is the targeted CHA complement of VTA (spec §4.8). VTA propagates +// types through project code only: library packages have no SSA bodies, so +// a value that passes through library storage — put in a context.Context or +// a sync.Map, returned by a library call, read from a library struct's field +// — reaches an interface call with no concrete type VTA can see, and only +// the D7 interface method (empty footprint) would remain. For an interface +// call whose receiver originates in library code (libraryOrigin), every +// project type whose method set implements the interface is a target: the +// method it resolves to, mapped as the join maps a callee. It reads SSA +// instructions and go/types only, never builds code, so it is safe before +// VTA. +type cha struct { + j *join + types []types.Type // every project named type T and *T that is not an interface or generic + byIface typeutil.Map // interface type → its implementers among types + msets typeutil.MethodSetCache +} + +func newCHA(j *join, in Input) *cha { + c := &cha{j: j} + seen := map[*types.TypeName]bool{} + for _, p := range projectPackages(in) { + if p.TypesInfo == nil { + continue + } + for _, f := range p.Syntax { + ast.Inspect(f, func(n ast.Node) bool { + ts, ok := n.(*ast.TypeSpec) + if !ok { + return true + } + tn, ok := p.TypesInfo.Defs[ts.Name].(*types.TypeName) + if !ok || tn.IsAlias() || seen[tn] { + return true + } + seen[tn] = true + named, ok := tn.Type().(*types.Named) + if !ok || named.TypeParams() != nil || types.IsInterface(named) { + return true + } + c.types = append(c.types, named, types.NewPointer(named)) + return true + }) + } + } + return c +} + +// implementers returns the project types of c.types that implement iface +// (T and *T alike: an interface may hold either). +func (c *cha) implementers(iface types.Type) []types.Type { + if v := c.byIface.At(iface); v != nil { + return v.([]types.Type) + } + it, _ := iface.Underlying().(*types.Interface) + var out []types.Type + if it != nil { + for _, t := range c.types { + if types.Implements(t, it) { + out = append(out, t) + } + } + } + c.byIface.Set(iface, out) + return out +} + +// addCHA adds the CHA complement's targets to every interface call of a +// body-joined project function (funcs) whose receiver originates in library +// code. A target is the method each implementer's method set selects: +// a unit, an interface method (a struct embedding an interface), a +// bodiless declaration, or a library method's external (a promoted library +// method). It never adds a call_graph edge (only VTA's callees are edges). +func (d *dispatch) addCHA(c *cha, funcs map[*ssa.Function]bool) { + for fn := range funcs { + forCalls(fn, func(call ssa.CallInstruction) { + cc := call.Common() + if !cc.IsInvoke() || cc.Method == nil { + return + } + if _, isParam := types.Unalias(cc.Value.Type()).(*types.TypeParam); isParam { + return + } + unit, key, ok := c.j.callSite(call) + if !ok || !c.libraryOrigin(cc.Value, map[ssa.Value]bool{}) { + return + } + site := siteKey{unit, key} + for _, t := range c.implementers(cc.Value.Type()) { + sel := c.msets.MethodSet(t).Lookup(cc.Method.Pkg(), cc.Method.Name()) + if sel == nil { + continue + } + if tg, ok := d.funcTarget(c.j, sel.Obj().(*types.Func)); ok { + d.add(site, tg) + } + } + }) + } +} + +// funcTarget maps a declared function or method to its target, as +// calleeTarget maps its SSA function: the unit it declares, an interface +// method, a bodiless declaration, or a library function's external. A +// universe method (error.Error, promoted through an embedded error) has no +// package and no target, as for methodTarget. +func (d *dispatch) funcTarget(j *join, obj *types.Func) (Target, bool) { + if obj.Pkg() == nil { + return Target{}, false + } + if j.isLibrary(obj) { + sig := semantic_analysis.CalleeSignatureOf(obj) + d.external(obj, sig) + return Target{External: sig}, true + } + k, ok := declKey(j.fset, j.projectDir, obj.Origin()) + if !ok { + return Target{}, false + } + if _, unit := j.units[k]; unit || j.iface[k] || j.bodiless[k] { + return Target{Callable: k}, true + } + return Target{}, false +} + +// libraryOrigin reports whether v, an interface call's receiver, may hold a +// value whose dynamic type VTA cannot see: following Phi edges, Extract, +// ChangeInterface and loads back to +// +// - a TypeAssert (the asserted value came from anywhere); +// - the result of a call to a library function or of an interface call of +// a library interface's method (its callee may be library code); +// - a load of a library global, or through a library struct's field, or +// through a pointer that itself originates in library code. +// +// A value project code built (MakeInterface), a parameter, a free variable +// or a load of project storage is VTA's: false. The remaining gap is a +// library-originated value first stored in a project struct field or +// global and then read back (spec §4.8). +func (c *cha) libraryOrigin(v ssa.Value, seen map[ssa.Value]bool) bool { + if seen[v] { + return false + } + seen[v] = true + switch v := v.(type) { + case *ssa.Phi: + for _, e := range v.Edges { + if c.libraryOrigin(e, seen) { + return true + } + } + case *ssa.Extract: + return c.libraryOrigin(v.Tuple, seen) + case *ssa.ChangeInterface: + return c.libraryOrigin(v.X, seen) + case *ssa.TypeAssert: + return true + case *ssa.Call: + return c.libraryCall(v.Common()) + case *ssa.UnOp: + if v.Op == token.MUL { + return c.libraryLoad(v.X, seen) + } + case *ssa.Field: + if c.libraryType(v.X.Type()) { + return true + } + return c.libraryOrigin(v.X, seen) + } + return false +} + +// libraryCall reports whether cc calls a library function, or a library +// interface's method. +func (c *cha) libraryCall(cc *ssa.CallCommon) bool { + if cc.IsInvoke() { + return cc.Method != nil && cc.Method.Pkg() != nil && c.j.isLibrary(cc.Method) + } + callee := cc.StaticCallee() + if callee == nil { + return false + } + obj, ok := callee.Object().(*types.Func) + return ok && obj.Pkg() != nil && c.j.isLibrary(obj) +} + +// libraryLoad reports whether a load through the pointer x reads library +// storage, or a pointer that originates in library code. +func (c *cha) libraryLoad(x ssa.Value, seen map[ssa.Value]bool) bool { + switch x := x.(type) { + case *ssa.Global: + return x.Pkg != nil && !c.j.projPaths[x.Pkg.Pkg.Path()] + case *ssa.FieldAddr: + if ptr, ok := x.X.Type().Underlying().(*types.Pointer); ok && c.libraryType(ptr.Elem()) { + return true + } + return c.libraryOrigin(x.X, seen) + case *ssa.IndexAddr: + return c.libraryOrigin(x.X, seen) + } + return c.libraryOrigin(x, seen) +} + +// libraryType reports whether t is a named type declared outside the +// project. +func (c *cha) libraryType(t types.Type) bool { + named, ok := types.Unalias(t).(*types.Named) + if !ok { + return false + } + pkg := named.Obj().Pkg() + return pkg != nil && !c.j.projPaths[pkg.Path()] +} diff --git a/internal/sdg/cha_test.go b/internal/sdg/cha_test.go new file mode 100644 index 0000000..01f7d5b --- /dev/null +++ b/internal/sdg/cha_test.go @@ -0,0 +1,83 @@ +package sdg + +import ( + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +const libstorePkg = "example.com/dataflow/libstore" + +// libstoreKey is the L3Key of a callable declared in +// testdata/dataflow/libstore/libstore.go. +func libstoreKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("libstore", "libstore.go"), Signature: libstorePkg + "." + name} +} + +func libstoreTarget(name string) Target { return Target{Callable: libstoreKey(name)} } + +// TestCHA_LibraryOriginatedReceivers (F3): HandleCtx's +// ctx.Value(k).(Store).Find(id) and HandleMap's v.(Store).Find(id) call Find +// on a value that came out of library storage (context.Context, sync.Map): +// VTA sees no concrete type, so the site's targets include every project +// type implementing Store — sqlStore.Find and memStore.Find, and the Find of +// any other package's type with the same method set — beside the interface +// method (D7). The callers' footprints see the global +// sqlStore.Find writes. HandleLocal's s.Find(id), on a value project code +// built, gets VTA's memStore.Find and no CHA target. +func TestCHA_LibraryOriginatedReceivers(t *testing.T) { + in, res, d := runDispatch(t) + all := []Target{libstoreTarget("Store.Find"), libstoreTarget("memStore.Find"), libstoreTarget("sqlStore.Find")} + fps := footprints(in, d) + for _, c := range []struct{ fn, site string }{ + {"HandleCtx", "46:9"}, + {"HandleMap", "59:9"}, + } { + got := d.targets(libstoreKey(c.fn), c.site) + for _, w := range all { + if !slices.Contains(got, w) { + t.Errorf("targets(%s, %s) = %v, want %v among them", c.fn, c.site, got, w) + } + } + // Every project type with Find(string) string implements Store + // structurally (wrap's too): each target is some type's Find. + for _, tg := range got { + if !strings.HasSuffix(tg.Callable.Signature, ".Find") { + t.Errorf("targets(%s, %s) has %v, not a Find", c.fn, c.site, tg) + } + } + if fp := fps[libstoreTarget(c.fn)]; !slices.Contains(fp.GlobalWrites, libstorePkg+"::lastQuery") { + t.Errorf("%s writes %v, want %s::lastQuery", c.fn, fp.GlobalWrites, libstorePkg) + } + } + want := []Target{libstoreTarget("Store.Find"), libstoreTarget("memStore.Find")} + if got := d.targets(libstoreKey("HandleLocal"), "72:9"); !slices.Equal(got, want) { + t.Errorf("targets(HandleLocal, 72:9) = %v, want %v (no CHA target for a project-built value)", got, want) + } + // CHA targets are not VTA's: no call_graph edge (F7). + for _, e := range res.CallEdges { + if (e.Source == libstorePkg+".HandleCtx" || e.Source == libstorePkg+".HandleMap") && e.Target != "" { + t.Errorf("call_graph edge %s -> %s for a CHA target", e.Source, e.Target) + } + } +} + +// TestCHA_UniverseMethodNoTarget: HandleErr's context.Cause(ctx).Error() +// is an interface call on a library call's result; errStore embeds the +// universe error, so its method set's Error has no package. The CHA +// complement adds no target for it (and does not panic): the site keeps +// what VTA and D7 give it. +func TestCHA_UniverseMethodNoTarget(t *testing.T) { + _, _, d := runDispatch(t) + if _, ok := d.sites[siteKey{libstoreKey("HandleErr"), "95:9/2"}]; !ok { + t.Fatal("HandleErr's context.Cause call has no targets: the site keys moved") + } + for _, tg := range d.targets(libstoreKey("HandleErr"), "95:9") { + if tg.Callable.Signature != "" { + t.Errorf("targets(HandleErr, 95:9) has %v, want no in-project target", tg) + } + } +} diff --git a/internal/sdg/run.go b/internal/sdg/run.go index 51be8f7..43ba730 100644 --- a/internal/sdg/run.go +++ b/internal/sdg/run.go @@ -97,6 +97,7 @@ func dispatchRun(in Input) (Result, *dispatch) { deg = append(deg, *dg) } d.addInvokes(j, funcs) + d.addCHA(newCHA(j, in), funcs) start = time.Now() cg, err := runVTA(all) diff --git a/internal/sdg/ssa_test.go b/internal/sdg/ssa_test.go index 5144cdb..7297f0f 100644 --- a/internal/sdg/ssa_test.go +++ b/internal/sdg/ssa_test.go @@ -54,7 +54,8 @@ func loadInputOpts(t *testing.T, name string, skipTests bool) Input { // skipped), sorted. var dataflowPkgs = []string{ "example.com/dataflow/asm", "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/httpsql", "example.com/dataflow/initclosure", "example.com/dataflow/inits", - "example.com/dataflow/l1", "example.com/dataflow/linedir", "example.com/dataflow/sdg", "example.com/dataflow/variant/p", "example.com/dataflow/variant/u", "example.com/dataflow/vend", + "example.com/dataflow/l1", "example.com/dataflow/libstore", + "example.com/dataflow/linedir", "example.com/dataflow/sdg", "example.com/dataflow/variant/p", "example.com/dataflow/variant/u", "example.com/dataflow/vend", "example.com/dataflow/wrap", "example.com/dataflow/xfile", "example.com/dataflow/xtest", "example.com/dataflow/xtest/user", } diff --git a/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go b/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go index e813ee4..687448a 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go @@ -56,3 +56,29 @@ func TestGateL4_FlowBehindSyntheticWrappers(t *testing.T) { } } } + +// TestGateL4_FlowThroughLibraryStorage (F3): on testdata/dataflow/libstore, +// a Store put in a context.Context or a sync.Map comes back out of library +// code and is called; each handler's id reaches Exec's query formal through +// the CHA complement's sqlStore.Find, and the handler's formal_out of the +// global sqlStore.Find writes exists. +func TestGateL4_FlowThroughLibraryStorage(t *testing.T) { + a := buildL4(t, "dataflow/libstore").l4 + if len(a.Degraded) != 0 { + t.Fatalf("libstore degraded: %+v", a.Degraded) + } + mod := v2.ModuleID(a.Application.ID, "libstore.go") + pkg := "example.com/dataflow/libstore." + fn := func(name string) string { return v2.CallableID(mod, pkg+name) } + sink := l4Vertex(t, a, fn("Exec"), "formal_in", "query") + g := newSDGGraph(a) + for _, h := range []string{"HandleCtx", "HandleMap"} { + src := l4Vertex(t, a, fn(h), "formal_in", "id") + if p := g.path(src, sink); p == nil { + t.Errorf("%s's id does not reach Exec's query", h) + } + if !hasVertex(a, fn(h), "formal_out", ":example.com/dataflow/libstore::lastQuery") { + t.Errorf("%s has no formal_out of the global sqlStore.Find writes", h) + } + } +} diff --git a/testdata/dataflow/libstore/libstore.go b/testdata/dataflow/libstore/libstore.go new file mode 100644 index 0000000..2580904 --- /dev/null +++ b/testdata/dataflow/libstore/libstore.go @@ -0,0 +1,96 @@ +// Package libstore is the L4 fixture for dynamic types that pass through +// library storage (spec §4.8, CHA complement): a Store put in a +// context.Context or a sync.Map comes back out of library code, which has +// no SSA bodies, so VTA sees no concrete type at the interface call. The +// call's targets then include every project type implementing Store. An +// interface call on a value project code built (HandleLocal) keeps VTA's +// targets only. Tests pin keys by line:col, so keep each statement on its +// own line and append new functions at the end. +package libstore + +import ( + "context" + "sync" +) + +// Store is the interface the handlers call. +type Store interface { + Find(id string) string +} + +// lastQuery is the global sqlStore.Find writes. +var lastQuery string + +type sqlStore struct{} + +// Find builds the query from id and runs it: the sink is Exec's query. +func (s *sqlStore) Find(id string) string { + lastQuery = id + return Exec("select " + id) +} + +// Exec is the sink. +func Exec(query string) string { + return query +} + +type storeKey struct{} + +// WithStore puts s in ctx. +func WithStore(ctx context.Context, s Store) context.Context { + return context.WithValue(ctx, storeKey{}, s) +} + +// HandleCtx takes the Store back out of ctx and calls Find. +func HandleCtx(ctx context.Context, id string) string { + return ctx.Value(storeKey{}).(Store).Find(id) +} + +var stores sync.Map + +// Register puts s in the stores map. +func Register(name string, s Store) { + stores.Store(name, s) +} + +// HandleMap loads the Store from the stores map and calls Find. +func HandleMap(name, id string) string { + v, _ := stores.Load(name) + return v.(Store).Find(id) +} + +type memStore struct{} + +// Find returns id. +func (memStore) Find(id string) string { + return id +} + +// HandleLocal calls Find on a Store project code built: VTA sees memStore. +func HandleLocal(id string) string { + var s Store = memStore{} + return s.Find(id) +} + +// MainCtx and MainMap each store a sqlStore in library storage and call one +// handler: no host variable is shared between the handlers' call sites. +func MainCtx() { + ctx := WithStore(context.Background(), &sqlStore{}) + HandleCtx(ctx, "a") +} + +func MainMap() { + Register("db", &sqlStore{}) + HandleMap("db", "b") +} + +// errStore embeds the universe error: its Error method has no package. +type errStore struct { + error +} + +// HandleErr calls Error on an error that came out of a library call: the +// CHA complement meets errStore, whose Error is the universe method. +func HandleErr(ctx context.Context) string { + return context.Cause(ctx).Error() +} From 0767221aac61e15c05cbb1fac92b5e3bd7eb2c0a Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 19:20:44 -0400 Subject: [PATCH 53/65] fix(l4): emit partial_types The documented code was never produced. It lists the project packages go/packages loaded with errors, sorted. --- internal/core/sdg_test.go | 2 +- internal/sdg/ir.go | 13 +++++++------ internal/sdg/join.go | 22 ++++++++++++++++++++++ internal/sdg/join_test.go | 17 ++++++++++------- internal/sdg/run.go | 10 ++++++++-- 5 files changed, 48 insertions(+), 16 deletions(-) diff --git a/internal/core/sdg_test.go b/internal/core/sdg_test.go index e69d61b..b517072 100644 --- a/internal/core/sdg_test.go +++ b/internal/core/sdg_test.go @@ -71,7 +71,7 @@ func TestAnalyze_Level4StrictFailsOnDegraded(t *testing.T) { if err == nil { t.Fatal("--strict -a 4 on a degraded project succeeded") } - for _, code := range []string{"partial_ssa", "no_ssa"} { + for _, code := range []string{"partial_types", "partial_ssa", "no_ssa"} { if !strings.Contains(err.Error(), code) { t.Errorf("error %q does not name %s", err, code) } diff --git a/internal/sdg/ir.go b/internal/sdg/ir.go index e32fd17..776e59d 100644 --- a/internal/sdg/ir.go +++ b/internal/sdg/ir.go @@ -107,10 +107,11 @@ type Models map[string][][2]string // Degraded codes (spec §4.15; an open vocabulary of plain strings). const ( - CodeNoSSA = "no_ssa" - CodePartialSSA = "partial_ssa" - CodeNoVTA = "no_vta" - CodeUnmappedSSA = "unmapped_ssa" - CodeL3Skipped = "l3_skipped" - CodeRangeFunc = "range_func" + CodeNoSSA = "no_ssa" + CodePartialSSA = "partial_ssa" + CodePartialTypes = "partial_types" + CodeNoVTA = "no_vta" + CodeUnmappedSSA = "unmapped_ssa" + CodeL3Skipped = "l3_skipped" + CodeRangeFunc = "range_func" ) diff --git a/internal/sdg/join.go b/internal/sdg/join.go index 8f47eaf..4fba7d9 100644 --- a/internal/sdg/join.go +++ b/internal/sdg/join.go @@ -417,6 +417,28 @@ func homePath(fn *ssa.Function) string { return "" } +// partialTypes is the partial_types entry (spec §4.15): the project +// packages (every variant, by path) go/packages loaded with errors — list, +// parse or type errors — or that are ill-typed through a broken dependency, +// sorted; nil when every package loaded clean. Their types (and so L1–L3 +// over them) are partial; partial_ssa says which also got no SSA. +func partialTypes(in Input) *Degraded { + pkgs := map[string]bool{} + for _, p := range projectPackages(in) { + if len(p.Errors) > 0 || p.IllTyped { + pkgs[p.PkgPath] = true + } + } + if len(pkgs) == 0 { + return nil + } + return &Degraded{ + Code: CodePartialTypes, + Detail: fmt.Sprintf("%d package(s) loaded with errors (list, parse or type errors, or a broken dependency)", len(pkgs)), + Packages: sortedKeys(pkgs), + } +} + // l3Skipped is the l3_skipped entry (spec §4.15, A16): the units with no L3 // result (a recovered panic, or a body call with no L1 key), or nil. func l3Skipped(in Input) *Degraded { diff --git a/internal/sdg/join_test.go b/internal/sdg/join_test.go index a056b8b..74959fc 100644 --- a/internal/sdg/join_test.go +++ b/internal/sdg/join_test.go @@ -653,19 +653,22 @@ func TestSDG_L3SkippedDegrades(t *testing.T) { } } -// TestSDG_IllTypedPackageDegrades (Review Focus 1): testdata/typeerr does not -// type-check, so it gets no SSA: partial_ssa lists it (then no_ssa), nothing -// panics, and its L3 results are those of -a 3. +// TestSDG_IllTypedPackageDegrades (Review Focus 1, F4): testdata/typeerr +// does not type-check: partial_types lists it (its go/packages errors), it +// gets no SSA (partial_ssa, then no_ssa), nothing panics, and its L3 +// results are those of -a 3. func TestSDG_IllTypedPackageDegrades(t *testing.T) { in := loadInput(t, "typeerr") plain := dataflow.Run(in.Fset, in.Units, 3, 1) res := Run(in) codes := codesOf(res.Degraded) - if !slices.Equal(codes, []string{CodePartialSSA, CodeNoSSA}) { - t.Fatalf("codes = %v, want [partial_ssa no_ssa]", codes) + if !slices.Equal(codes, []string{CodePartialTypes, CodePartialSSA, CodeNoSSA}) { + t.Fatalf("codes = %v, want [partial_types partial_ssa no_ssa]", codes) } - if !slices.Equal(res.Degraded[0].Packages, []string{"example.com/typeerr"}) { - t.Errorf("partial_ssa packages = %v", res.Degraded[0].Packages) + for _, d := range res.Degraded[:2] { + if !slices.Equal(d.Packages, []string{"example.com/typeerr"}) { + t.Errorf("%s packages = %v", d.Code, d.Packages) + } } if len(in.L3) == 0 || len(in.L3) != len(plain) { t.Fatalf("L3 has %d results, -a 3 has %d", len(in.L3), len(plain)) diff --git a/internal/sdg/run.go b/internal/sdg/run.go index 43ba730..3cca6eb 100644 --- a/internal/sdg/run.go +++ b/internal/sdg/run.go @@ -35,7 +35,8 @@ const l4GCPercent = 50 // // Order: the join is computed before VTA, so it never reads a program a VTA // panic may have wedged; after no_vta, later stages use the program for the -// join alone. The degradations are, in order: partial_ssa / no_ssa (SSA), +// join alone. The degradations are, in order: partial_types (the load), +// partial_ssa / no_ssa (SSA), // l3_skipped and range_func (from the units, whether or not SSA built), // unmapped_ssa (the join), no_vta. // @@ -68,8 +69,13 @@ func run(in Input) (Result, *dispatch) { // dispatchRun is run up to the dispatch: SSA, the join and VTA. func dispatchRun(in Input) (Result, *dispatch) { d := newDispatch(in) + var deg []Degraded + if dg := partialTypes(in); dg != nil { + deg = append(deg, *dg) + } start := time.Now() - prog, pkgs, deg := buildSSA(in) + prog, pkgs, ssaDeg := buildSSA(in) + deg = append(deg, ssaDeg...) utils.Info("L4: SSA built %d project packages in %s", len(pkgs), time.Since(start).Round(time.Millisecond)) for _, dg := range []*Degraded{l3Skipped(in), rangeFunc(in)} { if dg != nil { From 1be164329ca0e3b9c2b1b2fb1be0a20ec6687f12 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 19:24:55 -0400 Subject: [PATCH 54/65] fix(neo4j): clean push clears a stale degraded_json SET n += row.p keeps a property the row lacks, so a clean push over a graph a degraded run wrote kept its degraded_json. The application row now carries an explicit null. Also: Degraded.Code is an open vocabulary. --- internal/neo4j/bolt.go | 17 ++++++++- internal/neo4j/bolt_integration_test.go | 51 +++++++++++++++++++++++++ internal/neo4j/bolt_session_test.go | 47 +++++++++++++++++++++++ internal/schema/v2/schema.go | 4 +- 4 files changed, 116 insertions(+), 3 deletions(-) diff --git a/internal/neo4j/bolt.go b/internal/neo4j/bolt.go index 9bf4fb1..60d2916 100644 --- a/internal/neo4j/bolt.go +++ b/internal/neo4j/bolt.go @@ -433,10 +433,25 @@ func (w *boltWriter) read(fn func(neo4j.ManagedTransaction) error) error { // nodeParams converts node rows to driver params. Props map directly (the // driver stores Go ints as integers, strings as strings, []string as a list). +// clearedAppProps are the :GoApplication props a push writes as an +// explicit null when the projection pruned them: `SET n += row.p` keeps a +// property row.p lacks, so a clean run over a graph a degraded run wrote +// would otherwise keep the stale degraded_json (F6). A null in SET += +// removes the property. (The snapshot script wipes the application first.) +var clearedAppProps = []string{"degraded_json"} + func nodeParams(g []NodeRow) []map[string]any { out := make([]map[string]any, len(g)) for i, n := range g { - out[i] = map[string]any{"k": n.Value, "p": propsToAny(n.Props)} + p := propsToAny(n.Props) + if len(n.Labels) > 0 && n.Labels[0] == LabelApplication { + for _, k := range clearedAppProps { + if _, ok := p[k]; !ok { + p[k] = nil + } + } + } + out[i] = map[string]any{"k": n.Value, "p": p} } return out } diff --git a/internal/neo4j/bolt_integration_test.go b/internal/neo4j/bolt_integration_test.go index f5b2c12..d99671d 100644 --- a/internal/neo4j/bolt_integration_test.go +++ b/internal/neo4j/bolt_integration_test.go @@ -250,3 +250,54 @@ func TestIntegration_LazyL4PushOverLowerLevelGraph(t *testing.T) { } } } + +// TestIntegration_CleanLazyPushClearsDegradedJSON (F6): a graph a degraded +// run wrote carries :GoApplication.degraded_json; a clean lazy push of the +// same app removes it (the application row's explicit null), so the SDK no +// longer reads a stale degradation. +func TestIntegration_CleanLazyPushClearsDegradedJSON(t *testing.T) { + ctx := context.Background() + cfg := itConfig(t) + a := buildDataflowL4(t) + if len(a.Degraded) != 0 { + t.Fatalf("fixture degraded: %+v", a.Degraded) + } + degraded := Project(a) + for i, n := range degraded.Nodes { + if n.Labels[0] == LabelApplication { + p := Props{} + for k, v := range n.Props { + p[k] = v + } + p["degraded_json"] = `[{"code":"no_vta"}]` + degraded.Nodes[i].Props = p + } + } + seed := cfg + seed.Eager, seed.FullRun = true, true + if err := WriteBolt(ctx, degraded, a.Application.ID, seed); err != nil { + t.Fatalf("seed: %v", err) + } + read := func() any { + drv, err := neo4j.NewDriverWithContext(cfg.URI, neo4j.BasicAuth(cfg.User, cfg.Password, "")) + if err != nil { + t.Fatalf("driver: %v", err) + } + defer drv.Close(ctx) + res, err := neo4j.ExecuteQuery(ctx, drv, "MATCH (a:GoApplication {id: $id}) RETURN a.degraded_json", + map[string]any{"id": a.Application.ID}, neo4j.EagerResultTransformer, neo4j.ExecuteQueryWithDatabase(cfg.Database)) + if err != nil || len(res.Records) != 1 { + t.Fatalf("read degraded_json: %v (%d records)", err, len(res.Records)) + } + return res.Records[0].Values[0] + } + if v := read(); v != `[{"code":"no_vta"}]` { + t.Fatalf("seeded degraded_json = %v", v) + } + if err := WriteBolt(ctx, Project(a), a.Application.ID, cfg); err != nil { + t.Fatalf("lazy push: %v", err) + } + if v := read(); v != nil { + t.Errorf("clean lazy push left degraded_json = %v", v) + } +} diff --git a/internal/neo4j/bolt_session_test.go b/internal/neo4j/bolt_session_test.go index d73f0bb..1cc8d6f 100644 --- a/internal/neo4j/bolt_session_test.go +++ b/internal/neo4j/bolt_session_test.go @@ -339,3 +339,50 @@ func TestBolt_LazyPushForcesFullOnUnreadableStoredLevel(t *testing.T) { } } } + +// TestBolt_CleanPushClearsDegradedJSON (F6): the projector prunes a nil +// degraded_json, and `SET n += row.p` keeps a property row.p lacks, so a +// clean push over a graph an earlier degraded run wrote would leave the +// stale degraded_json behind (and the SDK would refuse "no flow" +// certificates). The application row therefore carries an explicit null +// when nothing is degraded, which SET += removes; a degraded run keeps its +// JSON string. +func TestBolt_CleanPushClearsDegradedJSON(t *testing.T) { + a := buildDataflowL4(t) + if len(a.Degraded) != 0 { + t.Fatalf("fixture degraded: %+v", a.Degraded) + } + appProps := func(rows GraphRows) map[string]any { + t.Helper() + db := &fakeDB{level: 4} + w := &boltWriter{ctx: context.Background(), session: db, rows: rows, appID: a.Application.ID} + if err := w.run(); err != nil { + t.Fatal(err) + } + for _, st := range db.writes { + if !strings.Contains(st.q, "MERGE (n:"+LabelApplication+" ") { + continue + } + for _, r := range st.params["rows"].([]map[string]any) { + if r["k"] == a.Application.ID { + return r["p"].(map[string]any) + } + } + } + t.Fatal("no :GoApplication upsert") + return nil + } + p := appProps(Project(a)) + if v, has := p["degraded_json"]; !has || v != nil { + t.Errorf("clean push: degraded_json = %v (present %v), want an explicit null", v, has) + } + rows := Project(a) + for i, n := range rows.Nodes { + if n.Labels[0] == LabelApplication { + rows.Nodes[i].Props = Props{"degraded_json": `[{"code":"no_vta"}]`, "max_level": int64(4)} + } + } + if v := appProps(rows)["degraded_json"]; v != `[{"code":"no_vta"}]` { + t.Errorf("degraded push: degraded_json = %v, want the JSON string", v) + } +} diff --git a/internal/schema/v2/schema.go b/internal/schema/v2/schema.go index 81b5b0d..2decfd9 100644 --- a/internal/schema/v2/schema.go +++ b/internal/schema/v2/schema.go @@ -63,8 +63,8 @@ type Analysis struct { } // Degraded is one degradation marker: an L4 step that fell back to a coarser -// result. Code is a closed vocabulary; Detail, Packages and Callables say what -// was affected. +// result. Code is an open vocabulary of plain strings (spec §4.15); Detail, +// Packages and Callables say what was affected. type Degraded struct { Code string `json:"code"` Detail string `json:"detail,omitempty"` From ce41831224881e8bf43cf22505615774885a7fae Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 19:27:21 -0400 Subject: [PATCH 55/65] docs: PR #16 merged, L4 live; partial_types and SSA variants in the spec; stale comments --- README.md | 10 ++++---- docs/design/specs/v2-l3-l4-dataflow.md | 24 ++++++++++++------- internal/syntactic_analysis/v2emit/emit_l3.go | 5 ++-- testdata/dataflow/linedir/linedir.go | 6 ++--- 4 files changed, 27 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 0e949be..b132e47 100644 --- a/README.md +++ b/README.md @@ -324,10 +324,12 @@ subset of `-a 4` output, and `-j 1` and `-j N` output are byte-identical. `/` tail, and a value is a list of `[in, out]` pairs of `of` names (`[]` = no summary edges). A model changes summaries, **not statement-level reachability**: in `n, err := strconv.Atoi(id)` the value assigned at the call site keeps its reaching-definition - edge from `id`, which is correct data flow. A model is therefore not a sanitizer. Cutting such a - flow is the SDK taint analysis's job ([codellm-devkit/.github#103](https://github.com/codellm-devkit/.github/issues/103)). A variable cut on `n` is also bypassed, by - implicit flow through `err` and by the coarse host wiring. An entry naming an `of` the external lacks falls back to all-to-all, with a warning. A - key no call site targets is ignored, with a warning. The flag is valid only at `-a 4` and with + edge from `id`, which is correct data flow. A model is therefore not a sanitizer. Cutting such + a flow is the SDK taint analysis's job + ([codellm-devkit/.github#103](https://github.com/codellm-devkit/.github/issues/103)). A variable + cut on `n` is also bypassed, by implicit flow through `err` and by the coarse host wiring. An + entry naming an `of` the external lacks falls back to all-to-all, with a warning. A key no call + site targets is ignored, with a warning. The flag is valid only at `-a 4` and with `--emit neo4j`. - **`-t` / `--target-files` is not supported at `-a 4`** (whole-program): it is an error. Use `-a 3 -t`, or run without `-t`. diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index 4dd428f..2ed5b75 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -1,8 +1,8 @@ # Spec: Go L3/L4 dataflow (CFG/CDG/DDG + SDG) with SDK taint (codeanalyzer-go) -**Status:** Phases 0–1 implemented (PR #16, open, not yet merged); Phase 2 (L4) implemented on +**Status:** Phases 0–1 merged (PR #16, merge commit `f9a39bb`); Phase 2 (L4) live on `feat/l4-sdg` and evaluated (`docs/design/notes/l3-dataflow-metrics.md` § L4); L4 design amended by -the 2026-10-25 re-check (§4.16) and the 2026-10-26/27 rulings · +the 2026-10-25 re-check (§4.16), the 2026-10-26/27 rulings and the final-review fixes (§4.8) · **Date:** 2026-10-23, amended 2026-10-25, 2026-10-27 · **Repos:** `codeanalyzer-go`, `python-sdk` · **Supersedes:** the "L3/L4 deferred" entries in `docs/design/roadmap.md`. @@ -88,7 +88,7 @@ siblings is required too; and the id grammar has to change for polyglot apps any As of 2026-10-25 (code at `3504445`, branch `feat/l3-dataflow` = `feat/l4-sdg`): -- **Phases 0 and 1 are implemented together on PR #16** (§9; open, not yet merged). L1/L2/L3 on +- **Phases 0 and 1 were implemented together on PR #16** (§9; merged as `f9a39bb`). L1/L2/L3 on v2, Neo4j projection live through L3 (as of this snapshot; L4 is now live too, Status header). - **Phase 0 (live):** `can:///go/…` ids (§4.1, `internal/schema/v2/ids.go`); L2 library calls with `application.external_symbols` (§4.14); co-positioned call keys `l:c/n`; `is_deferred`; @@ -590,8 +590,12 @@ built per D2 (amended, A12): `ssautil.Packages` over the **same** `go/packages` - **SSA construction (amended, A12; D2) [call].** The load is `NeedSyntax|NeedTypesInfo|NeedDeps`, so `AllPackages` would create, and `Build` would compile, **every dependency body** (stdlib included). Instead: - - `ssautil.Packages(roots, ssa.InstantiateGenerics|ssa.BuildSerially)` over `builder.Pkgs()`: - bodies only for project packages; dependencies come from types. + - `ssautil.Packages(roots, ssa.InstantiateGenerics|ssa.BuildSerially)` over every loaded + project package variant (`builder.Loaded()`): bodies only for project packages; dependencies + come from types. With tests loaded, `builder.Pkgs()` (keyed by path) holds only the test + variant `p [p.test]`; the plain `p` other packages import is a root too, or VTA misses the + types that flow through it [final review F2]. Both variants share syntax and join the same + units. - Build **per package** (`pkg.Build()`) in our own goroutine under `recover` — `Program.Build` and VTA panic inside their own goroutines, out of reach of the per-callable `recover`. A panic degrades (`partial_ssa` / `no_ssa`, §4.15). @@ -890,7 +894,9 @@ callables) but marks the envelope: - `no_vta` — VTA failed (incl. a recovered VTA panic). - `partial_ssa` — listed packages had type errors (`pkg.IllTyped`) or a recovered per-package build panic, so no SSA. - - `partial_types` — packages with `go/packages` errors at any level. + - `partial_types` — project packages `go/packages` loaded with errors (list, parse or type + errors, or ill-typed through a broken dependency), every test variant included; `packages[]` + sorted. A package also listed under `partial_ssa` got no SSA. - `unmapped_ssa` — SSA functions or call instructions that map to no callable or call node (§4.8); counted, listed where nameable (A16). - `l3_skipped` — callables whose L3 result was dropped (contained panic, call-count mismatch); @@ -1161,11 +1167,11 @@ JSON. The full results, insights and limitations are in `l3-dataflow-metrics.md` | Phase | Repo | Content | Release | -> **Phases 0 and 1 ship together in one PR / one release** [user, 2026-10-23]; they stay separate rows for traceability. **Both are implemented, on PR #16 (open, not yet merged).** +> **Phases 0 and 1 ship together in one PR / one release** [user, 2026-10-23]; they stay separate rows for traceability. **Both are implemented and merged (PR #16, `f9a39bb`); Phase 2 (L4) is live on `feat/l4-sdg`.** |---|---|---|---| -| 0 | analyzer | **implemented (PR #16, open).** id grammar `can:///go/…`; **L2 external calls (§4.14)**; L1 co-positioned call keys; `is_deferred`; `typeKind` alias/defined; `GO_DDG` `_k`; call-site position index (perf) | analyzer next **breaking** version (wheel is at 0.1.0 with no local tags — builder confirms the released version and bumps the major, or the minor if still 0.x) | -| 1 | analyzer | **implemented (PR #16, open).** L3: body statements, CFG (+defer chain, panic), CDG, syntactic DDG, CLI `-a 3`/`--graph-field-depth`, Neo4j CFG/CDG/DDG, gates | minor | +| 0 | analyzer | **merged (PR #16, `f9a39bb`).** id grammar `can:///go/…`; **L2 external calls (§4.14)**; L1 co-positioned call keys; `is_deferred`; `typeKind` alias/defined; `GO_DDG` `_k`; call-site position index (perf) | analyzer next **breaking** version (wheel is at 0.1.0 with no local tags — builder confirms the released version and bumps the major, or the minor if still 0.x) | +| 1 | analyzer | **merged (PR #16, `f9a39bb`).** L3: body statements, CFG (+defer chain, panic), CDG, syntactic DDG, CLI `-a 3`/`--graph-field-depth`, Neo4j CFG/CDG/DDG, gates | minor | | 2 | analyzer | **Prerequisites, in order (A24):** (1) #26 shared call/closure walker (incl. `//line` spans), first — returns `(*ast.CallExpr, key)`, `(*ast.FuncLit, closure name)` and call → host maps, replacing the index-paired L1/L3 walks; (2) #17 closure call resolution, after #26 — closures carry a declaring `Path` and `call_graph` gains edges from closures; (3) the #22 L3 fixes (D1, §4.5), then the #28 satellite-`parent` tests and a Go 1.23+ fixture — before L4 port wiring (changing L3 after L4 would re-baseline every L4 gate); (4) #27 streaming JSON and AST release — before the real-app L4 eval, can run in parallel with the L4 core; (5) schema and CLI scaffolding (A14, A18, A19, A17 declarations, a `-a 1..4` monotonicity harness) — the first L4 task. Dependencies: #26 → #17 → (VTA, summaries); #22 fixes → port wiring and gates; scaffolding → all of L4; #27 → the Kubernetes eval. **Then L4:** SSA spike (D3 measurement) and build (D2), SSA → L3 join, VTA, `degraded` marker (§4.15), external stand-ins (`external_symbols` `receiver`/`body`), vertices, param edges, summaries, port wiring, type-based alias delta, Neo4j L4 overlay, `--emit neo4j` at L4 (per D4), real-app eval | minor | | 3 | python-sdk | Go v2 models (first the §7 prerequisites: `GoType.kind`, L3 fields), facade, local + Neo4j backends, L1–L4 accessors, taint e2e | minor; pins the analyzer only after Phase 2 is released | diff --git a/internal/syntactic_analysis/v2emit/emit_l3.go b/internal/syntactic_analysis/v2emit/emit_l3.go index 3805138..bc43c28 100644 --- a/internal/syntactic_analysis/v2emit/emit_l3.go +++ b/internal/syntactic_analysis/v2emit/emit_l3.go @@ -63,8 +63,9 @@ func (o l3Overlay) apply(c *v2.Callable, li *lineIndex, key dataflow.L3Key) { // l3BodyNode is a non-L1 body node: its kind, and its span from the node's // [line, col] range (zero for @entry/@exit). A call node reaching here is a -// call L1 did not key — impossible when Run's count check passed — and keeps -// the call shape with a null callee. +// call L1 did not key — impossible once Run paired every body call with an +// L1 key by Lparen (a unit with an unpaired call gets no L3 result) — and +// keeps the call shape with a null callee. func l3BodyNode(n dataflow.Node, li *lineIndex) v2.BodyNode { bn := v2.BodyNode{Kind: string(n.Kind), Parent: n.Parent} if n.Kind != dataflow.KindEntry && n.Kind != dataflow.KindExit { diff --git a/testdata/dataflow/linedir/linedir.go b/testdata/dataflow/linedir/linedir.go index aa68c34..bd2d02d 100644 --- a/testdata/dataflow/linedir/linedir.go +++ b/testdata/dataflow/linedir/linedir.go @@ -1,13 +1,13 @@ // Package linedir has calls under a //line directive, the shape of generated -// code (goyacc, templ): L1 records them at their virtual positions, far past -// the file's real line count, and L2 must still resolve them. +// code (goyacc, templ): L1 records them at their real positions in this file +// (the directive is not applied), and L2 must still resolve them. package linedir import "os" func helper() string { return "" } -// Caller's calls sit on virtual lines of virtual.y. +// Caller's calls sit on virtual lines of virtual.y, real lines 13 and 15. func Caller() { //line virtual.y:900:1 os.Getenv(helper()) From ffda1c27f60af282177801180a1b3a258354e013 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 19:38:16 -0400 Subject: [PATCH 56/65] docs: the CHA complement's origin walk is intraprocedural A library-originated value returned by or passed into a project function is part of the remaining gap, not only one stored in project storage. --- README.md | 7 ++++--- docs/design/notes/l3-dataflow-metrics.md | 5 +++-- docs/design/specs/v2-l3-l4-dataflow.md | 20 ++++++++++++-------- internal/sdg/cha.go | 8 +++++--- 4 files changed, 24 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index b132e47..983b30d 100644 --- a/README.md +++ b/README.md @@ -337,9 +337,10 @@ subset of `-a 4` output, and `-j 1` and `-j N` output are byte-identical. - library code calling back into the project (`sort.Slice`, `http.HandleFunc`) is not an edge; - an interface call on a value that came out of library code (`ctx.Value(k).(Store)`, - `sync.Map.Load`) binds every project type implementing the interface (a targeted CHA - complement), but a value first stored in a project struct field or global and read back - later gets VTA's targets only, which miss the concrete type; + `sync.Map.Load`) in the same function binds every project type implementing the interface + (a targeted CHA complement); a value that first passes through project code (a struct field + or global, a project function's result or parameter) gets VTA's targets only, which miss + the concrete type; - range-over-func is not modelled (`range_func`); - summaries are close to all-to-all, by design; - `summary` edges whose two ends are both `:` vertices are omitted diff --git a/docs/design/notes/l3-dataflow-metrics.md b/docs/design/notes/l3-dataflow-metrics.md index 317e785..ab172ec 100644 --- a/docs/design/notes/l3-dataflow-metrics.md +++ b/docs/design/notes/l3-dataflow-metrics.md @@ -1102,8 +1102,9 @@ over-approximation or reported as degraded. every project type implementing the interface. Measured (go1.26.4, `-j 8`): chi +54 targets at 15 sites (the `w.(http.Flusher)` pattern), param_in 1,895 → 1,961 (+3.5%), param_out 1,649 → 1,770 (+7.3%); nats-server +317 targets at 63 sites, param_in 862,765 → 863,075 - (+0.04%), param_out 431,327 → 431,854 (+0.12%). Remaining gap: a library-originated value - first stored in a project struct field or global, then read back and called, gets VTA's + (+0.04%), param_out 431,327 → 431,854 (+0.12%). Remaining gap (the origin walk is + intraprocedural): a library-originated value that reaches the call through project code — a + project struct field or global, a project function's result or parameter — gets VTA's targets only. - **A15: summaries are near all-to-all** (74–94% of sites, table above). Under the coarse host wiring, a summary edge adds no reachability for the SDK's context-insensitive walk. It is a diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index 2ed5b75..a3746f2 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -573,9 +573,12 @@ built per D2 (amended, A12): `ssautil.Packages` over the **same** `go/packages` implements the interface (`types.Implements`) adds the method its method set selects, mapped by the join. A receiver that originates in project code (a `MakeInterface`, a parameter, a free variable, a load of project storage) gets no CHA target: VTA covers it. Not a `call_graph` edge - (F7). **Remaining gap:** a library-originated value first stored in a project struct field or - global, then read back and called (the load is project storage, so the walk stops), is VTA's - alone and misses the concrete type — recorded in §6 beside library callbacks. + (F7). The origin walk is intraprocedural. **Remaining gap:** a library-originated value that + reaches the call through project code — first stored in a project struct field or global and + read back, or returned by a project function (`func FromCtx(ctx) Store { return + ctx.Value(k).(Store) }`), or passed in as a parameter — is VTA's alone (the walk stops at the + project load, call result or parameter) and misses the concrete type. Recorded in §6 beside + library callbacks. - **Interfaces with no symbol-table node (ruling, Task 10) [call].** `error.Error()` (the universe `error`), methods of interface literals (`interface{ M() }`), and methods promoted through an embedded interface whose origin is declared outside the symbol table have no callable id and no @@ -1018,11 +1021,12 @@ addition now; the cross-language SDK taint revisit is deferred (§11). is not an edge. Taint *starting* in a handler is unaffected (sources are the handler's own params). Recorded limitation. - **Not modelled:** a value that came out of library code (`ctx.Value(k)`, `sync.Map.Load`, a - library call's result), first stored in a project struct field or global and then read back - and called through an interface. The targeted CHA complement (§4.8) covers the call made on the - library-originated value directly; once the value is in project storage the receiver's origin - is a project load, and VTA, which sees no concrete type in library code, finds none. The call - keeps its D7 interface method (pass-through, empty footprint). Recorded limitation. + library call's result) and reaches an interface call through project code: stored in a + project struct field or global and read back, returned by a project function, or passed as a + parameter. The targeted CHA complement (§4.8) covers the call made on the library-originated + value in the same function; otherwise the receiver's origin is a project load, call result or + parameter, and VTA, which sees no concrete type in library code, finds none. The call keeps + its D7 interface method (pass-through, empty footprint). Recorded limitation. - **SDK addressing:** `("query", "database/sql.DB.Query")` resolves through the Go backend's `resolve_callable` (domain = analysed callables ∪ stand-ins) then `resolve_value` over the stand-in body's `formal_in` `of`. Shared `commons/` code is untouched. diff --git a/internal/sdg/cha.go b/internal/sdg/cha.go index 86cd357..4d252b7 100644 --- a/internal/sdg/cha.go +++ b/internal/sdg/cha.go @@ -147,9 +147,11 @@ func (d *dispatch) funcTarget(j *join, obj *types.Func) (Target, bool) { // through a pointer that itself originates in library code. // // A value project code built (MakeInterface), a parameter, a free variable -// or a load of project storage is VTA's: false. The remaining gap is a -// library-originated value first stored in a project struct field or -// global and then read back (spec §4.8). +// or a load of project storage is VTA's: false. The walk is +// intraprocedural, so the remaining gap is a library-originated value that +// reaches the call through project code: stored in a project struct field +// or global and read back, or a project function's result or parameter +// (spec §4.8). func (c *cha) libraryOrigin(v ssa.Value, seen map[ssa.Value]bool) bool { if seen[v] { return false From 7762b6e10611a0ca7fa4e32c19afb4fda546c228 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 21:29:01 -0400 Subject: [PATCH 57/65] fix(l4): library-origin is interprocedural for the CHA complement A library-originated value that reached an interface call through a project function's result or parameter, a closure capture, a field, a global or a container element kept only the D7 interface method. A whole-program monotone fixpoint now tracks it (storage by container type), and map lookups, index and channel receives count as origins. CHA runs under recover: a panic drops its targets and records no_cha. --- internal/sdg/cha.go | 143 ++-- internal/sdg/cha_origin.go | 676 ++++++++++++++++++ internal/sdg/ir.go | 1 + internal/sdg/libflow_test.go | 89 +++ internal/sdg/run.go | 7 +- internal/sdg/ssa_test.go | 2 +- .../v2emit/gate_l4_wrap_test.go | 26 + testdata/dataflow/libflow/libflow.go | 173 +++++ .../vendor/example.com/vlib/box/box.go | 36 + testdata/dataflow/vendor/modules.txt | 1 + 10 files changed, 1061 insertions(+), 93 deletions(-) create mode 100644 internal/sdg/cha_origin.go create mode 100644 internal/sdg/libflow_test.go create mode 100644 testdata/dataflow/libflow/libflow.go create mode 100644 testdata/dataflow/vendor/example.com/vlib/box/box.go diff --git a/internal/sdg/cha.go b/internal/sdg/cha.go index 4d252b7..fc91d95 100644 --- a/internal/sdg/cha.go +++ b/internal/sdg/cha.go @@ -1,8 +1,8 @@ package sdg import ( + "fmt" "go/ast" - "go/token" "go/types" "golang.org/x/tools/go/ssa" @@ -17,7 +17,7 @@ import ( // a sync.Map, returned by a library call, read from a library struct's field // — reaches an interface call with no concrete type VTA can see, and only // the D7 interface method (empty footprint) would remain. For an interface -// call whose receiver originates in library code (libraryOrigin), every +// call whose receiver may carry such a value (origins), every // project type whose method set implements the interface is a target: the // method it resolves to, mapped as the join maps a callee. It reads SSA // instructions and go/types only, never builds code, so it is safe before @@ -78,14 +78,23 @@ func (c *cha) implementers(iface types.Type) []types.Type { return out } -// addCHA adds the CHA complement's targets to every interface call of a -// body-joined project function (funcs) whose receiver originates in library -// code. A target is the method each implementer's method set selects: -// a unit, an interface method (a struct embedding an interface), a -// bodiless declaration, or a library method's external (a promoted library -// method). It never adds a call_graph edge (only VTA's callees are edges). +// addCHA adds the CHA complement's targets (spec §4.8) to every interface +// call of a body-joined project function (funcs) whose receiver may carry +// a library-originated dynamic type: the origins fixpoint over the whole +// program (cha_origin.go). A target is the method each implementer's method +// set selects: a unit, an interface method (a struct embedding an +// interface), a bodiless declaration, or a library method's external (a +// promoted library method). It never adds a call_graph edge (only VTA's +// callees are edges). Functions are visited in a sorted order and targets +// are kept sorted (d.add), so the result is deterministic. func (d *dispatch) addCHA(c *cha, funcs map[*ssa.Function]bool) { - for fn := range funcs { + o := newOrigins(c, funcs) + o.solve() + for _, fn := range o.order { + if chaHook != nil { + chaHook(fn) + } + var lo map[ssa.Value]bool forCalls(fn, func(call ssa.CallInstruction) { cc := call.Common() if !cc.IsInvoke() || cc.Method == nil { @@ -95,7 +104,13 @@ func (d *dispatch) addCHA(c *cha, funcs map[*ssa.Function]bool) { return } unit, key, ok := c.j.callSite(call) - if !ok || !c.libraryOrigin(cc.Value, map[ssa.Value]bool{}) { + if !ok { + return + } + if lo == nil { + lo = o.eval(fn) // o.cur is nil: no reader is registered + } + if !lo[cc.Value] { return } site := siteKey{unit, key} @@ -112,6 +127,34 @@ func (d *dispatch) addCHA(c *cha, funcs map[*ssa.Function]bool) { } } +// runCHA is addCHA under recover (N1): it reads go/types and SSA in ways +// VTA does not, so an unforeseen shape must not abort the analyzer. On a +// panic every CHA target is dropped — the dispatch is restored to what it +// held before — and the error is returned (degraded no_cha). +func (d *dispatch) runCHA(j *join, in Input, funcs map[*ssa.Function]bool) (err error) { + saved := make(map[siteKey][]Target, len(d.sites)) + for k, ts := range d.sites { + saved[k] = ts + } + newExt := make(map[string]bool, len(d.newExt)) + for k := range d.newExt { + newExt[k] = true + } + defer func() { + if r := recover(); r != nil { + d.sites = saved + for k := range d.newExt { + if !newExt[k] { + delete(d.newExt, k) + } + } + err = fmt.Errorf("panic: %v", r) + } + }() + d.addCHA(newCHA(j, in), funcs) + return nil +} + // funcTarget maps a declared function or method to its target, as // calleeTarget maps its SSA function: the unit it declares, an interface // method, a bodiless declaration, or a library function's external. A @@ -136,86 +179,6 @@ func (d *dispatch) funcTarget(j *join, obj *types.Func) (Target, bool) { return Target{}, false } -// libraryOrigin reports whether v, an interface call's receiver, may hold a -// value whose dynamic type VTA cannot see: following Phi edges, Extract, -// ChangeInterface and loads back to -// -// - a TypeAssert (the asserted value came from anywhere); -// - the result of a call to a library function or of an interface call of -// a library interface's method (its callee may be library code); -// - a load of a library global, or through a library struct's field, or -// through a pointer that itself originates in library code. -// -// A value project code built (MakeInterface), a parameter, a free variable -// or a load of project storage is VTA's: false. The walk is -// intraprocedural, so the remaining gap is a library-originated value that -// reaches the call through project code: stored in a project struct field -// or global and read back, or a project function's result or parameter -// (spec §4.8). -func (c *cha) libraryOrigin(v ssa.Value, seen map[ssa.Value]bool) bool { - if seen[v] { - return false - } - seen[v] = true - switch v := v.(type) { - case *ssa.Phi: - for _, e := range v.Edges { - if c.libraryOrigin(e, seen) { - return true - } - } - case *ssa.Extract: - return c.libraryOrigin(v.Tuple, seen) - case *ssa.ChangeInterface: - return c.libraryOrigin(v.X, seen) - case *ssa.TypeAssert: - return true - case *ssa.Call: - return c.libraryCall(v.Common()) - case *ssa.UnOp: - if v.Op == token.MUL { - return c.libraryLoad(v.X, seen) - } - case *ssa.Field: - if c.libraryType(v.X.Type()) { - return true - } - return c.libraryOrigin(v.X, seen) - } - return false -} - -// libraryCall reports whether cc calls a library function, or a library -// interface's method. -func (c *cha) libraryCall(cc *ssa.CallCommon) bool { - if cc.IsInvoke() { - return cc.Method != nil && cc.Method.Pkg() != nil && c.j.isLibrary(cc.Method) - } - callee := cc.StaticCallee() - if callee == nil { - return false - } - obj, ok := callee.Object().(*types.Func) - return ok && obj.Pkg() != nil && c.j.isLibrary(obj) -} - -// libraryLoad reports whether a load through the pointer x reads library -// storage, or a pointer that originates in library code. -func (c *cha) libraryLoad(x ssa.Value, seen map[ssa.Value]bool) bool { - switch x := x.(type) { - case *ssa.Global: - return x.Pkg != nil && !c.j.projPaths[x.Pkg.Pkg.Path()] - case *ssa.FieldAddr: - if ptr, ok := x.X.Type().Underlying().(*types.Pointer); ok && c.libraryType(ptr.Elem()) { - return true - } - return c.libraryOrigin(x.X, seen) - case *ssa.IndexAddr: - return c.libraryOrigin(x.X, seen) - } - return c.libraryOrigin(x, seen) -} - // libraryType reports whether t is a named type declared outside the // project. func (c *cha) libraryType(t types.Type) bool { diff --git a/internal/sdg/cha_origin.go b/internal/sdg/cha_origin.go new file mode 100644 index 0000000..6a5b22e --- /dev/null +++ b/internal/sdg/cha_origin.go @@ -0,0 +1,676 @@ +package sdg + +import ( + "go/token" + "go/types" + "sort" + + "golang.org/x/tools/go/ssa" + "golang.org/x/tools/go/types/typeutil" +) + +// origins is the interprocedural "may carry a library-originated dynamic +// type" analysis behind the CHA complement (spec §4.8). VTA propagates +// types through project bodies only (its graph is field-based for struct +// fields and type-based for pointer, slice, map and channel contents), so +// what it cannot see is a value that came out of library code: no flow +// reaches it. The analysis tracks where such values go, through project +// code, to a monotone fixpoint over the whole program (the project +// functions VTA runs over): +// +// - Seeds (per value): a TypeAssert; the result of a library function, a +// project function with no body, a library interface's method, recover(); +// a load of a library global or through a library struct's field. +// - Local steps: Phi, Extract, ChangeInterface, ChangeType, Convert, Slice, +// Field and FieldAddr / IndexAddr of an LO base; a map Lookup, an Index, +// a channel receive and a range Next over an LO container (N2). +// - Interprocedural facts (finite, so the fixpoint terminates): +// ret(F, i) result i of project function F (any Return operand i is LO); +// param(p) a parameter (receiver included) some call passes an LO actual; +// free(v) a closure capture bound to an LO value (MakeClosure); +// global(G) a project global an LO value is stored to; +// field(S, i) field i of struct type S (by type: every instance); +// alloc(a) an address-taken local; +// seq(T) an element of any slice or array of element type T; +// mapkey(T), mapval(T), chan(T) map keys, map values and channel +// elements of type T (any container of that element type); +// unknown(T) a T stored through a pointer of unknown provenance (a +// parameter, a call result); every precise load of a T reads it too; +// any(T) any store of a T; a load through such a pointer reads it. +// A pointer, slice, map or channel of interface elements passed to +// library code (errors.As(err, &target)) marks unknown(T) / any(T) and +// the container's element cell: the library may write into it. +// - Call targets: the static callee; for an interface call, the method +// each project implementer's method set selects (CHA), and the result is +// LO too when the receiver is LO or a library type the project converts +// to an interface implements it; for a call through a function value, +// every address-taken project function of an identical signature, and +// the result is LO when the value is LO or a library function of that +// signature is used as a value. VTA's targets are not used: the analysis +// runs before VTA (the program must not be read after a no_vta), and +// these sets are supersets of VTA's. +// +// Each fact only goes from false to true. A function is re-evaluated when a +// fact it read changes (a reader-tracked worklist); a function's values are +// computed by a local fixpoint (Phi cycles). The least fixpoint is unique, +// so the result is independent of the visiting order; the order is +// deterministic anyway (functions sorted by package path and name, readers +// in registration order). It reads SSA instructions and go/types only and +// builds no code. +type origins struct { + c *cha + prog *ssa.Program + funcs map[*ssa.Function]bool + order []*ssa.Function + + facts map[fact]bool + readers map[fact][]*ssa.Function + readBy map[fact]map[*ssa.Function]bool + cur *ssa.Function // the function being evaluated (reader registration); nil = none + + work []*ssa.Function + queued map[*ssa.Function]bool + + typeIDs typeutil.Map // type → int id (fact keys) + nextID int + byFuncSig typeutil.Map // signature → []*ssa.Function: address-taken project functions + libFuncs typeutil.Map // signature → true: address-taken library functions + libConv []types.Type // library types project code converts to an interface + libImpl typeutil.Map // interface → bool: a libConv type implements it + implFns map[implKey][]*ssa.Function + implLib map[implKey]bool // an implementer's method is library code (no body) +} + +type factKind uint8 + +const ( + fRet factKind = iota + fParam + fFree + fGlobal + fField + fAlloc + fSeq + fMapKey + fMapVal + fChan + fUnknown + fAny +) + +// fact is one interprocedural fact: kind, an SSA object (ret, param, free, +// global, alloc) and up to two ints (a result index; a type id and a field +// index). +type fact struct { + k factKind + p any + a, b int +} + +type implKey struct { + iface int // type id of the interface + pkg *types.Package + name string +} + +// chaHook, when set, runs on each function the CHA complement scans (a test +// seam for the recovered-panic path, N1). +var chaHook func(*ssa.Function) + +func newOrigins(c *cha, funcs map[*ssa.Function]bool) *origins { + o := &origins{ + c: c, + funcs: funcs, + facts: map[fact]bool{}, + readers: map[fact][]*ssa.Function{}, + readBy: map[fact]map[*ssa.Function]bool{}, + queued: map[*ssa.Function]bool{}, + implFns: map[implKey][]*ssa.Function{}, + implLib: map[implKey]bool{}, + } + for fn := range funcs { + o.order = append(o.order, fn) + o.prog = fn.Prog + } + sort.Slice(o.order, func(i, j int) bool { + a, b := o.order[i], o.order[j] + if pa, pb := homePath(a), homePath(b); pa != pb { + return pa < pb + } + return a.String() < b.String() + }) + var conv typeutil.Map + var buf [10]*ssa.Value + for _, fn := range o.order { + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + var callee *ssa.Value + if call, ok := instr.(ssa.CallInstruction); ok && !call.Common().IsInvoke() { + callee = &call.Common().Value + } + for _, op := range instr.Operands(buf[:0]) { + if g, ok := (*op).(*ssa.Function); ok && op != callee { + o.addressTaken(g) + } + } + if mc, ok := instr.(*ssa.MakeClosure); ok { + o.addressTaken(mc.Fn.(*ssa.Function)) + } + if mi, ok := instr.(*ssa.MakeInterface); ok && c.libraryType(derefType(mi.X.Type())) && conv.At(mi.X.Type()) == nil { + conv.Set(mi.X.Type(), true) + o.libConv = append(o.libConv, mi.X.Type()) + } + } + } + } + return o +} + +func derefType(t types.Type) types.Type { + if p, ok := types.Unalias(t).Underlying().(*types.Pointer); ok { + return p.Elem() + } + return t +} + +func (o *origins) addressTaken(g *ssa.Function) { + if o.funcs[g] { + list, _ := o.byFuncSig.At(g.Signature).([]*ssa.Function) + for _, f := range list { + if f == g { + return + } + } + o.byFuncSig.Set(g.Signature, append(list, g)) + return + } + o.libFuncs.Set(g.Signature, true) +} + +func (o *origins) typeID(t types.Type) int { + if v := o.typeIDs.At(t); v != nil { + return v.(int) + } + o.nextID++ + o.typeIDs.Set(t, o.nextID) + return o.nextID +} + +// get reads fact f, registering the current function as its reader. +func (o *origins) get(f fact) bool { + if o.cur != nil { + set := o.readBy[f] + if set == nil { + set = map[*ssa.Function]bool{} + o.readBy[f] = set + } + if !set[o.cur] { + set[o.cur] = true + o.readers[f] = append(o.readers[f], o.cur) + } + } + return o.facts[f] +} + +// set makes fact f true and re-queues its readers. +func (o *origins) set(f fact) { + if o.facts[f] { + return + } + o.facts[f] = true + for _, r := range o.readers[f] { + o.enqueue(r) + } +} + +func (o *origins) enqueue(fn *ssa.Function) { + if !o.queued[fn] { + o.queued[fn] = true + o.work = append(o.work, fn) + } +} + +// solve runs the worklist to the fixpoint. +func (o *origins) solve() { + for _, fn := range o.order { + o.enqueue(fn) + } + for len(o.work) > 0 { + fn := o.work[0] + o.work = o.work[1:] + o.queued[fn] = false + o.cur = fn + lo := o.eval(fn) + o.effects(fn, lo) + o.cur = nil + } +} + +// eval is fn's local fixpoint: the set of its values (parameters, captures +// and instruction values) that may carry a library-originated dynamic type, +// under the current facts. +func (o *origins) eval(fn *ssa.Function) map[ssa.Value]bool { + lo := map[ssa.Value]bool{} + for _, p := range fn.Params { + if o.get(fact{k: fParam, p: p}) { + lo[p] = true + } + } + for _, fv := range fn.FreeVars { + if o.get(fact{k: fFree, p: fv}) { + lo[fv] = true + } + } + for changed := true; changed; { + changed = false + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + v, ok := instr.(ssa.Value) + if !ok || lo[v] { + continue + } + if o.valueLO(v, lo) { + lo[v] = true + changed = true + } + } + } + } + return lo +} + +// valueLO is the rule for one instruction value. +func (o *origins) valueLO(v ssa.Value, lo map[ssa.Value]bool) bool { + switch v := v.(type) { + case *ssa.Phi: + for _, e := range v.Edges { + if lo[e] { + return true + } + } + case *ssa.Extract: + return o.tupleLO(v.Tuple, v.Index, lo) + case *ssa.ChangeInterface: + return lo[v.X] + case *ssa.ChangeType: + return lo[v.X] + case *ssa.Convert: + return lo[v.X] + case *ssa.MultiConvert: + return lo[v.X] + case *ssa.SliceToArrayPointer: + return lo[v.X] + case *ssa.Slice: + return lo[v.X] + case *ssa.Range: + return lo[v.X] + case *ssa.TypeAssert: + return true + case *ssa.Call: + return o.callLO(v.Common(), 0, lo) + case *ssa.UnOp: + switch v.Op { + case token.MUL: + return o.loadLO(v.X, lo) + case token.ARROW: + return o.recvLO(v.X, lo) + } + case *ssa.Field: + st := v.X.Type() + return o.c.libraryType(st) || lo[v.X] || o.get(o.fieldFact(st, v.Field)) + case *ssa.FieldAddr, *ssa.IndexAddr: + var x ssa.Value + if fa, ok := v.(*ssa.FieldAddr); ok { + x = fa.X + } else { + x = v.(*ssa.IndexAddr).X + } + return lo[x] + case *ssa.Lookup: + return o.lookupLO(v, lo) + case *ssa.Index: + return lo[v.X] || o.get(fact{k: fSeq, a: o.typeID(v.Type())}) + } + return false +} + +// tupleLO is the rule for component i of a tuple-valued instruction. +func (o *origins) tupleLO(t ssa.Value, i int, lo map[ssa.Value]bool) bool { + switch t := t.(type) { + case *ssa.Call: + return o.callLO(t.Common(), i, lo) + case *ssa.TypeAssert: + return i == 0 + case *ssa.Lookup: + return i == 0 && o.lookupLO(t, lo) + case *ssa.UnOp: + return i == 0 && t.Op == token.ARROW && o.recvLO(t.X, lo) + case *ssa.Next: + if t.IsString || i == 0 { + return false + } + r, ok := t.Iter.(*ssa.Range) + if !ok { + return true + } + if lo[r.X] { + return true + } + m, ok := types.Unalias(r.X.Type()).Underlying().(*types.Map) + if !ok { + return false + } + if i == 1 { + return o.get(fact{k: fMapKey, a: o.typeID(m.Key())}) + } + return o.get(fact{k: fMapVal, a: o.typeID(m.Elem())}) + case *ssa.Select: + k := i - 2 // results: index, recvOk, then one per receive state + for _, st := range t.States { + if st.Dir != types.RecvOnly { + continue + } + if k == 0 { + return o.recvLO(st.Chan, lo) + } + k-- + } + } + return false +} + +func (o *origins) lookupLO(l *ssa.Lookup, lo map[ssa.Value]bool) bool { + if lo[l.X] { + return true + } + if m, ok := types.Unalias(l.X.Type()).Underlying().(*types.Map); ok { + return o.get(fact{k: fMapVal, a: o.typeID(m.Elem())}) + } + return false +} + +func (o *origins) recvLO(ch ssa.Value, lo map[ssa.Value]bool) bool { + if lo[ch] { + return true + } + if c, ok := types.Unalias(ch.Type()).Underlying().(*types.Chan); ok { + return o.get(fact{k: fChan, a: o.typeID(c.Elem())}) + } + return false +} + +func (o *origins) fieldFact(structType types.Type, i int) fact { + return fact{k: fField, a: o.typeID(types.Unalias(structType).Underlying()), b: i} +} + +// loadLO is the rule for a load *x. +func (o *origins) loadLO(x ssa.Value, lo map[ssa.Value]bool) bool { + t := o.typeID(derefType(x.Type())) + unknown := func() bool { return o.get(fact{k: fUnknown, a: t}) } + switch x := x.(type) { + case *ssa.Global: + if x.Pkg != nil && !o.c.j.projPaths[x.Pkg.Pkg.Path()] { + return true + } + return o.get(fact{k: fGlobal, p: x}) || unknown() + case *ssa.FieldAddr: + st := derefType(x.X.Type()) + return o.c.libraryType(st) || lo[x.X] || o.get(o.fieldFact(st, x.Field)) || unknown() + case *ssa.IndexAddr: + return lo[x.X] || o.get(fact{k: fSeq, a: t}) || unknown() + case *ssa.Alloc: + return o.get(fact{k: fAlloc, p: x}) || unknown() + } + return lo[x] || o.get(fact{k: fAny, a: t}) +} + +// callLO is the rule for result i of the call cc. +func (o *origins) callLO(cc *ssa.CallCommon, i int, lo map[ssa.Value]bool) bool { + if b, ok := cc.Value.(*ssa.Builtin); ok { + switch b.Name() { + case "recover": + return true + case "append": + for _, a := range cc.Args { + if lo[a] { + return true + } + } + } + return false + } + if cc.IsInvoke() { + if cc.Method == nil || cc.Method.Pkg() == nil { + return lo[cc.Value] + } + if o.c.j.isLibrary(cc.Method) || lo[cc.Value] || o.libImplements(cc.Value.Type()) { + return true + } + k := o.implKey(cc) + if o.implLib[k] { + return true + } + for _, g := range o.implementers(cc, k) { + if o.get(fact{k: fRet, p: g, a: i}) { + return true + } + } + return false + } + if g := cc.StaticCallee(); g != nil { + if !o.funcs[g] || len(g.Blocks) == 0 { + return true // library code, or a function with no SSA body (assembly) + } + return o.get(fact{k: fRet, p: g, a: i}) + } + sig := cc.Signature() + if lo[cc.Value] || o.libFuncs.At(sig) != nil { + return true + } + list, _ := o.byFuncSig.At(sig).([]*ssa.Function) + for _, g := range list { + if o.get(fact{k: fRet, p: g, a: i}) { + return true + } + } + return false +} + +// libImplements reports whether a library type the project converts to an +// interface implements iface: a call on such a value may run library code. +func (o *origins) libImplements(iface types.Type) bool { + if v := o.libImpl.At(iface); v != nil { + return v.(bool) + } + it, _ := types.Unalias(iface).Underlying().(*types.Interface) + found := false + for _, t := range o.libConv { + if it != nil && types.Implements(t, it) { + found = true + break + } + } + o.libImpl.Set(iface, found) + return found +} + +func (o *origins) implKey(cc *ssa.CallCommon) implKey { + return implKey{o.typeID(cc.Value.Type()), cc.Method.Pkg(), cc.Method.Name()} +} + +// implementers returns the project functions an interface call cc may run: +// the method each project implementer's method set selects, when it has a +// body (implLib records one that is library code). +func (o *origins) implementers(cc *ssa.CallCommon, k implKey) []*ssa.Function { + if fns, ok := o.implFns[k]; ok { + return fns + } + var fns []*ssa.Function + seen := map[*ssa.Function]bool{} + for _, t := range o.c.implementers(cc.Value.Type()) { + sel := o.c.msets.MethodSet(t).Lookup(k.pkg, k.name) + if sel == nil { + continue + } + obj := sel.Obj().(*types.Func) + if obj.Pkg() == nil { + continue + } + if o.c.j.isLibrary(obj) { + o.implLib[k] = true + continue + } + if g := o.prog.FuncValue(obj); g != nil && o.funcs[g] && !seen[g] { + seen[g] = true + fns = append(fns, g) + } + } + o.implFns[k] = fns + return fns +} + +// effects applies fn's instructions to the facts, under its values lo. +func (o *origins) effects(fn *ssa.Function, lo map[ssa.Value]bool) { + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + switch instr := instr.(type) { + case *ssa.Return: + for i, r := range instr.Results { + if lo[r] { + o.set(fact{k: fRet, p: fn, a: i}) + } + } + case ssa.CallInstruction: + o.callEffects(instr.Common(), lo) + case *ssa.MakeClosure: + g := instr.Fn.(*ssa.Function) + for i, v := range instr.Bindings { + if lo[v] && i < len(g.FreeVars) { + o.set(fact{k: fFree, p: g.FreeVars[i]}) + } + } + case *ssa.Store: + if lo[instr.Val] { + o.store(instr.Addr) + } + case *ssa.MapUpdate: + m, ok := types.Unalias(instr.Map.Type()).Underlying().(*types.Map) + if !ok { + continue + } + if lo[instr.Value] { + o.set(fact{k: fMapVal, a: o.typeID(m.Elem())}) + } + if lo[instr.Key] { + o.set(fact{k: fMapKey, a: o.typeID(m.Key())}) + } + case *ssa.Send: + if lo[instr.X] { + if c, ok := types.Unalias(instr.Chan.Type()).Underlying().(*types.Chan); ok { + o.set(fact{k: fChan, a: o.typeID(c.Elem())}) + } + } + } + } + } +} + +// store records an LO value stored through addr. +func (o *origins) store(addr ssa.Value) { + t := o.typeID(derefType(addr.Type())) + o.set(fact{k: fAny, a: t}) + switch a := addr.(type) { + case *ssa.Global: + o.set(fact{k: fGlobal, p: a}) + case *ssa.FieldAddr: + o.set(o.fieldFact(derefType(a.X.Type()), a.Field)) + case *ssa.IndexAddr: + o.set(fact{k: fSeq, a: t}) + case *ssa.Alloc: + o.set(fact{k: fAlloc, p: a}) + default: + o.set(fact{k: fUnknown, a: t}) + } +} + +// callEffects passes cc's LO actuals to its targets' parameters, and marks +// what an interface container passed to library code may receive. +func (o *origins) callEffects(cc *ssa.CallCommon, lo map[ssa.Value]bool) { + if _, ok := cc.Value.(*ssa.Builtin); ok { + if b := cc.Value.(*ssa.Builtin); b.Name() == "copy" && len(cc.Args) == 2 && lo[cc.Args[1]] { + if s, ok := types.Unalias(cc.Args[0].Type()).Underlying().(*types.Slice); ok { + o.set(fact{k: fSeq, a: o.typeID(s.Elem())}) + } + } + return + } + pass := func(g *ssa.Function, args []ssa.Value) { + for i, a := range args { + if lo[a] && i < len(g.Params) { + o.set(fact{k: fParam, p: g.Params[i]}) + } + } + } + if cc.IsInvoke() { + if cc.Method == nil || cc.Method.Pkg() == nil { + return + } + if o.c.j.isLibrary(cc.Method) { + o.escape(cc.Args) + } + args := append([]ssa.Value{cc.Value}, cc.Args...) + for _, g := range o.implementers(cc, o.implKey(cc)) { + pass(g, args) + } + return + } + if g := cc.StaticCallee(); g != nil { + if !o.funcs[g] || len(g.Blocks) == 0 { + o.escape(cc.Args) + return + } + pass(g, cc.Args) + return + } + list, _ := o.byFuncSig.At(cc.Signature()).([]*ssa.Function) + for _, g := range list { + pass(g, cc.Args) + } + if o.libFuncs.At(cc.Signature()) != nil { + o.escape(cc.Args) + } +} + +// escape marks what library code may write into the interface containers +// args hand it (through a MakeInterface too): a pointer to an interface +// (errors.As), a slice, map or channel of interface elements. +func (o *origins) escape(args []ssa.Value) { + for _, a := range args { + if mi, ok := a.(*ssa.MakeInterface); ok { + a = mi.X + } + switch t := types.Unalias(a.Type()).Underlying().(type) { + case *types.Pointer: + if types.IsInterface(t.Elem()) { + id := o.typeID(t.Elem()) + o.set(fact{k: fUnknown, a: id}) + o.set(fact{k: fAny, a: id}) + } + case *types.Slice: + if types.IsInterface(t.Elem()) { + o.set(fact{k: fSeq, a: o.typeID(t.Elem())}) + } + case *types.Map: + if types.IsInterface(t.Elem()) { + o.set(fact{k: fMapVal, a: o.typeID(t.Elem())}) + } + if types.IsInterface(t.Key()) { + o.set(fact{k: fMapKey, a: o.typeID(t.Key())}) + } + case *types.Chan: + if types.IsInterface(t.Elem()) { + o.set(fact{k: fChan, a: o.typeID(t.Elem())}) + } + } + } +} diff --git a/internal/sdg/ir.go b/internal/sdg/ir.go index 776e59d..a8224c9 100644 --- a/internal/sdg/ir.go +++ b/internal/sdg/ir.go @@ -111,6 +111,7 @@ const ( CodePartialSSA = "partial_ssa" CodePartialTypes = "partial_types" CodeNoVTA = "no_vta" + CodeNoCHA = "no_cha" CodeUnmappedSSA = "unmapped_ssa" CodeL3Skipped = "l3_skipped" CodeRangeFunc = "range_func" diff --git a/internal/sdg/libflow_test.go b/internal/sdg/libflow_test.go new file mode 100644 index 0000000..9de7b76 --- /dev/null +++ b/internal/sdg/libflow_test.go @@ -0,0 +1,89 @@ +package sdg + +import ( + "path/filepath" + "slices" + "testing" + + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +const libflowPkg = "example.com/dataflow/libflow" + +func libflowKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("libflow", "libflow.go"), Signature: libflowPkg + "." + name} +} + +// TestCHA_InterproceduralLibraryOrigin (round 2): a Store that came out of +// library code reaches an interface call through project code — a project +// function's result, a parameter, a struct field set in another method, a +// global, a closure capture — or through a library container element (map +// lookup, slice and array index, channel receive). Each site's targets +// include sqlStore.Find (the CHA complement), and the caller's footprint +// sees the global sqlStore.Find writes. +func TestCHA_InterproceduralLibraryOrigin(t *testing.T) { + in, _, d := runDispatch(t) + find := Target{Callable: libflowKey("sqlStore.Find")} + fps := footprints(in, d) + for _, c := range []struct{ fn, site string }{ + {"ViaReturn", "47:9"}, + {"use", "53:9"}, + {"handler.ViaField", "84:9"}, + {"ViaGlobal", "96:9"}, + {"ViaCapture.closure_1", "104:10"}, + {"ViaMap", "112:9"}, + {"ViaSlice", "117:9"}, + {"ViaChan", "122:9"}, + {"ViaArray", "127:9"}, + } { + if _, ok := d.sites[siteKey{libflowKey(c.fn), c.site}]; !ok { + t.Errorf("%s has no site %s: the keys moved", c.fn, c.site) + continue + } + if got := d.targets(libflowKey(c.fn), c.site); !slices.Contains(got, find) { + t.Errorf("targets(%s, %s) = %v, want sqlStore.Find among them", c.fn, c.site, got) + } + if fp := fps[Target{Callable: libflowKey(c.fn)}]; !slices.Contains(fp.GlobalWrites, libflowPkg+"::Last") { + t.Errorf("%s writes %v, want %s::Last", c.fn, fp.GlobalWrites, libflowPkg) + } + } +} + +// TestCHA_ProjectOnlyOriginNoTargets (round 2): ProjectOnly's +// relay(s).Find(id) is a call through a project function's result whose +// every value project code built (a memStore): it gets VTA's memStore.Find +// and the interface method, and no CHA target. +func TestCHA_ProjectOnlyOriginNoTargets(t *testing.T) { + _, _, d := runDispatch(t) + want := []Target{{Callable: libflowKey("Store.Find")}, {Callable: libflowKey("memStore.Find")}} + if got := d.targets(libflowKey("ProjectOnly"), "134:9"); !slices.Equal(got, want) { + t.Errorf("targets(ProjectOnly, 134:9) = %v, want %v", got, want) + } +} + +// TestCHA_PanicDegradesToNoCHA (N1): a panic inside the CHA complement is +// recovered: every CHA target is dropped (the dispatch is what it would be +// without CHA), no_cha is recorded, and VTA still runs. +func TestCHA_PanicDegradesToNoCHA(t *testing.T) { + in := loadInput(t, "dataflow") + defer func() { chaHook = nil }() + chaHook = func(fn *ssa.Function) { + if fn.Name() == "ViaMap" { + panic("cha boom") + } + } + res, d := run(in) + if codes := codesOf(res.Degraded); !slices.Equal(codes, []string{CodeNoCHA}) { + t.Fatalf("codes = %v, want [no_cha]", codes) + } + // ViaReturn's site keeps only D7 (VTA cannot see the sqlStore). + if got, want := d.targets(libflowKey("ViaReturn"), "47:9"), []Target{{Callable: libflowKey("Store.Find")}}; !slices.Equal(got, want) { + t.Errorf("targets(ViaReturn, 47:9) = %v, want %v (no CHA target after no_cha)", got, want) + } + // VTA ran: HandleDirect's VTA target is there. + if got := d.targets(wrapKey("HandleDirect"), "33:9"); !slices.Contains(got, Target{Callable: wrapKey("sqlStore.Find")}) { + t.Errorf("targets(HandleDirect, 33:9) = %v, want VTA's sqlStore.Find", got) + } +} diff --git a/internal/sdg/run.go b/internal/sdg/run.go index 3cca6eb..3aafa6b 100644 --- a/internal/sdg/run.go +++ b/internal/sdg/run.go @@ -38,7 +38,7 @@ const l4GCPercent = 50 // join alone. The degradations are, in order: partial_types (the load), // partial_ssa / no_ssa (SSA), // l3_skipped and range_func (from the units, whether or not SSA built), -// unmapped_ssa (the join), no_vta. +// unmapped_ssa (the join), no_cha (the CHA complement), no_vta. // // Run lowers the process-wide GC target to l4GCPercent while it runs and // restores the previous one on return (debug.SetGCPercent is global: a @@ -103,7 +103,10 @@ func dispatchRun(in Input) (Result, *dispatch) { deg = append(deg, *dg) } d.addInvokes(j, funcs) - d.addCHA(newCHA(j, in), funcs) + if err := d.runCHA(j, in, funcs); err != nil { + utils.Debug("L4: CHA complement failed: %v", err) + deg = append(deg, Degraded{Code: CodeNoCHA, Detail: err.Error()}) + } start = time.Now() cg, err := runVTA(all) diff --git a/internal/sdg/ssa_test.go b/internal/sdg/ssa_test.go index 7297f0f..3bf3178 100644 --- a/internal/sdg/ssa_test.go +++ b/internal/sdg/ssa_test.go @@ -54,7 +54,7 @@ func loadInputOpts(t *testing.T, name string, skipTests bool) Input { // skipped), sorted. var dataflowPkgs = []string{ "example.com/dataflow/asm", "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/httpsql", "example.com/dataflow/initclosure", "example.com/dataflow/inits", - "example.com/dataflow/l1", "example.com/dataflow/libstore", + "example.com/dataflow/l1", "example.com/dataflow/libflow", "example.com/dataflow/libstore", "example.com/dataflow/linedir", "example.com/dataflow/sdg", "example.com/dataflow/variant/p", "example.com/dataflow/variant/u", "example.com/dataflow/vend", "example.com/dataflow/wrap", "example.com/dataflow/xfile", "example.com/dataflow/xtest", "example.com/dataflow/xtest/user", } diff --git a/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go b/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go index 687448a..2cff0b1 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go @@ -82,3 +82,29 @@ func TestGateL4_FlowThroughLibraryStorage(t *testing.T) { } } } + +// TestGateL4_FlowThroughProjectCode (round 2): on testdata/dataflow/libflow, +// a sqlStore put in library storage reaches an interface call through +// project code (a result, a parameter, a field, a global, a capture) or a +// library container (map, slice, array, channel); each handler's id reaches +// Exec's query formal through the CHA complement's sqlStore.Find. +func TestGateL4_FlowThroughProjectCode(t *testing.T) { + a := buildL4(t, "dataflow/libflow").l4 + if len(a.Degraded) != 0 { + t.Fatalf("libflow degraded: %+v", a.Degraded) + } + mod := v2.ModuleID(a.Application.ID, "libflow.go") + pkg := "example.com/dataflow/libflow." + fn := func(name string) string { return v2.CallableID(mod, pkg+name) } + method := func(typ, m string) string { return v2.CallableID(v2.TypeID(mod, pkg+typ), pkg+typ+"."+m) } + sink := l4Vertex(t, a, fn("Exec"), "formal_in", "query") + g := newSDGGraph(a) + for _, src := range []string{ + fn("ViaReturn"), fn("ViaParam"), method("handler", "ViaField"), fn("ViaGlobal"), fn("ViaCapture"), + fn("ViaMap"), fn("ViaSlice"), fn("ViaChan"), fn("ViaArray"), + } { + if p := g.path(l4Vertex(t, a, src, "formal_in", "id"), sink); p == nil { + t.Errorf("%s's id does not reach Exec's query", src) + } + } +} diff --git a/testdata/dataflow/libflow/libflow.go b/testdata/dataflow/libflow/libflow.go new file mode 100644 index 0000000..570887f --- /dev/null +++ b/testdata/dataflow/libflow/libflow.go @@ -0,0 +1,173 @@ +// Package libflow is the L4 fixture for interprocedural library origin +// (spec §4.8, CHA complement): a Store that came out of library code +// (context.Context, the vendored box) reaches an interface call through +// project code — a project function's result, a parameter, a struct field, +// a global, a closure capture, a map, slice or channel element. VTA sees no +// concrete type in library code, so each call's targets include every +// project type implementing Store. Tests pin keys by line:col, so keep each +// statement on its own line and append new functions at the end. +package libflow + +import ( + "context" + + "example.com/vlib/box" +) + +// Store is the interface the handlers call. +type Store interface { + Find(id string) string +} + +// Last is the global sqlStore.Find writes. +var Last string + +type sqlStore struct{} + +// Find builds the query from id and runs it: the sink is Exec's query. +func (s *sqlStore) Find(id string) string { + Last = id + return Exec("select " + id) +} + +// Exec is the sink. +func Exec(query string) string { + return query +} + +type key struct{} + +// fromCtx returns the Store in ctx: its result is library-originated. +func fromCtx(ctx context.Context) Store { + return ctx.Value(key{}).(Store) +} + +// ViaReturn calls Find on a project function's library-originated result. +func ViaReturn(ctx context.Context, id string) string { + return fromCtx(ctx).Find(id) +} + +// use calls Find on its parameter: one caller passes a library-originated +// Store, another a project-built one. +func use(s Store, id string) string { + return s.Find(id) +} + +// ViaParam passes a library-originated Store to use. +func ViaParam(ctx context.Context, id string) string { + return use(ctx.Value(key{}).(Store), id) +} + +type memStore struct{} + +// Find returns id. +func (memStore) Find(id string) string { + return id +} + +// UseLocal passes a project-built Store to use. +func UseLocal(id string) string { + return use(memStore{}, id) +} + +type handler struct { + store Store +} + +// init stores a library-originated Store in a field. +func (h *handler) init(ctx context.Context) { + h.store = ctx.Value(key{}).(Store) +} + +// ViaField calls Find on the field another method set. +func (h *handler) ViaField(id string) string { + return h.store.Find(id) +} + +var global Store + +// SetGlobal stores a library-originated Store in a global. +func SetGlobal(ctx context.Context) { + global = ctx.Value(key{}).(Store) +} + +// ViaGlobal calls Find on the global. +func ViaGlobal(id string) string { + return global.Find(id) +} + +// ViaCapture calls Find inside a closure on a captured library-originated +// Store. +func ViaCapture(ctx context.Context, id string) string { + s := ctx.Value(key{}).(Store) + f := func() string { + return s.Find(id) + } + return f() +} + +// ViaMap looks up a library map (comma-ok free): the element is +// library-originated. +func ViaMap(id string) string { + return box.Map()["db"].Find(id) +} + +// ViaSlice indexes a library slice. +func ViaSlice(id string) string { + return box.List()[0].Find(id) +} + +// ViaChan receives from a library channel. +func ViaChan(id string) string { + return (<-box.Chan("db")).Find(id) +} + +// ViaArray indexes a library array value. +func ViaArray(id string) string { + return box.First("db")[0].Find(id) +} + +// ProjectOnly calls Find on a Store only project code ever built: no CHA +// target. +func ProjectOnly(id string) string { + var s Store = memStore{} + return relay(s).Find(id) +} + +// relay returns its parameter. +func relay(s Store) Store { + return s +} + +// The entries: each puts a sqlStore in library storage and calls one +// handler, so no host variable is shared between the handlers' sites. +func MainReturn() { + ViaReturn(context.WithValue(context.Background(), key{}, &sqlStore{}), "a") +} + +func MainParam() { + ViaParam(context.WithValue(context.Background(), key{}, &sqlStore{}), "b") +} + +func MainField() { + h := &handler{} + h.init(context.WithValue(context.Background(), key{}, &sqlStore{})) + h.ViaField("c") +} + +func MainGlobal() { + SetGlobal(context.WithValue(context.Background(), key{}, &sqlStore{})) + ViaGlobal("d") +} + +func MainCapture() { + ViaCapture(context.WithValue(context.Background(), key{}, &sqlStore{}), "e") +} + +func MainBox() { + box.Put("db", &sqlStore{}) + ViaMap("f") + ViaSlice("g") + ViaChan("h") + ViaArray("i") +} diff --git a/testdata/dataflow/vendor/example.com/vlib/box/box.go b/testdata/dataflow/vendor/example.com/vlib/box/box.go new file mode 100644 index 0000000..fd604fa --- /dev/null +++ b/testdata/dataflow/vendor/example.com/vlib/box/box.go @@ -0,0 +1,36 @@ +// Package box is a vendored library of interface containers: what it hands +// out comes from library code, which L4 builds no SSA for. Its elements +// have its own interface type, so a caller needs no type assertion. +package box + +// Finder is the library's interface; project types implement it. +type Finder interface { + Find(id string) string +} + +var store = map[string]Finder{} + +// Put stores f under k. +func Put(k string, f Finder) { store[k] = f } + +// Map returns the stored values by key. +func Map() map[string]Finder { return store } + +// List returns the stored values. +func List() []Finder { + var out []Finder + for _, f := range store { + out = append(out, f) + } + return out +} + +// First returns the value under k as a one-element array. +func First(k string) [1]Finder { return [1]Finder{store[k]} } + +// Chan returns a channel that yields the value under k once. +func Chan(k string) <-chan Finder { + ch := make(chan Finder, 1) + ch <- store[k] + return ch +} diff --git a/testdata/dataflow/vendor/modules.txt b/testdata/dataflow/vendor/modules.txt index 06b192c..ee62d69 100644 --- a/testdata/dataflow/vendor/modules.txt +++ b/testdata/dataflow/vendor/modules.txt @@ -1,3 +1,4 @@ # example.com/vlib v0.0.0 ## explicit example.com/vlib +example.com/vlib/box From 34f70bba1c035a8e8cc5f49f2bdf646eaff91dba Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 21:30:13 -0400 Subject: [PATCH 58/65] docs: interprocedural library origin and no_cha --- CLAUDE.md | 2 +- README.md | 11 ++-- docs/design/notes/l3-dataflow-metrics.md | 29 +++++---- docs/design/specs/v2-l3-l4-dataflow.md | 76 ++++++++++++++++-------- docs/design/specs/v2-schema-structure.md | 2 +- 5 files changed, 77 insertions(+), 43 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index a1e5a28..f4425d4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -120,7 +120,7 @@ Reuses Group A ids/spans and the L3 vocabulary unchanged; every L4 fact is an ad statement-level reachability; sanitizer semantics are the SDK's [user, 2026-10-27; codellm-devkit/.github#103]. - **`degraded[]`** envelope leaf `{code, detail?, packages[]?, callables[]?}`, absent when - nothing degraded; codes `no_ssa | no_vta | partial_ssa | partial_types | unmapped_ssa | + nothing degraded; codes `no_ssa | no_vta | no_cha | partial_ssa | partial_types | unmapped_ssa | l3_skipped | range_func` (open vocabulary). `--strict` exits non-zero on any. - **Rulings:** global → global summary edges are omitted [user, 2026-10-26; #30]; memory is measured and reported per app, not gated (D3) [user, 2026-10-26]; range-over-func is not diff --git a/README.md b/README.md index 983b30d..ffea422 100644 --- a/README.md +++ b/README.md @@ -313,6 +313,7 @@ subset of `-a 4` output, and `-j 1` and `-j N` output are byte-identical. and `callables[]`. The codes are: - `no_ssa` / `partial_ssa`: SSA failed, in whole or for some packages; - `no_vta`: VTA failed; + - `no_cha`: the CHA complement for library-originated values failed (its targets are dropped); - `partial_types`: some packages have type errors; - `unmapped_ssa`: SSA functions or calls that map to no callable; - `l3_skipped`: callables without an L3 result, which get a pass-through summary; @@ -336,11 +337,11 @@ subset of `-a 4` output, and `-j 1` and `-j N` output are byte-identical. - **Known limits** (see `docs/design/notes/l3-dataflow-metrics.md` § L4): - library code calling back into the project (`sort.Slice`, `http.HandleFunc`) is not an edge; - - an interface call on a value that came out of library code (`ctx.Value(k).(Store)`, - `sync.Map.Load`) in the same function binds every project type implementing the interface - (a targeted CHA complement); a value that first passes through project code (a struct field - or global, a project function's result or parameter) gets VTA's targets only, which miss - the concrete type; + - an interface call on a value that may have come out of library code (`ctx.Value(k).(Store)`, + `sync.Map.Load`, a library container), directly or through project code (results, + parameters, captures, fields, globals, container elements), binds every project type + implementing the interface (a CHA complement). Storage is tracked by container type (every + instance of a struct field), which is sound but coarser than VTA; - range-over-func is not modelled (`range_func`); - summaries are close to all-to-all, by design; - `summary` edges whose two ends are both `:` vertices are omitted diff --git a/docs/design/notes/l3-dataflow-metrics.md b/docs/design/notes/l3-dataflow-metrics.md index ab172ec..923dcff 100644 --- a/docs/design/notes/l3-dataflow-metrics.md +++ b/docs/design/notes/l3-dataflow-metrics.md @@ -877,7 +877,7 @@ dispatch targets to `call_graph` (`prov:["vta"]`), library callees get stand-in 7. **Degradation is reported, never silent.** Five of the six apps and the Kubernetes root use range-over-func somewhere (`range_func`, D8). Four apps and the root carry `unmapped_ssa`, which comes almost entirely from closures in package-level `var` initializers. No app hit - `no_ssa`, `no_vta`, `partial_ssa`, `partial_types` or `l3_skipped`. + `no_ssa`, `no_vta`, `no_cha`, `partial_ssa`, `partial_types` or `l3_skipped`. ### Per-app results (L4) @@ -1096,16 +1096,23 @@ over-approximation or reported as degraded. - **Library callbacks are not edges.** In `sort.Slice(xs, less)` and `http.HandleFunc(path, h)`, the function value flows into the stand-in, but the library calling it back is not modelled. Taint that *starts* in a handler (its own parameters) is unaffected. -- **Library-stored dynamic types (final review F3).** VTA sees no concrete type in library code. - An interface call whose receiver comes straight out of library code (`ctx.Value(k).(Store)`, - `sync.Map.Load`, a library call's result, a library field) gets a targeted CHA complement: - every project type implementing the interface. Measured (go1.26.4, `-j 8`): chi +54 targets at - 15 sites (the `w.(http.Flusher)` pattern), param_in 1,895 → 1,961 (+3.5%), param_out - 1,649 → 1,770 (+7.3%); nats-server +317 targets at 63 sites, param_in 862,765 → 863,075 - (+0.04%), param_out 431,327 → 431,854 (+0.12%). Remaining gap (the origin walk is - intraprocedural): a library-originated value that reaches the call through project code — a - project struct field or global, a project function's result or parameter — gets VTA's - targets only. +- **Library-originated dynamic types (final review F3, round 2).** VTA sees no concrete type in + library code. An interface call whose receiver may carry a value that came out of library code + (`ctx.Value(k).(Store)`, `sync.Map.Load`, a library call's result, field or container element), + directly or through project code (results, parameters, closure captures, fields, globals, + container elements; a whole-program fixpoint), gets a CHA complement: every project type + implementing the interface. Storage is tracked field-insensitively by container type. CHA + targets added over no CHA (go1.27.3, `-j 8`; round 1 = intraprocedural origin): + + | app | round 1: targets / sites | round 2: targets / sites | param_in, no CHA → round 2 | param_out, no CHA → round 2 | CHA time | + |---|---|---|---|---|---| + | chi | +54 / 15 | +168 / 75 | 1,895 → 2,431 (+28.3%) | 1,649 → 2,242 (+36.0%) | 6 ms | + | memos | +495 / 223 | +803 / 347 | 59,025 → 63,547 (+7.7%) | 64,804 → 68,487 (+5.7%) | 174 ms | + | nats-server | +317 / 63 | +2,290 / 883 | 862,765 → 864,091 (+0.15%) | 431,327 → 433,667 (+0.54%) | 456 ms | + + Peak RSS and overlay time do not move measurably. On chi most new sites are + `next.ServeHTTP(w, r)` and `w.Write`/`w.WriteHeader` on the `http.ResponseWriter` parameter of + a handler, which `net/http` (library code) calls: each gains chi's ResponseWriter wrappers. - **A15: summaries are near all-to-all** (74–94% of sites, table above). Under the coarse host wiring, a summary edge adds no reachability for the SDK's context-insensitive walk. It is a shortcut hop, and a precondition for a future HRB two-phase walk (spec §11). diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index a3746f2..73c7b58 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -562,23 +562,48 @@ built per D2 (amended, A12): `ssautil.Packages` over the **same** `go/packages` interface — VTA's concrete callees sit only there. The wrapper's VTA out-edges are followed transitively (cycle-safe) and their joined callees are targets, and `call_graph` edges, of the original site. -- **Targeted CHA complement [final review F3; controller ruling].** VTA propagates types through - project bodies only, so a value that passes through library storage — `ctx.Value(k).(Store)`, - `sync.Map.Load`, a library call's result — reaches an interface call with no concrete type VTA - sees, and only the D7 interface method (empty footprint) would remain. For an `invoke` site whose - receiver SSA value originates — through `Phi`, `Extract`, `ChangeInterface`, field reads and - `UnOp` loads — in a `TypeAssert`, the result of a call to a non-project function (or of a - library interface's method), or a load of a library global or through a library-typed - pointer or field, every **project** named type `T` or `*T` (non-generic) whose method set - implements the interface (`types.Implements`) adds the method its method set selects, mapped by - the join. A receiver that originates in project code (a `MakeInterface`, a parameter, a free - variable, a load of project storage) gets no CHA target: VTA covers it. Not a `call_graph` edge - (F7). The origin walk is intraprocedural. **Remaining gap:** a library-originated value that - reaches the call through project code — first stored in a project struct field or global and - read back, or returned by a project function (`func FromCtx(ctx) Store { return - ctx.Value(k).(Store) }`), or passed in as a parameter — is VTA's alone (the walk stops at the - project load, call result or parameter) and misses the concrete type. Recorded in §6 beside - library callbacks. +- **Targeted CHA complement [final review F3; controller ruling; interprocedural: user, + 2026-10-27].** VTA propagates types through project bodies only, so a value that passes through + library code — `ctx.Value(k).(Store)`, `sync.Map.Load`, a library call's result, a library + container's element — reaches an interface call with no concrete type VTA sees, and only the D7 + interface method (empty footprint) would remain. A whole-program monotone fixpoint + (`cha_origin.go`) computes the SSA values that **may carry a library-originated dynamic type** + (LO): + - *Seeds:* a `TypeAssert`; the result of a library function, of a function with no SSA body, of + a library interface's method, of `recover()`; a load of a library global or through a library + struct's field. + - *Local steps:* `Phi`, `Extract`, `ChangeInterface`/`ChangeType`/`Convert`, `Slice`, `Field` + and `FieldAddr`/`IndexAddr` of an LO base; a map `Lookup`, an `Index`, a channel receive + (`UnOp ARROW`, `select`) and a range `Next` over an LO container (N2). + - *Interprocedural:* a project function's result *i* (any `Return` operand *i* is LO) flows to + every call of it; an LO actual makes the callee's parameter LO (receiver included); an LO + `MakeClosure` binding makes the closure's free variable LO. Call targets are the static + callee; for an interface call, the method each project implementer selects (CHA) — and the + result is LO when the receiver is LO or a library type the project converts to an interface + implements it; for a call through a function value, every address-taken project function + of an identical signature — and the result is LO when the value is LO or a library function + of that signature is used as a value. VTA's targets are not used: the fixpoint runs before + VTA (so the program is never read after a `no_vta`), over supersets of them. + - *Storage, field-insensitive by container type:* an LO store marks the project global, or the + **(struct type, field index)** (every instance), or the slice/array element type, map key / + value type or channel element type of the container (any container of that type); a load of + that global, field or element is LO. A store through a pointer of unknown provenance (a + parameter, a call result) marks its element type for every load; a pointer, slice, map or + channel of interface elements passed to library code (`errors.As(err, &target)`) marks its + cells (the library may write them). + - *Termination and determinism:* each fact only goes false → true and the facts are finite + (per function result, parameter, capture, global, struct field and type), so the reader-tracked + worklist ends; a function is re-evaluated (a local fixpoint over its Phi cycles) only when a + fact it read changed. The least fixpoint is unique, so the result does not depend on the + visiting order, which is deterministic anyway (functions sorted by package path, name). + + Every `invoke` site in a joined project function whose receiver is LO gets, beside D7 and VTA, + every **project** named type `T` or `*T` (non-generic) whose method set implements the interface + (`types.Implements`) — the method its method set selects, mapped by the join. A receiver no + library value can reach (only project-built values: `MakeInterface` of project types) gets no + CHA target. Not a `call_graph` edge (F7). The complement and the fixpoint run under `recover`: + a panic drops every CHA target and degrades `no_cha` (N1). Reflection, `unsafe` and cgo stay + unsound spots (§4.7). - **Interfaces with no symbol-table node (ruling, Task 10) [call].** `error.Error()` (the universe `error`), methods of interface literals (`interface{ M() }`), and methods promoted through an embedded interface whose origin is declared outside the symbol table have no callable id and no @@ -895,6 +920,8 @@ callables) but marks the envelope: - Codes (open vocabulary, plain strings): - `no_ssa` — SSA build failed entirely. - `no_vta` — VTA failed (incl. a recovered VTA panic). + - `no_cha` — the CHA complement (§4.8) panicked (recovered): its targets are dropped, so an + interface call on a library-originated value keeps only D7 and VTA's targets. - `partial_ssa` — listed packages had type errors (`pkg.IllTyped`) or a recovered per-package build panic, so no SSA. - `partial_types` — project packages `go/packages` loaded with errors (list, parse or type @@ -917,7 +944,7 @@ callables) but marks the envelope: - **Go leaves (all additive, none in the shared keystone yet):** the `degraded` envelope field itself; the entry's `callables[]` (`can://` ids, sorted; carried by `l3_skipped` and `range_func`); the codes beyond the original four (`partial_types`, `unmapped_ssa`, - `l3_skipped`, `range_func`); and, in the vertex vocabulary it degrades over, the `_recv` `of` + `l3_skipped`, `range_func`, `no_cha`); and, in the vertex vocabulary it degrades over, the `_recv` `of` for an unnamed or `_` receiver (D6) and `:` (§4.6). - Go-only leaf now; candidate for the keystone alongside the cross-language taint revisit (§11). @@ -1020,13 +1047,12 @@ addition now; the cross-language SDK taint revisit is deferred (§11). `http.HandleFunc(path, h)`): the function-value argument flows in, but the library calling it back is not an edge. Taint *starting* in a handler is unaffected (sources are the handler's own params). Recorded limitation. -- **Not modelled:** a value that came out of library code (`ctx.Value(k)`, `sync.Map.Load`, a - library call's result) and reaches an interface call through project code: stored in a - project struct field or global and read back, returned by a project function, or passed as a - parameter. The targeted CHA complement (§4.8) covers the call made on the library-originated - value in the same function; otherwise the receiver's origin is a project load, call result or - parameter, and VTA, which sees no concrete type in library code, finds none. The call keeps - its D7 interface method (pass-through, empty footprint). Recorded limitation. +- **Library-originated dynamic types** (`ctx.Value(k)`, `sync.Map.Load`, a library call's + result or container element) reach interface calls through project code; VTA sees no + concrete type in library code, so the CHA complement (§4.8) binds every project implementer + at each call the interprocedural origin fixpoint reaches. Its storage tracking is + field-insensitive by container type (every instance of a struct field, every container of an + element type): sound, coarser than VTA. - **SDK addressing:** `("query", "database/sql.DB.Query")` resolves through the Go backend's `resolve_callable` (domain = analysed callables ∪ stand-ins) then `resolve_value` over the stand-in body's `formal_in` `of`. Shared `commons/` code is untouched. diff --git a/docs/design/specs/v2-schema-structure.md b/docs/design/specs/v2-schema-structure.md index 425bcc1..8d50f2d 100644 --- a/docs/design/specs/v2-schema-structure.md +++ b/docs/design/specs/v2-schema-structure.md @@ -16,7 +16,7 @@ the JSON tags from that file. ├─ k_limit # L3+ only: access-path bound (--graph-field-depth, default 3) ├─ degraded[] # L4; optional: absent when nothing degraded │ └─ { code, detail?, packages[]?, callables[]? } -│ # code: no_ssa | no_vta | partial_ssa | partial_types | +│ # code: no_ssa | no_vta | no_cha | partial_ssa | partial_types | │ # unmapped_ssa | l3_skipped | range_func (open vocabulary); │ # callables[] = can:// ids (l3_skipped, range_func) ├─ analyzer From d6908f88f47ce98981136ebe751c37e6a8e18fa6 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 22:31:35 -0400 Subject: [PATCH 59/65] fix(l4): library-origin reads only fixed or tracked facts callLO read the implementer-is-library-code memo before implementers filled it, and the memo is not a fact, so an invoke through a promoted library method got no CHA target when its caller was visited first. The memo is now filled with the implementer list it belongs to. runCHA clones the site slices it restores on a panic. --- internal/sdg/cha.go | 3 +- internal/sdg/cha_origin.go | 47 ++++-- internal/sdg/libflow_test.go | 135 ++++++++++++++++++ testdata/dataflow/libflow/libflow.go | 47 ++++++ .../vendor/example.com/vlib/box/box.go | 7 + 5 files changed, 224 insertions(+), 15 deletions(-) diff --git a/internal/sdg/cha.go b/internal/sdg/cha.go index fc91d95..a1b76ed 100644 --- a/internal/sdg/cha.go +++ b/internal/sdg/cha.go @@ -4,6 +4,7 @@ import ( "fmt" "go/ast" "go/types" + "slices" "golang.org/x/tools/go/ssa" "golang.org/x/tools/go/types/typeutil" @@ -134,7 +135,7 @@ func (d *dispatch) addCHA(c *cha, funcs map[*ssa.Function]bool) { func (d *dispatch) runCHA(j *join, in Input, funcs map[*ssa.Function]bool) (err error) { saved := make(map[siteKey][]Target, len(d.sites)) for k, ts := range d.sites { - saved[k] = ts + saved[k] = slices.Clone(ts) // d.add inserts in place: the backing array is not ours } newExt := make(map[string]bool, len(d.newExt)) for k := range d.newExt { diff --git a/internal/sdg/cha_origin.go b/internal/sdg/cha_origin.go index 6a5b22e..6c88544 100644 --- a/internal/sdg/cha_origin.go +++ b/internal/sdg/cha_origin.go @@ -3,6 +3,7 @@ package sdg import ( "go/token" "go/types" + "slices" "sort" "golang.org/x/tools/go/ssa" @@ -76,9 +77,11 @@ type origins struct { byFuncSig typeutil.Map // signature → []*ssa.Function: address-taken project functions libFuncs typeutil.Map // signature → true: address-taken library functions libConv []types.Type // library types project code converts to an interface - libImpl typeutil.Map // interface → bool: a libConv type implements it - implFns map[implKey][]*ssa.Function - implLib map[implKey]bool // an implementer's method is library code (no body) + // Memos of pure functions of the program (fixed before the worklist; + // never facts): each entry is computed whole the first time it is read. + libImpl typeutil.Map // interface → bool: a libConv type implements it + implFns map[implKey][]*ssa.Function // filled with implLib, by implementers + implLib map[implKey]bool // an implementer's selected method is library code } type factKind uint8 @@ -113,6 +116,10 @@ type implKey struct { name string } +// originSeedOrder, when set, permutes the order the origin worklist is +// seeded in (a test seam: the least fixpoint must not depend on it). +var originSeedOrder func([]*ssa.Function) + // chaHook, when set, runs on each function the CHA complement scans (a test // seam for the recovered-panic path, N1). var chaHook func(*ssa.Function) @@ -232,7 +239,12 @@ func (o *origins) enqueue(fn *ssa.Function) { // solve runs the worklist to the fixpoint. func (o *origins) solve() { - for _, fn := range o.order { + seed := o.order + if originSeedOrder != nil { + seed = slices.Clone(o.order) + originSeedOrder(seed) + } + for _, fn := range seed { o.enqueue(fn) } for len(o.work) > 0 { @@ -446,11 +458,11 @@ func (o *origins) callLO(cc *ssa.CallCommon, i int, lo map[ssa.Value]bool) bool if o.c.j.isLibrary(cc.Method) || lo[cc.Value] || o.libImplements(cc.Value.Type()) { return true } - k := o.implKey(cc) - if o.implLib[k] { + fns, lib := o.implementers(cc) + if lib { return true } - for _, g := range o.implementers(cc, k) { + for _, g := range fns { if o.get(fact{k: fRet, p: g, a: i}) { return true } @@ -498,12 +510,18 @@ func (o *origins) implKey(cc *ssa.CallCommon) implKey { return implKey{o.typeID(cc.Value.Type()), cc.Method.Pkg(), cc.Method.Name()} } -// implementers returns the project functions an interface call cc may run: -// the method each project implementer's method set selects, when it has a -// body (implLib records one that is library code). -func (o *origins) implementers(cc *ssa.CallCommon, k implKey) []*ssa.Function { +// implementers returns the project functions an interface call cc may run +// — the method each project implementer's method set selects, when it has a +// body — and whether some implementer's selected method is library code (a +// promoted library method: its result may carry a library-originated +// type). Both are a pure function of the program (the project types and +// their method sets, fixed before the worklist), computed together and +// memoized together: neither is ever read before it is filled, so no +// evaluation depends on which call was evaluated first. +func (o *origins) implementers(cc *ssa.CallCommon) ([]*ssa.Function, bool) { + k := o.implKey(cc) if fns, ok := o.implFns[k]; ok { - return fns + return fns, o.implLib[k] } var fns []*ssa.Function seen := map[*ssa.Function]bool{} @@ -526,7 +544,7 @@ func (o *origins) implementers(cc *ssa.CallCommon, k implKey) []*ssa.Function { } } o.implFns[k] = fns - return fns + return fns, o.implLib[k] } // effects applies fn's instructions to the facts, under its values lo. @@ -619,7 +637,8 @@ func (o *origins) callEffects(cc *ssa.CallCommon, lo map[ssa.Value]bool) { o.escape(cc.Args) } args := append([]ssa.Value{cc.Value}, cc.Args...) - for _, g := range o.implementers(cc, o.implKey(cc)) { + fns, _ := o.implementers(cc) + for _, g := range fns { pass(g, args) } return diff --git a/internal/sdg/libflow_test.go b/internal/sdg/libflow_test.go index 9de7b76..b94fed0 100644 --- a/internal/sdg/libflow_test.go +++ b/internal/sdg/libflow_test.go @@ -1,8 +1,11 @@ package sdg import ( + "maps" + "math/rand" "path/filepath" "slices" + "strconv" "testing" "golang.org/x/tools/go/ssa" @@ -37,6 +40,8 @@ func TestCHA_InterproceduralLibraryOrigin(t *testing.T) { {"ViaSlice", "117:9"}, {"ViaChan", "122:9"}, {"ViaArray", "127:9"}, + {"ViaPromoted", "195:9"}, + {"zzzViaPromoted", "205:9"}, } { if _, ok := d.sites[siteKey{libflowKey(c.fn), c.site}]; !ok { t.Errorf("%s has no site %s: the keys moved", c.fn, c.site) @@ -87,3 +92,133 @@ func TestCHA_PanicDegradesToNoCHA(t *testing.T) { t.Errorf("targets(HandleDirect, 33:9) = %v, want VTA's sqlStore.Find", got) } } + +// chaTargets is every CHA run's dispatch over in, with the origin fixpoint +// seeded in the order perm gives (nil: the sorted order), and the LO values +// of every project function, by function and instruction index. +func chaTargets(t *testing.T, in Input, perm func([]*ssa.Function)) (map[siteKey][]Target, map[string]bool) { + t.Helper() + prog, pkgs, _ := buildSSA(in) + funcs, _, err := funcSet(prog, pkgs) + if err != nil { + t.Fatal(err) + } + j := newJoin(in) + d := newDispatch(in) + defer func() { originSeedOrder = nil }() + originSeedOrder = perm + d.addCHA(newCHA(j, in), funcs) + o := newOrigins(newCHA(j, in), funcs) + originSeedOrder = perm + o.solve() + lo := map[string]bool{} + for _, fn := range o.order { + vals := o.eval(fn) + i := 0 + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + if v, ok := instr.(ssa.Value); ok && vals[v] { + lo[fn.String()+"#"+strconv.Itoa(i)] = true + } + i++ + } + } + for k, p := range fn.Params { + if vals[p] { + lo[fn.String()+"#p"+strconv.Itoa(k)] = true + } + } + } + return d.sites, lo +} + +// TestCHA_OriginOrderIndependent (round 3): the origin fixpoint is the +// least fixpoint, independent of the order functions are visited in. The +// worklist seeded in reverse and in two shuffled orders gives the same LO +// values and the same CHA targets as the sorted order. +func TestCHA_OriginOrderIndependent(t *testing.T) { + in := loadInput(t, "dataflow") + wantSites, wantLO := chaTargets(t, in, nil) + if !wantLO["example.com/dataflow/libflow.zzzGet#p0"] && len(wantLO) == 0 { + t.Fatal("no LO values: the fixture is vacuous") + } + perms := map[string]func([]*ssa.Function){ + "reverse": func(fs []*ssa.Function) { slices.Reverse(fs) }, + "shuffle1": func(fs []*ssa.Function) { + r := rand.New(rand.NewSource(1)) + r.Shuffle(len(fs), func(i, j int) { fs[i], fs[j] = fs[j], fs[i] }) + }, + "shuffle2": func(fs []*ssa.Function) { + r := rand.New(rand.NewSource(2)) + r.Shuffle(len(fs), func(i, j int) { fs[i], fs[j] = fs[j], fs[i] }) + }, + } + for name, perm := range perms { + sites, lo := chaTargets(t, in, perm) + if !maps.Equal(lo, wantLO) { + var diff []string + for k := range lo { + if !wantLO[k] { + diff = append(diff, "+"+k) + } + } + for k := range wantLO { + if !lo[k] { + diff = append(diff, "-"+k) + } + } + slices.Sort(diff) + t.Errorf("%s: LO differs from the sorted order: %v", name, diff) + } + if !maps.EqualFunc(sites, wantSites, slices.Equal[[]Target]) { + for k, ts := range sites { + if !slices.Equal(ts, wantSites[k]) { + t.Errorf("%s: site %v targets %v, sorted order %v", name, k, ts, wantSites[k]) + } + } + } + } +} + +// TestCHA_PanicRestoresEverySite (round 3): a panic after the CHA +// complement added targets restores every site to exactly what it held +// before CHA ran. d.add inserts in place, so a saved slice that shares its +// backing array with the live one would be shifted under it. +func TestCHA_PanicRestoresEverySite(t *testing.T) { + in := loadInput(t, "dataflow") + prog, pkgs, _ := buildSSA(in) + funcs, _, err := funcSet(prog, pkgs) + if err != nil { + t.Fatal(err) + } + j := newJoin(in) + d := newDispatch(in) + d.addInvokes(j, funcs) + // Spare capacity in every site, as append-grown slices have. + for k, ts := range d.sites { + d.sites[k] = slices.Grow(ts, 8) + } + before := map[siteKey][]Target{} + for k, ts := range d.sites { + before[k] = slices.Clone(ts) + } + defer func() { chaHook = nil }() + seen := 0 + chaHook = func(*ssa.Function) { + seen++ + if seen == len(funcs) { // the last function: every earlier site has its CHA targets + panic("cha boom") + } + } + if err := d.runCHA(j, in, funcs); err == nil { + t.Fatal("runCHA did not report the panic") + } + if len(d.sites) != len(before) { + t.Errorf("%d sites after the panic, %d before", len(d.sites), len(before)) + } + for k, ts := range before { + if got := d.sites[k]; !slices.Equal(got, ts) { + t.Errorf("site %v = %v after the panic, want %v", k, got, ts) + } + } +} diff --git a/testdata/dataflow/libflow/libflow.go b/testdata/dataflow/libflow/libflow.go index 570887f..0485b55 100644 --- a/testdata/dataflow/libflow/libflow.go +++ b/testdata/dataflow/libflow/libflow.go @@ -171,3 +171,50 @@ func MainBox() { ViaChan("h") ViaArray("i") } + +// Getter is a project interface whose one implementer's method is promoted +// library code (box.Holder.Get): a call of it returns a library-originated +// value whatever the visiting order of the origin fixpoint. +type Getter interface { + Get() box.Finder +} + +type holder struct { + box.Holder +} + +// zzzGet calls Get through the interface; its name sorts after +// ViaPromoted, so the fixpoint visits ViaPromoted first. +func zzzGet(g Getter) box.Finder { + return g.Get() +} + +// ViaPromoted calls Find on what a promoted library method returned, +// through a project function. +func ViaPromoted(g Getter, id string) string { + return zzzGet(g).Find(id) +} + +func MainPromoted() { + ViaPromoted(holder{box.Holder{F: &sqlStore{}}}, "j") +} + +// zzzViaPromoted is ViaPromoted visited after zzzGet: with an order- +// dependent fixpoint the two sites would differ. +func zzzViaPromoted(g Getter, id string) string { + return zzzGet(g).Find(id) +} + +func MainPromoted2() { + zzzViaPromoted(holder{box.Holder{F: &sqlStore{}}}, "k") +} + +// aaaPeek calls Get through the interface too; its name sorts first, so +// the fixpoint meets the Getter.Get call here before zzzGet. +func aaaPeek(g Getter) { + g.Get() +} + +func MainPeek() { + aaaPeek(holder{}) +} diff --git a/testdata/dataflow/vendor/example.com/vlib/box/box.go b/testdata/dataflow/vendor/example.com/vlib/box/box.go index fd604fa..2245620 100644 --- a/testdata/dataflow/vendor/example.com/vlib/box/box.go +++ b/testdata/dataflow/vendor/example.com/vlib/box/box.go @@ -34,3 +34,10 @@ func Chan(k string) <-chan Finder { ch <- store[k] return ch } + +// Holder holds a Finder; a project type embedding it promotes Get, whose +// code is library code. +type Holder struct{ F Finder } + +// Get returns the held Finder. +func (h Holder) Get() Finder { return h.F } From 0e2eb2074c4b002d54c9ea358097d95a77b866bd Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Tue, 27 Oct 2026 22:31:35 -0400 Subject: [PATCH 60/65] docs: what drives the CHA complement's volume; the fixpoint's order independence --- docs/design/notes/l3-dataflow-metrics.md | 20 +++++++++++++++++--- docs/design/specs/v2-l3-l4-dataflow.md | 9 +++++++-- 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/docs/design/notes/l3-dataflow-metrics.md b/docs/design/notes/l3-dataflow-metrics.md index 923dcff..8b2d690 100644 --- a/docs/design/notes/l3-dataflow-metrics.md +++ b/docs/design/notes/l3-dataflow-metrics.md @@ -1110,9 +1110,23 @@ over-approximation or reported as degraded. | memos | +495 / 223 | +803 / 347 | 59,025 → 63,547 (+7.7%) | 64,804 → 68,487 (+5.7%) | 174 ms | | nats-server | +317 / 63 | +2,290 / 883 | 862,765 → 864,091 (+0.15%) | 431,327 → 433,667 (+0.54%) | 456 ms | - Peak RSS and overlay time do not move measurably. On chi most new sites are - `next.ServeHTTP(w, r)` and `w.Write`/`w.WriteHeader` on the `http.ResponseWriter` parameter of - a handler, which `net/http` (library code) calls: each gains chi's ResponseWriter wrappers. + Peak RSS and overlay time do not move measurably. The parameters of a handler that `net/http` + calls are **not** seeds (a handler's `w.Write` gains nothing). What drives the growth, measured + by ablating one seed kind or propagation rule at a time (a site counts for a rule when + disabling the rule removes some of its CHA targets; the kinds overlap, since several seeds + often reach one site): + - **chi** (75 sites): the seeds are type assertions (`ctx.Value(k).(*Context)`, `r.Context()` + values, `w.(http.Flusher)`; 69 sites with type assertions alone) and library call results + (46 alone). Their reach comes from the type-keyed storage cells: 39 sites (78 targets) need + the load through a closure capture of a pointer cell whose element type some LO store + marked (`any(T)`: the captured `next http.Handler` of each middleware closure, after an + LO `http.Handler` is stored anywhere), and 9 need a field read of an LO base. Most of these + sites are `next.ServeHTTP(w, r)` in middleware closures, which gain chi's 2 handler types. + - **memos** (347 sites): interface type assertions (needed by 281 sites), library call results + (34), field reads of an LO base (83). + - **nats-server** (883 sites): library call results (needed by 118), field reads of an LO base + (597), capture-cell loads (113); most sites have several sufficient seeds (concrete and + interface type assertions, library fields and globals, promoted library methods). - **A15: summaries are near all-to-all** (74–94% of sites, table above). Under the coarse host wiring, a summary edge adds no reachability for the SDK's context-insensitive walk. It is a shortcut hop, and a precondition for a future HRB two-phase walk (spec §11). diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index 73c7b58..2afdc05 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -594,8 +594,13 @@ built per D2 (amended, A12): `ssautil.Packages` over the **same** `go/packages` - *Termination and determinism:* each fact only goes false → true and the facts are finite (per function result, parameter, capture, global, struct field and type), so the reader-tracked worklist ends; a function is re-evaluated (a local fixpoint over its Phi cycles) only when a - fact it read changed. The least fixpoint is unique, so the result does not depend on the - visiting order, which is deterministic anyway (functions sorted by package path, name). + fact it read changed. Every input an evaluation reads is either fixed before the worklist + starts (the program, the project types and their method sets, the address-taken function + sets, and memos of pure functions of them, such as which implementers' methods are library + code) or a tracked fact that re-queues its readers when it turns true. So each evaluation is + a monotone function of the facts, and the result is the least fixpoint, which is unique and + independent of the visiting order (`TestCHA_OriginOrderIndependent` permutes the seed + order); the order is deterministic anyway (functions sorted by package path, name). Every `invoke` site in a joined project function whose receiver is LO gets, beside D7 and VTA, every **project** named type `T` or `*T` (non-generic) whose method set implements the interface From e203f8cbbd9be190e12963fc9ea07d37d289e393 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Wed, 28 Oct 2026 08:18:41 -0400 Subject: [PATCH 61/65] fix: analysis.json honours the umask and is synced before the rename --- internal/core/analyzer.go | 13 ++++++++--- internal/core/umask_other.go | 9 ++++++++ internal/core/umask_unix.go | 16 +++++++++++++ internal/core/umask_unix_test.go | 39 ++++++++++++++++++++++++++++++++ 4 files changed, 74 insertions(+), 3 deletions(-) create mode 100644 internal/core/umask_other.go create mode 100644 internal/core/umask_unix.go create mode 100644 internal/core/umask_unix_test.go diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index e55dee4..98ef382 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -54,7 +54,7 @@ func (a *Analyzer) L3() map[dataflow.L3Key]dataflow.Result { return a.l3 } // L4 returns the SDG result of the last Analyze run. Nil below level 4. func (a *Analyzer) L4() *sdg.Result { return a.l4 } -// KLimit is the access-path bound used at L3: opts.FieldDepth, or the +// KLimit is the access-path bound used at L3 and L4: opts.FieldDepth, or the // --graph-field-depth default of 3 when a non-CLI caller left it 0. func KLimit(opts options.AnalysisOptions) int { if opts.FieldDepth > 0 { @@ -248,7 +248,9 @@ func (a *Analyzer) finalizeAndCache(app *schema.GoApplication) (*schema.GoApplic return app, nil } -// saveCache persists the application to cache as analysis_cache.json. +// saveCache persists the application to cache as analysis_cache.json, in the +// internal v1 shape. Nothing reads it back: it is a debugging aid, not an +// input, so a v1 shape change (e.g. closures gaining path) affects nothing. func (a *Analyzer) saveCache(app *schema.GoApplication) error { if err := utils.EnsureDir(a.opts.CacheDir); err != nil { return err @@ -323,11 +325,16 @@ func WriteOutput(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Resul } tmp := f.Name() err = StreamJSON(f, app, l3, l4, opts) + if err == nil { + err = f.Sync() // the rename below must not publish unflushed bytes + } if cerr := f.Close(); err == nil { err = cerr } if err == nil { - err = os.Chmod(tmp, 0o644) + // CreateTemp makes the file 0600; give it the mode os.Create would, + // 0666 less the process umask. + err = os.Chmod(tmp, 0o666&^currentUmask()) } if err == nil { err = os.Rename(tmp, dst) diff --git a/internal/core/umask_other.go b/internal/core/umask_other.go new file mode 100644 index 0000000..b8914ad --- /dev/null +++ b/internal/core/umask_other.go @@ -0,0 +1,9 @@ +//go:build !unix + +package core + +import "io/fs" + +// currentUmask returns 0 where the platform has no umask (Windows): the file +// gets mode 0666, which os.Create would also give it. +func currentUmask() fs.FileMode { return 0 } diff --git a/internal/core/umask_unix.go b/internal/core/umask_unix.go new file mode 100644 index 0000000..311b90d --- /dev/null +++ b/internal/core/umask_unix.go @@ -0,0 +1,16 @@ +//go:build unix + +package core + +import ( + "io/fs" + "syscall" +) + +// currentUmask returns the process umask. The syscall can only set it, so it +// is set to 0 and restored at once, before the chmod that uses the result. +func currentUmask() fs.FileMode { + m := syscall.Umask(0) + syscall.Umask(m) + return fs.FileMode(m) +} diff --git a/internal/core/umask_unix_test.go b/internal/core/umask_unix_test.go new file mode 100644 index 0000000..dc4ff93 --- /dev/null +++ b/internal/core/umask_unix_test.go @@ -0,0 +1,39 @@ +//go:build unix + +package core_test + +import ( + "os" + "path/filepath" + "syscall" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/core" + "github.com/codellm-devkit/codeanalyzer-go/internal/options" +) + +// TestWriteOutput_ModeHonoursUmask: analysis.json gets the mode os.Create +// would give it (0666 less the umask), not CreateTemp's 0600, and the temp +// file is gone after the rename. +func TestWriteOutput_ModeHonoursUmask(t *testing.T) { + for _, mask := range []int{0o022, 0o077} { + old := syscall.Umask(mask) + outDir := t.TempDir() + err := core.WriteOutput(sharedGreeterL1, nil, nil, options.AnalysisOptions{OutputDir: outDir, Format: "json"}) + syscall.Umask(old) + if err != nil { + t.Fatalf("umask %#o: WriteOutput: %v", mask, err) + } + st, err := os.Stat(filepath.Join(outDir, "analysis.json")) + if err != nil { + t.Fatal(err) + } + if want := os.FileMode(0o666 &^ mask); st.Mode().Perm() != want { + t.Errorf("umask %#o: mode = %v, want %v", mask, st.Mode().Perm(), want) + } + left, _ := filepath.Glob(filepath.Join(outDir, ".analysis.json.*")) + if len(left) != 0 { + t.Errorf("umask %#o: temp files left behind: %v", mask, left) + } + } +} From e950b686af181c7f1ff5dad0c9a6e584c40a1f9b Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Wed, 28 Oct 2026 08:18:41 -0400 Subject: [PATCH 62/65] fix(l4): bind receivers without a signature, classify fileless callees as L2 does, list range-over-func in package-level literals --- internal/schema/schema.go | 4 +- internal/schema/v2/schema.go | 4 +- internal/sdg/join.go | 40 ++++++++++++++++- internal/sdg/join_test.go | 43 +++++++++++++++++++ internal/sdg/vertices.go | 10 ++++- internal/sdg/vertices_test.go | 22 ++++++++++ internal/semantic_analysis/call_graph.go | 38 ++++++++++------ internal/semantic_analysis/call_graph_test.go | 22 ++++++++++ testdata/rangefuncpkg/go.mod | 3 ++ testdata/rangefuncpkg/pkglevel.go | 17 ++++++++ 10 files changed, 183 insertions(+), 20 deletions(-) create mode 100644 testdata/rangefuncpkg/go.mod create mode 100644 testdata/rangefuncpkg/pkglevel.go diff --git a/internal/schema/schema.go b/internal/schema/schema.go index 7c9e96d..84335d5 100644 --- a/internal/schema/schema.go +++ b/internal/schema/schema.go @@ -106,7 +106,9 @@ type GoCallsite struct { // GoCallable represents a function, method, or function literal in Go. // receiver_type / receiver_name are non-empty for methods; empty for functions. type GoCallable struct { - Name string `json:"name"` + Name string `json:"name"` + // Path is the declaring file (a symbol_table key). A closure carries its + // enclosing callable's declaring file, so v1 inner_callables have a path. Path string `json:"path"` Signature string `json:"signature"` // signatureOf() output — edge id Comments []GoComment `json:"comments"` diff --git a/internal/schema/v2/schema.go b/internal/schema/v2/schema.go index 2decfd9..8042056 100644 --- a/internal/schema/v2/schema.go +++ b/internal/schema/v2/schema.go @@ -48,8 +48,8 @@ type Span struct { // Analysis is the top-level payload written to analysis.json / stdout. Its // siblings of `application` carry the manifest; consumers read `max_level` -// rather than sniffing for keys. k_limit (the L3 field-path depth bound) is -// present only at max_level >= 3. +// rather than sniffing for keys. k_limit (the field-path depth bound of the +// L3 and L4 access paths) is present only at max_level >= 3. type Analysis struct { SchemaVersion string `json:"schema_version"` Language string `json:"language"` diff --git a/internal/sdg/join.go b/internal/sdg/join.go index 4fba7d9..8bcaa5f 100644 --- a/internal/sdg/join.go +++ b/internal/sdg/join.go @@ -49,6 +49,7 @@ type join struct { bodiless map[dataflow.L3Key]bool // (declaring file, signature) of every bodiless declared callable calls map[dataflow.L3Key]map[token.Pos]string // unit → call Lparen → L1/L3 call key projPaths map[string]bool // import paths of the project packages + modPath string // the project module path from go.mod ("" when unavailable) units map[dataflow.L3Key]dataflow.Unit // by key results map[dataflow.L3Key]dataflow.Result // in.L3 pkgs map[string]*packages.Package // in.Pkgs @@ -67,6 +68,7 @@ func newJoin(in Input) *join { bodiless: map[dataflow.L3Key]bool{}, calls: make(map[dataflow.L3Key]map[token.Pos]string, len(in.Units)), projPaths: map[string]bool{}, + modPath: semantic_analysis.ModulePath(in.ProjectDir), units: make(map[dataflow.L3Key]dataflow.Unit, len(in.Units)), results: in.L3, pkgs: in.Pkgs, @@ -294,7 +296,8 @@ func (j *join) isLibrary(obj *types.Func) bool { } return utils.IsVendored(utils.RelativePath(j.projectDir, tf.Name())) } - return !j.projPaths[obj.Pkg().Path()] + // No source file: L2's rule, so the two levels agree on the callee. + return semantic_analysis.LibraryByPath(obj.Pkg().Path(), j.modPath, j.projPaths[obj.Pkg().Path()]) } // joinCoverage counts how the project functions and their call instructions @@ -483,7 +486,14 @@ func rangeFunc(in Input) *Degraded { } } } - if len(keys) == 0 { + // A function literal in a package-level declaration (`var f = func(){…}`) + // has no callable, so it is no unit; its package is still listed. + for _, p := range projectPackages(in) { + if pkgLevelRangesOverFunc(p) { + pkgs[p.PkgPath] = true + } + } + if len(keys) == 0 && len(pkgs) == 0 { return nil } sortKeys(keys) @@ -521,6 +531,32 @@ func rangesOverFunc(u dataflow.Unit) bool { return found } +// pkgLevelRangesOverFunc reports whether a function literal in one of p's +// package-level declarations (outside every FuncDecl) ranges over a function. +func pkgLevelRangesOverFunc(p *packages.Package) bool { + if p.TypesInfo == nil { + return false + } + found := false + for _, f := range p.Syntax { + for _, d := range f.Decls { + if _, ok := d.(*ast.GenDecl); !ok { + continue + } + ast.Inspect(d, func(n ast.Node) bool { + if rs, ok := n.(*ast.RangeStmt); ok && dataflow.RangesOverFunc(rs.X, p.TypesInfo) { + found = true + } + return !found + }) + if found { + return true + } + } + } + return false +} + func sortKeys(keys []dataflow.L3Key) { sort.Slice(keys, func(i, j int) bool { if keys[i].Path != keys[j].Path { diff --git a/internal/sdg/join_test.go b/internal/sdg/join_test.go index 74959fc..706af23 100644 --- a/internal/sdg/join_test.go +++ b/internal/sdg/join_test.go @@ -2,6 +2,7 @@ package sdg import ( "go/ast" + "go/token" "go/types" "path/filepath" "slices" @@ -768,3 +769,45 @@ func TestJoin_ExternalOf(t *testing.T) { } func ptrTo(t types.Type) types.Type { return types.NewPointer(t) } + +// TestJoin_IsLibraryWithoutAFileFollowsL2: an object with no source file (no +// position) is classified by L2's rule — the go.mod module prefix — not by +// whether its package happened to be loaded, so L2 and L4 agree. +func TestJoin_IsLibraryWithoutAFileFollowsL2(t *testing.T) { + j := newJoin(loadInput(t, "dataflow")) + if j.modPath != "example.com/dataflow" { + t.Fatalf("modPath = %q, want example.com/dataflow", j.modPath) + } + sig := types.NewSignatureType(nil, nil, nil, nil, nil, false) + unloaded := types.NewFunc(token.NoPos, types.NewPackage("example.com/dataflow/notloaded", "notloaded"), "F", sig) + if j.isLibrary(unloaded) { + t.Error("an unloaded package under the module path is project code at L2, but L4 called it a library") + } + other := types.NewFunc(token.NoPos, types.NewPackage("example.com/other", "other"), "F", sig) + if !j.isLibrary(other) { + t.Error("a package outside the module path must be a library") + } +} + +// TestSDG_RangeFuncInPackageLevelLiteral: rangefuncpkg ranges over a function +// only inside a package-level `var Sum = func(){…}`. That literal has no +// callable (it is no unit, and is unmapped_ssa), yet range_func must still +// list its package, so the SDK does not certify "no flow" there. +func TestSDG_RangeFuncInPackageLevelLiteral(t *testing.T) { + res := Run(loadInput(t, "rangefuncpkg")) + var rf *Degraded + for i := range res.Degraded { + if res.Degraded[i].Code == CodeRangeFunc { + rf = &res.Degraded[i] + } + } + if rf == nil { + t.Fatalf("no range_func entry; degraded = %v", codesOf(res.Degraded)) + } + if !slices.Equal(rf.Packages, []string{"example.com/rangefuncpkg"}) || len(rf.Callables) != 0 { + t.Errorf("range_func = %+v, want package example.com/rangefuncpkg and no callables", rf) + } + if !hasCode(res.Degraded, CodeUnmappedSSA) { + t.Error("the package-level literal should also be unmapped_ssa") + } +} diff --git a/internal/sdg/vertices.go b/internal/sdg/vertices.go index cb0fc62..a98cf56 100644 --- a/internal/sdg/vertices.go +++ b/internal/sdg/vertices.go @@ -508,6 +508,7 @@ type callShape struct { nParams int values int results int + noSig bool // the call's type is unknown: nParams is the value count } func callShapeOf(call *ast.CallExpr, info *types.Info) callShape { @@ -529,7 +530,14 @@ func callShapeOf(call *ast.CallExpr, info *types.Info) callShape { } } if sig == nil { + // No signature (a package with type errors): the value count stands + // in for the param count, but it must never make bind read a bound + // method value f(x) on a one-param method as a method expression + // (nParams == params+1). Only a selection go/types recorded as + // MethodExpr is one, and without a signature there is none; cap + // nParams so the receiver binds through the function value. s.nParams = s.values + s.noSig = true return s } s.nParams = sig.Params().Len() @@ -560,7 +568,7 @@ func (s callShape) bind(fs *formalSet) map[int]int { base, off := 0, 0 // declared param d is fs.inPos[base+d] and binds call param d+off if fs.recv { base = 1 - if !s.method && s.nParams == fs.params+1 { + if !s.method && !s.noSig && s.nParams == fs.params+1 { off = 1 out[0] = 0 // a method expression always passes its receiver } else { diff --git a/internal/sdg/vertices_test.go b/internal/sdg/vertices_test.go index 864d48c..ab85d6b 100644 --- a/internal/sdg/vertices_test.go +++ b/internal/sdg/vertices_test.go @@ -380,3 +380,25 @@ func TestVertices_Deterministic(t *testing.T) { } } } + +// TestCallShape_NoSignatureNeverBindsMethodExpression: without type info (a +// package with type errors), a bound method value f(x) on a one-param method +// passes 2 values, the declared param count plus one. That must not read as +// a method expression binding x to the receiver: the function value binds +// the receiver, x binds the declared param. +func TestCallShape_NoSignatureNeverBindsMethodExpression(t *testing.T) { + fs := &formalSet{recv: true, params: 1} + noSig := callShape{values: 2, nParams: 2, results: -1, noSig: true} + got := noSig.bind(fs) + if got[slotRecv] != 0 { + t.Errorf("no signature: receiver not bound through the function value: %v", got) + } + if _, ok := got[0]; ok && got[0] == 0 { + t.Errorf("no signature: value 0 bound to the receiver (read as a method expression): %v", got) + } + // With a known signature the same counts are a method expression. + expr := callShape{values: 2, nParams: 2, results: 1} + if got := expr.bind(fs); got[0] != 0 || got[1] != 1 { + t.Errorf("method expression: want value 0 → receiver, value 1 → param, got %v", got) + } +} diff --git a/internal/semantic_analysis/call_graph.go b/internal/semantic_analysis/call_graph.go index 7619193..33fc1f5 100644 --- a/internal/semantic_analysis/call_graph.go +++ b/internal/semantic_analysis/call_graph.go @@ -215,26 +215,24 @@ func (cg *CallGraphBuilder) isLibrary(fn *types.Func) bool { // (the symbol table skips it), so it is a library too. return utils.IsVendored(utils.RelativePath(root, file)) } - if mod := cg.modulePath(); mod != "" { - p := fn.Pkg().Path() - return p != mod && !strings.HasPrefix(p, mod+"/") - } _, inProject := cg.pkgs[fn.Pkg().Path()] - return !inProject + return LibraryByPath(fn.Pkg().Path(), cg.modulePath(), inProject) } -func absPath(p string) string { - if abs, err := filepath.Abs(p); err == nil { - return abs +// LibraryByPath is the fallback library test for an object with no source +// file: by the project module's path prefix when go.mod names one, else by +// whether its package is a loaded project package. L2 and L4 share it so the +// two levels classify a callee the same way. +func LibraryByPath(pkgPath, modulePath string, projectPkg bool) bool { + if modulePath != "" { + return pkgPath != modulePath && !strings.HasPrefix(pkgPath, modulePath+"/") } - return p + return !projectPkg } -func (cg *CallGraphBuilder) absProjectDir() string { return absPath(cg.projectDir) } - -// modulePath reads the project's module path from go.mod ("" when unavailable). -func (cg *CallGraphBuilder) modulePath() string { - data, err := os.ReadFile(filepath.Join(cg.projectDir, "go.mod")) +// ModulePath reads the module path from dir's go.mod ("" when unavailable). +func ModulePath(dir string) string { + data, err := os.ReadFile(filepath.Join(dir, "go.mod")) if err != nil { return "" } @@ -246,6 +244,18 @@ func (cg *CallGraphBuilder) modulePath() string { return "" } +func absPath(p string) string { + if abs, err := filepath.Abs(p); err == nil { + return abs + } + return p +} + +func (cg *CallGraphBuilder) absProjectDir() string { return absPath(cg.projectDir) } + +// modulePath reads the project's module path from go.mod ("" when unavailable). +func (cg *CallGraphBuilder) modulePath() string { return ModulePath(cg.projectDir) } + // recordExternal registers a library callee, keyed by its full signature. func (cg *CallGraphBuilder) recordExternal(fn *types.Func, fullSig string) { if _, ok := cg.externals[fullSig]; ok { diff --git a/internal/semantic_analysis/call_graph_test.go b/internal/semantic_analysis/call_graph_test.go index 8aae710..6124394 100644 --- a/internal/semantic_analysis/call_graph_test.go +++ b/internal/semantic_analysis/call_graph_test.go @@ -224,3 +224,25 @@ func TestCallGraph_ClosureSitesResolve(t *testing.T) { } t.Error("no edge from the closure to flow.f") } + +// TestLibraryByPath: the no-source-file fallback classifies by the module +// path prefix when go.mod names one (so an unloaded project package is still +// project code), else by whether the package was loaded. +func TestLibraryByPath(t *testing.T) { + for _, c := range []struct { + pkg, mod string + loaded bool + want bool + }{ + {"example.com/app", "example.com/app", false, false}, + {"example.com/app/internal/x", "example.com/app", false, false}, // unloaded project package + {"example.com/application", "example.com/app", true, true}, // prefix is segment-wise + {"fmt", "example.com/app", false, true}, + {"example.com/app/x", "", true, false}, // no go.mod: loaded = project + {"example.com/app/x", "", false, true}, + } { + if got := semantic_analysis.LibraryByPath(c.pkg, c.mod, c.loaded); got != c.want { + t.Errorf("LibraryByPath(%q, %q, %v) = %v, want %v", c.pkg, c.mod, c.loaded, got, c.want) + } + } +} diff --git a/testdata/rangefuncpkg/go.mod b/testdata/rangefuncpkg/go.mod new file mode 100644 index 0000000..f3c8730 --- /dev/null +++ b/testdata/rangefuncpkg/go.mod @@ -0,0 +1,3 @@ +module example.com/rangefuncpkg + +go 1.23 diff --git a/testdata/rangefuncpkg/pkglevel.go b/testdata/rangefuncpkg/pkglevel.go new file mode 100644 index 0000000..a227c26 --- /dev/null +++ b/testdata/rangefuncpkg/pkglevel.go @@ -0,0 +1,17 @@ +// Package rangefuncpkg ranges over a function only inside a package-level +// function literal: that literal has no callable (it is no L3 unit), so the +// range_func entry must still list this package. +package rangefuncpkg + +func seq(yield func(int) bool) { + yield(1) +} + +// Sum is a package-level variable holding a literal that ranges over seq. +var Sum = func() int { + s := 0 + for v := range seq { + s += v + } + return s +} From ab6fb7a3e121981a03bc2866cbb466bc5b91255b Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Wed, 28 Oct 2026 08:18:41 -0400 Subject: [PATCH 63/65] test(l4): self-test the superset gate; determinism compares stand-ins --- internal/sdg/summary_test.go | 16 +++++++- .../syntactic_analysis/v2emit/gate_l4_test.go | 37 +++++++++++++++++++ 2 files changed, 52 insertions(+), 1 deletion(-) diff --git a/internal/sdg/summary_test.go b/internal/sdg/summary_test.go index cb32a97..2f3781b 100644 --- a/internal/sdg/summary_test.go +++ b/internal/sdg/summary_test.go @@ -2,6 +2,7 @@ package sdg import ( "cmp" + "maps" "slices" "strings" "testing" @@ -370,12 +371,25 @@ func TestSDG_Deterministic(t *testing.T) { if !slices.Equal(a.ParamIn, b.ParamIn) || !slices.Equal(a.ParamOut, b.ParamOut) { t.Error("param edges differ across runs") } + if len(a.Callables) != len(b.Callables) { + t.Errorf("callables: %d vs %d across runs", len(a.Callables), len(b.Callables)) + } for k, c := range a.Callables { - d := b.Callables[k] + d, ok := b.Callables[k] + if !ok { + t.Errorf("%v only in the first run", k) + continue + } if !slices.Equal(c.Vertices, d.Vertices) || !slices.Equal(c.Ddg, d.Ddg) || !slices.Equal(c.Summary, d.Summary) { t.Errorf("%v differs across runs", k) } } + if len(a.Externals) == 0 { + t.Fatal("no stand-ins: the Externals comparison would be vacuous") + } + if !maps.EqualFunc(a.Externals, b.Externals, slices.Equal[[]Vertex]) { + t.Error("stand-in vertices differ across runs") + } } // runPortsInput is runPorts' input and dispatch. diff --git a/internal/syntactic_analysis/v2emit/gate_l4_test.go b/internal/syntactic_analysis/v2emit/gate_l4_test.go index f530575..2c61a2d 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_test.go @@ -881,3 +881,40 @@ func TestGateL4_PointsToEdges(t *testing.T) { t.Errorf("AliasPtr points-to ddg = %v, want %v", got, want) } } + +// TestSupersetL4_DetectsEveryViolation: the -a 3 ⊆ -a 4 checker is a gate, so +// it must itself fail on each way a superset can break, and accept a pure +// addition. Without this a bug in supersetL4 would make that gate vacuous. +func TestSupersetL4_DetectsEveryViolation(t *testing.T) { + parse := func(s string) any { + var v any + if err := json.Unmarshal([]byte(s), &v); err != nil { + t.Fatal(err) + } + return v + } + base := `{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1,2]}}}}` + for name, c := range map[string]struct { + b string + want string // a substring of the error; "" means no error + }{ + "identical": {base, ""}, + "pure addition": {`{"application":{"call_graph":[{"src":"a","dst":"b"},{"src":"a","dst":"c"}],"x":{"f":{"ddg":[{"src":"0","dst":"1","var":"w"},{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1,2,3],"new":1}},"extra":true}}`, ""}, + "removed key": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"list":[1,2]}}}}`, "k missing at L4"}, + "changed value": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"w","list":[1,2]}}}}`, "value changed"}, + "shrunk list": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1]}}}}`, "list shrank"}, + "changed type": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":{"a":1}}}}}`, "list shrank or changed type"}, + "removed ddg edge": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"0","dst":"1","var":"w"}],"k":"v","list":[1,2]}}}}`, "missing or changed at L4"}, + "changed call_graph": {`{"application":{"call_graph":[{"src":"a","dst":"z"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1,2]}}}}`, "call_graph: edge"}, + } { + var errs []string + supersetL4("$", parse(base), parse(c.b), &errs) + got := strings.Join(errs, "; ") + switch { + case c.want == "" && len(errs) > 0: + t.Errorf("%s: want no error, got %s", name, got) + case c.want != "" && !strings.Contains(got, c.want): + t.Errorf("%s: want an error containing %q, got %q", name, c.want, got) + } + } +} From 98ebfefbcce492353b1b08f9a5bbbf067a77bfd7 Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Wed, 28 Oct 2026 08:33:51 -0400 Subject: [PATCH 64/65] fix: create analysis.json's temp file with the umask, not a umask round-trip; bind on go/types' method-expression kind --- internal/core/analyzer.go | 27 ++++++++++++++----- internal/core/umask_other.go | 9 ------- internal/core/umask_unix.go | 16 ----------- ..._unix_test.go => writeoutput_unix_test.go} | 0 internal/sdg/join.go | 3 ++- internal/sdg/vertices.go | 27 ++++++++++--------- internal/sdg/vertices_test.go | 6 +++++ .../syntactic_analysis/v2emit/gate_l4_test.go | 18 +++++++------ 8 files changed, 52 insertions(+), 54 deletions(-) delete mode 100644 internal/core/umask_other.go delete mode 100644 internal/core/umask_unix.go rename internal/core/{umask_unix_test.go => writeoutput_unix_test.go} (100%) diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index 98ef382..2d541bf 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -16,11 +16,15 @@ package core import ( "bufio" "encoding/json" + "errors" "fmt" "io" + "io/fs" + "math/rand/v2" "os" "os/exec" "path/filepath" + "strconv" "strings" "github.com/codellm-devkit/codeanalyzer-go/internal/analysis" @@ -319,11 +323,10 @@ func WriteOutput(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Resul return err } dst := filepath.Join(opts.OutputDir, "analysis.json") - f, err := os.CreateTemp(opts.OutputDir, ".analysis.json.*") + f, tmp, err := createTemp(opts.OutputDir, ".analysis.json.") if err != nil { return err } - tmp := f.Name() err = StreamJSON(f, app, l3, l4, opts) if err == nil { err = f.Sync() // the rename below must not publish unflushed bytes @@ -331,11 +334,6 @@ func WriteOutput(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Resul if cerr := f.Close(); err == nil { err = cerr } - if err == nil { - // CreateTemp makes the file 0600; give it the mode os.Create would, - // 0666 less the process umask. - err = os.Chmod(tmp, 0o666&^currentUmask()) - } if err == nil { err = os.Rename(tmp, dst) } @@ -344,3 +342,18 @@ func WriteOutput(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Resul } return err } + +// createTemp creates a new file prefix+ in dir with mode 0666, so the +// kernel applies the process umask as os.Create would (os.CreateTemp uses +// 0600). O_EXCL makes a name collision an error, and the name is retried. +func createTemp(dir, prefix string) (*os.File, string, error) { + for range 10000 { + name := filepath.Join(dir, prefix+strconv.FormatUint(uint64(rand.Uint32()), 36)) + f, err := os.OpenFile(name, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0o666) + if errors.Is(err, fs.ErrExist) { + continue + } + return f, name, err + } + return nil, "", fmt.Errorf("creating a temporary file in %s: too many name collisions", dir) +} diff --git a/internal/core/umask_other.go b/internal/core/umask_other.go deleted file mode 100644 index b8914ad..0000000 --- a/internal/core/umask_other.go +++ /dev/null @@ -1,9 +0,0 @@ -//go:build !unix - -package core - -import "io/fs" - -// currentUmask returns 0 where the platform has no umask (Windows): the file -// gets mode 0666, which os.Create would also give it. -func currentUmask() fs.FileMode { return 0 } diff --git a/internal/core/umask_unix.go b/internal/core/umask_unix.go deleted file mode 100644 index 311b90d..0000000 --- a/internal/core/umask_unix.go +++ /dev/null @@ -1,16 +0,0 @@ -//go:build unix - -package core - -import ( - "io/fs" - "syscall" -) - -// currentUmask returns the process umask. The syscall can only set it, so it -// is set to 0 and restored at once, before the chmod that uses the result. -func currentUmask() fs.FileMode { - m := syscall.Umask(0) - syscall.Umask(m) - return fs.FileMode(m) -} diff --git a/internal/core/umask_unix_test.go b/internal/core/writeoutput_unix_test.go similarity index 100% rename from internal/core/umask_unix_test.go rename to internal/core/writeoutput_unix_test.go diff --git a/internal/sdg/join.go b/internal/sdg/join.go index 8bcaa5f..e0064ae 100644 --- a/internal/sdg/join.go +++ b/internal/sdg/join.go @@ -288,7 +288,8 @@ func (j *join) externalOf(fn *ssa.Function) (sig string, ok bool) { } // isLibrary mirrors the L2 rule: declared in a file outside the project or -// vendored; without a file, in a package that is not a project package. +// vendored; without a file, by L2's semantic_analysis.LibraryByPath (the +// go.mod module prefix, else whether the package is a project package). func (j *join) isLibrary(obj *types.Func) bool { if tf := j.fset.File(obj.Pos()); tf != nil { if !utils.IsWithin(j.projectDir, tf.Name()) { diff --git a/internal/sdg/vertices.go b/internal/sdg/vertices.go index a98cf56..b7067c1 100644 --- a/internal/sdg/vertices.go +++ b/internal/sdg/vertices.go @@ -504,11 +504,12 @@ func unionByOf(order []Target, p *ports, list func(*formalSet) []string) []strin // function type has, how many values it passes (f(g()) passes each g // result), and how many results it has (-1 when its type is unknown). type callShape struct { - method bool - nParams int - values int - results int - noSig bool // the call's type is unknown: nParams is the value count + method bool + nParams int + values int + results int + noSig bool // the call's type is unknown: nParams is the value count + methodExpr bool // go/types recorded the callee as a method expression T.M } func callShapeOf(call *ast.CallExpr, info *types.Info) callShape { @@ -521,8 +522,9 @@ func callShapeOf(call *ast.CallExpr, info *types.Info) callShape { } } if sel, ok := ast.Unparen(call.Fun).(*ast.SelectorExpr); ok { - if x := info.Selections[sel]; x != nil && x.Kind() == types.MethodVal { - s.method = true + if x := info.Selections[sel]; x != nil { + s.method = x.Kind() == types.MethodVal + s.methodExpr = x.Kind() == types.MethodExpr } } if t := info.TypeOf(call.Fun); t != nil { @@ -531,11 +533,10 @@ func callShapeOf(call *ast.CallExpr, info *types.Info) callShape { } if sig == nil { // No signature (a package with type errors): the value count stands - // in for the param count, but it must never make bind read a bound - // method value f(x) on a one-param method as a method expression - // (nParams == params+1). Only a selection go/types recorded as - // MethodExpr is one, and without a signature there is none; cap - // nParams so the receiver binds through the function value. + // in for the param count. The count alone must not decide that a call + // is a method expression (a bound method value f(x) on a one-param + // method passes params+1 values too), so bind then relies only on + // the selection go/types recorded (methodExpr). s.nParams = s.values s.noSig = true return s @@ -568,7 +569,7 @@ func (s callShape) bind(fs *formalSet) map[int]int { base, off := 0, 0 // declared param d is fs.inPos[base+d] and binds call param d+off if fs.recv { base = 1 - if !s.method && !s.noSig && s.nParams == fs.params+1 { + if !s.method && (s.methodExpr || !s.noSig && s.nParams == fs.params+1) { off = 1 out[0] = 0 // a method expression always passes its receiver } else { diff --git a/internal/sdg/vertices_test.go b/internal/sdg/vertices_test.go index ab85d6b..7b2bbf4 100644 --- a/internal/sdg/vertices_test.go +++ b/internal/sdg/vertices_test.go @@ -401,4 +401,10 @@ func TestCallShape_NoSignatureNeverBindsMethodExpression(t *testing.T) { if got := expr.bind(fs); got[0] != 0 || got[1] != 1 { t.Errorf("method expression: want value 0 → receiver, value 1 → param, got %v", got) } + // Without a signature, a selection go/types recorded as MethodExpr is + // still a method expression. + exprNoSig := callShape{values: 2, nParams: 2, results: -1, noSig: true, methodExpr: true} + if got := exprNoSig.bind(fs); got[0] != 0 || got[1] != 1 { + t.Errorf("method expression without a signature: want value 0 → receiver, value 1 → param, got %v", got) + } } diff --git a/internal/syntactic_analysis/v2emit/gate_l4_test.go b/internal/syntactic_analysis/v2emit/gate_l4_test.go index 2c61a2d..42b4891 100644 --- a/internal/syntactic_analysis/v2emit/gate_l4_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l4_test.go @@ -898,14 +898,16 @@ func TestSupersetL4_DetectsEveryViolation(t *testing.T) { b string want string // a substring of the error; "" means no error }{ - "identical": {base, ""}, - "pure addition": {`{"application":{"call_graph":[{"src":"a","dst":"b"},{"src":"a","dst":"c"}],"x":{"f":{"ddg":[{"src":"0","dst":"1","var":"w"},{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1,2,3],"new":1}},"extra":true}}`, ""}, - "removed key": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"list":[1,2]}}}}`, "k missing at L4"}, - "changed value": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"w","list":[1,2]}}}}`, "value changed"}, - "shrunk list": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1]}}}}`, "list shrank"}, - "changed type": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":{"a":1}}}}}`, "list shrank or changed type"}, - "removed ddg edge": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"0","dst":"1","var":"w"}],"k":"v","list":[1,2]}}}}`, "missing or changed at L4"}, - "changed call_graph": {`{"application":{"call_graph":[{"src":"a","dst":"z"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1,2]}}}}`, "call_graph: edge"}, + "identical": {base, ""}, + "pure addition": {`{"application":{"call_graph":[{"src":"a","dst":"b"},{"src":"a","dst":"c"}],"x":{"f":{"ddg":[{"src":"0","dst":"1","var":"w"},{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1,2,3],"new":1}},"extra":true}}`, ""}, + "removed key": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"list":[1,2]}}}}`, "k missing at L4"}, + "changed value": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"w","list":[1,2]}}}}`, "value changed"}, + "shrunk list": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1]}}}}`, "list shrank"}, + "changed type": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":{"a":1}}}}}`, "list shrank or changed type"}, + "removed ddg edge": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"0","dst":"1","var":"w"}],"k":"v","list":[1,2]}}}}`, "missing or changed at L4"}, + "object to scalar": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":5}}}`, "object changed type"}, + "edge list to object": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":{},"k":"v","list":[1,2]}}}}`, ".ddg: changed type"}, + "changed call_graph": {`{"application":{"call_graph":[{"src":"a","dst":"z"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1,2]}}}}`, "call_graph: edge"}, } { var errs []string supersetL4("$", parse(base), parse(c.b), &errs) From 278b98143851c96850935b45bc82cb5eaf19742b Mon Sep 17 00:00:00 2001 From: Saurabh Sinha Date: Wed, 28 Oct 2026 10:34:51 -0400 Subject: [PATCH 65/65] docs: L4 evaluation at the final head --- README.md | 6 +- docs/design/notes/l3-dataflow-metrics.md | 216 ++++++++++++++--------- docs/design/notes/l3-eval-2026-10.md | 33 +++- docs/design/specs/v2-l3-l4-dataflow.md | 9 +- 4 files changed, 170 insertions(+), 94 deletions(-) diff --git a/README.md b/README.md index ffea422..11f93f5 100644 --- a/README.md +++ b/README.md @@ -666,9 +666,9 @@ check the following. VTA `call_graph` edges, external `receiver` / `body`, `degraded`). Code that assumes every body node has a `span`, or that every `ddg` edge has `prov: ["ssa"]`, must handle the L4 kinds. `-a 4 -t` is an error (whole-program; D9). `--library-models` is new and valid only at L4. - **Measured cost** (2026-10-27, analyzer built with go1.27.3, eight real apps). On 7 of 8 - apps, the peak RSS of `-a 4` is 1.05–1.4× that of `-a 3`; Kubernetes is 1.05×, at 7.7 GB. - nats-server is the outlier at 3.6×, because its globals fan out through one recursive 822-callable + **Measured cost** (2026-10-28, analyzer built with go1.27.3, eight real apps). On 7 of 8 + apps, the peak RSS of `-a 4` is 1.0–1.5× that of `-a 3`; Kubernetes takes 38 s at 7.9 GB. + nats-server is the outlier at 3.8×, because its globals fan out through one recursive 822-callable SCC. The JSON output is 1.9–23× that of `-a 3`. Per-app figures are in `docs/design/notes/l3-dataflow-metrics.md` § L4. - **Neo4j:** diff --git a/docs/design/notes/l3-dataflow-metrics.md b/docs/design/notes/l3-dataflow-metrics.md index 8b2d690..a078db5 100644 --- a/docs/design/notes/l3-dataflow-metrics.md +++ b/docs/design/notes/l3-dataflow-metrics.md @@ -806,15 +806,17 @@ dispatch targets to `call_graph` (`prov:["vta"]`), library callees get stand-in `prov:["points-to"]`. The semantics are in [`v2-l3-l4-dataflow.md`](../specs/v2-l3-l4-dataflow.md) §4.6–§4.16 and §6. -- **When and where:** measured 2026-10-27 on the same host, an Apple M3 Max (16 cores, 64 GB, +- **When and where:** measured 2026-10-28 on the same host, an Apple M3 Max (16 cores, 64 GB, macOS 15). The analyzer was built by `scripts/eval_l3.sh` with **go1.27.3** (`runtime.Version()`), and every run used `GOTOOLCHAIN=go1.27.3`. Every number in this section is go1.27.3 unless it says otherwise: figures quoted from earlier task reports carry their toolchain in parentheses. Absolute RSS differs by about 1 GB between go1.26.4 and go1.27.3 on nats-server (Task 15b), so do not compare figures across toolchains. -- **Binary:** `feat/l4-sdg` at `c230592`. The `+dirty` marker covers uncommitted docs only; the - analyzer code is that commit's. It includes the Task 17 alias rounds (`04244db`, `18fca40`), - the capture fixes (`078616f`, `cd37383`) and the Task 18a fix-ups. +- **Binary:** `feat/l4-sdg` at `98ebfef`, a clean tree: the final head of the branch, every + review fix included. The first L4 eval (2026-10-27, `c230592` on the pre-rebase branch; its + analyzer and scripts are `a2e7ea0` here) predates the final-review fixes (VTA through + wrappers, SSA for every package variant, the library-origin CHA complement); § What changed + since the earlier measurements compares the two. - **Protocol:** `CANGO_LEVELS="2 3 4" scripts/eval_l3.sh` over the same eight prepared trees and pinned commits as the L3 sections. Each app runs the L3 sequence (§ How to reproduce) and then the **`-a 4` triple**: `-a 4 -j 1 -v`, `-a 4 -j 16`, and a second `-a 4 -j 16`, all @@ -833,47 +835,51 @@ dispatch targets to `call_graph` (`prov:["vta"]`), library callees get stand-in |---|---|---| | `-a 4` complete, no crash | 6/6 | 2/2 | | `-j 1` = `-j 16` = `-j 16`, byte-identical | 6/6 | 2/2 | -| wall `-a 3` → `-a 4`, `-j 1` | 12.9 → 45.4 s | 13.5 → 31.5 s / 5.6 → 11.0 s | -| peak RSS `-a 4 -j 1` (× `-a 3`) | 290 MB (1.08×) to 5.9 GB (1.14×); nats-server 4.1 GB (3.64×) | **7.7 GB (1.05×)** / 2.4 GB (1.15×) | -| `analysis.json` `-a 3` → `-a 4` | 263 MB → 2.66 GB (10.1×; nats-server alone 1.78 GB) | 343 MB → 1.34 GB (3.9×) / 193 → 360 MB (1.9×) | -| vertices | 2.35M | 1.15M / 242k | -| `param_in` / `param_out` | 1.16M / 0.70M | 449k / 380k / 91k / 88k | -| `summary` | 3.55M | 1.26M / 206k | -| port `ddg` (`reaching-defs`) / `points-to` `ddg` | 4.69M / 653k | 1.84M / 373k / 439k / 39k | -| stand-ins (externals with a body) | 4,575 | 1,677 / 129 | -| bound call sites with a VTA-only target | 2.6–6.6% | 11.3% (11,294) / 0.4% | +| wall `-a 3` → `-a 4`, `-j 1` | 12.7 → 48.0 s | 12.8 → 38.4 s / 5.2 → 10.6 s | +| peak RSS `-a 4 -j 1` (× `-a 3`) | 287 MB (1.15×) to 5.9 GB (1.17×); nats-server 4.1 GB (3.77×) | **7.9 GB (0.99–1.13×)** / 2.5 GB (1.23×) | +| `analysis.json` `-a 3` → `-a 4` | 263 MB → 2.83 GB (10.8×; nats-server alone 1.78 GB) | 343 MB → 1.43 GB (4.2×) / 193 → 360 MB (1.9×) | +| vertices | 2.49M | 1.22M / 242k | +| `param_in` / `param_out` | 1.24M / 0.77M | 495k / 432k / 91k / 88k | +| `summary` | 3.79M | 1.36M / 206k | +| port `ddg` (`reaching-defs`) / `points-to` `ddg` | 4.91M / 654k | 1.94M / 374k / 439k / 39k | +| stand-ins (externals with a body) | 4,591 | 1,679 / 129 | +| bound call sites with a VTA-only target | 2.8–13.0% | 13.5% (13,770) / 0.4% | | consistency checks (L3 and L4) | all PASS | all PASS | | degraded codes | `range_func` (5 apps), `unmapped_ssa` (4) | `range_func`, `unmapped_ssa` / none | **Key takeaways.** -1. **The scale gate passes.** The Kubernetes root completes `-a 4` in 31.5 s at `-j 1` (30.0 s at - `-j 16`) with a whole-process peak of 7.7 GB, only 1.05× its `-a 3` peak, and its three - outputs are byte-identical. The `go/packages` load still dominates RSS. L4 adds 18 s of wall - time: SSA takes 1.0 s, VTA 2.3 s and the overlay 10.3 s, and the rest is emitting the 1.34 GB - file. -2. **Memory is not the problem L4 has; output size is.** RSS grows by 5–40% on every app but - one. nats-server is the outlier at 3.6× (4.1 GB). It has one giant recursive SCC whose - global footprint (A4) gives 91% of its 1.58M vertices a `:` `of`. The JSON grows - 4–10× on the hand-written apps and 22.7× on nats-server (78 MB → 1.78 GB). Most of those +1. **The scale gate passes.** The Kubernetes root completes `-a 4` in 38.4 s at `-j 1` (36.2 s at + `-j 16`) with a whole-process peak of 7.9 GB, and its three outputs are byte-identical. Its + `-a 3` peak varies from run to run (7.0–8.0 GB over three runs, against a steady 7.9 GB at + `-a 4`), so the ratio is 0.99–1.13×; the `go/packages` load still dominates RSS. L4 adds + 26 s of wall time: SSA takes 0.9 s, the CHA complement's library-origin fixpoint about 6 s, + VTA 2.4 s and the overlay 11.1 s, and the rest is emitting the 1.43 GB file. +2. **Memory is not the problem L4 has; output size is.** RSS grows by 15–50% on five of the + six apps, by up to 13% on the Kubernetes root and by 23% on `k8s-api`. nats-server is the + outlier at 3.8× (4.1 GB). It has one giant recursive SCC whose global footprint (A4) gives + 91% of its 1.59M vertices a `:` `of`. The JSON grows + 3–10× on the hand-written apps and 22.7× on nats-server (78 MB → 1.78 GB). Most of those bytes are `:` vertices, their port wiring, and param edges carrying full global ids. 3. **Summaries are near all-to-all, as A15 predicts.** 74–94% of call sites with summary edges have every `actual_in` → `actual_out` pair. At statement granularity the host already links every use to every def (the L3 call rule), so a summary contributes the callee's global and by-ref footprint, not intra-caller precision. 4. **VTA matters on a small share of call sites, but they are the ones taint needs.** Of the call - sites that bind a target, 2.6–6.6% gain one that only VTA found on the six apps, and 11.3% on - the Kubernetes root: interface calls (`store.Driver` in memos, `Repo` in the done fixture), - function values, and the D7 interface-method binding on top. `k8s-api`, generated marshallers - with no interfaces, has 0.4%. -5. **The points-to delta is moderate and strictly additive.** It is 7–19% of the port `ddg` on - the six apps (nats-server 9.6%) and 20% / 9% on Kubernetes. The superset gate holds: every - `-a 3` edge is in the `-a 4` file unchanged. + sites that bind a target, 2.8–6.7% gain one that only VTA or the CHA complement found on five + of the six apps, 13.0% on openbao and 13.5% on the Kubernetes root: interface calls + (`store.Driver` in memos, `Repo` in the done fixture), function values, receivers that came + out of library code, and the D7 interface-method binding on top. `k8s-api`, generated + marshallers with no interfaces, has 0.4%. +5. **The points-to delta is moderate and strictly additive.** It is 4–28% of the port `ddg` on + five of the six apps (nats-server 9.6%), 52% on cobra, whose port `ddg` is small, and 19% / 9% + on Kubernetes. The superset gate holds: every `-a 3` edge is in the `-a 4` file unchanged. 6. **Global port wiring is the volume driver.** On nats-server, 2.87M of 3.09M port `ddg` edges (93%) carry a global `var`. On listmonk, one function, `cmd.initHTTPHandlers` (a route table that registers about 300 handlers), holds 475k of the app's 523k port edges (91%). It is the host of hundreds of calls, each with the same global actuals, and the coarse host wiring - multiplies them: every host use feeds every `actual_in`. + multiplies them: every host use feeds every `actual_in`. (Both splits are from the first L4 + eval; neither app's port `ddg` has moved by more than 0.1% since.) 7. **Degradation is reported, never silent.** Five of the six apps and the Kubernetes root use range-over-func somewhere (`range_func`, D8). Four apps and the root carry `unmapped_ssa`, which comes almost entirely from closures in package-level `var` initializers. No app hit @@ -886,52 +892,58 @@ go1.27.3, `--eager`, one timed run per configuration. | app | wall `-a 3 -j 1` | wall `-a 4 -j 1` | wall `-a 4 -j 16` | user CPU `-a 4 -j 1` | RSS `-a 3 -j 1` | RSS `-a 4 -j 1` | RSS × | JSON `-a 3` | JSON `-a 4` | JSON × | SSA | VTA | overlay | `-a 4` identical | |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| -| cobra | 0.45 | 0.54 | 0.52 | 1.13 | 268 | 290 | 1.08 | 2.1 | 6.7 | 3.2 | 9 ms | 5 ms | 74 ms | yes | -| listmonk | 0.93 | 2.37 | 2.37 | 5.84 | 989 | 1,384 | 1.40 | 10.6 | 103.3 | 9.8 | 33 ms | 31 ms | 1.04 s | yes | -| memos | 2.47 | 4.68 | 4.43 | 13.62 | 2,501 | 3,153 | 1.26 | 43.2 | 174.5 | 4.0 | 127 ms | 148 ms | 1.27 s | yes | -| nats-server | 2.31 | 23.48 | 23.27 | 42.70 | 1,128 | 4,103 | **3.64** | 78.2 | 1,775.2 | **22.7** | 224 ms | 183 ms | 14.76 s | yes | -| openbao | 5.01 | 10.77 | 10.27 | 30.13 | 5,173 | 5,893 | 1.14 | 101.6 | 473.4 | 4.7 | 292 ms | 403 ms | 3.59 s | yes | -| dae (linux) | 1.71 | 3.53 | 3.23 | 8.65 | 1,385 | 1,743 | 1.26 | 27.5 | 130.7 | 4.8 | 125 ms | 69 ms | 1.04 s | yes | -| **kubernetes** | 13.49 | **31.53** | 30.01 | 68.56 | 7,315 | **7,697** | 1.05 | 343.2 | 1,341.5 | 3.9 | 971 ms | 2.30 s | 10.31 s | **yes** | -| k8s-api | 5.55 | 10.98 | 9.44 | 24.27 | 2,044 | 2,356 | 1.15 | 192.8 | 360.2 | 1.9 | 517 ms | 291 ms | 3.44 s | yes | - -`-j 16` barely helps at `-a 4`, by at most 1.5 s: SSA, VTA and the summary pass run serially by -design (D2, `ssa.BuildSerially`; the SCC pass is serial for determinism), and emission is one -stream. +| cobra | 0.44 | 0.55 | 0.54 | 1.19 | 250 | 287 | 1.15 | 2.1 | 6.8 | 3.2 | 13 ms | 5 ms | 69 ms | yes | +| listmonk | 0.90 | 2.37 | 2.38 | 5.85 | 912 | 1,369 | 1.50 | 10.6 | 103.3 | 9.8 | 35 ms | 29 ms | 1.00 s | yes | +| memos | 2.48 | 5.05 | 4.79 | 13.52 | 2,319 | 2,894 | 1.25 | 43.2 | 184.6 | 4.3 | 190 ms | 144 ms | 1.29 s | yes | +| nats-server | 2.29 | 23.82 | 23.26 | 43.53 | 1,110 | 4,182 | **3.77** | 78.2 | 1,776.7 | **22.7** | 217 ms | 182 ms | 14.68 s | yes | +| openbao | 4.77 | 12.06 | 11.44 | 32.30 | 5,072 | 5,925 | 1.17 | 101.6 | 569.7 | 5.6 | 300 ms | 390 ms | 3.96 s | yes | +| dae (linux) | 1.83 | 4.17 | 3.89 | 10.35 | 1,492 | 2,104 | 1.41 | 27.5 | 193.3 | 7.0 | 82 ms | 89 ms | 1.39 s | yes | +| **kubernetes** | 12.75 | **38.37** | 36.16 | 79.10 | 8,103 | **8,076** | 1.00 | 343.2 | 1,425.8 | 4.2 | 935 ms | 2.44 s | 11.06 s | **yes** | +| k8s-api | 5.22 | 10.58 | 9.50 | 24.87 | 2,052 | 2,533 | 1.23 | 192.8 | 360.2 | 1.9 | 443 ms | 295 ms | 3.41 s | yes | + +`-j 16` barely helps at `-a 4`, by at most 2.2 s: SSA, VTA, the CHA complement and the summary +pass run serially by design (D2, `ssa.BuildSerially`; the SCC pass is serial for determinism), +and emission is one stream. + +**Kubernetes RSS is noisy at `-a 3`.** Two repeat runs of the same binary, `-j 1 --eager`, gave +`-a 3` 13.73 s / 7,139 MB and 13.57 s / 8,213 MB, and `-a 4` 40.08 s / 8,079 MB and 39.17 s / +8,124 MB. The `-a 4` peak is steady at 7.9 GB; the `-a 3` peak moves by about 1 GB with GC +timing during the load, so the ratio over the three paired runs is 0.99–1.13×. | app | vertices | `formal_in` / `formal_out` | `actual_in` / `actual_out` | `:` share of vertices | `param_in` | `param_out` | `summary` | sites with summary: all-to-all | port `ddg` | `points-to` `ddg` | stand-ins (formals) | |---|---|---|---|---|---|---|---|---|---|---|---| -| cobra | 7,491 | 945 / 965 | 2,593 / 2,988 | 21.1% | 2,635 | 3,038 | 6,050 | 1,232: 74.1% | 10,790 | 5,637 | 91 (337) | -| listmonk | 58,546 | 3,698 / 3,721 | 26,432 / 24,695 | 46.6% | 28,270 | 26,653 | 40,912 | 6,105: 80.3% | 522,668 | 20,767 | 506 (2,021) | -| memos | 171,002 | 26,770 / 27,174 | 55,933 / 61,125 | 28.3% | 59,004 | 64,762 | 216,629 | 19,147: 86.6% | 237,601 | 68,956 | 830 (3,248) | -| nats-server | 1,584,949 | 324,205 / 155,426 | 734,728 / 370,590 | **91.2%** | 862,759 | 431,317 | 2,576,921 | 29,878: 78.9% | 3,093,971 | 295,776 | 667 (2,503) | -| openbao | 402,746 | 74,643 / 50,842 | 154,802 / 122,459 | 50.0% | 164,146 | 132,226 | 550,412 | 36,933: 82.0% | 605,515 | 195,288 | 1,639 (6,816) | -| dae | 120,882 | 18,889 / 17,257 | 42,266 / 42,470 | 49.6% | 43,611 | 43,881 | 155,698 | 12,163: 82.7% | 215,620 | 67,138 | 842 (3,298) | -| **kubernetes** | 1,154,746 | 229,049 / 197,197 | 394,069 / 334,431 | 40.9% | 448,676 | 380,030 | 1,260,761 | 97,185: 76.0% | 1,840,380 | 373,490 | 1,677 (6,447) | +| cobra | 7,500 | 945 / 965 | 2,596 / 2,994 | 21.1% | 2,638 | 3,044 | 6,056 | 1,235: 74.2% | 10,799 | 5,637 | 91 (337) | +| listmonk | 58,546 | 3,698 / 3,721 | 26,432 / 24,695 | 46.6% | 28,279 | 26,666 | 40,912 | 6,105: 80.3% | 522,668 | 20,767 | 506 (2,021) | +| memos | 181,608 | 29,866 / 29,282 | 59,137 / 63,323 | 32.5% | 63,547 | 68,487 | 236,583 | 19,178: 85.5% | 250,469 | 68,956 | 834 (3,273) | +| nats-server | 1,585,369 | 324,214 / 155,426 | 734,885 / 370,844 | **91.1%** | 864,091 | 433,667 | 2,577,267 | 30,005: 79.0% | 3,094,417 | 295,787 | 667 (2,503) | +| openbao | 477,113 | 92,049 / 61,133 | 183,481 / 140,450 | 57.2% | 212,468 | 171,261 | 680,354 | 37,783: 81.7% | 724,250 | 195,336 | 1,646 (6,848) | +| dae | 177,435 | 30,942 / 28,116 | 59,920 / 58,457 | 65.6% | 71,926 | 69,991 | 246,279 | 12,210: 81.6% | 311,385 | 67,149 | 847 (3,317) | +| **kubernetes** | 1,215,278 | 241,510 / 205,241 | 416,744 / 351,783 | 43.1% | 495,377 | 431,709 | 1,361,329 | 99,349: 75.7% | 1,943,025 | 373,553 | 1,679 (6,456) | | k8s-api | 242,414 | 27,556 / 35,489 | 91,428 / 87,941 | 7.2% | 91,428 | 87,941 | 205,643 | 40,856: 93.7% | 439,414 | 38,963 | 129 (812) | Notes: - The `-v` log's "port edges" count includes the `points-to` edges. The table splits them: port = `reaching-defs`, plus `points-to`. - A stand-in is an external with a `body`. Every external a call site targets has one, except - for the two to five per app whose formals are empty (`runtime.Gosched`). + for the zero to thirteen per app whose formals are empty (`runtime.Gosched`). | app | call nodes | sites that bind a target | with a VTA-only target | with ≥ 2 targets | `call_graph` VTA edges | VTA function set (library targets) | VTA edges | sites with targets (log) | non-call capture sites | degraded | |---|---|---|---|---|---|---|---|---|---|---| -| cobra | 1,474 | 1,240 | 32 (2.6%) | 14 | 45 | 411 (83) | 1,294 | 1,244 | 37 | none | -| listmonk | 7,057 | 6,126 | 220 (3.6%) | 63 | 272 | 1,809 (640) | 7,304 | 6,194 | 47 | `range_func` (1 pkg, 1 callable) | -| memos | 23,781 | 19,209 | 997 (5.2%) | 516 | 1,059 | 7,893 (1,443) | 21,417 | 19,436 | 380 | `range_func` (7, 10); `unmapped_ssa` (2 fn / 4 calls) | -| nats-server | 38,335 | 30,007 | 1,633 (5.4%) | 898 | 2,133 | 10,473 (718) | 39,668 | 30,679 | 587 | `range_func` (2, 27); `unmapped_ssa` (25 / 4) | -| openbao | 51,991 | 37,139 | 2,461 (6.6%) | 1,802 | 4,389 | 19,978 (3,181) | 58,876 | 38,377 | 822 | `range_func` (5, 6); `unmapped_ssa` (10 / 12) | -| dae | 15,354 | 12,218 | 628 (5.1%) | 228 | 888 | 4,498 (1,286) | 14,104 | 12,598 | 263 | `range_func` (5, 5); `unmapped_ssa` (40 / 125) | -| **kubernetes** | 193,481 | 99,757 | 11,294 (11.3%) | 4,652 | 15,897 | 94,601 (41,107) | 249,493 | 102,407 | 3,890 | `range_func` (26, 49); `unmapped_ssa` (3,720 / 39,700) | +| cobra | 1,474 | 1,243 | 35 (2.8%) | 14 | 45 | 411 (83) | 1,294 | 1,247 | 37 | none | +| listmonk | 7,057 | 6,126 | 224 (3.7%) | 67 | 272 | 1,809 (640) | 7,304 | 6,194 | 47 | `range_func` (1 pkg, 1 callable) | +| memos | 23,781 | 19,248 | 1,277 (6.6%) | 797 | 1,069 | 7,893 (1,443) | 21,417 | 19,475 | 380 | `range_func` (7, 10); `unmapped_ssa` (2 fn / 4 calls) | +| nats-server | 38,335 | 30,134 | 1,800 (6.0%) | 1,069 | 2,133 | 10,473 (718) | 39,668 | 30,806 | 587 | `range_func` (2, 27); `unmapped_ssa` (25 / 4) | +| openbao | 51,991 | 37,999 | 4,928 (13.0%) | 4,226 | 4,743 | 19,978 (3,181) | 58,876 | 39,232 | 822 | `range_func` (5, 6); `unmapped_ssa` (10 / 12) | +| dae | 15,354 | 12,269 | 817 (6.7%) | 425 | 890 | 4,498 (1,286) | 14,104 | 12,645 | 263 | `range_func` (5, 5); `unmapped_ssa` (40 / 125) | +| **kubernetes** | 193,481 | 101,948 | 13,770 (13.5%) | 5,514 | 15,943 | 94,601 (41,107) | 249,493 | 104,586 | 3,890 | `range_func` (26, 49); `unmapped_ssa` (3,720 / 39,700) | | k8s-api | 87,808 | 40,856 | 172 (0.4%) | 0 | 172 | 26,527 (8,570) | 52,526 | 40,856 | 175 | none | - **"Sites that bind a target"** are call nodes with at least one `param_in` from an `actual_in`. A call node binds nothing when its callee has no formals (`runtime.Gosched`), when the CFG pruned it, or when it is unresolved: a builtin, a conversion, or a function value VTA did not resolve. -- **"VTA-only"** means at least one bound target is not the call node's static `callee`. +- **"VTA-only"** means at least one bound target is not the call node's static `callee`. It + includes the CHA complement's targets (§ Limitations (L4)), which the dispatch adds before VTA. - **"Sites with targets (log)"** is the analyzer's own dispatch count. It also counts sites whose only target has no formals. - **Degraded:** the `range_func` figures are (packages, callables) and the `unmapped_ssa` @@ -939,20 +951,53 @@ Notes: ### What changed since the earlier measurements -The task reports measured nats-server at several points on the branch, at `-j 8`, on the -toolchain given with each figure. On the eval binary, go1.27.3 `-j 1`: -- **summary** is 2,576,921, against 2,547,371 after the Task 17 alias round (`04244db`; +**Since the first L4 eval (`c230592`, 2026-10-27, same host and toolchain).** The final-review +fixes added dispatch targets, and every new target adds formals, actuals and their edges. The +L3 part of every `-a 4` file is unchanged (its body, `cfg`, `cdg` and `ssa`-`ddg` counts equal the +`-a 3` file's on all eight apps). Growth in vertices / `param_in` / `summary`, and in bound sites +with a VTA-only target: + +| app | vertices | `param_in` | `summary` | VTA-only sites | wall `-a 4 -j 1` | +|---|---|---|---|---|---| +| cobra | +0.1% | +0.1% | +0.1% | 32 → 35 | 0.54 → 0.55 s | +| listmonk | 0 | +0.03% | 0 | 220 → 224 | 2.37 → 2.37 s | +| memos | +6.2% | +7.7% | +9.2% | 997 → 1,277 | 4.68 → 5.05 s | +| nats-server | +0.03% | +0.15% | +0.01% | 1,633 → 1,800 | 23.48 → 23.82 s | +| openbao | +18.5% | +29.4% | +23.6% | 2,461 → 4,928 | 10.77 → 12.06 s | +| dae | +46.8% | +64.9% | +58.2% | 628 → 817 | 3.53 → 4.17 s | +| kubernetes | +5.2% | +10.4% | +8.0% | 11,294 → 13,770 | 31.53 → 38.37 s | +| k8s-api | 0 | 0 | 0 | 172 → 172 | 10.98 → 10.58 s | + +Bisected on openbao, dae and Kubernetes (a build at each fix commit, `-a 4 -j 1`): +- **The library-origin CHA complement is most of the volume change.** Its intraprocedural first + round (`0a3ea12`) and its interprocedural fixpoint (`7762b6e`) take openbao from 402,746 to + 412,702 to 477,113 vertices, dae from 121,056 to 147,785 to 177,435, and Kubernetes from + 1,164,744 to 1,191,953 to 1,215,278. The earlier fixes (VTA through wrappers `4fb14b5`, SSA for + every package variant `3752427`, `call_graph` edges from VTA only `4a2121d`) add about 2.7k + `param_in` and no vertices on openbao, and 10k vertices on Kubernetes (1,154,746 → 1,164,744). +- **The Kubernetes wall-time increase is the interprocedural fixpoint.** With timestamped `-v` + logs, the time between the SSA join and the VTA line is 8.7 s on `98ebfef` against 2.4 s on + `c230592`. VTA itself takes 2.3–2.4 s in both, so the complement takes about 6 s. The overlay + grows by 0.7 s with the extra vertices, and emission by 0.7 s. On the other apps the + complement takes 0.04–0.7 s (openbao 0.7 s, nats-server 0.5 s). It has no `-v` line of its + own. The bisect builds show the same step: Kubernetes `-a 4` goes from 32.4 s at `0a3ea12` to + 37.7 s at `7762b6e`. + +**Against the task reports.** They measured nats-server at several points on the branch, at +`-j 8`, on the toolchain given with each figure. On the eval binary, go1.27.3 `-j 1`: +- **summary** is 2,577,267, against 2,547,371 after the Task 17 alias round (`a2be9a4`; go1.26.4) (+1.2%); -- **points-to** is 295,776, against 290,499 then and 292,707 after round 2 (`18fca40`; both +- **points-to** is 295,787, against 290,499 then and 292,707 after round 2 (`c74c11b`; both go1.26.4); -- **param_out** is 431,317, against 431k at Task 14 (go1.27.3). +- **param_out** is 433,667, against 431k at Task 14 (go1.27.3). -The growth since `18fca40` comes from: -- the reference-holding capture `formal_out` (`078616f`, MR5); -- the L3 host weak-def of every reference-holding capture (`cd37383`). +The growth since `c74c11b` comes from: +- the reference-holding capture `formal_out` (`7ca9c57`, MR5); +- the L3 host weak-def of every reference-holding capture (`40b03ea`); +- the CHA complement (above). -Both add edges only. The L3 `ddg` on nats-server went from 345,318 (§ Per-app counts) to 358,979 -(+4.0%). That includes `cd37383`'s capture weak defs (chi +0.26%, go1.26.4) and the D1 L3 fixes since the +All add edges only. The L3 `ddg` on nats-server went from 345,318 (§ Per-app counts) to 358,979 +(+4.0%). That includes `40b03ea`'s capture weak defs (chi +0.26%, go1.26.4) and the D1 L3 fixes since the L3 sections were measured. Wall time and RSS in those reports were measured for one stage, not end to end, on another toolchain. Compare the numbers here only with each other. @@ -962,7 +1007,7 @@ unmapped calls, against 3,754 / 39,728 at Task 9 (go1.27.3). Classified from `-v `var _ = Describe(…, func(){…})`; 3,555 of them under `test/`); - 3 are a second `func init()` in one file (`init#N`). -The "method declared above its receiver type" cause Task 9 found is gone (fixed in `5c57a1a`). +The "method declared above its receiver type" cause Task 9 found is gone (fixed in `c6f2bd4`). None of these is a join bug. The other apps have the same cause in smaller numbers (memos 2, dae 40, openbao 10, nats-server 25 functions). @@ -970,7 +1015,8 @@ dae 40, openbao 10, nats-server 25 functions). `sdg.Run` lowers the process GC target to 50 while it runs (Task 15b), and L1–L3 and the emission keep the default. Here is what that costs, measured end to end against a local build -identical except for `l4GCPercent = 100` (not committed). Both builds are go1.27.3, `-a 4 -j 1 +identical except for `l4GCPercent = 100` (not committed), on the first L4 eval's binary +(`c230592`; not re-run on `98ebfef`). Both builds are go1.27.3, `-a 4 -j 1 --eager`, two runs each, and every output is byte-identical to the eval's. | app | GC target in L4 | wall (s) | user CPU (s) | sys (s) | peak RSS (MB) | @@ -993,7 +1039,7 @@ identical except for `l4GCPercent = 100` (not committed). Both builds are go1.27 **Where the end-to-end peak sits.** With the levers on, the `-a 4` peak is no longer in `sdg.Run`. It is in the emission, which streams the envelope but builds each module's subtree whole before -writing it. The largest L4 modules are: +writing it. The largest L4 modules (`c230592`) are: | app | module | JSON | |---|---|---| @@ -1005,7 +1051,8 @@ writing it. The largest L4 modules are: The application-scope `param_in` and `param_out` lists stream (nats-server: 368 MB and 180 MB of JSON). The next lever, if a larger app needs one, is to stream each callable within a module -(spec §4.13, §11). It is not needed at this corpus's scale: Kubernetes is 1.05× its `-a 3` peak. +(spec §4.13, §11). It is not needed at this corpus's scale: Kubernetes is 0.99–1.13× its `-a 3` +peak. ### Consistency at scale (L4) @@ -1020,7 +1067,7 @@ Every check passes on every app: | every `summary` endpoint is a vertex of its callable | 0 failures | | no `summary` edge has `:` vertices at both ends (#30) | 0 | | L3 checks over the L3 part of every `-a 4` body (one `@entry`/`@exit`, reachability, endpoints, `prov == ["ssa"]` on the L3 edges) | all PASS | -| `-a 3` ⊆ `-a 4` | gated on the fixtures (`TestGateL4_SupersetOfL3`); on cobra, listmonk, memos and dae the L3 `body` / `cfg` / `cdg` / `ssa`-`ddg` counts of the `-a 4` file equal the `-a 3` file's exactly (counts, not an edge-by-edge diff; the other four apps were not compared) | +| `-a 3` ⊆ `-a 4` | gated on the fixtures (`TestGateL4_SupersetOfL3`); on all eight apps the L3 `body` / `cfg` / `cdg` / `ssa`-`ddg` counts of the `-a 4` file equal the `-a 3` file's exactly (counts, not an edge-by-edge diff) | The Go-side gates do the arity and kind checks per edge on the fixtures (`TestGateL4_ArityMatchesFormals`, `TestGateL4_NoDanglingEndpoints`, @@ -1031,11 +1078,11 @@ into `sqlRepo.Find`. A handler that never queries does not reach it, and neither entry, `Audit`, reach `Find`'s `id`. `TestGateL4_HTTPToSQLSanitizerIsSDKSemantics` pins the sanitizer variant (§ Limitations). -**Metamorphic relations.** `scripts/metamorphic.sh` (memos, the default) passes on this HEAD with -0 violations in all 14 relations. It covered 20–32 sites for MR1–MR6, MR14 and MR15, and 33M -reachability pairs for MR5 and MR6. The run took about 2 min. The last `--all` run (memos and -nats-server, 0 violations, 23 min, 16.2 GB; go1.27.3, the script's `EVAL_GO`) is from Task 17b -on `e160b17` and `4a6c885`. +**Metamorphic relations.** `scripts/metamorphic.sh` (memos, the default) passes on `98ebfef` with +0 violations in all 14 relations. It covered 17–32 sites for MR1–MR6, MR14 and MR15, and 80M +reachability pairs for MR5 and MR6 (15M for MR14). The run took 105 s. The last `--all` run +(memos and nats-server, 0 violations, 23 min, 16.2 GB; go1.27.3, the script's `EVAL_GO`) is from +Task 17b on `7c29d44` and `3331abe`. ### Insights @@ -1050,10 +1097,11 @@ on `e160b17` and `4a6c885`. member (A4, transitive), so 91% of its vertices are `:` vertices. The cut of global → global summaries (#30) already removed 98M summary edges there (Task 15, go1.27.3; spec §4.9). The vertices and their port edges remain, and they are what sources and sinks can name. -- **VTA is cheap.** Under 2.3 s on Kubernetes, with a function set of 94,601, of which 41,107 are - target-only library functions. The SSA build is about 1 s. The dominant L4 compute cost is the - overlay: flow graphs, summaries and port wiring, where `graphOf`/`bodyFlows` and GC dominate - (Task 18a profile, go1.26.4). The dominant wall-time cost on the large files is emission. +- **VTA is cheap.** About 2.4 s on Kubernetes, with a function set of 94,601, of which 41,107 are + target-only library functions. The SSA build is about 1 s. The CHA complement's + library-origin fixpoint costs more than VTA on Kubernetes (about 6 s) and under 1 s elsewhere. + The dominant L4 compute cost is the overlay: flow graphs, summaries and port wiring, where + `graphOf`/`bodyFlows` and GC dominate (Task 18a profile, go1.26.4). The dominant wall-time cost on the large files is emission. - **k8s-api is the counter-example.** Generated marshallers have few interface calls (0.4% VTA), few globals (7% of vertices) and leaf-heavy call graphs. L4 adds only 1.9× bytes there, the smallest growth in the corpus, because its L3 `ddg` already dominates the file. @@ -1144,7 +1192,7 @@ over-approximation or reported as degraded. 217k of 290k on nats-server at Task 17 (go1.26.4). - `MayHoldRef` also admits value structs that hold a reference, so `s1.n` → `s2.n` between two distinct value copies is a spurious edge. - - Both are sound over-approximations: 7–20% of port `ddg` here. + - Both are sound over-approximations: 4–28% of port `ddg` here (52% on cobra's small one). - **One-level alias offsets only.** - Not caught: offsets deeper than one step (`&x.f.g` against a `*G`), two prefixes of one base that alias each other, and an interface holding a value struct whose reference field is @@ -1152,7 +1200,7 @@ over-approximation or reported as degraded. - These are false negatives of the type-based oracle that an Andersen points-to over `go/ssa` would close (§11). They are the one class of L4 gap that is neither over-approximated nor degraded. -- **L3 capture weak-def breadth (`cd37383`).** +- **L3 capture weak-def breadth (`40b03ea`).** - At L3, a closure's host weak-defines every reference-holding captured base, read-only captures included. That is broader than needed: a read-only capture could weak-define `x.*` without downgrading the base. diff --git a/docs/design/notes/l3-eval-2026-10.md b/docs/design/notes/l3-eval-2026-10.md index 753c2db..8c04d19 100644 --- a/docs/design/notes/l3-eval-2026-10.md +++ b/docs/design/notes/l3-eval-2026-10.md @@ -1,4 +1,4 @@ -# L3 / L4 real-app evaluation (2026-10-24, L4 2026-10-27) +# L3 / L4 real-app evaluation (2026-10-24, L4 2026-10-27 and 2026-10-28) Open-source Go apps at pinned commits, analyzed with `cango -a 3` via `scripts/eval_l3.sh`. Host: Apple M3 Max, 16 cores, 64 GB, macOS. Analyzer built with go1.27.3 (see Finding 2). Every @@ -10,10 +10,37 @@ and "external" split `call_graph` edges by whether `dst` is an `@external` id. L3 volume, edge/node-kind breakdowns, cost, output size, the protobuf-marshaller measurement and consistency checks over these outputs: [`l3-dataflow-metrics.md`](l3-dataflow-metrics.md). +## L4 (`-a 4`) evaluation at the final head (2026-10-28) + +The same pass, rerun on `feat/l4-sdg` at `98ebfef` (every review fix in; go1.27.3, host and +flags as below). **Gate: passed** again: all eight apps complete `-a 4`, and the three `-a 4` +outputs are byte-identical on every app. + +| app | wall `-a 3 -j 1` (s) | wall `-a 4 -j 1` (s) | wall `-a 4 -j 16` (s) | RSS `-a 3` (MB) | RSS `-a 4` (MB) | × | `analysis.json` `-a 4` (MB) | `-a 4` identical | +|---|---|---|---|---|---|---|---|---| +| cobra | 0.44 | 0.55 | 0.54 | 250 | 287 | 1.15 | 6.8 | yes | +| listmonk | 0.90 | 2.37 | 2.38 | 912 | 1,369 | 1.50 | 103.3 | yes | +| memos | 2.48 | 5.05 | 4.79 | 2,319 | 2,894 | 1.25 | 184.6 | yes | +| nats-server | 2.29 | 23.82 | 23.26 | 1,110 | 4,182 | 3.77 | 1,776.7 | yes | +| openbao | 4.77 | 12.06 | 11.44 | 5,072 | 5,925 | 1.17 | 569.7 | yes | +| dae (linux) | 1.83 | 4.17 | 3.89 | 1,492 | 2,104 | 1.41 | 193.3 | yes | +| **kubernetes** | 12.75 | **38.37** | 36.16 | 8,103 | **8,076** | 1.00 | 1,425.8 | **yes** | +| k8s-api | 5.22 | 10.58 | 9.50 | 2,052 | 2,533 | 1.23 | 360.2 | yes | + +- The degraded codes are the same as on 2026-10-27. +- The Kubernetes root's `-a 3` peak is noisy (7.0–8.0 GB over three runs). Its `-a 4` peak is a + steady 7.9 GB, so the ratio is 0.99–1.13×. +- **What changed.** The final-review fixes added dispatch targets, so `-a 4` grew: + - vertices grew by 5–47% on openbao, dae, memos and Kubernetes, and by under 0.2% elsewhere. + - the Kubernetes root's `-a 4` wall time rose from 31.5 to 38.4 s, about 6 s of it in the + CHA complement's interprocedural library-origin fixpoint. + - Per-app counts and the bisect are in [`l3-dataflow-metrics.md` § L4](l3-dataflow-metrics.md#l4). + ## L4 (`-a 4`) evaluation (2026-10-27) One `CANGO_LEVELS="2 3 4" scripts/eval_l3.sh` pass over the same eight prepared trees, with the -binary the script builds from `feat/l4-sdg` at `c230592`: +binary the script builds from `feat/l4-sdg` at `c230592` (pre-rebase; `a2e7ea0` after the rebase +onto `main`), before the final-review fixes: - `runtime.Version()` go1.27.3; runs use `GOTOOLCHAIN=go1.27.3`; - the host as above, j = 16, every run `--eager`. @@ -43,7 +70,7 @@ Memory is reported, not gated (D3). Peak RSS is for the whole process. where the peak sits, the consistency checks and the limitations are in [`l3-dataflow-metrics.md` § L4](l3-dataflow-metrics.md#l4). - **Findings.** The eval found no crash and no nondeterminism. One script fix was needed: - `eval_l3.sh` learned the `-a 4` triple and its byte check (`c230592`). `--all` still includes + `eval_l3.sh` learned the `-a 4` triple and its byte check (`a2e7ea0`). `--all` still includes cockroach, which is not prepared on this host, so the run names the eight apps. ## Corpus change: Kubernetes replaces cockroach as the scale benchmark diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index 2afdc05..e0b8875 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -1193,10 +1193,11 @@ the six apps and both Kubernetes modules complete `-a 4` without crash and deter (`-j 1`/`-j N` and two runs on Kubernetes, not just fixtures); memory is reported per app as absolute `-a 4` peak RSS and its ratio to `-a 3`, not gated (D3, §4.13); no hard time budget, but timings are reported and a regression budget is set after the first measurement. Results go in `l3-dataflow-metrics.md` § L4. -**Measured 2026-10-27 (go1.27.3, `c230592`):** the gate passed on all eight. Every app completes -`-a 4` and its `-j 1` / `-j 16` / rerun outputs are byte-identical. The Kubernetes root takes -31.5 s at a 7.7 GB peak (1.05× `-a 3`). nats-server is the outlier at 3.6× RSS and 1.78 GB of -JSON. The full results, insights and limitations are in `l3-dataflow-metrics.md` § L4. +**Measured 2026-10-28 (go1.27.3, `98ebfef`, the final head):** the gate passed on all eight. +Every app completes `-a 4` and its `-j 1` / `-j 16` / rerun outputs are byte-identical. The +Kubernetes root takes 38.4 s at a 7.9 GB peak (0.99–1.13× a noisy `-a 3` peak). nats-server is +the outlier at 3.8× RSS and 1.78 GB of JSON. (The first measurement, 2026-10-27, before the +final-review fixes: 31.5 s at 7.7 GB.) The full results, insights and limitations are in `l3-dataflow-metrics.md` § L4. ## 9. Phasing and release