diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 380ebc1..3016800 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -24,6 +24,9 @@ jobs: - name: Test run: go test ./... -count=1 + - name: Metamorphic script arguments + run: scripts/metamorphic_args_test.sh + # The release binary is built with the latest Go (release.yml), so test on # it too; the job above tests the go.mod minimum. test-stable: diff --git a/CLAUDE.md b/CLAUDE.md index b918fbf..f4425d4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -95,10 +95,39 @@ committed spec is the full transcript: `docs/design/specs/v2-l1-emission.md`. - **`k_limit`** (envelope, L3+ only) = `--graph-field-depth` (default 3), the access-path bound of the syntactic `ddg` (`x.f.g.h` → `x.f.g.*`); `ddg.prov` is `["ssa"]` (parity). -### Deferred (not this train) -- L4 (SDG param edges) — reuses Group A ids/spans and the L3 vocabulary unchanged. Designed in - `docs/design/specs/v2-l3-l4-dataflow.md`. A Go-specific `select` cfg kind was not added - (`select` is a `switch` with `switch_case` edges). +### L4 SDG (`-a 4`; spec `docs/design/specs/v2-l3-l4-dataflow.md` §4.6–§4.16, §6) +Reuses Group A ids/spans and the L3 vocabulary unchanged; every L4 fact is an addition +(`-a 3` ⊆ `-a 4`, gated). Leaves: +- **Vertices** (body nodes, span-less, no `callee`): `@formal_in:`, `@formal_out[:]`, + `/actual_in:`, `/actual_out[:]` (Python's `IdentityMap` indexing: + `formal_in`/`actual_in` always indexed; an out is bare iff its group has one vertex). Actuals + carry `parent` = the call key (a non-call capture site: the literal's host). **`of`** (D6, + Python spelling, which the SDK's resolver reads): the param/receiver name; unnamed or `_` param + `_` (declared index, receiver excluded); unnamed or `_` receiver **`_recv`**; results + `` / **`:`** (never user result names; `:` is a Go leaf); + `:`; `:::`. The SDK reads `of`, never + `parameters[]` (`formal_in:0` is a method's receiver). +- **Edges:** application `param_in[]` / `param_out[]` `{src, dst, var}` (global ids, `var` = the + callee formal's `of`); callable `summary[] {src, dst}` (local keys, actual_in → actual_out); + port wiring in `ddg` with `prov: ["reaching-defs"]` (Python precedent; coarse host wiring: + host → every actual_in, every actual_out → host); may-alias `ddg` with `prov: ["points-to"]`. +- **Dispatch:** VTA over the project's functions (library targets in the set); `call_graph` + gains VTA-only edges `prov: ["vta"]`, weight 1 (an L2 edge is never changed). **D7:** an + interface call binds the interface method (pass-through summary) *and* every VTA target. +- **Stand-ins:** `external_symbols[].receiver {name, type}` and `.body` (formal vertices only, + ids `@formal_in:`); summary all-to-all unless `--library-models` overrides it + (keys `/`, values `[in, out]` `of` pairs). A model changes summaries, not + statement-level reachability; sanitizer semantics are the SDK's [user, 2026-10-27; + codellm-devkit/.github#103]. +- **`degraded[]`** envelope leaf `{code, detail?, packages[]?, callables[]?}`, absent when + nothing degraded; codes `no_ssa | no_vta | no_cha | partial_ssa | partial_types | unmapped_ssa | + l3_skipped | range_func` (open vocabulary). `--strict` exits non-zero on any. +- **Rulings:** global → global summary edges are omitted [user, 2026-10-26; #30]; memory is + measured and reported per app, not gated (D3) [user, 2026-10-26]; range-over-func is not + modelled, degraded `range_func` (D8); `-a 4 -t` is an error and `--emit neo4j -t` projects + at L3 with a warning (D9). +- A Go-specific `select` cfg kind was not added (`select` is a `switch` with `switch_case` + edges). ## Neo4j-projection decisions @@ -185,11 +214,17 @@ shared neutral vocabulary verbatim; never rename a sibling's shared name). Spec: It carries `call_node` (JSON `parent` verbatim — the **local** key, see § Flattened properties) and `var` (JSON `of`; L4 — absent at L3). `GO_HAS_BODY_NODE` may originate at `:GoExternal` (L4 stand-ins). -### Reserved L4 overlay (declared now, emits nothing until L4 lands) -- `schema.go` still declares `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY` (`Reserved`) with - parity-correct names; the projector emits **zero rows** for them. When L4 lands they fill - with **no schema-contract change**. (`--emit neo4j` is "full-depth = max implemented" — - L3 today; the graph grows additively.) +### L4 overlay (live; `--emit neo4j` runs at L4, D4) +- `GO_PARAM_IN {var}`, `GO_PARAM_OUT {var}` and `GO_SUMMARY` are live (no longer `Reserved`; + same names as declared, no contract change). One row per JSON `param_in[]` / `param_out[]` / + `summary[]` entry; endpoints are `:GoBodyNode` global ids. Port and points-to ddg ride + `GO_DDG {var, prov}` (`_k = var|prov`). +- Vertices are `:GoBodyNode`s with `var` (= JSON `of`) and `call_node` (actuals); stand-in + vertices hang off their `:GoExternal` via `GO_HAS_BODY_NODE`. `:GoApplication.degraded_json` + carries the envelope's `degraded[]`. +- **Lazy push (D5):** every L4 row is upserted on every push (L4 facts are non-local); stale L4 + rows need `--eager`. A stored `max_level` below the push's (or unreadable) forces a full + upsert; a lower-level push leaves old L4 rows. `--emit neo4j -t` projects at L3 (D9). ### SDK facade (`GoAnalysisBackend`, on the `python-sdk` fork — frontend rung) - **Type-centric surface** (Go is the SDK's first class-less facade; no C precedent exists @@ -202,10 +237,13 @@ shared neutral vocabulary verbatim; never rename a sibling's shared name). Spec: getters + `get_source`. Slicing / CFG / CDG / DDG / reachability are **omitted** (they need L3/L4 the analyzer does not produce) per the ABC contract. (2026-10-24: L3 is now emitted at `-a 3` — per-callable `cfg`/`cdg`/`ddg` and `GO_CFG_NEXT`/`GO_CDG`/`GO_DDG` — - so L3 accessors are supportable now; interprocedural slicing and taint still need L4: - `param_in`/`param_out`/`summary` and the points-to `ddg`.) + so L3 accessors are supportable now. 2026-10-27: L4 is emitted at `-a 4` and by + `--emit neo4j` — `param_in`/`param_out`/`summary` and the points-to `ddg` — so + interprocedural slicing and taint are supportable too.) ### Deferred (not this train) -- SDK consumer slicing/taint surface and L3/L4 facade methods — arrive with the L3/L4 - analyzer train, reusing this graph vocabulary unchanged. +- SDK consumer slicing/taint surface and L3/L4 facade methods — the analyzer now emits L4; + the SDK side reuses this graph vocabulary unchanged. - Struct-tag modeling. (`defer_resume` landed with L3; see § L3 body and edges.) +- `GO_EMBEDS` / `GO_SATISFIES` resolution: `base_types[]` holds raw type strings and + `interfaces[]` is never computed (#31, pre-existing). diff --git a/README.md b/README.md index 7050aa0..11f93f5 100644 --- a/README.md +++ b/README.md @@ -59,7 +59,10 @@ The binary is fully self-contained: a single static executable with no runtime d - **Call graph** — a resolver-based call graph via `go/types`: each call site is resolved to its full import-path signature. Level 2 emits edges to project callees and to library callees (homed in `application.external_symbols` under `can:///@external//`). - Level 3 (`-a 3`) adds per-callable `cfg`, `cdg` and `ddg`. + Level 3 (`-a 3`) adds per-callable `cfg`, `cdg` and `ddg`. Level 4 (`-a 4`) adds the + interprocedural system dependence graph (SDG): parameter vertices, `param_in` / `param_out`, + call-site `summary` edges, VTA dispatch targets and library stand-ins — what the SDK's + slicing and `taint()` walk. - **Generics-aware** — Go 1.18+ generics (`Set[T]`, union-constraint interfaces, multi-type-param functions) are modeled in the unified type/callable schema. - **Self-contained binary** — a single static executable (`go build`, `CGO_ENABLED=0`); no runtime @@ -162,22 +165,24 @@ Aliases: cango, codeanalyzer-go Flags: - -a, --analysis-level int Analysis level: 1=symbol table, 2=+call graph, 3=+intraprocedural cfg/cdg/ddg (default 1) + -a, --analysis-level int Analysis level: 1=symbol table, 2=+call graph, 3=+intraprocedural cfg/cdg/ddg, 4=+interprocedural SDG (default 1) --app-name string Application anchor name for can:// ids and Neo4j :Application (default: input dir name) -c, --cache-dir string Cache directory (default: ~/.cldk/go-cache) --eager Force clean rebuild (ignore cache) --emit string Output projection: json|neo4j|schema (default "json") -f, --format string Output format: json|msgpack (default "json") - --graph-field-depth int Field-path depth bound for L3 dataflow nodes (k limit; applies at -a 3 only) (default 3) + --graph-field-depth int Field-path depth bound for L3 dataflow nodes (k limit; applies at -a 3 and -a 4) (default 3) -h, --help help for cango -i, --input string Project root to analyze (required) -j, --jobs int Worker parallelism (default: CPU cores) + --library-models string JSON file of library stand-in flows overriding the all-to-all default (-a 4 and --emit neo4j only) --neo4j-database string Neo4j database (env NEO4J_DATABASE, optional) --neo4j-password string Neo4j password (env NEO4J_PASSWORD, default neo4j) --neo4j-uri string Live Bolt push target (env NEO4J_URI); omit to write graph.cypher --neo4j-user string Neo4j username (env NEO4J_USERNAME, default neo4j) -o, --output string Output directory for analysis.json (default: stdout) --skip-tests Skip *_test.go files (default true) + --strict Fail non-zero on any L4 degradation (default: degrade and record it) -t, --target-files strings Restrict analysis to specific files (incremental mode) -v, --verbose count Verbosity (repeat for more detail) --version Print version and exit @@ -275,7 +280,7 @@ are build artifacts, not project source), so only your own declarations are emit | 1 | `-a 1` (default) | Symbol table only — types, functions, call sites | Implemented | | 2 | `-a 2` | Level 1 + resolver-based call graph via `go/types` | Implemented | | 3 | `-a 3` | Level 2 + per-callable statement nodes and `cfg` / `cdg` / `ddg` | Implemented | -| 4 | `-a 4` | Level 3 + interprocedural parameter edges | Not implemented (rejected) | +| 4 | `-a 4` | Level 3 + interprocedural SDG: VTA dispatch, parameter vertices, `param_in` / `param_out`, `summary`, port and `points-to` `ddg`, library stand-ins | Implemented | **Level 1** loads each package with `packages.NeedSyntax | NeedTypes | NeedTypesInfo` and walks the AST file by file. Call sites are recorded with `callee_signature = null` at this stage. @@ -289,6 +294,60 @@ recorded in the envelope as `k_limit`. It is a flag error below `-a 3` and with `--emit neo4j`. Callables are analyzed in parallel (`-j`), and the output is byte-identical for every `-j`. L3 only adds to the L2 output: every L2 fact stays as it is. +**Level 4** is whole-program. It builds `go/ssa` for the project packages over the same load and +runs VTA (`golang.org/x/tools/go/callgraph/vta`) over the project's functions, so interface and +function-value calls get their concrete targets: `call_graph` gains those edges with +`prov: ["vta"]` (an L2 edge is never changed). Every callable gets `formal_in` / `formal_out` +vertices (parameters, receiver, closure captures, the package variables it reads or writes, +results), and every resolved call site gets `actual_in` / `actual_out` vertices, wired into the +body's `ddg` (`prov: ["reaching-defs"]`). `param_in` / `param_out` (application scope) join each +call site to every target, including the interface method itself for an interface call, and a +bottom-up pass over the call graph's SCCs puts `summary` edges (`actual_in` → `actual_out`) at +each call site. Library callees get a **stand-in** in `external_symbols` (`body` holds only its +formal vertices; summary all inputs → all outputs unless `--library-models` overrides it). A +type-based may-alias oracle adds `ddg` edges with `prov: ["points-to"]`. `-a 3` output is a +subset of `-a 4` output, and `-j 1` and `-j N` output are byte-identical. + +- **Degradation.** When L4 cannot build all it needs, it still emits and records why in the + envelope's `degraded[]`. Each entry has a `code`, an optional `detail`, and optional `packages[]` + and `callables[]`. The codes are: + - `no_ssa` / `partial_ssa`: SSA failed, in whole or for some packages; + - `no_vta`: VTA failed; + - `no_cha`: the CHA complement for library-originated values failed (its targets are dropped); + - `partial_types`: some packages have type errors; + - `unmapped_ssa`: SSA functions or calls that map to no callable; + - `l3_skipped`: callables without an L3 result, which get a pass-through summary; + - `range_func`: range-over-func, which is not modelled at L4. + + It also logs one warning per entry. `--strict` makes any degradation a non-zero exit. +- **`--library-models `** overrides stand-in summaries: + `{"strconv/Atoi": [], "strings/TrimSpace": [["s", ""]]}`. A key is the external id's + `/` tail, and a value is a list of `[in, out]` pairs of `of` names (`[]` = no + summary edges). A model changes summaries, **not statement-level reachability**: in + `n, err := strconv.Atoi(id)` the value assigned at the call site keeps its reaching-definition + edge from `id`, which is correct data flow. A model is therefore not a sanitizer. Cutting such + a flow is the SDK taint analysis's job + ([codellm-devkit/.github#103](https://github.com/codellm-devkit/.github/issues/103)). A variable + cut on `n` is also bypassed, by implicit flow through `err` and by the coarse host wiring. An + entry naming an `of` the external lacks falls back to all-to-all, with a warning. A key no call + site targets is ignored, with a warning. The flag is valid only at `-a 4` and with + `--emit neo4j`. +- **`-t` / `--target-files` is not supported at `-a 4`** (whole-program): it is an error. Use + `-a 3 -t`, or run without `-t`. +- **Known limits** (see `docs/design/notes/l3-dataflow-metrics.md` § L4): + - library code calling back into the project (`sort.Slice`, `http.HandleFunc`) is not an + edge; + - an interface call on a value that may have come out of library code (`ctx.Value(k).(Store)`, + `sync.Map.Load`, a library container), directly or through project code (results, + parameters, captures, fields, globals, container elements), binds every project type + implementing the interface (a CHA complement). Storage is tracked by container type (every + instance of a struct field), which is sound but coarser than VTA; + - range-over-func is not modelled (`range_func`); + - summaries are close to all-to-all, by design; + - `summary` edges whose two ends are both `:` vertices are omitted + (codeanalyzer-go#30). Reachability is unchanged, but a depth-bounded slice can stop earlier + than it would through that shortcut. + **Level 2** adds a resolver pass: for each call site, `go/types` resolves the callee to its full import-path signature (`pkgImportPath.TypeName.MethodName`). A project callee becomes an edge to its callable; a **library callee** (stdlib, a dependency, or a package vendored under `vendor/`) becomes an edge to an `@external` symbol, recorded once in `application.external_symbols`. A callee that is neither (an interface method of a project type, a conversion, a builtin, a call through a function value) gets no edge. Every resolved call node's `callee` is backfilled. ## Output schema @@ -367,9 +426,9 @@ node carries a `can:///…` `id`, a `kind`, and a `span`: ``` Key v2 schema properties: -- **Envelope** — `schema_version` (`"2.0.0"`), `language` (`"go"`), `max_level` (1, 2 or 3), - `k_limit` (L3 only: the `--graph-field-depth` value) and `analyzer{name,version}` wrap a - single `application` node. +- **Envelope** — `schema_version` (`"2.0.0"`), `language` (`"go"`), `max_level` (1–4), + `k_limit` (L3+: the `--graph-field-depth` value), `degraded[]` (L4, only when something + degraded) and `analyzer{name,version}` wrap a single `application` node. - **One containment tree** — `application → module → type → callable → body`, keyed as `symbol_table` (modules, by **project-relative file path**), `types`, `callables`, `body`. - **`id`** — every node carries a durable `can:///go///` id; @@ -404,6 +463,15 @@ Key v2 schema properties: A call nested in a statement carries `parent`, the statement's key. The callable gains `cfg[] {src, dst, kind}`, `cdg[] {src, dst}` and `ddg[] {src, dst, var, prov}`, whose endpoints are `body` keys. See `docs/design/specs/v2-schema-structure.md`. +- **L4 (`-a 4`)** — `body` gains span-less vertices `@formal_in:`, `@formal_out[:]`, + `/actual_in:`, `/actual_out[:]`, each with `of` (the bound name: + a parameter or receiver name, `_`, `_recv`, ``, `:`, + `:`, `:::`); actuals carry `parent` = the call key. + `ddg` gains `prov: ["reaching-defs"]` (port wiring) and `["points-to"]` edges; a callable gains + `summary[] {src, dst}` (local keys); the application gains `param_in[]` / `param_out[]` + `{src, dst, var}` (global body-node ids, `var` = the callee formal's `of`); an external gains + `receiver {name, type}` and a stand-in `body` of its formal vertices (ids + `@formal_in:`). ## Neo4j projection (`--emit neo4j`) @@ -439,10 +507,27 @@ What a re-emit replaces depends on the path: remain beside the new ones. Pass `--eager` when you push an L3 graph after editing source. The purge needs a full run: with `--target-files` it is skipped, because the push carries only the targeted packages. - -> **`--emit neo4j` is always full depth.** The graph is projected at the max implemented level (L3 -> today — symbol table, call graph, and every callable's body nodes with `GO_CFG_NEXT` / `GO_CDG` / -> `GO_DDG`) and canonical schema v2. Do **not** pass `-a`/`--analysis-level` +- **L4 rows on a lazy push (D5).** L4 facts are not local to a module: editing a callee changes + its callers' summaries and vertices. So a lazy push upserts **every** L4 row (vertices, + `GO_PARAM_IN` / `GO_PARAM_OUT`, `GO_SUMMARY`, port `GO_DDG`) on every push, not only the changed + modules'. It still adds and never removes: stale L4 rows stay until an `--eager` full run. +- **Level changes.** A lazy push whose level is above the stored `:GoApplication.max_level` (a graph + last pushed at L3, or with `-t`), or whose stored level is unreadable, upserts every module, so + no L4 row is dropped for a missing L3 endpoint. A push at a *lower* level onto a higher graph + diffs as usual and leaves the old L4 rows in place (additive). An **interrupted** push can leave + `:GoApplication` at the new `max_level` while some modules still hold older rows; the next lazy + push then trusts the diff. Recover with an `--eager` full run. +- **`--target-files` (D9).** L4 is whole-program, so `--emit neo4j -t` projects the targeted modules + at **L3** and writes no L4 rows; they keep the last full run's facts until the next full run. It + warns once on stderr: `--target-files: projecting the graph at level 3 (level 4 is + whole-program); L4 rows keep the last full run's facts until a full run` (plus `; + --library-models is ignored at level 3` when that flag is set). + +> **`--emit neo4j` is always full depth.** The graph is projected at the max implemented level (L4 +> — symbol table, call graph, every callable's body nodes with `GO_CFG_NEXT` / `GO_CDG` / +> `GO_DDG`, and the SDG: vertex `:GoBodyNode`s with `var`, `GO_PARAM_IN` / `GO_PARAM_OUT {var}`, +> `GO_SUMMARY`, stand-in vertices under their `:GoExternal`, and `degraded_json` on +> `:GoApplication`) and canonical schema v2. Do **not** pass `-a`/`--analysis-level` > with it — an explicit level is a flag error (`--analysis-level does not apply to --emit neo4j`), > because there is no shallower graph to ask for. @@ -576,8 +661,24 @@ check the following. as plain `line:col`. - Calls inside `go` statement arguments, such as `g()` in `go f(g())`, are now call sites. - Consumers that count call nodes will see more of them. +- **L4 (`-a 4`) is new.** It is additive: an `-a 3` consumer sees the same document plus new + keys (vertices, `of`, `summary`, `param_in` / `param_out`, `points-to` / `reaching-defs` `ddg`, + VTA `call_graph` edges, external `receiver` / `body`, `degraded`). Code that assumes every body + node has a `span`, or that every `ddg` edge has `prov: ["ssa"]`, must handle the L4 kinds. + `-a 4 -t` is an error (whole-program; D9). `--library-models` is new and valid only at L4. + **Measured cost** (2026-10-28, analyzer built with go1.27.3, eight real apps). On 7 of 8 + apps, the peak RSS of `-a 4` is 1.0–1.5× that of `-a 3`; Kubernetes takes 38 s at 7.9 GB. + nats-server is the outlier at 3.8×, because its globals fan out through one recursive 822-callable + SCC. The JSON output is 1.9–23× that of `-a 3`. Per-app figures are in + `docs/design/notes/l3-dataflow-metrics.md` § L4. - **Neo4j:** - - `--emit neo4j` projects L3, adding `GO_CFG_NEXT`, `GO_CDG` and `GO_DDG`. + - `--emit neo4j` projects **L4** (D4), adding `GO_CFG_NEXT`, `GO_CDG` and `GO_DDG`, and the SDG + rows `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY`, vertex `:GoBodyNode`s with `var`, and + `degraded_json`. `-a` with it is still an error. + - A lazy push upserts every L4 row on every push and removes none (D5); use `--eager` to drop + stale ones. `--emit neo4j -t` projects at L3 with a warning (D9). A push at a higher level + than the stored graph upserts everything; a lower-level push leaves the old L4 rows. See + [Neo4j projection](#neo4j-projection---emit-neo4j). - `:GoBodyNode.call_node` is the enclosing statement's local key (the JSON `parent`), not a global id. - `--eager` now works on Neo4j 5.x. @@ -593,10 +694,13 @@ check the following. | Type resolver | `golang.org/x/tools/go/packages` | Single API for both AST + full type resolution; handles modules natively | | Build/dep materialization | `go mod download` | Required before `packages.Load` so the module cache is warm; result cached by `go.sum` hash | | Packaging | Native binary (`go build`) | Zero-runtime-dep distribution; matches Rust/C++ analyzers | -| Analysis depth | Levels 1–3 | Symbol table, resolver call graph, intraprocedural cfg/cdg/ddg | +| Analysis depth | Levels 1–4 | Symbol table, resolver call graph, intraprocedural cfg/cdg/ddg, interprocedural SDG | | Call-graph dispatch | Declared-type resolution via `go/types.Selections` | CHA-equivalent; sufficient for cross-package reachability at Level 1 | | CFG (L3) | Hand-built from `go/ast` (`internal/dataflow`) | Adds what `go/cfg` does not model as edges: panic `exception` edges and the `defer` chain (`defer_resume`). A differential test checks reachability against `golang.org/x/tools/go/cfg` on every fixture body | | Def-use / DDG (L3) | Syntactic reaching definitions over k-limited access paths | `go/types` objects as bases (shadowing-safe); weak defs through reference arguments; `prov: ["ssa"]` for parity with the Java and Python L3 | +| SSA (L4) | `golang.org/x/tools/go/ssa` via `ssautil.Packages` over the same load, project packages only, built per package under `recover` | Bodies only for the project (dependencies from types); a panic degrades instead of aborting | +| Dispatch oracle (L4) | VTA (`go/callgraph/vta`) over the project's functions, library targets included | Concrete targets for interface and function-value calls; an interface call also binds the interface method (pass-through) | +| Alias oracle (L4) | Type-based may-alias (structural pointee compatibility, interface pairs, address-taken values, one-level field offset) | Additive `points-to` `ddg` edges; an interface so Andersen points-to can replace it without a schema change | ### Package structure @@ -610,6 +714,7 @@ codeanalyzer-go/ │ ├── syntactic_analysis/ # SymbolTableBuilder (packages.Load → AST walk) │ ├── semantic_analysis/ # CallGraphBuilder (go/types resolver) │ ├── dataflow/ # L3: body nodes, CFG, CDG, def-use, DDG; Run fans out per callable +│ ├── sdg/ # L4: SSA, VTA dispatch, footprints, vertices, port wiring, summaries, stand-ins, may-alias │ ├── analysis/ # Pluggable pass interface + registry (topo-ordered pipeline) │ ├── frameworks/ # BaseEntrypointFinder — extension seam for framework passes │ └── utils/ # DiscoverGoFiles, IsVendored, IsTestFile, logging diff --git a/cmd/codeanalyzer/main.go b/cmd/codeanalyzer/main.go index 296c5ba..9e0a941 100644 --- a/cmd/codeanalyzer/main.go +++ b/cmd/codeanalyzer/main.go @@ -40,6 +40,8 @@ func rootCmd() *cobra.Command { appName string level int fieldDepth int + strict bool + libraryModels string targetFiles []string skipTests bool eager bool @@ -79,9 +81,15 @@ via CLDK(language="go").analysis(project_path=...).`, if err := validateAnalysisLevel(level); err != nil { return err } + if level == int(options.LevelSDG) && len(targetFiles) > 0 { + return fmt.Errorf("--target-files is not supported at analysis level 4 (whole-program); run without -t, or use -a 3") + } if err := validateFieldDepth(cmd, level, fieldDepth); err != nil { return err } + if cmd.Flags().Changed("library-models") && level < int(options.LevelSDG) { + return fmt.Errorf("--library-models applies only at analysis level 4") + } case options.EmitNeo4j: if cmd.Flags().Changed("analysis-level") { return fmt.Errorf("--analysis-level does not apply to --emit neo4j; the graph is always projected at full depth") @@ -91,8 +99,17 @@ via CLDK(language="go").analysis(project_path=...).`, } // Force full depth: the graph carries every implemented // level's facts in both projections. (Output is always the - // canonical v2 tree.) - level = int(options.LevelDataflow) + // canonical v2 tree.) L4 is whole-program, so a targeted run + // projects at L3 and says so. + level = int(options.LevelSDG) + if len(targetFiles) > 0 { + level = int(options.LevelDataflow) + msg := "--target-files: projecting the graph at level 3 (level 4 is whole-program); L4 rows keep the last full run's facts until a full run" + if cmd.Flags().Changed("library-models") { + msg += "; --library-models is ignored at level 3" + } + utils.Warn("%s", msg) + } case options.EmitSchema: path, err := neo4j.EmitSchema(outputDir) if err != nil { @@ -136,6 +153,8 @@ via CLDK(language="go").analysis(project_path=...).`, Emit: options.EmitTarget(emit), AppName: appName, Level: options.AnalysisLevel(level), + Strict: strict, + LibraryModels: libraryModels, AnalyzerVersion: version, TargetFiles: targetFiles, SkipTests: skipTests, @@ -160,7 +179,7 @@ via CLDK(language="go").analysis(project_path=...).`, // JSON path emits) and project it. --neo4j-uri set → live Bolt push; // absent → a graph.cypher snapshot in the output dir. if emitTarget == options.EmitNeo4j { - payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, core.KLimit(opts), analyzer.L3()) + payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, core.KLimit(opts), analyzer.L3(), analyzer.L4()) boltCfg := neo4j.BoltConfig{ URI: opts.Neo4jURI, User: opts.Neo4jUser, @@ -177,18 +196,13 @@ via CLDK(language="go").analysis(project_path=...).`, return nil } - // When no --output dir is given, write JSON to cobra's output + // When no --output dir is given, stream JSON to cobra's output // writer so tests can capture it via cmd.SetOut. Both paths render - // through core so the v1/v2 schema selection stays in one place. + // through core so the schema selection stays in one place. if outputDir == "" { - data, err := core.RenderJSON(app, analyzer.L3(), opts) - if err != nil { - return err - } - _, err = cmd.OutOrStdout().Write(data) - return err + return core.StreamJSON(cmd.OutOrStdout(), app, analyzer.L3(), analyzer.L4(), opts) } - return core.WriteOutput(app, analyzer.L3(), opts) + return core.WriteOutput(app, analyzer.L3(), analyzer.L4(), opts) }, } @@ -200,9 +214,12 @@ via CLDK(language="go").analysis(project_path=...).`, f.StringVar(&appName, "app-name", "", "Application anchor name for can:// ids and Neo4j :Application (default: input dir name)") f.IntVarP(&level, "analysis-level", "a", 1, - "Analysis level: 1=symbol table, 2=+call graph, 3=+intraprocedural cfg/cdg/ddg") + "Analysis level: 1=symbol table, 2=+call graph, 3=+intraprocedural cfg/cdg/ddg, 4=+interprocedural SDG") f.IntVar(&fieldDepth, "graph-field-depth", 3, - "Field-path depth bound for L3 dataflow nodes (k limit; applies at -a 3 only)") + "Field-path depth bound for L3 dataflow nodes (k limit; applies at -a 3 and -a 4)") + f.BoolVar(&strict, "strict", false, "Fail non-zero on any L4 degradation (default: degrade and record it)") + f.StringVar(&libraryModels, "library-models", "", + "JSON file of library stand-in flows overriding the all-to-all default (-a 4 and --emit neo4j only)") f.StringSliceVarP(&targetFiles, "target-files", "t", nil, "Restrict analysis to specific files (incremental mode)") f.BoolVar(&skipTests, "skip-tests", true, "Skip *_test.go files") @@ -221,17 +238,12 @@ via CLDK(language="go").analysis(project_path=...).`, return cmd } -// validateAnalysisLevel accepts the implemented levels (1..3). Level 4 is -// reserved for a later release and gets its own message. +// validateAnalysisLevel accepts the implemented levels (1..4). func validateAnalysisLevel(level int) error { - switch { - case level >= 1 && level <= int(options.LevelDataflow): + if level >= 1 && level <= int(options.LevelSDG) { return nil - case level == 4: - return fmt.Errorf("analysis level 4 is not implemented yet") - default: - return fmt.Errorf("analysis level %d is not supported; use 1, 2 or 3", level) } + return fmt.Errorf("analysis level %d is not supported; use 1, 2, 3 or 4", level) } // validateFieldDepth checks --graph-field-depth. The flag only means something diff --git a/cmd/codeanalyzer/main_test.go b/cmd/codeanalyzer/main_test.go index cc35426..7da9999 100644 --- a/cmd/codeanalyzer/main_test.go +++ b/cmd/codeanalyzer/main_test.go @@ -353,22 +353,6 @@ func TestRootCmd_Level3Accepted(t *testing.T) { } } -func TestRootCmd_Level4RejectedInThisRelease(t *testing.T) { - td := cliTestdataDir() - - _, _, err := runCmd( - "--input", filepath.Join(td, "greeter"), - "--analysis-level", "4", - "--cache-dir", t.TempDir(), - ) - if err == nil { - t.Fatal("expected error for -a 4, got nil") - } - if err.Error() != "analysis level 4 is not implemented yet" { - t.Errorf("error = %q, want %q", err.Error(), "analysis level 4 is not implemented yet") - } -} - func TestRootCmd_UnsupportedLevelRejected(t *testing.T) { td := cliTestdataDir() @@ -503,3 +487,295 @@ func TestRootCmd_MalformedSourceDoesNotPanic(t *testing.T) { }) } } + +// ── L4 scaffolding ─────────────────────────────────────────────────────────── + +const ( + errL4WithTargetFiles = "--target-files is not supported at analysis level 4 (whole-program); run without -t, or use -a 3" + warnNeo4jTargetFiles = "--target-files: projecting the graph at level 3 (level 4 is whole-program); L4 rows keep the last full run's facts until a full run" +) + +func TestCLI_Level4Accepted(t *testing.T) { + td := cliTestdataDir() + out, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-level", "4", + "--graph-field-depth", "2", + "--cache-dir", t.TempDir(), + ) + if err != nil { + t.Fatalf("-a 4 should succeed; got %v", err) + } + var v struct { + MaxLevel int `json:"max_level"` + KLimit int `json:"k_limit"` + } + if jsonErr := json.Unmarshal([]byte(out), &v); jsonErr != nil { + t.Fatalf("stdout is not valid JSON: %v", jsonErr) + } + if v.MaxLevel != 4 { + t.Errorf("max_level = %d, want 4", v.MaxLevel) + } + if v.KLimit != 2 { + t.Errorf("k_limit = %d, want 2 (--graph-field-depth is valid at -a 4)", v.KLimit) + } +} + +func TestCLI_Level4WithTargetFilesRejected(t *testing.T) { + td := cliTestdataDir() + _, _, err := runCmd( + "--input", filepath.Join(td, "multipackage"), + "--analysis-level", "4", + "--target-files", filepath.Join(td, "multipackage", "server", "server.go"), + "--cache-dir", t.TempDir(), + ) + if err == nil { + t.Fatal("expected an error for -a 4 -t, got nil") + } + if err.Error() != errL4WithTargetFiles { + t.Errorf("error = %q, want %q", err.Error(), errL4WithTargetFiles) + } +} + +func TestCLI_Level3WithTargetFilesStillWorks(t *testing.T) { + td := cliTestdataDir() + out, _, err := runCmd( + "--input", filepath.Join(td, "multipackage"), + "--analysis-level", "3", + "--target-files", filepath.Join(td, "multipackage", "server", "server.go"), + "--cache-dir", t.TempDir(), + ) + if err != nil { + t.Fatalf("-a 3 -t should succeed; got %v", err) + } + if !strings.Contains(out, `"server/server.go"`) { + t.Error("-a 3 -t: server/server.go should appear in output") + } +} + +func TestCLI_Neo4jWithTargetFilesFallsBackToL3(t *testing.T) { + td := cliTestdataDir() + out := t.TempDir() + var runErr error + stderr := captureStderr(t, func() { + _, _, runErr = runCmd( + "--input", filepath.Join(td, "multipackage"), + "--emit", "neo4j", + "--target-files", filepath.Join(td, "multipackage", "server", "server.go"), + "--output", out, + "--cache-dir", t.TempDir(), + ) + }) + if runErr != nil { + t.Fatalf("--emit neo4j -t should succeed; got %v", runErr) + } + if n := strings.Count(stderr, warnNeo4jTargetFiles); n != 1 { + t.Errorf("D9 warning appeared %d times, want exactly 1; stderr:\n%s", n, stderr) + } + data, err := os.ReadFile(filepath.Join(out, "graph.cypher")) + if err != nil { + t.Fatalf("graph.cypher not written: %v", err) + } + cy := string(data) + // Effective level 3: the :GoApplication node carries max_level 3 and the + // L3 overlay is present. + if !strings.Contains(cy, "max_level: 3") && !strings.Contains(cy, `"max_level":3`) && !strings.Contains(cy, "max_level = 3") { + t.Errorf("graph.cypher does not record max_level 3") + } + for _, rel := range []string{"GO_CFG_NEXT", "GO_DDG"} { + if !strings.Contains(cy, rel) { + t.Errorf("graph.cypher should carry the L3 %s relationship", rel) + } + } + // No L4 rows (R-P8): no MERGE of a GO_PARAM_IN / GO_PARAM_OUT / + // GO_SUMMARY relationship, no port GO_DDG row, no vertex body node. The + // same fixture without -t does carry each of them, so the guard is real. + for rel, n := range l4CypherRows(cy) { + if n != 0 { + t.Errorf("graph.cypher under -t carries %d %s rows", n, rel) + } + } + full := t.TempDir() + if _, _, err := runCmd( + "--input", filepath.Join(td, "multipackage"), + "--emit", "neo4j", + "--output", full, + "--cache-dir", t.TempDir(), + ); err != nil { + t.Fatalf("--emit neo4j should succeed; got %v", err) + } + data, err = os.ReadFile(filepath.Join(full, "graph.cypher")) + if err != nil { + t.Fatal(err) + } + for rel, n := range l4CypherRows(string(data)) { + if n == 0 { + t.Errorf("graph.cypher without -t carries no %s rows", rel) + } + } +} + +// l4CypherRows counts the L4 rows of a graph.cypher script: the rows of +// each UNWIND batch that MERGEs a GO_PARAM_IN / GO_PARAM_OUT / GO_SUMMARY +// relationship, the port GO_DDG rows (prov reaching-defs), and the vertex +// :GoBodyNode rows (kind formal_in, formal_out, actual_in, actual_out). +func l4CypherRows(cy string) map[string]int { + n := map[string]int{"GO_PARAM_IN": 0, "GO_PARAM_OUT": 0, "GO_SUMMARY": 0, "GO_DDG reaching-defs": 0, "vertex": 0} + for _, stmt := range strings.Split(cy, ";\n") { + rows := strings.Count(stmt, "\n {") + for _, rel := range []string{"GO_PARAM_IN", "GO_PARAM_OUT", "GO_SUMMARY"} { + if strings.Contains(stmt, "MERGE (a)-[r:"+rel+"]->(b)") { + n[rel] += rows + } + } + if strings.Contains(stmt, "-[r:GO_DDG {_k: row.k}]->") { + n["GO_DDG reaching-defs"] += strings.Count(stmt, "|reaching-defs'") + } + if strings.Contains(stmt, "MERGE (n:GoBodyNode") { + for _, k := range []string{"formal_in", "formal_out", "actual_in", "actual_out"} { + n["vertex"] += strings.Count(stmt, "kind: '"+k+"'") + } + } + } + return n +} + +func TestCLI_Neo4jWithoutTargetFilesWarnsNothing(t *testing.T) { + td := cliTestdataDir() + var runErr error + stderr := captureStderr(t, func() { + _, _, runErr = runCmd( + "--input", filepath.Join(td, "greeter"), + "--emit", "neo4j", + "--output", t.TempDir(), + "--cache-dir", t.TempDir(), + ) + }) + if runErr != nil { + t.Fatalf("--emit neo4j should succeed; got %v", runErr) + } + if strings.Contains(stderr, "projecting the graph at level 3") { + t.Errorf("the D9 warning must appear only under -t; stderr:\n%s", stderr) + } +} + +func TestCLI_StrictFlag(t *testing.T) { + td := cliTestdataDir() + if rootCmd().Flags().Lookup("strict") == nil { + t.Fatal("--strict flag is not declared") + } + out, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-level", "4", + "--strict", + "--cache-dir", t.TempDir(), + ) + if err != nil { + t.Fatalf("-a 4 --strict should succeed on a clean project; got %v", err) + } + if !json.Valid([]byte(out)) { + t.Error("stdout is not valid JSON") + } +} + +// ── --library-models ───────────────────────────────────────────────────────── + +const errLibraryModelsLevel = "--library-models applies only at analysis level 4" + +func writeModelsFile(t *testing.T, body string) string { + t.Helper() + path := filepath.Join(t.TempDir(), "models.json") + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + return path +} + +func TestCLI_LibraryModelsBelowLevel4Rejected(t *testing.T) { + td := cliTestdataDir() + models := writeModelsFile(t, `{"strconv/Atoi": []}`) + for _, level := range []string{"1", "2", "3"} { + _, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-level", level, + "--library-models", models, + "--cache-dir", t.TempDir(), + ) + if err == nil || err.Error() != errLibraryModelsLevel { + t.Errorf("-a %s --library-models: error = %v, want %q", level, err, errLibraryModelsLevel) + } + } +} + +func TestCLI_LibraryModelsAtLevel4AndNeo4j(t *testing.T) { + td := cliTestdataDir() + models := writeModelsFile(t, `{"fmt/Sprintf": [["format", ""]]}`) + out, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-level", "4", + "--library-models", models, + "--cache-dir", t.TempDir(), + ) + if err != nil { + t.Fatalf("-a 4 --library-models should succeed; got %v", err) + } + if !json.Valid([]byte(out)) { + t.Error("stdout is not valid JSON") + } + if _, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--emit", "neo4j", + "--library-models", models, + "--output", t.TempDir(), + "--cache-dir", t.TempDir(), + ); err != nil { + t.Fatalf("--emit neo4j --library-models should succeed; got %v", err) + } +} + +// TestCLI_Neo4jTargetFilesWithLibraryModelsWarnsOnce: --emit neo4j -t runs at +// L3, so --library-models is unused there; the D9 warning says so on its one +// stderr line. +func TestCLI_Neo4jTargetFilesWithLibraryModelsWarnsOnce(t *testing.T) { + td := cliTestdataDir() + models := writeModelsFile(t, `{"fmt/Sprintf": []}`) + var runErr error + stderr := captureStderr(t, func() { + _, _, runErr = runCmd( + "--input", filepath.Join(td, "multipackage"), + "--emit", "neo4j", + "--target-files", filepath.Join(td, "multipackage", "server", "server.go"), + "--library-models", models, + "--output", t.TempDir(), + "--cache-dir", t.TempDir(), + ) + }) + if runErr != nil { + t.Fatalf("--emit neo4j -t --library-models should succeed; got %v", runErr) + } + want := "[codeanalyzer-go WARN] " + warnNeo4jTargetFiles + "; --library-models is ignored at level 3\n" + if n := strings.Count(stderr, want); n != 1 { + t.Errorf("want exactly one line %q; stderr:\n%s", want, stderr) + } + if n := strings.Count(stderr, "projecting the graph at level 3"); n != 1 { + t.Errorf("D9 warning appeared %d times, want 1; stderr:\n%s", n, stderr) + } +} + +func TestCLI_LibraryModelsBadFileErrors(t *testing.T) { + td := cliTestdataDir() + for name, path := range map[string]string{ + "missing": filepath.Join(t.TempDir(), "absent.json"), + "malformed": writeModelsFile(t, `{"fmt/Sprintf": [`), + } { + _, _, err := runCmd( + "--input", filepath.Join(td, "greeter"), + "--analysis-level", "4", + "--library-models", path, + "--cache-dir", t.TempDir(), + ) + if err == nil || !strings.Contains(err.Error(), path) { + t.Errorf("%s: error = %v, want one naming %s", name, err, path) + } + } +} diff --git a/docs/design/Neo4j_projection/issues/child-m1-schema-contract.md b/docs/design/Neo4j_projection/issues/child-m1-schema-contract.md index 3837bd5..c095bc4 100644 --- a/docs/design/Neo4j_projection/issues/child-m1-schema-contract.md +++ b/docs/design/Neo4j_projection/issues/child-m1-schema-contract.md @@ -28,7 +28,9 @@ JSON hot path) that is the declarative contract: `GO_HAS_METHOD`/`GO_HAS_FIELD`/`GO_HAS_BODY_NODE`), `GO_CALLS`/`GO_RESOLVES_TO`, `GO_EMBEDS`/`GO_SATISFIES`, `GO_IMPORTS`, the neutral repo edges, and `GO_PROVIDES`/ `GO_UNRESOLVED_IMPORT`. -- [ ] The **reserved L3/L4 overlay** vocabulary declared now (zero rows until the levels land): +- [ ] The **reserved L3/L4 overlay** vocabulary declared now (zero rows until the levels land; + *status 2026-10-27: both overlays are now live, `--emit neo4j` projects L4 and nothing is + `Reserved`*): `GO_HAS_BODY_NODE`/`GO_CFG_NEXT`(`_k=kind`)/`GO_CDG`/`GO_DDG`(`_k=var,prov`)/`GO_PARAM_IN`/ `GO_PARAM_OUT`/`GO_SUMMARY` + body-node statement kinds. - [ ] `_k` relationship discriminants on `GO_CFG_NEXT`, `GO_DDG`, `DECLARES_DEPENDENCY`. diff --git a/docs/design/Neo4j_projection/issues/parent-neo4j-projection.md b/docs/design/Neo4j_projection/issues/parent-neo4j-projection.md index d301b06..5613c48 100644 --- a/docs/design/Neo4j_projection/issues/parent-neo4j-projection.md +++ b/docs/design/Neo4j_projection/issues/parent-neo4j-projection.md @@ -43,7 +43,8 @@ Children (filed just-in-time on their repos, attached as sub-issues of this one) `N="Go"`/`P="GO_"`; one `:GoType` + `kind` property; `GO_EMBEDS` vs `GO_SATISFIES`; flattened `source_file`/`error_channel`/`is_goroutine`/`is_deferred`; narrower label set (no decorator/attribute/variable); neutral repo layer reused verbatim; full L3/L4 overlay -vocabulary declared now, zero rows until the levels land; type-centric SDK facade. +vocabulary declared now, zero rows until the levels land (*status 2026-10-27: the L3 and L4 +overlays are now live; `--emit neo4j` projects L4*); type-centric SDK facade. **Describe alternatives you've considered** diff --git a/docs/design/Neo4j_projection/roadmap.md b/docs/design/Neo4j_projection/roadmap.md index 3a209a0..740e1b9 100644 --- a/docs/design/Neo4j_projection/roadmap.md +++ b/docs/design/Neo4j_projection/roadmap.md @@ -192,9 +192,10 @@ and read-only Neo4j backends. - **`taint_flows` / slicing / reachability / source-sink** — the analyzer is a pure graph **provider**; labeled traversal is the SDK's job as Cypher queries, a later train. No analyzer surface for it. -- **L3/L4 overlay population** — the vocabulary is declared (decision L1) but emits zero rows - until the deferred L3/L4 analysis train lands. When it does, `project.go` fills the reserved - sections with **no `schema.go` change**. +- ~~**L3/L4 overlay population**~~ — done: the L3 train and then the L4 train (`feat/l4-sdg`) + filled the declared overlay in `project.go` with no contract change; `--emit neo4j` projects + at L4 and nothing in `schema.go` is `Reserved`. (The milestone text above — "zero rows at + L2" — describes the M1–M3 scope as built then.) - **Go-specific `cfg` edge kinds** (`defer_resume`, `select`) — recorded when L3 is designed. - **Struct-tag modeling** — Go has struct tags; v2 doesn't model them yet. Not forced in here. - **TS-SDK `GoAnalysis` encoding** — the facade vocabulary is designed once (M7 Python diff --git a/docs/design/Neo4j_projection/specs/neo4j-projection.md b/docs/design/Neo4j_projection/specs/neo4j-projection.md index b8e840a..bd1f350 100644 --- a/docs/design/Neo4j_projection/specs/neo4j-projection.md +++ b/docs/design/Neo4j_projection/specs/neo4j-projection.md @@ -80,7 +80,7 @@ or a verbatim reuse of shared vocabulary. "Ref" names the sibling precedent. | **CG1** | `call_graph` → **`GO_CALLS`** (weight, prov); `error_channel[]` → **string-array prop** on `:GoCallable` | `GO_CALLS` parity with `PY_CALLS` (endpoints = callable id or `:GoExternal` ghost; no dangling). Go errors are returned values, not thrown types — a value-shaped fact rides the node, not a `GO_RAISES` edge. | | **I1** | `GO_IMPORTS` (aggregated per target) → `:GoModule` (in-project) or `:GoExternal` ghost (stdlib/external, keyed by import path) | Parity with aggregated `PY_IMPORTS` + edge-only-when-resolved. Go imports are overwhelmingly external → ghost targets, never dangling, never minted as `:Package` (packages come from dependencies, not imports). | | **A1** | Declare neutral `:Artifact`/`:Package`/`:ConfigKey` + `HAS_ARTIFACT`/`DECLARES_DEPENDENCY`(`_k=kind`)/`LOCKS`/`DEFINES_CONFIG` **verbatim**; project only facts the analyzer produces | These are the cross-language **merge targets** — reused unchanged so a sibling over the same repo converges. go.mod deps → `:Package` `pkg:golang/…`; absent facts → **zero rows** (declared contract, empty projection). `GO_PROVIDES`/`GO_UNRESOLVED_IMPORT` stay `GO_`-prefixed (this analyzer's claims). | -| **L1** | Declare the **full L3/L4 overlay vocabulary now**; projector emits **zero rows** until the levels land | `GO_HAS_BODY_NODE`, `GO_CFG_NEXT`(`_k=kind`), `GO_CDG`, `GO_DDG`(`_k=var,prov`), `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY` + body-node statement kinds. `--emit neo4j` = "full-depth = max implemented" (**L3 today**: `GO_CFG_NEXT`, `GO_CDG`, `GO_DDG` and the L3 body nodes are live; only `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY` remain reserved for L4). The contract is declared once; L3 filled it with **no schema change**, and L4 will too. | +| **L1** | Declare the **full L3/L4 overlay vocabulary up front**; the projector emitted **zero rows** for a level until it landed | `GO_HAS_BODY_NODE`, `GO_CFG_NEXT`(`_k=kind`), `GO_CDG`, `GO_DDG`(`_k=var,prov`), `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY` + body-node statement kinds. `--emit neo4j` = "full-depth = max implemented" (**L4**, D4): the L3 overlay (`GO_CFG_NEXT`, `GO_CDG`, `GO_DDG`, L3 body nodes) and the L4 overlay (`GO_PARAM_IN {var}`, `GO_PARAM_OUT {var}`, `GO_SUMMARY`, vertex body nodes with `var`, stand-in vertices, `degraded_json`) are all live; nothing is `Reserved` (`schema.go`). The contract was declared once; L3 and L4 each filled it with **no schema change**. | | **K1** | `_k` discriminants: `GO_CFG_NEXT` per `kind`; `GO_DDG` per `(var,prov)`; `DECLARES_DEPENDENCY` per `kind` | Same mechanism as the siblings — parallel edges survive MERGE instead of collapsing. | | **V1** | `SCHEMA_VERSION = "2.0.0"`, stamped on `:GoApplication`; additive-MINOR suspended for the 2.0.0 line | Parity with the sibling versioning ruling; consumers gate on the analyzer-version floor. | @@ -109,7 +109,7 @@ Every `can://`-keyed node also carries `:GoCanNode`. - **Imports:** `GO_IMPORTS` (aggregated) - **Repository (neutral):** `HAS_ARTIFACT`, `DECLARES_DEPENDENCY` (`_k`), `LOCKS`, `DEFINES_CONFIG`; (`GO_`) `GO_PROVIDES`, `GO_UNRESOLVED_IMPORT` - **L3 overlay (live):** `GO_CFG_NEXT` (`kind`, `_k=kind`), `GO_CDG`, `GO_DDG` (`var`, `prov`, `_k=var|prov`) -- **Reserved L4 overlay (zero rows until L4 lands):** `GO_PARAM_IN`, `GO_PARAM_OUT`, `GO_SUMMARY` +- **L4 overlay (live):** `GO_PARAM_IN` (`var`), `GO_PARAM_OUT` (`var`), `GO_SUMMARY`; port and points-to `ddg` ride `GO_DDG`. One row per JSON `param_in[]` / `param_out[]` / `summary[]` entry; a lazy push upserts every L4 row (D5). ### CLI surface (parity with TS/Python) @@ -185,8 +185,8 @@ lockstep."** Concretely: - **Analyzer** ships the `internal/neo4j/` projection as part of the v2 line. `--emit neo4j` is independently landable behind the existing flag stubs; it produces a graph at - full-depth = L3 today (L3 landed additively, with no contract change) and grows additively - again when L4 arrives (no contract change — L1 reserve). + full-depth = L4, the max implemented level (L3 and then L4 landed additively, each with no + contract change — the L1 reserve). - **SDK** (`GoAnalysisBackend`) pins the analyzer version **only after** the analyzer v2 release is cut — same ordering constraint as the JSON wiring (the SDK's Go models can't parse v2 until then, and there is no Go wiring yet, so this is net-new, not a migration). diff --git a/docs/design/notes/l3-dataflow-metrics.md b/docs/design/notes/l3-dataflow-metrics.md index eb619f2..a078db5 100644 --- a/docs/design/notes/l3-dataflow-metrics.md +++ b/docs/design/notes/l3-dataflow-metrics.md @@ -31,8 +31,8 @@ degraded-input details. This note reuses its timings and extends its counts. rerun on the same trees where the earlier outputs were not kept (dae linux, both Kubernetes modules). -One analysis engine, so there is no engine comparison here. Interprocedural (L4) results will be -added under [§ L4](#l4-to-be-added). +One analysis engine, so there is no engine comparison here. Interprocedural (L4) results, measured +2026-10-27, are under [§ L4](#l4). ## Results at a glance @@ -102,8 +102,14 @@ scripts/eval_l3.sh cobra listmonk memos nats-server openbao dae kubernetes k8s-a # L3 metrics and consistency checks, one JSON object per app: python3 -I scripts/l3_metrics.py cobra=/tmp/eval-out/cobra-a3j1/analysis.json \ k8s-api=/tmp/eval-out/k8s-api-a3j1/analysis.json ... > metrics.jsonl + +# § L4: add the -a 4 triple (-j 1, -j N, rerun; byte-compared) and its counts: +CANGO_LEVELS="2 3 4" scripts/eval_l3.sh cobra listmonk memos nats-server openbao dae kubernetes k8s-api +python3 -I scripts/l3_metrics.py cobra=/tmp/eval-out/cobra-a4j1/analysis.json ... > metrics-a4.jsonl ``` +(`--all` also names cockroach, which is not prepared on this host, so the apps are listed.) + - **What `--prepare` does.** For each app in the script's manifest: a shallow, blob-filtered fetch of the pinned commit into `/tmp/eval/` (an existing clone is reused only at that commit), a tool check, the app's prep (dae: the `dae_bpf_headers` submodule and `make ebpf`, which needs @@ -112,6 +118,9 @@ python3 -I scripts/l3_metrics.py cobra=/tmp/eval-out/cobra-a3j1/analysis.json \ - **What `eval_l3.sh` does.** It runs a warm-up, `-a 2 -j 1`, `-a 3 -j 1` and two `-a 3 -j N`, all `--eager`, with each app's manifest env (dae: `GOOS=linux`). It records wall time and peak RSS from `/usr/bin/time -l`, then compares the three `-a 3` outputs byte for byte. +- **Metamorphic relations (`scripts/metamorphic.sh`).** Run the default, `scripts/metamorphic.sh` + (memos, about 2 min), for any L3/L4 change; run `scripts/metamorphic.sh --all` (memos and + nats-server, about 25 min) before a release and for the Task 18 eval. - **What `scripts/l3_metrics.py` does.** It produces every count, ratio, distribution and check in this note. It has no dependencies and loads one file at a time; the 318 MB Kubernetes root file takes about 10 s. The per-family protobuf breakdown and the before/after edge diffs were @@ -786,26 +795,432 @@ in any total above. node against every def in the body. A reviewer probe of one 6,000-case switch-heavy function took 15 s. Nothing in this corpus comes close. -## L4 (to be added) - -L4 (interprocedural SDG) results will be added here, over the same corpus and pinned commits, with -the same reproduction harness (`scripts/eval_l3.sh --prepare`, then run). Planned measurements: - -- **The points-to delta.** `ddg` edges with `prov:["points-to"]` added on top of the L3 `ssa` - edges, which must be strictly additive (§4.10): count, share, per-app, and which access-path - shapes the type-based oracle links. -- **Parameter wiring.** `@formal_in` / `@formal_out` vertices and the per-call-site actual - vertices (§4.6), and `GO_PARAM_IN` / `GO_PARAM_OUT` edges (§4.7). These are counted per callable - and per resolved call site, against L2's `call_graph` edges. -- **Summary edges** (§4.9): count, SCC sizes, fixpoint iterations on recursive SCCs, and the - cost of the bottom-up pass. -- **Library stand-ins** (§6): the number of external stand-ins with modelled bodies, and how many - `formal_in`/`formal_out` vertices they contribute. -- **VTA dispatch** (§4.8): call edges added over the `go/types` call graph. -- **Cost and size.** Wall time, RSS and `analysis.json` growth of `-a 4` over `-a 3`, plus the - `-j` determinism check, with Kubernetes as the scale benchmark. -- **Consistency at scale.** Every param edge joins a real call site to a real formal, and every - summary edge spans a call site's actual-in to actual-out. -- **Effect on the L3 caveats above.** Whether points-to edges make the weak-def tail denser, and - whether the protobuf-marshaller triangle compounds across `MarshalToSizedBuffer(dAtA[:i])` - calls once they are wired interprocedurally. +## L4 + +L4 is the interprocedural system dependence graph (SDG). At `-a 4`, on top of L3, every callable +gets `formal_in` / `formal_out` vertices, and every resolved call site gets `actual_in` / +`actual_out` vertices. These are wired into the callable's `ddg` (`prov:["reaching-defs"]`). The +application carries `param_in` / `param_out`, every call site carries `summary` edges, VTA adds +dispatch targets to `call_graph` (`prov:["vta"]`), library callees get stand-in formals in +`external_symbols[].body`, and a type-based may-alias oracle adds `ddg` edges with +`prov:["points-to"]`. The semantics are in [`v2-l3-l4-dataflow.md`](../specs/v2-l3-l4-dataflow.md) +§4.6–§4.16 and §6. + +- **When and where:** measured 2026-10-28 on the same host, an Apple M3 Max (16 cores, 64 GB, + macOS 15). The analyzer was built by `scripts/eval_l3.sh` with **go1.27.3** (`runtime.Version()`), + and every run used `GOTOOLCHAIN=go1.27.3`. Every number in this section is go1.27.3 unless it + says otherwise: figures quoted from earlier task reports carry their toolchain in parentheses. + Absolute RSS differs by about 1 GB between go1.26.4 and go1.27.3 on nats-server (Task 15b), so + do not compare figures across toolchains. +- **Binary:** `feat/l4-sdg` at `98ebfef`, a clean tree: the final head of the branch, every + review fix included. The first L4 eval (2026-10-27, `c230592` on the pre-rebase branch; its + analyzer and scripts are `a2e7ea0` here) predates the final-review fixes (VTA through + wrappers, SSA for every package variant, the library-origin CHA complement); § What changed + since the earlier measurements compares the two. +- **Protocol:** `CANGO_LEVELS="2 3 4" scripts/eval_l3.sh` over the same eight prepared trees and + pinned commits as the L3 sections. Each app runs the L3 sequence (§ How to reproduce) and then + the **`-a 4` triple**: `-a 4 -j 1 -v`, `-a 4 -j 16`, and a second `-a 4 -j 16`, all + `--eager`. The three `-a 4` files are compared byte for byte. Wall time, user CPU and peak RSS + come from `/usr/bin/time -l`. That peak is the **whole process**: load, L1–L3, SSA, VTA, the + L4 overlay and the streamed emission. L4 counts come from `scripts/l3_metrics.py` (its new `l4` + object) and the `-v` log. +- **Gate (spec §8):** every app completes `-a 4` with no crash, and its three outputs are + byte-identical. Memory is **measured and reported, not gated** (D3, user ruling 2026-10-26): + L4 is whole-program, and the former "≤ 1.5× `-a 3`" budget is dropped. The table reports the + absolute peak and its ratio to `-a 3`. + +### Results at a glance (L4) + +| | 6 apps | Kubernetes (`kubernetes` + `k8s-api`) | +|---|---|---| +| `-a 4` complete, no crash | 6/6 | 2/2 | +| `-j 1` = `-j 16` = `-j 16`, byte-identical | 6/6 | 2/2 | +| wall `-a 3` → `-a 4`, `-j 1` | 12.7 → 48.0 s | 12.8 → 38.4 s / 5.2 → 10.6 s | +| peak RSS `-a 4 -j 1` (× `-a 3`) | 287 MB (1.15×) to 5.9 GB (1.17×); nats-server 4.1 GB (3.77×) | **7.9 GB (0.99–1.13×)** / 2.5 GB (1.23×) | +| `analysis.json` `-a 3` → `-a 4` | 263 MB → 2.83 GB (10.8×; nats-server alone 1.78 GB) | 343 MB → 1.43 GB (4.2×) / 193 → 360 MB (1.9×) | +| vertices | 2.49M | 1.22M / 242k | +| `param_in` / `param_out` | 1.24M / 0.77M | 495k / 432k / 91k / 88k | +| `summary` | 3.79M | 1.36M / 206k | +| port `ddg` (`reaching-defs`) / `points-to` `ddg` | 4.91M / 654k | 1.94M / 374k / 439k / 39k | +| stand-ins (externals with a body) | 4,591 | 1,679 / 129 | +| bound call sites with a VTA-only target | 2.8–13.0% | 13.5% (13,770) / 0.4% | +| consistency checks (L3 and L4) | all PASS | all PASS | +| degraded codes | `range_func` (5 apps), `unmapped_ssa` (4) | `range_func`, `unmapped_ssa` / none | + +**Key takeaways.** + +1. **The scale gate passes.** The Kubernetes root completes `-a 4` in 38.4 s at `-j 1` (36.2 s at + `-j 16`) with a whole-process peak of 7.9 GB, and its three outputs are byte-identical. Its + `-a 3` peak varies from run to run (7.0–8.0 GB over three runs, against a steady 7.9 GB at + `-a 4`), so the ratio is 0.99–1.13×; the `go/packages` load still dominates RSS. L4 adds + 26 s of wall time: SSA takes 0.9 s, the CHA complement's library-origin fixpoint about 6 s, + VTA 2.4 s and the overlay 11.1 s, and the rest is emitting the 1.43 GB file. +2. **Memory is not the problem L4 has; output size is.** RSS grows by 15–50% on five of the + six apps, by up to 13% on the Kubernetes root and by 23% on `k8s-api`. nats-server is the + outlier at 3.8× (4.1 GB). It has one giant recursive SCC whose global footprint (A4) gives + 91% of its 1.59M vertices a `:` `of`. The JSON grows + 3–10× on the hand-written apps and 22.7× on nats-server (78 MB → 1.78 GB). Most of those + bytes are `:` vertices, their port wiring, and param edges carrying full global ids. +3. **Summaries are near all-to-all, as A15 predicts.** 74–94% of call sites with summary edges + have every `actual_in` → `actual_out` pair. At statement granularity the host already links + every use to every def (the L3 call rule), so a summary contributes the callee's global and + by-ref footprint, not intra-caller precision. +4. **VTA matters on a small share of call sites, but they are the ones taint needs.** Of the call + sites that bind a target, 2.8–6.7% gain one that only VTA or the CHA complement found on five + of the six apps, 13.0% on openbao and 13.5% on the Kubernetes root: interface calls + (`store.Driver` in memos, `Repo` in the done fixture), function values, receivers that came + out of library code, and the D7 interface-method binding on top. `k8s-api`, generated + marshallers with no interfaces, has 0.4%. +5. **The points-to delta is moderate and strictly additive.** It is 4–28% of the port `ddg` on + five of the six apps (nats-server 9.6%), 52% on cobra, whose port `ddg` is small, and 19% / 9% + on Kubernetes. The superset gate holds: every `-a 3` edge is in the `-a 4` file unchanged. +6. **Global port wiring is the volume driver.** On nats-server, 2.87M of 3.09M port `ddg` edges + (93%) carry a global `var`. On listmonk, one function, `cmd.initHTTPHandlers` (a route table + that registers about 300 handlers), holds 475k of the app's 523k port edges (91%). It is the + host of hundreds of calls, each with the same global actuals, and the coarse host wiring + multiplies them: every host use feeds every `actual_in`. (Both splits are from the first L4 + eval; neither app's port `ddg` has moved by more than 0.1% since.) +7. **Degradation is reported, never silent.** Five of the six apps and the Kubernetes root use + range-over-func somewhere (`range_func`, D8). Four apps and the root carry `unmapped_ssa`, + which comes almost entirely from closures in package-level `var` initializers. No app hit + `no_ssa`, `no_vta`, `no_cha`, `partial_ssa`, `partial_types` or `l3_skipped`. + +### Per-app results (L4) + +Times are seconds, RSS is MB (2^20 bytes), and JSON is MB (10^6 bytes). Every row is +go1.27.3, `--eager`, one timed run per configuration. + +| app | wall `-a 3 -j 1` | wall `-a 4 -j 1` | wall `-a 4 -j 16` | user CPU `-a 4 -j 1` | RSS `-a 3 -j 1` | RSS `-a 4 -j 1` | RSS × | JSON `-a 3` | JSON `-a 4` | JSON × | SSA | VTA | overlay | `-a 4` identical | +|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| +| cobra | 0.44 | 0.55 | 0.54 | 1.19 | 250 | 287 | 1.15 | 2.1 | 6.8 | 3.2 | 13 ms | 5 ms | 69 ms | yes | +| listmonk | 0.90 | 2.37 | 2.38 | 5.85 | 912 | 1,369 | 1.50 | 10.6 | 103.3 | 9.8 | 35 ms | 29 ms | 1.00 s | yes | +| memos | 2.48 | 5.05 | 4.79 | 13.52 | 2,319 | 2,894 | 1.25 | 43.2 | 184.6 | 4.3 | 190 ms | 144 ms | 1.29 s | yes | +| nats-server | 2.29 | 23.82 | 23.26 | 43.53 | 1,110 | 4,182 | **3.77** | 78.2 | 1,776.7 | **22.7** | 217 ms | 182 ms | 14.68 s | yes | +| openbao | 4.77 | 12.06 | 11.44 | 32.30 | 5,072 | 5,925 | 1.17 | 101.6 | 569.7 | 5.6 | 300 ms | 390 ms | 3.96 s | yes | +| dae (linux) | 1.83 | 4.17 | 3.89 | 10.35 | 1,492 | 2,104 | 1.41 | 27.5 | 193.3 | 7.0 | 82 ms | 89 ms | 1.39 s | yes | +| **kubernetes** | 12.75 | **38.37** | 36.16 | 79.10 | 8,103 | **8,076** | 1.00 | 343.2 | 1,425.8 | 4.2 | 935 ms | 2.44 s | 11.06 s | **yes** | +| k8s-api | 5.22 | 10.58 | 9.50 | 24.87 | 2,052 | 2,533 | 1.23 | 192.8 | 360.2 | 1.9 | 443 ms | 295 ms | 3.41 s | yes | + +`-j 16` barely helps at `-a 4`, by at most 2.2 s: SSA, VTA, the CHA complement and the summary +pass run serially by design (D2, `ssa.BuildSerially`; the SCC pass is serial for determinism), +and emission is one stream. + +**Kubernetes RSS is noisy at `-a 3`.** Two repeat runs of the same binary, `-j 1 --eager`, gave +`-a 3` 13.73 s / 7,139 MB and 13.57 s / 8,213 MB, and `-a 4` 40.08 s / 8,079 MB and 39.17 s / +8,124 MB. The `-a 4` peak is steady at 7.9 GB; the `-a 3` peak moves by about 1 GB with GC +timing during the load, so the ratio over the three paired runs is 0.99–1.13×. + +| app | vertices | `formal_in` / `formal_out` | `actual_in` / `actual_out` | `:` share of vertices | `param_in` | `param_out` | `summary` | sites with summary: all-to-all | port `ddg` | `points-to` `ddg` | stand-ins (formals) | +|---|---|---|---|---|---|---|---|---|---|---|---| +| cobra | 7,500 | 945 / 965 | 2,596 / 2,994 | 21.1% | 2,638 | 3,044 | 6,056 | 1,235: 74.2% | 10,799 | 5,637 | 91 (337) | +| listmonk | 58,546 | 3,698 / 3,721 | 26,432 / 24,695 | 46.6% | 28,279 | 26,666 | 40,912 | 6,105: 80.3% | 522,668 | 20,767 | 506 (2,021) | +| memos | 181,608 | 29,866 / 29,282 | 59,137 / 63,323 | 32.5% | 63,547 | 68,487 | 236,583 | 19,178: 85.5% | 250,469 | 68,956 | 834 (3,273) | +| nats-server | 1,585,369 | 324,214 / 155,426 | 734,885 / 370,844 | **91.1%** | 864,091 | 433,667 | 2,577,267 | 30,005: 79.0% | 3,094,417 | 295,787 | 667 (2,503) | +| openbao | 477,113 | 92,049 / 61,133 | 183,481 / 140,450 | 57.2% | 212,468 | 171,261 | 680,354 | 37,783: 81.7% | 724,250 | 195,336 | 1,646 (6,848) | +| dae | 177,435 | 30,942 / 28,116 | 59,920 / 58,457 | 65.6% | 71,926 | 69,991 | 246,279 | 12,210: 81.6% | 311,385 | 67,149 | 847 (3,317) | +| **kubernetes** | 1,215,278 | 241,510 / 205,241 | 416,744 / 351,783 | 43.1% | 495,377 | 431,709 | 1,361,329 | 99,349: 75.7% | 1,943,025 | 373,553 | 1,679 (6,456) | +| k8s-api | 242,414 | 27,556 / 35,489 | 91,428 / 87,941 | 7.2% | 91,428 | 87,941 | 205,643 | 40,856: 93.7% | 439,414 | 38,963 | 129 (812) | + +Notes: +- The `-v` log's "port edges" count includes the `points-to` edges. The table splits them: + port = `reaching-defs`, plus `points-to`. +- A stand-in is an external with a `body`. Every external a call site targets has one, except + for the zero to thirteen per app whose formals are empty (`runtime.Gosched`). + +| app | call nodes | sites that bind a target | with a VTA-only target | with ≥ 2 targets | `call_graph` VTA edges | VTA function set (library targets) | VTA edges | sites with targets (log) | non-call capture sites | degraded | +|---|---|---|---|---|---|---|---|---|---|---| +| cobra | 1,474 | 1,243 | 35 (2.8%) | 14 | 45 | 411 (83) | 1,294 | 1,247 | 37 | none | +| listmonk | 7,057 | 6,126 | 224 (3.7%) | 67 | 272 | 1,809 (640) | 7,304 | 6,194 | 47 | `range_func` (1 pkg, 1 callable) | +| memos | 23,781 | 19,248 | 1,277 (6.6%) | 797 | 1,069 | 7,893 (1,443) | 21,417 | 19,475 | 380 | `range_func` (7, 10); `unmapped_ssa` (2 fn / 4 calls) | +| nats-server | 38,335 | 30,134 | 1,800 (6.0%) | 1,069 | 2,133 | 10,473 (718) | 39,668 | 30,806 | 587 | `range_func` (2, 27); `unmapped_ssa` (25 / 4) | +| openbao | 51,991 | 37,999 | 4,928 (13.0%) | 4,226 | 4,743 | 19,978 (3,181) | 58,876 | 39,232 | 822 | `range_func` (5, 6); `unmapped_ssa` (10 / 12) | +| dae | 15,354 | 12,269 | 817 (6.7%) | 425 | 890 | 4,498 (1,286) | 14,104 | 12,645 | 263 | `range_func` (5, 5); `unmapped_ssa` (40 / 125) | +| **kubernetes** | 193,481 | 101,948 | 13,770 (13.5%) | 5,514 | 15,943 | 94,601 (41,107) | 249,493 | 104,586 | 3,890 | `range_func` (26, 49); `unmapped_ssa` (3,720 / 39,700) | +| k8s-api | 87,808 | 40,856 | 172 (0.4%) | 0 | 172 | 26,527 (8,570) | 52,526 | 40,856 | 175 | none | + +- **"Sites that bind a target"** are call nodes with at least one `param_in` from an `actual_in`. + A call node binds nothing when its callee has no formals (`runtime.Gosched`), when the CFG + pruned it, or when it is unresolved: a builtin, a conversion, or a function value VTA did not + resolve. +- **"VTA-only"** means at least one bound target is not the call node's static `callee`. It + includes the CHA complement's targets (§ Limitations (L4)), which the dispatch adds before VTA. +- **"Sites with targets (log)"** is the analyzer's own dispatch count. It also counts sites whose + only target has no formals. +- **Degraded:** the `range_func` figures are (packages, callables) and the `unmapped_ssa` + figures are (functions / calls), from the `-v` line. + +### What changed since the earlier measurements + +**Since the first L4 eval (`c230592`, 2026-10-27, same host and toolchain).** The final-review +fixes added dispatch targets, and every new target adds formals, actuals and their edges. The +L3 part of every `-a 4` file is unchanged (its body, `cfg`, `cdg` and `ssa`-`ddg` counts equal the +`-a 3` file's on all eight apps). Growth in vertices / `param_in` / `summary`, and in bound sites +with a VTA-only target: + +| app | vertices | `param_in` | `summary` | VTA-only sites | wall `-a 4 -j 1` | +|---|---|---|---|---|---| +| cobra | +0.1% | +0.1% | +0.1% | 32 → 35 | 0.54 → 0.55 s | +| listmonk | 0 | +0.03% | 0 | 220 → 224 | 2.37 → 2.37 s | +| memos | +6.2% | +7.7% | +9.2% | 997 → 1,277 | 4.68 → 5.05 s | +| nats-server | +0.03% | +0.15% | +0.01% | 1,633 → 1,800 | 23.48 → 23.82 s | +| openbao | +18.5% | +29.4% | +23.6% | 2,461 → 4,928 | 10.77 → 12.06 s | +| dae | +46.8% | +64.9% | +58.2% | 628 → 817 | 3.53 → 4.17 s | +| kubernetes | +5.2% | +10.4% | +8.0% | 11,294 → 13,770 | 31.53 → 38.37 s | +| k8s-api | 0 | 0 | 0 | 172 → 172 | 10.98 → 10.58 s | + +Bisected on openbao, dae and Kubernetes (a build at each fix commit, `-a 4 -j 1`): +- **The library-origin CHA complement is most of the volume change.** Its intraprocedural first + round (`0a3ea12`) and its interprocedural fixpoint (`7762b6e`) take openbao from 402,746 to + 412,702 to 477,113 vertices, dae from 121,056 to 147,785 to 177,435, and Kubernetes from + 1,164,744 to 1,191,953 to 1,215,278. The earlier fixes (VTA through wrappers `4fb14b5`, SSA for + every package variant `3752427`, `call_graph` edges from VTA only `4a2121d`) add about 2.7k + `param_in` and no vertices on openbao, and 10k vertices on Kubernetes (1,154,746 → 1,164,744). +- **The Kubernetes wall-time increase is the interprocedural fixpoint.** With timestamped `-v` + logs, the time between the SSA join and the VTA line is 8.7 s on `98ebfef` against 2.4 s on + `c230592`. VTA itself takes 2.3–2.4 s in both, so the complement takes about 6 s. The overlay + grows by 0.7 s with the extra vertices, and emission by 0.7 s. On the other apps the + complement takes 0.04–0.7 s (openbao 0.7 s, nats-server 0.5 s). It has no `-v` line of its + own. The bisect builds show the same step: Kubernetes `-a 4` goes from 32.4 s at `0a3ea12` to + 37.7 s at `7762b6e`. + +**Against the task reports.** They measured nats-server at several points on the branch, at +`-j 8`, on the toolchain given with each figure. On the eval binary, go1.27.3 `-j 1`: +- **summary** is 2,577,267, against 2,547,371 after the Task 17 alias round (`a2be9a4`; + go1.26.4) (+1.2%); +- **points-to** is 295,787, against 290,499 then and 292,707 after round 2 (`c74c11b`; both + go1.26.4); +- **param_out** is 433,667, against 431k at Task 14 (go1.27.3). + +The growth since `c74c11b` comes from: +- the reference-holding capture `formal_out` (`7ca9c57`, MR5); +- the L3 host weak-def of every reference-holding capture (`40b03ea`); +- the CHA complement (above). + +All add edges only. The L3 `ddg` on nats-server went from 345,318 (§ Per-app counts) to 358,979 +(+4.0%). That includes `40b03ea`'s capture weak defs (chi +0.26%, go1.26.4) and the D1 L3 fixes since the +L3 sections were measured. Wall time and RSS in those reports were measured for one stage, not +end to end, on another toolchain. Compare the numbers here only with each other. + +**`unmapped_ssa`, re-measured.** The Kubernetes root has 3,720 unmapped functions and 39,700 +unmapped calls, against 3,754 / 39,728 at Task 9 (go1.27.3). Classified from `-vv`: +- 3,717 are closures in package-level `var` initializers (`init$N`; Ginkgo + `var _ = Describe(…, func(){…})`; 3,555 of them under `test/`); +- 3 are a second `func init()` in one file (`init#N`). + +The "method declared above its receiver type" cause Task 9 found is gone (fixed in `c6f2bd4`). +None of these is a join bug. The other apps have the same cause in smaller numbers (memos 2, +dae 40, openbao 10, nats-server 25 functions). + +### Cost of the memory levers + +`sdg.Run` lowers the process GC target to 50 while it runs (Task 15b), and L1–L3 and the +emission keep the default. Here is what that costs, measured end to end against a local build +identical except for `l4GCPercent = 100` (not committed), on the first L4 eval's binary +(`c230592`; not re-run on `98ebfef`). Both builds are go1.27.3, `-a 4 -j 1 +--eager`, two runs each, and every output is byte-identical to the eval's. + +| app | GC target in L4 | wall (s) | user CPU (s) | sys (s) | peak RSS (MB) | +|---|---|---|---|---|---| +| nats-server | 50 (shipped) | 23.69 / 23.56 | 43.26 / 43.19 | 2.19 / 2.29 | 3,951 / 4,157 | +| nats-server | 100 | 23.40 / 23.00 | 34.80 / 34.61 | 1.87 / 2.16 | 4,463 / 4,563 | +| kubernetes | 50 (shipped) | 31.66 / 31.74 | 67.36 / 67.53 | 12.60 / 16.49 | 7,713 / 7,742 | +| kubernetes | 100 | 31.55 / 30.82 | 59.70 / 59.92 | 12.78 / 12.63 | 8,254 / 8,518 | + +- **CPU.** GOGC=50 costs **+24.6% user CPU** on nats-server (+8.5 s) and **+12.8%** on Kubernetes + (+7.6 s). +- **Wall time.** It costs +0.4 s on nats-server and +0.5 s on Kubernetes, about 1–2%. The extra + GC work runs on idle cores of a 16-core host. **On a CPU-starved host the extra CPU becomes wall + time**, up to the user-CPU delta. +- **Memory.** It saves 9–12% of peak RSS on nats-server (about 450 MB) and 7–9% on Kubernetes + (about 650 MB). +- The other levers (per-SCC cache release, interned port names) have no CPU cost to report. + Task 15b measured them: nats-server's live heap after fill went from 1.81 to 1.45 GB + (go1.26.4). + +**Where the end-to-end peak sits.** With the levers on, the `-a 4` peak is no longer in `sdg.Run`. +It is in the emission, which streams the envelope but builds each module's subtree whole before +writing it. The largest L4 modules (`c230592`) are: + +| app | module | JSON | +|---|---|---| +| nats-server | `server/server.go` | 340 MB | +| nats-server | `server/jetstream.go` | 164 MB | +| nats-server | `server/client.go` | 131 MB | +| kubernetes | `test/integration/dra/dra.go` | 66 MB | +| listmonk | `cmd/handlers.go` | 58 MB | + +The application-scope `param_in` and `param_out` lists stream (nats-server: 368 MB and 180 MB of +JSON). The next lever, if a larger app needs one, is to stream each callable within a module +(spec §4.13, §11). It is not needed at this corpus's scale: Kubernetes is 0.99–1.13× its `-a 3` +peak. + +### Consistency at scale (L4) + +`scripts/l3_metrics.py` runs its L4 checks on every callable and every edge of the eight `-a 4` +files. It applies the L3 checks of § Internal-consistency checks to the L3 part of the body only, +with vertices and port and points-to edges removed, so the L3 figures compare across levels. +Every check passes on every app: + +| check | result | +|---|---| +| every `param_in` / `param_out` endpoint is a body node id (a callable's or a stand-in's) | 0 dangling, 8/8 apps | +| every `summary` endpoint is a vertex of its callable | 0 failures | +| no `summary` edge has `:` vertices at both ends (#30) | 0 | +| L3 checks over the L3 part of every `-a 4` body (one `@entry`/`@exit`, reachability, endpoints, `prov == ["ssa"]` on the L3 edges) | all PASS | +| `-a 3` ⊆ `-a 4` | gated on the fixtures (`TestGateL4_SupersetOfL3`); on all eight apps the L3 `body` / `cfg` / `cdg` / `ssa`-`ddg` counts of the `-a 4` file equal the `-a 3` file's exactly (counts, not an edge-by-edge diff) | + +The Go-side gates do the arity and kind checks per edge on the fixtures +(`TestGateL4_ArityMatchesFormals`, `TestGateL4_NoDanglingEndpoints`, +`TestGateL4_VertexIdsAreGlobalOrdinals`). The HTTP → SQL "done" fixture +(`testdata/dataflow/httpsql`, `TestGateL4_HTTPToSQLPath`) does a BFS over the emitted JSON graph. +The handler's `r` reaches `database/sql.DB.Query`'s `query` formal through a VTA-bound `param_in` +into `sqlRepo.Find`. A handler that never queries does not reach it, and neither does a separate +entry, `Audit`, reach `Find`'s `id`. `TestGateL4_HTTPToSQLSanitizerIsSDKSemantics` pins the +sanitizer variant (§ Limitations). + +**Metamorphic relations.** `scripts/metamorphic.sh` (memos, the default) passes on `98ebfef` with +0 violations in all 14 relations. It covered 17–32 sites for MR1–MR6, MR14 and MR15, and 80M +reachability pairs for MR5 and MR6 (15M for MR14). The run took 105 s. The last `--all` run +(memos and nats-server, 0 violations, 23 min, 16.2 GB; go1.27.3, the script's `EVAL_GO`) is from +Task 17b on `7c29d44` and `3331abe`. + +### Insights + +- **The coarse host wiring is cheap where calls are sparse, quadratic where they are dense.** A + statement that hosts n calls, each with g global actuals, gets about n·g port edges in each + direction, and a body that hosts many such statements multiplies that again. listmonk's + `initHTTPHandlers` (one function, 91% of the app's port edges) and nats-server's + `client.parse` (162k) are the extremes. Python's reaching-def-matched host wiring (spec §4.7, + optional) would cut this volume without losing a flow; it is not built. +- **Globals are most of L4 on server-shaped code.** nats-server's `Server` methods form one large + recursive SCC. Every package variable any member reads or writes becomes a formal of every + member (A4, transitive), so 91% of its vertices are `:` vertices. The cut of global → + global summaries (#30) already removed 98M summary edges there (Task 15, go1.27.3; spec §4.9). The vertices and + their port edges remain, and they are what sources and sinks can name. +- **VTA is cheap.** About 2.4 s on Kubernetes, with a function set of 94,601, of which 41,107 are + target-only library functions. The SSA build is about 1 s. The CHA complement's + library-origin fixpoint costs more than VTA on Kubernetes (about 6 s) and under 1 s elsewhere. + The dominant L4 compute cost is the overlay: flow graphs, summaries and port wiring, where + `graphOf`/`bodyFlows` and GC dominate (Task 18a profile, go1.26.4). The dominant wall-time cost on the large files is emission. +- **k8s-api is the counter-example.** Generated marshallers have few interface calls (0.4% VTA), + few globals (7% of vertices) and leaf-heavy call graphs. L4 adds only 1.9× bytes there, the + smallest growth in the corpus, because its L3 `ddg` already dominates the file. + +### Limitations (L4) + +These are known, by design or measured, and each is a source of noise (extra edges) or of a gap +(no edge does not mean no flow). Under the anti-goal, every gap is either closed by +over-approximation or reported as degraded. + +- **D7 noise: every interface call also binds the interface method.** + - Its pass-through summary is all-to-all, so an interface call with concrete VTA targets still + carries the all-to-all edges of the interface stand-in. + - Over-approximation by design: a call whose VTA set is empty still keeps its flow. + - Interfaces with no symbol-table node (the universe `error.Error()`, interface literals) get + no interface-method target. They keep only their VTA targets, and with none the L3 host + pass-through applies (spec §4.8). A sink cannot be named as `error.Error`. +- **D8: range-over-func is not modelled.** `for v := range seq` routes the loop body through a + synthetic `yield`, so what the iterator passes to `yield` does not reach `v` + interprocedurally. Every program that uses it is degraded `range_func`: 5 of the 6 apps and + the Kubernetes root. The SDK then refuses to certify "no flow" over that graph (spec §4.15). + That is safe, but it means `exhausted` is never returned on most real apps until D8 is + modelled. +- **Library models and sanitizers change summaries, not statement-level reachability** + [user, 2026-10-27]. + - L4 is reaching-definitions data flow. In `n, err := strconv.Atoi(id)` the host statement + links `id` to `n` and `err` (the L3 call rule), and that is correct. + - `--library-models` `strconv/Atoi: []` removes only the Atoi call site's `summary` edges, so + the done fixture's sanitized path still reaches `DB.Query`. + - An SDK callable sanitizer cuts only nodes under the sanitizer's id, never the caller's host. + - A variable cut on `n` within `SanitizedFind` is bypassed three ways, each correct + dependence: + 1. `err` → `if err != nil` → the Query statement (cdg, implicit flow); + 2. `return nil, err` → the caller → the receiver `actual_in`; + 3. the caller's host feeding `id` into the receiver `actual_in` (coarse host wiring). + - Sanitizer semantics are the SDK taint analysis's job: [codellm-devkit/.github#103](https://github.com/codellm-devkit/.github/issues/103). + `TestGateL4_HTTPToSQLSanitizerIsSDKSemantics` pins each fact. + - The same statement-level call rule exists in codeanalyzer-python + (`access_paths.py:500-501`, `:350-356`), so this is cross-language. +- **Library callbacks are not edges.** In `sort.Slice(xs, less)` and `http.HandleFunc(path, h)`, + the function value flows into the stand-in, but the library calling it back is not modelled. + Taint that *starts* in a handler (its own parameters) is unaffected. +- **Library-originated dynamic types (final review F3, round 2).** VTA sees no concrete type in + library code. An interface call whose receiver may carry a value that came out of library code + (`ctx.Value(k).(Store)`, `sync.Map.Load`, a library call's result, field or container element), + directly or through project code (results, parameters, closure captures, fields, globals, + container elements; a whole-program fixpoint), gets a CHA complement: every project type + implementing the interface. Storage is tracked field-insensitively by container type. CHA + targets added over no CHA (go1.27.3, `-j 8`; round 1 = intraprocedural origin): + + | app | round 1: targets / sites | round 2: targets / sites | param_in, no CHA → round 2 | param_out, no CHA → round 2 | CHA time | + |---|---|---|---|---|---| + | chi | +54 / 15 | +168 / 75 | 1,895 → 2,431 (+28.3%) | 1,649 → 2,242 (+36.0%) | 6 ms | + | memos | +495 / 223 | +803 / 347 | 59,025 → 63,547 (+7.7%) | 64,804 → 68,487 (+5.7%) | 174 ms | + | nats-server | +317 / 63 | +2,290 / 883 | 862,765 → 864,091 (+0.15%) | 431,327 → 433,667 (+0.54%) | 456 ms | + + Peak RSS and overlay time do not move measurably. The parameters of a handler that `net/http` + calls are **not** seeds (a handler's `w.Write` gains nothing). What drives the growth, measured + by ablating one seed kind or propagation rule at a time (a site counts for a rule when + disabling the rule removes some of its CHA targets; the kinds overlap, since several seeds + often reach one site): + - **chi** (75 sites): the seeds are type assertions (`ctx.Value(k).(*Context)`, `r.Context()` + values, `w.(http.Flusher)`; 69 sites with type assertions alone) and library call results + (46 alone). Their reach comes from the type-keyed storage cells: 39 sites (78 targets) need + the load through a closure capture of a pointer cell whose element type some LO store + marked (`any(T)`: the captured `next http.Handler` of each middleware closure, after an + LO `http.Handler` is stored anywhere), and 9 need a field read of an LO base. Most of these + sites are `next.ServeHTTP(w, r)` in middleware closures, which gain chi's 2 handler types. + - **memos** (347 sites): interface type assertions (needed by 281 sites), library call results + (34), field reads of an LO base (83). + - **nats-server** (883 sites): library call results (needed by 118), field reads of an LO base + (597), capture-cell loads (113); most sites have several sufficient seeds (concrete and + interface type assertions, library fields and globals, promoted library methods). +- **A15: summaries are near all-to-all** (74–94% of sites, table above). Under the coarse host + wiring, a summary edge adds no reachability for the SDK's context-insensitive walk. It is a + shortcut hop, and a precondition for a future HRB two-phase walk (spec §11). +- **#30: the global → global summary cut, and the depth-bounded slice caveat.** + - Summary edges with both ends `:` are omitted [user, 2026-10-26]. Measured: 97.7% of + 101M summary edges on nats-server (Task 14, go1.27.3). + - Reachability is unchanged. But python-sdk's slices and taint walks are depth-bounded + (`DEFAULT_DEPTH = 5` for slices). A dropped shortcut makes the remaining route longer (into + the callee's body and back), so a depth-bounded slice may stop short of a node it would + otherwise reach. + - They return with the HRB walk (codeanalyzer-go#30). +- **Option A alias volume.** + - The one-level offset also applies to reference-compatible field types (`m := s.cache; + p := r.ptr`), not only to `&x.f` (Task 17 ruling). That is most of the `points-to` volume: + 217k of 290k on nats-server at Task 17 (go1.26.4). + - `MayHoldRef` also admits value structs that hold a reference, so `s1.n` → `s2.n` between two + distinct value copies is a spurious edge. + - Both are sound over-approximations: 4–28% of port `ddg` here (52% on cobra's small one). +- **One-level alias offsets only.** + - Not caught: offsets deeper than one step (`&x.f.g` against a `*G`), two prefixes of one base + that alias each other, and an interface holding a value struct whose reference field is + written through the interface (spec §4.10). + - These are false negatives of the type-based oracle that an Andersen points-to over `go/ssa` + would close (§11). They are the one class of L4 gap that is neither over-approximated nor + degraded. +- **L3 capture weak-def breadth (`40b03ea`).** + - At L3, a closure's host weak-defines every reference-holding captured base, read-only + captures included. That is broader than needed: a read-only capture could weak-define + `x.*` without downgrading the base. + - Sound. It costs about 0.26% L3 `ddg` on chi (go1.26.4). +- **The stored literal reached through an unresolved function value (Task 13).** + - A global written only inside a literal that is stored and later called through a function + value nothing resolves has no `param_out` from that write. Inside the encloser, the global's + `formal_out` is fed only by its `formal_in` (the identity edge). + - The flow is still reachable at the caller, whose host statement weak-defines the global. + - `TestGateL4_StoredLiteralGlobalReachable` pins both the reachable flow and the + intra-procedural gap, so the gap closing fails the test on purpose. +- **#29: cgo.** cgo-generated wrapper functions have no L1 callable, so SSA functions and calls + inside them are `unmapped_ssa`. A flow through C is not modelled (non-goal). The fixture test + `TestJoin_CgoNamesIssue29` skips when cgo is off. +- **#31: `base_types` / `interfaces`.** + - This predates L4 and is not an L4 change, but L4 consumers meet it. `base_types[]` holds raw + type strings, not type ids, so `GO_EMBEDS` never resolves, and `interfaces[]` is never + computed (`GO_SATISFIES` is empty). + - VTA does not need either; it reads `go/types` directly. +- **Memory and size are reported, not bounded** (D3). The largest file in the corpus is 1.78 GB + (nats-server). Loading it whole in Python, as `scripts/l3_metrics.py` does, needs about + 10–20 GB. The SDK's JSON backend loads it whole too. For apps that size the Neo4j backend is + the practical one. diff --git a/docs/design/notes/l3-eval-2026-10.md b/docs/design/notes/l3-eval-2026-10.md index 4bddc41..8c04d19 100644 --- a/docs/design/notes/l3-eval-2026-10.md +++ b/docs/design/notes/l3-eval-2026-10.md @@ -1,4 +1,4 @@ -# L3 real-app evaluation (2026-10-24) +# L3 / L4 real-app evaluation (2026-10-24, L4 2026-10-27 and 2026-10-28) Open-source Go apps at pinned commits, analyzed with `cango -a 3` via `scripts/eval_l3.sh`. Host: Apple M3 Max, 16 cores, 64 GB, macOS. Analyzer built with go1.27.3 (see Finding 2). Every @@ -10,6 +10,69 @@ and "external" split `call_graph` edges by whether `dst` is an `@external` id. L3 volume, edge/node-kind breakdowns, cost, output size, the protobuf-marshaller measurement and consistency checks over these outputs: [`l3-dataflow-metrics.md`](l3-dataflow-metrics.md). +## L4 (`-a 4`) evaluation at the final head (2026-10-28) + +The same pass, rerun on `feat/l4-sdg` at `98ebfef` (every review fix in; go1.27.3, host and +flags as below). **Gate: passed** again: all eight apps complete `-a 4`, and the three `-a 4` +outputs are byte-identical on every app. + +| app | wall `-a 3 -j 1` (s) | wall `-a 4 -j 1` (s) | wall `-a 4 -j 16` (s) | RSS `-a 3` (MB) | RSS `-a 4` (MB) | × | `analysis.json` `-a 4` (MB) | `-a 4` identical | +|---|---|---|---|---|---|---|---|---| +| cobra | 0.44 | 0.55 | 0.54 | 250 | 287 | 1.15 | 6.8 | yes | +| listmonk | 0.90 | 2.37 | 2.38 | 912 | 1,369 | 1.50 | 103.3 | yes | +| memos | 2.48 | 5.05 | 4.79 | 2,319 | 2,894 | 1.25 | 184.6 | yes | +| nats-server | 2.29 | 23.82 | 23.26 | 1,110 | 4,182 | 3.77 | 1,776.7 | yes | +| openbao | 4.77 | 12.06 | 11.44 | 5,072 | 5,925 | 1.17 | 569.7 | yes | +| dae (linux) | 1.83 | 4.17 | 3.89 | 1,492 | 2,104 | 1.41 | 193.3 | yes | +| **kubernetes** | 12.75 | **38.37** | 36.16 | 8,103 | **8,076** | 1.00 | 1,425.8 | **yes** | +| k8s-api | 5.22 | 10.58 | 9.50 | 2,052 | 2,533 | 1.23 | 360.2 | yes | + +- The degraded codes are the same as on 2026-10-27. +- The Kubernetes root's `-a 3` peak is noisy (7.0–8.0 GB over three runs). Its `-a 4` peak is a + steady 7.9 GB, so the ratio is 0.99–1.13×. +- **What changed.** The final-review fixes added dispatch targets, so `-a 4` grew: + - vertices grew by 5–47% on openbao, dae, memos and Kubernetes, and by under 0.2% elsewhere. + - the Kubernetes root's `-a 4` wall time rose from 31.5 to 38.4 s, about 6 s of it in the + CHA complement's interprocedural library-origin fixpoint. + - Per-app counts and the bisect are in [`l3-dataflow-metrics.md` § L4](l3-dataflow-metrics.md#l4). + +## L4 (`-a 4`) evaluation (2026-10-27) + +One `CANGO_LEVELS="2 3 4" scripts/eval_l3.sh` pass over the same eight prepared trees, with the +binary the script builds from `feat/l4-sdg` at `c230592` (pre-rebase; `a2e7ea0` after the rebase +onto `main`), before the final-review fixes: +- `runtime.Version()` go1.27.3; runs use `GOTOOLCHAIN=go1.27.3`; +- the host as above, j = 16, every run `--eager`. + +After the L3 sequence, each app runs `-a 4 -j 1 -v`, `-a 4 -j 16`, and a second `-a 4 -j 16`, and +the three `-a 4` files are compared byte for byte. + +**Gate: passed.** All eight apps complete `-a 4` with exit 0 and no crash, and all three `-a 4` +outputs are byte-identical on every app. The `-a 3` checks above still pass on the same run. +Memory is reported, not gated (D3). Peak RSS is for the whole process. + +| app | status | wall `-a 3 -j 1` (s) | wall `-a 4 -j 1` (s) | wall `-a 4 -j 16` (s) | RSS `-a 3` (MB) | RSS `-a 4` (MB) | × | `analysis.json` `-a 4` (MB) | `-a 4` identical | degraded | +|---|---|---|---|---|---|---|---|---|---|---| +| cobra `adbc881` | ok | 0.45 | 0.54 | 0.52 | 268 | 290 | 1.08 | 6.7 | yes | none | +| listmonk `8216c4f` | ok | 0.93 | 2.37 | 2.37 | 989 | 1,384 | 1.40 | 103.3 | yes | `range_func` | +| memos `28719a9` | ok | 2.47 | 4.68 | 4.43 | 2,501 | 3,153 | 1.26 | 174.5 | yes | `range_func`, `unmapped_ssa` | +| nats-server `f8f6db8` | ok | 2.31 | 23.48 | 23.27 | 1,128 | 4,103 | 3.64 | 1,775.2 | yes | `range_func`, `unmapped_ssa` | +| openbao `1d58653` | ok | 5.01 | 10.77 | 10.27 | 5,173 | 5,893 | 1.14 | 473.4 | yes | `range_func`, `unmapped_ssa` | +| dae `e3fee8f` (linux) | ok | 1.71 | 3.53 | 3.23 | 1,385 | 1,743 | 1.26 | 130.7 | yes | `range_func`, `unmapped_ssa` | +| **kubernetes** `f78e722` | ok | 13.49 | **31.53** | 30.01 | 7,315 | **7,697** | 1.05 | 1,341.5 | **yes** | `range_func`, `unmapped_ssa` | +| k8s-api `f78e722` | ok | 5.55 | 10.98 | 9.44 | 2,044 | 2,356 | 1.15 | 360.2 | yes | none | + +- **Kubernetes is the scale gate.** The root module completes `-a 4` in 31.5 s at a 7.7 GB peak + (1.05× `-a 3`), deterministically. +- **nats-server is the memory and size outlier** (3.6× RSS, 1.78 GB of JSON). Its one large + recursive `Server` SCC turns most vertices into `:` formals and actuals. +- The L4 counts, the GOGC=50 CPU cost (+12.8% to +24.6% user CPU, about 1–2% wall on this host), + where the peak sits, the consistency checks and the limitations are in + [`l3-dataflow-metrics.md` § L4](l3-dataflow-metrics.md#l4). +- **Findings.** The eval found no crash and no nondeterminism. One script fix was needed: + `eval_l3.sh` learned the `-a 4` triple and its byte check (`a2e7ea0`). `--all` still includes + cockroach, which is not prepared on this host, so the run names the eight apps. + ## Corpus change: Kubernetes replaces cockroach as the scale benchmark - **Why.** cockroach is now under the proprietary CockroachDB Software License, so it cannot be a diff --git a/docs/design/roadmap.md b/docs/design/roadmap.md index aa00b94..42143df 100644 --- a/docs/design/roadmap.md +++ b/docs/design/roadmap.md @@ -10,7 +10,7 @@ A schema **major**: move the Go analyzer off the v1 keystone booleans, `signature` as id) onto the v2 canonical schema (additive tree, `can://` ids, `span` with byte offsets, `source` per module, typed edge lists). The SDK moves in lockstep as a coordinated major release. **Reach for this train: L1 + L2 to parity.** -L3 shipped in release A (below); L4 remains deferred (see Not now). +L3 shipped in release A (below); L4 is implemented on `feat/l4-sdg` (release B, below). Golden rule carried from `schema-migration.md`: keep everything that *computes* facts (parser, resolver, call-graph builder); replace only what *serializes* them. @@ -51,7 +51,7 @@ C (edge shape) ─┴─► L2 emission (call_graph rename @ application sc │ A/B/C ─► D (Neo4j relabel) ─► SDK v2 model remap ─► pin SDK→analyzer (once BOTH cut) │ - L3 (release A) ─► L4 (deferred) ┘ reuse A & C unchanged + L3 (release A) ─► L4 (release B) ┘ reuse A & C unchanged ``` ## Release trains @@ -70,15 +70,33 @@ A/B/C ─► D (Neo4j relabel) ─► SDK v2 model remap ─► pin SDK→analyz found two issues, both fixed before release: L2 call-graph dedup on repeated `.init` signatures (nondeterministic, dropped edges) and the Neo4j `--eager` purge failing on Neo4j 5.x. The release tag is the user's decision. -- **Later (deferred).** L4 (SDG `param_in`/`param_out`/`summary`, needs a points-to oracle). - Reuses group A & C vocabulary; no new schema-major coordination beyond what Train 1 - establishes. +- **Release B — L4 (`-a 4`, SDG).** Implemented on `feat/l4-sdg` + (spec `docs/design/specs/v2-l3-l4-dataflow.md` Phase 2). It adds: + - VTA dispatch over `go/ssa`; + - formal/actual vertices, `param_in`/`param_out`, call-site `summary` edges and port-wiring + `ddg`; + - library stand-ins with `--library-models`; + - a type-based may-alias oracle (`points-to` `ddg`); + - the `degraded` envelope marker and `--strict`; + - the Neo4j `GO_PARAM_IN`/`GO_PARAM_OUT`/`GO_SUMMARY` overlay, with `--emit neo4j` at L4. + + It reuses the group A and C vocabulary; every L4 fact is additive (`-a 3` ⊆ `-a 4`). The + real-app evaluation is in `docs/design/notes/l3-dataflow-metrics.md` § L4 and + `l3-eval-2026-10.md`. Kubernetes `-a 4` completes deterministically, and memory is reported + per app, not gated (D3). Follow-ups: + - #30: global → global summaries for an HRB walk; + - #31: `GO_EMBEDS`/`GO_SATISFIES`; + - range-over-func (D8); + - library callbacks; + - Andersen points-to; + - export-data loading. + + The python-sdk Go L3/L4 surface (spec §7) is the next rung. ## Not now (considered, deliberately excluded) -- **L4 dataflow (SDG)** — deferred; needs a points-to oracle and interprocedural - construction. (L3 is in release A.) -- **Points-to oracle** — L4 prerequisite; parked with L4. +- **Andersen points-to over `go/ssa`** — a precision upgrade replacing L4's type-based may-alias + oracle behind the same interface (`prov` stays `points-to`); not this train. - **`--materialize-expressions` / `--materialize-basic-blocks`** — optional body node kinds; out of scope for the parity migration. - **New framework detection** — orthogonal enrichment axis diff --git a/docs/design/specs/v2-l3-l4-dataflow.md b/docs/design/specs/v2-l3-l4-dataflow.md index a36eebc..e0b8875 100644 --- a/docs/design/specs/v2-l3-l4-dataflow.md +++ b/docs/design/specs/v2-l3-l4-dataflow.md @@ -1,7 +1,10 @@ # Spec: Go L3/L4 dataflow (CFG/CDG/DDG + SDG) with SDK taint (codeanalyzer-go) -**Status:** design, awaiting approval · **Date:** 2026-10-23 · **Repos:** `codeanalyzer-go`, -`python-sdk` · **Supersedes:** the "L3/L4 deferred" entries in `docs/design/roadmap.md`. +**Status:** Phases 0–1 merged (PR #16, merge commit `f9a39bb`); Phase 2 (L4) live on +`feat/l4-sdg` and evaluated (`docs/design/notes/l3-dataflow-metrics.md` § L4); L4 design amended by +the 2026-10-25 re-check (§4.16), the 2026-10-26/27 rulings and the final-review fixes (§4.8) · +**Date:** 2026-10-23, amended 2026-10-25, 2026-10-27 · **Repos:** `codeanalyzer-go`, `python-sdk` · +**Supersedes:** the "L3/L4 deferred" entries in `docs/design/roadmap.md`. ## 1. Intent @@ -67,35 +70,48 @@ siblings is required too; and the id grammar has to change for polyglot apps any case is called out. - **Golden rule** (from the v2 migration): keep everything that computes facts; add L3/L4 as new computation; the v2 emitter stays the single serialization point. -- **Substrate:** `golang.org/x/tools v0.46.0` is already a dependency — `go/cfg`, `go/ssa`, - `go/ssa/ssautil`, `go/callgraph/vta`, `go/callgraph/cha` are available with no new module. +- **Substrate:** `golang.org/x/tools v0.52.0` (`go.mod`, `go 1.26.0`) is already a dependency — + `go/cfg`, `go/ssa`, `go/ssa/ssautil` and `go/callgraph/vta` are available with no new module. + VTA is seeded with no initial call graph (its internal lazy CHA, §4.8); `go/callgraph/cha` is + not used. VTA's package doc still says "experimental" and carries a TODO on generic + instantiation wrappers, so every VTA call runs under `recover` (§4.8, §4.15). `x/tools/go/pointer` is deprecated: **do not adopt**. - Level flag semantics (`cli-contract.md`): `-a 3` implies `-a 2`; `-a 4` implies `-a 3`; an `-a 3` run must not pay L4 cost; `--emit neo4j` runs at max implemented depth (now 4) and rejects `-a` and `--graph-field-depth`. - python-sdk rules: the Iron Rule (no public accessor changes name/signature/return type); new - capability only as new methods; models are `extra="forbid"` mirrors of v2. + capability only as new methods; models are `extra="ignore"` mirrors of v2 (PR #434's choice; + the TypeScript and Java wire models are `extra="ignore"` since python-sdk #386, per their + module docstrings on `main` — an additive analyzer release parses without an SDK edit; §7). ## 3. Current state (what exists) -- `codeanalyzer-go` L1/L2 on v2, Neo4j projection live. Analysis loads packages once via - `go/packages` (`NeedSyntax|NeedTypes|NeedTypesInfo|NeedDeps`); the internal v1-shaped model - (`internal/schema/schema.go`) is re-serialized by `internal/syntactic_analysis/v2emit`. -- No per-statement facts survive into the internal model; L3 must work from `pkg.Syntax` + - `pkg.TypesInfo` directly. -- `internal/schema/v2/ids.go` already defines `OrdinalID` (`@:`) and `TagID` - (`@`). `internal/neo4j/schema.go` already reserves the L3/L4 relationships. -- Known defects to fix on the way: `GO_DDG` discriminant is `var` only (must be `(var, prov)` per - CLAUDE.md K1 and siblings); `typeKind` never emits `alias`/`defined`; `is_deferred` is declared but - never populated; L2 call-site resolution re-walks the whole package AST per site (parked memory - `parked-callsite-resolution-perf`) — must be fixed before L3/L4 multiply call-site lookups. -- L2 dispatch: an interface-method call's callee is the interface method (CHA-style); no concrete - targets. This is insufficient for must-have flow (3). -- python-sdk `main` (2.0.0, c125d48) has **no Go support at all** — verified 2026-10-23: no tracked - Go files, `CLDK` has only `java`/`python`/`typescript` factories, the legacy `analysis()` shim - dispatches only those three. Branch `add-go-support` (July 2026, never merged, 332 commits behind) - is an old **v1-shaped** implementation and is obsolete. Since then PR #434 (unmerged) adds v2 - L1/L2 Go support; the SDK work builds on it (§7). +As of 2026-10-25 (code at `3504445`, branch `feat/l3-dataflow` = `feat/l4-sdg`): + +- **Phases 0 and 1 were implemented together on PR #16** (§9; merged as `f9a39bb`). L1/L2/L3 on + v2, Neo4j projection live through L3 (as of this snapshot; L4 is now live too, Status header). +- **Phase 0 (live):** `can:///go/…` ids (§4.1, `internal/schema/v2/ids.go`); L2 library calls + with `application.external_symbols` (§4.14); co-positioned call keys `l:c/n`; `is_deferred`; + `typeKind` emits `alias`/`defined`; `GO_DDG` `_k = var|prov`; the per-package call-site index + (`callsite_index.go`) replacing the per-site AST walk; body-node `id` from `-a 1`. +- **Phase 1 (live):** `-a 3` and `--graph-field-depth`; per-callable statement body nodes, `cfg` + (defer chain, panic edges), `cdg`, syntactic `ddg` (`prov:["ssa"]`); `k_limit`; Neo4j + `GO_CFG_NEXT`/`GO_CDG`/`GO_DDG`. Package `internal/dataflow`; join key to the tree is + `dataflow.L3Key{Path, Signature}` (`run.go`); ids minted only in the emitter. +- **Load:** `go/packages` once, `NeedSyntax|NeedTypes|NeedTypesInfo|NeedDeps` — every dependency + (stdlib included) is parsed and type-checked from source. That dominates RSS: the Kubernetes + root peaks at 7.1 GB at `-a 3` (`docs/design/notes/l3-dataflow-metrics.md`). +- **L2 dispatch** is still `go/types`: an interface-method call's callee is the bodiless interface + method (`collectInterfaceMethods`); no concrete targets. Insufficient for must-have flow (3); L4 + adds VTA (§4.8). +- **Not yet built (Phase 2)** — *superseded: Phase 2 is now implemented; see the Status header. + Kept as the 2026-10-25 snapshot.* `-a 4` is rejected by the CLI; no `LevelSDG` in `internal/options`; + no `--strict`; `--emit neo4j` forces `LevelDataflow` (L3); `v2.Analysis` has no `degraded` and + the package doc says L4 fields are not declared; `ExternalSymbol` has no receiver or body; + `GO_PARAM_IN`/`GO_PARAM_OUT`/`GO_SUMMARY` stay `Reserved`. +- **python-sdk:** Go v2 L1/L2 support is on PR #434 (branch `pr-434-v3`; merged into python-sdk + `main` on 2026-10-26, after this section was written); the SDK work builds on it (§7). It cannot yet parse `-a 3` output (§7 prerequisites). `add-go-support` is + obsolete. ## 4. Decisions @@ -261,8 +277,13 @@ truncated). Match Java's `AccessPath` / Python's `access_paths` semantics: every argument `a` that is pointer-, slice-, map-, chan-, func- or interface-typed, and the receiver if it is a pointer receiver or reference-typed** (pointer, slice, map, chan, func or interface — e.g. `hs.Write(b)` on an interface, `h.Set(k, v)` on a map-typed `http.Header`; the - callee may write through it; Python's suffixed-weak-def precedent). Value-typed struct arguments and - receivers are not weak-defined (copy semantics). A **slice expression** argument (`r.Read(buf[:n])`, + callee may write through it; Python's suffixed-weak-def precedent). **Amended (D1, gap 2):** the + predicate is widened to **"may hold a reference"** — the reference kinds above, or a struct or + array whose fields/elements transitively contain one (cycle-safe). A value struct or array that + may hold a reference (`fill(cfg)` with `cfg.items []T`) weak-defines `a.*` (whole base, not per + field, to stay k-safe). A value struct or array that holds no reference is not weak-defined (copy + semantics). The **same** predicate function is the L4 by-ref `formal_out` rule (§4.6), so caller + weak defs and callee ports agree. A **slice expression** argument (`r.Read(buf[:n])`, `io.ReadFull(r, hdr[:])`, and the destination of `copy(dst[i:], src)` / `clear(s[:n])`) shares its elements with its base, so it weak-defines the base's elements **`base[*]`** — slice, array or pointer-to-array base alike (`&`, `*` and parentheses around the base are stripped). `[*]` rather @@ -298,29 +319,45 @@ truncated). Match Java's `AccessPath` / Python's `access_paths` semantics: Differential check (non-gating, reported): compare scalar def-use against `go/ssa` (the construction guide's answer key) on fixtures; mismatches are triaged, not asserted. -#### L3 precision (known gaps) +#### L3 precision (known gaps) — amended 2026-10-25 (D1) [user] The L3 `ddg` is **syntactic and intraprocedural**. It over-approximates where it models a flow -(weak defs, bidirectional prefix matching), but it is **not refutation-complete**: the absence of -a `ddg` path does not prove the absence of a flow. It does not model: - -- **Writes to captured variables inside closures.** `func(){ x = secret }()` or - `once.Do(func(){ cfg = load() })` writes `x` / `cfg` in the closure's own body; the enclosing - callable sees only a use of the captures at the closure expression, never a def, so a later - read of `x` in the enclosing callable does not link to the closure's write. -- **Value-typed struct arguments or receivers that contain slice, map or pointer fields.** A - value struct is copied (no weak def, §4.5 Calls), but its reference-typed fields alias the - caller's: `fill(cfg)` with `cfg.items []T` may write `cfg.items[*]`, which L3 does not record. -- ~~Deferred calls reading receiver or argument state at exit~~ — **closed at L3, except for - reassignment**: the deferred call node also uses every path it may write through - (`defer w.Flush()` uses `w.*`, `defer fill(b[1:])` uses `b[*]`), so a write after the - registration reaches it. The paths are keyed on the *variable* at exit, not on the value - bound at registration, so a reassignment after the `defer` (`defer fill(b); …; b = o`) can - add a spurious edge, and can kill an earlier def on the normal path. - -The first two are addressed at **L4**: capture formals (`@formal_in`/`@formal_out` for captured -variables, §4.6 closures rule) carry closure writes back to the enclosing callable, and the -points-to delta (§4.10) links writes through reference fields of copied structs. +(weak defs, bidirectional prefix matching). The re-check found that L4 as first designed closed +neither gap 2 (the §4.10 oracle aliases only reference-typed bases, and the by-ref predicate +excluded value structs) nor gap 3. **Decision (D1): all three gaps are closed at L3**, before L4 +port wiring (§9 Phase 2 prerequisites), and L4 still adds capture ports on top: + +- **Gap 1 — writes to captured variables inside closures.** `func(){ x = secret }()` or + `once.Do(func(){ cfg = load() })` writes `x` / `cfg` in the closure's own body. **L3 fix:** the + node containing the literal weak-defines each captured variable the literal (or a literal nested + in it) may write; the write-set is computed syntactically per literal, and (amended 2026-10-27, + from metamorphic relation MR5) also holds `x.*` for every captured `x` that may hold a reference + (`MayHoldRef`): a write through a local alias, `q := p; q.n = v`, writes no path of `p`. In + sequential code a + closure cannot run before it exists, so this is complete apart from goroutine interleaving (a + non-goal). Expect more `ddg` edges at capture sites (measured on Kubernetes). **L4** then adds a + traceable source through capture ports (§4.6 closures rule). +- **Gap 2 — value structs or arrays with reference fields.** `fill(cfg)` with `cfg.items []T` may + write `cfg.items[*]`. **L3 fix:** the "may hold a reference" predicate (§4.5 Calls) weak-defines + `a.*`. Weak defs drive the `ddg` long tail (`l3-dataflow-metrics.md` takeaway 4), so the change + is gated on a Kubernetes measurement. **L4** reuses the same predicate for by-ref `formal_out` + (§4.6). +- **Gap 3 — deferred operands after reassignment.** The deferred call node already uses every path + it may write through (`defer w.Flush()` uses `w.*`, `defer fill(b[1:])` uses `b[*]`). The real + miss was `defer fill(b); b[0] = s; b = o`: `b = o` kills `b[*]` before the exit, losing the write + `fill` reads. **L3 fix:** every write whose base is an operand base of a registered `defer` in the + same callable also weak-defines `:`, which closes the false negative. L4 does not + touch gap 3. + +**Remaining documented over-approximation:** the deferred call's exit-time uses stay keyed on the +*variable* at exit, not on the value bound at registration (they are the points-to hook), so a +reassignment after the `defer` can still add a **spurious** edge. That is extra edges, not a +dropped flow. + +With these fixes the L3 `ddg` misses no flow it models, but L3 is still **intraprocedural**: a flow +through a callee is not an L3 fact, and the absence of an L3 `ddg` path does not prove the +absence of a flow. A `go/ssa` def-use differential (built with the L4 SSA join, §4.8) is +reported, non-gating. ### 4.6 L4 vertices [call; naming follows Python because the SDK reads it] @@ -328,19 +365,49 @@ Per callable (formals), all with `kind` + `of`, no span: | Vertex | `of` | When | |---|---|---| -| `@formal_in:` | real parameter name; receiver name for the receiver | receiver first (index 0) if method, then declared params in order (unnamed `_`/blank params get `_`) | +| `@formal_in:` | real parameter name; receiver name for the receiver | receiver first (index 0) if method, then declared params in order (unnamed or `_` params: `_`; unnamed or `_` receiver: `_recv` — D6) | | `@formal_in:` | `:` | each variable a closure captures (after declared params) | | `@formal_in:` | `:::` | each package-level variable the callable (transitively, via summary) reads | -| `@formal_out` / `@formal_out:` | `` | **one per result position** — Go's multiple returns: `` for a single result; for >1 results, `@formal_out:` with `of` = the named result's name if named else `:` | -| `@formal_out:` | parameter name | each pointer-/reference-typed parameter (incl. pointer receiver) the callable may write through (by-ref out) | -| `@formal_out:` | `:::` | each package-level variable written | +| `@formal_out` / `@formal_out:` | `` / `:` | **one per result position** — `` for a single result; `:` for each position of a multi-result, **never the user's result names** (D6) | +| `@formal_out:` | parameter name | each parameter (incl. the receiver, pointer or value) satisfying the **shared by-ref predicate** the callable may write through (by-ref out) | +| `@formal_out:` | `:` | each captured variable a closure writes (strong or weak def), or that satisfies the by-ref predicate (a write through a local alias, `q := p; q.n = v`, is no def of `p`) | +| `@formal_out:` | `:::` | each package-level variable written (transitively, via summary) | + +**Indexing and order (amended, A9) [call].** Python's `IdentityMap` rule verbatim +(`codeanalyzer-python dataflow/identity.py`): `formal_in` and `actual_in` are always indexed; a bare +`@formal_out` / `/actual_out` **iff that group has exactly one vertex**, else every vertex in +the group is indexed — so a single-result function with a by-ref or global out has +`@formal_out:0` (``), `@formal_out:1`, …. `formal_out` order: results by position, then +by-ref params (receiver first) in declaration order, then captures, then globals sorted by `of`. +`formal_in` order: receiver, declared params, captures, globals sorted (matches `EntryBases`). + +**`of` naming (D6) [call].** +- Unnamed or `_` param: `_`, i = the **declared-param index, receiver excluded** — the same + spelling `external_symbols[].parameters[]` uses. (In-project `parameters[].name` stays `_`; the + vertex `of` is the authority.) +- Unnamed or `_` receiver: `_recv`. +- Captures `:`; globals `:::` (matches the L3 `ddg` base + spelling `pkgpath::Name`; the Go SDK matches on import-path segments, §7). +- **The SDK reads `of`, never an index into `parameters[]`**: `formal_in:0` is the receiver for a + method, so Java's `formal_in:` = `parameters[n]` convention does not hold for Go. + +**By-ref predicate (amended, A3/A10) [call].** By-ref `formal_out` (callee-side `mutated_params`) +uses **the same function** as the L3 caller-side call rule (§4.5 Calls): reference-typed (pointer, +slice, map, chan, func, interface) **or "may hold a reference"** (struct/array transitively +containing one, D1 gap 2), for parameters and the receiver alike — a pointer receiver, and a value +receiver of reference kind (map-typed `http.Header`) or one that may hold a reference. Caller weak +defs and callee ports therefore agree. Per call site with a resolved callee (in-project **or stand-in**, §6): | Vertex | `of` | `parent` | |---|---|---| -| `/actual_in:` | the bound callee formal's name (mirrors the callee's `formal_in:` `of`) | call-site local id | -| `/actual_out` / `…/actual_out:` | mirrors callee's `formal_out` `of` | call-site local id | +| `/actual_in:` | the bound callee formal's name (mirrors the callee's `formal_in:` `of`) | the call key | +| `/actual_out` / `…/actual_out:` | mirrors callee's `formal_out` `of` | the call key | + +Vertices are **per call node** (Java style), keyed on the L1 call key (A11). Python groups actuals +per statement; that is a recorded parity difference the SDK does not depend on. A call node the +CFG pruned (unreachable, kept in `body` with no edges) gets **no** vertices. **Why Python's naming, not the keystone's `$ret`/`argN` [call, the one non-trivial parity choice]:** the SDK's `resolve_value` resolves a source/sink `(name, within)` against `formal_in` `of`, and @@ -349,7 +416,7 @@ With Python's spelling the shared resolver works unchanged; Java/TS's `$ret`/`ar Go-specific translation layer. The `:` multi-result form is a Go-only leaf extension of ``. Sources/sinks address `formal_in` only, so `:` never reaches `value_candidate`; it does reach `body_node_kind` (which labels path hops), where `RETURN_MARKER` -is compared by equality — so an unnamed multi-result hop would be labelled `return` with name +is compared by equality — so a multi-result hop would be labelled `return` with name `:1`. The Go backend normalises this locally before calling the shared helper; shared `commons/` stays untouched. @@ -360,37 +427,72 @@ miss later assignments (false refutation). Rule: - The closure callable gets `@formal_in:` `of:":"` per captured variable and `@formal_out:` `of:":"` per captured variable it **writes** (strong or weak - def). -- In the enclosing callable, the `FuncLit` has a **capture site**: the statement containing the - `FuncLit` is the host; vertices `/actual_in:` and `/actual_out:` - (`` = `line:col` of the `func` keyword; `parent` = host statement's local id). - `param_in`/`param_out` connect them to the closure's capture formals. No `summary` edges at a - capture site (it is not a call; flow through the closure is via its own body). + def) **or that satisfies the shared by-ref predicate** (amended 2026-10-27, from metamorphic + relation MR5): as for a by-ref param, a write through a local alias (`q := p; q.n = v`) is no + def of the capture at L3, so a reference-holding capture always gets its `formal_out`. +- In the enclosing callable, the `FuncLit` has a **capture site** (amended, A6 — a call's key is + `call.Pos()` = `Fun.Pos()` = `FuncLit.Pos()`, so keying the site at the `func` keyword collides + with the call node of `func(x){…}(v)`, `defer func(){…}()`, `go func(){…}()`): + - **Directly called literal:** the captures bind **at that call node**. Their + `/actual_in:` / `actual_out:` indices continue after the declared arguments' + (matching the closure's `formal_in` order: params, then captures). For `defer func(){…}()` this + is the deferred call node in the exit chain. + - **Any other literal** (stored, passed, returned): the capture site is keyed at the literal's + position, taking the next free `/n` suffix the way `BodyNodes.add` does, so it never moves an + L1 call key (chains such as `func() T {…}().M()` already shift to `l:c/2`). + - The site's vertices carry `parent` = the innermost CFG node that contains the literal (the + host); at a directly called literal, `parent` = the call key, as for any call (§4.6 table). + - `param_in`/`param_out` connect the site to the closure's capture formals. No `summary` edges at + a non-call capture site (it is not a call; flow through the closure is via its own body). - **Flow-insensitive wiring for captured variables** (port wiring, `prov:["reaching-defs"]`): *every* def of a captured variable anywhere in the enclosing callable → the capture site's `actual_in`; the capture site's `actual_out` → *every* use of that variable in the enclosing - callable. This over-approximates timing (covers defer, goroutines and callbacks alike) and is the - same weak-update posture as the rest of the design. + callable, **and** (amended, A7) the enclosing callable's own `@formal_out` port for that variable + when it is a named result, a by-ref param base or a global. Without the latter, + `defer func(){ err = wrap(err) }()` — whose deferred call node only *uses* named results, with + nothing defining them after it — never reaches `@formal_out(err)`: a false refutation on the + commonest Go error idiom. This over-approximates timing (covers defer, goroutines and callbacks + alike) and is the same weak-update posture as the rest of the design. - Calls *of* the closure value (`f := func(){…}; f()`) resolve to the closure callable via VTA at L4 and get ordinary call-site vertices for its declared parameters. Go specifics: - **Variadic** `f(xs ...T)`: excess actuals collapse onto the last formal (weak, as Java). - **Multi-value call as args** `f(g())`: each `g` result binds to the corresponding `f` formal. -- **Method value / expression** (`f := s.M; f(x)`; `T.M(s, x)`): resolved callee is `M`; receiver - binds `formal_in:0`. +- **Method expression** (`T.M(s, x)`): resolved callee is `M`; `s` binds `formal_in:0`. +- **Method value** (amended, A8) (`f := s.M; f(x)`): the receiver is not an argument of the call, + and VTA resolves the call to `M$bound` (unwrapped to `M`, §4.8). For a bound-method callee the + call's **function-value operand** (`f`) feeds `actual_in:0`, and the arguments bind + `formal_in:1…`. L3 already carries `s` into `f` (the `f := s.M` statement uses `s`). - **Global state as extra parameters** (keystone L4 rule): globals read/written are computed in the summary pass and propagated up the call graph, so a caller's `formal_in`/`formal_out` set includes globals its callees touch (Python precedent). -- **Global filter — only globals that can carry data [call, from builder check].** Python emits - 546k `:` formals on one real app; transitive propagation is quadratic at cockroach scale. - A package-level variable is modelled as an extra parameter only if it is **mutable in a way that - can carry a value**: it has at least one write (strong or weak def, incl. a method call through it - or `&v` escaping) in any project function body including `init`, **or** its initializer contains a +- **Transitive globals at hosts (amended, A4) [call].** L3 `@entry` defines only the package vars a + body touches **directly** (not those of nested literals), and a satellite call has an empty + DefUse, so a global G only a callee reads has no def reaching `/actual_in(G)`, and a global + only a callee writes never defines G at the host. Adopt Python's `augmented_facts` + re-solve + (`summaries.py`): at each host statement, add the callee's `global_reads` as uses and its + `global_writes` as **weak** defs; `@formal_in(G)` is the def of G for transitively read globals. + Re-run `BuildDDG` and emit only the **new** edges, `prov:["reaching-defs"]`. Weak defs kill + nothing, so the L3 `ssa` set is untouched and L3 ⊆ L4 holds. +- **Global filter — only globals that can carry data [call, from builder check; amended A5].** + Python emits 546k `:` formals on one real app; transitive propagation is quadratic at + Kubernetes scale. A **project** package-level variable is modelled as an extra parameter only if + it is **mutable in a way that can carry a value**: it has at least one write in any project + function body **including closures and `init`** — a strong def, a **weak def from the call rule** + (`mu.Lock()` on a global with a pointer receiver), or `&v` — **or** its initializer contains a call (e.g. `os.Getenv`). A global initialized only from literals/constants and never written - cannot carry taint, so dropping it removes no flow. Constants and functions are never globals. - This filter is sound under the over-approximation rule; output size is measured in the eval (§8) - and a size budget set from the first measurement. + cannot carry taint, so dropping it removes no flow. **Non-project (library) package vars always + pass the filter** (`os::Args`, `os::Stdin`, `flag` vars — the L3 `@entry` already defines them via + `isPkgVar`); without that they would be unnameable as sources. Constants and functions are never + globals. This filter is sound under the over-approximation rule; `:` volume is measured in + the eval (§8) and a size budget set from the first measurement. +- **Range-over-func (amended, A21; D8) [call].** `for v := range seq` has no `CallExpr` (no L1 call + node), and SSA routes the loop body through a synthetic yield function, so data the iterator + passes to `yield` (a `rows.Scan` result inside a DB iterator) never reaches `v` + interprocedurally. **Not modelled this train:** a program that uses range-over-func gets degraded + code `range_func` (§4.15), and a fixture pins the gap. Modelling `for … range f` as a call site + of `f` is a follow-up (§11). ### 4.7 L4 edges [call] @@ -403,11 +505,18 @@ Go specifics: statement def-use): `@formal_in:k → first-use statements of that base` (replaces the `@entry` def for that base for port purposes — keep both, the `@entry` edge stays for L3 monotonicity); `return stmt → - @formal_out[:i]` (`var` ``/name), statement writing a by-ref param/global → + @formal_out[:i]` (`var` = the formal's `of`, `` / `:` — never the user's + result name, D6), statement writing a by-ref param/global → corresponding `@formal_out`; `host stmt → /actual_in:i` (`var` = arg spelling or `argN` if not an access path); `/actual_out[:i] → host stmt` (`var` = the LHS base it defines). **This wiring is what the SDK's `_ports_carry_dependence` probe checks; without it taint certifies false refutations.** +- **Port host (amended, A11) [call].** Go records the host directly: host = the call node's + `parent` if set, else the call node itself (a bare call, a `go` call, a deferred call — each is + its own CFG node). For a deferred call the host is the deferred call node in the exit chain; its + DefUse already carries `:` plus the weak defs of §4.5. Wiring is Java's coarse + host → `actual_in` (every host use feeds every `actual_in`; sound). Python's reaching-def-matched + variant is optional. A pruned call node (no vertices, §4.6) gets no port wiring. - `summary` (callable scope, local ids): `/actual_in:i → /actual_out[:j]` iff the callee's summary says formal `i` may flow to output `j`. - Edge-only-when-resolved: a call site whose callee cannot be resolved at all (e.g. a call through a @@ -421,13 +530,16 @@ Go specifics: ### 4.8 Dispatch resolution: VTA for L2 and L4 [call] Must-have flow (3) needs concrete targets for interface and function-value calls. Use -`x/tools/go/callgraph/vta` over `go/ssa` (built with `ssautil.AllPackages` from the same -`go/packages` load, `ssa.InstantiateGenerics`), seeded with a CHA call graph -(`callgraph/cha`) — the construction guide's recommended Go oracle. - -- **L2 refinement:** `call_graph` gains VTA-derived edges for interface/func-value call sites, `prov` - `["vta"]`, merged with existing `["go/types"]` edges via `MergeEdges` (union prov, sum weight). - This is additive (monotone). Body `callee` stays a single id (the static callee — the interface +`x/tools/go/callgraph/vta` over `go/ssa` — the construction guide's recommended Go oracle — with SSA +built per D2 (amended, A12): `ssautil.Packages` over the **same** `go/packages` load, not +`AllPackages`. + +- **L2 refinement (amended, Task 10):** `call_graph` gains VTA-derived edges for interface/func-value + call sites, `prov` `["vta"]`, weight 1 — **only** for a `(src, dst)` pair the L2 resolver did not + already find. An edge L2 found that VTA also finds is left exactly as L2 emitted it (its `prov` + is not unioned, its weight not summed): the `-a 3` `call_graph` is a byte-for-byte subset of the + `-a 4` one (the monotonicity gate). `Result.CallEdges` holds only the VTA-only pairs, so + `Result.ApplyTo`'s `MergeEdges` appends them and never unions into an L2 edge. Body `callee` stays a single id (the static callee — the interface method for an interface call) — the schema has one slot. Multiple concrete targets are carried by `call_graph` and, at L4, by one `param_in`/`param_out` set **per target** from the same `actual_in` vertices (the actual vertices are per call site, not per target). @@ -437,22 +549,116 @@ Must-have flow (3) needs concrete targets for interface and function-value calls `actual_*` have a target for the static edge) and a **pass-through summary** (like a stand-in, §6), so a call whose VTA target set is empty still does not drop flow. When VTA finds concrete targets, `param_in`/`param_out` go to each concrete target **and** to the interface method; the - call-site `summary` edges are the union over targets. -- **SSA/VTA scope [call, from builder check]:** build SSA with `ssautil.AllPackages` over the - loaded packages (dependency SSA is needed for types and wrappers), but give VTA a function set of - **project functions only** (the SSA functions whose position lies in a project file, plus their - anonymous functions), seeded with a CHA graph restricted the same way. VTA needs no `main`/roots, - so libraries work. Library callbacks are therefore not discovered (§6, as intended) and cost stays - bounded by project size. Packages that fail to type-check get no SSA (`AllPackages` yields nil); - see §4.12 degradation. -- **SSA → `can://` mapping [call]:** map an SSA function to a callable by source position: - `fn.Origin()` for generic instantiations; synthetic wrappers (bound `$bound`, thunk `$thunk`, - promoted-method and pointer wrappers from embedding) unwrap to the method they wrap; anonymous - functions (`f$1`) map to the L1 closure whose `FuncLit` position equals `fn.Pos()` (L1 closure - naming `closure_N` stays; the position is the join key). A target that maps to nothing in-project - is an external stand-in (§6) if it has a package path outside the project, else dropped with a - debug log. Build an `ast` position → callable index once (also fixes the parked per-site AST walk). -- **VTA runs only at `-a 4`** (and `--emit neo4j`, which is `-a 4`): SSA construction is the + call-site `summary` edges are the union over targets. **D7:** an interface call *always* binds to + the interface method (pass-through) and every VTA target, even when every target is in-project — + per the anti-goal. Cost: every interface call (`io.Writer`) carries an all-to-all summary. + A target VTA did not produce — the D7 interface method, or a CHA-complement target (below) — is + a dispatch target with its param edges but **no `call_graph` edge** [final review F7]: `prov + ["vta"]` means VTA found it, and L2 already carries the static edge where it resolved. +- **Through synthetic wrappers [final review F1].** When VTA's callee at a project call site is a + synthetic wrapper with no syntax (`$bound`, `$thunk`, a promotion or pointer wrapper), the + wrapper's own call of what it wraps is a synthetic call site no call node joins. For an + interface method's wrapper — a method value `f := s.Find; f(id)`, or a struct embedding the + interface — VTA's concrete callees sit only there. The wrapper's VTA out-edges are followed + transitively (cycle-safe) and their joined callees are targets, and `call_graph` edges, of the + original site. +- **Targeted CHA complement [final review F3; controller ruling; interprocedural: user, + 2026-10-27].** VTA propagates types through project bodies only, so a value that passes through + library code — `ctx.Value(k).(Store)`, `sync.Map.Load`, a library call's result, a library + container's element — reaches an interface call with no concrete type VTA sees, and only the D7 + interface method (empty footprint) would remain. A whole-program monotone fixpoint + (`cha_origin.go`) computes the SSA values that **may carry a library-originated dynamic type** + (LO): + - *Seeds:* a `TypeAssert`; the result of a library function, of a function with no SSA body, of + a library interface's method, of `recover()`; a load of a library global or through a library + struct's field. + - *Local steps:* `Phi`, `Extract`, `ChangeInterface`/`ChangeType`/`Convert`, `Slice`, `Field` + and `FieldAddr`/`IndexAddr` of an LO base; a map `Lookup`, an `Index`, a channel receive + (`UnOp ARROW`, `select`) and a range `Next` over an LO container (N2). + - *Interprocedural:* a project function's result *i* (any `Return` operand *i* is LO) flows to + every call of it; an LO actual makes the callee's parameter LO (receiver included); an LO + `MakeClosure` binding makes the closure's free variable LO. Call targets are the static + callee; for an interface call, the method each project implementer selects (CHA) — and the + result is LO when the receiver is LO or a library type the project converts to an interface + implements it; for a call through a function value, every address-taken project function + of an identical signature — and the result is LO when the value is LO or a library function + of that signature is used as a value. VTA's targets are not used: the fixpoint runs before + VTA (so the program is never read after a `no_vta`), over supersets of them. + - *Storage, field-insensitive by container type:* an LO store marks the project global, or the + **(struct type, field index)** (every instance), or the slice/array element type, map key / + value type or channel element type of the container (any container of that type); a load of + that global, field or element is LO. A store through a pointer of unknown provenance (a + parameter, a call result) marks its element type for every load; a pointer, slice, map or + channel of interface elements passed to library code (`errors.As(err, &target)`) marks its + cells (the library may write them). + - *Termination and determinism:* each fact only goes false → true and the facts are finite + (per function result, parameter, capture, global, struct field and type), so the reader-tracked + worklist ends; a function is re-evaluated (a local fixpoint over its Phi cycles) only when a + fact it read changed. Every input an evaluation reads is either fixed before the worklist + starts (the program, the project types and their method sets, the address-taken function + sets, and memos of pure functions of them, such as which implementers' methods are library + code) or a tracked fact that re-queues its readers when it turns true. So each evaluation is + a monotone function of the facts, and the result is the least fixpoint, which is unique and + independent of the visiting order (`TestCHA_OriginOrderIndependent` permutes the seed + order); the order is deterministic anyway (functions sorted by package path, name). + + Every `invoke` site in a joined project function whose receiver is LO gets, beside D7 and VTA, + every **project** named type `T` or `*T` (non-generic) whose method set implements the interface + (`types.Implements`) — the method its method set selects, mapped by the join. A receiver no + library value can reach (only project-built values: `MakeInterface` of project types) gets no + CHA target. Not a `call_graph` edge (F7). The complement and the fixpoint run under `recover`: + a panic drops every CHA target and degrades `no_cha` (N1). Reflection, `unsafe` and cgo stay + unsound spots (§4.7). +- **Interfaces with no symbol-table node (ruling, Task 10) [call].** `error.Error()` (the universe + `error`), methods of interface literals (`interface{ M() }`), and methods promoted through an + embedded interface whose origin is declared outside the symbol table have no callable id and no + external id, so such a call gets **no** interface-method target. It keeps its VTA targets; + when VTA finds none the call is unresolved, and the L3 host pass-through (§4.7: the call + statement uses its receiver and arguments and defines its LHS) keeps the intra-caller flow. No + false refutation; but a sink cannot be named as `error.Error`. A stand-in for the universe + `error.Error` is a follow-up (§11). +- **VTA-only externals (amended, A19).** A VTA target outside the project that the `go/types` + resolver never saw (e.g. `(*bytes.Buffer).Write` reached through `io.Writer`) is **added** to + `external_symbols` at `-a 4`, with its stand-in body (§6). Monotone: `-a 3` ⊆ `-a 4`. +- **SSA construction (amended, A12; D2) [call].** The load is + `NeedSyntax|NeedTypesInfo|NeedDeps`, so `AllPackages` would create, and `Build` would compile, + **every dependency body** (stdlib included). Instead: + - `ssautil.Packages(roots, ssa.InstantiateGenerics|ssa.BuildSerially)` over every loaded + project package variant (`builder.Loaded()`): bodies only for project packages; dependencies + come from types. With tests loaded, `builder.Pkgs()` (keyed by path) holds only the test + variant `p [p.test]`; the plain `p` other packages import is a root too, or VTA misses the + types that flow through it [final review F2]. Both variants share syntax and join the same + units. + - Build **per package** (`pkg.Build()`) in our own goroutine under `recover` — `Program.Build` + and VTA panic inside their own goroutines, out of reach of the per-callable `recover`. A panic + degrades (`partial_ssa` / `no_ssa`, §4.15). + - Ill-typed packages (`pkg.IllTyped`) get no SSA and degrade to `partial_ssa`. +- **VTA scope [call, from builder check].** Give `vta.CallGraph(funcs, nil)` a function set of + **project functions only** (the SSA functions of project packages, plus their anonymous + functions). `initial = nil`: VTA uses its internal lazy CHA (no separate `callgraph/cha` seed). + VTA needs no `main`/roots, so libraries work. Library callbacks are therefore not discovered (§6, + as intended) and cost stays bounded by project size. VTA runs under `recover`; a panic degrades to + `no_vta`. Targets are sorted by `can://` id (determinism: never key on SSA names or map order). +- **SSA → L3 join (amended, A12) [call].** By **AST pointer identity**, never by position: + - Function: `fn.Syntax()` == the L3 unit's `Unit.Fn` (`l3units.go`); generic instances via + `fn.Origin()`. This holds because SSA is built from the same `*ast.File`s; positions break under + `//line`, and SSA's anonymous-function numbering (`f$1`) follows evaluation order, not + `ast.Inspect` order (L1 closure naming `closure_N` stays). + - Call instruction: `CallCommon.Pos()` == `CallExpr.Lparen` → the `*ast.CallExpr` → its L1 key. + Never by `line:col`. + - Synthetics unwrap: `$bound` / `$thunk` and promoted-method / pointer wrappers to the method + they wrap; instantiation wrappers to the origin; range-over-func `yield` synthetics fold into + `fn.Parent()`; `init#N` to its `init`; "from type information" synthetics have no syntax and map + to no callable. + - An SSA function or call that maps to nothing in-project is an external stand-in (§6) if it has a + package path outside the project; otherwise it is counted and degrades with `unmapped_ssa` + (§4.15). +- **Mapping to `can://` ids (amended, A13) [call].** The L4 IR keys vertices by `(L3Key, local key)` + and cross-callable edges by `L3Key` pairs and external sigs — no ids in the IR. The emitter mints + global ids from an `L3Key` → callable-id map built in the same walk as `flow.apply` (`emit.go`). + Do not reuse `sigIndex.bySigPath`: it keys on the **nesting** module path, not the declaring file. + (The per-site AST walk the old text tied to this is already fixed by `callsite_index.go`.) +- **VTA runs only at `-a 4`** (and `--emit neo4j`, which is `-a 4` per D4, §4.12): SSA construction is the dominant cost; `-a 2`/`-a 3` keep today's cheap `go/types` resolver (an `-a 3` run must not pay L4 cost). VTA edges therefore appear in `call_graph` only at `-a 4` (still monotone: L3 ⊆ L4). - Unsound spots (reflection, `unsafe`, cgo, plugins): no targets ⇒ handled by the unresolved-call @@ -470,10 +676,43 @@ Context-insensitive. Per callable compute: results, by-ref params, written globals — by forward reachability over the callable's `ddg ∪ cdg` from each input's `@entry`/`formal_in` def, bridging call sites via callees' summaries (`summary` edges). Including `cdg` (implicit flow) matches Python and over-approximates. -- `global_reads`, `global_writes`, `mutated_params` (written through, from the weak-def rule), - propagated up the SCC DAG. +- **What summaries refine (amended, A15).** At statement granularity the host already links every + use to every def (the L3 call rule), so a callee summary adds **no intra-caller precision**; it + contributes only the callee's global footprint (A4, §4.6) and its by-ref writes. Summary edges + will be close to all-to-all — a precision limit, not a bug (Python's stated posture, + `summaries.py`). +- `global_reads`, `global_writes`, `mutated_params` (written through, from the shared by-ref + predicate, §4.6), propagated up the SCC DAG and injected at hosts as uses / weak defs (A4). - No hammock regions (neither sibling built them; recorded as a known simplification). - Stand-ins (§6) contribute their modelled summary (default all-inputs → all-outputs). +- **Summaries are not sanitizers [user, 2026-10-27].** L4 is reaching-definitions data flow, not + taint. A modelled summary (`--library-models`), like an SDK callable sanitizer, changes the + call site's `summary` edges and the callee's flows, **not statement-level reachability**: the + value a call site assigns (`n, err := strconv.Atoi(id)`) keeps its reaching-definition edge + from `id` through the host statement (the L3 call rule, A15), and that edge is correct. A + variable cut (`("n", within=…)`) is bypassed too, by implicit flow (`err` → `if err != nil` → + the next statements, cdg), by results returned to the caller, and by the coarse host wiring + (the caller's host feeds every `actual_in`, receiver included). Deciding that a call + sanitizes is the SDK taint analysis's job: codellm-devkit/.github#103. The done fixture pins + all of this (`TestGateL4_HTTPToSQLSanitizerIsSDKSemantics`). +- **Global → global summary edges are not emitted [user, 2026-10-26].** A `summary` edge whose + `actual_in` and `actual_out` both have a `:` `of` is omitted; every edge with at least + one non-global end is kept, and the flows (and so the global formals, actuals and param edges) + are computed in full. Under the coarse host wiring (A15) a summary edge only shortcuts + host → `actual_in` → … → `actual_out` → host, and sources and sinks address `formal_in`, so a + global → global summary adds no reachability for an HRB-unaware walk. Measured on nats-server: + 97.7% of 101M summary edges were global → global; the cut took `sdg.Run`'s peak RSS from + 11.1 GB to ~4.4 GB (2.34M summary edges left). They return with the HRB two-phase walk (§11). + Tracked as **codeanalyzer-go#30**. Caveat: the SDK's slices and taint walks are + **depth-bounded** (python-sdk `DEFAULT_DEPTH = 5` for slices); a dropped global → global + shortcut makes the remaining route longer (into the callee's body and back), so a depth-bounded + slice may stop short of a node it would have reached through the shortcut. Reachability with an + unbounded depth is unchanged. +- **Unanalysed callables (amended, A16).** A callable whose L3 result was dropped (a contained + panic or a call-count mismatch, `run.go`) has no `ddg`, so taint entering through `param_in` + would stop at its `formal_in`. Such callables get formals and a **pass-through summary** (like an + interface method) and are listed under degraded code `l3_skipped` (§4.15). The same applies to SSA + functions that map to no callable (`unmapped_ssa`). - Caching of summaries in `cache_dir` and SCC-wavefront parallelism: **builder's choice** whether to implement in this train; determinism is mandatory either way. @@ -492,6 +731,47 @@ oracle"); `go/pointer` (deprecated). The oracle's API is frozen as an interface so an Andersen implementation can replace it later without schema change (`prov` stays `points-to`). +**As built (Task 17 and its fix round 1) [call].** `AliasOracle.MayAlias(d, u)` asks whether a write +to `d` may change memory a read of `u` observes. The oracle is built per callable +(`newAliasOracle(unit, root)`). The solver re-runs the same reaching-definitions fixpoint over the +port wiring's augmented `du'`. It never adds a kill across bases. It consults the oracle only for a +def and a use of different bases inside a **one-step offset window**: after at most one leading step +is dropped from each, the suffixes are prefix-compatible. +- **Anchors.** The oracle matches each path at offset 0 (its base) or offset 1 (its base and first + step). The anchored prefix is one of two things: + - a *reference*: the object it points to. A bare def of a reference rebinds it and writes + nothing that another base observes. + - an *address-taken value*: its own storage. +- **Compatible references.** + - Pointee types are compared structurally on the underlying types, with `types.Identical` on the + element types. This covers pointers, slices, maps (key and element), chans of either direction, + and identical signatures. So `type Bytes []byte` aliases `[]byte`, and `http.Header` aliases + `map[string][]string`. + - Any two interface (or type-parameter) types are compatible. + - An interface is compatible with a type that implements it. +- **Address-taken values.** A value x is address-taken when the callable's top-level declaration + (closures included) contains any of: + - `&x`, `&x.f…` or `&x[i]…`; + - `x[:]` or `x[i:j]` on an array value; + - a call or method value of a pointer-receiver method on x or `x.f…`. + + Package-level variables are always address-taken. +- **Address-taken matching.** An address-taken value of type T may alias: + - a `*T`; + - an interface that `*T` implements; + - for an array `[N]E`, a `[]E`. + + A field whose address is taken, or a field of reference type, matches at offset 1. So `p := &o.f; + p.g = v` reaches `o.f.g`, and `m := s.cache; m[k] = v` reaches `s.cache[k]`. +- **Never alias.** Two distinct value variables never share storage. Neither do value copies that are + not address-taken. +- **Known gaps (false negatives).** + - Offsets deeper than one step (`&x.f.g` against a `*G`). + - Two prefixes of one base that alias each other. + - An interface holding a value struct whose reference field is written through the interface. + + The Andersen upgrade closes them. + ### 4.11 Neo4j projection [call] Fill the reserved overlay, no contract renames: @@ -505,46 +785,109 @@ Fill the reserved overlay, no contract renames: flat span props siblings use if the SDK backend needs them (builder confirms against `PyNeo4jBackend`). - `GO_HAS_BODY_NODE` callable → every body node (incl. synthetic). -- `GO_CFG_NEXT {kind}` `_k=kind`; `GO_CDG`; `GO_DDG {var, prov}` **`_k=var|prov`** (fix the - current `var`-only discriminant); `GO_SUMMARY`; `GO_PARAM_IN {var}` / `GO_PARAM_OUT {var}`. +- `GO_CFG_NEXT {kind}` `_k=kind`; `GO_CDG`; `GO_DDG {var, prov}` `_k=var|prov` (live since + Phase 0/1); `GO_SUMMARY`; `GO_PARAM_IN {var}` / `GO_PARAM_OUT {var}`. **Amended (A17):** today + `schema.go` declares the param rels with no props; Phase 2 adds `var` to both (Python parity, + `codeanalyzer-python neo4j/schema.py`). - Stand-ins (§6) project onto the existing `:GoExternal:GoSymbol:GoCanNode` ghost (same - `can:///@external/…` id, so today's call-graph ghosts and stand-ins are one node), gaining - `signature`/`module` props and `GO_HAS_BODY_NODE` to their `formal_*` vertices. Their vertex ids - are `@formal_in:`, so the SDK's `under_callable` id-prefix rule covers them. - `RelHasBodyNode.From` widens to `{GoCallable, GoExternal}` in `schema.go`. -- Flip `Reserved:false`; conformance test must see every declared relationship emitted on the + `can:///@external/…` id, so today's call-graph ghosts and stand-ins are one node; it already + carries `signature`/`module`). Their vertex ids are `@formal_in:`, so the SDK's + `under_callable` id-prefix rule covers them. `GO_HAS_BODY_NODE` already allows a `GoExternal` + source. **Amended (A17):** the projector today creates `:GoExternal` only as call-endpoint ghosts + and never walks an external's body; Phase 2 adds a **stand-in pass** that emits each external's + `body` vertices as `:GoBodyNode` with `GO_HAS_BODY_NODE` from the `:GoExternal`. +- **`degraded_json`** (amended, A17) — a new `:GoApplication` property carrying the envelope's + `degraded[]` (§4.15). +- Flip `Reserved:false` (done: nothing in `schema.go` is `Reserved`); conformance test must see every declared relationship emitted on the fixture; cross-projection gate: Neo4j counts == JSON counts modulo `HAS_*`. -- `--emit neo4j` now runs `-a 4`. +- `--emit neo4j` now runs `-a 4` (D4, §4.12). +- **Lazy Bolt push (amended, A17; D5) [call].** The lazy push diffs per module by `content_hash`, + but L4 facts are non-local: editing a callee changes callers' `summary` edges, actual vertices and + even `formal_in` indices (a transitive global set grows). Policy: at L4 a lazy push **upserts + every L4 row on every push** (additive — adds, never removes). Removing stale L4 rows needs + `--eager`. The README documents this under Neo4j and in Upgrading. App-wide L4 purge-and-rewrite + on a lazy push is deferred (§11). ### 4.12 CLI and envelope [call] - `-a 3`, `-a 4` accepted; `--graph-field-depth ` (default 3, error below `-a 3`, error with `--emit neo4j`). +- **Phase 2 CLI tasks (amended, A14).** Today `-a 4` is rejected, there is no `LevelSDG` in + `internal/options`, no `--strict`, `--emit neo4j` forces `LevelDataflow`, and the + `--graph-field-depth` help says "at -a 3 only". Phase 2 adds `LevelSDG`, accepts `-a 4`, adds + `--strict`, sets the neo4j level per D4, handles `-t` per D9, and updates the help to "L3+". +- **`--emit neo4j` level (D4) [user].** `--emit neo4j` runs at max implemented depth = **L4**, and + `-a` / `--graph-field-depth` with it stay an error — the same as Python and Java v2. (Letting + `--emit neo4j` accept `-a ≤ max` would be a CLI-contract change shared with the siblings; not + decided here.) +- **`--target-files` with L4 (D9) [user].** L4 is whole-program, so a targeted run cannot produce + correct summaries or `param_in` edges. Incremental L4 is deferred to the CLDK-wide + incremental-analysis design (§11). + - `-a 1..3 -t` (JSON): unchanged, still works. + - `-a 4 -t` (JSON): an error, exact text: + `--target-files is not supported at analysis level 4 (whole-program); run without -t, or use -a 3` + - `--emit neo4j -t`: runs at **L3**, not L4, and pushes the targeted modules' L1–L3 rows as today. + It writes no L4 rows: the targeted modules' L4 rows stay as the last full run left them until + the next full run (`--eager` removes stale ones, D5). It logs one warning on stderr, exact text: + `--target-files: projecting the graph at level 3 (level 4 is whole-program); L4 rows keep the last full run's facts until a full run` + - The README documents both rules under Neo4j and in Upgrading. - Envelope: emit `k_limit` at L3+ (Python precedent; harmless superset of the keystone's "at L4"). - `max_level` = requested level. -- `--strict`: fail non-zero when SSA/VTA construction fails at `-a 4` (default: degrade — keep L3, +- **Envelope and tree fields (amended, A18).** `v2.Analysis` gains `degraded` (§4.15); `BodyNode` + gains `of`; `Callable` gains `summary`; `Application` gains `param_in`/`param_out`; + `ExternalSymbol` gains `receiver` and `body` (§6). All absent below `-a 4` (the `schema.go` + package doc's "L4 fields … are not declared here" is retired). +- `--strict`: fail non-zero on any degradation at `-a 4` (§4.15) (default: degrade — keep L3, derived vertices and param edges from `go/types` dispatch, log a warning to stderr, and record - the `degraded` envelope marker, §4.15), Java precedent. + the `degraded` envelope marker), Java precedent. - `-j N`: parallelize L3 per callable and L4 per SCC wavefront if implemented; collect then sort by id; never mint ids or emit inside a worker. Byte-identical to `-j 1`. ### 4.13 Caps [call] No hard node/time caps (neither sibling has them; a cap silently drops edges ⇒ false refutation). -Termination comes from k-limiting and finite summary domains. Memory: build L3 per callable and -discard ASTs-derived scratch after emission; SSA program is shared and freed after L4. - -### 4.14 L2 external calls — fixing the current gap (Phase 0) [user: fix it; details are calls] - -Today the L2 resolver resolves library callees (`internal/semantic_analysis/call_graph.go:105-131`) -and then drops them: no `call_graph` edge, and the emitter leaves the call node's `callee` `null` -(`v2emit/emit.go:207-216`). Python and TS emit these; Java emits them behind `--external-calls` -(always under `--emit neo4j`). Go fixes this at L2: +Termination comes from k-limiting and finite summary domains. + +**Memory (amended, A20; D3) [user, 2026-10-26: measured, not gated].** Today L3 nodes keep their AST (`ir.go` `Stmt`), the +whole envelope is marshalled into one `[]byte` (`core/analyzer.go`), and most RSS is `go/packages` +type info (Kubernetes root: 7.1 GB at `-a 3`). L4 adds SSA bodies for project packages, VTA, +summaries and roughly 1.5–2.5× the body nodes (the ports). +- **Prerequisites:** #27 streaming JSON output and AST release (before the real-app L4 eval). +- L4 needs the ASTs to bind actuals, so each callable's L4 vertices are built **before** its AST + references are dropped. The SSA program is shared and freed after L4. +- **Memory is measured, not gated (D3, amended) [user, 2026-10-26].** L4 is whole-program + analysis and needs more memory than L3; the former "≤ 1.5× `-a 3`" budget is dropped. The + real-app eval (§8) reports, per app, the absolute `-a 4` peak RSS and its ratio to `-a 3`. + Loading dependencies from export data (dropping `NeedDeps` syntax) is a separate follow-up, not + this train (§11). +- **Output-neutral levers (Task 15b).** `sdg.Run` releases each unit's flow graph and call-site + bindings once its call-graph SCC is solved and its summary edges are produced, and the cached + port wiring once `fill` has copied it into the result; it interns the formal keys and `of` + strings (every `:` spelling) so identical spellings share one string; and it lowers + the process GC target to 50 while it runs (`debug.SetGCPercent`, restored on return, never + raised above the caller's), so L1–L3 and emission keep the default. None changes a fact: the + result is deeply equal with them off (`TestSDG_OutputUnchangedByMemoryLevers`). +- **Where the `-a 4` peak sits (measured, Task 16/18).** With the levers on, the end-to-end + peak is no longer in `sdg.Run` but in the emission: the streaming writer still builds one + module's subtree whole before writing it, and an L4 module of a large file is big + (nats-server's `server/server.go` module is 340 MB of JSON, measured in the 2026-10-27 eval, + `l3-dataflow-metrics.md` § L4). The next lever, if one is needed, + is to stream per callable within a module; measured numbers per app are in + `l3-dataflow-metrics.md` § L4. + +### 4.14 L2 external calls (Phase 0, implemented on PR #16) [user: fix it; details are calls] + +Before Phase 0 the L2 resolver resolved library callees (`internal/semantic_analysis/call_graph.go`) +and then dropped them: no `call_graph` edge, and the emitter left the call node's `callee` `null`. +Python and TS emit these; Java emits them behind `--external-calls` (always under +`--emit neo4j`). Go now emits them at L2: - **`application.external_symbols{}`** (the Python/Java name) — one entry per distinct external callee: `id` (`can:///@external//`, §6), `kind:"external"`, `name`, `module` (import path), `signature`. Parameters/return type are added here too (cheap from - `go/types`; span-less `{name, type, is_variadic?}`); the L4 `body{}` vertices come in Phase 2. + `go/types`; span-less `{name, type, is_variadic?}`, unnamed → `_`, receiver excluded). The + receiver is not recorded today (`recordExternal` drops it); Phase 2 adds the optional + `receiver {name, type}` leaf and the L4 `body{}` vertices (§6). An external id may name a library *interface* method (e.g. `@external/io/Writer.Write`), and a method on a generic library type records the declared (origin) signature. - **`call_graph`** keeps edges to external targets (`dst` = external id, `prov:["go/types"]`). @@ -559,29 +902,43 @@ and then drops them: no `call_graph` edge, and the emitter leaves the call node' statement defining the LHS uses `b`. (Builtins likewise: L3 "call uses every argument, defines its LHS".) - This is an intended **L2 output change shipped in Phase 0** (more `call_graph` edges, non-null - `callee` on library calls, a new `external_symbols` map). -- Neo4j: the existing `:GoExternal` ghost path in `project.go` (`callEndpoint`) starts receiving - rows; `GO_RESOLVES_TO` extends to external callees. `GO_CALLS` endpoint set already allows - `GoExternal`. -- Gate update: the L2 fixture gate (`gate_l2_test.go`) currently asserts external callees stay - `null`; it changes to assert the external id plus a matching `external_symbols` entry, and no - dangling endpoint. + `callee` on library calls, a new `external_symbols` map; README Upgrading). +- Neo4j: the `:GoExternal` ghost path in `project.go` (`callEndpoint`) receives these rows; + `GO_RESOLVES_TO` extends to external callees; `GO_CALLS` endpoints include `GoExternal`. +- Gate: the L2 fixture gate (`gate_l2_test.go`) asserts the external id plus a matching + `external_symbols` entry, and no dangling endpoint. ### 4.15 Degradation marker [user] -When `-a 4` cannot build what L4 needs — SSA/VTA construction fails, or some packages failed to -type-check and got no SSA — the analyzer still emits (L3 + derived vertices + `go/types` dispatch) -but marks the envelope: +When `-a 4` cannot build what L4 needs — SSA/VTA construction fails, some packages failed to +type-check and got no SSA, or some callables or constructs were not analysed — the analyzer still +emits (L3 + derived vertices + `go/types` dispatch, with pass-through summaries for unanalysed +callables) but marks the envelope: ```json "degraded": [{"code": "no_vta", "detail": "...", "packages": ["example.com/x/pkg"]}] ``` - Envelope leaf, sibling of `max_level`; **absent** when nothing degraded (absence = no fact). - Codes (open vocabulary, plain strings): `no_ssa` (SSA build failed entirely), `no_vta` (VTA - failed), `partial_ssa` (listed packages had type errors / no SSA), `partial_types` (packages with - `go/packages` errors at any level). `max_level` still states the requested level. -- Projected to Neo4j as a `degraded_json` property on `:GoApplication`. + Entry shape `{code, detail?, packages[]?, callables[]?}` (`callables[]` = `can://` ids). + `max_level` still states the requested level. +- Codes (open vocabulary, plain strings): + - `no_ssa` — SSA build failed entirely. + - `no_vta` — VTA failed (incl. a recovered VTA panic). + - `no_cha` — the CHA complement (§4.8) panicked (recovered): its targets are dropped, so an + interface call on a library-originated value keeps only D7 and VTA's targets. + - `partial_ssa` — listed packages had type errors (`pkg.IllTyped`) or a recovered per-package + build panic, so no SSA. + - `partial_types` — project packages `go/packages` loaded with errors (list, parse or type + errors, or ill-typed through a broken dependency), every test variant included; `packages[]` + sorted. A package also listed under `partial_ssa` got no SSA. + - `unmapped_ssa` — SSA functions or call instructions that map to no callable or call node + (§4.8); counted, listed where nameable (A16). + - `l3_skipped` — callables whose L3 result was dropped (contained panic, call-count mismatch); + listed in `callables[]`; they get formals and a pass-through summary (§4.9, A16). + - `range_func` — the program uses range-over-func, which is not modelled (§4.6, D8); listed + in `callables[]`. +- Projected to Neo4j as a `degraded_json` property on `:GoApplication` (§4.11). - `--strict` turns any degradation into a non-zero exit. - **SDK:** the Go backends read it at attach. When present, `taint()` returns **no `exhausted` pairs** and adds an `unresolved` diagnostic naming the degradation (`complete=False`), so a @@ -589,19 +946,52 @@ but marks the envelope: returned. Same rule for `paths_between`-style refutations. Implemented in the Go backend (the diagnostic goes into the `blocked` ledger the walk already returns, which `taint_verdict` honours — shared `commons/` unchanged). +- **Go leaves (all additive, none in the shared keystone yet):** the `degraded` envelope field + itself; the entry's `callables[]` (`can://` ids, sorted; carried by `l3_skipped` and + `range_func`); the codes beyond the original four (`partial_types`, `unmapped_ssa`, + `l3_skipped`, `range_func`, `no_cha`); and, in the vertex vocabulary it degrades over, the `_recv` `of` + for an unnamed or `_` receiver (D6) and `:` (§4.6). - Go-only leaf now; candidate for the keystone alongside the cross-language taint revisit (§11). +### 4.16 Decisions of the 2026-10-25 re-check + +The L4 design was re-checked against the shipped L1–L3 code (`3504445`) and the siblings; the +amendments are folded into the sections above. Decisions: + +- **D1 (#22) [user]:** all three L3 gaps are closed at L3, and L4 still adds capture ports; the + `go/ssa` def-use differential is built with the L4 SSA join and does not gate (§4.5). +- **D2 [call]:** SSA comes from `ssautil.Packages` over the same load, built per package with + `ssa.InstantiateGenerics|ssa.BuildSerially` under `recover` (§4.8). +- **D3 [user; amended user, 2026-10-26]:** memory is measured and reported per app (absolute + `-a 4` peak RSS and its ratio to `-a 3`), not gated: L4 is whole-program. (Was: `-a 4` peak + RSS ≤ 1.5× `-a 3` on the Kubernetes root.) Export-data loading is a separate follow-up (§4.13). +- **D4 [user]:** `--emit neo4j` runs at max implemented depth = L4; `-a`/`--graph-field-depth` + with it stay an error, as in Python and Java v2 (§4.12). +- **D5 [call]:** a lazy Bolt push upserts every L4 row on every push; removing stale L4 rows needs + `--eager`, documented in the README (§4.11). +- **D6 [call]:** `of` = `` / `:` (never user names); unnamed or `_` params + `_` (declared-param index, receiver excluded); unnamed or `_` receiver `_recv`; captures + `:`; globals `:::` (§4.6). +- **D7 [call]:** an interface call always binds to the interface method (pass-through summary) + **and** to every VTA target (§4.8). +- **D8 [call]:** range-over-func is not modelled; a program that uses it gets degraded code + `range_func`, and a fixture pins the gap (§4.6, §4.15). +- **D9 [user]:** L4 is whole-program; `-a 1..3 -t` unchanged; `-a 4 -t` (JSON) is an error; + `--emit neo4j -t` projects at L3 with a one-line stderr warning; exact texts and the full rule in + §4.12; incremental L4 deferred to the CLDK roadmap (§11). + ## 5. Flow coverage map (user goal 4 → mechanism) | Flow | Mechanism | Level | |---|---|---| | (1) through calls | formal/actual vertices, `param_in/out`, `summary`, port wiring | L4 | | (3) interface / func-value dispatch | VTA targets → per-target `param_in/out` | L4 | -| (2) fields & pointers | k-limited access paths; weak-def through reference args; type-based may-alias `points-to` delta | L3 + L4 | -| closures | captures as `:` formals; nested callable's own SDG; capture bound at the `FuncLit` statement as `actual_in` (Python precedent) | L4 | +| (2) fields & pointers | k-limited access paths; weak-def through reference args and "may hold a reference" value structs/arrays (D1 gap 2); type-based may-alias `points-to` delta | L3 + L4 | +| closures | L3: the literal's host weak-defines the captures it may write (D1 gap 1). L4: captures as `:` formals; nested callable's own SDG; capture site at the direct call node or the literal (A6), wired flow-insensitively incl. into `@formal_out` (A7) | L3 + L4 | | goroutines | `go f(x)` is a call for data purposes (vertices + edges); no interleaving | L4 | | channels | `ch[*]` container path; cross-goroutine via args/captures/fields | L3 + L4 | | library calls | stand-ins with default pass-through summary | L4 | +| range-over-func iterators | **not modelled** (D8): degraded code `range_func`, fixture pins the gap | — | ## 6. Library stand-ins (sinks/sources naming library functions) [user] @@ -617,14 +1007,18 @@ addition now; the cross-language SDK taint revisit is deferred (§11). **extended** on Go with `callables`-like content: each external entry carries `kind:"external"`, `name`, `module` (import path), `signature`, `parameters[]` (real names from `go/types`; unnamed → `_`), `return_type`, and at L4 a `body{}` holding only its `@formal_in:` / - `@formal_out[:i]` vertices. The map itself lands at L2 in Phase 0 (§4.14); Phase 2 adds only - `body`. Reachable by containment from `application` ⇒ keystone-legal. + `@formal_out[:i]` vertices. The map itself landed at L2 in Phase 0 (§4.14). **Amended (A19):** + Phase 2 adds two optional leaves — `receiver {name, type}` (a method's receiver; it gives + `formal_in:0`'s `of`, `_recv` when unnamed) and `body{}`. `ExternalParameter` (span-less, `_`) + fits unchanged. Externals only VTA finds are added at `-a 4` (§4.8; monotone). Reachable by + containment from `application` ⇒ keystone-legal. - **Id:** `can:///@external//` where `` is the existing `calleeSignatureOf` canonical form minus the package path (`DB.Query`, `Sprintf`). Language-neutral `@external` per Python/TS/Java. -- **Vertices:** `formal_in` per receiver+param (`of` = real name), `formal_out` per result - (`` / `:`), plus by-ref `formal_out` for each pointer/slice/map/interface param - and pointer receiver (library may write through them — `json.Unmarshal(data, &v)`, `buf.Write`). +- **Vertices:** `formal_in` per receiver+param (`of` = real name, else `_recv` / `_`, D6), + `formal_out` per result (`` / `:`), plus by-ref `formal_out` for each param and + receiver satisfying the shared by-ref predicate (§4.6) (library may write through them — + `json.Unmarshal(data, &v)`, `buf.Write`). Indexing per §4.6 (A9). - **Stand-in summary (pass-through default):** every `formal_in` → every `formal_out`. Materialized as `summary` edges at each **caller's** call site (`actual_in:i → actual_out:j`), *not* as edges inside the stand-in. The stand-in has no outgoing `param_out` into unrelated callers beyond the @@ -635,12 +1029,35 @@ addition now; the cross-language SDK taint revisit is deferred (§11). present"); not made worse by stand-ins. Recorded as a known over-approximation. - **Models file (optional overrides):** `--library-models ` (builder may also embed a small default set), mapping an external id/signature to an explicit flow list, e.g. - `strconv.Atoi: [] ` (no flow), `strings.TrimSpace: [[0,""]]`. Starts empty or + `{"strconv/Atoi": [], "strings/TrimSpace": [["s",""]]}`. Starts empty or near-empty; no catalogue of sources/sinks — that stays the caller's (SDK rule). + **Format (amended, Task 15):** a JSON object; key = `/`, the external id's + tail (`strconv/Atoi`, `database/sql/DB.Query`); value = a list of `[in, out]` pairs of `of` + strings (`"s"`, `"_0"`, `"_recv"`, `""`, `":1"`, a by-ref param's name). + `[]` = no summary edges at that external's call sites (not "no flow": the call site's + assigned values keep their statement-level reaching-definition edges, and cutting those is + SDK taint semantics, §4.9, codellm-devkit/.github#103); a missing key = the all-to-all + default. Valid only at `-a 4` and with + `--emit neo4j` (`-a 1..3`: error `--library-models applies only at analysis level 4`); a + missing, unreadable or malformed file is an error naming the path. An entry naming an `of` + the external lacks is a warning and is ignored for that external, which keeps the all-to-all + default (a typo never refutes a flow); an entry for an external no call site targets (or a + misspelt key, `strconv.Atoi`) is ignored with one warning per key, in sorted key order: + `--library-models: no call site targets ; entry ignored (keys are /, e.g. strconv/Atoi)`. + A method of a generic library type is keyed by its declared (origin) signature + (`sync/atomic/Pointer.Store`), so one entry covers every instantiation. Under + `--emit neo4j -t` (L3, D9) the file is still read and checked but unused, and the D9 warning + ends `; --library-models is ignored at level 3`. - **Not modelled:** callbacks invoked by library code (`sort.Slice(xs, less)`, `http.HandleFunc(path, h)`): the function-value argument flows in, but the library calling it back is not an edge. Taint *starting* in a handler is unaffected (sources are the handler's own params). Recorded limitation. +- **Library-originated dynamic types** (`ctx.Value(k)`, `sync.Map.Load`, a library call's + result or container element) reach interface calls through project code; VTA sees no + concrete type in library code, so the CHA complement (§4.8) binds every project implementer + at each call the interprocedural origin fixpoint reaches. Its storage tracking is + field-insensitive by container type (every instance of a struct field, every container of an + element type): sound, coarser than VTA. - **SDK addressing:** `("query", "database/sql.DB.Query")` resolves through the Go backend's `resolve_callable` (domain = analysed callables ∪ stand-ins) then `resolve_value` over the stand-in body's `formal_in` `of`. Shared `commons/` code is untouched. @@ -648,10 +1065,28 @@ addition now; the cross-language SDK taint revisit is deferred (§11). ## 7. SDK (python-sdk) [user: in scope; Claude may write it] Build on python-sdk PR #434 (L1/L2 Go support: models, ABC, local + Neo4j backends, facade, factory; -unmerged as of 2026-10-23), fixing it first against the v2-only analyzer. `add-go-support` is obsolete. - -- **Models** `cldk/models/go/`: `extra="ignore"` (PR #434's choice, matching TS #386) mirror of Go v2 (module/type/field/callable/body - node/edges/external symbols), mirroring how `cldk/models/python` and `java` model L3/L4. +branch `pr-434-v3`, unmerged as of 2026-10-25), fixing it first against the v2-only analyzer. +`add-go-support` is obsolete. + +- **Phase 3 prerequisites (amended, A22)** — on `pr-434-v3` today: + - `GoType.kind: Literal["struct","interface"]` **rejects** `alias`/`defined`, which the analyzer + emits today (`extra="ignore"` does not help with a Literal mismatch). Fix first, so the SDK can + parse current output at all. + - `GoBodyNode` lacks `id`/`parent`/`of`/`is_deferred`; there is no `external_symbols`/ + `param_in`/`summary`/`degraded`, and no `_ports_carry_dependence`. Add the L3 fields first (so + `-a 3` output parses), then L4. + - `extra` policy is **`ignore`** (§2). +- **Id splitting (A22).** External ids contain `/@external/` before the vertex `@`, so the Go + backend must **never split a body-node id on its first `@`**; strip the known callable id prefix + instead. +- **Global names (A22).** `value_candidate` turns `:example.com/lib::V` into + `example.com/lib.V`, so a `lib.V` query fails `segment_match` (dot-separated). The Go backend + normalises the import path, or matches on `/` as well (D6 keeps `:::`). +- **`of`, not `parameters[]` (A9).** Go backends resolve formals by `of`; `formal_in:0` is the + receiver for a method. +- **Models** `cldk/models/go/`: `extra="ignore"` (PR #434's choice, matching TS/Java since #386) + mirror of Go v2 (module/type/field/callable/body node/edges/external symbols incl. `receiver` and + `body`, `degraded`), mirroring how `cldk/models/python` and `java` model L3/L4. - **Facade** `GoAnalysis` via `CLDK.go(...)` and the legacy shim; `GoAnalysisBackend` ABC with `N="Go"`, `P="GO_"`. - **Surface:** the decided CLAUDE.md SDK facade (type-centric: `get_types`/`get_type`, @@ -694,7 +1129,14 @@ with ≥2 implementations; func-value call; `a→b→c` chain; mutual recursion; function read in another; multi-file flow; cross-file method; multi-result returns; variadic; library pass-through and library sink (`database/sql`, `os/exec`, `fmt`); generics (existing `testdata/generics`: instantiation → `Origin()` mapping, generic interface dispatch); an HTTP-handler → -SQL-sink end-to-end taint fixture with a sanitizer variant (the "done" fixture). +SQL-sink end-to-end taint fixture with a sanitizer variant (the "done" fixture; the sanitizer +variant pins what L4 gives a sanitizer, and its semantics are the SDK's, codellm-devkit/.github#103). + +**Added by the 2026-10-25 re-check (A23):** a Go 1.23+ module (range-over-func — pins the D8 gap — +and range over int; `testdata/dataflow` declares `go 1.21`, #28); a method value (A8); a directly +called literal with captures (A6); a deferred closure that writes a named result (A7); a value +struct holding a slice (D1 gap 2); a forced L3-skipped callable (A16); an interface call whose +concrete target is a library type (A19); one fixture per SSA wrapper kind (§4.8 unwrap list). **Analyzer gates** (each level gated before advancing — `codeanalyzer-backend` HARD-GATE): CFG gate, dominance gate (exact CDG), PDG-slice gate (exact set), DFG gates, L4 gates (no dangling; @@ -703,35 +1145,71 @@ determinism (`-j 1` vs `-j N`, two runs), monotonicity (`-a 1..4` superset via a Python's `test_v2_superset.py`), cross-projection (Neo4j vs JSON), Neo4j conformance, SDK model validation of every level's output. +**Metamorphic relations** (`internal/metamorphic`; Task 17b, amended 2026-10-27). A relation +transforms a copy of a module, analyzes it in-process at `-a 4`, and compares the result with +the original's through a position map. Callables are keyed by signature (plus the declaring file +for the per-file `init`), not by module id. There are three oracles: **exact** (the normalized +views are equal), **delta** (the view changes by exactly the edges each site declares, nothing +else changes, and the expected CDG is a reference post-dominator CDG of the expected CFG), and +**reach** (one-way: every `formal_in` → `actual_in`/`formal_out` flow of a sample of callables +survives over ddg ∪ cdg ∪ summary ∪ param_in ∪ param_out). MR1 adds blank lines and comments +(exact). MR2 reverses the top-level declarations (exact). MR3 moves a function and a method to +a new file (exact). MR4 alpha-renames locals (exact modulo spelling). MR5 introduces an alias +(reach). MR6 extracts a helper (reach). MR7 runs `--target-files` at `-a 3` against a full run +(exact on that file). MR8 wraps a statement in `if c {}`. MR9 inserts a kill. MR10 inserts a +copy. MR11 swaps two independent statements. MR12 adds an early `if c { return }`. MR8–MR12 +use the delta oracle, on local scalars in callables with an L3 result and no defer or goto. +MR13 replaces a sink argument with a constant: it is a precision relation, so the flow into +that `actual_in` must be gone and nothing else may be added. MR14 routes a call through a +one-implementation interface, a closure or `defer` (reach, plus the summary edges for the +interface form). MR15 extracts a helper and inlines it back (exact, byte-identical source). +Loops, `select`/`switch`/`fallthrough`, panic edges and the defer chain are out of scope for +the delta relations. The fixtures (`testdata/dataflow` flow and sdg, chi) run in `go test`; +the eval apps run behind the `metamorphic` build tag (`scripts/metamorphic.sh`). + **SDK gates:** slice gate (exact set on a `can://…@line:col` criterion); taint gate (known -source→sink found; sanitizer variant reported cut, not dropped; contiguous witness); a pair with no +source→sink found; sanitizer variant — pending codellm-devkit/.github#103, see "Done"; contiguous witness); a pair with no flow lands in `exhausted`; context-sensitivity only as far as summaries support; JSON and Neo4j backends agree; e2e level matrix. -**"Done"** (user-confirmed intent, designer-worded): through `CLDK.go(...)`, on the HTTP→SQL -fixture, `taint()` finds the flow from a handler's request parameter to `database/sql.DB.Query`'s -`query` through an interface call; a callable sanitizer cuts it; an unrelated pair is `exhausted`; -identical answers from the JSON and Neo4j backends. - -**Real-world evaluation** (non-gating metrics + crash/determinism/time gates): the prior eval set — -cobra (small), dae, openbao, cockroach (scale) — plus listmonk and memos (HTTP → SQL; memos through -the `store.Driver` interface) and nats-server (concurrency / CFG stress) [user, 2026-10-23]. dae and -cockroach need their code-generation step (bpf2go, protobuf) before analysis. Record per app: wall time and peak RSS at `-a 3`/`-a 4`, body/edge counts, % call -sites with resolved targets (go/types vs VTA), % externals with stand-ins, determinism. Gate: all -four complete `-a 4` without crash and deterministically; no hard time budget, but timings are -reported and a regression budget is set after the first measurement. +**"Done"** (user-confirmed intent, designer-worded; amended [user, 2026-10-27]): through +`CLDK.go(...)`, on the HTTP→SQL fixture, `taint()` finds the flow from a handler's request +parameter to `database/sql.DB.Query`'s `query` through an interface call; an unrelated pair is +`exhausted`; identical answers from the JSON and Neo4j backends. Sanitizer handling (callable or +variable cuts) is SDK taint semantics, tracked in codellm-devkit/.github#103: L4's output is +correct reaching-definitions data flow, and neither a library model nor a variable cut on the +sanitized value severs the sanitized path (§4.9). The analyzer side is gated by +`TestGateL4_HTTPToSQLPath` and `TestGateL4_HTTPToSQLSanitizerIsSDKSemantics`. + +**Real-world evaluation** (non-gating metrics + crash/determinism/time gates; amended, A23): the six +apps — cobra (small), dae, openbao, listmonk and memos (HTTP → SQL; memos through the +`store.Driver` interface), nats-server (concurrency / CFG stress) — plus **Kubernetes as the scale +benchmark**, run as two modules: the root module `kubernetes` and `k8s-api` +(`docs/design/notes/l3-dataflow-metrics.md` § Corpus). cockroach is no longer a benchmark. dae +needs its code-generation step (bpf2go) before analysis (`scripts/eval_l3.sh --prepare`). Record per +app: wall time and peak RSS at `-a 3`/`-a 4`, body/edge counts, % call sites with resolved targets +(go/types vs VTA), % externals with stand-ins, `:` formal volume (A5), determinism. Gate: +the six apps and both Kubernetes modules complete `-a 4` without crash and deterministically +(`-j 1`/`-j N` and two runs on Kubernetes, not just fixtures); memory is reported per app as +absolute `-a 4` peak RSS and its ratio to `-a 3`, not gated (D3, §4.13); no hard time budget, but timings are reported and a regression budget is set after +the first measurement. Results go in `l3-dataflow-metrics.md` § L4. +**Measured 2026-10-28 (go1.27.3, `98ebfef`, the final head):** the gate passed on all eight. +Every app completes `-a 4` and its `-j 1` / `-j 16` / rerun outputs are byte-identical. The +Kubernetes root takes 38.4 s at a 7.9 GB peak (0.99–1.13× a noisy `-a 3` peak). nats-server is +the outlier at 3.8× RSS and 1.78 GB of JSON. (The first measurement, 2026-10-27, before the +final-review fixes: 31.5 s at 7.7 GB.) The full results, insights and limitations are in `l3-dataflow-metrics.md` § L4. ## 9. Phasing and release | Phase | Repo | Content | Release | -> **Phases 0 and 1 ship together in one PR / one release** [user, 2026-10-23]; they stay separate rows for traceability. +> **Phases 0 and 1 ship together in one PR / one release** [user, 2026-10-23]; they stay separate rows for traceability. **Both are implemented and merged (PR #16, `f9a39bb`); Phase 2 (L4) is live on `feat/l4-sdg`.** |---|---|---|---| -| 0 | analyzer | id grammar `can:///go/…`; **L2 external calls (§4.14)**; L1 co-positioned call keys; `is_deferred`; `typeKind` alias/defined; `GO_DDG` `_k`; call-site position index (perf) | analyzer next **breaking** version (wheel is at 0.1.0 with no local tags — builder confirms the released version and bumps the major, or the minor if still 0.x) | -| 1 | analyzer | L3: body statements, CFG (+defer chain, panic), CDG, syntactic DDG, CLI `-a 3`/`--graph-field-depth`, Neo4j CFG/CDG/DDG, gates | minor | -| 2 | analyzer | L4: SSA+VTA, `degraded` marker (§4.15), external stand-ins (`external_symbols`), vertices, param edges, summaries, port wiring, type-based alias delta, Neo4j L4 overlay, `--emit neo4j` at L4, real-app eval | minor | -| 3 | python-sdk | Go v2 models, facade, local + Neo4j backends, L1–L4 accessors, taint e2e | minor; pins the analyzer only after Phase 2 is released | +| 0 | analyzer | **merged (PR #16, `f9a39bb`).** id grammar `can:///go/…`; **L2 external calls (§4.14)**; L1 co-positioned call keys; `is_deferred`; `typeKind` alias/defined; `GO_DDG` `_k`; call-site position index (perf) | analyzer next **breaking** version (wheel is at 0.1.0 with no local tags — builder confirms the released version and bumps the major, or the minor if still 0.x) | +| 1 | analyzer | **merged (PR #16, `f9a39bb`).** L3: body statements, CFG (+defer chain, panic), CDG, syntactic DDG, CLI `-a 3`/`--graph-field-depth`, Neo4j CFG/CDG/DDG, gates | minor | +| 2 | analyzer | **Prerequisites, in order (A24):** (1) #26 shared call/closure walker (incl. `//line` spans), first — returns `(*ast.CallExpr, key)`, `(*ast.FuncLit, closure name)` and call → host maps, replacing the index-paired L1/L3 walks; (2) #17 closure call resolution, after #26 — closures carry a declaring `Path` and `call_graph` gains edges from closures; (3) the #22 L3 fixes (D1, §4.5), then the #28 satellite-`parent` tests and a Go 1.23+ fixture — before L4 port wiring (changing L3 after L4 would re-baseline every L4 gate); (4) #27 streaming JSON and AST release — before the real-app L4 eval, can run in parallel with the L4 core; (5) schema and CLI scaffolding (A14, A18, A19, A17 declarations, a `-a 1..4` monotonicity harness) — the first L4 task. Dependencies: #26 → #17 → (VTA, summaries); #22 fixes → port wiring and gates; scaffolding → all of L4; #27 → the Kubernetes eval. **Then L4:** SSA spike (D3 measurement) and build (D2), SSA → L3 join, VTA, `degraded` marker (§4.15), external stand-ins (`external_symbols` `receiver`/`body`), vertices, param edges, summaries, port wiring, type-based alias delta, Neo4j L4 overlay, `--emit neo4j` at L4 (per D4), real-app eval | minor | +| 3 | python-sdk | Go v2 models (first the §7 prerequisites: `GoType.kind`, L3 fields), facade, local + Neo4j backends, L1–L4 accessors, taint e2e | minor; pins the analyzer only after Phase 2 is released | Tracking: `codellm-devkit/.github` parent issue + one sub-issue per PR (Phase 0+1 as one) (filed just-in-time), per `issue-and-pr-tracking.md`. Each phase gets its own implementation plan. @@ -754,16 +1232,34 @@ dispatch + type-based may-alias). multi-node selectors in `commons`); promote `external_symbols.body` stand-ins into the keystone if adopted. - Andersen points-to over `go/ssa` replacing the type-based oracle (`prov` unchanged). -- Context-sensitive (HRB two-phase) taint traversal in the SDK. +- Context-sensitive (HRB two-phase) taint traversal in the SDK. With it, reinstate the global → + global `summary` edges omitted at L4 (§4.9, [user, 2026-10-26]; codeanalyzer-go#30 — also + the depth-bounded-slice caveat there). +- A stand-in for the universe `error.Error` (and interface literals), so an interface call on + them gets a D7 target (§4.8, Task 10 ruling). +- Stream per callable within a module during emission, if the emission-held module peak + (§4.13) matters on a larger app. +- `GO_EMBEDS` / `GO_SATISFIES` from `base_types[]` / `interfaces[]`: `base_types` holds raw type + strings, not type ids, and `interfaces[]` is never computed (codeanalyzer-go#31; pre-existing, + not an L4 change). - Library callback modelling (`sort.Slice`, `http.HandleFunc`). +- Sanitizer semantics in the SDK taint analysis (callable and variable cuts that account for the + host statement, implicit flow and coarse host wiring): codellm-devkit/.github#103. - Promote `callable.source_file` to the keystone (existing deferral). - Struct-tag modelling (existing deferral). +- **Incremental L4 / incremental Neo4j analysis — CLDK roadmap** (D9): re-analysis after a PR merges + without a whole-program run. +- Range-over-func modelling (D8): `for … range f` as a call site of `f`, wiring data reaching the + iterator's `yield` arguments to the loop variables. +- App-wide L4 purge-and-rewrite on a lazy Bolt push (D5 option (c)). +- Loading dependencies from export data instead of source (drop `NeedDeps` syntax) — a large RSS + cut at every level, but it changes the L1/L2 load path (D3). ## 12. Risks | Risk | Mitigation | |---|---| -| SSA/VTA cost on cockroach-scale | SSA only at `-a 4`; measure in Phase 2 eval; `--strict` / degrade path | +| SSA/VTA cost at Kubernetes scale | SSA only at `-a 4`, project packages only (D2); memory measured and reported per app (D3); #27 streaming; `--strict` / degrade path | | False refutations from a missed construct | over-approximation posture; "ports carry dependence" probe; per-construct fixtures; unresolved-call intra-caller pass-through | | Over-approximation noise (defer chain, pass-through, CDG in summaries) | acceptable by anti-goal; library models file narrows hot spots | | Breaking id change disrupts users | ship alone in Phase 0, before SDK pin; no released SDK consumes Go output | diff --git a/docs/design/specs/v2-schema-structure.md b/docs/design/specs/v2-schema-structure.md index aa7b45a..8d50f2d 100644 --- a/docs/design/specs/v2-schema-structure.md +++ b/docs/design/specs/v2-schema-structure.md @@ -12,7 +12,13 @@ the JSON tags from that file. ├─ schema_version # "2.0.0" ├─ language # "go" ├─ max_level # 1 (symbol table) | 2 (+ call graph) | 3 (+ cfg/cdg/ddg) +│ # | 4 (+ SDG: vertices, param edges, summaries) ├─ k_limit # L3+ only: access-path bound (--graph-field-depth, default 3) +├─ degraded[] # L4; optional: absent when nothing degraded +│ └─ { code, detail?, packages[]?, callables[]? } +│ # code: no_ssa | no_vta | no_cha | partial_ssa | partial_types | +│ # unmapped_ssa | l3_skipped | range_func (open vocabulary); +│ # callables[] = can:// ids (l3_skipped, range_func) ├─ analyzer │ ├─ name # "codeanalyzer-go" │ └─ version @@ -45,8 +51,13 @@ the JSON tags from that file. │ └─ │ ├─ src # can:// callable id │ ├─ dst # can:// callable id, or an @external id - │ ├─ prov[] # optional: e.g. ["go/types"] + │ ├─ prov[] # optional: e.g. ["go/types"]; L4 adds VTA-only edges, prov + │ │ # ["vta"], weight 1 (an L2 edge is never changed) │ └─ weight # optional + ├─ param_in[] # L4; optional: { src, dst, var } — global ids, + │ # actual_in → formal_in; var = the callee formal's `of` + ├─ param_out[] # L4; optional: { src, dst, var } — global ids, + │ # formal_out → actual_out; sorted (src, dst) └─ external_symbols # L2+; optional: { "<@external id>": external } └─ # library callee: `can:///@external//` ├─ id @@ -57,7 +68,11 @@ the JSON tags from that file. │ # library interface method (e.g. "Writer.Write") ├─ parameters[] # { name, type, is_variadic? } (no span: not in-project); │ # receiver excluded; unnamed params are "_" - └─ return_type # optional + ├─ return_type # optional + ├─ receiver # L4; optional: { name, type } of a method's + │ # receiver (gives formal_in:0's `of`; "_recv" if unnamed) + └─ body # L4; optional: stand-in formal vertices only, + # { "@formal_in:" | "@formal_out[:]": vertex } ``` ## `` node (recursive) @@ -83,6 +98,8 @@ The same shape whether it sits under a type's `callables` (methods), a module's ├─ metrics # optional: { cyclomatic: N } ├─ callables # optional: nested closures (same shape) ├─ body # { "": body node }; key = "[/n]" | "@entry" | "@exit" +│ # | L4: "@formal_in:" | "@formal_out[:]" +│ # | "/actual_in:" | "/actual_out[:]" │ ├─ # L1+ │ │ ├─ kind # "call" │ │ ├─ id # L1+: @ @@ -91,13 +108,25 @@ The same shape whether it sits under a type's `callables` (methods), a module's │ │ ├─ callee # null | can:// id (key always present) │ │ ├─ is_goroutine # optional: present only when true │ │ └─ is_deferred # optional: present only when true -│ └─ # L3+ -│ ├─ kind # statement | branch | loop | switch | return | entry | exit -│ ├─ id # @ (@entry → @entry) -│ └─ span { start, end, bytes } # the statement; absent on entry / exit +│ ├─ # L3+ +│ │ ├─ kind # statement | branch | loop | switch | return | entry | exit +│ │ ├─ id # @ (@entry → @entry) +│ │ └─ span { start, end, bytes } # the statement; absent on entry / exit +│ └─ # L4: SDG port, span-less +│ ├─ kind # formal_in | formal_out | actual_in | actual_out +│ ├─ id # @ +│ ├─ of # L4: the bound name — param / receiver name, "_", +│ │ # "_recv", "", ":", +│ │ # ":", ":::" +│ └─ parent # actual_* only: the call key; for a non-call closure +│ # capture site, the key of the node containing the literal ├─ cfg[] # L3+, optional: { src, dst, kind } — local body keys ├─ cdg[] # L3+, optional: { src, dst } -└─ ddg[] # L3+, optional: { src, dst, var, prov } (prov = ["ssa"]) +├─ ddg[] # L3+, optional: { src, dst, var, prov } (prov = ["ssa"]; +│ # L4 adds ["reaching-defs"] port wiring, ["points-to"]) +└─ summary[] # L4; optional: { src, dst } — local keys, + # /actual_in: → /actual_out[:]; + # no global → global edge (codeanalyzer-go#30) ``` ### L3 body nodes and edges @@ -123,11 +152,17 @@ The same shape whether it sits under a type's `callables` (methods), a module's - **`ddg`** comes from syntactic reaching definitions over k-limited access paths (`base(.field|[*])*`, past `k` → `.*`). `var` is the source spelling of the path, and `prov` is always `["ssa"]` at L3. Sorted by `(src, dst, var)`. It is **not - refutation-complete**: writes to captured variables inside closures and writes through - reference-typed fields of value-typed struct arguments/receivers are not modelled at L3 - (no edge ≠ no flow); L4 closes them. A deferred call's receiver/argument paths are keyed on - the variable at exit, so a reassignment after the `defer` can add a spurious edge or kill an - earlier def. See `v2-l3-l4-dataflow.md` §4.5 "L3 precision". + refutation-complete** (no edge ≠ no flow): it is alias-free (two bases never interact; L4's + `points-to` edges add that). The D1 gaps of the 2026-10-25 re-check — closure capture writes, + writes through reference fields of value structs, and deferred-operand writes — are closed at + L3 (#22). A deferred call's receiver/argument paths are keyed on the variable at exit, so a + reassignment after the `defer` can add a spurious edge. See `v2-l3-l4-dataflow.md` §4.5 + "L3 precision". +- **L4 vertex keys (L4).** Formals `@formal_in:`, `@formal_out` / `@formal_out:`; + actuals `/actual_in:`, `/actual_out` / `…/actual_out:`. `formal_in` and + `actual_in` are always indexed; `formal_out`/`actual_out` are bare iff the group has exactly one + vertex. The SDK reads `of`, never an index into `parameters[]` (`formal_in:0` is a method's + receiver). See `v2-l3-l4-dataflow.md` §4.6–§4.7. - **Spans.** A statement node's span is its AST node's range, and a clause head ends at its colon. For a callable with `source_file` (a method declared apart from its receiver type, or a closure inside one), body spans index `symbol_table[source_file].source`, as the @@ -143,6 +178,10 @@ span └─ bytes [from, to] # UTF-8 byte offsets into the owning module.source ``` +`start` / `end` are real file positions (`token.FileSet.PositionFor(p, false)`): `//line` +directives are never applied, so in generated code that carries them (goyacc, templ) every +span, and every body key derived from one, still indexes `module.source`. + ## Presence rules - **Always present:** every `id` (body nodes included, from L1 on — parity with @@ -154,6 +193,11 @@ span - **L3+ only:** `k_limit`; body node `parent`; non-call body nodes; a callable's `cfg` / `cdg` / `ddg`. These are omitted for a callable with no body. Each is a pure addition: the `-a 2` document is a subset of the `-a 3` one. +- **L4 only:** `degraded` (and only when something degraded), `formal_*` / `actual_*` + vertices and their `of`, a callable's `summary`, `param_in` / `param_out`, + `external_symbols[].receiver` / `.body`, `ddg` edges with `prov` `reaching-defs` / `points-to`, + `call_graph` `prov` `vta`. Bodiless interface methods and stand-ins get formals too. Each is a + pure addition: `-a 3` ⊆ `-a 4`. - **`omitempty` — emitted only when non-empty:** `content_hash`, `imports[].alias`, `base_types`, `interfaces`, `fields`, `source_file`, `return_type`, `error_channel`, `metrics`, a callable's nested `callables`, `parameters[].is_variadic`, edge `prov` / diff --git a/internal/astwalk/walk.go b/internal/astwalk/walk.go new file mode 100644 index 0000000..7a23c5e --- /dev/null +++ b/internal/astwalk/walk.go @@ -0,0 +1,73 @@ +// Package astwalk is the one call-site / closure walker of a callable body, +// shared by L1 (the symbol-table builder records call sites and names +// closures with it) and L3/L4 (the dataflow body builder pairs its call nodes +// with those sites). One walk, so the two levels can never drift apart. It +// imports only the standard library, so any package may use it. +package astwalk + +import ( + "go/ast" + "strconv" +) + +// Call is one call expression of a body, with the go / defer statement flag +// of the statement whose own call it is. +type Call struct { + Expr *ast.CallExpr + Goroutine bool // the call of a `go` statement + Deferred bool // the call of a `defer` statement +} + +// Calls lists body's call expressions in L1 recording order: pre-order — an +// outer call before the calls nested in it, a `go` / `defer` statement's call +// before the calls in its function value and arguments — and no function +// literal bodies (each literal is its own callable; see Literals). A nil body +// has no calls. +func Calls(body *ast.BlockStmt) []Call { + if body == nil { + return nil + } + var out []Call + goroutine := map[*ast.CallExpr]bool{} + deferred := map[*ast.CallExpr]bool{} + ast.Inspect(body, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.GoStmt: + goroutine[n.Call] = true + case *ast.DeferStmt: + deferred[n.Call] = true + case *ast.FuncLit: + return false + case *ast.CallExpr: + out = append(out, Call{Expr: n, Goroutine: goroutine[n], Deferred: deferred[n]}) + } + return true + }) + return out +} + +// Literal is one outermost function literal of a body and its closure name. +type Literal struct { + Lit *ast.FuncLit + Name string // closure_1, closure_2, … +} + +// Literals lists the outermost function literals of body in ast.Inspect +// order, named closure_1, closure_2, …. A literal nested in another is not +// listed: it is the enclosing literal's own (Literals(lit.Body)). A nil body +// has none. +func Literals(body *ast.BlockStmt) []Literal { + if body == nil { + return nil + } + var out []Literal + ast.Inspect(body, func(n ast.Node) bool { + lit, ok := n.(*ast.FuncLit) + if !ok { + return true + } + out = append(out, Literal{Lit: lit, Name: "closure_" + strconv.Itoa(len(out)+1)}) + return false + }) + return out +} diff --git a/internal/astwalk/walk_test.go b/internal/astwalk/walk_test.go new file mode 100644 index 0000000..5ca854a --- /dev/null +++ b/internal/astwalk/walk_test.go @@ -0,0 +1,127 @@ +package astwalk_test + +import ( + "go/ast" + "go/parser" + "go/token" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" +) + +// parseFlow parses testdata/dataflow/flow/flow.go and returns its functions by +// name. +func parseFlow(t *testing.T) (*token.FileSet, map[string]*ast.FuncDecl) { + t.Helper() + _, thisFile, _, _ := runtime.Caller(0) + path := filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "dataflow", "flow", "flow.go") + fset := token.NewFileSet() + f, err := parser.ParseFile(fset, path, nil, parser.SkipObjectResolution) + if err != nil { + t.Fatal(err) + } + decls := map[string]*ast.FuncDecl{} + for _, d := range f.Decls { + if fd, ok := d.(*ast.FuncDecl); ok && fd.Recv == nil { + decls[fd.Name.Name] = fd + } + } + return fset, decls +} + +// describe renders a call as its source text plus its go/defer flags. +func describe(fset *token.FileSet, src []byte, c astwalk.Call) string { + tf := fset.File(c.Expr.Pos()) + s := string(src[tf.Offset(c.Expr.Pos()):tf.Offset(c.Expr.End())]) + if c.Goroutine { + s += " [go]" + } + if c.Deferred { + s += " [defer]" + } + return s +} + +// TestWalk_CallsOrderMatchesL1: Calls lists a body's calls in L1 recording +// order — pre-order (an outer call before the calls nested in it, a go/defer +// statement's call before the calls in its operands), no function literal +// bodies — with the go/defer flag on the statement's own call only. +func TestWalk_CallsOrderMatchesL1(t *testing.T) { + fset, decls := parseFlow(t) + _, thisFile, _, _ := runtime.Caller(0) + src := mustRead(t, filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "dataflow", "flow", "flow.go")) + for fn, want := range map[string][]string{ + "GoArgs": {"f(g(x)) [go]", "g(x)", "func(y int) { _ = y }(h()) [go]", "h()"}, + "Chain": {"s.A().B()", "s.A()"}, + "Nested": {"f(g(y))", "g(y)"}, + "EarlyReturnDefer": {"cleanup() [defer]", "work(g(a)) [defer]", "g(a)", "cleanup()"}, + "Closure": {"h()"}, + } { + d, ok := decls[fn] + if !ok { + t.Fatalf("flow.go has no %s", fn) + } + var got []string + for _, c := range astwalk.Calls(d.Body) { + got = append(got, describe(fset, src, c)) + } + if strings.Join(got, " | ") != strings.Join(want, " | ") { + t.Errorf("%s: Calls = %q, want %q", fn, got, want) + } + } + if got := astwalk.Calls(nil); len(got) != 0 { + t.Errorf("Calls(nil) = %v, want none", got) + } +} + +const literalsSrc = `package p + +func outer(g func(func())) { + a := func() { + _ = func() {} // nested: a's own literal, not outer's + } + g(func() {}) + defer func() {}() + _ = a +} +` + +// TestWalk_LiteralsNaming: Literals returns only the outermost function +// literals of a body, in ast.Inspect order, named closure_1, closure_2, …; +// a literal nested in another is the inner literal's own closure_1. +func TestWalk_LiteralsNaming(t *testing.T) { + fset := token.NewFileSet() + f, err := parser.ParseFile(fset, "p.go", literalsSrc, 0) + if err != nil { + t.Fatal(err) + } + body := f.Decls[0].(*ast.FuncDecl).Body + lits := astwalk.Literals(body) + var got []string + for _, l := range lits { + got = append(got, l.Name+"@"+strings.TrimPrefix(fset.Position(l.Lit.Pos()).String(), "p.go:")) + } + if want := "closure_1@4:7,closure_2@7:4,closure_3@8:8"; strings.Join(got, ",") != want { + t.Errorf("Literals = %v, want %s", got, want) + } + inner := astwalk.Literals(lits[0].Lit.Body) + if len(inner) != 1 || inner[0].Name != "closure_1" || fset.Position(inner[0].Lit.Pos()).Line != 5 { + t.Errorf("nested Literals = %+v, want one closure_1 on line 5", inner) + } + if got := astwalk.Literals(nil); len(got) != 0 { + t.Errorf("Literals(nil) = %v, want none", got) + } +} + +func mustRead(t *testing.T, path string) []byte { + t.Helper() + data, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + return data +} diff --git a/internal/core/analyzer.go b/internal/core/analyzer.go index 2329528..2d541bf 100644 --- a/internal/core/analyzer.go +++ b/internal/core/analyzer.go @@ -10,19 +10,28 @@ // 3. Resolver-based call graph (semantic_analysis) — if level >= 2 // 4. Pass pipeline (analysis/registry) // 5. Intraprocedural dataflow (dataflow) — if level >= 3 +// 6. System dependence graph (sdg) — if level >= 4 package core import ( + "bufio" "encoding/json" + "errors" "fmt" + "io" + "io/fs" + "math/rand/v2" "os" "os/exec" "path/filepath" + "strconv" + "strings" "github.com/codellm-devkit/codeanalyzer-go/internal/analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/options" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" @@ -36,6 +45,9 @@ type Analyzer struct { // schema.GoApplication: schema must not depend on dataflow, and the // cached v1 model has no L3 facts. l3 map[dataflow.L3Key]dataflow.Result + // l4 is the Phase 6 result (nil below L4). Pass it to RenderJSON / + // WriteOutput / v2emit.Emit with L3(). + l4 *sdg.Result } // L3 returns the dataflow side table of the last Analyze run: one result per @@ -43,7 +55,10 @@ type Analyzer struct { // level 3. Pass it to RenderJSON / WriteOutput / v2emit.Emit. func (a *Analyzer) L3() map[dataflow.L3Key]dataflow.Result { return a.l3 } -// KLimit is the access-path bound used at L3: opts.FieldDepth, or the +// L4 returns the SDG result of the last Analyze run. Nil below level 4. +func (a *Analyzer) L4() *sdg.Result { return a.l4 } + +// KLimit is the access-path bound used at L3 and L4: opts.FieldDepth, or the // --graph-field-depth default of 3 when a non-CLI caller left it 0. func KLimit(opts options.AnalysisOptions) int { if opts.FieldDepth > 0 { @@ -67,6 +82,13 @@ func (a *Analyzer) Analyze() (*schema.GoApplication, error) { a.opts.InputPath = abs utils.Info("analyzing project: %s", a.opts.InputPath) + // The L4 library models are read first, so a bad file fails the run + // before any analysis (also under --emit neo4j -t, which runs at L3). + models, err := sdg.LoadModels(a.opts.LibraryModels) + if err != nil { + return nil, err + } + // ── Phase 1: Project materialization ────────────────────────────────────── if err := a.materialize(); err != nil { // Degrade gracefully — log but don't abort. Partial types are better than nothing. @@ -111,14 +133,81 @@ func (a *Analyzer) Analyze() (*schema.GoApplication, error) { } // ── Phase 5: Intraprocedural dataflow (L3) ──────────────────────────────── + var units []dataflow.Unit if a.opts.Level >= options.LevelDataflow { - a.l3 = dataflow.Run(builder.Fset(), builder.L3Units(symbolTable), KLimit(a.opts), a.opts.Jobs) + units = builder.L3Units(symbolTable) + if a.opts.Level >= options.LevelSDG { + // L4 reads each result's def/use sets and call keys (the SSA + // join); Release drops them after sdg.Run. + a.l3 = dataflow.RunWithSideTables(builder.Fset(), units, KLimit(a.opts), a.opts.Jobs) + } else { + a.l3 = dataflow.Run(builder.Fset(), units, KLimit(a.opts), a.opts.Jobs) + } utils.Info("dataflow: %d callables", len(a.l3)) } + // ── Phase 6: System dependence graph (L4) ───────────────────────────────── + if a.opts.Level >= options.LevelSDG { + res := sdg.Run(sdg.Input{ + Fset: builder.Fset(), + Pkgs: builder.Pkgs(), + Variants: builder.Loaded(), + ProjectDir: a.opts.InputPath, + SymbolTable: symbolTable, + Externals: app.Externals, + CallGraph: app.CallGraph, + Units: units, + L3: a.l3, + K: KLimit(a.opts), + Jobs: a.opts.Jobs, + Models: models, + }) + a.l4 = &res + if err := a.reportDegraded(res.Degraded); err != nil { + return nil, err + } + // VTA's dispatch edges and the externals only VTA reached (spec §4.8): + // additions to the L2 call graph, so -a 3 ⊆ -a 4. + res.ApplyTo(app) + } + + // The last reader of Node.Stmt and of the L4 side tables is done + // (dataflow.Run, or sdg.Run at L4, which joins SSA to them and binds + // call-site actuals from the AST; the emitters read keys, spans and + // edges only), so drop them now: the parsed packages are then + // unreachable once Analyze returns, before the cache write and the + // emission. (#27) + dataflow.ReleaseAll(a.l3) + return a.finalizeAndCache(app) } +// reportDegraded logs one warning per L4 degradation, or, under --strict, +// fails with an error naming every degraded code. +func (a *Analyzer) reportDegraded(deg []sdg.Degraded) error { + if len(deg) == 0 { + return nil + } + if a.opts.Strict { + codes := make([]string, len(deg)) + for i, d := range deg { + codes[i] = d.Code + } + return fmt.Errorf("--strict: analysis level 4 degraded: %s", strings.Join(codes, ", ")) + } + for _, d := range deg { + msg := "L4 degraded: " + d.Code + if d.Detail != "" { + msg += " (" + d.Detail + ")" + } + if len(d.Packages) > 0 { + msg += ": " + strings.Join(d.Packages, ", ") + } + utils.Warn("%s", msg) + } + return nil +} + // materialize runs `go mod download` to ensure the module graph is available // for go/packages to resolve imports and types. Idempotent and cached. func (a *Analyzer) materialize() error { @@ -163,7 +252,9 @@ func (a *Analyzer) finalizeAndCache(app *schema.GoApplication) (*schema.GoApplic return app, nil } -// saveCache persists the application to cache as analysis_cache.json. +// saveCache persists the application to cache as analysis_cache.json, in the +// internal v1 shape. Nothing reads it back: it is a debugging aid, not an +// input, so a v1 shape change (e.g. closures gaining path) affects nothing. func (a *Analyzer) saveCache(app *schema.GoApplication) error { if err := utils.EnsureDir(a.opts.CacheDir); err != nil { return err @@ -176,44 +267,93 @@ func (a *Analyzer) saveCache(app *schema.GoApplication) error { return os.WriteFile(cachePath, data, 0o644) } -// RenderJSON serializes the analysis to JSON as the canonical v2 containment -// tree via the v2emit emitter. This is the single serialization point, so both -// the stdout and file output paths agree. Only "json" is supported; other -// formats return an explicit error rather than silently falling back. -// -// l3 is the Analyzer's L3() side table (nil below level 3). At level 3 the -// envelope carries k_limit = KLimit(opts) whether or not l3 is set. -func RenderJSON(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, opts options.AnalysisOptions) ([]byte, error) { - format := opts.Format - if format == "" { - format = "json" - } - switch format { - case "json": - // only supported format +// checkFormat rejects every output format but json (the default when empty) +// with an explicit error rather than silently falling back. +func checkFormat(opts options.AnalysisOptions) error { + switch opts.Format { + case "", "json": + return nil case "msgpack": - return nil, fmt.Errorf("msgpack output is not yet implemented; use --format json") + return fmt.Errorf("msgpack output is not yet implemented; use --format json") default: - return nil, fmt.Errorf("unsupported output format %q; supported: json", format) + return fmt.Errorf("unsupported output format %q; supported: json", opts.Format) } +} - payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, KLimit(opts), l3) +// RenderJSON serializes the analysis to JSON as the canonical v2 containment +// tree via the v2emit emitter, as one []byte. It is the reference the +// streaming StreamJSON / WriteOutput paths are byte-identical to; the CLI +// streams instead, so the document never exists in memory at once. +// +// l3 is the Analyzer's L3() side table (nil below level 3) and l4 its L4() +// result (nil below level 4). At level 3 the envelope carries k_limit = +// KLimit(opts) whether or not l3 is set. +func RenderJSON(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, l4 *sdg.Result, opts options.AnalysisOptions) ([]byte, error) { + if err := checkFormat(opts); err != nil { + return nil, err + } + payload := v2emit.Emit(app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, KLimit(opts), l3, l4) return json.Marshal(payload) } -// WriteOutput serializes the analysis (in the selected schema) and writes it to -// outputDir/analysis.json, or to stdout when outputDir is empty. -func WriteOutput(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, opts options.AnalysisOptions) error { - data, err := RenderJSON(app, l3, opts) - if err != nil { +// StreamJSON writes the bytes RenderJSON returns to w, emitting and encoding +// one module at a time (v2emit.EmitTo) through a buffered writer. +func StreamJSON(w io.Writer, app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, l4 *sdg.Result, opts options.AnalysisOptions) error { + if err := checkFormat(opts); err != nil { return err } - if opts.OutputDir == "" { - _, err = os.Stdout.Write(data) + bw := bufio.NewWriterSize(w, 1<<20) + if err := v2emit.EmitTo(bw, app, opts.AppName, opts.InputPath, int(opts.Level), opts.AnalyzerVersion, KLimit(opts), l3, l4); err != nil { return err } + return bw.Flush() +} + +// WriteOutput streams the analysis to outputDir/analysis.json, or to stdout +// when outputDir is empty. The file is written beside its destination and +// renamed into place, so a failed run never leaves a truncated analysis.json. +func WriteOutput(app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result, l4 *sdg.Result, opts options.AnalysisOptions) error { + if err := checkFormat(opts); err != nil { + return err + } + if opts.OutputDir == "" { + return StreamJSON(os.Stdout, app, l3, l4, opts) + } if err := utils.EnsureDir(opts.OutputDir); err != nil { return err } - return os.WriteFile(filepath.Join(opts.OutputDir, "analysis.json"), data, 0o644) + dst := filepath.Join(opts.OutputDir, "analysis.json") + f, tmp, err := createTemp(opts.OutputDir, ".analysis.json.") + if err != nil { + return err + } + err = StreamJSON(f, app, l3, l4, opts) + if err == nil { + err = f.Sync() // the rename below must not publish unflushed bytes + } + if cerr := f.Close(); err == nil { + err = cerr + } + if err == nil { + err = os.Rename(tmp, dst) + } + if err != nil { + _ = os.Remove(tmp) + } + return err +} + +// createTemp creates a new file prefix+ in dir with mode 0666, so the +// kernel applies the process umask as os.Create would (os.CreateTemp uses +// 0600). O_EXCL makes a name collision an error, and the name is retried. +func createTemp(dir, prefix string) (*os.File, string, error) { + for range 10000 { + name := filepath.Join(dir, prefix+strconv.FormatUint(uint64(rand.Uint32()), 36)) + f, err := os.OpenFile(name, os.O_RDWR|os.O_CREATE|os.O_EXCL, 0o666) + if errors.Is(err, fs.ErrExist) { + continue + } + return f, name, err + } + return nil, "", fmt.Errorf("creating a temporary file in %s: too many name collisions", dir) } diff --git a/internal/core/analyzer_test.go b/internal/core/analyzer_test.go index 3cc70a5..a7198dd 100644 --- a/internal/core/analyzer_test.go +++ b/internal/core/analyzer_test.go @@ -1,7 +1,9 @@ package core_test import ( + "bytes" "encoding/json" + "fmt" "os" "path/filepath" "testing" @@ -139,7 +141,7 @@ func TestCallGraph_CallSitesBackfilled(t *testing.T) { func TestWriteOutput_ValidJSON(t *testing.T) { outDir := t.TempDir() - if err := core.WriteOutput(sharedGreeterL2, nil, options.AnalysisOptions{OutputDir: outDir, Format: "json"}); err != nil { + if err := core.WriteOutput(sharedGreeterL2, nil, nil, options.AnalysisOptions{OutputDir: outDir, Format: "json"}); err != nil { t.Fatalf("WriteOutput: %v", err) } data, err := os.ReadFile(filepath.Join(outDir, "analysis.json")) @@ -161,7 +163,7 @@ func TestWriteOutput_ValidJSON(t *testing.T) { func TestWriteOutput_EmptyFormatDefaultsToJSON(t *testing.T) { outDir := t.TempDir() - if err := core.WriteOutput(sharedGreeterL1, nil, options.AnalysisOptions{OutputDir: outDir, Format: ""}); err != nil { + if err := core.WriteOutput(sharedGreeterL1, nil, nil, options.AnalysisOptions{OutputDir: outDir, Format: ""}); err != nil { t.Fatalf("WriteOutput with empty format: %v", err) } if _, err := os.Stat(filepath.Join(outDir, "analysis.json")); err != nil { @@ -171,18 +173,81 @@ func TestWriteOutput_EmptyFormatDefaultsToJSON(t *testing.T) { func TestWriteOutput_MsgpackNotImplemented(t *testing.T) { outDir := t.TempDir() - if err := core.WriteOutput(sharedGreeterL1, nil, options.AnalysisOptions{OutputDir: outDir, Format: "msgpack"}); err == nil { + if err := core.WriteOutput(sharedGreeterL1, nil, nil, options.AnalysisOptions{OutputDir: outDir, Format: "msgpack"}); err == nil { t.Fatal("expected error for --format msgpack, got nil") } } func TestWriteOutput_UnknownFormatErrors(t *testing.T) { outDir := t.TempDir() - if err := core.WriteOutput(sharedGreeterL1, nil, options.AnalysisOptions{OutputDir: outDir, Format: "csv"}); err == nil { + if err := core.WriteOutput(sharedGreeterL1, nil, nil, options.AnalysisOptions{OutputDir: outDir, Format: "csv"}); err == nil { t.Fatal("expected error for unknown format, got nil") } } +// TestWriteOutput_StreamsRenderJSONBytes: the streamed file and writer paths +// write exactly RenderJSON's bytes at -a 3 and -a 4 (the L4 param edges and +// degraded marker included), and Analyze has already released every L3 +// node's AST reference (#27). +func TestWriteOutput_StreamsRenderJSONBytes(t *testing.T) { + for _, level := range []options.AnalysisLevel{options.LevelDataflow, options.LevelSDG} { + t.Run(fmt.Sprintf("a%d", level), func(t *testing.T) { streamsRenderJSONBytes(t, "dataflow", level) }) + } + t.Run("a4-degraded", func(t *testing.T) { streamsRenderJSONBytes(t, "rangefunc", options.LevelSDG) }) +} + +func streamsRenderJSONBytes(t *testing.T, fixture string, level options.AnalysisLevel) { + opts := options.AnalysisOptions{ + InputPath: filepath.Join(testdataDir(), fixture), AppName: fixture, Level: level, + SkipTests: true, CacheDir: t.TempDir(), OutputDir: t.TempDir(), + } + a := core.New(opts) + app, err := a.Analyze() + if err != nil { + t.Fatal(err) + } + for key, r := range a.L3() { + for _, n := range r.Nodes { + if n.Stmt != nil { + t.Fatalf("%v: node %s still holds its AST after Analyze", key, n.Key) + } + } + } + want, err := core.RenderJSON(app, a.L3(), a.L4(), opts) + if err != nil { + t.Fatal(err) + } + if level == options.LevelSDG { + key := `"param_in":[` + if fixture == "rangefunc" { + key = `"degraded":[` + } + if !bytes.Contains(want, []byte(key)) { + t.Fatalf("-a 4 JSON of %s has no %s", fixture, key) + } + } + var buf bytes.Buffer + if err := core.StreamJSON(&buf, app, a.L3(), a.L4(), opts); err != nil { + t.Fatal(err) + } + if !bytes.Equal(buf.Bytes(), want) { + t.Error("StreamJSON bytes differ from RenderJSON") + } + if err := core.WriteOutput(app, a.L3(), a.L4(), opts); err != nil { + t.Fatal(err) + } + got, err := os.ReadFile(filepath.Join(opts.OutputDir, "analysis.json")) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(got, want) { + t.Error("analysis.json differs from RenderJSON") + } + if ents, _ := os.ReadDir(opts.OutputDir); len(ents) != 1 { + t.Errorf("output dir holds %d entries, want only analysis.json", len(ents)) + } +} + // ── Caching tests ───────────────────────────────────────────────────────────── // These tests must run their own analysis to exercise the caching machinery. @@ -294,15 +359,24 @@ func TestCaching_EagerForcesRebuild(t *testing.T) { // ── Helpers ─────────────────────────────────────────────────────────────────── +// allSignatures collects every callable signature, closures included (a closure +// is a call-graph edge source). func allSignatures(app *schema.GoApplication) map[string]bool { sigs := map[string]bool{} + var add func(sig string, c schema.GoCallable) + add = func(sig string, c schema.GoCallable) { + sigs[sig] = true + for _, ic := range c.InnerCallables { + add(ic.Signature, ic) + } + } for _, f := range app.SymbolTable { - for sig := range f.Functions { - sigs[sig] = true + for sig, c := range f.Functions { + add(sig, c) } for _, t := range f.Types { - for sig := range t.Methods { - sigs[sig] = true + for sig, c := range t.Methods { + add(sig, c) } } } diff --git a/internal/core/dataflow_test.go b/internal/core/dataflow_test.go index 362bf18..222d7e9 100644 --- a/internal/core/dataflow_test.go +++ b/internal/core/dataflow_test.go @@ -23,7 +23,7 @@ func TestAnalyze_Level3RunsDataflow(t *testing.T) { if len(a.L3()) == 0 { t.Fatal("-a 3 produced no dataflow results") } - data, err := core.RenderJSON(app, a.L3(), opts) + data, err := core.RenderJSON(app, a.L3(), a.L4(), opts) if err != nil { t.Fatal(err) } diff --git a/internal/core/sdg_test.go b/internal/core/sdg_test.go new file mode 100644 index 0000000..b517072 --- /dev/null +++ b/internal/core/sdg_test.go @@ -0,0 +1,157 @@ +package core_test + +import ( + "os" + "path/filepath" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/core" + "github.com/codellm-devkit/codeanalyzer-go/internal/options" +) + +// TestAnalyze_Level4RunsSDG: -a 4 runs the L4 pass (SSA + VTA) after L3 and +// keeps its result; a clean project degrades nothing. Below -a 4 there is no +// L4 result. +func TestAnalyze_Level4RunsSDG(t *testing.T) { + opts := options.AnalysisOptions{ + InputPath: greeterDir(t), Level: options.LevelSDG, SkipTests: true, CacheDir: t.TempDir(), Strict: true, + } + a := core.New(opts) + if _, err := a.Analyze(); err != nil { + t.Fatal(err) + } + if a.L4() == nil { + t.Fatal("-a 4 produced no L4 result") + } + if len(a.L4().Degraded) != 0 { + t.Errorf("degraded = %+v, want none", a.L4().Degraded) + } + if len(a.L3()) == 0 { + t.Fatal("-a 4 produced no L3 results") + } + for k, r := range a.L3() { + for _, n := range r.Nodes { + if n.Stmt != nil { + t.Fatalf("%v: node %s keeps its Stmt after -a 4", k, n.Key) + } + } + if r.DefUse != nil || r.CallKeys != nil { + t.Fatalf("%v keeps its L4 side tables after -a 4", k) + } + } + + opts.Level = options.LevelDataflow + a3 := core.New(opts) + if _, err := a3.Analyze(); err != nil { + t.Fatal(err) + } + if a3.L4() != nil { + t.Error("-a 3 ran the L4 pass") + } +} + +// TestAnalyze_Level4StrictFailsOnDegraded: an ill-typed package gets no SSA. +// By default -a 4 degrades and still succeeds; with --strict Analyze fails +// and the error names the degraded codes. +func TestAnalyze_Level4StrictFailsOnDegraded(t *testing.T) { + opts := options.AnalysisOptions{ + InputPath: filepath.Join(testdataDir(), "typeerr"), Level: options.LevelSDG, SkipTests: true, CacheDir: t.TempDir(), + } + a := core.New(opts) + if _, err := a.Analyze(); err != nil { + t.Fatalf("non-strict -a 4 failed: %v", err) + } + if a.L4() == nil || len(a.L4().Degraded) == 0 { + t.Fatalf("L4 = %+v, want a degraded result", a.L4()) + } + + opts.Strict = true + _, err := core.New(opts).Analyze() + if err == nil { + t.Fatal("--strict -a 4 on a degraded project succeeded") + } + for _, code := range []string{"partial_types", "partial_ssa", "no_ssa"} { + if !strings.Contains(err.Error(), code) { + t.Errorf("error %q does not name %s", err, code) + } + } +} + +// TestAnalyze_Level4MergesVTAEdges: at -a 4 the analyzer merges sdg.Run's +// VTA edges into the call graph and its VTA-only externals into +// app.Externals; -a 3 has neither. +func TestAnalyze_Level4MergesVTAEdges(t *testing.T) { + opts := options.AnalysisOptions{ + InputPath: filepath.Join(testdataDir(), "dataflow"), AppName: "dataflow", Level: options.LevelSDG, + SkipTests: true, CacheDir: t.TempDir(), + } + const src, dst = "example.com/dataflow/sdg.Buffered", "bytes.Buffer.Write" + vtaEdge := func(level options.AnalysisLevel) (vta, ext bool) { + opts.Level = level + app, err := core.New(opts).Analyze() + if err != nil { + t.Fatal(err) + } + for _, e := range app.CallGraph { + if e.Source == src && e.Target == dst { + vta = len(e.Provenance) == 1 && e.Provenance[0] == "vta" + } + } + _, ext = app.Externals[dst] + return vta, ext + } + if vta, ext := vtaEdge(options.LevelSDG); !vta || !ext { + t.Errorf("-a 4: VTA edge %s -> %s = %v, external = %v; want both", src, dst, vta, ext) + } + if vta, ext := vtaEdge(options.LevelDataflow); vta || ext { + t.Errorf("-a 3: VTA edge = %v, external = %v; want neither", vta, ext) + } +} + +// TestAnalyze_LibraryModels: --library-models is loaded into the L4 input. By +// default Greet's call of fmt.Sprintf has summary edges; the model +// fmt/Sprintf: [] (no flow) leaves it none. A missing or malformed file +// fails the run, naming the path. +func TestAnalyze_LibraryModels(t *testing.T) { + summaries := func(t *testing.T, models string) int { + t.Helper() + a := core.New(options.AnalysisOptions{ + InputPath: greeterDir(t), Level: options.LevelSDG, SkipTests: true, CacheDir: t.TempDir(), LibraryModels: models, + }) + if _, err := a.Analyze(); err != nil { + t.Fatal(err) + } + n := 0 + for k, c := range a.L4().Callables { + if strings.HasSuffix(k.Signature, "Greeter.Greet") { + n += len(c.Summary) + } + } + return n + } + if n := summaries(t, ""); n == 0 { + t.Fatal("Greet has no summary edges by default") + } + dir := t.TempDir() + good := filepath.Join(dir, "models.json") + if err := os.WriteFile(good, []byte(`{"fmt/Sprintf": []}`), 0o644); err != nil { + t.Fatal(err) + } + if n := summaries(t, good); n != 0 { + t.Errorf("fmt/Sprintf modelled with no flow: Greet has %d summary edges", n) + } + + bad := filepath.Join(dir, "bad.json") + if err := os.WriteFile(bad, []byte(`{"fmt/Sprintf": [`), 0o644); err != nil { + t.Fatal(err) + } + for _, path := range []string{filepath.Join(dir, "absent.json"), bad} { + _, err := core.New(options.AnalysisOptions{ + InputPath: greeterDir(t), Level: options.LevelSDG, SkipTests: true, CacheDir: t.TempDir(), LibraryModels: path, + }).Analyze() + if err == nil || !strings.Contains(err.Error(), path) { + t.Errorf("%s: Analyze error = %v, want one naming the path", path, err) + } + } +} diff --git a/internal/core/testsetup_test.go b/internal/core/testsetup_test.go index db04684..f68d89b 100644 --- a/internal/core/testsetup_test.go +++ b/internal/core/testsetup_test.go @@ -76,7 +76,7 @@ func runTestMain(m *testing.M) int { fmt.Fprintf(os.Stderr, "testsetup %s: Analyze: %v\n", f.name, err) return 1 } - if err := core.WriteOutput(app, nil, opts); err != nil { + if err := core.WriteOutput(app, nil, nil, opts); err != nil { fmt.Fprintf(os.Stderr, "testsetup %s: WriteOutput: %v\n", f.name, err) return 1 } diff --git a/internal/core/writeoutput_unix_test.go b/internal/core/writeoutput_unix_test.go new file mode 100644 index 0000000..dc4ff93 --- /dev/null +++ b/internal/core/writeoutput_unix_test.go @@ -0,0 +1,39 @@ +//go:build unix + +package core_test + +import ( + "os" + "path/filepath" + "syscall" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/core" + "github.com/codellm-devkit/codeanalyzer-go/internal/options" +) + +// TestWriteOutput_ModeHonoursUmask: analysis.json gets the mode os.Create +// would give it (0666 less the umask), not CreateTemp's 0600, and the temp +// file is gone after the rename. +func TestWriteOutput_ModeHonoursUmask(t *testing.T) { + for _, mask := range []int{0o022, 0o077} { + old := syscall.Umask(mask) + outDir := t.TempDir() + err := core.WriteOutput(sharedGreeterL1, nil, nil, options.AnalysisOptions{OutputDir: outDir, Format: "json"}) + syscall.Umask(old) + if err != nil { + t.Fatalf("umask %#o: WriteOutput: %v", mask, err) + } + st, err := os.Stat(filepath.Join(outDir, "analysis.json")) + if err != nil { + t.Fatal(err) + } + if want := os.FileMode(0o666 &^ mask); st.Mode().Perm() != want { + t.Errorf("umask %#o: mode = %v, want %v", mask, st.Mode().Perm(), want) + } + left, _ := filepath.Glob(filepath.Join(outDir, ".analysis.json.*")) + if len(left) != 0 { + t.Errorf("umask %#o: temp files left behind: %v", mask, left) + } + } +} diff --git a/internal/dataflow/accesspath.go b/internal/dataflow/accesspath.go index c860636..9415792 100644 --- a/internal/dataflow/accesspath.go +++ b/internal/dataflow/accesspath.go @@ -43,7 +43,14 @@ func (p Path) Covers(q Path) bool { } ph, ps := splitSpell(p.Spell) qh, qs := splitSpell(q.Spell) - if ph != qh || len(ps) > len(qs) { + return ph == qh && stepsCover(ps, qs) +} + +// stepsCover is Covers on step lists alone, bases ignored: ps is a prefix of +// qs, a final wildcard of ps matching any non-empty remainder. The alias +// rule (BuildDDGAlias) compares the suffixes of two different bases with it. +func stepsCover(ps, qs []string) bool { + if len(ps) > len(qs) { return false } for i, s := range ps { @@ -271,6 +278,21 @@ func baseSpell(v types.Object) string { return v.Name() } +// SpellBase returns the base spelling of an emitted path spelling: "x" of +// "x.f[*]", "pkgpath::Name" of "pkgpath::Name.F", a pseudo-base whole. The +// L4 port wiring maps a ddg edge's var to the formal of its base with it. +func SpellBase(s string) string { + head, _ := splitSpell(s) + return head +} + +// BaseSpell is baseSpell for package sdg: the L4 global formals and +// footprints spell a package-level variable as its L3 ddg base. +func BaseSpell(v types.Object) string { return baseSpell(v) } + +// IsPkgVar is isPkgVar for package sdg (the L4 global footprints). +func IsPkgVar(obj types.Object) bool { return isPkgVar(obj) } + // isPkgVar reports whether obj is a package-level variable. It checks // membership of the package scope by identity, which holds for packages // type-checked from source and imported from export data alike. diff --git a/internal/dataflow/bodynodes.go b/internal/dataflow/bodynodes.go index cdaafca..89f36d2 100644 --- a/internal/dataflow/bodynodes.go +++ b/internal/dataflow/bodynodes.go @@ -6,6 +6,7 @@ import ( "sort" "strconv" + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" ) @@ -34,12 +35,12 @@ import ( // statement gets one only as a label target. Function literal bodies are // skipped: closures are their own callables. // -// calls holds the L1 call-node keys in L1 recording order (v2.CallBodyKeys -// over the callable's call sites). They are paired with the body's call -// expressions by recording index (ruling R5), never by position: under a -// //line directive the recorded line:col does not name the call's token.Pos. -// The caller checks len(calls) == RecordedCallCount(body); surplus keys are -// ignored and calls past the end of the list get no call node. +// calls holds the L1 call nodes: each call-node key (v2.CallBodyKeys over the +// callable's call sites) with the raw CallExpr.Lparen of its site. They are +// paired with the body's call expressions by Lparen — never by index or by +// line:col — so their order is irrelevant. A call no entry names gets no call +// node; entries naming no call of the body are ignored. Run skips a unit +// with an unpaired call rather than build it. // // Keys: call nodes take their L1 key. Statement nodes take // v2.LocalID(line, col) of their anchor; a statement starting with a call @@ -52,11 +53,14 @@ import ( // synthetic nodes have NoPos and a nil Stmt. A nil body yields an empty slice // and an empty map (no entry/exit: a bodyless callable has no flow). // +// Spans: a node's End is clampEnd'ed to Pos's file (a malformed file's +// recovered nodes can end past it). +// // The returned map keys every node's Stmt to its key. It also maps the AST // nodes folded into a node to that node's key: a bare or `go` call's // *ast.CallExpr, an if/for/switch init, a for post, a type switch guard, a comm // clause's statement, and a *ast.LabeledStmt (to its inner statement's node). -func BuildNodes(fset *token.FileSet, body *ast.BlockStmt, calls []string) ([]Node, map[ast.Node]string) { +func BuildNodes(fset *token.FileSet, body *ast.BlockStmt, calls []UnitCall) ([]Node, map[ast.Node]string) { if body == nil { return []Node{}, map[ast.Node]string{} } @@ -66,8 +70,8 @@ func BuildNodes(fset *token.FileSet, body *ast.BlockStmt, calls []string) ([]Nod taken: make(map[string]bool, len(calls)), keyOf: map[ast.Node]string{}, } - for _, k := range calls { - b.taken[k] = true + for _, c := range calls { + b.taken[c.Key] = true } b.stmtList(body.List) @@ -75,7 +79,8 @@ func BuildNodes(fset *token.FileSet, body *ast.BlockStmt, calls []string) ([]Nod out := make([]Node, 0, len(b.nodes)+2) out = append(out, Node{Key: EntryKey, Kind: KindEntry}) for _, n := range b.nodes { - p, e := fset.Position(n.Pos), fset.Position(n.End) + n.End = clampEnd(fset, n.Pos, n.End) + p, e := fset.PositionFor(n.Pos, false), fset.PositionFor(n.End, false) n.StartLC, n.EndLC = [2]int{p.Line, p.Column}, [2]int{e.Line, e.Column} out = append(out, n) } @@ -83,43 +88,57 @@ func BuildNodes(fset *token.FileSet, body *ast.BlockStmt, calls []string) ([]Nod return out, b.keyOf } -// recordedCalls lists body's call expressions in the L1 recording order of -// buildCallSites (internal/syntactic_analysis): pre-order — an outer call -// before the calls nested in it, a `go` / `defer` statement's call before the -// calls in its function value and arguments — and no function literal bodies. -// The two walks must stay identical: keys pair with calls by index (R5). -func recordedCalls(body *ast.BlockStmt) []*ast.CallExpr { - var out []*ast.CallExpr - ast.Inspect(body, func(n ast.Node) bool { - switch n := n.(type) { - case *ast.FuncLit: - return false - case *ast.CallExpr: - out = append(out, n) - } - return true - }) - return out +// clampEnd keeps a node's End inside the file its Pos is in (#28). The +// parser recovers from a syntax error with nodes whose End is one past the +// end of their file: in a shared FileSet that position has no file (the last +// file) or is the next file's base. Such an End is clamped to the end of +// Pos's file: the recovered node really runs to there (recovery swallowed +// the rest of the file), so the span keeps all of its text and stays in its +// own file. An End before Pos in the same file (no known source of one) is +// clamped to Pos, an empty span: no text is invented. A Pos with no file is +// left as it is. +func clampEnd(fset *token.FileSet, pos, end token.Pos) token.Pos { + tf := fset.File(pos) + if tf == nil { + return end + } + if fset.File(end) != tf { + return tf.Pos(tf.Size()) + } + if end < pos { + return pos + } + return end } -// RecordedCallCount is the number of L1 call sites body records: the length -// the calls argument of BuildNodes must have. +// RecordedCallCount is the number of L1 call sites body records +// (astwalk.Calls): a cross-check on the length of a unit's calls. func RecordedCallCount(body *ast.BlockStmt) int { - if body == nil { - return 0 - } - return len(recordedCalls(body)) + return len(astwalk.Calls(body)) } -// assignCallKeys pairs the i-th recorded call with the i-th L1 key. -func assignCallKeys(body *ast.BlockStmt, calls []string) map[*ast.CallExpr]string { - out := map[*ast.CallExpr]string{} - for i, call := range recordedCalls(body) { - if i < len(calls) { - out[call] = calls[i] +// assignCallKeys pairs each recorded call of body with the L1 key whose +// Lparen is the call's. +func assignCallKeys(body *ast.BlockStmt, calls []UnitCall) map[*ast.CallExpr]string { + out, _ := pairCalls(body, calls) + return out +} + +// pairCalls pairs each recorded call of body with the L1 key whose Lparen is +// the call's, and returns the pairs and the number of recorded calls. +func pairCalls(body *ast.BlockStmt, calls []UnitCall) (map[*ast.CallExpr]string, int) { + byLparen := make(map[token.Pos]string, len(calls)) + for _, c := range calls { + byLparen[c.Lparen] = c.Key + } + recorded := astwalk.Calls(body) + out := make(map[*ast.CallExpr]string, len(recorded)) + for _, c := range recorded { + if k, ok := byLparen[c.Expr.Lparen]; ok { + out[c.Expr] = k } } - return out + return out, len(recorded) } type builder struct { @@ -270,15 +289,24 @@ func (b *builder) folded(k string, s ast.Stmt) { // position's LocalID, or the next free "/n" suffix when an L1 call (or an // earlier node) already holds it. func (b *builder) add(kind NodeKind, n ast.Node, pos, end token.Pos) string { - p := b.fset.Position(pos) - base := v2.LocalID(p.Line, p.Column) + p := b.fset.PositionFor(pos, false) + key := NextFreeKey(v2.LocalID(p.Line, p.Column), b.taken) + b.nodes = append(b.nodes, Node{Key: key, Kind: kind, Pos: pos, End: end, Stmt: n}) + b.keyOf[n] = key + return key +} + +// NextFreeKey returns base, or base + "/n" for the least n ≥ 2 that taken +// does not hold, and marks it taken. It is the one rule for keying a node at +// a position an L1 call (or an earlier node) may already hold: body nodes +// (BuildNodes) and the L4 capture sites both use it, so neither ever moves +// an L1 call key. +func NextFreeKey(base string, taken map[string]bool) string { key := base - for i := 2; b.taken[key]; i++ { + for i := 2; taken[key]; i++ { key = base + "/" + strconv.Itoa(i) } - b.taken[key] = true - b.nodes = append(b.nodes, Node{Key: key, Kind: kind, Pos: pos, End: end, Stmt: n}) - b.keyOf[n] = key + taken[key] = true return key } diff --git a/internal/dataflow/bodynodes_test.go b/internal/dataflow/bodynodes_test.go index e84afb4..fe6cf44 100644 --- a/internal/dataflow/bodynodes_test.go +++ b/internal/dataflow/bodynodes_test.go @@ -2,9 +2,11 @@ package dataflow_test import ( "go/ast" + "go/parser" "go/token" "go/types" "path/filepath" + "reflect" "runtime" "sync" "testing" @@ -63,17 +65,21 @@ func loadFlow(t *testing.T) *flowFixture { } // callKeys builds the L1 call-node keys exactly as the driver does: keys from -// v2.CallBodyKeys over the call sites' [line, col], in recording order. -// BuildNodes pairs them with the AST calls by index (ruling R5), never by -// rebuilding a token.Pos. -func (fx *flowFixture) callKeys(t *testing.T, fn string) []string { +// v2.CallBodyKeys over the call sites' [line, col], in recording order, each +// with its site's Lparen. BuildNodes pairs them with the AST calls by Lparen. +func (fx *flowFixture) callKeys(t *testing.T, fn string) []dataflow.UnitCall { t.Helper() sites := fx.sites[fn] pos := make([][2]int, len(sites)) for i, cs := range sites { pos[i] = [2]int{cs.StartLine, cs.StartColumn} } - return v2.CallBodyKeys(pos) + keys := v2.CallBodyKeys(pos) + out := make([]dataflow.UnitCall, len(sites)) + for i, cs := range sites { + out[i] = dataflow.UnitCall{Key: keys[i], Lparen: cs.Lparen} + } + return out } func (fx *flowFixture) decl(t *testing.T, fn string) *ast.FuncDecl { @@ -222,8 +228,8 @@ func TestBodyNodes_KindsPerConstruct(t *testing.T) { func TestBodyNodes_BareCallSharesKey(t *testing.T) { fx := loadFlow(t) l1 := map[string]bool{} - for _, k := range fx.callKeys(t, "EarlyReturnDefer") { - l1[k] = true + for _, c := range fx.callKeys(t, "EarlyReturnDefer") { + l1[c.Key] = true } _, byKey, keyOf := build(t, "EarlyReturnDefer") n := wantKind(t, "EarlyReturnDefer", byKey, "131:2", dataflow.KindCall) @@ -353,9 +359,9 @@ func TestBodyNodes_CallKeysMatchL1(t *testing.T) { fx := loadFlow(t) for fn := range fx.decls { _, byKey, _ := build(t, fn) - for _, k := range fx.callKeys(t, fn) { - if n, ok := byKey[k]; !ok || n.Kind != dataflow.KindCall { - t.Errorf("%s: L1 call key %s has no call node", fn, k) + for _, c := range fx.callKeys(t, fn) { + if n, ok := byKey[c.Key]; !ok || n.Kind != dataflow.KindCall { + t.Errorf("%s: L1 call key %s has no call node", fn, c.Key) } } } @@ -370,3 +376,102 @@ func TestBodyNodes_BodylessCallableEmitsNothing(t *testing.T) { t.Errorf("nil body: want empty (non-nil) slice and map") } } + +// TestBuildNodes_PairsByLparenNotIndex: BuildNodes pairs each L1 call key +// with the call whose Lparen it carries, so the order of the calls slice is +// irrelevant: a permuted slice keys every call exactly as the recording-order +// slice does — co-positioned chained calls (s.A().B() at l:c, s.A() at +// l:c/2) included. +func TestBuildNodes_PairsByLparenNotIndex(t *testing.T) { + fx := loadFlow(t) + for _, fn := range []string{"EarlyReturnDefer", "GoArgs", "Chain", "Nested", "Spawn"} { + body := fx.decl(t, fn).Body + calls := fx.callKeys(t, fn) + if len(calls) < 2 { + t.Fatalf("%s: %d calls, want ≥ 2 to permute", fn, len(calls)) + } + rev := make([]dataflow.UnitCall, len(calls)) + for i, c := range calls { + rev[len(calls)-1-i] = c + } + nodes, keyOf := dataflow.BuildNodes(fx.fset, body, calls) + nodes2, keyOf2 := dataflow.BuildNodes(fx.fset, body, rev) + if !reflect.DeepEqual(nodes, nodes2) || !reflect.DeepEqual(keyOf, keyOf2) { + t.Errorf("%s: permuted calls build different nodes", fn) + } + want := map[token.Pos]string{} + for _, c := range calls { + want[c.Lparen] = c.Key + } + n := 0 + for node, key := range keyOf2 { + if call, ok := node.(*ast.CallExpr); ok { + n++ + if want[call.Lparen] != key { + t.Errorf("%s: call at %v keyed %s, want %s", fn, fx.fset.Position(call.Pos()), key, want[call.Lparen]) + } + } + } + if n != len(calls) { + t.Errorf("%s: %d call expressions keyed, want %d", fn, n, len(calls)) + } + } +} + +// TestBuildNodes_EndClampedToFile (#28): in testdata/malformed the parser +// recovers from each syntax error with nodes whose End lies one past the end +// of the file: in the shared FileSet it has no file (the last file) or is the +// next file's base. BuildNodes clamps such an End to the end of Pos's file +// (and an End before Pos to Pos), so every node's [StartLC, EndLC] is a +// non-inverted range inside the node's own file. +func TestBuildNodes_EndClampedToFile(t *testing.T) { + _, thisFile, _, _ := runtime.Caller(0) + dir := filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "malformed") + fset := token.NewFileSet() + // Parse in name order into one FileSet, as go/packages does, so a bad + // End of one file is the next file's base. + var files []*ast.File + for _, name := range []string{"case.go", "doc.go", "if.go", "range.go", "recv.go"} { + f, _ := parser.ParseFile(fset, filepath.Join(dir, name), nil, parser.SkipObjectResolution) + if f == nil { + t.Fatalf("parse %s: no file", name) + } + files = append(files, f) + } + bad, checked := 0, 0 + for _, f := range files { + tf := fset.File(f.Pos()) + for _, d := range f.Decls { + fd, ok := d.(*ast.FuncDecl) + if !ok || fd.Body == nil { + continue + } + if fset.File(fd.End()) != tf { + bad++ + } + nodes, _ := dataflow.BuildNodes(fset, fd.Body, gateCallKeys(fset, fd.Body)) + for _, n := range nodes { + if n.Pos == token.NoPos { + continue + } + checked++ + p := fset.PositionFor(n.Pos, false) + if fset.File(n.End) != tf && n.End != tf.Pos(tf.Size()) { + t.Errorf("%s %s: End %v is outside Pos's file", p.Filename, n.Key, n.End) + } + if n.End < n.Pos { + t.Errorf("%s %s: End %v before Pos %v", p.Filename, n.Key, n.End, n.Pos) + } + if n.EndLC[0] < n.StartLC[0] || n.EndLC[0] == n.StartLC[0] && n.EndLC[1] < n.StartLC[1] { + t.Errorf("%s %s: EndLC %v before StartLC %v", p.Filename, n.Key, n.EndLC, n.StartLC) + } + if n.EndLC[0] > tf.LineCount() || n.EndLC[0] == 0 { + t.Errorf("%s %s: EndLC %v outside the file's %d lines", p.Filename, n.Key, n.EndLC, tf.LineCount()) + } + } + } + } + if bad < 4 || checked == 0 { + t.Errorf("malformed fixture: %d functions with a bad End, %d nodes checked; want ≥ 4 and > 0", bad, checked) + } +} diff --git a/internal/dataflow/cfg_exits.go b/internal/dataflow/cfg_exits.go index 4c7c39a..4e90b2d 100644 --- a/internal/dataflow/cfg_exits.go +++ b/internal/dataflow/cfg_exits.go @@ -319,7 +319,7 @@ func canPanic(n ast.Node, info *types.Info) bool { case *ast.ForStmt: parts = []ast.Node{s.Init, s.Cond, s.Post} case *ast.RangeStmt: - if rangesOverFunc(s.X, info) { + if RangesOverFunc(s.X, info) { return true // each iteration calls the iterator } parts = []ast.Node{s.Key, s.Value, s.X} @@ -493,10 +493,11 @@ func sliceToArray(call *ast.CallExpr, info *types.Info) bool { return ok } -// rangesOverFunc reports whether a range statement over x iterates a +// RangesOverFunc reports whether a range statement over x iterates a // function (range-over-func). Without info it cannot tell and says no; the -// header's own expressions are still inspected. -func rangesOverFunc(x ast.Expr, info *types.Info) bool { +// header's own expressions are still inspected. L4 lists the callables that +// hold one under degraded code range_func (D8). +func RangesOverFunc(x ast.Expr, info *types.Info) bool { if info == nil || x == nil { return false } diff --git a/internal/dataflow/cfg_test.go b/internal/dataflow/cfg_test.go index 90d3bd3..f575e2c 100644 --- a/internal/dataflow/cfg_test.go +++ b/internal/dataflow/cfg_test.go @@ -9,12 +9,14 @@ import ( "path/filepath" "reflect" "runtime" + "sort" "strings" "testing" "golang.org/x/tools/go/cfg" "golang.org/x/tools/go/packages" + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" ) @@ -768,28 +770,23 @@ func gateOne(t *testing.T, name string, nodes []dataflow.Node, es []dataflow.Cfg return chain } -// gateCallKeys lists the key of every call in body in the L1 recording order -// (pre-order, an outer call before its nested calls; no function literal -// bodies), as -// v2.CallBodyKeys assigns them from each call's (line-directive adjusted) -// position. -func gateCallKeys(fset *token.FileSet, body *ast.BlockStmt) []string { - var calls []*ast.CallExpr - ast.Inspect(body, func(n ast.Node) bool { - switch n := n.(type) { - case *ast.FuncLit: - return false - case *ast.CallExpr: - calls = append(calls, n) - } - return true - }) +// gateCallKeys lists the L1 call node of every call in body in the L1 +// recording order (astwalk.Calls), each keyed as v2.CallBodyKeys assigns it +// from the call's real (not //line-adjusted) position and carrying its +// Lparen. +func gateCallKeys(fset *token.FileSet, body *ast.BlockStmt) []dataflow.UnitCall { + calls := astwalk.Calls(body) pos := make([][2]int, len(calls)) for i, c := range calls { - p := fset.Position(c.Pos()) + p := fset.PositionFor(c.Expr.Pos(), false) pos[i] = [2]int{p.Line, p.Column} } - return v2.CallBodyKeys(pos) + keys := v2.CallBodyKeys(pos) + out := make([]dataflow.UnitCall, len(calls)) + for i, c := range calls { + out[i] = dataflow.UnitCall{Key: keys[i], Lparen: c.Expr.Lparen} + } + return out } // ---- canPanic rules (spec §4.3.2) ---- @@ -1301,3 +1298,66 @@ func TestCFG_DifferentialCatchesMissingEdge(t *testing.T) { t.Errorf("differential missed the cut edges; got %v", errs) } } + +// TestCFG_RangeIntAndRangeFunc (#28): exact CFGs for the Go 1.23 range forms +// in testdata/rangefunc (a go 1.23 module: testdata/dataflow is go 1.21). A +// range over an integer is a plain loop header; a range over an iterator +// function calls the iterator on every trip, so its header may panic as well +// (canPanic): an exception edge to @exit. Both agree with go/cfg. +func TestCFG_RangeIntAndRangeFunc(t *testing.T) { + _, thisFile, _, _ := runtime.Caller(0) + fset, cases := loadDiffCases(t, filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "rangefunc")) + want := map[string][]dataflow.CfgEdge{ + "example.com/rangefunc.RangeInt": { + e(entry, "8:2", "fallthrough"), + e("8:2", "9:2", "fallthrough"), + e("9:2", "10:3", "true"), + e("9:2", "12:2", "false"), + e("10:3", "9:2", "loop_back"), + e("12:2", exit, "return"), + }, + "example.com/rangefunc.RangeFunc": { + e(entry, "18:2", "fallthrough"), + e("18:2", "19:2", "fallthrough"), + e("19:2", "20:3", "true"), + e("19:2", "22:2", "false"), + e("19:2", exit, "exception"), + e("20:3", "19:2", "loop_back"), + e("22:2", exit, "return"), + }, + } + seen := 0 + for _, c := range cases { + w, ok := want[c.name] + if !ok { + continue + } + seen++ + nodes, keyOf := dataflow.BuildNodes(fset, c.body, gateCallKeys(fset, c.body)) + kept, got := dataflow.BuildCFG(nodes, keyOf, c.body, c.info) + checkContract(t, c.name, kept, got) + sortCfg(w) // BuildCFG orders by (src, dst, kind) as strings + if !reflect.DeepEqual(got, w) { + t.Errorf("%s:\n got %v\nwant %v", c.name, got, w) + } + if diffOne(t.Errorf, c.name, c.body, c.info, keyOf, got) == 0 { + t.Errorf("%s: differential checked no pairs", c.name) + } + } + if seen != len(want) { + t.Errorf("rangefunc: found %d of %d functions", seen, len(want)) + } +} + +func sortCfg(es []dataflow.CfgEdge) { + sort.Slice(es, func(i, j int) bool { + a, b := es[i], es[j] + if a.Src != b.Src { + return a.Src < b.Src + } + if a.Dst != b.Dst { + return a.Dst < b.Dst + } + return a.Kind < b.Kind + }) +} diff --git a/internal/dataflow/defuse.go b/internal/dataflow/defuse.go index 4162c57..87452ff 100644 --- a/internal/dataflow/defuse.go +++ b/internal/dataflow/defuse.go @@ -18,11 +18,18 @@ import ( // // A strong def never has a wildcard step (any write through an index, a // dereference or past k is weak), so a kill never removes more than the -// written path covers. The `:` pseudo path (nil Base) appears only as -// a registration's weak def and its deferred call node's use. Each list is -// deduplicated and sorted by (Spell, base position). +// written path covers. The `:` pseudo path (nil Base) appears as a +// registration's weak def, its deferred call node's use, and the weak def of +// a node writing one of the deferred call's operand bases. +// +// Shallow is the subset of Defs that kills only its own path, not the +// extensions below it: `s = append(s, …)` rebinds s, but the new s keeps the +// old elements, so it kills s and not s[*] or s.* (#28). +// +// Each list is deduplicated and sorted by (Spell, base position). type DefUse struct { Defs, WeakDefs, Uses []Path + Shallow []Path } // DeferPrefix starts the pseudo path a defer registration weak-defines and its @@ -35,6 +42,10 @@ const DeferPrefix = ":" // BuildNodes' map, used to find the deferred call node's key from a defer // registration (keyOf[DeferStmt.Call]). info must be non-nil. // +// The capture-write and deferred-operand rules below need facts about the +// whole callable, which DefUseOf derives from fn and keyOf on every call; +// DefUses computes them once for a node set and returns the same result. +// // Rules by node: // // - @entry defines a base path for every object in entryBases; @exit has @@ -60,7 +71,8 @@ const DeferPrefix = ":" // nested in them) and weak-defines the pseudo path ":", // = keyOf[DeferStmt.Call], the deferred call node's Key. The // deferred call node (Deferred) uses that pseudo path, every named result -// of fn and, for `defer func(){…}()`, what the closure captures; it +// of fn and, for `defer func(){…}()`, what the closure captures (and +// weak-defines what it writes of them: capture writes, below); it // weak-defines what the call may write through (call rule), uses those // same paths (it reads receiver and argument state at exit, not at // registration) and never strong-defines anything (R11: chain @@ -83,15 +95,71 @@ const DeferPrefix = ":" // weak-defines a.* for each argument of pointer, slice, map, chan, func or // interface type (`&x` gives x.*; a slice expression of a path gives its // base's elements: buf[:n], arr[:] and (*pa)[i:] give buf[*], arr[*], pa[*]) -// and recv.* for a receiver that is -// reference-typed (the same kinds) or bound to a pointer-receiver method. A value-typed struct -// argument is not weak-defined. Conversions write nothing; builtins write +// and recv.* for a receiver that is reference-typed (the same kinds) or bound +// to a pointer-receiver method. A value struct or array argument or receiver +// that may hold a reference (MayHoldRef: a field or element, transitively, of +// a reference kind) weak-defines a.* too, the whole base, never per field +// (D1 gap 2); one that holds no reference is copied and not weak-defined. +// Conversions write nothing; builtins write // nothing except append(dst, …), copy(dst, …), delete(m, …) and clear(x), // which weak-define dst[*], m[*], x[*] (copy(dst[i:], …) and append(dst[:0], // …) give dst[*] too: append may write into dst's backing array); append's -// result also flows to the assignment's LHS. +// result also flows to the assignment's LHS. `s = append(s, …)` (the LHS is +// append's destination path, sliceBase stripped) strong-defines s as +// Shallow: the new s keeps the old elements, so an earlier `s[0] = v` still +// reaches a later read of s[0] (#28). +// +// Callable-level rules (spec §4.5 "L3 precision", D1). They only add edges: +// +// - Capture writes (gap 1). A function literal may run at any later point +// (called in place, stored, passed to once.Do, deferred, launched), and +// its writes to the variables it captures are writes of this callable. +// Let W be the union of hostWrites over the outermost literals of fn's +// body (a literal nested in another belongs to the outer one's unit, and +// CaptureWrites already includes what it writes). The host of a literal +// (the node whose DefUse is computed from an expression containing it: +// for `defer func(){…}()` the deferred call node, for `go func(){…}()` +// the go node) weak-defines each path of CaptureWrites(lit), and x.* for +// every capture x that may hold a reference (hostWrites: a write +// through a local alias, `q := p; q.n = v`, is no path of p). W is the +// union of those. Every strong def in fn whose base is a base of W is +// weak instead (@entry excepted: +// nothing reaches it to kill). This is flow-insensitive per captured +// variable on purpose: `h := func(){ x = 1 }; x = 2; h()` must not let +// x = 2 kill the closure's write. +// - Deferred operands (gap 3). For each `defer` in fn's body (not in a +// nested literal), its operand bases are the receiver's base of a method +// value and each argument's base (sliceBase stripped, & stripped); a +// function literal's captures do not count (the deferred call node +// already uses them at exit). Every non-@entry, non-deferred node that +// defines (strongly or weakly) a path on such a base also weak-defines +// that defer's ":", which nothing kills, so `defer +// fill(b); b[0] = s; b = o` keeps b[0] = s → fill. A def before the +// registration gets it too: harmless, an over-approximation. func DefUseOf(n Node, info *types.Info, k int, entryBases []types.Object, fn ast.Node, keyOf map[ast.Node]string) DefUse { - c := &collector{info: info, k: k, seen: map[pathKind]bool{}} + return defUseOf(n, info, k, entryBases, fn, keyOf, newFacts(fn, info, k, keyOf)) +} + +// DefUses is DefUseOf of every node of nodes, keyed by Key, with the +// callable-level facts computed once. +func DefUses(nodes []Node, info *types.Info, k int, entryBases []types.Object, fn ast.Node, keyOf map[ast.Node]string) map[string]DefUse { + return defUses(nodes, info, k, entryBases, fn, keyOf, newFacts(fn, info, k, keyOf)) +} + +func defUses(nodes []Node, info *types.Info, k int, entryBases []types.Object, fn ast.Node, keyOf map[ast.Node]string, f *facts) map[string]DefUse { + out := make(map[string]DefUse, len(nodes)) + for _, n := range nodes { + out[n.Key] = defUseOf(n, info, k, entryBases, fn, keyOf, f) + } + return out +} + +// defUseOf is DefUseOf with the callable-level facts given; nil facts apply +// none of the D1 rules (the pre-D1 sets, for the gates): no callable-level +// rule, and the call rule writes through reference kinds only (refTyped, not +// MayHoldRef). +func defUseOf(n Node, info *types.Info, k int, entryBases []types.Object, fn ast.Node, keyOf map[ast.Node]string, f *facts) DefUse { + c := &collector{info: info, k: k, seen: map[pathKind]bool{}, preD1: f == nil} switch { case n.Kind == KindEntry: for _, b := range entryBases { @@ -103,12 +171,14 @@ func DefUseOf(n Node, info *types.Info, k int, entryBases []types.Object, fn ast if !ok { break } + c.facts = f c.add(kindUse, Path{Spell: DeferPrefix + n.Key}) for _, r := range namedResults(fn, info) { c.add(kindUse, Path{Base: r, Spell: r.Name()}) } - if lit, ok := ast.Unparen(call.Fun).(*ast.FuncLit); ok { - c.captures(lit) + lit, _ := ast.Unparen(call.Fun).(*ast.FuncLit) + if lit != nil { + c.captureUses(lit) } c.callWrites(call) // The call runs at exit and reads, then, the state behind what it may @@ -117,17 +187,43 @@ func DefUseOf(n Node, info *types.Info, k int, entryBases []types.Object, fn ast for _, p := range c.du.WeakDefs { c.add(kindUse, p) } + if lit != nil { + // The closure's own writes (gap 1): it already uses its captures. + c.captureWrites(lit) + } default: c.fn = fn c.keyOf = keyOf + c.facts = f c.stmt(n.Stmt) + c.deferOperandWrites() } sortPaths(c.du.Defs) sortPaths(c.du.WeakDefs) sortPaths(c.du.Uses) + sortPaths(c.du.Shallow) return c.du } +// deferOperandWrites weak-defines ":" of every defer whose +// operand base this node defines. +func (c *collector) deferOperandWrites() { + if c.facts == nil || len(c.facts.deferOps) == 0 { + return + } + var spells []string + for _, list := range [][]Path{c.du.Defs, c.du.WeakDefs} { + for _, p := range list { + if p.Base != nil { + spells = append(spells, c.facts.deferOps[p.Base]...) + } + } + } + for _, s := range spells { + c.add(kindWeak, Path{Spell: s}) + } +} + type pathKind struct { p Path kind int @@ -144,22 +240,28 @@ type collector struct { k int fn ast.Node keyOf map[ast.Node]string + facts *facts // callable-level facts; nil applies no callable-level rule du DefUse seen map[pathKind]bool + shallow map[Path]bool weakOnly bool // record every def as weak (a range with `=`) + preD1 bool // call rule over reference kinds only (the pre-D1 gate sets) } // fork returns an empty collector with c's settings. func (c *collector) fork() *collector { - return &collector{info: c.info, k: c.k, fn: c.fn, keyOf: c.keyOf, seen: map[pathKind]bool{}} + return &collector{info: c.info, k: c.k, fn: c.fn, keyOf: c.keyOf, facts: c.facts, seen: map[pathKind]bool{}, preD1: c.preD1} } // mergeLoop adds a's facts to c, keeping a strong def strong only when other -// strong-defines the same path too. +// strong-defines the same path too (Shallow when either side's is). func (c *collector) mergeLoop(a, other *collector) { for _, p := range a.du.Defs { if other.seen[pathKind{p, kindDef}] { c.add(kindDef, p) + if a.shallow[p] || other.shallow[p] { + c.markShallow(p) + } } else { c.add(kindWeak, p) } @@ -190,6 +292,9 @@ func sortPaths(ps []Path) { } func (c *collector) add(kind int, p Path) { + if kind == kindDef && c.facts != nil && p.Base != nil && c.facts.weakBases[p.Base] { + kind = kindWeak // a base a literal of this callable may write (D1 gap 1) + } if c.seen[pathKind{p, kind}] { return } @@ -204,6 +309,20 @@ func (c *collector) add(kind int, p Path) { } } +// markShallow records strong def p as Shallow; a no-op unless p is in Defs. +func (c *collector) markShallow(p Path) { + if !c.seen[pathKind{p, kindDef}] { + return + } + if c.shallow == nil { + c.shallow = map[Path]bool{} + } + if !c.shallow[p] { + c.shallow[p] = true + c.du.Shallow = append(c.du.Shallow, p) + } +} + func (c *collector) addUse(b pathB) { c.add(kindUse, b.limit(c.k)) } func (c *collector) addWeak(b pathB) { c.add(kindWeak, b.limit(c.k)) } @@ -215,8 +334,11 @@ func (c *collector) stmt(s ast.Node) { for _, r := range s.Rhs { c.use(r) } - for _, l := range s.Lhs { + for i, l := range s.Lhs { c.assignTo(l, compound) + if !compound && len(s.Lhs) == len(s.Rhs) { + c.appendOnto(l, s.Rhs[i]) + } } case *ast.IncDecStmt: c.assignTo(s.X, true) @@ -474,13 +596,63 @@ func (c *collector) recv(ch ast.Expr) { c.indexReads(ch) } -// captures uses every variable function literal lit captures. +// appendOnto marks the strong def of lhs Shallow when rhs is append(lhs, …) +// (or append(lhs[:j], …)): the new slice keeps lhs's elements, so the def +// kills lhs but not lhs[*] (#28). callWrites already weak-defines lhs[*]. +func (c *collector) appendOnto(lhs, rhs ast.Expr) { + if c.facts == nil { + return + } + call, ok := ast.Unparen(rhs).(*ast.CallExpr) + if !ok || len(call.Args) == 0 { + return + } + if name, ok := builtinName(call.Fun, c.info); !ok || name != "append" { + return + } + l, ok := pathOf(lhs, c.info) + if !ok || l.base == nil { + return + } + dst, _ := sliceBase(call.Args[0]) + d, ok := pathOf(dst, c.info) + if !ok { + return + } + if p := l.limit(c.k); p == d.limit(c.k) { + c.markShallow(p) + } +} + +// captures uses every variable function literal lit captures and, with the +// callable-level facts, weak-defines what lit may write of them (D1 gap 1). func (c *collector) captures(lit *ast.FuncLit) { + c.captureUses(lit) + c.captureWrites(lit) +} + +// captureUses uses every variable function literal lit captures. +func (c *collector) captureUses(lit *ast.FuncLit) { for _, v := range freeVars(lit, c.info) { c.add(kindUse, Path{Base: v, Spell: v.Name()}) } } +// captureWrites weak-defines hostWrites(lit), with the callable-level facts +// only. +func (c *collector) captureWrites(lit *ast.FuncLit) { + if c.facts == nil { + return + } + ws, ok := c.facts.capW[lit] + if !ok { + ws = hostWrites(lit, c.info, c.k) + } + for _, p := range ws { + c.add(kindWeak, p) + } +} + // isQualified reports whether e is a qualified identifier pkg.Name. func isQualified(e *ast.SelectorExpr, info *types.Info) bool { x, ok := e.X.(*ast.Ident) diff --git a/internal/dataflow/defuse_calls.go b/internal/dataflow/defuse_calls.go index 2e223fa..9f8ac28 100644 --- a/internal/dataflow/defuse_calls.go +++ b/internal/dataflow/defuse_calls.go @@ -4,6 +4,7 @@ import ( "go/ast" "go/token" "go/types" + "sort" ) // The call rule of DefUseOf (spec §4.5), split into what a call reads and @@ -30,12 +31,13 @@ func (c *collector) callReads(call *ast.CallExpr, captures bool) { } } -// callWrites collects the weak defs of call itself: a.* for each -// reference-typed argument (&x gives x.*), recv.* for a reference-typed -// receiver or a pointer-receiver method, and the element writes of append, -// copy, delete and clear. append(dst, …) may write into dst's backing array -// (spare capacity, or dst[:0]), so it weak-defines dst[*] besides its result -// flowing to the assignment. Conversions and other builtins write nothing. +// callWrites collects the weak defs of call itself: a.* for each argument +// that may hold a reference (MayHoldRef: &x gives x.*), recv.* for a +// receiver that may hold one or is bound to a pointer-receiver method, and +// the element writes of append, copy, delete and clear. append(dst, …) may +// write into dst's backing array (spare capacity, or dst[:0]), so it +// weak-defines dst[*] besides its result flowing to the assignment. +// Conversions and other builtins write nothing. func (c *collector) callWrites(call *ast.CallExpr) { if isType(call.Fun, c.info) { return @@ -54,18 +56,27 @@ func (c *collector) callWrites(call *ast.CallExpr) { } if sel, ok := ast.Unparen(call.Fun).(*ast.SelectorExpr); ok { if s := c.info.Selections[sel]; s != nil && s.Kind() == types.MethodVal { - if ptrRecv(s.Obj()) || refTyped(c.info.TypeOf(sel.X)) { + if ptrRecv(s.Obj()) || c.holdsRef(c.info.TypeOf(sel.X)) { c.writeThrough(sel.X) } } } for _, a := range call.Args { - if refTyped(c.info.TypeOf(a)) { + if c.holdsRef(c.info.TypeOf(a)) { c.writeThrough(a) } } } +// holdsRef is the call rule's write-through predicate: MayHoldRef, or the +// reference kinds alone (refTyped) for the pre-D1 gate sets. +func (c *collector) holdsRef(t types.Type) bool { + if c.preD1 { + return refTyped(t) + } + return MayHoldRef(t) +} + // writeThrough weak-defines a.* (x.* for &x) when a is a path, and base[*] // when a slices a path (buf[:n], arr[:], (*p)[i:]): the callee writes the // elements the slice shares with its base. @@ -149,9 +160,9 @@ func isPointer(t types.Type) bool { return ok } -// refTyped reports whether a value of type t lets a callee write state the -// caller sees: pointer, slice, map, chan, func or interface (a type -// parameter's underlying type is its constraint interface, so it counts). +// refTyped reports whether t is itself a reference kind: pointer, slice, +// map, chan, func or interface (a type parameter's underlying type is its +// constraint interface, so it counts). func refTyped(t types.Type) bool { switch underlying(t).(type) { case *types.Pointer, *types.Slice, *types.Map, *types.Chan, *types.Signature, *types.Interface: @@ -159,3 +170,107 @@ func refTyped(t types.Type) bool { } return false } + +// MayHoldRef reports whether a value of type t, passed by value, lets a +// callee write state the caller sees (spec §4.5 Calls, D1 gap 2): t is a +// reference kind (refTyped), or a struct or array any of whose fields or +// elements, transitively through named types, may hold one (a value +// cfg{items []int} shares items' backing array with its copy). A struct or +// array that holds no reference is copied whole and is false. Cycle-safe: a +// named type already on the walk is not re-entered (a value-only cycle, +// possible only in a package with type errors, holds nothing new). The call +// rule and the L4 by-ref formal_out rule share it. +// +// Only a struct or array needs the walk and its cycle set: every other type +// is decided by refTyped alone, with no allocation. +func MayHoldRef(t types.Type) bool { + if t == nil { + return false + } + if refTyped(t) { + return true + } + switch t.Underlying().(type) { + case *types.Struct, *types.Array: + return mayHoldRef(t, map[*types.Named]bool{}) + } + return false +} + +func mayHoldRef(t types.Type, seen map[*types.Named]bool) bool { + if t == nil { + return false + } + if n, ok := types.Unalias(t).(*types.Named); ok { + if seen[n] { + return false + } + seen[n] = true + } + if refTyped(t) { + return true + } + switch u := t.Underlying().(type) { + case *types.Struct: + for i := 0; i < u.NumFields(); i++ { + if mayHoldRef(u.Field(i).Type(), seen) { + return true + } + } + case *types.Array: + return mayHoldRef(u.Elem(), seen) + } + return false +} + +// deferOperandBases maps each operand base of every `defer` in body (not in a +// nested function literal) to the defers' ":" pseudo paths, +// sorted: the base of a method value's receiver and of each argument path +// (sliceBase and & stripped; a non-path argument has none). A function +// literal's captures are not operands: the deferred call node already uses +// them at exit. A defer keyOf does not name is skipped. +func deferOperandBases(body *ast.BlockStmt, info *types.Info, keyOf map[ast.Node]string) map[types.Object][]string { + out := map[types.Object][]string{} + add := func(e ast.Expr, spell string) { + e, _ = sliceBase(e) + e = ast.Unparen(e) + if u, ok := e.(*ast.UnaryExpr); ok && u.Op == token.AND { + e = u.X + } + b, ok := pathOf(e, info) + if !ok || b.base == nil { + return + } + for _, s := range out[b.base] { + if s == spell { + return + } + } + out[b.base] = append(out[b.base], spell) + } + ast.Inspect(body, func(n ast.Node) bool { + switch d := n.(type) { + case *ast.FuncLit: + return false + case *ast.DeferStmt: + key, ok := keyOf[d.Call] + if !ok { + return true + } + spell := DeferPrefix + key + if sel, ok := ast.Unparen(d.Call.Fun).(*ast.SelectorExpr); ok { + if s := info.Selections[sel]; s != nil && s.Kind() == types.MethodVal { + add(sel.X, spell) + } + } + for _, a := range d.Call.Args { + add(a, spell) + } + } + return true + }) + for _, ss := range out { + sort.Strings(ss) + } + return out +} diff --git a/internal/dataflow/defuse_entry.go b/internal/dataflow/defuse_entry.go index a52543d..36b77a4 100644 --- a/internal/dataflow/defuse_entry.go +++ b/internal/dataflow/defuse_entry.go @@ -3,7 +3,10 @@ package dataflow import ( "go/ast" "go/types" + "slices" "sort" + + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" ) // What @entry defines (EntryBases) and the callable-level bases DefUseOf @@ -107,6 +110,10 @@ func EntryBases(fn ast.Node, info *types.Info) []types.Object { return append(out, globals...) } +// FreeVars is freeVars, exported for the L4 capture formals (spec §4.6): +// one capture list, in one order, for L3's @entry and L4's ports. +func FreeVars(lit *ast.FuncLit, info *types.Info) []types.Object { return freeVars(lit, info) } + // freeVars returns the variables function literal lit captures: every // variable its body (nested literals included) uses that is declared outside // lit and not at package scope, by position, then name. @@ -134,3 +141,112 @@ func freeVars(lit *ast.FuncLit, info *types.Info) []types.Object { }) return out } + +// facts are the callable-level inputs of DefUseOf's D1 rules (spec §4.5 "L3 +// precision"), computed once per callable from its body. +type facts struct { + // capW is hostWrites of each outermost function literal of the body. + capW map[*ast.FuncLit][]Path + // weakBases is the base of every path in W, the union of capW: a strong + // def on one of them is weak (gap 1). + weakBases map[types.Object]bool + // deferOps maps an operand base of a registered defer to the ":" + // pseudo paths of the defers it is an operand of, sorted (gap 3). + deferOps map[types.Object][]string +} + +// newFacts computes fn's callable-level facts; nil for a nil info or a +// callable without a body. +func newFacts(fn ast.Node, info *types.Info, k int, keyOf map[ast.Node]string) *facts { + body := funcBody(fn) + if info == nil || body == nil { + return nil + } + f := &facts{capW: map[*ast.FuncLit][]Path{}, weakBases: map[types.Object]bool{}} + for _, l := range astwalk.Literals(body) { + ws := hostWrites(l.Lit, info, k) + f.capW[l.Lit] = ws + for _, p := range ws { + f.weakBases[p.Base] = true + } + } + f.deferOps = deferOperandBases(body, info, keyOf) + return f +} + +// hostWrites is what the host of function literal lit weak-defines (D1 gap +// 1): CaptureWrites(lit), plus x.* for every capture x whose type may hold a +// reference (MayHoldRef). The literal may write through x without writing a +// path of x (`q := p; q.n = v` defines q.n, never p.*), exactly as a callee +// may write through a reference argument (the call rule), so a reference- +// holding capture is written through by the shared by-ref predicate, as at +// L4 (sdg writtenCaptures). +func hostWrites(lit *ast.FuncLit, info *types.Info, k int) []Path { + ws := CaptureWrites(lit, info, k) + seen := map[Path]bool{} + for _, p := range ws { + seen[p] = true + } + var add []Path + for _, v := range freeVars(lit, info) { + if !MayHoldRef(v.Type()) { + continue + } + p := pathB{base: v, head: baseSpell(v)}.with(".*").limit(k) + if !seen[p] { + seen[p] = true + add = append(add, p) + } + } + if len(add) == 0 { + return ws + } + out := append(slices.Clip(ws), add...) + sortPaths(out) + return out +} + +// CaptureWrites returns the paths function literal lit, or any literal nested +// in it, may write (strong or weak defs, by DefUseOf's def rules, k-limited), +// restricted to bases that are free variables of lit (freeVars: a variable +// declared inside lit, or at package scope, is not a capture). Sorted like a +// DefUse list (by spelling, then base position). +// +// A write is collected wherever it sits in lit's body: in a nested block, a +// loop or branch body, a deferred call (whose writes the deferred call node +// performs) or a nested literal's body. +func CaptureWrites(lit *ast.FuncLit, info *types.Info, k int) []Path { + if lit == nil || info == nil { + return nil + } + free := map[types.Object]bool{} + for _, v := range freeVars(lit, info) { + free[v] = true + } + if len(free) == 0 { + return nil + } + c := &collector{info: info, k: k, seen: map[pathKind]bool{}} + ast.Inspect(lit.Body, func(n ast.Node) bool { + switch s := n.(type) { + case *ast.DeferStmt: + c.stmt(s) + c.callWrites(s.Call) + case ast.Stmt: + c.stmt(s) + } + return true + }) + seen := map[Path]bool{} + var out []Path + for _, list := range [][]Path{c.du.Defs, c.du.WeakDefs} { + for _, p := range list { + if p.Base != nil && free[p.Base] && !seen[p] { + seen[p] = true + out = append(out, p) + } + } + } + sortPaths(out) + return out +} diff --git a/internal/dataflow/defuse_test.go b/internal/dataflow/defuse_test.go index d23d0ed..2618924 100644 --- a/internal/dataflow/defuse_test.go +++ b/internal/dataflow/defuse_test.go @@ -163,13 +163,15 @@ func TestDefUse_FieldDefIsPrefixDef(t *testing.T) { func TestDefUse_CallWeakDefsRefArgs(t *testing.T) { fx := loadDU(t) du, _ := fx.du(t, "Calls", "mut(&req", 0) - wantDU(t, "mut(...)", du, nil, ss("m.*", "p.*", "req.*", "s.*"), ss("m", "p", "req", "s", "t")) + // t is a value T, whose field s is a slice: it may hold a reference, so + // the callee may write through it (D1 gap 2). + wantDU(t, "mut(...)", du, nil, ss("m.*", "p.*", "req.*", "s.*", "t.*"), ss("m", "p", "req", "s", "t")) du, _ = fx.du(t, "Calls", "t.Ptr()", 0) wantDU(t, "t.Ptr()", du, nil, ss("t.*"), ss("t")) du, _ = fx.du(t, "Calls", "p.Ptr()", 0) wantDU(t, "p.Ptr()", du, nil, ss("p.*"), ss("p")) du, _ = fx.du(t, "Calls", "t.Val()", 0) - wantDU(t, "t.Val()", du, nil, nil, ss("t")) + wantDU(t, "t.Val()", du, nil, ss("t.*"), ss("t")) // value receiver holding T.s (D1 gap 2) du, _ = fx.du(t, "Calls", "n := len(s)", 0) wantDU(t, "n := len(s)", du, ss("n"), nil, ss("s")) du, _ = fx.du(t, "Calls", "s = append(s, n)", 0) @@ -267,12 +269,14 @@ func TestDefUse_DeferRegistrationCarriesArgs(t *testing.T) { wantDU(t, "mu.Unlock() (deferred)", call, nil, ss("mu.*"), ss(d, "err", "mu.*", "n")) // defer func(){…}(): nothing is evaluated at registration but the func - // value; the deferred call uses what the closure captures. + // value; the deferred call uses what the closure captures and (D1 gap 1) + // weak-defines what it writes of them: err = nil, and err.* (an + // interface may hold a reference: hostWrites). reg, _ = fx.du(t, "Defers", "defer func() {", 0) call, ck = fx.du(t, "Defers", "defer func() {", len("defer ")) d = ":" + ck wantDU(t, "defer func(){…}", reg, nil, ss(d), nil) - wantDU(t, "func(){…}() (deferred)", call, nil, nil, ss(d, "err", "n", "x")) + wantDU(t, "func(){…}() (deferred)", call, nil, ss("err", "err.*"), ss(d, "err", "n", "x")) reg, _ = fx.du(t, "Defers", "defer mut2(s, x)", 0) call, ck = fx.du(t, "Defers", "defer mut2(s, x)", len("defer ")) @@ -329,7 +333,7 @@ func TestDefUse_EntryDefinesParamsReceiverNamedResults(t *testing.T) { t.Errorf("EntryBases(closure) = %v, want %v", names, want) } h, _ := fx.du(t, "Entry", "h := func", 0) - wantDU(t, "h := func…", h, ss("h"), nil, ss("a", "r")) + wantDU(t, "h := func…", h, ss("h"), ss("r.*"), ss("a", "r")) // A flow-fixture function with no package vars: params only. wantDU(t, "Bump @entry", flowDU(t, "Bump", dataflow.EntryKey), ss("n"), nil, nil) @@ -392,7 +396,7 @@ func TestDefUse_ReferenceTypedReceiverIsWeakDefined(t *testing.T) { du, _ = fx.du(t, "Recv", "h.Set(", 0) wantDU(t, "h.Set(...)", du, nil, ss("h.*"), ss("h")) du, _ = fx.du(t, "Recv", "\tt.Val()\n}\n\nfunc IfShadow", 1) - wantDU(t, "t.Val() value struct", du, nil, nil, ss("t")) + wantDU(t, "t.Val() value struct", du, nil, ss("t.*"), ss("t")) // T.s is a slice (D1 gap 2) } func TestDefUse_IfInitShadows(t *testing.T) { @@ -419,3 +423,190 @@ func TestDefUse_SubsliceArgs(t *testing.T) { wantDU(t, "clear(m)", flowDU(t, "ClearAll", "507:2"), nil, ss("m[*]"), ss("m")) wantDU(t, "clear(s[:n])", flowDU(t, "ClearAll", "508:2"), nil, ss("s[*]"), ss("n", "s")) } + +// firstLit is the first function literal in fn's body (outermost first). +func firstLit(t *testing.T, fn string) *ast.FuncLit { + t.Helper() + var lit *ast.FuncLit + ast.Inspect(loadFlow(t).decl(t, fn).Body, func(n ast.Node) bool { + if f, ok := n.(*ast.FuncLit); ok && lit == nil { + lit = f + } + return lit == nil + }) + if lit == nil { + t.Fatalf("%s: no function literal", fn) + } + return lit +} + +// TestDefUse_CaptureWrites (D1 gap 1): CaptureWrites is what a literal or a +// literal nested in it may write of the literal's free variables; the host +// weak-defines it, and a strong def of a captured-and-written base is weak. +func TestDefUse_CaptureWrites(t *testing.T) { + fx := loadFlow(t) + // CaptureNested's f: the nested literal's x = y (y is f's own), and the + // fill(buf) write through buf; z is only read. + wantSet(t, "CaptureWrites(CaptureNested f)", dataflow.CaptureWrites(firstLit(t, "CaptureNested"), fx.info, kTest), "buf.*", "x") + wantSet(t, "CaptureWrites(Closure h)", dataflow.CaptureWrites(firstLit(t, "Closure"), fx.info, kTest)) + + wantDU(t, "CaptureNested 613:2", flowDU(t, "CaptureNested", "613:2"), ss("f"), ss("buf.*", "x"), ss("buf", "x", "z")) + // x, z := 0, 0: x is written by the literal, so its def is weak; z stays. + wantDU(t, "CaptureNested 612:2", flowDU(t, "CaptureNested", "612:2"), ss("z"), ss("x"), nil) + wantDU(t, "CaptureWrite 564:2", flowDU(t, "CaptureWrite", "564:2"), nil, ss("x"), ss("x")) + wantDU(t, "CaptureThenReassign 573:2", flowDU(t, "CaptureThenReassign", "573:2"), nil, ss("x"), nil) + wantDU(t, "OnceDo 588:2", flowDU(t, "OnceDo", "588:2"), nil, ss("cfg", "once.*"), ss("cfg", "once")) + // The deferred closure call node is the host, not the registration: + // err = wrapErr(err) writes err, and err.* through the interface argument. + wantDU(t, "DeferWrap 557:2", flowDU(t, "DeferWrap", "557:2"), nil, ss(":557:8"), nil) + wantDU(t, "DeferWrap 557:8", flowDU(t, "DeferWrap", "557:8"), nil, ss("err", "err.*"), ss(":557:8", "err")) + wantDU(t, "DeferWrap 555:2", flowDU(t, "DeferWrap", "555:2"), nil, ss("err"), nil) + // go func(){ x = <-ch }(): the go node is the host; ch, a channel, may + // hold a reference, so it weak-defines ch.* too (hostWrites). + wantDU(t, "CaptureGo 627:5", flowDU(t, "CaptureGo", "627:5"), nil, ss("ch.*", "x"), ss("ch", "x")) + wantDU(t, "CaptureGo 626:2", flowDU(t, "CaptureGo", "626:2"), nil, ss("x"), nil) +} + +// TestDefUse_AppendOntoIsShallow (#28): s = append(s, n) strong-defines s +// Shallow (it kills s, not s[*]) besides append's s[*] weak def. +func TestDefUse_AppendOntoIsShallow(t *testing.T) { + du := flowDU(t, "AppendKeepsElems", "604:2") + wantDU(t, "AppendKeepsElems 604:2", du, ss("s"), ss("s[*]"), ss("n", "s")) + wantSet(t, "AppendKeepsElems 604:2 Shallow", du.Shallow, "s") + // out := append(dst[:0], x): the LHS is not the destination. + wantSet(t, "AppendReuse 541:2 Shallow", flowDU(t, "AppendReuse", "541:2").Shallow) +} + +// TestDefUse_DeferOperandWrites (D1 gap 3): a node writing a base the +// registered defer fill(b) takes as an operand also weak-defines its pseudo +// path; the registration and the deferred call node are unchanged. +func TestDefUse_DeferOperandWrites(t *testing.T) { + wantDU(t, "DeferReassign 596:2", flowDU(t, "DeferReassign", "596:2"), nil, ss(":595:8", "b[*]"), ss("b", "s")) + wantDU(t, "DeferReassign 597:2", flowDU(t, "DeferReassign", "597:2"), ss("b"), ss(":595:8"), ss("o")) + wantDU(t, "DeferReassign 595:2", flowDU(t, "DeferReassign", "595:2"), nil, ss(":595:8"), ss("b")) + wantDU(t, "DeferReassign 595:8", flowDU(t, "DeferReassign", "595:8"), nil, ss("b.*"), ss(":595:8", "b.*")) + wantDU(t, "DeferReassign @entry", flowDU(t, "DeferReassign", dataflow.EntryKey), ss("b", "o", "s"), nil, nil) +} + +// TestMayHoldRef (D1 gap 2): the reference kinds, and structs and arrays +// whose fields or elements transitively may hold one, through named types, +// cycle-safe; a reference-free struct or array, a string and a basic do not. +func TestMayHoldRef(t *testing.T) { + i := types.Typ[types.Int] + sl := types.NewSlice(i) + st := func(ts ...types.Type) *types.Struct { + var fs []*types.Var + for n, ft := range ts { + fs = append(fs, types.NewField(0, nil, "f"+string(rune('a'+n)), ft, false)) + } + return types.NewStruct(fs, nil) + } + named := func(name string, u types.Type) *types.Named { + n := types.NewNamed(types.NewTypeName(0, nil, name, nil), nil, nil) + if u != nil { + n.SetUnderlying(u) + } + return n + } + empty := types.NewInterfaceType(nil, nil) + tp := types.NewTypeParam(types.NewTypeName(0, nil, "P", nil), empty) + + // A cycle through value types only (invalid Go, but go/types builds it in + // a package with type errors): A{b B}, B{a A, n int}. + a, b := named("A", nil), named("B", nil) + a.SetUnderlying(st(b)) + b.SetUnderlying(st(a, i)) + // The same cycle with a slice two levels down: C{d D}, D{c C, s []int}. + c, d := named("C", nil), named("D", nil) + c.SetUnderlying(st(d)) + d.SetUnderlying(st(c, sl)) + + for _, tc := range []struct { + name string + t types.Type + want bool + }{ + {"pointer", types.NewPointer(i), true}, + {"slice", sl, true}, + {"map", types.NewMap(i, i), true}, + {"chan", types.NewChan(types.SendRecv, i), true}, + {"func", types.NewSignatureType(nil, nil, nil, nil, nil, false), true}, + {"interface", empty, true}, + {"type parameter", tp, true}, + {"int", i, false}, + {"string", types.Typ[types.String], false}, + {"nil", nil, false}, + {"plain struct", st(i, types.Typ[types.String]), false}, + {"struct with slice", st(i, sl), true}, + {"named struct with slice", named("Cfg", st(sl)), true}, + {"struct nesting a struct with a map", st(st(i, types.NewMap(i, i))), true}, + {"array of int", types.NewArray(i, 2), false}, + {"array of slices", types.NewArray(sl, 2), true}, + {"array of structs with a pointer", types.NewArray(st(types.NewPointer(i)), 3), true}, + {"struct of arrays of int", st(types.NewArray(i, 4)), false}, + {"value cycle, no reference", a, false}, + {"value cycle, reference below", c, true}, + } { + if got := dataflow.MayHoldRef(tc.t); got != tc.want { + t.Errorf("MayHoldRef(%s) = %v, want %v", tc.name, got, tc.want) + } + } +} + +// TestDefUse_ValueStructMayHoldRef (D1 gap 2): a value argument or receiver +// that may hold a reference weak-defines a.* (whole base, never per field); +// a reference-free one is only read. +func TestDefUse_ValueStructMayHoldRef(t *testing.T) { + wantDU(t, "FillCfg 651:2", flowDU(t, "FillCfg", "651:2"), nil, ss("c.*"), ss("c")) + wantDU(t, "ValueRefs 659:2", flowDU(t, "ValueRefs", "659:2"), nil, ss("c.*"), ss("c")) + wantDU(t, "ValueRefs 660:2", flowDU(t, "ValueRefs", "660:2"), nil, nil, ss("p")) + wantDU(t, "ValueRefs 661:2", flowDU(t, "ValueRefs", "661:2"), nil, ss("a.*", "n.*"), ss("a", "b", "n")) + // A value receiver holding no reference is a copy: read only. + wantDU(t, "PlainRecv 698:2", flowDU(t, "PlainRecv", "698:2"), nil, nil, ss("p")) +} + +// TestDefUse_SatelliteParents (#28): a call in a folded for init, cond or +// post, a case list, a select comm clause (send value, receive operand) or a +// range key or expression is a satellite of the header or clause head that +// expression folds into, and carries an empty DefUse. +func TestDefUse_SatelliteParents(t *testing.T) { + _, byKey, _ := build(t, "SatParents") + want := map[string]string{ + "674:11": "674:2", "674:21": "674:2", "674:31": "674:2", // for init, cond, post + "678:7": "678:2", "678:13": "678:2", // case f(1), g(2): + "682:13": "682:2", // case ch <- f(s): + "683:14": "683:2", // case v := <-chanOf(ch): + "687:9": "687:2", "687:26": "687:2", // for xs[f(2)], x = range ids(xs) + } + calls := 0 + for k, n := range byKey { + if n.Kind != dataflow.KindCall { + continue + } + calls++ + p, ok := want[k] + if !ok { + t.Errorf("SatParents: unexpected call node %s (parent %q)", k, n.Parent) + continue + } + if n.Parent != p { + t.Errorf("SatParents: call %s Parent = %q, want %q", k, n.Parent, p) + } + wantDU(t, "SatParents "+k, flowDU(t, "SatParents", k), nil, nil, nil) + } + if calls != len(want) { + t.Errorf("SatParents: %d call nodes, want %d", calls, len(want)) + } + // The hosts carry their satellites' flow. + wantKind(t, "SatParents", byKey, "674:2", dataflow.KindLoop) + wantKind(t, "SatParents", byKey, "678:2", dataflow.KindStatement) + wantKind(t, "SatParents", byKey, "682:2", dataflow.KindStatement) + wantKind(t, "SatParents", byKey, "683:2", dataflow.KindStatement) + wantKind(t, "SatParents", byKey, "687:2", dataflow.KindLoop) + wantDU(t, "SatParents 682:2", flowDU(t, "SatParents", "682:2"), nil, ss("ch[*]"), ss("ch", "s")) + // <-chanOf(ch) receives from a call, not a path: the call reads ch and + // writes through it. + wantDU(t, "SatParents 683:2", flowDU(t, "SatParents", "683:2"), ss("v"), ss("ch.*"), ss("ch")) + // `=` range: weak key and value; ids(xs) writes through xs. + wantDU(t, "SatParents 687:2", flowDU(t, "SatParents", "687:2"), nil, ss("x", "xs.*", "xs[*]"), ss("xs")) +} diff --git a/internal/dataflow/export_test.go b/internal/dataflow/export_test.go index 0b9aab0..f6568bf 100644 --- a/internal/dataflow/export_test.go +++ b/internal/dataflow/export_test.go @@ -1,8 +1,21 @@ package dataflow +import ( + "go/ast" + "go/types" +) + // CanPanic exposes canPanic to the external test package. var CanPanic = canPanic // PostDominators exposes the immediate post-dominator map to the external // test package. var PostDominators = postDominators + +// DefUsesPreD1 is DefUses without the D1 rules (capture writes, +// deferred-operand writes, append's Shallow def, writes through value +// structs and arrays that may hold a reference): the L3 def/use sets before +// #22, for the gate that the D1 rules only add ddg edges. +func DefUsesPreD1(nodes []Node, info *types.Info, k int, entryBases []types.Object, fn ast.Node, keyOf map[ast.Node]string) map[string]DefUse { + return defUses(nodes, info, k, entryBases, fn, keyOf, nil) +} diff --git a/internal/dataflow/ir.go b/internal/dataflow/ir.go index 41b2fa2..b0895ff 100644 --- a/internal/dataflow/ir.go +++ b/internal/dataflow/ir.go @@ -39,8 +39,9 @@ const ( // - Key is the local key: "l:c" (v2.LocalID of the node's anchor position), // "l:c/2", … for co-positioned nodes, or EntryKey / ExitKey. // - Pos/End is the span of Stmt; NoPos for synthetic nodes. StartLC/EndLC -// are their [line, col] (fset.Position, so //line directives apply, as -// they do to the L1 call-site positions); zero for synthetic nodes. +// are their real [line, col] (fset.PositionFor(p, false): //line +// directives are not applied, as they are not to the L1 call-site +// positions); zero for synthetic nodes. // - Stmt is the AST node the body node stands for: // a statement (*ast.ExprStmt for a bare call, *ast.GoStmt for a `go` call), // a clause head (*ast.CaseClause / *ast.CommClause), @@ -75,9 +76,20 @@ type CdgEdge struct{ Src, Dst string } type DdgEdge struct{ Src, Dst, Var string } // Result is the L3 overlay of one callable body. +// +// DefUse and CallKeys are side tables for L4 (ruling R-P2), filled only by +// RunWithSideTables and dropped by Release; the emitters never read them: +// +// - DefUse is DefUses over Nodes, keyed by node key. +// - CallKeys maps the raw Lparen of every call the body pairs with an L1 +// call key to that key: the SSA join's call key (CallCommon.Pos() is the +// CallExpr's Lparen). type Result struct { Nodes []Node Cfg []CfgEdge Cdg []CdgEdge Ddg []DdgEdge + + DefUse map[string]DefUse + CallKeys map[token.Pos]string } diff --git a/internal/dataflow/reachdefs.go b/internal/dataflow/reachdefs.go index 97dd592..ac65bc1 100644 --- a/internal/dataflow/reachdefs.go +++ b/internal/dataflow/reachdefs.go @@ -1,6 +1,7 @@ package dataflow import ( + "go/types" "math/bits" "sort" ) @@ -16,9 +17,15 @@ import ( // - GEN is every def and weak def of the node. // - KILL is every def d, anywhere in the body, with p.Covers(d.path) for a // strong def p of the node (`x = …` kills x, x.f, x[*]; `x.f = …` kills -// x.f… but not x). Weak defs kill nothing. Strong defs carry no wildcard -// (DefUseOf), so a kill never over-reaches. A deferred call node only -// weak-defines (R11), so nothing is killed through the exit chain. +// x.f… but not x); for a Shallow strong def p, only the defs of p itself +// (`s = append(s, …)` kills s, not s[*]: #28). Weak defs kill nothing. +// Strong defs carry no wildcard (DefUseOf), so a kill never over-reaches. +// A deferred call node only weak-defines (R11), so nothing is killed +// through the exit chain. GEN is applied after KILL, so a node's own weak +// def (the s[*] of `s = append(s, …)`) survives its own strong def. +// Covers needs equal bases (and, for a nil base, equal spelling heads), +// so p is tested only against the defs of its kill bucket (#27 item 1): +// O(defs per base) per strong def, not O(defs in the body). // // A node's uses read IN — the defs reaching it before its own defs apply — // so `y = y + 1` in a loop links to itself across the back edge, never @@ -36,6 +43,69 @@ import ( // cfg or the DefUse lists. The output is deduplicated, sorted by (Src, Dst, // Var) and non-nil. Its prov (["ssa"] at L3) is added at emission. func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { + ddg, _ := solveDDG(nodes, cfg, du, nil) + return ddg +} + +// BuildDDGAlias is BuildDDG with a may-alias oracle (spec §4.10): the same +// reaching-definitions fixpoint, kills and use matching, plus one rule. A +// use u at n also links from a def d in IN(n) whose base differs from u's +// when mayAlias(d, u) holds: a write to d may change the memory a read of u +// observes. The oracle decides on the whole paths (which of their prefixes +// may share memory, and whether d writes through a reference or only +// rebinds it), so it is asymmetric: d is the write, u the read. +// +// The solver consults it only within the one-step offset window: for some +// i, j ∈ {0, 1}, d's steps after its first i and u's after its first j are +// prefix-compatible, in Covers' two-directional sense with the bases +// ignored (d.f covers u.f.g; u.* covers d.f; p.g against x.f.g at j = 1). +// An oracle accepting a pair outside the window is not asked about it. +// +// Alias never kills: a strong def through one base kills only the defs of +// its own base (BuildDDG's KILL), so a def that may be overwritten through +// an alias still reaches. The edge is d.node → n with Var = u.Spell, as for +// BuildDDG. The output is BuildDDG's plus the alias edges, deduplicated, +// sorted by (Src, Dst, Var) and non-nil; a nil mayAlias yields BuildDDG. +func BuildDDGAlias(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias func(a, b Path) bool) []DdgEdge { + ddg, alias := solveDDG(nodes, cfg, du, mayAlias) + if len(alias) == 0 { + return ddg + } + set := make(map[DdgEdge]bool, len(ddg)+len(alias)) + for _, e := range ddg { + set[e] = true + } + for _, e := range alias { + set[e.DdgEdge] = true + } + return sortedDdg(set) +} + +// AliasEdge is an edge only BuildDDGAlias's alias rule yields. Def is the +// base spelling of the def it links from when the edge leaves @entry (which +// defines several bases; the L4 port wiring starts the edge at that base's +// formal_in), else "". +type AliasEdge struct { + DdgEdge + Def string +} + +// BuildDDGSplit is BuildDDGAlias split in two: BuildDDG's edges, sorted by +// (Src, Dst, Var), and the edges the alias rule yields, sorted by (Src, Dst, +// Var, Def). An alias edge's DdgEdge is in the first list only when it +// leaves @entry from a def whose base is not the use's (Def ≠ the var's +// base): it then stands for another def than the base edge on that triple +// (fix round 2, N1). One fixpoint +// serves both: the L4 port wiring (package sdg) maps the first to +// reaching-defs port edges and the second to its points-to delta. +func BuildDDGSplit(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias func(a, b Path) bool) (ddg []DdgEdge, alias []AliasEdge) { + return solveDDG(nodes, cfg, du, mayAlias) +} + +// solveDDG is the fixpoint and use matching of BuildDDG and BuildDDGAlias: +// the BuildDDG edges, and (mayAlias non-nil) the alias-only edges, both +// sorted, the first non-nil. +func solveDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse, mayAlias func(a, b Path) bool) (ddg []DdgEdge, alias []AliasEdge) { g := newPdomGraph(nodes, cfg) n := len(g.keys) @@ -60,6 +130,18 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { } } + // Kill index (#27 item 1): p.Covers(d) and p == d both need the same + // Base, so a strong def is tested only against the defs of its bucket. + // A nil-base path (a pseudo path, an unresolved identifier) is bucketed + // by its spelling head: equal heads are necessary for Covers. + buckets := map[killBucket][]int{} + defBucket := make([]killBucket, len(defs)) + for i, d := range defs { + b := bucketOf(d.path) + defBucket[i] = b + buckets[b] = append(buckets[b], i) + } + words := (len(defs) + 63) / 64 newSet := func() []uint64 { return make([]uint64, words) } gen := make([][]uint64, n) @@ -69,9 +151,13 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { for _, i := range genOf[v] { gen[v][i/64] |= 1 << (i % 64) } + shallow := map[Path]bool{} + for _, p := range du[k].Shallow { + shallow[p] = true + } for _, p := range du[k].Defs { - for i, d := range defs { - if p.Covers(d.path) { + for _, i := range buckets[bucketOf(p)] { + if d := defs[i]; p == d.path || !shallow[p] && p.Covers(d.path) { kill[v][i/64] |= 1 << (i % 64) } } @@ -115,18 +201,27 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { } } + // Use matching is bucketed like the kills: interference (Covers either + // way) needs the same bucket, so a reaching def is tested only against + // the uses of its bucket — O(defs + matches) per node rather than + // O(defs × uses), which the L4 re-solve's global uses at hosts need. edges := map[DdgEdge]bool{} for v, k := range g.keys { uses := du[k].Uses if len(uses) == 0 { continue } + useBuckets := make(map[killBucket][]Path, len(uses)) + for _, u := range uses { + b := bucketOf(u) + useBuckets[b] = append(useBuckets[b], u) + } for w, x := range in[v] { for x != 0 { b := bits.TrailingZeros64(x) x &^= 1 << b d := defs[w*64+b] - for _, u := range uses { + for _, u := range useBuckets[defBucket[w*64+b]] { if d.path.Covers(u) || u.Covers(d.path) { edges[DdgEdge{Src: g.keys[d.node], Dst: k, Var: u.Spell}] = true } @@ -135,19 +230,142 @@ func BuildDDG(nodes []Node, cfg []CfgEdge, du map[string]DefUse) []DdgEdge { } } - res := make([]DdgEdge, 0, len(edges)) + if mayAlias != nil { + alias = aliasEdges(g.keys, du, in, func(i int) Path { return defs[i].path }, func(i int) string { return g.keys[defs[i].node] }, edges, mayAlias) + } + return sortedDdg(edges), alias +} + +// aliasEdges is BuildDDGAlias's alias rule over a solved fixpoint: at each +// node with uses, each reaching def is matched against each use of a +// different base within the offset window (inWindow), and the oracle +// decides. Edges already in base are skipped. The result is sorted (nil +// when empty). +func aliasEdges(keys []string, du map[string]DefUse, in [][]uint64, pathOf func(int) Path, nodeOf func(int) string, base map[DdgEdge]bool, mayAlias func(a, b Path) bool) []AliasEdge { + type use struct { + path Path + steps []string + } + steps := map[int][]string{} // def index → its steps; absent: not yet split + edges := map[AliasEdge]bool{} + for v, k := range keys { + var uses []use + for _, u := range du[k].Uses { + _, s := splitSpell(u.Spell) + uses = append(uses, use{u, s}) + } + if len(uses) == 0 { + continue + } + for w, x := range in[v] { + for x != 0 { + b := bits.TrailingZeros64(x) + x &^= 1 << b + i := w*64 + b + ds, ok := steps[i] + if !ok { + _, ds = splitSpell(pathOf(i).Spell) + steps[i] = ds + } + d := pathOf(i) + src := nodeOf(i) + def := "" + if src == EntryKey { + def = SpellBase(d.Spell) + } + for _, u := range uses { + if u.path.Base == d.Base || !inWindow(ds, u.steps) || !mayAlias(d, u.path) { + continue + } + // An edge from @entry whose def base is not the use's maps + // to another formal_in than the base edge on its triple: + // it is kept, and the wiring dedupes after mapping. + e := AliasEdge{DdgEdge{Src: src, Dst: k, Var: u.path.Spell}, def} + if !base[e.DdgEdge] || def != "" && def != SpellBase(u.path.Spell) { + edges[e] = true + } + } + } + } + } + if len(edges) == 0 { + return nil + } + res := make([]AliasEdge, 0, len(edges)) for e := range edges { res = append(res, e) } sort.Slice(res, func(i, j int) bool { a, b := res[i], res[j] - if a.Src != b.Src { - return a.Src < b.Src + if a.DdgEdge != b.DdgEdge { + return ddgLess(a.DdgEdge, b.DdgEdge) } - if a.Dst != b.Dst { - return a.Dst < b.Dst - } - return a.Var < b.Var + return a.Def < b.Def }) return res } + +// inWindow reports whether two step lists are prefix-compatible after +// dropping at most one leading step from each (BuildDDGAlias's window). +func inWindow(ds, us []string) bool { + for i := 0; i <= min(1, len(ds)); i++ { + for j := 0; j <= min(1, len(us)); j++ { + if StepsCompatible(ds[i:], us[j:]) { + return true + } + } + } + return false +} + +// StepsCompatible reports whether two step lists (the steps of two paths +// after their bases, or after a common offset) are prefix-compatible: one +// covers the other (Covers with the bases ignored). The alias oracle +// compares the steps below two may-alias prefixes with it. +func StepsCompatible(a, b []string) bool { return stepsCover(a, b) || stepsCover(b, a) } + +// PathSteps returns the steps of a path spelling after its base ("x.f[*]" +// → [".f", "[*]"]); a pseudo path has none. +func PathSteps(spell string) []string { + _, s := splitSpell(spell) + return s +} + +func ddgLess(a, b DdgEdge) bool { + if a.Src != b.Src { + return a.Src < b.Src + } + if a.Dst != b.Dst { + return a.Dst < b.Dst + } + return a.Var < b.Var +} + +func sortedDdg(edges map[DdgEdge]bool) []DdgEdge { + res := make([]DdgEdge, 0, len(edges)) + for e := range edges { + res = append(res, e) + } + sortDdg(res) + return res +} + +func sortDdg(res []DdgEdge) { + sort.Slice(res, func(i, j int) bool { return ddgLess(res[i], res[j]) }) +} + +// killBucket is the kill index key of a path: its Base, or for a nil Base +// its spelling head (splitSpell's base spelling; a pseudo path is all head). +// Two paths in different buckets never cover one another. +type killBucket struct { + base types.Object + head string +} + +func bucketOf(p Path) killBucket { + if p.Base != nil { + return killBucket{base: p.Base} + } + head, _ := splitSpell(p.Spell) + return killBucket{head: head} +} diff --git a/internal/dataflow/reachdefs_test.go b/internal/dataflow/reachdefs_test.go index 2868f0b..9f2de2b 100644 --- a/internal/dataflow/reachdefs_test.go +++ b/internal/dataflow/reachdefs_test.go @@ -7,6 +7,7 @@ import ( "path/filepath" "reflect" "runtime" + "slices" "sort" "testing" @@ -15,15 +16,10 @@ import ( func dd(src, dst, v string) dataflow.DdgEdge { return dataflow.DdgEdge{Src: src, Dst: dst, Var: v} } -// duMap is DefUseOf for every node of the pruned set, keyed by node Key — -// BuildDDG's input, computed the way the analyzer computes it. +// duMap is DefUses of the pruned set, keyed by node Key — BuildDDG's input, +// computed the way the analyzer computes it. func duMap(nodes []dataflow.Node, info *types.Info, fn ast.Node, keyOf map[ast.Node]string) map[string]dataflow.DefUse { - bases := dataflow.EntryBases(fn, info) - out := map[string]dataflow.DefUse{} - for _, n := range nodes { - out[n.Key] = dataflow.DefUseOf(n, info, kTest, bases, fn, keyOf) - } - return out + return dataflow.DefUses(nodes, info, kTest, dataflow.EntryBases(fn, info), fn, keyOf) } // ddgOf builds fn's pruned nodes, CFG, CDG and DDG from the flow fixture with @@ -340,12 +336,13 @@ func TestSlice_BackwardExact(t *testing.T) { // TestDDG_CorpusGate runs BuildDDG over every function and function literal // body in testdata/dataflow and testdata/chi: no panic, the output contract // (checkDDG: endpoints are pruned non-satellite nodes, sorted, deduped), two -// builds deep-equal, and the result independent of input order (nodes, CFG -// edges and every DefUse list reversed). +// builds deep-equal, the result independent of input order (nodes, CFG +// edges and every DefUse list reversed), and a superset of the DDG of the +// pre-D1 def/use sets (the D1 rules only add edges). func TestDDG_CorpusGate(t *testing.T) { _, thisFile, _, _ := runtime.Caller(0) root := filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata") - total, edges := 0, 0 + total, edges, oldEdges := 0, 0, 0 for _, dir := range []string{"dataflow", "chi"} { fset, cases := loadDiffCases(t, filepath.Join(root, dir)) for _, c := range cases { @@ -357,6 +354,14 @@ func TestDDG_CorpusGate(t *testing.T) { checkDDG(t, c.name, kept, got) edges += len(got) + // DefUseOf (callable facts derived per call) agrees with DefUses. + bases := dataflow.EntryBases(c.fn, c.info) + for _, n := range kept { + if one := dataflow.DefUseOf(n, c.info, kTest, bases, c.fn, keyOf); !reflect.DeepEqual(one, du[n.Key]) { + t.Errorf("%s %s: DefUseOf %+v != DefUses %+v", c.name, n.Key, one, du[n.Key]) + } + } + if again := dataflow.BuildDDG(kept, es, duMap(kept, c.info, c.fn, keyOf)); !reflect.DeepEqual(got, again) { t.Errorf("%s: two builds differ", c.name) } @@ -366,17 +371,31 @@ func TestDDG_CorpusGate(t *testing.T) { reverse(re) rdu := map[string]dataflow.DefUse{} for k, v := range du { - rdu[k] = dataflow.DefUse{Defs: reversed(v.Defs), WeakDefs: reversed(v.WeakDefs), Uses: reversed(v.Uses)} + rdu[k] = dataflow.DefUse{Defs: reversed(v.Defs), WeakDefs: reversed(v.WeakDefs), Uses: reversed(v.Uses), Shallow: reversed(v.Shallow)} } if perm := dataflow.BuildDDG(rn, re, rdu); !reflect.DeepEqual(got, perm) { t.Errorf("%s: DDG depends on input order", c.name) } + + // The D1 rules (#22, #28) only add edges: every edge of the + // pre-D1 def/use sets is still there. + have := map[dataflow.DdgEdge]bool{} + for _, g := range got { + have[g] = true + } + old := dataflow.BuildDDG(kept, es, dataflow.DefUsesPreD1(kept, c.info, kTest, dataflow.EntryBases(c.fn, c.info), c.fn, keyOf)) + for _, o := range old { + if !have[o] { + t.Errorf("%s: D1 rules dropped pre-D1 ddg %s→%s on %q", c.name, o.Src, o.Dst, o.Var) + } + } + oldEdges += len(old) } } if total < 100 || edges < 1000 { t.Errorf("ddg gate covered only %d bodies / %d edges", total, edges) } - t.Logf("ddg gate: %d bodies, %d edges", total, edges) + t.Logf("ddg gate: %d bodies, %d edges (%d pre-D1, all kept)", total, edges, oldEdges) } func reverse[T any](s []T) { @@ -435,11 +454,430 @@ func TestDDG_AppendWritesDstBackingArray(t *testing.T) { // chain is return 558:2 → 557:8 (err = wrapErr(err)) → 556:8 (observe(err)) // → @exit. A deferred call node never strong-defines, so err = wrapErr(nil) // (555:2) reaches the return, the rewriting closure and, through it, the -// observer at the end of the chain. +// observer at the end of the chain. The rewriting closure weak-defines its +// captured err (D1 gap 1), so its write reaches the observer too. func TestDDG_DeferredNamedResultWriteNeverKills(t *testing.T) { hasDDG(t, "DeferWrap", dd("555:2", "558:2", "err"), // the bare return reads the named result dd("555:2", "557:8", "err"), // the rewriting closure reads it at exit dd("555:2", "556:8", "err"), // and the def survives past it + dd("557:8", "556:8", "err"), // the rewrite's capture write (D1 gap 1) + ) +} + +// TestDDG_ClosureCaptureWrites (D1 gap 1): the node hosting a function literal +// weak-defines what the literal may write of its captures, and every strong +// def of a captured-and-written variable in the callable is weak, so the +// closure's write reaches the later read. +func TestDDG_ClosureCaptureWrites(t *testing.T) { + // func(){ x = 1 }() at 564:2 → return x. + hasDDG(t, "CaptureWrite", dd("564:2", "565:2", "x")) + // h := func(){ x = 1 } at 572:2; x = 2 (573:2) does not kill it. + hasDDG(t, "CaptureThenReassign", dd("572:2", "575:2", "x")) + // once.Do(func(){ cfg = load() }) at 588:2 → return cfg. + hasDDG(t, "OnceDo", dd("588:2", "589:2", "cfg")) + // f := func(){ func(){ x = y }() } at 613:2: the nested literal's write. + hasDDG(t, "CaptureNested", dd("613:2", "620:2", "x")) + // go func(){ x = <-ch }() at 627:5: the go node hosts the literal. + hasDDG(t, "CaptureGo", dd("627:5", "628:2", "x")) + // func(){ q := p; q.n = 1 }() at 716:2 writes through p by an alias: + // no path of p, but p may hold a reference, so the host weak-defines + // p.*, which reaches return p.n (found by metamorphic relation MR5). + hasDDG(t, "CaptureAliasWrite", dd("716:2", "717:2", "p.n")) +} + +// TestDDG_DeferOperandAfterReassign (D1 gap 3): in `defer fill(b); b[0] = s; +// b = o` the element write also weak-defines the defer's pseudo path, which +// b = o does not kill, so it reaches the deferred fill (595:8). +func TestDDG_DeferOperandAfterReassign(t *testing.T) { + hasDDG(t, "DeferReassign", dd("596:2", "595:8", ":595:8")) +} + +// TestDDG_AppendKeepsElementDefs (#28): s = append(s, n) may keep s's backing +// array, so it does not kill the earlier s[0] = 5. +func TestDDG_AppendKeepsElementDefs(t *testing.T) { + hasDDG(t, "AppendKeepsElems", dd("603:2", "605:2", "s[*]")) +} + +// TestDDG_ValueStructWithSliceIsWrittenThrough (D1 gap 2): fillCfg(c) with a +// value struct c whose field items is a slice may write c.items[*]; the call +// weak-defines c.*, which reaches the later read of c.items[0]. +func TestDDG_ValueStructWithSliceIsWrittenThrough(t *testing.T) { + wantDDG(t, "FillCfg", + dd(entry, "651:2", "c"), + dd(entry, "652:2", "c.items[*]"), + dd("651:2", "652:2", "c.items[*]"), + ) + // A value receiver over a slice field (c.reset()) and value arguments + // that may hold a reference (n nests one, a is an array of slices) are + // written through; p (ints only) and b ([2]int) are not. + hasDDG(t, "ValueRefs", + dd("659:2", "662:2", "c.items[*]"), + dd("661:2", "662:2", "n.in.items[*]"), + dd("661:2", "662:2", "a[*][*]"), ) + _, _, got := ddgOf(t, "ValueRefs") + for _, g := range got { + if g.Src != entry && (g.Var == "p.a" || g.Var == "b[*]") { + t.Errorf("ValueRefs: a reference-free value operand is written through: %v", g) + } + } +} + +// flatDef is a definition of the flat reference: a node and a path. +type flatDef struct { + Node string + Path dataflow.Path +} + +// flatReaching is a reference reaching-definitions fixpoint with the kill +// rule of BuildDDG tested against every definition of the body (no +// bucketing), on plain map sets: IN of every CFG node. +func flatReaching(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]dataflow.DefUse) map[string]map[flatDef]bool { + type def = flatDef + isNode := map[string]bool{} + for _, n := range nodes { + if n.Parent == "" { + isNode[n.Key] = true + } + } + pred := map[string][]string{} + for _, e := range cfg { + if isNode[e.Src] && isNode[e.Dst] { + pred[e.Dst] = append(pred[e.Dst], e.Src) + } + } + var all []def + for k := range isNode { + d := du[k] + for _, list := range [][]dataflow.Path{d.Defs, d.WeakDefs} { + for _, p := range list { + all = append(all, def{k, p}) + } + } + } + killed := func(k string, d def) bool { + shallow := map[dataflow.Path]bool{} + for _, p := range du[k].Shallow { + shallow[p] = true + } + for _, p := range du[k].Defs { + if p == d.Path || !shallow[p] && p.Covers(d.Path) { + return true + } + } + return false + } + in, out := map[string]map[def]bool{}, map[string]map[def]bool{} + for k := range isNode { + in[k], out[k] = map[def]bool{}, map[def]bool{} + } + for changed := true; changed; { + changed = false + for k := range isNode { + for _, p := range pred[k] { + for d := range out[p] { + in[k][d] = true + } + } + next := map[def]bool{} + for d := range in[k] { + if !killed(k, d) { + next[d] = true + } + } + for _, d := range all { + if d.Node == k { + next[d] = true + } + } + if len(next) != len(out[k]) { + changed = true + } else { + for d := range next { + if !out[k][d] { + changed = true + break + } + } + } + out[k] = next + } + } + return in +} + +// flatDDG is the reference DDG over flatReaching: each use links from each +// reaching def it interferes with. TestDDG_KillBucketsMatchFlat compares +// BuildDDG against it. +func flatDDG(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]dataflow.DefUse) []dataflow.DdgEdge { + return flatDDGAlias(nodes, cfg, du, nil) +} + +// anyBase is the broadest oracle: any two resolved bases may alias. The flat +// alias gate uses it so that the matching, not the oracle, is under test. +func anyBase(a, b dataflow.Path) bool { return a.Base != nil && b.Base != nil } + +// flatDDGAlias is BuildDDGAlias's reference, formulated independently: over +// flatReaching, a use u also links from a reaching def d of another base +// when mayAlias(d, u) and, dropping at most one leading step from each, d's +// remaining steps rebased onto u's base interfere with u's remaining steps +// rebased likewise (Covers either way). nil mayAlias is flatDDG. +func flatDDGAlias(nodes []dataflow.Node, cfg []dataflow.CfgEdge, du map[string]dataflow.DefUse, mayAlias func(a, b dataflow.Path) bool) []dataflow.DdgEdge { + edges := map[dataflow.DdgEdge]bool{} + for k, ins := range flatReaching(nodes, cfg, du) { + for d := range ins { + for _, u := range du[k].Uses { + if d.Path.Covers(u) || u.Covers(d.Path) { + edges[dataflow.DdgEdge{Src: d.Node, Dst: k, Var: u.Spell}] = true + continue + } + if mayAlias == nil || d.Path.Base == nil || u.Base == d.Path.Base { + continue + } + if flatWindow(d.Path, u) && mayAlias(d.Path, u) { + edges[dataflow.DdgEdge{Src: d.Node, Dst: k, Var: u.Spell}] = true + } + } + } + } + res := []dataflow.DdgEdge{} + for e := range edges { + res = append(res, e) + } + sort.Slice(res, func(i, j int) bool { return ddgLess(res[i], res[j]) }) + return res +} + +// flatWindow is the one-step offset window on spellings: both suffixes, +// each with up to one leading step dropped, rebased onto u's base. +func flatWindow(d, u dataflow.Path) bool { + tails := func(p dataflow.Path) []string { + suf := p.Spell[len(dataflow.SpellBase(p.Spell)):] + out := []string{suf} + if steps := dataflow.PathSteps(p.Spell); len(steps) > 0 { + out = append(out, suf[len(steps[0]):]) + } + return out + } + head := dataflow.SpellBase(u.Spell) + for _, ds := range tails(d) { + for _, us := range tails(u) { + a := dataflow.Path{Base: u.Base, Spell: head + ds} + b := dataflow.Path{Base: u.Base, Spell: head + us} + if a.Covers(b) || b.Covers(a) { + return true + } + } + } + return false +} + +// TestDDG_KillBucketsMatchFlat (#27 item 1): BuildDDG tests kills only within +// the bucket of a def's base (nil-base paths by spelling head); it equals the +// flat reference over the whole flow and chi corpus and over a hand-built +// line mixing shadowed bases, unresolved (nil-base) paths sharing a spelling +// with a real base, and pseudo paths. +func TestDDG_KillBucketsMatchFlat(t *testing.T) { + _, thisFile, _, _ := runtime.Caller(0) + root := filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata") + total, aliased := 0, 0 + for _, dir := range []string{"dataflow", "chi"} { + fset, cases := loadDiffCases(t, filepath.Join(root, dir)) + for _, c := range cases { + nodes, keyOf := dataflow.BuildNodes(fset, c.body, gateCallKeys(fset, c.body)) + kept, es := dataflow.BuildCFG(nodes, keyOf, c.body, c.info) + du := duMap(kept, c.info, c.fn, keyOf) + got, want := dataflow.BuildDDG(kept, es, du), flatDDG(kept, es, du) + if !reflect.DeepEqual(got, want) { + t.Errorf("%s: bucketed DDG (%d edges) != flat (%d edges)", c.name, len(got), len(want)) + } + total += len(got) + gotA, wantA := dataflow.BuildDDGAlias(kept, es, du, anyBase), flatDDGAlias(kept, es, du, anyBase) + if !reflect.DeepEqual(gotA, wantA) { + t.Errorf("%s: bucketed alias DDG (%d edges) != flat (%d edges)", c.name, len(gotA), len(wantA)) + } + aliased += len(gotA) - len(got) + } + } + if total < 1000 { + t.Errorf("kill-bucket gate covered only %d edges", total) + } + if aliased < 100 { + t.Errorf("alias gate covered only %d alias edges", aliased) + } + + // Hand-built: x (outer), x (shadowing inner, same spelling), the + // unresolved x and u (nil base), and a : pseudo path. + xo := types.NewVar(token.NoPos, nil, "x", types.Typ[types.Int]) + xi := types.NewVar(token.NoPos, nil, "x", types.Typ[types.Int]) + P := func(b types.Object, s ...string) []dataflow.Path { + var out []dataflow.Path + for _, sp := range s { + out = append(out, dataflow.Path{Base: b, Spell: sp}) + } + return out + } + cat := func(ps ...[]dataflow.Path) []dataflow.Path { + var out []dataflow.Path + for _, p := range ps { + out = append(out, p...) + } + return out + } + keys := []string{entry, "1:1", "1:2", "1:3", "1:4", "1:5", "1:6", exit} + var nodes []dataflow.Node + var cfg []dataflow.CfgEdge + for i, k := range keys { + nodes = append(nodes, dataflow.Node{Key: k}) + if i > 0 { + cfg = append(cfg, dataflow.CfgEdge{Src: keys[i-1], Dst: k, Kind: "fallthrough"}) + } + } + cfg = append(cfg, dataflow.CfgEdge{Src: "1:5", Dst: "1:2", Kind: "loop_back"}) + du := map[string]dataflow.DefUse{ + entry: {Defs: cat(P(xo, "x"), P(xi, "x.f"))}, + "1:1": {Defs: P(nil, "x", "u.f"), WeakDefs: P(nil, ":9:9", "u.*")}, + "1:2": {Defs: P(xi, "x"), Uses: cat(P(xo, "x"), P(nil, "x", "u"))}, + "1:3": {Defs: P(nil, "u.f"), Uses: cat(P(xi, "x.f"), P(nil, "u.f.g"))}, + "1:4": {Defs: cat(P(xo, "x"), P(nil, ":9:9", "u")), Shallow: P(xo, "x"), WeakDefs: P(xo, "x[*]")}, + "1:5": {Uses: cat(P(xo, "x[*]"), P(nil, "x", "u", ":9:9"), P(xi, "x"))}, + "1:6": {Defs: P(nil, "u"), Uses: cat(P(xo, "x"), P(nil, "u.f", ":9:9"))}, + } + got, want := dataflow.BuildDDG(nodes, cfg, du), flatDDG(nodes, cfg, du) + if !reflect.DeepEqual(got, want) { + t.Errorf("hand-built kills:\n got %v\nwant %v", got, want) + } + if gotA, wantA := dataflow.BuildDDGAlias(nodes, cfg, du, anyBase), flatDDGAlias(nodes, cfg, du, anyBase); !reflect.DeepEqual(gotA, wantA) { + t.Errorf("hand-built alias kills:\n got %v\nwant %v", gotA, wantA) + } + // The unresolved u (1:4) kills u.f of 1:1 and 1:3 by spelling head: only + // 1:4's own u reaches 1:6's u.f. + for _, g := range got { + if g.Dst == "1:6" && g.Var == "u.f" && g.Src != "1:4" { + t.Errorf("hand-built kills: %v survives the unresolved u = … at 1:4", g) + } + } +} + +// TestDDGAlias_Rule pins BuildDDGAlias on a hand-built line, with p and q +// pointers of one type and v an int, under an oracle that, like the L4 +// typeOracle, lets only a write through a pointer (a def with a step) alias: +// +// @entry defines p, q, v +// 1:1 p.f = v +// 1:2 q.f = v; q.g.h = v (strong defs through q: kill q.f…, q.g.h… only) +// 1:3 use p.f, q.f.g, q, p.g, p.h.f.k, p.k.m +// 1:4 q = nil (a bare def: rebinds q, writes no aliased memory) +// 1:5 use p.f +// +// 1:1's p.f reaches 1:2's bare use q. At 1:3, by alias: 1:2's q.f may have +// written p.f, yet 1:1's p.f still reaches by its own base (alias never +// kills); 1:1's p.f reaches q.f.g (its suffix covers) and the bare use q. +// The one-step offset window admits p.g and p.k.m against q.f (one leading +// step dropped from each leaves [] against [] or [.m]: q.f's storage may be +// what p.k points to), and p.h.f.k against q.f (.h dropped); the oracle +// decides them (yes). p.k.m against q.g.h is outside the window ([.h] +// against [.m] at best): the oracle is never asked about that pair. At 1:5 nothing aliases: 1:4 +// killed q's defs by its own base, and its bare def of q aliases nothing, +// like @entry's. +func TestDDGAlias_Rule(t *testing.T) { + ptr := types.NewPointer(types.NewStruct(nil, nil)) + p := types.NewVar(token.NoPos, nil, "p", ptr) + q := types.NewVar(token.NoPos, nil, "q", ptr) + v := types.NewVar(token.NoPos, nil, "v", types.Typ[types.Int]) + P := func(b types.Object, s string) dataflow.Path { return dataflow.Path{Base: b, Spell: s} } + keys := []string{entry, "1:1", "1:2", "1:3", "1:4", "1:5", exit} + var nodes []dataflow.Node + var cfg []dataflow.CfgEdge + for i, k := range keys { + nodes = append(nodes, dataflow.Node{Key: k}) + if i > 0 { + cfg = append(cfg, dataflow.CfgEdge{Src: keys[i-1], Dst: k, Kind: "fallthrough"}) + } + } + du := map[string]dataflow.DefUse{ + entry: {Defs: []dataflow.Path{P(p, "p"), P(q, "q"), P(v, "v")}}, + "1:1": {Defs: []dataflow.Path{P(p, "p.f")}, Uses: []dataflow.Path{P(p, "p"), P(v, "v")}}, + "1:2": {Defs: []dataflow.Path{P(q, "q.f"), P(q, "q.g.h")}, Uses: []dataflow.Path{P(q, "q"), P(v, "v")}}, + "1:3": {Uses: []dataflow.Path{P(p, "p.f"), P(q, "q.f.g"), P(q, "q"), P(p, "p.g"), P(p, "p.h.f.k"), P(p, "p.k.m")}}, + "1:4": {Defs: []dataflow.Path{P(q, "q")}}, + "1:5": {Uses: []dataflow.Path{P(p, "p.f")}}, + } + asked := map[[2]string]bool{} + mayAlias := func(a, b dataflow.Path) bool { + asked[[2]string{a.Spell, b.Spell}] = true + return a.Base != v && b.Base != v && a.Base != nil && b.Base != nil && len(dataflow.PathSteps(a.Spell)) > 0 + } + base := dataflow.BuildDDG(nodes, cfg, du) + got := dataflow.BuildDDGAlias(nodes, cfg, du, mayAlias) + ddg, alias := dataflow.BuildDDGSplit(nodes, cfg, du, mayAlias) + if !reflect.DeepEqual(ddg, base) { + t.Errorf("split ddg != BuildDDG:\n got %v\nwant %v", ddg, base) + } + var plain []dataflow.DdgEdge + for _, e := range alias { + if e.Def != "" { + t.Errorf("%v: Def set off @entry", e) + } + plain = append(plain, e.DdgEdge) + } + want := []dataflow.DdgEdge{ + dd("1:1", "1:2", "q"), // the bare use q at 1:2 reads the object p.f is in + dd("1:1", "1:3", "q"), + dd("1:1", "1:3", "q.f.g"), + dd("1:2", "1:3", "p.f"), + dd("1:2", "1:3", "p.g"), + dd("1:2", "1:3", "p.h.f.k"), + dd("1:2", "1:3", "p.k.m"), + } + if !reflect.DeepEqual(plain, want) { + t.Errorf("alias edges:\n got %v\nwant %v", plain, want) + } + if asked[[2]string{"q.g.h", "p.k.m"}] { + t.Errorf("oracle asked about q.g.h, p.k.m outside the window") + } + if !asked[[2]string{"q.f", "p.k.m"}] { + t.Errorf("oracle not asked about q.f, p.k.m inside the window") + } + for _, e := range base { + if !slices.Contains(got, e) { + t.Errorf("BuildDDGAlias lost BuildDDG edge %v", e) + } + } + if len(got) != len(base)+len(alias) { + t.Errorf("BuildDDGAlias = %d edges, want BuildDDG %d + alias %d", len(got), len(base), len(alias)) + } + for _, e := range []dataflow.DdgEdge{dd("1:1", "1:3", "p.f"), dd("1:1", "1:5", "p.f")} { + if !slices.Contains(base, e) { + t.Errorf("%v: q.f = v killed p.f across bases: %v", e, base) + } + } + if nilA := dataflow.BuildDDGAlias(nodes, cfg, du, nil); !reflect.DeepEqual(nilA, base) { + t.Errorf("nil oracle: got %v, want BuildDDG %v", nilA, base) + } +} + +// TestDDGAlias_EntryDefKept (fix round 2, N1): @entry defines both r (a +// value whose address may be taken) and q (a pointer); 1:1 reads q.f. The +// base edge @entry → 1:1 (q.f) is q's own; the alias edge from r's def on +// the same triple is a different def (another formal_in at L4), so it is +// kept with Def "r". An oracle aliasing q with itself adds nothing. +func TestDDGAlias_EntryDefKept(t *testing.T) { + r := types.NewVar(token.NoPos, nil, "r", types.NewStruct(nil, nil)) + q := types.NewVar(token.NoPos, nil, "q", types.NewPointer(r.Type())) + nodes := []dataflow.Node{{Key: entry}, {Key: "1:1"}, {Key: exit}} + cfg := []dataflow.CfgEdge{{Src: entry, Dst: "1:1", Kind: "fallthrough"}, {Src: "1:1", Dst: exit, Kind: "fallthrough"}} + du := map[string]dataflow.DefUse{ + entry: {Defs: []dataflow.Path{{Base: r, Spell: "r"}, {Base: q, Spell: "q"}}}, + "1:1": {Uses: []dataflow.Path{{Base: q, Spell: "q.f"}}}, + } + ddg, alias := dataflow.BuildDDGSplit(nodes, cfg, du, func(a, b dataflow.Path) bool { return true }) + if want := []dataflow.DdgEdge{dd(entry, "1:1", "q.f")}; !reflect.DeepEqual(ddg, want) { + t.Errorf("ddg = %v, want %v", ddg, want) + } + want := []dataflow.AliasEdge{{DdgEdge: dd(entry, "1:1", "q.f"), Def: "r"}} + if !reflect.DeepEqual(alias, want) { + t.Errorf("alias = %+v, want %+v", alias, want) + } } diff --git a/internal/dataflow/run.go b/internal/dataflow/run.go index fb3e01e..d36f48b 100644 --- a/internal/dataflow/run.go +++ b/internal/dataflow/run.go @@ -24,14 +24,22 @@ type L3Key struct{ Path, Signature string } // // - Fn is the *ast.FuncDecl or *ast.FuncLit; a closure is its own unit. // - Info is the type info of Fn's package (nil degrades: no def/use). -// - Calls is the callable's L1 call-node keys in L1 recording order -// (v2.CallBodyKeys over its call sites). BuildNodes pairs them with the -// body's calls by index (ruling R5); Run checks the counts first. +// - Calls is the callable's L1 call nodes: each L1 call-node key +// (v2.CallBodyKeys over its call sites) with its site's CallExpr.Lparen. +// BuildNodes pairs them with the body's calls by Lparen, so their order +// does not matter. type Unit struct { Key L3Key Fn ast.Node Info *types.Info - Calls []string + Calls []UnitCall +} + +// UnitCall is one L1 call node of a unit: its key and the raw Lparen of the +// call expression it was recorded from. +type UnitCall struct { + Key string + Lparen token.Pos } // Run builds the L3 overlay of every unit: the pruned body nodes, CFG, CDG @@ -42,10 +50,24 @@ type Unit struct { // // A unit that panics in any stage is skipped the same way (runOne). // -// A unit whose L1 call-key count differs from its body's recorded call count -// gets no result and a warning on stderr: pairing by index would mis-key its -// call nodes, so its body stays at L1 rather than being silently wrong. +// A unit with a body call that no L1 call key names (by Lparen) gets no +// result and a warning on stderr: a call node without its L1 key would be +// silently wrong, so its body stays at L1. A count mismatch between the L1 +// keys and the body's recorded calls (RecordedCallCount) is a cross-check +// only: it warns, and the unit is still built when every call is paired. func Run(fset *token.FileSet, units []Unit, k, jobs int) map[L3Key]Result { + return run(fset, units, k, jobs, false) +} + +// RunWithSideTables is Run that also keeps each result's DefUse and CallKeys +// (the -a 4 input). Below -a 4 nothing reads them, so Run drops them with the +// rest of each callable's working state instead of holding them all until +// Release. +func RunWithSideTables(fset *token.FileSet, units []Unit, k, jobs int) map[L3Key]Result { + return run(fset, units, k, jobs, true) +} + +func run(fset *token.FileSet, units []Unit, k, jobs int, keep bool) map[L3Key]Result { if jobs < 1 { jobs = runtime.GOMAXPROCS(0) } @@ -57,7 +79,7 @@ func Run(fset *token.FileSet, units []Unit, k, jobs int) map[L3Key]Result { go func() { defer wg.Done() for i := range next { - results[i] = runOne(fset, units[i], k) + results[i] = runOne(fset, units[i], k, keep) } }() } @@ -80,7 +102,7 @@ func Run(fset *token.FileSet, units []Unit, k, jobs int) map[L3Key]Result { // CFG, def-use or DDG builders) is recovered: that callable stays at L1 with a // warning (and, at -vv, the panic's stack), and the rest of the run — and the // L1/L2 output — is unaffected. -func runOne(fset *token.FileSet, u Unit, k int) (res *Result) { +func runOne(fset *token.FileSet, u Unit, k int, keep bool) (res *Result) { defer func() { if r := recover(); r != nil { utils.Warn("L3 skipped for %s (%s): %v", u.Key.Signature, u.Key.Path, r) @@ -92,23 +114,32 @@ func runOne(fset *token.FileSet, u Unit, k int) (res *Result) { if body == nil { return nil } - if n := RecordedCallCount(body); n != len(u.Calls) { - utils.Warn("L3 skipped for %s (%s): %d L1 call keys for %d calls in the body", u.Key.Signature, u.Key.Path, len(u.Calls), n) + paired, recorded := pairCalls(body, u.Calls) + missing := recorded - len(paired) + if recorded != len(u.Calls) { + utils.Warn("L3 for %s (%s): %d L1 call keys for %d calls in the body", u.Key.Signature, u.Key.Path, len(u.Calls), recorded) + } + if missing > 0 { + utils.Warn("L3 skipped for %s (%s): %d calls in the body have no L1 call key", u.Key.Signature, u.Key.Path, missing) return nil } nodes, keyOf := BuildNodes(fset, body, u.Calls) kept, cfg := BuildCFG(nodes, keyOf, body, u.Info) - bases := EntryBases(u.Fn, u.Info) - du := make(map[string]DefUse, len(kept)) - for _, n := range kept { - du[n.Key] = DefUseOf(n, u.Info, k, bases, u.Fn, keyOf) - } - return &Result{ + du := DefUses(kept, u.Info, k, EntryBases(u.Fn, u.Info), u.Fn, keyOf) + res = &Result{ Nodes: kept, Cfg: cfg, Cdg: BuildCDG(kept, cfg), Ddg: BuildDDG(kept, cfg, du), } + if keep { + res.DefUse = du + res.CallKeys = make(map[token.Pos]string, len(paired)) + for call, key := range paired { + res.CallKeys[call.Lparen] = key + } + } + return res } func funcBody(fn ast.Node) *ast.BlockStmt { @@ -120,3 +151,25 @@ func funcBody(fn ast.Node) *ast.BlockStmt { } return nil } + +// Release drops the result's references into the AST (each Node.Stmt) and +// its L4 side tables (DefUse, CallKeys), so a finished run no longer keeps +// the parsed packages reachable. Every other field (keys, spans, edges) +// stays: they are all the emitter reads. Call it once the last consumer is +// done: dataflow.Run below -a 4, sdg.Run at -a 4. A Result held in a map is +// a copy: release a map with ReleaseAll. +func (r *Result) Release() { + for i := range r.Nodes { + r.Nodes[i].Stmt = nil + } + r.DefUse = nil + r.CallKeys = nil +} + +// ReleaseAll releases every result of m in place. +func ReleaseAll(m map[L3Key]Result) { + for k, r := range m { + r.Release() + m[k] = r + } +} diff --git a/internal/dataflow/run_test.go b/internal/dataflow/run_test.go index 0e723c8..619963f 100644 --- a/internal/dataflow/run_test.go +++ b/internal/dataflow/run_test.go @@ -11,6 +11,7 @@ import ( "io" "os" "path/filepath" + "reflect" "runtime" "strings" "testing" @@ -49,7 +50,7 @@ func TestRun_JobsDeterminism(t *testing.T) { if len(l3) != len(units) { t.Errorf("-j %d: %d results for %d units", jobs, len(l3), len(units)) } - data, err := json.Marshal(v2emit.Emit(app, "chi", dir, 3, "det", 3, l3)) + data, err := json.Marshal(v2emit.Emit(app, "chi", dir, 3, "det", 3, l3, nil)) if err != nil { t.Fatal(err) } @@ -68,9 +69,9 @@ func TestRun_JobsDeterminism(t *testing.T) { } // lineDirectiveSrc maps its body to a different (virtual) file and line with -// a //line directive. L1 records call positions through fset.Position, which -// honours the directive, so the call keys name virtual lines; pairing them -// with the AST calls by recording index must still work (ruling R5). +// a //line directive. Positions are real file positions (PositionFor(p, +// false)), so the call keys and spans name the real lines 7–11 of p.go, not +// the virtual 900+, and calls pair with their keys by Lparen (#26). const lineDirectiveSrc = `package p func f(x int) int { return x } @@ -114,20 +115,22 @@ func funcDecl(t *testing.T, f *ast.File, name string) *ast.FuncDecl { return nil } -// TestRun_LineDirective: the call keys of a body under a //line directive -// land on the call nodes, and spans carry the directive-adjusted positions -// the keys name. +// TestRun_LineDirective: under a //line directive the call keys and spans +// are real positions, not the directive's virtual ones, and every call key +// lands on its call node. func TestRun_LineDirective(t *testing.T) { fset, f, info := parseCheck(t, lineDirectiveSrc) g := funcDecl(t, f, "g") - keys := gateCallKeys(fset, g.Body) - // Virtual positions; the file has only 12 real lines, so rebuilding a - // token.Pos from them (File.LineStart(900)) would panic. - if strings.Join(keys, ",") != "900:7,900:9,902:3" { - t.Fatalf("call keys = %v, want the virtual 900:7, 900:9, 902:3", keys) + calls := gateCallKeys(fset, g.Body) + var keys []string + for _, c := range calls { + keys = append(keys, c.Key) + } + if strings.Join(keys, ",") != "7:7,7:9,9:3" { + t.Fatalf("call keys = %v, want the real 7:7, 7:9, 9:3", keys) } key := dataflow.L3Key{Path: "p.go", Signature: "p.g"} - out := dataflow.Run(fset, []dataflow.Unit{{Key: key, Fn: g, Info: info, Calls: keys}}, 3, 1) + out := dataflow.Run(fset, []dataflow.Unit{{Key: key, Fn: g, Info: info, Calls: calls}}, 3, 1) res, ok := out[key] if !ok { t.Fatal("no result for g") @@ -141,29 +144,68 @@ func TestRun_LineDirective(t *testing.T) { t.Errorf("call key %s has no call node", k) } } - if n := byKey["901:2"]; n.Kind != dataflow.KindBranch || n.StartLC != [2]int{901, 2} || n.EndLC != [2]int{903, 3} { - t.Errorf("if node = %+v, want branch spanning 901:2–903:3", n) + if n := byKey["8:2"]; n.Kind != dataflow.KindBranch || n.StartLC != [2]int{8, 2} || n.EndLC != [2]int{10, 3} { + t.Errorf("if node = %+v, want branch spanning 8:2–10:3", n) } } -// TestRun_CallCountMismatchSkips: a unit whose L1 key list does not match -// its body's calls gets no result (never a silently mis-keyed one); other -// units are unaffected. -func TestRun_CallCountMismatchSkips(t *testing.T) { +// TestRun_UnpairedCallSkips: a unit with a body call that no L1 key names +// (by Lparen) gets no result (never a silently mis-keyed one); other units +// are unaffected. +func TestRun_UnpairedCallSkips(t *testing.T) { fset, f, info := parseCheck(t, lineDirectiveSrc) g, fn := funcDecl(t, f, "g"), funcDecl(t, f, "f") bad := dataflow.L3Key{Path: "p.go", Signature: "p.g"} good := dataflow.L3Key{Path: "p.go", Signature: "p.f"} - out := dataflow.Run(fset, []dataflow.Unit{ - {Key: bad, Fn: g, Info: info, Calls: []string{"900:7"}}, // 1 key, 3 calls - {Key: good, Fn: fn, Info: info, Calls: nil}, - }, 3, 2) + var out map[dataflow.L3Key]dataflow.Result + stderr := captureStderr(t, func() { + out = dataflow.Run(fset, []dataflow.Unit{ + {Key: bad, Fn: g, Info: info, Calls: gateCallKeys(fset, g.Body)[:1]}, // 1 key, 3 calls + {Key: good, Fn: fn, Info: info, Calls: nil}, + }, 3, 2) + }) if _, ok := out[bad]; ok { - t.Error("mismatched unit produced a result") + t.Error("unit with unpaired calls produced a result") } if r, ok := out[good]; !ok || len(r.Cfg) == 0 { t.Errorf("good unit result = %+v, %v", r, ok) } + if !strings.Contains(stderr, "L3 skipped for p.g (p.go): 2 calls in the body have no L1 call key") { + t.Errorf("want the unpaired-call warning; got:\n%s", stderr) + } +} + +// TestRun_CallCountMismatchWarnsOnly: the L1-key count is a cross-check, not +// the pairing: a surplus key (here one naming no call of the body) warns, and +// the unit is still built with every call keyed by its Lparen. +func TestRun_CallCountMismatchWarnsOnly(t *testing.T) { + fset, f, info := parseCheck(t, lineDirectiveSrc) + g := funcDecl(t, f, "g") + key := dataflow.L3Key{Path: "p.go", Signature: "p.g"} + calls := append(gateCallKeys(fset, g.Body), dataflow.UnitCall{Key: "99:1", Lparen: token.NoPos}) + var out map[dataflow.L3Key]dataflow.Result + stderr := captureStderr(t, func() { + out = dataflow.Run(fset, []dataflow.Unit{{Key: key, Fn: g, Info: info, Calls: calls}}, 3, 1) + }) + if !strings.Contains(stderr, "4 L1 call keys for 3 calls in the body") { + t.Errorf("want the count cross-check warning; got:\n%s", stderr) + } + res, ok := out[key] + if !ok { + t.Fatal("count mismatch alone skipped the unit") + } + byKey := map[string]dataflow.Node{} + for _, n := range res.Nodes { + byKey[n.Key] = n + } + for _, c := range calls[:3] { + if n, ok := byKey[c.Key]; !ok || n.Kind != dataflow.KindCall { + t.Errorf("call key %s has no call node", c.Key) + } + } + if _, ok := byKey["99:1"]; ok { + t.Error("surplus key produced a node") + } } // TestRun_ResultContract: on the flow fixture every result has @entry and @@ -326,9 +368,9 @@ func captureStderr(t *testing.T, fn func()) string { return <-done } -// TestRun_FlowFixtureHasNoCallCountMismatch: L1's call walk (buildCallSites) -// and L3's (recordedCalls) agree on every flow-fixture body — including calls -// in a `go` statement's arguments — so no unit is skipped with a warning. +// TestRun_FlowFixtureHasNoCallCountMismatch: every L1 call site of every +// flow-fixture body pairs with its call by Lparen — including calls in a `go` +// statement's arguments — so no unit warns or is skipped. func TestRun_FlowFixtureHasNoCallCountMismatch(t *testing.T) { _, thisFile, _, _ := runtime.Caller(0) dir := filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "dataflow") @@ -340,10 +382,113 @@ func TestRun_FlowFixtureHasNoCallCountMismatch(t *testing.T) { units := b.L3Units(st) var out map[dataflow.L3Key]dataflow.Result stderr := captureStderr(t, func() { out = dataflow.Run(b.Fset(), units, 3, 4) }) - if strings.Contains(stderr, "L1 call keys for") { - t.Errorf("call-count mismatch warnings:\n%s", stderr) + if strings.Contains(stderr, "L1 call keys for") || strings.Contains(stderr, "have no L1 call key") { + t.Errorf("call pairing warnings:\n%s", stderr) } if len(out) != len(units) { t.Errorf("%d results for %d units", len(out), len(units)) } } + +// TestResult_ReleaseDropsASTs: Release nils every Node.Stmt, and the -a 3 +// JSON emitted after it is byte-identical to the JSON emitted before it (no +// emitter reads Stmt). +func TestResult_ReleaseDropsASTs(t *testing.T) { + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "dataflow")) + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatal(err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + out := dataflow.Run(b.Fset(), b.L3Units(st), 3, 4) + emit := func() []byte { + data, err := json.Marshal(v2emit.Emit(app, "dataflow", dir, 3, "rel", 3, out, nil)) + if err != nil { + t.Fatal(err) + } + return data + } + before := emit() + stmts := 0 + for _, r := range out { + for _, n := range r.Nodes { + if n.Stmt != nil { + stmts++ + } + } + } + dataflow.ReleaseAll(out) + if stmts == 0 { + t.Fatal("no node had a Stmt before Release") + } + for key, r := range out { + for _, n := range r.Nodes { + if n.Stmt != nil { + t.Errorf("%v: node %s keeps its Stmt after Release", key, n.Key) + } + } + } + if !bytes.Equal(before, emit()) { + t.Error("-a 3 JSON changed after Release") + } + for key, r := range out { + if r.DefUse != nil || r.CallKeys != nil { + t.Errorf("%v: DefUse / CallKeys kept after Release", key) + } + } +} + +// TestRun_KeepsDefUseAndCallKeys: with side tables (-a 4), until Release, a +// result keeps its nodes' def/use sets (keyed by node key) and its call keys +// by Lparen, which the L4 SSA join and the go/ssa differential read (ruling +// R-P2). Plain Run (-a 3) keeps neither, and its graphs are the same. +func TestRun_KeepsDefUseAndCallKeys(t *testing.T) { + fset, f, info := parseCheck(t, lineDirectiveSrc) + g := funcDecl(t, f, "g") + calls := gateCallKeys(fset, g.Body) + key := dataflow.L3Key{Path: "p.go", Signature: "p.g"} + units := []dataflow.Unit{{Key: key, Fn: g, Info: info, Calls: calls}} + res, ok := dataflow.RunWithSideTables(fset, units, 3, 1)[key] + if !ok { + t.Fatal("no result for g") + } + plain := dataflow.Run(fset, units, 3, 1)[key] + if plain.DefUse != nil || plain.CallKeys != nil { + t.Error("Run kept the L4 side tables") + } + if !reflect.DeepEqual(plain.Cfg, res.Cfg) || !reflect.DeepEqual(plain.Ddg, res.Ddg) || !reflect.DeepEqual(plain.Cdg, res.Cdg) { + t.Error("side tables changed the graphs") + } + if len(res.CallKeys) != len(calls) { + t.Fatalf("CallKeys = %v, want one per call (%d)", res.CallKeys, len(calls)) + } + for _, c := range calls { + if res.CallKeys[c.Lparen] != c.Key { + t.Errorf("CallKeys[%v] = %q, want %q", c.Lparen, res.CallKeys[c.Lparen], c.Key) + } + } + if len(res.DefUse) != len(res.Nodes) { + t.Fatalf("DefUse has %d entries for %d nodes", len(res.DefUse), len(res.Nodes)) + } + for _, n := range res.Nodes { + if _, ok := res.DefUse[n.Key]; !ok { + t.Errorf("node %s has no DefUse", n.Key) + } + } + res.Release() + if res.DefUse != nil || res.CallKeys != nil { + t.Error("Release kept DefUse / CallKeys") + } + for _, n := range res.Nodes { + if n.Stmt != nil { + t.Errorf("node %s keeps its Stmt after Release", n.Key) + } + } +} diff --git a/internal/metamorphic/compare.go b/internal/metamorphic/compare.go new file mode 100644 index 0000000..d4e930e --- /dev/null +++ b/internal/metamorphic/compare.go @@ -0,0 +1,1301 @@ +package metamorphic + +// The normalized view of an analysis and the three oracles over it. +// +// A View keys callables by Identity: the signature, which stays the same when +// a declaration moves between files (MR3), plus the declaring file for the +// per-file `init` and its closures. It does not key them by can:// id, which +// names the module. Body keys are local ("l:c[/n][/actual_*]", "@entry", …). +// Edges are strings "kind|src|dst|label[|prov]" over local keys. The +// application edges (call_graph, param_in, param_out) are over Node strings +// "identity|key". +// +// Map carries a view through a transform's position map into the +// transformed module's coordinates, so the oracles compare like with like: +// +// - Exact: the mapped original equals the transformed view. +// - Delta: the mapped original, with each expected Delta applied, equals +// the transformed view. So after − before and before − after are exactly +// the expected additions and removals, and nothing outside the delta's +// callable changed. +// - Reach: every sink reachable from a sampled source in the original is +// still reachable, through the mapped nodes, in the transformed view. The +// check is one-way: over-approximation may add flows. + +import ( + "encoding/json" + "fmt" + "io" + "maps" + "math/bits" + "math/rand/v2" + "regexp" + "slices" + "sort" + "strconv" + "strings" + "sync" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// View is the normalized analysis. +type View struct { + Calls map[string]*CV + // App holds "cg|src|dst|prov" over identities, and "pin|src|dst|var" / + // "pout|src|dst|var" over Node strings. + App map[string]bool + + // callerOnce guards callerOf, which is derived from App on first use + // (called, Callers). A View is shared by parallel relations, so its + // lazily built state must be built once. + callerOnce sync.Once + callerOf map[string][]string // callee identity → caller identities, by param_in +} + +// CV is one callable (or a library stand-in, Kind "external"). +type CV struct { + Ident string + File string // declaring file (source_file, else the module) + Line int // the declaration's first line + Kind string + L3 bool // has a cfg (an L3 result) + // Body maps each body key to "kind|parent|of|flags|callee". + Body map[string]string + Edges map[string]bool +} + +func (c *CV) clone() *CV { + n := *c + n.Body, n.Edges = maps.Clone(c.Body), maps.Clone(c.Edges) + return &n +} + +// Node is the application-scope name of body key key of callable ident. +func Node(ident, key string) string { return ident + "|" + key } + +func splitNode(n string) (string, string) { + i := strings.LastIndexByte(n, '|') + return n[:i], n[i+1:] +} + +// externalIdent is the Identity of an @external id: "@ext:" + its tail. +func externalIdent(id string) (string, bool) { + i := strings.Index(id, "/@external/") + if i < 0 { + return "", false + } + return "@ext:" + id[i+len("/@external/"):], true +} + +// NodeAttr is the body attribute string of a node of kind with no parent, +// `of`, flags or callee. +func NodeAttr(kind string) string { return kind + "||||" } + +// Collect normalizes an analysis.json stream, decoding one module at a +// time. Its application fields come in json.Marshal order: symbol_table +// before call_graph, external_symbols, param_in and param_out, so every +// callable id is known before an edge names it. +func Collect(r io.Reader) (*View, error) { + dec := json.NewDecoder(r) + v := &View{Calls: map[string]*CV{}, App: map[string]bool{}} + ids := map[string]string{} + callee := map[*CV]map[string]string{} + delim := func(want json.Delim) error { + t, err := dec.Token() + if err != nil { + return err + } + if d, ok := t.(json.Delim); !ok || d != want { + return fmt.Errorf("collect: got %v, want %v", t, want) + } + return nil + } + skip := func() error { + var raw json.RawMessage + return dec.Decode(&raw) + } + if err := delim('{'); err != nil { + return nil, err + } + for dec.More() { + t, err := dec.Token() + if err != nil { + return nil, err + } + if t != "application" { + if err := skip(); err != nil { + return nil, err + } + continue + } + if err := delim('{'); err != nil { + return nil, err + } + for dec.More() { + k, err := dec.Token() + if err != nil { + return nil, err + } + switch k { + case "symbol_table": + if err := delim('{'); err != nil { + return nil, err + } + for dec.More() { + rel, err := dec.Token() + if err != nil { + return nil, err + } + var m v2.Module + if err := dec.Decode(&m); err != nil { + return nil, err + } + v.addModule(rel.(string), &m, ids, callee) + } + if err := delim('}'); err != nil { + return nil, err + } + case "call_graph": + var es []v2.Edge + if err := dec.Decode(&es); err != nil { + return nil, err + } + for _, e := range es { + v.App["cg|"+resolveID(ids, e.Src)+"|"+resolveID(ids, e.Dst)+"|"+strings.Join(e.Prov, ",")] = true + } + case "external_symbols": + var ext map[string]v2.ExternalSymbol + if err := dec.Decode(&ext); err != nil { + return nil, err + } + for id, e := range ext { + ident, _ := externalIdent(id) + c := &CV{Ident: ident, Kind: "external", Body: map[string]string{}, Edges: map[string]bool{}} + for key, n := range e.Body { + c.Body[key] = n.Kind + "||" + n.Of + "||" + } + v.Calls[ident] = c + } + case "param_in", "param_out": + tag := "pin" + if k == "param_out" { + tag = "pout" + } + if err := delim('['); err != nil { + return nil, err + } + for dec.More() { + var e v2.ParamEdge + if err := dec.Decode(&e); err != nil { + return nil, err + } + v.App[tag+"|"+globalNode(ids, e.Src)+"|"+globalNode(ids, e.Dst)+"|"+e.Var] = true + } + if err := delim(']'); err != nil { + return nil, err + } + default: + if err := skip(); err != nil { + return nil, err + } + } + } + if err := delim('}'); err != nil { + return nil, err + } + } + for c, m := range callee { + for key, id := range m { + c.Body[key] += resolveID(ids, id) + } + } + return v, nil +} + +func resolveID(ids map[string]string, id string) string { + if ident, ok := ids[id]; ok { + return ident + } + if ident, ok := externalIdent(id); ok { + return ident + } + return "?" + id +} + +// globalNode is the Node of a global body-node id (v2.GlobalOrdinal). +func globalNode(ids map[string]string, gid string) string { + i := strings.LastIndexByte(gid, '@') + if i < 0 { + return "?" + gid + } + owner, local := gid[:i], gid[i+1:] + if local == "" || local[0] < '0' || local[0] > '9' { + local = "@" + local + } + return Node(resolveID(ids, owner), local) +} + +func (v *View) addModule(rel string, m *v2.Module, ids map[string]string, callee map[*CV]map[string]string) { + var walk func(c *v2.Callable, file string, topInit bool) + walk = func(c *v2.Callable, file string, topInit bool) { + if c.SourceFile != "" { + file = c.SourceFile + } + ident := c.Signature + if topInit { + ident = identityOf(c.Signature, file) + } + cv := &CV{Ident: ident, File: file, Line: c.Span.Start[0], Kind: c.Kind, L3: len(c.Cfg) > 0, + Body: make(map[string]string, len(c.Body)), Edges: map[string]bool{}} + calls := map[string]string{} + for key, n := range c.Body { + flags := "" + if n.IsGoroutine { + flags += "g" + } + if n.IsDeferred { + flags += "d" + } + cv.Body[key] = n.Kind + "|" + n.Parent + "|" + n.Of + "|" + flags + "|" + if n.Callee != nil { + calls[key] = *n.Callee + } + } + for _, e := range c.Cfg { + cv.Edges["cfg|"+e.Src+"|"+e.Dst+"|"+e.Kind] = true + } + for _, e := range c.Cdg { + cv.Edges["cdg|"+e.Src+"|"+e.Dst+"|"] = true + } + for _, e := range c.Ddg { + cv.Edges["ddg|"+e.Src+"|"+e.Dst+"|"+e.Var+"|"+strings.Join(e.Prov, ",")] = true + } + for _, e := range c.Summary { + cv.Edges["sum|"+e.Src+"|"+e.Dst+"|"] = true + } + if prev, dup := v.Calls[ident]; dup { + // Two callables share an identity: keep both visible, so the + // comparer reports the collision instead of hiding one. + ident += "#dup:" + prev.File + ":" + strconv.Itoa(prev.Line) + "/" + file + ":" + strconv.Itoa(cv.Line) + cv.Ident = ident + } + ids[c.ID] = ident + v.Calls[ident] = cv + callee[cv] = calls + for _, k := range slices.Sorted(maps.Keys(c.Callables)) { + inner := c.Callables[k] + walk(&inner, file, topInit) + } + } + for _, k := range slices.Sorted(maps.Keys(m.Functions)) { + fn := m.Functions[k] + walk(&fn, rel, strings.HasSuffix(fn.Signature, ".init")) + } + for _, tn := range slices.Sorted(maps.Keys(m.Types)) { + t := m.Types[tn] + for _, k := range slices.Sorted(maps.Keys(t.Callables)) { + c := t.Callables[k] + walk(&c, rel, false) + } + } +} + +// callerIndex builds callerOf once. +func (v *View) callerIndex() map[string][]string { + v.callerOnce.Do(func() { + set := map[string]map[string]bool{} + for e := range v.App { + f := strings.Split(e, "|") // pin | src ident | src key | dst ident | dst key | var + if f[0] != "pin" { + continue + } + if set[f[3]] == nil { + set[f[3]] = map[string]bool{} + } + set[f[3]][f[1]] = true + } + v.callerOf = make(map[string][]string, len(set)) + for callee, cs := range set { + v.callerOf[callee] = slices.Sorted(maps.Keys(cs)) + } + }) + return v.callerOf +} + +// called reports whether some param_in edge targets callable ident. +func (v *View) called(ident string) bool { + return len(v.callerIndex()[ident]) > 0 +} + +// Callers is ident and every callable that reaches it over param_in edges +// (its transitive callers), the callables whose summary edges a change of +// ident's flows may change. +func (v *View) Callers(ident string) map[string]bool { + idx := v.callerIndex() + out := map[string]bool{ident: true} + stack := []string{ident} + for len(stack) > 0 { + n := stack[len(stack)-1] + stack = stack[:len(stack)-1] + for _, c := range idx[n] { + if !out[c] { + out[c] = true + stack = append(stack, c) + } + } + } + return out +} + +// ─── names (MR4) ────────────────────────────────────────────────────────────── + +// Fix rewrites a transformed view's spellings back to the original's. +type Fix func(string) string + +// StripSuffix is the Fix undoing a rename that appended suffix. +func StripSuffix(suffix string) Fix { + return func(s string) string { return strings.ReplaceAll(s, suffix, "") } +} + +// Apply returns v with fix applied to every `of` and every var label. +func (v *View) Apply(fix Fix) *View { + out := &View{Calls: make(map[string]*CV, len(v.Calls)), App: make(map[string]bool, len(v.App))} + for id, c := range v.Calls { + n := *c + n.Body, n.Edges = make(map[string]string, len(c.Body)), make(map[string]bool, len(c.Edges)) + for k, b := range c.Body { + f := strings.Split(b, "|") + f[2] = fix(f[2]) + n.Body[k] = strings.Join(f, "|") + } + for e := range c.Edges { + f := strings.Split(e, "|") + if f[0] == "ddg" { + f[3] = fix(f[3]) + } + n.Edges[strings.Join(f, "|")] = true + } + out.Calls[id] = &n + } + for e := range v.App { + if strings.HasPrefix(e, "pin|") || strings.HasPrefix(e, "pout|") { + i := strings.LastIndexByte(e, '|') + e = e[:i+1] + fix(e[i+1:]) + } + out.App[e] = true + } + return out +} + +// ─── mapping ────────────────────────────────────────────────────────────────── + +// Mapper maps an old source position to a new one. +type Mapper interface { + Map(Pos) (Pos, bool) +} + +// identity maps every position to itself. +type identity struct{} + +func (identity) Map(p Pos) (Pos, bool) { return p, true } + +// Compose is the map a, then b. +func Compose(a, b Mapper) Mapper { return composed{a, b} } + +type composed struct{ a, b Mapper } + +func (c composed) Map(p Pos) (Pos, bool) { + q, ok := c.a.Map(p) + if !ok { + return Pos{}, false + } + return c.b.Map(q) +} + +var posKey = regexp.MustCompile(`^([0-9]+):([0-9]+)(.*)$`) +var deferVar = regexp.MustCompile(`:([0-9]+):([0-9]+)`) + +// mapper carries the keys of one view through pm. +type mapper struct { + pm Mapper + unmapped []string +} + +// key maps local key k of a callable declared in file. It returns the new +// key and the new file. +func (m *mapper) key(file, k string) (string, string, bool) { + g := posKey.FindStringSubmatch(k) + if g == nil { + return k, file, true + } + l, _ := strconv.Atoi(g[1]) + c, _ := strconv.Atoi(g[2]) + p, ok := m.pm.Map(Pos{File: file, Line: l, Col: c}) + if !ok { + return k, file, false + } + return v2.LocalID(p.Line, p.Col) + g[3], p.File, true +} + +func (m *mapper) must(ident, file, k string) string { + nk, _, ok := m.key(file, k) + if !ok { + m.unmapped = append(m.unmapped, ident+" "+k) + return "?" + k + } + return nk +} + +func (m *mapper) label(file, s string) string { + return deferVar.ReplaceAllStringFunc(s, func(x string) string { + g := deferVar.FindStringSubmatch(x) + nk, _, ok := m.key(file, g[1]+":"+g[2]) + if !ok { + return x + } + return ":" + nk + }) +} + +func (m *mapper) callable(id string, c *CV) *CV { + n := &CV{Ident: c.Ident, File: c.File, Line: c.Line, Kind: c.Kind, L3: c.L3, + Body: make(map[string]string, len(c.Body)), Edges: make(map[string]bool, len(c.Edges))} + if c.File != "" { + if p, ok := m.pm.Map(Pos{File: c.File, Line: c.Line, Col: 1}); ok { + n.File, n.Line = p.File, p.Line + } else { + m.unmapped = append(m.unmapped, id+" declaration") + } + } + for k, b := range c.Body { + nk := m.must(id, c.File, k) + f := strings.Split(b, "|") + if f[1] != "" { + f[1] = m.must(id, c.File, f[1]) + } + n.Body[nk] = strings.Join(f, "|") + } + for e := range c.Edges { + f := strings.Split(e, "|") + f[1], f[2] = m.must(id, c.File, f[1]), m.must(id, c.File, f[2]) + if f[0] == "ddg" { + f[3] = m.label(c.File, f[3]) + } + n.Edges[strings.Join(f, "|")] = true + } + return n +} + +// Map returns v in the coordinates pm maps into, and the keys that did not +// map (sorted). A nil pm maps every key to itself. +func (v *View) Map(pm Mapper) (*View, []string) { + if pm == nil { + pm = identity{} + } + m := &mapper{pm: pm} + out := &View{Calls: make(map[string]*CV, len(v.Calls)), App: make(map[string]bool, len(v.App))} + for id, c := range v.Calls { + out.Calls[id] = m.callable(id, c) + } + node := func(nd string) string { + ident, k := splitNode(nd) + c, ok := v.Calls[ident] + if !ok { + return nd + } + return Node(ident, m.must(ident, c.File, k)) + } + for e := range v.App { + f := strings.Split(e, "|") + if f[0] == "pin" || f[0] == "pout" { + // tag | src ident | src key | dst ident | dst key | var + out.App[f[0]+"|"+node(f[1]+"|"+f[2])+"|"+node(f[3]+"|"+f[4])+"|"+f[5]] = true + continue + } + out.App[e] = true + } + sort.Strings(m.unmapped) + return out, m.unmapped +} + +// ─── exact ──────────────────────────────────────────────────────────────────── + +// Scope limits a comparison. Only, when set, compares just those callables +// and no application edges. Skip leaves those callables out, together with +// every application edge that touches them. KeysOnly compares body keys but +// not their attributes. +type Scope struct { + Only map[string]bool + Skip map[string]bool + KeysOnly bool +} + +func (s Scope) in(ident string) bool { + if s.Only != nil && !s.Only[ident] { + return false + } + return !s.Skip[ident] +} + +func (s Scope) appIn(e string) bool { + if s.Only != nil { + return false + } + f := strings.Split(e, "|") + for _, x := range f[1:] { + if s.Skip[x] { + return false + } + } + return true +} + +// Diff lists where want and got differ within scope, at most limit lines +// (all when limit <= 0). +func Diff(want, got *View, s Scope, limit int) []string { + var out []string + add := func(format string, a ...any) { out = append(out, fmt.Sprintf(format, a...)) } + for _, id := range unionKeys(want.Calls, got.Calls) { + if !s.in(id) { + continue + } + w, g := want.Calls[id], got.Calls[id] + switch { + case g == nil: + add("%s: callable missing", id) + continue + case w == nil: + add("%s: callable added (%s:%d)", id, g.File, g.Line) + continue + } + if w.Kind != g.Kind || w.File != g.File || w.Line != g.Line { + add("%s: %s %s:%d -> %s %s:%d", id, w.Kind, w.File, w.Line, g.Kind, g.File, g.Line) + } + for _, k := range unionKeys(w.Body, g.Body) { + wb, wok := w.Body[k] + gb, gok := g.Body[k] + switch { + case !gok: + add("%s: body node %s (%s) missing", id, k, wb) + case !wok: + add("%s: body node %s (%s) added", id, k, gb) + case wb != gb && !s.KeysOnly: + add("%s: body node %s: %s -> %s", id, k, wb, gb) + } + } + for _, e := range unionKeys(w.Edges, g.Edges) { + switch { + case !g.Edges[e]: + add("%s: edge %s missing", id, e) + case !w.Edges[e]: + add("%s: edge %s added", id, e) + } + } + } + for _, e := range unionKeys(want.App, got.App) { + if !s.appIn(e) { + continue + } + switch { + case !got.App[e]: + add("app: %s missing", e) + case !want.App[e]: + add("app: %s added", e) + } + } + if limit > 0 && len(out) > limit { + n := len(out) + out = append(out[:limit:limit], fmt.Sprintf("… and %d more", n-limit)) + } + return out +} + +func unionKeys[V any](a, b map[string]V) []string { + set := make(map[string]bool, len(a)+len(b)) + for k := range a { + set[k] = true + } + for k := range b { + set[k] = true + } + return slices.Sorted(maps.Keys(set)) +} + +// ─── delta ──────────────────────────────────────────────────────────────────── + +// Ref names a body node in a delta: a key of the original (Pre), mapped +// through the position map, or a mark the transform set in its new text. +type Ref struct { + Pre string + Mark string +} + +// Pre and Mark build Refs. +func Pre(key string) Ref { return Ref{Pre: key} } +func Mark(name string) Ref { return Ref{Mark: name} } + +func (r Ref) String() string { + if r.Mark != "" { + return "mark:" + r.Mark + } + return "pre:" + r.Pre +} + +// DEdge is one expected edge change. Kind is cfg, cdg, ddg or sum. Label is +// the cfg kind or the ddg var (empty otherwise), and Prov is the ddg +// provenance ("ssa" when empty). +type DEdge struct { + Kind string + Src, Dst Ref + Label string + Prov string +} + +// Edge constructors, so a relation reads like the table. +func CFG(src, dst Ref, kind string) DEdge { return DEdge{Kind: "cfg", Src: src, Dst: dst, Label: kind} } +func CDG(src, dst Ref) DEdge { return DEdge{Kind: "cdg", Src: src, Dst: dst} } +func DDG(src, dst Ref, v string) DEdge { return DEdge{Kind: "ddg", Src: src, Dst: dst, Label: v} } +func Port(src, dst Ref, v string) DEdge { + return DEdge{Kind: "ddg", Src: src, Dst: dst, Label: v, Prov: "reaching-defs"} +} + +// Delta is the expected change of one callable (an Identity of the +// original). Add and Remove are the edges added and removed, and Nodes the +// body nodes added (mark → kind). With CDGFromCFG the expected CDG is not +// listed: it is the reference CDG of the expected CFG (RefCDG). With AtMost +// the relation is a precision one: any subset of Remove may go, and nothing +// may be added. +type Delta struct { + Site string + Callable string + Add []DEdge + Remove []DEdge + Nodes map[string]string + CDGFromCFG bool + AtMost bool +} + +func (r Ref) resolve(m *mapper, file string, marks map[string]Pos) (string, error) { + if r.Mark != "" { + p, ok := marks[r.Mark] + if !ok { + return "", fmt.Errorf("no mark %s", r.Mark) + } + return v2.LocalID(p.Line, p.Col), nil + } + k, _, ok := m.key(file, r.Pre) + if !ok { + return "", fmt.Errorf("pre key %s does not map", r.Pre) + } + return k, nil +} + +func (e DEdge) str(m *mapper, file string, marks map[string]Pos) (string, error) { + s, err := e.Src.resolve(m, file, marks) + if err != nil { + return "", err + } + d, err := e.Dst.resolve(m, file, marks) + if err != nil { + return "", err + } + out := e.Kind + "|" + s + "|" + d + "|" + e.Label + if e.Kind == "ddg" { + out += "|" + cmpOr(e.Prov, "ssa") + } + return out, nil +} + +func cmpOr(s, def string) string { + if s == "" { + return def + } + return s +} + +// Expect applies ds to mapped (pre mapped through pm) and returns the view +// the transformed module must have. A removed edge the original lacks, and +// an added one it already has, are errors of the relation (its delta is +// wrong), as is a declared CDG that disagrees with RefCDG of the expected +// CFG. +func Expect(pre, mapped *View, pm Mapper, marks map[string]Pos, ds []Delta) (*View, error) { + if pm == nil { + pm = identity{} + } + out := &View{Calls: maps.Clone(mapped.Calls), App: mapped.App} + m := &mapper{pm: pm} + for _, d := range ds { + c0, c := pre.Calls[d.Callable], mapped.Calls[d.Callable] + if c0 == nil || c == nil { + return nil, fmt.Errorf("delta %s: no callable %s", d.Site, d.Callable) + } + if d.AtMost { + continue // CheckAtMost + } + n := c.clone() + for _, e := range d.Remove { + s, err := e.str(m, c0.File, marks) + if err != nil { + return nil, fmt.Errorf("delta %s: %w", d.Site, err) + } + if !n.Edges[s] { + return nil, fmt.Errorf("delta %s: removed edge %s is not in the original", d.Site, s) + } + delete(n.Edges, s) + } + for _, e := range d.Add { + s, err := e.str(m, c0.File, marks) + if err != nil { + return nil, fmt.Errorf("delta %s: %w", d.Site, err) + } + if n.Edges[s] { + return nil, fmt.Errorf("delta %s: added edge %s is already there", d.Site, s) + } + n.Edges[s] = true + } + for mark, kind := range d.Nodes { + k, err := Mark(mark).resolve(m, c0.File, marks) + if err != nil { + return nil, fmt.Errorf("delta %s: %w", d.Site, err) + } + n.Body[k] = NodeAttr(kind) + } + ref := RefCDG(n) + if d.CDGFromCFG { + for e := range n.Edges { + if strings.HasPrefix(e, "cdg|") { + delete(n.Edges, e) + } + } + for e := range ref { + n.Edges[e] = true + } + } else if diff := cdgDiff(n, ref); diff != "" { + return nil, fmt.Errorf("delta %s: declared CDG disagrees with the reference CDG of the expected CFG: %s", d.Site, diff) + } + out.Calls[d.Callable] = n + } + return out, nil +} + +func cdgDiff(c *CV, ref map[string]bool) string { + var out []string + for e := range c.Edges { + if strings.HasPrefix(e, "cdg|") && !ref[e] { + out = append(out, "-"+e) + } + } + for e := range ref { + if !c.Edges[e] { + out = append(out, "+"+e) + } + } + sort.Strings(out) + return strings.Join(out, " ") +} + +// CheckAtMost checks the AtMost deltas of ds: in each callable got is +// mapped minus some subset of Remove, with nothing added. +func CheckAtMost(pre, mapped, got *View, pm Mapper, marks map[string]Pos, ds []Delta) []string { + if pm == nil { + pm = identity{} + } + var out []string + m := &mapper{pm: pm} + for _, d := range ds { + if !d.AtMost { + continue + } + c0, w, g := pre.Calls[d.Callable], mapped.Calls[d.Callable], got.Calls[d.Callable] + if c0 == nil || w == nil || g == nil { + out = append(out, d.Site+": callable missing") + continue + } + allowed := map[string]bool{} + for _, e := range d.Remove { + if s, err := e.str(m, c0.File, marks); err == nil { + allowed[s] = true + } + } + for _, e := range unionKeys(w.Edges, g.Edges) { + switch { + case !g.Edges[e] && !allowed[e]: + out = append(out, fmt.Sprintf("%s: %s removed", d.Site, e)) + case !w.Edges[e]: + out = append(out, fmt.Sprintf("%s: %s added", d.Site, e)) + } + } + } + return out +} + +// ─── reference CDG ──────────────────────────────────────────────────────────── + +// RefCDG is the control-dependence graph of c's cfg edges, computed +// independently of the analyzer: post-dominator sets by fixed-point +// iteration, then x is control dependent on a when some edge a→b has x +// post-dominating b (reflexively) but not strictly post-dominating a +// (Ferrante–Ottenstein–Warren); self-loops are dropped, and every CFG node +// but @entry/@exit with no controller is anchored at @entry (the analyzer's +// documented convention, spec §4.4). +func RefCDG(c *CV) map[string]bool { + succ := map[string][]string{} + set := map[string]bool{"@exit": true} + for e := range c.Edges { + if !strings.HasPrefix(e, "cfg|") { + continue + } + f := strings.Split(e, "|") + succ[f[1]] = append(succ[f[1]], f[2]) + set[f[1]], set[f[2]] = true, true + } + nodes := slices.Sorted(maps.Keys(set)) + idx := make(map[string]int, len(nodes)) + for i, n := range nodes { + idx[n] = i + } + words := (len(nodes) + 63) / 64 + full := make([]uint64, words) + for i := range nodes { + full[i/64] |= 1 << (i % 64) + } + exit := idx["@exit"] + pdom := make([][]uint64, len(nodes)) + for i := range nodes { + pdom[i] = slices.Clone(full) + } + pdom[exit] = make([]uint64, words) + pdom[exit][exit/64] |= 1 << (exit % 64) + for changed := true; changed; { + changed = false + for i, n := range nodes { + if i == exit { + continue + } + nw := slices.Clone(full) + if len(succ[n]) == 0 { + nw = make([]uint64, words) // reaches no exit + } + for _, s := range succ[n] { + for w := range nw { + nw[w] &= pdom[idx[s]][w] + } + } + nw[i/64] |= 1 << (i % 64) + if !slices.Equal(nw, pdom[i]) { + pdom[i], changed = nw, true + } + } + } + has := func(set []uint64, j int) bool { return set[j/64]&(1<<(j%64)) != 0 } + out := map[string]bool{} + controlled := map[string]bool{} + for _, a := range nodes { + ia := idx[a] + for _, b := range succ[a] { + ib := idx[b] + for w, word := range pdom[ib] { + for word != 0 { + j := w*64 + bits.TrailingZeros64(word) + word &= word - 1 + if j == ia || (has(pdom[ia], j) && j != ia) { + continue + } + if nodes[j] == "@exit" { + continue + } + out["cdg|"+a+"|"+nodes[j]+"|"] = true + controlled[nodes[j]] = true + } + } + } + } + for _, n := range nodes { + if n != "@entry" && n != "@exit" && !controlled[n] { + out["cdg|@entry|"+n+"|"] = true + } + } + return out +} + +// ─── reachability ───────────────────────────────────────────────────────────── + +// Graph is the application-scope dependence graph over Node strings: ddg ∪ +// summary ∪ param_in ∪ param_out, plus cdg unless dataOnly. +type Graph map[string][]string + +// Graph builds v's dependence graph. +func (v *View) Graph(dataOnly bool) Graph { + g := Graph{} + for id, c := range v.Calls { + for e := range c.Edges { + f := strings.SplitN(e, "|", 4) + if f[0] == "cfg" || (dataOnly && f[0] == "cdg") { + continue + } + s := Node(id, f[1]) + g[s] = append(g[s], Node(id, f[2])) + } + } + for e := range v.App { + f := strings.Split(e, "|") + if f[0] != "pin" && f[0] != "pout" { + continue + } + s := f[1] + "|" + f[2] + g[s] = append(g[s], f[3]+"|"+f[4]) + } + return g +} + +// Reach is the set of nodes reachable from src, src included. +func (g Graph) Reach(src string) map[string]bool { + seen := map[string]bool{src: true} + stack := []string{src} + for len(stack) > 0 { + n := stack[len(stack)-1] + stack = stack[:len(stack)-1] + for _, d := range g[n] { + if !seen[d] { + seen[d] = true + stack = append(stack, d) + } + } + } + return seen +} + +var vertexTail = regexp.MustCompile(`^(.*)/(actual_in|actual_out)(:[0-9]+)?$`) + +// locator finds the counterpart in post of a node of pre. It tries the +// mapped key in the same callable first. Failing that, it tries any +// callable of post that holds the mapped position in the mapped file, since +// a statement can move into a new callable (MR6, MR14). A call-site vertex +// (/actual_*:i) matches by its call key, kind and `of`, not by index. +type locator struct { + pre, post *View + m *mapper + byFileKey map[string][]string + // vertex indexes each call-site vertex of post by callable, call key, + // kind and `of` (the least key when several match). + vertex map[[4]string]string + memo map[string]string // locate's results ("" = none) +} + +func newLocator(pre, post *View, pm Mapper) *locator { + if pm == nil { + pm = identity{} + } + l := &locator{pre: pre, post: post, m: &mapper{pm: pm}, byFileKey: map[string][]string{}, vertex: map[[4]string]string{}, memo: map[string]string{}} + for id, c := range post.Calls { + for k, b := range c.Body { + if h := vertexTail.FindStringSubmatch(k); h != nil { + vk := [4]string{id, h[1], h[2], strings.Split(b, "|")[2]} + if prev, ok := l.vertex[vk]; !ok || k < prev { + l.vertex[vk] = k + } + continue + } + if posKey.MatchString(k) { + l.byFileKey[c.File+" "+k] = append(l.byFileKey[c.File+" "+k], id) + } + } + } + for _, ids := range l.byFileKey { + slices.Sort(ids) + } + return l +} + +func (l *locator) locate(node string) (string, bool) { + if r, ok := l.memo[node]; ok { + return r, r != "" + } + r, ok := l.find(node) + l.memo[node] = r + return r, ok +} + +func (l *locator) find(node string) (string, bool) { + ident, key := splitNode(node) + c0 := l.pre.Calls[ident] + if c0 == nil { + return "", false + } + nk, nf, ok := l.m.key(c0.File, key) + if !ok { + return "", false + } + g := vertexTail.FindStringSubmatch(nk) + cands := []string{ident} + if posKey.MatchString(nk) { + base := nk + if g != nil { + base = g[1] + } + cands = append(cands, l.byFileKey[nf+" "+base]...) + } + of := "" + if b, ok := c0.Body[key]; ok { + of = strings.Split(b, "|")[2] + } + for _, id := range cands { + c := l.post.Calls[id] + if c == nil { + continue + } + if g == nil { + if _, ok := c.Body[nk]; ok { + return Node(id, nk), true + } + continue + } + if k, ok := l.vertex[[4]string{id, g[1], g[2], of}]; ok { + return Node(id, k), true + } + } + return "", false +} + +// ReachResult is the outcome of the reachability oracle. Lost and +// Unlocated hold at most maxReport entries each; NLost and NUnlocated count +// them all. +type ReachResult struct { + Callables, Sources, Pairs int + Lost []string // "source -> sink", no longer reachable + Unlocated []string // sources or sinks with no counterpart + NLost, NUnlocated int +} + +const maxReport = 200 + +func (r *ReachResult) lost(s string) { + r.NLost++ + if len(r.Lost) < maxReport { + r.Lost = append(r.Lost, s) + } +} + +func (r *ReachResult) unlocated(s string) { + r.NUnlocated++ + if len(r.Unlocated) < maxReport { + r.Unlocated = append(r.Unlocated, s) + } +} + +// igraph is a Graph over interned node indices. +type igraph struct { + idx map[string]int32 + name []string + succ [][]int32 +} + +func intern(g Graph) *igraph { + ig := &igraph{idx: make(map[string]int32, len(g))} + id := func(n string) int32 { + i, ok := ig.idx[n] + if !ok { + i = int32(len(ig.name)) + ig.idx[n] = i + ig.name = append(ig.name, n) + ig.succ = append(ig.succ, nil) + } + return i + } + for _, n := range slices.Sorted(maps.Keys(g)) { + s := id(n) + for _, d := range g[n] { + ig.succ[s] = append(ig.succ[s], id(d)) + } + } + return ig +} + +// reach marks in seen (len = node count, cleared by the caller) every node +// reachable from src, and returns them in visit order. +func (ig *igraph) reach(src int32, seen []bool, out []int32) []int32 { + out = append(out[:0], src) + seen[src] = true + for i := 0; i < len(out); i++ { + for _, d := range ig.succ[out[i]] { + if !seen[d] { + seen[d] = true + out = append(out, d) + } + } + } + return out +} + +// CheckReach samples up to n callables of pre with an L3 result (those in +// prefer first, then the others, shuffled with seed). For every @formal_in +// of each, it checks that every sink reachable in pre is reachable from the +// source's counterpart in post. A sink is a formal_out vertex or the +// actual_in of a call node. The actual_in of a capture site is not a sink: +// it feeds only its closure's formal_in (param_in; a capture site has no +// summary edge, spec §4.6), so whatever a flow into it reaches downstream is +// checked at that sink. +// +// cut lists pre nodes whose out-edges the pre graph leaves out (a relation +// that legitimately removes flows through them, MR14). +func CheckReach(pre, post *View, pm Mapper, prefer []string, n int, seed uint64, cut ...string) ReachResult { + gPre, gPost := pre.Graph(false), post.Graph(false) + for _, c := range cut { + delete(gPre, c) + } + iPre, iPost := intern(gPre), intern(gPost) + seenPre, seenPost := make([]bool, len(iPre.name)), make([]bool, len(iPost.name)) + var bufPre, bufPost []int32 + // sinkPost[i] is the post counterpart of pre node i (-1 none, -2 not a + // sink, -3 not yet computed). + sinkPost := make([]int32, len(iPre.name)) + for i := range sinkPost { + sinkPost[i] = -3 + } + l := newLocator(pre, post, pm) + var res ReachResult + for _, id := range SampleCallables(pre, prefer, n, seed) { + res.Callables++ + c := pre.Calls[id] + for _, k := range slices.Sorted(maps.Keys(c.Body)) { + if !strings.HasPrefix(k, "@formal_in:") { + continue + } + src := Node(id, k) + psrc, ok := l.locate(src) + if !ok { + res.unlocated("source " + src) + continue + } + res.Sources++ + si, ok := iPre.idx[src] + if !ok { + continue // no out-edge: reaches nothing + } + bufPost = bufPost[:0] + if pi, ok := iPost.idx[psrc]; ok { + bufPost = iPost.reach(pi, seenPost, bufPost) + } + bufPre = iPre.reach(si, seenPre, bufPre) + var lost []string + for _, ni := range bufPre { + if ni == si { + continue + } + if sinkPost[ni] == -3 { + sinkPost[ni] = -2 + if sink := iPre.name[ni]; pre.isSink(sink) { + sinkPost[ni] = -1 + if ps, ok := l.locate(sink); ok { + if pi, ok := iPost.idx[ps]; ok { + sinkPost[ni] = pi + } else { + sinkPost[ni] = int32(len(iPost.name)) // located, no edges: unreachable + } + } + } + } + switch p := sinkPost[ni]; { + case p == -2: + continue + case p == -1: + res.Pairs++ + res.unlocated("sink " + iPre.name[ni]) + default: + res.Pairs++ + if int(p) >= len(seenPost) || !seenPost[p] { + lost = append(lost, iPre.name[ni]) + } + } + } + for _, ni := range bufPre { + seenPre[ni] = false + } + for _, ni := range bufPost { + seenPost[ni] = false + } + slices.Sort(lost) + for _, sink := range lost { + res.lost(src + " -> " + sink) + } + } + } + return res +} + +func (v *View) isSink(node string) bool { + id, k := splitNode(node) + if strings.HasPrefix(k, "@formal_out") { + return true + } + g := vertexTail.FindStringSubmatch(k) + if g == nil || g[2] != "actual_in" { + return false + } + c := v.Calls[id] + return c != nil && strings.HasPrefix(c.Body[g[1]], "call|") +} + +// SampleCallables picks up to n callables of v with an L3 result: those in +// prefer first, then the rest, each group shuffled with seed. n <= 0 picks +// all of them. +func SampleCallables(v *View, prefer []string, n int, seed uint64) []string { + r := rand.New(rand.NewPCG(seed, 2)) + taken := map[string]bool{} + pick := func(ids []string) []string { + var out []string + for _, id := range ids { + if c := v.Calls[id]; c != nil && c.L3 && !taken[id] { + taken[id] = true + out = append(out, id) + } + } + slices.Sort(out) + r.Shuffle(len(out), func(i, j int) { out[i], out[j] = out[j], out[i] }) + return out + } + out := pick(prefer) + if n <= 0 || len(out) < n { + out = append(out, pick(slices.Collect(maps.Keys(v.Calls)))...) + } + if n > 0 && len(out) > n { + out = out[:n] + } + slices.Sort(out) + return out +} + +// ReachesFrom reports whether any node of srcs reaches dst over g. +func (g Graph) ReachesFrom(srcs []string, dst string) bool { + for _, s := range srcs { + if g.Reach(s)[dst] { + return true + } + } + return false +} + +// Witness is a shortest path src → dst over g (nil when none). +func (g Graph) Witness(src, dst string) []string { + prev := map[string]string{src: ""} + q := []string{src} + for len(q) > 0 { + n := q[0] + q = q[1:] + if n == dst { + var p []string + for x := dst; x != ""; x = prev[x] { + p = append([]string{x}, p...) + } + return p + } + for _, d := range g[n] { + if _, ok := prev[d]; !ok { + prev[d] = n + q = append(q, d) + } + } + } + return nil +} diff --git a/internal/metamorphic/mr_test.go b/internal/metamorphic/mr_test.go new file mode 100644 index 0000000..386f0dd --- /dev/null +++ b/internal/metamorphic/mr_test.go @@ -0,0 +1,1179 @@ +package metamorphic + +// The metamorphic relations on the fixtures (testdata/dataflow's flow and +// sdg packages, and chi), run by go test ./... The same harness runs them on +// the prepared eval apps behind the `metamorphic` build tag +// (realapp_test.go, scripts/metamorphic.sh). +// +// Each relation transforms a copy of the module, analyzes it, and checks +// one of three oracles against the original's analysis: +// +// - exact: the analysis is unchanged up to the position map (MR1-MR4, +// MR7, MR15); +// - delta: it changes by exactly the delta each site declares (MR8-MR12), +// or by at most a declared removal (MR13, a precision relation); +// - reach: every flow of the original survives (MR5, MR6, MR14). +// +// Every relation logs one MRSTAT line (sites, skips, pairs checked), which +// scripts/metamorphic.sh collects. + +import ( + "fmt" + "go/ast" + "maps" + "os" + "path/filepath" + "reflect" + "runtime" + "slices" + "strings" + "sync" + "testing" + "time" +) + +// seed is the fixed seed of every sample and site choice. +const seed = 1 + +// target is one module the relations run on. +type target struct { + name string + dir string + // pkgDirs restricts the transformed files to these package directories + // (all when empty). + pkgDirs []string + // nFiles and nCallables bound the samples (all when 0). + nFiles, nCallables int + + once sync.Once + ws *Workspace + pre *analyzed // -a 4 + pre3 *analyzed // -a 3 + err error + mu sync.Mutex +} + +// analyzed is one analyzed module. +type analyzed struct { + view *View + errPkgs map[string]bool // packages with load errors + dur time.Duration +} + +func analyze(dir string, level int, targets []string) (*analyzed, error) { + start := time.Now() + r, err := Analyze(dir, level, targets) + if err != nil { + return nil, err + } + a := &analyzed{view: r.View, errPkgs: map[string]bool{}, dur: time.Since(start)} + for _, e := range r.Errors { + a.errPkgs[e[:strings.Index(e, ": ")]] = true + } + return a, nil +} + +func testdata(name string) string { + _, file, _, _ := runtime.Caller(0) + return filepath.Join(filepath.Dir(file), "..", "..", "testdata", name) +} + +// examples is testdata/metamorphic: small functions written to be eligible +// sites of every relation, and the convention pins (conventions.go). +var examples = &target{name: "examples", dir: testdata("metamorphic")} + +var fixtures = []*target{ + {name: "dataflow", dir: testdata("dataflow"), pkgDirs: []string{"flow", "sdg"}}, + {name: "chi", dir: testdata("chi"), nFiles: 12, nCallables: 60}, + examples, +} + +// load loads the target's workspace and its original analysis once. +func (tg *target) load(t *testing.T) { + t.Helper() + tg.once.Do(func() { + tg.ws, tg.err = LoadWorkspace(tg.dir) + if tg.err == nil { + tg.pre, tg.err = analyze(tg.ws.Dir, 4, nil) + } + }) + if tg.err != nil { + t.Fatalf("%s: %v", tg.name, tg.err) + } +} + +func (tg *target) level3(t *testing.T) *analyzed { + tg.mu.Lock() + defer tg.mu.Unlock() + if tg.pre3 == nil { + a, err := analyze(tg.ws.Dir, 3, nil) + if err != nil { + t.Fatal(err) + } + tg.pre3 = a + } + return tg.pre3 +} + +// files is the sample of eligible files the relations transform. +func (tg *target) files() []string { + return Sample(tg.ws.Eligible(tg.pkgDirs...), tg.nFiles, seed) +} + +// relation is one metamorphic relation. +type relation struct { + name string + t Transform + oracle string // exact | delta | reach + fix Fix // spelling fix for the transformed view (MR4) +} + +// outcome is what one relation found on one target. +type outcome struct { + sites int + skipped map[string]int + violations []string + pairs int + summaries int // MR14a summary edges checked + dur time.Duration +} + +func (o *outcome) fail(format string, a ...any) { + o.violations = append(o.violations, fmt.Sprintf(format, a...)) +} + +// logStat writes the relation's MRSTAT line. +func logStat(t *testing.T, mr, tg string, o *outcome) { + t.Helper() + var sk []string + for _, k := range slices.Sorted(maps.Keys(o.skipped)) { + sk = append(sk, fmt.Sprintf("%s=%d", k, o.skipped[k])) + } + t.Logf("MRSTAT %s %s sites=%d pairs=%d summaries=%d violations=%d time=%s skipped=[%s]", + mr, tg, o.sites, o.pairs, o.summaries, len(o.violations), o.dur.Round(time.Millisecond), strings.Join(sk, "; ")) +} + +// transformAndAnalyze applies r to tg and analyzes the result at -a 4. A +// transformed module that type-checks worse than the original is a test +// bug and fails the test. +func transformAndAnalyze(t *testing.T, tg *target, tr Transform, ws *Workspace, files []string, pre *View) (*Result, *analyzed) { + t.Helper() + res, err := ws.Apply(tr, files, filepath.Join(t.TempDir(), "m"), seed, pre) + if err != nil { + t.Fatalf("%s on %s: %v", tr.Name, tg.name, err) + } + if len(res.Files) == 0 { + return res, nil + } + post, err := analyze(res.Dir, 4, nil) + if err != nil { + t.Fatalf("%s on %s: analyze: %v", tr.Name, tg.name, err) + } + checkTypes(t, tg, tr.Name, res, post) + return res, post +} + +// checkTypes fails when the transformed module has a package with load +// errors the original had none in. +func checkTypes(t *testing.T, tg *target, mr string, res *Result, post *analyzed) { + t.Helper() + for p := range post.errPkgs { + if !tg.pre.errPkgs[p] { + r, _ := Analyze(res.Dir, 1, nil) + var msgs []string + for _, e := range r.Errors { + if strings.HasPrefix(e, p+": ") { + msgs = append(msgs, e) + } + } + t.Fatalf("%s on %s: the transformed module does not type-check (a transform bug): %s (kept at %s)", + mr, tg.name, strings.Join(msgs, "; "), keep(t, res.Dir)) + } + } +} + +// keep copies a failing transformed module out of the test's temp dir. +func keep(t *testing.T, dir string) string { + dst, err := os.MkdirTemp("", "mr-fail-*") + if err != nil { + return dir + } + if err := copyTree(dir, dst); err != nil { + return dir + } + return dst +} + +// runRelation runs r on tg and reports its violations. +func runRelation(t *testing.T, tg *target, r relation) *outcome { + t.Helper() + tg.load(t) + start := time.Now() + o := &outcome{skipped: map[string]int{}} + defer func() { o.dur = time.Since(start) }() + pre := tg.pre.view + res, post := transformAndAnalyze(t, tg, r.t, tg.ws, tg.files(), pre) + maps.Copy(o.skipped, res.Skipped) + o.sites = len(res.Sites) + for _, v := range res.Violations { + o.fail("%s", v) + } + if post == nil { + return o + } + got := post.view + if r.fix != nil { + got = got.Apply(r.fix) + } + mapped, unmapped := pre.Map(res.Map) + if r.oracle != "reach" && len(unmapped) > 0 { + o.fail("keys of the original do not map: %s", strings.Join(head(unmapped, 10), ", ")) + return o + } + switch r.oracle { + case "exact": + for _, d := range Diff(mapped, got, Scope{}, 40) { + o.fail("%s", d) + } + case "delta": + checkDelta(o, pre, mapped, got, res) + case "reach": + checkReach(o, tg, pre, got, res) + } + if len(o.violations) > 0 { + t.Logf("%s on %s: transformed module kept at %s", r.name, tg.name, keep(t, res.Dir)) + } + return o +} + +func head(s []string, n int) []string { + if len(s) > n { + return append(slices.Clone(s[:n]), fmt.Sprintf("… and %d more", len(s)-n)) + } + return s +} + +// checkDelta is the delta oracle: the transformed view equals the original +// with every site's delta applied (MR8-MR12), and for a precision site +// (MR13) the site's checks hold. +func checkDelta(o *outcome, pre, mapped, got *View, res *Result) { + var ds []Delta + atMost := map[string]bool{} + for _, s := range res.Sites { + if s.Delta != nil { + ds = append(ds, *s.Delta) + if s.Delta.AtMost { + atMost[s.Delta.Callable] = true + } + } + } + want, err := Expect(pre, mapped, res.Map, res.Marks, ds) + if err != nil { + o.fail("relation: %v", err) + return + } + if len(atMost) == 0 { + for _, d := range Diff(want, got, Scope{}, 40) { + o.fail("%s", d) + } + return + } + // A precision relation: nothing is added anywhere. At a site, its + // declared removals may go (CheckAtMost). A summary edge may go only in + // a site callable or one of its transitive callers (over param_in): + // the constant cuts a flow of the site callable, which changes its own + // summary edges and those of every caller whose summary is derived from + // its flows, and nothing else. + sumOK := map[string]bool{} + for id := range atMost { + maps.Copy(sumOK, pre.Callers(id)) + } + for _, d := range Diff(want, got, Scope{Skip: atMost}, 0) { + if id, rest, ok := strings.Cut(d, ": edge sum|"); ok && strings.HasSuffix(rest, " missing") && sumOK[id] { + continue + } + o.fail("%s", d) + } + for _, d := range CheckAtMost(pre, mapped, got, res.Map, res.Marks, ds) { + // A site callable is its own caller set's root, so its summary + // edges may go too. + if strings.Contains(d, ": sum|") && strings.HasSuffix(d, " removed") { + continue + } + o.fail("%s", d) + } + g := got.Graph(true) + l := newLocator(pre, got, res.Map) + for _, s := range res.Sites { + if len(s.NoFlow) == 0 { + continue + } + c := got.Calls[s.Callable] + var srcs []string + for _, k := range slices.Sorted(maps.Keys(c.Body)) { + if strings.HasPrefix(k, "@formal_in:") && !strings.HasPrefix(strings.Split(c.Body[k], "|")[2], ":") { + srcs = append(srcs, Node(s.Callable, k)) + } + } + for _, ref := range s.NoFlow { + dst, ok := l.locate(Node(s.Callable, ref.Pre)) + if !ok { + o.fail("%s: no counterpart of %s", s.Desc, ref.Pre) + continue + } + if g.ReachesFrom(srcs, dst) { + o.fail("%s: a formal_in still reaches %s (a constant argument cannot carry a flow)", s.Desc, dst) + } + } + } +} + +// checkReach is the reachability oracle, plus MR14's summary checks. +func checkReach(o *outcome, tg *target, pre, got *View, res *Result) { + var cut []string + for _, s := range res.Sites { + cut = append(cut, s.Cut...) + } + rr := CheckReach(pre, got, res.Map, res.Touched, tg.nCallables, seed, cut...) + o.pairs = rr.Pairs + if len(res.Sites) > 0 && rr.Pairs == 0 { + o.fail("%d sites but no source-sink pair checked: the oracle checked nothing", len(res.Sites)) + } + gPre := pre.Graph(false) + for i, l := range head(rr.Lost, 40) { + msg := "lost flow " + l + if p := strings.SplitN(l, " -> ", 2); i < 5 && len(p) == 2 { + msg += "\n\t\twitness in the original: " + strings.Join(gPre.Witness(p[0], p[1]), "\n\t\t ") + } + o.fail("%s", msg) + } + for _, u := range head(rr.Unlocated, 20) { + o.fail("no counterpart: %s", u) + } + if rr.NLost > len(rr.Lost) || rr.NUnlocated > len(rr.Unlocated) { + o.fail("%d lost flows and %d unlocated nodes in all", rr.NLost, rr.NUnlocated) + } + l := newLocator(pre, got, res.Map) + for _, s := range res.Sites { + for _, sc := range s.Summary { + o.summaries++ + src, ok1 := l.locate(Node(s.Callable, sc.Src.Pre)) + dst, ok2 := l.locate(Node(s.Callable, sc.Dst.Pre)) + if !ok1 || !ok2 { + o.fail("%s: summary %s: no counterpart", s.Desc, sc.Pre) + continue + } + id, sk := splitNode(src) + _, dk := splitNode(dst) + if c := got.Calls[id]; c == nil || !c.Edges["sum|"+sk+"|"+dk+"|"] { + o.fail("%s: summary %s has no counterpart %s -> %s", s.Desc, sc.Pre, sk, dk) + } + } + } +} + +// ─── the relations ──────────────────────────────────────────────────────────── + +var relations = []relation{ + {name: "MR1", t: MR1, oracle: "exact"}, + {name: "MR2", t: MR2, oracle: "exact"}, + {name: "MR3", t: MR3, oracle: "exact"}, + {name: "MR4", t: MR4, oracle: "exact", fix: StripSuffix(Suffix4)}, + {name: "MR5", t: MR5, oracle: "reach"}, + {name: "MR6", t: MR6, oracle: "reach"}, + {name: "MR8", t: MR8, oracle: "delta"}, + {name: "MR9", t: MR9, oracle: "delta"}, + {name: "MR10", t: MR10, oracle: "delta"}, + {name: "MR11", t: MR11, oracle: "delta"}, + {name: "MR12", t: MR12, oracle: "delta"}, + {name: "MR13", t: MR13, oracle: "delta"}, + {name: "MR14", t: MR14, oracle: "reach"}, +} + +// knownSkips names the relation/target pairs skipped for a filed issue. +var knownSkips = map[string]string{} + +func runOn(t *testing.T, targets []*target, r relation, requireSites bool) { + sites, ran := 0, 0 + var mu sync.Mutex + t.Run(r.name, func(t *testing.T) { + for _, tg := range targets { + t.Run(tg.name, func(t *testing.T) { + if why, ok := knownSkips[r.name+"/"+tg.name]; ok { + t.Skip(why) + } + t.Parallel() + o := runRelation(t, tg, r) + mu.Lock() + sites += o.sites + ran++ + mu.Unlock() + logStat(t, r.name, tg.name, o) + for _, v := range o.violations { + t.Errorf("%s on %s: %s", r.name, tg.name, v) + } + }) + } + }) + if requireSites && ran == len(targets) && sites == 0 { + t.Errorf("%s exercised no site on the fixtures", r.name) + } +} + +func TestMR_Fixtures(t *testing.T) { + for _, r := range relations { + runOn(t, fixtures, r, true) + } + t.Run("MR7", func(t *testing.T) { + for _, tg := range fixtures { + t.Run(tg.name, func(t *testing.T) { + t.Parallel() + o := runMR7(t, tg) + logStat(t, "MR7", tg.name, o) + for _, v := range o.violations { + t.Errorf("MR7 on %s: %s", tg.name, v) + } + }) + } + }) + t.Run("MR15", func(t *testing.T) { + for _, tg := range fixtures { + t.Run(tg.name, func(t *testing.T) { + t.Parallel() + o := runMR15(t, tg) + logStat(t, "MR15", tg.name, o) + for _, v := range o.violations { + t.Errorf("MR15 on %s: %s", tg.name, v) + } + }) + } + }) +} + +// runMR7 is MR7: --target-files at -a 3 on one sampled file equals the +// full -a 3 run on that file's callables. +func runMR7(t *testing.T, tg *target) *outcome { + tg.load(t) + start := time.Now() + o := &outcome{skipped: map[string]int{}} + defer func() { o.dur = time.Since(start) }() + full := tg.level3(t) + for _, rel := range Sample(tg.files(), 2, seed) { + part, err := analyze(tg.ws.Dir, 3, []string{filepath.Join(tg.ws.Dir, rel)}) + if err != nil { + t.Fatal(err) + } + only := map[string]bool{} + for id, c := range full.view.Calls { + if c.File == rel { + only[id] = true + } + } + o.sites++ + for _, d := range Diff(full.view, part.view, Scope{Only: only}, 40) { + o.fail("%s: %s", rel, d) + } + } + return o +} + +// runMR15 is MR15: MR6 then Inline6 gives back the original source, byte +// for byte, and the original analysis exactly. +func runMR15(t *testing.T, tg *target) *outcome { + tg.load(t) + start := time.Now() + o := &outcome{skipped: map[string]int{}} + defer func() { o.dur = time.Since(start) }() + files := tg.files() + ext, err := tg.ws.Apply(MR6, files, filepath.Join(t.TempDir(), "x"), seed, nil) + if err != nil { + t.Fatal(err) + } + maps.Copy(o.skipped, ext.Skipped) + if len(ext.Files) == 0 { + return o + } + ws2, err := LoadWorkspace(ext.Dir) + if err != nil { + t.Fatal(err) + } + inl, err := ws2.Apply(Inline6, ext.Files, filepath.Join(t.TempDir(), "i"), seed, nil) + if err != nil { + t.Fatal(err) + } + o.sites = len(inl.Sites) + for _, rel := range ext.Files { + a, _ := os.ReadFile(filepath.Join(tg.ws.Dir, rel)) + b, _ := os.ReadFile(filepath.Join(inl.Dir, rel)) + if string(a) != string(b) { + o.fail("%s: inlining the MR6 helpers does not give back the original source", rel) + } + } + post, err := analyze(inl.Dir, 4, nil) + if err != nil { + t.Fatal(err) + } + mapped, unmapped := tg.pre.view.Map(Compose(ext.Map, inl.Map)) + if len(unmapped) > 0 { + o.fail("keys do not map: %s", strings.Join(head(unmapped, 10), ", ")) + } + for _, d := range Diff(mapped, post.view, Scope{}, 40) { + o.fail("%s", d) + } + return o +} + +// ─── harness tests (Step 1) ─────────────────────────────────────────────────── + +// TestPosMap_RoundTrip: every transform's PosMap maps each old statement and +// call it keeps to a node of the same kind, in the transformed module (which +// type-checks: LoadWorkspace would report its errors). +func TestPosMap_RoundTrip(t *testing.T) { + tg := fixtures[0] + tg.load(t) + for _, tr := range []Transform{MR1, MR2, MR3, MR4, MR5, MR6, MR8, MR9, MR10, MR11, MR12, MR13, MR14} { + t.Run(tr.Name, func(t *testing.T) { + t.Parallel() + res, err := tg.ws.Apply(tr, tg.files(), filepath.Join(t.TempDir(), "m"), seed, tg.pre.view) + if err != nil { + t.Fatal(err) + } + if len(res.Files) == 0 { + t.Fatalf("%s changed no file", tr.Name) + } + ws2, err := LoadWorkspace(res.Dir) + if err != nil { + t.Fatal(err) + } + for _, f := range ws2.Files { + if f.Skip == "package has errors" { + t.Fatalf("%s: %s does not type-check: %v", tr.Name, f.Rel, f.Pkg.Errors) + } + } + checked := 0 + for _, rel := range res.Files { + old := tg.ws.Files[rel] + ast.Inspect(old.AST, func(n ast.Node) bool { + switch n.(type) { + case ast.Stmt, *ast.CallExpr: + default: + return true + } + if _, isBlock := n.(*ast.BlockStmt); isBlock { + return true + } + p, ok := res.Map.Map(old.pos(n.Pos())) + if !ok { + return true // deleted or replaced + } + nf := ws2.Files[p.File] + if nf == nil { + t.Errorf("%s: %v maps to %s, not a file", tr.Name, old.pos(n.Pos()), p.File) + return true + } + want := reflect.TypeOf(n) + if !nodeAt(nf, p, want) && !kindChanged(tr, n, want) { + t.Errorf("%s: %s %v maps to %v, which holds no %v", tr.Name, want, old.pos(n.Pos()), p, want) + } + checked++ + return true + }) + } + if checked == 0 { + t.Errorf("%s: nothing checked", tr.Name) + } + }) + } +} + +// kindChanged reports whether tr changes the kind of statement n on +// purpose: MR14 defers a call statement, so the expression statement +// becomes a defer statement. +func kindChanged(tr Transform, n ast.Node, want reflect.Type) bool { + return tr.Name == "MR14" && want == reflect.TypeOf(&ast.ExprStmt{}) +} + +// nodeAt reports whether f has a node of type want starting at p. +func nodeAt(f *SrcFile, p Pos, want reflect.Type) bool { + off, ok := f.lines.offset(p.Line, p.Col) + if !ok { + return false + } + found := false + ast.Inspect(f.AST, func(n ast.Node) bool { + if n == nil || found { + return false + } + if f.off(n.Pos()) == off && reflect.TypeOf(n) == want { + found = true + } + return f.off(n.Pos()) <= off && off < f.off(n.End())+1 + }) + return found +} + +// TestCompare_DetectsChange: each oracle flags a hand-made change. An +// untransformed copy compares equal; removing one ddg edge fails the exact +// oracle; a delta naming that removal passes; and removing a param_in edge +// that carried a flow fails the reachability oracle. +func TestCompare_DetectsChange(t *testing.T) { + tg := fixtures[0] + tg.load(t) + pre := tg.pre.view + cp, err := analyze(copyOf(t, tg.ws.Dir), 4, nil) + if err != nil { + t.Fatal(err) + } + if d := Diff(pre, cp.view, Scope{}, 10); len(d) > 0 { + t.Fatalf("an untransformed copy differs: %v", d) + } + + id, edge := firstEdge(pre, "ddg") + got := cloneView(pre) + delete(got.Calls[id].Edges, edge) + d := Diff(pre, got, Scope{}, 0) + if len(d) != 1 || !strings.Contains(d[0], edge) || !strings.HasSuffix(d[0], "missing") { + t.Fatalf("exact oracle on a one-edge removal: %v", d) + } + f := strings.Split(edge, "|") + want, err := Expect(pre, pre, nil, nil, []Delta{{Site: "test", Callable: id, + Remove: []DEdge{{Kind: "ddg", Src: Pre(f[1]), Dst: Pre(f[2]), Label: f[3], Prov: f[4]}}}}) + if err != nil { + t.Fatal(err) + } + if d := Diff(want, got, Scope{}, 0); len(d) > 0 { + t.Errorf("delta oracle with the removal declared: %v", d) + } + if d := Diff(want, pre, Scope{}, 0); len(d) != 1 { + t.Errorf("delta oracle with the removal declared, edge kept: %v", d) + } + if _, err := Expect(pre, got, nil, nil, []Delta{{Site: "test", Callable: id, + Remove: []DEdge{{Kind: "ddg", Src: Pre(f[1]), Dst: Pre(f[2]), Label: f[3], Prov: f[4]}}}}); err == nil { + t.Error("a delta removing an edge the original lacks is not reported") + } + + // Reachability: drop every param_in edge; some flow is lost. + cut := cloneView(pre) + for e := range cut.App { + if strings.HasPrefix(e, "pin|") { + delete(cut.App, e) + } + } + if rr := CheckReach(pre, cut, nil, nil, 0, seed); len(rr.Lost) == 0 || rr.Pairs == 0 { + t.Errorf("reachability oracle without param_in: %d pairs, none lost", rr.Pairs) + } + if rr := CheckReach(pre, pre, nil, nil, 0, seed); len(rr.Lost) > 0 || len(rr.Unlocated) > 0 { + t.Errorf("reachability oracle on itself: lost %v, unlocated %v", head(rr.Lost, 5), head(rr.Unlocated, 5)) + } +} + +// TestRefCDG: the analyzer's CDG equals the reference CDG of its CFG on +// every fixture callable (the delta relations derive expected CDGs with it). +func TestRefCDG(t *testing.T) { + for _, tg := range fixtures { + tg.load(t) + n := 0 + for _, id := range slices.Sorted(maps.Keys(tg.pre.view.Calls)) { + c := tg.pre.view.Calls[id] + if !c.L3 { + continue + } + n++ + if d := cdgDiff(c, RefCDG(c)); d != "" { + t.Errorf("%s %s: cdg differs from the reference: %s", tg.name, id, d) + } + } + floor := 50 + if tg == examples { + floor = 30 // a small module of focused examples + } + if n < floor { + t.Errorf("%s: only %d callables", tg.name, n) + } + } +} + +func firstEdge(v *View, kind string) (string, string) { + for _, id := range slices.Sorted(maps.Keys(v.Calls)) { + for _, e := range slices.Sorted(maps.Keys(v.Calls[id].Edges)) { + if strings.HasPrefix(e, kind+"|") { + return id, e + } + } + } + return "", "" +} + +func cloneView(v *View) *View { + out := &View{Calls: map[string]*CV{}, App: maps.Clone(v.App)} + for id, c := range v.Calls { + out.Calls[id] = c.clone() + } + return out +} + +func copyOf(t *testing.T, dir string) string { + dst := filepath.Join(t.TempDir(), "copy") + if err := copyTree(dir, dst); err != nil { + t.Fatal(err) + } + return dst +} + +// ─── one relation per oracle, on one named example ──────────────────────────── +// +// The table above runs every relation on whole samples. These three pin one +// site each on testdata/metamorphic, so a relation's shape reads end to end. +// Keys are found by statement text (stmtKey), never by line number. + +const exPkg = "example.com/metamorphic." + +// stmtKey is the body key of the one statement of function fn in examples +// file rel whose text is text, or starts with text followed by a token +// boundary (so "x := a + 1" never matches "x := a + 10"). Zero or several +// matches fail the test. +func stmtKey(t *testing.T, rel, fn, text string) string { + t.Helper() + f := examples.ws.Files[rel] + if f == nil { + t.Fatalf("no file %s", rel) + } + var keys []string + for _, fd := range funcDecls(f) { + if fd.Name.Name != fn { + continue + } + ast.Inspect(fd.Body, func(n ast.Node) bool { + st, ok := n.(ast.Stmt) + if !ok { + return true + } + if _, block := st.(*ast.BlockStmt); block { + return true + } + if src := string(f.Src[f.off(st.Pos()):f.off(st.End())]); textMatch(src, text) { + keys = append(keys, f.key(st.Pos())) + } + return true + }) + } + if len(keys) != 1 { + t.Fatalf("%s: %d statements %q in %s (%v), want exactly one", rel, len(keys), text, fn, keys) + } + return keys[0] +} + +// textMatch reports whether src is text, or text then a token boundary: a +// character that cannot continue text's last token. +func textMatch(src, text string) bool { + if !strings.HasPrefix(src, text) { + return false + } + if len(src) == len(text) { + return true + } + word := func(c byte) bool { + return c == '_' || c >= '0' && c <= '9' || c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' + } + return !(word(text[len(text)-1]) && word(src[len(text)])) +} + +// siteIn is the site of res in callable id. +func siteIn(t *testing.T, res *Result, id string) Site { + t.Helper() + for _, s := range res.Sites { + if s.Callable == id { + return s + } + } + t.Fatalf("no site in %s", id) + return Site{} +} + +// TestMR2_ReverseDeclarations_Exact: reversing mr01_04_layout.go's +// declarations (the method Scale goes back below its type) leaves the -a 4 +// analysis exactly as it was, once every key moves to its statement's new +// line. +func TestMR2_ReverseDeclarations_Exact(t *testing.T) { + tg := examples + tg.load(t) + res, post := transformAndAnalyze(t, tg, MR2, tg.ws, []string{"mr01_04_layout.go"}, tg.pre.view) + mapped, unmapped := tg.pre.view.Map(res.Map) + if len(unmapped) > 0 { + t.Fatalf("keys do not map: %v", unmapped) + } + if d := Diff(mapped, post.view, Scope{}, 20); len(d) > 0 { + t.Errorf("MR2 changed the analysis:\n%s", strings.Join(d, "\n")) + } +} + +// TestMR9_InsertKill_Delta: in KillOneUse, inserting `x = 0` before +// `sinkInt(x)` moves both def edges of `x := a + 1` on x (to the call and +// to the return) to the kill and splices the kill into the CFG; the CDG is +// the reference CDG of that CFG, and nothing else in the module changes. +func TestMR9_InsertKill_Delta(t *testing.T) { + tg := examples + tg.load(t) + res, post := transformAndAnalyze(t, tg, MR9, tg.ws, []string{"mr09_kill.go"}, tg.pre.view) + site := siteIn(t, res, exPkg+"KillOneUse") + var k Ref // the kill, the node the transform adds (its one mark) + for m := range site.Delta.Nodes { + k = Mark(m) + } + d := stmtKey(t, "mr09_kill.go", "KillOneUse", "x := a + 1") + u := stmtKey(t, "mr09_kill.go", "KillOneUse", "sinkInt(x)") + r := stmtKey(t, "mr09_kill.go", "KillOneUse", "return x") + want := Delta{Site: site.Desc, Callable: site.Callable, CDGFromCFG: true, Nodes: site.Delta.Nodes, + Remove: []DEdge{ + DDG(Pre(d), Pre(u), "x"), + DDG(Pre(d), Pre(r), "x"), + CFG(Pre(d), Pre(u), "fallthrough"), + }, + Add: []DEdge{ + DDG(k, Pre(u), "x"), + DDG(k, Pre(r), "x"), + CFG(Pre(d), k, "fallthrough"), + CFG(k, Pre(u), "fallthrough"), + }, + } + mapped, unmapped := tg.pre.view.Map(res.Map) + if len(unmapped) > 0 { + t.Fatalf("keys do not map: %v", unmapped) + } + var ds []Delta + for _, s := range res.Sites { + if s.Callable == want.Callable { + ds = append(ds, want) + } else { + ds = append(ds, *s.Delta) + } + } + exp, err := Expect(tg.pre.view, mapped, res.Map, res.Marks, ds) + if err != nil { + t.Fatal(err) + } + if diff := Diff(exp, post.view, Scope{}, 20); len(diff) > 0 { + t.Errorf("MR9 delta:\n%s", strings.Join(diff, "\n")) + } +} + +// TestMR5_IntroduceAlias_Reach: rewriting `p.n = v` in SetPtr as +// `q := p; q.n = v` keeps every flow from SetPtr's formal_ins: v still +// reaches the formal_out of p, now through the alias oracle. +func TestMR5_IntroduceAlias_Reach(t *testing.T) { + tg := examples + tg.load(t) + res, post := transformAndAnalyze(t, tg, MR5, tg.ws, []string{"mr05_alias.go"}, tg.pre.view) + f := exPkg + "SetPtr" + if !slices.Contains(res.Touched, f) { + t.Fatalf("MR5 did not rewrite %s", f) + } + rr := CheckReach(tg.pre.view, post.view, res.Map, []string{f}, 1, seed) + if rr.Pairs == 0 { + t.Fatal("no flow checked") + } + if len(rr.Lost) > 0 || len(rr.Unlocated) > 0 { + t.Errorf("MR5: lost %v, unlocated %v", rr.Lost, rr.Unlocated) + } + // The flow the alias carries: v (formal_in 1) → p's formal_out. + if !post.view.Graph(true).Reach(Node(f, "@formal_in:1"))[Node(f, "@formal_out")] { + t.Error("after MR5, v no longer reaches SetPtr's formal_out for p") + } +} + +// ─── the examples module ────────────────────────────────────────────────────── + +// TestExamples_EveryRelationHasASite: every relation MR1-MR15 transforms at +// least one site of testdata/metamorphic, and the relations with a file of +// their own land on the functions written for them: MR13 on mr13_*, each +// MR14 form (a interface, b closure, c defer) on mr14_*, MR7 exact on a +// named file, MR15 byte-identical on the MR6 example. +func TestExamples_EveryRelationHasASite(t *testing.T) { + tg := examples + tg.load(t) + sitesIn := func(res *Result, file string) []Site { + var out []Site + for _, s := range res.Sites { + if strings.HasPrefix(s.Desc, file+":") { + out = append(out, s) + } + } + return out + } + for _, r := range relations { + res, err := tg.ws.Apply(r.t, tg.files(), filepath.Join(t.TempDir(), r.name), seed, tg.pre.view) + if err != nil { + t.Fatal(err) + } + t.Logf("%s: %d sites", r.name, len(res.Sites)) + if len(res.Sites) == 0 { + t.Errorf("%s has no site on the examples module (skipped: %v)", r.name, res.Skipped) + } + switch r.name { + case "MR13": + want := []string{"ConstScalar", "ConstTwoArgs", "ConstInLoop"} + for _, fn := range want { + if !slices.ContainsFunc(sitesIn(res, "mr13_const_sink.go"), func(s Site) bool { return s.Callable == exPkg+fn }) { + t.Errorf("MR13 has no site in %s (mr13_const_sink.go)", fn) + } + } + case "MR14": + forms := map[string]string{ // the function written for each form → its Desc + "RouteIface": "call sinkInt through mr14i", + "RouteClosure": "call through a closure", + "RouteDefer": "defer the call", + } + for fn, form := range forms { + if !slices.ContainsFunc(sitesIn(res, "mr14_route.go"), func(s Site) bool { + return s.Callable == exPkg+fn && strings.Contains(s.Desc, form) + }) { + t.Errorf("MR14 form %q has no site in %s (mr14_route.go)", form, fn) + } + } + } + } + + // MR7: --target-files on mr09_kill.go at -a 3 is exact on its callables. + full := tg.level3(t) + const rel7 = "mr09_kill.go" + part, err := analyze(tg.ws.Dir, 3, []string{filepath.Join(tg.ws.Dir, rel7)}) + if err != nil { + t.Fatal(err) + } + only := map[string]bool{} + for id, c := range full.view.Calls { + if c.File == rel7 { + only[id] = true + } + } + if len(only) < 3 { + t.Fatalf("MR7: only %d callables in %s", len(only), rel7) + } + if d := Diff(full.view, part.view, Scope{Only: only}, 20); len(d) > 0 { + t.Errorf("MR7 on %s:\n%s", rel7, strings.Join(d, "\n")) + } + + // MR15: MR6 extracts a helper from Extract, and Inline6 gives + // mr06_extract.go back byte for byte. + const rel15 = "mr06_extract.go" + ext, err := tg.ws.Apply(MR6, []string{rel15}, filepath.Join(t.TempDir(), "mr15x"), seed, nil) + if err != nil { + t.Fatal(err) + } + if !slices.ContainsFunc(ext.Sites, func(s Site) bool { return s.Callable == exPkg+"Extract" }) { + t.Fatalf("MR15: MR6 extracted nothing from Extract: %v", ext.Skipped) + } + mid, _ := os.ReadFile(filepath.Join(ext.Dir, rel15)) + if !strings.Contains(string(mid), "func "+Prefix6) { + t.Fatalf("MR15: %s has no MR6 helper after the extraction", rel15) + } + ws2, err := LoadWorkspace(ext.Dir) + if err != nil { + t.Fatal(err) + } + inl, err := ws2.Apply(Inline6, []string{rel15}, filepath.Join(t.TempDir(), "mr15i"), seed, nil) + if err != nil { + t.Fatal(err) + } + orig, _ := os.ReadFile(filepath.Join(tg.ws.Dir, rel15)) + back, _ := os.ReadFile(filepath.Join(inl.Dir, rel15)) + if string(orig) != string(back) { + t.Errorf("MR15: inlining does not restore %s byte for byte:\n--- original\n%s\n--- inlined\n%s", rel15, orig, back) + } +} + +// ─── convention pins (testdata/metamorphic/conventions.go) ──────────────────── +// +// Task 17b encoded four analyzer conventions as oracle rules. Each test pins +// the analyzer fact its rule rests on, so a change that invalidates the rule +// fails here first, naming it, instead of silently weakening a relation. + +// TestConvention_CaptureSiteActualInIsNotASink pins rule (i), the +// reachability oracle's sink set: a capture site's actual_in has exactly +// one out-edge, the param_in to its closure's capture formal_in (no ddg, +// no summary), so it is not a sink. A call node's actual_in is one. +func TestConvention_CaptureSiteActualInIsNotASink(t *testing.T) { + const rule = "rule (i), capture-site actual_ins are not reachability sinks (CheckReach, View.isSink)" + tg := examples + tg.load(t) + v := tg.pre.view + id := exPkg + "ConvCaptureSite" + site := stmtKey(t, "conventions.go", "ConvCaptureSite", "f := func()") // the literal's host + litKey := "" + for k, b := range v.Calls[id].Body { + if strings.HasSuffix(k, "/actual_in:0") && strings.Split(b, "|")[1] == site { + litKey = strings.TrimSuffix(k, "/actual_in:0") + } + } + if litKey == "" { + t.Fatalf("%s: ConvCaptureSite has no capture site hosted at %s", rule, site) + } + ain := Node(id, litKey+"/actual_in:0") + g := v.Graph(false) + succ := g[ain] + if len(succ) != 1 || !strings.HasPrefix(succ[0], id+".closure_1|@formal_in:") { + t.Errorf("%s: %s's out-edges are %v, want exactly the param_in to closure_1's capture formal_in", rule, ain, succ) + } + if v.isSink(ain) { + t.Errorf("%s: %s is treated as a sink", rule, ain) + } + call := Node(exPkg+"KillOneUse", stmtKey(t, "mr09_kill.go", "KillOneUse", "sinkInt(x)")+"/actual_in:0") + if !v.isSink(call) { + t.Errorf("%s: the call actual_in %s is not a sink", rule, call) + } +} + +// TestConvention_GuardSelfLoopOnlyInLoops pins rule (ii), MR8's guard edge: +// the header `if c := true; c` folds init and condition into one node, so +// the guard c adds the ddg self-loop B→B on c exactly when the wrapped +// statement is on a CFG cycle (here, in a loop body), and no edge in +// straight-line code. +func TestConvention_GuardSelfLoopOnlyInLoops(t *testing.T) { + const rule = "rule (ii), MR8's guard self-loop B→B on c iff the wrapped statement is in a loop" + tg := examples + tg.load(t) + res, post := transformAndAnalyze(t, tg, MR8, tg.ws, []string{"conventions.go"}, tg.pre.view) + for fn, wantLoop := range map[string]bool{"ConvGuardInLoop": true, "ConvGuardStraight": false} { + id := exPkg + fn + s := siteIn(t, res, id) + var b string + for m := range s.Delta.Nodes { + b = res.Marks[m].Key() + } + var guard []string + for e := range post.view.Calls[id].Edges { + if f := strings.Split(e, "|"); f[0] == "ddg" && strings.HasPrefix(f[3], "mr8c") { + guard = append(guard, e) + } + } + // The guard exists: B is a branch node, its source text is the + // `if := true; {` header MR8 wrote, and B → the wrapped + // statement is its true edge. Its variable is read from that text, + // so the no-edge check below cannot pass for a renamed guard. + if a := post.view.Calls[id].Body[b]; !strings.HasPrefix(a, "branch|") { + t.Fatalf("%s: %s: node %s is %q, want the guard branch", rule, fn, b, a) + } + g := guardVar(t, res, "conventions.go", b) + var on []string + for e := range post.view.Calls[id].Edges { + if f := strings.Split(e, "|"); f[0] == "ddg" && f[3] == g { + on = append(on, e) + } + } + if len(guard) != len(on) { + t.Errorf("%s: %s: guard edges %v are not the edges on %s, %v", rule, fn, guard, g, on) + } + switch { + case wantLoop && (len(on) != 1 || on[0] != "ddg|"+b+"|"+b+"|"+g+"|ssa"): + t.Errorf("%s: %s: edges on %s are %v, want the one self-loop on %s", rule, fn, g, on, b) + case !wantLoop && len(on) != 0: + t.Errorf("%s: %s: edges on %s are %v, want none", rule, fn, g, on) + } + } +} + +// guardVar is the variable of the `if := true; {` header MR8 wrote +// at body key b of file rel in the transformed module res. +func guardVar(t *testing.T, res *Result, rel, b string) string { + t.Helper() + var p Pos + for _, m := range res.Marks { + if m.File == rel && m.Key() == b { + p = m + } + } + src, err := os.ReadFile(filepath.Join(res.Dir, p.File)) + if err != nil { + t.Fatal(err) + } + off, ok := newLineTable(src).offset(p.Line, p.Col) + if !ok { + t.Fatalf("no position %v", p) + } + head := string(src[off:]) + var c string + if _, err := fmt.Sscanf(head, "if %s := true;", &c); err != nil || !strings.HasPrefix(head, "if "+c+" := true; "+c+" {") { + t.Fatalf("no guard header at %v: %.40q", p, head) + } + return c +} + +// TestConvention_DiscardedResultReachesOnlyViaHost pins rule (iii), MR14(b)'s +// cut: a discarded result's actual_out has the one out-edge actual_out → +// call node (the coarse host wiring, spec §4.7), and the call node reaches +// the rest of the callable only by control dependence (it may panic, an +// exception edge), so the closure form may lose that path. +func TestConvention_DiscardedResultReachesOnlyViaHost(t *testing.T) { + const rule = "rule (iii), MR14(b) cuts a discarded actual_out (reaches on only via host cdg)" + tg := examples + tg.load(t) + c := tg.pre.view.Calls[exPkg+"ConvDiscardedResult"] + call := stmtKey(t, "conventions.go", "ConvDiscardedResult", "p.sum()") + ret := stmtKey(t, "conventions.go", "ConvDiscardedResult", "return p.a") + out := edgesOf(c, "ddg") + from := filter(out, func(e edgeRec) bool { return e.src == call+"/actual_out" }) + if len(from) != 1 || from[0].dst != call || from[0].label != "" { + t.Errorf("%s: %s/actual_out edges %v, want only → the call node on ", rule, call, from) + } + if !c.Edges["cdg|"+call+"|"+ret+"|"] { + t.Errorf("%s: no cdg %s → %s (the call may panic)", rule, call, ret) + } + if len(filter(out, func(e edgeRec) bool { return e.src == call && e.dst == ret })) != 0 { + t.Errorf("%s: the call node has a ddg edge to the return: the result is not discarded", rule) + } +} + +// TestConvention_NamedResultPortMovesToKill pins rule (iv), MR9 on a named +// result: the def `n = a + 1` reaches @formal_out through the @exit port +// edge (reaching-defs, var ), and the kill MR9 inserts before the +// return takes it over. +func TestConvention_NamedResultPortMovesToKill(t *testing.T) { + const rule = "rule (iv), MR9's kill takes over a named result's D → @formal_out port edge" + tg := examples + tg.load(t) + id := exPkg + "ConvNamedKill" + d := stmtKey(t, "conventions.go", "ConvNamedKill", "n = a + 1") + port := "ddg|" + d + "|@formal_out||reaching-defs" + if !tg.pre.view.Calls[id].Edges[port] { + t.Fatalf("%s: no port edge %s before the kill", rule, port) + } + res, post := transformAndAnalyze(t, tg, MR9, tg.ws, []string{"conventions.go"}, tg.pre.view) + s := siteIn(t, res, id) + var k string + for m := range s.Delta.Nodes { + k = res.Marks[m].Key() + } + nd, _, ok := (&mapper{pm: res.Map}).key("conventions.go", d) + if !ok { + t.Fatalf("%s: %s does not map", rule, d) + } + edges := post.view.Calls[id].Edges + if edges["ddg|"+nd+"|@formal_out||reaching-defs"] { + t.Errorf("%s: the def %s still reaches @formal_out past the kill", rule, nd) + } + if !edges["ddg|"+k+"|@formal_out||reaching-defs"] { + t.Errorf("%s: the kill %s has no port edge to @formal_out", rule, k) + } +} + +// TestStmtKey_TokenBoundary: stmtKey's text match needs the whole text or a +// token boundary after it, so one statement prefix never matches a longer +// literal or identifier. +func TestStmtKey_TokenBoundary(t *testing.T) { + for _, c := range []struct { + src, text string + want bool + }{ + {"x := a + 1", "x := a + 1", true}, + {"x := a + 10", "x := a + 1", false}, + {"x := a + 1 // c", "x := a + 1", true}, + {"sinkInt(x)", "sinkInt(x)", true}, + {"sinkInt(x)", "sinkInt(", true}, + {"return xy", "return x", false}, + {"return x * x", "return x", true}, + } { + if got := textMatch(c.src, c.text); got != c.want { + t.Errorf("textMatch(%q, %q) = %v, want %v", c.src, c.text, got, c.want) + } + } +} diff --git a/internal/metamorphic/realapp_test.go b/internal/metamorphic/realapp_test.go new file mode 100644 index 0000000..b12d95d --- /dev/null +++ b/internal/metamorphic/realapp_test.go @@ -0,0 +1,96 @@ +//go:build metamorphic + +package metamorphic + +// The metamorphic relations on the prepared eval apps (scripts/eval_l3.sh +// --prepare memos nats-server), run by scripts/metamorphic.sh: +// +// go test -tags metamorphic -run TestMR_RealApps ./internal/metamorphic/ +// +// Each app is $EVAL_SRC/ (default /tmp/eval). A missing app is +// skipped. MR_APPS (space-separated) picks the apps, MR_RELATIONS the +// relations (all when unset). Each relation samples MR_SAMPLE_FILES files +// (default 20) and MR_SAMPLE_CALLABLES callables (default 50) with seed 1. + +import ( + "os" + "path/filepath" + "slices" + "strconv" + "strings" + "testing" +) + +func realTargets(t *testing.T) []*target { + src := os.Getenv("EVAL_SRC") + if src == "" { + src = "/tmp/eval" + } + names := strings.Fields(os.Getenv("MR_APPS")) + if len(names) == 0 { + names = []string{"memos", "nats-server"} + } + var out []*target + for _, n := range names { + dir := filepath.Join(src, n) + if _, err := os.Stat(filepath.Join(dir, "go.mod")); err != nil { + t.Logf("%s: not prepared at %s; skipped", n, dir) + continue + } + out = append(out, &target{name: n, dir: dir, nFiles: envInt(t, "MR_SAMPLE_FILES", 20), nCallables: envInt(t, "MR_SAMPLE_CALLABLES", 50)}) + } + return out +} + +// envInt is the positive integer in env var name, or def when unset. +func envInt(t *testing.T, name string, def int) int { + v := os.Getenv(name) + if v == "" { + return def + } + n, err := strconv.Atoi(v) + if err != nil || n <= 0 { + t.Fatalf("%s=%q: want a positive integer", name, v) + } + return n +} + +func TestMR_RealApps(t *testing.T) { + tgs := realTargets(t) + if len(tgs) == 0 { + t.Skip("no prepared eval app under $EVAL_SRC") + } + want := strings.Fields(os.Getenv("MR_RELATIONS")) + pick := func(name string) bool { return len(want) == 0 || slices.Contains(want, name) } + for _, tg := range tgs { + t.Run(tg.name, func(t *testing.T) { + tg.load(t) + t.Logf("MRLOAD %s files=%d eligible=%d skipped=%v analysis=%s", + tg.name, len(tg.ws.Files), len(tg.ws.Eligible()), tg.ws.SkipCounts(), tg.pre.dur.Round(1e6)) + for _, r := range relations { + if !pick(r.name) { + continue + } + t.Run(r.name, func(t *testing.T) { + if why, ok := knownSkips[r.name+"/"+tg.name]; ok { + t.Skip(why) + } + o := runRelation(t, tg, r) + logStat(t, r.name, tg.name, o) + for _, v := range o.violations { + t.Errorf("%s on %s: %s", r.name, tg.name, v) + } + }) + } + if pick("MR15") { + t.Run("MR15", func(t *testing.T) { + o := runMR15(t, tg) + logStat(t, "MR15", tg.name, o) + for _, v := range o.violations { + t.Errorf("MR15 on %s: %s", tg.name, v) + } + }) + } + }) + } +} diff --git a/internal/metamorphic/run.go b/internal/metamorphic/run.go new file mode 100644 index 0000000..8c96d98 --- /dev/null +++ b/internal/metamorphic/run.go @@ -0,0 +1,100 @@ +// Package metamorphic holds the metamorphic relations (MRs) over the L3/L4 +// analysis: source transforms that must leave the analysis unchanged up to a +// known renaming (the exact oracle), or must keep every flow the original had +// (the one-way reachability oracle). Fixtures only catch an invariance or +// soundness bug when someone writes that exact case; the relations check it +// on any module, real apps included (spec §8, Metamorphic relations). +package metamorphic + +import ( + "fmt" + "io" + "path/filepath" + "sort" + + "github.com/codellm-devkit/codeanalyzer-go/internal/analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" +) + +// AppName is the --app-name every run uses, so both sides of a relation +// mint the same can:// prefix (external ids included). +const AppName = "mr" + +// Run is one in-process analysis: its normalized view and the load errors +// of the project packages (from the analyzer's own go/packages load). +type Run struct { + View *View + Errors []string +} + +// Analyze runs the analyzer pipeline over dir at level (3 or 4) the way +// core.Analyzer.Analyze does — symbol table, call graph, pass pipeline, +// dataflow, and at level 4 sdg.Run with its call-graph additions — minus +// `go mod download` (the inputs are already materialized), and emits the v2 +// payload. targets is --target-files (absolute paths); nil is a full run. +// Test files are skipped (the CLI default). +func Analyze(dir string, level int, targets []string) (*Run, error) { + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(targets, true) + if err != nil { + return nil, fmt.Errorf("symbol table: %w", err) + } + var errs []string + for _, p := range b.Pkgs() { + for _, e := range p.Errors { + errs = append(errs, p.PkgPath+": "+e.Msg) + } + } + sort.Strings(errs) + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + app := &schema.GoApplication{SymbolTable: st, CallGraph: cg.Build(st), Externals: cg.Externals(), Entrypoints: map[string][]schema.GoEntrypoint{}} + app.Externals = cg.Externals() + if err := analysis.RunPipeline(app, analysis.AnalysisContext{ProjectDir: dir}); err != nil { + return nil, err + } + units := b.L3Units(st) + const k, jobs = 3, 4 + var res map[dataflow.L3Key]dataflow.Result + var l4 *sdg.Result + if level >= 4 { + res = dataflow.RunWithSideTables(b.Fset(), units, k, jobs) + r := sdg.Run(sdg.Input{ + Fset: b.Fset(), Pkgs: b.Pkgs(), Variants: b.Loaded(), ProjectDir: dir, SymbolTable: st, + Externals: app.Externals, CallGraph: app.CallGraph, + Units: units, L3: res, K: k, Jobs: jobs, + }) + r.ApplyTo(app) + l4 = &r + } else { + res = dataflow.Run(b.Fset(), units, k, jobs) + } + dataflow.ReleaseAll(res) + // The payload is streamed (EmitTo) into the view one module at a time, + // so a large app's tree never exists in memory at once. + pr, pw := io.Pipe() + go func() { + pw.CloseWithError(v2emit.EmitTo(pw, app, AppName, dir, level, "metamorphic", k, res, l4)) + }() + v, err := Collect(pr) + pr.CloseWithError(err) + if err != nil { + return nil, err + } + return &Run{View: v, Errors: errs}, nil +} + +// realDir is dir with symlinks resolved: go/packages reports resolved file +// names, and the analyzer keys files relative to the input root, so a root +// under a symlink (macOS /var → /private/var) would put every file outside it. +func realDir(dir string) (string, error) { + abs, err := filepath.Abs(dir) + if err != nil { + return "", err + } + return filepath.EvalSymlinks(abs) +} diff --git a/internal/metamorphic/transform.go b/internal/metamorphic/transform.go new file mode 100644 index 0000000..30ab5be --- /dev/null +++ b/internal/metamorphic/transform.go @@ -0,0 +1,1438 @@ +package metamorphic + +// Source transforms. Each one rewrites some files of a module copy and +// returns a PosMap from every old position it kept to its new position, the +// marks it set in its new text, and its sites (what the relations check). +// +// The transforms find their sites with go/ast and go/types. The workspace is +// loaded with the analyzer's own go/packages mode. They never re-print a +// whole file: go/printer would reformat it and move free-floating comments, +// and with them every position in the file. Instead a new file is spliced +// from byte ranges of the old one, copied verbatim, and new text. That keeps +// the PosMap exact: a copied byte maps by its offset in the copy, and the +// start of a replaced range (an anchor) maps to the start of its +// replacement. + +import ( + "bytes" + "errors" + "fmt" + "go/ast" + "go/token" + "go/types" + "io/fs" + "math/rand/v2" + "os" + "path/filepath" + "slices" + "sort" + "strconv" + "strings" + + "golang.org/x/tools/go/packages" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// ─── positions ──────────────────────────────────────────────────────────────── + +// Pos is a source position: a file relative to the module root, a 1-based +// line and a 1-based byte column (the go/token convention body keys use). +type Pos struct { + File string + Line, Col int +} + +// Key is p's body key, "line:col". +func (p Pos) Key() string { return v2.LocalID(p.Line, p.Col) } + +// lineTable holds the offset of each line start of a file. +type lineTable []int + +func newLineTable(src []byte) lineTable { + t := lineTable{0} + for i, c := range src { + if c == '\n' { + t = append(t, i+1) + } + } + return t +} + +func (t lineTable) offset(line, col int) (int, bool) { + if line < 1 || line > len(t) || col < 1 { + return 0, false + } + return t[line-1] + col - 1, true +} + +func (t lineTable) pos(off int) (int, int) { + i := sort.Search(len(t), func(i int) bool { return t[i] > off }) - 1 + return i + 1, off - t[i] + 1 +} + +// seg maps old bytes [a, z) of a file to newFile at offset at (a copy), or, +// for an anchor, the single old offset a to at. +type seg struct { + a, z int + newFile string + at int +} + +// PosMap maps old positions to new ones. A file no transform rewrote maps to +// itself. In a rewritten file a position maps when its byte was copied or it +// is an anchor, and is unmapped (deleted or replaced text) otherwise. Every +// old byte is copied at most once, so the copies of a file are disjoint. +type PosMap struct { + old map[string]lineTable + new map[string]lineTable + copies map[string][]seg // by old file, sorted by a + anchors map[string]map[int]seg +} + +func newPosMap() *PosMap { + return &PosMap{old: map[string]lineTable{}, new: map[string]lineTable{}, + copies: map[string][]seg{}, anchors: map[string]map[int]seg{}} +} + +// Map returns p's new position. +func (m *PosMap) Map(p Pos) (Pos, bool) { + lt, ok := m.old[p.File] + if !ok { + return p, true + } + off, ok := lt.offset(p.Line, p.Col) + if !ok { + return Pos{}, false + } + s, ok := m.anchors[p.File][off] + if !ok { + cs := m.copies[p.File] + i := sort.Search(len(cs), func(i int) bool { return cs[i].a > off }) - 1 + if i < 0 || off >= cs[i].z { + return Pos{}, false + } + s = cs[i] + } + l, c := m.new[s.newFile].pos(s.at + off - s.a) + return Pos{File: s.newFile, Line: l, Col: c}, true +} + +// ─── workspace ──────────────────────────────────────────────────────────────── + +// Workspace is a module loaded for transformation: every project file with +// its syntax, types and source. +type Workspace struct { + Dir string + Fset *token.FileSet + Files map[string]*SrcFile // by path relative to Dir +} + +// SrcFile is one project file. +type SrcFile struct { + Rel string + Pkg *packages.Package + AST *ast.File + Src []byte + TF *token.File + lines lineTable + // Skip is why the file is not eligible for transformation ("" when it is). + Skip string +} + +func (f *SrcFile) off(p token.Pos) int { return f.TF.Offset(p) } + +// pos is the old position of p. +func (f *SrcFile) pos(p token.Pos) Pos { + l, c := f.lines.pos(f.off(p)) + return Pos{File: f.Rel, Line: l, Col: c} +} + +// key is the body key of p. +func (f *SrcFile) key(p token.Pos) string { return f.pos(p).Key() } + +// LoadWorkspace loads the module at dir with the analyzer's go/packages mode +// (test files excluded, as at the CLI default). +func LoadWorkspace(dir string) (*Workspace, error) { + dir, err := realDir(dir) + if err != nil { + return nil, err + } + fset := token.NewFileSet() + cfg := &packages.Config{ + Mode: packages.NeedName | packages.NeedFiles | packages.NeedSyntax | packages.NeedTypes | + packages.NeedTypesInfo | packages.NeedImports | packages.NeedDeps, + Dir: dir, Fset: fset, + } + pkgs, err := packages.Load(cfg, "./...") + if err != nil { + return nil, err + } + w := &Workspace{Dir: dir, Fset: fset, Files: map[string]*SrcFile{}} + for _, p := range pkgs { + for _, af := range p.Syntax { + tf := fset.File(af.Pos()) + if tf == nil || !utils.IsWithin(dir, tf.Name()) { + continue + } + rel := filepath.ToSlash(utils.RelativePath(dir, tf.Name())) + if utils.IsTestFile(rel) || utils.IsVendored(rel) { + continue + } + src, err := os.ReadFile(tf.Name()) + if err != nil { + return nil, err + } + f := &SrcFile{Rel: rel, Pkg: p, AST: af, Src: src, TF: tf, lines: newLineTable(src)} + f.Skip = skipReason(f) + w.Files[rel] = f + } + } + return w, nil +} + +// skipReason is why f is not transformed: cgo, //line directives and +// generated code (spans follow other rules there), type errors in its +// package (the transforms need its types), or a source that does not match +// its parse. +func skipReason(f *SrcFile) string { + switch { + case len(f.Pkg.Errors) > 0: + return "package has errors" + case f.TF.Size() != len(f.Src): + return "source differs from its parse" + case ast.IsGenerated(f.AST): + return "generated" + case f.Pkg.TypesInfo == nil: + return "no types" + } + for _, s := range f.AST.Imports { + if s.Path.Value == `"C"` { + return "cgo" + } + } + for _, cg := range f.AST.Comments { + for _, c := range cg.List { + if strings.HasPrefix(c.Text, "//line ") || strings.HasPrefix(c.Text, "/*line ") { + return "//line directive" + } + } + } + return "" +} + +// Eligible returns the transformable files, sorted, restricted to the +// package directories in dirs when dirs is non-empty. +func (w *Workspace) Eligible(dirs ...string) []string { + var out []string + for rel, f := range w.Files { + if f.Skip != "" { + continue + } + if len(dirs) > 0 && !slices.Contains(dirs, filepath.ToSlash(filepath.Dir(rel))) { + continue + } + out = append(out, rel) + } + sort.Strings(out) + return out +} + +// SkipCounts counts the ineligible files by reason. +func (w *Workspace) SkipCounts() map[string]int { + out := map[string]int{} + for _, f := range w.Files { + if f.Skip != "" { + out[f.Skip]++ + } + } + return out +} + +// Sample returns n of files chosen with seed (all when n <= 0 or n >= len). +func Sample(files []string, n int, seed uint64) []string { + out := slices.Clone(files) + sort.Strings(out) + if n <= 0 || n >= len(out) { + return out + } + r := rand.New(rand.NewPCG(seed, 0)) + r.Shuffle(len(out), func(i, j int) { out[i], out[j] = out[j], out[i] }) + out = out[:n] + sort.Strings(out) + return out +} + +// ─── transform driver ───────────────────────────────────────────────────────── + +// Transform is one source transform, applied file by file. edit returns +// nil, or errSkip, to leave a file unchanged. +type Transform struct { + Name string + edit func(c *tctx, f *SrcFile) (*fileEdit, error) +} + +// piece is a run of a new file's text: a literal, a copy of old bytes +// [a, z) of f, an anchor (old offset a of f maps to here; writes nothing) +// or a mark (records here under its name; writes nothing). +type piece struct { + lit string + f *SrcFile + a, z int + anchor bool + mark string +} + +func lit(s string) piece { return piece{lit: s} } +func cp(f *SrcFile, a, z int) piece { return piece{f: f, a: a, z: z} } +func anchorAt(f *SrcFile, a int) piece { return piece{f: f, a: a, z: a, anchor: true} } +func mark(name string) piece { return piece{mark: name} } + +type fileEdit struct { + pieces []piece // the old file's new text + created []newFile // files added next to it + touched []string // Identity of every callable changed + sites []Site + // skips counts the sites declined, by reason. + skips map[string]int + // violations are analyzer faults the transform found while choosing + // sites (a CDG that is not the reference CDG of its CFG). + violations []string +} + +func (e *fileEdit) skip(reason string) { + if e.skips == nil { + e.skips = map[string]int{} + } + e.skips[reason]++ +} + +type newFile struct { + rel string + pieces []piece +} + +// Site is one place a transform changed, with what its relation expects. +type Site struct { + Desc string // file:line and what was done + Callable string // Identity, in the original + // Delta is the expected change (delta and precision relations). + Delta *Delta + // NoFlow lists vertices (refs into Callable) that no @formal_in of + // Callable may reach over data edges after the transform (MR13). + NoFlow []Ref + // Reloc maps a call key of Callable in the original to its call node + // after the transform, in RelocTo (MR14a: the call moved into a method). + Reloc map[string]Ref + RelocTo string + // Summary lists summary edges that must exist after the transform in + // Callable, each the counterpart of one in the original (MR14a). + Summary []SumCheck + // Cut lists the Nodes of the original whose out-edges the reachability + // oracle leaves out of the original's graph (MR14b, see there). + Cut []string +} + +// SumCheck is one expected summary edge. +type SumCheck struct { + Pre string // the original edge, for the message + Src, Dst Ref +} + +// tctx is the per-run state of a transform. +type tctx struct { + w *Workspace + pre *View // the original's analysis (nil when the transform needs none) + rng *rand.Rand + n int // a counter for fresh names and marks +} + +func (c *tctx) fresh(prefix string) string { + c.n++ + return prefix + strconv.Itoa(c.n) +} + +// Result is a transformed module. +type Result struct { + Dir string // the transformed copy's root + Map *PosMap // old → new positions + Marks map[string]Pos + Files []string // rewritten + Created []string + // Touched is the Identity of every callable a transform changed, sorted. + Touched []string + Sites []Site + // Skipped counts, per reason, the files and sites left unchanged. + Skipped map[string]int + // Violations are analyzer faults found while choosing sites; the + // relation reports each one. + Violations []string +} + +// errSkip is returned by an edit that leaves its file unchanged. +type errSkip string + +func (e errSkip) Error() string { return string(e) } + +// Apply copies the module to dst, applies t to files with seed and returns +// the result. pre is the original's analysis, which the delta transforms +// read to choose and describe their sites. +func (w *Workspace) Apply(t Transform, files []string, dst string, seed uint64, pre *View) (*Result, error) { + if err := copyTree(w.Dir, dst); err != nil { + return nil, err + } + dst, err := realDir(dst) + if err != nil { + return nil, err + } + c := &tctx{w: w, pre: pre, rng: rand.New(rand.NewPCG(seed, 1))} + res := &Result{Dir: dst, Skipped: map[string]int{}, Map: newPosMap(), Marks: map[string]Pos{}} + touched := map[string]bool{} + for _, rel := range files { + f := w.Files[rel] + if f == nil || f.Skip != "" { + return nil, fmt.Errorf("%s: not an eligible file", rel) + } + fe, err := t.edit(c, f) + if fe != nil { + res.Violations = append(res.Violations, fe.violations...) + } + var skip errSkip + if errors.As(err, &skip) { + res.Skipped["file: "+string(skip)]++ + if fe != nil { + for r, n := range fe.skips { + res.Skipped["site: "+r] += n + } + } + continue + } + if err != nil { + return nil, fmt.Errorf("%s %s: %w", t.Name, rel, err) + } + if fe == nil || fe.pieces == nil { + res.Skipped["file: no site"]++ + continue + } + for r, n := range fe.skips { + res.Skipped["site: "+r] += n + } + if err := res.write(dst, rel, fe.pieces); err != nil { + return nil, err + } + res.Files = append(res.Files, rel) + for _, nf := range fe.created { + if _, err := os.Stat(filepath.Join(dst, nf.rel)); err == nil { + return nil, fmt.Errorf("%s: %s exists", t.Name, nf.rel) + } + if err := res.write(dst, nf.rel, nf.pieces); err != nil { + return nil, err + } + res.Created = append(res.Created, nf.rel) + } + for _, id := range fe.touched { + touched[id] = true + } + res.Sites = append(res.Sites, fe.sites...) + } + res.Touched = slices.Sorted(func(yield func(string) bool) { + for id := range touched { + if !yield(id) { + return + } + } + }) + for _, s := range res.Map.copies { + sort.Slice(s, func(i, j int) bool { return s[i].a < s[j].a }) + } + return res, nil +} + +// write assembles pieces into dst/rel and records their segments and marks. +func (r *Result) write(dst, rel string, ps []piece) error { + var buf bytes.Buffer + marks := map[string]int{} + for _, p := range ps { + switch { + case p.mark != "": + if _, dup := r.Marks[p.mark]; dup { + return fmt.Errorf("%s: duplicate mark %s", rel, p.mark) + } + marks[p.mark] = buf.Len() + continue + case p.f == nil: + buf.WriteString(p.lit) + continue + case p.a < 0 || p.z > len(p.f.Src) || p.a > p.z: + return fmt.Errorf("%s: bad piece [%d,%d) of %s", rel, p.a, p.z, p.f.Rel) + } + old := p.f.Rel + if _, ok := r.Map.old[old]; !ok { + r.Map.old[old] = p.f.lines + r.Map.anchors[old] = map[int]seg{} + } + s := seg{a: p.a, z: p.z, newFile: rel, at: buf.Len()} + if p.anchor { + r.Map.anchors[old][p.a] = s + continue + } + if p.z > p.a { + r.Map.copies[old] = append(r.Map.copies[old], s) + buf.Write(p.f.Src[p.a:p.z]) + } + } + lt := newLineTable(buf.Bytes()) + r.Map.new[rel] = lt + for name, off := range marks { + l, c := lt.pos(off) + r.Marks[name] = Pos{File: rel, Line: l, Col: c} + } + return os.WriteFile(filepath.Join(dst, rel), buf.Bytes(), 0o644) +} + +// copyTree copies the regular files under src to dst, skipping VCS and +// JavaScript dependency trees. +func copyTree(src, dst string) error { + return filepath.WalkDir(src, func(p string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + rel, _ := filepath.Rel(src, p) + if d.IsDir() { + if n := d.Name(); rel != "." && (n == ".git" || n == "node_modules") { + return filepath.SkipDir + } + return os.MkdirAll(filepath.Join(dst, rel), 0o755) + } + if !d.Type().IsRegular() { + return nil + } + data, err := os.ReadFile(p) + if err != nil { + return err + } + return os.WriteFile(filepath.Join(dst, rel), data, 0o644) + }) +} + +// edit is one change to a file: old bytes [a, z) replaced by out. With +// anchor, old offset a maps to the start of the replacement. +type edit struct { + a, z int + out []piece + anchor bool +} + +func insert(at int, out ...piece) edit { return edit{a: at, z: at, out: out} } + +// applyEdits is f's text with edits applied (they must not overlap; an +// insertion at an offset comes before a replacement starting there). +func applyEdits(f *SrcFile, edits []edit) ([]piece, error) { + sort.SliceStable(edits, func(i, j int) bool { + if edits[i].a != edits[j].a { + return edits[i].a < edits[j].a + } + return edits[i].z < edits[j].z + }) + var ps []piece + cur := 0 + for _, e := range edits { + if e.a < cur { + return nil, fmt.Errorf("overlapping edits at offset %d", e.a) + } + if e.a > cur { + ps = append(ps, cp(f, cur, e.a)) + } + if e.anchor { + ps = append(ps, anchorAt(f, e.a)) + } + ps = append(ps, e.out...) + cur = e.z + } + if cur < len(f.Src) { + ps = append(ps, cp(f, cur, len(f.Src))) + } + return ps, nil +} + +// ─── shared helpers ─────────────────────────────────────────────────────────── + +func identityOf(sig, file string) string { return sig + "#" + file } + +func lineStart(src []byte, off int) int { + return bytes.LastIndexByte(src[:off], '\n') + 1 +} + +// lineEnd is the offset just past the '\n' ending the line holding off +// (len(src) on the last line). +func lineEnd(src []byte, off int) int { + if i := bytes.IndexByte(src[off:], '\n'); i >= 0 { + return off + i + 1 + } + return len(src) +} + +// blankTail reports whether src[off:] up to the end of its line is blank or +// a line comment. +func blankTail(src []byte, off int) bool { + rest := strings.TrimLeft(string(src[off:lineEnd(src, off)]), " \t\r\n") + return rest == "" || strings.HasPrefix(rest, "//") +} + +func blankHead(src []byte, off int) bool { + return strings.TrimLeft(string(src[lineStart(src, off):off]), " \t") == "" +} + +func declDoc(d ast.Decl) *ast.CommentGroup { + switch d := d.(type) { + case *ast.FuncDecl: + return d.Doc + case *ast.GenDecl: + return d.Doc + } + return nil +} + +// chunk is the whole lines of decl i of f with its doc comment: [a, z). ok +// is false when another declaration shares one of those lines. +func chunk(f *SrcFile, i int) (a, z int, ok bool) { + d := f.AST.Decls[i] + start := d.Pos() + if doc := declDoc(d); doc != nil { + start = doc.Pos() + } + so, eo := f.off(start), f.off(d.End()) + if !blankHead(f.Src, so) || !blankTail(f.Src, eo) { + return 0, 0, false + } + a, z = lineStart(f.Src, so), lineEnd(f.Src, eo) + if i > 0 && f.off(f.AST.Decls[i-1].End()) > a { + return 0, 0, false + } + if i+1 < len(f.AST.Decls) { + n := f.AST.Decls[i+1] + ns := n.Pos() + if doc := declDoc(n); doc != nil { + ns = doc.Pos() + } + if f.off(ns) < z { + return 0, 0, false + } + } + return a, z, true +} + +func isImport(d ast.Decl) bool { + g, ok := d.(*ast.GenDecl) + return ok && g.Tok == token.IMPORT +} + +// funcIdentity is the comparer's Identity of FuncDecl fd of f. +func funcIdentity(f *SrcFile, fd *ast.FuncDecl) string { + obj := f.Pkg.TypesInfo.Defs[fd.Name] + if obj == nil { + return "" + } + sig := syntactic_analysis.SignatureOf(obj) + if fd.Recv == nil && fd.Name.Name == "init" { + return identityOf(sig, f.Rel) + } + return sig +} + +// localVar is obj as a function-local variable: a parameter, result, +// receiver or local, not a field or a package-level variable. +func localVar(f *SrcFile, obj types.Object) (*types.Var, bool) { + v, ok := obj.(*types.Var) + if !ok || v.IsField() || v.Pkg() != f.Pkg.Types || v.Parent() == nil || v.Parent() == v.Pkg().Scope() { + return nil, false + } + return v, true +} + +func contains[T ast.Node](n ast.Node, pred func(T) bool) bool { + found := false + ast.Inspect(n, func(n ast.Node) bool { + if x, ok := n.(T); ok && pred(x) { + found = true + } + return !found + }) + return found +} + +func hasGoto(n ast.Node) bool { + return contains(n, func(b *ast.BranchStmt) bool { return b.Tok == token.GOTO }) +} + +// usesPrefix reports whether any identifier of f's package starts with prefix. +func usesPrefix(f *SrcFile, prefix string) bool { + for _, af := range f.Pkg.Syntax { + if contains(af, func(id *ast.Ident) bool { return strings.HasPrefix(id.Name, prefix) }) { + return true + } + } + return false +} + +func funcDecls(f *SrcFile) []*ast.FuncDecl { + var out []*ast.FuncDecl + for _, d := range f.AST.Decls { + if fd, ok := d.(*ast.FuncDecl); ok { + out = append(out, fd) + } + } + return out +} + +// stmtLists calls fn with every statement list of body outside function +// literals: blocks, and case and comm clause bodies. +func stmtLists(body *ast.BlockStmt, fn func([]ast.Stmt)) { + ast.Inspect(body, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.FuncLit: + return false + case *ast.BlockStmt: + fn(n.List) + case *ast.CaseClause: + fn(n.Body) + case *ast.CommClause: + fn(n.Body) + } + return true + }) +} + +// ─── MR1: blank lines and comments ──────────────────────────────────────────── + +// MR1 inserts a comment and blank lines at the top of each file and before +// every top-level declaration. +var MR1 = Transform{Name: "MR1", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + edits := []edit{insert(0, lit("// mr1: inserted at the top\n\n"))} + prevEnd := 0 + n := 0 + for _, d := range f.AST.Decls { + start := d.Pos() + if doc := declDoc(d); doc != nil { + start = doc.Pos() + } + at := lineStart(f.Src, f.off(start)) + if at >= prevEnd && at > 0 { + edits = append(edits, insert(at, lit("\n// mr1: inserted between declarations\n\n\n"))) + n++ + } + prevEnd = f.off(d.End()) + } + ps, err := applyEdits(f, edits) + return &fileEdit{pieces: ps, sites: []Site{{Desc: fmt.Sprintf("%s: %d insertions", f.Rel, n+1)}}}, err +}} + +// ─── MR2: reverse the top-level declarations ────────────────────────────────── + +// MR2 reverses the order of a file's top-level declarations after its +// imports, so methods land above their types. Each declaration moves with its +// doc comment and the comments before it. +var MR2 = Transform{Name: "MR2", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + decls := f.AST.Decls + first := 0 + for first < len(decls) && isImport(decls[first]) { + first++ + } + if len(decls)-first < 2 { + return nil, errSkip("fewer than 2 declarations") + } + header := lineEnd(f.Src, f.off(f.AST.Name.End())) + if first > 0 { + header = lineEnd(f.Src, f.off(decls[first-1].End())) + } + // Chunk i runs from the end of decl i-1's last line to the end of decl + // i's last line: it holds the comments before decl i. + var bounds []int + for i := first; i < len(decls); i++ { + if _, _, ok := chunk(f, i); !ok { + return nil, errSkip("declarations share a line") + } + bounds = append(bounds, lineEnd(f.Src, f.off(decls[i].End()))) + } + bounds[len(bounds)-1] = len(f.Src) + ps := []piece{cp(f, 0, header)} + for i := len(bounds) - 1; i >= 0; i-- { + a := header + if i > 0 { + a = bounds[i-1] + } + ps = append(ps, cp(f, a, bounds[i])) + if !bytes.HasSuffix(f.Src[a:bounds[i]], []byte("\n")) { + ps = append(ps, lit("\n")) + } + } + var touched []string + for _, fd := range funcDecls(f) { + touched = append(touched, funcIdentity(f, fd)) + } + return &fileEdit{pieces: ps, touched: touched, + sites: []Site{{Desc: fmt.Sprintf("%s: %d declarations reversed", f.Rel, len(bounds))}}}, nil +}} + +// ─── MR3: move a function and a method to a new file ────────────────────────── + +// MR3 moves one top-level function and one method of a file (never an init) +// into a new file of the same package, which imports what they use. An +// import of the old file that only they used becomes a blank import, so the +// old file keeps its lines and still type-checks. +var MR3 = Transform{Name: "MR3", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + info := f.Pkg.TypesInfo + for _, s := range f.AST.Imports { + if s.Name != nil && s.Name.Name == "." { + return nil, errSkip("dot import") + } + } + var fns, meths []int + for i, d := range f.AST.Decls { + fd, ok := d.(*ast.FuncDecl) + if !ok || fd.Name.Name == "init" || fd.Name.Name == "_" { + continue + } + if _, _, ok := chunk(f, i); !ok { + continue + } + if fd.Recv == nil { + fns = append(fns, i) + } else { + meths = append(meths, i) + } + } + var move []int + if len(fns) > 0 { + move = append(move, fns[c.rng.IntN(len(fns))]) + } + if len(meths) > 0 { + move = append(move, meths[c.rng.IntN(len(meths))]) + } + if len(move) == 0 { + return nil, errSkip("nothing to move") + } + slices.Sort(move) + + pkgUses := func(n ast.Node) map[*types.PkgName]bool { + out := map[*types.PkgName]bool{} + ast.Inspect(n, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + if pn, ok := info.Uses[id].(*types.PkgName); ok { + out[pn] = true + } + } + return true + }) + return out + } + moved, kept := map[*types.PkgName]bool{}, map[*types.PkgName]bool{} + for i, d := range f.AST.Decls { + dst := kept + if slices.Contains(move, i) { + dst = moved + } + for pn := range pkgUses(d) { + dst[pn] = true + } + } + + var edits []edit + var imports []string + for _, s := range f.AST.Imports { + pn := importName(info, s) + if pn == nil || pn.Name() == "_" { + continue + } + if moved[pn] { + imports = append(imports, pn.Name()+" "+s.Path.Value) + } + if !kept[pn] { + edits = append(edits, edit{a: f.off(s.Pos()), z: f.off(s.End()), out: []piece{lit("_ " + s.Path.Value)}}) + } + } + created := []piece{lit("package " + f.AST.Name.Name + "\n\n")} + if len(imports) > 0 { + sort.Strings(imports) + created = append(created, lit("import (\n\t"+strings.Join(imports, "\n\t")+"\n)\n")) + } + base := strings.TrimSuffix(filepath.Base(f.Rel), ".go") + "_mr3.go" + rel := filepath.ToSlash(filepath.Join(filepath.Dir(f.Rel), base)) + var touched []string + var sites []Site + for _, i := range move { + a, z, _ := chunk(f, i) + edits = append(edits, edit{a: a, z: z}) + created = append(created, lit("\n"), cp(f, a, z)) + if !bytes.HasSuffix(f.Src[a:z], []byte("\n")) { + created = append(created, lit("\n")) + } + fd := f.AST.Decls[i].(*ast.FuncDecl) + id := funcIdentity(f, fd) + touched = append(touched, id) + sites = append(sites, Site{Desc: fmt.Sprintf("%s:%d: %s moved to %s", f.Rel, f.pos(fd.Pos()).Line, fd.Name.Name, rel), Callable: id}) + } + ps, err := applyEdits(f, edits) + if err != nil { + return nil, err + } + return &fileEdit{pieces: ps, touched: touched, sites: sites, created: []newFile{{rel: rel, pieces: created}}}, nil +}} + +// importName is the package name import spec s declares in its file. +func importName(info *types.Info, s *ast.ImportSpec) *types.PkgName { + if s.Name != nil { + pn, _ := info.Defs[s.Name].(*types.PkgName) + return pn + } + pn, _ := info.Implicits[s].(*types.PkgName) + return pn +} + +// ─── MR4: alpha-rename locals ───────────────────────────────────────────────── + +// Suffix4 is the suffix MR4 appends to every local variable and parameter +// name. A file whose package already has an identifier containing it is +// skipped, so the comparer can undo the rename by stripping it. +const Suffix4 = "_mr4" + +// MR4 renames every local variable, parameter, named result and receiver of +// the file's functions (closures included) to name+Suffix4. +var MR4 = Transform{Name: "MR4", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + for _, af := range f.Pkg.Syntax { + if contains(af, func(id *ast.Ident) bool { return strings.Contains(id.Name, Suffix4) }) { + return nil, errSkip("name clash") + } + } + info := f.Pkg.TypesInfo + var edits []edit + var touched []string + n := 0 + rename := func(id *ast.Ident) { + edits = append(edits, edit{a: f.off(id.Pos()), z: f.off(id.End()), anchor: true, out: []piece{lit(id.Name + Suffix4)}}) + n++ + } + for _, fd := range funcDecls(f) { + before := n + ast.Inspect(fd, func(nd ast.Node) bool { + switch x := nd.(type) { + case *ast.TypeSwitchStmt: + // `switch v := x.(type)`: v has no object of its own; each + // clause has an implicit one, which its uses resolve to. + if as, ok := x.Assign.(*ast.AssignStmt); ok { + if id, ok := as.Lhs[0].(*ast.Ident); ok && id.Name != "_" { + rename(id) + } + } + case *ast.Ident: + if x.Name == "_" { + return true + } + obj := info.Defs[x] + if obj == nil { + obj = info.Uses[x] + } + if _, ok := localVar(f, obj); ok { + rename(x) + } + } + return true + }) + if n > before { + touched = append(touched, funcIdentity(f, fd)) + } + } + if len(edits) == 0 { + return nil, errSkip("no locals") + } + ps, err := applyEdits(f, edits) + return &fileEdit{pieces: ps, touched: touched, + sites: []Site{{Desc: fmt.Sprintf("%s: %d identifiers renamed", f.Rel, n)}}}, err +}} + +// ─── MR5: introduce an alias ────────────────────────────────────────────────── + +// MR5 rewrites each `x.f = e` of the file, where x is a local and f a direct +// field of it, as `q := x; q.f = e` when x is a pointer to a struct, or as +// `q := &x; q.f = e` when x is a struct value. q is fresh. The write now goes +// through an alias of x. Other bases are left alone: a value copy does not +// alias. +var MR5 = Transform{Name: "MR5", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + if usesPrefix(f, "mr5q") { + return nil, errSkip("name clash") + } + info := f.Pkg.TypesInfo + fe := &fileEdit{} + var edits []edit + for _, fd := range funcDecls(f) { + if fd.Body == nil || hasGoto(fd.Body) { + continue + } + before := len(fe.sites) + stmtListsAll(fd.Body, func(list []ast.Stmt) { + for _, s := range list { + as, ok := s.(*ast.AssignStmt) + if !ok || as.Tok != token.ASSIGN || len(as.Lhs) != 1 || len(as.Rhs) != 1 { + continue + } + sel, ok := as.Lhs[0].(*ast.SelectorExpr) + if !ok { + continue + } + x, ok := sel.X.(*ast.Ident) + if !ok { + continue + } + v, ok := localVar(f, info.Uses[x]) + if !ok { + continue + } + if s := info.Selections[sel]; s == nil || s.Kind() != types.FieldVal || len(s.Index()) != 1 { + continue + } + var rhs string + switch t := v.Type().(type) { + case *types.Pointer: + if _, ok := t.Elem().Underlying().(*types.Struct); !ok { + fe.skip("MR5: not a struct pointer") + continue + } + rhs = x.Name + case *types.Named: + if _, ok := t.Underlying().(*types.Struct); !ok { + fe.skip("MR5: not a struct") + continue + } + rhs = "&" + x.Name + default: + fe.skip("MR5: not a struct or a pointer to one") + continue + } + q := c.fresh("mr5q") + at := f.off(as.Pos()) + edits = append(edits, + insert(at, lit(q+" := "+rhs+"; ")), + edit{a: at, z: f.off(x.End()), anchor: true, out: []piece{lit(q)}}) + fe.sites = append(fe.sites, Site{Desc: fmt.Sprintf("%s:%d: %s via %s", f.Rel, f.lines.lineOf(at), x.Name, q), Callable: funcIdentity(f, fd)}) + } + }) + if len(fe.sites) > before { + fe.touched = append(fe.touched, funcIdentity(f, fd)) + } + } + if len(edits) == 0 { + return fe, errSkip("no field write") + } + ps, err := applyEdits(f, edits) + fe.pieces = ps + return fe, err +}} + +// stmtListsAll is stmtLists including function literal bodies. +func stmtListsAll(body *ast.BlockStmt, fn func([]ast.Stmt)) { + ast.Inspect(body, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.BlockStmt: + fn(n.List) + case *ast.CaseClause: + fn(n.Body) + case *ast.CommClause: + fn(n.Body) + } + return true + }) +} + +// ─── MR6: extract a helper ──────────────────────────────────────────────────── + +// Prefix6 starts every MR6 helper's name. +const Prefix6 = "mr6h_" + +// MR6 extracts, in each function of the file, one straight-line run of 2-4 +// statements into a new top-level helper at the end of the file. The helper +// takes every outer local the run names and returns them (in-out, so a +// write to a copy is assigned back), then the locals the run declares that +// the function uses after it: +// +// var n1 T1; o1, o2, n1 = mr6h_f_1(o1, o2) +// +// A run qualifies when: +// - its statements are simple (assignment, expression, ++/--, send, var +// declaration) and hold no function literal or recover(); +// - it names no outer local that a closure captures or that is +// address-taken anywhere in the function (copy-in/copy-out would then +// change what the program does); +// - it names no function-local type or constant; +// - every type in the helper's signature can be spelled in the file. +// +// Generic functions and functions with a goto are skipped. +var MR6 = Transform{Name: "MR6", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + if usesPrefix(f, Prefix6) { + return nil, errSkip("name clash") + } + fe := &fileEdit{} + var edits []edit + var helpers []piece + for _, fd := range funcDecls(f) { + if fd.Body == nil || fd.Type.TypeParams != nil || hasGoto(fd.Body) || recvGeneric(fd) { + continue + } + runs := mr6Runs(f, fd) + if len(runs) == 0 { + continue + } + r := runs[c.rng.IntN(len(runs))] + name := c.fresh(Prefix6 + fd.Name.Name + "_") + var params, results, args, outs []string + for _, v := range r.outer { + params = append(params, v.v.Name()+" "+v.t) + results = append(results, v.t) + args = append(args, v.v.Name()) + outs = append(outs, v.v.Name()) + } + var decl string + for _, v := range r.fresh { + decl += "var " + v.v.Name() + " " + v.t + "; " + results = append(results, v.t) + outs = append(outs, v.v.Name()) + } + call := name + "(" + strings.Join(args, ", ") + ")" + if len(outs) > 0 { + call = decl + strings.Join(outs, ", ") + " = " + call + } + edits = append(edits, edit{a: r.a, z: r.z, out: []piece{lit(call)}}) + head := "\n\n// " + name + " is extracted by MR6.\nfunc " + name + "(" + strings.Join(params, ", ") + ")" + if len(results) > 0 { + head += " (" + strings.Join(results, ", ") + ")" + } + helpers = append(helpers, lit(head+" {\n\t"), cp(f, r.a, r.z)) + if len(outs) > 0 { + helpers = append(helpers, lit("\n\treturn "+strings.Join(outs, ", "))) + } + helpers = append(helpers, lit("\n}\n")) + id := funcIdentity(f, fd) + fe.touched = append(fe.touched, id) + fe.sites = append(fe.sites, Site{Desc: fmt.Sprintf("%s:%d: %d statements into %s", f.Rel, f.lines.lineOf(r.a), r.n, name), Callable: id}) + } + if len(edits) == 0 { + return fe, errSkip("no run") + } + edits = append(edits, insert(len(f.Src), helpers...)) + ps, err := applyEdits(f, edits) + fe.pieces = ps + return fe, err +}} + +func (t lineTable) lineOf(off int) int { + l, _ := t.pos(off) + return l +} + +func recvGeneric(fd *ast.FuncDecl) bool { + if fd.Recv == nil || len(fd.Recv.List) == 0 { + return false + } + t := fd.Recv.List[0].Type + if s, ok := t.(*ast.StarExpr); ok { + t = s.X + } + switch t.(type) { + case *ast.IndexExpr, *ast.IndexListExpr: + return true + } + return false +} + +type runVar struct { + v *types.Var + t string // the type, spelled in the file +} + +type run struct { + a, z int + n int + outer []runVar // outer locals named, by first use + fresh []runVar // locals declared in the run and used after it +} + +// mr6Runs returns the qualifying runs of fd (see MR6). +func mr6Runs(f *SrcFile, fd *ast.FuncDecl) []run { + info := f.Pkg.TypesInfo + pinned := pinnedVars(f, fd.Body) + var out []run + stmtLists(fd.Body, func(list []ast.Stmt) { + for i := range list { + for n := 2; n <= 4 && i+n <= len(list); n++ { + w := list[i : i+n] + if slices.ContainsFunc(w, func(s ast.Stmt) bool { return !simpleStmt(info, s) }) { + break + } + if r, ok := makeRun(f, fd, w, pinned); ok { + out = append(out, r) + } + } + } + }) + return out +} + +func simpleStmt(info *types.Info, s ast.Stmt) bool { + switch s := s.(type) { + case *ast.AssignStmt, *ast.ExprStmt, *ast.IncDecStmt, *ast.SendStmt: + case *ast.DeclStmt: + if g, ok := s.Decl.(*ast.GenDecl); !ok || g.Tok != token.VAR { + return false + } + default: + return false + } + return !contains(s, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.FuncLit: + return true + case *ast.Ident: + b, isB := info.Uses[n].(*types.Builtin) + return isB && b.Name() == "recover" + } + return false + }) +} + +// pinnedVars are the locals of body whose identity matters beyond their +// value: captured by a function literal, or address-taken (&x, an array +// sliced, a pointer-receiver method called on an addressable value). +func pinnedVars(f *SrcFile, body ast.Node) map[*types.Var]bool { + info := f.Pkg.TypesInfo + out := map[*types.Var]bool{} + base := func(e ast.Expr) *types.Var { + for { + switch x := e.(type) { + case *ast.ParenExpr: + e = x.X + case *ast.SelectorExpr: + if s := info.Selections[x]; s != nil && s.Indirect() { + return nil // through a pointer: not the variable itself + } + e = x.X + case *ast.IndexExpr: + if _, isArr := info.TypeOf(x.X).Underlying().(*types.Array); !isArr { + return nil + } + e = x.X + case *ast.Ident: + v, _ := localVar(f, info.Uses[x]) + return v + default: + return nil + } + } + } + ast.Inspect(body, func(n ast.Node) bool { + switch x := n.(type) { + case *ast.FuncLit: + ast.Inspect(x.Body, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + if v, ok := localVar(f, info.Uses[id]); ok && (v.Pos() < x.Pos() || v.Pos() >= x.End()) { + out[v] = true + } + } + return true + }) + case *ast.UnaryExpr: + if x.Op == token.AND { + if v := base(x.X); v != nil { + out[v] = true + } + } + case *ast.SliceExpr: + if t := info.TypeOf(x.X); t != nil { + if _, isArr := t.Underlying().(*types.Array); isArr { + if v := base(x.X); v != nil { + out[v] = true + } + } + } + case *ast.SelectorExpr: + if s := info.Selections[x]; s != nil && s.Kind() == types.MethodVal { + if sig, ok := s.Obj().Type().(*types.Signature); ok && sig.Recv() != nil { + if _, ptrRecv := sig.Recv().Type().(*types.Pointer); ptrRecv { + if v := base(x.X); v != nil { + out[v] = true + } + } + } + } + } + return true + }) + return out +} + +func makeRun(f *SrcFile, fd *ast.FuncDecl, w []ast.Stmt, pinned map[*types.Var]bool) (run, bool) { + info := f.Pkg.TypesInfo + r := run{a: f.off(w[0].Pos()), z: f.off(w[len(w)-1].End()), n: len(w)} + lo, hi := w[0].Pos(), w[len(w)-1].End() + seen := map[*types.Var]bool{} + var fresh []*types.Var + ok := true + for _, s := range w { + ast.Inspect(s, func(n ast.Node) bool { + id, isID := n.(*ast.Ident) + if !isID || !ok { + return ok + } + obj := info.Uses[id] + if obj == nil { + obj = info.Defs[id] + } + if obj == nil { + return true + } + // A function-local type, constant or function name would not be + // in scope in the helper. + if _, isVar := obj.(*types.Var); !isVar { + if _, isPkg := obj.(*types.PkgName); !isPkg && obj.Pkg() == f.Pkg.Types && + obj.Parent() != nil && obj.Parent() != f.Pkg.Types.Scope() { + ok = false + } + return true + } + v, local := localVar(f, obj) + if !local || seen[v] { + return true + } + seen[v] = true + if v.Pos() >= lo && v.Pos() < hi { + fresh = append(fresh, v) + return true + } + if pinned[v] { + ok = false + return false + } + t, spelled := spell(f, v.Type()) + if !spelled { + ok = false + return false + } + r.outer = append(r.outer, runVar{v, t}) + return true + }) + } + if !ok || len(r.outer) == 0 { + return run{}, false + } + names := map[string]bool{} + for _, v := range r.outer { + if names[v.v.Name()] { + return run{}, false + } + names[v.v.Name()] = true + } + for _, v := range fresh { + if names[v.Name()] { + return run{}, false + } + if !usedAfter(info, fd, v, hi) { + continue + } + t, spelled := spell(f, v.Type()) + if !spelled { + return run{}, false + } + r.fresh = append(r.fresh, runVar{v, t}) + } + return r, true +} + +func usedAfter(info *types.Info, fd *ast.FuncDecl, v *types.Var, after token.Pos) bool { + return contains(fd.Body, func(id *ast.Ident) bool { return id.Pos() >= after && info.Uses[id] == v }) +} + +// spell is t written in f, package-qualified by f's import names. It fails +// for a type f cannot name: an unexported or function-local named type, a +// type parameter, or a package f does not import. +func spell(f *SrcFile, t types.Type) (string, bool) { + names := map[string]string{} + for _, s := range f.AST.Imports { + if pn := importName(f.Pkg.TypesInfo, s); pn != nil && pn.Name() != "_" && pn.Name() != "." { + names[pn.Imported().Path()] = pn.Name() + } + } + ok := true + named := func(o *types.TypeName) { + if o.Pkg() == nil { + return + } + if o.Parent() != o.Pkg().Scope() || (o.Pkg() != f.Pkg.Types && (!o.Exported() || names[o.Pkg().Path()] == "")) { + ok = false + } + } + var check func(t types.Type, depth int) + check = func(t types.Type, depth int) { + if !ok || depth > 20 { + ok = false + return + } + switch t := t.(type) { + case *types.Named: + named(t.Obj()) + if args := t.TypeArgs(); args != nil { + for i := range args.Len() { + check(args.At(i), depth+1) + } + } + case *types.Alias: + named(t.Obj()) + case *types.TypeParam, *types.Tuple: + ok = false + case *types.Pointer: + check(t.Elem(), depth+1) + case *types.Slice: + check(t.Elem(), depth+1) + case *types.Array: + check(t.Elem(), depth+1) + case *types.Chan: + check(t.Elem(), depth+1) + case *types.Map: + check(t.Key(), depth+1) + check(t.Elem(), depth+1) + case *types.Signature: + for _, tup := range []*types.Tuple{t.Params(), t.Results()} { + for i := range tup.Len() { + check(tup.At(i).Type(), depth+1) + } + } + case *types.Struct: + for i := range t.NumFields() { + if !t.Field(i).Exported() && t.Field(i).Pkg() != f.Pkg.Types { + ok = false + } + check(t.Field(i).Type(), depth+1) + } + case *types.Interface: + for i := range t.NumExplicitMethods() { + check(t.ExplicitMethod(i).Type(), depth+1) + } + for i := range t.NumEmbeddeds() { + check(t.EmbeddedType(i), depth+1) + } + case *types.Basic: + if t.Info()&types.IsUntyped != 0 || t.Kind() == types.UnsafePointer { + ok = false + } + } + } + check(t, 0) + if !ok { + return "", false + } + return types.TypeString(t, func(p *types.Package) string { + if p == f.Pkg.Types { + return "" + } + return names[p.Path()] + }), true +} diff --git a/internal/metamorphic/transform_delta.go b/internal/metamorphic/transform_delta.go new file mode 100644 index 0000000..c888f2a --- /dev/null +++ b/internal/metamorphic/transform_delta.go @@ -0,0 +1,1461 @@ +package metamorphic + +// The expected-delta transforms (MR8–MR12), the precision transform (MR13), +// the call-routing transform (MR14) and extract-then-inline (MR15). +// +// The delta transforms work only where the analysis is exact (brief +// amendment 2026-10-27). Their sites are statements over local scalar +// variables (basic types, never captured or address-taken) built from +// identifiers, literals and operators that cannot panic. The callable must +// have an L3 result and no defer, goto or label. Loops, select / switch / +// fallthrough, panic edges and the defer chain are out of scope: a site's +// own CFG edges are never exception or loop_back edges. Each delta is +// computed from the original's own edges (the pre view), so the relation +// states what changes, not what the edges are. +// +// Each callable gets at most one site, so deltas never interact. Sites are +// chosen with the transform's seeded generator. + +import ( + "fmt" + "go/ast" + "go/token" + "go/types" + "maps" + "slices" + "strconv" + "strings" +) + +// ─── pre-view queries ───────────────────────────────────────────────────────── + +type edgeRec struct{ kind, src, dst, label, prov string } + +func (e edgeRec) dedge() DEdge { + return DEdge{Kind: e.kind, Src: Pre(e.src), Dst: Pre(e.dst), Label: e.label, Prov: e.prov} +} + +func edgesOf(cv *CV, kind string) []edgeRec { + var out []edgeRec + for _, e := range slices.Sorted(maps.Keys(cv.Edges)) { + f := strings.Split(e, "|") + if f[0] != kind { + continue + } + r := edgeRec{kind: f[0], src: f[1], dst: f[2], label: f[3]} + if len(f) > 4 { + r.prov = f[4] + } + out = append(out, r) + } + return out +} + +func filter(es []edgeRec, pred func(edgeRec) bool) []edgeRec { + var out []edgeRec + for _, e := range es { + if pred(e) { + out = append(out, e) + } + } + return out +} + +func into(cv *CV, kind, dst string) []edgeRec { + return filter(edgesOf(cv, kind), func(e edgeRec) bool { return e.dst == dst }) +} + +func outOf(cv *CV, kind, src string) []edgeRec { + return filter(edgesOf(cv, kind), func(e edgeRec) bool { return e.src == src }) +} + +// cfgReach is the set of CFG nodes reachable from src in cv. +func cfgReach(cv *CV, src string) map[string]bool { + succ := map[string][]string{} + for _, e := range edgesOf(cv, "cfg") { + succ[e.src] = append(succ[e.src], e.dst) + } + seen := map[string]bool{src: true} + stack := []string{src} + for len(stack) > 0 { + n := stack[len(stack)-1] + stack = stack[:len(stack)-1] + for _, d := range succ[n] { + if !seen[d] { + seen[d] = true + stack = append(stack, d) + } + } + } + return seen +} + +// onCycle reports whether CFG node k reaches itself. +func onCycle(cv *CV, k string) bool { + for _, e := range outOf(cv, "cfg", k) { + if cfgReach(cv, e.dst)[k] { + return true + } + } + return false +} + +// cdgMismatch prefixes the deltaScope reason for a callable whose CDG is +// not the reference CDG of its CFG. +const cdgMismatch = "cdg differs from the reference: " + +// ─── eligibility ────────────────────────────────────────────────────────────── + +// scope is the per-callable state of a delta transform. +type scope struct { + f *SrcFile + fd *ast.FuncDecl + id string + cv *CV + pre *View + pinned map[*types.Var]bool +} + +// deltaScope returns fd's scope, or why it is not eligible. +func deltaScope(c *tctx, f *SrcFile, fd *ast.FuncDecl) (*scope, string) { + if fd.Body == nil { + return nil, "no body" + } + if fd.Type.TypeParams != nil || recvGeneric(fd) { + return nil, "generic" + } + if contains(fd.Body, func(n ast.Node) bool { + switch n.(type) { + case *ast.DeferStmt, *ast.LabeledStmt: + return true + } + return false + }) || hasGoto(fd.Body) { + return nil, "defer, goto or label" + } + id := funcIdentity(f, fd) + cv := c.pre.Calls[id] + if cv == nil || !cv.L3 { + return nil, "no L3 result" + } + if d := cdgDiff(cv, RefCDG(cv)); d != "" { + // Its CDG is not the reference CDG of its CFG: an analyzer CDG bug, + // reported as a violation (deltaTransform), never as a skip. + return nil, cdgMismatch + d + } + return &scope{f: f, fd: fd, id: id, cv: cv, pre: c.pre, pinned: pinnedVars(f, fd.Body)}, "" +} + +// scalar is obj as a local scalar of s: a function-local variable of basic +// type, neither captured nor address-taken. +func (s *scope) scalar(obj types.Object) (*types.Var, bool) { + v, ok := localVar(s.f, obj) + if !ok || s.pinned[v] { + return nil, false + } + b, ok := v.Type().Underlying().(*types.Basic) + if !ok || b.Info()&(types.IsNumeric|types.IsString|types.IsBoolean) == 0 { + return nil, false + } + return v, true +} + +func (s *scope) obj(id *ast.Ident) types.Object { + info := s.f.Pkg.TypesInfo + if o := info.Defs[id]; o != nil { + return o + } + return info.Uses[id] +} + +// safeExpr reports whether e cannot panic and names only local scalars and +// constants: identifiers, literals, parentheses and operators other than +// division, remainder and shifts. +func (s *scope) safeExpr(e ast.Expr) bool { + switch e := e.(type) { + case *ast.BasicLit: + return true + case *ast.Ident: + switch o := s.f.Pkg.TypesInfo.Uses[e].(type) { + case *types.Const, *types.Nil: + return true + case *types.Var: + _, ok := s.scalar(o) + return ok + } + return false + case *ast.ParenExpr: + return s.safeExpr(e.X) + case *ast.UnaryExpr: + return (e.Op == token.SUB || e.Op == token.ADD || e.Op == token.NOT || e.Op == token.XOR) && s.safeExpr(e.X) + case *ast.BinaryExpr: + switch e.Op { + case token.QUO, token.REM, token.SHL, token.SHR: + return false + } + return s.safeExpr(e.X) && s.safeExpr(e.Y) + } + return false +} + +// facts are a safe statement's defined and used variables, by object. +type facts struct { + defs, uses []*types.Var + names map[string]bool // every identifier it names + define bool // a := statement +} + +func addVar(list []*types.Var, v *types.Var) []*types.Var { + if slices.Contains(list, v) { + return list + } + return append(list, v) +} + +// safeStmt returns st's facts when st is a safe simple statement: an +// assignment (=, :=, or an operator assignment that cannot panic) of safe +// expressions to local scalars, or ++/-- of one. +func (s *scope) safeStmt(st ast.Stmt) (*facts, bool) { + fs := &facts{names: map[string]bool{}} + ast.Inspect(st, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + fs.names[id.Name] = true + } + return true + }) + usesOf := func(e ast.Expr) { + ast.Inspect(e, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + if v, ok := s.scalar(s.f.Pkg.TypesInfo.Uses[id]); ok { + fs.uses = addVar(fs.uses, v) + } + } + return true + }) + } + lhs := func(e ast.Expr, use bool) bool { + id, ok := e.(*ast.Ident) + if !ok || id.Name == "_" { + return false + } + v, ok := s.scalar(s.obj(id)) + if !ok { + return false + } + fs.defs = addVar(fs.defs, v) + if use { + fs.uses = addVar(fs.uses, v) + } + return true + } + switch st := st.(type) { + case *ast.AssignStmt: + switch st.Tok { + case token.ASSIGN, token.DEFINE: + case token.ADD_ASSIGN, token.SUB_ASSIGN, token.MUL_ASSIGN, token.AND_ASSIGN, token.OR_ASSIGN, + token.XOR_ASSIGN, token.AND_NOT_ASSIGN: + default: + return nil, false + } + if len(st.Lhs) != len(st.Rhs) { + return nil, false + } + op := st.Tok != token.ASSIGN && st.Tok != token.DEFINE + for _, r := range st.Rhs { + if !s.safeExpr(r) { + return nil, false + } + usesOf(r) + } + for _, l := range st.Lhs { + if !lhs(l, op) { + return nil, false + } + } + fs.define = st.Tok == token.DEFINE + case *ast.IncDecStmt: + if !lhs(st.X, true) { + return nil, false + } + default: + return nil, false + } + return fs, true +} + +// safeReturn reports whether st is a return of safe expressions. +func (s *scope) safeReturn(st ast.Stmt) bool { + r, ok := st.(*ast.ReturnStmt) + if !ok { + return false + } + for _, e := range r.Results { + if !s.safeExpr(e) { + return false + } + } + return true +} + +// resultNamed reports whether name is a named result of s's function. +func (s *scope) resultNamed(name string) bool { + if s.fd.Type.Results == nil { + return false + } + for _, f := range s.fd.Type.Results.List { + for _, n := range f.Names { + if n.Name == name { + return true + } + } + } + return false +} + +// key is the body key of statement st, which must be a CFG node of s.cv. +func (s *scope) key(st ast.Stmt) (string, bool) { + k := s.f.key(st.Pos()) + b, ok := s.cv.Body[k] + if !ok || strings.Split(b, "|")[1] != "" { + return "", false + } + return k, true +} + +// only reports whether k's only CFG out-edge is a fallthrough to a node +// other than @exit, and returns it. +func (s *scope) only(k string) (edgeRec, bool) { + out := outOf(s.cv, "cfg", k) + if len(out) != 1 || out[0].label != "fallthrough" || out[0].dst == "@exit" { + return edgeRec{}, false + } + return out[0], true +} + +func zeroLit(t types.Type) string { + b := t.Underlying().(*types.Basic) + switch { + case b.Info()&types.IsString != 0: + return `""` + case b.Info()&types.IsBoolean != 0: + return "false" + } + return "0" +} + +// deltaTransform runs pick on every eligible function of the file, at most +// one site each. +func deltaTransform(name string, pick func(c *tctx, s *scope, fe *fileEdit) []edit) Transform { + return Transform{Name: name, edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + if c.pre == nil { + return nil, fmt.Errorf("%s needs the original's analysis", name) + } + if usesPrefix(f, strings.ToLower(name)) { + return nil, errSkip(name + ": name clash") + } + fe := &fileEdit{} + var edits []edit + for _, fd := range funcDecls(f) { + s, why := deltaScope(c, f, fd) + if strings.HasPrefix(why, cdgMismatch) { + fe.violations = append(fe.violations, fmt.Sprintf("%s: %s %s", f.Rel, funcIdentity(f, fd), why)) + continue + } + if s == nil { + fe.skip(name + ": callable: " + why) + continue + } + es := pick(c, s, fe) + if len(es) == 0 { + fe.skip(name + ": callable: no eligible site") + continue + } + edits = append(edits, es...) + fe.touched = append(fe.touched, s.id) + } + if len(edits) == 0 { + return fe, errSkip(name + ": no eligible site") + } + ps, err := applyEdits(f, edits) + fe.pieces = ps + return fe, err + }} +} + +// choose returns one of n candidates with c's generator. +func (c *tctx) choose(n int) int { return c.rng.IntN(n) } + +func desc(s *scope, st ast.Node, what string) string { + return fmt.Sprintf("%s:%d: %s: %s", s.f.Rel, s.f.pos(st.Pos()).Line, s.fd.Name.Name, what) +} + +// statements is every statement list of s's body outside function literals, +// with each statement. +func (s *scope) lists() [][]ast.Stmt { + var out [][]ast.Stmt + stmtLists(s.fd.Body, func(l []ast.Stmt) { out = append(out, l) }) + return out +} + +// ─── delta builder ──────────────────────────────────────────────────────────── + +// dbuild accumulates a Delta over a callable's pre view. Its DDG helpers +// keep the L4 port mirror of an @entry edge in step: an L3 edge @entry → n +// on a base that has a formal_in is mirrored as @formal_in: → n on the +// same var, prov reaching-defs (spec §4.7 port wiring). +type dbuild struct { + d *Delta + cv *CV + called bool +} + +func newDelta(s *scope, site string, nodes map[string]string) *dbuild { + return &dbuild{d: &Delta{Site: site, Callable: s.id, Nodes: nodes}, cv: s.cv, called: s.pre.called(s.id)} +} + +// formalIn is the @formal_in key whose `of` is name ("" when none). +func (b *dbuild) formalIn(name string) string { + for k, attr := range b.cv.Body { + if strings.HasPrefix(k, "@formal_in:") && strings.Split(attr, "|")[2] == name { + return k + } + } + return "" +} + +func (b *dbuild) rm(e edgeRec) { + b.d.Remove = append(b.d.Remove, e.dedge()) + if e.kind == "ddg" && e.src == "@entry" { + if fi := b.formalIn(e.label); fi != "" && b.cv.Edges["ddg|"+fi+"|"+e.dst+"|"+e.label+"|reaching-defs"] { + b.d.Remove = append(b.d.Remove, Port(Pre(fi), Pre(e.dst), e.label)) + } + } +} + +func (b *dbuild) add(e DEdge) { + for _, a := range b.d.Add { + if a == e { + return + } + } + b.d.Add = append(b.d.Add, e) + if e.Kind == "ddg" && e.Src.Pre == "@entry" { + if fi := b.formalIn(e.Label); fi != "" { + b.add(Port(Pre(fi), e.Dst, e.Label)) + } + } +} + +// ssaOn are the ssa ddg edges of es on var v; ok is false when es holds a +// non-ssa edge on v other than an @formal_in port mirror (a site the delta +// rules do not describe). +func ssaOn(es []edgeRec, v string) ([]edgeRec, bool) { + var out []edgeRec + for _, e := range es { + if e.label != v { + continue + } + switch { + case e.prov == "ssa": + out = append(out, e) + case e.prov == "reaching-defs" && strings.HasPrefix(e.src, "@formal_in:"): + default: + return nil, false + } + } + return out, true +} + +// sym is e with its refs spelled symbolically: a pre key as itself, a mark +// as "mark:". +func (e DEdge) sym() string { + ref := func(r Ref) string { + if r.Mark != "" { + return "mark:" + r.Mark + } + return r.Pre + } + out := e.Kind + "|" + ref(e.Src) + "|" + ref(e.Dst) + "|" + e.Label + if e.Kind == "ddg" { + out += "|" + cmpOr(e.Prov, "ssa") + } + return out +} + +// localFlows is the set of (formal_in, formal_out) pairs of a callable's +// edges, the flows its summary is computed from: reachability over its ddg, +// cdg and summary edges (sdg graphOf). +func localFlows(edges map[string]bool) map[[2]string]bool { + succ := map[string][]string{} + var srcs []string + for e := range edges { + f := strings.Split(e, "|") + if f[0] == "cfg" { + continue + } + succ[f[1]] = append(succ[f[1]], f[2]) + if strings.HasPrefix(f[1], "@formal_in:") { + srcs = append(srcs, f[1]) + } + } + out := map[[2]string]bool{} + for _, s := range srcs { + for n := range (Graph(succ)).Reach(s) { + if strings.HasPrefix(n, "@formal_out") { + out[[2]string{s, n}] = true + } + } + } + return out +} + +// preserves reports whether the delta leaves the callable's flows as they +// are. A delta that changes them would change the summary edges of every +// caller, outside the callable, which the delta oracle does not predict. +// +// The predicted edges name new nodes by mark ("mark:"); its CDG is +// derived from its CFG when the delta says so. +func (b *dbuild) preserves() bool { + post := maps.Clone(b.cv.Edges) + for _, e := range b.d.Remove { + delete(post, e.sym()) + } + for _, e := range b.d.Add { + post[e.sym()] = true + } + if b.d.CDGFromCFG { + for e := range post { + if strings.HasPrefix(e, "cdg|") { + delete(post, e) + } + } + for e := range RefCDG(&CV{Edges: post}) { + post[e] = true + } + } + return maps.Equal(localFlows(b.cv.Edges), localFlows(post)) +} + +// done records the delta as a site of fe, or the reason it is not one. +// +// A callable no in-project call site targets (no param_in edge into it) may +// change its flows: no caller's summary edges depend on them. +func (b *dbuild) done(fe *fileEdit, name string) bool { + if b.called && !b.preserves() { + fe.skip(name + ": site: changes the callable's flows") + return false + } + fe.sites = append(fe.sites, Site{Desc: b.d.Site, Callable: b.d.Callable, Delta: b.d}) + return true +} + +// ─── MR8: wrap a statement in `if c { S }` ──────────────────────────────────── + +// MR8 wraps a statement S in `if c := true; c { S }`. S is a safe +// assignment that reads every variable it writes (x++, x += e, x = x + e), +// so every def reaching S is visible as one of its ddg in-edges, and it +// falls through to its successor Y. Expected delta: +// - CFG: each X→S becomes X→B (same kind); B→S true; B→Y false. +// - CDG: derived from the expected CFG (the reference CDG). +// - DDG: S is now conditional, so each def D reaching S on x also reaches +// each use U that S's def of x reached: D→U on x, unless already there +// (with its @formal_in port mirror when D is @entry). The guard c is +// defined by the header's init and used by its condition, both folded +// into the header node B (spec §4.2). Within a node uses precede defs, +// so c adds no edge in straight-line code; when S lies on a CFG cycle +// (inside a loop body) the init's def reaches the condition around the +// loop, and c adds the self-loop B→B (a sound over-approximation of the +// header folding). +var MR8 = deltaTransform("MR8", func(c *tctx, s *scope, fe *fileEdit) []edit { + type cand struct { + st ast.Stmt + k string + y edgeRec + fs *facts + } + var cands []cand + for _, list := range s.lists() { + for _, st := range list { + fs, ok := s.safeStmt(st) + if !ok || fs.define || slices.ContainsFunc(fs.defs, func(v *types.Var) bool { return !slices.Contains(fs.uses, v) }) { + continue + } + k, ok := s.key(st) + if !ok { + continue + } + y, ok := s.only(k) + if !ok { + continue + } + cands = append(cands, cand{st, k, y, fs}) + } + } + for len(cands) > 0 { + i := c.choose(len(cands)) + x := cands[i] + cands = slices.Delete(cands, i, i+1) + b, g := c.fresh("mr8b"), c.fresh("mr8c") + db := newDelta(s, desc(s, x.st, "wrap in if"), map[string]string{b: "branch"}) + db.d.CDGFromCFG = true + for _, e := range into(s.cv, "cfg", x.k) { + db.rm(e) + db.add(CFG(Pre(e.src), Mark(b), e.label)) + } + db.add(CFG(Mark(b), Pre(x.k), "true")) + db.add(CFG(Mark(b), Pre(x.y.dst), "false")) + if onCycle(s.cv, x.k) { + db.add(DDG(Mark(b), Mark(b), g)) + } + ok := true + for _, v := range x.fs.defs { + ins, ok1 := ssaOn(into(s.cv, "ddg", x.k), v.Name()) + outs, ok2 := ssaOn(outOf(s.cv, "ddg", x.k), v.Name()) + if !ok1 || !ok2 { + ok = false + break + } + for _, in := range ins { + for _, o := range outs { + if !s.cv.Edges["ddg|"+in.src+"|"+o.dst+"|"+v.Name()+"|ssa"] { + db.add(DDG(Pre(in.src), Pre(o.dst), v.Name())) + } + } + } + } + if !ok { + fe.skip("MR8: site: non-ssa ddg on the variable") + continue + } + if !db.done(fe, "MR8") { + continue + } + a, z := s.f.off(x.st.Pos()), s.f.off(x.st.End()) + return []edit{ + insert(a, mark(b), lit("if "+g+" := true; "+g+" { ")), + insert(z, lit(" }")), + } + } + return nil +}) + +// ─── MR9: insert a kill ─────────────────────────────────────────────────────── + +// MR9 inserts the kill `x = k` (k the zero literal of x's type) between a +// safe statement D that defines local scalar x and the next statement U, +// which D falls through to and which reads x (D→U on x). Every path from D +// passes the kill, so expected delta: +// - DDG: each D→W on x becomes K→W on x. +// - CFG: D→U becomes D→K, K→U (fallthrough). +// - CDG: derived (K gets D's controllers). +var MR9 = deltaTransform("MR9", func(c *tctx, s *scope, fe *fileEdit) []edit { + type cand struct { + d, u ast.Stmt + dk, uk string + v *types.Var + } + var cands []cand + for _, list := range s.lists() { + for i := 0; i+1 < len(list); i++ { + fs, ok := s.safeStmt(list[i]) + if !ok { + continue + } + dk, ok1 := s.key(list[i]) + uk, ok2 := s.key(list[i+1]) + if !ok1 || !ok2 { + continue + } + if y, ok := s.only(dk); !ok || y.dst != uk { + continue + } + for _, v := range fs.defs { + if s.cv.Edges["ddg|"+dk+"|"+uk+"|"+v.Name()+"|ssa"] { + cands = append(cands, cand{list[i], list[i+1], dk, uk, v}) + } + } + } + } + for len(cands) > 0 { + i := c.choose(len(cands)) + x := cands[i] + cands = slices.Delete(cands, i, i+1) + k := c.fresh("mr9k") + name := x.v.Name() + db := newDelta(s, desc(s, x.u, "kill "+name+" before"), map[string]string{k: "statement"}) + db.d.CDGFromCFG = true + // Every def edge of D on x moves to the kill: the ssa edges to its + // uses, and the reaching-defs port edge to the formal_out of a + // named result x (the @exit rule, spec §4.7), whose var is the + // formal_out's `of`. + outs := filter(outOf(s.cv, "ddg", x.dk), func(e edgeRec) bool { + return e.label == name || e.prov == "reaching-defs" && strings.HasPrefix(e.dst, "@formal_out") && s.resultNamed(name) + }) + if slices.ContainsFunc(outs, func(e edgeRec) bool { return e.prov != "ssa" && !strings.HasPrefix(e.dst, "@formal_out") }) { + fe.skip("MR9: site: non-ssa ddg on the variable") + continue + } + for _, e := range outs { + db.rm(e) + db.add(DEdge{Kind: "ddg", Src: Mark(k), Dst: Pre(e.dst), Label: e.label, Prov: e.prov}) + } + db.rm(edgeRec{kind: "cfg", src: x.dk, dst: x.uk, label: "fallthrough"}) + db.add(CFG(Pre(x.dk), Mark(k), "fallthrough")) + db.add(CFG(Mark(k), Pre(x.uk), "fallthrough")) + if !db.done(fe, "MR9") { + continue + } + return []edit{insert(s.f.off(x.u.Pos()), mark(k), lit(name+" = "+zeroLit(x.v.Type())+"; "))} + } + return nil +}) + +// ─── MR10: insert a copy ────────────────────────────────────────────────────── + +// MR10 inserts `y := x` before a statement U and rewrites U's one read of +// local scalar x to y. U is a safe statement or a safe return that names x +// only there. Expected delta: +// - DDG: each D→U on x becomes D→C on x (with its port mirror when D is +// @entry); C→U on y. +// - CFG: each X→U becomes X→C (same kind); C→U fallthrough. +// - CDG: derived (C gets U's controllers). +var MR10 = deltaTransform("MR10", func(c *tctx, s *scope, fe *fileEdit) []edit { + if usesPrefix(s.f, "mr10y") { + return nil + } + type cand struct { + u ast.Stmt + uk string + id *ast.Ident + v *types.Var + } + var cands []cand + info := s.f.Pkg.TypesInfo + for _, list := range s.lists() { + for _, st := range list { + fs, ok := s.safeStmt(st) + if !ok && !s.safeReturn(st) { + continue + } + uk, ok := s.key(st) + if !ok { + continue + } + byVar := map[*types.Var][]*ast.Ident{} + ast.Inspect(st, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + if v, ok := s.scalar(s.obj(id)); ok { + byVar[v] = append(byVar[v], id) + } + } + return true + }) + for _, v := range slices.SortedFunc(maps.Keys(byVar), func(a, b *types.Var) int { return int(a.Pos() - b.Pos()) }) { + ids := byVar[v] + if len(ids) != 1 || info.Uses[ids[0]] != v { + continue // named twice, or only written + } + if fs != nil && slices.Contains(fs.defs, v) { + continue // an operator assignment reads and writes x + } + cands = append(cands, cand{st, uk, ids[0], v}) + } + } + } + for len(cands) > 0 { + i := c.choose(len(cands)) + x := cands[i] + cands = slices.Delete(cands, i, i+1) + name := x.v.Name() + ins, ok := ssaOn(into(s.cv, "ddg", x.uk), name) + if !ok || len(ins) == 0 { + fe.skip("MR10: site: no ssa def of the variable reaches it") + continue + } + cm, y := c.fresh("mr10c"), c.fresh("mr10y") + db := newDelta(s, desc(s, x.u, "copy "+name+" to "+y), map[string]string{cm: "statement"}) + db.d.CDGFromCFG = true + for _, e := range ins { + db.rm(e) + db.add(DDG(Pre(e.src), Mark(cm), name)) + } + db.add(DDG(Mark(cm), Pre(x.uk), y)) + for _, e := range into(s.cv, "cfg", x.uk) { + db.rm(e) + db.add(CFG(Pre(e.src), Mark(cm), e.label)) + } + db.add(CFG(Mark(cm), Pre(x.uk), "fallthrough")) + if !db.done(fe, "MR10") { + continue + } + return []edit{ + insert(s.f.off(x.u.Pos()), mark(cm), lit(y+" := "+name+"; ")), + {a: s.f.off(x.id.Pos()), z: s.f.off(x.id.End()), anchor: true, out: []piece{lit(y)}}, + } + } + return nil +}) + +// ─── MR11: swap two independent statements ──────────────────────────────────── + +// MR11 swaps adjacent safe statements S1; S2 that share no variable by name +// (neither writes a name the other mentions, so a := cannot shadow). Their +// text moves, and their keys map to the new positions. Expected delta: +// - DDG: none (after mapping). +// - CFG: each X→S1 becomes X→S2; S1→S2 becomes S2→S1; S2's out-edges +// leave S1 instead (same kinds). +// - CDG: derived. It equals the original's, since S1 and S2 sit in one +// basic block. +var MR11 = deltaTransform("MR11", func(c *tctx, s *scope, fe *fileEdit) []edit { + type cand struct { + s1, s2 ast.Stmt + k1, k2 string + } + var cands []cand + for _, list := range s.lists() { + for i := 0; i+1 < len(list); i++ { + f1, ok1 := s.safeStmt(list[i]) + f2, ok2 := s.safeStmt(list[i+1]) + if !ok1 || !ok2 { + continue + } + if slices.ContainsFunc(f1.defs, func(v *types.Var) bool { return f2.names[v.Name()] }) || + slices.ContainsFunc(f2.defs, func(v *types.Var) bool { return f1.names[v.Name()] }) { + continue + } + k1, ok1 := s.key(list[i]) + k2, ok2 := s.key(list[i+1]) + if !ok1 || !ok2 { + continue + } + if y, ok := s.only(k1); !ok || y.dst != k2 { + continue + } + if _, ok := s.only(k2); !ok { + continue + } + cands = append(cands, cand{list[i], list[i+1], k1, k2}) + } + } + if len(cands) == 0 { + return nil + } + x := cands[c.choose(len(cands))] + db := newDelta(s, desc(s, x.s1, "swap with the next statement"), nil) + db.d.CDGFromCFG = true + for _, e := range into(s.cv, "cfg", x.k1) { + db.rm(e) + db.add(CFG(Pre(e.src), Pre(x.k2), e.label)) + } + db.rm(edgeRec{kind: "cfg", src: x.k1, dst: x.k2, label: "fallthrough"}) + db.add(CFG(Pre(x.k2), Pre(x.k1), "fallthrough")) + for _, e := range outOf(s.cv, "cfg", x.k2) { + db.rm(e) + db.add(CFG(Pre(x.k1), Pre(e.dst), e.label)) + } + if !db.done(fe, "MR11") { + return nil + } + f := s.f + a1, z1 := f.off(x.s1.Pos()), f.off(x.s1.End()) + a2, z2 := f.off(x.s2.Pos()), f.off(x.s2.End()) + return []edit{{a: a1, z: z2, out: []piece{cp(f, a2, z2), cp(f, z1, a2), cp(f, a1, z1)}}} +}) + +// ─── MR12: an early return ──────────────────────────────────────────────────── + +// MR12 inserts `if c := true; c { return }` before a safe +// statement S of the function's top-level statement list. The function's +// results are unnamed, so the return names no variable. Expected delta: +// - CFG: each X→S becomes X→B (same kind); B→R true; R→@exit return; +// B→S false. +// - CDG: derived. Every node after S that @entry controlled is now +// controlled by B. +// - DDG: R → each @formal_out of a result (reaching-defs), the return's +// port wiring. S is not in a loop, so the guard c adds no edge (see +// MR8). +var MR12 = deltaTransform("MR12", func(c *tctx, s *scope, fe *fileEdit) []edit { + for k, attr := range s.cv.Body { + if strings.HasPrefix(k, "@formal_out") && !strings.HasPrefix(strings.Split(attr, "|")[2], "") { + return nil // a by-ref, capture or global out: an early exit adds def → exit paths + } + } + res := s.fd.Type.Results + var zeros []string + if res != nil { + for _, fl := range res.List { + if len(fl.Names) > 0 { + return nil + } + z, ok := zeroValue(s.f, s.f.Pkg.TypesInfo.TypeOf(fl.Type)) + if !ok { + return nil + } + zeros = append(zeros, z) + } + } + type cand struct { + st ast.Stmt + k string + } + var cands []cand + for _, st := range s.fd.Body.List { + if _, ok := s.safeStmt(st); !ok && !s.safeReturn(st) { + continue + } + if k, ok := s.key(st); ok { + cands = append(cands, cand{st, k}) + } + } + for len(cands) > 0 { + i := c.choose(len(cands)) + x := cands[i] + cands = slices.Delete(cands, i, i+1) + b, r, g := c.fresh("mr12b"), c.fresh("mr12r"), c.fresh("mr12c") + db := newDelta(s, desc(s, x.st, "early return before"), map[string]string{b: "branch", r: "return"}) + db.d.CDGFromCFG = true + for _, e := range into(s.cv, "cfg", x.k) { + db.rm(e) + db.add(CFG(Pre(e.src), Mark(b), e.label)) + } + db.add(CFG(Mark(b), Mark(r), "true")) + db.add(CFG(Mark(r), Pre("@exit"), "return")) + db.add(CFG(Mark(b), Pre(x.k), "false")) + for _, k := range slices.Sorted(maps.Keys(s.cv.Body)) { + if of := strings.Split(s.cv.Body[k], "|")[2]; strings.HasPrefix(k, "@formal_out") && strings.HasPrefix(of, "") { + db.add(Port(Mark(r), Pre(k), of)) + } + } + if !db.done(fe, "MR12") { + continue + } + ret := "return" + if len(zeros) > 0 { + ret += " " + strings.Join(zeros, ", ") + } + return []edit{insert(s.f.off(x.st.Pos()), mark(b), lit("if "+g+" := true; "+g+" { "), mark(r), lit(ret+" }; "))} + } + return nil +}) + +// zeroValue spells t's zero value in f, using no variable. +func zeroValue(f *SrcFile, t types.Type) (string, bool) { + if t == nil { + return "", false + } + if _, isTP := t.(*types.TypeParam); isTP { + return "", false + } + switch u := t.Underlying().(type) { + case *types.Basic: + switch { + case u.Info()&types.IsString != 0: + return `""`, true + case u.Info()&types.IsBoolean != 0: + return "false", true + case u.Info()&types.IsNumeric != 0: + return "0", true + case u.Kind() == types.UnsafePointer: + return "nil", true + } + return "", false + case *types.Pointer, *types.Slice, *types.Map, *types.Chan, *types.Signature, *types.Interface: + return "nil", true + case *types.Struct, *types.Array: + s, ok := spell(f, t) + if !ok { + return "", false + } + return s + "{}", true + } + return "", false +} + +// ─── MR13: a constant sink argument ─────────────────────────────────────────── + +// MR13 replaces the argument x of a call statement `g(…, x, …)` with the +// zero literal of x's type. x is a local scalar that the callable also names +// elsewhere, and every other argument is a constant. g is a plain function +// (no receiver, not variadic) whose formal_outs are only its results, and +// that touches no global, so the argument's value can only come from x. +// Before the change, some non-global @formal_in of the caller reaches the +// argument's actual_in over data edges. +// +// Expected (precision): afterwards no non-global @formal_in of the caller +// reaches that actual_in over data edges (ddg ∪ summary ∪ param_in ∪ +// param_out). So "no flow" can happen. +// Nothing may be added anywhere, and nothing removed but the caller's ddg +// edges into the call on x and summary edges anywhere (they follow from the +// flows that went). +var MR13 = Transform{Name: "MR13", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + if c.pre == nil { + return nil, fmt.Errorf("MR13 needs the original's analysis") + } + info := f.Pkg.TypesInfo + fe := &fileEdit{} + var edits []edit + g := c.pre.Graph(true) + for _, fd := range funcDecls(f) { + if fd.Body == nil { + continue + } + id := funcIdentity(f, fd) + cv := c.pre.Calls[id] + if cv == nil || !cv.L3 { + fe.skip("MR13: callable: no L3 result") + continue + } + s := &scope{f: f, fd: fd, id: id, cv: cv, pre: c.pre, pinned: pinnedVars(f, fd.Body)} + uses := map[*types.Var]int{} + ast.Inspect(fd.Body, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + if v, ok := info.Uses[id].(*types.Var); ok { + uses[v]++ + } + } + return true + }) + var srcs []string + for k, b := range cv.Body { + if strings.HasPrefix(k, "@formal_in:") && !strings.HasPrefix(strings.Split(b, "|")[2], ":") { + srcs = append(srcs, Node(id, k)) + } + } + slices.Sort(srcs) + var cands []mr13Cand + for _, list := range s.lists() { + for _, st := range list { + x, ok := mr13Site(s, st, uses) + if !ok { + continue + } + if !g.ReachesFrom(srcs, Node(id, x.vk)) { + fe.skip("MR13: site: no flow to the argument before") + continue + } + cands = append(cands, x) + } + } + if len(cands) == 0 { + fe.skip("MR13: callable: no eligible site") + continue + } + x := cands[c.choose(len(cands))] + d := &Delta{Site: desc(s, x.arg, "constant argument for "+x.v.Name()), Callable: id, AtMost: true} + for _, e := range filter(into(cv, "ddg", x.ck), func(e edgeRec) bool { return e.label == x.v.Name() }) { + d.Remove = append(d.Remove, e.dedge()) + } + fe.sites = append(fe.sites, Site{Desc: d.Site, Callable: id, Delta: d, NoFlow: []Ref{Pre(x.vk)}}) + fe.touched = append(fe.touched, id) + edits = append(edits, edit{a: f.off(x.arg.Pos()), z: f.off(x.arg.End()), anchor: true, out: []piece{lit(zeroLit(x.v.Type()))}}) + } + if len(edits) == 0 { + return fe, errSkip("MR13: no eligible site") + } + ps, err := applyEdits(f, edits) + fe.pieces = ps + return fe, err +}} + +type mr13Cand struct { + arg *ast.Ident + v *types.Var + ck, vk string +} + +// mr13Site is st as an MR13 site (see MR13). +func mr13Site(s *scope, st ast.Stmt, uses map[*types.Var]int) (mr13Cand, bool) { + info := s.f.Pkg.TypesInfo + es, ok := st.(*ast.ExprStmt) + if !ok { + return mr13Cand{}, false + } + call, ok := es.X.(*ast.CallExpr) + if !ok || call.Ellipsis.IsValid() { + return mr13Cand{}, false + } + fn, ok := ast.Unparen(call.Fun).(*ast.Ident) + if !ok { + return mr13Cand{}, false + } + callee, ok := info.Uses[fn].(*types.Func) + if !ok { + return mr13Cand{}, false + } + sig := callee.Type().(*types.Signature) + if sig.Variadic() || sig.Recv() != nil || sig.TypeParams() != nil || sig.Params().Len() != len(call.Args) { + return mr13Cand{}, false + } + ck := s.f.key(call.Pos()) + cb, ok := s.cv.Body[ck] + if !ok { + return mr13Cand{}, false + } + ccv := s.pre.Calls[strings.Split(cb, "|")[4]] + if ccv == nil { + return mr13Cand{}, false + } + for _, b := range ccv.Body { + f := strings.Split(b, "|") + if strings.HasPrefix(f[2], ":") || f[0] == "formal_out" && !strings.HasPrefix(f[2], "") { + return mr13Cand{}, false + } + } + var x mr13Cand + for i, a := range call.Args { + if tv, ok := info.Types[a]; ok && tv.Value != nil { + continue + } + id, ok := a.(*ast.Ident) + if !ok || x.arg != nil { + return mr13Cand{}, false + } + v, ok := s.scalar(info.Uses[id]) + // A local must keep a use, or it no longer compiles; a parameter + // need not. + if !ok || uses[v] < 2 && !isParam(s.fd, v) { + return mr13Cand{}, false + } + pname := sig.Params().At(i).Name() + if pname == "" || pname == "_" { + pname = "_" + strconv.Itoa(i) + } + x = mr13Cand{arg: id, v: v, ck: ck} + for _, k := range slices.Sorted(maps.Keys(s.cv.Body)) { + if strings.HasPrefix(k, ck+"/actual_in:") && strings.Split(s.cv.Body[k], "|")[2] == pname { + x.vk = k + } + } + } + return x, x.arg != nil && x.vk != "" +} + +// isParam reports whether v is a parameter, receiver or result of fd. +func isParam(fd *ast.FuncDecl, v *types.Var) bool { + for _, fl := range []*ast.FieldList{fd.Recv, fd.Type.Params, fd.Type.Results} { + if fl == nil { + continue + } + for _, f := range fl.List { + for _, n := range f.Names { + if n.Pos() == v.Pos() { + return true + } + } + } + } + return false +} + +// ─── MR14: route a call ─────────────────────────────────────────────────────── + +// MR14 routes one direct call statement `f(args)` per function, cycling +// through three forms: +// +// (a) through a single-implementation interface: +// mr14iN(mr14tN{}).f(args), where mr14tN.f calls f with the same +// parameter names; +// (b) through an in-place closure: func() { f(args) }(), in a function +// with no other literal (a new one would renumber closure_N); +// (c) deferred: defer f(args), only when the call is the function's last +// top-level statement, so it runs where it did (at exit, after every +// other statement). +// +// Oracle: one-way reachability. For (a), every summary edge of the +// original call site must have its counterpart (same `of` at both ends) at +// the new call site. +// +// Two flows the original has are not semantic and are tightened away +// (found on the fixtures, ruled false positives of the relation): +// - (c) anywhere but last: a deferred call runs at exit, so what it +// writes can no longer reach the statements after it +// (`defer readFull(r, arr[:]); return arr[0]` returns before +// readFull runs). Restricted to the last statement. +// - (b): a discarded result reaches the rest of the caller only by the +// coarse wiring (its actual_out → the call node, its host, spec §4.7) +// and then the call node's control dependence, which models that the +// call may panic, an exception edge (out of scope). Inside the +// closure that path ends at the closure. The original's graph leaves +// out the out-edges of the routed call's result actual_outs (Site.Cut). +var MR14 = Transform{Name: "MR14", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + if c.pre == nil { + return nil, fmt.Errorf("MR14 needs the original's analysis") + } + if usesPrefix(f, "mr14") { + return nil, errSkip("name clash") + } + info := f.Pkg.TypesInfo + fe := &fileEdit{} + var edits []edit + var tail []piece + for _, fd := range funcDecls(f) { + if fd.Body == nil { + continue + } + id := funcIdentity(f, fd) + cv := c.pre.Calls[id] + if cv == nil || !cv.L3 { + fe.skip("MR14: callable: no L3 result") + continue + } + type cand struct { + call *ast.CallExpr + callee *types.Func + ck string + ifaceOK bool + last bool + } + var cands []cand + last := map[ast.Stmt]bool{} + if n := len(fd.Body.List); n > 0 { + last[fd.Body.List[n-1]] = true + } + stmtLists(fd.Body, func(list []ast.Stmt) { + for _, st := range list { + es, ok := st.(*ast.ExprStmt) + if !ok { + continue + } + call, ok := es.X.(*ast.CallExpr) + if !ok { + continue + } + var callee *types.Func + switch fn := ast.Unparen(call.Fun).(type) { + case *ast.Ident: + callee, _ = info.Uses[fn].(*types.Func) + case *ast.SelectorExpr: + callee, _ = info.Uses[fn.Sel].(*types.Func) + } + if callee == nil { + continue + } + ck := f.key(call.Pos()) + if _, ok := cv.Body[ck+"/actual_in:0"]; !ok { + continue // no resolved callee with vertices + } + cands = append(cands, cand{call, callee, ck, ifaceForm(f, call, callee), last[st]}) + } + }) + if len(cands) == 0 { + fe.skip("MR14: callable: no direct call statement") + continue + } + x := cands[c.choose(len(cands))] + // A new literal would renumber the function's later closures + // (closure_N follows source order), so (b) needs a function + // without literals. + noLit := !contains(fd.Body, func(*ast.FuncLit) bool { return true }) + forms := []int{} + if x.ifaceOK { + forms = append(forms, 0) + } + if noLit { + forms = append(forms, 1) + } + if x.last { + forms = append(forms, 2) + } + if len(forms) == 0 { + fe.skip("MR14: site: no applicable form") + continue + } + form := forms[c.n%len(forms)] + a, z := f.off(x.call.Pos()), f.off(x.call.End()) + site := Site{Callable: id} + switch form { + case 0: + n := c.fresh("") + iface, impl := "mr14i"+n, "mr14t"+n + sig := x.callee.Type().(*types.Signature) + var params, names []string + for i := range sig.Params().Len() { + p := sig.Params().At(i) + t, _ := spell(f, p.Type()) + params = append(params, p.Name()+" "+t) + names = append(names, p.Name()) + } + var results []string + for i := range sig.Results().Len() { + t, _ := spell(f, sig.Results().At(i).Type()) + results = append(results, t) + } + msig := "(" + strings.Join(params, ", ") + ")" + if len(results) > 0 { + msig += " (" + strings.Join(results, ", ") + ")" + } + body := x.callee.Name() + "(" + strings.Join(names, ", ") + ")" + if len(results) > 0 { + body = "return " + body + } + m := x.callee.Name() + tail = append(tail, lit("\n\n// "+iface+" routes "+m+" through an interface (MR14).\ntype "+iface+" interface{ "+m+msig+" }\n\ntype "+impl+" struct{}\n\nfunc ("+impl+") "+m+msig+" { "+body+" }\n")) + // Only the function part is replaced: the arguments keep their + // text and positions. + edits = append(edits, edit{a: a, z: f.off(x.call.Lparen), anchor: true, out: []piece{lit(iface + "(" + impl + "{})." + m)}}) + site.Desc = desc(&scope{f: f, fd: fd}, x.call, "call "+m+" through "+iface) + for _, e := range edgesOf(cv, "sum") { + if strings.HasPrefix(e.src, x.ck+"/") && strings.HasPrefix(e.dst, x.ck+"/") { + site.Summary = append(site.Summary, SumCheck{Pre: e.src + " -> " + e.dst, Src: Pre(e.src), Dst: Pre(e.dst)}) + } + } + case 1: + edits = append(edits, insert(a, lit("func() { ")), insert(z, lit(" }()"))) + site.Desc = desc(&scope{f: f, fd: fd}, x.call, "call through a closure") + for k, b := range cv.Body { + if strings.HasPrefix(k, x.ck+"/actual_out") && strings.HasPrefix(strings.Split(b, "|")[2], "") { + site.Cut = append(site.Cut, Node(id, k)) + } + } + case 2: + edits = append(edits, insert(a, lit("defer "))) + site.Desc = desc(&scope{f: f, fd: fd}, x.call, "defer the call") + } + c.n++ + fe.sites = append(fe.sites, site) + fe.touched = append(fe.touched, id) + } + if len(edits) == 0 { + return fe, errSkip("MR14: no eligible site") + } + if len(tail) > 0 { + edits = append(edits, insert(len(f.Src), tail...)) + } + ps, err := applyEdits(f, edits) + fe.pieces = ps + return fe, err +}} + +// ifaceForm reports whether call can be routed through an interface: f is a +// non-generic, non-variadic function of the file's package, called by its +// bare name, with named, distinct parameters whose types (and results') the +// file can spell. +func ifaceForm(f *SrcFile, call *ast.CallExpr, callee *types.Func) bool { + if _, ok := ast.Unparen(call.Fun).(*ast.Ident); !ok || callee.Pkg() != f.Pkg.Types { + return false + } + sig := callee.Type().(*types.Signature) + if sig.Recv() != nil || sig.Variadic() || sig.TypeParams() != nil || call.Ellipsis.IsValid() { + return false + } + seen := map[string]bool{} + for i := range sig.Params().Len() { + p := sig.Params().At(i) + if p.Name() == "" || p.Name() == "_" || seen[p.Name()] || p.Name() == callee.Name() { + return false + } + seen[p.Name()] = true + if _, ok := spell(f, p.Type()); !ok { + return false + } + } + for i := range sig.Results().Len() { + if _, ok := spell(f, sig.Results().At(i).Type()); !ok { + return false + } + } + return sig.Params().Len() > 0 +} + +// ─── MR15: inline the MR6 helpers back ──────────────────────────────────────── + +// Inline6 undoes MR6 on its output: each `[var …; ]outs = mr6h_…(args)` +// statement becomes the helper's statements again, and the helpers are +// deleted. MR15 is MR6 then Inline6, which must give back the original +// source byte for byte, and the original analysis exactly. +var Inline6 = Transform{Name: "Inline6", edit: func(c *tctx, f *SrcFile) (*fileEdit, error) { + helpers := map[string]*ast.FuncDecl{} + for _, fd := range funcDecls(f) { + if strings.HasPrefix(fd.Name.Name, Prefix6) { + helpers[fd.Name.Name] = fd + } + } + if len(helpers) == 0 { + return nil, errSkip("no MR6 helper") + } + fe := &fileEdit{} + var edits []edit + for _, fd := range funcDecls(f) { + if helpers[fd.Name.Name] != nil || fd.Body == nil { + continue + } + stmtLists(fd.Body, func(list []ast.Stmt) { + for i, st := range list { + name := helperCalled(st) + h := helpers[name] + if h == nil { + continue + } + // MR6 writes one `var` per fresh local: the call's results + // beyond its arguments. + start := i - helperFresh(st) + if start < 0 || slices.ContainsFunc(list[start:i], func(d ast.Stmt) bool { return !isVarDecl(d) }) { + return + } + body := h.Body.List + if len(body) > 0 { + if _, ok := body[len(body)-1].(*ast.ReturnStmt); ok { + body = body[:len(body)-1] + } + } + if len(body) == 0 { + continue + } + edits = append(edits, edit{a: f.off(list[start].Pos()), z: f.off(st.End()), + out: []piece{cp(f, f.off(body[0].Pos()), f.off(body[len(body)-1].End()))}}) + fe.sites = append(fe.sites, Site{Desc: f.Rel + ": inline " + name}) + } + }) + } + for _, h := range helpers { + a := f.off(h.Doc.Pos()) - 2 + z := f.off(h.End()) + 1 + if a < 0 || string(f.Src[a:a+2]) != "\n\n" || z > len(f.Src) || f.Src[z-1] != '\n' { + return nil, fmt.Errorf("%s: helper %s is not as MR6 writes it", f.Rel, h.Name.Name) + } + edits = append(edits, edit{a: a, z: z}) + } + ps, err := applyEdits(f, edits) + fe.pieces = ps + return fe, err +}} + +// helperFresh is the number of fresh locals an MR6 call statement assigns. +func helperFresh(st ast.Stmt) int { + as, ok := st.(*ast.AssignStmt) + if !ok { + return 0 + } + return len(as.Lhs) - len(as.Rhs[0].(*ast.CallExpr).Args) +} + +func isVarDecl(st ast.Stmt) bool { + d, ok := st.(*ast.DeclStmt) + if !ok { + return false + } + g, ok := d.Decl.(*ast.GenDecl) + return ok && g.Tok == token.VAR +} + +// helperCalled is the MR6 helper st calls, or "". +func helperCalled(st ast.Stmt) string { + var call ast.Expr + switch st := st.(type) { + case *ast.AssignStmt: + if len(st.Rhs) == 1 { + call = st.Rhs[0] + } + case *ast.ExprStmt: + call = st.X + } + ce, ok := call.(*ast.CallExpr) + if !ok { + return "" + } + id, ok := ce.Fun.(*ast.Ident) + if !ok || !strings.HasPrefix(id.Name, Prefix6) { + return "" + } + return id.Name +} diff --git a/internal/neo4j/bolt.go b/internal/neo4j/bolt.go index 6dc6c08..60d2916 100644 --- a/internal/neo4j/bolt.go +++ b/internal/neo4j/bolt.go @@ -16,17 +16,29 @@ import ( // // Algorithm: // 1. ensure constraints + indexes; -// 2. schema-version gate: read :GoApplication.schema_version scoped to this -// app's id; a mismatch forces a full re-UPSERT (never a delete — that is +// 2. stored-application gate: read :GoApplication.schema_version and +// max_level scoped to this app's id; a version mismatch, or a stored level +// below this push's, forces a full re-UPSERT (never a delete — that is // --eager's job); // 3. partition nodes by owning module (NodeRow.module); shared nodes (externals, -// neutral repo nodes) have none and are MERGE-only, always upserted; +// neutral repo nodes) have none and are MERGE-only, always upserted; L4 +// vertex nodes (a callable's or a stand-in's formal/actual vertices) are +// their own group; // 4. --eager? scoped purge of everything under the app prefix, then rebuild; // 5. diff each module's content_hash against the DB → the changed set; // 6. per changed module: on --eager, purge its owned edges + stale decls, then // upsert its nodes; WITHOUT --eager the push only adds/updates; -// 7. upsert edges owned by a changed module (or shared); -// 8. on a full --eager run, orphan-prune modules whose source vanished. +// 7. upsert every L4 vertex node, changed module or not (D5, below); +// 8. upsert every edge (an unchanged edge's MERGE is a no-op); +// 9. on a full --eager run, orphan-prune modules whose source vanished. +// +// D5 (spec §4.11): L4 facts are not module-local — editing a callee changes +// its callers' summary edges, actual vertices and even formal_in indices — +// so the content_hash diff gates the L1–L3 rows only. A lazy push upserts +// every L4 row on every push: the vertex nodes (step 7), the stand-in +// bodies among them, and the GO_PARAM_IN / GO_PARAM_OUT / GO_SUMMARY rows +// and the port GO_DDG rows (prov ≠ ssa) with every other edge (step 8). It +// is additive: a stale L4 row stays until an --eager push. // // A push NEVER deletes by default (#171): steps 4/6/8 run under --eager only. // Every destructive statement is scoped on the can:/// id prefix (#173), @@ -79,11 +91,12 @@ func (w *boltWriter) run() error { return err } - // Schema-version gate: a producer/consumer mismatch forces a full re-upsert - // (it never deletes). We detect it but the upsert path below is already a - // full MERGE of every current node, so a mismatch simply means we don't - // trust the per-module content_hash diff and upsert everything. - forceFull, err := w.schemaVersionMismatch() + // Stored-application gate: a producer/consumer version mismatch, or a + // graph pushed at a lower level, forces a full re-upsert (it never + // deletes). The upsert path below is already a full MERGE of every + // current node, so it simply means we don't trust the per-module + // content_hash diff and upsert everything. + forceFull, err := w.storedAppStale() if err != nil { return err } @@ -95,9 +108,9 @@ func (w *boltWriter) run() error { } } - // Partition nodes by owning module; shared nodes (module == "") are always - // upserted. - byModule, shared := partitionNodes(w.rows.Nodes) + // Partition nodes by owning module; shared nodes (module == "") and L4 + // vertex nodes are always upserted. + byModule, shared, l4 := partitionNodes(w.rows.Nodes) // Diff content_hash per module unless a mismatch/eager forces everything. changed := map[string]bool{} @@ -129,12 +142,15 @@ func (w *boltWriter) run() error { return err } } + // L4 vertex nodes, after the module purges so an --eager purge of a + // changed module cannot delete them (D5). + if err := w.upsertNodes(l4); err != nil { + return err + } - // Upsert edges owned by a changed module (owner = source node's module) or - // shared. Determining ownership precisely needs the source node's module; we - // conservatively upsert every edge whose source id is under a changed module - // prefix, plus all shared/app-level edges. A MERGE-upsert of an unchanged - // edge is a harmless no-op. + // Upsert every edge: the L4 rows must be (D5), and for the L1–L3 rows + // the precise owner (the source node's module) buys nothing — a + // MERGE-upsert of an unchanged edge is a harmless no-op. if err := w.upsertEdges(w.rows.Edges); err != nil { return err } @@ -169,14 +185,28 @@ func (w *boltWriter) ensureDDL() error { }) } -// schemaVersionMismatch reads the stored :GoApplication.schema_version for THIS -// app and compares to the producer's SchemaVersion. A mismatch (or absence) -// returns true → the caller upserts everything rather than trusting the diff. -func (w *boltWriter) schemaVersionMismatch() (bool, error) { +// storedAppStale reads the stored :GoApplication.schema_version and +// max_level for THIS app and reports whether the content_hash diff cannot be +// trusted, so the caller upserts everything: +// +// - a stored schema_version different from the producer's SchemaVersion; +// - a stored max_level below this push's. A module's content_hash covers +// its source, not the level it was analysed at, so an unchanged module +// of a graph pushed at L3 still lacks what an L4 push adds to it — its L4 +// vertices are upserted (D5), but rows the diff skips stay at L3, and an +// edge row whose endpoint is missing is dropped by its MATCH silently; +// - a stored max_level present but not an integer: it cannot be compared, +// so the push fails safe. +// +// No stored application (first push) forces nothing beyond the normal +// upsert, nor does a stored level at or above this push's. +func (w *boltWriter) storedAppStale() (bool, error) { var stored string + var storedLevel int64 + haveLevel, badLevel := false, false err := w.read(func(tx neo4j.ManagedTransaction) error { res, err := tx.Run(w.ctx, - "MATCH (a:GoApplication {id: $id}) RETURN a.schema_version AS v", + "MATCH (a:GoApplication {id: $id}) RETURN a.schema_version AS v, a.max_level AS l", map[string]any{"id": w.appID}) if err != nil { return err @@ -185,15 +215,33 @@ func (w *boltWriter) schemaVersionMismatch() (bool, error) { if v, ok := res.Record().Get("v"); ok && v != nil { stored, _ = v.(string) } + if l, ok := res.Record().Get("l"); ok && l != nil { + storedLevel, haveLevel = l.(int64) + badLevel = !haveLevel + } } return res.Err() }) if err != nil { return false, err } - // No stored version (first push) is not a "mismatch" that forces anything - // beyond the normal upsert; a DIFFERENT stored version does. - return stored != "" && stored != SchemaVersion, nil + if stored != "" && stored != SchemaVersion || badLevel { + return true, nil + } + return haveLevel && storedLevel < int64(w.pushLevel()), nil +} + +// pushLevel is the max_level of the application row this push carries (0 +// when it carries none). +func (w *boltWriter) pushLevel() int { + for _, n := range w.rows.Nodes { + if n.Labels[0] == LabelApplication && n.Value == w.appID { + if l, ok := n.Props["max_level"].(int); ok { + return l + } + } + } + return 0 } // ─── destructive steps (--eager only) ───────────────────────────────────────── @@ -385,10 +433,25 @@ func (w *boltWriter) read(fn func(neo4j.ManagedTransaction) error) error { // nodeParams converts node rows to driver params. Props map directly (the // driver stores Go ints as integers, strings as strings, []string as a list). +// clearedAppProps are the :GoApplication props a push writes as an +// explicit null when the projection pruned them: `SET n += row.p` keeps a +// property row.p lacks, so a clean run over a graph a degraded run wrote +// would otherwise keep the stale degraded_json (F6). A null in SET += +// removes the property. (The snapshot script wipes the application first.) +var clearedAppProps = []string{"degraded_json"} + func nodeParams(g []NodeRow) []map[string]any { out := make([]map[string]any, len(g)) for i, n := range g { - out[i] = map[string]any{"k": n.Value, "p": propsToAny(n.Props)} + p := propsToAny(n.Props) + if len(n.Labels) > 0 && n.Labels[0] == LabelApplication { + for _, k := range clearedAppProps { + if _, ok := p[k]; !ok { + p[k] = nil + } + } + } + out[i] = map[string]any{"k": n.Value, "p": p} } return out } @@ -418,16 +481,34 @@ func propsToAny(p Props) map[string]any { // ─── partitioning & hashing ─────────────────────────────────────────────────── -func partitionNodes(nodes []NodeRow) (byModule map[string][]NodeRow, shared []NodeRow) { +// partitionNodes splits nodes into the module-owned L1–L3 nodes (by owning +// module), the shared nodes (no module), and the L4 vertex nodes, owned by +// a module or by a stand-in, which a push always upserts (D5). +func partitionNodes(nodes []NodeRow) (byModule map[string][]NodeRow, shared, l4 []NodeRow) { byModule = map[string][]NodeRow{} for _, n := range nodes { - if n.Module == "" { + switch { + case isL4Node(n): + l4 = append(l4, n) + case n.Module == "": shared = append(shared, n) - continue + default: + byModule[n.Module] = append(byModule[n.Module], n) } - byModule[n.Module] = append(byModule[n.Module], n) } - return byModule, shared + return byModule, shared, l4 +} + +// isL4Node reports an L4 vertex: a :GoBodyNode of a formal / actual kind. +func isL4Node(n NodeRow) bool { + if n.Labels[0] != LabelBodyNode { + return false + } + switch n.Props["kind"] { + case "formal_in", "formal_out", "actual_in", "actual_out": + return true + } + return false } // moduleHash returns the content_hash the projector put on the :GoModule node in diff --git a/internal/neo4j/bolt_integration_test.go b/internal/neo4j/bolt_integration_test.go index 72ac8b7..d99671d 100644 --- a/internal/neo4j/bolt_integration_test.go +++ b/internal/neo4j/bolt_integration_test.go @@ -16,10 +16,13 @@ package neo4j import ( "context" + "fmt" "os" "testing" "github.com/neo4j/neo4j-go-driver/v5/neo4j" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" ) func itConfig(t *testing.T) BoltConfig { @@ -128,3 +131,173 @@ func countSubgraph(t *testing.T, ctx context.Context, cfg BoltConfig, appID stri } return nodes, edges } + +// TestIntegration_LazyPushRestoresL4Rows (D5): after an eager push of the +// -a 4 dataflow graph, delete its L4 rows (every GO_PARAM_IN / GO_PARAM_OUT +// / GO_SUMMARY relationship and every vertex body node); a lazy push of the +// same rows — every module unchanged, so its L1–L3 rows are skipped — puts +// every one of them back. +func TestIntegration_LazyPushRestoresL4Rows(t *testing.T) { + ctx := context.Background() + cfg := itConfig(t) + a := buildDataflowL4(t) + rows := Project(a) + seed := cfg + seed.Eager, seed.FullRun = true, true + if err := WriteBolt(ctx, rows, a.Application.ID, seed); err != nil { + t.Fatalf("seed: %v", err) + } + want := l4Counts(t, ctx, cfg, a.Application.ID) + for k, n := range want { + if n == 0 { + t.Fatalf("seeded graph has no %s", k) + } + } + if want["GO_PARAM_IN"] != countEdges(rows, RelParamIn) || want["GO_SUMMARY"] != countEdges(rows, RelSummary) { + t.Errorf("seeded L4 counts %v differ from the projection", want) + } + runIT(t, ctx, cfg, "MATCH (n:GoBodyNode) WHERE n.id STARTS WITH $p AND n.kind IN ['formal_in','formal_out','actual_in','actual_out'] DETACH DELETE n", + map[string]any{"p": a.Application.ID + "/"}) + if got := l4Counts(t, ctx, cfg, a.Application.ID); got["vertex"] != 0 || got["GO_PARAM_IN"] != 0 { + t.Fatalf("L4 rows not deleted: %v", got) + } + if err := WriteBolt(ctx, rows, a.Application.ID, cfg); err != nil { + t.Fatalf("lazy push: %v", err) + } + if got := l4Counts(t, ctx, cfg, a.Application.ID); fmt.Sprint(got) != fmt.Sprint(want) { + t.Errorf("lazy push restored %v, want %v", got, want) + } +} + +func runIT(t *testing.T, ctx context.Context, cfg BoltConfig, q string, params map[string]any) { + t.Helper() + drv, err := neo4j.NewDriverWithContext(cfg.URI, neo4j.BasicAuth(cfg.User, cfg.Password, "")) + if err != nil { + t.Fatalf("driver: %v", err) + } + defer drv.Close(ctx) + if _, err := neo4j.ExecuteQuery(ctx, drv, q, params, neo4j.EagerResultTransformer, neo4j.ExecuteQueryWithDatabase(cfg.Database)); err != nil { + t.Fatalf("%s: %v", q, err) + } +} + +// l4Counts counts this app's L4 rows: vertex body nodes and the three L4 +// relationship types, plus the port GO_DDG rows. +func l4Counts(t *testing.T, ctx context.Context, cfg BoltConfig, appID string) map[string]int { + t.Helper() + drv, err := neo4j.NewDriverWithContext(cfg.URI, neo4j.BasicAuth(cfg.User, cfg.Password, "")) + if err != nil { + t.Fatalf("driver: %v", err) + } + defer drv.Close(ctx) + p := map[string]any{"p": appID + "/"} + out := map[string]int{} + for k, q := range map[string]string{ + "vertex": "MATCH (n:GoBodyNode) WHERE n.id STARTS WITH $p AND n.kind IN ['formal_in','formal_out','actual_in','actual_out'] RETURN count(n)", + "GO_PARAM_IN": "MATCH (n:GoBodyNode)-[r:GO_PARAM_IN]->() WHERE n.id STARTS WITH $p RETURN count(r)", + "GO_PARAM_OUT": "MATCH (n:GoBodyNode)-[r:GO_PARAM_OUT]->() WHERE n.id STARTS WITH $p RETURN count(r)", + "GO_SUMMARY": "MATCH (n:GoBodyNode)-[r:GO_SUMMARY]->() WHERE n.id STARTS WITH $p RETURN count(r)", + "GO_DDG port": "MATCH (n:GoBodyNode)-[r:GO_DDG]->() WHERE n.id STARTS WITH $p AND r.prov <> ['ssa'] RETURN count(r)", + } { + res, err := neo4j.ExecuteQuery(ctx, drv, q, p, neo4j.EagerResultTransformer, neo4j.ExecuteQueryWithDatabase(cfg.Database)) + if err != nil { + t.Fatalf("%s: %v", q, err) + } + out[k] = int(res.Records[0].Values[0].(int64)) + } + return out +} + +// TestIntegration_LazyL4PushOverLowerLevelGraph: a graph first pushed at a +// lower level (L2, L3), then pushed lazily at L4 with every module's source +// unchanged, ends equal to an eager L4 push. The stored max_level below 4 +// forces the full upsert; without it the content_hash diff skips the +// unchanged modules, and an L2 graph lacks the L3 statement nodes the L4 +// port and cfg/ddg rows end at, so their MATCH drops them silently. +func TestIntegration_LazyL4PushOverLowerLevelGraph(t *testing.T) { + ctx := context.Background() + cfg := itConfig(t) + eager := cfg + eager.Eager, eager.FullRun = true, true + l4 := buildDataflowL4(t) + id := l4.Application.ID + rows4 := Project(l4) + if err := WriteBolt(ctx, rows4, id, eager); err != nil { + t.Fatalf("eager L4: %v", err) + } + wantN, wantE := countSubgraph(t, ctx, cfg, id) + want := l4Counts(t, ctx, cfg, id) + + for _, seed := range []*v2.Analysis{buildDataflowAt(t, 2), buildDataflowL3(t)} { + if seed.Application.ID != id { + t.Fatalf("app ids differ: %s, %s", seed.Application.ID, id) + } + if err := WriteBolt(ctx, Project(seed), id, eager); err != nil { + t.Fatalf("eager L%d: %v", seed.MaxLevel, err) + } + if got := l4Counts(t, ctx, cfg, id); got["GO_PARAM_IN"] != 0 { + t.Fatalf("L%d graph has L4 rows: %v", seed.MaxLevel, got) + } + if err := WriteBolt(ctx, rows4, id, cfg); err != nil { + t.Fatalf("lazy L4: %v", err) + } + gotN, gotE := countSubgraph(t, ctx, cfg, id) + if got := l4Counts(t, ctx, cfg, id); fmt.Sprint(got) != fmt.Sprint(want) { + t.Errorf("lazy L4 over L%d: L4 rows %v, want %v", seed.MaxLevel, got, want) + } + if gotN != wantN || gotE != wantE { + t.Errorf("lazy L4 over L%d: %d nodes, %d edges; eager L4 %d, %d", seed.MaxLevel, gotN, gotE, wantN, wantE) + } + } +} + +// TestIntegration_CleanLazyPushClearsDegradedJSON (F6): a graph a degraded +// run wrote carries :GoApplication.degraded_json; a clean lazy push of the +// same app removes it (the application row's explicit null), so the SDK no +// longer reads a stale degradation. +func TestIntegration_CleanLazyPushClearsDegradedJSON(t *testing.T) { + ctx := context.Background() + cfg := itConfig(t) + a := buildDataflowL4(t) + if len(a.Degraded) != 0 { + t.Fatalf("fixture degraded: %+v", a.Degraded) + } + degraded := Project(a) + for i, n := range degraded.Nodes { + if n.Labels[0] == LabelApplication { + p := Props{} + for k, v := range n.Props { + p[k] = v + } + p["degraded_json"] = `[{"code":"no_vta"}]` + degraded.Nodes[i].Props = p + } + } + seed := cfg + seed.Eager, seed.FullRun = true, true + if err := WriteBolt(ctx, degraded, a.Application.ID, seed); err != nil { + t.Fatalf("seed: %v", err) + } + read := func() any { + drv, err := neo4j.NewDriverWithContext(cfg.URI, neo4j.BasicAuth(cfg.User, cfg.Password, "")) + if err != nil { + t.Fatalf("driver: %v", err) + } + defer drv.Close(ctx) + res, err := neo4j.ExecuteQuery(ctx, drv, "MATCH (a:GoApplication {id: $id}) RETURN a.degraded_json", + map[string]any{"id": a.Application.ID}, neo4j.EagerResultTransformer, neo4j.ExecuteQueryWithDatabase(cfg.Database)) + if err != nil || len(res.Records) != 1 { + t.Fatalf("read degraded_json: %v (%d records)", err, len(res.Records)) + } + return res.Records[0].Values[0] + } + if v := read(); v != `[{"code":"no_vta"}]` { + t.Fatalf("seeded degraded_json = %v", v) + } + if err := WriteBolt(ctx, Project(a), a.Application.ID, cfg); err != nil { + t.Fatalf("lazy push: %v", err) + } + if v := read(); v != nil { + t.Errorf("clean lazy push left degraded_json = %v", v) + } +} diff --git a/internal/neo4j/bolt_session_test.go b/internal/neo4j/bolt_session_test.go index ce13d83..1cc8d6f 100644 --- a/internal/neo4j/bolt_session_test.go +++ b/internal/neo4j/bolt_session_test.go @@ -72,3 +72,317 @@ func TestEagerPurge_OnlyOnFullRun(t *testing.T) { } } } + +// fakeDB is a DB-free SessionWithContext for the push algorithm: reads +// answer the stored-application gate with SchemaVersion and level (when +// non-zero; else no stored level) and the content_hash diff with +// hashes[module id] (an unchanged module when it equals the local hash); +// every write statement is recorded with its parameters. +type fakeDB struct { + neo4j.SessionWithContext + hashes map[string]string + level int64 + // rawLevel, when non-nil, is the stored max_level returned verbatim + // (any type), overriding level. + rawLevel any + writes []fakeStmt +} + +type fakeStmt struct { + q string + params map[string]any +} + +type fakeTx struct { + neo4j.ManagedTransaction + db *fakeDB +} + +func (s *fakeDB) Run(_ context.Context, q string, params map[string]any, _ ...func(*neo4j.TransactionConfig)) (neo4j.ResultWithContext, error) { + s.writes = append(s.writes, fakeStmt{q, params}) + return consumedResult{}, nil +} + +func (s *fakeDB) ExecuteWrite(ctx context.Context, work neo4j.ManagedTransactionWork, _ ...func(*neo4j.TransactionConfig)) (any, error) { + return work(fakeTx{db: s}) +} + +func (s *fakeDB) ExecuteRead(ctx context.Context, work neo4j.ManagedTransactionWork, _ ...func(*neo4j.TransactionConfig)) (any, error) { + return work(fakeTx{db: s}) +} + +func (tx fakeTx) Run(_ context.Context, q string, params map[string]any) (neo4j.ResultWithContext, error) { + switch { + case strings.Contains(q, "a.schema_version"): + var lvl any + if tx.db.level != 0 { + lvl = tx.db.level + } + if tx.db.rawLevel != nil { + lvl = tx.db.rawLevel + } + return &fakeResult{rec: &neo4j.Record{Keys: []string{"v", "l"}, Values: []any{SchemaVersion, lvl}}}, nil + case strings.Contains(q, "m.content_hash"): + h, ok := tx.db.hashes[params["id"].(string)] + if !ok { + return &fakeResult{}, nil + } + return &fakeResult{rec: &neo4j.Record{Keys: []string{"h"}, Values: []any{h}}}, nil + } + tx.db.writes = append(tx.db.writes, fakeStmt{q, params}) + return &fakeResult{}, nil +} + +type fakeResult struct { + neo4j.ResultWithContext + rec *neo4j.Record + done bool +} + +func (r *fakeResult) Next(context.Context) bool { + if r.rec == nil || r.done { + return false + } + r.done = true + return true +} +func (r *fakeResult) Record() *neo4j.Record { return r.rec } +func (r *fakeResult) Err() error { return nil } + +// pushedIDs returns the node keys (row.k of node MERGEs) and the edge +// (type, a, b) triples a push wrote. +func (s *fakeDB) pushed() (nodes map[string]bool, edges map[string]int) { + nodes, edges = map[string]bool{}, map[string]int{} + for _, st := range s.writes { + rows, _ := st.params["rows"].([]map[string]any) + switch { + case strings.Contains(st.q, "MERGE (n:"): + for _, r := range rows { + nodes[r["k"].(string)] = true + } + case strings.Contains(st.q, "MERGE (a)-[r:"): + typ := st.q[strings.Index(st.q, "[r:")+3:] + typ = typ[:strings.IndexAny(typ, " ]")] + edges[typ] += len(rows) + } + } + return nodes, edges +} + +// TestBolt_LazyPushUpsertsL4RowsOfUnchangedModules (D5): on a lazy push +// where every module's content_hash matches the DB, the L1–L3 nodes of the +// (unchanged) modules are skipped, but every L4 row is upserted — each param +// vertex of a callable, each stand-in body node, and every GO_PARAM_IN / +// GO_PARAM_OUT / GO_SUMMARY row and port GO_DDG row (prov ≠ ssa). On an +// eager push nothing is skipped. +func TestBolt_LazyPushUpsertsL4RowsOfUnchangedModules(t *testing.T) { + a := buildDataflowL4(t) + rows := Project(a) + byModule, _, _ := partitionNodes(rows.Nodes) + hashes := map[string]string{} + for m, ns := range byModule { + hashes[m] = moduleHash(ns) + if hashes[m] == "" { + t.Fatalf("module %s has no content_hash", m) + } + } + db := &fakeDB{hashes: hashes} + w := &boltWriter{ctx: context.Background(), session: db, rows: rows, appID: a.Application.ID} + if err := w.run(); err != nil { + t.Fatal(err) + } + nodes, edges := db.pushed() + + var vertices, l3Skipped int + for _, n := range rows.Nodes { + if n.Labels[0] != LabelBodyNode { + continue + } + kind, _ := n.Props["kind"].(string) + switch { + case isVertex(kind): + vertices++ + if !nodes[n.Value] { + t.Errorf("L4 vertex %s of an unchanged module not upserted", n.Value) + } + case n.Module != "": + if nodes[n.Value] { + t.Errorf("L3 body node %s of an unchanged module upserted on a lazy push", n.Value) + } + l3Skipped++ + } + } + if vertices == 0 || l3Skipped == 0 { + t.Fatalf("fixture: %d vertices, %d L3 body nodes", vertices, l3Skipped) + } + standIns := 0 + for _, ext := range a.Application.ExternalSymbols { + for _, n := range ext.Body { + standIns++ + if !nodes[n.ID] { + t.Errorf("stand-in vertex %s not upserted", n.ID) + } + } + } + if standIns == 0 { + t.Fatal("fixture has no stand-in body") + } + portRows := 0 + for _, e := range rows.Edges { + if p, _ := e.Props["prov"].([]string); e.Type == RelDDG && strings.Join(p, ",") != "ssa" { + portRows++ + } + } + for _, rel := range []string{RelParamIn, RelParamOut, RelSummary, RelDDG} { + if want := countEdges(rows, rel); want == 0 || edges[rel] != want { + t.Errorf("%s: pushed %d rows, projected %d", rel, edges[rel], want) + } + } + if portRows == 0 { + t.Error("fixture has no port GO_DDG row") + } + + // Eager: every node of every module is upserted. + db = &fakeDB{hashes: hashes} + w = &boltWriter{ctx: context.Background(), session: db, rows: rows, appID: a.Application.ID, cfg: BoltConfig{Eager: true, FullRun: true}} + if err := w.run(); err != nil { + t.Fatal(err) + } + nodes, _ = db.pushed() + for _, n := range rows.Nodes { + if !nodes[n.Value] { + t.Fatalf("eager push skipped %s", n.Value) + } + } +} + +// TestBolt_LazyPushForcesFullOnLowerStoredLevel: a graph first pushed at a +// lower level (an L3 push) has no L4 rows, and a lazy L4 push must not skip +// an unchanged module's L3 nodes: they may be missing L4 vertices' parents, +// and an edge MATCH on a missing endpoint drops the row silently. A stored +// max_level below the current one forces every module's upsert; an equal +// (or higher) stored level keeps the content_hash diff. +func TestBolt_LazyPushForcesFullOnLowerStoredLevel(t *testing.T) { + a := buildDataflowL4(t) + if a.MaxLevel != 4 { + t.Fatalf("fixture max_level = %d, want 4", a.MaxLevel) + } + rows := Project(a) + byModule, _, _ := partitionNodes(rows.Nodes) + hashes := map[string]string{} + for m, ns := range byModule { + hashes[m] = moduleHash(ns) + } + push := func(stored int64) map[string]bool { + db := &fakeDB{hashes: hashes, level: stored} + w := &boltWriter{ctx: context.Background(), session: db, rows: rows, appID: a.Application.ID} + if err := w.run(); err != nil { + t.Fatal(err) + } + nodes, _ := db.pushed() + return nodes + } + moduleNodes := 0 + for _, ns := range byModule { + moduleNodes += len(ns) + } + if moduleNodes == 0 { + t.Fatal("fixture has no module-owned nodes") + } + for _, stored := range []int64{1, 3} { + nodes := push(stored) + for m, ns := range byModule { + for _, n := range ns { + if !nodes[n.Value] { + t.Fatalf("stored level %d: module %s node %s not upserted", stored, m, n.Value) + } + } + } + } + for _, stored := range []int64{4, 5} { + nodes := push(stored) + for _, ns := range byModule { + for _, n := range ns { + if nodes[n.Value] { + t.Fatalf("stored level %d: unchanged module node %s upserted", stored, n.Value) + } + } + } + } +} + +// TestBolt_LazyPushForcesFullOnUnreadableStoredLevel: a stored max_level +// that is present but not an integer (a float, a string — a graph written by +// another tool, or hand-edited) cannot be compared with this push's level, +// so the lazy push fails safe and upserts every module, as for a lower one. +func TestBolt_LazyPushForcesFullOnUnreadableStoredLevel(t *testing.T) { + a := buildDataflowL4(t) + rows := Project(a) + byModule, _, _ := partitionNodes(rows.Nodes) + hashes := map[string]string{} + for m, ns := range byModule { + hashes[m] = moduleHash(ns) + } + for _, raw := range []any{float64(4), "4", true} { + db := &fakeDB{hashes: hashes, rawLevel: raw} + w := &boltWriter{ctx: context.Background(), session: db, rows: rows, appID: a.Application.ID} + if err := w.run(); err != nil { + t.Fatal(err) + } + nodes, _ := db.pushed() + for m, ns := range byModule { + for _, n := range ns { + if !nodes[n.Value] { + t.Fatalf("stored max_level %T(%v): module %s node %s not upserted", raw, raw, m, n.Value) + } + } + } + } +} + +// TestBolt_CleanPushClearsDegradedJSON (F6): the projector prunes a nil +// degraded_json, and `SET n += row.p` keeps a property row.p lacks, so a +// clean push over a graph an earlier degraded run wrote would leave the +// stale degraded_json behind (and the SDK would refuse "no flow" +// certificates). The application row therefore carries an explicit null +// when nothing is degraded, which SET += removes; a degraded run keeps its +// JSON string. +func TestBolt_CleanPushClearsDegradedJSON(t *testing.T) { + a := buildDataflowL4(t) + if len(a.Degraded) != 0 { + t.Fatalf("fixture degraded: %+v", a.Degraded) + } + appProps := func(rows GraphRows) map[string]any { + t.Helper() + db := &fakeDB{level: 4} + w := &boltWriter{ctx: context.Background(), session: db, rows: rows, appID: a.Application.ID} + if err := w.run(); err != nil { + t.Fatal(err) + } + for _, st := range db.writes { + if !strings.Contains(st.q, "MERGE (n:"+LabelApplication+" ") { + continue + } + for _, r := range st.params["rows"].([]map[string]any) { + if r["k"] == a.Application.ID { + return r["p"].(map[string]any) + } + } + } + t.Fatal("no :GoApplication upsert") + return nil + } + p := appProps(Project(a)) + if v, has := p["degraded_json"]; !has || v != nil { + t.Errorf("clean push: degraded_json = %v (present %v), want an explicit null", v, has) + } + rows := Project(a) + for i, n := range rows.Nodes { + if n.Labels[0] == LabelApplication { + rows.Nodes[i].Props = Props{"degraded_json": `[{"code":"no_vta"}]`, "max_level": int64(4)} + } + } + if v := appProps(rows)["degraded_json"]; v != `[{"code":"no_vta"}]` { + t.Errorf("degraded push: degraded_json = %v, want the JSON string", v) + } +} diff --git a/internal/neo4j/bolt_test.go b/internal/neo4j/bolt_test.go index bb3723a..cd6972f 100644 --- a/internal/neo4j/bolt_test.go +++ b/internal/neo4j/bolt_test.go @@ -19,13 +19,16 @@ func TestPartitionNodes_SharedHaveNoModule(t *testing.T) { b.Node([]string{SymbolLabel, LabelExternal, MarkerLabel}, "id", "can://app/@external/fmt", nil, "") // shared rows := b.Finish() - byModule, shared := partitionNodes(rows.Nodes) + byModule, shared, l4 := partitionNodes(rows.Nodes) if len(byModule["x.go"]) != 2 { t.Errorf("module x.go should own 2 nodes, got %d", len(byModule["x.go"])) } if len(shared) != 1 { t.Errorf("expected 1 shared (no-module) node, got %d", len(shared)) } + if len(l4) != 0 { + t.Errorf("expected no L4 node, got %d", len(l4)) + } } func TestModuleHash_ReadsModuleNode(t *testing.T) { diff --git a/internal/neo4j/project.go b/internal/neo4j/project.go index 8c74346..9584338 100644 --- a/internal/neo4j/project.go +++ b/internal/neo4j/project.go @@ -18,9 +18,9 @@ import ( // every id the app owns, which is what makes prefix scoping exact. // // Walk order (spine): application → per module (types → callables → fields → -// body call nodes) → call_graph → imports → neutral artifact/package layer → -// reserved L3/L4 overlay (zero rows at the current max level). Every endpoint -// resolves to a declared node or an id-keyed :GoExternal ghost — nothing dangles. +// body nodes and the L3/L4 intra-callable overlay) → call_graph → imports → +// stand-in bodies → param edges. Every endpoint resolves to a declared node or +// an id-keyed :GoExternal ghost — nothing dangles. // Project is the pure (tree) → GraphRows function. appID is analysis.Application.ID // (the can:// root); it is read from the envelope, never recomposed. @@ -37,6 +37,8 @@ func Project(a *v2.Analysis) GraphRows { p.projectModule(mod) } p.projectCallGraph() + p.projectStandIns() + p.projectParamEdges() return p.b.Finish() } @@ -59,11 +61,25 @@ func (p *projector) projectApplication() { "max_level": a.MaxLevel, "analyzer_name": a.Analyzer.Name, "analyzer_version": a.Analyzer.Version, + "degraded_json": degradedJSON(a.Degraded), }, "", // the application root belongs to no single module ) } +// degradedJSON is the envelope's degraded[] (spec §4.15) as a JSON string, +// nil (pruned) when nothing degraded. +func degradedJSON(d []v2.Degraded) Value { + if len(d) == 0 { + return nil + } + data, err := json.Marshal(d) + if err != nil { + return nil + } + return string(data) +} + func (p *projector) appRef() NodeRef { return NodeRef{Label: LabelApplication, KeyProp: "id", Value: p.analysis.Application.ID} } @@ -174,6 +190,9 @@ func (p *projector) projectCallable(parentRef NodeRef, module string, c v2.Calla return cRef } +// projectBodyNode emits one body node of the callable (or, for a stand-in, +// the :GoExternal) at callableRef and its GO_HAS_BODY_NODE. module is the +// owning module, "" for a stand-in's vertices (shared, like the ghost). func (p *projector) projectBodyNode(callableRef NodeRef, module, callableID, local string, bn v2.BodyNode) { // GlobalOrdinal, not callableID+"@"+local: synthetic keys ("@entry") already // carry the "@", and the JSON body node id uses the same rule. @@ -195,8 +214,8 @@ func (p *projector) projectBodyNode(callableRef NodeRef, module, callableID, loc "is_deferred": boolOrNil(bn.IsDeferred), "span_json": bodySpanJSON(bn.Span), "call_node": callNode, - // "var" (JSON `of`) is L4: the v2 BodyNode carries no such field yet, - // and a nil prop is pruned, so L3 emits none. + // var is JSON `of`: set on L4 vertices only (pruned when empty). + "var": emptyToNil(bn.Of), }, module, ) @@ -213,10 +232,11 @@ func (p *projector) projectBodyNode(callableRef NodeRef, module, callableID, loc } } -// projectFlow emits a callable's L3 intra-callable overlay. JSON edge endpoints -// are LOCAL body keys; the graph addresses :GoBodyNode by its global id, so each -// endpoint goes through v2.GlobalOrdinal. Parallel edges survive via _k: -// GO_CFG_NEXT per kind, GO_DDG per (var, prov). +// projectFlow emits a callable's intra-callable overlay: L3's cfg / cdg / ddg +// and L4's port ddg edges (prov reaching-defs / points-to) and summary. JSON +// edge endpoints are LOCAL body keys; the graph addresses :GoBodyNode by its +// global id, so each endpoint goes through v2.GlobalOrdinal. Parallel edges +// survive via _k: GO_CFG_NEXT per kind, GO_DDG per (var, prov). func (p *projector) projectFlow(c v2.Callable) { ref := func(local string) NodeRef { return NodeRef{Label: LabelBodyNode, KeyProp: "id", Value: v2.GlobalOrdinal(c.ID, local)} @@ -234,6 +254,40 @@ func (p *projector) projectFlow(c v2.Callable) { Props{"var": e.Var, "prov": stringsOrNil(prov)}, e.Var+"|"+strings.Join(prov, ",")) } + for _, e := range c.Summary { + p.b.Edge(RelSummary, ref(e.Src), ref(e.Dst), nil, "") + } +} + +// ─── L4: stand-ins and param edges ─────────────────────────────────────────── + +// projectStandIns emits each library stand-in's body (its formal vertices, +// spec §6) as :GoBodyNode rows owned by the external's :GoExternal ghost — +// the same id-keyed node call_graph endpoints use — via GO_HAS_BODY_NODE. +// Stand-ins are app-scoped: their rows belong to no module. +func (p *projector) projectStandIns() { + for id, ext := range p.analysis.Application.ExternalSymbols { + if len(ext.Body) == 0 { + continue + } + ghost := p.callEndpoint(id) + for local, bn := range ext.Body { + p.projectBodyNode(ghost, "", id, local, bn) + } + } +} + +// projectParamEdges emits GO_PARAM_IN / GO_PARAM_OUT {var}, one row per JSON +// param_in / param_out edge. Endpoints are body node ids already (the +// emitter minted them), addressed as :GoBodyNode. +func (p *projector) projectParamEdges() { + ref := func(id string) NodeRef { return NodeRef{Label: LabelBodyNode, KeyProp: "id", Value: id} } + for _, e := range p.analysis.Application.ParamIn { + p.b.Edge(RelParamIn, ref(e.Src), ref(e.Dst), Props{"var": e.Var}, "") + } + for _, e := range p.analysis.Application.ParamOut { + p.b.Edge(RelParamOut, ref(e.Src), ref(e.Dst), Props{"var": e.Var}, "") + } } // ─── imports ────────────────────────────────────────────────────────────────── diff --git a/internal/neo4j/project_test.go b/internal/neo4j/project_test.go index a2d545e..cebe5f9 100644 --- a/internal/neo4j/project_test.go +++ b/internal/neo4j/project_test.go @@ -1,14 +1,17 @@ package neo4j import ( + "encoding/json" "path/filepath" "runtime" "strings" + "sync" "testing" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis/v2emit" @@ -36,25 +39,40 @@ func buildMultipackageL2(t *testing.T) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} - return v2emit.Emit(app, "multipackage", dir, 2, "neo4j-gate", 0, nil) + return v2emit.Emit(app, "multipackage", dir, 2, "neo4j-gate", 0, nil, nil) } func TestProject_NoDanglingEndpoints(t *testing.T) { - rows := Project(buildMultipackageL2(t)) + for _, c := range []struct { + name string + a func(*testing.T) *v2.Analysis + }{ + {"multipackage L2", buildMultipackageL2}, + {"dataflow L4", buildDataflowL4}, + } { + rows := Project(c.a(t)) - // Every (label, value) that was emitted as a node. - emitted := make(map[string]bool, len(rows.Nodes)) - for _, n := range rows.Nodes { - for _, l := range n.Labels { - emitted[l+"\x00"+n.Value] = true + // Every (label, value) that was emitted as a node. + emitted := make(map[string]bool, len(rows.Nodes)) + for _, n := range rows.Nodes { + for _, l := range n.Labels { + emitted[l+"\x00"+n.Value] = true + } } - } - for _, e := range rows.Edges { - if !emitted[e.From.Label+"\x00"+e.From.Value] { - t.Errorf("%s edge FROM a non-emitted node: %s %s", e.Type, e.From.Label, e.From.Value) + for _, e := range rows.Edges { + if !emitted[e.From.Label+"\x00"+e.From.Value] { + t.Errorf("%s: %s edge FROM a non-emitted node: %s %s", c.name, e.Type, e.From.Label, e.From.Value) + } + if !emitted[e.To.Label+"\x00"+e.To.Value] { + t.Errorf("%s: %s edge TO a non-emitted node: %s %s", c.name, e.Type, e.To.Label, e.To.Value) + } } - if !emitted[e.To.Label+"\x00"+e.To.Value] { - t.Errorf("%s edge TO a non-emitted node: %s %s", e.Type, e.To.Label, e.To.Value) + if c.name == "dataflow L4" { + for _, rel := range []string{RelParamIn, RelParamOut, RelSummary} { + if countEdges(rows, rel) == 0 { + t.Errorf("%s: no %s row checked", c.name, rel) + } + } } } } @@ -201,6 +219,12 @@ func TestProject_ExternalGhostCarriesSignature(t *testing.T) { // buildDataflowL3 builds testdata/dataflow through the real -a 3 wiring (symbol // table, resolver, dataflow driver) and emits the v2 L3 analysis. func buildDataflowL3(t *testing.T) *v2.Analysis { + t.Helper() + return buildDataflowAt(t, 3) +} + +// buildDataflowAt is buildDataflowL3 at level 2 or 3 (no dataflow below 3). +func buildDataflowAt(t *testing.T, level int) *v2.Analysis { t.Helper() _, thisFile, _, _ := runtime.Caller(0) dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "dataflow")) @@ -215,8 +239,11 @@ func buildDataflowL3(t *testing.T) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + if level < 3 { + return v2emit.Emit(app, "dataflow", dir, level, "neo4j-gate", 3, nil, nil) + } res := dataflow.Run(b.Fset(), b.L3Units(st), 3, 4) - return v2emit.Emit(app, "dataflow", dir, 3, "neo4j-gate", 3, res) + return v2emit.Emit(app, "dataflow", dir, level, "neo4j-gate", 3, res, nil) } // walkCallables visits every callable (methods and nested closures included). @@ -498,3 +525,224 @@ func TestProject_CallNodeResolvesWithinOwner(t *testing.T) { t.Errorf("only %d call_node props checked", checked) } } + +// ─── L4 overlay ────────────────────────────────────────────────────────────── + +var ( + l4Once sync.Once + l4Analysis *v2.Analysis +) + +// buildDataflowL4 builds testdata/dataflow through the real -a 4 wiring +// (symbol table, resolver, dataflow, sdg.Run merged into the application) +// and emits the v2 L4 analysis. Built once: the projection never mutates it. +func buildDataflowL4(t *testing.T) *v2.Analysis { + t.Helper() + l4Once.Do(func() { + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", "dataflow")) + if err != nil { + t.Fatalf("resolving fixture dir: %v", err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + units := b.L3Units(st) + res := dataflow.RunWithSideTables(b.Fset(), units, 3, 4) + l4 := sdg.Run(sdg.Input{ + Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Externals: app.Externals, CallGraph: app.CallGraph, Units: units, L3: res, K: 3, Jobs: 4, + }) + l4.ApplyTo(app) + l4Analysis = v2emit.Emit(app, "dataflow", dir, 4, "neo4j-gate", 3, res, &l4) + }) + if l4Analysis == nil { + t.Fatal("L4 fixture build failed") + } + return l4Analysis +} + +func isVertex(kind string) bool { + switch kind { + case "formal_in", "formal_out", "actual_in", "actual_out": + return true + } + return false +} + +// TestProject_L4CountsMatchJSON is the cross-projection gate at -a 4: per +// kind, the graph carries exactly the JSON's facts — GO_PARAM_IN / OUT per +// param edge, GO_SUMMARY per summary edge, GO_DDG per ddg edge (port edges +// included), one :GoBodyNode and one GO_HAS_BODY_NODE per body node of a +// callable or a stand-in. +func TestProject_L4CountsMatchJSON(t *testing.T) { + a := buildDataflowL4(t) + rows := Project(a) + var ddg, port, summary, body, vertices int + walkCallables(a, func(c v2.Callable) { + ddg += len(c.Ddg) + for _, e := range c.Ddg { + if strings.Join(e.Prov, ",") != "ssa" { + port++ + } + } + summary += len(c.Summary) + body += len(c.Body) + for _, n := range c.Body { + if isVertex(n.Kind) { + vertices++ + } + } + }) + standIns := 0 + for _, ext := range a.Application.ExternalSymbols { + standIns += len(ext.Body) + } + if port == 0 || summary == 0 || vertices == 0 || standIns == 0 || len(a.Application.ParamIn) == 0 || len(a.Application.ParamOut) == 0 { + t.Fatalf("fixture has no L4 content: port=%d summary=%d vertices=%d stand-ins=%d param_in=%d param_out=%d", + port, summary, vertices, standIns, len(a.Application.ParamIn), len(a.Application.ParamOut)) + } + portRows := 0 + for _, e := range rows.Edges { + if e.Type == RelDDG { + if p, _ := e.Props["prov"].([]string); strings.Join(p, ",") != "ssa" { + portRows++ + if e.Key != e.Props["var"].(string)+"|"+strings.Join(p, ",") { + t.Errorf("port GO_DDG _k = %q", e.Key) + } + } + } + } + nodes, vertexRows := 0, 0 + for _, n := range rows.Nodes { + if n.Labels[0] == LabelBodyNode { + nodes++ + if k, _ := n.Props["kind"].(string); isVertex(k) { + vertexRows++ + } + } + } + for _, c := range []struct { + name string + got, want int + }{ + {RelParamIn, countEdges(rows, RelParamIn), len(a.Application.ParamIn)}, + {RelParamOut, countEdges(rows, RelParamOut), len(a.Application.ParamOut)}, + {RelSummary, countEdges(rows, RelSummary), summary}, + {RelDDG, countEdges(rows, RelDDG), ddg}, + {"GO_DDG (port)", portRows, port}, + {RelHasBodyNode, countEdges(rows, RelHasBodyNode), body + standIns}, + {":GoBodyNode", nodes, body + standIns}, + {":GoBodyNode (vertex)", vertexRows, vertices + standIns}, + } { + if c.got != c.want { + t.Errorf("%s rows = %d, JSON = %d", c.name, c.got, c.want) + } + } +} + +// TestProject_L4VertexProps: a vertex projects as a :GoBodyNode with var = +// its `of` and call_node = its JSON parent (the local key); a stand-in's +// vertices hang off its :GoExternal ghost by GO_HAS_BODY_NODE; param edges +// carry var and join two :GoBodyNode ids. +func TestProject_L4VertexProps(t *testing.T) { + a := buildDataflowL4(t) + rows := Project(a) + byID := map[string]NodeRow{} + for _, n := range rows.Nodes { + byID[n.Labels[0]+"\x00"+n.Value] = n + } + walkCallables(a, func(c v2.Callable) { + for _, n := range c.Body { + if !isVertex(n.Kind) { + continue + } + row, ok := byID[LabelBodyNode+"\x00"+n.ID] + if !ok { + t.Errorf("vertex %s not projected", n.ID) + continue + } + if row.Props["var"] != n.Of { + t.Errorf("%s var = %v, want %q", n.ID, row.Props["var"], n.Of) + } + if cn, has := row.Props["call_node"]; (n.Parent != "") != has || (has && cn != n.Parent) { + t.Errorf("%s call_node = %v, want parent %q", n.ID, cn, n.Parent) + } + if _, has := row.Props["span_json"]; has { + t.Errorf("%s carries span_json", n.ID) + } + } + }) + owner := map[string]NodeRef{} + for _, e := range rows.Edges { + if e.Type == RelHasBodyNode { + owner[e.To.Value] = e.From + } + } + checked := 0 + for id, ext := range a.Application.ExternalSymbols { + if _, ok := byID[SymbolLabel+"\x00"+id]; len(ext.Body) > 0 && !ok { + t.Errorf("stand-in %s has no :GoExternal node", id) + } + for _, n := range ext.Body { + if from := owner[n.ID]; from.Value != id || from.Label != SymbolLabel { + t.Errorf("stand-in vertex %s: GO_HAS_BODY_NODE from %+v, want %s", n.ID, from, id) + } + if row := byID[LabelBodyNode+"\x00"+n.ID]; row.Props["var"] != n.Of || row.Module != "" { + t.Errorf("stand-in vertex %s row = %+v", n.ID, row) + } + checked++ + } + } + if checked == 0 { + t.Fatal("no stand-in vertex checked") + } + for _, e := range rows.Edges { + if e.Type != RelParamIn && e.Type != RelParamOut { + continue + } + if e.From.Label != LabelBodyNode || e.To.Label != LabelBodyNode || e.Props["var"] == nil { + t.Errorf("%s row %+v", e.Type, e) + } + } +} + +// TestProject_DegradedJSONOnApplication: the envelope's degraded[] rides +// :GoApplication as the JSON string prop degraded_json, absent when the +// run degraded nothing. +func TestProject_DegradedJSONOnApplication(t *testing.T) { + a := overlayFixture() + a.MaxLevel = 4 + a.Degraded = []v2.Degraded{ + {Code: "no_vta", Detail: "panic: x"}, + {Code: "range_func", Packages: []string{"example.com/p"}, Callables: []string{"can://go/x/a.go/f()"}}, + } + want, err := json.Marshal(a.Degraded) + if err != nil { + t.Fatal(err) + } + app := func(rows GraphRows) NodeRow { + for _, n := range rows.Nodes { + if n.Labels[0] == LabelApplication { + return n + } + } + t.Fatal("no :GoApplication row") + return NodeRow{} + } + if got := app(Project(a)).Props["degraded_json"]; got != string(want) { + t.Errorf("degraded_json = %v, want %s", got, want) + } + a.Degraded = nil + if got, has := app(Project(a)).Props["degraded_json"]; has { + t.Errorf("clean run carries degraded_json %v", got) + } + if _, has := app(Project(buildDataflowL4(t))).Props["degraded_json"]; has { + t.Errorf("dataflow (clean) carries degraded_json") + } +} diff --git a/internal/neo4j/schema.go b/internal/neo4j/schema.go index 6fae733..928e1b4 100644 --- a/internal/neo4j/schema.go +++ b/internal/neo4j/schema.go @@ -125,8 +125,9 @@ const ( RelCDG = "GO_CDG" RelDDG = "GO_DDG" - // Reserved L4 overlay — declared NOW, emits ZERO rows until L4 lands - // (CLAUDE.md). The projector fills these with no schema-contract change. + // L4 overlay (LIVE at -a 4): interprocedural edges between :GoBodyNode + // vertex ids (GO_PARAM_IN/OUT carry var) and call-site summary edges. + // Declared since L1 and filled with no schema-contract change. RelParamIn = "GO_PARAM_IN" RelParamOut = "GO_PARAM_OUT" RelSummary = "GO_SUMMARY" @@ -175,8 +176,8 @@ type RelSpec struct { // present in the graph: it is emitted via the RowBuilder's edge-only-when- // resolved path and kept at finish() only if the target was emitted. Deferred bool - // Reserved marks the L4 overlay edges: declared now, emits zero rows at - // the current max level (L3). + // Reserved marks an edge declared ahead of the level that fills it: it + // emits zero rows. None is Reserved today (the L4 overlay is live). Reserved bool } @@ -184,7 +185,7 @@ type RelSpec struct { var Nodes = []NodeSpec{ { Label: LabelApplication, MergeLabel: LabelApplication, Key: "id", CanNode: true, - Props: []string{"schema_version", "language", "max_level", "analyzer_name", "analyzer_version"}, + Props: []string{"schema_version", "language", "max_level", "analyzer_name", "analyzer_version", "degraded_json"}, }, { Label: LabelModule, MergeLabel: LabelModule, Key: "id", CanNode: true, @@ -207,11 +208,13 @@ var Nodes = []NodeSpec{ }, { Label: LabelBodyNode, MergeLabel: LabelBodyNode, Key: "id", CanNode: true, - // kind "call" at L1, statement|branch|loop|switch|return|entry|exit at L3; - // is_goroutine/is_deferred pruned when false. call_node = the enclosing - // statement's LOCAL key (JSON `parent` verbatim, resolved against the - // owning callable; Python parity); var = JSON `of` (L4 — absent at L3). - // span_json absent on the synthetic entry/exit nodes. + // kind "call" at L1, statement|branch|loop|switch|return|entry|exit at L3, + // formal_in|formal_out|actual_in|actual_out at L4; is_goroutine / + // is_deferred pruned when false. call_node = the enclosing statement's + // (an actual vertex's: its call's) LOCAL key — JSON `parent` verbatim, + // resolved against the owning callable or stand-in; Python parity. + // var = JSON `of` (L4 vertices only). span_json absent on the + // synthetic entry/exit nodes and on vertices. Props: []string{"kind", "is_goroutine", "is_deferred", "span_json", "call_node", "var"}, }, { @@ -268,8 +271,8 @@ var Rels = []RelSpec{ {Type: RelCDG, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}}, {Type: RelDDG, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Props: []string{"var", "prov"}, Discriminant: "var|prov"}, - // Reserved L4 overlay (zero rows until L4 lands). - {Type: RelParamIn, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Reserved: true}, - {Type: RelParamOut, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Reserved: true}, - {Type: RelSummary, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Reserved: true}, + // L4 overlay (live at -a 4). + {Type: RelParamIn, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Props: []string{"var"}}, + {Type: RelParamOut, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}, Props: []string{"var"}}, + {Type: RelSummary, From: []string{LabelBodyNode}, To: []string{LabelBodyNode}}, } diff --git a/internal/neo4j/schema_test.go b/internal/neo4j/schema_test.go index 212cd61..4069038 100644 --- a/internal/neo4j/schema_test.go +++ b/internal/neo4j/schema_test.go @@ -138,6 +138,46 @@ func TestSchemaDocument_RoundTrips(t *testing.T) { func TestConformance_ProjectorEmitsOnlyDeclared(t *testing.T) { t.Run("L2", func(t *testing.T) { checkConformance(t, Project(buildMultipackageL2(t))) }) t.Run("L3", func(t *testing.T) { checkConformance(t, Project(buildDataflowL3(t))) }) + t.Run("L4", func(t *testing.T) { checkConformance(t, Project(buildDataflowL4(t))) }) +} + +// unproducedRels are the declared relationships whose facts the Go analyzer +// does not produce yet, so no fixture can emit a row of them (declared +// contract, empty projection — CLAUDE.md § Repository-artifact layer): +// the neutral repository layer and its two GO_ resolution claims, and the +// inheritance pair (base_types[] are type strings, not type ids, so the +// deferred GO_EMBEDS never resolves; interfaces[] is not computed). +var unproducedRels = map[string]bool{ + RelHasArtifact: true, RelDeclaresDependency: true, RelLocks: true, RelDefinesConfig: true, + RelProvides: true, RelUnresolvedImport: true, + RelEmbeds: true, RelSatisfies: true, +} + +// TestConformance_EveryDeclaredRelEmitted: the -a 4 projection of the +// dataflow fixture emits at least one row of every declared relationship +// the analyzer produces facts for — the L4 overlay (GO_PARAM_IN, +// GO_PARAM_OUT, GO_SUMMARY) included — and none of a Reserved one. +func TestConformance_EveryDeclaredRelEmitted(t *testing.T) { + rows := Project(buildDataflowL4(t)) + for _, r := range Rels { + n := countEdges(rows, r.Type) + switch { + case r.Reserved && n > 0: + t.Errorf("%s is Reserved but has %d rows", r.Type, n) + case !r.Reserved && !unproducedRels[r.Type] && n == 0: + t.Errorf("%s is declared but the fixture emits no row of it", r.Type) + } + } +} + +// TestContract_L4OverlayNotReserved: the L4 overlay is live — no declared +// relationship is Reserved any more. +func TestContract_L4OverlayNotReserved(t *testing.T) { + for _, r := range Rels { + if r.Reserved { + t.Errorf("%s is still Reserved", r.Type) + } + } } func checkConformance(t *testing.T, rows GraphRows) { @@ -218,7 +258,7 @@ func checkConformance(t *testing.T, rows GraphRows) { func TestContract_L3OverlayNotReserved(t *testing.T) { want := map[string]bool{ RelCFGNext: false, RelCDG: false, RelDDG: false, - RelParamIn: true, RelParamOut: true, RelSummary: true, + RelParamIn: false, RelParamOut: false, RelSummary: false, } for _, r := range Rels { if res, ok := want[r.Type]; ok { @@ -252,3 +292,34 @@ func TestContract_L3OverlayNotReserved(t *testing.T) { } } } + +// TestContract_L4RelsCarryVar: the L4 param relationships declare the `var` +// property they carry; :GoApplication declares degraded_json for the +// degradation marker. +func TestContract_L4RelsCarryVar(t *testing.T) { + seen := map[string]bool{} + for _, r := range Rels { + switch r.Type { + case RelParamIn, RelParamOut: + seen[r.Type] = true + if len(r.Props) != 1 || r.Props[0] != "var" { + t.Errorf("%s Props = %v, want [var]", r.Type, r.Props) + } + } + } + if len(seen) != 2 { + t.Errorf("param relationships declared: %v", seen) + } + for _, n := range Nodes { + if n.Label != LabelApplication { + continue + } + found := false + for _, p := range n.Props { + found = found || p == "degraded_json" + } + if !found { + t.Errorf(":GoApplication props %v lack degraded_json", n.Props) + } + } +} diff --git a/internal/options/options.go b/internal/options/options.go index 1006679..01a1af8 100644 --- a/internal/options/options.go +++ b/internal/options/options.go @@ -11,6 +11,8 @@ const ( LevelCallGraph AnalysisLevel = 2 // LevelDataflow adds intraprocedural CFG/CDG/DDG on top of the call graph. LevelDataflow AnalysisLevel = 3 + // LevelSDG adds the interprocedural system dependence graph (L4) on top of L3. + LevelSDG AnalysisLevel = 4 ) // EmitTarget selects the analyzer's output projection (cli-contract.md § Neo4j). @@ -38,9 +40,14 @@ type AnalysisOptions struct { // AppName anchors application identity (can:// and the Neo4j // :Application node). Defaults to the input directory's base name. AppName string - // AnalysisLevel controls symbol-table-only (1), + call graph (2), or - // + intraprocedural dataflow (3). + // AnalysisLevel controls symbol-table-only (1), + call graph (2), + // + intraprocedural dataflow (3), or + the interprocedural SDG (4). Level AnalysisLevel + // Strict makes the run fail non-zero on any L4 degradation (--strict). + Strict bool + // LibraryModels is the --library-models JSON file overriding library + // stand-in summaries at L4 (sdg.LoadModels). Empty = the defaults. + LibraryModels string // AnalyzerVersion is the analyzer's own version, recorded in the v2 // manifest's analyzer{} tag. Set by the CLI from its build-stamped value. AnalyzerVersion string diff --git a/internal/schema/schema.go b/internal/schema/schema.go index 0161769..84335d5 100644 --- a/internal/schema/schema.go +++ b/internal/schema/schema.go @@ -5,6 +5,8 @@ // transformation. Design decisions are recorded in .claude/SCHEMA_DECISIONS.md. package schema +import "go/token" + // ─── Leaf models ───────────────────────────────────────────────────────────── // GoImport represents a single import declaration in a Go source file. @@ -92,6 +94,11 @@ type GoCallsite struct { StartColumn int `json:"start_column"` EndLine int `json:"end_line"` EndColumn int `json:"end_column"` + // Lparen is the call's raw CallExpr.Lparen in the builder's FileSet: the + // in-memory join key from an L1 site to its *ast.CallExpr (L3 body nodes, + // and go/ssa CallCommon.Pos() at L4). Never serialized; meaningless once + // the FileSet is gone. + Lparen token.Pos `json:"-"` } // ─── Callable ───────────────────────────────────────────────────────────────── @@ -99,7 +106,9 @@ type GoCallsite struct { // GoCallable represents a function, method, or function literal in Go. // receiver_type / receiver_name are non-empty for methods; empty for functions. type GoCallable struct { - Name string `json:"name"` + Name string `json:"name"` + // Path is the declaring file (a symbol_table key). A closure carries its + // enclosing callable's declaring file, so v1 inner_callables have a path. Path string `json:"path"` Signature string `json:"signature"` // signatureOf() output — edge id Comments []GoComment `json:"comments"` @@ -186,6 +195,13 @@ type GoCallEdge struct { // whose signature is already unique (the common case). Internal-only; not // part of the v2 output schema. SourcePath string `json:"-"` + + // TargetPath is the relative file declaring the target callable, set only + // when the signature alone may not name it: a VTA target can be a closure + // of a package-level init (".init.closure_1" repeats once per file + // with an init). Empty for L2 edges (a static call never targets an init + // or a closure) and for an external target. Internal-only, like SourcePath. + TargetPath string `json:"-"` } // ─── Root object ────────────────────────────────────────────────────────────── @@ -208,6 +224,10 @@ type GoExternalCallee struct { Name string `json:"name"` Params []GoParameter `json:"parameters"` ReturnType string `json:"return_type,omitempty"` + // Receiver is the method receiver (nil for a plain function): named by the + // receiver variable, or "_recv" when unnamed or "_"; typed by the origin's + // receiver type string. + Receiver *GoParameter `json:"receiver,omitempty"` } // GoEntrypoint marks a callable as a framework entry point. diff --git a/internal/schema/v2/schema.go b/internal/schema/v2/schema.go index 2293c09..8042056 100644 --- a/internal/schema/v2/schema.go +++ b/internal/schema/v2/schema.go @@ -6,9 +6,11 @@ // The model is ONE additive containment tree // (application → module → type → callable → body) with typed edge overlays // laid over it (a Code Property Graph). This package declares the L1 tree, the -// L2 refinement/edge slots and the L3 body completion (statement nodes, body -// node id/parent, the callable's cfg/cdg/ddg). L4 fields (param_*) are not -// declared here — they arrive in their own train and reuse these ids/spans +// L2 refinement/edge slots, the L3 body completion (statement nodes, body +// node id/parent, the callable's cfg/cdg/ddg) and the L4 fields (the envelope's +// degraded marker, body node `of`, a callable's summary, the application's +// param_in/param_out, an external symbol's receiver and body). The L4 fields +// are all omitempty and absent below -a 4; they reuse the L1-L3 ids/spans // unchanged. // // Conventions from the keystone, held here: @@ -46,8 +48,8 @@ type Span struct { // Analysis is the top-level payload written to analysis.json / stdout. Its // siblings of `application` carry the manifest; consumers read `max_level` -// rather than sniffing for keys. k_limit (the L3 field-path depth bound) is -// present only at max_level >= 3. +// rather than sniffing for keys. k_limit (the field-path depth bound of the +// L3 and L4 access paths) is present only at max_level >= 3. type Analysis struct { SchemaVersion string `json:"schema_version"` Language string `json:"language"` @@ -55,6 +57,19 @@ type Analysis struct { KLimit int `json:"k_limit,omitempty"` Analyzer AnalyzerTag `json:"analyzer"` Application Application `json:"application"` + // Degraded records the L4 degradation markers (spec §4.15); absent when + // the run degraded nothing and below -a 4. + Degraded []Degraded `json:"degraded,omitempty"` +} + +// Degraded is one degradation marker: an L4 step that fell back to a coarser +// result. Code is an open vocabulary of plain strings (spec §4.15); Detail, +// Packages and Callables say what was affected. +type Degraded struct { + Code string `json:"code"` + Detail string `json:"detail,omitempty"` + Packages []string `json:"packages,omitempty"` + Callables []string `json:"callables,omitempty"` } // AnalyzerTag identifies the producing analyzer and version. @@ -78,6 +93,19 @@ type Application struct { // reaches, keyed by their `can:///@external//` id. // Absent when the analysis saw no library call. ExternalSymbols map[string]ExternalSymbol `json:"external_symbols,omitempty"` + // ParamIn and ParamOut are the L4 interprocedural parameter edges between + // global body-node ids. Absent below -a 4. + ParamIn []ParamEdge `json:"param_in,omitempty"` + ParamOut []ParamEdge `json:"param_out,omitempty"` +} + +// ParamEdge is an L4 interprocedural parameter edge (call-site actual → +// callee formal for param_in, callee formal/return → call-site result for +// param_out) on the access path Var. +type ParamEdge struct { + Src string `json:"src"` + Dst string `json:"dst"` + Var string `json:"var"` } // ExternalSymbol is a library callable that call-graph edges and call nodes @@ -93,6 +121,12 @@ type ExternalSymbol struct { Signature string `json:"signature"` Parameters []ExternalParameter `json:"parameters"` ReturnType string `json:"return_type,omitempty"` + // Receiver is the method receiver (absent on a plain function): the name + // is the receiver variable, or _recv when unnamed or `_`; the type is the + // origin's receiver type string. + Receiver *ExternalParameter `json:"receiver,omitempty"` + // Body holds the L4 stand-in body nodes of a library callee (spec §6). + Body map[string]BodyNode `json:"body,omitempty"` } // ExternalParameter is a library callee's formal parameter. It has no span: the @@ -197,6 +231,9 @@ type Callable struct { Cfg []CfgEdge `json:"cfg,omitempty"` Cdg []CdgEdge `json:"cdg,omitempty"` Ddg []DdgEdge `json:"ddg,omitempty"` + // Summary is the L4 summary edge list (formal-in → formal-out through the + // callable); endpoints are local body keys. Absent below -a 4. + Summary []SummaryEdge `json:"summary,omitempty"` // Callables are nested closures / function literals, each with its own // can:// id (replaces v1 InnerCallables). Callables map[string]Callable `json:"callables,omitempty"` @@ -238,6 +275,16 @@ type BodyNode struct { IsGoroutine bool `json:"is_goroutine,omitempty"` // IsDeferred is true when the call is preceded by `defer` (net-new vs v1). IsDeferred bool `json:"is_deferred,omitempty"` + // Of names the value an L4 vertex carries (, :, a + // parameter name, :, :::). Absent below + // -a 4 and on L1-L3 nodes. + Of string `json:"of,omitempty"` +} + +// SummaryEdge is an L4 summary edge between two local body keys of one callable. +type SummaryEdge struct { + Src string `json:"src"` + Dst string `json:"dst"` } // MarshalJSON writes the callee key on call nodes only: it is the call diff --git a/internal/sdg/alias.go b/internal/sdg/alias.go new file mode 100644 index 0000000..b905363 --- /dev/null +++ b/internal/sdg/alias.go @@ -0,0 +1,444 @@ +package sdg + +import ( + "go/ast" + "go/token" + "go/types" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// ProvPointsTo is the prov of every may-alias ddg edge (spec §4.10): the L4 +// semantic delta over L3's "ssa" edges and the "reaching-defs" port wiring. +const ProvPointsTo = "points-to" + +// AliasOracle answers may-alias queries between two access paths of one +// callable (spec §4.10): MayAlias(d, u) reports whether a write to d may +// change memory a read of u observes, d and u having different bases. The +// interface is frozen so that a points-to analysis (an Andersen solve over +// go/ssa) can replace the type-based MVP without a schema change; prov stays +// "points-to". +// +// The DDG solver (dataflow.BuildDDGAlias) consults it only within its +// one-step offset window, so an oracle need not accept pairs outside it. +type AliasOracle interface { + MayAlias(a, b dataflow.Path) bool +} + +// newAliasOracle builds the oracle of unit u; root is the top-level +// declaration enclosing u.Fn (u.Fn itself for a declaration, the outermost +// enclosing function for a closure), which typeOracle scans for taken +// addresses. nil turns the alias pass off. A test seam: TestAlias_AddsOnly +// compares Run with it off and on. +var newAliasOracle func(u dataflow.Unit, root ast.Node) AliasOracle = newTypeOracle + +// typeOracle is the type-based may-alias MVP of one callable (spec §4.10, +// rulings of Task 17 and its fix round 1). +// +// A path is matched through its anchors: a prefix of it, at offset 0 (the +// base) or 1 (the base and its first step), that names one object, with the +// steps after the prefix (rem) addressing memory inside that object. An +// anchor is +// +// - a reference anchor: the prefix has a reference type (pointer, slice, +// map, chan, func, interface or type parameter); the object is what it +// points to. A write reaches the object only through a step after the +// prefix: a bare def `p = …` (or `x.f = …` at offset 1) rebinds the +// reference and writes nothing another base observes; +// - a storage anchor: the prefix is an addressable variable whose address +// may be taken (taken below); the object is its own storage, so a bare +// def `x = …` overwrites it. A reference-typed variable whose address is +// taken has both anchors. +// +// At offset 1 the first step is a field (".f") or element ("[*]") whose +// declared type decides the anchor: a reference-typed field or element is a +// reference anchor (it may share its target with another base: `m := +// s.cache; m[k] = v` reaches `s.cache[k]`); a field or element whose address +// is taken is a storage anchor. A path that ends at the base, or whose first +// step is the ".*" wildcard, stands for every first step, so every field and +// element of the base's type is tried. +// +// MayAlias(d, u) holds iff a writing anchor of d and an anchor of u may be +// the same object and their rems are prefix-compatible +// (dataflow.StepsCompatible). Two anchors may be the same object iff +// +// - both are references with compatible types (I1, I2): any two interface +// (or type-parameter) types; an interface and a type that implements it +// (a type parameter: that satisfies its constraint); else, on the +// underlying types, pointers with identical elements, slices with +// identical elements, a pointer to an array [N]E and a slice []E, maps +// with identical keys and elements, chans with identical elements +// whatever their directions, identical signatures; +// - a reference and a storage of type T (I3): a pointer to T, an +// interface *T implements, or (T an array [N]E) a slice []E; +// - never two storages: distinct variables never share storage. +// +// Taken addresses are scanned over root, closures included (an &x on a +// captured x may sit in the encloser): &x, &x.f…, &x[i]… and x[:], x[i:j] +// on an array value make x and (x, its first step) taken; so does a call or +// method value of a pointer-receiver method on an addressable value x or +// x.f… (the callee may keep the pointer). A package-level variable, and +// each of its fields and elements, is always taken: &G may sit in any +// function. +// +// A nil base (a ":" pseudo path, an unresolved identifier) never +// aliases. Globals are bases like any other; so are a closure's captures. +// +// Known gaps, all false negatives: offsets deeper than one step (&x.f.g +// against a *G; q := x.f.g against x.f.g.h); two paths of one base whose +// prefixes alias (x.f and x.g pointing to one object: the solver only asks +// about different bases); and an interface whose dynamic value is a value +// struct holding a reference (i.* written through the interface against the +// struct's reference field). +type typeOracle struct { + taken map[types.Object]bool + takenSub map[subKey]bool + anchors map[dataflow.Path][]anchor + memo map[[2]dataflow.Path]bool +} + +type subKey struct { + base types.Object + step string +} + +type anchor struct { + ref bool // a reference anchor; else a storage anchor + t types.Type // the reference's type, or the storage's + rem []string // the steps after the prefix (nil: every one) + write bool // a def of the path writes inside the object +} + +func newTypeOracle(u dataflow.Unit, root ast.Node) AliasOracle { + o := &typeOracle{ + taken: map[types.Object]bool{}, + takenSub: map[subKey]bool{}, + anchors: map[dataflow.Path][]anchor{}, + memo: map[[2]dataflow.Path]bool{}, + } + if root != nil && u.Info != nil { + o.scan(root, u.Info) + } + return o +} + +// scan records the addresses root takes (typeOracle's doc). +func (o *typeOracle) scan(root ast.Node, info *types.Info) { + ast.Inspect(root, func(n ast.Node) bool { + switch e := n.(type) { + case *ast.UnaryExpr: + if e.Op == token.AND { + o.take(e.X, info) + } + case *ast.SliceExpr: + if isArray(info.TypeOf(e.X)) { + o.take(e.X, info) + } + case *ast.SelectorExpr: + if ptrMethodOnValue(e, info) { + o.take(e.X, info) + } + } + return true + }) +} + +// take records x (an addressed operand) and its base's first step. +func (o *typeOracle) take(x ast.Expr, info *types.Info) { + p, ok := dataflow.PathOf(x, info, 2) + if !ok || p.Base == nil { + return + } + o.taken[p.Base] = true + if s := dataflow.PathSteps(p.Spell); len(s) > 0 && s[0] != ".*" { + o.takenSub[subKey{p.Base, s[0]}] = true + } +} + +// ptrMethodOnValue reports whether sel is a pointer-receiver method (call or +// value) selected on a non-pointer, non-interface operand: Go takes its +// address implicitly. +func ptrMethodOnValue(sel *ast.SelectorExpr, info *types.Info) bool { + s := info.Selections[sel] + if s == nil || s.Kind() != types.MethodVal || s.Indirect() { + return false + } + sig, ok := s.Obj().Type().(*types.Signature) + if !ok || sig.Recv() == nil { + return false + } + if _, ptr := sig.Recv().Type().Underlying().(*types.Pointer); !ptr { + return false + } + t := info.TypeOf(sel.X) + if t == nil || types.IsInterface(t) { + return false + } + _, ptr := t.Underlying().(*types.Pointer) + return !ptr +} + +func isArray(t types.Type) bool { + if t == nil { + return false + } + _, ok := t.Underlying().(*types.Array) + return ok +} + +func (o *typeOracle) MayAlias(d, u dataflow.Path) bool { + if d.Base == nil || u.Base == nil { + return false + } + if d.Base == u.Base { + return true + } + k := [2]dataflow.Path{d, u} + if v, ok := o.memo[k]; ok { + return v + } + v := o.mayAlias(d, u) + o.memo[k] = v + return v +} + +func (o *typeOracle) mayAlias(d, u dataflow.Path) bool { + ua := o.anchorsOf(u) + for _, a := range o.anchorsOf(d) { + if !a.write { + continue + } + for _, b := range ua { + if sameObject(a, b) && dataflow.StepsCompatible(a.rem, b.rem) { + return true + } + } + } + return false +} + +// anchorsOf returns p's anchors (typeOracle's doc), memoized. +func (o *typeOracle) anchorsOf(p dataflow.Path) []anchor { + if as, ok := o.anchors[p]; ok { + return as + } + var as []anchor + bt := p.Base.Type() + steps := dataflow.PathSteps(p.Spell) + global := dataflow.IsPkgVar(p.Base) + baseRef := refType(bt) + // A reference-typed base whose address is taken has both anchors: what + // it points to, and its own storage (`pp := &p; *pp = b` rewrites p, so + // p.n then reads b.n: fix round 2, N2). + if baseRef { + as = append(as, anchor{ref: true, t: bt, rem: steps, write: len(steps) > 0}) + } + if global || o.taken[p.Base] { + as = append(as, anchor{t: bt, rem: steps, write: true}) + } + var firsts []string + var rem []string + if len(steps) > 0 && steps[0] != ".*" { + firsts, rem = steps[:1], steps[1:] + } else { + firsts = childSteps(bt) // every first step; rem nil + } + for _, s := range firsts { + ft := childType(bt, s) + if ft == nil { + continue + } + if refType(ft) { + // A write through the field's reference needs a step past it: one + // more listed step, or a wildcard standing for it. + w := len(steps) >= 2 || len(steps) == 1 && steps[0] == ".*" + as = append(as, anchor{ref: true, t: ft, rem: rem, write: w}) + } + if global || o.takenSub[subKey{p.Base, s}] { + // The field's own storage: written by any def reaching it — not by + // a bare def of a pointer base, which only rebinds the pointer. + w := len(steps) > 0 || !baseRef + as = append(as, anchor{t: ft, rem: rem, write: w}) + } + } + o.anchors[p] = as + return as +} + +// childSteps returns every first step a path on a base of type t may take: +// its struct fields (through a pointer) by name, and "[*]" for a container. +func childSteps(t types.Type) []string { + t = deref(t) + var out []string + switch u := t.Underlying().(type) { + case *types.Struct: + for i := 0; i < u.NumFields(); i++ { + out = append(out, "."+u.Field(i).Name()) + } + case *types.Slice, *types.Array, *types.Map, *types.Chan: + out = append(out, "[*]") + } + return out +} + +// childType is the declared type of step s below a base of type t (a field +// through a pointer, an element of a slice, array, array pointer, map or +// chan), nil when there is none. +func childType(t types.Type, s string) types.Type { + t = deref(t) + switch u := t.Underlying().(type) { + case *types.Struct: + for i := 0; i < u.NumFields(); i++ { + if "."+u.Field(i).Name() == s { + return u.Field(i).Type() + } + } + case *types.Slice: + if s == "[*]" { + return u.Elem() + } + case *types.Array: + if s == "[*]" { + return u.Elem() + } + case *types.Map: + if s == "[*]" { + return u.Elem() + } + case *types.Chan: + if s == "[*]" { + return u.Elem() + } + } + return nil +} + +func deref(t types.Type) types.Type { + if p, ok := t.Underlying().(*types.Pointer); ok { + return p.Elem() + } + return t +} + +// refType reports whether t is a reference kind: pointer, slice, map, chan, +// func, interface or type parameter. +func refType(t types.Type) bool { + switch t.Underlying().(type) { + case *types.Pointer, *types.Slice, *types.Map, *types.Chan, *types.Signature, *types.Interface: + return true + } + return false +} + +func sameObject(a, b anchor) bool { + switch { + case a.ref && b.ref: + return refsMayAlias(a.t, b.t) + case a.ref: + return refToStorage(a.t, b.t) + case b.ref: + return refToStorage(b.t, a.t) + } + return false +} + +// refsMayAlias reports whether two references of types a and b may point to +// one object (I1, I2: typeOracle's doc). +func refsMayAlias(a, b types.Type) bool { + ia, ib := types.IsInterface(a), types.IsInterface(b) + switch { + case ia && ib: + return true + case ia: + return implementsIface(b, a) + case ib: + return implementsIface(a, b) + } + if arrayPtrSlice(a, b) || arrayPtrSlice(b, a) { + return true + } + switch ua := a.Underlying().(type) { + case *types.Pointer: + ub, ok := b.Underlying().(*types.Pointer) + return ok && types.Identical(ua.Elem(), ub.Elem()) + case *types.Slice: + ub, ok := b.Underlying().(*types.Slice) + return ok && types.Identical(ua.Elem(), ub.Elem()) + case *types.Map: + ub, ok := b.Underlying().(*types.Map) + return ok && types.Identical(ua.Key(), ub.Key()) && types.Identical(ua.Elem(), ub.Elem()) + case *types.Chan: + ub, ok := b.Underlying().(*types.Chan) + return ok && types.Identical(ua.Elem(), ub.Elem()) + case *types.Signature: + return types.Identical(ua, b.Underlying()) + } + return false +} + +// arrayPtrSlice reports whether a is a pointer to an array [N]E and b a +// slice []E: pa[:] shares the array (fix round 2, O1). +func arrayPtrSlice(a, b types.Type) bool { + pa, ok := a.Underlying().(*types.Pointer) + if !ok { + return false + } + arr, ok := pa.Elem().Underlying().(*types.Array) + if !ok { + return false + } + sl, ok := b.Underlying().(*types.Slice) + return ok && types.Identical(arr.Elem(), sl.Elem()) +} + +// refToStorage reports whether a reference of type r may point to the +// storage of a value of type t (I3: typeOracle's doc). +func refToStorage(r, t types.Type) bool { + if types.IsInterface(r) { + return implementsIface(types.NewPointer(t), r) + } + switch ur := r.Underlying().(type) { + case *types.Pointer: + return types.Identical(ur.Elem(), t) + case *types.Slice: + at, ok := t.Underlying().(*types.Array) + return ok && types.Identical(ur.Elem(), at.Elem()) + } + return false +} + +// implementsIface reports whether a value of type t may be held by a value +// of interface type iface: t implements it, or (iface a type parameter) +// satisfies its constraint. +func implementsIface(t, iface types.Type) bool { + if tp, ok := iface.(*types.TypeParam); ok { + c, ok := tp.Constraint().Underlying().(*types.Interface) + return ok && types.Satisfies(t, c) + } + it, ok := iface.Underlying().(*types.Interface) + return ok && types.Implements(t, it) +} + +// rootOf returns the top-level declaration enclosing unit u's function: u.Fn +// for a declaration, else the outermost unit function of u's file whose +// extent contains u.Fn (a package-level literal is its own root). +func (p *ports) rootOf(u dataflow.Unit) ast.Node { + if _, ok := u.Fn.(*ast.FuncDecl); ok || u.Fn == nil { + return u.Fn + } + if p.fnsByPath == nil { + p.fnsByPath = map[string][]ast.Node{} + for _, v := range p.in.Units { + if v.Fn != nil { + p.fnsByPath[v.Key.Path] = append(p.fnsByPath[v.Key.Path], v.Fn) + } + } + } + root := u.Fn + for _, fn := range p.fnsByPath[u.Key.Path] { + if fn.Pos() <= root.Pos() && root.End() <= fn.End() && fn.Pos() < root.Pos() { + root = fn + } + } + return root +} diff --git a/internal/sdg/alias_test.go b/internal/sdg/alias_test.go new file mode 100644 index 0000000..fb40112 --- /dev/null +++ b/internal/sdg/alias_test.go @@ -0,0 +1,362 @@ +package sdg + +import ( + "flag" + "fmt" + "go/ast" + "go/token" + "go/types" + "os" + "path/filepath" + "reflect" + "slices" + "strings" + "testing" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +var update = flag.Bool("update", false, "rewrite golden files") + +func pt(src, dst, v string) PortEdge { + return PortEdge{Src: src, Dst: dst, Var: v, Prov: ProvPointsTo} +} + +func pointsTo(es []PortEdge) []PortEdge { + var out []PortEdge + for _, e := range es { + if e.Prov == ProvPointsTo { + out = append(out, e) + } + } + return out +} + +// runAlias runs L4 over testdata/dataflow with the alias pass on or off. +func runAlias(t *testing.T, on bool) Result { + t.Helper() + defer func(v func(dataflow.Unit, ast.Node) AliasOracle) { newAliasOracle = v }(newAliasOracle) + if !on { + newAliasOracle = nil + } + return Run(loadInput(t, "dataflow")) +} + +// TestAlias_PointerCopyLinks: in AliasPtr (p := req; p.UserID = id; return +// req.UserID) the write through the copy p reaches the read through req only +// by may-alias, as a points-to edge 360:2 → 361:2 on req.UserID; the same +// write reaches req's by-ref formal_out (var its `of`, as wire maps an +// @exit edge). Those are AliasPtr's only points-to edges. +// +// AliasPtrWhole returns the bare req (ruling, Task 17: the use side is +// unrestricted): the write p.UserID is prefix-compatible with the whole read. +// The fix-round-1 fixtures (address-taken values, reference fields copied +// out, unrelated interfaces) are pinned whole too. +func TestAlias_PointerCopyLinks(t *testing.T) { + res := runAlias(t, true) + for _, c := range []struct { + fn string + want []PortEdge + }{ + {"AliasPtr", []PortEdge{pt("360:2", "361:2", "req.UserID"), pt("360:2", "@formal_out:1", "req")}}, + {"AliasPtrWhole", []PortEdge{pt("368:2", "369:2", "req"), pt("368:2", "@formal_out:1", "req")}}, + // aliasAssign (q := p; q.n = v): q.n = v reaches p's formal_out. + {"aliasAssign", []PortEdge{pt("153:2", "@formal_out", "p")}}, + // Fix round 1, I3: address-taken values. The value's own def + // (`var req Req`, storage) reaches the use p of `p.UserID = id`: p + // points into it. + {"AliasPtrValue", []PortEdge{pt("388:2", "390:2", "p"), pt("390:2", "391:2", "req.UserID")}}, + {"AliasArraySlice", []PortEdge{pt("396:2", "398:2", "s"), pt("398:2", "399:2", "arr[*]")}}, + {"AliasFieldAddr", []PortEdge{pt("405:2", "407:2", "p"), pt("407:2", "408:2", "o.Inner.n")}}, + // Option A: reference-typed fields copied out (one-step offset). + {"AliasRefField", []PortEdge{pt("424:2", "425:2", "s.m[*]"), pt("424:2", "@formal_out:1", "s")}}, + {"AliasPtrField", []PortEdge{pt("436:2", "437:2", "r.ptr.n"), pt("436:2", "@formal_out:1", "r")}}, + // I2: an io.Writer and an io.Reader holding one *bytes.Buffer. + {"AliasIfaces", []PortEdge{pt("445:2", "446:2", "r"), pt("445:2", "@formal_out:1", "f"), pt("446:2", "@formal_out:1", "f")}}, + // Fix round 2, N1: alias edges from @entry whose def is another base + // than the use's start at the def's formal_in, though @entry → the + // use is also the use base's own edge: req (a value param whose + // address is taken) and the value global GReq (&GReq in takeGReq) + // reach q.UserID. + {"AliasEntryValue", []PortEdge{pt("@formal_in:0", "457:2", "q.UserID"), pt("@formal_in:0", "@formal_out:1", "q")}}, + {"AliasEntryGlobal", []PortEdge{pt("@formal_in:1", "469:2", "q.UserID"), pt("@formal_in:1", "@formal_out:1", "q")}}, + // N2: an address-taken reference has its storage anchor too. + {"AliasPtrPtr", []PortEdge{pt("474:2", "476:2", "pp"), pt("476:2", "477:2", "p.n")}}, + {"AliasSlicePtr", []PortEdge{pt("483:2", "484:2", "s[*]"), pt("483:2", "@formal_out:1", "s"), pt("@formal_in:0", "483:2", "ps")}}, + // O1: a slice of an array pointer aliases it. + {"AliasArrPtr", []PortEdge{pt("490:2", "491:2", "pa[*]"), pt("490:2", "@formal_out:1", "pa")}}, + } { + if got := pointsTo(res.Callables[sdgKey(c.fn)].Ddg); !slices.Equal(got, c.want) { + t.Errorf("%s points-to edges:\n got %+v\n want %+v", c.fn, got, c.want) + } + } +} + +// TestAlias_ValueCopiesNeverAlias: AliasValue copies req by value (no +// reference inside Req) and AliasValueCopies copies s twice, no address +// taken: no points-to edge. The oracle table pins the type rules (typeOracle's +// doc; fix round 1, I1–I3). +func TestAlias_ValueCopiesNeverAlias(t *testing.T) { + res := runAlias(t, true) + for _, fn := range []string{"AliasValue", "AliasValueCopies"} { + if got := pointsTo(res.Callables[sdgKey(fn)].Ddg); len(got) != 0 { + t.Errorf("%s has points-to edges %+v", fn, got) + } + } + + pkg := types.NewPackage("example.com/p", "p") + newNamed := func(name string, u types.Type) *types.Named { + return types.NewNamed(types.NewTypeName(token.NoPos, pkg, name, nil), u, nil) + } + field := func(name string, t types.Type) *types.Var { return types.NewField(token.NoPos, pkg, name, t, false) } + S := newNamed("S", types.NewStruct([]*types.Var{field("f", types.Typ[types.Int])}, nil)) + S.AddMethod(types.NewFunc(token.NoPos, pkg, "M", types.NewSignatureType(types.NewVar(token.NoPos, pkg, "s", types.NewPointer(S)), nil, nil, nil, nil, false))) + method := func(name string) *types.Interface { + return types.NewInterfaceType([]*types.Func{types.NewFunc(token.NoPos, pkg, name, types.NewSignatureType(nil, nil, nil, nil, nil, false))}, nil).Complete() + } + I, J := newNamed("I", method("M")), newNamed("J", method("N")) + Inner := newNamed("Inner", types.NewStruct([]*types.Var{field("n", types.Typ[types.Int])}, nil)) + Outer := newNamed("Outer", types.NewStruct([]*types.Var{types.NewField(token.NoPos, pkg, "Inner", Inner, true)}, nil)) + bytesT := types.NewSlice(types.Typ[types.Byte]) + Bytes := newNamed("Bytes", bytesT) + header := types.NewMap(types.Typ[types.String], types.NewSlice(types.Typ[types.String])) + Header := newNamed("Header", header) + ch := func(d types.ChanDir) types.Type { return types.NewChan(d, types.Typ[types.Int]) } + fn := types.NewSignatureType(nil, nil, nil, types.NewTuple(types.NewVar(token.NoPos, pkg, "", types.Typ[types.Int])), nil, false) + arr := types.NewArray(types.Typ[types.Int], 4) + + o := newTypeOracle(dataflow.Unit{}, nil).(*typeOracle) + vars := map[string]*types.Var{} + P := func(name string, t types.Type, spell string) dataflow.Path { + v, ok := vars[name] + if !ok { + v = types.NewVar(token.NoPos, pkg, name, t) + vars[name] = v + } + return dataflow.Path{Base: v, Spell: spell} + } + gv := types.NewVar(token.NoPos, pkg, "G", types.NewPointer(S)) + pkg.Scope().Insert(gv) + gs := types.NewVar(token.NoPos, pkg, "GS", S) + pkg.Scope().Insert(gs) + taken := P("x", S, "x.f") + o.taken[taken.Base] = true + tarr := P("arr", arr, "arr[*]") + o.taken[tarr.Base] = true + o.takenSub[subKey{tarr.Base, "[*]"}] = true + o.taken[P("tp", types.NewPointer(S), "tp").Base] = true + to := P("o", Outer, "o.Inner.n") + o.taken[to.Base] = true + o.takenSub[subKey{to.Base, ".Inner"}] = true + + for _, c := range []struct { + what string + d, u dataflow.Path + want bool + }{ + {"value structs", P("a1", S, "a1.f"), P("b1", S, "b1.f"), false}, + {"ints", P("a2", types.Typ[types.Int], "a2"), P("b2", types.Typ[types.Int], "b2"), false}, + {"identical pointers", P("a3", types.NewPointer(S), "a3.f"), P("b3", types.NewPointer(S), "b3.f"), true}, + {"bare def of a pointer rebinds it", P("a3", nil, "a3"), P("b3", nil, "b3.f"), false}, + {"pointer, untaken value", P("a3", nil, "a3.f"), P("b4", S, "b4.f"), false}, + {"different pointees", P("a3", nil, "a3.f"), P("b5", types.NewPointer(types.Typ[types.Int]), "b5.*"), false}, + {"*S implements I", P("a3", nil, "a3.f"), P("i1", I, "i1.*"), true}, + {"I, *S", P("i1", nil, "i1.*"), P("a3", nil, "a3.f"), true}, + {"value S, I (only *S implements)", P("a1", nil, "a1.f"), P("i1", nil, "i1.*"), false}, + {"*S does not implement J", P("a3", nil, "a3.f"), P("j1", J, "j1.*"), false}, + {"I2: two unrelated interfaces", P("i1", nil, "i1.*"), P("j1", nil, "j1.*"), true}, + {"I1: Bytes vs []byte", P("s1", Bytes, "s1[*]"), P("s2", bytesT, "s2[*]"), true}, + {"I1: Header vs map[string][]string", P("h1", Header, "h1[*]"), P("h2", header, "h2[*]"), true}, + {"I1: chan int vs <-chan int", P("c1", ch(types.SendRecv), "c1[*]"), P("c2", ch(types.RecvOnly), "c2[*]"), true}, + {"I1: chan<- int vs chan int", P("c3", ch(types.SendOnly), "c3[*]"), P("c1", nil, "c1[*]"), true}, + {"I1: []int vs []string", P("s3", types.NewSlice(types.Typ[types.Int]), "s3[*]"), P("s4", types.NewSlice(types.Typ[types.String]), "s4[*]"), false}, + {"I1: identical funcs", P("f1", fn, "f1.*"), P("f2", fn, "f2.*"), true}, + {"O1: *[4]int vs []int", P("pa", types.NewPointer(arr), "pa[*]"), P("s3", nil, "s3[*]"), true}, + {"O1: []int vs *[4]int", P("s3", nil, "s3[*]"), P("pa", nil, "pa[*]"), true}, + {"O1: *[4]int vs []string", P("pa", nil, "pa[*]"), P("s4", nil, "s4[*]"), false}, + {"N2: taken pointer's storage, bare def", P("tp", types.NewPointer(S), "tp"), P("pp", types.NewPointer(types.NewPointer(S)), "pp.*"), true}, + {"global and local pointer", dataflow.Path{Base: gv, Spell: "example.com/p::G.f"}, P("a3", nil, "a3.f"), true}, + {"I3: *S write, taken S read", P("a3", nil, "a3.f"), taken, true}, + {"I3: taken S write, *S read", P("x", nil, "x.f"), P("a3", nil, "a3.f"), true}, + {"I3: taken S bare def, *S read", P("x", nil, "x"), P("a3", nil, "a3.f"), true}, + {"I3: global value is always taken", P("a3", nil, "a3.f"), dataflow.Path{Base: gs, Spell: "example.com/p::GS.f"}, true}, + {"I3: two taken values never share storage", P("x", nil, "x.f"), dataflow.Path{Base: gs, Spell: "example.com/p::GS.f"}, false}, + {"I3: sliced array vs []int", P("s3", nil, "s3[*]"), tarr, true}, + {"I3: &o.Inner offset, p.n vs o.Inner.n", P("p1", types.NewPointer(Inner), "p1.n"), to, true}, + {"I3: &o.Inner offset, reverse", to, P("p1", nil, "p1.n"), true}, + {"I3: *Inner vs untaken Outer field", P("p1", nil, "p1.n"), P("o2", Outer, "o2.Inner.n"), false}, + {"ref field: m vs s.m", P("m1", Header, "m1[*]"), P("hs", newNamed("HS", types.NewStruct([]*types.Var{field("m", header)}, nil)), "hs.m[*]"), true}, + {"ref field: bare read of the struct", P("m1", nil, "m1[*]"), P("hs", nil, "hs"), true}, + {"ref field: field rebind writes no target", P("hs", nil, "hs.m"), P("m1", nil, "m1[*]"), false}, + {"nil base", dataflow.Path{Spell: "u.f"}, P("a3", nil, "a3.f"), false}, + {"pseudo path", dataflow.Path{Spell: ":9:9"}, P("a3", nil, "a3.f"), false}, + } { + if got := o.MayAlias(c.d, c.u); got != c.want { + t.Errorf("%s: MayAlias(%s, %s) = %v, want %v", c.what, c.d.Spell, c.u.Spell, got, c.want) + } + } +} + +// TestAlias_AddsOnly: the alias pass only adds. With the oracle off vs on, +// Run's Result is identical except that (rulings, Task 17) the on run's +// callable ddg holds the points-to edges besides the off run's edges, which +// it keeps unchanged, and its summary edges are a superset (the flows read +// the points-to edges, spec §4.9). No points-to edge repeats the (src, dst, +// var) of an L3 or reaching-defs edge, every endpoint is a body node or a +// vertex of the callable, no summary edge is global → global (#30), and +// skipped units get none. +func TestAlias_AddsOnly(t *testing.T) { + in := loadInput(t, "dataflow") + defer func(v func(dataflow.Unit, ast.Node) AliasOracle) { newAliasOracle = v }(newAliasOracle) + newAliasOracle = nil + off := Run(in) + newAliasOracle = newTypeOracle + on := Run(loadInput(t, "dataflow")) + + var nPT, nSumm int + for k, c := range on.Callables { + o := off.Callables[k] + var rest []PortEdge + for _, e := range c.Ddg { + if e.Prov != ProvPointsTo { + rest = append(rest, e) + } + } + if !slices.Equal(rest, o.Ddg) { + t.Errorf("%s: non-points-to ddg changed:\n off %+v\n on %+v", k.Signature, o.Ddg, rest) + } + for _, e := range o.Summary { + if !slices.Contains(c.Summary, e) { + t.Errorf("%s: summary edge %+v lost", k.Signature, e) + } + } + nSumm += len(c.Summary) - len(o.Summary) + if !reflect.DeepEqual(c.Vertices, o.Vertices) { + t.Errorf("%s: vertices changed", k.Signature) + } + of := map[string]string{} + for _, v := range c.Vertices { + of[v.Key] = v.Of + } + for _, e := range c.Summary { + if strings.HasPrefix(of[e.Src], ofGlobal) && strings.HasPrefix(of[e.Dst], ofGlobal) { + t.Errorf("%s: global → global summary %+v", k.Signature, e) + } + } + ends := map[string]bool{} + l3 := map[dataflow.DdgEdge]bool{} + if r, ok := in.L3[k]; ok { + for _, n := range r.Nodes { + ends[n.Key] = true + } + for _, e := range r.Ddg { + l3[e] = true + } + } + for _, v := range c.Vertices { + ends[v.Key] = true + } + rd := map[dataflow.DdgEdge]bool{} + for _, e := range rest { + rd[dataflow.DdgEdge{Src: e.Src, Dst: e.Dst, Var: e.Var}] = true + } + for _, e := range pointsTo(c.Ddg) { + nPT++ + t3 := dataflow.DdgEdge{Src: e.Src, Dst: e.Dst, Var: e.Var} + if l3[t3] || rd[t3] { + t.Errorf("%s: points-to %+v repeats an L3 or reaching-defs edge", k.Signature, e) + } + if !ends[e.Src] || !ends[e.Dst] || e.Dst == dataflow.ExitKey || e.Src == dataflow.EntryKey { + t.Errorf("%s: points-to %+v dangles or touches @entry/@exit", k.Signature, e) + } + } + for i := 1; i < len(c.Ddg); i++ { + if comparePortEdge(c.Ddg[i-1], c.Ddg[i]) >= 0 { + t.Errorf("%s: ddg not sorted/unique at %+v", k.Signature, c.Ddg[i]) + } + } + } + if nPT == 0 || nSumm == 0 { + t.Fatalf("alias pass added %d points-to and %d summary edges: the gate covers nothing", nPT, nSumm) + } + t.Logf("dataflow: %d points-to edges, %d summary edges added", nPT, nSumm) + + off.Callables, on.Callables = nil, nil + if !reflect.DeepEqual(off, on) { + t.Errorf("Run's result outside Callables differs with the alias pass on") + } +} + +// TestAlias_SummaryThroughAlias: aliasAssign's v reaches its p only through +// the alias q (q := p; q.n = v), so its caller CallAliasAssign gains the +// summary edge from v's actual_in to p's actual_out with the pass on, and +// lacks it off. +func TestAlias_SummaryThroughAlias(t *testing.T) { + want := SummaryEdge{Src: "382:2/actual_in:1", Dst: "382:2/actual_out"} + if got := runAlias(t, true).Callables[sdgKey("CallAliasAssign")].Summary; !slices.Contains(got, want) { + t.Errorf("CallAliasAssign summary %+v lacks %+v", got, want) + } + if got := runAlias(t, false).Callables[sdgKey("CallAliasAssign")].Summary; slices.Contains(got, want) { + t.Errorf("CallAliasAssign summary %+v has %+v without the alias pass", got, want) + } +} + +// TestAlias_SkippedAndNoVTA: an l3_skipped unit gets no points-to edge (its +// pass-through wiring is already all-to-all), and under no_vta the pass +// still runs (it is intraprocedural; it reads no SSA). +func TestAlias_SkippedAndNoVTA(t *testing.T) { + in := loadInput(t, "dataflow") + delete(in.L3, sdgKey("AliasPtr")) + res := Run(in) + if got := pointsTo(res.Callables[sdgKey("AliasPtr")].Ddg); len(got) != 0 { + t.Errorf("skipped AliasPtr has points-to edges %+v", got) + } + + in = loadInput(t, "dataflow") + defer func(f func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph) { vtaCallGraph = f }(vtaCallGraph) + vtaCallGraph = func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph { panic("vta boom") } + res = Run(in) + if len(res.Degraded) == 0 || res.Degraded[len(res.Degraded)-1].Code != CodeNoVTA { + t.Fatalf("degraded = %+v, want no_vta", res.Degraded) + } + if got := pointsTo(res.Callables[sdgKey("AliasPtr")].Ddg); !slices.Contains(got, pt("360:2", "361:2", "req.UserID")) { + t.Errorf("no_vta AliasPtr points-to edges %+v lack 360:2 → 361:2", got) + } +} + +// TestWiring_GoldenWithAliasOff (M5) pins wire's output for every sdg +// fixture unit with the alias pass off, byte for byte, against +// testdata/wire_sdg_off.golden, which was generated by the pre-refactor +// wiring (7ec77e7) over this fixture: the port() refactor of the wiring is +// output-neutral. Regenerate with -update only for an intended change. +func TestWiring_GoldenWithAliasOff(t *testing.T) { + defer func(v func(dataflow.Unit, ast.Node) AliasOracle) { newAliasOracle = v }(newAliasOracle) + newAliasOracle = nil + in, _, p := runPorts(t) + var b strings.Builder + for _, u := range in.Units { + if u.Key.Path != sdgFile { + continue + } + fmt.Fprintf(&b, "%s\n", u.Key.Signature) + for _, e := range p.wire(u.Key) { + fmt.Fprintf(&b, "\t%s -> %s %s %s\n", e.Src, e.Dst, e.Var, e.Prov) + } + } + golden := filepath.Join("testdata", "wire_sdg_off.golden") + if *update { + if err := os.WriteFile(golden, []byte(b.String()), 0o644); err != nil { + t.Fatal(err) + } + } + want, err := os.ReadFile(golden) + if err != nil { + t.Fatal(err) + } + if b.String() != string(want) { + t.Errorf("wire output (alias off) differs from %s; diff with -update", golden) + } +} diff --git a/internal/sdg/cha.go b/internal/sdg/cha.go new file mode 100644 index 0000000..a1b76ed --- /dev/null +++ b/internal/sdg/cha.go @@ -0,0 +1,192 @@ +package sdg + +import ( + "fmt" + "go/ast" + "go/types" + "slices" + + "golang.org/x/tools/go/ssa" + "golang.org/x/tools/go/types/typeutil" + + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" +) + +// cha is the targeted CHA complement of VTA (spec §4.8). VTA propagates +// types through project code only: library packages have no SSA bodies, so +// a value that passes through library storage — put in a context.Context or +// a sync.Map, returned by a library call, read from a library struct's field +// — reaches an interface call with no concrete type VTA can see, and only +// the D7 interface method (empty footprint) would remain. For an interface +// call whose receiver may carry such a value (origins), every +// project type whose method set implements the interface is a target: the +// method it resolves to, mapped as the join maps a callee. It reads SSA +// instructions and go/types only, never builds code, so it is safe before +// VTA. +type cha struct { + j *join + types []types.Type // every project named type T and *T that is not an interface or generic + byIface typeutil.Map // interface type → its implementers among types + msets typeutil.MethodSetCache +} + +func newCHA(j *join, in Input) *cha { + c := &cha{j: j} + seen := map[*types.TypeName]bool{} + for _, p := range projectPackages(in) { + if p.TypesInfo == nil { + continue + } + for _, f := range p.Syntax { + ast.Inspect(f, func(n ast.Node) bool { + ts, ok := n.(*ast.TypeSpec) + if !ok { + return true + } + tn, ok := p.TypesInfo.Defs[ts.Name].(*types.TypeName) + if !ok || tn.IsAlias() || seen[tn] { + return true + } + seen[tn] = true + named, ok := tn.Type().(*types.Named) + if !ok || named.TypeParams() != nil || types.IsInterface(named) { + return true + } + c.types = append(c.types, named, types.NewPointer(named)) + return true + }) + } + } + return c +} + +// implementers returns the project types of c.types that implement iface +// (T and *T alike: an interface may hold either). +func (c *cha) implementers(iface types.Type) []types.Type { + if v := c.byIface.At(iface); v != nil { + return v.([]types.Type) + } + it, _ := iface.Underlying().(*types.Interface) + var out []types.Type + if it != nil { + for _, t := range c.types { + if types.Implements(t, it) { + out = append(out, t) + } + } + } + c.byIface.Set(iface, out) + return out +} + +// addCHA adds the CHA complement's targets (spec §4.8) to every interface +// call of a body-joined project function (funcs) whose receiver may carry +// a library-originated dynamic type: the origins fixpoint over the whole +// program (cha_origin.go). A target is the method each implementer's method +// set selects: a unit, an interface method (a struct embedding an +// interface), a bodiless declaration, or a library method's external (a +// promoted library method). It never adds a call_graph edge (only VTA's +// callees are edges). Functions are visited in a sorted order and targets +// are kept sorted (d.add), so the result is deterministic. +func (d *dispatch) addCHA(c *cha, funcs map[*ssa.Function]bool) { + o := newOrigins(c, funcs) + o.solve() + for _, fn := range o.order { + if chaHook != nil { + chaHook(fn) + } + var lo map[ssa.Value]bool + forCalls(fn, func(call ssa.CallInstruction) { + cc := call.Common() + if !cc.IsInvoke() || cc.Method == nil { + return + } + if _, isParam := types.Unalias(cc.Value.Type()).(*types.TypeParam); isParam { + return + } + unit, key, ok := c.j.callSite(call) + if !ok { + return + } + if lo == nil { + lo = o.eval(fn) // o.cur is nil: no reader is registered + } + if !lo[cc.Value] { + return + } + site := siteKey{unit, key} + for _, t := range c.implementers(cc.Value.Type()) { + sel := c.msets.MethodSet(t).Lookup(cc.Method.Pkg(), cc.Method.Name()) + if sel == nil { + continue + } + if tg, ok := d.funcTarget(c.j, sel.Obj().(*types.Func)); ok { + d.add(site, tg) + } + } + }) + } +} + +// runCHA is addCHA under recover (N1): it reads go/types and SSA in ways +// VTA does not, so an unforeseen shape must not abort the analyzer. On a +// panic every CHA target is dropped — the dispatch is restored to what it +// held before — and the error is returned (degraded no_cha). +func (d *dispatch) runCHA(j *join, in Input, funcs map[*ssa.Function]bool) (err error) { + saved := make(map[siteKey][]Target, len(d.sites)) + for k, ts := range d.sites { + saved[k] = slices.Clone(ts) // d.add inserts in place: the backing array is not ours + } + newExt := make(map[string]bool, len(d.newExt)) + for k := range d.newExt { + newExt[k] = true + } + defer func() { + if r := recover(); r != nil { + d.sites = saved + for k := range d.newExt { + if !newExt[k] { + delete(d.newExt, k) + } + } + err = fmt.Errorf("panic: %v", r) + } + }() + d.addCHA(newCHA(j, in), funcs) + return nil +} + +// funcTarget maps a declared function or method to its target, as +// calleeTarget maps its SSA function: the unit it declares, an interface +// method, a bodiless declaration, or a library function's external. A +// universe method (error.Error, promoted through an embedded error) has no +// package and no target, as for methodTarget. +func (d *dispatch) funcTarget(j *join, obj *types.Func) (Target, bool) { + if obj.Pkg() == nil { + return Target{}, false + } + if j.isLibrary(obj) { + sig := semantic_analysis.CalleeSignatureOf(obj) + d.external(obj, sig) + return Target{External: sig}, true + } + k, ok := declKey(j.fset, j.projectDir, obj.Origin()) + if !ok { + return Target{}, false + } + if _, unit := j.units[k]; unit || j.iface[k] || j.bodiless[k] { + return Target{Callable: k}, true + } + return Target{}, false +} + +// libraryType reports whether t is a named type declared outside the +// project. +func (c *cha) libraryType(t types.Type) bool { + named, ok := types.Unalias(t).(*types.Named) + if !ok { + return false + } + pkg := named.Obj().Pkg() + return pkg != nil && !c.j.projPaths[pkg.Path()] +} diff --git a/internal/sdg/cha_origin.go b/internal/sdg/cha_origin.go new file mode 100644 index 0000000..6c88544 --- /dev/null +++ b/internal/sdg/cha_origin.go @@ -0,0 +1,695 @@ +package sdg + +import ( + "go/token" + "go/types" + "slices" + "sort" + + "golang.org/x/tools/go/ssa" + "golang.org/x/tools/go/types/typeutil" +) + +// origins is the interprocedural "may carry a library-originated dynamic +// type" analysis behind the CHA complement (spec §4.8). VTA propagates +// types through project bodies only (its graph is field-based for struct +// fields and type-based for pointer, slice, map and channel contents), so +// what it cannot see is a value that came out of library code: no flow +// reaches it. The analysis tracks where such values go, through project +// code, to a monotone fixpoint over the whole program (the project +// functions VTA runs over): +// +// - Seeds (per value): a TypeAssert; the result of a library function, a +// project function with no body, a library interface's method, recover(); +// a load of a library global or through a library struct's field. +// - Local steps: Phi, Extract, ChangeInterface, ChangeType, Convert, Slice, +// Field and FieldAddr / IndexAddr of an LO base; a map Lookup, an Index, +// a channel receive and a range Next over an LO container (N2). +// - Interprocedural facts (finite, so the fixpoint terminates): +// ret(F, i) result i of project function F (any Return operand i is LO); +// param(p) a parameter (receiver included) some call passes an LO actual; +// free(v) a closure capture bound to an LO value (MakeClosure); +// global(G) a project global an LO value is stored to; +// field(S, i) field i of struct type S (by type: every instance); +// alloc(a) an address-taken local; +// seq(T) an element of any slice or array of element type T; +// mapkey(T), mapval(T), chan(T) map keys, map values and channel +// elements of type T (any container of that element type); +// unknown(T) a T stored through a pointer of unknown provenance (a +// parameter, a call result); every precise load of a T reads it too; +// any(T) any store of a T; a load through such a pointer reads it. +// A pointer, slice, map or channel of interface elements passed to +// library code (errors.As(err, &target)) marks unknown(T) / any(T) and +// the container's element cell: the library may write into it. +// - Call targets: the static callee; for an interface call, the method +// each project implementer's method set selects (CHA), and the result is +// LO too when the receiver is LO or a library type the project converts +// to an interface implements it; for a call through a function value, +// every address-taken project function of an identical signature, and +// the result is LO when the value is LO or a library function of that +// signature is used as a value. VTA's targets are not used: the analysis +// runs before VTA (the program must not be read after a no_vta), and +// these sets are supersets of VTA's. +// +// Each fact only goes from false to true. A function is re-evaluated when a +// fact it read changes (a reader-tracked worklist); a function's values are +// computed by a local fixpoint (Phi cycles). The least fixpoint is unique, +// so the result is independent of the visiting order; the order is +// deterministic anyway (functions sorted by package path and name, readers +// in registration order). It reads SSA instructions and go/types only and +// builds no code. +type origins struct { + c *cha + prog *ssa.Program + funcs map[*ssa.Function]bool + order []*ssa.Function + + facts map[fact]bool + readers map[fact][]*ssa.Function + readBy map[fact]map[*ssa.Function]bool + cur *ssa.Function // the function being evaluated (reader registration); nil = none + + work []*ssa.Function + queued map[*ssa.Function]bool + + typeIDs typeutil.Map // type → int id (fact keys) + nextID int + byFuncSig typeutil.Map // signature → []*ssa.Function: address-taken project functions + libFuncs typeutil.Map // signature → true: address-taken library functions + libConv []types.Type // library types project code converts to an interface + // Memos of pure functions of the program (fixed before the worklist; + // never facts): each entry is computed whole the first time it is read. + libImpl typeutil.Map // interface → bool: a libConv type implements it + implFns map[implKey][]*ssa.Function // filled with implLib, by implementers + implLib map[implKey]bool // an implementer's selected method is library code +} + +type factKind uint8 + +const ( + fRet factKind = iota + fParam + fFree + fGlobal + fField + fAlloc + fSeq + fMapKey + fMapVal + fChan + fUnknown + fAny +) + +// fact is one interprocedural fact: kind, an SSA object (ret, param, free, +// global, alloc) and up to two ints (a result index; a type id and a field +// index). +type fact struct { + k factKind + p any + a, b int +} + +type implKey struct { + iface int // type id of the interface + pkg *types.Package + name string +} + +// originSeedOrder, when set, permutes the order the origin worklist is +// seeded in (a test seam: the least fixpoint must not depend on it). +var originSeedOrder func([]*ssa.Function) + +// chaHook, when set, runs on each function the CHA complement scans (a test +// seam for the recovered-panic path, N1). +var chaHook func(*ssa.Function) + +func newOrigins(c *cha, funcs map[*ssa.Function]bool) *origins { + o := &origins{ + c: c, + funcs: funcs, + facts: map[fact]bool{}, + readers: map[fact][]*ssa.Function{}, + readBy: map[fact]map[*ssa.Function]bool{}, + queued: map[*ssa.Function]bool{}, + implFns: map[implKey][]*ssa.Function{}, + implLib: map[implKey]bool{}, + } + for fn := range funcs { + o.order = append(o.order, fn) + o.prog = fn.Prog + } + sort.Slice(o.order, func(i, j int) bool { + a, b := o.order[i], o.order[j] + if pa, pb := homePath(a), homePath(b); pa != pb { + return pa < pb + } + return a.String() < b.String() + }) + var conv typeutil.Map + var buf [10]*ssa.Value + for _, fn := range o.order { + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + var callee *ssa.Value + if call, ok := instr.(ssa.CallInstruction); ok && !call.Common().IsInvoke() { + callee = &call.Common().Value + } + for _, op := range instr.Operands(buf[:0]) { + if g, ok := (*op).(*ssa.Function); ok && op != callee { + o.addressTaken(g) + } + } + if mc, ok := instr.(*ssa.MakeClosure); ok { + o.addressTaken(mc.Fn.(*ssa.Function)) + } + if mi, ok := instr.(*ssa.MakeInterface); ok && c.libraryType(derefType(mi.X.Type())) && conv.At(mi.X.Type()) == nil { + conv.Set(mi.X.Type(), true) + o.libConv = append(o.libConv, mi.X.Type()) + } + } + } + } + return o +} + +func derefType(t types.Type) types.Type { + if p, ok := types.Unalias(t).Underlying().(*types.Pointer); ok { + return p.Elem() + } + return t +} + +func (o *origins) addressTaken(g *ssa.Function) { + if o.funcs[g] { + list, _ := o.byFuncSig.At(g.Signature).([]*ssa.Function) + for _, f := range list { + if f == g { + return + } + } + o.byFuncSig.Set(g.Signature, append(list, g)) + return + } + o.libFuncs.Set(g.Signature, true) +} + +func (o *origins) typeID(t types.Type) int { + if v := o.typeIDs.At(t); v != nil { + return v.(int) + } + o.nextID++ + o.typeIDs.Set(t, o.nextID) + return o.nextID +} + +// get reads fact f, registering the current function as its reader. +func (o *origins) get(f fact) bool { + if o.cur != nil { + set := o.readBy[f] + if set == nil { + set = map[*ssa.Function]bool{} + o.readBy[f] = set + } + if !set[o.cur] { + set[o.cur] = true + o.readers[f] = append(o.readers[f], o.cur) + } + } + return o.facts[f] +} + +// set makes fact f true and re-queues its readers. +func (o *origins) set(f fact) { + if o.facts[f] { + return + } + o.facts[f] = true + for _, r := range o.readers[f] { + o.enqueue(r) + } +} + +func (o *origins) enqueue(fn *ssa.Function) { + if !o.queued[fn] { + o.queued[fn] = true + o.work = append(o.work, fn) + } +} + +// solve runs the worklist to the fixpoint. +func (o *origins) solve() { + seed := o.order + if originSeedOrder != nil { + seed = slices.Clone(o.order) + originSeedOrder(seed) + } + for _, fn := range seed { + o.enqueue(fn) + } + for len(o.work) > 0 { + fn := o.work[0] + o.work = o.work[1:] + o.queued[fn] = false + o.cur = fn + lo := o.eval(fn) + o.effects(fn, lo) + o.cur = nil + } +} + +// eval is fn's local fixpoint: the set of its values (parameters, captures +// and instruction values) that may carry a library-originated dynamic type, +// under the current facts. +func (o *origins) eval(fn *ssa.Function) map[ssa.Value]bool { + lo := map[ssa.Value]bool{} + for _, p := range fn.Params { + if o.get(fact{k: fParam, p: p}) { + lo[p] = true + } + } + for _, fv := range fn.FreeVars { + if o.get(fact{k: fFree, p: fv}) { + lo[fv] = true + } + } + for changed := true; changed; { + changed = false + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + v, ok := instr.(ssa.Value) + if !ok || lo[v] { + continue + } + if o.valueLO(v, lo) { + lo[v] = true + changed = true + } + } + } + } + return lo +} + +// valueLO is the rule for one instruction value. +func (o *origins) valueLO(v ssa.Value, lo map[ssa.Value]bool) bool { + switch v := v.(type) { + case *ssa.Phi: + for _, e := range v.Edges { + if lo[e] { + return true + } + } + case *ssa.Extract: + return o.tupleLO(v.Tuple, v.Index, lo) + case *ssa.ChangeInterface: + return lo[v.X] + case *ssa.ChangeType: + return lo[v.X] + case *ssa.Convert: + return lo[v.X] + case *ssa.MultiConvert: + return lo[v.X] + case *ssa.SliceToArrayPointer: + return lo[v.X] + case *ssa.Slice: + return lo[v.X] + case *ssa.Range: + return lo[v.X] + case *ssa.TypeAssert: + return true + case *ssa.Call: + return o.callLO(v.Common(), 0, lo) + case *ssa.UnOp: + switch v.Op { + case token.MUL: + return o.loadLO(v.X, lo) + case token.ARROW: + return o.recvLO(v.X, lo) + } + case *ssa.Field: + st := v.X.Type() + return o.c.libraryType(st) || lo[v.X] || o.get(o.fieldFact(st, v.Field)) + case *ssa.FieldAddr, *ssa.IndexAddr: + var x ssa.Value + if fa, ok := v.(*ssa.FieldAddr); ok { + x = fa.X + } else { + x = v.(*ssa.IndexAddr).X + } + return lo[x] + case *ssa.Lookup: + return o.lookupLO(v, lo) + case *ssa.Index: + return lo[v.X] || o.get(fact{k: fSeq, a: o.typeID(v.Type())}) + } + return false +} + +// tupleLO is the rule for component i of a tuple-valued instruction. +func (o *origins) tupleLO(t ssa.Value, i int, lo map[ssa.Value]bool) bool { + switch t := t.(type) { + case *ssa.Call: + return o.callLO(t.Common(), i, lo) + case *ssa.TypeAssert: + return i == 0 + case *ssa.Lookup: + return i == 0 && o.lookupLO(t, lo) + case *ssa.UnOp: + return i == 0 && t.Op == token.ARROW && o.recvLO(t.X, lo) + case *ssa.Next: + if t.IsString || i == 0 { + return false + } + r, ok := t.Iter.(*ssa.Range) + if !ok { + return true + } + if lo[r.X] { + return true + } + m, ok := types.Unalias(r.X.Type()).Underlying().(*types.Map) + if !ok { + return false + } + if i == 1 { + return o.get(fact{k: fMapKey, a: o.typeID(m.Key())}) + } + return o.get(fact{k: fMapVal, a: o.typeID(m.Elem())}) + case *ssa.Select: + k := i - 2 // results: index, recvOk, then one per receive state + for _, st := range t.States { + if st.Dir != types.RecvOnly { + continue + } + if k == 0 { + return o.recvLO(st.Chan, lo) + } + k-- + } + } + return false +} + +func (o *origins) lookupLO(l *ssa.Lookup, lo map[ssa.Value]bool) bool { + if lo[l.X] { + return true + } + if m, ok := types.Unalias(l.X.Type()).Underlying().(*types.Map); ok { + return o.get(fact{k: fMapVal, a: o.typeID(m.Elem())}) + } + return false +} + +func (o *origins) recvLO(ch ssa.Value, lo map[ssa.Value]bool) bool { + if lo[ch] { + return true + } + if c, ok := types.Unalias(ch.Type()).Underlying().(*types.Chan); ok { + return o.get(fact{k: fChan, a: o.typeID(c.Elem())}) + } + return false +} + +func (o *origins) fieldFact(structType types.Type, i int) fact { + return fact{k: fField, a: o.typeID(types.Unalias(structType).Underlying()), b: i} +} + +// loadLO is the rule for a load *x. +func (o *origins) loadLO(x ssa.Value, lo map[ssa.Value]bool) bool { + t := o.typeID(derefType(x.Type())) + unknown := func() bool { return o.get(fact{k: fUnknown, a: t}) } + switch x := x.(type) { + case *ssa.Global: + if x.Pkg != nil && !o.c.j.projPaths[x.Pkg.Pkg.Path()] { + return true + } + return o.get(fact{k: fGlobal, p: x}) || unknown() + case *ssa.FieldAddr: + st := derefType(x.X.Type()) + return o.c.libraryType(st) || lo[x.X] || o.get(o.fieldFact(st, x.Field)) || unknown() + case *ssa.IndexAddr: + return lo[x.X] || o.get(fact{k: fSeq, a: t}) || unknown() + case *ssa.Alloc: + return o.get(fact{k: fAlloc, p: x}) || unknown() + } + return lo[x] || o.get(fact{k: fAny, a: t}) +} + +// callLO is the rule for result i of the call cc. +func (o *origins) callLO(cc *ssa.CallCommon, i int, lo map[ssa.Value]bool) bool { + if b, ok := cc.Value.(*ssa.Builtin); ok { + switch b.Name() { + case "recover": + return true + case "append": + for _, a := range cc.Args { + if lo[a] { + return true + } + } + } + return false + } + if cc.IsInvoke() { + if cc.Method == nil || cc.Method.Pkg() == nil { + return lo[cc.Value] + } + if o.c.j.isLibrary(cc.Method) || lo[cc.Value] || o.libImplements(cc.Value.Type()) { + return true + } + fns, lib := o.implementers(cc) + if lib { + return true + } + for _, g := range fns { + if o.get(fact{k: fRet, p: g, a: i}) { + return true + } + } + return false + } + if g := cc.StaticCallee(); g != nil { + if !o.funcs[g] || len(g.Blocks) == 0 { + return true // library code, or a function with no SSA body (assembly) + } + return o.get(fact{k: fRet, p: g, a: i}) + } + sig := cc.Signature() + if lo[cc.Value] || o.libFuncs.At(sig) != nil { + return true + } + list, _ := o.byFuncSig.At(sig).([]*ssa.Function) + for _, g := range list { + if o.get(fact{k: fRet, p: g, a: i}) { + return true + } + } + return false +} + +// libImplements reports whether a library type the project converts to an +// interface implements iface: a call on such a value may run library code. +func (o *origins) libImplements(iface types.Type) bool { + if v := o.libImpl.At(iface); v != nil { + return v.(bool) + } + it, _ := types.Unalias(iface).Underlying().(*types.Interface) + found := false + for _, t := range o.libConv { + if it != nil && types.Implements(t, it) { + found = true + break + } + } + o.libImpl.Set(iface, found) + return found +} + +func (o *origins) implKey(cc *ssa.CallCommon) implKey { + return implKey{o.typeID(cc.Value.Type()), cc.Method.Pkg(), cc.Method.Name()} +} + +// implementers returns the project functions an interface call cc may run +// — the method each project implementer's method set selects, when it has a +// body — and whether some implementer's selected method is library code (a +// promoted library method: its result may carry a library-originated +// type). Both are a pure function of the program (the project types and +// their method sets, fixed before the worklist), computed together and +// memoized together: neither is ever read before it is filled, so no +// evaluation depends on which call was evaluated first. +func (o *origins) implementers(cc *ssa.CallCommon) ([]*ssa.Function, bool) { + k := o.implKey(cc) + if fns, ok := o.implFns[k]; ok { + return fns, o.implLib[k] + } + var fns []*ssa.Function + seen := map[*ssa.Function]bool{} + for _, t := range o.c.implementers(cc.Value.Type()) { + sel := o.c.msets.MethodSet(t).Lookup(k.pkg, k.name) + if sel == nil { + continue + } + obj := sel.Obj().(*types.Func) + if obj.Pkg() == nil { + continue + } + if o.c.j.isLibrary(obj) { + o.implLib[k] = true + continue + } + if g := o.prog.FuncValue(obj); g != nil && o.funcs[g] && !seen[g] { + seen[g] = true + fns = append(fns, g) + } + } + o.implFns[k] = fns + return fns, o.implLib[k] +} + +// effects applies fn's instructions to the facts, under its values lo. +func (o *origins) effects(fn *ssa.Function, lo map[ssa.Value]bool) { + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + switch instr := instr.(type) { + case *ssa.Return: + for i, r := range instr.Results { + if lo[r] { + o.set(fact{k: fRet, p: fn, a: i}) + } + } + case ssa.CallInstruction: + o.callEffects(instr.Common(), lo) + case *ssa.MakeClosure: + g := instr.Fn.(*ssa.Function) + for i, v := range instr.Bindings { + if lo[v] && i < len(g.FreeVars) { + o.set(fact{k: fFree, p: g.FreeVars[i]}) + } + } + case *ssa.Store: + if lo[instr.Val] { + o.store(instr.Addr) + } + case *ssa.MapUpdate: + m, ok := types.Unalias(instr.Map.Type()).Underlying().(*types.Map) + if !ok { + continue + } + if lo[instr.Value] { + o.set(fact{k: fMapVal, a: o.typeID(m.Elem())}) + } + if lo[instr.Key] { + o.set(fact{k: fMapKey, a: o.typeID(m.Key())}) + } + case *ssa.Send: + if lo[instr.X] { + if c, ok := types.Unalias(instr.Chan.Type()).Underlying().(*types.Chan); ok { + o.set(fact{k: fChan, a: o.typeID(c.Elem())}) + } + } + } + } + } +} + +// store records an LO value stored through addr. +func (o *origins) store(addr ssa.Value) { + t := o.typeID(derefType(addr.Type())) + o.set(fact{k: fAny, a: t}) + switch a := addr.(type) { + case *ssa.Global: + o.set(fact{k: fGlobal, p: a}) + case *ssa.FieldAddr: + o.set(o.fieldFact(derefType(a.X.Type()), a.Field)) + case *ssa.IndexAddr: + o.set(fact{k: fSeq, a: t}) + case *ssa.Alloc: + o.set(fact{k: fAlloc, p: a}) + default: + o.set(fact{k: fUnknown, a: t}) + } +} + +// callEffects passes cc's LO actuals to its targets' parameters, and marks +// what an interface container passed to library code may receive. +func (o *origins) callEffects(cc *ssa.CallCommon, lo map[ssa.Value]bool) { + if _, ok := cc.Value.(*ssa.Builtin); ok { + if b := cc.Value.(*ssa.Builtin); b.Name() == "copy" && len(cc.Args) == 2 && lo[cc.Args[1]] { + if s, ok := types.Unalias(cc.Args[0].Type()).Underlying().(*types.Slice); ok { + o.set(fact{k: fSeq, a: o.typeID(s.Elem())}) + } + } + return + } + pass := func(g *ssa.Function, args []ssa.Value) { + for i, a := range args { + if lo[a] && i < len(g.Params) { + o.set(fact{k: fParam, p: g.Params[i]}) + } + } + } + if cc.IsInvoke() { + if cc.Method == nil || cc.Method.Pkg() == nil { + return + } + if o.c.j.isLibrary(cc.Method) { + o.escape(cc.Args) + } + args := append([]ssa.Value{cc.Value}, cc.Args...) + fns, _ := o.implementers(cc) + for _, g := range fns { + pass(g, args) + } + return + } + if g := cc.StaticCallee(); g != nil { + if !o.funcs[g] || len(g.Blocks) == 0 { + o.escape(cc.Args) + return + } + pass(g, cc.Args) + return + } + list, _ := o.byFuncSig.At(cc.Signature()).([]*ssa.Function) + for _, g := range list { + pass(g, cc.Args) + } + if o.libFuncs.At(cc.Signature()) != nil { + o.escape(cc.Args) + } +} + +// escape marks what library code may write into the interface containers +// args hand it (through a MakeInterface too): a pointer to an interface +// (errors.As), a slice, map or channel of interface elements. +func (o *origins) escape(args []ssa.Value) { + for _, a := range args { + if mi, ok := a.(*ssa.MakeInterface); ok { + a = mi.X + } + switch t := types.Unalias(a.Type()).Underlying().(type) { + case *types.Pointer: + if types.IsInterface(t.Elem()) { + id := o.typeID(t.Elem()) + o.set(fact{k: fUnknown, a: id}) + o.set(fact{k: fAny, a: id}) + } + case *types.Slice: + if types.IsInterface(t.Elem()) { + o.set(fact{k: fSeq, a: o.typeID(t.Elem())}) + } + case *types.Map: + if types.IsInterface(t.Elem()) { + o.set(fact{k: fMapVal, a: o.typeID(t.Elem())}) + } + if types.IsInterface(t.Key()) { + o.set(fact{k: fMapKey, a: o.typeID(t.Key())}) + } + case *types.Chan: + if types.IsInterface(t.Elem()) { + o.set(fact{k: fChan, a: o.typeID(t.Elem())}) + } + } + } +} diff --git a/internal/sdg/cha_test.go b/internal/sdg/cha_test.go new file mode 100644 index 0000000..01f7d5b --- /dev/null +++ b/internal/sdg/cha_test.go @@ -0,0 +1,83 @@ +package sdg + +import ( + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +const libstorePkg = "example.com/dataflow/libstore" + +// libstoreKey is the L3Key of a callable declared in +// testdata/dataflow/libstore/libstore.go. +func libstoreKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("libstore", "libstore.go"), Signature: libstorePkg + "." + name} +} + +func libstoreTarget(name string) Target { return Target{Callable: libstoreKey(name)} } + +// TestCHA_LibraryOriginatedReceivers (F3): HandleCtx's +// ctx.Value(k).(Store).Find(id) and HandleMap's v.(Store).Find(id) call Find +// on a value that came out of library storage (context.Context, sync.Map): +// VTA sees no concrete type, so the site's targets include every project +// type implementing Store — sqlStore.Find and memStore.Find, and the Find of +// any other package's type with the same method set — beside the interface +// method (D7). The callers' footprints see the global +// sqlStore.Find writes. HandleLocal's s.Find(id), on a value project code +// built, gets VTA's memStore.Find and no CHA target. +func TestCHA_LibraryOriginatedReceivers(t *testing.T) { + in, res, d := runDispatch(t) + all := []Target{libstoreTarget("Store.Find"), libstoreTarget("memStore.Find"), libstoreTarget("sqlStore.Find")} + fps := footprints(in, d) + for _, c := range []struct{ fn, site string }{ + {"HandleCtx", "46:9"}, + {"HandleMap", "59:9"}, + } { + got := d.targets(libstoreKey(c.fn), c.site) + for _, w := range all { + if !slices.Contains(got, w) { + t.Errorf("targets(%s, %s) = %v, want %v among them", c.fn, c.site, got, w) + } + } + // Every project type with Find(string) string implements Store + // structurally (wrap's too): each target is some type's Find. + for _, tg := range got { + if !strings.HasSuffix(tg.Callable.Signature, ".Find") { + t.Errorf("targets(%s, %s) has %v, not a Find", c.fn, c.site, tg) + } + } + if fp := fps[libstoreTarget(c.fn)]; !slices.Contains(fp.GlobalWrites, libstorePkg+"::lastQuery") { + t.Errorf("%s writes %v, want %s::lastQuery", c.fn, fp.GlobalWrites, libstorePkg) + } + } + want := []Target{libstoreTarget("Store.Find"), libstoreTarget("memStore.Find")} + if got := d.targets(libstoreKey("HandleLocal"), "72:9"); !slices.Equal(got, want) { + t.Errorf("targets(HandleLocal, 72:9) = %v, want %v (no CHA target for a project-built value)", got, want) + } + // CHA targets are not VTA's: no call_graph edge (F7). + for _, e := range res.CallEdges { + if (e.Source == libstorePkg+".HandleCtx" || e.Source == libstorePkg+".HandleMap") && e.Target != "" { + t.Errorf("call_graph edge %s -> %s for a CHA target", e.Source, e.Target) + } + } +} + +// TestCHA_UniverseMethodNoTarget: HandleErr's context.Cause(ctx).Error() +// is an interface call on a library call's result; errStore embeds the +// universe error, so its method set's Error has no package. The CHA +// complement adds no target for it (and does not panic): the site keeps +// what VTA and D7 give it. +func TestCHA_UniverseMethodNoTarget(t *testing.T) { + _, _, d := runDispatch(t) + if _, ok := d.sites[siteKey{libstoreKey("HandleErr"), "95:9/2"}]; !ok { + t.Fatal("HandleErr's context.Cause call has no targets: the site keys moved") + } + for _, tg := range d.targets(libstoreKey("HandleErr"), "95:9") { + if tg.Callable.Signature != "" { + t.Errorf("targets(HandleErr, 95:9) has %v, want no in-project target", tg) + } + } +} diff --git a/internal/sdg/differential_test.go b/internal/sdg/differential_test.go new file mode 100644 index 0000000..5d785ed --- /dev/null +++ b/internal/sdg/differential_test.go @@ -0,0 +1,418 @@ +package sdg + +import ( + "fmt" + "go/ast" + "go/token" + "os" + "sort" + "strings" + "testing" + + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// The go/ssa def-use differential (#22, spec §4.5 "L3 precision"): an +// independent answer key for the syntactic L3 ddg. It does not gate: L3 and +// SSA model different things (SSA is value flow over registers, L3 is +// access-path flow between statements), so mismatches are reported, and the +// test asserts only that the mapping layer works (ruling R-P6). + +// ssaPair is one SSA def-use pair of a callable mapped to its L3 body keys. +type ssaPair struct { + Unit dataflow.L3Key + Def, Use string // local body keys + What string // the SSA value, for the report +} + +// bodyIndex maps a raw position to the innermost CFG body node of one L3 +// result containing it. Satellite call nodes are skipped (their def/use +// belongs to the host statement, DefUseOf), and so are deferred call nodes: +// SSA evaluates a deferred call's function value and arguments at the +// registration, the `defer` statement node. A position in the signature +// (a parameter spilled to the heap) is @entry; one at the closing brace (the +// implicit return) is @exit. +type bodyIndex struct { + nodes []dataflow.Node + lbrace, rbrace token.Pos +} + +func newBodyIndex(r dataflow.Result, fn ast.Node) bodyIndex { + var ns []dataflow.Node + for _, n := range r.Nodes { + if n.Pos.IsValid() && n.Parent == "" && !n.Deferred { + ns = append(ns, n) + } + } + b := bodyIndex{nodes: ns} + switch fn := fn.(type) { + case *ast.FuncDecl: + b.lbrace, b.rbrace = fn.Body.Lbrace, fn.Body.Rbrace + case *ast.FuncLit: + b.lbrace, b.rbrace = fn.Body.Lbrace, fn.Body.Rbrace + } + return b +} + +func (b bodyIndex) at(pos token.Pos) (string, bool) { + if b.lbrace.IsValid() { + if pos < b.lbrace { + return dataflow.EntryKey, true + } + if pos >= b.rbrace { + return dataflow.ExitKey, true + } + } + best := -1 + for i, n := range b.nodes { + if n.Pos <= pos && pos < n.End { + if best < 0 || n.End-n.Pos < b.nodes[best].End-b.nodes[best].Pos { + best = i + } + } + } + if best < 0 { + return "", false + } + return b.nodes[best].Key, true +} + +// ssaPairs returns the def-use pairs of the SSA functions of funcs that join +// a unit by body, mapped to body keys: for each SSA value with a definition +// site (a Parameter or FreeVar: @entry; an instruction: the body node at its +// position) and each of its referrers, the referrer's body node. Phis and +// position-less value instructions (Extract, implicit conversions, +// MakeClosure) are looked through on both sides; a Store through a value is +// a write, not a use. Pairs within one node are dropped (L3 edges are +// between nodes). +func ssaPairs(j *join, funcs map[*ssa.Function]bool) (pairs []ssaPair, instrs, unmappedInstrs int) { + idx := map[dataflow.L3Key]bodyIndex{} + for fn := range funcs { + unit, kind := j.resolve(fn) + if kind != joinBody { + continue + } + r, ok := j.results[unit] + if !ok { + continue + } + bi, ok := idx[unit] + if !ok { + bi = newBodyIndex(r, j.units[unit].Fn) + idx[unit] = bi + } + keyOf := func(instr ssa.Instruction) (string, bool) { + p := instr.Pos() + if !p.IsValid() { + return "", false + } + instrs++ + k, ok := bi.at(p) + if !ok { + unmappedInstrs++ + } + return k, ok + } + seen := map[ssaPair]bool{} + add := func(def, use, what string) { + if def == use { + return + } + p := ssaPair{Unit: unit, Def: def, Use: use, What: what} + if !seen[p] { + seen[p] = true + pairs = append(pairs, p) + } + } + var values []ssa.Value + for _, p := range fn.Params { + values = append(values, p) + } + for _, fv := range fn.FreeVars { + values = append(values, fv) + } + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + if v, ok := instr.(ssa.Value); ok { + values = append(values, v) + } + } + } + for _, v := range values { + var def string + switch v := v.(type) { + case *ssa.Parameter, *ssa.FreeVar: + def = dataflow.EntryKey + case *ssa.Phi: + continue // looked through from its operands' side + case ssa.Instruction: + k, ok := keyOf(v) + if !ok { + continue // position-less: looked through from its operands + } + def = k + } + for _, use := range useSites(v, keyOf) { + add(def, use, v.Name()+" = "+v.String()) + } + } + } + sort.Slice(pairs, func(a, b int) bool { + x, y := pairs[a], pairs[b] + if x.Unit != y.Unit { + return x.Unit.Path+x.Unit.Signature < y.Unit.Path+y.Unit.Signature + } + if x.Def != y.Def { + return x.Def < y.Def + } + if x.Use != y.Use { + return x.Use < y.Use + } + return x.What < y.What + }) + return pairs, instrs, unmappedInstrs +} + +// useSites returns the body keys of v's referrers, looking through phis and +// position-less value instructions. +func useSites(v ssa.Value, keyOf func(ssa.Instruction) (string, bool)) []string { + var out []string + seen := map[ssa.Value]bool{} + var walk func(ssa.Value) + walk = func(v ssa.Value) { + if seen[v] { + return + } + seen[v] = true + refs := v.Referrers() + if refs == nil { + return + } + for _, r := range *refs { + if phi, ok := r.(*ssa.Phi); ok { + walk(phi) + continue + } + if st, ok := r.(*ssa.Store); ok && st.Addr == v && st.Val != v { + continue // a write through v, not a read of it + } + if writeOnlyAddr(r, v) { + continue // &v.f / &v[i] taken only to store through: a write + } + if k, ok := keyOf(r); ok { + out = append(out, k) + continue + } + if rv, ok := r.(ssa.Value); ok { + walk(rv) + } + } + } + walk(v) + return out +} + +// writeOnlyAddr reports whether r is a field or element address of v whose +// every referrer stores through it. +func writeOnlyAddr(r ssa.Instruction, v ssa.Value) bool { + var refs *[]ssa.Instruction + switch a := r.(type) { + case *ssa.FieldAddr: + if a.X != v { + return false + } + refs = a.Referrers() + case *ssa.IndexAddr: + if a.X != v { + return false + } + refs = a.Referrers() + default: + return false + } + if refs == nil || len(*refs) == 0 { + return false + } + for _, x := range *refs { + if st, ok := x.(*ssa.Store); !ok || st.Addr != r.(ssa.Value) { + return false + } + } + return true +} + +// Triage classes of an SSA pair with no direct L3 ddg edge. All but +// classGap are known differences of representation: the flow is in L3. +const ( + classTransitive = "transitive in L3 (SSA copy-propagates through copies, closure values and captured cells)" + classClause = "bound at a different node (switch tag / select comm / type-switch binding: header in one, clause in the other)" + classDefer = "deferred call: SSA evaluates at the registration, L3 reads at the deferred call node (two-node rule)" + classSlot = "SSA bookkeeping (anonymous result slot)" + classGap = "candidate gap: no L3 ddg path" +) + +// classify triages a mapped SSA pair p of result r that has no direct L3 +// ddg edge. +func classify(r dataflow.Result, p ssaPair) string { + succ := map[string][]string{} + for _, e := range r.Ddg { + succ[e.Src] = append(succ[e.Src], e.Dst) + } + reach := func(src, dst string) bool { + seen := map[string]bool{src: true} + for work := []string{src}; len(work) > 0; work = work[1:] { + for _, m := range succ[work[0]] { + if m == dst { + return true + } + if !seen[m] { + seen[m] = true + work = append(work, m) + } + } + } + return false + } + node := map[string]dataflow.Node{} + for _, n := range r.Nodes { + node[n.Key] = n + } + // header returns the innermost switch/select node holding node n. + header := func(n dataflow.Node) (dataflow.Node, bool) { + var best dataflow.Node + found := false + for _, h := range r.Nodes { + if h.Kind == dataflow.KindSwitch && h.Pos < n.Pos && n.End <= h.End && (!found || h.End-h.Pos < best.End-best.Pos) { + best, found = h, true + } + } + return best, found + } + clauses := func(h dataflow.Node) []dataflow.Node { + var out []dataflow.Node + for _, n := range r.Nodes { + switch n.Stmt.(type) { + case *ast.CaseClause, *ast.CommClause: + if hh, ok := header(n); ok && hh.Key == h.Key { + out = append(out, n) + } + } + } + return out + } + d, u := node[p.Def], node[p.Use] + switch s := u.Stmt.(type) { + case *ast.DeferStmt: + for _, n := range r.Nodes { + if n.Deferred && n.Pos == s.Call.Pos() && (p.Def == n.Key || reach(p.Def, n.Key)) { + return classDefer + } + } + case *ast.CaseClause, *ast.CommClause, *ast.SelectStmt: + h := u + if _, ok := u.Stmt.(*ast.SelectStmt); !ok { + h, _ = header(u) + } + if p.Def == h.Key || reach(p.Def, h.Key) { + return classClause + } + for _, c := range clauses(h) { + if p.Def == c.Key || reach(p.Def, c.Key) { + return classClause + } + } + } + // A value SSA binds at a header (a select's received tuple) or at a + // clause (a type switch's per-clause assertion) that L3 binds at the + // other. + if h, ok := header(d); ok && reach(h.Key, p.Use) { + return classClause + } + if d.Kind == dataflow.KindSwitch { + for _, c := range clauses(d) { + if reach(c.Key, p.Use) { + return classClause + } + } + } + if strings.Contains(p.What, " = local ") && strings.HasSuffix(p.What, " ()") { + return classSlot + } + if reach(p.Def, p.Use) { + return classTransitive + } + return classGap +} + +// TestSSADifferential_Report: on the fixtures, every project SSA function +// and call joins, and at least one SSA def-use pair maps to body keys; then +// it logs every mapped pair with no L3 ddg edge between the same two nodes +// (any variable), triaged (classify). Mismatches never fail the test. Set +// CANGO_SSA_DIFF_OUT= to also write the report there. +func TestSSADifferential_Report(t *testing.T) { + var report strings.Builder + for _, name := range []string{"dataflow", "chi"} { + in := loadInput(t, name) + _, j, funcs := joined(t, in) + // The mapping layer is complete: every function joins (but those of + // declarations the symbol table dropped, pinned by + // TestJoin_EveryProjectFunctionMaps) and no positioned call of a + // joined function fails to join. + dropped := droppedDecls(in) + for fn := range funcs { + if _, kind := j.resolve(fn); kind == joinNone && fn.Synthetic != synthPackageInit && !inDropped(fn, dropped) { + t.Errorf("%s: %s joins nothing", name, fn) + } + } + for _, b := range describeUnmapped(j, funcs) { + if strings.HasPrefix(b, "call ") { + t.Errorf("%s: unmapped %s", name, b) + } + } + pairs, instrs, unmapped := ssaPairs(j, funcs) + if len(pairs) == 0 { + t.Errorf("%s: no SSA def-use pair mapped to body keys", name) + } + if unmapped > 0 { + t.Errorf("%s: %d positioned SSA instructions of joined functions lie in no body node", name, unmapped) + } + has := map[[2]string]map[dataflow.L3Key]bool{} + for k, r := range in.L3 { + for _, e := range r.Ddg { + p := [2]string{e.Src, e.Dst} + if has[p] == nil { + has[p] = map[dataflow.L3Key]bool{} + } + has[p][k] = true + } + } + missing := 0 + classes := map[string]int{} + for _, p := range pairs { + if has[[2]string{p.Def, p.Use}][p.Unit] { + continue + } + missing++ + class := classify(in.L3[p.Unit], p) + classes[class]++ + fmt.Fprintf(&report, "%s\t%s\t%s\t%s -> %s\t%s\t%s\n", name, p.Unit.Path, p.Unit.Signature, p.Def, p.Use, p.What, class) + } + t.Logf("%s: %d SSA def-use pairs mapped (%d positioned instructions, %d outside every body node); %d with no L3 ddg edge", + name, len(pairs), instrs, unmapped, missing) + for _, c := range []string{classTransitive, classClause, classDefer, classSlot, classGap} { + if classes[c] > 0 { + t.Logf("%s: %4d %s", name, classes[c], c) + } + } + } + if report.Len() > 0 { + t.Logf("SSA def-use pairs with no L3 ddg edge (fixture, file, callable, def -> use, SSA value, class):\n%s", report.String()) + } + if out := os.Getenv("CANGO_SSA_DIFF_OUT"); out != "" { + if err := os.WriteFile(out, []byte(report.String()), 0o644); err != nil { + t.Error(err) + } + } +} diff --git a/internal/sdg/footprint.go b/internal/sdg/footprint.go new file mode 100644 index 0000000..efdd424 --- /dev/null +++ b/internal/sdg/footprint.go @@ -0,0 +1,904 @@ +package sdg + +import ( + "fmt" + "go/ast" + "go/token" + "go/types" + "slices" + "strings" + + "golang.org/x/tools/go/packages" + + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// Footprint is what a call target touches beyond its formals' values (spec +// §4.6, §4.9): the package-level variables it reads and writes, itself or +// through any callee, spelled "pkgpath::Name" (the L3 ddg base spelling), and +// the params and receiver it may write through (by-ref outs), by base name, +// or by the D6 `of` (`_`, `_recv`) for an unnamed one. Each list is sorted +// and deduplicated. Lists may be shared between footprints: callers must not +// modify them. +type Footprint struct { + GlobalReads, GlobalWrites []string + Mutated []string +} + +// globals is a callable's global read and write sets (sorted spellings). +type globals struct{ reads, writes []string } + +// footprints computes the footprint of every unit and of every dispatch +// target (spec §4.9). It reads the L3 side tables, the dispatch and go/types +// only, never the SSA program, so it is safe after a no_vta. +// +// - A unit with an L3 result (analysed): its direct globals are the package-level +// variable bases of its DefUse (uses are reads; strong and weak defs are +// writes; @entry, which defines every global the body touches, is not a +// write) and every global it takes the address of (&v, or a pointer +// method value v.M): a write through that address is not a def of v at +// L3. Globals the filter drops (globalFilter, A5) are removed. Its +// footprint is propagated bottom-up over the call graph (callGraph), so +// it holds every global any callee touches. +// - A unit footprints cannot analyse (an L3 result dropped, l3_skipped; +// no type info): its direct globals are syntactic — every package-level +// variable its Fn names, nested literals included, as both read and +// written (mentions) — and it keeps its callees (callGraph). +// - Mutated (A10, fix I1): for an analysed unit, a receiver or param p +// whose type may hold a reference (dataflow.MayHoldRef) and that some +// node other than @entry uses or defines. A use is enough: a write +// through a local alias (`q := p; q.x = v`, a range variable over p) or +// passing the alias on is no def of p at L3, and no alias analysis +// tells them apart. A by-ref formal never used stays out. It is not +// propagated: a caller passing p by ref uses p, and a closure writing +// through a captured p makes its host use p (captureUses), so the +// encloser's own DefUse records it. +// - A stand-in, an interface method, a bodiless declaration and an +// unanalysed unit: Mutated = every by-ref formal (byRefFormals); only +// the unanalysed unit has globals. +func footprints(in Input, d *dispatch) map[Target]Footprint { + nodes, callees := callGraph(in, d) + gf := newGlobalFilter(in) + formals := newFormalIndex(in, d) + + direct := map[Target]globals{} + mutated := map[Target][]string{} + for _, u := range in.Units { + t := Target{Callable: u.Key} + if !analysed(in, u) { + direct[t] = gf.mentions(u) // no DefUse: every global it names may be read or written + mutated[t] = byRefFormals(formals.of(t)) + continue + } + r := in.L3[u.Key] + direct[t] = gf.direct(u, r) + mutated[t] = mutatedFormals(u, r) + } + gf.initializers() + for t, g := range direct { + direct[t] = globals{reads: gf.keep(g.reads), writes: gf.keep(g.writes)} + } + + prop := propagate(nodes, callees, direct) + out := make(map[Target]Footprint, len(nodes)) + for _, t := range nodes { + if _, ok := direct[t]; ok { + g := prop[t] + out[t] = Footprint{GlobalReads: g.reads, GlobalWrites: g.writes, Mutated: mutated[t]} + continue + } + out[t] = Footprint{Mutated: byRefFormals(formals.of(t))} + } + return out +} + +// analysed reports whether unit u has the L3 facts footprints read: an L3 +// result with its DefUse side table, and type info. +func analysed(in Input, u dataflow.Unit) bool { + r, ok := in.L3[u.Key] + return ok && r.DefUse != nil && u.Info != nil +} + +// callGraph returns the call graph footprints propagate over (spec §4.9): +// every unit and every target, sorted by compareTarget, and each unit's +// callees, sorted: +// +// - the union of its call sites' dispatch targets (L2's static callee ∪ +// VTA ∪ the interface method, D7). A call node the CFG pruned keeps its +// targets (an over-approximation); +// - for an unanalysed unit dispatch seeded no site for (fix I2), its L2 +// call_graph edges (in.CallGraph): the callable of the edge's +// TargetPath when it has one (a repeated signature, such as one init +// closure per file, names the right file), else the target mapped as +// newDispatch maps a static callee; +// - every function literal of its body (outermost: a nested one is its +// enclosing literal's; fixes I3, N1). One handed to library code +// (once.Do, errgroup.Go, http.HandleFunc, sort.Slice) or stored may run +// during the call with no call edge to it. One called in place +// (`func(){…}()`, deferred or launched) has a call edge only from VTA +// (L2 resolves no FuncLit callee), so without the literal edge a +// no_vta run would lose its globals. This mirrors the L3 capture-write +// rule; Compact drops the duplicate when VTA's edge is there too. +func callGraph(in Input, d *dispatch) ([]Target, map[Target][]Target) { + callees := map[Target][]Target{} + nodeSet := map[Target]bool{} + byFn := make(map[ast.Node]dataflow.L3Key, len(in.Units)) + for _, u := range in.Units { + nodeSet[Target{Callable: u.Key}] = true + byFn[u.Fn] = u.Key + } + seeded := map[dataflow.L3Key]bool{} + for site, ts := range d.sites { + for _, t := range ts { + nodeSet[t] = true + } + seeded[site.unit] = true + u := Target{Callable: site.unit} + callees[u] = append(callees[u], ts...) + } + l2 := map[dataflow.L3Key][]schema.GoCallEdge{} + for _, e := range in.CallGraph { + k := dataflow.L3Key{Path: e.SourcePath, Signature: e.Source} + l2[k] = append(l2[k], e) + } + for _, u := range in.Units { + src := Target{Callable: u.Key} + if !seeded[u.Key] && !analysed(in, u) { + for _, e := range l2[u.Key] { + t, ok := Target{Callable: dataflow.L3Key{Path: e.TargetPath, Signature: e.Target}}, true + if e.TargetPath == "" { + t = Target{External: e.Target} + if _, ext := in.Externals[e.Target]; !ext { + var k dataflow.L3Key + k, ok = d.bySig[e.Target] + t = Target{Callable: k} + } + } + if ok { + nodeSet[t] = true + callees[src] = append(callees[src], t) + } + } + } + for _, lit := range astwalk.Literals(fnBody(u.Fn)) { + if k, ok := byFn[lit.Lit]; ok { + callees[src] = append(callees[src], Target{Callable: k}) + } + } + } + for u, ts := range callees { + slices.SortFunc(ts, compareTarget) + callees[u] = slices.Compact(ts) + } + nodes := make([]Target, 0, len(nodeSet)) + for t := range nodeSet { + nodes = append(nodes, t) + } + slices.SortFunc(nodes, compareTarget) + return nodes, callees +} + +// fnBody is the body of a *ast.FuncDecl or *ast.FuncLit (nil without one). +func fnBody(fn ast.Node) *ast.BlockStmt { + switch fn := fn.(type) { + case *ast.FuncDecl: + return fn.Body + case *ast.FuncLit: + return fn.Body + } + return nil +} + +// propagate closes the direct global sets over callees, bottom-up over the +// SCCs (callees first): a target's set is the union of the direct sets of +// every target it reaches, itself included. That is the least fixpoint of +// "a target's set ⊇ its own ∪ each callee's"; every member of an SCC reaches +// every other, so the fixpoint gives all members one set, the union of the +// members' direct sets and of the sets of the SCCs they call (computed +// earlier), and one pass per SCC reaches it. When the union adds nothing to +// one contribution, that contribution's slice is shared rather than copied. +func propagate(nodes []Target, callees map[Target][]Target, direct map[Target]globals) map[Target]globals { + comps := sccs(nodes, func(t Target) []Target { return callees[t] }) + compOf := make(map[Target]int, len(nodes)) + out := make(map[Target]globals, len(nodes)) + for ci, comp := range comps { + for _, m := range comp { + compOf[m] = ci + } + var reads, writes [][]string + for _, m := range comp { + if g, ok := direct[m]; ok { + reads, writes = append(reads, g.reads), append(writes, g.writes) + } + for _, c := range callees[m] { + if compOf[c] != ci { + g := out[c] + reads, writes = append(reads, g.reads), append(writes, g.writes) + } + } + } + g := globals{reads: unionSorted(reads), writes: unionSorted(writes)} + for _, m := range comp { + out[m] = g + } + } + return out +} + +// unionSorted is the sorted union of the sorted, deduplicated lists. It +// returns the largest list itself when the union is no larger, and nil when +// it is empty. +func unionSorted(lists [][]string) []string { + var largest []string + n := 0 + for _, l := range lists { + if len(l) > len(largest) { + largest = l + } + n += len(l) + } + if n == len(largest) { + if len(largest) == 0 { + return nil + } + return largest + } + all := make([]string, 0, n) + for _, l := range lists { + all = append(all, l...) + } + slices.Sort(all) + all = slices.Compact(all) + if len(all) == len(largest) { + return largest + } + return slices.Clip(all) +} + +// mutatedFormals is the Mutated set of analysed unit u from its L3 result r +// (A10, fix I1): the receiver and params that may hold a reference and that +// a node other than @entry uses or defines, by name, sorted. +func mutatedFormals(u dataflow.Unit, r dataflow.Result) []string { + byRef := map[types.Object]bool{} + for _, obj := range formalObjs(u.Fn, u.Info) { + if dataflow.MayHoldRef(obj.Type()) { + byRef[obj] = true + } + } + if len(byRef) == 0 { + return nil + } + hit := map[string]bool{} + for key, du := range r.DefUse { + if key == dataflow.EntryKey { + continue + } + for _, list := range [][]dataflow.Path{du.Uses, du.Defs, du.WeakDefs} { + for _, p := range list { + if byRef[p.Base] { + hit[p.Base.Name()] = true + } + } + } + } + return sortedKeys(hit) +} + +// formalObjs returns the receiver and parameter objects of fn (a +// *ast.FuncDecl or *ast.FuncLit), in declaration order; "_" and unnamed ones +// have none. +func formalObjs(fn ast.Node, info *types.Info) []types.Object { + var lists []*ast.FieldList + switch fn := fn.(type) { + case *ast.FuncDecl: + lists = []*ast.FieldList{fn.Recv, fn.Type.Params} + case *ast.FuncLit: + lists = []*ast.FieldList{fn.Type.Params} + } + var out []types.Object + for _, fl := range lists { + if fl == nil { + continue + } + for _, f := range fl.List { + for _, name := range f.Names { + if obj := info.Defs[name]; obj != nil && name.Name != "_" { + out = append(out, obj) + } + } + } + } + return out +} + +// globalFilter is the A5 global filter (spec §4.6): a package-level +// variable is kept iff it is declared outside the project (os::Args), or +// some project code writes it — a strong or weak def in any unit (init and +// closures included), &v or a pointer method value, a write in a unit L3 +// could not analyse, or a write in a package-level initializer — or its +// initializer contains a call. +type globalFilter struct { + fset *token.FileSet + pkgs []*packages.Package + project map[string]bool // import paths of the project packages + kept map[string]bool // project globals that pass, by spelling + spellings map[types.Object]string +} + +func newGlobalFilter(in Input) *globalFilter { + f := &globalFilter{ + fset: in.Fset, + pkgs: projectPackages(in), + project: map[string]bool{}, + kept: map[string]bool{}, + spellings: map[types.Object]string{}, + } + for _, p := range f.pkgs { + f.project[p.PkgPath] = true + } + return f +} + +// spell returns the spelling of obj when it is a package-level variable. +func (f *globalFilter) spell(obj types.Object) (string, bool) { + if s, ok := f.spellings[obj]; ok { + return s, s != "" + } + s := "" + if dataflow.IsPkgVar(obj) { + s = dataflow.BaseSpell(obj) + } + f.spellings[obj] = s + return s, s != "" +} + +// write records a write of obj when it is a project global. +func (f *globalFilter) write(obj types.Object) { + if s, ok := f.spell(obj); ok && f.project[obj.Pkg().Path()] { + f.kept[s] = true + } +} + +// keep returns the globals of list (sorted spellings) the filter keeps. +func (f *globalFilter) keep(list []string) []string { + var out []string + for _, s := range list { + pkg, _, _ := strings.Cut(s, "::") + if !f.project[pkg] || f.kept[s] { + out = append(out, s) + } + } + return out +} + +// direct returns the unfiltered direct globals of unit u (see footprints) +// and records its writes of project globals. +func (f *globalFilter) direct(u dataflow.Unit, r dataflow.Result) globals { + reads, writes := map[string]bool{}, map[string]bool{} + read := func(obj types.Object) { + if s, ok := f.spell(obj); ok { + reads[s] = true + } + } + write := func(obj types.Object) { + if s, ok := f.spell(obj); ok { + writes[s] = true + f.write(obj) + } + } + for key, du := range r.DefUse { + if key == dataflow.EntryKey { + continue + } + for _, p := range du.Uses { + read(p.Base) + } + for _, list := range [][]dataflow.Path{du.Defs, du.WeakDefs} { + for _, p := range list { + write(p.Base) + } + } + } + addressTaken(u.Fn, u.Info, write) + return globals{reads: sortedKeys(reads), writes: sortedKeys(writes)} +} + +// addressTaken calls f on the base of every &x and of every pointer method +// value x.M (a call x.M() included) whose receiver x is not a pointer: the +// address may be written through. It walks fn's body, not its function +// literals: they are their own units. +func addressTaken(fn ast.Node, info *types.Info, f func(types.Object)) { + body := fnBody(fn) + if body == nil { + return + } + base := func(e ast.Expr) { + if p, ok := dataflow.PathOf(e, info, 1); ok && p.Base != nil { + f(p.Base) + } + } + ast.Inspect(body, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.FuncLit: + return false + case *ast.UnaryExpr: + if n.Op == token.AND { + base(n.X) + } + case *ast.SelectorExpr: + sel := info.Selections[n] + if sel == nil || sel.Kind() != types.MethodVal { + break + } + if m, ok := sel.Obj().(*types.Func); ok { + if recv := m.Signature().Recv(); recv != nil && isPtr(recv.Type()) && !isPtr(info.TypeOf(n.X)) { + base(n.X) + } + } + } + return true + }) +} + +func isPtr(t types.Type) bool { + if t == nil { + return false + } + _, ok := t.Underlying().(*types.Pointer) + return ok +} + +// mentions returns the syntactic direct globals of unanalysed unit u (fix +// I2): every package-level variable its Fn names, nested literals included, +// as both read and written (with no DefUse, any of them may be either), and +// records the writes. With no type info nothing resolves a name, so it is +// every package-level variable of u's package. +func (f *globalFilter) mentions(u dataflow.Unit) globals { + set := map[string]bool{} + note := func(obj types.Object) { + if s, ok := f.spell(obj); ok { + set[s] = true + f.write(obj) + } + } + if u.Info == nil { + f.packageVars(u, note) + } else { + ast.Inspect(u.Fn, func(n ast.Node) bool { + if id, ok := n.(*ast.Ident); ok { + note(u.Info.Uses[id]) + } + return true + }) + } + all := sortedKeys(set) + return globals{reads: all, writes: all} +} + +// packageVars calls note on every object of the scope of the project +// package that declares u. +func (f *globalFilter) packageVars(u dataflow.Unit, note func(types.Object)) { + tf := f.fset.File(u.Fn.Pos()) + if tf == nil { + return + } + for _, p := range f.pkgs { + if !slices.Contains(p.GoFiles, tf.Name()) || p.Types == nil { + continue + } + scope := p.Types.Scope() + for _, name := range scope.Names() { + note(scope.Lookup(name)) + } + } +} + +// initializers applies the filter's rules to the package-level variable +// declarations of every project package: a variable whose initializer +// contains a call (not a conversion or a builtin, and not in a function +// literal's body: that runs only when the literal is called) is kept; so is +// every project global an initializer may write — one named inside a call's +// operands, under &, or in a function literal (L1 has no callable for a +// package-level literal, so no DefUse says what it writes). Last, a chained +// initializer (fix M4): a variable whose initializer names a library +// package-level variable or a kept one is kept, to a fixpoint (`var argv = +// os.Args`; `var cfgDir = home + "/x"` with home kept). +func (f *globalFilter) initializers() { + type chain struct{ names, named []types.Object } + var chains []chain + for _, p := range f.pkgs { + info := p.TypesInfo + if info == nil { + continue + } + for _, file := range p.Syntax { + for _, decl := range file.Decls { + gd, ok := decl.(*ast.GenDecl) + if !ok || gd.Tok != token.VAR { + continue + } + for _, spec := range gd.Specs { + vs := spec.(*ast.ValueSpec) + c := chain{} + for _, name := range vs.Names { + c.names = append(c.names, info.Defs[name]) + } + for _, v := range vs.Values { + if hasCall(v, info) { + for _, obj := range c.names { + f.write(obj) + } + } + f.initWrites(v, info, false) + c.named = append(c.named, namedVars(v, info)...) + } + if len(c.named) > 0 { + chains = append(chains, c) + } + } + } + } + } + for changed := true; changed; { + changed = false + for _, c := range chains { + if !slices.ContainsFunc(c.named, f.carries) || !slices.ContainsFunc(c.names, f.dropped) { + continue + } + for _, obj := range c.names { + f.write(obj) + } + changed = true + } + } +} + +// carries reports whether package-level variable obj passes the filter as +// it stands: a library variable, or a kept project one. +func (f *globalFilter) carries(obj types.Object) bool { + s, ok := f.spell(obj) + return ok && (!f.project[obj.Pkg().Path()] || f.kept[s]) +} + +// dropped reports whether obj is a project package-level variable the +// filter does not keep (yet). +func (f *globalFilter) dropped(obj types.Object) bool { + s, ok := f.spell(obj) + return ok && f.project[obj.Pkg().Path()] && !f.kept[s] +} + +// namedVars returns the package-level variables e names outside a function +// literal's body (a literal's body runs only when it is called). +func namedVars(e ast.Expr, info *types.Info) []types.Object { + var out []types.Object + ast.Inspect(e, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.FuncLit: + return false + case *ast.Ident: + if obj := info.Uses[n]; dataflow.IsPkgVar(obj) { + out = append(out, obj) + } + } + return true + }) + return out +} + +// hasCall reports whether e calls a function outside a function literal's +// body: a conversion or a builtin call does not count, but its operands are +// searched. +func hasCall(e ast.Expr, info *types.Info) bool { + found := false + ast.Inspect(e, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.FuncLit: + return false + case *ast.CallExpr: + if tv, ok := info.Types[n.Fun]; !ok || !(tv.IsType() || tv.IsBuiltin()) { + found = true + } + } + return !found + }) + return found +} + +// initWrites records as written every project global n names in a hot +// position: inside a call (its function value and operands), under &, or in +// a function literal. +func (f *globalFilter) initWrites(n ast.Node, info *types.Info, hot bool) { + ast.Inspect(n, func(m ast.Node) bool { + switch m := m.(type) { + case *ast.Ident: + if hot { + f.write(info.Uses[m]) + } + case *ast.CallExpr: + if !hot { + f.initWrites(m, info, true) + return false + } + case *ast.FuncLit: + f.initWrites(m.Body, info, true) + return false + case *ast.UnaryExpr: + if m.Op == token.AND && !hot { + f.initWrites(m.X, info, true) + return false + } + } + return true + }) +} + +// formal is one receiver or declared parameter of a call target: its D6 +// `of` name and its type (nil when only the type's string is known). +type formal struct { + name string + typ types.Type + typStr string + recv bool + variadic bool +} + +// byRefFormals returns the names of the formals that are by-ref (the shared +// predicate dataflow.MayHoldRef on a known type, else refTypeString), +// sorted. +func byRefFormals(fs []formal) []string { + var out []string + for _, f := range fs { + if f.typ != nil && dataflow.MayHoldRef(f.typ) || f.typ == nil && (f.variadic || refTypeString(f.typStr)) { + out = append(out, f.name) + } + } + slices.Sort(out) + return out +} + +// refTypeString is the by-ref predicate on a type known only by its string +// (spec §4.6, the conservative fallback): false only for a basic type, or an +// array of one; a named type that cannot be resolved may hold a reference. +func refTypeString(s string) bool { + for strings.HasPrefix(s, "[") { + i := strings.IndexByte(s, ']') + if i <= 1 { // "[]T": a slice + return true + } + s = s[i+1:] + } + if b, ok := types.Universe.Lookup(s).(*types.TypeName); ok { + if _, basic := b.Type().(*types.Basic); basic { + return false + } + } + return true +} + +// formalIndex finds the formals of call targets, with their go/types types +// wherever the load has them: +// +// - a unit: its Fn's signature (types.Info); +// - an interface method or a bodiless declaration (no unit): the +// *types.Func a project package's scope declares under that key; +// - an external: the names from its GoExternalCallee (already D6), the +// types from the *types.Func its import path's package declares, found +// through the load's import graph (every external is reached from +// project code, so its package is loaded, from source or export data). +// +// When no type is found, the symbol table's or the external's type strings +// are used with refTypeString. +type formalIndex struct { + in Input + units map[dataflow.L3Key]dataflow.Unit + decls map[dataflow.L3Key]*types.Func // project funcs and methods by (declaring file, signature) + strs map[dataflow.L3Key]schema.GoCallable + ext map[string]schema.GoExternalCallee + pkgByPath map[string]*types.Package +} + +func newFormalIndex(in Input, d *dispatch) *formalIndex { + x := &formalIndex{ + in: in, + units: make(map[dataflow.L3Key]dataflow.Unit, len(in.Units)), + decls: map[dataflow.L3Key]*types.Func{}, + strs: map[dataflow.L3Key]schema.GoCallable{}, + ext: map[string]schema.GoExternalCallee{}, + pkgByPath: map[string]*types.Package{}, + } + for _, u := range in.Units { + x.units[u.Key] = u + } + for sig, e := range in.Externals { + x.ext[sig] = e + } + for sig, e := range d.newExt { + x.ext[sig] = e + } + var walk func(path string, c schema.GoCallable) + walk = func(path string, c schema.GoCallable) { + x.strs[dataflow.L3Key{Path: path, Signature: c.Signature}] = c + for _, ic := range c.InnerCallables { + walk(ic.Path, ic) + } + } + for path, f := range in.SymbolTable { + for _, c := range f.Functions { + walk(c.Path, c) + } + for _, t := range f.Types { + for _, c := range t.Methods { + if c.Path == "" { // an interface method, keyed by its interface's file + walk(path, c) + continue + } + walk(c.Path, c) + } + } + } + for _, p := range projectPackages(in) { + if p.Types == nil { + continue + } + scope := p.Types.Scope() + for _, name := range scope.Names() { + switch obj := scope.Lookup(name).(type) { + case *types.Func: + x.addDecl(obj) + case *types.TypeName: + named, ok := obj.Type().(*types.Named) + if !ok { + continue + } + for i := 0; i < named.NumMethods(); i++ { + x.addDecl(named.Method(i)) + } + if it, ok := named.Underlying().(*types.Interface); ok { + for i := 0; i < it.NumExplicitMethods(); i++ { + x.addDecl(it.ExplicitMethod(i)) + } + } + } + } + } + var visit func(p *packages.Package) + visit = func(p *packages.Package) { + if p == nil || p.Types == nil || x.pkgByPath[p.PkgPath] != nil { + return + } + x.pkgByPath[p.PkgPath] = p.Types + for _, q := range p.Imports { + visit(q) + } + } + for _, p := range in.Pkgs { + visit(p) + } + return x +} + +func (x *formalIndex) addDecl(fn *types.Func) { + if k, ok := declKey(x.in.Fset, x.in.ProjectDir, fn); ok { + x.decls[k] = fn + } +} + +// of returns t's receiver (first) and declared params. +func (x *formalIndex) of(t Target) []formal { + if t.External != "" { + return x.external(t.External) + } + if u, ok := x.units[t.Callable]; ok && u.Info != nil { + if sig := unitSignature(u); sig != nil { + return sigFormals(sig) + } + } + if fn, ok := x.decls[t.Callable]; ok { + return sigFormals(fn.Signature()) + } + c, ok := x.strs[t.Callable] + if !ok { + utils.Debug("L4: no formals for %v", t) + return nil + } + var out []formal + if c.ReceiverType != "" { + out = append(out, formal{name: recvName(c.ReceiverName), typStr: c.ReceiverType, recv: true}) + } + for i, p := range c.Parameters { + out = append(out, formal{name: paramName(p.Name, i), typStr: p.Type, variadic: p.IsVariadic}) + } + return out +} + +// external returns the formals of the external with full signature sig. +func (x *formalIndex) external(sig string) []formal { + e, ok := x.ext[sig] + if !ok { + utils.Debug("L4: no external %s", sig) + return nil + } + var out []formal + if e.Receiver != nil { + out = append(out, formal{name: e.Receiver.Name, typStr: e.Receiver.Type, recv: true}) + } + for _, p := range e.Params { + out = append(out, formal{name: p.Name, typStr: p.Type, variadic: p.IsVariadic}) + } + if fn := x.libraryFunc(e); fn != nil { + if typed := sigFormals(fn.Signature()); len(typed) == len(out) { + for i := range out { + out[i].typ = typed[i].typ + } + } + } + return out +} + +// libraryFunc looks up e's *types.Func: a function of its package's scope, +// or a method (Sig "Type.Method") of a type it declares. +func (x *formalIndex) libraryFunc(e schema.GoExternalCallee) *types.Func { + pkg := x.pkgByPath[e.ImportPath] + if pkg == nil { + return nil + } + typ, method, isMethod := strings.Cut(e.Sig, ".") + if !isMethod { + fn, _ := pkg.Scope().Lookup(e.Sig).(*types.Func) + return fn + } + tn, ok := pkg.Scope().Lookup(typ).(*types.TypeName) + if !ok { + return nil + } + obj, _, _ := types.LookupFieldOrMethod(tn.Type(), true, pkg, method) + fn, _ := obj.(*types.Func) + return fn +} + +// unitSignature is the signature of u's Fn. +func unitSignature(u dataflow.Unit) *types.Signature { + switch fn := u.Fn.(type) { + case *ast.FuncDecl: + if f, ok := u.Info.Defs[fn.Name].(*types.Func); ok { + return f.Signature() + } + case *ast.FuncLit: + sig, _ := u.Info.TypeOf(fn).(*types.Signature) + return sig + } + return nil +} + +// sigFormals returns sig's receiver (an interface method's receiver is its +// interface) and params, named by D6. +func sigFormals(sig *types.Signature) []formal { + var out []formal + if r := sig.Recv(); r != nil { + out = append(out, formal{name: recvName(r.Name()), typ: r.Type(), recv: true}) + } + for i := 0; i < sig.Params().Len(); i++ { + p := sig.Params().At(i) + out = append(out, formal{name: paramName(p.Name(), i), typ: p.Type(), variadic: sig.Variadic() && i == sig.Params().Len()-1}) + } + return out +} + +// recvName and paramName are the D6 `of` of a receiver and of declared +// param i (receiver excluded). +func recvName(name string) string { + if name == "" || name == "_" { + return "_recv" + } + return name +} + +func paramName(name string, i int) string { + if name == "" || name == "_" { + return fmt.Sprintf("_%d", i) + } + return name +} diff --git a/internal/sdg/footprint_test.go b/internal/sdg/footprint_test.go new file mode 100644 index 0000000..fe80df7 --- /dev/null +++ b/internal/sdg/footprint_test.go @@ -0,0 +1,355 @@ +package sdg + +import ( + "math/rand" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" +) + +const configSpell = sdgPkg + "::Config" + +// globalsKey is the L3Key of a callable declared in +// testdata/dataflow/sdg/globals.go. +func globalsKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("sdg", "globals.go"), Signature: sdgPkg + "." + name} +} + +func runFootprints(t *testing.T) (Input, *dispatch, map[Target]Footprint) { + t.Helper() + in, _, d := runDispatch(t) + return in, d, footprints(in, d) +} + +func fpOf(t *testing.T, fps map[Target]Footprint, tg Target) Footprint { + t.Helper() + fp, ok := fps[tg] + if !ok { + t.Fatalf("no footprint for %v", tg) + } + return fp +} + +// TestFootprint_TransitiveGlobalReads: useConfig never names Config, but it +// calls readConfig, which reads it; initConfig writes it. +func TestFootprint_TransitiveGlobalReads(t *testing.T) { + _, _, fps := runFootprints(t) + want := []string{configSpell} + if got := fpOf(t, fps, sdgTarget("useConfig")); !slices.Equal(got.GlobalReads, want) || len(got.GlobalWrites) != 0 { + t.Errorf("useConfig = %+v, want reads %v, no writes", got, want) + } + if got := fpOf(t, fps, sdgTarget("initConfig")); !slices.Equal(got.GlobalWrites, want) || len(got.GlobalReads) != 0 { + t.Errorf("initConfig = %+v, want writes %v, no reads", got, want) + } +} + +// TestFootprint_GlobalFilter (A5): constOnly (literal initializer, never +// written) is dropped everywhere; os::Args (a library variable) is kept, and +// so are home (its initializer calls os.Getenv) and counter (written only +// through &counter). +func TestFootprint_GlobalFilter(t *testing.T) { + _, _, fps := runFootprints(t) + for tg, fp := range fps { + for _, g := range slices.Concat(fp.GlobalReads, fp.GlobalWrites) { + if strings.HasSuffix(g, "::constOnly") { + t.Errorf("%v has filtered global %s", tg, g) + } + } + } + if got := fpOf(t, fps, sdgTarget("readConst")); len(got.GlobalReads) != 0 { + t.Errorf("readConst reads %v, want none", got.GlobalReads) + } + for _, c := range []struct { + fn string + want string + }{ + {"readArgs", "os::Args"}, + {"readHome", sdgPkg + "::home"}, + {"readCounter", sdgPkg + "::counter"}, + } { + if got := fpOf(t, fps, Target{Callable: globalsKey(c.fn)}); !slices.Equal(got.GlobalReads, []string{c.want}) { + t.Errorf("%s reads %v, want [%s]", c.fn, got.GlobalReads, c.want) + } + } +} + +// TestFootprint_MutualRecursionFixpoint: readConfig and useConfig are one +// SCC and agree on its footprint. On a synthetic graph where the fact sits on +// the SCC member sorted last, only a fixpoint gives it to the first one (and +// to a caller of the SCC). +func TestFootprint_MutualRecursionFixpoint(t *testing.T) { + in, d, fps := runFootprints(t) + nodes, callees := callGraph(in, d) + comps := sccs(nodes, func(tg Target) []Target { return callees[tg] }) + want := []Target{sdgTarget("readConfig"), sdgTarget("useConfig")} + if !slices.ContainsFunc(comps, func(c []Target) bool { return slices.Equal(c, want) }) { + t.Errorf("no SCC %v", want) + } + for _, tg := range want { + if got := fpOf(t, fps, tg); !slices.Equal(got.GlobalReads, []string{configSpell}) { + t.Errorf("%v reads %v, want [%s]", tg, got.GlobalReads, configSpell) + } + } + + a, b, c := sdgTarget("a"), sdgTarget("b"), sdgTarget("c") + syn := map[Target][]Target{a: {b}, b: {a}, c: {a}} + direct := map[Target]globals{b: {reads: []string{"p::G"}, writes: []string{"p::H"}}} + got := propagate([]Target{c, b, a}, syn, direct) + for _, tg := range []Target{a, b, c} { + if g := got[tg]; !slices.Equal(g.reads, []string{"p::G"}) || !slices.Equal(g.writes, []string{"p::H"}) { + t.Errorf("%v = %+v, want reads [p::G], writes [p::H]", tg, g) + } + } +} + +// TestFootprint_Mutated (A10, fix I1): a callable mutates a by-ref param or +// receiver (the shared predicate) it uses or writes; a stand-in, an interface +// method and a bodiless declaration mutate every by-ref formal. +func TestFootprint_Mutated(t *testing.T) { + _, _, fps := runFootprints(t) + for _, c := range []struct { + tg Target + want []string + }{ + {sdgTarget("mutate"), []string{"p", "q", "s"}}, // n is a copy holding no reference; q is used (`_ = q`) + {sdgTarget("sqlStore.setDSN"), []string{"s"}}, + {sdgTarget("memStore.Get"), []string{"s"}}, // s holds a map and is used: a use is enough (I1) + {sdgTarget("Store.Get"), []string{"_recv"}}, // interface method: the receiver is an interface + {Target{External: "io.Writer.Write"}, []string{"_recv", "p"}}, + {Target{External: "bytes.Buffer.Write"}, []string{"b", "p"}}, + {Target{Callable: dataflow.L3Key{Path: filepath.Join("asm", "asm.go"), Signature: "example.com/dataflow/asm.Add"}}, nil}, + } { + if got := fpOf(t, fps, c.tg); !slices.Equal(got.Mutated, c.want) { + t.Errorf("%v mutates %v, want %v", c.tg, got.Mutated, c.want) + } + } +} + +// TestFootprint_L3SkippedMutatesEveryByRefFormal: with its L3 result gone, +// mutate has no direct globals and mutates every by-ref formal (p, s and q; +// n holds no reference). +func TestFootprint_L3SkippedMutatesEveryByRefFormal(t *testing.T) { + in, _, d := runDispatch(t) + l3 := make(map[dataflow.L3Key]dataflow.Result, len(in.L3)) + for k, r := range in.L3 { + if k != sdgKey("mutate") { + l3[k] = r + } + } + in.L3 = l3 + got := fpOf(t, footprints(in, d), sdgTarget("mutate")) + if want := []string{"p", "q", "s"}; !slices.Equal(got.Mutated, want) || len(got.GlobalReads)+len(got.GlobalWrites) != 0 { + t.Errorf("l3-skipped mutate = %+v, want mutated %v, no globals", got, want) + } +} + +// TestSDG_GenericInstancesShareOneCallable (Review Focus 2): the join maps a +// generic instance to its origin, so testdata/dataflow/generic has one +// footprint per origin callable and none per instance. +func TestSDG_GenericInstancesShareOneCallable(t *testing.T) { + in, _, fps := runFootprints(t) + units := map[dataflow.L3Key]bool{} + for _, u := range in.Units { + if strings.HasPrefix(u.Key.Path, "generic"+string(filepath.Separator)) { + units[u.Key] = true + } + } + got := map[dataflow.L3Key]bool{} + for tg := range fps { + if strings.HasPrefix(tg.Callable.Path, "generic"+string(filepath.Separator)) { + got[tg.Callable] = true + } + } + if len(units) == 0 || len(got) != len(units) { + t.Fatalf("generic footprint keys = %v, want the %d units %v", got, len(units), units) + } + for k := range got { + if !units[k] || strings.Contains(k.Signature, "[") { + t.Errorf("footprint key %v is not an origin unit", k) + } + } + twice := dataflow.L3Key{Path: filepath.Join("generic", "b.go"), Signature: "example.com/dataflow/generic.Twice"} + if !units[twice] { + t.Fatalf("no unit %v", twice) + } + storeA := fpOf(t, fps, Target{Callable: dataflow.L3Key{Path: filepath.Join("generic", "a.go"), Signature: "example.com/dataflow/generic.StoreA"}}) + if want := []string{"example.com/dataflow/generic::pa"}; !slices.Equal(storeA.GlobalWrites, want) { + t.Errorf("StoreA writes %v, want %v (pa.Store has a pointer receiver)", storeA.GlobalWrites, want) + } +} + +// TestSCC_Deterministic: shuffling the node order and every successor list +// gives the same SCCs, callees first, each sorted. +func TestSCC_Deterministic(t *testing.T) { + n := func(s string) Target { return Target{External: s} } + edges := map[Target][]Target{ + n("a"): {n("b")}, + n("b"): {n("c"), n("e")}, + n("c"): {n("a"), n("d")}, + n("d"): {}, + n("e"): {n("f")}, + n("f"): {n("e"), n("g")}, + n("g"): {n("g")}, + n("h"): {n("a"), n("x")}, // x is reachable only: not in nodes + } + var nodes []Target + for k := range edges { + nodes = append(nodes, k) + } + slices.SortFunc(nodes, compareTarget) + want := sccs(nodes, func(tg Target) []Target { return edges[tg] }) + wantStr := [][]Target{{n("d")}, {n("g")}, {n("e"), n("f")}, {n("a"), n("b"), n("c")}, {n("x")}, {n("h")}} + if !slices.EqualFunc(want, wantStr, slices.Equal) { + t.Fatalf("sccs = %v, want %v", want, wantStr) + } + rng := rand.New(rand.NewSource(1)) + for i := 0; i < 50; i++ { + shuf := slices.Clone(nodes) + rng.Shuffle(len(shuf), func(i, j int) { shuf[i], shuf[j] = shuf[j], shuf[i] }) + se := map[Target][]Target{} + for k, v := range edges { + v = slices.Clone(v) + rng.Shuffle(len(v), func(i, j int) { v[i], v[j] = v[j], v[i] }) + se[k] = v + } + if got := sccs(shuf, func(tg Target) []Target { return se[tg] }); !slices.EqualFunc(got, want, slices.Equal) { + t.Fatalf("shuffle %d: sccs = %v, want %v", i, got, want) + } + } +} + +// TestFootprint_MutatedThroughAlias (fix I1): a by-ref formal used at any +// node other than @entry is mutated — a write through a local alias (a range +// variable, q := p) or passing the alias on is not a def of the formal at L3. +// One never used is not. +func TestFootprint_MutatedThroughAlias(t *testing.T) { + _, _, fps := runFootprints(t) + for _, c := range []struct { + fn string + want []string + }{ + {"aliasRange", []string{"s"}}, + {"aliasAssign", []string{"p"}}, + {"aliasCall", []string{"p"}}, + {"unusedRef", nil}, + } { + if got := fpOf(t, fps, sdgTarget(c.fn)); !slices.Equal(got.Mutated, c.want) { + t.Errorf("%s mutates %v, want %v", c.fn, got.Mutated, c.want) + } + } +} + +// TestFootprint_L3SkippedKeepsGlobalsAndCallees (fix I2): with skipX's L3 +// result gone, skipX still has the global it names (g1, as read and written) +// and its callee skipY's (g2), and so does its caller. +func TestFootprint_L3SkippedKeepsGlobalsAndCallees(t *testing.T) { + in, _, d := runDispatch(t) + skipped := globalsKey("skipX") + l3 := make(map[dataflow.L3Key]dataflow.Result, len(in.L3)) + for k, r := range in.L3 { + if k != skipped { + l3[k] = r + } + } + in.L3 = l3 + fps := footprints(in, d) + g1, g2 := sdgPkg+"::g1", sdgPkg+"::g2" + for _, k := range []dataflow.L3Key{skipped, globalsKey("callsSkipX")} { + got := fpOf(t, fps, Target{Callable: k}) + if !slices.Equal(got.GlobalReads, []string{g1, g2}) || !slices.Contains(got.GlobalWrites, g1) { + t.Errorf("%s = %+v, want reads [%s %s] and writes ⊇ [%s]", k.Signature, got, g1, g2, g1) + } + } + + // With no dispatch sites either (a call-count mismatch leaves + // newDispatch nothing for it), the L2 call graph still gives its callees. + for site := range d.sites { + if site.unit == skipped { + delete(d.sites, site) + } + } + got := fpOf(t, footprints(in, d), Target{Callable: globalsKey("callsSkipX")}) + if !slices.Contains(got.GlobalReads, g2) { + t.Errorf("callsSkipX reads %v without skipX's dispatch sites, want ⊇ [%s]", got.GlobalReads, g2) + } +} + +// TestFootprint_LiteralPassedToLibrary (fix I3): Get hands a literal to +// once.Do, which never calls it, so no call edge reaches the literal; the +// encloser → literal edge still gives Get and its caller the literal's write +// of inst. +func TestFootprint_LiteralPassedToLibrary(t *testing.T) { + _, _, fps := runFootprints(t) + inst := sdgPkg + "::inst" + for _, fn := range []string{"Get", "callsGet"} { + if got := fpOf(t, fps, Target{Callable: globalsKey(fn)}); !slices.Contains(got.GlobalWrites, inst) { + t.Errorf("%s writes %v, want ⊇ [%s]", fn, got.GlobalWrites, inst) + } + } +} + +// TestFootprint_ChainedInitializers (fix M4): a global initialized from a +// library variable (argv = os.Args) or from a kept global (cfgDir = home + +// "/x") is kept. +func TestFootprint_ChainedInitializers(t *testing.T) { + _, _, fps := runFootprints(t) + for _, c := range []struct{ fn, want string }{ + {"readArgv", sdgPkg + "::argv"}, + {"readCfgDir", sdgPkg + "::cfgDir"}, + } { + if got := fpOf(t, fps, Target{Callable: globalsKey(c.fn)}); !slices.Equal(got.GlobalReads, []string{c.want}) { + t.Errorf("%s reads %v, want [%s]", c.fn, got.GlobalReads, c.want) + } + } +} + +// TestFootprint_CalledLiteralsWithoutVTA (fix N1): with VTA failed, only VTA +// would give a call edge to a literal called in place (L2 resolves no +// FuncLit callee), so the encloser → literal edge must cover every literal: +// calledLits writes litG (`func(){…}()`) and litH (`defer func(){…}()`). +func TestFootprint_CalledLiteralsWithoutVTA(t *testing.T) { + in := loadInput(t, "dataflow") + defer func(f func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph) { vtaCallGraph = f }(vtaCallGraph) + vtaCallGraph = func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph { panic("vta boom") } + _, d := run(in) + got := fpOf(t, footprints(in, d), Target{Callable: globalsKey("calledLits")}) + if want := []string{sdgPkg + "::litG", sdgPkg + "::litH"}; !slices.Equal(got.GlobalWrites, want) { + t.Errorf("calledLits writes %v under no_vta, want %v", got.GlobalWrites, want) + } +} + +// TestFootprint_L2FallbackHonoursTargetPath (fix N2): an L2 edge with a +// TargetPath maps to that file's callable, not to the lowest-path twin of a +// repeated signature (both files' init closures are +// initclosure.init.closure_1). The edge is synthetic, from an L3-skipped +// unit with no dispatch sites. +func TestFootprint_L2FallbackHonoursTargetPath(t *testing.T) { + in, _, d := runDispatch(t) + const pkg = "example.com/dataflow/initclosure" + src := globalsKey("callsSkipX") + b := dataflow.L3Key{Path: filepath.Join("initclosure", "b.go"), Signature: pkg + ".init.closure_1"} + l3 := make(map[dataflow.L3Key]dataflow.Result, len(in.L3)) + for k, r := range in.L3 { + if k != src { + l3[k] = r + } + } + in.L3 = l3 + for site := range d.sites { + if site.unit == src { + delete(d.sites, site) + } + } + in.CallGraph = []schema.GoCallEdge{{Source: src.Signature, SourcePath: src.Path, Target: b.Signature, TargetPath: b.Path}} + _, callees := callGraph(in, d) + if got, want := callees[Target{Callable: src}], []Target{{Callable: b}}; !slices.Equal(got, want) { + t.Errorf("callees(%s) = %v, want %v", src.Signature, got, want) + } +} diff --git a/internal/sdg/ir.go b/internal/sdg/ir.go new file mode 100644 index 0000000..a8224c9 --- /dev/null +++ b/internal/sdg/ir.go @@ -0,0 +1,118 @@ +// Package sdg builds the L4 overlay (-a 4): the interprocedural system +// dependence graph over the L3 results. It builds go/ssa over the analyzer's +// go/packages load (bodies for project packages only), resolves dynamic +// calls with VTA, and derives formal/actual vertices, param edges and +// summaries. +// +// The IR here holds no can:// ids: vertices are keyed by (L3Key, local key) +// and cross-callable edges by L3Key pairs and external signatures. The v2 +// emitter mints the global ids. This package must not import v2emit or core. +package sdg + +import ( + "go/token" + + "golang.org/x/tools/go/packages" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" +) + +// Target is a call target: an in-project callable or a library stand-in. +type Target struct { + Callable dataflow.L3Key // in-project callable; interface methods use the interface's declaring file + External string // full callee signature (key of app.Externals) for a stand-in; else "" +} + +// VertexKind is an L4 port vertex kind. +type VertexKind string + +// The L4 port vertex kinds (shared vocabulary). +const ( + KindFormalIn VertexKind = "formal_in" + KindFormalOut VertexKind = "formal_out" + KindActualIn VertexKind = "actual_in" + KindActualOut VertexKind = "actual_out" +) + +// Vertex is one L4 port vertex of a callable, keyed by its local key. +type Vertex struct { + Key, Of, Parent string + Kind VertexKind +} + +// Ref names a vertex of a target by its local key. +type Ref struct { + Owner Target + Key string +} + +// ParamEdge is a param_in / param_out edge between two targets' vertices. +type ParamEdge struct { + Src, Dst Ref + Var string +} + +// PortEdge is a callable-local ddg edge added at L4. +type PortEdge struct{ Src, Dst, Var, Prov string } // callable-local ddg addition; Prov "reaching-defs" | "points-to" + +// SummaryEdge is a call-site summary edge, actual_in to actual_out (local keys). +type SummaryEdge struct{ Src, Dst string } + +// CallableSDG is the L4 overlay of one callable. +type CallableSDG struct { + Vertices []Vertex + Ddg []PortEdge + Summary []SummaryEdge +} + +// Degraded is one degradation marker entry (spec §4.15). +type Degraded struct { + Code, Detail string + Packages []string + Callables []dataflow.L3Key +} + +// Result is the L4 overlay of a run. +type Result struct { + Callables map[dataflow.L3Key]CallableSDG + Externals map[string][]Vertex // stand-in formal vertices by full callee sig + NewExternals map[string]schema.GoExternalCallee // externals only VTA reached + ParamIn, ParamOut []ParamEdge + CallEdges []schema.GoCallEdge // prov ["vta"] + Degraded []Degraded +} + +// Input is what Run needs from the earlier phases. +type Input struct { + Fset *token.FileSet + Pkgs map[string]*packages.Package + Variants []*packages.Package // every loaded package, test variants included (SymbolTableBuilder.Loaded); nil = Pkgs' values + ProjectDir string + SymbolTable map[string]schema.GoFile + Externals map[string]schema.GoExternalCallee + CallGraph []schema.GoCallEdge // L2 edges (app.CallGraph) + Units []dataflow.Unit + L3 map[dataflow.L3Key]dataflow.Result + K, Jobs int + Models Models // --library-models (LoadModels); nil = the defaults +} + +// Models overrides library stand-in summaries (spec §6), keyed by the +// external id's / tail (`strconv/Atoi`): each pair is a +// flow [in, out] from the stand-in's formal_in of `of` in to its formal_out +// of `of` out. An empty list is no flow; an external with no entry keeps +// every input → every output. +type Models map[string][][2]string + +// Degraded codes (spec §4.15; an open vocabulary of plain strings). +const ( + CodeNoSSA = "no_ssa" + CodePartialSSA = "partial_ssa" + CodePartialTypes = "partial_types" + CodeNoVTA = "no_vta" + CodeNoCHA = "no_cha" + CodeUnmappedSSA = "unmapped_ssa" + CodeL3Skipped = "l3_skipped" + CodeRangeFunc = "range_func" +) diff --git a/internal/sdg/join.go b/internal/sdg/join.go new file mode 100644 index 0000000..e0064ae --- /dev/null +++ b/internal/sdg/join.go @@ -0,0 +1,577 @@ +package sdg + +import ( + "fmt" + "go/ast" + "go/token" + "go/types" + "sort" + "strings" + + "golang.org/x/tools/go/packages" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// SSA Synthetic tags the join tells apart (x/tools v0.52 go/ssa). +const ( + synthPackageInit = "package initializer" + synthYield = "range-over-func yield" + synthInstWrapper = "instantiation wrapper" +) + +// joinKind is how an SSA function joined its callable. +type joinKind int + +const ( + joinNone joinKind = iota + joinBody // its code is a unit's body: by syntax, origin, or a folded yield + joinWrapper // a synthetic wrapper (or a types-only stub) of a unit: by Object() + joinIface // an interface method ($bound / $thunk of it): a symbol-table key, no unit + joinBodiless // a declared function with no Go body (assembly, linkname), or a wrapper of one: a symbol-table key, no unit +) + +// join maps go/ssa back to the L3 units (spec §4.8, SSA → L3 join). It keys +// only on AST pointer identity and raw Lparen positions, never on line:col or +// SSA names, so //line directives, SSA's anonymous-function numbering and +// build order cannot move a mapping. Its maps are read-only after newJoin, so +// it is safe for concurrent use, and it never builds SSA code (it may be +// used after a no_vta). +type join struct { + fset *token.FileSet + projectDir string + byFn map[ast.Node]dataflow.L3Key // Unit.Fn → unit + iface map[dataflow.L3Key]bool // (interface's file, signature) of every interface method + bodiless map[dataflow.L3Key]bool // (declaring file, signature) of every bodiless declared callable + calls map[dataflow.L3Key]map[token.Pos]string // unit → call Lparen → L1/L3 call key + projPaths map[string]bool // import paths of the project packages + modPath string // the project module path from go.mod ("" when unavailable) + units map[dataflow.L3Key]dataflow.Unit // by key + results map[dataflow.L3Key]dataflow.Result // in.L3 + pkgs map[string]*packages.Package // in.Pkgs +} + +// newJoin indexes in's units. A unit's call keys are its L3 result's +// CallKeys; a unit with no result (l3_skipped), or one built without side +// tables, falls back to its L1 call keys (Unit.Calls), which the emitter +// writes either way. +func newJoin(in Input) *join { + j := &join{ + fset: in.Fset, + projectDir: in.ProjectDir, + byFn: make(map[ast.Node]dataflow.L3Key, len(in.Units)), + iface: map[dataflow.L3Key]bool{}, + bodiless: map[dataflow.L3Key]bool{}, + calls: make(map[dataflow.L3Key]map[token.Pos]string, len(in.Units)), + projPaths: map[string]bool{}, + modPath: semantic_analysis.ModulePath(in.ProjectDir), + units: make(map[dataflow.L3Key]dataflow.Unit, len(in.Units)), + results: in.L3, + pkgs: in.Pkgs, + } + for _, u := range in.Units { + j.byFn[u.Fn] = u.Key + j.units[u.Key] = u + if r, ok := in.L3[u.Key]; ok && r.CallKeys != nil { + j.calls[u.Key] = r.CallKeys + continue + } + m := make(map[token.Pos]string, len(u.Calls)) + for _, c := range u.Calls { + m[c.Lparen] = c.Key + } + j.calls[u.Key] = m + } + // Callables with a symbol-table key but no unit (no body, so no L3): + // - interface methods, keyed by the interface's file (c.Path is ""); + // a $bound / $thunk of one names it; + // - declared functions and methods with no Go body (assembly or + // linkname implementations); only a FuncDecl with a nil body counts, + // so a declaration L3Units skipped despite its body (a second init in + // one file) never lands here. + declared := map[dataflow.L3Key]bool{} + for path, f := range in.SymbolTable { + for _, c := range f.Functions { + declared[dataflow.L3Key{Path: c.Path, Signature: c.Signature}] = true + } + for _, t := range f.Types { + for _, c := range t.Methods { + if c.Path == "" { + j.iface[dataflow.L3Key{Path: path, Signature: c.Signature}] = true + } else { + declared[dataflow.L3Key{Path: c.Path, Signature: c.Signature}] = true + } + } + } + } + for _, p := range projectPackages(in) { + j.projPaths[p.PkgPath] = true + if p.TypesInfo == nil { + continue + } + for _, f := range p.Syntax { + for _, d := range f.Decls { + fd, ok := d.(*ast.FuncDecl) + if !ok || fd.Body != nil { + continue + } + if obj, ok := p.TypesInfo.Defs[fd.Name].(*types.Func); ok { + if k, ok := declKey(j.fset, j.projectDir, obj); ok && declared[k] { + j.bodiless[k] = true + } + } + } + } + } + return j +} + +// fnUnit returns the L3 unit fn joins. ok means a unit exists: j.units[k] +// holds it (its Fn and Info are set), though its L3 result may be absent +// (l3_skipped). A function that names a callable with no unit — an +// interface method (ifaceMethod) or a bodiless declaration (bodilessDecl) — +// is !ok here. The join: +// +// 1. fn.Syntax() is a unit's Fn (a declared function or method, init#N, +// a function literal, a generic instance or instantiation wrapper — +// both keep their origin's syntax; a wrapper joins as a wrapper). +// 2. Else, for a top-level function, fn.Origin()'s syntax: an instance +// with no syntax of its own (go/ssa v0.52 always copies it, so this is +// a guard). Closures never call Origin: for a closure of an instance it +// may build a package. +// 3. Else, for a function with no syntax — a synthetic wrapper ($bound, +// $thunk, promoted or pointer wrapper) or a bodiless stub of a project +// function seen through another test variant of its package — the +// function fn.Object() names, by (declaring file, SignatureOf), the +// symbol table's own canonicalizer. +// 4. A range-over-func yield folds into fn.Parent(). +// +// The package initializer, closures of package-level variable initializers +// and "from type information" functions declared outside the project map to +// nothing. +func (j *join) fnUnit(fn *ssa.Function) (dataflow.L3Key, bool) { + k, kind := j.resolve(fn) + if kind != joinBody && kind != joinWrapper { + return dataflow.L3Key{}, false + } + if _, ok := j.units[k]; !ok { + return dataflow.L3Key{}, false + } + return k, true +} + +// ifaceMethod returns the symbol-table key of the interface method fn +// stands for — the interface's declaring file and the method's signature, +// as the symbol table keys bodiless interface methods (GoType.Methods with +// Path "") — for a $bound (method value) or $thunk (method expression) of an +// interface method. The key has no unit and no L3 result: fnUnit is !ok for +// fn. Later stages bind such a target to the interface method's formals and +// pass-through summary (spec §4.8, D7). +func (j *join) ifaceMethod(fn *ssa.Function) (dataflow.L3Key, bool) { + if k, kind := j.resolve(fn); kind == joinIface { + return k, true + } + return dataflow.L3Key{}, false +} + +// bodilessDecl returns the symbol-table key of the declared function or +// method fn stands for when its declaration has no Go body (implemented in +// assembly, or by //go:linkname): fn itself (a bodiless SSA function with +// that FuncDecl as syntax) or a wrapper of it. The key has no unit and no +// L3 result: fnUnit is !ok for fn. Like an interface method, it is a target +// with formals and a pass-through summary. +func (j *join) bodilessDecl(fn *ssa.Function) (dataflow.L3Key, bool) { + if k, kind := j.resolve(fn); kind == joinBodiless { + return k, true + } + return dataflow.L3Key{}, false +} + +func (j *join) resolve(fn *ssa.Function) (dataflow.L3Key, joinKind) { + for fn != nil && fn.Synthetic == synthYield { + fn = fn.Parent() + } + if fn == nil { + return dataflow.L3Key{}, joinNone + } + if syn := fn.Syntax(); syn != nil { + k, ok := j.byFn[syn] + switch { + case !ok: + if fd, isDecl := syn.(*ast.FuncDecl); isDecl && fd.Body == nil { + break // a bodiless declaration: by Object(), below + } + return dataflow.L3Key{}, joinNone + case strings.HasPrefix(fn.Synthetic, synthInstWrapper): + // It keeps its origin's syntax, but its body is a synthetic + // call of the origin. + return k, joinWrapper + default: + return k, joinBody + } + } + if fn.Parent() == nil { + if o := fn.Origin(); o != nil && o != fn { + if k, ok := j.byFn[o.Syntax()]; ok { + return k, joinBody + } + } + } + if fn.Synthetic == synthPackageInit { + return dataflow.L3Key{}, joinNone + } + if obj, ok := fn.Object().(*types.Func); ok { + if k, ok := declKey(j.fset, j.projectDir, obj.Origin()); ok { + switch { + case j.units[k].Fn != nil: + return k, joinWrapper + case j.iface[k]: + return k, joinIface + case j.bodiless[k]: + return k, joinBodiless + } + } + } + return dataflow.L3Key{}, joinNone +} + +// declKey is obj's (declaring file, signature) when it is declared in a +// project file of projectDir. +func declKey(fset *token.FileSet, projectDir string, obj *types.Func) (dataflow.L3Key, bool) { + tf := fset.File(obj.Pos()) + if tf == nil || !utils.IsWithin(projectDir, tf.Name()) { + return dataflow.L3Key{}, false + } + rel := utils.RelativePath(projectDir, tf.Name()) + if utils.IsVendored(rel) { + return dataflow.L3Key{}, false + } + return dataflow.L3Key{Path: rel, Signature: syntactic_analysis.SignatureOf(obj)}, true +} + +// callSite returns the unit and L1/L3 call key of instr: its function must +// join by body (a wrapper's calls are synthetic) and its CallCommon.Pos() — +// the CallExpr's Lparen — must be one of the unit's call keys. A call with +// no position (one go/ssa synthesizes: the defer stack, a nil check, an +// init call, a range-over-func iterator call) maps to nothing. +func (j *join) callSite(instr ssa.CallInstruction) (unit dataflow.L3Key, callKey string, ok bool) { + pos := instr.Common().Pos() + if !pos.IsValid() { + return dataflow.L3Key{}, "", false + } + k, kind := j.resolve(instr.Parent()) + if kind != joinBody { + return dataflow.L3Key{}, "", false + } + key, ok := j.calls[k][pos] + if !ok { + return dataflow.L3Key{}, "", false + } + return k, key, true +} + +// externalOf returns the full callee signature (the app.Externals key) of a +// function fn.Object() names outside the project: a library function or +// method, a wrapper of one, or its "from type information" stub. The +// signature is L2's (semantic_analysis.CalleeSignatureOf), so L2 and L4 +// external ids agree. +func (j *join) externalOf(fn *ssa.Function) (sig string, ok bool) { + obj, ok := fn.Object().(*types.Func) + if !ok || obj.Pkg() == nil || !j.isLibrary(obj) { + return "", false + } + return semantic_analysis.CalleeSignatureOf(obj), true +} + +// isLibrary mirrors the L2 rule: declared in a file outside the project or +// vendored; without a file, by L2's semantic_analysis.LibraryByPath (the +// go.mod module prefix, else whether the package is a project package). +func (j *join) isLibrary(obj *types.Func) bool { + if tf := j.fset.File(obj.Pos()); tf != nil { + if !utils.IsWithin(j.projectDir, tf.Name()) { + return true + } + return utils.IsVendored(utils.RelativePath(j.projectDir, tf.Name())) + } + // No source file: L2's rule, so the two levels agree on the callee. + return semantic_analysis.LibraryByPath(obj.Pkg().Path(), j.modPath, j.projPaths[obj.Pkg().Path()]) +} + +// joinCoverage counts how the project functions and their call instructions +// joined. Only call instructions with a source position count: a call +// go/ssa synthesizes has none and no call node. +type joinCoverage struct { + Funcs, Unmapped, Wrappers, Inits int // Inits: package initializers (no callable by design) + NoUnit int // mapped to a callable with no unit: interface methods, bodiless declarations + Calls, UnmappedCalls, Synthetic int // Calls: positioned calls in body-joined functions + unmappedPkgs map[string]bool +} + +// coverage joins every function of funcs (the project functions) and its +// calls. A package initializer has no callable (L1 records none for +// package-level variable initializers) and its calls are not counted; a +// closure in such an initializer is unmapped. +func (j *join) coverage(funcs map[*ssa.Function]bool) joinCoverage { + c := joinCoverage{unmappedPkgs: map[string]bool{}} + for fn := range funcs { + if fn.Synthetic == synthPackageInit { + c.Inits++ + continue + } + c.Funcs++ + _, kind := j.resolve(fn) + switch kind { + case joinNone: + c.Unmapped++ + c.unmappedPkgs[homePath(fn)] = true + utils.Debug("L4: unmapped SSA function %s [%s]", fn, fn.Synthetic) + case joinWrapper: + c.Wrappers++ + case joinIface, joinBodiless: + c.NoUnit++ + } + forCalls(fn, func(call ssa.CallInstruction) { + switch { + case !call.Common().Pos().IsValid(): + c.Synthetic++ + case kind == joinWrapper || kind == joinIface || kind == joinBodiless: + c.Synthetic++ // a wrapper's calls are synthetic + default: + c.Calls++ + if _, _, ok := j.callSite(call); !ok { + c.UnmappedCalls++ + c.unmappedPkgs[homePath(fn)] = true + utils.Debug("L4: unmapped SSA call %s in %s at %s", call, fn, j.fset.PositionFor(call.Common().Pos(), false)) + } + } + }) + } + return c +} + +// degraded is the unmapped_ssa entry for c, or nil. A listed package with +// cgo files gets a note: cgo files are not in the symbol table yet (#29). +func (j *join) degraded(c joinCoverage) *Degraded { + if c.Unmapped == 0 && c.UnmappedCalls == 0 { + return nil + } + pkgs := sortedKeys(c.unmappedPkgs) + detail := fmt.Sprintf("%d SSA function(s) and %d call instruction(s) map to no callable or call node", c.Unmapped, c.UnmappedCalls) + for _, p := range pkgs { + if hasCgo(j.fset, j.pkgs[p]) { + detail += "; cgo files are not in the symbol table (#29)" + break + } + } + return &Degraded{Code: CodeUnmappedSSA, Detail: detail, Packages: pkgs} +} + +// hasCgo reports whether p compiles cgo-generated files: go/packages then +// parses the generated files (in the build cache) instead of the ones in +// GoFiles. The load does not request CompiledGoFiles, so the test is a +// Syntax file that GoFiles does not list. +func hasCgo(fset *token.FileSet, p *packages.Package) bool { + if p == nil { + return false + } + files := make(map[string]bool, len(p.GoFiles)) + for _, f := range p.GoFiles { + files[f] = true + } + for _, f := range p.Syntax { + if tf := fset.File(f.Pos()); tf != nil && !files[tf.Name()] { + return true + } + } + return false +} + +// forCalls calls f on every call instruction of fn's body. +func forCalls(fn *ssa.Function, f func(ssa.CallInstruction)) { + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + if call, ok := instr.(ssa.CallInstruction); ok { + f(call) + } + } + } +} + +// homePath is the import path of the package fn's code comes from: its own +// package, its generic origin's, its enclosing function's, or (for a +// wrapper) the wrapped object's. +func homePath(fn *ssa.Function) string { + for f := fn; f != nil; f = f.Parent() { + if f.Pkg != nil { + return f.Pkg.Pkg.Path() + } + if f.Parent() == nil { + if o := f.Origin(); o != nil && o.Pkg != nil { + return o.Pkg.Pkg.Path() + } + if obj := f.Object(); obj != nil && obj.Pkg() != nil { + return obj.Pkg().Path() + } + } + } + return "" +} + +// partialTypes is the partial_types entry (spec §4.15): the project +// packages (every variant, by path) go/packages loaded with errors — list, +// parse or type errors — or that are ill-typed through a broken dependency, +// sorted; nil when every package loaded clean. Their types (and so L1–L3 +// over them) are partial; partial_ssa says which also got no SSA. +func partialTypes(in Input) *Degraded { + pkgs := map[string]bool{} + for _, p := range projectPackages(in) { + if len(p.Errors) > 0 || p.IllTyped { + pkgs[p.PkgPath] = true + } + } + if len(pkgs) == 0 { + return nil + } + return &Degraded{ + Code: CodePartialTypes, + Detail: fmt.Sprintf("%d package(s) loaded with errors (list, parse or type errors, or a broken dependency)", len(pkgs)), + Packages: sortedKeys(pkgs), + } +} + +// l3Skipped is the l3_skipped entry (spec §4.15, A16): the units with no L3 +// result (a recovered panic, or a body call with no L1 key), or nil. +func l3Skipped(in Input) *Degraded { + var keys []dataflow.L3Key + for _, u := range in.Units { + if _, ok := in.L3[u.Key]; !ok { + keys = append(keys, u.Key) + } + } + if len(keys) == 0 { + return nil + } + sortKeys(keys) + return &Degraded{ + Code: CodeL3Skipped, + Detail: fmt.Sprintf("%d callable(s) have no L3 result (a recovered panic or a call with no L1 key)", len(keys)), + Callables: keys, + } +} + +// rangeFunc is the range_func entry (spec §4.15, D8): the units whose own +// body (not a nested literal: that is its own unit) ranges over a function, +// and their packages, or nil. It reads the AST only, so it holds whether or +// not SSA was built. +func rangeFunc(in Input) *Degraded { + pkgOf := map[string]string{} + for _, p := range projectPackages(in) { + for _, f := range p.GoFiles { + pkgOf[f] = p.PkgPath + } + } + var keys []dataflow.L3Key + pkgs := map[string]bool{} + for _, u := range in.Units { + if !rangesOverFunc(u) { + continue + } + keys = append(keys, u.Key) + if tf := in.Fset.File(u.Fn.Pos()); tf != nil { + if p, ok := pkgOf[tf.Name()]; ok { + pkgs[p] = true + } + } + } + // A function literal in a package-level declaration (`var f = func(){…}`) + // has no callable, so it is no unit; its package is still listed. + for _, p := range projectPackages(in) { + if pkgLevelRangesOverFunc(p) { + pkgs[p.PkgPath] = true + } + } + if len(keys) == 0 && len(pkgs) == 0 { + return nil + } + sortKeys(keys) + return &Degraded{ + Code: CodeRangeFunc, + Detail: "range-over-func is not modelled at L4 (D8): what the iterator passes to yield does not reach the loop variables", + Packages: sortedKeys(pkgs), + Callables: keys, + } +} + +func rangesOverFunc(u dataflow.Unit) bool { + var body *ast.BlockStmt + switch fn := u.Fn.(type) { + case *ast.FuncDecl: + body = fn.Body + case *ast.FuncLit: + body = fn.Body + } + if body == nil || u.Info == nil { + return false + } + found := false + ast.Inspect(body, func(n ast.Node) bool { + switch n := n.(type) { + case *ast.FuncLit: + return false + case *ast.RangeStmt: + if dataflow.RangesOverFunc(n.X, u.Info) { + found = true + } + } + return !found + }) + return found +} + +// pkgLevelRangesOverFunc reports whether a function literal in one of p's +// package-level declarations (outside every FuncDecl) ranges over a function. +func pkgLevelRangesOverFunc(p *packages.Package) bool { + if p.TypesInfo == nil { + return false + } + found := false + for _, f := range p.Syntax { + for _, d := range f.Decls { + if _, ok := d.(*ast.GenDecl); !ok { + continue + } + ast.Inspect(d, func(n ast.Node) bool { + if rs, ok := n.(*ast.RangeStmt); ok && dataflow.RangesOverFunc(rs.X, p.TypesInfo) { + found = true + } + return !found + }) + if found { + return true + } + } + } + return false +} + +func sortKeys(keys []dataflow.L3Key) { + sort.Slice(keys, func(i, j int) bool { + if keys[i].Path != keys[j].Path { + return keys[i].Path < keys[j].Path + } + return keys[i].Signature < keys[j].Signature + }) +} + +func sortedKeys(m map[string]bool) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + sort.Strings(out) + return out +} diff --git a/internal/sdg/join_test.go b/internal/sdg/join_test.go new file mode 100644 index 0000000..706af23 --- /dev/null +++ b/internal/sdg/join_test.go @@ -0,0 +1,813 @@ +package sdg + +import ( + "go/ast" + "go/token" + "go/types" + "path/filepath" + "slices" + "sort" + "strings" + "testing" + + "golang.org/x/tools/go/ssa" + "golang.org/x/tools/go/ssa/ssautil" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// joined builds SSA over in and the join, and returns them with the +// project-function set the join is measured on: every function of the +// program whose code is project code (projectFuncs' rule) — not only the VTA +// seeds, so wrappers built on demand and every test variant count too. +func joined(t *testing.T, in Input) (*ssa.Program, *join, map[*ssa.Function]bool) { + t.Helper() + prog, pkgs, deg := buildSSA(in) + for _, d := range deg { + t.Logf("degraded: %+v", d) + } + if len(pkgs) == 0 { + t.Fatal("no SSA package built") + } + j := newJoin(in) + project := map[string]bool{} + for _, p := range pkgs { + project[p.Pkg.Path()] = true + } + funcs := map[*ssa.Function]bool{} + for fn := range ssautil.AllFunctions(prog) { + if project[homePath(fn)] && !strings.HasPrefix(fn.Synthetic, "from type information") { + funcs[fn] = true + } + } + project2, _ := projectFuncs(prog, pkgs) + for fn := range project2 { + funcs[fn] = true + } + return prog, j, funcs +} + +// describeUnmapped lists the functions and calls of funcs that join nothing. +func describeUnmapped(j *join, funcs map[*ssa.Function]bool) []string { + var out []string + for fn := range funcs { + if fn.Synthetic == synthPackageInit { + continue + } + _, kind := j.resolve(fn) + if kind == joinNone { + out = append(out, "func "+fn.String()+" ["+fn.Synthetic+"]") + continue + } + if kind != joinBody { + continue + } + forCalls(fn, func(c ssa.CallInstruction) { + if c.Common().Pos().IsValid() { + if _, _, ok := j.callSite(c); !ok { + out = append(out, "call "+c.String()+" in "+fn.String()) + } + } + }) + } + sort.Strings(out) + return out +} + +// droppedDecls returns the bodies of the project FuncDecls that have no L3 +// unit: the declarations the symbol table dropped (cgo wrappers, #29, and +// methods whose receiver type it did not find — today also a method declared +// before its receiver type in the same file). Their SSA functions, and +// closures in them, cannot join: that is unmapped_ssa by design. +func droppedDecls(in Input) map[ast.Node]bool { + units := map[ast.Node]bool{} + for _, u := range in.Units { + units[u.Fn] = true + } + out := map[ast.Node]bool{} + for _, p := range projectPackages(in) { + for _, f := range p.Syntax { + tf := in.Fset.File(f.Pos()) + if tf == nil || !utils.IsWithin(in.ProjectDir, tf.Name()) { + continue + } + for _, d := range f.Decls { + if fd, ok := d.(*ast.FuncDecl); ok && fd.Body != nil && !units[fd] { + out[fd] = true + } + } + } + } + return out +} + +// inDropped reports whether fn's code is (inside) a dropped declaration. +func inDropped(fn *ssa.Function, dropped map[ast.Node]bool) bool { + for f := fn; f != nil; f = f.Parent() { + if dropped[f.Syntax()] { + return true + } + } + return false +} + +// TestJoin_EveryProjectFunctionMaps: on the fixtures every project SSA +// function joins a callable and every positioned call instruction in a +// body-joined function joins its call node, except the functions of +// declarations the symbol table dropped (chi has one: +// http2FancyWriter.Push is declared before its receiver type). Every unit is +// the image of at least one SSA function. +func TestJoin_EveryProjectFunctionMaps(t *testing.T) { + wantDropped := map[string][]string{} + for _, name := range []string{"dataflow", "chi", "generics", "rangefunc"} { + t.Run(name, func(t *testing.T) { + in := loadInput(t, name) + _, j, funcs := joined(t, in) + dropped := droppedDecls(in) + var lost []string + for fn := range funcs { + if _, kind := j.resolve(fn); kind == joinNone && fn.Synthetic != synthPackageInit { + if !inDropped(fn, dropped) { + t.Errorf("%s [%s] joins nothing and is not a dropped declaration", fn, fn.Synthetic) + } + lost = append(lost, fn.String()) + } + } + sort.Strings(lost) + if !slices.Equal(lost, wantDropped[name]) { + t.Errorf("unmapped functions = %v, want %v", lost, wantDropped[name]) + } + for _, b := range describeUnmapped(j, funcs) { + if strings.HasPrefix(b, "call ") { + t.Errorf("unmapped %s", b) + } + } + c := j.coverage(funcs) + if d := j.degraded(c); (d != nil) != (len(wantDropped[name]) > 0) { + t.Errorf("degraded = %+v", d) + } + hit := map[dataflow.L3Key]bool{} + for fn := range funcs { + if k, kind := j.resolve(fn); kind == joinBody { + hit[k] = true + } + } + for _, u := range in.Units { + if !hit[u.Key] { + t.Errorf("unit %v is the image of no SSA function", u.Key) + } + } + t.Logf("%s: %d funcs (%d by wrapper, %d unmapped, +%d package inits), %d positioned calls (%d unmapped, +%d synthetic), %d units", + name, c.Funcs, c.Wrappers, c.Unmapped, c.Inits, c.Calls, c.UnmappedCalls, c.Synthetic, len(in.Units)) + }) + } +} + +// fnByName returns the SSA function of package pkg named name (a member, or +// a method "T.M" of a named type, pointer receiver first). +func fnByName(t *testing.T, prog *ssa.Program, pkg, name string) *ssa.Function { + t.Helper() + p := prog.ImportedPackage(pkg) + if p == nil { + t.Fatalf("no SSA package %s", pkg) + } + if typ, meth, ok := strings.Cut(name, "."); ok { + T := p.Type(typ) + if T == nil { + t.Fatalf("%s has no type %s", pkg, typ) + } + for _, recv := range []bool{true, false} { + mset := prog.MethodSets.MethodSet(T.Type()) + if recv { + mset = prog.MethodSets.MethodSet(ptrTo(T.Type())) + } + if sel := mset.Lookup(p.Pkg, meth); sel != nil { + return prog.MethodValue(sel) + } + } + t.Fatalf("%s.%s has no method %s", pkg, typ, meth) + } + fn := p.Func(name) + if fn == nil { + t.Fatalf("%s has no func %s", pkg, name) + } + return fn +} + +// TestJoin_MultipleInitsAndLineDirectives (Review Focus 4): each init#N joins +// its own file's init by AST pointer, and the calls of each join that file's +// call keys; calls under a //line directive join by Lparen. +func TestJoin_MultipleInitsAndLineDirectives(t *testing.T) { + in := loadInput(t, "dataflow") + prog, j, _ := joined(t, in) + + const initSig = "example.com/dataflow/inits.init" + files := map[string]string{} // init#N → file + inits := prog.ImportedPackage("example.com/dataflow/inits") + for _, name := range []string{"init#1", "init#2"} { + fn := inits.Func(name) + if fn == nil { + t.Fatalf("no %s", name) + } + k, ok := j.fnUnit(fn) + if !ok || k.Signature != initSig { + t.Fatalf("%s joins %v, %v; want an %s unit", name, k, ok, initSig) + } + files[name] = k.Path + calls := 0 + forCalls(fn, func(c ssa.CallInstruction) { + if !c.Common().Pos().IsValid() { + return + } + u, key, ok := j.callSite(c) + if !ok || u != k { + t.Errorf("%s: call %s joins (%v, %q, %v), want its own unit %v", name, c, u, key, ok, k) + } + calls++ + }) + if calls != 2 { + t.Errorf("%s: %d positioned calls, want 2 (setup, os.Getenv)", name, calls) + } + } + if files["init#1"] == files["init#2"] { + t.Errorf("both inits join %s", files["init#1"]) + } + want := []string{filepath.Join("inits", "a.go"), filepath.Join("inits", "b.go")} + got := []string{files["init#1"], files["init#2"]} + sort.Strings(got) + if !slices.Equal(got, want) { + t.Errorf("init files = %v, want %v", got, want) + } + // The synthetic package initializer calls each init#N with no position. + if _, ok := j.fnUnit(inits.Func("init")); ok { + t.Error("the package initializer joins a callable") + } + + caller := fnByName(t, prog, "example.com/dataflow/linedir", "Caller") + k, ok := j.fnUnit(caller) + if !ok || k.Signature != "example.com/dataflow/linedir.Caller" { + t.Fatalf("linedir.Caller joins %v, %v", k, ok) + } + u := j.units[k] + wantKeys := map[string]bool{} + for _, c := range u.Calls { + wantKeys[c.Key] = true + } + gotKeys := map[string]bool{} + forCalls(caller, func(c ssa.CallInstruction) { + if _, key, ok := j.callSite(c); ok { + gotKeys[key] = true + } + }) + if len(wantKeys) != 3 || len(gotKeys) != len(wantKeys) { + t.Errorf("linedir.Caller call keys = %v, want every L1 key %v", gotKeys, wantKeys) + } + // The real line of os.Getenv(helper()) is 13: keys are real positions, + // and a virtual virtual.y:900 position would have joined nothing. + for key := range gotKeys { + if strings.HasPrefix(key, "900:") { + t.Errorf("call key %s is a //line virtual position", key) + } + } +} + +// TestJoin_ExternalTestPackage (Review Focus 5). With tests loaded, the +// external test package xtest_test and the test variant "xtest [xtest.test]" +// are SSA packages too; package user imports the plain xtest, so the program +// holds two SSA packages with path xtest, sharing a.go's syntax. Every +// project function of both still joins (variant-safe by AST pointer), the +// test files' functions join the test files' callables, and the plain and +// the variant Add join the same callable. With --skip-tests the test +// functions are absent and nothing dangles. +func TestJoin_ExternalTestPackage(t *testing.T) { + const xt = "example.com/dataflow/xtest" + aGo, testGo := filepath.Join("xtest", "a.go"), filepath.Join("xtest", "a_test.go") + for _, skip := range []bool{false, true} { + t.Run(map[bool]string{false: "tests", true: "skip-tests"}[skip], func(t *testing.T) { + in := loadInputOpts(t, "dataflow", skip) + prog, j, funcs := joined(t, in) + if bad := describeUnmapped(j, funcs); len(bad) > 0 { + t.Errorf("%d unmapped:\n %s", len(bad), strings.Join(bad, "\n ")) + } + + // Every SSA (*Counter).Add, of every variant: with tests, the + // test variant "xtest [xtest.test]" and the plain xtest that + // user imports are both roots built from a.go's shared syntax + // (F2: the plain one's bodies carry the types user sees). Both + // join the one callable by syntax. + wantAdd := dataflow.L3Key{Path: aGo, Signature: xt + ".Counter.Add"} + kinds := map[joinKind]int{} + for fn := range ssautil.AllFunctions(prog) { + if fn.Name() == "Add" && fn.Signature.Recv() != nil && homePath(fn) == xt && !strings.HasPrefix(fn.Synthetic, "wrapper") { + k, kind := j.resolve(fn) + if k != wantAdd { + t.Errorf("%s [%s] (pkg %p) joins %v, %v; want %v", fn, fn.Synthetic, fn.Pkg, k, kind, wantAdd) + } + kinds[kind]++ + } + } + wantBodies := map[bool]int{true: 1, false: 2}[skip] + if kinds[joinBody] != wantBodies || len(kinds) != 1 { + t.Errorf("(*Counter).Add joins %v, want %d SSA bodies and nothing else", kinds, wantBodies) + } + // user.Use calls the plain xtest's stubs; they still name the + // in-project callables, never an external. + use := fnByName(t, prog, xt+"/user", "Use") + forCalls(use, func(c ssa.CallInstruction) { + callee := c.Common().StaticCallee() + if callee == nil { + return + } + if _, ok := j.fnUnit(callee); !ok { + t.Errorf("user.Use callee %s [%s] joins nothing", callee, callee.Synthetic) + } + if sig, ok := j.externalOf(callee); ok { + t.Errorf("user.Use callee %s is external %s", callee, sig) + } + }) + + var testFns []dataflow.L3Key + for fn := range funcs { + if k, kind := j.resolve(fn); kind == joinBody && k.Path == testGo { + testFns = append(testFns, k) + } + } + sortKeys(testFns) + testFns = slices.Compact(testFns) + if skip { + if len(testFns) != 0 { + t.Errorf("test callables joined with --skip-tests: %v", testFns) + } + if p := prog.ImportedPackage(xt + "_test"); p != nil { + t.Errorf("xtest_test has an SSA package with --skip-tests") + } + return + } + want := []dataflow.L3Key{ + {Path: testGo, Signature: xt + "_test.TestAdd"}, + {Path: testGo, Signature: xt + "_test.TestAdd.closure_1"}, + {Path: testGo, Signature: xt + "_test.helper"}, + } + if !slices.Equal(testFns, want) { + t.Errorf("test callables joined = %v, want %v", testFns, want) + } + // helper's c.Add(2) joins its call key in a_test.go. + helper := fnByName(t, prog, xt+"_test", "helper") + n := 0 + forCalls(helper, func(c ssa.CallInstruction) { + if u, key, ok := j.callSite(c); ok && u == want[2] && key != "" { + n++ + } + }) + if n != 1 { + t.Errorf("helper: %d calls joined, want 1", n) + } + }) + } +} + +// TestJoin_WrappersUnwrap: each SSA wrapper kind (§4.8 unwrap list) joins +// the method it wraps: $bound (method value), $thunk (method expression), +// the pointer wrapper of a value method, and a promoted method of an +// embedding struct. Their calls are synthetic: none joins a call node. +func TestJoin_WrappersUnwrap(t *testing.T) { + in := loadInput(t, "dataflow") + prog, j, funcs := joined(t, in) + const xt = "example.com/dataflow/xtest" + add := dataflow.L3Key{Path: filepath.Join("xtest", "a.go"), Signature: xt + ".Counter.Add"} + get := dataflow.L3Key{Path: filepath.Join("xtest", "a.go"), Signature: xt + ".Counter.Get"} + + kinds := map[string]dataflow.L3Key{} + for fn := range funcs { + if homePath(fn) != xt || fn.Synthetic == "" { + continue + } + // Keyed by fn.String(): an interface method's wrappers share the + // concrete method's short names (Add$bound, Add$thunk). + if (strings.HasPrefix(fn.Synthetic, "bound method wrapper") || strings.HasPrefix(fn.Synthetic, "thunk")) && !strings.Contains(fn.String(), "Adder") { + kinds[fn.String()] = mustJoin(t, j, fn) // concrete-method wrappers; Adder's are checked below + } + if fn.Syntax() == nil && fn.Synthetic != synthPackageInit { + if _, kind := j.resolve(fn); kind == joinWrapper { + forCalls(fn, func(c ssa.CallInstruction) { + if _, _, ok := j.callSite(c); ok { + t.Errorf("wrapper %s: call %s joined a call node", fn, c) + } + }) + } + } + } + for _, name := range []string{"(*" + xt + ".Counter).Add$bound", "(*" + xt + ".Counter).Add$thunk"} { + if k, ok := kinds[name]; !ok || k != add { + t.Errorf("%s joins %v (seen %v), want %v (wrappers seen: %v)", name, k, ok, add, kinds) + } + } + // An interface method has a symbol-table key but no unit: its $bound and + // $thunk are !ok for fnUnit and name it through ifaceMethod. + iface := dataflow.L3Key{Path: filepath.Join("xtest", "a.go"), Signature: xt + ".Adder.Add"} + if _, ok := j.units[iface]; ok { + t.Fatalf("interface method %v has a unit", iface) + } + var ifaceWrappers int + for fn := range funcs { + if obj := fn.Object(); obj != nil && fn.Syntax() == nil && obj.Pkg() != nil && obj.Pkg().Path() == xt && obj.Name() == "Add" && types.IsInterface(fn.Object().Type().(*types.Signature).Recv().Type()) { + if k, ok := j.fnUnit(fn); ok { + t.Errorf("fnUnit(%s [%s]) = %v, true; want !ok (an interface method has no unit)", fn, fn.Synthetic, k) + } + if k, ok := j.ifaceMethod(fn); !ok || k != iface { + t.Errorf("ifaceMethod(%s [%s]) = %v, %v; want %v, true", fn, fn.Synthetic, k, ok, iface) + } + if _, ok := j.bodilessDecl(fn); ok { + t.Errorf("bodilessDecl(%s) is ok for an interface method", fn) + } + ifaceWrappers++ + } + } + if ifaceWrappers != 2 { + t.Errorf("%d wrappers of the interface method Adder.Add, want 2 ($bound, $thunk)", ifaceWrappers) + } + // A concrete method and its wrappers still join their unit, and are not + // interface methods. + for _, fn := range []*ssa.Function{fnByName(t, prog, xt, "Counter.Add"), fnByName(t, prog, xt, "Outer.Add")} { + if k, ok := j.fnUnit(fn); !ok || k != add || j.units[k].Fn == nil { + t.Errorf("fnUnit(%s) = %v, %v; want the unit %v", fn, k, ok, add) + } + if _, ok := j.ifaceMethod(fn); ok { + t.Errorf("ifaceMethod(%s) is ok for a concrete method", fn) + } + } + if k := mustJoin(t, j, fnByName(t, prog, xt, "Counter.Get")); k != get { + t.Errorf("(*Counter).Get (pointer wrapper) joins %v, want %v", k, get) + } + if k := mustJoin(t, j, fnByName(t, prog, xt, "Outer.Add")); k != add { + t.Errorf("Outer.Add (promoted) joins %v, want %v", k, add) + } +} + +// TestJoin_BodilessDeclHasNoUnit: a function declared without a Go body +// (testdata/dataflow/asm, implemented in assembly) is in the symbol table +// but has no unit. Its SSA function is !ok for fnUnit and names it through +// bodilessDecl; it is not unmapped, and its caller's call still joins. +func TestJoin_BodilessDeclHasNoUnit(t *testing.T) { + in := loadInput(t, "dataflow") + prog, j, funcs := joined(t, in) + const pkg = "example.com/dataflow/asm" + want := dataflow.L3Key{Path: filepath.Join("asm", "asm.go"), Signature: pkg + ".Add"} + if _, ok := j.units[want]; ok { + t.Fatalf("bodiless %v has a unit", want) + } + add := fnByName(t, prog, pkg, "Add") + if k, ok := j.fnUnit(add); ok { + t.Errorf("fnUnit(asm.Add) = %v, true; want !ok", k) + } + if k, ok := j.bodilessDecl(add); !ok || k != want { + t.Errorf("bodilessDecl(asm.Add) = %v, %v; want %v, true", k, ok, want) + } + if _, ok := j.ifaceMethod(add); ok { + t.Error("ifaceMethod(asm.Add) is ok") + } + if c := j.coverage(funcs); c.NoUnit == 0 { + t.Errorf("coverage %+v counts no function without a unit", c) + } + n := 0 + forCalls(fnByName(t, prog, pkg, "Use"), func(c ssa.CallInstruction) { + if _, _, ok := j.callSite(c); ok { + n++ + } + }) + if n != 1 { + t.Errorf("asm.Use: %d calls joined, want 1", n) + } +} + +func mustJoin(t *testing.T, j *join, fn *ssa.Function) dataflow.L3Key { + t.Helper() + k, ok := j.fnUnit(fn) + if !ok { + t.Errorf("%s [%s] joins nothing", fn, fn.Synthetic) + } + return k +} + +// TestJoin_GenericInstancesJoinOrigin: with InstantiateGenerics, every +// instance of a generic function or method joins its one origin callable, +// and its calls join the origin's call keys. +func TestJoin_GenericInstancesJoinOrigin(t *testing.T) { + in := loadInput(t, "generics") + _, j, funcs := joined(t, in) + byOrigin := map[dataflow.L3Key]int{} + for fn := range funcs { + if !strings.HasPrefix(fn.Synthetic, "instance of") { + continue + } + k, kind := j.resolve(fn) + if kind != joinBody { + t.Errorf("instance %s joins %v, %v", fn, k, kind) + continue + } + byOrigin[k]++ + } + add := dataflow.L3Key{Path: filepath.Join("set", "set.go"), Signature: "example.com/generics/set.Set.Add"} + if byOrigin[add] == 0 { + t.Errorf("no instance of Set.Add joins %v (instances: %v)", add, byOrigin) + } + + // Twice[T] calls Ident[T] through an instantiation wrapper (its type + // arguments are parameterized): it joins Ident as a wrapper. + in = loadInput(t, "dataflow") + _, j, funcs = joined(t, in) + ident := dataflow.L3Key{Path: filepath.Join("generic", "b.go"), Signature: "example.com/dataflow/generic.Ident"} + wrappers := 0 + for fn := range funcs { + if strings.HasPrefix(fn.Synthetic, synthInstWrapper) { + k, kind := j.resolve(fn) + if k != ident || kind != joinWrapper { + t.Errorf("%s joins %v, %v; want %v as a wrapper", fn, k, kind, ident) + } + wrappers++ + } + } + if wrappers == 0 { + t.Error("no instantiation wrapper in testdata/dataflow/generic") + } +} + +// TestJoin_RangeFuncYieldFolds: a range-over-func yield function folds into +// its enclosing callable, and the call in the loop body joins that +// callable's call key. +func TestJoin_RangeFuncYieldFolds(t *testing.T) { + in := loadInput(t, "rangefunc") + prog, j, _ := joined(t, in) + outer := fnByName(t, prog, "example.com/rangefunc", "RangeFuncCall") + want, ok := j.fnUnit(outer) + if !ok { + t.Fatal("RangeFuncCall joins nothing") + } + var yield *ssa.Function + for _, a := range outer.AnonFuncs { + if a.Synthetic == synthYield { + yield = a + } + } + if yield == nil { + t.Fatal("RangeFuncCall has no yield function") + } + if k, ok := j.fnUnit(yield); !ok || k != want { + t.Errorf("yield joins %v, %v; want %v", k, ok, want) + } + n := 0 + forCalls(yield, func(c ssa.CallInstruction) { + if u, _, ok := j.callSite(c); ok && u == want { + n++ + } + }) + if n != 1 { + t.Errorf("yield: %d calls joined (sink(v)), want 1", n) + } +} + +// TestJoin_UnmappedDegrades: a function literal in a package-level variable +// initializer has no callable, so its SSA function (a child of the package +// initializer) is unmapped_ssa, counted, with its package listed. +func TestJoin_UnmappedDegrades(t *testing.T) { + in := loadInput(t, "unmapped") + _, j, funcs := joined(t, in) + c := j.coverage(funcs) + if c.Unmapped != 1 { + t.Errorf("coverage = %+v, want 1 unmapped function", c) + } + d := j.degraded(c) + if d == nil || d.Code != CodeUnmappedSSA || !slices.Equal(d.Packages, []string{"example.com/unmapped"}) { + t.Fatalf("degraded = %+v, want unmapped_ssa [example.com/unmapped]", d) + } + if strings.Contains(d.Detail, "#29") { + t.Errorf("detail %q names #29 for a package with no cgo", d.Detail) + } + res := Run(in) + if !hasCode(res.Degraded, CodeUnmappedSSA) { + t.Errorf("Run degraded = %+v, want unmapped_ssa", res.Degraded) + } +} + +// TestJoin_CgoNamesIssue29: a cgo package's functions have no callable +// (#29), so they are unmapped_ssa and the detail names the issue. +func TestJoin_CgoNamesIssue29(t *testing.T) { + in := loadInput(t, "cgo") + res := Run(in) + var d *Degraded + for i := range res.Degraded { + if res.Degraded[i].Code == CodeUnmappedSSA { + d = &res.Degraded[i] + } + } + if d == nil { + t.Skipf("no unmapped_ssa (cgo disabled?): %+v", res.Degraded) + } + if !strings.Contains(d.Detail, "#29") || !slices.Equal(d.Packages, []string{"example.com/cgo/c"}) { + t.Errorf("unmapped_ssa = %+v, want package example.com/cgo/c and a #29 note", d) + } +} + +func hasCode(deg []Degraded, code string) bool { + for _, d := range deg { + if d.Code == code { + return true + } + } + return false +} + +// TestSDG_L3SkippedDegrades (A16): a unit with no L3 result is listed by key +// under l3_skipped. +func TestSDG_L3SkippedDegrades(t *testing.T) { + in := loadInput(t, "dataflow") + if hasCode(Run(in).Degraded, CodeL3Skipped) { + t.Fatal("clean fixture has l3_skipped") + } + drop := []dataflow.L3Key{in.Units[3].Key, in.Units[0].Key} + l3 := map[dataflow.L3Key]dataflow.Result{} + for k, r := range in.L3 { + l3[k] = r + } + for _, k := range drop { + delete(l3, k) + } + in.L3 = l3 + res := Run(in) + var got *Degraded + for i := range res.Degraded { + if res.Degraded[i].Code == CodeL3Skipped { + got = &res.Degraded[i] + } + } + if got == nil { + t.Fatalf("degraded = %+v, want l3_skipped", res.Degraded) + } + want := []dataflow.L3Key{in.Units[0].Key, in.Units[3].Key} // units are sorted by key + if !slices.Equal(got.Callables, want) { + t.Errorf("l3_skipped callables = %v, want %v", got.Callables, want) + } + // A skipped unit still joins: its calls fall back to its L1 call keys. + if hasCode(res.Degraded, CodeUnmappedSSA) { + t.Errorf("dropping L3 results unmapped SSA: %+v", res.Degraded) + } +} + +// TestSDG_IllTypedPackageDegrades (Review Focus 1, F4): testdata/typeerr +// does not type-check: partial_types lists it (its go/packages errors), it +// gets no SSA (partial_ssa, then no_ssa), nothing panics, and its L3 +// results are those of -a 3. +func TestSDG_IllTypedPackageDegrades(t *testing.T) { + in := loadInput(t, "typeerr") + plain := dataflow.Run(in.Fset, in.Units, 3, 1) + res := Run(in) + codes := codesOf(res.Degraded) + if !slices.Equal(codes, []string{CodePartialTypes, CodePartialSSA, CodeNoSSA}) { + t.Fatalf("codes = %v, want [partial_types partial_ssa no_ssa]", codes) + } + for _, d := range res.Degraded[:2] { + if !slices.Equal(d.Packages, []string{"example.com/typeerr"}) { + t.Errorf("%s packages = %v", d.Code, d.Packages) + } + } + if len(in.L3) == 0 || len(in.L3) != len(plain) { + t.Fatalf("L3 has %d results, -a 3 has %d", len(in.L3), len(plain)) + } + for k, r := range plain { + got := in.L3[k] + if !slices.Equal(got.Cfg, r.Cfg) || !slices.Equal(got.Cdg, r.Cdg) || !slices.Equal(got.Ddg, r.Ddg) || len(got.Nodes) != len(r.Nodes) { + t.Errorf("%v: L3 differs from -a 3", k) + } + } +} + +func codesOf(deg []Degraded) []string { + out := make([]string, len(deg)) + for i, d := range deg { + out[i] = d.Code + } + return out +} + +// TestSDG_RangeFuncDegrades (D8): a program that ranges over a function is +// degraded range_func, listing the callables (by key) and their package; +// the join itself is complete. +func TestSDG_RangeFuncDegrades(t *testing.T) { + in := loadInput(t, "rangefunc") + res := Run(in) + if codes := codesOf(res.Degraded); !slices.Equal(codes, []string{CodeRangeFunc}) { + t.Fatalf("codes = %v, want [range_func]", codes) + } + d := res.Degraded[0] + want := []dataflow.L3Key{ + {Path: "r.go", Signature: "example.com/rangefunc.RangeFunc"}, + {Path: "r.go", Signature: "example.com/rangefunc.RangeFuncCall"}, + } + if !slices.Equal(d.Callables, want) || !slices.Equal(d.Packages, []string{"example.com/rangefunc"}) { + t.Errorf("range_func = %+v, want callables %v in example.com/rangefunc", d, want) + } + if hasCode(Run(loadInput(t, "dataflow")).Degraded, CodeRangeFunc) { + t.Error("dataflow (go 1.21, no range-over-func) is range_func") + } +} + +// TestJoin_Deterministic: two independent loads and SSA builds give the same +// join of every project function, compared by (function string, key) — the +// join must not depend on SSA build or map order. +func TestJoin_Deterministic(t *testing.T) { + render := func() []string { + in := loadInput(t, "chi") + _, j, funcs := joined(t, in) + var out []string + for fn := range funcs { + k, kind := j.resolve(fn) + line := fn.String() + " → " + k.Path + " " + k.Signature + " " + string(rune('0'+kind)) + forCalls(fn, func(c ssa.CallInstruction) { + if _, key, ok := j.callSite(c); ok { + line += " " + key + } + }) + out = append(out, line) + } + sort.Strings(out) + return out + } + if a, b := render(), render(); !slices.Equal(a, b) { + t.Error("two runs join differently") + } +} + +// TestJoin_ExternalOf: a library callee's signature is L2's: the call of +// os.Getenv in inits is os.Getenv, (*strings.Builder) methods are +// strings.Builder., and a project function is not external. +func TestJoin_ExternalOf(t *testing.T) { + in := loadInput(t, "dataflow") + prog, j, _ := joined(t, in) + init1 := prog.ImportedPackage("example.com/dataflow/inits").Func("init#1") + var got []string + forCalls(init1, func(c ssa.CallInstruction) { + if callee := c.Common().StaticCallee(); callee != nil { + if sig, ok := j.externalOf(callee); ok { + got = append(got, sig) + } else if _, ok := j.fnUnit(callee); !ok { + t.Errorf("callee %s is neither external nor in-project", callee) + } + } + }) + if !slices.Equal(got, []string{"os.Getenv"}) { + t.Errorf("externals of init#1 = %v, want [os.Getenv]", got) + } + if fn := prog.ImportedPackage("example.com/vlib"); fn != nil { + if sig, ok := j.externalOf(fn.Func("F")); !ok || sig != "example.com/vlib.F" { + t.Errorf("vendored vlib.F: externalOf = %q, %v; want example.com/vlib.F", sig, ok) + } + } + if _, ok := j.externalOf(fnByName(t, prog, "example.com/dataflow/flow", "IfElse")); ok { + t.Error("flow.IfElse is external") + } +} + +func ptrTo(t types.Type) types.Type { return types.NewPointer(t) } + +// TestJoin_IsLibraryWithoutAFileFollowsL2: an object with no source file (no +// position) is classified by L2's rule — the go.mod module prefix — not by +// whether its package happened to be loaded, so L2 and L4 agree. +func TestJoin_IsLibraryWithoutAFileFollowsL2(t *testing.T) { + j := newJoin(loadInput(t, "dataflow")) + if j.modPath != "example.com/dataflow" { + t.Fatalf("modPath = %q, want example.com/dataflow", j.modPath) + } + sig := types.NewSignatureType(nil, nil, nil, nil, nil, false) + unloaded := types.NewFunc(token.NoPos, types.NewPackage("example.com/dataflow/notloaded", "notloaded"), "F", sig) + if j.isLibrary(unloaded) { + t.Error("an unloaded package under the module path is project code at L2, but L4 called it a library") + } + other := types.NewFunc(token.NoPos, types.NewPackage("example.com/other", "other"), "F", sig) + if !j.isLibrary(other) { + t.Error("a package outside the module path must be a library") + } +} + +// TestSDG_RangeFuncInPackageLevelLiteral: rangefuncpkg ranges over a function +// only inside a package-level `var Sum = func(){…}`. That literal has no +// callable (it is no unit, and is unmapped_ssa), yet range_func must still +// list its package, so the SDK does not certify "no flow" there. +func TestSDG_RangeFuncInPackageLevelLiteral(t *testing.T) { + res := Run(loadInput(t, "rangefuncpkg")) + var rf *Degraded + for i := range res.Degraded { + if res.Degraded[i].Code == CodeRangeFunc { + rf = &res.Degraded[i] + } + } + if rf == nil { + t.Fatalf("no range_func entry; degraded = %v", codesOf(res.Degraded)) + } + if !slices.Equal(rf.Packages, []string{"example.com/rangefuncpkg"}) || len(rf.Callables) != 0 { + t.Errorf("range_func = %+v, want package example.com/rangefuncpkg and no callables", rf) + } + if !hasCode(res.Degraded, CodeUnmappedSSA) { + t.Error("the package-level literal should also be unmapped_ssa") + } +} diff --git a/internal/sdg/libflow_test.go b/internal/sdg/libflow_test.go new file mode 100644 index 0000000..b94fed0 --- /dev/null +++ b/internal/sdg/libflow_test.go @@ -0,0 +1,224 @@ +package sdg + +import ( + "maps" + "math/rand" + "path/filepath" + "slices" + "strconv" + "testing" + + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +const libflowPkg = "example.com/dataflow/libflow" + +func libflowKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("libflow", "libflow.go"), Signature: libflowPkg + "." + name} +} + +// TestCHA_InterproceduralLibraryOrigin (round 2): a Store that came out of +// library code reaches an interface call through project code — a project +// function's result, a parameter, a struct field set in another method, a +// global, a closure capture — or through a library container element (map +// lookup, slice and array index, channel receive). Each site's targets +// include sqlStore.Find (the CHA complement), and the caller's footprint +// sees the global sqlStore.Find writes. +func TestCHA_InterproceduralLibraryOrigin(t *testing.T) { + in, _, d := runDispatch(t) + find := Target{Callable: libflowKey("sqlStore.Find")} + fps := footprints(in, d) + for _, c := range []struct{ fn, site string }{ + {"ViaReturn", "47:9"}, + {"use", "53:9"}, + {"handler.ViaField", "84:9"}, + {"ViaGlobal", "96:9"}, + {"ViaCapture.closure_1", "104:10"}, + {"ViaMap", "112:9"}, + {"ViaSlice", "117:9"}, + {"ViaChan", "122:9"}, + {"ViaArray", "127:9"}, + {"ViaPromoted", "195:9"}, + {"zzzViaPromoted", "205:9"}, + } { + if _, ok := d.sites[siteKey{libflowKey(c.fn), c.site}]; !ok { + t.Errorf("%s has no site %s: the keys moved", c.fn, c.site) + continue + } + if got := d.targets(libflowKey(c.fn), c.site); !slices.Contains(got, find) { + t.Errorf("targets(%s, %s) = %v, want sqlStore.Find among them", c.fn, c.site, got) + } + if fp := fps[Target{Callable: libflowKey(c.fn)}]; !slices.Contains(fp.GlobalWrites, libflowPkg+"::Last") { + t.Errorf("%s writes %v, want %s::Last", c.fn, fp.GlobalWrites, libflowPkg) + } + } +} + +// TestCHA_ProjectOnlyOriginNoTargets (round 2): ProjectOnly's +// relay(s).Find(id) is a call through a project function's result whose +// every value project code built (a memStore): it gets VTA's memStore.Find +// and the interface method, and no CHA target. +func TestCHA_ProjectOnlyOriginNoTargets(t *testing.T) { + _, _, d := runDispatch(t) + want := []Target{{Callable: libflowKey("Store.Find")}, {Callable: libflowKey("memStore.Find")}} + if got := d.targets(libflowKey("ProjectOnly"), "134:9"); !slices.Equal(got, want) { + t.Errorf("targets(ProjectOnly, 134:9) = %v, want %v", got, want) + } +} + +// TestCHA_PanicDegradesToNoCHA (N1): a panic inside the CHA complement is +// recovered: every CHA target is dropped (the dispatch is what it would be +// without CHA), no_cha is recorded, and VTA still runs. +func TestCHA_PanicDegradesToNoCHA(t *testing.T) { + in := loadInput(t, "dataflow") + defer func() { chaHook = nil }() + chaHook = func(fn *ssa.Function) { + if fn.Name() == "ViaMap" { + panic("cha boom") + } + } + res, d := run(in) + if codes := codesOf(res.Degraded); !slices.Equal(codes, []string{CodeNoCHA}) { + t.Fatalf("codes = %v, want [no_cha]", codes) + } + // ViaReturn's site keeps only D7 (VTA cannot see the sqlStore). + if got, want := d.targets(libflowKey("ViaReturn"), "47:9"), []Target{{Callable: libflowKey("Store.Find")}}; !slices.Equal(got, want) { + t.Errorf("targets(ViaReturn, 47:9) = %v, want %v (no CHA target after no_cha)", got, want) + } + // VTA ran: HandleDirect's VTA target is there. + if got := d.targets(wrapKey("HandleDirect"), "33:9"); !slices.Contains(got, Target{Callable: wrapKey("sqlStore.Find")}) { + t.Errorf("targets(HandleDirect, 33:9) = %v, want VTA's sqlStore.Find", got) + } +} + +// chaTargets is every CHA run's dispatch over in, with the origin fixpoint +// seeded in the order perm gives (nil: the sorted order), and the LO values +// of every project function, by function and instruction index. +func chaTargets(t *testing.T, in Input, perm func([]*ssa.Function)) (map[siteKey][]Target, map[string]bool) { + t.Helper() + prog, pkgs, _ := buildSSA(in) + funcs, _, err := funcSet(prog, pkgs) + if err != nil { + t.Fatal(err) + } + j := newJoin(in) + d := newDispatch(in) + defer func() { originSeedOrder = nil }() + originSeedOrder = perm + d.addCHA(newCHA(j, in), funcs) + o := newOrigins(newCHA(j, in), funcs) + originSeedOrder = perm + o.solve() + lo := map[string]bool{} + for _, fn := range o.order { + vals := o.eval(fn) + i := 0 + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + if v, ok := instr.(ssa.Value); ok && vals[v] { + lo[fn.String()+"#"+strconv.Itoa(i)] = true + } + i++ + } + } + for k, p := range fn.Params { + if vals[p] { + lo[fn.String()+"#p"+strconv.Itoa(k)] = true + } + } + } + return d.sites, lo +} + +// TestCHA_OriginOrderIndependent (round 3): the origin fixpoint is the +// least fixpoint, independent of the order functions are visited in. The +// worklist seeded in reverse and in two shuffled orders gives the same LO +// values and the same CHA targets as the sorted order. +func TestCHA_OriginOrderIndependent(t *testing.T) { + in := loadInput(t, "dataflow") + wantSites, wantLO := chaTargets(t, in, nil) + if !wantLO["example.com/dataflow/libflow.zzzGet#p0"] && len(wantLO) == 0 { + t.Fatal("no LO values: the fixture is vacuous") + } + perms := map[string]func([]*ssa.Function){ + "reverse": func(fs []*ssa.Function) { slices.Reverse(fs) }, + "shuffle1": func(fs []*ssa.Function) { + r := rand.New(rand.NewSource(1)) + r.Shuffle(len(fs), func(i, j int) { fs[i], fs[j] = fs[j], fs[i] }) + }, + "shuffle2": func(fs []*ssa.Function) { + r := rand.New(rand.NewSource(2)) + r.Shuffle(len(fs), func(i, j int) { fs[i], fs[j] = fs[j], fs[i] }) + }, + } + for name, perm := range perms { + sites, lo := chaTargets(t, in, perm) + if !maps.Equal(lo, wantLO) { + var diff []string + for k := range lo { + if !wantLO[k] { + diff = append(diff, "+"+k) + } + } + for k := range wantLO { + if !lo[k] { + diff = append(diff, "-"+k) + } + } + slices.Sort(diff) + t.Errorf("%s: LO differs from the sorted order: %v", name, diff) + } + if !maps.EqualFunc(sites, wantSites, slices.Equal[[]Target]) { + for k, ts := range sites { + if !slices.Equal(ts, wantSites[k]) { + t.Errorf("%s: site %v targets %v, sorted order %v", name, k, ts, wantSites[k]) + } + } + } + } +} + +// TestCHA_PanicRestoresEverySite (round 3): a panic after the CHA +// complement added targets restores every site to exactly what it held +// before CHA ran. d.add inserts in place, so a saved slice that shares its +// backing array with the live one would be shifted under it. +func TestCHA_PanicRestoresEverySite(t *testing.T) { + in := loadInput(t, "dataflow") + prog, pkgs, _ := buildSSA(in) + funcs, _, err := funcSet(prog, pkgs) + if err != nil { + t.Fatal(err) + } + j := newJoin(in) + d := newDispatch(in) + d.addInvokes(j, funcs) + // Spare capacity in every site, as append-grown slices have. + for k, ts := range d.sites { + d.sites[k] = slices.Grow(ts, 8) + } + before := map[siteKey][]Target{} + for k, ts := range d.sites { + before[k] = slices.Clone(ts) + } + defer func() { chaHook = nil }() + seen := 0 + chaHook = func(*ssa.Function) { + seen++ + if seen == len(funcs) { // the last function: every earlier site has its CHA targets + panic("cha boom") + } + } + if err := d.runCHA(j, in, funcs); err == nil { + t.Fatal("runCHA did not report the panic") + } + if len(d.sites) != len(before) { + t.Errorf("%d sites after the panic, %d before", len(d.sites), len(before)) + } + for k, ts := range before { + if got := d.sites[k]; !slices.Equal(got, ts) { + t.Errorf("site %v = %v after the panic, want %v", k, got, ts) + } + } +} diff --git a/internal/sdg/memlevers_test.go b/internal/sdg/memlevers_test.go new file mode 100644 index 0000000..165f00d --- /dev/null +++ b/internal/sdg/memlevers_test.go @@ -0,0 +1,117 @@ +package sdg + +import ( + "reflect" + "runtime/debug" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// TestSDG_OutputUnchangedByMemoryLevers: the memory levers (early cache +// release, interned port names, the L4 GC target) change memory, never +// facts: Run's result on dataflow and chi is deeply equal with them off +// (memLevers = false) and on. +func TestSDG_OutputUnchangedByMemoryLevers(t *testing.T) { + for _, name := range []string{"dataflow", "chi"} { + t.Run(name, func(t *testing.T) { + defer func(v bool) { memLevers = v }(memLevers) + memLevers = false + off := Run(loadInput(t, name)) + memLevers = true + on := Run(loadInput(t, name)) + if len(on.Callables) == 0 || len(on.ParamIn) == 0 { + t.Fatalf("empty overlay: %d callables, %d param_in", len(on.Callables), len(on.ParamIn)) + } + if !reflect.DeepEqual(off, on) { + for k, c := range off.Callables { + if !reflect.DeepEqual(c, on.Callables[k]) { + t.Errorf("callable %v differs:\n off %+v\n on %+v", k, c, on.Callables[k]) + break + } + } + t.Fatalf("Run(%s) differs with the memory levers on", name) + } + }) + } +} + +// TestRun_RestoresGCPercent: Run lowers the GC target only while it runs: +// the caller's setting is back on return, and a lower one is never raised. +func TestRun_RestoresGCPercent(t *testing.T) { + in := loadInput(t, "greeter") + for _, pct := range []int{200, 20} { + prev := debug.SetGCPercent(pct) + Run(in) + got := debug.SetGCPercent(prev) + if got != pct { + t.Errorf("GC percent after Run = %d, want the caller's %d", got, pct) + } + } +} + +// TestSDG_ReleasedCachesAreNotRebuilt (Task 15b): with the memory levers on, +// a unit's flow graph, call sites and wiring are each built at most once, +// and fill never recomputes summary edges releaseSCC produced ("summ" +// misses: none) — +// a cache released before its last reader is rebuilt silently, which costs +// time and memory but changes no fact, so only a miss count catches it — +// and fill leaves the per-unit caches empty. +func TestSDG_ReleasedCachesAreNotRebuilt(t *testing.T) { + for _, name := range []string{"dataflow", "chi"} { + t.Run(name, func(t *testing.T) { + defer func(v bool) { memLevers = v }(memLevers) + memLevers = true + builds := map[string]map[dataflow.L3Key]int{} + defer func(f func(string, dataflow.L3Key)) { onCacheMiss = f }(onCacheMiss) + onCacheMiss = func(cache string, k dataflow.L3Key) { + if builds[cache] == nil { + builds[cache] = map[dataflow.L3Key]int{} + } + builds[cache][k]++ + } + in := loadInput(t, name) + res, d := dispatchRun(in) + b := newSDGBuilder(in, d) + b.fill(&res) + for _, cache := range []string{"graph", "sites", "wired"} { + if len(builds[cache]) == 0 { + t.Errorf("no %s cache built", cache) + } + for k, n := range builds[cache] { + if n > 1 { + t.Errorf("%s of %s built %d times: released before its last reader", cache, k.Signature, n) + } + } + } + if n := len(builds["summ"]); n != 0 { + t.Errorf("fill recomputed the summary edges of %d units: releaseSCC's were released before fill read them", n) + } + if len(b.graph)+len(b.sites)+len(b.wired)+len(b.summ) != 0 { + t.Errorf("after fill: %d graphs, %d sites, %d wirings, %d summaries still cached", + len(b.graph), len(b.sites), len(b.wired), len(b.summ)) + } + }) + } +} + +// TestSDG_SummaryMissHookFires: the "summ" miss TestSDG_ReleasedCachesAreNotRebuilt +// counts is live — with the memory levers off, releaseSCC never runs, so +// fill computes every unit's summary edges through the miss path. +func TestSDG_SummaryMissHookFires(t *testing.T) { + defer func(v bool) { memLevers = v }(memLevers) + memLevers = false + misses := 0 + defer func(f func(string, dataflow.L3Key)) { onCacheMiss = f }(onCacheMiss) + onCacheMiss = func(cache string, _ dataflow.L3Key) { + if cache == "summ" { + misses++ + } + } + in := loadInput(t, "dataflow") + res, d := dispatchRun(in) + newSDGBuilder(in, d).fill(&res) + if misses != len(in.Units) { + t.Errorf("summ misses with the levers off = %d, want one per unit (%d)", misses, len(in.Units)) + } +} diff --git a/internal/sdg/models.go b/internal/sdg/models.go new file mode 100644 index 0000000..75b28f1 --- /dev/null +++ b/internal/sdg/models.go @@ -0,0 +1,145 @@ +package sdg + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "io" + "os" + "slices" + + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// warnf logs a warning (a test seam over utils.Warn). +var warnf = utils.Warn + +// LoadModels reads a --library-models file (spec §6): a JSON object whose +// keys are the external id's / tail (`strconv/Atoi`, +// `database/sql/DB.Query`) and whose values are lists of [in, out] pairs of +// `of` strings (`"s"`, `"_0"`, `"_recv"`, `""`, `":1"`, a +// by-ref param's name). An empty list is no flow. An empty path gives +// Models{}. +// +// Only the shape is checked here: which `of`s an external has is known once +// its formals are, so a pair naming one it lacks is reported by the summary +// pass (opaqueFlows), which then keeps that external's default. +func LoadModels(path string) (Models, error) { + if path == "" { + return Models{}, nil + } + data, err := os.ReadFile(path) + if err != nil { + return nil, fmt.Errorf("--library-models: %w", err) + } + m, err := parseModels(data) + if err != nil { + return nil, fmt.Errorf("--library-models %s: %w", path, err) + } + return m, nil +} + +func parseModels(data []byte) (Models, error) { + var raw map[string]json.RawMessage + dec := json.NewDecoder(bytes.NewReader(data)) + if err := dec.Decode(&raw); err != nil { + return nil, err + } + if _, err := dec.Token(); !errors.Is(err, io.EOF) { + return nil, errors.New("data after the top-level object") + } + if raw == nil { + return nil, errors.New("want a JSON object, got null") + } + out := make(Models, len(raw)) + for key, v := range raw { + var pairs [][]string + if err := json.Unmarshal(v, &pairs); err != nil { + return nil, fmt.Errorf("%s: want a list of [in, out] pairs: %w", key, err) + } + if pairs == nil { + return nil, fmt.Errorf("%s: want a list of [in, out] pairs, got null", key) + } + flows := make([][2]string, 0, len(pairs)) + for _, p := range pairs { + if len(p) != 2 { + return nil, fmt.Errorf("%s: pair %q: want [in, out]", key, p) + } + flows = append(flows, [2]string{p[0], p[1]}) + } + out[key] = flows + } + return out, nil +} + +// modelKey is the Models key of external t: /. +func (b *sdgBuilder) modelKey(t Target) (string, bool) { + e, ok := b.p.fx.ext[t.External] + if !ok { + return "", false + } + return e.ImportPath + "/" + e.Sig, true +} + +// model returns the Models entry of stand-in t, if any. +func (b *sdgBuilder) model(t Target) ([][2]string, string, bool) { + if t.External == "" || len(b.in.Models) == 0 { + return nil, "", false + } + key, ok := b.modelKey(t) + if !ok { + return nil, "", false + } + m, ok := b.in.Models[key] + return m, key, ok +} + +// modelFlows is stand-in t's flows under its model, or false when it has +// none or the model names an `of` t lacks: the caller then keeps the +// all-to-all default (a typo must never refute a flow). The unknown `of`s +// are one warning. +func (b *sdgBuilder) modelFlows(t Target, fi *formalInfo) (*flowSet, bool) { + model, key, ok := b.model(t) + if !ok { + return nil, false + } + b.modelled[key] = true + f := newFlowSet(len(fi.inOf), len(fi.outOf)) + var unknown []string + for _, pair := range model { + i, j := slices.Index(fi.inOf, pair[0]), slices.Index(fi.outOf, pair[1]) + if i < 0 { + unknown = append(unknown, "in "+pair[0]) + } + if j < 0 { + unknown = append(unknown, "out "+pair[1]) + } + if i >= 0 && j >= 0 { + f.set(i, j) + } + } + if len(unknown) > 0 { + warnf("--library-models: %s has no formal %v (formal_in %v, formal_out %v); keeping its default, every input to every output", + key, unknown, fi.inOf, fi.outOf) + return nil, false + } + return f, true +} + +// unusedModels warns, once per key and in sorted key order, of every Models +// entry no targeted external has: an uncalled external, or a misspelt key +// (`strconv.Atoi` for `strconv/Atoi`), which would otherwise silently keep +// the default. +func (b *sdgBuilder) unusedModels() { + keys := make([]string, 0, len(b.in.Models)) + for key := range b.in.Models { + if !b.modelled[key] { + keys = append(keys, key) + } + } + slices.Sort(keys) + for _, key := range keys { + warnf("--library-models: no call site targets %s; entry ignored (keys are /, e.g. strconv/Atoi)", key) + } +} diff --git a/internal/sdg/models_test.go b/internal/sdg/models_test.go new file mode 100644 index 0000000..9872e52 --- /dev/null +++ b/internal/sdg/models_test.go @@ -0,0 +1,258 @@ +package sdg + +import ( + "cmp" + "fmt" + "os" + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// stdlibKey is the L3Key of a callable declared in +// testdata/dataflow/sdg/stdlib.go. +func stdlibKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("sdg", "stdlib.go"), Signature: sdgPkg + "." + name} +} + +func byKey(vs []Vertex) []Vertex { + vs = slices.Clone(vs) + slices.SortFunc(vs, func(a, b Vertex) int { return cmp.Compare(a.Key, b.Key) }) + return vs +} + +// TestStandIn_VerticesFromSignature: fmt.Sprintf's stand-in has formal_in +// format and a (its declared params, named from go/types) and formal_out +// ; a ([]any, a slice) is also a by-ref out (the shared predicate). +func TestStandIn_VerticesFromSignature(t *testing.T) { + _, res, _ := runDispatch(t) + vs, ok := res.Externals["fmt.Sprintf"] + if !ok { + t.Fatalf("Externals lack fmt.Sprintf: %v", keysOf(res.Externals)) + } + checkVertices(t, "stand-in fmt.Sprintf", vs, + byKey([]Vertex{fin("0", "format"), fin("1", "a"), fout(":0", ""), fout(":1", "a")})) +} + +// TestStandIn_ByRefOut: json.Unmarshal(data []byte, v any) may write through +// v, so v is a formal_out after the result (data, a slice, is one too). +func TestStandIn_ByRefOut(t *testing.T) { + _, res, _ := runDispatch(t) + vs, ok := res.Externals["encoding/json.Unmarshal"] + if !ok { + t.Fatalf("Externals lack encoding/json.Unmarshal: %v", keysOf(res.Externals)) + } + checkVertices(t, "stand-in encoding/json.Unmarshal", vs, byKey([]Vertex{ + fin("0", "data"), fin("1", "v"), + fout(":0", ""), fout(":1", "data"), fout(":2", "v"), + })) +} + +// TestStandIn_EveryTargetedExternal: Externals holds exactly the externals +// some call site targets, each with the formals ports gives it; one with no +// formals at all (runtime.Gosched) is listed with an empty vertex list. +func TestStandIn_EveryTargetedExternal(t *testing.T) { + in, res, d := runDispatch(t) + p := newPorts(in, d) + want := map[string]bool{} + for _, ts := range d.sites { + for _, tg := range ts { + if tg.External != "" { + want[tg.External] = true + } + } + } + if got := keysOf(res.Externals); !slices.Equal(got, keysOf(want)) { + t.Errorf("Externals = %v, want %v", got, keysOf(want)) + } + for sig, vs := range res.Externals { + fi, fo := p.formals(Target{External: sig}) + checkVertices(t, "stand-in "+sig, vs, byKey(slices.Concat(fi, fo))) + } + vs, ok := res.Externals["runtime.Gosched"] + if !ok || vs == nil || len(vs) != 0 { + t.Errorf("runtime.Gosched: listed %v, vertices %#v; want an empty, non-nil list", ok, vs) + } +} + +// fillWith fills the L4 overlay of the dataflow fixture under models, and +// returns it with the warnings it logged. +func fillWith(t *testing.T, models Models) (Result, []string) { + t.Helper() + in, d := runPortsInput(t) + in.Models = models + var warns []string + defer func(f func(string, ...any)) { warnf = f }(warnf) + warnf = func(format string, args ...any) { warns = append(warns, fmt.Sprintf(format, args...)) } + var res Result + newSDGBuilder(in, d).fill(&res) + return res, warns +} + +const atoiSite = "27:10" + +// TestModels_OverrideCutsFlow: the model strconv/Atoi: [] (no flow) leaves +// atoi's call of it with no summary edge; by default it has one per result. +func TestModels_OverrideCutsFlow(t *testing.T) { + res, _ := fillWith(t, nil) + checkSummary(t, "default summary of atoi", siteSummaryOf(res, stdlibKey("atoi"), atoiSite), []SummaryEdge{ + se(atoiSite+"/actual_in:0", atoiSite+"/actual_out:0"), se(atoiSite+"/actual_in:0", atoiSite+"/actual_out:1"), + }) + res, warns := fillWith(t, Models{"strconv/Atoi": {}}) + if s := siteSummaryOf(res, stdlibKey("atoi"), atoiSite); len(s) != 0 { + t.Errorf("strconv/Atoi modelled with no flow; summary = %+v", s) + } + if len(warns) != 0 { + t.Errorf("warnings: %v", warns) + } +} + +// TestModels_PairsByOf: a model's pairs name formals by `of`: s → :0 +// keeps that one edge and drops the error result's. +func TestModels_PairsByOf(t *testing.T) { + res, warns := fillWith(t, Models{"strconv/Atoi": {{"s", ":0"}}}) + checkSummary(t, "summary of atoi", siteSummaryOf(res, stdlibKey("atoi"), atoiSite), + []SummaryEdge{se(atoiSite+"/actual_in:0", atoiSite+"/actual_out:0")}) + if len(warns) != 0 { + t.Errorf("warnings: %v", warns) + } +} + +// TestModels_UnknownOfFallsBackToAllToAll: a pair naming an `of` the +// external lacks is a warning, and the external keeps its all-to-all +// default: a typo never refutes a flow. +func TestModels_UnknownOfFallsBackToAllToAll(t *testing.T) { + for _, c := range []struct { + pair [2]string + unknown string + }{{[2]string{"str", ":0"}, "in str"}, {[2]string{"s", ""}, "out "}} { + pair := c.pair + res, warns := fillWith(t, Models{"strconv/Atoi": {{"s", ":1"}, pair}}) + checkSummary(t, fmt.Sprint("summary of atoi under ", pair), siteSummaryOf(res, stdlibKey("atoi"), atoiSite), []SummaryEdge{ + se(atoiSite+"/actual_in:0", atoiSite+"/actual_out:0"), se(atoiSite+"/actual_in:0", atoiSite+"/actual_out:1"), + }) + if len(warns) != 1 || !strings.Contains(warns[0], "strconv/Atoi") || !strings.Contains(warns[0], c.unknown) { + t.Errorf("pair %v: warnings %q, want one naming strconv/Atoi and the unknown of", pair, warns) + } + } +} + +// TestModels_UncalledEntryIgnored: an entry for an external no site targets +// (a misspelt key too: strconv.Atoi for strconv/Atoi) changes nothing and +// is one warning per key, in sorted key order. +func TestModels_UncalledEntryIgnored(t *testing.T) { + res, warns := fillWith(t, Models{"strconv.Atoi": {}, "example.com/nosuch/F": {}, "strconv/Atoi": {{"s", ":0"}, {"s", ":1"}}}) + want := []string{ + "--library-models: no call site targets example.com/nosuch/F; entry ignored (keys are /, e.g. strconv/Atoi)", + "--library-models: no call site targets strconv.Atoi; entry ignored (keys are /, e.g. strconv/Atoi)", + } + if !slices.Equal(warns, want) { + t.Errorf("warnings:\n got %q\n want %q", warns, want) + } + if s := siteSummaryOf(res, stdlibKey("atoi"), atoiSite); len(s) != 2 { + t.Errorf("summary of atoi = %+v, want the default", s) + } +} + +// TestModels_GenericExternalKey: a method of a generic library type is +// keyed by its origin signature (sync/atomic/Pointer.Store, never an +// instance's type arguments), so one entry covers every instantiation: +// StoreA (atomic.Pointer[A]) and StoreB (atomic.Pointer[B]) both lose +// their summary edges under `[]`. +func TestModels_GenericExternalKey(t *testing.T) { + storeA := dataflow.L3Key{Path: filepath.Join("generic", "a.go"), Signature: "example.com/dataflow/generic.StoreA"} + storeB := dataflow.L3Key{Path: filepath.Join("generic", "b.go"), Signature: "example.com/dataflow/generic.StoreB"} + in, d := runPortsInput(t) + b := newSDGBuilder(in, d) + key, ok := b.modelKey(Target{External: "sync/atomic.Pointer.Store"}) + if want := "sync/atomic/Pointer.Store"; !ok || key != want { + t.Fatalf("modelKey(sync/atomic.Pointer.Store) = %q, %v; want %q", key, ok, want) + } + res, _ := fillWith(t, nil) + for _, k := range []dataflow.L3Key{storeA, storeB} { + if s := res.Callables[k].Summary; len(s) != 4 { + t.Errorf("default summary of %s = %+v, want 4 edges (x, val → x, val)", k.Signature, s) + } + } + res, warns := fillWith(t, Models{"sync/atomic/Pointer.Store": {}}) + for _, k := range []dataflow.L3Key{storeA, storeB} { + if s := res.Callables[k].Summary; len(s) != 0 { + t.Errorf("sync/atomic/Pointer.Store modelled with no flow; summary of %s = %+v", k.Signature, s) + } + } + if len(warns) != 0 { + t.Errorf("warnings: %v", warns) + } +} + +func writeModels(t *testing.T, body string) string { + t.Helper() + path := filepath.Join(t.TempDir(), "models.json") + if err := os.WriteFile(path, []byte(body), 0o644); err != nil { + t.Fatal(err) + } + return path +} + +// TestModels_Load: a models file maps / to [in, out] +// pairs; [] is no flow; an empty path is no models. +func TestModels_Load(t *testing.T) { + m, err := LoadModels(writeModels(t, `{ + "strconv/Atoi": [], + "strings/TrimSpace": [["s", ""]], + "database/sql/DB.Query": [["query", ":0"], ["args", ":0"]] + }`)) + if err != nil { + t.Fatal(err) + } + want := Models{ + "strconv/Atoi": {}, + "strings/TrimSpace": {{"s", ""}}, + "database/sql/DB.Query": {{"query", ":0"}, {"args", ":0"}}, + } + if len(m) != len(want) { + t.Fatalf("models = %v, want %v", m, want) + } + for k, v := range want { + got, ok := m[k] + if !ok || !slices.Equal(got, v) || got == nil { + t.Errorf("models[%s] = %#v, want %#v", k, got, v) + } + } + m, err = LoadModels("") + if err != nil || m == nil || len(m) != 0 { + t.Errorf(`LoadModels("") = %v, %v; want Models{}, nil`, m, err) + } +} + +// TestModels_BadFileErrors: a missing file, malformed JSON and a value that +// is not a list of [in, out] string pairs are errors naming the path. +func TestModels_BadFileErrors(t *testing.T) { + missing := filepath.Join(t.TempDir(), "absent.json") + paths := map[string]string{"missing": missing} + for name, body := range map[string]string{ + "malformed": `{"strconv/Atoi": [`, + "not object": `[]`, + "null": `null`, + "null value": `{"strconv/Atoi": null}`, + "short pair": `{"strconv/Atoi": [["s"]]}`, + "long pair": `{"strconv/Atoi": [["s", ":0", "x"]]}`, + "number": `{"strconv/Atoi": [[0, ":0"]]}`, + "trailing": `{} {}`, + } { + paths[name] = writeModels(t, body) + } + for name, path := range paths { + m, err := LoadModels(path) + if err == nil { + t.Errorf("%s: LoadModels = %v, want an error", name, m) + continue + } + if !strings.Contains(err.Error(), path) { + t.Errorf("%s: error %q does not name %s", name, err, path) + } + } +} diff --git a/internal/sdg/params.go b/internal/sdg/params.go new file mode 100644 index 0000000..1d602ab --- /dev/null +++ b/internal/sdg/params.go @@ -0,0 +1,169 @@ +package sdg + +import ( + "cmp" + "slices" + "strings" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// params returns the param_in and param_out edges of unit k (spec §4.7), +// one set per target: +// +// - at each resolved call site with vertices: param_in from each actual_in +// to each target's formal_in it binds, and param_out from each target's +// formal_out to the actual_out it binds; +// - at each capture site (A6): param_in from each capture's actual_in to +// the closure's `:` formal_in, and param_out from the closure's +// `:` formal_out to the site's actual_out. +// +// Var is the target formal's `of`. An actual whose `of` a target lacks (a +// global another target touches) binds nothing in it, so it gets no edge. +func (b *sdgBuilder) params(k dataflow.L3Key) (in, out []ParamEdge) { + u, ok := b.p.units[k] + if !ok { + return nil, nil + } + caller := Target{Callable: k} + taken := make(map[string]bool, len(u.Calls)) + for _, c := range u.Calls { + taken[c.Key] = true + s := b.p.callSite(k, c.Key, b.p.siteTargets(k, c.Key)) + for _, pb := range s.BindIn { + fi := b.formalInfo(pb.Target) + if j, ok := fi.inPos[pb.Formal]; ok { + in = append(in, ParamEdge{Src: Ref{caller, pb.Actual}, Dst: Ref{pb.Target, pb.Formal}, Var: fi.inOf[j]}) + } + } + for _, pb := range s.BindOut { + fi := b.formalInfo(pb.Target) + if j, ok := fi.outPos[pb.Formal]; ok { + out = append(out, ParamEdge{Src: Ref{pb.Target, pb.Formal}, Dst: Ref{caller, pb.Actual}, Var: fi.outOf[j]}) + } + } + } + for key := range b.p.keptKeys(k) { + taken[key] = true + } + for _, cs := range b.p.capSites(u, taken) { + lk, ok := b.p.byFn[cs.lit] + if !ok { + continue + } + lit := Target{Callable: lk} + fs := b.p.formalSet(lit) + for i, of := range cs.ofs { + if f, ok := fs.inByOf[of]; ok { + in = append(in, ParamEdge{Src: Ref{caller, cs.in[i]}, Dst: Ref{lit, f}, Var: of}) + } + if cs.out[i] == "" { + continue + } + if f, ok := fs.outByOf[of]; ok { + out = append(out, ParamEdge{Src: Ref{lit, f}, Dst: Ref{caller, cs.out[i]}, Var: of}) + } + } + } + return in, out +} + +// compareParamEdge orders param edges by (Src owner, Src key, Dst owner, +// Dst key, Var), owners by compareTarget: stable before ids are minted. +func compareParamEdge(a, b ParamEdge) int { + return cmp.Or( + compareTarget(a.Src.Owner, b.Src.Owner), strings.Compare(a.Src.Key, b.Src.Key), + compareTarget(a.Dst.Owner, b.Dst.Owner), strings.Compare(a.Dst.Key, b.Dst.Key), + strings.Compare(a.Var, b.Var), + ) +} + +// releaseUnit drops what fill cached for unit k once its overlay is in the +// Result: the wiring (the Result shares its slice), its summary edges and +// call sites, and the ports' call and kept-node indexes of k (read only by +// k's own vertices, wiring and params). +func (b *sdgBuilder) releaseUnit(k dataflow.L3Key) { + delete(b.wired, k) + delete(b.summ, k) + delete(b.sites, k) + delete(b.graph, k) + delete(b.p.calls, k) + delete(b.p.kept, k) +} + +func sortParamEdges(es []ParamEdge) []ParamEdge { + slices.SortFunc(es, compareParamEdge) + return slices.Compact(es) +} + +// fill computes the flows and fills res's L4 overlay, serially: +// +// - Callables[k] for every unit: its vertices, port edges and call-site +// summary edges; +// - ParamIn / ParamOut over every unit's call and capture sites, sorted +// by compareParamEdge; +// - Externals[sig]: the formal vertices (sorted by key) of every stand-in +// some call site targets (spec §6), an empty list for one with no +// formals (runtime.Gosched); +// - Callables[k] (vertices only) for every interface method or bodiless +// declaration some param edge reaches (spec §4.8), so no param edge +// dangles. +func (b *sdgBuilder) fill(res *Result) { + b.computeFlows() + res.Callables = make(map[dataflow.L3Key]CallableSDG, len(b.in.Units)) + var ins, outs []ParamEdge + for _, u := range b.in.Units { + k := u.Key + res.Callables[k] = CallableSDG{ + Vertices: b.p.vertices(k), + Ddg: b.portEdges(k), + Summary: b.summary(k), + } + pi, po := b.params(k) + ins, outs = append(ins, pi...), append(outs, po...) + if memLevers { + // The Result now holds unit k's overlay; nothing below reads + // its caches again. + b.releaseUnit(k) + } + } + res.ParamIn, res.ParamOut = sortParamEdges(ins), sortParamEdges(outs) + + formals := func(t Target) []Vertex { + in, out := b.p.formals(t) + vs := slices.Concat(in, out) + if vs == nil { + vs = []Vertex{} + } + slices.SortFunc(vs, func(a, b Vertex) int { return strings.Compare(a.Key, b.Key) }) + return vs + } + for _, ts := range b.d.sites { + for _, t := range ts { + if t.External == "" { + continue + } + if res.Externals == nil { + res.Externals = map[string][]Vertex{} + } + if _, done := res.Externals[t.External]; !done { + res.Externals[t.External] = formals(t) + } + } + } + + // The in-project targets with no unit some param edge reaches. + for _, list := range [][]ParamEdge{res.ParamIn, res.ParamOut} { + for _, e := range list { + for _, t := range []Target{e.Src.Owner, e.Dst.Owner} { + if _, unit := b.p.units[t.Callable]; unit || t.External != "" { + continue + } + if _, done := res.Callables[t.Callable]; !done { + res.Callables[t.Callable] = CallableSDG{Vertices: formals(t)} + } + } + } + } + b.unusedModels() +} diff --git a/internal/sdg/run.go b/internal/sdg/run.go new file mode 100644 index 0000000..3aafa6b --- /dev/null +++ b/internal/sdg/run.go @@ -0,0 +1,187 @@ +package sdg + +import ( + "fmt" + "runtime/debug" + "time" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/callgraph/vta" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// vtaCallGraph is vta.CallGraph (a test seam for the recovered-panic path). +var vtaCallGraph = vta.CallGraph + +// memLevers enables the output-neutral memory levers of Run (spec §4.13): +// early release of the builder's per-unit caches, interned port names, and +// the lower GC target. A test seam: TestSDG_OutputUnchangedByMemoryLevers +// turns it off to check the result is the same. +var memLevers = true + +// l4GCPercent is the GC target during Run. L4's live heap grows by the +// whole-program overlay, and GOGC=100 headroom doubles the transient +// garbage on top of it; 50 trades some GC time for that headroom. +const l4GCPercent = 50 + +// Run builds the L4 overlay of in. It builds SSA over the project packages, +// joins it to the L3 units, resolves every call site's dispatch targets +// (L2's static callee ∪ VTA's callees), and returns the call-graph edges and +// externals only VTA found, the degradation, and the SDG: every unit's +// vertices, port edges and summaries, the param edges, and the stand-in +// formals. +// +// Order: the join is computed before VTA, so it never reads a program a VTA +// panic may have wedged; after no_vta, later stages use the program for the +// join alone. The degradations are, in order: partial_types (the load), +// partial_ssa / no_ssa (SSA), +// l3_skipped and range_func (from the units, whether or not SSA built), +// unmapped_ssa (the join), no_cha (the CHA complement), no_vta. +// +// Run lowers the process-wide GC target to l4GCPercent while it runs and +// restores the previous one on return (debug.SetGCPercent is global: a +// caller running other work concurrently shares the lower target). A target +// already at or below it, or GC off (GOGC=off), is kept. L1–L3 and the +// emission run at the caller's setting. +func Run(in Input) Result { + if memLevers { + prev := debug.SetGCPercent(l4GCPercent) + if prev < l4GCPercent { + debug.SetGCPercent(prev) + } + defer debug.SetGCPercent(prev) + } + res, _ := run(in) + return res +} + +// run is Run that also returns the call-site dispatch. Every site keeps its +// static L2 targets whether or not SSA or VTA ran, so the SDG is built on +// every path. +func run(in Input) (Result, *dispatch) { + res, d := dispatchRun(in) + overlay(in, d, &res) + return res, d +} + +// dispatchRun is run up to the dispatch: SSA, the join and VTA. +func dispatchRun(in Input) (Result, *dispatch) { + d := newDispatch(in) + var deg []Degraded + if dg := partialTypes(in); dg != nil { + deg = append(deg, *dg) + } + start := time.Now() + prog, pkgs, ssaDeg := buildSSA(in) + deg = append(deg, ssaDeg...) + utils.Info("L4: SSA built %d project packages in %s", len(pkgs), time.Since(start).Round(time.Millisecond)) + for _, dg := range []*Degraded{l3Skipped(in), rangeFunc(in)} { + if dg != nil { + deg = append(deg, *dg) + } + } + if len(pkgs) == 0 { + return d.result(deg), d + } + + // The VTA function set is built under recover: it builds wrappers on + // demand, and a panic there may wedge the program, so neither the join + // nor VTA then reads it. + start = time.Now() + funcs, all, err := funcSet(prog, pkgs) + if err != nil { + utils.Debug("L4: VTA function set failed: %v", err) + return d.result(append(deg, Degraded{Code: CodeNoVTA, Detail: err.Error()})), d + } + j := newJoin(in) + cov := j.coverage(funcs) + utils.Info("L4: SSA join: %d functions (%d by wrapper, %d without a unit), %d calls; %d functions, %d calls unmapped in %s", + cov.Funcs, cov.Wrappers, cov.NoUnit, cov.Calls, cov.Unmapped, cov.UnmappedCalls, time.Since(start).Round(time.Millisecond)) + if dg := j.degraded(cov); dg != nil { + deg = append(deg, *dg) + } + d.addInvokes(j, funcs) + if err := d.runCHA(j, in, funcs); err != nil { + utils.Debug("L4: CHA complement failed: %v", err) + deg = append(deg, Degraded{Code: CodeNoCHA, Detail: err.Error()}) + } + + start = time.Now() + cg, err := runVTA(all) + if err != nil { + utils.Debug("L4: VTA failed: %v", err) + deg = append(deg, Degraded{Code: CodeNoVTA, Detail: err.Error()}) + } else { + d.addVTA(j, cg) + utils.Info("L4: VTA over %d functions (%d library targets): %d edges in %s", + len(all), len(all)-len(funcs), edgeCount(cg), time.Since(start).Round(time.Millisecond)) + } + res := d.result(deg) + utils.Info("L4: dispatch: %d call sites with targets; %d call_graph edges and %d externals added by VTA", + len(d.sites), len(res.CallEdges), len(res.NewExternals)) + return res, d +} + +// overlay fills res's vertices, port edges, summaries and param edges +// (summary.go, params.go), serially. It reads the dispatch, the L3 results +// and go/types, never the SSA program, so it runs after a no_vta too (the +// dispatch then holds the L2 static targets). +func overlay(in Input, d *dispatch, res *Result) { + start := time.Now() + newSDGBuilder(in, d).fill(res) + var nv, nd, np, ns int + for _, c := range res.Callables { + nv, nd, ns = nv+len(c.Vertices), nd+len(c.Ddg), ns+len(c.Summary) + for _, e := range c.Ddg { + if e.Prov == ProvPointsTo { + np++ + } + } + } + utils.Info("L4: %d vertices, %d port edges (%d points-to), %d summary edges, %d param_in, %d param_out, %d stand-ins in %s", + nv, nd, np, ns, len(res.ParamIn), len(res.ParamOut), len(res.Externals), time.Since(start).Round(time.Millisecond)) +} + +// funcSet is projectFuncs under recover: it builds functions on demand +// (MethodValue wrappers), so a builder panic is turned into an error. It +// returns the project functions (the join is measured on them) and the VTA +// set: those plus the target-only library functions. +func funcSet(prog *ssa.Program, pkgs []*ssa.Package) (funcs, all map[*ssa.Function]bool, err error) { + defer func() { + if r := recover(); r != nil { + funcs, all, err = nil, nil, fmt.Errorf("panic: %v", r) + } + }() + funcs, targetOnly := projectFuncs(prog, pkgs) + all = make(map[*ssa.Function]bool, len(funcs)+len(targetOnly)) + for fn := range funcs { + all[fn] = true + } + for fn := range targetOnly { + all[fn] = true + } + return funcs, all, nil +} + +// runVTA runs VTA over funcs with no initial graph (VTA then uses its +// internal lazy CHA over that set), turning a panic into an error. VTA builds +// functions on demand (method lookups), so it may wedge the program: after +// an error, later stages must not read the program beyond the join. +func runVTA(funcs map[*ssa.Function]bool) (cg *callgraph.Graph, err error) { + defer func() { + if r := recover(); r != nil { + cg, err = nil, fmt.Errorf("panic: %v", r) + } + }() + return vtaCallGraph(funcs, nil), nil +} + +func edgeCount(cg *callgraph.Graph) int { + n := 0 + for _, node := range cg.Nodes { + n += len(node.Out) + } + return n +} diff --git a/internal/sdg/scc.go b/internal/sdg/scc.go new file mode 100644 index 0000000..8b63133 --- /dev/null +++ b/internal/sdg/scc.go @@ -0,0 +1,85 @@ +package sdg + +import "slices" + +// sccs returns the strongly connected components of the graph nodes and succ +// span (spec §4.9), callees first: a component comes after every component it +// reaches. Nodes only succ reaches are included. Each component is sorted by +// compareTarget. +// +// It is Tarjan's algorithm with an explicit stack, so a deep call chain +// cannot overflow the goroutine stack. The roots are visited in +// compareTarget order and each successor list is sorted and deduplicated +// first, so the result depends on neither the order of nodes nor the order +// succ returns. +func sccs(nodes []Target, succ func(Target) []Target) [][]Target { + roots := slices.Clone(nodes) + slices.SortFunc(roots, compareTarget) + roots = slices.Compact(roots) + + type vertex struct { + index, low int + onStack bool + } + type frame struct { + v Target + succ []Target + next int + } + state := make(map[Target]*vertex, len(roots)) + var stack []Target // Tarjan's stack + var calls []frame // the explicit DFS stack + var out [][]Target + + visit := func(v Target) { + n := len(state) + state[v] = &vertex{index: n, low: n, onStack: true} + stack = append(stack, v) + s := slices.Clone(succ(v)) + slices.SortFunc(s, compareTarget) + calls = append(calls, frame{v: v, succ: slices.Compact(s)}) + } + for _, r := range roots { + if state[r] != nil { + continue + } + visit(r) + for len(calls) > 0 { + top := len(calls) - 1 + f := &calls[top] + sv := state[f.v] + if f.next < len(f.succ) { + w := f.succ[f.next] + f.next++ + switch sw := state[w]; { + case sw == nil: + visit(w) + case sw.onStack: + sv.low = min(sv.low, sw.index) + } + continue + } + v := f.v + calls = calls[:top] + if top > 0 { + p := state[calls[top-1].v] + p.low = min(p.low, sv.low) + } + if sv.low != sv.index { + continue + } + i := len(stack) - 1 + for stack[i] != v { + i-- + } + comp := slices.Clone(stack[i:]) + stack = stack[:i] + for _, w := range comp { + state[w].onStack = false + } + slices.SortFunc(comp, compareTarget) + out = append(out, comp) + } + } + return out +} diff --git a/internal/sdg/ssa.go b/internal/sdg/ssa.go new file mode 100644 index 0000000..3177453 --- /dev/null +++ b/internal/sdg/ssa.go @@ -0,0 +1,316 @@ +package sdg + +import ( + "fmt" + "go/types" + "slices" + "sort" + "strings" + + "golang.org/x/tools/go/packages" + "golang.org/x/tools/go/ssa" + "golang.org/x/tools/go/ssa/ssautil" + "golang.org/x/tools/go/types/typeutil" + + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// buildHook, when set, runs just before each package's Build (a test seam +// for the recovered-panic path). +var buildHook func(*ssa.Package) + +// callHook, when set, runs on the static callee of every call instruction SSA +// emits, inside the build (a test seam for panics mid-build: after a builder +// has created shared generic instances, or while a wrapper is built on demand). +var callHook func(*types.Func) + +// installCallHook wires callHook into prog through the builder's no-return +// predicate, which SSA consults on every static call it emits. Production +// runs never set callHook, so prog keeps its nil predicate. +func installCallHook(prog *ssa.Program) { + if hook := callHook; hook != nil { + prog.SetNoReturn(func(f *types.Func) bool { hook(f); return false }) + } +} + +// buildSSA builds go/ssa over the project packages (projectPackages: every +// loaded variant) (spec §4.8, D2). +// +// ssautil.Packages gets only the project packages, so SSA bodies exist for +// them alone; every dependency (stdlib, modules, vendored code) is created +// from types, without bodies. The mode is InstantiateGenerics (VTA needs it) +// | BuildSerially. Each package is then built on its own, sorted by path, +// under recover: +// +// - A package that is ill-typed (pkg.IllTyped, which also covers a broken +// dependency) or that ssautil left nil gets no SSA. +// - A package whose Build panics gets no SSA either, and the whole program +// is discarded and rebuilt without it (it then enters from types, as a +// dependency). A builder that panics never marks the generic instances +// and wrappers it created as done, so any later build or method lookup +// sharing one would block forever — not a panic, and not recoverable. +// The rebuild repeats until a pass builds every remaining package, at +// most once per package. +// - Every package that got no SSA is listed under one partial_ssa entry; +// when no package built, no_ssa follows. +// +// It returns the built packages in path order. +func buildSSA(in Input) (*ssa.Program, []*ssa.Package, []Degraded) { + roots := projectPackages(in) + var failed []string + var ok []*packages.Package + for _, p := range roots { + if p.IllTyped { + failed = append(failed, p.PkgPath) + continue + } + ok = append(ok, p) + } + + var prog *ssa.Program + var built []*ssa.Package + for len(ok) > 0 { + var spkgs []*ssa.Package + prog, spkgs = ssautil.Packages(ok, ssa.InstantiateGenerics|ssa.BuildSerially) + installCallHook(prog) + built = built[:0] + var good []*packages.Package + panicked := -1 + for i, sp := range spkgs { + if sp == nil { + failed = append(failed, ok[i].PkgPath) + continue + } + if err := buildPackage(sp); err != nil { + utils.Debug("L4: SSA build of %s failed: %v; rebuilding without it", ok[i].PkgPath, err) + failed = append(failed, ok[i].PkgPath) + panicked = i + break + } + good = append(good, ok[i]) + built = append(built, sp) + } + if panicked < 0 { + break + } + // Drop this (possibly wedged) program; rebuild from the packages not + // yet known bad: the ones that built, and the ones not reached. + ok = append(good, ok[panicked+1:]...) + prog, built = nil, nil + } + + var deg []Degraded + if len(failed) > 0 { + sort.Strings(failed) + failed = slices.Compact(failed) // two variants of one path + deg = append(deg, Degraded{ + Code: CodePartialSSA, + Detail: fmt.Sprintf("%d package(s) have no SSA (type errors or a recovered build panic)", len(failed)), + Packages: failed, + }) + } + if len(built) == 0 && len(roots) > 0 { + deg = append(deg, Degraded{Code: CodeNoSSA, Detail: "no project package could be built in SSA form"}) + } + return prog, built, deg +} + +// buildPackage builds one SSA package, turning a panic into an error. +func buildPackage(p *ssa.Package) (err error) { + defer func() { + if r := recover(); r != nil { + err = fmt.Errorf("panic: %v", r) + } + }() + if buildHook != nil { + buildHook(p) + } + p.Build() + return nil +} + +// projectPackages returns the loaded packages that hold project code (a Go +// file under ProjectDir, not vendored), sorted by (path, ID): every variant +// of in.Variants, else in.Pkgs. With tests loaded, a package path has two +// variants — the test variant "p [p.test]" (in.Pkgs keeps only it) and the +// plain p every other package imports — and both are project packages: SSA +// needs the plain p's bodies for VTA to see the types that flow through it +// (F2). The variants share their files' syntax, so both join the same +// units. Generated test mains live in the build cache and so are never +// project packages. +func projectPackages(in Input) []*packages.Package { + all := in.Variants + if all == nil { + for _, p := range in.Pkgs { + all = append(all, p) + } + } + var out []*packages.Package + for _, p := range all { + if isProject(in.ProjectDir, p) { + out = append(out, p) + } + } + sort.Slice(out, func(i, j int) bool { + if out[i].PkgPath != out[j].PkgPath { + return out[i].PkgPath < out[j].PkgPath + } + return out[i].ID < out[j].ID + }) + return out +} + +func isProject(dir string, p *packages.Package) bool { + for _, f := range p.GoFiles { + if utils.IsWithin(dir, f) && !utils.IsVendored(utils.RelativePath(dir, f)) { + return true + } + } + return false +} + +// projectFuncs is the VTA function set (spec §4.8, VTA scope). funcs holds +// the SSA functions of the built project packages — package-level functions +// and inits, every declared method (generic origins included), the method-set +// wrappers of their non-generic named types — and, transitively, their +// anonymous functions and every function their code refers to that is project +// code (generic instances of project functions, $bound / $thunk wrappers). +// Its code is what VTA propagates types through, so VTA's cost stays bounded +// by the project's size; the SSA join is measured on it. +// +// targetOnly holds the functions declared outside the project that a dynamic +// call in project code may reach, which VTA can return only when they are in +// its set (it intersects the propagated types with a lazy CHA over the set): +// +// - a library function used as a value in project code (any operand but +// the static callee of a call: VTA resolves static calls without the set); +// - the methods (prog.MethodValue) of each type that a MakeInterface in +// project code converts to an interface — the dynamic types an interface +// call there may dispatch to — that are not project code: a library +// type's methods, or a project type's wrappers of promoted library +// methods. A project method found this way joins funcs. +// +// A library function has no SSA body (its package is created from types), so +// targetOnly adds no propagation work; a library promotion wrapper's body is +// a single synthetic call. +func projectFuncs(prog *ssa.Program, pkgs []*ssa.Package) (funcs, targetOnly map[*ssa.Function]bool) { + project := make(map[*types.Package]bool, len(pkgs)) + for _, p := range pkgs { + project[p.Pkg] = true + } + funcs = map[*ssa.Function]bool{} + targetOnly = map[*ssa.Function]bool{} + var work []*ssa.Function + add := func(fn *ssa.Function) { + if fn != nil && !funcs[fn] && isProjectFunc(fn, project) { + funcs[fn] = true + work = append(work, fn) + } + } + addTarget := func(fn *ssa.Function) { + switch { + case fn == nil || funcs[fn]: + case isProjectFunc(fn, project): + add(fn) + default: + targetOnly[fn] = true + } + } + var converted typeutil.Map // types already expanded by a MakeInterface + addMethods := func(t types.Type) { + if converted.At(t) != nil { + return + } + converted.Set(t, true) + if _, isParam := types.Unalias(t).(*types.TypeParam); isParam || types.IsInterface(t) { + return + } + mset := prog.MethodSets.MethodSet(t) + for i := 0; i < mset.Len(); i++ { + addTarget(prog.MethodValue(mset.At(i))) // nil for a generic method or type + } + } + for _, p := range pkgs { + names := make([]string, 0, len(p.Members)) + for name := range p.Members { + names = append(names, name) + } + sort.Strings(names) + for _, name := range names { + switch m := p.Members[name].(type) { + case *ssa.Function: + add(m) + case *ssa.Type: + named, ok := types.Unalias(m.Type()).(*types.Named) + if !ok { + continue + } + for i := 0; i < named.NumMethods(); i++ { + add(prog.FuncValue(named.Method(i))) + } + if named.TypeParams() == nil && !types.IsInterface(named) { + for _, t := range []types.Type{named, types.NewPointer(named)} { + mset := prog.MethodSets.MethodSet(t) + for i := 0; i < mset.Len(); i++ { + add(prog.MethodValue(mset.At(i))) + } + } + } + } + } + } + var buf [10]*ssa.Value + for len(work) > 0 { + fn := work[len(work)-1] + work = work[:len(work)-1] + for _, anon := range fn.AnonFuncs { + add(anon) + } + for _, b := range fn.Blocks { + for _, instr := range b.Instrs { + var callee *ssa.Value + if call, ok := instr.(ssa.CallInstruction); ok && !call.Common().IsInvoke() { + callee = &call.Common().Value + } + for _, op := range instr.Operands(buf[:0]) { + g, ok := (*op).(*ssa.Function) + switch { + case !ok: + case op == callee: + add(g) // a static callee: VTA needs it only as project code + default: + addTarget(g) + } + } + if mi, ok := instr.(*ssa.MakeInterface); ok { + addMethods(mi.X.Type()) + } + } + } + } + return funcs, targetOnly +} + +// isProjectFunc reports whether fn's code is project code: it belongs to a +// built project package, is an instance (or a closure of one) of a project +// generic, or is a synthetic wrapper ($bound, $thunk, promotion, pointer +// indirection) of a project method. +func isProjectFunc(fn *ssa.Function, project map[*types.Package]bool) bool { + for f := fn; f != nil; f = f.Parent() { + if f.Pkg != nil { + return project[f.Pkg.Pkg] + } + if o := f.Origin(); o != nil && o.Pkg != nil { + return project[o.Pkg.Pkg] + } + if f.Parent() == nil { + if strings.HasPrefix(f.Synthetic, "from type information") { + return false + } + if obj := f.Object(); obj != nil { + return project[obj.Pkg()] + } + } + } + return false +} diff --git a/internal/sdg/ssa_test.go b/internal/sdg/ssa_test.go new file mode 100644 index 0000000..3bf3178 --- /dev/null +++ b/internal/sdg/ssa_test.go @@ -0,0 +1,308 @@ +package sdg + +import ( + "go/types" + "path/filepath" + "runtime" + "slices" + "strings" + "testing" + "time" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + "golang.org/x/tools/go/ssa/ssautil" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// loadInput loads testdata/ the way the analyzer does at -a 4 +// (symbol-table builder, tests skipped, the L2 call graph, L3 with side tables) and returns the +// sdg input over it. +func loadInput(t *testing.T, name string) Input { + t.Helper() + return loadInputOpts(t, name, true) +} + +// loadInputOpts is loadInput with the --skip-tests choice. +func loadInputOpts(t *testing.T, name string, skipTests bool) Input { + t.Helper() + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "testdata", name)) + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, skipTests) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + units := b.L3Units(st) + return Input{ + Fset: b.Fset(), Pkgs: b.Pkgs(), Variants: b.Loaded(), ProjectDir: dir, SymbolTable: st, + Externals: cg.Externals(), CallGraph: edges, + Units: units, L3: dataflow.RunWithSideTables(b.Fset(), units, 3, 1), K: 3, Jobs: 1, + } +} + +// dataflowPkgs are the project packages of testdata/dataflow (tests +// skipped), sorted. +var dataflowPkgs = []string{ + "example.com/dataflow/asm", "example.com/dataflow/flow", "example.com/dataflow/generic", "example.com/dataflow/httpsql", "example.com/dataflow/initclosure", "example.com/dataflow/inits", + "example.com/dataflow/l1", "example.com/dataflow/libflow", "example.com/dataflow/libstore", + "example.com/dataflow/linedir", "example.com/dataflow/sdg", "example.com/dataflow/variant/p", "example.com/dataflow/variant/u", "example.com/dataflow/vend", + "example.com/dataflow/wrap", "example.com/dataflow/xfile", "example.com/dataflow/xtest", "example.com/dataflow/xtest/user", +} + +// TestSSA_BuildsProjectPackagesOnly: bodies are built for the project's +// packages only. Every function with syntax lies in a project file, and no +// function declared outside the project (stdlib, the vendored vlib) has blocks. +func TestSSA_BuildsProjectPackagesOnly(t *testing.T) { + in := loadInput(t, "dataflow") + prog, pkgs, deg := buildSSA(in) + if len(deg) != 0 { + t.Fatalf("degraded = %+v, want none", deg) + } + var built []string + for _, p := range pkgs { + built = append(built, p.Pkg.Path()) + } + want := dataflowPkgs + if !slices.Equal(built, want) { + t.Fatalf("built packages = %v, want %v", built, want) + } + + project := map[string]bool{} + for _, p := range want { + project[p] = true + } + withBody := 0 + for fn := range ssautil.AllFunctions(prog) { + if syn := fn.Syntax(); syn != nil { + file := in.Fset.File(syn.Pos()).Name() + if !utils.IsWithin(in.ProjectDir, file) || utils.IsVendored(utils.RelativePath(in.ProjectDir, file)) { + t.Errorf("%s has syntax outside the project: %s", fn, file) + } + } + // Wrappers ($bound, promotion, thunks) are synthesized with blocks + // for whatever they wrap; only declared code and instances count. + declared := fn.Synthetic == "" || strings.HasPrefix(fn.Synthetic, "instance of") + if declared && len(fn.Blocks) > 0 { + if !project[homePath(fn)] { + t.Errorf("%s (package %q) has blocks; only project bodies are built", fn, homePath(fn)) + } + withBody++ + } + } + if withBody == 0 { + t.Fatal("no project function was built") + } + if fn := prog.ImportedPackage("example.com/dataflow/flow").Func("IfElse"); fn == nil || len(fn.Blocks) == 0 { + t.Error("flow.IfElse has no SSA body") + } + osPkg := prog.ImportedPackage("os") + if osPkg == nil { + t.Fatal("dependency os has no SSA package") + } + if fn := osPkg.Func("Getenv"); fn == nil || len(fn.Blocks) != 0 { + t.Errorf("os.Getenv = %v, want a bodiless function from types", fn) + } + vlib := prog.ImportedPackage("example.com/vlib") + if vlib == nil { + t.Fatal("vendored dependency vlib has no SSA package") + } + if fn := vlib.Func("F"); fn == nil || len(fn.Blocks) != 0 { + t.Errorf("vlib.F = %v, want a bodiless function from types (vendored code is not the project)", fn) + } +} + +// TestSSA_PanicDegradesPackage: a panic while one package builds is +// recovered; that package is listed under partial_ssa and the rest still +// build. When every package panics, no_ssa is added. +func TestSSA_PanicDegradesPackage(t *testing.T) { + in := loadInput(t, "dataflow") + defer func() { buildHook = nil }() + + buildHook = func(p *ssa.Package) { + if p.Pkg.Path() == "example.com/dataflow/flow" { + panic("boom") + } + } + prog, pkgs, deg := buildSSA(in) + if len(deg) != 1 || deg[0].Code != "partial_ssa" || !slices.Equal(deg[0].Packages, []string{"example.com/dataflow/flow"}) { + t.Fatalf("degraded = %+v, want one partial_ssa naming example.com/dataflow/flow", deg) + } + for _, p := range pkgs { + if p.Pkg.Path() == "example.com/dataflow/flow" { + t.Error("the panicked package is still returned as built") + } + } + if len(pkgs) != len(dataflowPkgs)-1 { + t.Errorf("built %d packages, want the other %d", len(pkgs), len(dataflowPkgs)-1) + } + if fn := prog.ImportedPackage("example.com/dataflow/l1"); fn == nil || len(fn.Members) == 0 { + t.Error("l1 was not built after flow panicked") + } + + buildHook = func(*ssa.Package) { panic("boom") } + _, pkgs, deg = buildSSA(in) + if len(pkgs) != 0 { + t.Errorf("built %d packages, want 0", len(pkgs)) + } + var codes []string + for _, d := range deg { + codes = append(codes, d.Code) + } + if !slices.Equal(codes, []string{"partial_ssa", "no_ssa"}) { + t.Fatalf("codes = %v, want [partial_ssa no_ssa]", codes) + } + if !slices.Equal(deg[0].Packages, dataflowPkgs) { + t.Errorf("partial_ssa lists %v, want all %d packages", deg[0].Packages, len(dataflowPkgs)) + } + if res := Run(in); len(res.Degraded) != 2 || res.Degraded[1].Code != "no_ssa" { + t.Errorf("Run degraded = %+v, want partial_ssa + no_ssa and no VTA", res.Degraded) + } +} + +// TestSSA_IllTypedPackageDegrades: a package with type errors gets no SSA and +// is listed under partial_ssa; it is the only package, so no_ssa follows. +func TestSSA_IllTypedPackageDegrades(t *testing.T) { + in := loadInput(t, "typeerr") + _, pkgs, deg := buildSSA(in) + if len(pkgs) != 0 { + t.Errorf("built %d packages from an ill-typed project", len(pkgs)) + } + if len(deg) != 2 || deg[0].Code != "partial_ssa" || !slices.Equal(deg[0].Packages, []string{"example.com/typeerr"}) || deg[1].Code != "no_ssa" { + t.Fatalf("degraded = %+v, want partial_ssa [example.com/typeerr] then no_ssa", deg) + } +} + +// TestRun_CleanFixtureNotDegraded: SSA and VTA over the fixture degrade +// nothing. +func TestRun_CleanFixtureNotDegraded(t *testing.T) { + if res := Run(loadInput(t, "dataflow")); len(res.Degraded) != 0 { + t.Fatalf("degraded = %+v, want none", res.Degraded) + } +} + +// TestRun_VTAPanicDegrades: a VTA panic is recovered as no_vta. +func TestRun_VTAPanicDegrades(t *testing.T) { + in := loadInput(t, "dataflow") + defer func(f func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph) { vtaCallGraph = f }(vtaCallGraph) + vtaCallGraph = func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph { panic("vta boom") } + res := Run(in) + if len(res.Degraded) != 1 || res.Degraded[0].Code != "no_vta" { + t.Fatalf("degraded = %+v, want one no_vta", res.Degraded) + } +} + +// TestProjectFuncs_CoversClosuresAndMethods: the VTA function set holds the +// project's functions, their closures and methods, and nothing declared +// outside the project. +func TestProjectFuncs_CoversClosuresAndMethods(t *testing.T) { + in := loadInput(t, "dataflow") + prog, pkgs, _ := buildSSA(in) + funcs, _ := projectFuncs(prog, pkgs) + project := map[string]bool{} + for _, p := range pkgs { + project[p.Pkg.Path()] = true + } + anon, methods := 0, 0 + for fn := range funcs { + if !project[homePath(fn)] { + t.Errorf("%s (package %q) is outside the project", fn, homePath(fn)) + } + if fn.Parent() != nil { + anon++ + } + if fn.Signature.Recv() != nil { + methods++ + } + } + if anon == 0 || methods == 0 { + t.Errorf("funcs has %d closures and %d methods; want both > 0", anon, methods) + } + if !funcs[prog.ImportedPackage("example.com/dataflow/flow").Func("IfElse")] { + t.Error("flow.IfElse is not in the VTA function set") + } +} + +// panicOn returns a callHook that panics on a call to the function named name. +func panicOn(name string) func(*types.Func) { + return func(f *types.Func) { + if f.Name() == name { + panic("boom in " + name) + } + } +} + +// withTimeout runs f, failing the test if it does not return within d (a +// deadlocked SSA program blocks forever; recover cannot catch that). +func withTimeout(t *testing.T, d time.Duration, f func()) { + t.Helper() + done := make(chan struct{}) + go func() { defer close(done); f() }() + select { + case <-done: + case <-time.After(d): + t.Fatalf("did not finish within %s: the SSA program deadlocked", d) + } +} + +// TestSSA_MidBuildPanicDoesNotDeadlock: testdata/ssapanic packages a and b +// share the instance g.Map[int]. Building a creates it under a's builder, +// then panics on the call to Boom, so a's builder never marks that instance +// done. Package b, built next in the same program, would wait on it forever. +// buildSSA must instead drop the program and rebuild without a: the run +// finishes, partial_ssa names a alone, and b keeps its SSA body. +func TestSSA_MidBuildPanicDoesNotDeadlock(t *testing.T) { + in := loadInput(t, "ssapanic") + callHook = panicOn("Boom") + defer func() { callHook = nil }() + + var prog *ssa.Program + var pkgs []*ssa.Package + var deg []Degraded + withTimeout(t, 30*time.Second, func() { prog, pkgs, deg = buildSSA(in) }) + + if len(deg) != 1 || deg[0].Code != "partial_ssa" || !slices.Equal(deg[0].Packages, []string{"example.com/ssapanic/a"}) { + t.Fatalf("degraded = %+v, want one partial_ssa naming example.com/ssapanic/a", deg) + } + var built []string + for _, p := range pkgs { + built = append(built, p.Pkg.Path()) + } + if want := []string{"example.com/ssapanic/b", "example.com/ssapanic/g", "example.com/ssapanic/w"}; !slices.Equal(built, want) { + t.Fatalf("built = %v, want %v", built, want) + } + useB := prog.ImportedPackage("example.com/ssapanic/b").Func("UseB") + if useB == nil || len(useB.Blocks) == 0 { + t.Fatal("b.UseB has no SSA body after a's build panicked") + } + if a := prog.ImportedPackage("example.com/ssapanic/a"); a != nil { + t.Errorf("the rebuilt program still holds package a (%d members)", len(a.Members)) + } + if res := Run(in); len(res.Degraded) != 1 || res.Degraded[0].Code != "partial_ssa" { + t.Errorf("Run degraded = %+v, want only partial_ssa (VTA runs on the rebuilt program)", res.Degraded) + } +} + +// TestRun_FuncSetPanicDegradesToNoVTA: building the VTA function set builds +// method wrappers on demand ((*w.V).Never, which nothing calls, so the package +// build never made it). A panic there is recovered as no_vta. +func TestRun_FuncSetPanicDegradesToNoVTA(t *testing.T) { + in := loadInput(t, "ssapanic") + callHook = panicOn("Never") + defer func() { callHook = nil }() + + var res Result + withTimeout(t, 30*time.Second, func() { res = Run(in) }) + if len(res.Degraded) != 1 || res.Degraded[0].Code != "no_vta" { + t.Fatalf("degraded = %+v, want one no_vta", res.Degraded) + } +} diff --git a/internal/sdg/summary.go b/internal/sdg/summary.go new file mode 100644 index 0000000..61d93ea --- /dev/null +++ b/internal/sdg/summary.go @@ -0,0 +1,630 @@ +package sdg + +import ( + "math/bits" + "slices" + "strings" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// sdgBuilder computes the flow summaries (spec §4.9) and the L4 overlay of +// a run over one ports. Like ports, it is not safe for concurrent use. +type sdgBuilder struct { + in Input + d *dispatch + p *ports + flows map[Target]*flowSet + info map[Target]*formalInfo + sites map[dataflow.L3Key][]siteFlow + graph map[dataflow.L3Key]*unitGraph + wired map[dataflow.L3Key][]PortEdge + // summ is each unit's summary edges, produced as soon as its SCC is + // solved (releaseSCC); fill moves them into the Result. + summ map[dataflow.L3Key][]SummaryEdge + // modelled is the Models keys some targeted stand-in has. + modelled map[string]bool + done bool +} + +// onCacheMiss, when set, is called each time the builder builds unit k's +// entry of a per-unit cache ("graph", "sites" or "wired"), or recomputes +// its summary edges because releaseSCC's ("summ") are gone. A test seam: +// TestSDG_ReleasedCachesAreNotRebuilt checks that, with the memory levers +// on, no entry is built twice — a cache released before its last reader +// (releaseSCC, releaseUnit) is rebuilt silently, costing time, not facts. +var onCacheMiss func(cache string, k dataflow.L3Key) + +func newSDGBuilder(in Input, d *dispatch) *sdgBuilder { + return &sdgBuilder{ + in: in, + d: d, + p: newPorts(in, d), + flows: map[Target]*flowSet{}, + info: map[Target]*formalInfo{}, + sites: map[dataflow.L3Key][]siteFlow{}, + graph: map[dataflow.L3Key]*unitGraph{}, + wired: map[dataflow.L3Key][]PortEdge{}, + summ: map[dataflow.L3Key][]SummaryEdge{}, + modelled: map[string]bool{}, + } +} + +// flowSet is a target's flows: bit (i, j) is set iff formal_in i may flow +// to formal_out j. +type flowSet struct { + nIn, nOut, words int + bits []uint64 +} + +func newFlowSet(nIn, nOut int) *flowSet { + w := (nOut + 63) / 64 + return &flowSet{nIn: nIn, nOut: nOut, words: w, bits: make([]uint64, nIn*w)} +} + +func (f *flowSet) set(i, j int) { f.bits[i*f.words+j/64] |= 1 << (j % 64) } + +func (f *flowSet) has(i, j int) bool { + return f != nil && i < f.nIn && j < f.nOut && f.bits[i*f.words+j/64]&(1<<(j%64)) != 0 +} + +func (f *flowSet) count() int { + n := 0 + if f != nil { + for _, w := range f.bits { + n += bits.OnesCount64(w) + } + } + return n +} + +// equal reports whether f and g hold the same flows (nil holds none). +func (f *flowSet) equal(g *flowSet) bool { + if f.count() == 0 || g.count() == 0 { + return f.count() == g.count() + } + return f.nIn == g.nIn && f.nOut == g.nOut && slices.Equal(f.bits, g.bits) +} + +func (f *flowSet) pairs() [][2]int { + var out [][2]int + if f == nil { + return nil + } + for i := range f.nIn { + for j := range f.nOut { + if f.has(i, j) { + out = append(out, [2]int{i, j}) + } + } + } + return out +} + +// formalInfo is a target's formals by index: their keys' positions and +// their `of`s. +type formalInfo struct { + inPos, outPos map[string]int + inOf, outOf []string +} + +func (b *sdgBuilder) formalInfo(t Target) *formalInfo { + if fi, ok := b.info[t]; ok { + return fi + } + in, out := b.p.formals(t) + fi := &formalInfo{inPos: make(map[string]int, len(in)), outPos: make(map[string]int, len(out))} + for i, v := range in { + fi.inPos[v.Key] = i + fi.inOf = append(fi.inOf, v.Of) + } + for j, v := range out { + fi.outPos[v.Key] = j + fi.outOf = append(fi.outOf, v.Of) + } + b.info[t] = fi + return fi +} + +// siteFlow is one resolved call site of a unit: its actuals and, per +// target, which actual binds which of that target's formals (by index). +type siteFlow struct { + in, out []Vertex + binds []siteBind +} + +type siteBind struct { + t Target + ins, outs []bindPos +} + +// bindPos binds the actual at index actual (into siteFlow.in or .out) to a +// target's formal at index formal. +type bindPos struct{ actual, formal int } + +// sitesOf returns the resolved call sites of unit k that have vertices (the +// sites vertices() gives actuals), in u.Calls order. Capture sites are not +// calls (spec §4.6) and are not among them. +func (b *sdgBuilder) sitesOf(k dataflow.L3Key) []siteFlow { + if s, ok := b.sites[k]; ok { + return s + } + if onCacheMiss != nil { + onCacheMiss("sites", k) + } + var out []siteFlow + for _, c := range b.p.units[k].Calls { + s := b.p.callSite(k, c.Key, b.p.siteTargets(k, c.Key)) + if len(s.In)+len(s.Out) == 0 { + continue + } + sf := siteFlow{in: s.In, out: s.Out} + inIdx := make(map[string]int, len(s.In)) + for i, v := range s.In { + inIdx[v.Key] = i + } + outIdx := make(map[string]int, len(s.Out)) + for i, v := range s.Out { + outIdx[v.Key] = i + } + byT := map[Target]*siteBind{} + var order []Target + bind := func(t Target) *siteBind { + sb, ok := byT[t] + if !ok { + sb = &siteBind{t: t} + byT[t] = sb + order = append(order, t) + } + return sb + } + for _, pb := range s.BindIn { + if j, ok := b.formalInfo(pb.Target).inPos[pb.Formal]; ok { + sb := bind(pb.Target) + sb.ins = append(sb.ins, bindPos{inIdx[pb.Actual], j}) + } + } + for _, pb := range s.BindOut { + if j, ok := b.formalInfo(pb.Target).outPos[pb.Formal]; ok { + sb := bind(pb.Target) + sb.outs = append(sb.outs, bindPos{outIdx[pb.Actual], j}) + } + } + slices.SortFunc(order, compareTarget) + for _, t := range order { + sf.binds = append(sf.binds, *byT[t]) + } + out = append(out, sf) + } + b.sites[k] = out + return out +} + +// matrix is the site's summary under the current flows: bit i*len(out)+j is +// set iff some target has a flow from the formal actual_in i binds to the +// formal actual_out j binds. +func (b *sdgBuilder) matrix(s siteFlow) []uint64 { + n := len(s.in) * len(s.out) + m := make([]uint64, (n+63)/64) + for _, sb := range s.binds { + f := b.flows[sb.t] + if f.count() == 0 { + continue + } + for _, ip := range sb.ins { + for _, op := range sb.outs { + if f.has(ip.formal, op.formal) { + bit := ip.actual*len(s.out) + op.actual + m[bit/64] |= 1 << (bit % 64) + } + } + } + } + return m +} + +// summaryEdges is the summary edges of every call site of unit k under the +// current flows, sorted by (Src, Dst), except those from a global actual_in +// to a global actual_out (`of` `:` at both ends; ruling [user, +// 2026-10-26], spec §4.9). Under the coarse host wiring (A15) a summary edge +// only shortcuts host → actual_in → … → actual_out → host, and sources and +// sinks address formal_in, so a global → global summary adds no +// reachability for an HRB-unaware walk; it is omitted to bound the output +// (on nats-server 97.7% of all summary edges were global → global). They +// return with the deferred HRB two-phase walk (spec §11). The flows keep +// every pair: only the emitted list is filtered. +func (b *sdgBuilder) summaryEdges(k dataflow.L3Key) []SummaryEdge { + var out []SummaryEdge + for _, s := range b.sitesOf(k) { + m := b.matrix(s) + for i, ai := range s.in { + inGlobal := strings.HasPrefix(ai.Of, ofGlobal) + for j, ao := range s.out { + if inGlobal && strings.HasPrefix(ao.Of, ofGlobal) { + continue + } + if bit := i*len(s.out) + j; m[bit/64]&(1<<(bit%64)) != 0 { + out = append(out, SummaryEdge{Src: ai.Key, Dst: ao.Key}) + } + } + } + } + slices.SortFunc(out, compareSummaryEdge) + return slices.Compact(out) +} + +func compareSummaryEdge(a, b SummaryEdge) int { + if c := strings.Compare(a.Src, b.Src); c != 0 { + return c + } + return strings.Compare(a.Dst, b.Dst) +} + +// unitGraph is an analysed unit's flow graph without its summary edges: L3 +// ddg ∪ cdg ∪ the port wiring, over node indices. +type unitGraph struct { + idx map[string]int + succ [][]int32 + ins, outs []int // node of each formal_in / formal_out, by formal index +} + +// summary is unit k's summary edges: the ones releaseSCC produced, else +// summaryEdges under the current flows. +func (b *sdgBuilder) summary(k dataflow.L3Key) []SummaryEdge { + if s, ok := b.summ[k]; ok { + return s + } + if onCacheMiss != nil { + onCacheMiss("summ", k) + } + return b.summaryEdges(k) +} + +func (b *sdgBuilder) portEdges(k dataflow.L3Key) []PortEdge { + if e, ok := b.wired[k]; ok { + return e + } + if onCacheMiss != nil { + onCacheMiss("wired", k) + } + e := b.p.wire(k) + b.wired[k] = e + return e +} + +func (b *sdgBuilder) graphOf(k dataflow.L3Key) *unitGraph { + if g, ok := b.graph[k]; ok { + return g + } + if onCacheMiss != nil { + onCacheMiss("graph", k) + } + g := &unitGraph{idx: map[string]int{}} + node := func(key string) int { + i, ok := g.idx[key] + if !ok { + i = len(g.succ) + g.idx[key] = i + g.succ = append(g.succ, nil) + } + return i + } + edge := func(src, dst string) { + s := node(src) + g.succ[s] = append(g.succ[s], int32(node(dst))) + } + in, out := b.p.formals(Target{Callable: k}) + for _, v := range in { + g.ins = append(g.ins, node(v.Key)) + } + for _, v := range out { + g.outs = append(g.outs, node(v.Key)) + } + r := b.in.L3[k] + for _, e := range r.Ddg { + edge(e.Src, e.Dst) + } + for _, e := range r.Cdg { + edge(e.Src, e.Dst) + } + for _, e := range b.portEdges(k) { + edge(e.Src, e.Dst) + } + for _, s := range b.sitesOf(k) { + for _, v := range slices.Concat(s.in, s.out) { + node(v.Key) + } + } + for i, s := range g.succ { + slices.Sort(s) + g.succ[i] = slices.Compact(s) + } + b.graph[k] = g + return g +} + +// opaque reports whether t's flows are not computed from a body: a stand-in, +// an interface method, a bodiless declaration, or a unit without the L3 +// facts the port wiring needs (l3_skipped, A16). +func (b *sdgBuilder) opaque(t Target) bool { + if t.External != "" { + return true + } + u, ok := b.p.units[t.Callable] + return !ok || !analysed(b.in, u) +} + +// opaqueFlows is the flows of an opaque target: every formal_in → every +// formal_out (spec §6, §4.8 D7, A16), unless it is a stand-in with a valid +// Models entry (modelFlows). An interface method's flows are all-to-all +// whatever its VTA targets are, so the D7 union over them adds nothing. +func (b *sdgBuilder) opaqueFlows(t Target) *flowSet { + fi := b.formalInfo(t) + if f, ok := b.modelFlows(t, fi); ok { + return f + } + f := newFlowSet(len(fi.inOf), len(fi.outOf)) + for i := range f.nIn { + for j := range f.nOut { + f.set(i, j) + } + } + return f +} + +// bodyFlows is the flows of analysed unit k under its callees' current +// flows: forward reachability from each @formal_in over the unit graph plus +// the summary edges at its call sites; (i, j) iff @formal_out j is reached +// from @formal_in i. Reachability from all formal_ins is solved at once: the +// graph's SCCs reachable from a formal_in are condensed (one source bitset +// per SCC) and propagated once, in topological order. +func (b *sdgBuilder) bodyFlows(k dataflow.L3Key) *flowSet { + g := b.graphOf(k) + f := newFlowSet(len(g.ins), len(g.outs)) + if len(g.ins) == 0 || len(g.outs) == 0 { + return f + } + extra := map[int][]int32{} + for _, s := range b.sitesOf(k) { + m := b.matrix(s) + for i, ai := range s.in { + src := g.idx[ai.Key] + for j, ao := range s.out { + if bit := i*len(s.out) + j; m[bit/64]&(1<<(bit%64)) != 0 { + extra[src] = append(extra[src], int32(g.idx[ao.Key])) + } + } + } + } + succ := func(n int) ([]int32, []int32) { return g.succ[n], extra[n] } + comp, order := condense(len(g.succ), g.ins, succ) + w := (len(g.ins) + 63) / 64 + reach := make([]uint64, len(order)*w) + for i, n := range g.ins { + c := comp[n] + reach[int(c)*w+i/64] |= 1 << (i % 64) + } + // order is sinks first: walk it backwards, so every SCC is complete + // before it is pushed to its successors. + for ci := len(order) - 1; ci >= 0; ci-- { + src := reach[ci*w : (ci+1)*w] + for _, n := range order[ci] { + a, x := succ(n) + for _, list := range [2][]int32{a, x} { + for _, m := range list { + cm := int(comp[m]) + if cm == ci { + continue + } + dst := reach[cm*w : (cm+1)*w] + for z := range dst { + dst[z] |= src[z] + } + } + } + } + } + for j, n := range g.outs { + c := comp[n] + if c < 0 { + continue + } + row := reach[int(c)*w : (int(c)+1)*w] + for i := range g.ins { + if row[i/64]&(1<<(i%64)) != 0 { + f.set(i, j) + } + } + } + return f +} + +// condense returns the SCCs of the graph of n nodes and succ reachable from +// roots (iterative Tarjan): comp[v] is v's SCC index (-1 when unreached), +// and order the SCCs' members, sinks first (an SCC comes after every SCC it +// reaches). +func condense(n int, roots []int, succ func(int) ([]int32, []int32)) ([]int32, [][]int) { + comp := make([]int32, n) + index := make([]int32, n) + low := make([]int32, n) + for i := range comp { + comp[i], index[i] = -1, -1 + } + onStack := make([]bool, n) + var stack []int + var order [][]int + type frame struct{ v, next int } + var calls []frame + next := int32(0) + edge := func(v, i int) (int, bool) { + a, x := succ(v) + if i < len(a) { + return int(a[i]), true + } + if i -= len(a); i < len(x) { + return int(x[i]), true + } + return 0, false + } + visit := func(v int) { + index[v], low[v] = next, next + next++ + stack = append(stack, v) + onStack[v] = true + calls = append(calls, frame{v: v}) + } + for _, r := range roots { + if index[r] >= 0 { + continue + } + visit(r) + for len(calls) > 0 { + top := len(calls) - 1 + fr := &calls[top] + v := fr.v + if w, ok := edge(v, fr.next); ok { + fr.next++ + switch { + case index[w] < 0: + visit(w) + case onStack[w]: + low[v] = min(low[v], index[w]) + } + continue + } + calls = calls[:top] + if top > 0 { + p := calls[top-1].v + low[p] = min(low[p], low[v]) + } + if low[v] != index[v] { + continue + } + i := len(stack) - 1 + for stack[i] != v { + i-- + } + members := slices.Clone(stack[i:]) + stack = stack[:i] + c := int32(len(order)) + for _, m := range members { + onStack[m] = false + comp[m] = c + } + order = append(order, members) + } + } + return comp, order +} + +// solve recomputes t's flows and reports whether they changed. +func (b *sdgBuilder) solve(t Target) bool { + var f *flowSet + if b.opaque(t) { + if _, ok := b.flows[t]; ok { + return false // constant + } + f = b.opaqueFlows(t) + } else { + f = b.bodyFlows(t.Callable) + } + old, had := b.flows[t] + b.flows[t] = f + return !had && f.count() > 0 || had && !old.equal(f) +} + +// summarize computes the flows of every unit and call target (spec §4.9): +// (i, j) pairs, formal_in index i → formal_out index j, sorted. Callees come +// first over the call graph's SCCs (callGraph, sccs); a recursive SCC +// iterates to its least fixpoint. +func (b *sdgBuilder) summarize() map[Target][][2]int { + b.computeFlows() + out := make(map[Target][][2]int, len(b.flows)) + for t, f := range b.flows { + out[t] = f.pairs() + } + return out +} + +func (b *sdgBuilder) computeFlows() { + if b.done { + return + } + b.done = true + nodes, callees := callGraph(b.in, b.d) + comps := sccs(nodes, func(t Target) []Target { return callees[t] }) + if !memLevers { + solveSCCs(comps, callees, b.solve) + return + } + for _, comp := range comps { + solveSCCs([][]Target{comp}, callees, b.solve) + b.releaseSCC(comp) + } +} + +// releaseSCC produces the summary edges of every unit of the solved SCC comp +// and drops its unit graphs and call sites (a memory lever, spec §4.13). A +// unit's call-site targets are all its callees (callGraph), so once its SCC +// is solved every flow its summaries read is final: no SCC solved later +// changes them, and no other unit reads its graph or sites. +func (b *sdgBuilder) releaseSCC(comp []Target) { + for _, t := range comp { + if t.External != "" { + continue + } + k := t.Callable + if _, unit := b.p.units[k]; !unit { + continue + } + b.summ[k] = b.summaryEdges(k) + delete(b.graph, k) + delete(b.sites, k) + } +} + +// solveSCCs solves comps in order (callees first, as sccs returns them). A +// non-recursive singleton is solved once. A recursive SCC (several members, +// or one calling itself) is iterated to a fixpoint with a worklist: every +// member once, then each member whose callee in the SCC changed. solve +// recomputes a target's flows from its callees' current ones and reports a +// change; it must be monotone, so over finite flow sets this terminates. +func solveSCCs(comps [][]Target, callees map[Target][]Target, solve func(Target) bool) { + for _, comp := range comps { + if len(comp) == 1 && !slices.Contains(callees[comp[0]], comp[0]) { + solve(comp[0]) + continue + } + in := make(map[Target]bool, len(comp)) + for _, m := range comp { + in[m] = true + } + callers := map[Target][]Target{} + for _, m := range comp { + for _, c := range callees[m] { + if in[c] { + callers[c] = append(callers[c], m) + } + } + } + work := slices.Clone(comp) + queued := make(map[Target]bool, len(comp)) + for _, m := range comp { + queued[m] = true + } + for len(work) > 0 { + m := work[0] + work = work[1:] + queued[m] = false + if !solve(m) { + continue + } + for _, c := range callers[m] { + if !queued[c] { + queued[c] = true + work = append(work, c) + } + } + } + } +} diff --git a/internal/sdg/summary_test.go b/internal/sdg/summary_test.go new file mode 100644 index 0000000..2f3781b --- /dev/null +++ b/internal/sdg/summary_test.go @@ -0,0 +1,541 @@ +package sdg + +import ( + "cmp" + "maps" + "slices" + "strings" + "testing" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +func se(src, dst string) SummaryEdge { return SummaryEdge{Src: src, Dst: dst} } + +func checkSummary(t *testing.T, what string, got, want []SummaryEdge) { + t.Helper() + if !slices.Equal(got, want) { + t.Errorf("%s:\n got %+v\n want %+v", what, got, want) + } +} + +// siteSummaryOf returns the summary edges of res's unit k whose source is an +// actual of call (or capture) key call. +func siteSummaryOf(res Result, k dataflow.L3Key, call string) []SummaryEdge { + var out []SummaryEdge + for _, e := range res.Callables[k].Summary { + if strings.HasPrefix(e.Src, call+"/") { + out = append(out, e) + } + } + return out +} + +func checkFlows(t *testing.T, flows map[Target][][2]int, tg Target, want [][2]int) { + t.Helper() + got, ok := flows[tg] + if !ok { + t.Errorf("no flows for %v", tg) + return + } + if !slices.Equal(got, want) { + t.Errorf("flows of %v = %v, want %v", tg, got, want) + } +} + +// TestSummary_TransitiveFlow: a(x) → b(x) → c(x) returns x, so c and b each +// have the flow x → , and a's call of b has the summary edge +// actual_in:0 → actual_out. +func TestSummary_TransitiveFlow(t *testing.T) { + _, res, _ := runDispatch(t) + b := newSDGBuilder(runPortsInput(t)) + flows := b.summarize() + for _, name := range []string{"a", "b", "c"} { + checkFlows(t, flows, sdgTarget(name), [][2]int{{0, 0}}) + } + checkSummary(t, "summary of a", res.Callables[sdgKey("a")].Summary, + []SummaryEdge{se("308:9/actual_in:0", "308:9/actual_out")}) + checkSummary(t, "summary of b", res.Callables[sdgKey("b")].Summary, + []SummaryEdge{se("312:9/actual_in:0", "312:9/actual_out")}) + if s := res.Callables[sdgKey("c")].Summary; len(s) != 0 { + t.Errorf("c calls nothing; summary = %+v", s) + } +} + +// TestSummary_NoFlowNoEdge: ignore(x, y) returns y, so its only flow is y → +// , and its call site has no edge from actual_in:0. +func TestSummary_NoFlowNoEdge(t *testing.T) { + _, res, _ := runDispatch(t) + flows := newSDGBuilder(runPortsInput(t)).summarize() + checkFlows(t, flows, sdgTarget("ignore"), [][2]int{{1, 0}}) + checkSummary(t, "summary of callIgnore", res.Callables[sdgKey("callIgnore")].Summary, + []SummaryEdge{se("326:9/actual_in:1", "326:9/actual_out")}) +} + +// TestSummary_RecursiveFixpoint: dropX and dropY form one recursive SCC. +// Its fixpoint is the least one: dropX's x flows nowhere, so dropY's call +// of dropX has no edge from actual_in:0. +func TestSummary_RecursiveFixpoint(t *testing.T) { + _, res, _ := runDispatch(t) + flows := newSDGBuilder(runPortsInput(t)).summarize() + checkFlows(t, flows, sdgTarget("dropX"), [][2]int{{1, 0}}) + checkFlows(t, flows, sdgTarget("dropY"), [][2]int{{0, 0}}) + checkSummary(t, "summary of dropY", res.Callables[sdgKey("dropY")].Summary, + []SummaryEdge{se("339:9/actual_in:1", "339:9/actual_out")}) + checkSummary(t, "summary of dropX", res.Callables[sdgKey("dropX")].Summary, + []SummaryEdge{se("335:9/actual_in:0", "335:9/actual_out")}) +} + +// TestSummary_FixpointIterates: the SCC driver iterates a recursive SCC to +// its least fixpoint. In this synthetic graph p → q → p, p's flow exists only +// once q's does and q's only through r (outside the SCC) and p, so a single +// pass in member order (p first) misses p's flow. +func TestSummary_FixpointIterates(t *testing.T) { + p, q, r := Target{External: "p"}, Target{External: "q"}, Target{External: "r"} + callees := map[Target][]Target{p: {q}, q: {p, r}} + comps := sccs([]Target{p, q, r}, func(t Target) []Target { return callees[t] }) + flows := map[Target][][2]int{} + solves := 0 + solve := func(t Target) bool { + solves++ + var f [][2]int + switch t { + case r: + f = [][2]int{{0, 0}} + case q: // q passes r's flow on + f = flows[r] + case p: // p passes q's flow on + f = flows[q] + } + old, had := flows[t] + flows[t] = f + return !had && len(f) > 0 || had && !slices.Equal(old, f) + } + solveSCCs(comps, callees, solve) + for _, tg := range []Target{p, q, r} { + checkFlows(t, flows, tg, [][2]int{{0, 0}}) + } + if solves < 4 { + t.Errorf("solved %d times: the recursive SCC was not iterated", solves) + } +} + +// TestSummary_NoBodyTargetsAllToAll: an interface method, a stand-in and an +// l3_skipped unit get every formal_in → every formal_out; a Models entry +// overrides a stand-in, and an empty one cuts every flow. +func TestSummary_NoBodyTargetsAllToAll(t *testing.T) { + in, d, _ := runPorts(t) + flows := newSDGBuilder(in, d).summarize() + all := func(nIn, nOut int) [][2]int { + var out [][2]int + for i := range nIn { + for j := range nOut { + out = append(out, [2]int{i, j}) + } + } + return out + } + // Store.Get: formal_in _recv, k; formal_out , _recv. + checkFlows(t, flows, sdgTarget("Store.Get"), all(2, 2)) + write := Target{External: "io.Writer.Write"} + if _, ok := flows[write]; !ok { + t.Fatalf("no flows for %v; targets of Buffered: %v", write, d.targets(sdgKey("Buffered"), "94:2")) + } + checkFlows(t, flows, write, all(2, 4)) + + in.Models = Models{"io/Writer.Write": {{"p", ":0"}}} + flows = newSDGBuilder(in, d).summarize() + checkFlows(t, flows, write, [][2]int{{1, 0}}) + in.Models = Models{"io/Writer.Write": {}} + flows = newSDGBuilder(in, d).summarize() + checkFlows(t, flows, write, nil) + + skip := map[dataflow.L3Key]bool{sdgKey("ignore"): true} + l3 := make(map[dataflow.L3Key]dataflow.Result, len(in.L3)) + for k, r := range in.L3 { + if !skip[k] { + l3[k] = r + } + } + in.L3, in.Models = l3, nil + flows = newSDGBuilder(in, d).summarize() + checkFlows(t, flows, sdgTarget("ignore"), all(2, 1)) +} + +// TestParams_PerTargetWithVar: UseStore's s.Get(k) has one param_in set per +// target (the interface method and both implementations), each var that +// target's formal name, and one param_out set per target. +func TestParams_PerTargetWithVar(t *testing.T) { + _, res, _ := runDispatch(t) + use := sdgTarget("UseStore") + iface, mem, sql := sdgTarget("Store.Get"), sdgTarget("memStore.Get"), sdgTarget("sqlStore.Get") + ref := func(o Target, k string) Ref { return Ref{Owner: o, Key: k} } + var gotIn, gotOut []ParamEdge + for _, e := range res.ParamIn { + if e.Src.Owner == use { + gotIn = append(gotIn, e) + } + } + for _, e := range res.ParamOut { + if e.Dst.Owner == use { + gotOut = append(gotOut, e) + } + } + wantIn := []ParamEdge{ + {ref(use, "34:9/actual_in:0"), ref(iface, "@formal_in:0"), "_recv"}, + {ref(use, "34:9/actual_in:0"), ref(mem, "@formal_in:0"), "s"}, + {ref(use, "34:9/actual_in:0"), ref(sql, "@formal_in:0"), "s"}, + {ref(use, "34:9/actual_in:1"), ref(iface, "@formal_in:1"), "k"}, + {ref(use, "34:9/actual_in:1"), ref(mem, "@formal_in:1"), "k"}, + {ref(use, "34:9/actual_in:1"), ref(sql, "@formal_in:1"), "k"}, + } + slices.SortFunc(wantIn, compareParamEdge) + if !slices.Equal(gotIn, wantIn) { + t.Errorf("param_in of UseStore:\n got %+v\n want %+v", gotIn, wantIn) + } + wantOut := []ParamEdge{ + {ref(iface, "@formal_out:0"), ref(use, "34:9/actual_out:0"), ""}, + {ref(iface, "@formal_out:1"), ref(use, "34:9/actual_out:1"), "_recv"}, + {ref(mem, "@formal_out:0"), ref(use, "34:9/actual_out:0"), ""}, + {ref(mem, "@formal_out:1"), ref(use, "34:9/actual_out:1"), "s"}, + {ref(sql, "@formal_out:0"), ref(use, "34:9/actual_out:0"), ""}, + {ref(sql, "@formal_out:1"), ref(use, "34:9/actual_out:1"), "s"}, + } + slices.SortFunc(wantOut, compareParamEdge) + if !slices.Equal(gotOut, wantOut) { + t.Errorf("param_out of UseStore:\n got %+v\n want %+v", gotOut, wantOut) + } + // The interface method passes everything through (D7), so the site's + // summary is all-to-all. + checkSummary(t, "summary of UseStore", res.Callables[sdgKey("UseStore")].Summary, []SummaryEdge{ + se("34:9/actual_in:0", "34:9/actual_out:0"), se("34:9/actual_in:0", "34:9/actual_out:1"), + se("34:9/actual_in:1", "34:9/actual_out:0"), se("34:9/actual_in:1", "34:9/actual_out:1"), + }) +} + +// TestParams_CaptureSite: a stored literal's capture site gets param_in and +// param_out to the closure's capture formals, and no summary edge. +func TestParams_CaptureSite(t *testing.T) { + _, res, _ := runDispatch(t) + host, lit := sdgTarget("CaptureSite"), sdgTarget("CaptureSite.closure_1") + want := []ParamEdge{ + {Ref{host, "215:7/actual_in:0"}, Ref{lit, "@formal_in:0"}, ":x"}, + } + var got []ParamEdge + for _, e := range res.ParamIn { + if e.Src.Owner == host { + got = append(got, e) + } + } + if !slices.Equal(got, want) { + t.Errorf("param_in of CaptureSite:\n got %+v\n want %+v", got, want) + } + wantOut := []ParamEdge{ + {Ref{lit, "@formal_out:1"}, Ref{host, "215:7/actual_out"}, ":x"}, + } + got = nil + for _, e := range res.ParamOut { + if e.Dst.Owner == host { + got = append(got, e) + } + } + if !slices.Equal(got, wantOut) { + t.Errorf("param_out of CaptureSite:\n got %+v\n want %+v", got, wantOut) + } + if s := siteSummaryOf(res, sdgKey("CaptureSite"), "215:7"); len(s) != 0 { + t.Errorf("capture site has summary edges %+v", s) + } +} + +// vertexSet is the vertex keys of every owner in res. +func vertexSet(res Result) map[Target]map[string]bool { + out := map[Target]map[string]bool{} + add := func(t Target, vs []Vertex) { + m := map[string]bool{} + for _, v := range vs { + m[v.Key] = true + } + out[t] = m + } + for k, c := range res.Callables { + add(Target{Callable: k}, c.Vertices) + } + for sig, vs := range res.Externals { + add(Target{External: sig}, vs) + } + return out +} + +// TestParams_NoDanglingEndpoints: every param_in / param_out endpoint and +// every summary edge endpoint names a vertex of its owner; the lists are +// sorted and unique. +func TestParams_NoDanglingEndpoints(t *testing.T) { + check := func(t *testing.T, res Result) { + t.Helper() + if len(res.ParamIn) == 0 || len(res.ParamOut) == 0 { + t.Fatalf("param_in %d, param_out %d: want both", len(res.ParamIn), len(res.ParamOut)) + } + vs := vertexSet(res) + for _, list := range [][]ParamEdge{res.ParamIn, res.ParamOut} { + for _, e := range list { + for _, r := range []Ref{e.Src, e.Dst} { + if !vs[r.Owner][r.Key] { + t.Errorf("edge %+v: %v has no vertex %s", e, r.Owner, r.Key) + } + } + if e.Var == "" { + t.Errorf("edge %+v has no var", e) + } + } + if !slices.IsSortedFunc(list, compareParamEdge) { + t.Error("param edges are not sorted") + } + if len(slices.CompactFunc(slices.Clone(list), func(a, b ParamEdge) bool { return a == b })) != len(list) { + t.Error("param edges repeat") + } + } + for k, c := range res.Callables { + own := vs[Target{Callable: k}] + for _, e := range c.Summary { + if !own[e.Src] || !own[e.Dst] { + t.Errorf("%v: summary edge %+v dangles", k, e) + } + } + if !slices.IsSortedFunc(c.Summary, compareSummaryEdge) { + t.Errorf("%v: summary not sorted", k) + } + } + } + t.Run("vta", func(t *testing.T) { + _, res, _ := runDispatch(t) + check(t, res) + }) + t.Run("no_vta", func(t *testing.T) { + in := loadInput(t, "dataflow") + defer func(f func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph) { vtaCallGraph = f }(vtaCallGraph) + vtaCallGraph = func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph { panic("vta boom") } + res := Run(in) + if !hasCode(res.Degraded, CodeNoVTA) { + t.Fatalf("degraded = %+v, want no_vta", res.Degraded) + } + check(t, res) + }) +} + +// TestSDG_RunFillsCallablesAndExternals: Run gives every unit its vertices, +// port edges and summary, every stand-in target its formals, and an +// interface method its formals alone. +func TestSDG_RunFillsCallablesAndExternals(t *testing.T) { + in, res, _ := runDispatch(t) + for _, u := range in.Units { + if _, ok := res.Callables[u.Key]; !ok { + t.Errorf("no CallableSDG for unit %v", u.Key) + } + } + got := res.Callables[sdgKey("UseStore")] + if len(got.Vertices) == 0 || len(got.Ddg) == 0 { + t.Errorf("UseStore: %d vertices, %d port edges", len(got.Vertices), len(got.Ddg)) + } + iface := res.Callables[sdgKey("Store.Get")] + checkVertices(t, "interface method Store.Get", iface.Vertices, + []Vertex{fin("0", "_recv"), fin("1", "k"), fout(":0", ""), fout(":1", "_recv")}) + if len(iface.Ddg)+len(iface.Summary) != 0 { + t.Errorf("Store.Get has edges: %+v", iface) + } + vs, ok := res.Externals["io.Writer.Write"] + if !ok { + t.Fatalf("Externals lack io.Writer.Write: %v", keysOf(res.Externals)) + } + // p ([]byte) and the receiver are by-ref outs. + want := []Vertex{fin("0", "_recv"), fin("1", "p"), fout(":0", ":0"), fout(":1", ":1"), fout(":2", "_recv"), fout(":3", "p")} + slices.SortFunc(want, func(a, b Vertex) int { return cmp.Compare(a.Key, b.Key) }) + checkVertices(t, "stand-in io.Writer.Write", vs, want) +} + +func keysOf[V any](m map[string]V) []string { + var out []string + for k := range m { + out = append(out, k) + } + slices.Sort(out) + return out +} + +// TestSDG_Deterministic: two runs over two loads give equal L4 overlays. +func TestSDG_Deterministic(t *testing.T) { + _, a, _ := runDispatch(t) + _, b, _ := runDispatch(t) + if !slices.Equal(a.ParamIn, b.ParamIn) || !slices.Equal(a.ParamOut, b.ParamOut) { + t.Error("param edges differ across runs") + } + if len(a.Callables) != len(b.Callables) { + t.Errorf("callables: %d vs %d across runs", len(a.Callables), len(b.Callables)) + } + for k, c := range a.Callables { + d, ok := b.Callables[k] + if !ok { + t.Errorf("%v only in the first run", k) + continue + } + if !slices.Equal(c.Vertices, d.Vertices) || !slices.Equal(c.Ddg, d.Ddg) || !slices.Equal(c.Summary, d.Summary) { + t.Errorf("%v differs across runs", k) + } + } + if len(a.Externals) == 0 { + t.Fatal("no stand-ins: the Externals comparison would be vacuous") + } + if !maps.EqualFunc(a.Externals, b.Externals, slices.Equal[[]Vertex]) { + t.Error("stand-in vertices differ across runs") + } +} + +// runPortsInput is runPorts' input and dispatch. +func runPortsInput(t *testing.T) (Input, *dispatch) { + t.Helper() + in, d, _ := runPorts(t) + return in, d +} + +// TestSummary_ImplicitFlowViaCdg: implicit's x reaches no return over the +// ddg, only over the cdg (the returns depend on the if), so the flow x → +// needs the cdg in the graph (spec §4.9, Python precedent). +func TestSummary_ImplicitFlowViaCdg(t *testing.T) { + flows := newSDGBuilder(runPortsInput(t)).summarize() + checkFlows(t, flows, sdgTarget("implicit"), [][2]int{{0, 0}}) +} + +// TestSummary_CondensedReachMatchesBFS: bodyFlows (one pass over the +// condensed graph) equals a plain BFS from each formal_in over the same +// edges, on every analysed fixture unit, under the final callee flows. +func TestSummary_CondensedReachMatchesBFS(t *testing.T) { + b := newSDGBuilder(runPortsInput(t)) + b.computeFlows() + n := 0 + for _, u := range b.in.Units { + k := u.Key + if b.opaque(Target{Callable: k}) { + continue + } + g := b.graphOf(k) + succ := map[int][]int{} + for v, s := range g.succ { + for _, m := range s { + succ[v] = append(succ[v], int(m)) + } + } + for _, s := range b.sitesOf(k) { + m := b.matrix(s) + for i, ai := range s.in { + for j, ao := range s.out { + if bit := i*len(s.out) + j; m[bit/64]&(1<<(bit%64)) != 0 { + succ[g.idx[ai.Key]] = append(succ[g.idx[ai.Key]], g.idx[ao.Key]) + } + } + } + } + var want [][2]int + for i, src := range g.ins { + seen := map[int]bool{src: true} + work := []int{src} + for len(work) > 0 { + v := work[0] + work = work[1:] + for _, m := range succ[v] { + if !seen[m] { + seen[m] = true + work = append(work, m) + } + } + } + for j, o := range g.outs { + if seen[o] { + want = append(want, [2]int{i, j}) + } + } + } + slices.SortFunc(want, func(a, b [2]int) int { return cmp.Or(cmp.Compare(a[0], b[0]), cmp.Compare(a[1], b[1])) }) + if got := b.bodyFlows(k).pairs(); !slices.Equal(got, want) { + t.Errorf("%v: bodyFlows %v, BFS %v", k, got, want) + } + n++ + } + if n < 50 { + t.Errorf("checked %d units", n) + } +} + +// TestSummary_OmitsGlobalToGlobal (ruling [user, 2026-10-26], spec §4.9): a +// call site whose callee reads and writes globals gets no summary edge from +// a global actual_in to a global actual_out, but the callee's flows still +// hold those pairs, and every other summary edge stands. +func TestSummary_OmitsGlobalToGlobal(t *testing.T) { + in, res, _ := runDispatch(t) + b := newSDGBuilder(runPortsInput(t)) + flows := b.summarize() + + // appendConfig(v): formal_in v, Config; formal_out , Config. + // Its flows keep Config → Config. + checkFlows(t, flows, Target{Callable: globalsKey("appendConfig")}, [][2]int{{0, 0}, {0, 1}, {1, 0}, {1, 1}}) + checkSummary(t, "summary of useAppend", res.Callables[globalsKey("useAppend")].Summary, []SummaryEdge{ + se("100:9/actual_in:0", "100:9/actual_out:0"), se("100:9/actual_in:0", "100:9/actual_out:1"), + se("100:9/actual_in:1", "100:9/actual_out:0"), + }) + // Get reads and writes inst and once (formal_in inst, once; formal_out + // , inst, once): inst → inst and once → once stay in its flows. + get := Target{Callable: globalsKey("Get")} + fl := flows[get] + for _, p := range [][2]int{{0, 1}, {1, 2}} { + if !slices.Contains(fl, p) { + t.Errorf("flows of Get = %v lack the global → global pair %v", fl, p) + } + } + checkSummary(t, "summary of callsGet", res.Callables[globalsKey("callsGet")].Summary, []SummaryEdge{ + se("68:33/actual_in:0", "68:33/actual_out:0"), se("68:33/actual_in:1", "68:33/actual_out:0"), + }) + // useConfig and readConfig only read Config: their global → + // edges are unchanged. + checkSummary(t, "summary of useConfig", res.Callables[sdgKey("useConfig")].Summary, + []SummaryEdge{se("122:9/actual_in:0", "122:9/actual_out")}) + checkSummary(t, "summary of readConfig", res.Callables[sdgKey("readConfig")].Summary, + []SummaryEdge{se("116:10/actual_in:0", "116:10/actual_out")}) + + // No unit anywhere keeps a global → global summary edge. + for _, u := range in.Units { + c := res.Callables[u.Key] + of := map[string]string{} + for _, v := range c.Vertices { + of[v.Key] = v.Of + } + for _, e := range c.Summary { + if strings.HasPrefix(of[e.Src], ofGlobal) && strings.HasPrefix(of[e.Dst], ofGlobal) { + t.Errorf("%v: global → global summary edge %+v", u.Key, e) + } + } + } +} + +// TestSummary_CaptureWrittenThroughAlias: a literal that writes through a +// local alias of a captured pointer (`q := p; q.n = x`) writes no path of p, +// yet the write reaches p's pointee. p may hold a reference, so the literal +// has a capture formal_out for p (the shared by-ref predicate, A10, as a +// param in mutatedFormals): the closure has the flow x → p, and its capture +// site has an actual_out for p that reaches CaptureAlias's formal_out of p. +// Found by the metamorphic relation MR5 (introduce an alias), which turned +// CaptureParam's flow from x into a false "no flow". +func TestSummary_CaptureWrittenThroughAlias(t *testing.T) { + in, _, p := runPorts(t) + lit := Target{Callable: dataflow.L3Key{Path: sdgFile, Signature: sdgPkg + ".CaptureAlias.closure_1"}} + ins, outs := p.formals(lit) + if len(ins) != 2 || len(outs) != 1 || outs[0].Of != ":p" { + t.Fatalf("CaptureAlias.closure_1 formals = %+v / %+v, want 2 in, one out :p", ins, outs) + } + b := newSDGBuilder(in, p.d) + flows := b.summarize() + // closure_1: formal_in 0 = :p, 1 = :x; formal_out 0 = :p. + checkFlows(t, flows, lit, [][2]int{{0, 0}, {1, 0}}) + got := p.wire(sdgKey("CaptureAlias")) + checkHas(t, "wire(CaptureAlias)", got, pe("499:7/actual_out", "@formal_out", "p")) +} diff --git a/internal/sdg/testdata/wire_sdg_off.golden b/internal/sdg/testdata/wire_sdg_off.golden new file mode 100644 index 0000000..5f9d5c3 --- /dev/null +++ b/internal/sdg/testdata/wire_sdg_off.golden @@ -0,0 +1,451 @@ +example.com/dataflow/sdg.AliasArrPtr + 491:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 489:2 pa reaching-defs + @formal_in:0 -> 491:2 pa[*] reaching-defs + @formal_in:0 -> @formal_out:1 pa reaching-defs +example.com/dataflow/sdg.AliasArraySlice + 399:2 -> @formal_out reaching-defs + @formal_in:0 -> 398:2 v reaching-defs +example.com/dataflow/sdg.AliasEntryGlobal + 469:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 469:2 q.UserID reaching-defs + @formal_in:0 -> @formal_out:1 q reaching-defs + @formal_in:1 -> 468:2 example.com/dataflow/sdg::GReq.UserID reaching-defs +example.com/dataflow/sdg.AliasEntryValue + 455:3 -> @formal_out:1 q reaching-defs + 457:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 455:3 req reaching-defs + @formal_in:1 -> 457:2 q.UserID reaching-defs + @formal_in:1 -> @formal_out:1 q reaching-defs + @formal_in:2 -> 454:2 c reaching-defs +example.com/dataflow/sdg.AliasFieldAddr + 408:2 -> @formal_out reaching-defs + @formal_in:0 -> 407:2 v reaching-defs +example.com/dataflow/sdg.AliasIfaces + 445:2 -> 445:2/actual_in:0 _recv reaching-defs + 445:2 -> 445:2/actual_in:1 p reaching-defs + 445:2 -> @formal_out:2 p reaching-defs + 445:2/actual_out:0 -> 445:2 :0 reaching-defs + 445:2/actual_out:1 -> 445:2 :1 reaching-defs + 445:2/actual_out:2 -> 445:2 _recv reaching-defs + 445:2/actual_out:3 -> 445:2 p reaching-defs + 446:10/actual_out:0 -> 446:2 :0 reaching-defs + 446:10/actual_out:1 -> 446:2 :1 reaching-defs + 446:10/actual_out:2 -> 446:2 _recv reaching-defs + 446:10/actual_out:3 -> 446:2 p reaching-defs + 446:2 -> 446:10/actual_in:0 _recv reaching-defs + 446:2 -> 446:10/actual_in:1 p reaching-defs + 446:2 -> @formal_out:2 p reaching-defs + 447:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 443:2 f reaching-defs + @formal_in:0 -> 444:2 f reaching-defs + @formal_in:0 -> @formal_out:1 f reaching-defs + @formal_in:1 -> 445:2 p reaching-defs + @formal_in:1 -> 446:2 p reaching-defs + @formal_in:1 -> @formal_out:2 p reaching-defs +example.com/dataflow/sdg.AliasPtr + 361:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 359:2 req reaching-defs + @formal_in:0 -> 361:2 req.UserID reaching-defs + @formal_in:0 -> @formal_out:1 req reaching-defs + @formal_in:1 -> 360:2 id reaching-defs +example.com/dataflow/sdg.AliasPtrField + 437:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 435:2 r.ptr reaching-defs + @formal_in:0 -> 437:2 r.ptr.n reaching-defs + @formal_in:0 -> @formal_out:1 r reaching-defs + @formal_in:1 -> 436:2 v reaching-defs +example.com/dataflow/sdg.AliasPtrPtr + 477:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 474:2 a reaching-defs + @formal_in:0 -> @formal_out:1 a reaching-defs + @formal_in:1 -> 476:2 b reaching-defs + @formal_in:1 -> @formal_out:2 b reaching-defs +example.com/dataflow/sdg.AliasPtrValue + 391:2 -> @formal_out reaching-defs + @formal_in:0 -> 390:2 id reaching-defs +example.com/dataflow/sdg.AliasPtrWhole + 369:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 367:2 req reaching-defs + @formal_in:0 -> 369:2 req reaching-defs + @formal_in:0 -> @formal_out:1 req reaching-defs + @formal_in:1 -> 368:2 id reaching-defs +example.com/dataflow/sdg.AliasRefField + 425:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 423:2 s.m reaching-defs + @formal_in:0 -> 425:2 s.m[*] reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 424:2 k reaching-defs + @formal_in:1 -> 425:2 k reaching-defs + @formal_in:2 -> 424:2 v reaching-defs +example.com/dataflow/sdg.AliasSlicePtr + 484:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 482:2 s reaching-defs + @formal_in:0 -> 484:2 s[*] reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 483:2 b reaching-defs + @formal_in:1 -> @formal_out:2 b reaching-defs +example.com/dataflow/sdg.AliasValue + 376:2 -> @formal_out reaching-defs + @formal_in:0 -> 374:2 req reaching-defs + @formal_in:0 -> 376:2 req.UserID reaching-defs + @formal_in:1 -> 375:2 id reaching-defs +example.com/dataflow/sdg.AliasValueCopies + 417:2 -> @formal_out reaching-defs + @formal_in:0 -> 414:2 s reaching-defs + @formal_in:0 -> 415:2 s reaching-defs + @formal_in:1 -> 416:2 v reaching-defs +example.com/dataflow/sdg.Buffered + 93:2 -> 93:2/actual_in:0 _recv reaching-defs + 93:2 -> 93:2/actual_in:1 p reaching-defs + 93:2 -> @formal_out:0 w reaching-defs + 93:2 -> @formal_out:1 p reaching-defs + 93:2/actual_out:0 -> 93:2 :0 reaching-defs + 93:2/actual_out:1 -> 93:2 :1 reaching-defs + 93:2/actual_out:2 -> 93:2 _recv reaching-defs + 93:2/actual_out:3 -> 93:2 p reaching-defs + @formal_in:0 -> 93:2 w reaching-defs + @formal_in:0 -> @formal_out:0 w reaching-defs + @formal_in:1 -> 93:2 p reaching-defs + @formal_in:1 -> @formal_out:1 p reaching-defs +example.com/dataflow/sdg.CallAliasAssign + 382:2 -> 382:2/actual_in:0 p reaching-defs + 382:2 -> 382:2/actual_in:1 v reaching-defs + 382:2 -> @formal_out p reaching-defs + 382:2/actual_out -> 382:2 p reaching-defs + @formal_in:0 -> 382:2 p reaching-defs + @formal_in:0 -> @formal_out p reaching-defs + @formal_in:1 -> 382:2 v reaching-defs +example.com/dataflow/sdg.CallFuncValue + 51:2 -> 51:9/actual_in:0 x reaching-defs + 51:2 -> @formal_out:0 reaching-defs + 51:9/actual_out -> 51:2 reaching-defs + @formal_in:0 -> 51:2 f reaching-defs + @formal_in:0 -> @formal_out:1 f reaching-defs +example.com/dataflow/sdg.CallPromoted + 80:2 -> 80:9/actual_in:0 _recv reaching-defs + 80:2 -> @formal_out:0 reaching-defs + 80:2 -> @formal_out:1 o reaching-defs + 80:9/actual_out:0 -> 80:2 reaching-defs + 80:9/actual_out:1 -> 80:2 _recv reaching-defs + @formal_in:0 -> 80:2 o reaching-defs + @formal_in:0 -> @formal_out:1 o reaching-defs +example.com/dataflow/sdg.CallVariadic + 195:2 -> 195:7/actual_in:0 xs reaching-defs + 195:7/actual_out:0 -> 195:2 reaching-defs + 195:7/actual_out:1 -> 195:2 xs reaching-defs + 196:16/actual_out:0 -> 196:2 :0 reaching-defs + 196:16/actual_out:1 -> 196:2 :1 reaching-defs + 196:2 -> 196:7/actual_in:0 xs reaching-defs + 196:7/actual_out:0 -> 196:2 reaching-defs + 196:7/actual_out:1 -> 196:2 xs reaching-defs + 197:15/actual_out:0 -> 197:2 :0 reaching-defs + 197:15/actual_out:1 -> 197:2 :1 reaching-defs + 197:2 -> 197:7/actual_in:0 a reaching-defs + 197:2 -> 197:7/actual_in:1 b reaching-defs + 197:7/actual_out -> 197:2 reaching-defs + 198:2 -> @formal_out reaching-defs +example.com/dataflow/sdg.CaptureAlias + 499:2 -> 499:7/actual_in:0 :p reaching-defs + 499:2 -> @formal_out p reaching-defs + 499:7/actual_out -> 499:2 :p reaching-defs + 499:7/actual_out -> @formal_out p reaching-defs + @formal_in:0 -> 499:2 p reaching-defs + @formal_in:0 -> 499:7/actual_in:0 :p reaching-defs + @formal_in:0 -> @formal_out p reaching-defs + @formal_in:1 -> 499:2 x reaching-defs + @formal_in:1 -> 499:7/actual_in:1 :x reaching-defs +example.com/dataflow/sdg.CaptureAlias.closure_1 + @formal_in:0 -> 499:16 p reaching-defs + @formal_in:0 -> @formal_out :p reaching-defs + @formal_in:1 -> 499:24 x reaching-defs +example.com/dataflow/sdg.CaptureArg + 267:2 -> 267:7/actual_in:0 f reaching-defs + 267:7/actual_out:0 -> 267:2 reaching-defs + 267:7/actual_out:1 -> 267:2 f reaching-defs + 268:2 -> @formal_out reaching-defs + @formal_in:0 -> 267:13/actual_in:0 :v reaching-defs + @formal_in:0 -> 267:2 v reaching-defs +example.com/dataflow/sdg.CaptureArg.closure_1 + 267:26 -> @formal_out reaching-defs + @formal_in:0 -> 267:26 v reaching-defs +example.com/dataflow/sdg.CaptureNamed + 293:2 -> 293:7/actual_in:0 :err reaching-defs + 293:2 -> @formal_out reaching-defs + 293:7/actual_out -> 293:2 :err reaching-defs + 293:7/actual_out -> 295:2 :err reaching-defs + 293:7/actual_out -> @formal_out reaching-defs + @entry -> 293:7/actual_in:0 :err reaching-defs + @entry -> @formal_out reaching-defs +example.com/dataflow/sdg.CaptureNamed.closure_1 + 293:16 -> @formal_out :err reaching-defs +example.com/dataflow/sdg.CaptureParam + 301:2 -> 301:7/actual_in:0 :p reaching-defs + 301:2 -> @formal_out p reaching-defs + 301:7/actual_out -> 301:2 :p reaching-defs + 301:7/actual_out -> @formal_out p reaching-defs + @formal_in:0 -> 301:2 p reaching-defs + @formal_in:0 -> 301:7/actual_in:0 :p reaching-defs + @formal_in:0 -> @formal_out p reaching-defs +example.com/dataflow/sdg.CaptureParam.closure_1 + 301:16 -> @formal_out :p reaching-defs + @formal_in:0 -> 301:16 p reaching-defs + @formal_in:0 -> @formal_out :p reaching-defs +example.com/dataflow/sdg.CaptureSite + 214:2 -> 215:7/actual_in:0 :x reaching-defs + 215:2 -> 215:7/actual_in:0 :x reaching-defs + 215:7/actual_out -> 215:2 :x reaching-defs + 219:2 -> @formal_out reaching-defs + @formal_in:0 -> 214:2 v reaching-defs +example.com/dataflow/sdg.CaptureSite.closure_1 + 216:3 -> @formal_out:1 :x reaching-defs + 217:3 -> @formal_out:0 reaching-defs + @formal_in:0 -> 216:3 x reaching-defs +example.com/dataflow/sdg.ChainLit + 259:2/2 -> 259:2/actual_in:0 :x reaching-defs + 259:2/actual_out -> 259:2/2 reaching-defs + @formal_in:0 -> 259:2/2 v reaching-defs +example.com/dataflow/sdg.ChainLit.closure_1 + 259:25 -> @formal_out reaching-defs + @formal_in:0 -> 259:18 x reaching-defs +example.com/dataflow/sdg.DeadLit + @formal_in:0 -> 274:2 v reaching-defs +example.com/dataflow/sdg.DeadLit.closure_1 + 276:20 -> @formal_out reaching-defs + @formal_in:0 -> 276:20 v reaching-defs +example.com/dataflow/sdg.DirectLit + 226:2 -> 226:2/actual_in:0 a reaching-defs + 226:2 -> 226:2/actual_in:1 :y reaching-defs + 226:2/actual_out -> 226:2 :y reaching-defs + 229:2 -> @formal_out reaching-defs + @formal_in:0 -> 226:2 v reaching-defs +example.com/dataflow/sdg.DirectLit.closure_1 + 227:3 -> @formal_out :y reaching-defs + @formal_in:0 -> 227:3 a reaching-defs +example.com/dataflow/sdg.Inner.M + 65:2 -> @formal_out reaching-defs + @formal_in:0 -> 65:2 i.n reaching-defs +example.com/dataflow/sdg.MethodExpr + 209:2 -> 209:9/actual_in:0 s reaching-defs + 209:2 -> 209:9/actual_in:1 k reaching-defs + 209:2 -> @formal_out:0 reaching-defs + 209:2 -> @formal_out:1 s reaching-defs + 209:9/actual_out:0 -> 209:2 reaching-defs + 209:9/actual_out:1 -> 209:2 s reaching-defs + @formal_in:0 -> 209:2 s reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 209:2 k reaching-defs +example.com/dataflow/sdg.MethodExprVariadic + 287:2 -> 287:9/actual_in:0 _recv reaching-defs + 287:2 -> @formal_out reaching-defs + 287:9/actual_out -> 287:2 reaching-defs + @formal_in:0 -> 287:2 v reaching-defs +example.com/dataflow/sdg.MethodValue + 204:2 -> 204:9/actual_in:0 s reaching-defs + 204:2 -> 204:9/actual_in:1 k reaching-defs + 204:2 -> @formal_out:0 reaching-defs + 204:9/actual_out:0 -> 204:2 reaching-defs + 204:9/actual_out:1 -> 204:2 s reaching-defs + @formal_in:0 -> 203:2 s reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 204:2 k reaching-defs +example.com/dataflow/sdg.MixedValue + 241:2 -> 241:9/actual_in:0 s reaching-defs + 241:2 -> 241:9/actual_in:1 k reaching-defs + 241:2 -> @formal_out:0 reaching-defs + 241:9/actual_out:0 -> 241:2 reaching-defs + 241:9/actual_out:1 -> 241:2 s reaching-defs + @formal_in:0 -> 239:3 s reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 238:2 b reaching-defs +example.com/dataflow/sdg.Named + 179:2 -> @formal_out reaching-defs +example.com/dataflow/sdg.PassBuffer + 99:2 -> 99:2/actual_in:0 w reaching-defs + 99:2 -> 99:2/actual_in:1 p reaching-defs + 99:2/actual_out:0 -> 99:2 w reaching-defs + 99:2/actual_out:1 -> 99:2 p reaching-defs +example.com/dataflow/sdg.PassDouble + 56:2 -> 56:9/actual_in:0 f reaching-defs + 56:2 -> @formal_out reaching-defs + 56:9/actual_out:0 -> 56:2 reaching-defs + 56:9/actual_out:1 -> 56:2 f reaching-defs +example.com/dataflow/sdg.PassOuter + 86:2 -> 86:7/actual_in:0 o reaching-defs + 86:7/actual_out:0 -> 86:2 reaching-defs + 86:7/actual_out:1 -> 86:2 o reaching-defs + 87:2 -> 87:7/actual_in:0 o reaching-defs + 87:7/actual_out:0 -> 87:2 reaching-defs + 87:7/actual_out:1 -> 87:2 o reaching-defs + 88:2 -> @formal_out reaching-defs +example.com/dataflow/sdg.PrunedCall + 246:2 -> @formal_out reaching-defs + @formal_in:0 -> 246:2 v reaching-defs +example.com/dataflow/sdg.ScalarSink + 510:2 -> 510:2/actual_in:0 v reaching-defs + 510:2/actual_out -> 510:2 reaching-defs + 511:2 -> @formal_out reaching-defs + @formal_in:0 -> 509:2 a reaching-defs +example.com/dataflow/sdg.Stores + 40:2 -> 40:2/actual_in:0 s reaching-defs + 40:2 -> 40:2/actual_in:1 k reaching-defs + 40:2/actual_out:0 -> 40:2 reaching-defs + 40:2/actual_out:1 -> 40:2 s reaching-defs + 41:2 -> 41:2/actual_in:0 s reaching-defs + 41:2 -> 41:2/actual_in:1 k reaching-defs + 41:2/actual_out:0 -> 41:2 reaching-defs + 41:2/actual_out:1 -> 41:2 s reaching-defs + 42:2 -> 42:2/actual_in:0 s reaching-defs + 42:2 -> 42:2/actual_in:1 k reaching-defs + 42:2/actual_out:0 -> 42:2 reaching-defs + 42:2/actual_out:1 -> 42:2 s reaching-defs +example.com/dataflow/sdg.UseStore + 34:2 -> 34:9/actual_in:0 _recv reaching-defs + 34:2 -> 34:9/actual_in:1 k reaching-defs + 34:2 -> @formal_out:0 reaching-defs + 34:2 -> @formal_out:1 s reaching-defs + 34:9/actual_out:0 -> 34:2 reaching-defs + 34:9/actual_out:1 -> 34:2 _recv reaching-defs + @formal_in:0 -> 34:2 s reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 34:2 k reaching-defs +example.com/dataflow/sdg.Variadic + 185:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 185:2 xs reaching-defs + @formal_in:0 -> @formal_out:1 xs reaching-defs +example.com/dataflow/sdg.a + 308:2 -> 308:9/actual_in:0 x reaching-defs + 308:2 -> @formal_out reaching-defs + 308:9/actual_out -> 308:2 reaching-defs + @formal_in:0 -> 308:2 x reaching-defs +example.com/dataflow/sdg.aliasAssign + @formal_in:0 -> 152:2 p reaching-defs + @formal_in:0 -> @formal_out p reaching-defs + @formal_in:1 -> 153:2 v reaching-defs +example.com/dataflow/sdg.aliasCall + 159:2 -> 159:2/actual_in:0 p reaching-defs + @formal_in:0 -> 158:2 p reaching-defs + @formal_in:0 -> @formal_out p reaching-defs +example.com/dataflow/sdg.aliasRange + @formal_in:0 -> 145:2 s reaching-defs + @formal_in:0 -> @formal_out s reaching-defs + @formal_in:1 -> 146:3 v reaching-defs +example.com/dataflow/sdg.apply + 262:32 -> @formal_out:0 reaching-defs + 262:39/actual_out -> 262:32 reaching-defs + @formal_in:0 -> 262:32 f reaching-defs + @formal_in:0 -> @formal_out:1 f reaching-defs +example.com/dataflow/sdg.b + 312:2 -> 312:9/actual_in:0 x reaching-defs + 312:2 -> @formal_out reaching-defs + 312:9/actual_out -> 312:2 reaching-defs + @formal_in:0 -> 312:2 x reaching-defs +example.com/dataflow/sdg.c + 316:2 -> @formal_out reaching-defs + @formal_in:0 -> 316:2 x reaching-defs +example.com/dataflow/sdg.callIgnore + 326:2 -> 326:9/actual_in:0 x reaching-defs + 326:2 -> 326:9/actual_in:1 y reaching-defs + 326:2 -> @formal_out reaching-defs + 326:9/actual_out -> 326:2 reaching-defs + @formal_in:0 -> 326:2 v reaching-defs + @formal_in:1 -> 326:2 w reaching-defs +example.com/dataflow/sdg.callUnknown + 252:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 252:2 f reaching-defs + @formal_in:0 -> @formal_out:1 f reaching-defs +example.com/dataflow/sdg.double + 46:2 -> @formal_out reaching-defs + @formal_in:0 -> 46:2 x reaching-defs +example.com/dataflow/sdg.dropX + 333:3 -> @formal_out reaching-defs + 335:2 -> 335:9/actual_in:0 y reaching-defs + 335:2 -> @formal_out reaching-defs + 335:9/actual_out -> 335:2 reaching-defs + @formal_in:1 -> 332:2 y reaching-defs + @formal_in:1 -> 333:3 y reaching-defs + @formal_in:1 -> 335:2 y reaching-defs +example.com/dataflow/sdg.dropY + 339:2 -> 339:9/actual_in:0 x reaching-defs + 339:2 -> 339:9/actual_in:1 y reaching-defs + 339:2 -> @formal_out reaching-defs + 339:9/actual_out -> 339:2 reaching-defs + @formal_in:0 -> 339:2 y reaching-defs +example.com/dataflow/sdg.ignore + 322:2 -> @formal_out reaching-defs + @formal_in:1 -> 322:2 y reaching-defs +example.com/dataflow/sdg.implicit + 346:3 -> @formal_out reaching-defs + 348:2 -> @formal_out reaching-defs + @formal_in:0 -> 345:2 x reaching-defs +example.com/dataflow/sdg.initConfig + 111:2 -> @formal_out :example.com/dataflow/sdg::Config reaching-defs +example.com/dataflow/sdg.memStore.Get + 21:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 21:2 s.m[*] reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 21:2 k reaching-defs +example.com/dataflow/sdg.multiT.Multi + 174:2 -> @formal_out:0 :0 reaching-defs + 174:2 -> @formal_out:1 :1 reaching-defs + @formal_in:0 -> 174:2 s.n reaching-defs + @formal_in:1 -> 174:2 a reaching-defs +example.com/dataflow/sdg.mutate + 132:2 -> @formal_out:0 p reaching-defs + 133:2 -> @formal_out:1 s reaching-defs + @formal_in:0 -> 132:2 p reaching-defs + @formal_in:0 -> @formal_out:0 p reaching-defs + @formal_in:1 -> 133:2 s.m reaching-defs + @formal_in:1 -> @formal_out:1 s reaching-defs + @formal_in:3 -> 135:2 q reaching-defs + @formal_in:3 -> @formal_out:2 q reaching-defs +example.com/dataflow/sdg.pair + 188:26 -> @formal_out:0 :0 reaching-defs + 188:26 -> @formal_out:1 :1 reaching-defs +example.com/dataflow/sdg.readConfig + 116:10/actual_out -> 116:3 reaching-defs + 116:3 -> 116:10/actual_in:0 :example.com/dataflow/sdg::Config reaching-defs + 116:3 -> @formal_out reaching-defs + 118:2 -> @formal_out reaching-defs + @formal_in:0 -> 115:2 example.com/dataflow/sdg::Config reaching-defs + @formal_in:0 -> 116:3 example.com/dataflow/sdg::Config reaching-defs + @formal_in:0 -> 118:2 example.com/dataflow/sdg::Config reaching-defs +example.com/dataflow/sdg.readConst + 126:2 -> @formal_out reaching-defs +example.com/dataflow/sdg.sinkInt + 504:27 -> @formal_out reaching-defs + @formal_in:0 -> 504:27 v reaching-defs +example.com/dataflow/sdg.sqlStore.Get + 29:2 -> @formal_out:0 reaching-defs + @formal_in:0 -> 29:2 s.dsn reaching-defs + @formal_in:0 -> @formal_out:1 s reaching-defs + @formal_in:1 -> 29:2 k reaching-defs +example.com/dataflow/sdg.sqlStore.setDSN + 140:2 -> @formal_out s reaching-defs + @formal_in:0 -> 140:2 s reaching-defs + @formal_in:0 -> @formal_out s reaching-defs + @formal_in:1 -> 140:2 d reaching-defs +example.com/dataflow/sdg.sumT.Sum + 282:34 -> @formal_out:0 reaching-defs + @formal_in:1 -> 282:34 xs reaching-defs + @formal_in:1 -> @formal_out:1 xs reaching-defs +example.com/dataflow/sdg.takeGReq + 463:24 -> @formal_out:0 reaching-defs + @formal_in:0 -> 463:24 example.com/dataflow/sdg::GReq reaching-defs + @formal_in:0 -> @formal_out:1 :example.com/dataflow/sdg::GReq reaching-defs +example.com/dataflow/sdg.takeInner +example.com/dataflow/sdg.takeTwo + 190:30 -> @formal_out reaching-defs + @formal_in:0 -> 190:30 a reaching-defs + @formal_in:1 -> 190:30 b reaching-defs +example.com/dataflow/sdg.unusedRef + 166:2 -> @formal_out reaching-defs + @formal_in:1 -> 166:2 v reaching-defs +example.com/dataflow/sdg.upper + 232:31 -> @formal_out reaching-defs + @formal_in:0 -> 232:31 t reaching-defs +example.com/dataflow/sdg.useConfig + 122:2 -> 122:9/actual_in:0 :example.com/dataflow/sdg::Config reaching-defs + 122:2 -> @formal_out reaching-defs + 122:9/actual_out -> 122:2 reaching-defs + @formal_in:0 -> 122:2 example.com/dataflow/sdg::Config reaching-defs diff --git a/internal/sdg/variant_test.go b/internal/sdg/variant_test.go new file mode 100644 index 0000000..c835369 --- /dev/null +++ b/internal/sdg/variant_test.go @@ -0,0 +1,87 @@ +package sdg + +import ( + "path/filepath" + "reflect" + "slices" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +const variantPkg = "example.com/dataflow/variant/" + +var ( + variantUse = dataflow.L3Key{Path: filepath.Join("variant", "u", "u.go"), Signature: variantPkg + "u.Use"} + variantImpl = dataflow.L3Key{Path: filepath.Join("variant", "p", "p.go"), Signature: variantPkg + "p.impl.M"} +) + +// siteTargetsOf is the union of the dispatch targets of unit's call sites. +func siteTargetsOf(d *dispatch, unit dataflow.L3Key) []Target { + var out []Target + for site, ts := range d.sites { + if site.unit == unit { + out = append(out, ts...) + } + } + slices.SortFunc(out, compareTarget) + return slices.Compact(out) +} + +// TestSSA_EveryPackageVariant (F2): with tests loaded, go/packages yields +// the test variant "p [p.test]" beside the plain p that u imports. SSA +// must build both: u.Use's p.New().M(x) dispatches by VTA to impl.M only +// when the plain p, whose New u calls, has bodies. Use then writes the +// global impl.M writes, x reaches its , and nothing is degraded. +func TestSSA_EveryPackageVariant(t *testing.T) { + in := loadInputOpts(t, "dataflow", false) + res, d := run(in) + if len(res.Degraded) != 0 { + t.Fatalf("degraded = %+v, want none", res.Degraded) + } + impl := Target{Callable: variantImpl} + if ts := siteTargetsOf(d, variantUse); !slices.Contains(ts, impl) { + t.Errorf("Use's targets = %v, want impl.M among them", ts) + } + vta := false + for _, e := range res.CallEdges { + vta = vta || e.Source == variantUse.Signature && e.Target == variantImpl.Signature && slices.Equal(e.Provenance, []string{"vta"}) + } + if !vta { + t.Errorf("no VTA call_graph edge Use -> impl.M: %+v", res.CallEdges) + } + fp := footprints(in, d)[Target{Callable: variantUse}] + if !slices.Contains(fp.GlobalWrites, variantPkg+"p::Last") { + t.Errorf("Use writes %v, want %sp::Last", fp.GlobalWrites, variantPkg) + } + flows := newSDGBuilder(in, d).summarize() + if !slices.Contains(flows[Target{Callable: variantUse}], [2]int{0, 0}) { + t.Errorf("Use's flows = %v, want x → ([0 0])", flows[Target{Callable: variantUse}]) + } + // Every target and param edge appears once though two SSA variants of + // p join the same callables. + for site, ts := range d.sites { + if !slices.IsSortedFunc(ts, compareTarget) || len(slices.CompactFunc(slices.Clone(ts), func(a, b Target) bool { return a == b })) != len(ts) { + t.Errorf("site %v targets not sorted and unique: %v", site, ts) + } + } + if !slices.IsSortedFunc(res.ParamIn, compareParamEdge) || len(slices.Compact(slices.Clone(res.ParamIn))) != len(res.ParamIn) { + t.Error("param_in not sorted and unique") + } +} + +// TestSSA_VariantsNeutralWithoutTests (F2): with tests skipped there is one +// variant per path, so the L4 result is the same whether SSA is built from +// in.Variants or from in.Pkgs. +func TestSSA_VariantsNeutralWithoutTests(t *testing.T) { + in := loadInput(t, "dataflow") + if len(in.Variants) != len(in.Pkgs) { + t.Fatalf("%d variants, %d packages: want one variant per path with tests skipped", len(in.Variants), len(in.Pkgs)) + } + a := Run(in) + in.Variants = nil + b := Run(in) + if !reflect.DeepEqual(a, b) { + t.Error("L4 result differs between Variants and Pkgs with tests skipped") + } +} diff --git a/internal/sdg/vertices.go b/internal/sdg/vertices.go new file mode 100644 index 0000000..b7067c1 --- /dev/null +++ b/internal/sdg/vertices.go @@ -0,0 +1,819 @@ +package sdg + +import ( + "go/ast" + "go/token" + "go/types" + "slices" + "strconv" + "strings" + + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// The `of` markers of the L4 vertices (spec §4.6, D6): Python's spelling, +// which the SDK's resolve_value and value_candidate read. +const ( + ofReturn = "" + ofCapture = ":" + ofGlobal = ":" +) + +// ports builds the L4 port vertices (spec §4.6): the formals of every call +// target and the actuals of every resolved call site and capture site. It +// reads the L3 results, the dispatch, the footprints and go/types, never the +// SSA program, so it is safe after a no_vta. Formal sets are cached: it is +// not safe for concurrent use. +type ports struct { + in Input + d *dispatch + fps map[Target]Footprint + fx *formalIndex + units map[dataflow.L3Key]dataflow.Unit + byFn map[ast.Node]dataflow.L3Key + sets map[Target]*formalSet + calls map[dataflow.L3Key]map[string]*ast.CallExpr // unit → call key → call + kept map[dataflow.L3Key]map[string]bool // unit → its kept L3 node keys + names map[string]string // interned formal keys and `of`s (intern) + // fnsByPath is every unit function by file (rootOf, the alias oracle's + // address-taken scan root); built on first use. + fnsByPath map[string][]ast.Node +} + +func newPorts(in Input, d *dispatch) *ports { + p := &ports{ + in: in, + d: d, + fps: footprints(in, d), + fx: newFormalIndex(in, d), + units: make(map[dataflow.L3Key]dataflow.Unit, len(in.Units)), + byFn: make(map[ast.Node]dataflow.L3Key, len(in.Units)), + sets: map[Target]*formalSet{}, + calls: map[dataflow.L3Key]map[string]*ast.CallExpr{}, + kept: map[dataflow.L3Key]map[string]bool{}, + names: map[string]string{}, + } + for _, u := range in.Units { + p.units[u.Key] = u + p.byFn[u.Fn] = u.Key + } + return p +} + +// intern returns the one shared copy of s (a memory lever, spec §4.13): +// every target in a large SCC has the same few hundred `:` formals, +// and every target the same `@formal_in:` keys, so building each anew +// would hold one copy per vertex. +func (p *ports) intern(s string) string { + if !memLevers { + return s + } + if c, ok := p.names[s]; ok { + return c + } + p.names[s] = s + return s +} + +// formalSet is the formals of one target, with how a call site binds them: +// by position (receiver, declared params, results) or by `of` (captures, +// globals). +type formalSet struct { + in, out []Vertex + recv bool // inPos[0] is the receiver + params int // declared params + inPos []string // formal_in key of the receiver (when recv), then of each declared param + posOf []string // the `of` of each inPos entry + outPos []string // formal_out key of each inPos entry that is a by-ref out, else "" + results []string // formal_out key of each result + captures []string // `of` of each capture, in formal order + capOut []string // `of` of each written capture + globIn []string // `of` of each global formal_in, sorted + globOut []string // `of` of each global formal_out, sorted + inByOf map[string]string + outByOf map[string]string +} + +// formals returns t's formal_in and formal_out vertices (spec §4.6, A9, D6), +// in index order: +// +// - formal_in (`@formal_in:`, always indexed): the receiver (its name, +// `_recv` when unnamed or `_`), the declared params (`_` when unnamed +// or `_`, i the declared index), the captures of a closure +// (`:`, dataflow.FreeVars order), then the globals the +// footprint reads or writes (`:::`, sorted); +// - formal_out (`@formal_out` when there is exactly one, else +// `@formal_out:`): the results by position (`` for one, +// `:` for several, never the user's names), the by-ref +// params the footprint mutates in declaration order (receiver first, +// `of` = the formal's), the captures the closure writes +// (dataflow.CaptureWrites), then the global writes. +// +// t may be any target: a unit, an interface method or bodiless declaration +// (its *types.Func), or an external (its GoExternalCallee, typed through the +// load). The slices are shared: callers must not modify them. +func (p *ports) formals(t Target) (in, out []Vertex) { + fs := p.formalSet(t) + return fs.in, fs.out +} + +func (p *ports) formalSet(t Target) *formalSet { + if fs, ok := p.sets[t]; ok { + return fs + } + fs := &formalSet{inByOf: map[string]string{}, outByOf: map[string]string{}} + p.sets[t] = fs + formals := p.fx.of(t) + fp, ok := p.fps[t] + if !ok { + fp = Footprint{Mutated: byRefFormals(formals)} + } + mutated := map[string]bool{} + for _, m := range fp.Mutated { + mutated[m] = true + } + var ins, outs []string // `of`s + var outFrom []int // per out: the inPos entry it mirrors, or -1 + for i, f := range formals { + if i == 0 && f.recv { + fs.recv = true + } else { + fs.params++ + } + ins = append(ins, f.name) + fs.posOf = append(fs.posOf, f.name) + } + nRes := p.resultCount(t) + for i := range nRes { + outs = append(outs, p.intern(resultOf(i, nRes))) + outFrom = append(outFrom, -1) + } + for i, f := range formals { + if mutated[f.name] { + outs = append(outs, f.name) + outFrom = append(outFrom, i) + } + } + if u, ok := p.units[t.Callable]; ok && t.External == "" && u.Info != nil { + if lit, ok := u.Fn.(*ast.FuncLit); ok { + fs.captures, fs.capOut = captureOfs(lit, u.Info, p.in.K) + } + } + ins = append(ins, fs.captures...) + outs = append(outs, fs.capOut...) + for range fs.capOut { + outFrom = append(outFrom, -1) + } + for _, g := range unionSorted([][]string{fp.GlobalReads, fp.GlobalWrites}) { + fs.globIn = append(fs.globIn, p.intern(ofGlobal+g)) + } + for _, g := range fp.GlobalWrites { + fs.globOut = append(fs.globOut, p.intern(ofGlobal+g)) + outFrom = append(outFrom, -1) + } + ins = append(ins, fs.globIn...) + outs = append(outs, fs.globOut...) + + fs.outPos = make([]string, len(fs.posOf)) + for i, of := range ins { + key := p.intern("@formal_in:" + strconv.Itoa(i)) + fs.in = append(fs.in, Vertex{Key: key, Of: of, Kind: KindFormalIn}) + if i < len(fs.posOf) { + fs.inPos = append(fs.inPos, key) + } else { + fs.inByOf[of] = key + } + } + for i, of := range outs { + key := "@formal_out" + if len(outs) > 1 { + key = p.intern(key + ":" + strconv.Itoa(i)) + } + fs.out = append(fs.out, Vertex{Key: key, Of: of, Kind: KindFormalOut}) + switch { + case i < nRes: + fs.results = append(fs.results, key) + case outFrom[i] >= 0: + fs.outPos[outFrom[i]] = key + default: + fs.outByOf[of] = key + } + } + return fs +} + +// resultOf is the D6 `of` of result i of n. +func resultOf(i, n int) string { + if n == 1 { + return ofReturn + } + return ofReturn + ":" + strconv.Itoa(i) +} + +// captureOfs returns the `of` of every variable lit captures (dataflow.FreeVars +// order) and of those it writes (writtenCaptures), in that order. A +// capture's name is unique: every variable a literal names from outside it +// resolves in the one scope chain the literal sits in. +func captureOfs(lit *ast.FuncLit, info *types.Info, k int) (all, written []string) { + w := writtenCaptures(lit, info, k) + for _, v := range dataflow.FreeVars(lit, info) { + all = append(all, ofCapture+v.Name()) + if w[v] { + written = append(written, ofCapture+v.Name()) + } + } + return all, written +} + +// writtenCaptures is the set of captures of lit that get a capture +// formal_out (spec §4.6): the bases lit writes (dataflow.CaptureWrites), and, +// by the shared by-ref predicate (A10, fix I1, as for a param in +// mutatedFormals), every capture whose type may hold a reference +// (dataflow.MayHoldRef). A capture is used by definition, and a write +// through a local alias (`q := p; q.n = v`) or passing it on is no def of +// the capture at L3, so a use is enough. +func writtenCaptures(lit *ast.FuncLit, info *types.Info, k int) map[types.Object]bool { + w := map[types.Object]bool{} + for _, path := range dataflow.CaptureWrites(lit, info, k) { + w[path.Base] = true + } + for _, v := range dataflow.FreeVars(lit, info) { + if dataflow.MayHoldRef(v.Type()) { + w[v] = true + } + } + return w +} + +// resultCount is the number of results of t: from its go/types signature +// where the load has one, else from the symbol table's or the external's +// type strings. +func (p *ports) resultCount(t Target) int { + x := p.fx + if t.External != "" { + e, ok := x.ext[t.External] + if !ok { + return 0 + } + if fn := x.libraryFunc(e); fn != nil { + return fn.Signature().Results().Len() + } + return countResults(e.ReturnType) + } + if u, ok := x.units[t.Callable]; ok && u.Info != nil { + if sig := unitSignature(u); sig != nil { + return sig.Results().Len() + } + } + if fn, ok := x.decls[t.Callable]; ok { + return fn.Signature().Results().Len() + } + return len(x.strs[t.Callable].ReturnTypes) +} + +// countResults counts the results of a GoExternalCallee.ReturnType: "" is +// none, "(A, B)" (types.TypeString's tuple) is the top-level commas plus one, +// anything else is one. +func countResults(s string) int { + if s == "" { + return 0 + } + if !strings.HasPrefix(s, "(") || !strings.HasSuffix(s, ")") { + return 1 + } + n, depth := 1, 0 + for _, r := range s[1 : len(s)-1] { + switch r { + case '(', '[', '{': + depth++ + case ')', ']', '}': + depth-- + case ',': + if depth == 0 { + n++ + } + } + } + return n +} + +// slotRecv is the call-side slot of the receiver: x of a method call x.M(…), +// or the function-value operand f of f(…) for a bound-method target (A8). +// Slots 0, 1, … are the call's function-type params. +const slotRecv = -1 + +// portBind binds one actual of a call or capture site to one target's +// formal: the param_in / param_out endpoints. +type portBind struct { + Actual string + Target Target + Formal string +} + +// site is the actuals of one call or capture site, and how each binds to +// each target's formals (in: actual_in → formal_in; out: formal_out → +// actual_out). +type site struct { + In, Out []Vertex + BindIn, BindOut []portBind +} + +// actuals returns the actual_in and actual_out vertices of call site +// callKey of unit (spec §4.6, A8, A11), whose dispatch targets are ts +// (dispatch.targets, or siteTargets). Keys are `/actual_in:` +// and `/actual_out` (exactly one) or `/actual_out:`; +// Parent is callKey. A call with no target, or whose node the CFG pruned, +// has none. +func (p *ports) actuals(unit dataflow.L3Key, callKey string, ts []Target) (in, out []Vertex) { + s := p.callSite(unit, callKey, ts) + return s.In, s.Out +} + +// callSite is actuals with the bindings. The actuals are the union over ts: +// +// - by position: the receiver slot (x of x.M(…); a method expression's +// first argument; for a bound-method target, the function-value +// operand), then the call's function-type params in order. A variadic +// call's excess values collapse onto its last param, and f(g()) has one +// value per g result. Each slot's `of` is that of the formal it binds +// in the static callee (L2's), else in the first target (ts order) that +// binds it. +// - then, at a directly called literal (`func(…){…}(…)`, deferred or +// launched), the literal's captures after the declared args; +// - then the globals by `of`: the static callee's (sorted), then any +// other target's extras in target order. +// +// actual_out: the results by position, then the by-ref slots (a slot whose +// bound formal some target mutates), then the written captures of a +// directly called literal, then the global writes (by `of`, as for in). +func (p *ports) callSite(unit dataflow.L3Key, callKey string, ts []Target) site { + if len(ts) == 0 || p.pruned(unit, callKey) { + return site{} + } + u := p.units[unit] + call := p.callExpr(unit, callKey) + if call == nil { + return site{} + } + static, hasStatic := p.d.static[siteKey{unit, callKey}] + if !hasStatic || !slices.Contains(ts, static) { + static, hasStatic = Target{}, false + } + order := ts + if hasStatic { + order = append([]Target{static}, slices.DeleteFunc(slices.Clone(ts), func(t Target) bool { return t == static })...) + } + + shape := callShapeOf(call, u.Info) + type slotBind struct { + t Target + pos int // index into the target's inPos + } + binds := map[int][]slotBind{} + for _, t := range order { + fs := p.formalSet(t) + for slot, pos := range shape.bind(fs) { + binds[slot] = append(binds[slot], slotBind{t, pos}) + } + } + slots := make([]int, 0, len(binds)) + for slot := range binds { + slots = append(slots, slot) + } + slices.Sort(slots) + + var s site + addIn := func(of string) string { + key := callKey + "/actual_in:" + strconv.Itoa(len(s.In)) + s.In = append(s.In, Vertex{Key: key, Of: of, Parent: callKey, Kind: KindActualIn}) + return key + } + slotOf := map[int]string{} + for _, slot := range slots { + b := binds[slot][0] + fs := p.formalSet(b.t) + slotOf[slot] = fs.posOf[b.pos] + key := addIn(slotOf[slot]) + for _, b := range binds[slot] { + s.BindIn = append(s.BindIn, portBind{key, b.t, p.formalSet(b.t).inPos[b.pos]}) + } + } + + // A directly called literal binds its captures here (A6). + var lit Target + litOK := false + if fl, ok := ast.Unparen(call.Fun).(*ast.FuncLit); ok { + if k, ok := p.byFn[fl]; ok && slices.Contains(ts, Target{Callable: k}) { + lit, litOK = Target{Callable: k}, true + } + } + if litOK { + fs := p.formalSet(lit) + for _, of := range fs.captures { + s.BindIn = append(s.BindIn, portBind{addIn(of), lit, fs.inByOf[of]}) + } + } + for _, of := range unionByOf(order, p, func(fs *formalSet) []string { return fs.globIn }) { + key := addIn(of) + for _, t := range order { + if f, ok := p.formalSet(t).inByOf[of]; ok { + s.BindIn = append(s.BindIn, portBind{key, t, f}) + } + } + } + + // actual_out: collect (of, bindings), then key once the count is known. + type pending struct { + of string + binds []portBind // Actual filled in below + } + var outs []pending + nRes := shape.results + if nRes < 0 { + nRes = len(p.formalSet(order[0]).results) + } + for i := range nRes { + o := pending{of: p.intern(resultOf(i, nRes))} + for _, t := range order { + if fs := p.formalSet(t); i < len(fs.results) { + o.binds = append(o.binds, portBind{Target: t, Formal: fs.results[i]}) + } + } + outs = append(outs, o) + } + for _, slot := range slots { + o := pending{of: slotOf[slot]} + for _, b := range binds[slot] { + if f := p.formalSet(b.t).outPos[b.pos]; f != "" { + o.binds = append(o.binds, portBind{Target: b.t, Formal: f}) + } + } + if len(o.binds) > 0 { + outs = append(outs, o) + } + } + if litOK { + fs := p.formalSet(lit) + for _, of := range fs.capOut { + outs = append(outs, pending{of: of, binds: []portBind{{Target: lit, Formal: fs.outByOf[of]}}}) + } + } + for _, of := range unionByOf(order, p, func(fs *formalSet) []string { return fs.globOut }) { + o := pending{of: of} + for _, t := range order { + if f, ok := p.formalSet(t).outByOf[of]; ok { + o.binds = append(o.binds, portBind{Target: t, Formal: f}) + } + } + outs = append(outs, o) + } + for i, o := range outs { + key := callKey + "/actual_out" + if len(outs) > 1 { + key += ":" + strconv.Itoa(i) + } + s.Out = append(s.Out, Vertex{Key: key, Of: o.of, Parent: callKey, Kind: KindActualOut}) + for _, b := range o.binds { + b.Actual = key + s.BindOut = append(s.BindOut, b) + } + } + return s +} + +// unionByOf is the union of list(formalSet(t)) over order: the first +// target's list, then each later target's `of`s not seen yet, in order. +func unionByOf(order []Target, p *ports, list func(*formalSet) []string) []string { + var out []string + seen := map[string]bool{} + for _, t := range order { + for _, of := range list(p.formalSet(t)) { + if !seen[of] { + seen[of] = true + out = append(out, of) + } + } + } + return out +} + +// callShape is what a call expression says about how its values bind: is +// it a method call (x.M(…), the receiver is x), how many params its +// function type has, how many values it passes (f(g()) passes each g +// result), and how many results it has (-1 when its type is unknown). +type callShape struct { + method bool + nParams int + values int + results int + noSig bool // the call's type is unknown: nParams is the value count + methodExpr bool // go/types recorded the callee as a method expression T.M +} + +func callShapeOf(call *ast.CallExpr, info *types.Info) callShape { + s := callShape{values: len(call.Args), results: -1} + var sig *types.Signature + if info != nil { + if len(call.Args) == 1 { + if tup, ok := info.TypeOf(call.Args[0]).(*types.Tuple); ok { + s.values = tup.Len() + } + } + if sel, ok := ast.Unparen(call.Fun).(*ast.SelectorExpr); ok { + if x := info.Selections[sel]; x != nil { + s.method = x.Kind() == types.MethodVal + s.methodExpr = x.Kind() == types.MethodExpr + } + } + if t := info.TypeOf(call.Fun); t != nil { + sig, _ = t.Underlying().(*types.Signature) + } + } + if sig == nil { + // No signature (a package with type errors): the value count stands + // in for the param count. The count alone must not decide that a call + // is a method expression (a bound method value f(x) on a one-param + // method passes params+1 values too), so bind then relies only on + // the selection go/types recorded (methodExpr). + s.nParams = s.values + s.noSig = true + return s + } + s.nParams = sig.Params().Len() + s.results = sig.Results().Len() + return s +} + +// bind maps the call's slots to fs's positional formals (indices into +// fs.inPos): +// +// - no receiver: call param j binds declared param j; +// - a method call x.M(…): x (slotRecv) binds the receiver, param j +// declared param j; +// - a call with one more param than fs declares — a method expression +// T.M(s, x), or its thunk reached through a function value: param 0 +// binds the receiver, param j declared param j-1; +// - otherwise a bound method value (f := s.M; f(x), A8): the +// function-value operand (slotRecv) binds the receiver, param j declared +// param j. +// +// Values bind in order, one actual per bound param. Only a variadic call +// passes more values than it has params: the excess collapse onto the last +// param (weak, as Java), which value nParams-1 already binds, so they add +// no actual. A param no value reaches (a variadic call with no excess) +// binds nothing. +func (s callShape) bind(fs *formalSet) map[int]int { + out := map[int]int{} + base, off := 0, 0 // declared param d is fs.inPos[base+d] and binds call param d+off + if fs.recv { + base = 1 + if !s.method && (s.methodExpr || !s.noSig && s.nParams == fs.params+1) { + off = 1 + out[0] = 0 // a method expression always passes its receiver + } else { + out[slotRecv] = 0 + } + } + for j := range min(s.values, s.nParams) { + if d := j - off; d >= 0 && d < fs.params { + out[j] = base + d + } + } + return out +} + +// pruned reports whether unit's call node callKey is not among its kept L3 +// nodes. A unit with no L3 result (l3_skipped) has no CFG to prune by: its +// L1 call nodes all stand. +func (p *ports) pruned(unit dataflow.L3Key, callKey string) bool { + if _, ok := p.in.L3[unit]; !ok { + return false + } + return !p.keptKeys(unit)[callKey] +} + +// keptKeys is the set of unit's kept L3 node keys (empty with no result). +func (p *ports) keptKeys(unit dataflow.L3Key) map[string]bool { + keep, ok := p.kept[unit] + if !ok { + keep = map[string]bool{} + for _, n := range p.in.L3[unit].Nodes { + keep[n.Key] = true + } + p.kept[unit] = keep + } + return keep +} + +// callExpr returns the call expression of unit's call key, paired by Lparen +// as BuildNodes pairs it. +func (p *ports) callExpr(unit dataflow.L3Key, callKey string) *ast.CallExpr { + m, ok := p.calls[unit] + if !ok { + u := p.units[unit] + byLparen := make(map[token.Pos]string, len(u.Calls)) + for _, c := range u.Calls { + byLparen[c.Lparen] = c.Key + } + m = map[string]*ast.CallExpr{} + for _, c := range astwalk.Calls(fnBody(u.Fn)) { + if k, ok := byLparen[c.Expr.Lparen]; ok { + m[k] = c.Expr + } + } + p.calls[unit] = m + } + return m[callKey] +} + +// siteTargets is the dispatch targets of unit's call callKey, plus, for a +// literal called in place, that literal's callable: L2 resolves no FuncLit +// callee, so without VTA (no_vta) only the syntax names it, and its +// captures must still bind at the call. +func (p *ports) siteTargets(unit dataflow.L3Key, callKey string) []Target { + ts := p.d.targets(unit, callKey) + if call := p.callExpr(unit, callKey); call != nil { + if fl, ok := ast.Unparen(call.Fun).(*ast.FuncLit); ok { + if k, ok := p.byFn[fl]; ok { + t := Target{Callable: k} + if i, found := slices.BinarySearchFunc(ts, t, compareTarget); !found { + ts = slices.Insert(slices.Clone(ts), i, t) + } + } + } + } + return ts +} + +// captureSites returns the capture-site actuals of unit u (A6): for every +// outermost function literal of its body that is not called in place and +// captures a variable, `/actual_in:` per capture and +// `/actual_out[:]` per written capture (`of` +// `:`), with Parent the host: the innermost CFG node holding +// the literal (hostOf). is the literal's line:col, taking the next +// free "/n" among taken (dataflow.NextFreeKey, as BodyNodes keys a +// statement), so it never equals a body key. A literal whose host the CFG +// pruned is dead and has none. +// +// A unit with no L3 result (l3_skipped) has no CFG to host a literal: its +// host is the innermost L1 call node whose call holds the literal (`once.Do( +// func(){…})`), else @entry (ruling, Task 12 review). Its port wiring is +// pass-through (wire). +func (p *ports) captureSites(u dataflow.Unit, taken map[string]bool) []Vertex { + var out []Vertex + for _, cs := range p.capSites(u, taken) { + out = append(out, cs.vertices()...) + } + return out +} + +// capSite is one capture site: its key, its host, its literal, and each +// capture (in actual_in order) with its actual_in key and, when the literal writes it, +// its actual_out key. +type capSite struct { + key, host string + lit *ast.FuncLit + caps []types.Object + ofs []string + in, out []string // per capture; out[i] == "" when not written +} + +func (cs capSite) vertices() []Vertex { + var vs []Vertex + for i, of := range cs.ofs { + vs = append(vs, Vertex{Key: cs.in[i], Of: of, Parent: cs.host, Kind: KindActualIn}) + } + for i, of := range cs.ofs { + if cs.out[i] != "" { + vs = append(vs, Vertex{Key: cs.out[i], Of: of, Parent: cs.host, Kind: KindActualOut}) + } + } + return vs +} + +// capSites is captureSites with each site's capture objects. +func (p *ports) capSites(u dataflow.Unit, taken map[string]bool) []capSite { + _, analysed := p.in.L3[u.Key] + body := fnBody(u.Fn) + if body == nil || u.Info == nil { + return nil + } + direct := map[*ast.FuncLit]bool{} + for _, c := range astwalk.Calls(body) { + if fl, ok := ast.Unparen(c.Expr.Fun).(*ast.FuncLit); ok { + direct[fl] = true + } + } + var all []dataflow.Node // every body node, pruned ones included; built once + kept := p.keptKeys(u.Key) + var out []capSite + for _, l := range astwalk.Literals(body) { + if direct[l.Lit] { + continue + } + caps := dataflow.FreeVars(l.Lit, u.Info) + if len(caps) == 0 { + continue + } + var host string + if analysed { + if all == nil { + all, _ = dataflow.BuildNodes(p.in.Fset, body, u.Calls) + } + h, ok := hostOf(all, l.Lit) + if !ok || !kept[h] { + continue + } + host = h + } else { + host = p.skippedHost(u, l.Lit) + } + written := writtenCaptures(l.Lit, u.Info, p.in.K) + nOut := 0 + for _, v := range caps { + if written[v] { + nOut++ + } + } + pos := p.in.Fset.PositionFor(l.Lit.Pos(), false) + cs := capSite{key: dataflow.NextFreeKey(v2.LocalID(pos.Line, pos.Column), taken), host: host, lit: l.Lit, caps: caps} + j := 0 + for i, v := range caps { + cs.ofs = append(cs.ofs, ofCapture+v.Name()) + cs.in = append(cs.in, cs.key+"/actual_in:"+strconv.Itoa(i)) + o := "" + if written[v] { + o = cs.key + "/actual_out" + if nOut > 1 { + o += ":" + strconv.Itoa(j) + } + j++ + } + cs.out = append(cs.out, o) + } + out = append(out, cs) + } + return out +} + +// skippedHost is the host of a capture site in a unit with no L3 result: +// the innermost of its L1 call nodes whose call expression holds lit, else +// @entry. +func (p *ports) skippedHost(u dataflow.Unit, lit *ast.FuncLit) string { + host, best := dataflow.EntryKey, token.Pos(-1) + for _, c := range u.Calls { + call := p.callExpr(u.Key, c.Key) + if call == nil || call.Pos() > lit.Pos() || lit.End() > call.End() { + continue + } + if span := call.End() - call.Pos(); best < 0 || span < best { + host, best = c.Key, span + } + } + return host +} + +// hostOf is the key of the innermost CFG node of nodes (BuildNodes' output, +// before pruning) whose span holds lit. A satellite call node is no CFG +// node; nor is a deferred call node the host of a literal in its operands: +// they are evaluated at the defer statement, whose DefUse uses the +// literal's captures. +func hostOf(nodes []dataflow.Node, lit *ast.FuncLit) (string, bool) { + best, found := dataflow.Node{}, false + for _, n := range nodes { + if n.Parent != "" || n.Deferred || !n.Pos.IsValid() || n.Pos > lit.Pos() || lit.End() > n.End { + continue + } + if !found || n.End-n.Pos < best.End-best.Pos { + best, found = n, true + } + } + return best.Key, found +} + +// vertices returns every L4 vertex of unit k, sorted by key: its formals, +// the actuals of each resolved, unpruned call site (siteTargets), and its +// capture sites. +func (p *ports) vertices(k dataflow.L3Key) []Vertex { + u, ok := p.units[k] + if !ok { + return nil + } + in, out := p.formals(Target{Callable: k}) + vs := slices.Concat(in, out) + taken := make(map[string]bool, len(u.Calls)) + for _, c := range u.Calls { + taken[c.Key] = true + ai, ao := p.actuals(k, c.Key, p.siteTargets(k, c.Key)) + vs = append(vs, ai...) + vs = append(vs, ao...) + } + for key := range p.keptKeys(k) { + taken[key] = true + } + vs = append(vs, p.captureSites(u, taken)...) + slices.SortFunc(vs, func(a, b Vertex) int { return strings.Compare(a.Key, b.Key) }) + return vs +} diff --git a/internal/sdg/vertices_test.go b/internal/sdg/vertices_test.go new file mode 100644 index 0000000..7b2bbf4 --- /dev/null +++ b/internal/sdg/vertices_test.go @@ -0,0 +1,410 @@ +package sdg + +import ( + "fmt" + "path/filepath" + "slices" + "strings" + "testing" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// runPorts runs L4 over testdata/dataflow and returns its vertex builder. +func runPorts(t *testing.T) (Input, *dispatch, *ports) { + t.Helper() + in, _, d := runDispatch(t) + return in, d, newPorts(in, d) +} + +func fin(i, of string) Vertex { return Vertex{Key: "@formal_in:" + i, Of: of, Kind: KindFormalIn} } +func fout(i, of string) Vertex { return Vertex{Key: "@formal_out" + i, Of: of, Kind: KindFormalOut} } +func ain(call, i, of, parent string) Vertex { + return Vertex{Key: call + "/actual_in:" + i, Of: of, Parent: parent, Kind: KindActualIn} +} +func aout(call, i, of, parent string) Vertex { + return Vertex{Key: call + "/actual_out" + i, Of: of, Parent: parent, Kind: KindActualOut} +} + +func checkVertices(t *testing.T, what string, got, want []Vertex) { + t.Helper() + if !slices.Equal(got, want) { + t.Errorf("%s:\n got %+v\n want %+v", what, got, want) + } +} + +func checkFormals(t *testing.T, p *ports, tg Target, wantIn, wantOut []Vertex) { + t.Helper() + in, out := p.formals(tg) + checkVertices(t, fmt.Sprint("formal_in of ", tg), in, wantIn) + checkVertices(t, fmt.Sprint("formal_out of ", tg), out, wantOut) +} + +func checkActuals(t *testing.T, p *ports, d *dispatch, unit dataflow.L3Key, call string, wantIn, wantOut []Vertex) { + t.Helper() + in, out := p.actuals(unit, call, d.targets(unit, call)) + checkVertices(t, "actual_in of "+unit.Signature+"@"+call, in, wantIn) + checkVertices(t, "actual_out of "+unit.Signature+"@"+call, out, wantOut) +} + +// TestVertices_FormalOrderAndOf: formal_in is receiver, declared params +// (unnamed or _ as _, i the declared index), captures, globals; formal_out +// is results by position (:, never user names), by-ref params in +// declaration order (receiver first), written captures, global writes. +func TestVertices_FormalOrderAndOf(t *testing.T) { + _, _, p := runPorts(t) + checkFormals(t, p, sdgTarget("multiT.Multi"), + []Vertex{fin("0", "s"), fin("1", "a"), fin("2", "_1")}, + []Vertex{fout(":0", ":0"), fout(":1", ":1")}) + // Mutated is p, q, s: declaration order puts s before q. + checkFormals(t, p, sdgTarget("mutate"), + []Vertex{fin("0", "p"), fin("1", "s"), fin("2", "n"), fin("3", "q")}, + []Vertex{fout(":0", "p"), fout(":1", "s"), fout(":2", "q")}) + checkFormals(t, p, sdgTarget("sqlStore.setDSN"), + []Vertex{fin("0", "s"), fin("1", "d")}, + []Vertex{fout("", "s")}) + // A transitively read global is a formal_in after the declared params. + checkFormals(t, p, sdgTarget("useConfig"), + []Vertex{fin("0", ":"+configSpell)}, + []Vertex{fout("", "")}) + checkFormals(t, p, sdgTarget("initConfig"), + []Vertex{fin("0", ":"+configSpell)}, + []Vertex{fout("", ":"+configSpell)}) + // A closure: captures after its params; the written capture is an out. + checkFormals(t, p, sdgTarget("CaptureSite.closure_1"), + []Vertex{fin("0", ":x")}, + []Vertex{fout(":0", ""), fout(":1", ":x")}) + checkFormals(t, p, sdgTarget("DirectLit.closure_1"), + []Vertex{fin("0", "a"), fin("1", ":y")}, + []Vertex{fout("", ":y")}) + // An interface method (no unit) and a library interface method: the + // unnamed receiver is _recv. + checkFormals(t, p, sdgTarget("Store.Get"), + []Vertex{fin("0", "_recv"), fin("1", "k")}, + []Vertex{fout(":0", ""), fout(":1", "_recv")}) + checkFormals(t, p, Target{External: "io.Writer.Write"}, + []Vertex{fin("0", "_recv"), fin("1", "p")}, + []Vertex{fout(":0", ":0"), fout(":1", ":1"), fout(":2", "_recv"), fout(":3", "p")}) +} + +// TestVertices_SingleOutIsBare: a callable with exactly one formal_out, and a +// call site with exactly one actual_out, use the bare key. A named result is +// , never its name. +func TestVertices_SingleOutIsBare(t *testing.T) { + _, d, p := runPorts(t) + checkFormals(t, p, sdgTarget("Named"), nil, []Vertex{fout("", "")}) + checkActuals(t, p, d, sdgKey("useConfig"), "122:9", + []Vertex{ain("122:9", "0", ":"+configSpell, "122:9")}, + []Vertex{aout("122:9", "", "", "122:9")}) +} + +// TestVertices_VariadicCollapses: excess actuals of a variadic call collapse +// onto the last formal (one actual_in), each result of f(g()) is its own +// actual, and both results of pair() collapse onto Variadic's xs. +func TestVertices_VariadicCollapses(t *testing.T) { + _, d, p := runPorts(t) + k := sdgKey("CallVariadic") + // xs is a used slice: a by-ref out after the result. + checkActuals(t, p, d, k, "195:7", + []Vertex{ain("195:7", "0", "xs", "195:7")}, + []Vertex{aout("195:7", ":0", "", "195:7"), aout("195:7", ":1", "xs", "195:7")}) + checkActuals(t, p, d, k, "196:7", + []Vertex{ain("196:7", "0", "xs", "196:7")}, + []Vertex{aout("196:7", ":0", "", "196:7"), aout("196:7", ":1", "xs", "196:7")}) + checkActuals(t, p, d, k, "197:7", + []Vertex{ain("197:7", "0", "a", "197:7"), ain("197:7", "1", "b", "197:7")}, + []Vertex{aout("197:7", "", "", "197:7")}) + // pair() has no params: no actual_in; two results. + checkActuals(t, p, d, k, "196:16", nil, + []Vertex{aout("196:16", ":0", ":0", "196:16"), aout("196:16", ":1", ":1", "196:16")}) +} + +// TestVertices_MethodValueBindsReceiver (A8): a method call binds its +// receiver at actual_in:0; a bound method value's function-value operand +// binds it too; a method expression's first argument binds it. The actual's +// of is the static callee's formal (the interface method's _recv, not an +// implementation's name), or the first target's without one; a function +// value that may be a plain function or a bound method keeps the receiver +// slot at 0. +func TestVertices_MethodValueBindsReceiver(t *testing.T) { + _, d, p := runPorts(t) + recvOut := []Vertex{aout("204:9", ":0", "", "204:9"), aout("204:9", ":1", "s", "204:9")} + checkActuals(t, p, d, sdgKey("MethodValue"), "204:9", + []Vertex{ain("204:9", "0", "s", "204:9"), ain("204:9", "1", "k", "204:9")}, recvOut) + checkActuals(t, p, d, sdgKey("MethodExpr"), "209:9", + []Vertex{ain("209:9", "0", "s", "209:9"), ain("209:9", "1", "k", "209:9")}, + []Vertex{aout("209:9", ":0", "", "209:9"), aout("209:9", ":1", "s", "209:9")}) + checkActuals(t, p, d, sdgKey("UseStore"), "34:9", + []Vertex{ain("34:9", "0", "_recv", "34:9"), ain("34:9", "1", "k", "34:9")}, + []Vertex{aout("34:9", ":0", "", "34:9"), aout("34:9", ":1", "_recv", "34:9")}) + // Targets bytes.Buffer.Write (sorted first) and io.Writer.Write (static). + checkActuals(t, p, d, sdgKey("Buffered"), "93:2", + []Vertex{ain("93:2", "0", "_recv", "93:2"), ain("93:2", "1", "p", "93:2")}, + []Vertex{aout("93:2", ":0", ":0", "93:2"), aout("93:2", ":1", ":1", "93:2"), + aout("93:2", ":2", "_recv", "93:2"), aout("93:2", ":3", "p", "93:2")}) + // A variadic method expression passing only the receiver: v binds the + // receiver and nothing reaches xs (no actual for it, in or out). + checkActuals(t, p, d, sdgKey("MethodExprVariadic"), "287:9", + []Vertex{ain("287:9", "0", "_recv", "287:9")}, + []Vertex{aout("287:9", "", "", "287:9")}) + // sqlStore.Get (offset 1) and upper (offset 0): f binds 0 for the method. + checkActuals(t, p, d, sdgKey("MixedValue"), "241:9", + []Vertex{ain("241:9", "0", "s", "241:9"), ain("241:9", "1", "k", "241:9")}, + []Vertex{aout("241:9", ":0", "", "241:9"), aout("241:9", ":1", "s", "241:9")}) +} + +// TestVertices_BindsEachTargetFormal: a site binds each actual to the +// formal of every target that has it, by position (receiver, params, +// results, by-ref outs) or by `of` (captures, globals) — the param_in / +// param_out endpoints. An actual a target lacks binds nothing there. +func TestVertices_BindsEachTargetFormal(t *testing.T) { + _, d, p := runPorts(t) + k := sdgKey("MixedValue") + s := p.callSite(k, "241:9", d.targets(k, "241:9")) + get, up := sdgTarget("sqlStore.Get"), sdgTarget("upper") + wantIn := []portBind{ + {"241:9/actual_in:0", get, "@formal_in:0"}, + {"241:9/actual_in:1", get, "@formal_in:1"}, + {"241:9/actual_in:1", up, "@formal_in:0"}, + } + if !slices.Equal(s.BindIn, wantIn) { + t.Errorf("MixedValue BindIn = %v, want %v", s.BindIn, wantIn) + } + // sqlStore.Get's outs are and s (by-ref); upper's only . + wantOut := []portBind{ + {"241:9/actual_out:0", get, "@formal_out:0"}, + {"241:9/actual_out:0", up, "@formal_out"}, + {"241:9/actual_out:1", get, "@formal_out:1"}, + } + if !slices.Equal(s.BindOut, wantOut) { + t.Errorf("MixedValue BindOut = %v, want %v", s.BindOut, wantOut) + } + // A transitively read global binds by `of`; a directly called literal's + // capture binds to the literal's capture formal. + k = sdgKey("useConfig") + s = p.callSite(k, "122:9", d.targets(k, "122:9")) + if want := []portBind{{"122:9/actual_in:0", sdgTarget("readConfig"), "@formal_in:0"}}; !slices.Equal(s.BindIn, want) { + t.Errorf("useConfig BindIn = %v, want %v", s.BindIn, want) + } + k = sdgKey("DirectLit") + s = p.callSite(k, "226:2", d.targets(k, "226:2")) + lit := sdgTarget("DirectLit.closure_1") + if want := []portBind{{"226:2/actual_in:0", lit, "@formal_in:0"}, {"226:2/actual_in:1", lit, "@formal_in:1"}}; !slices.Equal(s.BindIn, want) { + t.Errorf("DirectLit BindIn = %v, want %v", s.BindIn, want) + } + if want := []portBind{{"226:2/actual_out", lit, "@formal_out"}}; !slices.Equal(s.BindOut, want) { + t.Errorf("DirectLit BindOut = %v, want %v", s.BindOut, want) + } +} + +// TestVertices_NoVTABindsDirectLiteral: L2 resolves no FuncLit callee, so +// without VTA only the syntax names a literal called in place; its declared +// args and captures still bind at the call. +func TestVertices_NoVTABindsDirectLiteral(t *testing.T) { + in := loadInput(t, "dataflow") + defer func(f func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph) { vtaCallGraph = f }(vtaCallGraph) + vtaCallGraph = func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph { panic("vta boom") } + _, d := run(in) + p := newPorts(in, d) + k := sdgKey("DirectLit") + if ts := d.targets(k, "226:2"); len(ts) != 0 { + t.Fatalf("no_vta targets(DirectLit, 226:2) = %v: the test no longer covers the gap", ts) + } + in1, out1 := p.actuals(k, "226:2", p.siteTargets(k, "226:2")) + checkVertices(t, "no_vta actual_in", in1, []Vertex{ain("226:2", "0", "a", "226:2"), ain("226:2", "1", ":y", "226:2")}) + checkVertices(t, "no_vta actual_out", out1, []Vertex{aout("226:2", "", ":y", "226:2")}) +} + +// TestVertices_DirectlyCalledLiteralBindsAtCall (A6): a literal called in +// place binds its captures at that call node after the declared args, and +// has no capture site; so does a deferred one and one whose call is a +// satellite whose statement shares its position. +func TestVertices_DirectlyCalledLiteralBindsAtCall(t *testing.T) { + _, d, p := runPorts(t) + checkActuals(t, p, d, sdgKey("DirectLit"), "226:2", + []Vertex{ain("226:2", "0", "a", "226:2"), ain("226:2", "1", ":y", "226:2")}, + []Vertex{aout("226:2", "", ":y", "226:2")}) + checkActuals(t, p, d, sdgKey("ChainLit"), "259:2", + []Vertex{ain("259:2", "0", ":x", "259:2")}, + []Vertex{aout("259:2", "", "", "259:2")}) + flowKey := func(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("flow", "flow.go"), Signature: "example.com/dataflow/flow." + name} + } + // defer func() { err = wrapErr(err) }() at 557:8: the deferred call node. + checkActuals(t, p, d, flowKey("DeferWrap"), "557:8", + []Vertex{ain("557:8", "0", ":err", "557:8")}, + []Vertex{aout("557:8", "", ":err", "557:8")}) + for _, k := range []dataflow.L3Key{sdgKey("DirectLit"), sdgKey("ChainLit"), flowKey("DeferWrap")} { + for _, v := range p.vertices(k) { + if v.Kind == KindActualIn || v.Kind == KindActualOut { + if _, ok := d.sites[siteKey{k, strings.SplitN(v.Key, "/actual", 2)[0]}]; !ok { + t.Errorf("%s: %s is a capture site, want only call-node actuals", k.Signature, v.Key) + } + } + } + } +} + +// TestVertices_CaptureSite (A6): a literal not called in place gets +// capture-site actuals keyed at its line:col, parented on the innermost CFG +// node holding it: the statement storing it, or the call it is passed to. +func TestVertices_CaptureSite(t *testing.T) { + _, _, p := runPorts(t) + got := capturesOf(p.vertices(sdgKey("CaptureSite"))) + checkVertices(t, "CaptureSite capture site", got, []Vertex{ + ain("215:7", "0", ":x", "215:2"), + aout("215:7", "", ":x", "215:2"), + }) + got = capturesOf(p.vertices(dataflow.L3Key{Path: filepath.Join("flow", "flow.go"), Signature: "example.com/dataflow/flow.OnceDo"})) + checkVertices(t, "OnceDo capture site", got, []Vertex{ + ain("588:10", "0", ":cfg", "588:2"), + aout("588:10", "", ":cfg", "588:2"), + }) + // Passed to a satellite call: the assignment, not the call, hosts it; + // the literal only reads v, so it has no actual_out. + got = capturesOf(p.vertices(sdgKey("CaptureArg"))) + checkVertices(t, "CaptureArg capture site", got, []Vertex{ain("267:13", "0", ":v", "267:2")}) + // A literal after a return is dead: its host is pruned. + if got := capturesOf(p.vertices(sdgKey("DeadLit"))); len(got) != 0 { + t.Errorf("DeadLit capture site = %v, want none", got) + } +} + +// capturesOf keeps the vertices of vs that are not at a call key (l:c or +// l:c/n followed by /actual_*) of a call: the capture-site actuals, whose +// prefix is a literal's key, told apart by their : of. +func capturesOf(vs []Vertex) []Vertex { + var out []Vertex + for _, v := range vs { + if (v.Kind == KindActualIn || v.Kind == KindActualOut) && strings.HasPrefix(v.Of, ":") && v.Parent != strings.SplitN(v.Key, "/actual", 2)[0] { + out = append(out, v) + } + } + return out +} + +// TestVertices_PrunedAndUnresolvedCallsHaveNone: a call node the CFG pruned +// and a call with no target get no vertices. +func TestVertices_PrunedAndUnresolvedCallsHaveNone(t *testing.T) { + _, d, p := runPorts(t) + if ts := d.targets(sdgKey("PrunedCall"), "247:9"); len(ts) == 0 { + t.Fatalf("PrunedCall@247:9 has no targets: the fixture no longer tests pruning") + } + for _, c := range []struct { + k dataflow.L3Key + call string + }{{sdgKey("PrunedCall"), "247:9"}, {sdgKey("callUnknown"), "252:9"}} { + in, out := p.actuals(c.k, c.call, d.targets(c.k, c.call)) + if len(in)+len(out) != 0 { + t.Errorf("%s@%s: actuals %v %v, want none", c.k.Signature, c.call, in, out) + } + for _, v := range p.vertices(c.k) { + if strings.HasPrefix(v.Key, c.call+"/") { + t.Errorf("%s has vertex %s", c.k.Signature, v.Key) + } + } + } +} + +// TestVertices_CaptureSiteKeyNeverCollides: for every callable of sdg and +// flow, vertex keys are unique, sorted, and never a body key (the kept nodes +// and every L1 call key); the key allocator skips taken keys like +// BodyNodes.add. +func TestVertices_CaptureSiteKeyNeverCollides(t *testing.T) { + in, _, p := runPorts(t) + n := 0 + for _, u := range in.Units { + dir := strings.SplitN(u.Key.Path, string(filepath.Separator), 2)[0] + if dir != "sdg" && dir != "flow" { + continue + } + body := map[string]bool{} + for _, nd := range in.L3[u.Key].Nodes { + body[nd.Key] = true + } + for _, c := range u.Calls { + body[c.Key] = true + } + vs := p.vertices(u.Key) + seen := map[string]bool{} + for i, v := range vs { + if body[v.Key] || seen[v.Key] { + t.Errorf("%s: vertex key %s collides", u.Key.Signature, v.Key) + } + seen[v.Key] = true + if i > 0 && vs[i-1].Key >= v.Key { + t.Errorf("%s: vertices not sorted at %s", u.Key.Signature, v.Key) + } + n++ + } + } + if n == 0 { + t.Fatal("no vertices on sdg and flow") + } + // The capture site takes the next free key the way BodyNodes does. + u := unitOf(t, in, sdgKey("CaptureSite")) + got := p.captureSites(u, map[string]bool{"215:7": true}) + if len(got) == 0 || !strings.HasPrefix(got[0].Key, "215:7/2/") { + t.Errorf("capture site with 215:7 taken = %v, want keyed 215:7/2", got) + } + taken := map[string]bool{"259:2": true, "259:2/2": true} + if got := dataflow.NextFreeKey("259:2", taken); got != "259:2/3" || !taken[got] { + t.Errorf("dataflow.NextFreeKey(259:2) = %s, want 259:2/3 (and taken)", got) + } + if got := dataflow.NextFreeKey("7:1", taken); got != "7:1" { + t.Errorf("dataflow.NextFreeKey(7:1) = %s, want 7:1", got) + } +} + +func unitOf(t *testing.T, in Input, k dataflow.L3Key) dataflow.Unit { + t.Helper() + for _, u := range in.Units { + if u.Key == k { + return u + } + } + t.Fatalf("no unit %v", k) + return dataflow.Unit{} +} + +// TestVertices_Deterministic: two builders over two loads agree. +func TestVertices_Deterministic(t *testing.T) { + in1, _, p1 := runPorts(t) + _, _, p2 := runPorts(t) + for _, u := range in1.Units { + if a, b := p1.vertices(u.Key), p2.vertices(u.Key); !slices.Equal(a, b) { + t.Errorf("%v: vertices differ across runs:\n%v\n%v", u.Key, a, b) + } + } +} + +// TestCallShape_NoSignatureNeverBindsMethodExpression: without type info (a +// package with type errors), a bound method value f(x) on a one-param method +// passes 2 values, the declared param count plus one. That must not read as +// a method expression binding x to the receiver: the function value binds +// the receiver, x binds the declared param. +func TestCallShape_NoSignatureNeverBindsMethodExpression(t *testing.T) { + fs := &formalSet{recv: true, params: 1} + noSig := callShape{values: 2, nParams: 2, results: -1, noSig: true} + got := noSig.bind(fs) + if got[slotRecv] != 0 { + t.Errorf("no signature: receiver not bound through the function value: %v", got) + } + if _, ok := got[0]; ok && got[0] == 0 { + t.Errorf("no signature: value 0 bound to the receiver (read as a method expression): %v", got) + } + // With a known signature the same counts are a method expression. + expr := callShape{values: 2, nParams: 2, results: 1} + if got := expr.bind(fs); got[0] != 0 || got[1] != 1 { + t.Errorf("method expression: want value 0 → receiver, value 1 → param, got %v", got) + } + // Without a signature, a selection go/types recorded as MethodExpr is + // still a method expression. + exprNoSig := callShape{values: 2, nParams: 2, results: -1, noSig: true, methodExpr: true} + if got := exprNoSig.bind(fs); got[0] != 0 || got[1] != 1 { + t.Errorf("method expression without a signature: want value 0 → receiver, value 1 → param, got %v", got) + } +} diff --git a/internal/sdg/vta.go b/internal/sdg/vta.go new file mode 100644 index 0000000..ee0b0d0 --- /dev/null +++ b/internal/sdg/vta.go @@ -0,0 +1,379 @@ +package sdg + +import ( + "cmp" + "go/types" + "slices" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" +) + +// siteKey names one call site: its unit and its L1/L3 call key. +type siteKey struct { + unit dataflow.L3Key + call string +} + +// l2Edge is the identity of an L2 call_graph edge: L2 dedups per (declaring +// file, source signature, target signature). +type l2Edge struct{ path, src, dst string } + +// dispatch holds every call site's targets (spec §4.8): the static callee L2 +// resolved ∪ the callees VTA found ∪, for an interface call, the interface +// method (D7). Targets are mapped to the L3 join's keys — a unit, an +// interface method or a bodiless declaration, or an external signature — and +// deduplicated after mapping, so a generic instance and its origin, or a +// wrapper and the method it wraps, are one target. A callee that maps to +// nothing (unmapped_ssa, already reported) is not a target. +type dispatch struct { + sites map[siteKey][]Target // each sorted by compareTarget, no duplicates + static map[siteKey]Target // the L2 static callee of a site, when it maps to a target + byVTA map[siteTarget]bool // the (site, target) pairs VTA produced: the call_graph edge candidates + bySig map[string]dataflow.L3Key // symbol-table callable of each signature (init excepted) + l2 map[l2Edge]bool + l2Ext map[string]schema.GoExternalCallee // in.Externals, read-only + newExt map[string]schema.GoExternalCallee // externals L2 never saw +} + +// siteTarget is one target of one call site. +type siteTarget struct { + site siteKey + t Target +} + +// compareTarget orders targets by Callable (Path, Signature), then External. +func compareTarget(a, b Target) int { + return cmp.Or( + cmp.Compare(a.Callable.Path, b.Callable.Path), + cmp.Compare(a.Callable.Signature, b.Callable.Signature), + cmp.Compare(a.External, b.External), + ) +} + +// newDispatch seeds every unit's call sites with their L2 static callee: the +// callee signature L2 backfilled into the symbol table, as an external when +// L2 tagged it with an import path, else as the symbol-table callable of that +// signature. It needs no SSA, so a run that degrades to no_ssa or no_vta +// keeps the static targets (go/types dispatch only). +func newDispatch(in Input) *dispatch { + d := &dispatch{ + sites: map[siteKey][]Target{}, + static: map[siteKey]Target{}, + byVTA: map[siteTarget]bool{}, + l2: make(map[l2Edge]bool, len(in.CallGraph)), + l2Ext: in.Externals, + newExt: map[string]schema.GoExternalCallee{}, + } + for _, e := range in.CallGraph { + d.l2[l2Edge{e.SourcePath, e.Source, e.Target}] = true + } + + callables := map[dataflow.L3Key]schema.GoCallable{} + bySig := map[string]dataflow.L3Key{} + keep := func(sig string, k dataflow.L3Key) { + // A signature names one callable, except init (never a callee). + if old, ok := bySig[sig]; !ok || k.Path < old.Path { + bySig[sig] = k + } + } + var walk func(c schema.GoCallable) + walk = func(c schema.GoCallable) { + callables[dataflow.L3Key{Path: c.Path, Signature: c.Signature}] = c + for _, ic := range c.InnerCallables { + walk(ic) + } + } + for path, f := range in.SymbolTable { + for _, c := range f.Functions { + keep(c.Signature, dataflow.L3Key{Path: c.Path, Signature: c.Signature}) + walk(c) + } + for _, t := range f.Types { + for _, c := range t.Methods { + if c.Path == "" { // an interface method, keyed by its interface's file + keep(c.Signature, dataflow.L3Key{Path: path, Signature: c.Signature}) + continue + } + keep(c.Signature, dataflow.L3Key{Path: c.Path, Signature: c.Signature}) + walk(c) + } + } + } + + d.bySig = bySig + for _, u := range in.Units { + c, ok := callables[u.Key] + if !ok || len(c.CallSites) != len(u.Calls) { // Unit.Calls is built index by index from CallSites + continue + } + for i, cs := range c.CallSites { + if cs.CalleeSignature == nil { + continue + } + sig := *cs.CalleeSignature + site := siteKey{u.Key, u.Calls[i].Key} + t := Target{External: sig} + if cs.CalleeImportPath == "" { + k, ok := bySig[sig] + if !ok { + continue + } + t = Target{Callable: k} + } + d.add(site, t) + d.static[site] = t + } + } + return d +} + +// targets returns the dispatch targets of the call site callKey of siteUnit, +// sorted by Target (Callable Path, Signature, then External): the static L2 +// callee ∪ the VTA callees, and for an interface call always the interface +// method (D7). The slice is shared: callers must not modify it. +func (d *dispatch) targets(siteUnit dataflow.L3Key, callKey string) []Target { + return d.sites[siteKey{siteUnit, callKey}] +} + +func (d *dispatch) add(site siteKey, t Target) { + ts := d.sites[site] + i, found := slices.BinarySearchFunc(ts, t, compareTarget) + if !found { + d.sites[site] = slices.Insert(ts, i, t) + } +} + +// addInvokes binds every interface call of a body-joined project function to +// its interface method (D7), whether or not L2 resolved it and whether or not +// VTA finds a concrete target. It reads instructions only, so it is safe +// before VTA. +func (d *dispatch) addInvokes(j *join, funcs map[*ssa.Function]bool) { + for fn := range funcs { + forCalls(fn, func(call ssa.CallInstruction) { + cc := call.Common() + if !cc.IsInvoke() { + return + } + unit, key, ok := j.callSite(call) + if !ok { + return + } + if t, ok := d.methodTarget(j, cc.Method); ok { + d.add(siteKey{unit, key}, t) + } + }) + } +} + +// methodTarget is the target of the interface method m: the symbol table's +// interface method for a project interface, else the external of a library +// one. A method of a universe type (error.Error) or of an interface the +// symbol table does not record (an interface literal) has no target. +func (d *dispatch) methodTarget(j *join, m *types.Func) (Target, bool) { + if m == nil || m.Pkg() == nil { + return Target{}, false + } + if j.isLibrary(m) { + sig := semantic_analysis.CalleeSignatureOf(m) + d.external(m, sig) + return Target{External: sig}, true + } + if k, ok := declKey(j.fset, j.projectDir, m.Origin()); ok && j.iface[k] { + return Target{Callable: k}, true + } + return Target{}, false +} + +// addVTA adds VTA's callees: every edge whose site joins a call node and +// whose callee maps to a target. A callee that is a synthetic wrapper — a +// $bound or $thunk, a promotion, pointer or instantiation wrapper: SSA code +// with no syntax of its own — calls what it wraps at a synthetic call site, +// which joins no call node; for an interface method's wrapper (a method value +// of an interface, a struct embedding an interface) VTA's concrete callees +// sit only there. So the wrapper's own VTA callees, transitively through +// nested wrappers, are targets of the original site too. +func (d *dispatch) addVTA(j *join, cg *callgraph.Graph) { + for _, node := range cg.Nodes { + for _, e := range node.Out { + if e.Site == nil || e.Callee == nil || e.Callee.Func == nil { + continue + } + unit, key, ok := j.callSite(e.Site) + if !ok { + continue + } + site := siteKey{unit, key} + for _, fn := range throughWrappers(cg, e.Callee.Func) { + if t, ok := d.calleeTarget(j, fn); ok { + d.add(site, t) + if !isIfaceMethod(fn) { + d.byVTA[siteTarget{site, t}] = true + } + } + } + } + } +} + +// throughWrappers returns fn and, when fn is a synthetic wrapper, the VTA +// callees of its calls, transitively through nested wrappers (each function +// once, so a cycle of wrappers ends). The order is the graph's; d.add sorts. +func throughWrappers(cg *callgraph.Graph, fn *ssa.Function) []*ssa.Function { + if !isWrapper(fn) { + return []*ssa.Function{fn} + } + seen := map[*ssa.Function]bool{fn: true} + out := []*ssa.Function{fn} + for work := []*ssa.Function{fn}; len(work) > 0; { + w := work[len(work)-1] + work = work[:len(work)-1] + node := cg.Nodes[w] + if node == nil { + continue + } + for _, e := range node.Out { + if e.Callee == nil || e.Callee.Func == nil || seen[e.Callee.Func] { + continue + } + g := e.Callee.Func + seen[g] = true + out = append(out, g) + if isWrapper(g) { + work = append(work, g) + } + } + } + return out +} + +// isIfaceMethod reports whether fn stands for an interface method (a $bound +// or $thunk of one): no concrete callee, so VTA found no edge to it — D7 +// binds the interface method itself (addInvokes). +func isIfaceMethod(fn *ssa.Function) bool { + obj, ok := fn.Object().(*types.Func) + if !ok { + return false + } + recv := obj.Type().(*types.Signature).Recv() + return recv != nil && types.IsInterface(recv.Type()) +} + +// isWrapper reports whether fn is SSA-synthesized code with no syntax of its +// own: a $bound, $thunk, promotion or pointer wrapper. A range-over-func +// yield and an instance keep syntax (their code is the unit's own); an +// instantiation wrapper keeps its origin's syntax but is a synthetic call of +// the origin, which the join already maps. +func isWrapper(fn *ssa.Function) bool { + return fn.Synthetic != "" && fn.Syntax() == nil && fn.Synthetic != synthPackageInit +} + +// calleeTarget maps an SSA callee to its target: the unit it joins (by body +// or as a wrapper; an instance joins its origin), an interface method, a +// bodiless declaration, or a library function's external. An unmapped project +// function is no target. +func (d *dispatch) calleeTarget(j *join, fn *ssa.Function) (Target, bool) { + if k, ok := j.fnUnit(fn); ok { + return Target{Callable: k}, true + } + if k, ok := j.ifaceMethod(fn); ok { + return Target{Callable: k}, true + } + if k, ok := j.bodilessDecl(fn); ok { + return Target{Callable: k}, true + } + if sig, ok := j.externalOf(fn); ok { + d.external(fn.Object().(*types.Func), sig) + return Target{External: sig}, true + } + return Target{}, false +} + +// external records the library callee obj (full signature sig) when L2 did +// not, with L2's own conversion, so its entry is the one L2 would have made. +func (d *dispatch) external(obj *types.Func, sig string) { + if _, ok := d.l2Ext[sig]; ok { + return + } + if _, ok := d.newExt[sig]; ok { + return + } + d.newExt[sig] = semantic_analysis.ExternalCallee(obj, sig) +} + +// result is the Result of the dispatch: one call_graph edge per (caller, +// target) pair VTA produced (byVTA) that L2 lacks, prov ["vta"], weight 1 — +// a target bound otherwise (the D7 interface method, L2's static callee) is +// no edge, though it keeps its param edges — Source the caller's +// signature and SourcePath its declaring file, as L2 has them (closures +// included), Target the callee signature (the full external signature for an +// external) and TargetPath the callee's declaring file (empty for an +// external), since a closure of an init has a per-file signature — sorted by +// (SourcePath, Source, Target, TargetPath); and the externals only VTA +// reached. +func (d *dispatch) result(deg []Degraded) Result { + type edgeID struct{ srcPath, src, dst, dstPath string } + seen := map[edgeID]bool{} + var ids []edgeID + for site, ts := range d.sites { + for _, t := range ts { + if !d.byVTA[siteTarget{site, t}] { + continue + } + id := edgeID{site.unit.Path, site.unit.Signature, t.External, ""} + if t.External == "" { + id.dst, id.dstPath = t.Callable.Signature, t.Callable.Path + } + if d.l2[l2Edge{id.srcPath, id.src, id.dst}] || seen[id] { + continue + } + seen[id] = true + ids = append(ids, id) + } + } + slices.SortFunc(ids, func(a, b edgeID) int { + return cmp.Or(cmp.Compare(a.srcPath, b.srcPath), cmp.Compare(a.src, b.src), + cmp.Compare(a.dst, b.dst), cmp.Compare(a.dstPath, b.dstPath)) + }) + res := Result{Degraded: deg} + for _, id := range ids { + res.CallEdges = append(res.CallEdges, schema.GoCallEdge{ + Source: id.src, + Target: id.dst, + Type: "CALL_DEP", + Weight: 1, + Provenance: []string{"vta"}, + Tags: map[string]string{}, + SourcePath: id.srcPath, + TargetPath: id.dstPath, + }) + } + if len(d.newExt) > 0 { + res.NewExternals = d.newExt + } + return res +} + +// ApplyTo merges the L4 call-graph additions into app (spec §4.8): the VTA +// edges into app.CallGraph and the VTA-only externals into app.Externals, so +// every new edge's target resolves. Both are additions: -a 3 ⊆ -a 4. +// CallEdges holds only pairs the L2 resolver did not find (result skips +// every d.l2 edge), so MergeEdges never meets an existing identity and +// appends: an L2 edge VTA also finds keeps its L2 prov and weight. +func (r *Result) ApplyTo(app *schema.GoApplication) { + if len(r.CallEdges) > 0 { + app.CallGraph = semantic_analysis.MergeEdges(app.CallGraph, r.CallEdges) + } + if len(r.NewExternals) > 0 && app.Externals == nil { + app.Externals = make(map[string]schema.GoExternalCallee, len(r.NewExternals)) + } + for sig, ext := range r.NewExternals { + if _, ok := app.Externals[sig]; !ok { + app.Externals[sig] = ext + } + } +} diff --git a/internal/sdg/vta_test.go b/internal/sdg/vta_test.go new file mode 100644 index 0000000..ddec8e4 --- /dev/null +++ b/internal/sdg/vta_test.go @@ -0,0 +1,327 @@ +package sdg + +import ( + "cmp" + "path/filepath" + "slices" + "testing" + + "golang.org/x/tools/go/callgraph" + "golang.org/x/tools/go/ssa" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" +) + +const sdgPkg = "example.com/dataflow/sdg" + +var sdgFile = filepath.Join("sdg", "sdg.go") + +// sdgKey is the L3Key of a callable declared in testdata/dataflow/sdg/sdg.go. +func sdgKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: sdgFile, Signature: sdgPkg + "." + name} +} + +func sdgTarget(name string) Target { return Target{Callable: sdgKey(name)} } + +// runDispatch runs L4 over testdata/dataflow and returns its result and the +// call-site dispatch. +func runDispatch(t *testing.T) (Input, Result, *dispatch) { + t.Helper() + in := loadInput(t, "dataflow") + res, d := run(in) + if len(res.Degraded) != 0 { + t.Fatalf("degraded = %+v, want none", res.Degraded) + } + return in, res, d +} + +// hasEdge reports whether edges holds src → dst (src declared in sdg.go) +// with prov ["vta"] and weight 1. +func hasVTAEdge(edges []schema.GoCallEdge, src, dst string) bool { + for _, e := range edges { + if e.Source == src && e.Target == dst && e.SourcePath == sdgFile { + return slices.Equal(e.Provenance, []string{"vta"}) && e.Weight == 1 + } + } + return false +} + +func hasEdgeTo(edges []schema.GoCallEdge, src, dst string) bool { + for _, e := range edges { + if e.Source == src && e.Target == dst { + return true + } + } + return false +} + +// TestVTA_InterfaceTargets: UseStore's s.Get(k) dispatches to the interface +// method (D7) and to both implementations; memStore's pointer wrapper (from +// UseStore(&memStore{})) maps to memStore.Get. Only the concrete targets are +// new call_graph edges: L2 already has UseStore → Store.Get. +func TestVTA_InterfaceTargets(t *testing.T) { + _, res, d := runDispatch(t) + got := d.targets(sdgKey("UseStore"), "34:9") + want := []Target{sdgTarget("Store.Get"), sdgTarget("memStore.Get"), sdgTarget("sqlStore.Get")} + if !slices.Equal(got, want) { + t.Fatalf("targets(UseStore, 34:9) = %v, want %v", got, want) + } + src := sdgPkg + ".UseStore" + for _, dst := range []string{sdgPkg + ".memStore.Get", sdgPkg + ".sqlStore.Get"} { + if !hasVTAEdge(res.CallEdges, src, dst) { + t.Errorf("CallEdges lack %s → %s (prov [vta], weight 1): %+v", src, dst, res.CallEdges) + } + } + if hasEdgeTo(res.CallEdges, src, sdgPkg+".Store.Get") { + t.Errorf("CallEdges repeat the L2 edge %s → Store.Get", src) + } +} + +// TestVTA_FuncValueTargets: CallFuncValue's f(1) has no static callee; VTA +// resolves it to double, the function PassDouble passes. +func TestVTA_FuncValueTargets(t *testing.T) { + _, res, d := runDispatch(t) + got := d.targets(sdgKey("CallFuncValue"), "51:9") + if want := []Target{sdgTarget("double")}; !slices.Equal(got, want) { + t.Fatalf("targets(CallFuncValue, 51:9) = %v, want %v", got, want) + } + if !hasVTAEdge(res.CallEdges, sdgPkg+".CallFuncValue", sdgPkg+".double") { + t.Errorf("CallEdges lack CallFuncValue → double: %+v", res.CallEdges) + } + // A static call keeps its one L2 target. + got = d.targets(sdgKey("PassDouble"), "56:9") + if want := []Target{sdgTarget("CallFuncValue")}; !slices.Equal(got, want) { + t.Errorf("targets(PassDouble, 56:9) = %v, want %v", got, want) + } +} + +// TestVTA_PromotedAndWrapperTargetsMap (Review Focus 3): VTA returns the +// synthetic wrappers — Outer's promotion wrapper and *Outer's +// promotion-and-indirection wrapper for o.M(), *memStore's pointer wrapper for +// s.Get(k) — and each maps to the declared method, not to nothing. +func TestVTA_PromotedAndWrapperTargetsMap(t *testing.T) { + in := loadInput(t, "dataflow") + prog, pkgs, _ := buildSSA(in) + _, all, err := funcSet(prog, pkgs) + if err != nil { + t.Fatal(err) + } + cg, err := runVTA(all) + if err != nil { + t.Fatal(err) + } + j := newJoin(in) + raw := func(unit dataflow.L3Key, key string) []*ssa.Function { + var out []*ssa.Function + for _, node := range cg.Nodes { + for _, e := range node.Out { + if u, k, ok := j.callSite(e.Site); ok && u == unit && k == key { + out = append(out, e.Callee.Func) + } + } + } + return out + } + for _, c := range []struct { + unit dataflow.L3Key + key string + wrappers int + want dataflow.L3Key + }{ + {sdgKey("CallPromoted"), "80:9", 2, sdgKey("Inner.M")}, + {sdgKey("UseStore"), "34:9", 1, sdgKey("memStore.Get")}, + } { + n := 0 + for _, fn := range raw(c.unit, c.key) { + if fn.Synthetic == "" { + continue + } + n++ + if k, ok := j.fnUnit(fn); !ok || k != c.want { + t.Errorf("wrapper %s [%s] maps to %v, %v; want %v", fn, fn.Synthetic, k, ok, c.want) + } + } + if n != c.wrappers { + t.Errorf("%v %s: %d wrapper callees, want %d (callees %v)", c.unit, c.key, n, c.wrappers, raw(c.unit, c.key)) + } + } + + _, res, d := runDispatch(t) + got := d.targets(sdgKey("CallPromoted"), "80:9") + if want := []Target{sdgTarget("Inner.M"), sdgTarget("Mer.M")}; !slices.Equal(got, want) { + t.Fatalf("targets(CallPromoted, 80:9) = %v, want %v", got, want) + } + if !hasVTAEdge(res.CallEdges, sdgPkg+".CallPromoted", sdgPkg+".Inner.M") { + t.Errorf("CallEdges lack CallPromoted → Inner.M: %+v", res.CallEdges) + } +} + +// TestVTA_LibraryTargetBecomesExternal (A19): Buffered's w.Write(p) reaches +// (*bytes.Buffer).Write only through VTA. The target is the external +// bytes.Buffer.Write, recorded in NewExternals with its receiver, beside the +// interface method io.Writer.Write (D7), which L2 already recorded. +func TestVTA_LibraryTargetBecomesExternal(t *testing.T) { + in, res, d := runDispatch(t) + got := d.targets(sdgKey("Buffered"), "93:2") + if want := []Target{{External: "bytes.Buffer.Write"}, {External: "io.Writer.Write"}}; !slices.Equal(got, want) { + t.Fatalf("targets(Buffered, 93:2) = %v, want %v", got, want) + } + if _, ok := in.Externals["bytes.Buffer.Write"]; ok { + t.Fatal("L2 already records bytes.Buffer.Write; the fixture no longer pins a VTA-only external") + } + if _, ok := in.Externals["io.Writer.Write"]; !ok { + t.Fatal("L2 does not record io.Writer.Write") + } + ext, ok := res.NewExternals["bytes.Buffer.Write"] + if !ok { + t.Fatalf("NewExternals lack bytes.Buffer.Write: %v", res.NewExternals) + } + if ext.ImportPath != "bytes" || ext.Sig != "Buffer.Write" || ext.Name != "Write" || ext.ReturnType != "(int, error)" { + t.Errorf("bytes.Buffer.Write = %+v", ext) + } + if len(ext.Params) != 1 || ext.Params[0].Name != "p" || ext.Params[0].Type != "[]byte" { + t.Errorf("bytes.Buffer.Write params = %+v, want [p []byte]", ext.Params) + } + if ext.Receiver == nil || ext.Receiver.Name != "b" || ext.Receiver.Type != "*bytes.Buffer" { + t.Errorf("bytes.Buffer.Write receiver = %+v, want b *bytes.Buffer", ext.Receiver) + } + if _, ok := res.NewExternals["io.Writer.Write"]; ok { + t.Error("NewExternals repeat the L2 external io.Writer.Write") + } + for sig := range res.NewExternals { + if _, ok := in.Externals[sig]; ok { + t.Errorf("NewExternals repeat the L2 external %s", sig) + } + } + if !hasVTAEdge(res.CallEdges, sdgPkg+".Buffered", "bytes.Buffer.Write") { + t.Errorf("CallEdges lack Buffered → bytes.Buffer.Write: %+v", res.CallEdges) + } +} + +// TestVTA_EdgesNewSortedAndResolvable: every VTA edge is new (not an L2 +// (caller, callee) pair), the list is sorted and duplicate-free, its source +// is a unit, and its target is a symbol-table callable or an external +// (L2's or a new one). +func TestVTA_EdgesNewSortedAndResolvable(t *testing.T) { + in, res, _ := runDispatch(t) + if len(res.CallEdges) == 0 { + t.Fatal("no VTA edges") + } + type key struct{ path, src, dst, dstPath string } + l2 := map[key]bool{} + for _, e := range in.CallGraph { + l2[key{e.SourcePath, e.Source, e.Target, ""}] = true + } + units := map[dataflow.L3Key]bool{} + for _, u := range in.Units { + units[u.Key] = true + } + sigs := map[string]bool{} // every callable the emitter indexes, closures included + var walk func(c schema.GoCallable) + walk = func(c schema.GoCallable) { + sigs[c.Signature] = true + for _, ic := range c.InnerCallables { + walk(ic) + } + } + for _, f := range in.SymbolTable { + for _, c := range f.Functions { + walk(c) + } + for _, ty := range f.Types { + for _, c := range ty.Methods { + walk(c) + } + } + } + for i, e := range res.CallEdges { + k := key{e.SourcePath, e.Source, e.Target, e.TargetPath} + if l2[key{k.path, k.src, k.dst, ""}] { + t.Errorf("edge %+v is already in L2", e) + } + if i > 0 { + p := res.CallEdges[i-1] + pk := key{p.SourcePath, p.Source, p.Target, p.TargetPath} + if c := cmp.Or(cmp.Compare(pk.path, k.path), cmp.Compare(pk.src, k.src), cmp.Compare(pk.dst, k.dst), cmp.Compare(pk.dstPath, k.dstPath)); c >= 0 { + t.Errorf("edges %+v, %+v out of order or duplicated", p, e) + } + } + if !units[dataflow.L3Key{Path: e.SourcePath, Signature: e.Source}] { + t.Errorf("edge source %s (%s) is not a unit", e.Source, e.SourcePath) + } + _, l2ext := in.Externals[e.Target] + _, newExt := res.NewExternals[e.Target] + if !sigs[e.Target] && !l2ext && !newExt { + t.Errorf("edge target %s resolves to nothing", e.Target) + } + } +} + +// TestVTA_Deterministic: two runs give the same targets, edges and externals. +func TestVTA_Deterministic(t *testing.T) { + _, a, da := runDispatch(t) + _, b, db := runDispatch(t) + if !slices.EqualFunc(a.CallEdges, b.CallEdges, func(x, y schema.GoCallEdge) bool { + return x.Source == y.Source && x.Target == y.Target && x.SourcePath == y.SourcePath && x.TargetPath == y.TargetPath && + slices.Equal(x.Provenance, y.Provenance) && x.Weight == y.Weight + }) { + t.Error("CallEdges differ between runs") + } + if len(da.sites) != len(db.sites) { + t.Fatalf("%d vs %d sites", len(da.sites), len(db.sites)) + } + for s, ts := range da.sites { + if !slices.Equal(ts, db.sites[s]) { + t.Errorf("site %v: %v vs %v", s, ts, db.sites[s]) + } + } + if len(a.NewExternals) != len(b.NewExternals) { + t.Errorf("%d vs %d new externals", len(a.NewExternals), len(b.NewExternals)) + } +} + +// TestVTA_NoVTAKeepsStaticTargets: when VTA fails, every site keeps its L2 +// target (the interface method for an interface call) and no edge is added. +func TestVTA_NoVTAKeepsStaticTargets(t *testing.T) { + in := loadInput(t, "dataflow") + defer func(f func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph) { vtaCallGraph = f }(vtaCallGraph) + vtaCallGraph = func(map[*ssa.Function]bool, *callgraph.Graph) *callgraph.Graph { panic("vta boom") } + res, d := run(in) + if len(res.CallEdges) != 0 || len(res.NewExternals) != 0 { + t.Errorf("no_vta run added %d edges, %d externals", len(res.CallEdges), len(res.NewExternals)) + } + if got, want := d.targets(sdgKey("UseStore"), "34:9"), []Target{sdgTarget("Store.Get")}; !slices.Equal(got, want) { + t.Errorf("targets(UseStore, 34:9) = %v, want %v", got, want) + } + if got, want := d.targets(sdgKey("Buffered"), "93:2"), []Target{{External: "io.Writer.Write"}}; !slices.Equal(got, want) { + t.Errorf("targets(Buffered, 93:2) = %v, want %v", got, want) + } + if got := d.targets(sdgKey("CallFuncValue"), "51:9"); len(got) != 0 { + t.Errorf("targets(CallFuncValue, 51:9) = %v, want none", got) + } +} + +// TestVTA_InitClosureTargetsKeepTheirFile: Run's h() reaches the closure of +// each file's init. Both share the signature initclosure.init.closure_1, so +// each target, and each edge's TargetPath, names its own declaring file. +func TestVTA_InitClosureTargetsKeepTheirFile(t *testing.T) { + _, res, d := runDispatch(t) + const pkg = "example.com/dataflow/initclosure" + a, b := filepath.Join("initclosure", "a.go"), filepath.Join("initclosure", "b.go") + sig := pkg + ".init.closure_1" + got := d.targets(dataflow.L3Key{Path: a, Signature: pkg + ".Run"}, "16:8") + want := []Target{{Callable: dataflow.L3Key{Path: a, Signature: sig}}, {Callable: dataflow.L3Key{Path: b, Signature: sig}}} + if !slices.Equal(got, want) { + t.Fatalf("targets(Run, 16:8) = %v, want %v", got, want) + } + files := map[string]bool{} + for _, e := range res.CallEdges { + if e.Source == pkg+".Run" && e.Target == sig { + files[e.TargetPath] = true + } + } + if !files[a] || !files[b] || len(files) != 2 { + t.Errorf("Run → %s edges have TargetPaths %v, want %s and %s", sig, files, a, b) + } +} diff --git a/internal/sdg/wiring.go b/internal/sdg/wiring.go new file mode 100644 index 0000000..87a7c84 --- /dev/null +++ b/internal/sdg/wiring.go @@ -0,0 +1,484 @@ +package sdg + +import ( + "cmp" + "go/ast" + "go/types" + "slices" + "strings" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +// ProvReachingDefs is the prov of every port-wiring ddg edge (spec §4.7, +// Python's label; L3's own edges are "ssa"). +const ProvReachingDefs = "reaching-defs" + +// wire returns the port wiring of unit k (spec §4.7, A4, A7, A11): the +// callable-local ddg edges, prov "reaching-defs", that connect its L4 +// vertices to its body. They come from re-solving reaching definitions over +// an augmented DefUse (resolve) and from the vertices themselves: +// +// - each L3 edge @entry → n on a base with a formal_in is mirrored as +// @formal_in: → n (same var); the L3 edge stays and is not repeated; +// - each re-solved edge d → @exit on an output base b is emitted as +// d → the formal_out of b (var its `of`), never into @exit; +// - each other re-solved edge not in L3 (from the A4 uses and weak defs) +// is emitted as is; +// - a new edge from @entry on a base with a formal_in starts at that +// formal_in instead (Python's formal_for remap); @entry defines a named +// result's zero value and keeps those edges; +// - every return node → each formal_out of an unnamed (or `_`) result +// (var `` / `:`); a named result reaches its +// formal_out through the @exit rule; +// - at each resolved call site (A11): host → each actual_in, and each +// actual_out → host (var the actual's `of`), host = the call node's +// Parent, else the call node (a bare, `go` or deferred call); +// - at each capture site (A6, A7): every node defining a captured base → +// its actual_in; its actual_out → every node using the base, and → the +// encloser's own formal_out for the base when it is an output base (a +// named result, a by-ref param, or a capture the encloser writes). +// +// An edge's var is the `of` of the vertex it touches, the formal_out's when +// it ends at one; a body → body edge keeps the L3 path spelling. +// +// The same re-solve, with aliasOracle consulted (dataflow.BuildDDGSplit), +// yields the points-to delta (spec §4.10): each alias-only edge, mapped by +// the rules above (port), prov "points-to", unless its (src, dst, var) +// already is an L3 or reaching-defs edge. It never removes or changes one: +// alias adds no kills. A skipped unit gets none (its wiring is already +// all-to-all). +// +// A unit with no L3 result (l3_skipped) has no body nodes to wire: it is +// wired pass-through (wireSkipped). +// +// The edges are sorted by (Src, Dst, Var, Prov) and unique; none repeats an +// L3 edge, and no (src, dst, var) has both provs. +func (p *ports) wire(k dataflow.L3Key) []PortEdge { + u, ok := p.units[k] + if !ok { + return nil + } + w := p.newWiring(u) + if w == nil { + return p.wireSkipped(u) + } + l3 := make(map[dataflow.DdgEdge]bool, len(w.r.Ddg)) + for _, e := range w.r.Ddg { + l3[e] = true + } + out := map[PortEdge]bool{} + // add drops an L3 edge. No rule below yields one (an edge touching a + // vertex is never L3's, and a body → body edge is emitted only when it is + // new), so this only guards the "no PortEdge repeats L3" contract. + add := func(src, dst, v string) { + if !l3[dataflow.DdgEdge{Src: src, Dst: dst, Var: v}] { + out[PortEdge{Src: src, Dst: dst, Var: v, Prov: ProvReachingDefs}] = true + } + } + + var mayAlias func(a, b dataflow.Path) bool + if newAliasOracle != nil { + mayAlias = newAliasOracle(u, p.rootOf(u)).MayAlias + } + resolved, alias := dataflow.BuildDDGSplit(w.r.Nodes, w.r.Cfg, w.du, mayAlias) + for _, e := range resolved { + if src, dst, v, ok := w.port(e, l3[e]); ok { + add(src, dst, v) + } + } + + for i, fo := range w.fs.results { + if i < len(w.resObjs) && w.resObjs[i] != nil { + continue + } + of := resultOf(i, len(w.fs.results)) + for _, n := range w.r.Nodes { + if n.Kind == dataflow.KindReturn { + add(n.Key, fo, of) + } + } + } + + for _, s := range w.sites { + for _, v := range s.site.In { + add(s.host, v.Key, v.Of) + } + for _, v := range s.site.Out { + add(v.Key, s.host, v.Of) + } + } + + for _, cs := range p.capSites(u, w.taken) { + for i, v := range cs.caps { + of := cs.ofs[i] + for _, n := range w.r.Nodes { + d := w.r.DefUse[n.Key] + if hasBase(d.Defs, v) || hasBase(d.WeakDefs, v) { + src := n.Key + if fi, ok := w.inByObj[v]; ok && src == dataflow.EntryKey { + src = fi + } + add(src, cs.in[i], of) + } + } + if cs.out[i] == "" { + continue + } + for _, n := range w.r.Nodes { + if n.Key != dataflow.ExitKey && hasBase(w.r.DefUse[n.Key].Uses, v) { + add(cs.out[i], n.Key, of) + } + } + if fo, ok := w.outByObj[v]; ok { + add(cs.out[i], fo.Key, fo.Of) + } + } + } + + // The points-to delta (spec §4.10): the alias-only edges of the same + // re-solve, mapped onto the vertices like the re-solved edges, minus any + // (src, dst, var) an L3 or reaching-defs edge already has. + for _, e := range alias { + src, dst, v, ok := w.portFrom(e.DdgEdge, e.Def, false) + if !ok || l3[dataflow.DdgEdge{Src: src, Dst: dst, Var: v}] { + continue + } + if !out[PortEdge{Src: src, Dst: dst, Var: v, Prov: ProvReachingDefs}] { + out[PortEdge{Src: src, Dst: dst, Var: v, Prov: ProvPointsTo}] = true + } + } + return sortedPortEdges(out) +} + +// port maps a re-solved ddg edge e of the unit onto its port edge (wire's +// rules), ok false when e yields none. inL3 says e is an L3 edge. +// +// - e → @exit on an output base: e.Src → its formal_out, var the +// formal_out's `of`; on any other base, nothing; +// - an L3 edge: only its formal_in mirror, when it leaves @entry on a base +// with a formal_in; else nothing (the L3 edge stays L3's); +// - any other edge: as is, from the base's formal_in instead of @entry. +func (w *wiring) port(e dataflow.DdgEdge, inL3 bool) (src, dst, v string, ok bool) { + return w.portFrom(e, dataflow.SpellBase(e.Var), inL3) +} + +// portFrom is port with the base whose formal_in an edge from @entry starts +// at given: the use's base for a re-solved edge (def and use share it), the +// def's for an alias edge (AliasEdge.Def: @entry defines a value base whose +// address is taken, and the use reads it through another base). +func (w *wiring) portFrom(e dataflow.DdgEdge, b string, inL3 bool) (src, dst, v string, ok bool) { + src = e.Src + fi, hasIn := w.inOf(b) + fromEntry := hasIn && src == dataflow.EntryKey + if fromEntry { + src = fi + } + switch { + case e.Dst == dataflow.ExitKey: + fo, ok := w.exitOut[e.Var] + if !ok { + return "", "", "", false + } + return src, fo.Key, fo.Of, true + case inL3: + return src, e.Dst, e.Var, fromEntry + default: + return src, e.Dst, e.Var, true + } +} + +// resolve re-solves unit k's reaching definitions over the augmented DefUse +// (newWiring): L3's ddg plus the A4 and @exit edges. nil without an L3 +// result. +func (p *ports) resolve(k dataflow.L3Key) []dataflow.DdgEdge { + u, ok := p.units[k] + if !ok { + return nil + } + w := p.newWiring(u) + if w == nil { + return nil + } + return dataflow.BuildDDG(w.r.Nodes, w.r.Cfg, w.du) +} + +// wiring is one analysed unit's port-wiring inputs. +type wiring struct { + r dataflow.Result + fs *formalSet + du map[string]dataflow.DefUse // du': L3's DefUse, augmented + exitOut map[string]Vertex // @exit use spelling → its formal_out + inByBase map[string]string // base spelling → formal_in key + inByObj map[types.Object]string // base object → formal_in key + outByObj map[types.Object]Vertex // output base object → its formal_out + resObjs []types.Object // per result position: its object, nil if unnamed or `_` + sites []hostedSite + taken map[string]bool // every L1 call key and kept node key (capture-site keys) +} + +type hostedSite struct { + host string + site site +} + +func (w *wiring) inOf(base string) (string, bool) { + k, ok := w.inByBase[base] + return k, ok +} + +// newWiring builds the augmented DefUse du' of analysed unit u (nil without +// an L3 DefUse): a copy of L3's, plus +// +// 1. (A4) at each host of a resolved, kept call site, the union over its +// targets of their footprints' GlobalReads as uses and GlobalWrites as +// weak defs (weak defs kill nothing, so every L3 edge survives); +// 2. at @entry, a def of every global formal_in base, so that a global +// only a callee reads still has a def (its formal_in) at the host; +// 3. at @exit, a use of every output base: the named results, the by-ref +// params in Footprint.Mutated (the receiver too), the captures the +// closure writes, and the footprint's GlobalWrites. +// +// A global is the unit's own object for it when its DefUse names one, else +// a nil-base path spelled "pkgpath::Name" (the spelling of an unresolved +// package member at L3), consistent within the unit. +func (p *ports) newWiring(u dataflow.Unit) *wiring { + r, ok := p.in.L3[u.Key] + if !ok || r.DefUse == nil || u.Info == nil { + return nil + } + t := Target{Callable: u.Key} + fs := p.formalSet(t) + fp := p.fps[t] + w := &wiring{ + r: r, + fs: fs, + du: make(map[string]dataflow.DefUse, len(r.DefUse)+1), + exitOut: map[string]Vertex{}, + inByBase: map[string]string{}, + inByObj: map[types.Object]string{}, + outByObj: map[types.Object]Vertex{}, + taken: make(map[string]bool, len(u.Calls)+len(r.Nodes)), + } + for key, d := range r.DefUse { + w.du[key] = d + } + globals := map[string]types.Object{} + for _, d := range r.DefUse { + for _, list := range [][]dataflow.Path{d.Defs, d.WeakDefs, d.Uses} { + for _, path := range list { + if dataflow.IsPkgVar(path.Base) { + globals[dataflow.BaseSpell(path.Base)] = path.Base + } + } + } + } + global := func(g string) dataflow.Path { return dataflow.Path{Base: globals[g], Spell: g} } + outKey := make(map[string]string, len(fs.out)) + outs := make(map[string]Vertex, len(fs.out)) + for _, v := range fs.out { + outKey[v.Of] = v.Key + outs[v.Key] = v + } + outVertex := func(key string) Vertex { return outs[key] } + + // formal_in by base: declared formals by name, captures, globals. + objs := formalObjs(u.Fn, u.Info) + byName := map[string]types.Object{} + for _, o := range objs { + byName[o.Name()] = o + } + for i, of := range fs.posOf { + w.inByBase[of] = fs.inPos[i] + if o, ok := byName[of]; ok { + w.inByObj[o] = fs.inPos[i] + } + } + var caps []types.Object + if lit, ok := u.Fn.(*ast.FuncLit); ok { + caps = dataflow.FreeVars(lit, u.Info) + } + for _, v := range caps { + if k, ok := fs.inByOf[ofCapture+v.Name()]; ok { + w.inByBase[v.Name()] = k + w.inByObj[v] = k + } + } + for _, of := range fs.globIn { + w.inByBase[strings.TrimPrefix(of, ofGlobal)] = fs.inByOf[of] + } + + // @exit uses: the output bases, each mapped to its formal_out. + var exitUses []dataflow.Path + exitUse := func(path dataflow.Path, key string) { + exitUses = append(exitUses, path) + w.exitOut[path.Spell] = outVertex(key) + if path.Base != nil { + w.outByObj[path.Base] = outVertex(key) + } + } + w.resObjs = resultObjs(u.Fn, u.Info) + for i, o := range w.resObjs { + if o != nil && i < len(fs.results) { + exitUse(dataflow.Path{Base: o, Spell: o.Name()}, fs.results[i]) + } + } + for i, of := range fs.posOf { + if key := fs.outPos[i]; key != "" { + if o, ok := byName[of]; ok { + exitUse(dataflow.Path{Base: o, Spell: o.Name()}, key) + } + } + } + for _, v := range caps { + if key, ok := fs.outByOf[ofCapture+v.Name()]; ok { + exitUse(dataflow.Path{Base: v, Spell: v.Name()}, key) + } + } + for _, g := range fp.GlobalWrites { + if key, ok := outKey[ofGlobal+g]; ok { + exitUse(global(g), key) + } + } + d := w.du[dataflow.ExitKey] + d.Uses = append(slices.Clip(d.Uses), exitUses...) + w.du[dataflow.ExitKey] = d + + var entryDefs []dataflow.Path + for _, of := range fs.globIn { + entryDefs = append(entryDefs, global(strings.TrimPrefix(of, ofGlobal))) + } + d = w.du[dataflow.EntryKey] + d.Defs = append(slices.Clip(d.Defs), entryDefs...) + w.du[dataflow.EntryKey] = d + + // Call sites: the hosts, their A4 facts and their actuals. + nodes := make(map[string]dataflow.Node, len(r.Nodes)) + for _, n := range r.Nodes { + nodes[n.Key] = n + w.taken[n.Key] = true + } + for _, c := range u.Calls { + w.taken[c.Key] = true + } + for _, c := range u.Calls { + n, ok := nodes[c.Key] + if !ok { + continue // pruned: no vertices, no wiring + } + host := c.Key + if n.Parent != "" { + host = n.Parent + } + if _, ok := nodes[host]; !ok { + continue + } + ts := p.siteTargets(u.Key, c.Key) + if len(ts) == 0 { + continue + } + var reads, writes [][]string + for _, t := range ts { + f := p.fps[t] + reads, writes = append(reads, f.GlobalReads), append(writes, f.GlobalWrites) + } + if rs, ws := unionSorted(reads), unionSorted(writes); len(rs)+len(ws) > 0 { + d := w.du[host] + uses, weak := slices.Clip(d.Uses), slices.Clip(d.WeakDefs) + for _, g := range rs { + uses = append(uses, global(g)) + } + for _, g := range ws { + weak = append(weak, global(g)) + } + d.Uses, d.WeakDefs = uses, weak + w.du[host] = d + } + if s := p.callSite(u.Key, c.Key, ts); len(s.In)+len(s.Out) > 0 { + w.sites = append(w.sites, hostedSite{host: host, site: s}) + } + } + return w +} + +// wireSkipped is the pass-through wiring of a unit with no L3 result +// (ruling, Task 12 review; l3_skipped is a degraded code): with no body +// nodes, every formal_in → every actual_in of its call and capture sites and +// → every formal_out, and every actual_out → every formal_out. Each edge's +// var is its destination vertex's `of`. +func (p *ports) wireSkipped(u dataflow.Unit) []PortEdge { + fin, fout := p.formals(Target{Callable: u.Key}) + var ins, outs []Vertex + taken := make(map[string]bool, len(u.Calls)) + for _, c := range u.Calls { + taken[c.Key] = true + s := p.callSite(u.Key, c.Key, p.siteTargets(u.Key, c.Key)) + ins, outs = append(ins, s.In...), append(outs, s.Out...) + } + for _, v := range p.captureSites(u, taken) { + if v.Kind == KindActualIn { + ins = append(ins, v) + } else { + outs = append(outs, v) + } + } + out := map[PortEdge]bool{} + for _, f := range fin { + for _, a := range slices.Concat(ins, fout) { + out[PortEdge{Src: f.Key, Dst: a.Key, Var: a.Of, Prov: ProvReachingDefs}] = true + } + } + for _, a := range outs { + for _, f := range fout { + out[PortEdge{Src: a.Key, Dst: f.Key, Var: f.Of, Prov: ProvReachingDefs}] = true + } + } + return sortedPortEdges(out) +} + +// resultObjs returns, per result position of fn, its named result object, +// nil for an unnamed or `_` one. +func resultObjs(fn ast.Node, info *types.Info) []types.Object { + var ft *ast.FuncType + switch fn := fn.(type) { + case *ast.FuncDecl: + ft = fn.Type + case *ast.FuncLit: + ft = fn.Type + } + if ft == nil || ft.Results == nil { + return nil + } + var out []types.Object + for _, f := range ft.Results.List { + if len(f.Names) == 0 { + out = append(out, nil) + continue + } + for _, name := range f.Names { + var o types.Object + if name.Name != "_" { + o = info.Defs[name] + } + out = append(out, o) + } + } + return out +} + +func hasBase(list []dataflow.Path, v types.Object) bool { + return slices.ContainsFunc(list, func(p dataflow.Path) bool { return p.Base == v }) +} + +func comparePortEdge(a, b PortEdge) int { + return cmp.Or(strings.Compare(a.Src, b.Src), strings.Compare(a.Dst, b.Dst), strings.Compare(a.Var, b.Var), strings.Compare(a.Prov, b.Prov)) +} + +func sortedPortEdges(set map[PortEdge]bool) []PortEdge { + out := make([]PortEdge, 0, len(set)) + for e := range set { + out = append(out, e) + } + slices.SortFunc(out, comparePortEdge) + return out +} diff --git a/internal/sdg/wiring_test.go b/internal/sdg/wiring_test.go new file mode 100644 index 0000000..0d45281 --- /dev/null +++ b/internal/sdg/wiring_test.go @@ -0,0 +1,358 @@ +package sdg + +import ( + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +func pe(src, dst, v string) PortEdge { + return PortEdge{Src: src, Dst: dst, Var: v, Prov: ProvReachingDefs} +} + +func flowKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("flow", "flow.go"), Signature: "example.com/dataflow/flow." + name} +} + +// fixtureUnits returns the units of the sdg and flow fixture packages. +func fixtureUnits(in Input) []dataflow.Unit { + var out []dataflow.Unit + for _, u := range in.Units { + dir := strings.SplitN(u.Key.Path, string(filepath.Separator), 2)[0] + if dir == "sdg" || dir == "flow" { + out = append(out, u) + } + } + return out +} + +func checkHas(t *testing.T, what string, got []PortEdge, want ...PortEdge) { + t.Helper() + for _, e := range want { + if !slices.Contains(got, e) { + t.Errorf("%s lacks %+v; got %+v", what, e, got) + } + } +} + +// reaches reports whether dst is reachable from src over edges. +func reaches(edges []PortEdge, src, dst string) bool { + seen := map[string]bool{src: true} + work := []string{src} + for len(work) > 0 { + n := work[0] + work = work[1:] + for _, e := range edges { + if e.Src == n && !seen[e.Dst] { + if e.Dst == dst { + return true + } + seen[e.Dst] = true + work = append(work, e.Dst) + } + } + } + return false +} + +// TestWiring_FormalInReachesUses: each @entry → n edge on a formal's base is +// mirrored as @formal_in: → n with the edge's var (the @entry edge stays +// L3's). UseStore's whole wiring is pinned: the return statement hosts the +// satellite s.Get(k), defines the result and writes through s (by-ref out). +func TestWiring_FormalInReachesUses(t *testing.T) { + _, _, p := runPorts(t) + got := p.wire(sdgKey("UseStore")) + want := []PortEdge{ + pe("34:2", "34:9/actual_in:0", "_recv"), + pe("34:2", "34:9/actual_in:1", "k"), + pe("34:2", "@formal_out:0", ""), + pe("34:2", "@formal_out:1", "s"), + pe("34:9/actual_out:0", "34:2", ""), + pe("34:9/actual_out:1", "34:2", "_recv"), + pe("@formal_in:0", "34:2", "s"), + pe("@formal_in:0", "@formal_out:1", "s"), + pe("@formal_in:1", "34:2", "k"), + } + if !slices.Equal(got, want) { + t.Errorf("wire(UseStore):\n got %+v\n want %+v", got, want) + } + // A field use keeps the edge's spelling. + checkHas(t, "wire(multiT.Multi)", p.wire(sdgKey("multiT.Multi")), + pe("@formal_in:0", "174:2", "s.n"), pe("@formal_in:1", "174:2", "a")) + // A closure's capture formal mirrors its @entry def of the capture. + checkHas(t, "wire(DeferWrap.closure_2)", p.wire(flowKey("DeferWrap.closure_2")), + pe("@formal_in:0", "557:17", "err")) +} + +// TestWiring_ReturnReachesFormalOut: with unnamed results, every return node +// reaches each result's formal_out (var its `of`); a named result reaches it +// through its defs reaching @exit, and the bare return adds nothing. +func TestWiring_ReturnReachesFormalOut(t *testing.T) { + _, _, p := runPorts(t) + checkHas(t, "wire(multiT.Multi)", p.wire(sdgKey("multiT.Multi")), + pe("174:2", "@formal_out:0", ":0"), pe("174:2", "@formal_out:1", ":1")) + got := p.wire(sdgKey("Named")) + checkHas(t, "wire(Named)", got, pe("179:2", "@formal_out", "")) + for _, e := range got { + if e.Src == "180:2" { + t.Errorf("wire(Named) has %+v: a bare return defines nothing", e) + } + } + // A by-ref param's writes reach its formal_out. + checkHas(t, "wire(mutate)", p.wire(sdgKey("mutate")), + pe("132:2", "@formal_out:0", "p"), pe("133:2", "@formal_out:1", "s")) +} + +// TestWiring_NamedResultViaDeferredClosure (A7): DeferWrap's deferred +// `func() { err = wrapErr(err) }()` hosts its own actuals: the capture's +// actual_out reaches the deferred call node, whose weak def of err reaches +// @formal_out. A stored literal writing a named result or a by-ref param +// wires its capture actual_out straight to the encloser's formal_out. +func TestWiring_NamedResultViaDeferredClosure(t *testing.T) { + _, _, p := runPorts(t) + got := p.wire(flowKey("DeferWrap")) + checkHas(t, "wire(DeferWrap)", got, + pe("557:8", "557:8/actual_in:0", ":err"), + pe("557:8/actual_out", "557:8", ":err"), + pe("557:8", "@formal_out", "")) + if !reaches(got, "557:8/actual_out", "@formal_out") { + t.Errorf("wire(DeferWrap): 557:8/actual_out does not reach @formal_out: %+v", got) + } + checkHas(t, "wire(CaptureNamed)", p.wire(sdgKey("CaptureNamed")), + pe("293:2", "293:7/actual_in:0", ":err"), + pe("293:7/actual_out", "293:2", ":err"), + pe("293:7/actual_out", "295:2", ":err"), + pe("293:7/actual_out", "@formal_out", ""), + pe("293:2", "@formal_out", "")) + checkHas(t, "wire(CaptureParam)", p.wire(sdgKey("CaptureParam")), + pe("@formal_in:0", "301:7/actual_in:0", ":p"), + pe("301:7/actual_out", "301:2", ":p"), + pe("301:7/actual_out", "@formal_out", "p"), + pe("301:2", "@formal_out", "p")) + // A capture site passed to a call (OnceDo): its defs and uses. + checkHas(t, "wire(OnceDo)", p.wire(flowKey("OnceDo")), + pe("587:2", "588:10/actual_in:0", ":cfg"), + pe("588:10/actual_out", "589:2", ":cfg")) +} + +// TestWiring_TransitiveGlobalAtHost (A4): useConfig reads Config only +// through readConfig. The host uses Config, which @formal_in(Config) defines, +// and feeds the call's actual_in for it. callsGet's callee writes inst: the +// actual_out feeds the host, whose weak def of inst reaches its formal_out. +func TestWiring_TransitiveGlobalAtHost(t *testing.T) { + _, _, p := runPorts(t) + got := p.wire(sdgKey("useConfig")) + checkHas(t, "wire(useConfig)", got, + pe("@formal_in:0", "122:2", configSpell), + pe("122:2", "122:9/actual_in:0", ":"+configSpell)) + if !reaches(got, "@formal_in:0", "122:9/actual_in:0") { + t.Errorf("wire(useConfig): @formal_in:0 does not reach 122:9/actual_in:0") + } + + k := globalsKey("callsGet") + inst := ":" + sdgPkg + "::inst" + var aout, fout string + for _, v := range p.vertices(k) { + switch { + case v.Kind == KindActualOut && v.Of == inst: + aout = v.Key + case v.Kind == KindFormalOut && v.Of == inst: + fout = v.Key + } + } + if aout == "" || fout == "" { + t.Fatalf("callsGet has no inst actual_out / formal_out: %+v", p.vertices(k)) + } + host := strings.SplitN(aout, "/actual_out", 2)[0] + if n := nodeOf(p.in.L3[k], host); n.Parent != "" { + host = n.Parent + } + got = p.wire(k) + checkHas(t, "wire(callsGet)", got, pe(aout, host, inst), pe(host, fout, inst)) +} + +func nodeOf(r dataflow.Result, key string) dataflow.Node { + for _, n := range r.Nodes { + if n.Key == key { + return n + } + } + return dataflow.Node{} +} + +// TestWiring_L3EdgesUntouched: the augmented re-solve keeps every L3 ddg +// edge (weak defs kill nothing), no port edge repeats an L3 edge, none ends +// at @exit, every prov is reaching-defs or (the alias delta, Task 17) +// points-to, every endpoint is a kept body node or a vertex of the callable, +// and the output is sorted and unique. +func TestWiring_L3EdgesUntouched(t *testing.T) { + in, _, p := runPorts(t) + n := 0 + for _, u := range in.Units { + r, ok := in.L3[u.Key] + if !ok { + continue + } + re := p.resolve(u.Key) + l3 := map[dataflow.DdgEdge]bool{} + for _, e := range r.Ddg { + l3[e] = true + if !slices.Contains(re, e) { + t.Errorf("%s: re-solve lost L3 edge %+v", u.Key.Signature, e) + } + } + ends := map[string]bool{} + for _, nd := range r.Nodes { + ends[nd.Key] = true + } + for _, v := range p.vertices(u.Key) { + ends[v.Key] = true + } + got := p.wire(u.Key) + for i, e := range got { + n++ + if l3[dataflow.DdgEdge{Src: e.Src, Dst: e.Dst, Var: e.Var}] { + t.Errorf("%s: port edge %+v repeats an L3 edge", u.Key.Signature, e) + } + if e.Dst == dataflow.ExitKey || e.Prov != ProvReachingDefs && e.Prov != ProvPointsTo || !ends[e.Src] || !ends[e.Dst] { + t.Errorf("%s: bad port edge %+v", u.Key.Signature, e) + } + if i > 0 && comparePortEdge(got[i-1], e) >= 0 { + t.Errorf("%s: port edges not sorted/unique at %+v", u.Key.Signature, e) + } + } + } + if n == 0 { + t.Fatal("no port edges on testdata/dataflow") + } +} + +// TestWiring_PortsCarryDependence mirrors the SDK's _ports_carry_dependence +// probe per callable: every sdg/flow callable with a formal_in has a port +// edge leaving one, unless none of its formal bases is used (no node but +// @entry uses its spelling, and no call site's targets read it, A4). +func TestWiring_PortsCarryDependence(t *testing.T) { + in, d, p := runPorts(t) + checked := 0 + for _, u := range fixtureUnits(in) { + if fails, ok := portsCarry(in, d, p, u); ok { + checked++ + } else if fails { + t.Errorf("%s: no port edge leaves a formal_in, though a formal base is used: %+v", u.Key.Signature, p.wire(u.Key)) + } + } + if checked < 20 { + t.Fatalf("only %d callables with a used formal_in: the gate covers too little", checked) + } +} + +// portsCarry checks u: ok when a port edge leaves one of its formal_ins; +// fails when none does though some formal base is used. Both false: exempt. +func portsCarry(in Input, d *dispatch, p *ports, u dataflow.Unit) (fails, ok bool) { + fin, _ := p.formals(Target{Callable: u.Key}) + if len(fin) == 0 { + return false, false + } + for _, e := range p.wire(u.Key) { + if strings.HasPrefix(e.Src, "@formal_in:") { + return false, true + } + } + return formalUsed(in, d, p, u, fin), false +} + +// formalUsed reports whether some formal_in base of u is used: a node other +// than @entry uses a path with its spelling, or (a global) a resolved call +// site's target reads it. A unit with no L3 result counts as using them all. +func formalUsed(in Input, d *dispatch, p *ports, u dataflow.Unit, fin []Vertex) bool { + r, ok := in.L3[u.Key] + if !ok { + return true + } + bases := map[string]bool{} + for _, v := range fin { + b := strings.TrimPrefix(strings.TrimPrefix(v.Of, ofCapture), ofGlobal) + bases[b] = true + } + for key, du := range r.DefUse { + if key == dataflow.EntryKey { + continue + } + for _, use := range du.Uses { + if bases[dataflow.SpellBase(use.Spell)] { + return true + } + } + } + for _, c := range u.Calls { + if p.pruned(u.Key, c.Key) { + continue + } + for _, t := range p.siteTargets(u.Key, c.Key) { + for _, g := range p.fps[t].GlobalReads { + if bases[g] { + return true + } + } + } + } + return false +} + +// TestWiring_L3SkippedPassThrough (ruling, Task 12 review): a unit with no +// L3 result gets capture sites hosted by the L1 call holding the literal, +// else @entry, and pass-through wiring: every formal_in feeds every actual_in, +// every actual_out every formal_out, and every formal_in every formal_out. +func TestWiring_L3SkippedPassThrough(t *testing.T) { + in, d, _ := runPorts(t) + skip := map[dataflow.L3Key]bool{sdgKey("CaptureSite"): true, sdgKey("CaptureArg"): true, flowKey("OnceDo"): true} + l3 := make(map[dataflow.L3Key]dataflow.Result, len(in.L3)) + for k, r := range in.L3 { + if !skip[k] { + l3[k] = r + } + } + in.L3 = l3 + p := newPorts(in, d) + + checkVertices(t, "skipped CaptureSite capture site", capturesOf(p.vertices(sdgKey("CaptureSite"))), []Vertex{ + ain("215:7", "0", ":x", "@entry"), + aout("215:7", "", ":x", "@entry"), + }) + checkVertices(t, "skipped CaptureArg capture site", capturesOf(p.vertices(sdgKey("CaptureArg"))), []Vertex{ + ain("267:13", "0", ":v", "267:7"), + }) + checkVertices(t, "skipped OnceDo capture site", capturesOf(p.vertices(flowKey("OnceDo"))), []Vertex{ + ain("588:10", "0", ":cfg", "588:2"), + aout("588:10", "", ":cfg", "588:2"), + }) + + got := p.wire(sdgKey("CaptureSite")) + want := []PortEdge{ + pe("215:7/actual_out", "@formal_out", ""), + pe("@formal_in:0", "215:7/actual_in:0", ":x"), + pe("@formal_in:0", "@formal_out", ""), + } + if !slices.Equal(got, want) { + t.Errorf("skipped wire(CaptureSite):\n got %+v\n want %+v", got, want) + } + checkHas(t, "skipped wire(CaptureArg)", p.wire(sdgKey("CaptureArg")), + pe("@formal_in:0", "267:13/actual_in:0", ":v"), + pe("@formal_in:0", "267:7/actual_in:0", "f"), + pe("267:7/actual_out:0", "@formal_out", ""), + pe("267:7/actual_out:1", "@formal_out", "")) +} + +// TestWiring_Deterministic: two builders over two loads agree. +func TestWiring_Deterministic(t *testing.T) { + in1, _, p1 := runPorts(t) + _, _, p2 := runPorts(t) + for _, u := range in1.Units { + if a, b := p1.wire(u.Key), p2.wire(u.Key); !slices.Equal(a, b) { + t.Errorf("%v: port edges differ across runs:\n%v\n%v", u.Key, a, b) + } + } +} diff --git a/internal/sdg/wrap_test.go b/internal/sdg/wrap_test.go new file mode 100644 index 0000000..fa187e6 --- /dev/null +++ b/internal/sdg/wrap_test.go @@ -0,0 +1,66 @@ +package sdg + +import ( + "path/filepath" + "slices" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" +) + +const wrapPkg = "example.com/dataflow/wrap" + +// wrapKey is the L3Key of a callable declared in testdata/dataflow/wrap/wrap.go. +func wrapKey(name string) dataflow.L3Key { + return dataflow.L3Key{Path: filepath.Join("wrap", "wrap.go"), Signature: wrapPkg + "." + name} +} + +// TestVTA_TargetsBehindSyntheticWrappers (F1): VTA returns a synthetic +// wrapper for HandleBound's f(id) ((Store).Find$bound) and HandleDecorated's +// l.Find(id) (logged's promotion wrapper); the wrapper's own call of Find is +// a synthetic call site no call node joins. The wrapper's VTA targets are the +// original site's: each handler's site reaches sqlStore.Find, as the direct +// interface call does, and so does the global sqlStore.Find writes. +func TestVTA_TargetsBehindSyntheticWrappers(t *testing.T) { + in, _, d := runDispatch(t) + find := Target{Callable: wrapKey("sqlStore.Find")} + fps := footprints(in, d) + const global = wrapPkg + "::lastQuery" + for _, c := range []struct{ fn, site string }{ + {"HandleDirect", "33:9"}, + {"HandleBound", "39:9"}, + {"HandleDecorated", "50:9"}, + } { + got := d.targets(wrapKey(c.fn), c.site) + if !slices.Contains(got, find) { + t.Errorf("targets(%s, %s) = %v, want sqlStore.Find among them", c.fn, c.site, got) + } + fp := fps[Target{Callable: wrapKey(c.fn)}] + if !slices.Contains(fp.GlobalWrites, global) { + t.Errorf("%s footprint writes %v, want %s", c.fn, fp.GlobalWrites, global) + } + } +} + +// TestVTA_NoEdgeForTargetsVTADidNotFind (F7): a call_graph edge prov ["vta"] +// is one VTA produced. HandleBound's f(id) has no L2 callee; its targets +// include the interface method Store.Find (bound by D7, from the $bound +// wrapper VTA returned, not a VTA callee of the site's own) and sqlStore.Find +// (VTA, through the wrapper). Only the latter is an edge. The interface +// method stays a target: its param edges are unchanged. +func TestVTA_NoEdgeForTargetsVTADidNotFind(t *testing.T) { + _, res, d := runDispatch(t) + src := wrapKey("HandleBound") + if ts := d.targets(src, "39:9"); !slices.Contains(ts, Target{Callable: wrapKey("Store.Find")}) { + t.Fatalf("targets(HandleBound, 39:9) = %v, want Store.Find among them", ts) + } + var got []string + for _, e := range res.CallEdges { + if e.Source == src.Signature { + got = append(got, e.Target) + } + } + if want := []string{wrapPkg + ".sqlStore.Find"}; !slices.Equal(got, want) { + t.Errorf("VTA edges from HandleBound = %v, want %v", got, want) + } +} diff --git a/internal/semantic_analysis/call_graph.go b/internal/semantic_analysis/call_graph.go index 8279c6c..33fc1f5 100644 --- a/internal/semantic_analysis/call_graph.go +++ b/internal/semantic_analysis/call_graph.go @@ -2,7 +2,7 @@ // // This stage uses the same golang.org/x/tools/go/packages load that built the // symbol table. For each recorded call site it resolves the callee to a -// *types.Func, derives its signature via signatureOf(), backfills +// *types.Func, derives its signature via SignatureOf(), backfills // callee_signature in place, and emits an identity-only GoCallEdge. // // Precision choice: declared-type dispatch (CHA-style). Pointer-receiver @@ -23,6 +23,7 @@ import ( "golang.org/x/tools/go/packages" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" "github.com/codellm-devkit/codeanalyzer-go/internal/utils" ) @@ -121,6 +122,8 @@ func buildKnownSigs(symbolTable map[string]schema.GoFile) map[string]bool { } // resolveCallable backfills callee_signature on each call site and produces edges. +// It then does the same for the callable's closures (InnerCallables) in sorted +// key order, recursively, after the callable's own sites. // // An in-project callee (in knownSigs) gets an edge. A library callee (a package // outside the project) is recorded in cg.externals, its site is tagged with the @@ -146,7 +149,7 @@ func (cg *CallGraphBuilder) resolveCallable( utils.Debug("unresolved call site: %s in %s", site.MethodName, callable.Signature) continue } - calleeSig := calleeSignatureOf(fn) + calleeSig := CalleeSignatureOf(fn) // Backfill the site whether the callee is in-project or a library. site.CalleeSignature = &calleeSig @@ -181,6 +184,15 @@ func (cg *CallGraphBuilder) resolveCallable( }) } + // Closures: their own sites, then theirs in turn. Each carries its enclosing + // callable's declaring file as Path, so the site index and the edge dedup + // treat it like any other callable. + for _, name := range sortedKeys(callable.InnerCallables) { + inner, innerEdges := cg.resolveCallable(pkg, callable.InnerCallables[name], seen, knownSigs) + callable.InnerCallables[name] = inner + edges = append(edges, innerEdges...) + } + return callable, edges } @@ -203,26 +215,24 @@ func (cg *CallGraphBuilder) isLibrary(fn *types.Func) bool { // (the symbol table skips it), so it is a library too. return utils.IsVendored(utils.RelativePath(root, file)) } - if mod := cg.modulePath(); mod != "" { - p := fn.Pkg().Path() - return p != mod && !strings.HasPrefix(p, mod+"/") - } _, inProject := cg.pkgs[fn.Pkg().Path()] - return !inProject + return LibraryByPath(fn.Pkg().Path(), cg.modulePath(), inProject) } -func absPath(p string) string { - if abs, err := filepath.Abs(p); err == nil { - return abs +// LibraryByPath is the fallback library test for an object with no source +// file: by the project module's path prefix when go.mod names one, else by +// whether its package is a loaded project package. L2 and L4 share it so the +// two levels classify a callee the same way. +func LibraryByPath(pkgPath, modulePath string, projectPkg bool) bool { + if modulePath != "" { + return pkgPath != modulePath && !strings.HasPrefix(pkgPath, modulePath+"/") } - return p + return !projectPkg } -func (cg *CallGraphBuilder) absProjectDir() string { return absPath(cg.projectDir) } - -// modulePath reads the project's module path from go.mod ("" when unavailable). -func (cg *CallGraphBuilder) modulePath() string { - data, err := os.ReadFile(filepath.Join(cg.projectDir, "go.mod")) +// ModulePath reads the module path from dir's go.mod ("" when unavailable). +func ModulePath(dir string) string { + data, err := os.ReadFile(filepath.Join(dir, "go.mod")) if err != nil { return "" } @@ -234,9 +244,19 @@ func (cg *CallGraphBuilder) modulePath() string { return "" } +func absPath(p string) string { + if abs, err := filepath.Abs(p); err == nil { + return abs + } + return p +} + +func (cg *CallGraphBuilder) absProjectDir() string { return absPath(cg.projectDir) } + +// modulePath reads the project's module path from go.mod ("" when unavailable). +func (cg *CallGraphBuilder) modulePath() string { return ModulePath(cg.projectDir) } + // recordExternal registers a library callee, keyed by its full signature. -// Parameter names and types come from the callee's *types.Signature; unnamed -// parameters become "_". The receiver is not a parameter. func (cg *CallGraphBuilder) recordExternal(fn *types.Func, fullSig string) { if _, ok := cg.externals[fullSig]; ok { return @@ -244,6 +264,15 @@ func (cg *CallGraphBuilder) recordExternal(fn *types.Func, fullSig string) { if cg.externals == nil { cg.externals = map[string]schema.GoExternalCallee{} } + cg.externals[fullSig] = ExternalCallee(fn, fullSig) +} + +// ExternalCallee is the external_symbols entry of the library callee fn, whose +// full signature (CalleeSignatureOf) is fullSig. L2 and the L4 VTA pass both +// build their entries with it, so an external has one shape whichever found it. +// Parameter names and types come from the callee's *types.Signature; unnamed +// parameters become "_". The receiver is not a parameter. +func ExternalCallee(fn *types.Func, fullSig string) schema.GoExternalCallee { importPath := fn.Pkg().Path() // Record the declared signature, not the instance the call site selected: a // method on a generic library type (atomic.Pointer[A].Store) must yield one @@ -278,19 +307,29 @@ func (cg *CallGraphBuilder) recordExternal(fn *types.Func, fullSig string) { ret = "(" + strings.Join(rets, ", ") + ")" } - cg.externals[fullSig] = schema.GoExternalCallee{ + var receiver *schema.GoParameter + if recv := sig.Recv(); recv != nil { + name := recv.Name() + if name == "" || name == "_" { + name = "_recv" + } + receiver = &schema.GoParameter{Name: name, Type: types.TypeString(recv.Type(), nil)} + } + + return schema.GoExternalCallee{ ImportPath: importPath, Sig: strings.TrimPrefix(fullSig, importPath+"."), Name: fn.Name(), Params: params, ReturnType: ret, + Receiver: receiver, } } // resolveCallSite resolves a single call site to its callee function by looking // it up in the package's call-site index under the key L1 recorded it with: -// relPath (the file declaring the enclosing callable) and the site's -// directive-adjusted start and end line:col. Returns nil when the site cannot be +// relPath (the file declaring the enclosing callable) and the site's real +// (not //line-adjusted) start and end line:col. Returns nil when the site cannot be // resolved (graceful fallback). func (cg *CallGraphBuilder) resolveCallSite(pkg *packages.Package, relPath string, site *schema.GoCallsite) *types.Func { if pkg.TypesInfo == nil { @@ -339,38 +378,12 @@ func (cg *CallGraphBuilder) resolveCallExpr(pkg *packages.Package, call *ast.Cal return nil } -// calleeSignatureOf is the call-site mirror of signatureOf — same canonicalization. -// Must produce byte-identical strings to signatureOf() in syntactic_analysis. -func calleeSignatureOf(obj types.Object) string { - if obj == nil { - return "" - } - pkg := obj.Pkg() - pkgPath := "" - if pkg != nil { - pkgPath = pkg.Path() - } - - switch o := obj.(type) { - case *types.Func: - sig := o.Type().(*types.Signature) - recv := sig.Recv() - if recv != nil { - recvType := recv.Type() - if ptr, ok := recvType.(*types.Pointer); ok { - recvType = ptr.Elem() - } - if named, ok := recvType.(*types.Named); ok { - typeName := named.Obj().Name() - return pkgPath + "." + typeName + "." + o.Name() - } - } - return pkgPath + "." + o.Name() - case *types.TypeName: - return pkgPath + "." + o.Name() - default: - return pkgPath + "." + obj.Name() - } +// CalleeSignatureOf is the signature of a resolved callee, the call-site side +// of the one canonicalizer (syntactic_analysis.SignatureOf), so caller-side +// and callee-side ids are identical by construction. The L4 SSA join uses it +// for external callees (ruling R-P5). +func CalleeSignatureOf(obj types.Object) string { + return syntactic_analysis.SignatureOf(obj) } // packageForFile returns the *packages.Package that contains the given relative file path. @@ -386,14 +399,17 @@ func (cg *CallGraphBuilder) packageForFile(relPath string) *packages.Package { } // MergeEdges merges two edge lists, unioning provenance and accumulating weight -// for duplicate (source, target) pairs. Mirrors Python's merge_edges(). +// for duplicate edges. Mirrors Python's merge_edges(), with the edge identity +// (SourcePath, Source, Target, TargetPath). A signature is not unique (every +// file's func init shares ".init", and so do its closures), so the +// declaring files keep two inits' edges apart. func MergeEdges(primary, secondary []schema.GoCallEdge) []schema.GoCallEdge { - type key struct{ src, tgt string } + type key struct{ path, src, tgt, tgtPath string } index := map[key]int{} result := make([]schema.GoCallEdge, 0, len(primary)+len(secondary)) add := func(e schema.GoCallEdge) { - k := key{e.Source, e.Target} + k := key{e.SourcePath, e.Source, e.Target, e.TargetPath} if idx, exists := index[k]; exists { // Merge provenance (union) and accumulate weight. provSet := map[string]bool{} diff --git a/internal/semantic_analysis/call_graph_test.go b/internal/semantic_analysis/call_graph_test.go index 84cb74f..6124394 100644 --- a/internal/semantic_analysis/call_graph_test.go +++ b/internal/semantic_analysis/call_graph_test.go @@ -120,6 +120,30 @@ func TestMergeEdges_OrderPreserved(t *testing.T) { } } +// TestMergeEdges_KeepsDeclaringFilesApart: two inits (one signature, two +// files) calling one callee are two edges, and so are two VTA edges to the +// same-signature closures of two files' inits; the same file pair merges. +func TestMergeEdges_KeepsDeclaringFilesApart(t *testing.T) { + at := func(e schema.GoCallEdge, src, dst string) schema.GoCallEdge { + e.SourcePath, e.TargetPath = src, dst + return e + } + primary := []schema.GoCallEdge{at(edge("p.init", "p.setup", 1, "go/types"), "a.go", "")} + secondary := []schema.GoCallEdge{ + at(edge("p.init", "p.setup", 1, "go/types"), "b.go", ""), + at(edge("p.Run", "p.init.closure_1", 1, "vta"), "a.go", "a.go"), + at(edge("p.Run", "p.init.closure_1", 1, "vta"), "a.go", "b.go"), + at(edge("p.init", "p.setup", 1, "vta"), "a.go", ""), + } + result := semantic_analysis.MergeEdges(primary, secondary) + if len(result) != 4 { + t.Fatalf("got %d edges, want 4: %+v", len(result), result) + } + if result[0].Weight != 2 || len(result[0].Provenance) != 2 { + t.Errorf("a.go init edge = %+v, want weight 2, prov [go/types vta]", result[0]) + } +} + // ── Call-site index ─────────────────────────────────────────────────────────── // TestCallGraph_IndexMatchesWalk locks the chi edge set (see chiGoldenEdges for the delta vs the walk) after @@ -161,3 +185,64 @@ func firstDiff(a, b []string) string { } return fmt.Sprintf("length differs (%d vs %d)", len(a), len(b)) } + +// TestCallGraph_ClosureSitesResolve: call sites inside function literals are +// backfilled and produce edges whose source is the closure signature and whose +// SourcePath is the declaring file (#17). +func TestCallGraph_ClosureSitesResolve(t *testing.T) { + dir, err := filepath.Abs("../../testdata/dataflow") + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + + const outer = "example.com/dataflow/flow.Closure" + lit, ok := st["flow/flow.go"].Functions[outer].InnerCallables["closure_1"] + if !ok { + t.Fatal("Closure has no closure_1") + } + if lit.Path != "flow/flow.go" { + t.Errorf("closure Path = %q, want flow/flow.go", lit.Path) + } + if len(lit.CallSites) != 1 || lit.CallSites[0].CalleeSignature == nil || + *lit.CallSites[0].CalleeSignature != "example.com/dataflow/flow.f" { + t.Fatalf("closure call site not resolved: %+v", lit.CallSites) + } + for _, e := range edges { + if e.Source == outer+".closure_1" && e.Target == "example.com/dataflow/flow.f" { + if e.SourcePath != "flow/flow.go" { + t.Errorf("edge SourcePath = %q, want flow/flow.go", e.SourcePath) + } + return + } + } + t.Error("no edge from the closure to flow.f") +} + +// TestLibraryByPath: the no-source-file fallback classifies by the module +// path prefix when go.mod names one (so an unloaded project package is still +// project code), else by whether the package was loaded. +func TestLibraryByPath(t *testing.T) { + for _, c := range []struct { + pkg, mod string + loaded bool + want bool + }{ + {"example.com/app", "example.com/app", false, false}, + {"example.com/app/internal/x", "example.com/app", false, false}, // unloaded project package + {"example.com/application", "example.com/app", true, true}, // prefix is segment-wise + {"fmt", "example.com/app", false, true}, + {"example.com/app/x", "", true, false}, // no go.mod: loaded = project + {"example.com/app/x", "", false, true}, + } { + if got := semantic_analysis.LibraryByPath(c.pkg, c.mod, c.loaded); got != c.want { + t.Errorf("LibraryByPath(%q, %q, %v) = %v, want %v", c.pkg, c.mod, c.loaded, got, c.want) + } + } +} diff --git a/internal/semantic_analysis/callsite_index.go b/internal/semantic_analysis/callsite_index.go index d4ec07e..243e5fc 100644 --- a/internal/semantic_analysis/callsite_index.go +++ b/internal/semantic_analysis/callsite_index.go @@ -12,12 +12,9 @@ import ( // callKey identifies a call expression exactly as L1 records its site: the // project-relative path of the file that contains it (the token.File name, the // same path as the enclosing callable's Path) and its start and end line:col -// from fset.Position, which applies //line directives. Start alone is -// ambiguous: a chained s.A().B() starts where its inner s.A() does. -// -// The key is never turned back into a token.Pos: under a //line directive the -// recorded line names a virtual line (often past the file's real line count), -// so rebuilding a position from it would be wrong or panic. +// as real file positions (fset.PositionFor(p, false): //line directives are +// not applied, as L1 does not apply them). Start alone is ambiguous: a chained +// s.A().B() starts where its inner s.A() does. type callKey struct { path string startLine, startCol, endLine, endColumn int @@ -26,9 +23,9 @@ type callKey struct { // callSiteIndex maps a call expression's callKey to the call. type callSiteIndex map[callKey]*ast.CallExpr -// newCallSiteIndex walks each file of pkg once and indexes every call. When -// two calls of one file share a key (only possible when //line directives map -// both to the same virtual range), the first in source order wins. +// newCallSiteIndex walks each file of pkg once and indexes every call. Real +// positions make the key unique per file (two calls never share both a start +// and an end); should one repeat, the first in source order wins. func newCallSiteIndex(projectDir string, fset *token.FileSet, pkg *packages.Package) callSiteIndex { idx := callSiteIndex{} for _, f := range pkg.Syntax { @@ -42,7 +39,7 @@ func newCallSiteIndex(projectDir string, fset *token.FileSet, pkg *packages.Pack if !ok { return true } - start, end := fset.Position(call.Pos()), fset.Position(call.End()) + start, end := fset.PositionFor(call.Pos(), false), fset.PositionFor(call.End(), false) k := callKey{rel, start.Line, start.Column, end.Line, end.Column} if _, dup := idx[k]; !dup { idx[k] = call diff --git a/internal/semantic_analysis/chi_golden_test.go b/internal/semantic_analysis/chi_golden_test.go index 1d09efb..3bb6e36 100644 --- a/internal/semantic_analysis/chi_golden_test.go +++ b/internal/semantic_analysis/chi_golden_test.go @@ -11,7 +11,9 @@ package semantic_analysis_test // - Throttle -> ThrottleWithOpts (throttle.go:33) is gained for the same reason; // - seven edges from the inner call of a chained call (e.g. // mx.NotFoundHandler().ServeHTTP, Chain(...).HandlerFunc) are gained, since -// the outer call no longer shadows the co-positioned inner one. +// the outer call no longer shadows the co-positioned inner one; +// - thirty-seven edges whose source is a closure (.closure_N) are gained: +// call sites inside function literals are resolved too (#17). var chiGoldenEdges = []string{ "github.com/go-chi/chi/v5.Context.RoutePattern -> github.com/go-chi/chi/v5.replaceWildcards", "github.com/go-chi/chi/v5.Middlewares.Handler -> github.com/go-chi/chi/v5.chain", @@ -33,14 +35,18 @@ var chiGoldenEdges = []string{ "github.com/go-chi/chi/v5.Mux.MethodNotAllowed -> github.com/go-chi/chi/v5.Chain", "github.com/go-chi/chi/v5.Mux.MethodNotAllowed -> github.com/go-chi/chi/v5.Middlewares.HandlerFunc", "github.com/go-chi/chi/v5.Mux.MethodNotAllowed -> github.com/go-chi/chi/v5.Mux.updateSubRoutes", + "github.com/go-chi/chi/v5.Mux.MethodNotAllowed.closure_1 -> github.com/go-chi/chi/v5.Mux.MethodNotAllowed", "github.com/go-chi/chi/v5.Mux.MethodNotAllowedHandler -> github.com/go-chi/chi/v5.methodNotAllowedHandler", "github.com/go-chi/chi/v5.Mux.Mount -> github.com/go-chi/chi/v5.Mux.MethodNotAllowed", "github.com/go-chi/chi/v5.Mux.Mount -> github.com/go-chi/chi/v5.Mux.NotFound", "github.com/go-chi/chi/v5.Mux.Mount -> github.com/go-chi/chi/v5.Mux.handle", "github.com/go-chi/chi/v5.Mux.Mount -> github.com/go-chi/chi/v5.node.findPattern", + "github.com/go-chi/chi/v5.Mux.Mount.closure_1 -> github.com/go-chi/chi/v5.Mux.nextRoutePath", + "github.com/go-chi/chi/v5.Mux.Mount.closure_1 -> github.com/go-chi/chi/v5.RouteContext", "github.com/go-chi/chi/v5.Mux.NotFound -> github.com/go-chi/chi/v5.Chain", "github.com/go-chi/chi/v5.Mux.NotFound -> github.com/go-chi/chi/v5.Middlewares.HandlerFunc", "github.com/go-chi/chi/v5.Mux.NotFound -> github.com/go-chi/chi/v5.Mux.updateSubRoutes", + "github.com/go-chi/chi/v5.Mux.NotFound.closure_1 -> github.com/go-chi/chi/v5.Mux.NotFound", "github.com/go-chi/chi/v5.Mux.Options -> github.com/go-chi/chi/v5.Mux.handle", "github.com/go-chi/chi/v5.Mux.Patch -> github.com/go-chi/chi/v5.Mux.handle", "github.com/go-chi/chi/v5.Mux.Post -> github.com/go-chi/chi/v5.Mux.handle", @@ -62,6 +68,7 @@ var chiGoldenEdges = []string{ "github.com/go-chi/chi/v5.Mux.routeHTTP -> github.com/go-chi/chi/v5.node.FindRoute", "github.com/go-chi/chi/v5.Mux.updateRouteHandler -> github.com/go-chi/chi/v5.chain", "github.com/go-chi/chi/v5.Mux.updateSubRoutes -> github.com/go-chi/chi/v5.node.routes", + "github.com/go-chi/chi/v5.NewMux.closure_1 -> github.com/go-chi/chi/v5.NewRouteContext", "github.com/go-chi/chi/v5.NewRouter -> github.com/go-chi/chi/v5.NewMux", "github.com/go-chi/chi/v5.URLParam -> github.com/go-chi/chi/v5.Context.URLParam", "github.com/go-chi/chi/v5.URLParam -> github.com/go-chi/chi/v5.RouteContext", @@ -94,14 +101,29 @@ var chiGoldenEdges = []string{ "github.com/go-chi/chi/v5.walk -> github.com/go-chi/chi/v5.Routes.Middlewares", "github.com/go-chi/chi/v5.walk -> github.com/go-chi/chi/v5.Routes.Routes", "github.com/go-chi/chi/v5.walk -> github.com/go-chi/chi/v5.walk", + "github.com/go-chi/chi/v5/middleware.BasicAuth.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.basicAuthFailed", + "github.com/go-chi/chi/v5/middleware.CleanPath.closure_1 -> github.com/go-chi/chi/v5.RouteContext", + "github.com/go-chi/chi/v5/middleware.ClientIPFromHeader.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.parseHeaderAddr", + "github.com/go-chi/chi/v5/middleware.ClientIPFromXFF.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.walkXFF", + "github.com/go-chi/chi/v5/middleware.ClientIPFromXFF.closure_1.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.inAnyPrefix", + "github.com/go-chi/chi/v5/middleware.ClientIPFromXFF.closure_1.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.parseHeaderAddr", + "github.com/go-chi/chi/v5/middleware.ClientIPFromXFFTrustedProxies.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.parseHeaderAddr", + "github.com/go-chi/chi/v5/middleware.ClientIPFromXFFTrustedProxies.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.walkXFF", "github.com/go-chi/chi/v5/middleware.Compress -> github.com/go-chi/chi/v5/middleware.NewCompressor", + "github.com/go-chi/chi/v5/middleware.Compressor.Handler.closure_1 -> github.com/go-chi/chi/v5/middleware.Compressor.selectEncoder", + "github.com/go-chi/chi/v5/middleware.Compressor.Handler.closure_1 -> github.com/go-chi/chi/v5/middleware.compressResponseWriter.Close", "github.com/go-chi/chi/v5/middleware.Compressor.selectEncoder -> github.com/go-chi/chi/v5/middleware.ioResetterWriter.Reset", "github.com/go-chi/chi/v5/middleware.Compressor.selectEncoder -> github.com/go-chi/chi/v5/middleware.matchAcceptEncoding", + "github.com/go-chi/chi/v5/middleware.ContentCharset.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.contentEncoding", "github.com/go-chi/chi/v5/middleware.DefaultLogFormatter.NewLogEntry -> github.com/go-chi/chi/v5/middleware.GetClientIP", "github.com/go-chi/chi/v5/middleware.DefaultLogFormatter.NewLogEntry -> github.com/go-chi/chi/v5/middleware.GetReqID", "github.com/go-chi/chi/v5/middleware.DefaultLogFormatter.NewLogEntry -> github.com/go-chi/chi/v5/middleware.cW", "github.com/go-chi/chi/v5/middleware.GetClientIP -> github.com/go-chi/chi/v5/middleware.GetClientIPAddr", + "github.com/go-chi/chi/v5/middleware.GetHead.closure_1 -> github.com/go-chi/chi/v5.NewRouteContext", + "github.com/go-chi/chi/v5/middleware.GetHead.closure_1 -> github.com/go-chi/chi/v5.RouteContext", + "github.com/go-chi/chi/v5/middleware.GetHead.closure_1 -> github.com/go-chi/chi/v5.Routes.Match", "github.com/go-chi/chi/v5/middleware.HeaderRoute.IsMatch -> github.com/go-chi/chi/v5/middleware.Pattern.Match", + "github.com/go-chi/chi/v5/middleware.HeaderRouter.Handler.closure_1 -> github.com/go-chi/chi/v5/middleware.HeaderRoute.IsMatch", "github.com/go-chi/chi/v5/middleware.HeaderRouter.Route -> github.com/go-chi/chi/v5/middleware.NewPattern", "github.com/go-chi/chi/v5/middleware.HeaderRouter.RouteAny -> github.com/go-chi/chi/v5/middleware.NewPattern", "github.com/go-chi/chi/v5/middleware.NewCompressor -> github.com/go-chi/chi/v5/middleware.Compressor.SetEncoder", @@ -111,8 +133,25 @@ var chiGoldenEdges = []string{ "github.com/go-chi/chi/v5/middleware.Profiler -> github.com/go-chi/chi/v5.Mux.HandleFunc", "github.com/go-chi/chi/v5/middleware.Profiler -> github.com/go-chi/chi/v5.Mux.Use", "github.com/go-chi/chi/v5/middleware.Profiler -> github.com/go-chi/chi/v5.NewRouter", + "github.com/go-chi/chi/v5/middleware.RealIP.closure_1 -> github.com/go-chi/chi/v5/middleware.realIP", + "github.com/go-chi/chi/v5/middleware.Recoverer.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.GetLogEntry", + "github.com/go-chi/chi/v5/middleware.Recoverer.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.LogEntry.Panic", + "github.com/go-chi/chi/v5/middleware.Recoverer.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.PrintPrettyStack", + "github.com/go-chi/chi/v5/middleware.RedirectSlashes.closure_1 -> github.com/go-chi/chi/v5.RouteContext", + "github.com/go-chi/chi/v5/middleware.RequestLogger.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.LogFormatter.NewLogEntry", + "github.com/go-chi/chi/v5/middleware.RequestLogger.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.NewWrapResponseWriter", + "github.com/go-chi/chi/v5/middleware.RequestLogger.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.WithLogEntry", + "github.com/go-chi/chi/v5/middleware.RequestLogger.closure_1.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.LogEntry.Write", + "github.com/go-chi/chi/v5/middleware.RequestLogger.closure_1.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.WrapResponseWriter.BytesWritten", + "github.com/go-chi/chi/v5/middleware.RequestLogger.closure_1.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.WrapResponseWriter.Status", + "github.com/go-chi/chi/v5/middleware.StripSlashes.closure_1 -> github.com/go-chi/chi/v5.RouteContext", + "github.com/go-chi/chi/v5/middleware.SupressNotFound.closure_1.closure_1 -> github.com/go-chi/chi/v5.Mux.NotFoundHandler", + "github.com/go-chi/chi/v5/middleware.SupressNotFound.closure_1.closure_1 -> github.com/go-chi/chi/v5.RouteContext", + "github.com/go-chi/chi/v5/middleware.SupressNotFound.closure_1.closure_1 -> github.com/go-chi/chi/v5.Routes.Match", "github.com/go-chi/chi/v5/middleware.Throttle -> github.com/go-chi/chi/v5/middleware.ThrottleWithOpts", "github.com/go-chi/chi/v5/middleware.ThrottleBacklog -> github.com/go-chi/chi/v5/middleware.ThrottleWithOpts", + "github.com/go-chi/chi/v5/middleware.ThrottleWithOpts.closure_1.closure_1 -> github.com/go-chi/chi/v5/middleware.throttler.setRetryAfterHeaderIfNeeded", + "github.com/go-chi/chi/v5/middleware.URLFormat.closure_1 -> github.com/go-chi/chi/v5.RouteContext", "github.com/go-chi/chi/v5/middleware.basicWriter.Write -> github.com/go-chi/chi/v5/middleware.basicWriter.maybeWriteHeader", "github.com/go-chi/chi/v5/middleware.basicWriter.maybeWriteHeader -> github.com/go-chi/chi/v5/middleware.basicWriter.WriteHeader", "github.com/go-chi/chi/v5/middleware.compressResponseWriter.Close -> github.com/go-chi/chi/v5/middleware.compressResponseWriter.writer", diff --git a/internal/syntactic_analysis/export.go b/internal/syntactic_analysis/export.go index fad49bb..a8d81c8 100644 --- a/internal/syntactic_analysis/export.go +++ b/internal/syntactic_analysis/export.go @@ -13,3 +13,9 @@ func (b *SymbolTableBuilder) Fset() *token.FileSet { return b.fset } // Pkgs returns the map of loaded packages keyed by package path. // Used by CallGraphBuilder for type-info lookups. func (b *SymbolTableBuilder) Pkgs() map[string]*packages.Package { return b.pkgs } + +// Loaded returns every loaded package with types, in load order. Unlike +// Pkgs it keeps every variant of a package path: with tests loaded, both +// the test variant "p [p.test]" and the plain p other packages import. +// L4 builds SSA over all of them (sdg.Input.Variants). +func (b *SymbolTableBuilder) Loaded() []*packages.Package { return b.loaded } diff --git a/internal/syntactic_analysis/l3units.go b/internal/syntactic_analysis/l3units.go index 3cac08d..07772d3 100644 --- a/internal/syntactic_analysis/l3units.go +++ b/internal/syntactic_analysis/l3units.go @@ -1,11 +1,11 @@ package syntactic_analysis import ( - "fmt" "go/ast" "go/types" "sort" + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" @@ -22,14 +22,13 @@ import ( // which the symbol table overwrote, from taking the survivor's key. // Declarations the symbol table dropped (cgo wrappers, methods whose // receiver type was not found) get no unit. -// - A FuncLit is matched to its L1 closure by the traversal that named it: -// buildInnerCallables numbers the outermost literals of a body -// closure_1, closure_2, … in ast.Inspect order and recurses per literal, -// so walking the same way pairs each literal with InnerCallables[name]. -// Its start line is checked too. Its key's Path is the enclosing -// callable's declaring file. +// - A FuncLit is matched to its L1 closure by name: buildInnerCallables +// names the outermost literals of a body with astwalk.Literals and +// recurses per literal, so walking the same way pairs each literal with +// InnerCallables[name]. Its start line is checked too. Its key's Path is +// the enclosing callable's declaring file. // - Calls is v2.CallBodyKeys over the callable's L1 call sites, the exact -// keys the L1 emitter writes. +// keys the L1 emitter writes, each with its site's Lparen. // // A mismatch (which would mean the walk drifted from L1's) is logged and the // callable is left at L1. @@ -73,7 +72,7 @@ func (b *SymbolTableBuilder) L3Units(symbolTable map[string]schema.GoFile) []dat Signature: b.declSignature(pkg, fd), } c, ok := byKey[key] - if !ok || seen[key] || c.StartLine != b.fset.Position(fd.Pos()).Line { + if !ok || seen[key] || c.StartLine != b.position(fd.Pos()).Line { continue } seen[key] = true @@ -92,26 +91,18 @@ func (b *SymbolTableBuilder) L3Units(symbolTable map[string]schema.GoFile) []dat return units } -// closureUnits appends a unit for every function literal of body, mirroring -// buildInnerCallables' naming walk. +// closureUnits appends a unit for every function literal of body, walking +// astwalk.Literals as buildInnerCallables does. func (b *SymbolTableBuilder) closureUnits(units []dataflow.Unit, path string, body *ast.BlockStmt, outer schema.GoCallable, info *types.Info) []dataflow.Unit { - n := 0 - ast.Inspect(body, func(node ast.Node) bool { - lit, ok := node.(*ast.FuncLit) - if !ok { - return true + for _, l := range astwalk.Literals(body) { + ic, ok := outer.InnerCallables[l.Name] + if !ok || ic.StartLine != b.position(l.Lit.Pos()).Line { + utils.Warn("L3 skipped for %s.%s (%s): no matching L1 closure", outer.Signature, l.Name, path) + continue } - n++ - name := fmt.Sprintf("closure_%d", n) - ic, ok := outer.InnerCallables[name] - if !ok || ic.StartLine != b.fset.Position(lit.Pos()).Line { - utils.Warn("L3 skipped for %s.%s (%s): no matching L1 closure", outer.Signature, name, path) - return false - } - units = append(units, unit(dataflow.L3Key{Path: path, Signature: ic.Signature}, lit, info, ic)) - units = b.closureUnits(units, path, lit.Body, ic, info) - return false - }) + units = append(units, unit(dataflow.L3Key{Path: path, Signature: ic.Signature}, l.Lit, info, ic)) + units = b.closureUnits(units, path, l.Lit.Body, ic, info) + } return units } @@ -120,5 +111,10 @@ func unit(key dataflow.L3Key, fn ast.Node, info *types.Info, c schema.GoCallable for i, cs := range c.CallSites { pos[i] = [2]int{cs.StartLine, cs.StartColumn} } - return dataflow.Unit{Key: key, Fn: fn, Info: info, Calls: v2.CallBodyKeys(pos)} + keys := v2.CallBodyKeys(pos) + calls := make([]dataflow.UnitCall, len(c.CallSites)) + for i, cs := range c.CallSites { + calls[i] = dataflow.UnitCall{Key: keys[i], Lparen: cs.Lparen} + } + return dataflow.Unit{Key: key, Fn: fn, Info: info, Calls: calls} } diff --git a/internal/syntactic_analysis/signature.go b/internal/syntactic_analysis/signature.go index cdfd71e..352f48c 100644 --- a/internal/syntactic_analysis/signature.go +++ b/internal/syntactic_analysis/signature.go @@ -6,17 +6,19 @@ import ( "go/types" ) -// signatureOf is the single canonicalizer for all signature strings in the analyzer. +// SignatureOf is the single canonicalizer for all signature strings in the analyzer. // It produces the edge id used in GoCallable.signature, GoType.signature, and every // call-graph edge source/target. All callers must use this function — never build // signatures ad-hoc. Caller-side and callee-side ids are then guaranteed identical. +// The call-graph builder (semantic_analysis.CalleeSignatureOf) and the L4 SSA join +// (internal/sdg) reach it through that one exported entry point. // // Format: // - Package function: "pkg/path.FuncName" // - Method: "pkg/path.TypeName.MethodName" // - Interface method: "pkg/path.InterfaceName.MethodName" // - Type: "pkg/path.TypeName" -func signatureOf(obj types.Object) string { +func SignatureOf(obj types.Object) string { if obj == nil { return "" } @@ -48,4 +50,3 @@ func signatureOf(obj types.Object) string { return fmt.Sprintf("%s.%s", pkgPath, obj.Name()) } } - diff --git a/internal/syntactic_analysis/symbol_table.go b/internal/syntactic_analysis/symbol_table.go index a0e2b35..9ada56b 100644 --- a/internal/syntactic_analysis/symbol_table.go +++ b/internal/syntactic_analysis/symbol_table.go @@ -12,6 +12,7 @@ import ( "golang.org/x/tools/go/packages" + "github.com/codellm-devkit/codeanalyzer-go/internal/astwalk" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" "github.com/codellm-devkit/codeanalyzer-go/internal/utils" ) @@ -24,8 +25,13 @@ import ( type SymbolTableBuilder struct { projectDir string fset *token.FileSet - // pkgs is the flat list of loaded packages, keyed by package path. + // pkgs is the flat list of loaded packages, keyed by package path. With + // tests loaded, a package's test variant "p [p.test]" replaces the plain + // p here (L1–L3 read the variant, which holds every file of p). pkgs map[string]*packages.Package + // loaded is every loaded package with types, in load order: test + // variants and the plain packages they shadow in pkgs alike. + loaded []*packages.Package } // NewSymbolTableBuilder creates a builder for projectDir. @@ -37,6 +43,16 @@ func NewSymbolTableBuilder(projectDir string) *SymbolTableBuilder { } } +// position is p's real file position: //line directives are NOT applied +// (token.FileSet.PositionFor(p, false)). Every line:col the builder records — +// call sites, callable / type / field / import spans — indexes the file that +// holds the node, the file module.source is read from, so spans slice to the +// node's text even in generated code (goyacc, templ) carrying //line +// directives. +func (b *SymbolTableBuilder) position(p token.Pos) token.Position { + return b.fset.PositionFor(p, false) +} + // Build loads all packages under projectDir (or only targetFiles if non-empty), // walks each file, and returns the symbol table keyed by relative file path. func (b *SymbolTableBuilder) Build(targetFiles []string, skipTests bool) (map[string]schema.GoFile, error) { @@ -86,6 +102,7 @@ func (b *SymbolTableBuilder) Build(targetFiles []string, skipTests bool) (map[st } if pkg.Types != nil { b.pkgs[pkg.PkgPath] = pkg + b.loaded = append(b.loaded, pkg) } } tooNew.warn() @@ -231,7 +248,14 @@ func (b *SymbolTableBuilder) buildGoFile( FileSize: fileSize, } - // Walk top-level declarations. + // Walk top-level declarations. A method may be declared above its receiver + // type, so methods are collected here and attached once every type of the + // file is known. + type pendingMethod struct { + typeName string + callable *schema.GoCallable + } + var methods []pendingMethod for _, decl := range astFile.Decls { switch d := decl.(type) { case *ast.GenDecl: @@ -242,18 +266,22 @@ func (b *SymbolTableBuilder) buildGoFile( continue } if d.Recv != nil { - // Method — attach to its type. if typeName := b.receiverTypeName(d.Recv); typeName != "" { - if gt, ok := gf.Types[typeName]; ok { - gt.Methods[callable.Signature] = *callable - gf.Types[typeName] = gt - } + methods = append(methods, pendingMethod{typeName, callable}) } } else { gf.Functions[callable.Signature] = *callable } } } + // Attach in source order. A receiver type in another file is left to + // reconcileCrossFileMethods. + for _, m := range methods { + if gt, ok := gf.Types[m.typeName]; ok { + gt.Methods[m.callable.Signature] = *m.callable + gf.Types[m.typeName] = gt + } + } return gf } @@ -268,8 +296,8 @@ func (b *SymbolTableBuilder) buildImports(astFile *ast.File) []schema.GoImport { if imp.Name != nil { alias = imp.Name.Name } - pos := b.fset.Position(imp.Pos()) - end := b.fset.Position(imp.End()) + pos := b.position(imp.Pos()) + end := b.position(imp.End()) imports = append(imports, schema.GoImport{ Module: path, Alias: alias, @@ -286,8 +314,8 @@ func (b *SymbolTableBuilder) buildFileComments(astFile *ast.File) []schema.GoCom var comments []schema.GoComment for _, cg := range astFile.Comments { for _, c := range cg.List { - pos := b.fset.Position(c.Pos()) - end := b.fset.Position(c.End()) + pos := b.position(c.Pos()) + end := b.position(c.End()) comments = append(comments, schema.GoComment{ Content: c.Text, StartLine: pos.Line, @@ -307,8 +335,8 @@ func (b *SymbolTableBuilder) docComments(doc *ast.CommentGroup) []schema.GoComme } var comments []schema.GoComment for _, c := range doc.List { - pos := b.fset.Position(c.Pos()) - end := b.fset.Position(c.End()) + pos := b.position(c.Pos()) + end := b.position(c.End()) comments = append(comments, schema.GoComment{ Content: c.Text, StartLine: pos.Line, @@ -355,15 +383,15 @@ func (b *SymbolTableBuilder) buildType( var typSig string if pkg.TypesInfo != nil { if obj, ok := pkg.TypesInfo.Defs[spec.Name]; ok && obj != nil { - typSig = signatureOf(obj) + typSig = SignatureOf(obj) } } if typSig == "" { typSig = pkg.Types.Path() + "." + typeName } - pos := b.fset.Position(spec.Pos()) - end := b.fset.Position(spec.End()) + pos := b.position(spec.Pos()) + end := b.position(spec.End()) gt := &schema.GoType{ Name: typeName, @@ -437,8 +465,8 @@ func (b *SymbolTableBuilder) buildStructFields(pkg *packages.Package, st *ast.St isEmbedded := len(field.Names) == 0 if isEmbedded { - pos := b.fset.Position(field.Pos()) - end := b.fset.Position(field.End()) + pos := b.position(field.Pos()) + end := b.position(field.End()) fields = append(fields, schema.GoField{ Name: typStr, Type: typStr, @@ -451,8 +479,8 @@ func (b *SymbolTableBuilder) buildStructFields(pkg *packages.Package, st *ast.St continue } for _, name := range field.Names { - pos := b.fset.Position(name.Pos()) - end := b.fset.Position(field.End()) + pos := b.position(name.Pos()) + end := b.position(field.End()) fields = append(fields, schema.GoField{ Name: name.Name, Type: typStr, @@ -491,8 +519,8 @@ func (b *SymbolTableBuilder) collectInterfaceMethods(pkg *packages.Package, it * continue } for _, name := range method.Names { - pos := b.fset.Position(name.Pos()) - end := b.fset.Position(method.End()) + pos := b.position(name.Pos()) + end := b.position(method.End()) var retType string var retTypes []string @@ -544,8 +572,8 @@ func (b *SymbolTableBuilder) buildCallable( } isExported := unicode.IsUpper(rune(name[0])) - pos := b.fset.Position(decl.Pos()) - end := b.fset.Position(decl.End()) + pos := b.position(decl.Pos()) + end := b.position(decl.End()) sig := b.declSignature(pkg, decl) @@ -561,7 +589,7 @@ func (b *SymbolTableBuilder) buildCallable( retType, retTypes := b.buildReturnTypes(pkg, decl.Type) bodyStart := pos.Line if decl.Body != nil { - bodyStart = b.fset.Position(decl.Body.Pos()).Line + bodyStart = b.position(decl.Body.Pos()).Line } callable := &schema.GoCallable{ @@ -589,18 +617,18 @@ func (b *SymbolTableBuilder) buildCallable( callable.Code = b.nodeSource(decl) callable.CallSites = b.buildCallSites(pkg, decl.Body) callable.LocalVariables = b.buildLocalVars(pkg, decl.Body) - callable.InnerCallables = b.buildInnerCallables(pkg, sig, decl.Body) + callable.InnerCallables = b.buildInnerCallables(pkg, sig, callable.Path, decl.Body) } return callable } -// declSignature is a FuncDecl's callable signature: signatureOf its object, +// declSignature is a FuncDecl's callable signature: SignatureOf its object, // or, without type info, [.].. func (b *SymbolTableBuilder) declSignature(pkg *packages.Package, decl *ast.FuncDecl) string { if pkg.TypesInfo != nil { if obj, ok := pkg.TypesInfo.Defs[decl.Name]; ok && obj != nil { - if sig := signatureOf(obj); sig != "" { + if sig := SignatureOf(obj); sig != "" { return sig } } @@ -626,8 +654,8 @@ func (b *SymbolTableBuilder) buildParams(pkg *packages.Package, ft *ast.FuncType if _, ok := field.Type.(*ast.Ellipsis); ok { isVariadic = true } - pos := b.fset.Position(field.Pos()) - end := b.fset.Position(field.End()) + pos := b.position(field.Pos()) + end := b.position(field.End()) if len(field.Names) == 0 { params = append(params, schema.GoParameter{ Name: "_", Type: typStr, IsVariadic: isVariadic, @@ -684,42 +712,23 @@ func (b *SymbolTableBuilder) joinReturnTypes(parts []string) string { // ─── Call sites ─────────────────────────────────────────────────────────────── -// buildCallSites walks the function body and records every call expression. -// callee_signature is left nil here — the resolver backfills it in the call-graph stage. +// buildCallSites records every call expression of the function body in the +// order astwalk.Calls lists them (the order L3 pairs its call nodes with by +// Lparen). callee_signature is left nil here — the resolver backfills it in +// the call-graph stage. func (b *SymbolTableBuilder) buildCallSites(pkg *packages.Package, body *ast.BlockStmt) []schema.GoCallsite { var sites []schema.GoCallsite - // goroutine / deferred hold the call of each `go` / `defer` statement seen - // so far. Pre-order traversal visits the statement before its call, so the - // CallExpr case can tag it, and continues into the call so calls in its - // function value and arguments (evaluated by the launching goroutine, at - // the statement) are recorded as ordinary sites. The L3 walk - // (dataflow.recordedCalls) must record the same calls in the same order. - goroutine := map[*ast.CallExpr]bool{} - deferred := map[*ast.CallExpr]bool{} - ast.Inspect(body, func(n ast.Node) bool { - switch node := n.(type) { - case *ast.GoStmt: - goroutine[node.Call] = true - case *ast.DeferStmt: - deferred[node.Call] = true - case *ast.FuncLit: - // Closure bodies are handled by buildInnerCallables; don't double-count. - return false - case *ast.CallExpr: - site := b.callExprToSite(pkg, node, goroutine[node]) - if site != nil { - site.IsDeferred = deferred[node] - sites = append(sites, *site) - } - } - return true - }) + for _, c := range astwalk.Calls(body) { + site := b.callExprToSite(pkg, c.Expr, c.Goroutine) + site.IsDeferred = c.Deferred + sites = append(sites, *site) + } return sites } func (b *SymbolTableBuilder) callExprToSite(pkg *packages.Package, call *ast.CallExpr, isGoroutine bool) *schema.GoCallsite { - pos := b.fset.Position(call.Pos()) - end := b.fset.Position(call.End()) + pos := b.position(call.Pos()) + end := b.position(call.End()) var methodName, receiverExpr, receiverType string isConstructor := false @@ -771,6 +780,7 @@ func (b *SymbolTableBuilder) callExprToSite(pkg *packages.Package, call *ast.Cal StartColumn: pos.Column, EndLine: end.Line, EndColumn: end.Column, + Lparen: call.Lparen, } } @@ -784,7 +794,7 @@ func (b *SymbolTableBuilder) buildLocalVars(pkg *packages.Package, body *ast.Blo if node.Tok.String() == ":=" { for i, lhs := range node.Lhs { if ident, ok := lhs.(*ast.Ident); ok { - pos := b.fset.Position(ident.Pos()) + pos := b.position(ident.Pos()) typStr := "" if pkg.TypesInfo != nil { if obj := pkg.TypesInfo.ObjectOf(ident); obj != nil { @@ -807,7 +817,7 @@ func (b *SymbolTableBuilder) buildLocalVars(pkg *packages.Package, body *ast.Blo for _, spec := range gen.Specs { if vs, ok := spec.(*ast.ValueSpec); ok { for i, name := range vs.Names { - pos := b.fset.Position(name.Pos()) + pos := b.position(name.Pos()) typStr := "" if vs.Type != nil { typStr = b.typeString(pkg, vs.Type) @@ -834,27 +844,25 @@ func (b *SymbolTableBuilder) buildLocalVars(pkg *packages.Package, body *ast.Blo return vars } -// buildInnerCallables walks body and collects each FuncLit as a named closure. -// Only the top level is captured; nested closures appear in the closure's own -// InnerCallables (populated when buildCallSites recurses into lit.Body). -func (b *SymbolTableBuilder) buildInnerCallables(pkg *packages.Package, outerSig string, body *ast.BlockStmt) map[string]schema.GoCallable { +// buildInnerCallables collects the outermost function literals of body as +// named closures (astwalk.Literals: closure_1, closure_2, … in ast.Inspect +// order). A nested literal is its enclosing closure's own inner callable, +// built by the recursion on lit.Body. Every closure carries the enclosing +// callable's declaring file as its Path (Go has no cross-file closures), so +// the resolver keys its call sites and edges like any other callable's. +func (b *SymbolTableBuilder) buildInnerCallables(pkg *packages.Package, outerSig, path string, body *ast.BlockStmt) map[string]schema.GoCallable { inner := map[string]schema.GoCallable{} - n := 0 - ast.Inspect(body, func(node ast.Node) bool { - lit, ok := node.(*ast.FuncLit) - if !ok { - return true - } - n++ - name := fmt.Sprintf("closure_%d", n) + for _, l := range astwalk.Literals(body) { + lit, name := l.Lit, l.Name sig := outerSig + "." + name - pos := b.fset.Position(lit.Pos()) - endPos := b.fset.Position(lit.End()) + pos := b.position(lit.Pos()) + endPos := b.position(lit.End()) _, retTypes := b.buildReturnTypes(pkg, lit.Type) retType := b.joinReturnTypes(retTypes) ic := schema.GoCallable{ Name: name, + Path: path, Signature: sig, Parameters: b.buildParams(pkg, lit.Type), ReturnType: retType, @@ -869,11 +877,10 @@ func (b *SymbolTableBuilder) buildInnerCallables(pkg *packages.Package, outerSig ic.Code = b.nodeSource(lit) ic.CallSites = b.buildCallSites(pkg, lit.Body) ic.LocalVariables = b.buildLocalVars(pkg, lit.Body) - ic.InnerCallables = b.buildInnerCallables(pkg, sig, lit.Body) + ic.InnerCallables = b.buildInnerCallables(pkg, sig, path, lit.Body) } inner[name] = ic - return false // don't recurse; nested closures are handled above - }) + } return inner } @@ -895,7 +902,7 @@ func (b *SymbolTableBuilder) buildPackageVars(pkg *packages.Package, astFile *as continue } for i, name := range vs.Names { - pos := b.fset.Position(name.Pos()) + pos := b.position(name.Pos()) typStr := "" if vs.Type != nil { typStr = b.typeString(pkg, vs.Type) @@ -1012,7 +1019,7 @@ func (b *SymbolTableBuilder) nodeSource(node ast.Node) string { if tf == nil || b.fset.File(node.End()) != tf { return "" } - pos := b.fset.Position(node.Pos()) + pos := b.position(node.Pos()) if pos.Filename == "" { return "" } diff --git a/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go b/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go index f178a3e..f89fb6a 100644 --- a/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go +++ b/internal/syntactic_analysis/v2emit/determinism_freshbuild_test.go @@ -64,7 +64,7 @@ func buildChiL2Fresh(t *testing.T, dir string) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} - return Emit(app, "chi", dir, 2, "gate-test", 0, nil) + return Emit(app, "chi", dir, 2, "gate-test", 0, nil, nil) } // TestDeterminism_FreshBuildsByteIdentical locks that independent from-disk diff --git a/internal/syntactic_analysis/v2emit/determinism_test.go b/internal/syntactic_analysis/v2emit/determinism_test.go index b7d4d81..e87e9cf 100644 --- a/internal/syntactic_analysis/v2emit/determinism_test.go +++ b/internal/syntactic_analysis/v2emit/determinism_test.go @@ -14,9 +14,14 @@ package v2emit // fails here rather than only in an external CLI diff. import ( + "bytes" "encoding/json" + "fmt" + "path/filepath" + "runtime" "testing" + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" @@ -41,7 +46,7 @@ func TestDeterminism_CallGraphStableAcrossEmits(t *testing.T) { // edges slice without a stable sort, repeated marshals still match (same // slice), so to actually exercise order-independence we marshal a shuffled // copy of the edge slice too. - first := mustMarshalCallGraph(t, Emit(app, "multipackage", dir, 2, "det-test", 0, nil)) + first := mustMarshalCallGraph(t, Emit(app, "multipackage", dir, 2, "det-test", 0, nil, nil)) // Reverse the input edge order and re-emit: a correct emitter sorts, so the // serialized call_graph must be unchanged. @@ -50,7 +55,7 @@ func TestDeterminism_CallGraphStableAcrossEmits(t *testing.T) { rev[len(edges)-1-i] = e } appRev := &schema.GoApplication{SymbolTable: st, CallGraph: rev} - second := mustMarshalCallGraph(t, Emit(appRev, "multipackage", dir, 2, "det-test", 0, nil)) + second := mustMarshalCallGraph(t, Emit(appRev, "multipackage", dir, 2, "det-test", 0, nil, nil)) if first != second { t.Errorf("call_graph not order-independent: emitCallGraph must sort edges to a total order\nfirst=%s\nsecond=%s", first, second) @@ -74,3 +79,120 @@ func mustMarshalCallGraph(t *testing.T, out *v2.Analysis) string { } return string(c.Application.CallGraph) } + +// TestEncode_ByteIdenticalToMarshal locks the streaming writers to +// json.Marshal byte for byte (#27): sorted map keys, omitempty / omitzero, +// the call-node-only callee, HTML escaping (module sources carry `<-` and +// `&`), and no trailing newline. Encode streams a built tree; EmitTo builds +// and streams one module at a time. Both must equal json.Marshal(Emit(...)). +func TestEncode_ByteIdenticalToMarshal(t *testing.T) { + for _, name := range []string{"chi", "dataflow", "generics"} { + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata", name)) + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("%s: symbol table build failed: %v", name, err) + } + apps := map[int]*schema.GoApplication{1: {SymbolTable: st, CallGraph: []schema.GoCallEdge{}}} + check := func(level int, app *schema.GoApplication, l3 map[dataflow.L3Key]dataflow.Result) { + t.Run(fmt.Sprintf("%s/a%d", name, level), func(t *testing.T) { + tree := Emit(app, name, dir, level, "enc-test", 3, l3, nil) + want, err := json.Marshal(tree) + if err != nil { + t.Fatal(err) + } + var enc bytes.Buffer + if err := Encode(&enc, tree); err != nil { + t.Fatal(err) + } + if !bytes.Equal(enc.Bytes(), want) { + t.Errorf("Encode differs from json.Marshal at byte %d (len %d vs %d)", firstDiff(enc.Bytes(), want), enc.Len(), len(want)) + } + var lazy bytes.Buffer + if err := EmitTo(&lazy, app, name, dir, level, "enc-test", 3, l3, nil); err != nil { + t.Fatal(err) + } + if !bytes.Equal(lazy.Bytes(), want) { + t.Errorf("EmitTo differs from json.Marshal(Emit) at byte %d (len %d vs %d)", firstDiff(lazy.Bytes(), want), lazy.Len(), len(want)) + } + }) + } + check(1, apps[1], nil) + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + check(2, app, nil) + check(3, app, dataflow.Run(b.Fset(), b.L3Units(st), 3, 4)) + } +} + +func firstDiff(a, b []byte) int { + for i := 0; i < len(a) && i < len(b); i++ { + if a[i] != b[i] { + return i + } + } + return min(len(a), len(b)) +} + +// TestEncode_L4EnvelopeFieldsByteIdentical: the L4 envelope and tree fields +// (degraded after application, param_in/out inside it, summary, of, receiver, +// external body) go through the streaming writer's envelope split unchanged. +func TestEncode_L4EnvelopeFieldsByteIdentical(t *testing.T) { + call := "x" + a := &v2.Analysis{ + SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: 4, KLimit: 3, + Analyzer: v2.AnalyzerTag{Name: "codeanalyzer-go", Version: "t"}, + Degraded: []v2.Degraded{{Code: "ssa_build", Detail: `has "symbol_table":{} & `, Packages: []string{"p"}, Callables: []string{"c"}}}, + Application: v2.Application{ + ID: "can://app", Kind: "application", + SymbolTable: map[string]v2.Module{ + "a.go": {ID: "m", Functions: map[string]v2.Callable{"f": { + ID: "f", Summary: []v2.SummaryEdge{{Src: "a", Dst: "b"}}, + Body: map[string]v2.BodyNode{ + "@fin": {Kind: "formal_in", Of: "p0"}, + "c1": {Kind: "call", Callee: &call}, + }, + }}}, + }, + ExternalSymbols: map[string]v2.ExternalSymbol{"e": { + ID: "e", Kind: "external", + Receiver: &v2.ExternalParameter{Name: "_recv", Type: "*strings.Builder"}, + Body: map[string]v2.BodyNode{"@entry": {Kind: "entry"}}, + }}, + ParamIn: []v2.ParamEdge{{Src: "s", Dst: "d", Var: "v"}}, + ParamOut: []v2.ParamEdge{{Src: "d", Dst: "s", Var: ""}}, + }, + } + want, err := json.Marshal(a) + if err != nil { + t.Fatal(err) + } + for _, k := range []string{`"degraded":`, `"param_in":`, `"param_out":`, `"summary":`, `"of":"p0"`, `"receiver":`} { + if !bytes.Contains(want, []byte(k)) { + t.Errorf("marshalled tree lacks %s", k) + } + } + var enc bytes.Buffer + if err := Encode(&enc, a); err != nil { + t.Fatal(err) + } + if !bytes.Equal(enc.Bytes(), want) { + t.Errorf("Encode differs from json.Marshal at byte %d", firstDiff(enc.Bytes(), want)) + } + // And absent when empty. + plain, _ := json.Marshal(&v2.Analysis{}) + for _, k := range []string{"degraded", "param_in", "param_out", "summary", `"of"`, "receiver"} { + if bytes.Contains(plain, []byte(k)) { + t.Errorf("empty analysis serialises %s", k) + } + } + n, _ := json.Marshal(v2.BodyNode{Kind: "statement"}) + if bytes.Contains(n, []byte(`"of"`)) { + t.Errorf("empty Of serialised: %s", n) + } +} diff --git a/internal/syntactic_analysis/v2emit/emit.go b/internal/syntactic_analysis/v2emit/emit.go index 888f8b6..8ff0b1e 100644 --- a/internal/syntactic_analysis/v2emit/emit.go +++ b/internal/syntactic_analysis/v2emit/emit.go @@ -1,14 +1,22 @@ package v2emit import ( + "bytes" + "encoding/json" + "fmt" + "io" + "iter" + "maps" "os" "path/filepath" + "slices" "sort" "strings" "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" "github.com/codellm-devkit/codeanalyzer-go/internal/schema" v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" ) // Emit transforms the v1 GoApplication into the v2 Analysis payload. appName is @@ -19,54 +27,242 @@ import ( // // kLimit is the L3 access-path bound (--graph-field-depth), written as the // envelope's k_limit at maxLevel >= 3 only. l3 is dataflow.Run's side table, -// nil below L3; see emit_l3.go for how it merges into each callable. +// nil below L3; see emit_l3.go for how it merges into each callable. l4 is +// sdg.Run's result, nil below L4 (and ignored there); see emit_l4.go for how +// it merges into the tree and mints every L4 id. // // This is a pure re-serialization: every fact comes from the v1 model and the -// L3 side table, except span byte offsets, which are derived from source via -// lineIndex. -func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, analyzerVersion string, kLimit int, l3 map[dataflow.L3Key]dataflow.Result) *v2.Analysis { - appID := v2.AppID(appName) +// L3/L4 side tables, except span byte offsets, which are derived from source +// via lineIndex. +func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, analyzerVersion string, kLimit int, l3 map[dataflow.L3Key]dataflow.Result, l4 *sdg.Result) *v2.Analysis { + e := newEmitter(app, appName, projectDir, maxLevel, l3, l4) + out := e.envelope(app, maxLevel, analyzerVersion, kLimit) + out.Application.SymbolTable = make(map[string]v2.Module, len(app.SymbolTable)) + for _, relPath := range sortedFileKeys(app.SymbolTable) { + out.Application.SymbolTable[relPath] = e.module(relPath, app.SymbolTable[relPath]) + } + late := e.sdg.late() + out.Application.ParamIn = collectEdges(late.paramIn) + out.Application.ParamOut = collectEdges(late.paramOut) + out.Degraded = late.degraded + return out +} - // The signature→can:// id index translates v1 identity (signature strings) - // into v2 node ids. It is what the L2 refinement uses to backfill body - // callees and to map call_graph edge endpoints onto real callable ids. +// EmitTo writes the analysis.json bytes of Emit's payload to w — the same bytes +// json.Marshal(Emit(...)) produces — without building the whole tree: each +// module is emitted, marshalled and dropped in symbol_table key order, so peak +// memory is the envelope plus the largest module, not the document. The L4 +// param edges follow the modules (they need every callable's id) and are +// minted and written one at a time. +func EmitTo(w io.Writer, app *schema.GoApplication, appName, projectDir string, maxLevel int, analyzerVersion string, kLimit int, l3 map[dataflow.L3Key]dataflow.Result, l4 *sdg.Result) error { + e := newEmitter(app, appName, projectDir, maxLevel, l3, l4) + out := e.envelope(app, maxLevel, analyzerVersion, kLimit) + return encode(w, out, func(yield func(string, v2.Module) bool) { + for _, relPath := range sortedFileKeys(app.SymbolTable) { + if !yield(relPath, e.module(relPath, app.SymbolTable[relPath])) { + return + } + } + }, e.sdg.late) +} + +// collectEdges is seq as a slice, nil when empty (omitempty). +func collectEdges(seq iter.Seq[v2.ParamEdge]) []v2.ParamEdge { + if seq == nil { + return nil + } + var out []v2.ParamEdge + for e := range seq { + out = append(out, e) + } + return out +} + +// Encode writes a to w as the exact bytes json.Marshal(a) produces (no +// trailing newline), streaming symbol_table one module at a time so the whole +// document never exists as one []byte. +func Encode(w io.Writer, a *v2.Analysis) error { + st := a.Application.SymbolTable + if st == nil { // json.Marshal writes null; nothing to stream + data, err := json.Marshal(a) + if err != nil { + return err + } + _, err = w.Write(data) + return err + } + keys := make([]string, 0, len(st)) + for k := range st { + keys = append(keys, k) + } + sort.Strings(keys) + env := *a + env.Application.SymbolTable = nil + return encode(w, &env, func(yield func(string, v2.Module) bool) { + for _, k := range keys { + if !yield(k, st[k]) { + return + } + } + }, func() lateFields { + lf := lateFields{degraded: a.Degraded} + if len(a.Application.ParamIn) > 0 { + lf.paramIn = slices.Values(a.Application.ParamIn) + } + if len(a.Application.ParamOut) > 0 { + lf.paramOut = slices.Values(a.Application.ParamOut) + } + return lf + }) +} + +// symbolTableHole is the envelope's symbol_table as json.Marshal writes it +// when the map is empty. Every field before it is a number or a JSON string; +// inside a string every '"' is written as \", and the needle's second '"' +// follows 'e', so its first match is the field itself. +const symbolTableHole = `"symbol_table":{}` + +// encode writes env (its SymbolTable, ParamIn, ParamOut and Degraded +// ignored) with symbol_table filled from modules, which must yield keys in +// sort.Strings order — the order json.Marshal writes a map[string] in — and +// then the fields late returns, called once the modules are written: +// param_in and param_out close the application (its last two fields) and +// degraded closes the envelope (its last field), as json.Marshal orders +// them. Keys are marshalled as JSON strings, which is how encoding/json +// writes a string map key (same escaping). +func encode(w io.Writer, env *v2.Analysis, modules iter.Seq2[string, v2.Module], late func() lateFields) error { + shell := *env + shell.Application.SymbolTable = map[string]v2.Module{} + shell.Application.ParamIn, shell.Application.ParamOut, shell.Degraded = nil, nil, nil + data, err := json.Marshal(&shell) + if err != nil { + return err + } + i := bytes.Index(data, []byte(symbolTableHole)) + if i < 0 { + return fmt.Errorf("v2emit: no %s in the envelope", symbolTableHole) + } + // With param_in, param_out and degraded omitted, the shell ends with the + // application's closing '}' and then the envelope's. + if !bytes.HasSuffix(data, []byte("}}")) { + return fmt.Errorf("v2emit: the envelope does not end with the application") + } + split := i + len(symbolTableHole) - 1 // at the closing '}' + if _, err := w.Write(data[:split]); err != nil { + return err + } + first := true + for k, m := range modules { + if err := writeEntry(w, k, m, first); err != nil { + return err + } + first = false + } + if _, err := w.Write(data[split : len(data)-2]); err != nil { + return err + } + lf := late() + if err := writeParamList(w, "param_in", lf.paramIn); err != nil { + return err + } + if err := writeParamList(w, "param_out", lf.paramOut); err != nil { + return err + } + if _, err := io.WriteString(w, "}"); err != nil { + return err + } + if len(lf.degraded) > 0 { + deg, err := json.Marshal(lf.degraded) + if err != nil { + return err + } + if _, err := io.WriteString(w, `,"degraded":`); err != nil { + return err + } + if _, err := w.Write(deg); err != nil { + return err + } + } + _, err = io.WriteString(w, "}") + return err +} + +func writeEntry(w io.Writer, key string, m v2.Module, first bool) error { + k, err := json.Marshal(key) + if err != nil { + return err + } + v, err := json.Marshal(m) + if err != nil { + return err + } + buf := make([]byte, 0, len(k)+len(v)+2) + if !first { + buf = append(buf, ',') + } + buf = append(buf, k...) + buf = append(buf, ':') + if _, err := w.Write(buf); err != nil { + return err + } + _, err = w.Write(v) + return err +} + +// emitter is the per-run state Emit and EmitTo share. +// +// The signature→can:// id index (idx) translates v1 identity (signature +// strings) into v2 node ids. It is what the L2 refinement uses to backfill +// body callees and to map call_graph edge endpoints onto real callable ids. +// +// srcs holds one lineIndex per source file, read once and shared. A method +// whose receiver type lives in a different file is nested under that type's +// module but must have its span computed against ITS OWN file (CLAUDE.md § +// Schema decisions → callable.source_file); the cache lets emitCallable reach +// any file's index without re-reading it per callable. +type emitter struct { + appID string + idx sigIndex + srcs *sourceCache + flow l3Overlay + sdg *l4Overlay // nil below L4 +} + +func newEmitter(app *schema.GoApplication, appName, projectDir string, maxLevel int, l3 map[dataflow.L3Key]dataflow.Result, l4 *sdg.Result) *emitter { + appID := v2.AppID(appName) idx := buildSigIndex(appID, app.SymbolTable) idx.externals = externalIDs(appID, app.Externals) + return &emitter{ + appID: appID, idx: idx, srcs: newSourceCache(projectDir), + flow: newL3Overlay(maxLevel, l3), sdg: newL4Overlay(maxLevel, l4, idx), + } +} - // One lineIndex per source file, read once and shared. A method whose - // receiver type lives in a different file is nested under that type's module - // but must have its span computed against ITS OWN file (CLAUDE.md § Schema - // decisions → callable.source_file); the cache lets emitCallable reach any - // file's index without re-reading it per callable. - srcs := newSourceCache(projectDir) - flow := newL3Overlay(maxLevel, l3) - +// envelope is the payload without its symbol_table (nil): the manifest, the +// call graph and the external symbols. +func (e *emitter) envelope(app *schema.GoApplication, maxLevel int, analyzerVersion string, kLimit int) *v2.Analysis { out := &v2.Analysis{ SchemaVersion: v2.SchemaVersion, Language: v2.Language, MaxLevel: maxLevel, Analyzer: v2.AnalyzerTag{Name: "codeanalyzer-go", Version: analyzerVersion}, Application: v2.Application{ - ID: appID, - Kind: "application", - SymbolTable: make(map[string]v2.Module, len(app.SymbolTable)), - CallGraph: emitCallGraph(app.CallGraph, idx), + ID: e.appID, + Kind: "application", + CallGraph: emitCallGraph(app.CallGraph, e.idx), }, } - out.Application.ExternalSymbols = emitExternalSymbols(appID, app.Externals) - if flow.on { + out.Application.ExternalSymbols = emitExternalSymbols(e.appID, app.Externals, e.sdg.standIns()) + if e.flow.on { out.KLimit = kLimit } - - // Deterministic module order does not affect the JSON map, but iterating - // sorted keeps any incidental ordering (and logs) stable across runs. - for _, relPath := range sortedFileKeys(app.SymbolTable) { - file := app.SymbolTable[relPath] - out.Application.SymbolTable[relPath] = emitModule(appID, relPath, file, idx, srcs, flow) - } return out } +func (e *emitter) module(relPath string, file schema.GoFile) v2.Module { + return emitModule(e.appID, relPath, file, e.idx, e.srcs, e.flow, e.sdg) +} + // emitModule builds a v2 module from a v1 GoFile, reading the file's source so // spans can slice off it. A file that cannot be read still emits its structure // (with empty source and zeroed byte spans) rather than dropping the module. @@ -74,7 +270,7 @@ func Emit(app *schema.GoApplication, appName, projectDir string, maxLevel int, a // modRel is this module's own file path (a symbol_table key). It is threaded to // callables so a method declared in another file can be detected and given a // span against its own source plus a source_file pointer. -func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *sourceCache, flow l3Overlay) v2.Module { +func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *sourceCache, flow l3Overlay, l4 *l4Overlay) v2.Module { li := srcs.index(modRel) source := li.source modID := v2.ModuleID(appID, modRel) @@ -94,10 +290,10 @@ func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *so } for name, t := range file.Types { - mod.Types[name] = emitType(modID, modRel, t, idx, srcs, flow) + mod.Types[name] = emitType(modID, modRel, t, idx, srcs, flow, l4) } for sig, fn := range file.Functions { - mod.Functions[sig] = emitCallable(modID, modRel, modRel, li, fn, idx, srcs, flow) + mod.Functions[sig] = emitCallable(modID, modRel, modRel, li, fn, idx, srcs, flow, l4) } return mod } @@ -106,7 +302,7 @@ func emitModule(appID, modRel string, file schema.GoFile, idx sigIndex, srcs *so // kind and splitting embedding (base_types) from computed satisfaction. Methods // resolve into the type's callables{}. modRel is the module the type lives in; // its methods may be declared in other files (handled in emitCallable). -func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceCache, flow l3Overlay) v2.Type { +func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceCache, flow l3Overlay, l4 *l4Overlay) v2.Type { li := srcs.index(modRel) typeID := v2.TypeID(modID, t.Signature) out := v2.Type{ @@ -128,7 +324,7 @@ func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceC } } for sig, m := range t.Methods { - out.Callables[sig] = emitCallable(typeID, modRel, modRel, li, m, idx, srcs, flow) + out.Callables[sig] = emitCallable(typeID, modRel, modRel, li, m, idx, srcs, flow, l4) } return out } @@ -141,8 +337,8 @@ func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceC // module its outermost enclosing callable nests under). declRel and declLI are // the file the callable's positions are relative to and its lineIndex: the // nesting file for a top-level callable, the enclosing callable's declaring -// file for a closure (Go has no cross-file closures; v1 closures carry no -// Path). A callable's own c.Path, when set, is its declaring file. +// file for a closure (Go has no cross-file closures; a v1 closure's Path is +// that same file). A callable's own c.Path, when set, is its declaring file. // // When the declaring file differs from the containing module — a method whose // receiver type is in another file, and every closure (nested ones included) @@ -155,7 +351,10 @@ func emitType(modID, modRel string, t schema.GoType, idx sigIndex, srcs *sourceC // signature), completes its body and adds cfg/cdg/ddg (emit_l3.go). Its spans // use the same declaring-file lineIndex, so a cross-file callable's statement // spans index its declaring file's source, like its own span. -func emitCallable(parentID, modRel, declRel string, declLI *lineIndex, c schema.GoCallable, idx sigIndex, srcs *sourceCache, flow l3Overlay) v2.Callable { +// +// At L4 the same key finds the callable's SDG overlay (emit_l4.go), applied +// after the L3 one, and records the callable's id for the param edges. +func emitCallable(parentID, modRel, declRel string, declLI *lineIndex, c schema.GoCallable, idx sigIndex, srcs *sourceCache, flow l3Overlay, l4 *l4Overlay) v2.Callable { callID := v2.CallableID(parentID, c.Signature) // Resolve the file this callable's positions are relative to. @@ -190,10 +389,12 @@ func emitCallable(parentID, modRel, declRel string, declLI *lineIndex, c schema. // Closures nest in the same containing module and are declared in the // same file as their enclosing callable. for sig, inner := range c.InnerCallables { - out.Callables[sig] = emitCallable(callID, modRel, declRel, li, inner, idx, srcs, flow) + out.Callables[sig] = emitCallable(callID, modRel, declRel, li, inner, idx, srcs, flow, l4) } } - flow.apply(&out, li, dataflow.L3Key{Path: declRel, Signature: c.Signature}) + key := dataflow.L3Key{Path: declRel, Signature: c.Signature} + flow.apply(&out, li, key) + l4.apply(&out, key) return out } @@ -262,22 +463,39 @@ func externalIDs(appID string, externals map[string]schema.GoExternalCallee) map } // emitExternalSymbols builds application.external_symbols, keyed by @external -// id. Nil (omitted from JSON) when no library callee was recorded. -func emitExternalSymbols(appID string, externals map[string]schema.GoExternalCallee) map[string]v2.ExternalSymbol { +// id. Nil (omitted from JSON) when no library callee was recorded. At L4 +// standIns (by full callee signature) gives each stand-in its body: its +// formal vertices, id v2.GlobalOrdinal(external id, key). +func emitExternalSymbols(appID string, externals map[string]schema.GoExternalCallee, standIns map[string][]sdg.Vertex) map[string]v2.ExternalSymbol { + for _, sig := range slices.Sorted(maps.Keys(standIns)) { + if _, ok := externals[sig]; !ok { + warnf("L4 emit: stand-in %s is not an external symbol; its body is dropped", sig) + } + } if len(externals) == 0 { return nil } out := make(map[string]v2.ExternalSymbol, len(externals)) - for _, ext := range externals { + for fullSig, ext := range externals { id := v2.ExternalID(appID, ext.ImportPath, ext.Sig) params := make([]v2.ExternalParameter, 0, len(ext.Params)) for _, p := range ext.Params { params = append(params, v2.ExternalParameter{Name: p.Name, Type: p.Type, IsVariadic: p.IsVariadic}) } - out[id] = v2.ExternalSymbol{ + sym := v2.ExternalSymbol{ ID: id, Kind: "external", Name: ext.Name, Module: ext.ImportPath, Signature: ext.Sig, Parameters: params, ReturnType: ext.ReturnType, } + if ext.Receiver != nil { + sym.Receiver = &v2.ExternalParameter{Name: ext.Receiver.Name, Type: ext.Receiver.Type} + } + if vs := standIns[fullSig]; len(vs) > 0 { + sym.Body = make(map[string]v2.BodyNode, len(vs)) + for _, v := range vs { + sym.Body[v.Key] = vertexNode(id, v) + } + } + out[id] = sym } return out } @@ -320,10 +538,15 @@ func emitCallGraph(edges []schema.GoCallEdge, idx sigIndex) []v2.Edge { // Resolve the source by (signature, declaring file): a package-level // init signature repeats once per file in a package, so the file is // what pins the edge to the init that actually contains the call site. - // The target stays signature-only — a call target is never an init - // (Go forbids calling init), and every other signature is unique. + // The target is signature-only unless the edge carries its declaring + // file: a static call target is never an init (Go forbids calling + // init), but a VTA target may be a closure of one, whose signature + // repeats per file. src, srcOK := idx.idForWithPath(e.Source, e.SourcePath) dst, dstOK := idx.resolve(e.Target) + if e.TargetPath != "" { + dst, dstOK = idx.idForWithPath(e.Target, e.TargetPath) + } if !srcOK || !dstOK { continue } diff --git a/internal/syntactic_analysis/v2emit/emit_l3.go b/internal/syntactic_analysis/v2emit/emit_l3.go index 3805138..bc43c28 100644 --- a/internal/syntactic_analysis/v2emit/emit_l3.go +++ b/internal/syntactic_analysis/v2emit/emit_l3.go @@ -63,8 +63,9 @@ func (o l3Overlay) apply(c *v2.Callable, li *lineIndex, key dataflow.L3Key) { // l3BodyNode is a non-L1 body node: its kind, and its span from the node's // [line, col] range (zero for @entry/@exit). A call node reaching here is a -// call L1 did not key — impossible when Run's count check passed — and keeps -// the call shape with a null callee. +// call L1 did not key — impossible once Run paired every body call with an +// L1 key by Lparen (a unit with an unpaired call gets no L3 result) — and +// keeps the call shape with a null callee. func l3BodyNode(n dataflow.Node, li *lineIndex) v2.BodyNode { bn := v2.BodyNode{Kind: string(n.Kind), Parent: n.Parent} if n.Kind != dataflow.KindEntry && n.Kind != dataflow.KindExit { diff --git a/internal/syntactic_analysis/v2emit/emit_l4.go b/internal/syntactic_analysis/v2emit/emit_l4.go new file mode 100644 index 0000000..feb3df3 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/emit_l4.go @@ -0,0 +1,326 @@ +package v2emit + +import ( + "cmp" + "encoding/json" + "fmt" + "io" + "iter" + "slices" + "strings" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" + "github.com/codellm-devkit/codeanalyzer-go/internal/utils" +) + +// warnf reports an L4 fact the emitter cannot place (a test seam). +var warnf = utils.Warn + +// l4Overlay merges sdg.Run's result into the emitted tree (spec §4.6, §4.7, +// §4.15). It is nil below -a 4, and then the tree is exactly the L3 one. +// +// The sdg IR holds no ids; this overlay mints every one of them: +// - a callable's vertices land in its body under their local keys, with +// id v2.GlobalOrdinal(callable id, key), kind, of and parent (actuals +// only); no span and, not being call nodes, no callee key; +// - its port edges join its ddg with prov [Prov]; +// - its summary edges are its summary; +// - a stand-in's formals land in its external_symbols entry's body, with +// id v2.GlobalOrdinal(external id, key) — @formal_in:; +// - param_in / param_out endpoints and the degraded callables resolve +// through ids, the sdg.Target → id map filled in the same walk that +// applies the L3 overlay (externals are known up front, from their +// import path and signature). +// +// A Ref or degraded callable that names no emitted node is an upstream bug: +// it is dropped with a warning, never emitted dangling. +type l4Overlay struct { + res *sdg.Result + ids map[sdg.Target]string + // unresolved records the targets already warned about. + unresolved map[sdg.Target]bool +} + +func newL4Overlay(maxLevel int, res *sdg.Result, idx sigIndex) *l4Overlay { + if maxLevel < 4 || res == nil { + return nil + } + o := &l4Overlay{ + res: res, + ids: make(map[sdg.Target]string, len(res.Callables)+len(idx.externals)), + unresolved: map[sdg.Target]bool{}, + } + for sig, id := range idx.externals { + o.ids[sdg.Target{External: sig}] = id + } + return o +} + +// apply records c's id under its target key (the callable's declaring file +// and signature, the key the L3 overlay uses) and adds its L4 overlay: the +// vertices to its body, the port edges to its ddg (re-sorted by (src, dst, +// var, prov)), and its summary. +func (o *l4Overlay) apply(c *v2.Callable, key dataflow.L3Key) { + if o == nil { + return + } + // Two callables sharing a target key route its param edges and degraded + // entries to the lexically smaller id: emission order walks maps, so + // last-write-wins would not be deterministic. + t := sdg.Target{Callable: key} + prev, dup := o.ids[t] + if dup && prev != c.ID { + warnf("L4 emit: %s and %s share the target key %s %s", prev, c.ID, key.Path, key.Signature) + } + if !dup || c.ID < prev { + o.ids[t] = c.ID + } + r, ok := o.res.Callables[key] + if !ok { + return + } + for _, v := range r.Vertices { + if _, clash := c.Body[v.Key]; clash { + warnf("L4 emit: vertex %s of %s collides with a body node; dropped", v.Key, c.ID) + continue + } + c.Body[v.Key] = vertexNode(c.ID, v) + } + if len(r.Ddg) > 0 { + ddg := slices.Grow(c.Ddg, len(r.Ddg)) + for _, e := range r.Ddg { + ddg = append(ddg, v2.DdgEdge{Src: e.Src, Dst: e.Dst, Var: e.Var, Prov: []string{e.Prov}}) + } + slices.SortStableFunc(ddg, compareDdg) + c.Ddg = ddg + } + if len(r.Summary) > 0 { + c.Summary = make([]v2.SummaryEdge, len(r.Summary)) + for i, e := range r.Summary { + c.Summary[i] = v2.SummaryEdge{Src: e.Src, Dst: e.Dst} + } + slices.SortFunc(c.Summary, func(a, b v2.SummaryEdge) int { + return cmp.Or(strings.Compare(a.Src, b.Src), strings.Compare(a.Dst, b.Dst)) + }) + } +} + +// vertexNode is the body node of vertex v of the callable or stand-in owner. +func vertexNode(owner string, v sdg.Vertex) v2.BodyNode { + return v2.BodyNode{Kind: string(v.Kind), ID: v2.GlobalOrdinal(owner, v.Key), Parent: v.Parent, Of: v.Of} +} + +// compareDdg orders a callable's ddg by (src, dst, var, prov). L3's own +// list is sorted by (src, dst, var) with prov ["ssa"] throughout, so the +// L3 edges keep their relative order. +func compareDdg(a, b v2.DdgEdge) int { + return cmp.Or( + strings.Compare(a.Src, b.Src), strings.Compare(a.Dst, b.Dst), strings.Compare(a.Var, b.Var), + slices.Compare(a.Prov, b.Prov), + ) +} + +// standIns returns the stand-in formals by full callee signature (nil +// below -a 4). +func (o *l4Overlay) standIns() map[string][]sdg.Vertex { + if o == nil { + return nil + } + return o.res.Externals +} + +// late returns the fields written after the symbol table — param_in, +// param_out and degraded — which need every callable's id, so they are +// computed once the module walk has filled ids. +func (o *l4Overlay) late() lateFields { + if o == nil { + return lateFields{} + } + return lateFields{ + paramIn: o.paramEdges("param_in", o.res.ParamIn), + paramOut: o.paramEdges("param_out", o.res.ParamOut), + degraded: o.degraded(), + } +} + +// mintedEdge is a param edge with its owners' ids resolved: its minted +// endpoints are GlobalOrdinal(src, e.Src.Key) and GlobalOrdinal(dst, +// e.Dst.Key), built only when the edge is written. +type mintedEdge struct { + src, dst string + e *sdg.ParamEdge +} + +// paramEdges returns list's edges with minted endpoints, sorted by (src, +// dst, var) over the minted ids, duplicates dropped. The sort works on the +// owner ids and local keys (compareOrdinal compares the joined ids without +// building them), so the whole list is never minted at once: on the +// streaming path each edge's ids exist only while it is written. +func (o *l4Overlay) paramEdges(list string, edges []sdg.ParamEdge) iter.Seq[v2.ParamEdge] { + ms := make([]mintedEdge, 0, len(edges)) + for i := range edges { + e := &edges[i] + src, okS := o.resolve(list, e.Src.Owner) + dst, okD := o.resolve(list, e.Dst.Owner) + if okS && okD { + ms = append(ms, mintedEdge{src: src, dst: dst, e: e}) + } + } + compare := func(a, b mintedEdge) int { + return cmp.Or( + compareOrdinal(a.src, a.e.Src.Key, b.src, b.e.Src.Key), + compareOrdinal(a.dst, a.e.Dst.Key, b.dst, b.e.Dst.Key), + strings.Compare(a.e.Var, b.e.Var), + ) + } + slices.SortFunc(ms, compare) + return func(yield func(v2.ParamEdge) bool) { + for i, m := range ms { + if i > 0 && compare(ms[i-1], m) == 0 { + continue + } + if !yield(v2.ParamEdge{ + Src: v2.GlobalOrdinal(m.src, m.e.Src.Key), + Dst: v2.GlobalOrdinal(m.dst, m.e.Dst.Key), + Var: m.e.Var, + }) { + return + } + } + } +} + +// resolve is t's id, warning once per target that names no emitted node. +func (o *l4Overlay) resolve(what string, t sdg.Target) (string, bool) { + if id, ok := o.ids[t]; ok { + return id, true + } + if !o.unresolved[t] { + o.unresolved[t] = true + warnf("L4 emit: %s names %s, which is not in the tree; its edges are dropped", what, describeTarget(t)) + } + return "", false +} + +func describeTarget(t sdg.Target) string { + if t.External != "" { + return "external " + t.External + } + return fmt.Sprintf("callable %s %s", t.Callable.Path, t.Callable.Signature) +} + +// degraded mints the degradation markers: each callable as its id (sorted, +// unique), entries sorted by (code, detail); nil when nothing degraded. +func (o *l4Overlay) degraded() []v2.Degraded { + if len(o.res.Degraded) == 0 { + return nil + } + out := make([]v2.Degraded, 0, len(o.res.Degraded)) + for _, d := range o.res.Degraded { + v := v2.Degraded{Code: d.Code, Detail: d.Detail, Packages: nonEmpty(d.Packages)} + for _, k := range d.Callables { + if id, ok := o.resolve("degraded "+d.Code, sdg.Target{Callable: k}); ok { + v.Callables = append(v.Callables, id) + } + } + slices.Sort(v.Callables) + v.Callables = slices.Compact(v.Callables) + out = append(out, v) + } + slices.SortStableFunc(out, func(a, b v2.Degraded) int { + return cmp.Or(strings.Compare(a.Code, b.Code), strings.Compare(a.Detail, b.Detail)) + }) + return out +} + +// compareOrdinal compares GlobalOrdinal(o1, k1) with GlobalOrdinal(o2, k2) +// byte-wise without building either. +func compareOrdinal(o1, k1, o2, k2 string) int { + if o1 == o2 { + return compareJoined([3]string{"", ordinalSep(k1), k1}, [3]string{"", ordinalSep(k2), k2}) + } + if n := min(len(o1), len(o2)); o1[:n] != o2[:n] { + return strings.Compare(o1, o2) + } + // One owner id is a prefix of the other (a callable and its closure). + return compareJoined([3]string{o1, ordinalSep(k1), k1}, [3]string{o2, ordinalSep(k2), k2}) +} + +// ordinalSep is what v2.GlobalOrdinal puts between a callable id and k. +func ordinalSep(k string) string { + if strings.HasPrefix(k, "@") { + return "" + } + return "@" +} + +// compareJoined compares the concatenations of a's and b's segments. +func compareJoined(a, b [3]string) int { + i, j, ai, bj := 0, 0, 0, 0 + for { + for i < len(a) && ai == len(a[i]) { + i, ai = i+1, 0 + } + for j < len(b) && bj == len(b[j]) { + j, bj = j+1, 0 + } + switch { + case i == len(a) && j == len(b): + return 0 + case i == len(a): + return -1 + case j == len(b): + return 1 + } + if ca, cb := a[i][ai], b[j][bj]; ca != cb { + if ca < cb { + return -1 + } + return 1 + } + ai, bj = ai+1, bj+1 + } +} + +// lateFields are the payload fields that follow the symbol table and need +// the whole tree's ids: the application's param_in / param_out (its last +// two fields) and the envelope's degraded (its last field). +type lateFields struct { + paramIn, paramOut iter.Seq[v2.ParamEdge] + degraded []v2.Degraded +} + +// writeParamList writes `,"":[e,e,…]` with each edge as json.Marshal +// writes it, or nothing when seq is empty (omitempty). +func writeParamList(w io.Writer, key string, seq iter.Seq[v2.ParamEdge]) error { + if seq == nil { + return nil + } + var err error + first := true + for e := range seq { + data, merr := json.Marshal(e) + if merr != nil { + err = merr + break + } + lead := "," + if first { + lead = `,"` + key + `":[` + first = false + } + if _, err = io.WriteString(w, lead); err != nil { + break + } + if _, err = w.Write(data); err != nil { + break + } + } + if err != nil || first { + return err + } + _, err = io.WriteString(w, "]") + return err +} diff --git a/internal/syntactic_analysis/v2emit/emit_l4_test.go b/internal/syntactic_analysis/v2emit/emit_l4_test.go new file mode 100644 index 0000000..cdf0d91 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/emit_l4_test.go @@ -0,0 +1,62 @@ +package v2emit + +import ( + "fmt" + "math/rand/v2" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" +) + +// TestCompareOrdinal_MatchesMintedIds: compareOrdinal orders (owner, key) +// pairs exactly as their minted ids v2.GlobalOrdinal(owner, key) compare, +// including an owner that is a prefix of the other (a callable and its +// closure) and the "@"-tag keys, whose ids carry no extra "@". +func TestCompareOrdinal_MatchesMintedIds(t *testing.T) { + owners := []string{ + "can://a/go/f.go/p.F", "can://a/go/f.go/p.F/p.F.closure_1", "can://a/go/f.go/p.F2", + "can://a/go/f.go/p.F@", "can://a/@external/fmt/Sprintf", "can://a/@external/fmt", + "can://a/go/f.go/T/p.T.M", "", + } + keys := []string{"@formal_in:0", "@formal_in:10", "@formal_out", "@entry", "12:3/actual_in:0", "12:3/2/actual_out:1", "9:1", "", "@", "/x"} + r := rand.New(rand.NewPCG(1, 2)) + pick := func(s []string) string { return s[r.IntN(len(s))] } + for range 20000 { + o1, k1, o2, k2 := pick(owners), pick(keys), pick(owners), pick(keys) + want := strings.Compare(v2.GlobalOrdinal(o1, k1), v2.GlobalOrdinal(o2, k2)) + if got := compareOrdinal(o1, k1, o2, k2); got != want { + t.Fatalf("compareOrdinal(%q, %q, %q, %q) = %d, want %d", o1, k1, o2, k2, got, want) + } + } +} + +// TestL4Overlay_DuplicateKeyKeepsSmallerID: two callables sharing a target +// key (declaring file and signature) route its param edges and degraded +// entries to the lexically smaller id, whichever is applied first — +// emission walks maps, so last-write-wins would be nondeterministic — and +// the clash is warned once, naming both ids. +func TestL4Overlay_DuplicateKeyKeepsSmallerID(t *testing.T) { + key := dataflow.L3Key{Path: "a.go", Signature: "example.com/a.F"} + small, big := "can://app/go/a.go/example.com/a.F", "can://app/go/b.go/example.com/a.F" + var warned []string + prev := warnf + warnf = func(format string, args ...any) { warned = append(warned, fmt.Sprintf(format, args...)) } + defer func() { warnf = prev }() + for _, order := range [][]string{{small, big}, {big, small}} { + warned = nil + o := newL4Overlay(4, &sdg.Result{}, sigIndex{}) + for _, id := range order { + o.apply(&v2.Callable{ID: id, Body: map[string]v2.BodyNode{}}, key) + } + if got := o.ids[sdg.Target{Callable: key}]; got != small { + t.Errorf("apply order %v: target routes to %s, want %s", order, got, small) + } + if len(warned) != 1 || !strings.Contains(warned[0], small) || !strings.Contains(warned[0], big) || + !strings.Contains(warned[0], "share the target key") { + t.Errorf("apply order %v: warnings %q, want one naming both ids", order, warned) + } + } +} diff --git a/internal/syntactic_analysis/v2emit/emit_test.go b/internal/syntactic_analysis/v2emit/emit_test.go index c5ae286..87fe4e2 100644 --- a/internal/syntactic_analysis/v2emit/emit_test.go +++ b/internal/syntactic_analysis/v2emit/emit_test.go @@ -37,7 +37,7 @@ func buildGreeter(t *testing.T) (*schema.GoApplication, string) { func TestEmit_Envelope(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1, "test", 0, nil) + out := Emit(app, "greeter", dir, 1, "test", 0, nil, nil) if out.SchemaVersion != "2.0.0" { t.Errorf("schema_version = %q, want 2.0.0", out.SchemaVersion) @@ -61,7 +61,7 @@ func TestEmit_Envelope(t *testing.T) { func TestEmit_ModuleSourceAndIDs(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1, "test", 0, nil) + out := Emit(app, "greeter", dir, 1, "test", 0, nil, nil) mod, ok := out.Application.SymbolTable["main.go"] if !ok { @@ -84,7 +84,7 @@ func TestEmit_ModuleSourceAndIDs(t *testing.T) { func TestEmit_CallableBodyAndSpanSlice(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1, "test", 0, nil) + out := Emit(app, "greeter", dir, 1, "test", 0, nil, nil) mod := out.Application.SymbolTable["main.go"] var main v2.Callable @@ -123,7 +123,7 @@ func TestEmit_CallableBodyAndSpanSlice(t *testing.T) { func TestEmit_CalleeSerializesAsNull(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1, "test", 0, nil) + out := Emit(app, "greeter", dir, 1, "test", 0, nil, nil) data, err := json.Marshal(out) if err != nil { t.Fatalf("marshal failed: %v", err) @@ -144,7 +144,7 @@ func TestEmit_CalleeSerializesAsNull(t *testing.T) { func TestEmit_TypeKindAndMethods(t *testing.T) { app, dir := buildGreeter(t) - out := Emit(app, "greeter", dir, 1, "test", 0, nil) + out := Emit(app, "greeter", dir, 1, "test", 0, nil, nil) // Find the Greeter type wherever it lives; assert struct kind + methods. var found bool diff --git a/internal/syntactic_analysis/v2emit/gate_l2_inits_test.go b/internal/syntactic_analysis/v2emit/gate_l2_inits_test.go index 529329c..287cd81 100644 --- a/internal/syntactic_analysis/v2emit/gate_l2_inits_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l2_inits_test.go @@ -37,7 +37,7 @@ func buildDataflowL2Fresh(t *testing.T) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} - return Emit(app, "dataflow", dir, 2, "gate-test", 0, nil) + return Emit(app, "dataflow", dir, 2, "gate-test", 0, nil, nil) } // TestGateL2_EveryInitKeepsItsEdges: each file's init has its own edge to the diff --git a/internal/syntactic_analysis/v2emit/gate_l2_test.go b/internal/syntactic_analysis/v2emit/gate_l2_test.go index 5e46baf..347d3db 100644 --- a/internal/syntactic_analysis/v2emit/gate_l2_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l2_test.go @@ -42,7 +42,7 @@ func buildMultipackageL2(t *testing.T) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) // mutates st in place: backfills callee_signature app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} - return Emit(app, "multipackage", dir, 2, "gate-test", 0, nil) + return Emit(app, "multipackage", dir, 2, "gate-test", 0, nil, nil) } // allCallableIDs collects every callable/type id in the emitted tree. @@ -189,12 +189,12 @@ func TestGateL2_SupersetOfL1(t *testing.T) { t.Fatalf("symbol table build failed: %v", err) } // L1 snapshot BEFORE the resolver mutates call sites. - l1 := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: nil}, "multipackage", dir, 1, "gate-test", 0, nil) + l1 := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: nil}, "multipackage", dir, 1, "gate-test", 0, nil, nil) // Now run the resolver (mutates st) and emit L2. cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) - l2 := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges}, "multipackage", dir, 2, "gate-test", 0, nil) + l2 := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges}, "multipackage", dir, 2, "gate-test", 0, nil, nil) if len(l1.Application.SymbolTable) != len(l2.Application.SymbolTable) { t.Fatalf("module count changed L1->L2: %d vs %d", @@ -337,6 +337,26 @@ func F() string { if len(sym.Parameters) != 1 || sym.Parameters[0].Name != "s" { t.Errorf("params = %+v, want exactly [s] (receiver excluded)", sym.Parameters) } + if sym.Receiver == nil || sym.Receiver.Name != "b" || sym.Receiver.Type != "*strings.Builder" { + t.Errorf("receiver = %+v, want {b *strings.Builder}", sym.Receiver) + } +} + +// TestGateL2_ExternalFunctionHasNoReceiver: only methods carry a receiver. +func TestGateL2_ExternalFunctionHasNoReceiver(t *testing.T) { + out := emitTemp(t, `package p + +import "strings" + +func F() string { return strings.ToUpper("x") } +`) + sym, ok := out.Application.ExternalSymbols[v2.ExternalID(out.Application.ID, "strings", "ToUpper")] + if !ok { + t.Fatalf("external_symbols missing strings.ToUpper") + } + if sym.Receiver != nil { + t.Errorf("function has receiver %+v", sym.Receiver) + } } // TestGateL2_ConversionIsNotACall locks that a type conversion is not minted as @@ -390,7 +410,7 @@ func emitTemp(t *testing.T, src string) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} - return Emit(app, "p", dir, 2, "gate-test", 0, nil) + return Emit(app, "p", dir, 2, "gate-test", 0, nil, nil) } // TestGateL2_ExternalParamsHaveNoSpan locks that external parameters serialize @@ -422,7 +442,7 @@ func TestGateL2_GenericLibraryMethodUsesDeclaredSignature(t *testing.T) { } cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) - out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "dataflow", dir, 2, "gate-test", 0, nil) + out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "dataflow", dir, 2, "gate-test", 0, nil, nil) ext := v2.ExternalID(out.Application.ID, "sync/atomic", "Pointer.Store") sym, ok := out.Application.ExternalSymbols[ext] if !ok { @@ -446,7 +466,7 @@ func TestGateL2_IncrementalModeKeepsProjectCalleesInternal(t *testing.T) { } cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) - out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "multipackage", dir, 2, "gate-test", 0, nil) + out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "multipackage", dir, 2, "gate-test", 0, nil, nil) const projectModule = "example.com/multipackage" for id, sym := range out.Application.ExternalSymbols { @@ -472,7 +492,7 @@ func TestGateL2_VendoredCalleeIsExternal(t *testing.T) { } cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) - out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "dataflow", dir, 2, "gate-test", 0, nil) + out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "dataflow", dir, 2, "gate-test", 0, nil, nil) ext := v2.ExternalID(out.Application.ID, "example.com/vlib", "F") sym, ok := out.Application.ExternalSymbols[ext] @@ -490,10 +510,11 @@ func TestGateL2_VendoredCalleeIsExternal(t *testing.T) { } } -// TestGateL2_LineDirectiveCallsResolve (I1): L1 records call sites through -// fset.Position, which applies //line directives, so a site under a directive -// names a virtual line:col (here line 900 of a 17-line file). The resolver -// must find it by that same key and backfill its callee and edge. +// TestGateL2_LineDirectiveCallsResolve (I1, #26): L1 records call sites at +// their real file positions, never the //line-adjusted ones, so a site under +// a directive keeps its real line:col (here 13:2, which the directive maps +// to virtual.y:900). The resolver must find it by that same key and backfill +// its callee and edge. func TestGateL2_LineDirectiveCallsResolve(t *testing.T) { out := buildDataflowL2Fresh(t) fn := functionByName(t, mustModule(t, out, "linedir/linedir.go"), "Caller") @@ -511,8 +532,8 @@ func TestGateL2_LineDirectiveCallsResolve(t *testing.T) { } callees[*n.Callee] = true } - if _, virtual := fn.Body["900:2"]; !virtual { - t.Fatalf("no call node at the virtual 900:2; body %v", keys(fn.Body)) + if _, real := fn.Body["13:2"]; !real { + t.Fatalf("no call node at the real 13:2; body %v", keys(fn.Body)) } edges := map[string]bool{} for _, e := range out.Application.CallGraph { @@ -543,7 +564,7 @@ func TestGateL2_ExternalUnnamedParamsArePositional(t *testing.T) { } cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) - out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "dataflow", dir, 2, "gate-test", 0, nil) + out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, "dataflow", dir, 2, "gate-test", 0, nil, nil) for sig, want := range map[string][]string{"G": {"_0", "_1"}, "H": {"_0", "n"}} { ext := v2.ExternalID(out.Application.ID, "example.com/vlib", sig) @@ -561,3 +582,143 @@ func TestGateL2_ExternalUnnamedParamsArePositional(t *testing.T) { } } } + +// emitDirL2 builds dir through the real builder + resolver and emits v2 at L2. +func emitDirL2(t *testing.T, dir, app string) *v2.Analysis { + t.Helper() + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + return Emit(&schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()}, app, dir, 2, "gate-test", 0, nil, nil) +} + +// assertNoDanglingEdges fails when an edge endpoint is neither a callable id +// nor an external_symbols key. +func assertNoDanglingEdges(t *testing.T, out *v2.Analysis) { + t.Helper() + ids := allCallableIDs(out) + for id := range out.Application.ExternalSymbols { + ids[id] = true + } + for i, e := range out.Application.CallGraph { + if !ids[e.Src] || !ids[e.Dst] { + t.Errorf("edge[%d] dangles: %s -> %s", i, e.Src, e.Dst) + } + } +} + +// TestGateL2_ClosureCallSitesResolve: a call inside a function literal resolves +// like any other (#17). flow.Closure's literal calls f(x) at 217:10. +func TestGateL2_ClosureCallSitesResolve(t *testing.T) { + out := buildDataflowL2Fresh(t) + mod := mustModule(t, out, "flow/flow.go") + closure := functionByName(t, mod, "Closure") + if len(closure.Callables) != 1 { + t.Fatalf("Closure has %d closures, want 1", len(closure.Callables)) + } + var lit v2.Callable + for _, c := range closure.Callables { + lit = c + } + wantCallee := functionByName(t, mod, "f").ID + + n, ok := lit.Body["217:10"] + if !ok { + t.Fatalf("closure body has no 217:10 call; keys=%v", keys(lit.Body)) + } + if n.Callee == nil || *n.Callee != wantCallee { + t.Errorf("closure call 217:10 callee = %v, want %s", n.Callee, wantCallee) + } + found := false + for _, e := range out.Application.CallGraph { + if e.Src == lit.ID && e.Dst == wantCallee { + found = true + if len(e.Prov) != 1 || e.Prov[0] != "go/types" { + t.Errorf("closure edge prov = %v, want [go/types]", e.Prov) + } + } + } + if !found { + t.Errorf("call_graph has no edge %s -> %s", lit.ID, wantCallee) + } + assertNoDanglingEdges(t, out) +} + +// TestGateL2_ChiClosureCallsResolve: on chi, closure call nodes resolve and no +// edge dangles. +func TestGateL2_ChiClosureCallsResolve(t *testing.T) { + out := buildChiL2Fresh(t, chiDir(t)) + resolved := 0 + var walk func(c v2.Callable, inClosure bool) + walk = func(c v2.Callable, inClosure bool) { + if inClosure { + for _, n := range c.Body { + if n.Kind == "call" && n.Callee != nil { + resolved++ + } + } + } + for _, in := range c.Callables { + walk(in, true) + } + } + for _, mod := range out.Application.SymbolTable { + for _, fn := range mod.Functions { + walk(fn, false) + } + for _, ty := range mod.Types { + for _, m := range ty.Callables { + walk(m, false) + } + } + } + if resolved == 0 { + t.Error("no closure call node has a callee on chi") + } + t.Logf("chi closure call nodes with a callee: %d", resolved) + assertNoDanglingEdges(t, out) +} + +// TestGateL2_CrossFileMethodClosureEdge: a closure inside a method declared in a +// different file than its receiver type nests under the type's module but is +// declared in the method's file. Its edge must resolve to the closure's id. +func TestGateL2_CrossFileMethodClosureEdge(t *testing.T) { + dir := t.TempDir() + for name, src := range map[string]string{ + "go.mod": "module example.com/p\n\ngo 1.21\n", + "a.go": "package p\n\ntype T struct{}\n", + "b.go": "package p\n\nfunc helper() {}\n\nfunc (t *T) Run() {\n\tfn := func() {\n\t\tinner := func() { helper() }\n\t\tinner()\n\t\thelper()\n\t}\n\tfn()\n}\n", + } { + if err := os.WriteFile(filepath.Join(dir, name), []byte(src), 0o644); err != nil { + t.Fatal(err) + } + } + out := emitDirL2(t, dir, "p") + run := mustModule(t, out, "a.go").Types["T"].Callables["example.com/p.T.Run"] + if len(run.Callables) != 1 { + t.Fatalf("Run has %d closures, want 1", len(run.Callables)) + } + helper := functionByName(t, mustModule(t, out, "b.go"), "helper").ID + var fn v2.Callable + for _, c := range run.Callables { + fn = c + } + var inner v2.Callable + for _, c := range fn.Callables { + inner = c + } + has := map[[2]string]bool{} + for _, e := range out.Application.CallGraph { + has[[2]string{e.Src, e.Dst}] = true + } + for _, c := range []v2.Callable{fn, inner} { + if c.ID == "" || !has[[2]string{c.ID, helper}] { + t.Errorf("missing edge %s -> %s", c.ID, helper) + } + } + assertNoDanglingEdges(t, out) +} diff --git a/internal/syntactic_analysis/v2emit/gate_l3_test.go b/internal/syntactic_analysis/v2emit/gate_l3_test.go index 8f6a976..eaf31f6 100644 --- a/internal/syntactic_analysis/v2emit/gate_l3_test.go +++ b/internal/syntactic_analysis/v2emit/gate_l3_test.go @@ -69,8 +69,8 @@ func buildL3(t *testing.T, name string) *l3Fixture { res := dataflow.Run(b.Fset(), b.L3Units(st), 3, 4) fx := &l3Fixture{ st: st, - l2: Emit(app, name, dir, 2, "gate-test", 3, nil), - l3: Emit(app, name, dir, 3, "gate-test", 3, res), + l2: Emit(app, name, dir, 2, "gate-test", 3, nil, nil), + l3: Emit(app, name, dir, 3, "gate-test", 3, res, nil), res: res, } l3Cache[name] = fx @@ -464,3 +464,94 @@ func TestGateL3_GoStatementArgumentCalls(t *testing.T) { } } } + +// TestGateL3_LineDirectiveRealPositions (#26): under //line directives the +// L3 body of linedir.Caller is built (its calls pair with their L1 keys by +// Lparen), its keys are real line:col, and every non-synthetic node's span +// slices module.source to its text. +func TestGateL3_LineDirectiveRealPositions(t *testing.T) { + fx := buildL3(t, "dataflow") + mod := mustModule(t, fx.l3, "linedir/linedir.go") + c := functionByName(t, mod, "Caller") + want := map[string]string{ + "@entry": "", "@exit": "", + "13:2": "os.Getenv(helper())", + "13:12": "helper()", + "15:2": "helper()", + } + if got, w := strings.Join(sortedKeys(c.Body), ","), strings.Join(sortedKeys(want), ","); got != w { + t.Fatalf("Caller body keys = %s, want %s", got, w) + } + for k, text := range want { + if text == "" { + continue + } + if got := c.Body[k].Span.Slice(mod.Source); got != text { + t.Errorf("Caller %s span slices to %q, want %q", k, got, text) + } + } + if len(c.Cfg) == 0 { + t.Error("Caller: empty cfg") + } +} + +// TestSymbolTable_MethodDeclaredAboveItsType: a method declared above its +// receiver type in the same file is kept. It has its call site at L1, the +// call to f resolves at -a 2 (callee id and call_graph edge), and at -a 3 it +// has body nodes and a ddg. +func TestSymbolTable_MethodDeclaredAboveItsType(t *testing.T) { + fx := buildL3(t, "dataflow") + const sig = "example.com/dataflow/flow.early.Bump" + const fID = "example.com/dataflow/flow.f" + + find := func(out *v2.Analysis) (v2.Callable, bool) { + for _, mod := range out.Application.SymbolTable { + for _, ty := range mod.Types { + for _, m := range ty.Callables { + if strings.HasSuffix(m.ID, "/"+sig) { + return m, true + } + } + } + } + return v2.Callable{}, false + } + + bump, ok := find(fx.l2) + if !ok { + t.Fatalf("no callable %s under type early at -a 2", sig) + } + var call *v2.BodyNode + for _, n := range bump.Body { + n := n + if n.Kind == "call" { + call = &n + } + } + if call == nil { + t.Fatalf("%s has no call site", sig) + } + if call.Callee == nil || !strings.HasSuffix(*call.Callee, "/"+fID) { + t.Fatalf("call to f not resolved at -a 2: callee = %v", call.Callee) + } + sawEdge := false + for _, e := range fx.l2.Application.CallGraph { + if e.Src == bump.ID && e.Dst == *call.Callee { + sawEdge = true + } + } + if !sawEdge { + t.Errorf("no call_graph edge %s -> %s", bump.ID, *call.Callee) + } + + b3, ok := find(fx.l3) + if !ok { + t.Fatalf("no callable %s at -a 3", sig) + } + if _, ok := b3.Body["@entry"]; !ok { + t.Errorf("%s has no @entry body node at -a 3 (keys %d)", sig, len(b3.Body)) + } + if len(b3.Ddg) == 0 { + t.Errorf("%s has no ddg at -a 3", sig) + } +} diff --git a/internal/syntactic_analysis/v2emit/gate_l4_done_test.go b/internal/syntactic_analysis/v2emit/gate_l4_done_test.go new file mode 100644 index 0000000..d3b4319 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/gate_l4_done_test.go @@ -0,0 +1,344 @@ +package v2emit + +// The L4 "done" gate (spec §8): on the HTTP → SQL fixture +// testdata/dataflow/httpsql, the emitted JSON graph carries a flow from an +// HTTP handler's request parameter to database/sql's DB.Query query +// argument through an interface call, and unrelated pairs are unreachable. +// Sanitizers are SDK taint semantics (.github#103); the second test pins +// what L4 gives a sanitizer to work with. + +import ( + "path/filepath" + "runtime" + "slices" + "strings" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" +) + +// sdgGraph is the emitted JSON's dependence graph over global body node ids: +// every callable's ddg, cdg and summary edges (local keys minted with +// v2.GlobalOrdinal) and the application's param_in and param_out — the +// edge families the SDK's slice and taint walks follow. Each edge records +// its family, so a path can be checked for the hops it takes. +type sdgGraph struct { + succ map[string][]sdgHop +} + +type sdgHop struct { + to, family string + v string // the edge's var (ddg, param_in, param_out); "" for cdg and summary +} + +func newSDGGraph(a *v2.Analysis) *sdgGraph { + g := &sdgGraph{succ: map[string][]sdgHop{}} + add := func(src, dst, family, v string) { g.succ[src] = append(g.succ[src], sdgHop{dst, family, v}) } + for _, c := range l4Callables(a) { + id := func(k string) string { return v2.GlobalOrdinal(c.ID, k) } + for _, e := range c.Ddg { + add(id(e.Src), id(e.Dst), "ddg", e.Var) + } + for _, e := range c.Cdg { + add(id(e.Src), id(e.Dst), "cdg", "") + } + for _, e := range c.Summary { + add(id(e.Src), id(e.Dst), "summary", "") + } + } + for _, e := range a.Application.ParamIn { + add(e.Src, e.Dst, "param_in", e.Var) + } + for _, e := range a.Application.ParamOut { + add(e.Src, e.Dst, "param_out", e.Var) + } + return g +} + +// path is a shortest path from src to dst by BFS, as the hops taken (nil if +// dst is unreachable). Successors are visited in edge order, so the path is +// deterministic. +func (g *sdgGraph) path(src, dst string) []sdgHop { + return g.pathExcept(src, dst, nil) +} + +// underCallable is python-sdk's under_callable for one id: the callable +// itself, a body node of it (q@…), or a node of a callable nested in it +// (q/…). +func underCallable(id, q string) bool { + return id == q || strings.HasPrefix(id, q+"@") || strings.HasPrefix(id, q+"/") +} + +// pathExcept is path over the graph without the hops skip rejects (skip +// gets the hop's source node and the hop); a nil skip keeps every hop. +func (g *sdgGraph) pathExcept(src, dst string, skip func(from string, h sdgHop) bool) []sdgHop { + type pred struct { + from string + hop sdgHop + } + prev := map[string]pred{src: {}} + for q := []string{src}; len(q) > 0; q = q[1:] { + n := q[0] + if n == dst { + var out []sdgHop + for n != src { + p := prev[n] + out = append(out, p.hop) + n = p.from + } + slices.Reverse(out) + return out + } + for _, h := range g.succ[n] { + if skip != nil && skip(n, h) { + continue + } + if _, seen := prev[h.to]; !seen { + prev[h.to] = pred{n, h} + q = append(q, h.to) + } + } + } + return nil +} + +// TestGateL4_HTTPToSQLPath: Handler's closure reads r.URL.Query().Get("id") +// and calls repo.Find(id) on the Repo interface; VTA binds sqlRepo.Find, +// which passes "…" + id to db.Query. A BFS over the emitted graph from the +// closure's r formal_in reaches the stand-in DB.Query's formal_in of +// "query", and the path enters sqlRepo.Find through a param_in. Health +// never queries: from its r, the sink is unreachable. +func TestGateL4_HTTPToSQLPath(t *testing.T) { + fx := buildL4(t, "dataflow/httpsql") + a := fx.l4 + if len(a.Degraded) != 0 { + t.Fatalf("httpsql degraded: %+v", a.Degraded) + } + app := a.Application.ID + if app != "can://httpsql" { + t.Fatalf("application id %s, want can://httpsql", app) + } + idx := bodyIndex(a) + vertex := func(owner, kind, of string) string { + t.Helper() + var found []string + for id, on := range idx { + if on.owner == owner && on.node.Kind == kind && on.node.Of == of { + found = append(found, id) + } + } + if len(found) != 1 { + t.Fatalf("%s has %d %s vertices of %q, want 1", owner, len(found), kind, of) + } + return found[0] + } + mod := v2.ModuleID(app, "httpsql.go") + pkg := "example.com/dataflow/httpsql." + fn := func(name string) string { return v2.CallableID(mod, pkg+name) } + closure := func(name string) string { return v2.CallableID(fn(name), pkg+name+".closure_1") } + method := func(typ, m string) string { return v2.CallableID(v2.TypeID(mod, pkg+typ), pkg+typ+"."+m) } + + sink := vertex(v2.ExternalID(app, "database/sql", "DB.Query"), "formal_in", "query") + if sink != "can://httpsql/@external/database/sql/DB.Query@formal_in:1" { + t.Fatalf("sink vertex %s", sink) + } + find := method("sqlRepo", "Find") + g := newSDGGraph(a) + + src := vertex(closure("Handler"), "formal_in", "r") + p := g.path(src, sink) + if p == nil { + t.Fatalf("no path from Handler's r (%s) to %s", src, sink) + } + intoFind := slices.ContainsFunc(p, func(h sdgHop) bool { + return h.family == "param_in" && idx[h.to].owner == find + }) + if !intoFind { + t.Errorf("path %v does not enter %s through a param_in", p, find) + } + // The interface call binds sqlRepo.Find by VTA. + vta := false + for _, e := range a.Application.CallGraph { + vta = vta || e.Src == closure("Handler") && e.Dst == find && slices.Contains(e.Prov, "vta") + } + if !vta { + t.Errorf("no VTA call_graph edge %s -> %s", closure("Handler"), find) + } + for i, h := range p { + t.Logf("hop %d: %s -> %s", i, h.family, h.to) + } + // The shortest path may take the receiver (the coarse host wiring feeds + // every actual_in of repo.Find(id)); the flow of id itself must exist + // too: r → Find's id formal_in → the query actual_in. + findID := vertex(find, "formal_in", "id") + if g.path(src, findID) == nil || g.path(findID, sink) == nil { + t.Errorf("no path %s → %s → %s", src, findID, sink) + } + + // A pair with no flow: Health never touches the database. + if p := g.path(vertex(fn("Health"), "formal_in", "r"), sink); p != nil { + t.Errorf("Health's r reaches %s: %v", sink, p) + } + + // A no-flow pair inside the DB plumbing, so the gate catches + // over-connection and not only an uncalled callable. Audit is an entry + // point of its own: it builds its own Repo inline and calls only Count, so + // it shares no host variable and no in-project callee with the handlers. + // Its user reaches the sink (through Count), but never Find's id, and + // Handler's r never reaches Count's user. Either fails if an interface + // site bound the wrong method or param edges were cross-wired between + // sites. Reordering Routes' registrations or Audit's statements, or moving + // Audit in the file, leaves both pairs unreachable (checked by mutation). + count := method("sqlRepo", "Count") + auditUser := vertex(fn("Audit"), "formal_in", "user") + if g.path(auditUser, sink) == nil { + t.Errorf("Audit's user does not reach %s (the pairs below would be vacuous)", sink) + } + if p := g.path(auditUser, findID); p != nil { + t.Errorf("Audit's user reaches %s: %v", findID, p) + } + if p := g.path(src, vertex(count, "formal_in", "user")); p != nil { + t.Errorf("Handler's r reaches %s's user: %v", count, p) + } +} + +// buildHTTPSQLWithModels is buildL4 over testdata/dataflow/httpsql with +// --library-models models, uncached. +func buildHTTPSQLWithModels(t *testing.T, models sdg.Models) *v2.Analysis { + t.Helper() + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata", "dataflow", "httpsql")) + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + units := b.L3Units(st) + res := dataflow.RunWithSideTables(b.Fset(), units, 3, 4) + l4 := sdg.Run(sdg.Input{ + Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Externals: app.Externals, CallGraph: app.CallGraph, + Units: units, L3: res, K: 3, Jobs: 4, Models: models, + }) + l4.ApplyTo(app) + return Emit(app, "httpsql", dir, 4, "gate-test", 3, res, &l4) +} + +// TestGateL4_HTTPToSQLSanitizerIsSDKSemantics pins what L4 does — and +// does not do — about SanitizedFind's strconv.Atoi [user, 2026-10-27]. L4 is +// reaching-definitions data flow, not taint: `n, err := strconv.Atoi(id)` +// does make n (and err) depend on id, and that edge is correct. Treating a +// call as a sanitizer is the SDK taint analysis's job (callable or +// variable cuts, including the bypasses below), tracked in +// codellm-devkit/.github#103. +// +// - (a) The model strconv/Atoi: [] removes the Atoi call site's summary +// edges and nothing else that matters here: SanitizedHandler's r still +// reaches DB.Query's query, because the host statement links id → n. +// - (b) A variable cut on n within sqlRepo.SanitizedFind — the SDK's pair +// cut: drop the edges whose var is n and whose source node is under that +// callable (python-sdk commons/graphs.py:380-383) — does NOT sever the +// path either. Three bypasses remain, each correct data or control +// dependence: +// 1. err → cdg → Query: the same statement defines err; `if err != nil` +// controls the Query statement (implicit flow); +// 2. err → return → caller → receiver: `return nil, err` carries err to +// the caller, whose host feeds the receiver actual_in back into +// SanitizedFind's s, then s.db → Query; +// 3. caller host → receiver actual_in → s.db: the caller's host +// statement uses id, and the coarse host wiring feeds every host use +// into every actual_in, receiver included. +func TestGateL4_HTTPToSQLSanitizerIsSDKSemantics(t *testing.T) { + plain := buildL4(t, "dataflow/httpsql").l4 + modelled := buildHTTPSQLWithModels(t, sdg.Models{"strconv/Atoi": {}}) + app := modelled.Application.ID + mod := v2.ModuleID(app, "httpsql.go") + pkg := "example.com/dataflow/httpsql." + sanFind := v2.CallableID(v2.TypeID(mod, pkg+"sqlRepo"), pkg+"sqlRepo.SanitizedFind") + handler := v2.CallableID(v2.CallableID(mod, pkg+"SanitizedHandler"), pkg+"SanitizedHandler.closure_1") + sink := v2.ExternalID(app, "database/sql", "DB.Query") + "@formal_in:1" + r := v2.GlobalOrdinal(handler, "@formal_in:1") + id := v2.GlobalOrdinal(sanFind, "@formal_in:1") + recv := v2.GlobalOrdinal(sanFind, "@formal_in:0") + + callable := func(a *v2.Analysis, cid string) v2.Callable { + for _, c := range l4Callables(a) { + if c.ID == cid { + return c + } + } + t.Fatalf("no callable %s", cid) + return v2.Callable{} + } + // The Atoi call site: the actual_in whose param_in reaches strconv/Atoi. + atoi := v2.ExternalID(app, "strconv", "Atoi") + var atoiSite string + for _, e := range modelled.Application.ParamIn { + if strings.HasPrefix(e.Dst, atoi+"@") && strings.HasPrefix(e.Src, sanFind+"@") { + atoiSite = strings.TrimPrefix(e.Src, sanFind+"@") + atoiSite = atoiSite[:strings.Index(atoiSite, "/actual_in")] + } + } + if atoiSite == "" { + t.Fatal("no param_in from SanitizedFind to strconv/Atoi") + } + sitesSummary := func(a *v2.Analysis) int { + n := 0 + for _, e := range callable(a, sanFind).Summary { + if strings.HasPrefix(e.Src, atoiSite+"/") { + n++ + } + } + return n + } + + // (a) The model removes the Atoi summary edges; the flow stays. + if n := sitesSummary(plain); n == 0 { + t.Fatalf("without a model, the Atoi site %s has no summary edges (the model check would be vacuous)", atoiSite) + } + if n := sitesSummary(modelled); n != 0 { + t.Errorf("strconv/Atoi: [] leaves %d summary edges at %s", n, atoiSite) + } + g := newSDGGraph(modelled) + if g.path(r, sink) == nil { + t.Errorf("with strconv/Atoi: [], SanitizedHandler's r no longer reaches %s; L4 must keep the id → n data flow (.github#103)", sink) + } + + // (b) A variable cut on n within SanitizedFind does not sever it. + cutN := func(src string, h sdgHop) bool { return h.v == "n" && underCallable(src, sanFind) } + if g.pathExcept(r, sink, cutN) == nil { + t.Errorf("a cut on n within SanitizedFind severs %s → %s: a bypass below is gone; update this test and .github#103", r, sink) + } + // Bypass 1: err → cdg → Query, within SanitizedFind alone. + local := func(src string, h sdgHop) bool { + return cutN(src, h) || !underCallable(h.to, sanFind) && h.to != sink + } + if p := g.pathExcept(id, sink, local); p == nil || !slices.ContainsFunc(p, func(h sdgHop) bool { return h.family == "cdg" }) { + t.Errorf("bypass 1 (err → cdg → Query) gone: path %v", p) + } + // Bypass 2: err → return → the caller → the receiver, with cdg excluded. + noCDG := func(src string, h sdgHop) bool { return cutN(src, h) || h.family == "cdg" } + p2 := g.pathExcept(id, sink, noCDG) + if p2 == nil || !slices.ContainsFunc(p2, func(h sdgHop) bool { return h.family == "param_out" }) || + !slices.ContainsFunc(p2, func(h sdgHop) bool { return h.family == "param_in" && h.to == recv }) { + t.Errorf("bypass 2 (err → return → caller → receiver) gone: path %v", p2) + } + // Bypass 3: the caller's host feeds id into the receiver actual_in. + // r reaches the receiver without entering SanitizedFind's body first. + outside := func(_ string, h sdgHop) bool { return underCallable(h.to, sanFind) && h.to != recv } + p3 := g.pathExcept(r, recv, outside) + if p3 == nil || g.pathExcept(recv, sink, cutN) == nil { + t.Errorf("bypass 3 (caller host → receiver actual_in → s.db) gone: %v", p3) + } +} diff --git a/internal/syntactic_analysis/v2emit/gate_l4_test.go b/internal/syntactic_analysis/v2emit/gate_l4_test.go new file mode 100644 index 0000000..42b4891 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/gate_l4_test.go @@ -0,0 +1,922 @@ +package v2emit + +// L4 conformance harness for the v2 emitter. +// +// buildL4 runs the -a 4 pipeline: the -a 3 one, then sdg.Run, whose VTA +// call_graph edges and VTA-only externals are merged into the application +// before the L4 payload is emitted (as core does), with the SDG overlay +// (vertices, port ddg, summaries, param edges, stand-in bodies, degraded). +// The monotonicity gate below holds over it (-a 3 ⊆ -a 4). + +import ( + "bytes" + "encoding/json" + "fmt" + "path/filepath" + "runtime" + "slices" + "strings" + "sync" + "testing" + + "github.com/codellm-devkit/codeanalyzer-go/internal/dataflow" + "github.com/codellm-devkit/codeanalyzer-go/internal/schema" + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" + "github.com/codellm-devkit/codeanalyzer-go/internal/sdg" + "github.com/codellm-devkit/codeanalyzer-go/internal/semantic_analysis" + "github.com/codellm-devkit/codeanalyzer-go/internal/syntactic_analysis" +) + +type l4Fixture struct { + st map[string]schema.GoFile + l3, l4 *v2.Analysis + res map[dataflow.L3Key]dataflow.Result + sdg *sdg.Result + dir string +} + +var ( + l4Mu sync.Mutex + l4Cache = map[string]*l4Fixture{} +) + +// buildL4 runs the -a 4 pipeline over testdata/ once and emits both the +// L3 and the L4 payload from the same resolved symbol table. The app name is +// name's last element (cango's default: the input directory's base name). +func buildL4(t *testing.T, name string) *l4Fixture { + t.Helper() + l4Mu.Lock() + defer l4Mu.Unlock() + if fx, ok := l4Cache[name]; ok { + return fx + } + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata", name)) + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + units := b.L3Units(st) + res := dataflow.RunWithSideTables(b.Fset(), units, 3, 4) + appName := filepath.Base(dir) + fx := &l4Fixture{st: st, l3: Emit(app, appName, dir, 3, "gate-test", 3, res, nil), res: res, dir: dir} + l4 := sdg.Run(sdg.Input{ + Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Externals: app.Externals, CallGraph: app.CallGraph, + Units: units, L3: res, K: 3, Jobs: 4, + }) + l4.ApplyTo(app) + fx.sdg = &l4 + fx.l4 = Emit(app, appName, dir, 4, "gate-test", 3, res, &l4) + l4Cache[name] = fx + return fx +} + +// supersetL4 records where a (-a 3) is not contained in b (-a 4): every object +// key and value, and every list entry, by position — except the sorted edge +// lists L4 adds entries into (call_graph gets VTA edges, a callable's ddg +// gets port edges), where every -a 3 edge must occur unchanged somewhere in +// the -a 4 list. +func supersetL4(path string, a, b any, errs *[]string) { + if path == "$.application.call_graph" || strings.HasSuffix(path, ".ddg") { + av, aok := a.([]any) + bv, bok := b.([]any) + if !aok || !bok { + *errs = append(*errs, path+": changed type") + return + } + have := map[string]bool{} + for _, e := range bv { + have[jsonKey(e)] = true + } + for _, e := range av { + if !have[jsonKey(e)] { + *errs = append(*errs, path+": edge "+jsonKey(e)+" missing or changed at L4") + } + } + return + } + switch av := a.(type) { + case map[string]any: + bv, ok := b.(map[string]any) + if !ok { + *errs = append(*errs, path+": object changed type") + return + } + for k, x := range av { + y, ok := bv[k] + if !ok { + *errs = append(*errs, path+"."+k+" missing at L4") + continue + } + supersetL4(path+"."+k, x, y, errs) + } + case []any: + bv, ok := b.([]any) + if !ok || len(av) > len(bv) { + *errs = append(*errs, path+": list shrank or changed type") + return + } + for i := range av { + supersetL4(path+"[]", av[i], bv[i], errs) + } + default: + if a != b { + *errs = append(*errs, path+": value changed") + } + } +} + +// TestGateL4_SupersetOfL3 is the monotonicity gate: everything in the -a 3 JSON +// is present and equal in the -a 4 JSON. Only max_level, the level marker +// itself, may differ. +func TestGateL4_SupersetOfL3(t *testing.T) { + for _, name := range l4Fixtures { + fx := buildL4(t, name) + a, b := toAny(t, fx.l3), toAny(t, fx.l4) + if b["max_level"] != float64(4) { + t.Errorf("%s: max_level = %v, want 4", name, b["max_level"]) + } + a["max_level"], b["max_level"] = nil, nil + var errs []string + supersetL4("$", a, b, &errs) + for i, e := range errs { + if i == 20 { + t.Errorf("%s: … %d more", name, len(errs)-20) + break + } + t.Errorf("%s: %s", name, e) + } + } +} + +func jsonKey(v any) string { + data, _ := json.Marshal(v) + return string(data) +} + +// TestGateL4_CallGraphHasVTAProv: at -a 4 the call graph gains VTA's dispatch +// edges (spec §4.8), prov ["vta"], weight 1, each one new: the concrete +// implementations of an interface call, the function passed to a func-value +// call, a promoted method, and a library method only VTA reaches, whose +// external (with its receiver) joins external_symbols. Every endpoint +// resolves, and the L2 edges are unchanged (the superset gate). +func TestGateL4_CallGraphHasVTAProv(t *testing.T) { + fx := buildL4(t, "dataflow") + app := fx.l4.Application + mod := v2.ModuleID(app.ID, "sdg/sdg.go") + fn := func(sig string) string { + return v2.CallableID(mod, "example.com/dataflow/sdg."+sig) + } + method := func(typ, m string) string { + return v2.CallableID(v2.TypeID(mod, "example.com/dataflow/sdg."+typ), "example.com/dataflow/sdg."+typ+"."+m) + } + bufWrite := v2.ExternalID(app.ID, "bytes", "Buffer.Write") + want := [][2]string{ + {fn("UseStore"), method("memStore", "Get")}, + {fn("UseStore"), method("sqlStore", "Get")}, + {fn("CallFuncValue"), fn("double")}, + {fn("CallPromoted"), method("Inner", "M")}, + {fn("Buffered"), bufWrite}, + } + l3 := map[[2]string]bool{} + for _, e := range fx.l3.Application.CallGraph { + l3[[2]string{e.Src, e.Dst}] = true + } + vta := map[[2]string]v2.Edge{} + for _, e := range app.CallGraph { + if slices.Contains(e.Prov, "vta") { + k := [2]string{e.Src, e.Dst} + if !slices.Equal(e.Prov, []string{"vta"}) || e.Weight != 1 || l3[k] { + t.Errorf("VTA edge %+v: want prov [vta], weight 1, not an L3 edge", e) + } + vta[k] = e + } + } + for _, k := range want { + if _, ok := vta[k]; !ok { + t.Errorf("no VTA edge %s -> %s", k[0], k[1]) + } + } + if l3[[2]string{fn("UseStore"), method("Store", "Get")}] == false { + t.Errorf("L2 edge UseStore -> Store.Get missing") + } + if _, ok := fx.l3.Application.ExternalSymbols[bufWrite]; ok { + t.Fatalf("-a 3 already has %s; the fixture no longer pins a VTA-only external", bufWrite) + } + ext, ok := app.ExternalSymbols[bufWrite] + if !ok { + t.Fatalf("external_symbols lack %s", bufWrite) + } + if ext.Receiver == nil || ext.Receiver.Name != "b" || ext.Receiver.Type != "*bytes.Buffer" || ext.Signature != "Buffer.Write" { + t.Errorf("%s = %+v (receiver %+v)", bufWrite, ext, ext.Receiver) + } + // Each file's init closure is its own target: the edges keep both. + initClosure := func(file string) string { + m := v2.ModuleID(app.ID, "initclosure/"+file) + init := v2.CallableID(m, "example.com/dataflow/initclosure.init") + return v2.CallableID(init, "example.com/dataflow/initclosure.init.closure_1") + } + run := v2.CallableID(v2.ModuleID(app.ID, "initclosure/a.go"), "example.com/dataflow/initclosure.Run") + for _, file := range []string{"a.go", "b.go"} { + if _, ok := vta[[2]string{run, initClosure(file)}]; !ok { + t.Errorf("no VTA edge Run -> %s's init closure", file) + } + } + for _, name := range []string{"dataflow", "chi", "generics"} { + assertNoDanglingEdges(t, buildL4(t, name).l4) + } +} + +// ─── the L4 overlay in the tree (Task 16) ──────────────────────────────────── + +// l4Fixtures are the fixtures the L4 tree gates run over. +var l4Fixtures = []string{"dataflow", "chi", "generics", "storedlit"} + +// isVertexKind reports an L4 port vertex kind. +func isVertexKind(k string) bool { + switch sdg.VertexKind(k) { + case sdg.KindFormalIn, sdg.KindFormalOut, sdg.KindActualIn, sdg.KindActualOut: + return true + } + return false +} + +// l4Callables is every callable of the tree (methods and closures included). +func l4Callables(a *v2.Analysis) []v2.Callable { + var out []v2.Callable + var walk func(c v2.Callable) + walk = func(c v2.Callable) { + out = append(out, c) + for _, in := range c.Callables { + walk(in) + } + } + for _, m := range a.Application.SymbolTable { + for _, ty := range m.Types { + for _, c := range ty.Callables { + walk(c) + } + } + for _, f := range m.Functions { + walk(f) + } + } + return out +} + +// bodyIndex maps every body node id of the tree — callables' and stand-ins' +// — to the node and the id of the callable or external that owns it. +type ownedNode struct { + owner string + key string + node v2.BodyNode +} + +func bodyIndex(a *v2.Analysis) map[string]ownedNode { + idx := map[string]ownedNode{} + for _, c := range l4Callables(a) { + for k, n := range c.Body { + idx[n.ID] = ownedNode{c.ID, k, n} + } + } + for id, ext := range a.Application.ExternalSymbols { + for k, n := range ext.Body { + idx[n.ID] = ownedNode{id, k, n} + } + } + return idx +} + +// TestGateL4_NoDanglingEndpoints: every L4 edge endpoint is a node of the +// tree — ddg and summary endpoints are body keys of their callable, param +// edge endpoints are body node ids (a callable's or a stand-in's) — every +// degraded callable is a callable id, and nothing the SDG computed is lost on +// the way: the tree carries every vertex, port edge, summary edge, param +// edge and stand-in formal of the sdg.Result. +func TestGateL4_NoDanglingEndpoints(t *testing.T) { + // rangefunc: the one fixture with degraded callables, so a dropped one + // fails the degraded rows of the table below. + for _, name := range append(slices.Clone(l4Fixtures), "rangefunc") { + fx := buildL4(t, name) + a := fx.l4 + idx := bodyIndex(a) + ids := allCallableIDs(a) + var vertices, port, summary int + for _, c := range l4Callables(a) { + for k, n := range c.Body { + if isVertexKind(n.Kind) { + vertices++ + } + if n.Parent != "" { + if _, ok := c.Body[n.Parent]; !ok { + t.Errorf("%s: %s: body %s parent %q is not a body key", name, c.ID, k, n.Parent) + } + } + } + for _, e := range c.Ddg { + for _, k := range []string{e.Src, e.Dst} { + if _, ok := c.Body[k]; !ok { + t.Errorf("%s: %s: ddg endpoint %q is not a body key", name, c.ID, k) + } + } + if !slices.Equal(e.Prov, []string{"ssa"}) { + port++ + } + } + for _, e := range c.Summary { + for _, k := range []string{e.Src, e.Dst} { + if n, ok := c.Body[k]; !ok || !isVertexKind(n.Kind) { + t.Errorf("%s: %s: summary endpoint %q is not a vertex", name, c.ID, k) + } + } + } + summary += len(c.Summary) + } + for list, edges := range map[string][]v2.ParamEdge{"param_in": a.Application.ParamIn, "param_out": a.Application.ParamOut} { + for _, e := range edges { + for _, id := range []string{e.Src, e.Dst} { + if _, ok := idx[id]; !ok { + t.Errorf("%s: %s endpoint %s is not a body node", name, list, id) + } + } + } + } + for _, d := range a.Degraded { + for _, id := range d.Callables { + if !ids[id] { + t.Errorf("%s: degraded %s names %s, not a callable id", name, d.Code, id) + } + } + } + + var wantV, wantP, wantS, wantX, wantD, gotD int + for _, d := range fx.sdg.Degraded { + ks := slices.Clone(d.Callables) + slices.SortFunc(ks, func(a, b dataflow.L3Key) int { return cmpStrings(a.Path, b.Path, a.Signature, b.Signature) }) + wantD += len(slices.Compact(ks)) + } + for _, d := range a.Degraded { + gotD += len(d.Callables) + } + for _, c := range fx.sdg.Callables { + wantV, wantP, wantS = wantV+len(c.Vertices), wantP+len(c.Ddg), wantS+len(c.Summary) + } + gotX := 0 + for _, ext := range a.Application.ExternalSymbols { + gotX += len(ext.Body) + } + for _, vs := range fx.sdg.Externals { + wantX += len(vs) + } + for _, c := range []struct { + what string + got, want int + }{ + {"vertices", vertices, wantV}, + {"port ddg edges", port, wantP}, + {"summary edges", summary, wantS}, + {"param_in", len(a.Application.ParamIn), len(fx.sdg.ParamIn)}, + {"param_out", len(a.Application.ParamOut), len(fx.sdg.ParamOut)}, + {"stand-in formals", gotX, wantX}, + {"degraded entries", len(a.Degraded), len(fx.sdg.Degraded)}, + {"degraded callables", gotD, wantD}, + } { + if c.got != c.want { + t.Errorf("%s: tree has %d %s, sdg.Result %d", name, c.got, c.what, c.want) + } + } + if name != "storedlit" && name != "rangefunc" && (wantV == 0 || len(fx.sdg.ParamIn) == 0 || wantS == 0) { + t.Errorf("%s: empty overlay (vertices %d, param_in %d, summary %d)", name, wantV, len(fx.sdg.ParamIn), wantS) + } + if name == "rangefunc" && wantD == 0 { + t.Errorf("rangefunc: no degraded callable") + } + t.Logf("%s: %d vertices, %d port ddg, %d summary, %d param_in, %d param_out, %d stand-in formals, %d degraded callables", + name, vertices, port, summary, len(a.Application.ParamIn), len(a.Application.ParamOut), gotX, gotD) + } +} + +// TestGateL4_VertexIdsAreGlobalOrdinals: every vertex — a callable's or a +// stand-in's — has id v2.GlobalOrdinal(owner id, key), a kind, an `of`, no +// span and no callee key; only actual vertices carry a parent. A stand-in +// vertex id is @formal_in: (one "@" after the external id). +func TestGateL4_VertexIdsAreGlobalOrdinals(t *testing.T) { + for _, name := range l4Fixtures { + fx := buildL4(t, name) + for id, on := range bodyIndex(fx.l4) { + n := on.node + if want := v2.GlobalOrdinal(on.owner, on.key); id != want { + t.Errorf("%s: body id %s, want %s", name, id, want) + } + if !isVertexKind(n.Kind) { + if n.Of != "" { + t.Errorf("%s: non-vertex %s carries of %q", name, id, n.Of) + } + continue + } + if n.Of == "" || n.Span != (v2.Span{}) || n.Callee != nil { + t.Errorf("%s: vertex %s = %+v, want an of, no span, no callee", name, id, n) + } + actual := n.Kind == string(sdg.KindActualIn) || n.Kind == string(sdg.KindActualOut) + if actual != (n.Parent != "") { + t.Errorf("%s: vertex %s kind %s parent %q", name, id, n.Kind, n.Parent) + } + data, err := json.Marshal(n) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(data), `"callee"`) || strings.Contains(string(data), `"span"`) { + t.Errorf("%s: vertex %s serialises %s", name, id, data) + } + } + for id, ext := range fx.l4.Application.ExternalSymbols { + for k := range ext.Body { + if !strings.HasPrefix(k, "@formal_") || strings.Count(ext.Body[k].ID[len(id):], "@") != 1 { + t.Errorf("%s: stand-in %s body key %s id %s", name, id, k, ext.Body[k].ID) + } + } + } + } +} + +// TestGateL4_ArityMatchesFormals: every param_in runs from an actual_in of +// the caller to a formal_in that exists in the target's body (a callable or +// a stand-in), and every param_out from such a formal_out to an actual_out; +// var is the target formal's `of` on both. +func TestGateL4_ArityMatchesFormals(t *testing.T) { + for _, name := range l4Fixtures { + fx := buildL4(t, name) + idx := bodyIndex(fx.l4) + check := func(list string, e v2.ParamEdge, actual, formal string, formalEnd string) { + src, dst := idx[e.Src], idx[e.Dst] + act, form := src, dst + if formalEnd == "src" { + act, form = dst, src + } + if act.node.Kind != actual || form.node.Kind != formal { + t.Errorf("%s: %s %s -> %s joins %s to %s", name, list, e.Src, e.Dst, src.node.Kind, dst.node.Kind) + } + if e.Var != form.node.Of { + t.Errorf("%s: %s %s -> %s var %q, target formal of %q", name, list, e.Src, e.Dst, e.Var, form.node.Of) + } + } + for _, e := range fx.l4.Application.ParamIn { + check("param_in", e, string(sdg.KindActualIn), string(sdg.KindFormalIn), "dst") + } + for _, e := range fx.l4.Application.ParamOut { + check("param_out", e, string(sdg.KindActualOut), string(sdg.KindFormalOut), "src") + } + } +} + +// TestGateL4_ParamEdgesSorted: param_in and param_out are sorted by +// (src, dst, var) over the minted ids and hold no duplicate; each +// callable's ddg is sorted by (src, dst, var, prov) and its summary by +// (src, dst). +func TestGateL4_ParamEdgesSorted(t *testing.T) { + for _, name := range l4Fixtures { + fx := buildL4(t, name) + for list, edges := range map[string][]v2.ParamEdge{"param_in": fx.l4.Application.ParamIn, "param_out": fx.l4.Application.ParamOut} { + for i := 1; i < len(edges); i++ { + a, b := edges[i-1], edges[i] + if cmpStrings(a.Src, b.Src, a.Dst, b.Dst, a.Var, b.Var) >= 0 { + t.Errorf("%s: %s[%d..%d] out of order: %+v, %+v", name, list, i-1, i, a, b) + } + } + } + for _, c := range l4Callables(fx.l4) { + for i := 1; i < len(c.Ddg); i++ { + a, b := c.Ddg[i-1], c.Ddg[i] + if cmpStrings(a.Src, b.Src, a.Dst, b.Dst, a.Var, b.Var, strings.Join(a.Prov, ","), strings.Join(b.Prov, ",")) >= 0 { + t.Errorf("%s: %s ddg[%d..%d] out of order: %+v, %+v", name, c.ID, i-1, i, a, b) + } + } + for i := 1; i < len(c.Summary); i++ { + a, b := c.Summary[i-1], c.Summary[i] + if cmpStrings(a.Src, b.Src, a.Dst, b.Dst) >= 0 { + t.Errorf("%s: %s summary[%d..%d] out of order", name, c.ID, i-1, i) + } + } + } + } +} + +// cmpStrings compares the pairs (xs[0], xs[1]), (xs[2], xs[3]), … in turn. +func cmpStrings(xs ...string) int { + for i := 0; i+1 < len(xs); i += 2 { + if c := strings.Compare(xs[i], xs[i+1]); c != 0 { + return c + } + } + return 0 +} + +// TestGateL4_UseStoreEdges pins the emitted L4 of sdg.UseStore (its +// interface call s.Get at 34:9 binds Store.Get, memStore.Get and +// sqlStore.Get): its vertices, its all-to-all summary, and its param edges +// with the minted ids. +func TestGateL4_UseStoreEdges(t *testing.T) { + fx := buildL4(t, "dataflow") + app := fx.l4.Application + mod := v2.ModuleID(app.ID, "sdg/sdg.go") + useStore := v2.CallableID(mod, "example.com/dataflow/sdg.UseStore") + method := func(typ, m string) string { + return v2.CallableID(v2.TypeID(mod, "example.com/dataflow/sdg."+typ), "example.com/dataflow/sdg."+typ+"."+m) + } + var c v2.Callable + for _, x := range l4Callables(fx.l4) { + if x.ID == useStore { + c = x + } + } + for k, of := range map[string]string{ + "@formal_in:0": "s", "@formal_in:1": "k", "@formal_out:0": "", "@formal_out:1": "s", + "34:9/actual_in:0": "_recv", "34:9/actual_in:1": "k", + "34:9/actual_out:0": "", "34:9/actual_out:1": "_recv", + } { + n, ok := c.Body[k] + if !ok || n.Of != of || n.ID != v2.GlobalOrdinal(useStore, k) { + t.Errorf("UseStore body %s = %+v, want of %q", k, n, of) + } + } + if len(c.Summary) != 4 { + t.Errorf("UseStore summary = %v, want all-to-all over 2×2", c.Summary) + } + in := map[[3]string]bool{} + for _, e := range app.ParamIn { + in[[3]string{e.Src, e.Dst, e.Var}] = true + } + out := map[[3]string]bool{} + for _, e := range app.ParamOut { + out[[3]string{e.Src, e.Dst, e.Var}] = true + } + site := func(k string) string { return v2.GlobalOrdinal(useStore, "34:9/"+k) } + for _, e := range [][3]string{ + {site("actual_in:0"), method("Store", "Get") + "@formal_in:0", "_recv"}, + {site("actual_in:0"), method("memStore", "Get") + "@formal_in:0", "s"}, + {site("actual_in:0"), method("sqlStore", "Get") + "@formal_in:0", "s"}, + {site("actual_in:1"), method("sqlStore", "Get") + "@formal_in:1", "k"}, + } { + if !in[e] { + t.Errorf("no param_in %v", e) + } + } + for _, e := range [][3]string{ + {method("Store", "Get") + "@formal_out:0", site("actual_out:0"), ""}, + {method("memStore", "Get") + "@formal_out:1", site("actual_out:1"), "s"}, + } { + if !out[e] { + t.Errorf("no param_out %v", e) + } + } + stores := v2.CallableID(mod, "example.com/dataflow/sdg.Stores") + if !in[[3]string{v2.GlobalOrdinal(stores, "40:2/actual_in:1"), useStore + "@formal_in:1", "k"}] { + t.Errorf("no param_in Stores@40:2/actual_in:1 -> UseStore@formal_in:1") + } +} + +// TestGateL4_StandInBody: a library callee the project calls carries its +// stand-in formals in external_symbols[].body (spec §6), and a param edge +// reaches them. +func TestGateL4_StandInBody(t *testing.T) { + fx := buildL4(t, "dataflow") + app := fx.l4.Application + id := v2.ExternalID(app.ID, "fmt", "Sprintf") + ext, ok := app.ExternalSymbols[id] + if !ok { + t.Fatalf("no external %s", id) + } + // a ...any is a by-ref out (an interface slice), so formal_out is indexed. + for k, of := range map[string]string{"@formal_in:0": "format", "@formal_in:1": "a", "@formal_out:0": "", "@formal_out:1": "a"} { + n, ok := ext.Body[k] + if !ok || n.Of != of || n.ID != id+k { + t.Errorf("%s body %s = %+v, want of %q", id, k, n, of) + } + } + reached := false + for _, e := range app.ParamIn { + reached = reached || e.Dst == id+"@formal_in:0" + } + if !reached { + t.Errorf("no param_in reaches %s@formal_in:0", id) + } + if _, ok := fx.l3.Application.ExternalSymbols[id]; !ok { + t.Fatalf("-a 3 lacks %s", id) + } + if fx.l3.Application.ExternalSymbols[id].Body != nil { + t.Errorf("-a 3 carries a stand-in body") + } +} + +// TestGateL4_DegradedMintsCallables: a degraded run's markers land on the +// envelope with each callable as its can:// id, sorted, entries by code. +func TestGateL4_DegradedMintsCallables(t *testing.T) { + fx := buildL4(t, "rangefunc") + deg := fx.l4.Degraded + if len(deg) == 0 { + t.Fatal("rangefunc: no degraded marker") + } + ids := allCallableIDs(fx.l4) + found := false + for i, d := range deg { + if i > 0 && cmpStrings(deg[i-1].Code, d.Code, deg[i-1].Detail, d.Detail) > 0 { + t.Errorf("degraded not sorted by code: %v", deg) + } + if !slices.IsSorted(d.Callables) { + t.Errorf("%s callables not sorted: %v", d.Code, d.Callables) + } + for _, id := range d.Callables { + if !ids[id] { + t.Errorf("%s names %s, not a callable id", d.Code, id) + } + } + if d.Code == sdg.CodeRangeFunc { + found = true + want := v2.CallableID(v2.ModuleID(fx.l4.Application.ID, "r.go"), "example.com/rangefunc.RangeFunc") + if !slices.Contains(d.Callables, want) { + t.Errorf("range_func callables %v lack %s", d.Callables, want) + } + } + } + if !found { + t.Errorf("degraded %+v has no range_func", deg) + } + if fx.l3.Degraded != nil { + t.Errorf("-a 3 carries degraded") + } + if len(buildL4(t, "dataflow").l4.Degraded) != 0 { + t.Errorf("dataflow degraded = %+v, want none", buildL4(t, "dataflow").l4.Degraded) + } +} + +// TestGateL4_StoredLiteralGlobalReachable (Task 13 review): storedlit's +// Publish writes the global sink only inside a literal it stores, and Fire +// calls that literal through a function value nothing resolves — so inside +// Publish, sink's formal_out is fed only by its formal_in (the identity +// edge). The flow v → sink → readSink → must still be reachable in +// the emitted graph (ddg ∪ cdg ∪ summary ∪ param_in ∪ param_out, the SDK's +// walk): the caller's host statement Publish(v) weak-defines sink, whose +// later use feeds readSink's global formal_in. +func TestGateL4_StoredLiteralGlobalReachable(t *testing.T) { + fx := buildL4(t, "storedlit") + a := fx.l4 + succ := map[string][]string{} + for _, c := range l4Callables(a) { + g := func(k string) string { return v2.GlobalOrdinal(c.ID, k) } + for _, e := range c.Ddg { + succ[g(e.Src)] = append(succ[g(e.Src)], g(e.Dst)) + } + for _, e := range c.Cdg { + succ[g(e.Src)] = append(succ[g(e.Src)], g(e.Dst)) + } + for _, e := range c.Summary { + succ[g(e.Src)] = append(succ[g(e.Src)], g(e.Dst)) + } + } + for _, e := range slices.Concat(a.Application.ParamIn, a.Application.ParamOut) { + succ[e.Src] = append(succ[e.Src], e.Dst) + } + reach := func(from string) map[string]bool { + seen := map[string]bool{from: true} + for work := []string{from}; len(work) > 0; { + n := work[len(work)-1] + work = work[:len(work)-1] + for _, m := range succ[n] { + if !seen[m] { + seen[m] = true + work = append(work, m) + } + } + } + return seen + } + mod := v2.ModuleID(a.Application.ID, "storedlit.go") + fn := func(name string) string { return v2.CallableID(mod, "example.com/storedlit."+name) } + idx := bodyIndex(a) + formal := func(owner, kind, of string) string { + for id, on := range idx { + if on.owner == owner && on.node.Kind == kind && on.node.Of == of { + return id + } + } + t.Fatalf("%s has no %s of %q", owner, kind, of) + return "" + } + src := formal(fn("PublishThenRead"), "formal_in", "v") + r := reach(src) + for _, dst := range []string{ + formal(fn("readSink"), "formal_in", ":example.com/storedlit::sink"), + formal(fn("PublishThenRead"), "formal_out", ""), + } { + if !r[dst] { + t.Errorf("%s does not reach %s", src, dst) + } + } + // The gap the caller covers: over Publish's own edges (ddg ∪ cdg ∪ + // summary), v does not reach sink's formal_out — sink's formal_out is fed + // only by its formal_in, the identity edge, as the literal's write has no + // param_out. Update this when the gap closes. + var pub v2.Callable + for _, c := range l4Callables(a) { + if c.ID == fn("Publish") { + pub = c + } + } + local := map[string][]string{} + for _, e := range pub.Ddg { + local[e.Src] = append(local[e.Src], e.Dst) + } + for _, e := range pub.Cdg { + local[e.Src] = append(local[e.Src], e.Dst) + } + for _, e := range pub.Summary { + local[e.Src] = append(local[e.Src], e.Dst) + } + keyOf := func(id string) string { return idx[id].key } + seen := map[string]bool{keyOf(formal(fn("Publish"), "formal_in", "v")): true} + for work := []string{keyOf(formal(fn("Publish"), "formal_in", "v"))}; len(work) > 0; { + n := work[len(work)-1] + work = work[:len(work)-1] + for _, m := range local[n] { + if !seen[m] { + seen[m] = true + work = append(work, m) + } + } + } + sinkOut := keyOf(formal(fn("Publish"), "formal_out", ":example.com/storedlit::sink")) + if seen[sinkOut] { + t.Errorf("Publish: v now reaches %s intra-procedurally; the stored-literal gap closed, update this test", sinkOut) + } + var feeders []string + for _, e := range pub.Ddg { + if e.Dst == sinkOut { + feeders = append(feeders, e.Src) + } + } + if !slices.Equal(feeders, []string{keyOf(formal(fn("Publish"), "formal_in", ":example.com/storedlit::sink"))}) { + t.Errorf("Publish: %s fed by %v, want only sink's formal_in (identity edge)", sinkOut, feeders) + } +} + +// buildL4Jobs runs the -a 4 pipeline over dir from scratch with -j jobs and +// returns EmitTo's bytes, checking they equal json.Marshal(Emit(...)). +func buildL4Jobs(t *testing.T, dir, name string, jobs int) []byte { + t.Helper() + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) + edges := cg.Build(st) + app := &schema.GoApplication{SymbolTable: st, CallGraph: edges, Externals: cg.Externals()} + units := b.L3Units(st) + res := dataflow.RunWithSideTables(b.Fset(), units, 3, jobs) + l4 := sdg.Run(sdg.Input{ + Fset: b.Fset(), Pkgs: b.Pkgs(), ProjectDir: dir, SymbolTable: st, + Externals: app.Externals, CallGraph: app.CallGraph, + Units: units, L3: res, K: 3, Jobs: jobs, + }) + l4.ApplyTo(app) + dataflow.ReleaseAll(res) + var buf bytes.Buffer + if err := EmitTo(&buf, app, name, dir, 4, "det", 3, res, &l4); err != nil { + t.Fatal(err) + } + want, err := json.Marshal(Emit(app, name, dir, 4, "det", 3, res, &l4)) + if err != nil { + t.Fatal(err) + } + if !bytes.Equal(buf.Bytes(), want) { + t.Fatalf("%s: EmitTo differs from json.Marshal(Emit) at byte %d (len %d vs %d)", name, firstDiff(buf.Bytes(), want), buf.Len(), len(want)) + } + return buf.Bytes() +} + +// TestGateL4_DeterministicAcrossJobs: the -a 4 JSON is byte-identical +// between -j 1 and -j 8 and across fresh runs, on sdg, flow and chi; the +// streaming writer matches json.Marshal of the built tree. +func TestGateL4_DeterministicAcrossJobs(t *testing.T) { + _, thisFile, _, _ := runtime.Caller(0) + td := filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata") + for name, rel := range map[string]string{ + "sdg": filepath.Join("dataflow", "sdg"), + "flow": filepath.Join("dataflow", "flow"), + "chi": "chi", + } { + dir, err := filepath.Abs(filepath.Join(td, rel)) + if err != nil { + t.Fatal(err) + } + first := buildL4Jobs(t, dir, name, 1) + if !bytes.Contains(first, []byte(`"param_in":[`)) { + t.Errorf("%s: -a 4 JSON has no param_in", name) + } + for i, jobs := range []int{8, 1, 8} { + if got := buildL4Jobs(t, dir, name, jobs); !bytes.Equal(got, first) { + t.Errorf("%s: run %d (-j %d) differs from -j 1 at byte %d", name, i+2, jobs, firstDiff(got, first)) + } + } + } +} + +// TestEmitL4_UnresolvedRefIsDroppedAndWarned: a param edge whose endpoint +// names no emitted callable is a bug upstream; the emitter never writes it +// with a dangling endpoint (it warns and drops it). +func TestEmitL4_UnresolvedRefIsDroppedAndWarned(t *testing.T) { + fx := buildL4(t, "dataflow") + res := *fx.sdg + ghost := sdg.Target{Callable: dataflow.L3Key{Path: "nowhere.go", Signature: "example.com/nowhere.F"}} + res.ParamIn = append(slices.Clone(res.ParamIn), sdg.ParamEdge{ + Src: sdg.Ref{Owner: ghost, Key: "1:1/actual_in:0"}, + Dst: sdg.Ref{Owner: ghost, Key: "@formal_in:0"}, + Var: "x", + }) + var warned []string + prev := warnf + warnf = func(format string, args ...any) { warned = append(warned, fmt.Sprintf(format, args...)) } + defer func() { warnf = prev }() + app := &schema.GoApplication{SymbolTable: fx.st, Externals: map[string]schema.GoExternalCallee{}} + out := Emit(app, "dataflow", fx.dir, 4, "t", 3, fx.res, &res) + for _, e := range out.Application.ParamIn { + if strings.Contains(e.Src, "nowhere") || strings.Contains(e.Dst, "nowhere") { + t.Errorf("dangling param_in emitted: %+v", e) + } + } + if len(warned) == 0 || !strings.Contains(strings.Join(warned, "\n"), "nowhere.go") { + t.Errorf("no warning names the unresolved ref; warned %q", warned) + } +} + +// TestGateL4_PointsToEdges (spec §4.10): AliasPtr's write through the copy +// p reaches the read through req, and req's formal_out, as ddg edges with +// prov ["points-to"], distinct from the reaching-defs edges on the same +// endpoints. +func TestGateL4_PointsToEdges(t *testing.T) { + fx := buildL4(t, "dataflow") + mod := v2.ModuleID(fx.l4.Application.ID, "sdg/sdg.go") + id := v2.CallableID(mod, "example.com/dataflow/sdg.AliasPtr") + var got [][3]string + for _, c := range l4Callables(fx.l4) { + if c.ID != id { + continue + } + for _, e := range c.Ddg { + if slices.Equal(e.Prov, []string{"points-to"}) { + got = append(got, [3]string{e.Src, e.Dst, e.Var}) + } + } + } + want := [][3]string{{"360:2", "361:2", "req.UserID"}, {"360:2", "@formal_out:1", "req"}} + if !slices.Equal(got, want) { + t.Errorf("AliasPtr points-to ddg = %v, want %v", got, want) + } +} + +// TestSupersetL4_DetectsEveryViolation: the -a 3 ⊆ -a 4 checker is a gate, so +// it must itself fail on each way a superset can break, and accept a pure +// addition. Without this a bug in supersetL4 would make that gate vacuous. +func TestSupersetL4_DetectsEveryViolation(t *testing.T) { + parse := func(s string) any { + var v any + if err := json.Unmarshal([]byte(s), &v); err != nil { + t.Fatal(err) + } + return v + } + base := `{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1,2]}}}}` + for name, c := range map[string]struct { + b string + want string // a substring of the error; "" means no error + }{ + "identical": {base, ""}, + "pure addition": {`{"application":{"call_graph":[{"src":"a","dst":"b"},{"src":"a","dst":"c"}],"x":{"f":{"ddg":[{"src":"0","dst":"1","var":"w"},{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1,2,3],"new":1}},"extra":true}}`, ""}, + "removed key": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"list":[1,2]}}}}`, "k missing at L4"}, + "changed value": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"w","list":[1,2]}}}}`, "value changed"}, + "shrunk list": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1]}}}}`, "list shrank"}, + "changed type": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":{"a":1}}}}}`, "list shrank or changed type"}, + "removed ddg edge": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":[{"src":"0","dst":"1","var":"w"}],"k":"v","list":[1,2]}}}}`, "missing or changed at L4"}, + "object to scalar": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":5}}}`, "object changed type"}, + "edge list to object": {`{"application":{"call_graph":[{"src":"a","dst":"b"}],"x":{"f":{"ddg":{},"k":"v","list":[1,2]}}}}`, ".ddg: changed type"}, + "changed call_graph": {`{"application":{"call_graph":[{"src":"a","dst":"z"}],"x":{"f":{"ddg":[{"src":"1","dst":"2","var":"v"}],"k":"v","list":[1,2]}}}}`, "call_graph: edge"}, + } { + var errs []string + supersetL4("$", parse(base), parse(c.b), &errs) + got := strings.Join(errs, "; ") + switch { + case c.want == "" && len(errs) > 0: + t.Errorf("%s: want no error, got %s", name, got) + case c.want != "" && !strings.Contains(got, c.want): + t.Errorf("%s: want an error containing %q, got %q", name, c.want, got) + } + } +} diff --git a/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go b/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go new file mode 100644 index 0000000..2cff0b1 --- /dev/null +++ b/internal/syntactic_analysis/v2emit/gate_l4_wrap_test.go @@ -0,0 +1,110 @@ +package v2emit + +import ( + "testing" + + v2 "github.com/codellm-devkit/codeanalyzer-go/internal/schema/v2" +) + +// l4Vertex is the one body node of owner with kind and of in a's tree. +func l4Vertex(t *testing.T, a *v2.Analysis, owner, kind, of string) string { + t.Helper() + var found []string + for id, on := range bodyIndex(a) { + if on.owner == owner && on.node.Kind == kind && on.node.Of == of { + found = append(found, id) + } + } + if len(found) != 1 { + t.Fatalf("%s has %d %s vertices of %q, want 1", owner, len(found), kind, of) + } + return found[0] +} + +// hasVertex reports whether owner has a body node of kind and of in a's tree. +func hasVertex(a *v2.Analysis, owner, kind, of string) bool { + for _, on := range bodyIndex(a) { + if on.owner == owner && on.node.Kind == kind && on.node.Of == of { + return true + } + } + return false +} + +// TestGateL4_FlowBehindSyntheticWrappers (F1): on testdata/dataflow/wrap, +// each handler's id reaches Exec's query formal — called directly through +// the interface, through a method value of it ((Store).Find$bound) and +// through a struct embedding it (a promotion wrapper) — and each handler's +// formal_out of the global sqlStore.Find writes exists. +func TestGateL4_FlowBehindSyntheticWrappers(t *testing.T) { + a := buildL4(t, "dataflow/wrap").l4 + if len(a.Degraded) != 0 { + t.Fatalf("wrap degraded: %+v", a.Degraded) + } + mod := v2.ModuleID(a.Application.ID, "wrap.go") + pkg := "example.com/dataflow/wrap." + fn := func(name string) string { return v2.CallableID(mod, pkg+name) } + sink := l4Vertex(t, a, fn("Exec"), "formal_in", "query") + g := newSDGGraph(a) + for _, h := range []string{"HandleDirect", "HandleBound", "HandleDecorated"} { + src := l4Vertex(t, a, fn(h), "formal_in", "id") + if p := g.path(src, sink); p == nil { + t.Errorf("%s's id does not reach Exec's query", h) + } + if !hasVertex(a, fn(h), "formal_out", ":example.com/dataflow/wrap::lastQuery") { + t.Errorf("%s has no formal_out of the global sqlStore.Find writes", h) + } + } +} + +// TestGateL4_FlowThroughLibraryStorage (F3): on testdata/dataflow/libstore, +// a Store put in a context.Context or a sync.Map comes back out of library +// code and is called; each handler's id reaches Exec's query formal through +// the CHA complement's sqlStore.Find, and the handler's formal_out of the +// global sqlStore.Find writes exists. +func TestGateL4_FlowThroughLibraryStorage(t *testing.T) { + a := buildL4(t, "dataflow/libstore").l4 + if len(a.Degraded) != 0 { + t.Fatalf("libstore degraded: %+v", a.Degraded) + } + mod := v2.ModuleID(a.Application.ID, "libstore.go") + pkg := "example.com/dataflow/libstore." + fn := func(name string) string { return v2.CallableID(mod, pkg+name) } + sink := l4Vertex(t, a, fn("Exec"), "formal_in", "query") + g := newSDGGraph(a) + for _, h := range []string{"HandleCtx", "HandleMap"} { + src := l4Vertex(t, a, fn(h), "formal_in", "id") + if p := g.path(src, sink); p == nil { + t.Errorf("%s's id does not reach Exec's query", h) + } + if !hasVertex(a, fn(h), "formal_out", ":example.com/dataflow/libstore::lastQuery") { + t.Errorf("%s has no formal_out of the global sqlStore.Find writes", h) + } + } +} + +// TestGateL4_FlowThroughProjectCode (round 2): on testdata/dataflow/libflow, +// a sqlStore put in library storage reaches an interface call through +// project code (a result, a parameter, a field, a global, a capture) or a +// library container (map, slice, array, channel); each handler's id reaches +// Exec's query formal through the CHA complement's sqlStore.Find. +func TestGateL4_FlowThroughProjectCode(t *testing.T) { + a := buildL4(t, "dataflow/libflow").l4 + if len(a.Degraded) != 0 { + t.Fatalf("libflow degraded: %+v", a.Degraded) + } + mod := v2.ModuleID(a.Application.ID, "libflow.go") + pkg := "example.com/dataflow/libflow." + fn := func(name string) string { return v2.CallableID(mod, pkg+name) } + method := func(typ, m string) string { return v2.CallableID(v2.TypeID(mod, pkg+typ), pkg+typ+"."+m) } + sink := l4Vertex(t, a, fn("Exec"), "formal_in", "query") + g := newSDGGraph(a) + for _, src := range []string{ + fn("ViaReturn"), fn("ViaParam"), method("handler", "ViaField"), fn("ViaGlobal"), fn("ViaCapture"), + fn("ViaMap"), fn("ViaSlice"), fn("ViaChan"), fn("ViaArray"), + } { + if p := g.path(l4Vertex(t, a, src, "formal_in", "id"), sink); p == nil { + t.Errorf("%s's id does not reach Exec's query", src) + } + } +} diff --git a/internal/syntactic_analysis/v2emit/gate_test.go b/internal/syntactic_analysis/v2emit/gate_test.go index 82a6626..4679a60 100644 --- a/internal/syntactic_analysis/v2emit/gate_test.go +++ b/internal/syntactic_analysis/v2emit/gate_test.go @@ -58,7 +58,7 @@ func buildMultipackageV2(t *testing.T) *v2.Analysis { t.Fatalf("symbol table build failed: %v", err) } app := &schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}} - return Emit(app, "multipackage", dir, 1, "gate-test", 0, nil) + return Emit(app, "multipackage", dir, 1, "gate-test", 0, nil, nil) } // mustModule fetches a module by relative path or fails. @@ -322,7 +322,7 @@ func TestGateL1_SupersetOfV1(t *testing.T) { t.Fatalf("symbol table build failed: %v", err) } app := &schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}} - out := Emit(app, "multipackage", dir, 1, "gate-test", 0, nil) + out := Emit(app, "multipackage", dir, 1, "gate-test", 0, nil, nil) // Every v1 file, type, method and function reappears in v2. for rel, v1file := range st { @@ -387,7 +387,7 @@ func buildDataflowL1V2(t *testing.T) *v2.Analysis { t.Fatalf("symbol table build failed: %v", err) } app := &schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}} - return Emit(app, "dataflow", dir, 1, "gate-test", 0, nil) + return Emit(app, "dataflow", dir, 1, "gate-test", 0, nil, nil) } func TestGateL1_TypeKindsAliasAndDefined(t *testing.T) { @@ -478,7 +478,7 @@ func buildDataflowL2V2(t *testing.T) *v2.Analysis { cg := semantic_analysis.NewCallGraphBuilder(dir, b.Fset(), b.Pkgs()) edges := cg.Build(st) app := &schema.GoApplication{SymbolTable: st, CallGraph: edges} - return Emit(app, "dataflow", dir, 2, "gate-test", 0, nil) + return Emit(app, "dataflow", dir, 2, "gate-test", 0, nil, nil) } // TestGateL2_CoPositionedCallsResolveSeparately: in s.A().B() the outer call @@ -620,3 +620,70 @@ func anyEdgeFromTo(edges []v2.Edge, src, dst string) bool { } return false } + +// TestGate_LineDirectiveSpansSliceToText (#26): positions are real file +// positions (token.FileSet.PositionFor(p, false)), never //line-adjusted, so +// under a directive every span still indexes module.source. In +// testdata/dataflow/linedir every call node's span slices to text that +// starts with its callee expression, every callable's to its declaration, +// and the call keys name the calls' real line:col. +func TestGate_LineDirectiveSpansSliceToText(t *testing.T) { + _, thisFile, _, _ := runtime.Caller(0) + dir, err := filepath.Abs(filepath.Join(filepath.Dir(thisFile), "..", "..", "..", "testdata", "dataflow")) + if err != nil { + t.Fatal(err) + } + b := syntactic_analysis.NewSymbolTableBuilder(dir) + st, err := b.Build(nil, true) + if err != nil { + t.Fatalf("symbol table build failed: %v", err) + } + out := Emit(&schema.GoApplication{SymbolTable: st, CallGraph: []schema.GoCallEdge{}}, "dataflow", dir, 1, "gate-test", 0, nil, nil) + const rel = "linedir/linedir.go" + mod := mustModule(t, out, rel) + calls := 0 + for sig, fn := range st[rel].Functions { + c := mod.Functions[sig] + text := c.Span.Slice(mod.Source) + if c.Span.Bytes[0] >= c.Span.Bytes[1] || !strings.HasPrefix(text, "func "+fn.Name+"(") { + t.Errorf("callable %s: span %v slices to %q, want func %s(…", sig, c.Span, text, fn.Name) + } + keys := v2.CallBodyKeys(siteStarts(fn.CallSites)) + for i, cs := range fn.CallSites { + n, ok := c.Body[keys[i]] + if !ok { + t.Errorf("%s: no call node %s; body %v", sig, keys[i], keys) + continue + } + calls++ + callee := cs.MethodName + if cs.ReceiverExpr != "" { + callee = cs.ReceiverExpr + "." + callee + } + if got := n.Span.Slice(mod.Source); n.Span.Bytes[0] >= n.Span.Bytes[1] || !strings.HasPrefix(got, callee+"(") { + t.Errorf("%s call %s: span %v slices to %q, want %s(…", sig, keys[i], n.Span, got, callee) + } + } + } + caller := functionByName(t, mod, "Caller") + if got := strings.Join(sortedKeys(caller.Body), ","); got != "13:12,13:2,15:2" { + t.Errorf("Caller call keys = %s, want the real 13:12,13:2,15:2", got) + } + if calls != 3 { + t.Errorf("checked %d call nodes, want 3", calls) + } +} + +func siteStarts(sites []schema.GoCallsite) [][2]int { + pos := make([][2]int, len(sites)) + for i, cs := range sites { + pos[i] = [2]int{cs.StartLine, cs.StartColumn} + } + return pos +} + +func sortedKeys[V any](m map[string]V) []string { + ks := keys(m) + sort.Strings(ks) + return ks +} diff --git a/internal/syntactic_analysis/v2emit/sigindex.go b/internal/syntactic_analysis/v2emit/sigindex.go index 992af19..00ed238 100644 --- a/internal/syntactic_analysis/v2emit/sigindex.go +++ b/internal/syntactic_analysis/v2emit/sigindex.go @@ -80,8 +80,14 @@ func (idx sigIndex) addCallable(parentID, relPath string, c schema.GoCallable) { } // Precise entry: a callable's declaring file plus its signature is unique - // even for init, because at most one init exists per file. + // even for init, because at most one init exists per file. Call-graph + // edges carry the declaring file (c.Path), which differs from relPath for + // a method declared apart from its receiver type, and for the closures in + // one; index both so either resolves. idx.bySigPath[sigPathKey{sig: c.Signature, path: relPath}] = callID + if c.Path != "" && c.Path != relPath { + idx.bySigPath[sigPathKey{sig: c.Signature, path: c.Path}] = callID + } for _, inner := range c.InnerCallables { idx.addCallable(callID, relPath, inner) diff --git a/scripts/eval_l3.sh b/scripts/eval_l3.sh index 0cb8fb7..0065c84 100755 --- a/scripts/eval_l3.sh +++ b/scripts/eval_l3.sh @@ -39,7 +39,12 @@ # 2. cango -a 3 -j 1 # 3. cango -a 3 -j N (N = CPU count) # 4. cango -a 3 -j N (second run) -# and compares the three -a 3 analysis.json files byte for byte. It records wall time, peak +# 5. cango -a 4 -j 1 -v (only when CANGO_LEVELS includes 4; steps 5-7) +# 6. cango -a 4 -j N +# 7. cango -a 4 -j N (second run) +# and compares the three -a 3 analysis.json files byte for byte, and the three -a 4 ones. A run that +# exits non-zero or times out (the -a 4 runs included), or a byte difference at either level, fails +# the app, and the script then exits 1. It records wall time, peak # RSS (macOS /usr/bin/time -l), module/callable/body-node/edge counts, external_symbols, the # call_graph split, the prepare-time type-check status, the number of type-error warnings # ("package ..." stderr lines), the analyzer's too-old-toolchain warning, and the number of @@ -59,8 +64,16 @@ # EVAL_JOBS N for the -j N runs (default: CPU count) # RUN_TIMEOUT per-run limit, seconds (default 3600); on expiry the run is killed, the app # is marked TIMEOUT and its remaining runs are skipped -# EVAL_PRUNE 1 = after the comparison, delete the warm-up and both -j N analysis.json files -# (keeps a2j1 and a3j1, and every .err/.time). For very large apps. +# CANGO_LEVELS levels to run (default "2 3"). Levels 2 and 3 always run (the L3 baseline); +# adding 4 (e.g. "2 3 4") also runs steps 5-7 and records, from the -a 4 -j 1 run: +# wall time, peak RSS (the whole process: L1-L3, SSA, the L4 overlay and the +# emission) and its ratio to -a 3, user CPU, analysis.json bytes, the SSA build, VTA +# and L4-overlay times, and from its -v log the L4 counts (vertices, param_in, +# param_out, summary, port ddg, points-to ddg, stand-ins, call sites with dispatch +# targets, unmapped SSA functions/calls) and the degraded codes; plus the -a 4 -j N +# wall time and the -a 4 byte-identity check. +# EVAL_PRUNE 1 = after the comparison, delete the warm-up, both -j N and every -a 4 +# analysis.json file (keeps a2j1 and a3j1, and every .err/.time). For very large apps. # Any other environment (GOOS, GOFLAGS, ...) is inherited by the analyzer unchanged. # # Requires: git, go, /usr/bin/time -l (BSD/macOS), cmp. jq is used for counts; without it the @@ -73,6 +86,9 @@ EVAL_GO=${EVAL_GO:-go1.27.3} DATE=$(date +%Y-%m-%d) EVAL_NOTE=${EVAL_NOTE:-$EVAL_OUT/l3-$DATE.md} RUN_TIMEOUT=${RUN_TIMEOUT:-3600} +CANGO_LEVELS=${CANGO_LEVELS:-"2 3"} +RUN_L4=0 +case " $CANGO_LEVELS " in *" 4 "*) RUN_L4=1 ;; esac EVAL_PRUNE=${EVAL_PRUNE:-0} JOBS=${EVAL_JOBS:-$( (sysctl -n hw.ncpu 2>/dev/null || nproc 2>/dev/null || echo 4) | head -1)} REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd) @@ -142,7 +158,7 @@ if [ "${1:-}" = "--clean" ]; then [ -d "$EVAL_OUT" ] || exit 0 case $EVAL_OUT in /|"$HOME"|"$EVAL_SRC") echo "refusing to clean $EVAL_OUT" >&2; exit 2 ;; esac for n in $(manifest_names) $(ls "$EVAL_OUT" 2>/dev/null | sed -n 's/-a3j1$//p'); do - for t in warm a2j1 a3j1 a3jN a3jN2; do + for t in warm a2j1 a3j1 a3jN a3jN2 a4j1 a4jN a4jN2; do rm -rf "$EVAL_OUT/$n-$t" "$EVAL_OUT/$n-$t.err" "$EVAL_OUT/$n-$t.time" "$EVAL_OUT/$n-$t.timeout" done rm -rf "$EVAL_OUT/cache-$n" @@ -321,15 +337,16 @@ JQ_COUNTS=' ([$a.call_graph[]|select(.dst|contains("/@external/")|not)]|length), ([$a.call_graph[]|select(.dst|contains("/@external/"))]|length) ] | @tsv' -# run_one NAME TAG LEVEL JOBS -> sets RC WALL RSS; output in $EVAL_OUT/NAME-TAG/analysis.json +# run_one NAME TAG LEVEL JOBS [FLAG...] -> sets RC WALL UCPU (user CPU) RSS BYTES; output in $EVAL_OUT/NAME-TAG/analysis.json run_one() { local name=$1 tag=$2 level=$3 jobs=$4 + shift 4 local dir=$EVAL_OUT/$name-$tag rm -rf "$dir"; mkdir -p "$dir" # shellcheck disable=SC2086 env $APP_ENV GOTOOLCHAIN="$RUN_GO" /usr/bin/time -l -o "$dir.time" \ "$CANGO" -i "$APP" --app-name "$name" -a "$level" -j "$jobs" \ - --eager -c "$EVAL_OUT/cache-$name" -o "$dir" >/dev/null 2>"$dir.err" & + --eager -c "$EVAL_OUT/cache-$name" -o "$dir" "$@" >/dev/null 2>"$dir.err" & local pid=$! waited=0 while kill -0 "$pid" 2>/dev/null; do if [ "$waited" -ge "$RUN_TIMEOUT" ]; then @@ -343,9 +360,30 @@ run_one() { RC=$? if [ -e "$dir.timeout" ]; then RC=124; fi WALL=$(awk '/ real /{print $1}' "$dir.time") + UCPU=$(awk '/ real /{print $3}' "$dir.time") RSS=$(awk '/maximum resident set size/{printf "%d", $1/1048576}' "$dir.time") + BYTES=$(wc -c <"$dir/analysis.json" 2>/dev/null | tr -d ' ') [ -n "$WALL" ] || WALL=n/a + [ -n "$UCPU" ] || UCPU=n/a [ -n "$RSS" ] || RSS=n/a + [ -n "$BYTES" ] || BYTES=n/a +} + +# l4_log ERRFILE -> the L4 columns of the note from an -a 4 -v stderr log: SSA, VTA and overlay +# times, then vertices | param_in | param_out | summary | port ddg | points-to | stand-ins | +# call sites with targets | unmapped functions/calls | degraded codes (sorted, comma-joined). +l4_log() { + local e=$1 ssa vta ov counts sites unm codes + ssa=$(sed -n 's/.*L4: SSA built .* in \(.*\)$/\1/p' "$e" 2>/dev/null | head -1) + vta=$(sed -n 's/.*L4: VTA over .* edges in \(.*\)$/\1/p' "$e" 2>/dev/null | head -1) + ov=$(sed -n 's/.*L4: [0-9]* vertices, .* stand-ins in \(.*\)$/\1/p' "$e" 2>/dev/null | head -1) + counts=$(sed -n 's/.*L4: \([0-9]*\) vertices, \([0-9]*\) port edges (\([0-9]*\) points-to), \([0-9]*\) summary edges, \([0-9]*\) param_in, \([0-9]*\) param_out, \([0-9]*\) stand-ins in .*/\1 | \5 | \6 | \4 | \2 | \3 | \7/p' "$e" 2>/dev/null | head -1) + sites=$(sed -n 's/.*L4: dispatch: \([0-9]*\) call sites with targets.*/\1/p' "$e" 2>/dev/null | head -1) + unm=$(sed -n 's/.*L4: SSA join: .*; \([0-9]*\) functions, \([0-9]*\) calls unmapped in .*/\1\/\2/p' "$e" 2>/dev/null | head -1) + codes=$(sed -n 's/.*L4 degraded: \([a-z_]*\) .*/\1/p' "$e" 2>/dev/null | sort -u | paste -sd, -) + [ -n "$counts" ] || counts="n/a | n/a | n/a | n/a | n/a | n/a | n/a" + printf '%s | %s | %s | %s | %s | %s | %s' "${ssa:-n/a}" "${vta:-n/a}" "${ov:-n/a}" "$counts" \ + "${sites:-n/a}" "${unm:-n/a}" "${codes:-none}" } BUILD_INFO=$(go version -m "$CANGO" 2>/dev/null) @@ -358,8 +396,14 @@ CANGO_DIRTY=$(printf '%s\n' "$BUILD_INFO" | awk -F'=' '/vcs.modified=/{print $2} echo echo "Analyzer: $("$CANGO" --version 2>&1 | head -1), built with ${CANGO_GO:-unknown} (runtime.Version), rev ${CANGO_REV:-unknown}; runs use GOTOOLCHAIN=$RUN_GO; host: $(uname -sm); j=$JOBS; all runs --eager." echo - echo "| app | status | type-check | wall a2 j1 (s) | wall a3 j1 (s) | wall a3 jN (s) | RSS a3 j1 (MB) | modules | callables | body nodes | cfg | cdg | ddg | ext symbols | cg in-project | cg external | -j identical | L3 skips | type errors | toolchain warn |" - echo "|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|" + if [ "$RUN_L4" -eq 1 ]; then + L4H=" wall a4 j1 (s) | wall a4 jN (s) | user CPU a4 j1 (s) | RSS a4 j1 (MB) | RSS a4/a3 | JSON a3 (B) | JSON a4 (B) | SSA | VTA | L4 overlay | vertices | param_in | param_out | summary | port ddg | points-to | stand-ins | sites with targets | unmapped fn/calls | degraded | a4 -j identical |" + L4S="---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|" + else + L4H=""; L4S="" + fi + echo "| app | status | type-check | wall a2 j1 (s) | wall a3 j1 (s) | wall a3 jN (s) | RSS a3 j1 (MB) | modules | callables | body nodes | cfg | cdg | ddg | ext symbols | cg in-project | cg external | -j identical | L3 skips | type errors | toolchain warn |$L4H" + echo "|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|$L4S" } >"$EVAL_NOTE" FAILS="" @@ -383,17 +427,40 @@ for T in $TARGETS; do status=ok rm -f "$EVAL_OUT/$name"-*.timeout - W3=n/a; WN=n/a; R3=n/a; RC3=0; RCN=0; RCN2=0 + W3=n/a; WN=n/a; R3=n/a; B3=n/a; RC3=0; RCN=0; RCN2=0 # Warm-up (discarded): populates the Go build/module cache and the OS file cache so the # timed -a 2 and -a 3 runs are comparable. run_one "$name" warm 2 1; RCW=$RC run_one "$name" a2j1 2 1; W2=$WALL; RC2=$RC - if [ "$RCW" -ne 124 ] && [ "$RC" -ne 124 ]; then run_one "$name" a3j1 3 1; W3=$WALL; R3=$RSS; RC3=$RC; fi + if [ "$RCW" -ne 124 ] && [ "$RC" -ne 124 ]; then run_one "$name" a3j1 3 1; W3=$WALL; R3=$RSS; RC3=$RC; B3=$BYTES; fi if [ "$RC3" -ne 124 ] && [ "$RC2" -ne 124 ]; then run_one "$name" a3jN 3 "$JOBS"; WN=$WALL; RCN=$RC if [ "$RCN" -ne 124 ]; then run_one "$name" a3jN2 3 "$JOBS"; RCN2=$RC; fi fi + L4COLS="" + if [ "$RUN_L4" -eq 1 ]; then + W4=n/a; WN4=n/a; U4=n/a; R4=n/a; B4=n/a; RATIO=n/a; RC4=0; RCN4=0; RCN42=0; ident4=n/a + if [ "$RC3" -ne 124 ] && [ "$RC2" -ne 124 ]; then + run_one "$name" a4j1 4 1 -v; W4=$WALL; U4=$UCPU; R4=$RSS; B4=$BYTES; RC4=$RC + if [ "$RC4" -ne 124 ]; then run_one "$name" a4jN 4 "$JOBS"; WN4=$WALL; RCN4=$RC; fi + if [ "$RC4" -ne 124 ] && [ "$RCN4" -ne 124 ]; then run_one "$name" a4jN2 4 "$JOBS"; RCN42=$RC; fi + # One status for the triple: the first non-zero rc (124 = timeout). + for r in $RC4 $RCN4 $RCN42; do [ "$RC4" -eq 0 ] && RC4=$r; done + if [ "$RC4" -ne 0 ]; then W4="$W4 (rc $RC4)" + else + a4=$EVAL_OUT/$name-a4j1/analysis.json + if cmp -s "$a4" "$EVAL_OUT/$name-a4jN/analysis.json" && cmp -s "$a4" "$EVAL_OUT/$name-a4jN2/analysis.json"; then + ident4=yes + else + ident4=NO + fi + fi + fi + case $R3$R4 in *[!0-9]*|"") ;; *) RATIO=$(awk -v a="$R4" -v b="$R3" 'BEGIN{printf "%.2f", a/b}') ;; esac + L4COLS=" $W4 | $WN4 | $U4 | $R4 | $RATIO | $B3 | $B4 | $(l4_log "$EVAL_OUT/$name-a4j1.err") | $ident4 |" + fi + ident=n/a if [ "$RC2" -eq 124 ] || [ "$RC3" -eq 124 ] || [ "$RCN" -eq 124 ] || [ "$RCN2" -eq 124 ]; then status="TIMEOUT(>${RUN_TIMEOUT}s; remaining runs skipped)" @@ -409,6 +476,18 @@ for T in $TARGETS; do ident=NO; status="FAIL(-j differs)"; FAILS="$FAILS $name" fi fi + # A failed or timed-out -a 4 run, or -a 4 outputs that differ, fail the app too (the -a 3 + # checks above stand). + s4="" + if [ "$RUN_L4" -eq 1 ] && [ "$RC4" -ne 0 ]; then + if [ "$RC4" -eq 124 ]; then s4="TIMEOUT(a4 >${RUN_TIMEOUT}s)"; else s4="FAIL(rc a4=$RC4)"; fi + elif [ "$RUN_L4" -eq 1 ] && [ "$ident4" = NO ]; then + s4="FAIL(a4 -j differs)" + fi + if [ -n "$s4" ]; then + if [ "$status" = ok ]; then status=$s4; else status="$status $s4"; fi + case " $FAILS " in *" $name "*) ;; *) FAILS="$FAILS $name" ;; esac + fi skips=$(grep -c 'L3 skipped' "$EVAL_OUT/$name-a3j1.err" 2>/dev/null || true) terrs=$(grep -c 'WARN\] package ' "$EVAL_OUT/$name-a3j1.err" 2>/dev/null || true) @@ -420,13 +499,14 @@ for T in $TARGETS; do IFS=$'\t' read -r c1 c2 c3 c4 c5 c6 c7 c8 c9 <>"$EVAL_NOTE" + "$ident" "$skips" "$terrs" "$twarn" "$L4COLS" >>"$EVAL_NOTE" if [ "$EVAL_PRUNE" = 1 ] && [ "$ident" = yes ]; then rm -f "$EVAL_OUT/$name-warm/analysis.json" "$EVAL_OUT/$name-a3jN/analysis.json" \ - "$EVAL_OUT/$name-a3jN2/analysis.json" + "$EVAL_OUT/$name-a3jN2/analysis.json" "$EVAL_OUT/$name-a4j1/analysis.json" \ + "$EVAL_OUT/$name-a4jN/analysis.json" "$EVAL_OUT/$name-a4jN2/analysis.json" fi done diff --git a/scripts/eval_l3_test.sh b/scripts/eval_l3_test.sh new file mode 100755 index 0000000..3f9107c --- /dev/null +++ b/scripts/eval_l3_test.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# eval_l3_test.sh - check scripts/eval_l3.sh's exit status against a stub analyzer (no analysis runs). +# Usage: scripts/eval_l3_test.sh (exits non-zero on a failed check; needs /usr/bin/time -l, i.e. macOS) +set -u +S=$(cd "$(dirname "$0")" && pwd)/eval_l3.sh +T=$(mktemp -d) +trap 'rm -rf "$T"' EXIT +mkdir -p "$T/app" +# The stub writes a fixed analysis.json to -o and exits $FAIL_LEVEL_RC when -a is $FAIL_LEVEL. +cat >"$T/cango" <<'STUB' +#!/usr/bin/env bash +[ "${1:-}" = "--version" ] && { echo "stub"; exit 0; } +out="" level="" +while [ $# -gt 0 ]; do + case $1 in -o) out=$2; shift ;; -a) level=$2; shift ;; esac + shift +done +[ -n "$out" ] && echo '{"application":{"symbol_table":{},"call_graph":[]}}' >"$out/analysis.json" +# Record the environment's USER each run sees (eval_l3.sh must not clobber it). +[ -n "$out" ] && printf '%s\n' "${USER-}" >>"${USER_LOG:-/dev/null}" +# NONDET_LEVEL: a run at that level appends a per-run nonce (a nondeterministic analyzer). +[ "$level" = "${NONDET_LEVEL:-}" ] && echo "$$-$RANDOM" >>"$out/analysis.json" +# At -a 4 with -v, print the L4 log lines eval_l3.sh parses. +if [ "$level" = 4 ]; then + echo "[codeanalyzer-go] L4: SSA built 2 project packages in 5ms" >&2 + echo "[codeanalyzer-go] L4: SSA join: 9 functions (1 by wrapper, 0 without a unit), 20 calls; 1 functions, 3 calls unmapped in 1ms" >&2 + echo "[codeanalyzer-go] L4: VTA over 12 functions (3 library targets): 30 edges in 2ms" >&2 + echo "[codeanalyzer-go] L4: dispatch: 7 call sites with targets; 4 call_graph edges and 1 externals added by VTA" >&2 + echo "[codeanalyzer-go] L4: 40 vertices, 50 port edges (6 points-to), 8 summary edges, 11 param_in, 10 param_out, 5 stand-ins in 4ms" >&2 + echo "[codeanalyzer-go WARN] L4 degraded: unmapped_ssa (1 SSA function(s) and 3 call instruction(s) map to no callable or call node): example.com/a" >&2 + echo "[codeanalyzer-go WARN] L4 degraded: range_func (range-over-func is not modelled at L4 (D8)): example.com/b" >&2 +fi +[ "$level" = "${FAIL_LEVEL:-}" ] && exit 3 +exit 0 +STUB +chmod +x "$T/cango" +fail=0 +check() { # want-rc want-note-substring env... + local want=$1 msg=$2; shift 2 + local rc + env CANGO="$T/cango" EVAL_OUT="$T/out" EVAL_NOTE="$T/note.md" EVAL_JOBS=2 "$@" "$S" "$T/app" >/dev/null 2>"$T/err"; rc=$? + if [ "$rc" != "$want" ] || ! grep -q -- "$msg" "$T/note.md"; then + echo "FAIL: $* eval_l3.sh -> rc $rc (want $want, note containing '$msg')"; sed 's/^/ /' "$T/err" "$T/note.md"; fail=1 + else + echo "ok: $* eval_l3.sh -> rc $rc" + fi +} +check 0 "| app | ok |" CANGO_LEVELS="2 3 4" +# The -a 4 triple: -j 1, -j N and a rerun, compared byte for byte; the L4 log counts and codes. +check 0 "| 40 | 11 | 10 | 8 | 50 | 6 | 5 | 7 | 1/3 | range_func,unmapped_ssa | yes |" CANGO_LEVELS="2 3 4" +check 1 "FAIL(a4 -j differs)" CANGO_LEVELS="2 3 4" NONDET_LEVEL=4 +check 1 "FAIL(-j differs)" CANGO_LEVELS="2 3" NONDET_LEVEL=3 +check 1 "FAIL(rc a4=3)" CANGO_LEVELS="2 3 4" FAIL_LEVEL=4 +check 1 "FAIL(rc a2=0 j1=3" CANGO_LEVELS="2 3" FAIL_LEVEL=3 +# $USER reaches every analyzer run unchanged (a script variable named USER would shadow it). +: >"$T/users" +env CANGO="$T/cango" EVAL_OUT="$T/out" EVAL_NOTE="$T/note.md" EVAL_JOBS=2 CANGO_LEVELS="2 3 4" \ + USER=evaluser USER_LOG="$T/users" "$S" "$T/app" >/dev/null 2>&1 +if [ "$(sort -u "$T/users")" = evaluser ] && [ "$(wc -l <"$T/users" | tr -d ' ')" -ge 8 ]; then + echo "ok: USER unchanged across all runs" +else + echo "FAIL: USER seen by the runs: $(sort -u "$T/users" | paste -sd, -)"; fail=1 +fi +exit $fail diff --git a/scripts/l3_metrics.py b/scripts/l3_metrics.py index 4394fe9..b5f3db5 100644 --- a/scripts/l3_metrics.py +++ b/scripts/l3_metrics.py @@ -8,7 +8,10 @@ switch / select / type-switch fan-out, defer-chain and exception-edge shape, the per-callable DDG distribution and its top callables, `var` classes (wildcard, `:` pseudo path), DDG volume by generated-file class and by callable name (e.g. protobuf `Marshal*` / `Unmarshal`), approximate bytes per section, and the internal-consistency -checks. The figures in docs/design/notes/l3-dataflow-metrics.md come from this script. It loads +checks. On an `-a 4` file it adds an `l4` object: vertices by kind and by `of` class, param_in / +param_out, summary edges (and how many are all-to-all at their call site), port / points-to ddg, +stand-ins and their formals, the share of call nodes with a VTA call_graph target, the degraded +codes, and the L4 consistency checks (no dangling endpoint, no global → global summary). The figures in docs/design/notes/l3-dataflow-metrics.md come from this script. It loads each file whole: a 1.1 GB analysis.json needs about 11 GB of RAM. """ import json @@ -20,6 +23,7 @@ CFG_KINDS = ["fallthrough", "true", "false", "switch_case", "loop_back", "exception", "return", "break", "continue", "defer_resume"] NODE_KINDS = ["call", "statement", "branch", "loop", "switch", "return", "entry", "exit"] +VERTEX_KINDS = ("formal_in", "formal_out", "actual_in", "actual_out") # L4 IF_ERR = re.compile(rb"^if\s+(?:[^{]*;\s*)?err\s*!=\s*nil\s*\{") @@ -102,8 +106,11 @@ def src(f): K["no_cfg"][c.get("kind") + (" with body" if c.get("body") else " no body")] += 1 continue C["with_cfg"] += 1 - body = c.get("body") or {} - cfg, cdg, ddg = c.get("cfg") or [], c.get("cdg") or [], c.get("ddg") or [] + # The L3 figures leave out the L4 overlay (port vertices, reaching-defs / points-to + # ddg), so they compare across -a 3 and -a 4; analyze_l4 counts the overlay. + body = {k: n for k, n in (c.get("body") or {}).items() if n.get("kind") not in VERTEX_KINDS} + cfg, cdg = c.get("cfg") or [], c.get("cdg") or [] + ddg = [e for e in c.get("ddg") or [] if e.get("prov") == ["ssa"]] if len(body) == 2: C["entry_exit_only_bodies"] += 1 @@ -234,19 +241,22 @@ def closure_of(start, adj): top.sort(key=lambda x: -x[0]) del top[5:] + l4 = analyze_l4(doc, st) if doc.get("max_level", 0) >= 4 else None ddg_per.sort() top.sort(key=lambda x: -x[0]) top_detail = [] for n, _, c in top[:5]: - vc = Counter(e["var"] for e in c["ddg"]) + l3ddg = [e for e in c["ddg"] if e.get("prov") == ["ssa"]] + vc = Counter(e["var"] for e in l3ddg) defs, uses = defaultdict(set), defaultdict(set) - for e in c["ddg"]: + for e in l3ddg: defs[e["var"]].add(e["src"]) uses[e["var"]].add(e["dst"]) top_detail.append({ - "signature": c.get("signature"), "ddg": n, "body": len(c["body"]), "cfg": len(c["cfg"]), + "signature": c.get("signature"), "ddg": n, "cfg": len(c["cfg"]), + "body": sum(1 for b in c["body"].values() if b.get("kind") not in VERTEX_KINDS), "cyclomatic": (c.get("metrics") or {}).get("cyclomatic"), - "node_kinds": dict(Counter(b.get("kind") for b in c["body"].values())), + "node_kinds": dict(Counter(b.get("kind") for b in c["body"].values() if b.get("kind") not in VERTEX_KINDS)), "distinct_vars": len(vc), "top_vars": [[v, k, len(defs[v]), len(uses[v])] for v, k in vc.most_common(4)]}) k1 = max(1, len(ddg_per) // 100) @@ -268,6 +278,102 @@ def closure_of(start, adj): "top1pct_share": round(sum(ddg_per[-k1:]) / max(1, C["ddg"]), 4)}, "ddg_top": top_detail, "check_failures": {k: v for k, v in chk.items() if v}, + **({"l4": l4} if l4 is not None else {}), + } + + + +def of_class(of): + for p in (":", ":", ""): + if of.startswith(p): + return p.rstrip(":") + return "_recv" if of == "_recv" else "param" + + +def analyze_l4(doc, st): + """The L4 overlay of an -a 4 analysis.json. + + Dispatch per call site comes from param_in: a site's targets are the owners of the formals + its actual_ins bind (`@formal_in:`); a target other than the call node's static + `callee` was added by VTA (an interface call's concrete methods, a func-value call's + functions). A site whose callee is not null but binds nothing has no vertices (a pruned call + node or an external with no formals).""" + a = doc["application"] + C, K, chk = Counter(), defaultdict(Counter), Counter() + ids, calls = set(), {} + for c, closure, mpath, sfile in callables(st): + body = c.get("body") or {} + sites = defaultdict(lambda: [0, 0]) # call key -> [#actual_in, #actual_out] + for k, n in body.items(): + ids.add(n.get("id")) + kd = n.get("kind") + if kd in VERTEX_KINDS: + K["vertices_by_kind"][kd] += 1 + K["vertex_of_class"][kd + " " + of_class(n.get("of", ""))] += 1 + if kd.startswith("actual"): + sites[n.get("parent")][kd == "actual_out"] += 1 + elif kd == "call": + calls[n["id"]] = n.get("callee") + per_site = Counter() + for e in c.get("summary") or []: + for x in (e["src"], e["dst"]): + chk["summary_endpoint_not_vertex"] += body.get(x, {}).get("kind") not in VERTEX_KINDS + if (body.get(e["src"], {}).get("of", "").startswith(":") + and body.get(e["dst"], {}).get("of", "").startswith(":")): + chk["summary_global_to_global"] += 1 + per_site[body.get(e["src"], {}).get("parent")] += 1 + C["summary"] += 1 + for site, (ni, no) in sites.items(): + if per_site[site]: + C["sites_with_summary"] += 1 + C["sites_with_summary_all_to_all"] += per_site[site] == ni * no + for e in c.get("ddg") or []: + prov = e.get("prov") or [] + if prov == ["reaching-defs"]: + C["port_ddg"] += 1 + elif prov == ["points-to"]: + C["points_to_ddg"] += 1 + C["vertices"] = sum(K["vertices_by_kind"].values()) + C["call_nodes"] = len(calls) + C["call_nodes_with_callee"] = sum(1 for v in calls.values() if v) + ext = a.get("external_symbols") or {} + C["external_symbols"] = len(ext) + for e in ext.values(): + b = e.get("body") or {} + C["stand_ins"] += bool(b) + C["stand_in_formals"] += len(b) + for n in b.values(): + ids.add(n.get("id")) + for e in a.get("call_graph") or []: + prov = e.get("prov") or [] + C["call_graph_vta"] += "vta" in prov + C["call_graph_vta_only"] += prov == ["vta"] + targets = defaultdict(set) + for lst in ("param_in", "param_out"): + es = a.get(lst) or [] + C[lst] = len(es) + for e in es: + chk[lst + "_endpoint_dangling"] += e["src"] not in ids or e["dst"] not in ids + if lst == "param_in" and "/actual_in:" in e["src"]: + targets[e["src"].rsplit("/actual_in:", 1)[0]].add(e["dst"].rsplit("@formal_in:", 1)[0]) + for site, ts in targets.items(): + if site not in calls: + C["capture_sites"] += 1 # a non-call capture site (a literal's position) + continue + C["call_sites_bound"] += 1 + extra = ts - {calls[site]} + C["call_sites_with_vta_target"] += bool(extra) + C["call_sites_multi_target"] += len(ts) > 1 + K["targets_per_bound_site"][min(len(ts), 10)] += 1 + deg = doc.get("degraded") or [] + return { + **dict(C), **{k: dict(v) for k, v in K.items()}, + "pct_call_nodes_with_vta_target": round(100 * C["call_sites_with_vta_target"] / max(1, C["call_nodes"]), 2), + "pct_bound_sites_with_vta_target": round(100 * C["call_sites_with_vta_target"] / max(1, C["call_sites_bound"]), 2), + "degraded": {d["code"]: {"packages": len(d.get("packages") or []), + "callables": len(d.get("callables") or []), + "detail": d.get("detail", "")} for d in deg}, + "check_failures": {k: v for k, v in chk.items() if v}, } diff --git a/scripts/metamorphic.sh b/scripts/metamorphic.sh new file mode 100755 index 0000000..ff4962d --- /dev/null +++ b/scripts/metamorphic.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash +# metamorphic.sh - run the L3/L4 metamorphic relations on prepared eval apps. +# +# Usage: +# scripts/metamorphic.sh the default set: memos (~2 min); run it for any L3/L4 change +# scripts/metamorphic.sh --all every app below: memos nats-server (~25 min; before a release) +# scripts/metamorphic.sh APP [APP...] the named apps, e.g. scripts/metamorphic.sh nats-server +# +# Each APP is a clone under $EVAL_SRC, prepared with scripts/eval_l3.sh --prepare APP. A missing +# app is skipped by the test (and reported). The relations transform a copy of the app in a temp +# dir and analyze it in-process; the app itself is never modified. Each relation samples +# MR_SAMPLE_FILES files and MR_SAMPLE_CALLABLES callables with seed 1 +# (internal/metamorphic/realapp_test.go). +# +# Environment: +# EVAL_SRC clones (default /tmp/eval) +# EVAL_GO Go toolchain for the test run (default go1.27.3; eval_l3.sh's default) +# MR_RELATIONS relations to run, space-separated (default: all of MR1-MR6, MR8-MR15) +# MR_SAMPLE_FILES files transformed per relation (default 20) +# MR_SAMPLE_CALLABLES callables sampled by the reachability oracle (default 50; 20 gives a +# quicker nats-server run) +# MR_TIMEOUT go test -timeout (default 120m) +# MR_LOG log file (default $EVAL_OUT/metamorphic-.log) +# +# Prints one MRSTAT line per relation and app (sites, reachability pairs, violations, time, +# skips) and exits non-zero on any violation. +set -u + +EVAL_SRC=${EVAL_SRC:-/tmp/eval} +EVAL_OUT=${EVAL_OUT:-/tmp/eval-out} +EVAL_GO=${EVAL_GO:-go1.27.3} +MR_TIMEOUT=${MR_TIMEOUT:-120m} +MR_LOG=${MR_LOG:-$EVAL_OUT/metamorphic-$(date +%Y-%m-%d).log} +REPO_ROOT=$(cd "$(dirname "$0")/.." && pwd) + +DEFAULT_APPS="memos" +ALL_APPS="memos nats-server" +for a in "$@"; do + case $a in + -h|--help) sed -n '2,12p' "$0" | sed 's/^# \{0,1\}//' >&2; exit 0 ;; + --all) [ $# -eq 1 ] || { echo "--all takes no other arguments" >&2; exit 2; } ;; + -*) echo "unknown flag: $a" >&2; exit 2 ;; + esac +done +case "${1:-}" in + --all) set -- $ALL_APPS ;; + "") set -- $DEFAULT_APPS ;; +esac +for a in "$@"; do + [ -f "$EVAL_SRC/$a/go.mod" ] || echo "warning: $a is not prepared at $EVAL_SRC/$a (skipped)" >&2 +done +mkdir -p "$(dirname "$MR_LOG")" + +start=$(date +%s) +(cd "$REPO_ROOT" && EVAL_SRC=$EVAL_SRC MR_APPS="$*" MR_RELATIONS="${MR_RELATIONS:-}" \ + MR_SAMPLE_FILES="${MR_SAMPLE_FILES:-20}" MR_SAMPLE_CALLABLES="${MR_SAMPLE_CALLABLES:-50}" \ + GOTOOLCHAIN=${GOTOOLCHAIN:-$EVAL_GO} \ + go test -tags metamorphic -count=1 -timeout "$MR_TIMEOUT" -v -run 'TestMR_RealApps' ./internal/metamorphic/) \ + >"$MR_LOG" 2>&1 +rc=$? +end=$(date +%s) + +grep -E 'MRLOAD|MRSTAT' "$MR_LOG" | sed 's/^ *[a-z_]*\.go:[0-9]*: //' +grep -E '^\s+(realapp|mr)_test\.go:[0-9]+: MR[0-9]+ on ' "$MR_LOG" | sed 's/^ *//' | head -200 +echo "log: $MR_LOG ($((end - start)) s, go test rc $rc)" +exit $rc diff --git a/scripts/metamorphic_args_test.sh b/scripts/metamorphic_args_test.sh new file mode 100755 index 0000000..f25885e --- /dev/null +++ b/scripts/metamorphic_args_test.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash +# metamorphic_args_test.sh - check scripts/metamorphic.sh's argument handling (no analysis runs). +# Usage: scripts/metamorphic_args_test.sh (exits non-zero on a failed check) +set -u +S=$(cd "$(dirname "$0")" && pwd)/metamorphic.sh +fail=0 +check() { # want-rc want-stderr-substring args... + local want=$1 msg=$2; shift 2 + local err rc + err=$("$S" "$@" 2>&1 >/dev/null); rc=$? + if [ "$rc" != "$want" ] || [[ "$err" != *"$msg"* ]]; then + echo "FAIL: metamorphic.sh $* -> rc $rc, stderr: $err (want rc $want, '$msg')"; fail=1 + else + echo "ok: metamorphic.sh $* -> rc $rc" + fi +} +check 2 "unknown flag: --al" --al +check 2 "unknown flag: -x" -x memos +check 2 "unknown flag: --memos" memos --memos +check 2 "--all takes no other arguments" --all memos +check 2 "--all takes no other arguments" memos --all +check 0 "Usage:" --help +exit $fail diff --git a/testdata/cgo/c/c.go b/testdata/cgo/c/c.go new file mode 100644 index 0000000..91956f7 --- /dev/null +++ b/testdata/cgo/c/c.go @@ -0,0 +1,10 @@ +// Package c has a cgo file. go/packages parses the cgo-generated copy of it +// (in the build cache) instead, so its functions have no symbol-table +// callable (#29) and their SSA functions are unmapped_ssa. +package c + +// int twice(int x) { return 2 * x; } +import "C" + +// Twice calls C. +func Twice(x int) int { return int(C.twice(C.int(x))) } diff --git a/testdata/cgo/c/d.go b/testdata/cgo/c/d.go new file mode 100644 index 0000000..2e90fca --- /dev/null +++ b/testdata/cgo/c/d.go @@ -0,0 +1,4 @@ +package c + +// Plain is a plain Go function in the cgo package. +func Plain() int { return 1 } diff --git a/testdata/cgo/go.mod b/testdata/cgo/go.mod new file mode 100644 index 0000000..585ff94 --- /dev/null +++ b/testdata/cgo/go.mod @@ -0,0 +1,3 @@ +module example.com/cgo + +go 1.21 diff --git a/testdata/dataflow/asm/asm.go b/testdata/dataflow/asm/asm.go new file mode 100644 index 0000000..4906011 --- /dev/null +++ b/testdata/dataflow/asm/asm.go @@ -0,0 +1,10 @@ +// Package asm declares a function with no Go body (implemented in assembly, +// asm.s). The symbol table records it as a callable, but it has no L3 unit: +// the SSA join maps it through bodilessDecl, not fnUnit. +package asm + +// Add is implemented in assembly. +func Add(a, b int) int + +// Use calls Add. +func Use() int { return Add(1, 2) } diff --git a/testdata/dataflow/asm/asm.s b/testdata/dataflow/asm/asm.s new file mode 100644 index 0000000..d17a030 --- /dev/null +++ b/testdata/dataflow/asm/asm.s @@ -0,0 +1,2 @@ +// Add is declared in asm.go; this file only lets the package build with a +// bodiless declaration. diff --git a/testdata/dataflow/flow/flow.go b/testdata/dataflow/flow/flow.go index 5c392f8..2795b3c 100644 --- a/testdata/dataflow/flow/flow.go +++ b/testdata/dataflow/flow/flow.go @@ -549,11 +549,170 @@ func observe(err error) { _ = err } // DeferWrap has a deferred closure that rewrites the named result err (R11). // The chain runs the rewrite, then the observer, then @exit. Deferred nodes -// never strong-define, and L3 records no def for the closure's write to its -// captured err, so the earlier def of err still reaches the observer. +// never strong-define; the rewrite weak-defines its captured err (D1), so the +// earlier def of err and the rewrite both reach the observer. func DeferWrap() (err error) { err = wrapErr(nil) defer func() { observe(err) }() defer func() { err = wrapErr(err) }() return } + +// CaptureWrite calls a literal that writes the captured x (D1 gap 1). +func CaptureWrite() int { + x := 0 + func() { x = 1 }() + return x +} + +// CaptureThenReassign stores a literal that writes x, rebinds x, then calls +// the literal: x = 2 must not kill the closure's write. +func CaptureThenReassign() int { + x := 0 + h := func() { x = 1 } + x = 2 + h() + return x +} + +type onceT struct{} + +func (o *onceT) Do(f func()) { f() } + +func load() int { return 1 } + +// OnceDo writes cfg through a literal passed to a method. +func OnceDo() int { + var once onceT + var cfg int + once.Do(func() { cfg = load() }) + return cfg +} + +// DeferReassign writes an element of b after registering fill(b), then +// rebinds b (D1 gap 3): the element write still reaches the deferred fill. +func DeferReassign(b, o []byte, s byte) { + defer fill(b) + b[0] = s + b = o +} + +// AppendKeepsElems writes an element, then appends in place (#28): append may +// keep the backing array, so the s[0] def survives s = append(s, n). +func AppendKeepsElems(s []int, n int) int { + s[0] = 5 + s = append(s, n) + return s[0] +} + +// CaptureNested writes x from a literal nested in the stored one, and buf's +// elements through a call. y is the outer literal's own variable and z is +// only read: neither is a capture write. +func CaptureNested(buf []byte) int { + x, z := 0, 0 + f := func() { + y := 0 + func() { x = y; y = 1 }() + fill(buf) + _ = z + } + f() + return x +} + +// CaptureGo launches a literal that writes the captured x: the go node is +// its host. +func CaptureGo(ch chan int) int { + x := 0 + go func() { x = <-ch }() + return x +} + +type cfg struct{ items []int } + +func fillCfg(c cfg) { c.items[0] = 1 } + +func (c cfg) reset() { c.items[0] = 0 } + +type plain struct{ a, b int } + +func usePlain(p plain) { _ = p } + +type nest struct { + in cfg + arr [2][]int +} + +func useNest(n nest, a [2][]int, b [2]int) { _, _, _ = n, a, b } + +// FillCfg passes a value struct holding a slice (D1 gap 2): the callee may +// write c.items[*], so the call weak-defines c.*, which reaches the read. +func FillCfg(c cfg) int { + fillCfg(c) + return c.items[0] +} + +// ValueRefs passes value operands that may hold a reference (a value +// receiver over a slice field, a struct nesting one, an array of slices) and +// two that cannot (a struct and an array of ints). +func ValueRefs(c cfg, p plain, n nest, a [2][]int, b [2]int) int { + c.reset() + usePlain(p) + useNest(n, a, b) + return c.items[0] + p.a + n.in.items[0] + a[0][0] + b[0] +} + +func chanOf(ch chan int) chan int { return ch } + +func ids(xs []int) []int { return xs } + +// SatParents puts calls in a folded for init and post, a case list, select +// comm clauses and a range key and expression (#28): each call is a +// satellite of the header or clause head its expression folds into. +func SatParents(n int, ch chan int, xs []int) int { + s := 0 + for i := f(0); i < g(n); i = f(i) + 1 { + s += i + } + switch n { + case f(1), g(2): + s++ + } + select { + case ch <- f(s): + case v := <-chanOf(ch): + s += v + } + var x int + for xs[f(2)], x = range ids(xs) { + s += x + } + return s +} + +func (p plain) sum() int { return p.a + p.b } + +// PlainRecv calls a value-receiver method on a struct that holds no +// reference: a copy, so nothing is written through it. +func PlainRecv(p plain) int { + p.sum() + return p.a +} + +// Bump is a method declared above its receiver type (L1 used to drop it). +func (e *early) Bump() int { return f(e.n) } + +type early struct{ n int } + +// aliasT has a scalar field n. +type aliasT struct{ n int } + +// CaptureAliasWrite writes through a local alias of the captured pointer p +// inside a literal called in place: the literal writes no path of p, but p +// may hold a reference, so its host weak-defines p.* and the write reaches +// the later read of p.n (found by metamorphic relation MR5). +func CaptureAliasWrite() int { + p := &aliasT{} + func() { q := p; q.n = 1 }() + return p.n +} diff --git a/testdata/dataflow/generic/b.go b/testdata/dataflow/generic/b.go index 713f17b..6bbafac 100644 --- a/testdata/dataflow/generic/b.go +++ b/testdata/dataflow/generic/b.go @@ -7,3 +7,16 @@ type B struct{ s string } var pb atomic.Pointer[B] func StoreB() { pb.Store(nil) } + +// Ident is generic; Twice passes it its own type parameter, so SSA builds +// Twice's call through an instantiation wrapper of Ident (the type +// arguments are parameterized), which joins Ident via Origin(). +func Ident[T any](v T) T { return v } + +func Twice[T any](v T) T { + f := Ident[T] + return f(v) +} + +// UseTwice instantiates Twice. +func UseTwice() int { return Twice(1) } diff --git a/testdata/dataflow/httpsql/httpsql.go b/testdata/dataflow/httpsql/httpsql.go new file mode 100644 index 0000000..5f4055e --- /dev/null +++ b/testdata/dataflow/httpsql/httpsql.go @@ -0,0 +1,95 @@ +// Package httpsql is the L4 "done" fixture (spec §8): an HTTP handler's +// request reaches database/sql's DB.Query through an interface call, and a +// second handler takes the same route through strconv.Atoi first. +package httpsql + +import ( + "database/sql" + "net/http" + "strconv" +) + +// Repo is the storage interface the handlers call; sqlRepo is its one +// implementation, so the interface calls bind sqlRepo's methods by VTA. +type Repo interface { + Find(id string) (*sql.Rows, error) + SanitizedFind(id string) (*sql.Rows, error) + Count(user string) (*sql.Rows, error) +} + +type sqlRepo struct{ db *sql.DB } + +// NewRepo returns the database-backed Repo. +func NewRepo(db *sql.DB) Repo { return &sqlRepo{db: db} } + +// Find builds the query from id: the sink. +func (s *sqlRepo) Find(id string) (*sql.Rows, error) { + return s.db.Query("SELECT name FROM users WHERE id = " + id) +} + +// SanitizedFind parses id as a number first and queries with the number. +func (s *sqlRepo) SanitizedFind(id string) (*sql.Rows, error) { + n, err := strconv.Atoi(id) + if err != nil { + return nil, err + } + return s.db.Query("SELECT name FROM users WHERE id = " + strconv.Itoa(n)) +} + +// Count counts a user's rows; only Audit calls it. +func (s *sqlRepo) Count(user string) (*sql.Rows, error) { + return s.db.Query("SELECT count(*) FROM events WHERE user = " + user) +} + +// Handler reads the id query parameter and looks it up. +func Handler(repo Repo) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + id := r.URL.Query().Get("id") + rows, err := repo.Find(id) + if err != nil { + http.Error(w, "lookup failed", http.StatusInternalServerError) + return + } + rows.Close() + } +} + +// SanitizedHandler is Handler through SanitizedFind. +func SanitizedHandler(repo Repo) http.HandlerFunc { + return func(w http.ResponseWriter, r *http.Request) { + id := r.URL.Query().Get("id") + rows, err := repo.SanitizedFind(id) + if err != nil { + http.Error(w, "lookup failed", http.StatusInternalServerError) + return + } + rows.Close() + } +} + +// Health answers without touching the database. +func Health(w http.ResponseWriter, r *http.Request) { + w.WriteHeader(http.StatusOK) +} + +// Audit is an entry point apart from Routes: it builds its own Repo inline +// (no call), so it shares no host variable and no in-project callee with the +// handlers; its only in-project callee is sqlRepo.Count. +func Audit(db *sql.DB, user string) error { + var repo Repo = &sqlRepo{db: db} + rows, err := repo.Count(user) + if err != nil { + return err + } + return rows.Close() +} + +// Routes wires the handlers to a database-backed Repo: without it no +// concrete type reaches the handlers' repo, and VTA binds repo.Find to +// nothing but the interface method. +func Routes(mux *http.ServeMux, db *sql.DB) { + repo := NewRepo(db) + mux.Handle("/user", Handler(repo)) + mux.Handle("/user/sanitized", SanitizedHandler(repo)) + mux.HandleFunc("/health", Health) +} diff --git a/testdata/dataflow/initclosure/a.go b/testdata/dataflow/initclosure/a.go new file mode 100644 index 0000000..377e27e --- /dev/null +++ b/testdata/dataflow/initclosure/a.go @@ -0,0 +1,19 @@ +// Package initclosure has a func init() with a closure in more than one file. +// The closures share the signature example.com/dataflow/initclosure.init.closure_1, +// so a call edge to one must carry its declaring file. +package initclosure + +var hooks []func() int + +func init() { + hooks = append(hooks, func() int { return 1 }) +} + +// Run calls every hook through a function value. +func Run() int { + n := 0 + for _, h := range hooks { + n += h() + } + return n +} diff --git a/testdata/dataflow/initclosure/b.go b/testdata/dataflow/initclosure/b.go new file mode 100644 index 0000000..be0e13b --- /dev/null +++ b/testdata/dataflow/initclosure/b.go @@ -0,0 +1,5 @@ +package initclosure + +func init() { + hooks = append(hooks, func() int { return 2 }) +} diff --git a/testdata/dataflow/libflow/libflow.go b/testdata/dataflow/libflow/libflow.go new file mode 100644 index 0000000..0485b55 --- /dev/null +++ b/testdata/dataflow/libflow/libflow.go @@ -0,0 +1,220 @@ +// Package libflow is the L4 fixture for interprocedural library origin +// (spec §4.8, CHA complement): a Store that came out of library code +// (context.Context, the vendored box) reaches an interface call through +// project code — a project function's result, a parameter, a struct field, +// a global, a closure capture, a map, slice or channel element. VTA sees no +// concrete type in library code, so each call's targets include every +// project type implementing Store. Tests pin keys by line:col, so keep each +// statement on its own line and append new functions at the end. +package libflow + +import ( + "context" + + "example.com/vlib/box" +) + +// Store is the interface the handlers call. +type Store interface { + Find(id string) string +} + +// Last is the global sqlStore.Find writes. +var Last string + +type sqlStore struct{} + +// Find builds the query from id and runs it: the sink is Exec's query. +func (s *sqlStore) Find(id string) string { + Last = id + return Exec("select " + id) +} + +// Exec is the sink. +func Exec(query string) string { + return query +} + +type key struct{} + +// fromCtx returns the Store in ctx: its result is library-originated. +func fromCtx(ctx context.Context) Store { + return ctx.Value(key{}).(Store) +} + +// ViaReturn calls Find on a project function's library-originated result. +func ViaReturn(ctx context.Context, id string) string { + return fromCtx(ctx).Find(id) +} + +// use calls Find on its parameter: one caller passes a library-originated +// Store, another a project-built one. +func use(s Store, id string) string { + return s.Find(id) +} + +// ViaParam passes a library-originated Store to use. +func ViaParam(ctx context.Context, id string) string { + return use(ctx.Value(key{}).(Store), id) +} + +type memStore struct{} + +// Find returns id. +func (memStore) Find(id string) string { + return id +} + +// UseLocal passes a project-built Store to use. +func UseLocal(id string) string { + return use(memStore{}, id) +} + +type handler struct { + store Store +} + +// init stores a library-originated Store in a field. +func (h *handler) init(ctx context.Context) { + h.store = ctx.Value(key{}).(Store) +} + +// ViaField calls Find on the field another method set. +func (h *handler) ViaField(id string) string { + return h.store.Find(id) +} + +var global Store + +// SetGlobal stores a library-originated Store in a global. +func SetGlobal(ctx context.Context) { + global = ctx.Value(key{}).(Store) +} + +// ViaGlobal calls Find on the global. +func ViaGlobal(id string) string { + return global.Find(id) +} + +// ViaCapture calls Find inside a closure on a captured library-originated +// Store. +func ViaCapture(ctx context.Context, id string) string { + s := ctx.Value(key{}).(Store) + f := func() string { + return s.Find(id) + } + return f() +} + +// ViaMap looks up a library map (comma-ok free): the element is +// library-originated. +func ViaMap(id string) string { + return box.Map()["db"].Find(id) +} + +// ViaSlice indexes a library slice. +func ViaSlice(id string) string { + return box.List()[0].Find(id) +} + +// ViaChan receives from a library channel. +func ViaChan(id string) string { + return (<-box.Chan("db")).Find(id) +} + +// ViaArray indexes a library array value. +func ViaArray(id string) string { + return box.First("db")[0].Find(id) +} + +// ProjectOnly calls Find on a Store only project code ever built: no CHA +// target. +func ProjectOnly(id string) string { + var s Store = memStore{} + return relay(s).Find(id) +} + +// relay returns its parameter. +func relay(s Store) Store { + return s +} + +// The entries: each puts a sqlStore in library storage and calls one +// handler, so no host variable is shared between the handlers' sites. +func MainReturn() { + ViaReturn(context.WithValue(context.Background(), key{}, &sqlStore{}), "a") +} + +func MainParam() { + ViaParam(context.WithValue(context.Background(), key{}, &sqlStore{}), "b") +} + +func MainField() { + h := &handler{} + h.init(context.WithValue(context.Background(), key{}, &sqlStore{})) + h.ViaField("c") +} + +func MainGlobal() { + SetGlobal(context.WithValue(context.Background(), key{}, &sqlStore{})) + ViaGlobal("d") +} + +func MainCapture() { + ViaCapture(context.WithValue(context.Background(), key{}, &sqlStore{}), "e") +} + +func MainBox() { + box.Put("db", &sqlStore{}) + ViaMap("f") + ViaSlice("g") + ViaChan("h") + ViaArray("i") +} + +// Getter is a project interface whose one implementer's method is promoted +// library code (box.Holder.Get): a call of it returns a library-originated +// value whatever the visiting order of the origin fixpoint. +type Getter interface { + Get() box.Finder +} + +type holder struct { + box.Holder +} + +// zzzGet calls Get through the interface; its name sorts after +// ViaPromoted, so the fixpoint visits ViaPromoted first. +func zzzGet(g Getter) box.Finder { + return g.Get() +} + +// ViaPromoted calls Find on what a promoted library method returned, +// through a project function. +func ViaPromoted(g Getter, id string) string { + return zzzGet(g).Find(id) +} + +func MainPromoted() { + ViaPromoted(holder{box.Holder{F: &sqlStore{}}}, "j") +} + +// zzzViaPromoted is ViaPromoted visited after zzzGet: with an order- +// dependent fixpoint the two sites would differ. +func zzzViaPromoted(g Getter, id string) string { + return zzzGet(g).Find(id) +} + +func MainPromoted2() { + zzzViaPromoted(holder{box.Holder{F: &sqlStore{}}}, "k") +} + +// aaaPeek calls Get through the interface too; its name sorts first, so +// the fixpoint meets the Getter.Get call here before zzzGet. +func aaaPeek(g Getter) { + g.Get() +} + +func MainPeek() { + aaaPeek(holder{}) +} diff --git a/testdata/dataflow/libstore/libstore.go b/testdata/dataflow/libstore/libstore.go new file mode 100644 index 0000000..2580904 --- /dev/null +++ b/testdata/dataflow/libstore/libstore.go @@ -0,0 +1,96 @@ +// Package libstore is the L4 fixture for dynamic types that pass through +// library storage (spec §4.8, CHA complement): a Store put in a +// context.Context or a sync.Map comes back out of library code, which has +// no SSA bodies, so VTA sees no concrete type at the interface call. The +// call's targets then include every project type implementing Store. An +// interface call on a value project code built (HandleLocal) keeps VTA's +// targets only. Tests pin keys by line:col, so keep each statement on its +// own line and append new functions at the end. +package libstore + +import ( + "context" + "sync" +) + +// Store is the interface the handlers call. +type Store interface { + Find(id string) string +} + +// lastQuery is the global sqlStore.Find writes. +var lastQuery string + +type sqlStore struct{} + +// Find builds the query from id and runs it: the sink is Exec's query. +func (s *sqlStore) Find(id string) string { + lastQuery = id + return Exec("select " + id) +} + +// Exec is the sink. +func Exec(query string) string { + return query +} + +type storeKey struct{} + +// WithStore puts s in ctx. +func WithStore(ctx context.Context, s Store) context.Context { + return context.WithValue(ctx, storeKey{}, s) +} + +// HandleCtx takes the Store back out of ctx and calls Find. +func HandleCtx(ctx context.Context, id string) string { + return ctx.Value(storeKey{}).(Store).Find(id) +} + +var stores sync.Map + +// Register puts s in the stores map. +func Register(name string, s Store) { + stores.Store(name, s) +} + +// HandleMap loads the Store from the stores map and calls Find. +func HandleMap(name, id string) string { + v, _ := stores.Load(name) + return v.(Store).Find(id) +} + +type memStore struct{} + +// Find returns id. +func (memStore) Find(id string) string { + return id +} + +// HandleLocal calls Find on a Store project code built: VTA sees memStore. +func HandleLocal(id string) string { + var s Store = memStore{} + return s.Find(id) +} + +// MainCtx and MainMap each store a sqlStore in library storage and call one +// handler: no host variable is shared between the handlers' call sites. +func MainCtx() { + ctx := WithStore(context.Background(), &sqlStore{}) + HandleCtx(ctx, "a") +} + +func MainMap() { + Register("db", &sqlStore{}) + HandleMap("db", "b") +} + +// errStore embeds the universe error: its Error method has no package. +type errStore struct { + error +} + +// HandleErr calls Error on an error that came out of a library call: the +// CHA complement meets errStore, whose Error is the universe method. +func HandleErr(ctx context.Context) string { + return context.Cause(ctx).Error() +} diff --git a/testdata/dataflow/linedir/linedir.go b/testdata/dataflow/linedir/linedir.go index aa68c34..bd2d02d 100644 --- a/testdata/dataflow/linedir/linedir.go +++ b/testdata/dataflow/linedir/linedir.go @@ -1,13 +1,13 @@ // Package linedir has calls under a //line directive, the shape of generated -// code (goyacc, templ): L1 records them at their virtual positions, far past -// the file's real line count, and L2 must still resolve them. +// code (goyacc, templ): L1 records them at their real positions in this file +// (the directive is not applied), and L2 must still resolve them. package linedir import "os" func helper() string { return "" } -// Caller's calls sit on virtual lines of virtual.y. +// Caller's calls sit on virtual lines of virtual.y, real lines 13 and 15. func Caller() { //line virtual.y:900:1 os.Getenv(helper()) diff --git a/testdata/dataflow/sdg/globals.go b/testdata/dataflow/sdg/globals.go new file mode 100644 index 0000000..e4905ca --- /dev/null +++ b/testdata/dataflow/sdg/globals.go @@ -0,0 +1,101 @@ +package sdg + +// The globals of the L4 footprint fixture that need the os import; they live +// apart from sdg.go so that its line:col keys stay put. + +import "os" + +// home's initializer calls a function: the global filter keeps it. +var home = os.Getenv("HOME") + +// counter is written only through its address: the global filter keeps it. +var counter int + +// readArgs reads os.Args, a non-project package variable. +func readArgs() []string { + return os.Args +} + +func readHome() string { + return home +} + +func bump() { + p := &counter + *p = *p + 1 +} + +func readCounter() int { + return counter +} + +// g1 and g2 are read by skipX and skipY; tests force skipX to have no L3 +// result. +var g1 = os.Getenv("G1") + +var g2 = os.Getenv("G2") + +func skipX() string { + return g1 + skipY() +} + +func skipY() string { + return g2 +} + +func callsSkipX() string { + return skipX() +} + +// lazyOnce stands for library code that runs a callback later: Do keeps f +// and never calls it, so no call edge reaches Get's literal. +type lazyOnce struct{ f func() } + +func (o *lazyOnce) Do(f func()) { o.f = f } + +var once lazyOnce + +var inst *Inner + +func build() *Inner { return &Inner{} } + +// Get writes inst only inside the literal it hands to once.Do. +func Get() *Inner { + once.Do(func() { inst = build() }) + return inst +} + +func callsGet() *Inner { return Get() } + +// argv is initialized from a library variable, cfgDir from a kept global: +// the chained initializer rule keeps both. +var argv = os.Args + +var cfgDir = home + "/x" + +func readArgv() []string { return argv } + +func readCfgDir() string { return cfgDir } + +// litG and litH are written only by literals called where they are written. +var litG, litH int + +// calledLits writes litG and litH only inside literals it calls in place +// (one deferred): with no VTA, no call edge reaches them. +func calledLits() { + defer func() { litH = 2 }() + func() { litG = 1 }() +} + +// appendConfig reads and writes Config (a weak write: Config flows to +// itself) and takes v: a call of it has the global → global summary +// Config → Config, which is not emitted, beside v → Config, v → +// and Config → , which are. +func appendConfig(v string) string { + Config += v + return Config +} + +func useAppend(v string) string { + return appendConfig(v) +} diff --git a/testdata/dataflow/sdg/sdg.go b/testdata/dataflow/sdg/sdg.go new file mode 100644 index 0000000..9bddab0 --- /dev/null +++ b/testdata/dataflow/sdg/sdg.go @@ -0,0 +1,512 @@ +// Package sdg is the L4 fixture for the dataflow train: one named function per +// interprocedural construct. Tests pin keys by line:col, so keep each +// statement on its own line and append new functions at the end. +package sdg + +import ( + "bytes" + "io" +) + +// Store is an interface with two implementations. +type Store interface { + Get(k string) string +} + +type memStore struct { + m map[string]string +} + +func (s memStore) Get(k string) string { + return s.m[k] +} + +type sqlStore struct { + dsn string +} + +func (s *sqlStore) Get(k string) string { + return s.dsn + k +} + +// UseStore dispatches s.Get through the interface. +func UseStore(s Store, k string) string { + return s.Get(k) +} + +// Stores passes both implementations to UseStore, memStore also by pointer +// (its pointer wrapper maps to memStore.Get). +func Stores() { + UseStore(memStore{}, "a") + UseStore(&memStore{}, "b") + UseStore(&sqlStore{}, "c") +} + +func double(x int) int { + return x * 2 +} + +// CallFuncValue calls f through a function value. +func CallFuncValue(f func(int) int) int { + return f(1) +} + +// PassDouble passes double to CallFuncValue. +func PassDouble() int { + return CallFuncValue(double) +} + +// Inner's method M is promoted to Outer. +type Inner struct { + n int +} + +func (i Inner) M() int { + return i.n +} + +// Outer embeds Inner. +type Outer struct { + Inner +} + +// Mer is satisfied by Outer and *Outer through the promoted M. +type Mer interface { + M() int +} + +// CallPromoted calls M through the interface. +func CallPromoted(o Mer) int { + return o.M() +} + +// PassOuter passes an Outer and an *Outer: the targets are the promotion +// wrapper and the promotion-and-pointer wrapper, both of Inner.M. +func PassOuter() int { + a := CallPromoted(Outer{}) + b := CallPromoted(&Outer{}) + return a + b +} + +// Buffered writes through an io.Writer. +func Buffered(w io.Writer, p []byte) { + w.Write(p) +} + +// PassBuffer passes a *bytes.Buffer: (*bytes.Buffer).Write is a target only +// VTA finds. +func PassBuffer() { + Buffered(&bytes.Buffer{}, nil) +} + +// Config is written only in initConfig and read only in readConfig, which +// recurses through useConfig: useConfig reads it transitively. +var Config string + +// constOnly is initialized from a literal and never written: the global +// filter drops it. +var constOnly = 3 + +func initConfig() { + Config = "on" +} + +func readConfig() string { + if Config == "" { + return useConfig() + } + return Config +} + +func useConfig() string { + return readConfig() +} + +func readConst() int { + return constOnly +} + +// mutate writes through p (a pointer) and s (a value holding a map); n is a +// value copy and q is never written through. +func mutate(p *Inner, s memStore, n Inner, q *Inner) { + p.n = 1 + s.m["k"] = "v" + n.n = 2 + _ = q +} + +// setDSN writes through its pointer receiver. +func (s *sqlStore) setDSN(d string) { + s.dsn = d +} + +// aliasRange writes through s's elements via the loop variable, an alias. +func aliasRange(s []*Inner, v int) { + for _, it := range s { + it.n = v + } +} + +// aliasAssign writes through p via the local alias q. +func aliasAssign(p *Inner, v int) { + q := p + q.n = v +} + +// aliasCall passes p's alias q to a callee. +func aliasCall(p *Inner) { + q := p + takeInner(q) +} + +func takeInner(p *Inner) {} + +// unusedRef never uses its by-ref param p. +func unusedRef(p *Inner, v int) int { + return v +} + +// multiT is a value receiver type that holds no reference. +type multiT struct{ n int } + +// Multi has a receiver, a named and a blank param, and two results. +func (s multiT) Multi(a int, _ string) (int, error) { + return a + s.n, nil +} + +// Named has one named result: its formal_out is , never n. +func Named() (n int) { + n = 1 + return +} + +// Variadic collapses excess actuals onto xs. +func Variadic(xs ...int) int { + return len(xs) +} + +func pair() (int, int) { return 1, 2 } + +func takeTwo(a, b int) int { return a + b } + +// CallVariadic passes three values, then pair's two results, to Variadic, +// and pair's results to takeTwo. +func CallVariadic() int { + a := Variadic(1, 2, 3) + b := Variadic(pair()) + c := takeTwo(pair()) + return a + b + c +} + +// MethodValue calls s.Get through a bound method value. +func MethodValue(s *sqlStore, k string) string { + f := s.Get + return f(k) +} + +// MethodExpr calls Get as a method expression: s binds the receiver. +func MethodExpr(s *sqlStore, k string) string { + return (*sqlStore).Get(s, k) +} + +// CaptureSite stores a literal that reads and writes the captured x. +func CaptureSite(v int) func() int { + x := v + f := func() int { + x++ + return x + } + return f +} + +// DirectLit calls a literal in place: a binds v, and the capture y binds at +// the call. +func DirectLit(v int) int { + y := 0 + func(a int) { + y = a + }(v) + return y +} + +func upper(t string) string { return t } + +// MixedValue calls a function value that is either a plain function or a +// bound method: only the method binds the function value as its receiver. +func MixedValue(s *sqlStore, b bool) string { + f := upper + if b { + f = s.Get + } + return f("k") +} + +// PrunedCall's second return is unreachable: its call node is pruned. +func PrunedCall(v int) int { + return v + return double(v) +} + +// callUnknown calls a function value no caller supplies: unresolved. +func callUnknown(f func(int) int) int { + return f(1) +} + +// ChainLit starts a statement with a called literal whose result is +// written: the call holds the literal's line:col, the statement its /2. +func ChainLit(v int) { + x := 0 + func() *Inner { _ = x; return &Inner{} }().n = v +} + +func apply(f func() int) int { return f() } + +// CaptureArg passes a capturing literal to a call nested in an assignment: +// the assignment, not the satellite call, hosts the capture site. +func CaptureArg(v int) int { + n := apply(func() int { return v }) + return n +} + +// DeadLit stores a capturing literal after a return, inside a kept branch: +// the literal is dead and has no capture site. +func DeadLit(v int) { + if v > 0 { + return + _ = func() int { return v } + } +} + +type sumT struct{} + +func (sumT) Sum(xs ...int) int { return len(xs) } + +// MethodExprVariadic passes only the receiver to a variadic method +// expression: nothing reaches xs. +func MethodExprVariadic(v sumT) int { + return sumT.Sum(v) +} + +// CaptureNamed stores a literal that rewrites the named result err: the +// capture site's actual_out reaches err's formal_out (A7). +func CaptureNamed() (err error) { + h := func() { err = nil } + h() + return +} + +// CaptureParam stores a literal that writes through the by-ref param p: the +// capture site's actual_out reaches p's formal_out (A7). +func CaptureParam(p *Inner) { + f := func() { p.n = 1 } + f() +} + +// a passes x to b, which passes it to c, which returns it: a's call of b +// has a summary edge from its actual_in:0 to its actual_out. +func a(x int) int { + return b(x) +} + +func b(x int) int { + return c(x) +} + +func c(x int) int { + return x +} + +// ignore never uses x: no summary edge leaves the actual_in:0 of a call +// of it. +func ignore(x, y int) int { + return y +} + +func callIgnore(v, w int) int { + return ignore(v, w) +} + +// dropX and dropY are mutually recursive and dropX never uses x: no summary +// edge leaves the actual_in:0 of dropY's call of dropX. +func dropX(x, y int) int { + if y == 0 { + return y + } + return dropY(y - 1) +} + +func dropY(y int) int { + return dropX(0, y) +} + +// implicit's result depends on x only through control: x reaches +// over the cdg. +func implicit(x int) int { + if x > 0 { + return 1 + } + return 0 +} + +// Req is a request whose field a pointer copy writes (Task 17, §4.10). +type Req struct { + UserID string +} + +// AliasPtr writes req.UserID through the pointer copy p: the write reaches +// the return's read of req.UserID only through the may-alias oracle. +func AliasPtr(req *Req, id string) string { + p := req + p.UserID = id + return req.UserID +} + +// AliasPtrWhole is AliasPtr returning the whole pointer: the bare use req +// is prefix-compatible with the aliased write p.UserID. +func AliasPtrWhole(req *Req, id string) *Req { + p := req + p.UserID = id + return req +} + +// AliasValue copies req by value: the write to the copy never reaches req. +func AliasValue(req Req, id string) string { + p := req + p.UserID = id + return req.UserID +} + +// CallAliasAssign calls aliasAssign, whose v reaches its p only through the +// alias q: the call has a summary edge from v's actual_in to p's actual_out. +func CallAliasAssign(p *Inner, v int) { + aliasAssign(p, v) +} + +// AliasPtrValue takes req's address (spec §4.10's motivating case): the +// write through p reaches the read of the value req. +func AliasPtrValue(id string) string { + var req Req + p := &req + p.UserID = id + return req.UserID +} + +// AliasArraySlice writes the array arr through its slice s. +func AliasArraySlice(v int) int { + var arr [4]int + s := arr[:] + s[0] = v + return arr[0] +} + +// AliasFieldAddr writes o's embedded Inner through its address p: a +// one-step offset (p.n is o.Inner.n). +func AliasFieldAddr(v int) int { + var o Outer + p := &o.Inner + p.n = v + return o.Inner.n +} + +// AliasValueCopies copies s twice by value, no address taken: the write to +// s1 never reaches s2 (Inner holds no reference). +func AliasValueCopies(s Inner, v int) int { + s1 := s + s2 := s + s1.n = v + return s2.n +} + +// AliasRefField copies the map field s.m out: the write through m reaches +// the read through s.m (a reference-typed field, one-step offset). +func AliasRefField(s memStore, k, v string) string { + m := s.m + m[k] = v + return s.m[k] +} + +type ptrHolder struct { + ptr *Inner +} + +// AliasPtrField copies the pointer field r.ptr out: the write through p +// reaches the read through r.ptr. +func AliasPtrField(r ptrHolder, v int) int { + p := r.ptr + p.n = v + return r.ptr.n +} + +// AliasIfaces holds one value in two unrelated interfaces: the write +// through w may change what r reads. +func AliasIfaces(f *bytes.Buffer, p []byte) int { + var r io.Reader = f + var w io.Writer = f + w.Write(p) + n, _ := r.Read(p) + return n +} + +// AliasEntryValue: q may point to the value param req, so req's input +// reaches the read q.UserID through q (an alias edge from @entry whose def +// is req, not q). +func AliasEntryValue(req Req, q *Req, c bool) string { + if c { + q = &req + } + return q.UserID +} + +// GReq is a value global whose address is taken (takeGReq). +var GReq Req + +func takeGReq() *Req { return &GReq } + +// AliasEntryGlobal reads GReq and q, which may point to GReq: GReq's input +// reaches the read q.UserID. +func AliasEntryGlobal(q *Req) string { + s := GReq.UserID + return s + q.UserID +} + +// AliasPtrPtr rewrites p through its address pp: p.n then reads b.n. +func AliasPtrPtr(a, b *Inner) int { + p := a + pp := &p + *pp = b + return p.n +} + +// AliasSlicePtr rewrites the slice s through its address ps. +func AliasSlicePtr(s, b []int) int { + ps := &s + *ps = b + return s[0] +} + +// AliasArrPtr writes the array pa points to through its slice s. +func AliasArrPtr(pa *[4]int) int { + s := pa[:] + s[0] = 1 + return pa[0] +} + +// CaptureAlias stores a literal that writes through a local alias of the +// captured pointer p: the write is no def of p, but p may hold a reference, +// so the literal has a capture formal_out for p (the by-ref predicate) and +// x reaches p's formal_out (found by metamorphic relation MR5). +func CaptureAlias(p *Inner, x int) { + f := func() { q := p; q.n = x } + f() +} + +// sinkInt is a sink with no by-ref or global output. +func sinkInt(v int) int { return v } + +// ScalarSink passes a local scalar computed from a parameter to sinkInt (the +// metamorphic relations' constant-argument and call-routing sites). +func ScalarSink(a int) int { + x := a + 1 + sinkInt(x) + return x +} diff --git a/testdata/dataflow/sdg/stdlib.go b/testdata/dataflow/sdg/stdlib.go new file mode 100644 index 0000000..89d9ebe --- /dev/null +++ b/testdata/dataflow/sdg/stdlib.go @@ -0,0 +1,34 @@ +package sdg + +// The library calls of the L4 stand-in fixture; they live apart from sdg.go +// so that its line:col keys stay put. + +import ( + "encoding/json" + "fmt" + "runtime" + "strconv" +) + +// format calls fmt.Sprintf: its stand-in has formal_ins format and a. +func format(a int) string { + return fmt.Sprintf("%d", a) +} + +// decode calls json.Unmarshal, whose v (any) is a by-ref out. +func decode(data []byte) map[string]int { + var m map[string]int + _ = json.Unmarshal(data, &m) + return m +} + +// atoi calls strconv.Atoi: s flows to both results unless a model says not. +func atoi(s string) int { + n, _ := strconv.Atoi(s) + return n +} + +// yield calls runtime.Gosched, an external with no formals at all. +func yield() { + runtime.Gosched() +} diff --git a/testdata/dataflow/variant/p/p.go b/testdata/dataflow/variant/p/p.go new file mode 100644 index 0000000..82a67bf --- /dev/null +++ b/testdata/dataflow/variant/p/p.go @@ -0,0 +1,26 @@ +// Package p is the L4 fixture for whole-program SSA with tests loaded +// (--skip-tests=false): p_test.go makes go/packages yield the test variant +// "p [p.test]" beside the plain p that package u imports. Only the plain p's +// impl reaches u's interface call, so SSA must build the plain p too. +package p + +// I is the interface u calls. +type I interface { + M(x string) string +} + +type impl struct{} + +// Last is the global impl.M writes. +var Last string + +// M records x and returns it. +func (impl) M(x string) string { + Last = x + return x +} + +// New returns the one implementation. +func New() I { + return impl{} +} diff --git a/testdata/dataflow/variant/p/p_test.go b/testdata/dataflow/variant/p/p_test.go new file mode 100644 index 0000000..4b65e52 --- /dev/null +++ b/testdata/dataflow/variant/p/p_test.go @@ -0,0 +1,9 @@ +package p + +import "testing" + +func TestM(t *testing.T) { + if New().M("a") != "a" { + t.Fatal("M") + } +} diff --git a/testdata/dataflow/variant/u/u.go b/testdata/dataflow/variant/u/u.go new file mode 100644 index 0000000..e306ca2 --- /dev/null +++ b/testdata/dataflow/variant/u/u.go @@ -0,0 +1,9 @@ +// Package u calls p's interface method on the value p.New returns. +package u + +import "example.com/dataflow/variant/p" + +// Use dispatches M through the interface p.New returns. +func Use(x string) string { + return p.New().M(x) +} diff --git a/testdata/dataflow/vendor/example.com/vlib/box/box.go b/testdata/dataflow/vendor/example.com/vlib/box/box.go new file mode 100644 index 0000000..2245620 --- /dev/null +++ b/testdata/dataflow/vendor/example.com/vlib/box/box.go @@ -0,0 +1,43 @@ +// Package box is a vendored library of interface containers: what it hands +// out comes from library code, which L4 builds no SSA for. Its elements +// have its own interface type, so a caller needs no type assertion. +package box + +// Finder is the library's interface; project types implement it. +type Finder interface { + Find(id string) string +} + +var store = map[string]Finder{} + +// Put stores f under k. +func Put(k string, f Finder) { store[k] = f } + +// Map returns the stored values by key. +func Map() map[string]Finder { return store } + +// List returns the stored values. +func List() []Finder { + var out []Finder + for _, f := range store { + out = append(out, f) + } + return out +} + +// First returns the value under k as a one-element array. +func First(k string) [1]Finder { return [1]Finder{store[k]} } + +// Chan returns a channel that yields the value under k once. +func Chan(k string) <-chan Finder { + ch := make(chan Finder, 1) + ch <- store[k] + return ch +} + +// Holder holds a Finder; a project type embedding it promotes Get, whose +// code is library code. +type Holder struct{ F Finder } + +// Get returns the held Finder. +func (h Holder) Get() Finder { return h.F } diff --git a/testdata/dataflow/vendor/modules.txt b/testdata/dataflow/vendor/modules.txt index 06b192c..ee62d69 100644 --- a/testdata/dataflow/vendor/modules.txt +++ b/testdata/dataflow/vendor/modules.txt @@ -1,3 +1,4 @@ # example.com/vlib v0.0.0 ## explicit example.com/vlib +example.com/vlib/box diff --git a/testdata/dataflow/wrap/wrap.go b/testdata/dataflow/wrap/wrap.go new file mode 100644 index 0000000..d511c2f --- /dev/null +++ b/testdata/dataflow/wrap/wrap.go @@ -0,0 +1,65 @@ +// Package wrap is the L4 fixture for VTA targets behind synthetic wrappers +// (spec §4.8): an interface call reaches sqlStore.Find directly, through a +// method value of the interface (VTA returns the (Store).Find$bound wrapper) +// and through a struct embedding the interface (VTA returns logged's +// promotion wrapper). Each wrapper's own call of Find is a synthetic call +// site; its VTA targets are the original site's. Tests pin keys by line:col, +// so keep each statement on its own line and append new functions at the end. +package wrap + +// Store is the interface the handlers call. +type Store interface { + Find(id string) string +} + +// lastQuery is the global sqlStore.Find writes. +var lastQuery string + +type sqlStore struct{} + +// Find builds the query from id and runs it: the sink is Exec's query. +func (s *sqlStore) Find(id string) string { + lastQuery = id + return Exec("select " + id) +} + +// Exec is the sink. +func Exec(query string) string { + return query +} + +// HandleDirect calls Find through the interface. +func HandleDirect(s Store, id string) string { + return s.Find(id) +} + +// HandleBound calls Find through a method value of the interface. +func HandleBound(s Store, id string) string { + f := s.Find + return f(id) +} + +// logged decorates a Store by embedding it: Find is promoted. +type logged struct { + Store +} + +// HandleDecorated calls Find on a decorated Store, through the interface. +func HandleDecorated(s Store, id string) string { + var l Store = logged{s} + return l.Find(id) +} + +// MainDirect, MainBound and MainDecorated each pass a sqlStore to one +// handler: no host variable is shared between the handlers' call sites. +func MainDirect() { + HandleDirect(&sqlStore{}, "a") +} + +func MainBound() { + HandleBound(&sqlStore{}, "b") +} + +func MainDecorated() { + HandleDecorated(&sqlStore{}, "c") +} diff --git a/testdata/dataflow/xtest/a.go b/testdata/dataflow/xtest/a.go new file mode 100644 index 0000000..435d601 --- /dev/null +++ b/testdata/dataflow/xtest/a.go @@ -0,0 +1,45 @@ +// Package xtest is called from an external test package (a_test.go, package +// xtest_test) and from an in-package test (b_test.go). With +// --skip-tests=false the loader also yields the test variant +// "xtest [xtest.test]"; package user imports the plain xtest, so SSA holds +// both. The join must map each variant's functions to its one callable. +// +// It also holds one function per SSA wrapper kind the join unwraps: a method +// value ($bound), a method expression ($thunk), a promoted method and a +// pointer wrapper of a value method. +package xtest + +// Counter counts. +type Counter struct{ n int } + +// Add adds d through a closure. +func (c *Counter) Add(d int) int { + inc := func() { c.n += d } + inc() + return c.n +} + +// Get is a value method: (*Counter).Get is its pointer wrapper. +func (c Counter) Get() int { return c.n } + +// Outer promotes Counter's methods. +type Outer struct{ *Counter } + +// New returns a counter. +func New() *Counter { return &Counter{} } + +// Bound returns a method value: SSA makes the closure (*Counter).Add$bound. +func Bound(c *Counter) func(int) int { return c.Add } + +// Thunk returns a method expression: SSA makes (*Counter).Add$thunk. +func Thunk() func(*Counter, int) int { return (*Counter).Add } + +// Adder is an interface: a method value of it is a $bound wrapper of the +// interface method, and a method expression a $thunk of it. +type Adder interface{ Add(int) int } + +// IfaceBound returns a method value of an interface. +func IfaceBound(a Adder) func(int) int { return a.Add } + +// IfaceThunk returns a method expression of an interface. +func IfaceThunk() func(Adder, int) int { return Adder.Add } diff --git a/testdata/dataflow/xtest/a_test.go b/testdata/dataflow/xtest/a_test.go new file mode 100644 index 0000000..385d24c --- /dev/null +++ b/testdata/dataflow/xtest/a_test.go @@ -0,0 +1,19 @@ +package xtest_test + +import ( + "testing" + + "example.com/dataflow/xtest" +) + +func helper(c *xtest.Counter) int { return c.Add(2) } + +func TestAdd(t *testing.T) { + c := xtest.New() + check := func(n int) { + if n != 2 { + t.Fatal(n) + } + } + check(helper(c)) +} diff --git a/testdata/dataflow/xtest/b_test.go b/testdata/dataflow/xtest/b_test.go new file mode 100644 index 0000000..118cf73 --- /dev/null +++ b/testdata/dataflow/xtest/b_test.go @@ -0,0 +1,12 @@ +package xtest + +import "testing" + +// TestNew is an in-package test: it makes the loader yield the variant +// "example.com/dataflow/xtest [example.com/dataflow/xtest.test]", which has +// the plain package's path and shares its parsed a.go. +func TestNew(t *testing.T) { + if New().Add(1) != 1 { + t.Fatal("Add") + } +} diff --git a/testdata/dataflow/xtest/user/user.go b/testdata/dataflow/xtest/user/user.go new file mode 100644 index 0000000..e68f75c --- /dev/null +++ b/testdata/dataflow/xtest/user/user.go @@ -0,0 +1,9 @@ +// Package user imports xtest from non-test code. With --skip-tests=false the +// loader gives it the plain xtest while xtest_test imports the test variant +// "xtest [xtest.test]": SSA then holds two packages for one import path. +package user + +import "example.com/dataflow/xtest" + +// Use calls into xtest. +func Use() int { return xtest.New().Add(3) } diff --git a/testdata/metamorphic/conventions.go b/testdata/metamorphic/conventions.go new file mode 100644 index 0000000..376df60 --- /dev/null +++ b/testdata/metamorphic/conventions.go @@ -0,0 +1,51 @@ +package examples + +// The oracle conventions internal/metamorphic encodes; each is pinned by a +// test there (TestConvention_*), which names the rule when it breaks. + +// ConvCaptureSite (i): the literal's capture site actual_in for p feeds only +// the closure's formal_in (param_in), never a summary or ddg edge: it is not +// a reachability sink. +func ConvCaptureSite(p *pt, v int) { + f := func() { p.n = v } + f() +} + +// ConvGuardInLoop (ii): MR8 wrapping `s += i` in a loop body gives the guard +// the self-loop B→B on c. +func ConvGuardInLoop(n int) int { + s := 0 + for i := 0; i < n; i++ { + s += i + s++ + } + return s +} + +// ConvGuardStraight (ii): MR8 wrapping `x += a` outside a loop gives no +// guard edge. +func ConvGuardStraight(a int) int { + x := a + x += a + return x +} + +// ConvDiscardedResult (iii): p.sum()'s discarded result reaches the return +// only through actual_out → call node (coarse wiring) and the call node's +// control dependence (it may panic): MR14(b) cuts that actual_out. +func ConvDiscardedResult(p pt2) int { + p.sum() + return p.a +} + +// pt2 has a value method. +type pt2 struct{ a, b int } + +func (p pt2) sum() int { return p.a + p.b } + +// ConvNamedKill (iv): `n = a + 1` reaches @formal_out through the @exit +// port edge; MR9's kill before the return takes it over. +func ConvNamedKill(a int) (n int) { + n = a + 1 + return +} diff --git a/testdata/metamorphic/doc.go b/testdata/metamorphic/doc.go new file mode 100644 index 0000000..afc2ca1 --- /dev/null +++ b/testdata/metamorphic/doc.go @@ -0,0 +1,12 @@ +// Package examples holds small functions written to be eligible sites of the +// metamorphic relations (internal/metamorphic): each one names the relation +// it targets and the effect the relation expects. conventions.go pins the +// oracle conventions those relations rely on. +package examples + +// sinkInt is the sink the call sites pass values to: it has no by-ref or +// global output, so a flow into its argument ends at the call. +func sinkInt(v int) int { return v } + +// sink2 is a two-argument sink. +func sink2(v, w int) int { return v + w } diff --git a/testdata/metamorphic/go.mod b/testdata/metamorphic/go.mod new file mode 100644 index 0000000..ec07201 --- /dev/null +++ b/testdata/metamorphic/go.mod @@ -0,0 +1,3 @@ +module example.com/metamorphic + +go 1.21 diff --git a/testdata/metamorphic/mr01_04_layout.go b/testdata/metamorphic/mr01_04_layout.go new file mode 100644 index 0000000..3183f53 --- /dev/null +++ b/testdata/metamorphic/mr01_04_layout.go @@ -0,0 +1,20 @@ +package examples + +// MR1-MR4, MR7, MR15 (layout, rename, target-files): a type with a method +// declared above it, a function, and locals to rename. Exact oracle: the +// analysis is unchanged up to positions (and spelling for MR4). + +// Scale is declared above its receiver type (MR2 reverses it back below). +func (c *counter) Scale(k int) int { + m := c.n * k + return m +} + +// counter is a struct with one field. +type counter struct{ n int } + +// Sum adds two values through locals (MR3 may move it; MR4 renames a, b, s). +func Sum(a, b int) int { + s := a + b + return s +} diff --git a/testdata/metamorphic/mr05_alias.go b/testdata/metamorphic/mr05_alias.go new file mode 100644 index 0000000..4870b2d --- /dev/null +++ b/testdata/metamorphic/mr05_alias.go @@ -0,0 +1,16 @@ +package examples + +// pt is the struct MR5's alias writes go through. +type pt struct{ n int } + +// SetPtr: MR5 rewrites p.n = v as `q := p; q.n = v`; v still reaches p's formal_out. +func SetPtr(p *pt, v int) { + p.n = v +} + +// SetValue: MR5 rewrites s.n = v as `q := &s; q.n = v`; v still reaches the return. +func SetValue(v int) int { + var s pt + s.n = v + return s.n +} diff --git a/testdata/metamorphic/mr06_extract.go b/testdata/metamorphic/mr06_extract.go new file mode 100644 index 0000000..0ce3078 --- /dev/null +++ b/testdata/metamorphic/mr06_extract.go @@ -0,0 +1,9 @@ +package examples + +// Extract: MR6 moves `x := a + 1; y := b * 2` into a helper taking a, b and +// returning x, y; a and b still reach the return through param_in/param_out. +func Extract(a, b int) int { + x := a + 1 + y := b * 2 + return x + y +} diff --git a/testdata/metamorphic/mr08_wrap.go b/testdata/metamorphic/mr08_wrap.go new file mode 100644 index 0000000..8aa4a70 --- /dev/null +++ b/testdata/metamorphic/mr08_wrap.go @@ -0,0 +1,27 @@ +package examples + +// WrapStraight: MR8 wraps `x += a` in `if c { … }`; the def before it now also +// reaches the return (x += a is conditional). +func WrapStraight(a int) int { + x := a + x += a + return x +} + +// WrapInLoop: MR8 wraps `s += i` (a loop-body statement); the guard's def +// reaches its own condition around the loop (B→B on c, see conventions.go). +func WrapInLoop(n int) int { + s := 0 + for i := 0; i < n; i++ { + s += i + s++ + } + return s +} + +// WrapBeforeReturn: MR8 wraps `y++`, the statement right before the return. +func WrapBeforeReturn(a int) int { + y := a * 2 + y++ + return y +} diff --git a/testdata/metamorphic/mr09_kill.go b/testdata/metamorphic/mr09_kill.go new file mode 100644 index 0000000..eb34712 --- /dev/null +++ b/testdata/metamorphic/mr09_kill.go @@ -0,0 +1,22 @@ +package examples + +// KillOneUse: MR9 inserts `x = 0` before sinkInt(x); x := a + 1 no longer +// reaches the call or the return, the kill does. +func KillOneUse(a int) int { + x := a + 1 + sinkInt(x) + return x +} + +// KillTwoUses: MR9 inserts `x = 0` before the return, which reads x twice. +func KillTwoUses(a int) int { + x := a + 1 + return x * x +} + +// KillNamed: MR9 inserts `n = 0` before the bare return; the kill takes over +// n's port edge into @formal_out (see conventions.go). +func KillNamed(a int) (n int) { + n = a + 1 + return +} diff --git a/testdata/metamorphic/mr10_copy.go b/testdata/metamorphic/mr10_copy.go new file mode 100644 index 0000000..b9f63ce --- /dev/null +++ b/testdata/metamorphic/mr10_copy.go @@ -0,0 +1,14 @@ +package examples + +// CopyInReturn: MR10 inserts `y := x` before the return and returns y*2; +// x's def now reaches the copy, the copy the return. +func CopyInReturn(a int) int { + x := a + 1 + return x * 2 +} + +// CopyInAssign: MR10 copies one of the reads of the assignment's right side. +func CopyInAssign(a, b int) int { + z := a - b + return z +} diff --git a/testdata/metamorphic/mr11_swap.go b/testdata/metamorphic/mr11_swap.go new file mode 100644 index 0000000..ac0d9ef --- /dev/null +++ b/testdata/metamorphic/mr11_swap.go @@ -0,0 +1,8 @@ +package examples + +// Swap: MR11 swaps the two independent definitions; ddg and cdg are unchanged. +func Swap(a, b int) int { + x := a + 1 + y := b + 2 + return x + y +} diff --git a/testdata/metamorphic/mr12_early_return.go b/testdata/metamorphic/mr12_early_return.go new file mode 100644 index 0000000..e3110c1 --- /dev/null +++ b/testdata/metamorphic/mr12_early_return.go @@ -0,0 +1,14 @@ +package examples + +// EarlyReturn: MR12 inserts `if c { return 0 }` before a statement; the +// statements after it become control dependent on the new branch. +func EarlyReturn(a int) int { + x := a + 1 + return x +} + +// EarlyReturnTwo: MR12 on a two-result function returns `0, ""` early. +func EarlyReturnTwo(a int) (int, string) { + x := a * 3 + return x, "ok" +} diff --git a/testdata/metamorphic/mr13_const_sink.go b/testdata/metamorphic/mr13_const_sink.go new file mode 100644 index 0000000..b734c78 --- /dev/null +++ b/testdata/metamorphic/mr13_const_sink.go @@ -0,0 +1,25 @@ +package examples + +// ConstScalar: MR13 replaces x in sinkInt(x) with 0; no formal_in reaches +// the call's actual_in any more. +func ConstScalar(a int) int { + x := a + 1 + sinkInt(x) + return x +} + +// ConstTwoArgs: MR13 replaces x in sink2(x, 3) with 0. +func ConstTwoArgs(a int) int { + x := a - 1 + sink2(x, 3) + return x +} + +// ConstInLoop: MR13 replaces x in the loop's sinkInt(x) with 0. +func ConstInLoop(a, n int) int { + x := a * 2 + for i := 0; i < n; i++ { + sinkInt(x) + } + return x +} diff --git a/testdata/metamorphic/mr14_route.go b/testdata/metamorphic/mr14_route.go new file mode 100644 index 0000000..8f8baea --- /dev/null +++ b/testdata/metamorphic/mr14_route.go @@ -0,0 +1,28 @@ +package examples + +import "strconv" + +// RouteIface: MR14(a) routes sinkInt(x) through a one-implementation +// interface (the literal rules out the closure form, and the call is not +// last, so not defer); the call's summary edge has its counterpart. +func RouteIface(a int) int { + x := a + 1 + _ = func() {} + sinkInt(x) + return x +} + +// RouteClosure: MR14(b) routes strconv.Itoa(x) through func(){ … }() (a +// library call: no interface form; not last: no defer). +func RouteClosure(a int) int { + x := a + 1 + strconv.Itoa(x) + return x +} + +// RouteDefer: MR14(c) defers strconv.Itoa(x), the last statement. +func RouteDefer(a int) { + x := a + 1 + _ = func() {} + strconv.Itoa(x) +} diff --git a/testdata/rangefunc/go.mod b/testdata/rangefunc/go.mod new file mode 100644 index 0000000..af1ce4b --- /dev/null +++ b/testdata/rangefunc/go.mod @@ -0,0 +1,3 @@ +module example.com/rangefunc + +go 1.23 diff --git a/testdata/rangefunc/r.go b/testdata/rangefunc/r.go new file mode 100644 index 0000000..eedc0b5 --- /dev/null +++ b/testdata/rangefunc/r.go @@ -0,0 +1,34 @@ +// Package rangefunc holds the Go 1.23 range forms (#28): a range over an +// integer and a range over an iterator function. testdata/dataflow declares +// go 1.21, which rejects both. +package rangefunc + +// RangeInt ranges over an integer: the header neither calls nor panics. +func RangeInt() int { + s := 0 + for i := range 10 { + s += i + } + return s +} + +// RangeFunc ranges over an iterator function: the header calls seq, so it +// may panic. +func RangeFunc(seq func(func(int) bool)) int { + s := 0 + for v := range seq { + s += v + } + return s +} + +func sink(int) {} + +// RangeFuncCall calls sink in the body of a range over an iterator: in SSA +// the call sits in the synthetic yield function, which folds into +// RangeFuncCall. +func RangeFuncCall(seq func(func(int) bool)) { + for v := range seq { + sink(v) + } +} diff --git a/testdata/rangefuncpkg/go.mod b/testdata/rangefuncpkg/go.mod new file mode 100644 index 0000000..f3c8730 --- /dev/null +++ b/testdata/rangefuncpkg/go.mod @@ -0,0 +1,3 @@ +module example.com/rangefuncpkg + +go 1.23 diff --git a/testdata/rangefuncpkg/pkglevel.go b/testdata/rangefuncpkg/pkglevel.go new file mode 100644 index 0000000..a227c26 --- /dev/null +++ b/testdata/rangefuncpkg/pkglevel.go @@ -0,0 +1,17 @@ +// Package rangefuncpkg ranges over a function only inside a package-level +// function literal: that literal has no callable (it is no L3 unit), so the +// range_func entry must still list this package. +package rangefuncpkg + +func seq(yield func(int) bool) { + yield(1) +} + +// Sum is a package-level variable holding a literal that ranges over seq. +var Sum = func() int { + s := 0 + for v := range seq { + s += v + } + return s +} diff --git a/testdata/ssapanic/a/a.go b/testdata/ssapanic/a/a.go new file mode 100644 index 0000000..837394b --- /dev/null +++ b/testdata/ssapanic/a/a.go @@ -0,0 +1,13 @@ +// Package a instantiates g.Map[int] and then calls Boom. A test hook panics +// on the call to Boom, after a's builder has created the shared instance. +package a + +import "example.com/ssapanic/g" + +func Boom() {} + +func UseA() int { + v := g.Map[int](1) + Boom() + return v +} diff --git a/testdata/ssapanic/b/b.go b/testdata/ssapanic/b/b.go new file mode 100644 index 0000000..e0611ea --- /dev/null +++ b/testdata/ssapanic/b/b.go @@ -0,0 +1,6 @@ +// Package b instantiates the same g.Map[int] as package a. +package b + +import "example.com/ssapanic/g" + +func UseB() int { return g.Map[int](2) } diff --git a/testdata/ssapanic/g/g.go b/testdata/ssapanic/g/g.go new file mode 100644 index 0000000..6d3c1cc --- /dev/null +++ b/testdata/ssapanic/g/g.go @@ -0,0 +1,5 @@ +// Package g declares a generic that packages a and b both instantiate at the +// same type argument, so they share one SSA instance. +package g + +func Map[T any](x T) T { return x } diff --git a/testdata/ssapanic/go.mod b/testdata/ssapanic/go.mod new file mode 100644 index 0000000..bc404d8 --- /dev/null +++ b/testdata/ssapanic/go.mod @@ -0,0 +1,3 @@ +module example.com/ssapanic + +go 1.21 diff --git a/testdata/ssapanic/w/w.go b/testdata/ssapanic/w/w.go new file mode 100644 index 0000000..efd57c8 --- /dev/null +++ b/testdata/ssapanic/w/w.go @@ -0,0 +1,7 @@ +// Package w has a value method that no code calls: its pointer wrapper +// (*V).Never is built only on demand, when the VTA function set is computed. +package w + +type V struct{} + +func (V) Never() {} diff --git a/testdata/storedlit/go.mod b/testdata/storedlit/go.mod new file mode 100644 index 0000000..e0929f2 --- /dev/null +++ b/testdata/storedlit/go.mod @@ -0,0 +1,3 @@ +module example.com/storedlit + +go 1.21 diff --git a/testdata/storedlit/storedlit.go b/testdata/storedlit/storedlit.go new file mode 100644 index 0000000..cacae73 --- /dev/null +++ b/testdata/storedlit/storedlit.go @@ -0,0 +1,31 @@ +// Package storedlit is the L4 fixture for a global written only inside a +// stored function literal that runs later through a function value no +// analysis resolves (Task 13 review): inside Publish, v reaches sink's +// formal_out only through the caller's host, never through the literal. +package storedlit + +import "sync/atomic" + +// sink is written only by the literal Publish stores. +var sink string + +var hook atomic.Value + +// Publish stores a literal that writes v into sink. +func Publish(v string) { + hook.Store(func() { sink = v }) +} + +// Fire calls the stored literal through an unresolved function value. +func Fire() { + hook.Load().(func())() +} + +func readSink() string { return sink } + +// PublishThenRead returns what the stored literal wrote. +func PublishThenRead(v string) string { + Publish(v) + Fire() + return readSink() +} diff --git a/testdata/unmapped/go.mod b/testdata/unmapped/go.mod new file mode 100644 index 0000000..f9322a7 --- /dev/null +++ b/testdata/unmapped/go.mod @@ -0,0 +1,3 @@ +module example.com/unmapped + +go 1.21 diff --git a/testdata/unmapped/u.go b/testdata/unmapped/u.go new file mode 100644 index 0000000..8a8602c --- /dev/null +++ b/testdata/unmapped/u.go @@ -0,0 +1,13 @@ +// Package unmapped holds a function literal in a package-level variable +// initializer. The symbol table has no callable for it (L1 records callables +// of declarations only), so its SSA function, a child of the synthetic +// package initializer, maps to nothing: unmapped_ssa. +package unmapped + +func helper() int { return 1 } + +// Handler is a package-level function literal. +var Handler = func() int { return helper() } + +// Use is a declared function: it maps. +func Use() int { return Handler() }