Repository navigation
Go L3 intraprocedural dataflow, app-outermost ids and L2 library calls - #16
Merged
Merged
Conversation
…ojectDir; span-less external params; chi externals determinism
BuildCFG now returns the pruned node set with the edges.
Adds a by-definition CDG reference check over the dataflow and chi corpora.
BuildNodes takes the keys in recording order instead of rebuilding a token.Pos from line:col, which is wrong under //line directives. Nodes also carry their directive-adjusted line:col range for emission.
dataflow.Run builds cfg/cdg/ddg per callable over -j workers; core runs it at -a 3 and v2emit merges the side table into each callable (statement nodes, id/parent, edge lists, k_limit). L1/L2 output is unchanged.
Also cover the unmatched-closure skip path.
GO_CFG_NEXT (_k=kind), GO_CDG and GO_DDG (_k=var|prov) are now live, keyed on body-node global ids. :GoBodyNode gains call_node and var.
The edge dedup was keyed on (signature, callee), so with several func init() in one package only the first-visited file's init kept an edge to a shared callee, and the visit order was random map order. Key the dedup on the declaring file too, and walk files, types and callables in sorted order.
Neo4j only accepts CALL { ... } IN TRANSACTIONS in an auto-commit
transaction; the purge ran inside ExecuteWrite, so every --eager push onto
existing data failed.
Per-app volume, kind breakdowns, DDG distribution, cost, size and consistency checks, with the extraction script.
r.Read(buf[:n]), io.ReadFull(r, hdr[:]), copy(dst[i:], src) and clear(s[:n]) recorded no write: pathOf has no SliceExpr case. Strip slice expressions to the base and weak-define its elements.
L1 records call sites via fset.Position, which applies //line directives; the index rebuilt a token.Pos from that virtual line:col, so sites under a directive lost their callee and edge. Key the index by (file, start, end) as L1 records them; drops the per-site file scan.
A field selector with no recorded selection, an undeclared name, or a member of a failed import dropped the def and turned the LHS into a use. Treat the selector as a field step, give unresolved names a spelling-identified base (weak root.*), and never make an LHS use-only.
External call edges and external_symbols exist; --emit neo4j is L3; ids are can://<app>/go/...; record the L3 Go leaves (defer two-node rule, defer_resume, body kinds, ids from L1, k_limit, span-less @entry/@EXIT). Mark the R19 closure source_file gap fixed.
Clone, prep, tool checks and type-check status per app; the run phase reads the prepared entries, builds the analyzer with EVAL_GO and adds a type-check column. Kubernetes (root and k8s.io/api) joins the manifest.
Rerun the six apps and both Kubernetes modules (root, k8s.io/api) with the post-fix analyzer; measure protobuf-marshaller DDG volume (I5) on k8s-api. cockroach kept as a labelled pre-fix subsection. l3_metrics.py groups DDG by callable name and counts zz_generated files as generated.
x/tools v0.46 cannot read Go 1.27's export data (version 5), so loads that read export data instead of type-checking deps from source fail on Go 1.27. v0.50+ supports it and requires Go 1.26, raising the go.mod minimum.
This was referenced Oct 25, 2026
Closed
On malformed source the parser can end a function one past its file, where the FileSet has no token.File; nodeSource dereferenced it and the analyzer panicked at every level. It now returns "" for such nodes.
buildCallSites stopped at a go statement's call, so g() in go f(g()) had no call node, callee or edge. It now tags the launched call is_goroutine and records nested calls as ordinary sites. The L3 mirror (recordedCalls) and the test mirror (gateCallKeys) change in lockstep; at L3 the nested calls are satellites of the go node.
append(dst[:0], x), or append into spare capacity, writes dst's backing array; it now weak-defines dst[*] alongside the LHS def. A per-callable panic also logs its stack at -vv.
…al params DeferWrap shows a deferred named-result write never kills; a recording session shows eagerPurge runs its IN TRANSACTIONS statement through Run, not ExecuteWrite; vlib.G/H show unnamed and blank library params become _<i>.
With --target-files the push carries only the targeted packages, so an app-wide purge first would leave the rest of the app missing from the graph.
Contributor
Author
sinha108
marked this pull request as ready for review
October 25, 2026 15:23
pavuluri
approved these changes
Oct 27, 2026
pavuluri
left a comment
pavuluri
left a comment
Contributor
There was a problem hiding this comment.
The PR is rather large with substantial code changes. The main addition is intraprocedural L3 analysis: for each callable, it produces a control-flow graph (CFG), control-dependence graph (CDG), and data-dependence graph (DDG), in addition to the existing symbol and call-graph information. It also projects those graphs into Neo4j. It includes extensive tests and a real-application evaluation, with good scale results.
pavuluri
pushed a commit
that referenced
this pull request
Oct 29, 2026
* docs: amend the L3/L4 spec to the re-checked L4 design * fix: share one call/closure walker; real positions under //line (#26) * fix(l2): resolve call sites inside closures (#17) * feat(l3): model closure capture writes and deferred-operand writes (#22) * feat(l3): writes through reference fields of value structs; bucketed kills * perf: stream analysis.json and release ASTs after use (#27) * feat(l4): accept -a 4 and declare the L4 schema fields * feat(l4): build go/ssa over the project load * fix(l4): rebuild SSA without a package whose build panicked; recover VTA set construction * feat(l4): join go/ssa to the L3 units * fix(l4): fnUnit joins only units; interface methods have their own accessor * fix(l1): keep methods declared above their receiver type * feat(l4): VTA dispatch targets in call_graph * feat(l4): global and mutation footprints over call-graph SCCs * fix(l4): footprints over-approximate aliases, skipped units, literals passed to libraries * fix(l4): every function literal is a callee of its encloser; honour TargetPath * feat(l4): formal and actual vertices * feat(l4): port wiring ddg edges * feat(l4): summaries and parameter edges * feat(l4): library stand-ins and --library-models * perf(l4): release SDG caches early, intern port names, lower GC target during L4 * feat(l4): emit the SDG in JSON and Neo4j * feat(l4): type-based may-alias points-to edges * fix(l4): may-alias compares pointee types, aliases interfaces and address-taken values * fix(l4): may-alias keeps distinct-def entry edges, both anchors at offset 0, array-pointer slices * fix(l4): give a reference-holding capture its formal_out A closure that writes through a local alias of a captured pointer (`q := p; q.n = x`) writes no path of p, so it had no capture formal_out for p and the flow x -> p was refuted. Apply the shared by-ref predicate (MayHoldRef) to captures, as to params. Found by metamorphic relation MR5. * test(l4): metamorphic relations for L3/L4 on fixtures and real apps * fix(l3): a closure's host writes through every reference-holding capture A literal writing through a local alias of a captured pointer (`q := p; q.n = x`) writes no path of p, so at -a 3 its host did not weak-define p.* and a later read of p.n had no edge from it. Add x.* for every capture x that may hold a reference, mirroring 078616f at L4. * test(l4): metamorphic harness race, CDG mismatch as violation, tighter MR13 * test(l4): metamorphic examples module and memos-default real-app run * test(l4): metamorphic script rejects unknown flags; unique statement lookup; stronger example checks * fix(neo4j): force a full push when the stored level is lower A lazy push diffed content_hash only, so an L4 push onto a graph first pushed at L3 skipped unchanged modules and their edges' MATCH dropped rows silently. A stored max_level below the push's now forces a full upsert, like a schema_version mismatch. * fix(l4): deterministic id for duplicate callable keys Two callables sharing an L3Key routed their L4 edges to whichever was emitted last, over map order. Keep the lexically smaller id. * test(l4): no-dangling gates cover the L4 projection and degraded callables * fix(eval): a failed -a 4 run fails the app and the script * ci: run the metamorphic script argument checks * test(l4): a released SDG cache rebuilt before fill ends fails Count per-unit cache builds through a test seam; with the memory levers on each graph, call-site list and wiring is built once and fill leaves the caches empty. * test(neo4j): live lazy L4 push over a lower-level graph equals an eager one * fix(neo4j): force a full push when the stored level is not an integer * test(l4): count summary recomputes; assert the duplicate-key warning * test(l4): HTTP-to-SQL done fixture * feat(eval): -a 4 in the evaluation CANGO_LEVELS="2 3 4" runs the -a 4 triple (-j 1, -j N, rerun) with a byte-identity check and records L4 counts, times, bytes and degraded codes; l3_metrics.py adds an l4 section. * docs: L4 evaluation, spec and README * docs: L4 overlay live in the Neo4j spec; measured memory; eval keeps $USER * test(l4): done gate checks a no-flow pair inside the DB plumbing * test(l4): done gate pins the sanitizer as a variable cut With strconv/Atoi: [] the Atoi summary edges go and the flow stays; a cut on n within SanitizedFind is bypassed by err (cdg, return) and the receiver (coarse host wiring), all correct dependence (codellm-devkit/.github#103). The no-flow pair moves to a separate entry, Audit, that shares no host variable. * docs: library models change summaries, not statement-level reachability * fix(l4): follow VTA through synthetic wrappers A method value of an interface or a struct embedding one makes VTA return a $bound or promotion wrapper whose own call of the concrete method sits at a synthetic site no call node joins; the wrapper's VTA callees are now targets of the original site. * fix(l4): SSA for every package variant With tests loaded the PkgPath-keyed package map keeps only the test variant, so the plain package other packages import had no SSA bodies and VTA missed the types flowing through it. L4 now builds SSA over every loaded variant. * fix(l4): call_graph edges only for targets VTA produced A D7 interface-method target (or L2's static callee) at a site with no L2 edge was emitted with prov ["vta"]. It stays a dispatch target with its param edges; only VTA's own callees become call_graph edges. * refactor(l4): declKey is a free function formalIndex built a partially initialised join just to call it. * fix(l4): CHA targets for interface calls on library-originated values VTA sees no concrete type in library code, so ctx.Value(k).(Store).Find or a sync.Map round-trip kept only the D7 interface method. An invoke site whose receiver originates in a type assertion, a library call or a library load now also targets every project type implementing the interface. Also documents F1 and F7 in spec §4.8. * fix(l4): emit partial_types The documented code was never produced. It lists the project packages go/packages loaded with errors, sorted. * fix(neo4j): clean push clears a stale degraded_json SET n += row.p keeps a property the row lacks, so a clean push over a graph a degraded run wrote kept its degraded_json. The application row now carries an explicit null. Also: Degraded.Code is an open vocabulary. * docs: PR #16 merged, L4 live; partial_types and SSA variants in the spec; stale comments * docs: the CHA complement's origin walk is intraprocedural A library-originated value returned by or passed into a project function is part of the remaining gap, not only one stored in project storage. * fix(l4): library-origin is interprocedural for the CHA complement A library-originated value that reached an interface call through a project function's result or parameter, a closure capture, a field, a global or a container element kept only the D7 interface method. A whole-program monotone fixpoint now tracks it (storage by container type), and map lookups, index and channel receives count as origins. CHA runs under recover: a panic drops its targets and records no_cha. * docs: interprocedural library origin and no_cha * fix(l4): library-origin reads only fixed or tracked facts callLO read the implementer-is-library-code memo before implementers filled it, and the memo is not a fact, so an invoke through a promoted library method got no CHA target when its caller was visited first. The memo is now filled with the implementer list it belongs to. runCHA clones the site slices it restores on a panic. * docs: what drives the CHA complement's volume; the fixpoint's order independence * fix: analysis.json honours the umask and is synced before the rename * fix(l4): bind receivers without a signature, classify fileless callees as L2 does, list range-over-func in package-level literals * test(l4): self-test the superset gate; determinism compares stand-ins * fix: create analysis.json's temp file with the umask, not a umask round-trip; bind on go/types' method-expression kind * docs: L4 evaluation at the final head
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds L3 intraprocedural dataflow to codeanalyzer-go at
-a 3, switches ids to the app-outermostcan://<app>/go/…grammar, and makes L2 keep calls to library functions.Motivation and Context
L3 brings the Go analyzer to parity with codeanalyzer-python and codeanalyzer-java. It is also what python-sdk's slicing and taint walk over: every callable now carries statement body nodes plus
cfg/cdg/ddgedges, in bothanalysis.jsonand the Neo4j projection.The id change (
can://go/<app>becomescan://<app>/go) puts the app outermost, as the Python and Java analyzers already do, so a polyglot app's analyses share onecan://<app>prefix.L2 previously resolved calls to library functions and then threw them away. It now keeps them as
call_graphedges into a newapplication.external_symbolsmap.Design:
docs/design/specs/v2-l3-l4-dataflow.md. L4 (interprocedural SDG) is a separate, later PR.How Has This Been Tested?
Unit and gate tests (
go test ./..., run in CI on Go 1.26, thego.modminimum, and on the latest stable Go):x/tools/go/cfg-j 1vs-j Nbyte-identityLive Neo4j 5.26: the
neo4j_itsuite (41 tests) and repeated--eagerpushes, run in a throwaway Docker container.Real-app evaluation, reproducible with
scripts/eval_l3.sh --preparethenscripts/eval_l3.sh --all:k8s.io/api)-j 1,-j 16and a second-j 16docs/design/notes/l3-dataflow-metrics.mdanddocs/design/notes/l3-eval-2026-10.mdk8s.io/api)cfg/cdg/ddgedges-a 2(-j 1)-a 3analysis.json,-a 2→-a 3k8s.io/api)Takeaways:
-a 3in 12.4 s at 7.1 GB.k8s.io/api, 627kddgedges come almost entirely fromgenerated.pb.go, and the largest marshaller has 14.8k edges.ddgis long-tailed. The top 1% of callables hold 18–53% of each app's edges; the largest is generated conversion code at 130k.Bugs found by the evaluation and reviews, fixed here with regression tests:
func init().--eagerpush failed on Neo4j 5.x.--eagercombined with--target-filespurged the whole app from Neo4j and pushed back only the targeted packages. The purge now runs only on a full run.//linedirectives.gostatement arguments, such asg()ingo f(g()), were not recorded.x/toolsv0.46 could not read Go 1.27's compiled package data; upgraded to v0.52.Breaking Changes
Ids:
can://go/<app>/<file>/…becomescan://<app>/go/<file>/…. External symbols arecan://<app>/@external/<import-path>/<sig>.Body nodes: they now carry
id(<callable-id>@<local>) at every level, including-a 1and-a 2, as codeanalyzer-python does.L2 output gains library edges.
call_graphgains edges to library callees, a call node'scalleeis set for them instead ofnull, andapplication.external_symbolsis new. A type conversion is still a call node, but itscalleestaysnull; it is no longer resolved to a<Type>.__new__pseudo-callee.L1 records more:
kindcan bealiasordefinedis_deferred: trues.A().B(), are separate call nodes keyedl:candl:c/2gostatement arguments are now call sitesConsumers that count call nodes, or parse body keys as plain
line:col, are affected.Neo4j:
--emit neo4jnow projects at L3, addingGO_CFG_NEXT,GO_CDGandGO_DDG.:GoBodyNode.call_nodeis the local parent key, matching Python.python-sdk: saved Go
analysis.jsonfixtures need regenerating, and the Go backend's reader must mapcall_nodeback toparent(python-sdk#434).Toolchain:
go.modmoves fromgo 1.25.0togo 1.26.0, becausegolang.org/x/toolsv0.52 requires it.goline prints one clear warning instead of degrading silently.Upgrade notes for all of the above are in the README's "Upgrading" section.
Types of changes
Checklist
Additional context
-a 3and--graph-field-depth <k>(default 3).go/ast;deferis two nodes plus an exit chain (defer_resumeedges), and panic-capable statements getexceptionedges.prov: ["ssa"].go/ssacomparison check.//linedirectives, and one call walker shared by L1 and L3.