feat(editor): complete columns of every CTE shape and ORDER BY aliases by datlechin · Pull Request #3713 · TableProApp/TablePro

datlechin · 2026-10-20T18:28:19Z

Summary

A CTE's columns now come from AS MATERIALIZED and AS NOT MATERIALIZED bodies, INSERT/UPDATE/DELETE/MERGE ... RETURNING, VALUES with a column list, TABLE t and nested WITH bodies.
* and t.* in a CTE or derived table, and RETURNING *, expand to the table's columns from the schema cache. RETURNING * adds the UPDATE ... FROM and DELETE ... USING tables, and a USING join lists its join columns once, as PostgreSQL 17 returns them.
A column list (WITH c (a, b) AS, AS x(a, b)) renames by position and keeps the rest. Implicit aliases (count(*) total) and LATERAL derived tables resolve.
ORDER BY offers the SELECT list's aliases ahead of the table columns, none inside an aggregate or WITHIN GROUP.
SET and an INSERT column list read the table the statement writes, not the first table found in the statement: UPDATE t SET .. FROM u, INSERT INTO t (..) SELECT .. FROM s, a CTE body, UPDATE o SET .. FROM orders o.
DerivedTableParser reads the analyzer's lexed code view as tokens, so a dollar quote, a backslash escape or a comment no longer breaks paren matching. CTE names come from it instead of two regexes, and a CTE binds only where it is in scope.
LATERAL is no longer read as a table, which fetched columns for a table named LATERAL on each keystroke.

Test plan

TableProAutocompleteTests (swift test): DerivedTableParserTests per shape, SQLContextAnalyzerStatementShapeTests, provider cases against a stub catalog.
Changed on purpose: "SELECT star yields no resolvable columns" now asserts the table the star leaves to the catalog.
Hosted autocomplete suites and the iOS completion suites.
RETURNING, MERGE, VALUES, USING and column-list widths checked against PostgreSQL 17.

# Conflicts: # apple/Packages/TableProCore/Sources/TableProAutocomplete/SQLContextAnalyzer.swift # apple/Packages/TableProCore/Tests/TableProAutocompleteTests/SQLCompletionProviderCatalogTests.swift # apple/TablePro/Core/Autocomplete/CompletionEngine.swift

J2TeamNNL

Review findings, mostly on completion scoping. Items 2 and 3 are worth checking first.

J2TeamNNL · 2026-10-21T09:29:58Z

-                continue
+            if nextCTE < ctes.count, ctes[nextCTE].open == index {
+                let cte = ctes[nextCTE]
+                found.append((DerivedTable(alias: cte.name, projection: cte.projection, isCommonTableExpression: true), depth))


visibleTables returns every CTE of the WITH list, including the CTE whose own body holds the cursor and CTEs defined after it. In WITH orders AS (SELECT id FROM orders WHERE |) SELECT ... the inner orders is the real table, but mergeDerivedTables swaps it for the CTE projection, so completion offers only id. collect appends every CTE to found unconditionally, so the in-scope rule applies to projections but not to visibleTables.

J2TeamNNL · 2026-10-21T09:29:58Z

+        default:
+            return nil
+        }
+        if stream.isWord(index, "ONLY") { index += 1 }


modifiedTable treats the first word after UPDATE as the table. UPDATE OR REPLACE users SET | (SQLite), UPDATE LOW_PRIORITY t SET and UPDATE TOP (n) t SET resolve the target to OR, LOW_PRIORITY or TOP. .set now prefers context.targetTable over tableReferences.first, so no columns are offered. This is a regression. path() rejects IGNORE through nonAliasWords but not these.

J2TeamNNL · 2026-10-21T09:29:58Z

-        }
-        for cteName in cteNames where present.insert(cteName.lowercased()).inserted {
-            references.append(TableReference(tableName: cteName, alias: nil))
+            references.append(TableReference(


Every visible CTE is appended to tableReferences with its columns, even when the query never reads it. WITH a AS (SELECT x FROM t), b AS (SELECT y FROM u) SELECT | FROM a now offers y from b. Only derived tables used in FROM or JOIN should add unqualified columns. Unreferenced CTEs should stay resolvable for b. and FROM completion only.

J2TeamNNL · 2026-10-21T09:29:58Z

+    ])
+
+    // Innermost first, so a nearer definition shadows an outer one of the same name.
+    func visibleTables(in stream: SQLTokenStream, cursor: Int) -> [DerivedTable] {


The whole statement is re-tokenized and every definition's projection is recomputed on each completion request, and several projections are computed twice (collect calls withList, mainStatement calls it again per inner body, definition and collect repeat one level down). This runs on every keystroke. Cache projections per call or project only what the cursor scope uses.

J2TeamNNL · 2026-10-21T09:29:58Z

+    }
+
+    // CTEs were bound to their own scope by the parser, so what is left to expand is a table the catalog knows.
+    private func derivedColumns(of reference: TableReference) async -> [String] {


derivedColumns(of:) awaits the catalog sequentially per table, and columnItems(for:) calls it once per reference. The same tables are queried again in allColumnsInScope. N derived tables using * mean N sequential awaits plus repeated fetches on every SELECT-list or WHERE completion. Use a TaskGroup or batch the lookups and reuse the results.

J2TeamNNL · 2026-10-21T09:29:58Z

+    }
+
+    // The open parentheses around `cursor`, outermost first.
+    func enclosingParens(before cursor: Int) -> [Int] {


The statement-body check in targetTable accepts a column-list paren whose first column is named like a body keyword. INSERT INTO t (values, |, (table, | or (update, | makes isStatementBody true for the column list, targetTable finds no statement and returns nil, and the result falls back to tableReferences.first. Check the token before the paren so INSERT column lists are excluded.

J2TeamNNL · 2026-10-21T09:29:58Z

+    let isCommonTableExpression: Bool
+
+    // What the statement text names, before the catalog expands a `*`.
+    var columns: [String] {


DerivedTable.columns has no production caller. The analyzer and provider use projection, so this property only exists for tests and can drift from projection.columns(expanding:). Delete it and have tests call projection.columns { _ in nil }.

J2TeamNNL · 2026-10-21T09:29:58Z

-        while i < end {
-            if let skipped = skipNonCode(in: ns, at: i, limit: end) {
-                i = skipped
+    // MARK: - Definitions


About 58 new // lines restate what the code does, for example // MARK: - Definitions and the one above nextFromItem. AGENTS.md says code comments say why, never what. Remove the ones that only describe the code beneath them.

J2TeamNNL · 2026-10-21T09:40:43Z

Fixes for the review findings, in J2TeamNNL@559b114

CTE visibility (visibleTables): fixed. A CTE body now sees only the CTEs before it, and itself only under RECURSIVE.
UPDATE modifiers: fixed. OR REPLACE, LOW_PRIORITY, IGNORE and TOP (n) are skipped in the parser and in the reference regex.
Unread CTEs: fixed. A CTE or derived table the query never reads stays resolvable for b., but adds no unqualified column.
Re-parsing cost: partly fixed. Projections are now cached per statement, and targetTable and firstSelect no longer project the WITH list. Skipped: the statement is tokenized once per request, and that cannot be avoided without a cross-request cache.
Sequential catalog awaits: fixed. Catalog lookups for all derived tables run in one task group and are shared across references. allColumnsInScope still reads the catalog itself.
Column list named like a body keyword: fixed. INSERT INTO t (values, is no longer a statement body.
DerivedTable.columns: fixed. Removed; tests call projection.columns(expanding:).
Comments that restate the code: fixed. Removed the MARK lines and the comments that only described the next line.

Checks: the autocomplete sources and tests compile with Xcode 26.3 in a scratch build, and all 441 tests pass. The app target was not built and SwiftLint was not run.

To take it: git fetch https://github.com/J2TeamNNL/TablePro review-fix/3713 && git cherry-pick 559b114c120936c194c3736b4681556178862916

datlechin added 2 commits October 21, 2026 01:28

feat(editor): complete columns of every CTE shape and ORDER BY aliases

8dfe2ab

docs(changelog): note CTE column completion and the SET target fix

78d8f97

datlechin mentioned this pull request Oct 20, 2026

docs(features): autocomplete reads every CTE shape and ORDER BY aliases TableProApp/docs#71

Merged

J2TeamNNL reviewed Oct 21, 2026

View reviewed changes

datlechin merged commit 91653dd into main Oct 21, 2026
9 checks passed

datlechin deleted the feat/3608-cte-completion branch October 21, 2026 11:14

Navigation Menu

Repository navigation

feat(editor): complete columns of every CTE shape and ORDER BY aliases - #3713

feat(editor): complete columns of every CTE shape and ORDER BY aliases#3713
datlechin merged 3 commits into
mainfrom
feat/3608-cte-completion

datlechin commented Oct 20, 2026 •

edited

Loading

Uh oh!

J2TeamNNL left a comment

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

datlechin commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

J2TeamNNL left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

J2TeamNNL commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datlechin commented Oct 20, 2026 •

edited

Loading