feat(import): import rows as Update, Insert or Update, or Insert, Skip Existing by datlechin · Pull Request #3743 · TableProApp/TablePro

datlechin · 2026-10-21T05:12:24Z

Summary

Row import (CSV, TSV, JSON, NDJSON, XLSX) into an existing table gets Import as: Insert, Update, Insert or Update, Insert, Skip Existing.
Match on offers the primary key and each unique index over plain columns (no partial or expression index). The sheet asks for a field on every key column, and for the Update modes a column outside the key.
Update writes only the mapped columns outside the key; the rest of the row keeps its values.
Per run of rows, one SELECT <pos>, COUNT(*) ... UNION ALL ... asks the server how many rows hold each key, so the database's own casting and collation decide a match. Count 1: UPDATE ... SET <mapped> WHERE <key>. Count 0: the existing grouped INSERT, or skipped for Update. Count > 1: the row is refused, and Skip and Continue records it against its line.
Writes keep file order. A key the file repeats (compared after folding case, accents, width, spaces and number spelling) starts a new run, so it sees the earlier write.
Not the engines' upsert forms. Measured on PostgreSQL 17.11, MySQL 8.4.11 and SQLite 3.54: an upsert of an existing row that leaves out a NOT NULL column fails (23502, 1364, NOT NULL constraint failed), MySQL ON DUPLICATE KEY updated the row matched by another unique key, PostgreSQL refuses one key twice per statement, and none says which rows it inserted.
Delete existing rows before import with any mode but Insert stops the import before it writes, with a message saying why.
The summary lists Matched, Updated, Inserted, Skipped, and Failed under Skip and Continue. A plain Insert reports the rows it inserted.
No PluginKit change: the plugins still hand rows to the sink; the mode and key live in the sheet and ImportDataSinkAdapter.
Importing several files in one run (feat(import): import several files or a folder in one run #3745) stays Insert only. Editing one file of the run shows no Import as or Match on.

Test plan

ImportDataSinkAdapterRowWritingTests runs each mode against an in-memory SQLite: unmapped NOT NULL column untouched, repeated and equivalent keys (7/07, NOCASE a/A), file order under a self-referencing foreign key, 250 keys over three lookups, ambiguous key refused, Skip and Continue through RowImportRunner, clearing refused.
ImportRowWritingTests (key options, exact spelling, validation, summary), SQLStatementGeneratorImportTests (lookup and keyed UPDATE SQL), ImportSummaryTextTests.
RowImportUpdateUITests: imports id,name rows 2,Bebop and 9,Polka into the fixture's genre as Update; genre 2 changes, 9 stays out, the summary reads Updated 1 and Skipped 1.

Screenshots

Not in this PR

ClickHouse before lightweight updates refuses UPDATE ... SET, so Update and Insert or Update fail on the first matched row there. Data Compare sync writes the same statement. Gating it needs a PluginKit capability.
A unique index whose own collation differs from its column's (SQLite code COLLATE NOCASE in the index) is matched with the column's comparison.
Two spellings of one new key that only the column type equates (timestamptz offsets) in one batch fail with a unique violation in the stop modes. Skip and Continue writes row by row and handles them.
A NULL in the key matches no row, including on SQL Server and PostgreSQL NULLS NOT DISTINCT, where a unique key holds one NULL.

Fixes #2521

…p Existing

…te-upsert

J2TeamNNL

Automated code review: inline findings below.

J2TeamNNL · 2026-10-21T09:26:42Z

            throw PluginImportError.importFailed(
                String(format: String(localized: "Could not build an INSERT for %@"), targetTable)
            )
        }


Duplicate-key detection in writeMatching uses a client-side approximation (foldedKeyValue) of the server's key equality. With accent/punctuation-insensitive collations, numeric-vs-text keys ('1.0' vs '1'), or UUIDs with different hyphenation/casing, two rows the server treats as one key land in the same probe run, both get count 0, and Insert or Update inserts both (unique violation or double insert). NULL keys are not tracked.

J2TeamNNL · 2026-10-21T09:26:42Z

+        }
+        return lines
+    }
+}


.nothingToUpdate is raised for Insert or Update as well as Update. For Insert or Update, mapping only the key columns is valid (insert missing keys, leave existing rows alone), yet the sheet blocks it. The same file works under 'Insert, Skip Existing'.

(Anchored to nearest changed line; relates to about line 462.)

J2TeamNNL · 2026-10-21T09:26:42Z

+                .folding(options: [.caseInsensitive, .diacriticInsensitive, .widthInsensitive], locale: nil)
+            // A Double, not a Decimal: a REAL column holds 0.1 and 0.100000000000000001 as one value.
+            guard folded.contains(where: { $0.isASCII && $0.isNumber }),
+                  folded.allSatisfy({ numberCharacters.contains($0) }),


Matching modes plus 'Delete existing rows first' are rejected only at run time in deleteAllRowsFromTargetTable. validationMessage doesn't flag it, so Import stays enabled and fails after the progress dialog opens. Surface the conflict inline in the sheet.

J2TeamNNL · 2026-10-21T09:26:42Z

+            String(format: "%.2f", result.executionTime)
+        )
+        if let rowTally {
+            return (rowTally.summaryLines(failed: result.skippedStatements) + [seconds]).joined(separator: "\n")


Whenever rowTally is present the summary shows tally lines even for plain Insert, replacing 'N statements executed, M failed'. Failed: is skippedStatements (statements) next to row-based Inserted:, so a rejected 100-row INSERT shows 'Inserted: 900 / Failed: 1' with mismatched units. It also changes existing Insert-mode wording.

J2TeamNNL · 2026-10-21T09:26:42Z

+            try await insertGroup(groupValues, columns: columns, generator: generator)
        }
    }



Tally counters (updated, inserted) are incremented per statement with no handling for the import transaction rolling back. After a later error or cancel rolls back, state.rowTally still reports e.g. 500 Updated, so the UI shows rolled-back rows as written.

(Anchored to nearest changed line; relates to about line 274.)

J2TeamNNL · 2026-10-21T09:26:42Z

+    }
+
+    private func mappedRow(_ values: [String: PluginCellValue]) -> MappedRow {
+        let (columns, mappedValues) = mappedColumnsAndValues(values)


Matching mode costs one probe per run of up to 100 keys plus one UPDATE per matched row, all sequential. A 1M-row Update over a remote DB is ~1M round-trips; repeated keys split runs and shrink probes toward single rows. No batching and no speed warning.

J2TeamNNL · 2026-10-21T09:26:43Z

        return ParameterizedStatement(sql: sql, parameters: bindParameters)
    }

+    // The server compares, so its casting and collation decide a match, as they do for the UPDATE that follows.


keyMatchCountStatement and keyedUpdateStatement use plain col = ? and bypass the generator's per-database handling in rowCondition / RowMatchTerm (SQLite affinity, typed-column casts). On a SQLite column with no affinity, file key '5' (text) probes = '5' and misses stored integer 5, so Insert or Update inserts a duplicate instead of updating.

J2TeamNNL · 2026-10-21T09:26:43Z

-            let columns = try await DatabaseManager.shared.withMetadataDriver(scope: scope) { driver in
-                try await driver.fetchColumns(table: table)
-            }.map(\.name)
+            let (columns, indexes) = try await DatabaseManager.shared.withMetadataDriver(scope: scope) { driver in


fetchIndexes failures are swallowed with try?. For a table whose only unique key is a unique index, a transient metadata error leaves matchKeys empty and the sheet says 'no primary key or unique index', which is wrong and gives no hint it's a retryable load failure.

J2TeamNNL · 2026-10-21T09:26:43Z

+        let writable = mapped.filter { !$0.columns.isEmpty }
+
+        if rowWriting.mode.matchesExistingRows {
+            try await writeMatching(writable, generator: rowGenerator)


keyValues(of:) throws for the whole import when a single row lacks a key value (e.g. a JSON record missing id). One sparse row aborts the import with no row number, and earlier writes may already be applied. Skip or count it as failed like other row-level errors.

J2TeamNNL · 2026-10-21T09:31:21Z

Fixes for the review findings are in J2TeamNNL@f66cc99.

Fixed (6 of 9):

Duplicate-key detection: UUID spellings with different hyphens, braces or case now share one fold. See the skipped note below for the rest.
Key probe and keyed UPDATE on SQLite: both now use the generator's per-database key condition, so text '5' matches a stored integer 5 in a column with no affinity. Test added.
Update-only error: "nothing to update" now blocks Update only. Insert or Update with just the key columns is allowed. The old test pinned the old behavior and is changed.
Summary wording: tally lines show only for the matching modes. Plain Insert keeps "N statements executed, M failed". Test added.
Unreadable indexes: the sheet now says so and offers Try Again when matching has no key because the index read failed.
Row with no key value: all keys of a batch are checked before any row is written, so the batch writes nothing. Skip and Continue already records the row by its line. Test added.

Skipped (3 of 9):

Delete existing rows with a matching mode, shown inline: the option lives in each plugin's own settings, and the sheet cannot read it. Showing it inline needs a new PluginKit protocol member and changes to three plugins. The run still stops before any write, with a clear message.
Tally after rollback: the tally is read only after the import returns successfully. Any error or cancel throws, so a rolled-back run never reaches the summary.
Speed of matching mode: an UPDATE per matched row is needed because each row has its own values. Probes are already batched. Batching writes is a separate change.
Also, the rest of the duplicate-key finding (accent or punctuation-insensitive collations): a client fold cannot know the server collation. The key is a primary key or unique index, so a collision fails loudly with the server's error instead of writing a silent duplicate.

Not run: build, tests, swiftlint (no local Xcode; the files only pass swiftc -parse). The new message "Could not read this table's indexes, so rows cannot be matched. Try again." still needs python3 scripts/localization.py extracted --add after a build.

Cherry-pick: git fetch https://github.com/J2TeamNNL/TablePro review-fix/3743 && git cherry-pick f66cc996c10316e55d56abec998ca4f92734f070

…te-upsert # Conflicts: # apple/TablePro/Resources/Localizable.xcstrings

…te-upsert # Conflicts: # apple/TablePro/Core/Plugins/ImportDataSinkAdapter.swift # apple/TablePro/Core/Services/Export/ImportService.swift

datlechin added 2 commits October 21, 2026 12:12

feat(import): import rows as Update, Insert or Update, or Insert, Ski…

826d7b1

…p Existing

docs(changelog): note row import update modes

b427314

datlechin mentioned this pull request Oct 21, 2026

docs(features): document Import as and Match on for row imports TableProApp/docs#96

Merged

datlechin added 2 commits October 21, 2026 13:38

Merge branch 'main' into feat/2521-import-update-upsert

704eccc

Merge remote-tracking branch 'origin/main' into feat/2521-import-upda…

2c6e32f

…te-upsert

J2TeamNNL reviewed Oct 21, 2026

View reviewed changes

datlechin added 2 commits October 21, 2026 18:22

Merge remote-tracking branch 'origin/main' into feat/2521-import-upda…

51a7e3c

…te-upsert # Conflicts: # apple/TablePro/Resources/Localizable.xcstrings

Merge remote-tracking branch 'origin/main' into feat/2521-import-upda…

4a6d1e2

…te-upsert # Conflicts: # apple/TablePro/Core/Plugins/ImportDataSinkAdapter.swift # apple/TablePro/Core/Services/Export/ImportService.swift

datlechin merged commit b0efd9d into main Oct 21, 2026
8 checks passed

datlechin mentioned this pull request Oct 21, 2026

ClickHouse: Import as Update and Insert or Update fail on the first matched row #3765

Closed

datlechin deleted the feat/2521-import-update-upsert branch October 22, 2026 07:05

Navigation Menu

Repository navigation

feat(import): import rows as Update, Insert or Update, or Insert, Skip Existing - #3743

feat(import): import rows as Update, Insert or Update, or Insert, Skip Existing#3743
datlechin merged 6 commits into
mainfrom
feat/2521-import-update-upsert

datlechin commented Oct 21, 2026 •

edited

Loading

Uh oh!

J2TeamNNL left a comment

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL Oct 21, 2026

Uh oh!

J2TeamNNL commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

datlechin commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Screenshots

Not in this PR

Uh oh!

J2TeamNNL left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

J2TeamNNL commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datlechin commented Oct 21, 2026 •

edited

Loading