Skip to content

[spark] Pass the IOManager to the data evolution partial-column writer - #10919

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/data-evolution-write-io-manager
Open

LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/data-evolution-write-io-manager

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #10918

DataEvolutionTableDataWrite writes the updated columns of a data evolution table for MERGE INTO and UPDATE. It created an IOManager but never passed it to the write, so with write-buffer-for-append = true the first spill of the append write buffer failed with a NullPointerException in ExternalBuffer.spill.

The write now gets the IOManager through withIOManager, like PaimonDataWrite and PaimonV2DataWriter.

Tests

RowTrackingTestBase "Data Evolution: merge into with a spilling append write buffer": a MERGE INTO on a data evolution table with write-buffer-for-append and a small write-buffer-size, so the buffer spills, then checks every updated row against its id. On master it fails with the NullPointerException.

API and Format

no

Documentation

no

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Spark MERGE INTO on a data evolution table fails with write-buffer-for-append

1 participant