Repository navigation
Conversation
roseduan
force-pushed
the
refactor_ao_parallel_scan
branch
7 times, most recently
from
November 2, 2026 07:24
3b34e37 to
8319276
Compare
roseduan
force-pushed
the
refactor_ao_parallel_scan
branch
4 times, most recently
from
November 4, 2026 07:50
7160aea to
4a7cf3f
Compare
roseduan
changed the title
roseduan
force-pushed
the
refactor_ao_parallel_scan
branch
from
November 4, 2026 10:21
177d265 to
a476b65
Compare
roseduan
marked this pull request as ready for review
November 4, 2026 10:21
There was a problem hiding this comment.
🟡 Changes recommended
The header-only skip mishandles pre-4.0 AOCS row counts and first-row tracking, potentially skipping or duplicating rows and misaligning columns.
1 open issue
What changed in this PR
This PR enables parallel scans to share work within append-optimized row (AO) and column (AOCS) segment files, improving parallelism for tables loaded into a single segment file.
Changes:
- Splits segment files into row chunks claimed through a shared counter, with format-specific reader positioning.
- Sizes planner worker counts by table size rather than segment-file count.
- Adds regression and concurrency coverage, documents the design, and adjusts existing tests.
| File | Description |
|---|---|
src/test/regress/sql/zlib.sql |
Keeps the spill fault test on a nonparallel path. |
src/test/regress/sql/cbdb_parallel.sql |
Adds RLE column-scan coverage and sets join-test parallelism. |
src/test/regress/output/uao_dml/ao_parallel_scan.source |
Adds expected chunked-scan results and plans. |
src/test/regress/input/uao_dml/ao_parallel_scan.source |
Tests chunk boundaries, positioning, visibility, and planning. |
src/test/regress/greenplum_schedule |
Schedules AO row and column scan tests. |
src/test/regress/expected/zlib.out |
Updates expected output for the nonparallel spill test. |
src/test/regress/expected/cbdb_parallel.out |
Records RLE scan results and join-test changes. |
src/test/isolation2/output/uao/parallel_scan_concurrency.source |
Adds expected concurrency and recovery results. |
src/test/isolation2/isolation2_schedule |
Schedules parallel-scan concurrency tests. |
src/test/isolation2/input/uao/parallel_scan_concurrency.source |
Tests concurrent writes, vacuum, snapshots, and worker errors. |
src/include/cdb/cdbappendonlyam.h |
Defines shared chunk sizing and AO scan state. |
src/include/cdb/cdbaocsam.h |
Declares AOCS chunk state and counting interface. |
src/backend/optimizer/path/allpaths.c |
Uses size-based worker selection for AO sequential scans. |
src/backend/access/table/tableam.c |
Initializes shared scans with chunk counts. |
src/backend/access/appendonly/README.parallel_scan.md |
Documents chunk allocation, reading, and limitations. |
src/backend/access/appendonly/appendonlyam.c |
Implements AO chunk allocation and block ownership. |
src/backend/access/aocs/aocsam.c |
Implements AOCS chunk allocation and per-column positioning. |
Review effort: Balanced
roseduan
force-pushed
the
refactor_ao_parallel_scan
branch
from
November 5, 2026 02:46
a476b65 to
9fe1b61
Compare
There was a problem hiding this comment.
🟡 Changes recommended
Block-positioning loops lack interrupt checks, delaying cancellation and timeouts, while regression helpers round numeric averages and may fail to exercise the actual chunk boundaries.
3 open issues
1 resolved since last review
Review effort: Balanced
A parallel scan of an AO/AOCS table used to give each worker whole segment files, and the planner capped the number of workers at pg_appendonly.segfilecount. A table loaded by one transaction usually has a single segment file per segment, so it was scanned by one process. Now each segment file is split into chunks of rows, and the workers claim chunks from a shared counter, as heap does with blocks. Every process computes the same chunk layout from pg_aoseg using the scan's snapshot, so the shared state holds only the number of chunks and the counter. A scan has about 2048 chunks, bounded in bytes by the table's average row size. A block belongs to the chunk that holds its first row. To reach its chunk, a process reads only block headers and skips other chunks' blocks without decompressing them. AOCS positions each column file separately. The planner now sizes AO/AOCS workers by table size, as for heap, instead of by segment file count.
roseduan
force-pushed
the
refactor_ao_parallel_scan
branch
from
November 5, 2026 06:24
9fe1b61 to
5e23560
Compare
There was a problem hiding this comment.
🔵 Needs a closer look
Storage-scan ownership and snapshot-sensitive column positioning changes require database-backed concurrency and isolation validation that this review environment cannot run.
No open issues 🎉
3 resolved since last review
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #ISSUE_Number
What does this PR do?
A parallel scan of an AO/AOCS table used to give each worker whole
segment files, and the planner capped the number of workers at
pg_appendonly.segfilecount. A table loaded by one transaction usually
has a single segment file per segment, so it was scanned by one
process.
Now each segment file is split into chunks of rows, and the workers
claim chunks from a shared counter, as heap does with blocks. Every
process computes the same chunk layout from pg_aoseg using the scan's
snapshot, so the shared state holds only the number of chunks and the
counter. A scan has about 2048 chunks, bounded in bytes by the
table's average row size.
A block belongs to the chunk that holds its first row. To reach its
chunk, a process reads only block headers and skips other chunks'
blocks without decompressing them. AOCS positions each column file
separately.
The planner now sizes AO/AOCS workers by table size, as for heap,
instead of by segment file count.
Type of Change
Breaking Changes
Test Plan
make installcheckmake -C src/test installcheck-cbdb-parallelImpact
Performance:
User-facing changes:
Dependencies:
Checklist
Additional Context
CI Skip Instructions