[lake/paimon] Support clean and legacy Paimon lake table schemas - #3982
Open
fhan688 wants to merge 5 commits into
Open
[lake/paimon] Support clean and legacy Paimon lake table schemas#3982fhan688 wants to merge 5 commits into
fhan688 wants to merge 5 commits into
Conversation
fhan688
force-pushed
the
Support-clean-and-legacy-Paimon-lake-table-schemas
branch
from
August 14, 2026 02:33
1799d67 to
8d9f5df
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR implements the Paimon portion of FIP-27 by supporting two physical Paimon lake table layouts: clean (new tables with only user columns) and legacy (existing tables that still contain __bucket, __offset, __timestamp as trailing physical columns). It introduces schema-based layout detection and threads the detected layout through writers/readers so existing tables remain readable/writable without migration.
Changes:
- Add centralized layout detection (
PaimonSystemColumns.detectLayout) and use it to preserve legacy physical layout when re-enabling tiering / checking schema compatibility. - Stop appending Paimon system columns for newly created tables and make schema evolution (add-column positioning) layout-aware.
- Thread layout through tiering writers and record readers so system columns are written/read only for legacy tables; clean tables use sentinel offset/timestamp values on read.
Reviewed changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| fluss-lake/fluss-lake-paimon/src/test/java/org/apache/fluss/lake/paimon/tiering/FlussRecordAsPaimonRowTest.java | Updates writer tests to pass explicit legacy layout (but does not yet add clean-layout coverage). |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/utils/PaimonTableValidation.java | Detects legacy layout and enriches new schemas with system columns for compatibility checks; timestamp-precision relaxation logic updated. |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/utils/PaimonSystemColumns.java | New single source of truth for system columns, layout enum, and schema-based layout detection with validation. |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/utils/PaimonRowAsFlussRow.java | Makes trailing system-column trimming explicit via a layout-derived count (avoids hard-coded global trimming). |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/utils/PaimonConversions.java | Stops adding system columns on create and makes schema-change generation layout-aware for add-column positioning. |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/tiering/RecordWriter.java | Threads LakeLayout into row conversion used by tiering writers. |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/tiering/PaimonLakeWriter.java | Detects target table layout once and passes it through writer implementations. |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/tiering/mergetree/MergeTreeWriter.java | Adds LakeLayout plumbing through constructors to maintain correct write layout. |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/tiering/FlussRecordAsPaimonRow.java | Makes row conversion layout-aware so system columns are emitted only for legacy tables. |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/tiering/append/AppendOnlyWriter.java | Threads LakeLayout into Arrow-batch helper creation and stores layout on the writer. |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/tiering/append/AppendOnlyArrowBatchHelper.java | Writes Arrow batches directly for clean tables; enriches with system vectors only for legacy tables. |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/source/PaimonRecordReader.java | Detects layout for reads; projects/reads system columns only for legacy tables and emits sentinel values for clean tables. |
| fluss-lake/fluss-lake-paimon/src/main/java/org/apache/fluss/lake/paimon/PaimonLakeCatalog.java | Moves system-column definition to PaimonSystemColumns and makes alter-table schema changes layout-aware. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+107
to
+111
| // Only legacy tables carry a trailing __timestamp system column. Clean tables have no | ||
| // system columns, so there is no precision to relax and we compare them directly. | ||
| if (existingFields.isEmpty()) { | ||
| return equalPhysicalSchema(existingSchema, newSchema); | ||
| } |
Comment on lines
+46
to
47
| public FlussRecordAsPaimonRow(int bucket, RowType tableTowType, LakeLayout lakeLayout) { | ||
| super(tableTowType); |
Comment on lines
78
to
80
| FlussRecordAsPaimonRow flussRecordAsPaimonRow = | ||
| new FlussRecordAsPaimonRow(tableBucket, tableRowType); | ||
| new FlussRecordAsPaimonRow(tableBucket, tableRowType, LakeLayout.LEGACY); | ||
| long logOffset = 0; |
Comment on lines
+115
to
119
| // Legacy tables carry __offset/__timestamp, which the iterator needs to recover the log | ||
| // offset and timestamp of each record; append them to the projection. | ||
| int offsetFieldPos = paimonFullRowType.getFieldIndex(OFFSET_COLUMN_NAME); | ||
| int timestampFieldPos = paimonFullRowType.getFieldIndex(TIMESTAMP_COLUMN_NAME); | ||
|
|
luoyuxia
reviewed
Aug 15, 2026
| * system columns, or carries them with an unexpected type, is neither a clean nor a valid legacy | ||
| * table and is rejected with a clear error. | ||
| */ | ||
| public class PaimonSystemColumns { |
Contributor
There was a problem hiding this comment.
I feel like it's a little complex to introduce a new class for it. I think we can just use a method like I did in https://github.com/apache/fluss/pull/2493/changes#diff-e125d6b59322af48ed676cbd230c662ea8e002be31fedfce8e0a66873bdfc23aR156
| addColumn.getComment(), | ||
| SchemaChange.Move.before( | ||
| addColumn.getName(), firstSystemColumnName))); | ||
| if (lakeLayout == LakeLayout.LEGACY) { |
Contributor
There was a problem hiding this comment.
also add a IT to verify that we can still add column for table with legacy system column
| @@ -46,6 +48,14 @@ | |||
| /** Record reader for paimon table. */ | |||
| public class PaimonRecordReader implements RecordReader { | |||
Contributor
There was a problem hiding this comment.
I feel like the changes is a little of complex, you can refer to https://github.com/apache/fluss/pull/2493/changes#diff-e125d6b59322af48ed676cbd230c662ea8e002be31fedfce8e0a66873bdfc23a to simplify the code
Contributor
There was a problem hiding this comment.
Also add a test to verify the tiering & union read still work for legacy table. See https://github.com/apache/fluss/pull/2493/changes#diff-9b69c1d14758f0ca8bc71ae7eb8186be3741545178e86226d312c76264f6a0caR418
Contributor
There was a problem hiding this comment.
Also add a test in this IT to cover the case:
- a table with legacy system column
- disable datalake
- enable datalake again
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #3902
Sub-task of the FIP-27 umbrella (#2411): Remove Mandatory System Columns From Fluss Lake Tables.
Today every Paimon lake table Fluss creates is forced to carry three mandatory system columns (
__bucket,__offset,__timestamp) as its last physical columns. They pollute the schema users see from Paimon and other engines, and impose a system-metadata-based physical layout.This PR implements the Paimon part of FIP-27: newly created Paimon lake tables use a clean physical schema containing only user-defined columns, while existing legacy tables that still carry the three system columns remain fully readable and writable without any schema migration. Both layouts are supported across create, tiering writers, readers, projections, schema evolution, and re-enabling tiering.
Brief change log
PaimonSystemColumnswith aLakeLayoutenum anddetectLayout(RowType). Detection is purely schema-based — no new table property or metadata, so existing tables are never migrated:CLEAN;LEGACY;InvalidTableException.PaimonConversions.toPaimonSchemano longer appends the system columns; new tables are clean. The user-column name-conflict check against system names is kept.toPaimonSchemaChangesnow takes the target layout — for a legacy table a new business column is still inserted before the first system column; for a clean table it is appended normally.PaimonTableValidation.isPaimonSchemaCompatibledetects the existing layout and, for a legacy table, enriches the freshly generated clean schema with the trailing system columns before comparison, so disabling and re-enabling tiering preserves the existing physical layout. The__timestampprecision-6→3 relaxation is guarded so it only applies to legacy tables (a clean table has no__timestamp).FlussRecordAsPaimonRowemits the three system values only for legacy tables; for clean tables the business-field count equals the full row and no system fields are written. Layout is threaded throughPaimonLakeWriter→RecordWriter→AppendOnlyWriter/MergeTreeWriter.AppendOnlyArrowBatchHelperwrites the originalVectorSchemaRootdirectly; for legacy tables it keeps enriching the batch with the__bucket/__offset/__timestampvectors.PaimonRecordReaderprojects and reads__offset/__timestamponly for legacy tables. For clean tables it emits a sentinel-1log offset / timestamp, consistent with the existingLakeRecordRecordEmittercontract (logOffset() >= 0marks the incremental phase) and the existingUNKNOWN_OFFSET = -1convention.PaimonRowAsFlussRowno longer assumes a fixed number of trailing system columns globally; the trailing-system-column count is explicit, which also fixes latent miscounts for nested/projected rows (e.g. the lookup path, where Paimon already projects system columns away).Tests
FlussRecordAsPaimonRowTestto the layout-aware writer (existing cases are legacy-layout).FULLstartup; disable+re-enable preserves layout; row and Arrow-batch writer paths; reader projections; schema evolution; and rejection of a partial/type-incompatible legacy layout. Cases should span log and primary-key tables, and partitioned and non-partitioned tables.mvn clean verifyhas not been run locally (JDK 11 build environment not available on my machine; only JDK 8). Verified signature consistency, imports, and static review; full compile + IT run is pending on a JDK 11 environment.API and Format
No public API change. This changes the physical schema of newly created Paimon lake tables (clean layout by default). Existing tables are not migrated and keep their current physical format. Compatibility / rolling-upgrade requirements are covered by the umbrella #2411 and documented in #3905:
FULLstartup mode must not read newly created clean tables.Fluss cluster.
Documentation
Feature behavior (clean vs. legacy layouts, detection, and the rolling-upgrade/compatibility matrix) is documented separately under #3905. No standalone doc change in this PR.