Add undo recovery support for aggregation tables #2545
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
platinumhamburg
force-pushed
the
undoinfra
branch
3 times, most recently
from
edd4525 to
a75f04f
Compare
This PR introduces the undo recovery mechanism for Flink sink writer to handle failure recovery scenarios with aggregation merge engine tables. Key components: - ByteArrayWrapper: Utility class for using byte arrays as map keys - UndoComputer: Computes undo operations by comparing checkpoint state with current log records, supporting both full row and partial update modes - UndoRecoveryExecutor: Executes undo operations using UpsertWriter - UndoRecoveryCoordinator: Coordinates the recovery process across buckets, managing log scanning, undo computation, and execution - BucketRecoveryContext: Holds per-bucket recovery state The undo recovery works by: 1. Scanning log records from checkpoint offset to current end offset 2. Computing inverse operations for uncommitted records 3. Executing undo operations to restore table state For partial update mode, INSERT records require full row deletion (not partial column deletion), which is handled by using a separate delete writer with null target columns.
platinumhamburg
force-pushed
the
undoinfra
branch
from
a75f04f to
ba699f0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the undo recovery mechanism for Flink sink writer to handle failure recovery scenarios with aggregation merge engine tables.
Key components:
The undo recovery works by:
For partial update mode, INSERT records require full row deletion (not partial column deletion), which is handled by using a separate delete writer with null target columns.
Purpose
Linked issue: close #2544 2544
Brief change log
Tests
API and Format
Documentation