[Flink] Add sourceScalingMaxParallelism metric for auto-scaling systems #7117
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
This PR adds a new metric
sourceScalingMaxParallelismto the Flink Source Enumerator. This metric provides a recommended upper bound of parallelism for auto-scaling systems to optimize resource allocation.Motivation
Auto-scaling systems need to understand the optimal parallelism for Paimon sources to:
The metric value:
Note: This is a recommendation, not a hard limit - users can still configure higher parallelism manually if needed.
Tests
Added unit tests in
FileStoreSourceMetricsTest.java:continuousFileStoreFixBucketEnumeratorMetricsTest()- Verifies metric equals bucket number for fixed bucket tablescontinuousFileStoreDynBucketEnumeratorMetricsTest()- Verifies metric equals current parallelism for dynamic bucket tablesAlso added
TestingMetricUtils.getGauge()helper method for testing Gauge metrics.API and Format
No API or storage format changes. This only adds a new metric.
Documentation
Updated
docs/content/maintenance/metrics.mdwith the new metric description.