Skip to content

Conversation

@wzhero1
Copy link
Contributor

@wzhero1 wzhero1 commented Jan 26, 2026

Purpose

This PR adds a new metric sourceScalingMaxParallelism to the Flink Source Enumerator. This metric provides a recommended upper bound of parallelism for auto-scaling systems to optimize resource allocation.

Motivation

Auto-scaling systems need to understand the optimal parallelism for Paimon sources to:

  1. Avoid over-provisioning resources for fixed-bucket tables (where parallelism shouldn't exceed bucket count)
  2. Make informed scaling decisions for dynamic-bucket tables

The metric value:

  • For fixed bucket tables: equals the bucket number
  • For dynamic bucket tables (bucket = -1): equals the current parallelism

Note: This is a recommendation, not a hard limit - users can still configure higher parallelism manually if needed.

Tests

Added unit tests in FileStoreSourceMetricsTest.java:

  • continuousFileStoreFixBucketEnumeratorMetricsTest() - Verifies metric equals bucket number for fixed bucket tables
  • continuousFileStoreDynBucketEnumeratorMetricsTest() - Verifies metric equals current parallelism for dynamic bucket tables

Also added TestingMetricUtils.getGauge() helper method for testing Gauge metrics.

API and Format

No API or storage format changes. This only adds a new metric.

Documentation

Updated docs/content/maintenance/metrics.md with the new metric description.

@wzhero1 wzhero1 force-pushed the feat/paimon-flink-available-max-parallelism-metrics branch from 117dbc1 to affa652 Compare January 26, 2026 07:08
@wzhero1 wzhero1 force-pushed the feat/paimon-flink-available-max-parallelism-metrics branch from affa652 to 438e986 Compare January 26, 2026 07:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant