add minor compaction for Overlord-based compaction supervisors using the MSQ engine by cecemei · Pull Request #19059 · apache/druid

cecemei · 2026-02-26T03:06:50Z

Description

This PR introduces minor compaction support in msq engine to Apache Druid, enabling compaction operations on only uncompacted segments within an interval while upgrading already-compacted segments for consistency.

Key Changes

CompactionMode Enum Extension

Added ALL_SEGMENTS mode (full compaction of all segments)
Added UNCOMPACTED_SEGMENTS_ONLY mode (compacts only new segments, upgrades existing ones)

UncompactedInputSpec
A new CompactionInputSpec implementation specifically for minor compaction:

Specifies uncompacted segments to compact within an interval
Required (non-nullable) interval and uncompactedSegments fields
Type: "uncompacted"
Replaces nullable field handling in CompactionIntervalSpec, making the intent of minor compaction explicit

SegmentUpgradeAction
A new task action that updates segment metadata without rewriting data, modifying partition numbers and shard specifications of already-compacted segments.
ShardSpec Interface Enhancements
Extended with mutation methods including withPartitionNum(), withCorePartitions(), and isNumChunkSupported() checks.
Policy-Level Control
Introduced minUncompactedBytesPercentForFullCompaction threshold in MostFragmentedIntervalFirstPolicy determining when incremental versus full compaction applies based on uncompacted-to-total segment byte ratios.
Builder Pattern Implementation
Replaced verbose constructors in UserCompactionTaskQueryTuningConfig with a builder pattern for improved readability.

Release Notes

This feature enables compacting newly ingested segments while preserving already-compacted data, with configuration through ratio thresholds in compaction policies. Support is currently limited to Overlord-based compaction supervisors using the MSQ engine.

This PR has:

server/src/test/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinatorTest.java

+      final DataSegment segment = new DataSegment(
+          "foo",
+          Intervals.of("2023-01-0" + i + "/2023-01-0" + (i + 1)),
+          "2023-01-0" + i,
+          ImmutableMap.of("path", "a-" + i),
+          ImmutableList.of("dim1"),
+          ImmutableList.of("m1"),
+          new DimensionRangeShardSpec(List.of("dim1"), null, null, i - 1, 8),
+          9,
+          100
+      );


capistrant

This is awesome. left some minor comments

server/src/main/java/org/apache/druid/server/compaction/CompactionMode.java

server/src/main/java/org/apache/druid/server/compaction/MostFragmentedIntervalFirstPolicy.java

server/src/main/java/org/apache/druid/server/compaction/CompactionCandidate.java

capistrant · 2026-02-26T17:58:54Z

indexing-service/src/main/java/org/apache/druid/indexing/compact/CompactionJobQueue.java

@@ -282,18 +283,20 @@ private boolean startJobIfPendingAndReady(
    }

    // Check if the job is already running, completed or skipped


nit: this comment might need a refresh to reflect how the status handling has changed versus the former code. I was confused about why PENDING and COMPLETE resulted in a throw until I reviewed deriveCompactionStatus

there's no logic change here, just made the code more defensive. updated the comment here. ihmo, CompactionStatus is not the best class to use here.

actually i guess there's some logic change, the policy check has been moved to CompactionConfigBasedJobTemplate since creating the job needs to know the compaction mode (which is decided by the policy), but still the computeCompactionStatus only returns running/skipped/pending state.

capistrant · 2026-02-26T18:15:59Z

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java

should we update the javadoc to call out how this method now offers special handling for incremental compaction that includes upgrading already compacted segments?

...ing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentUpgradeAction.java

...e/src/test/java/org/apache/druid/indexing/common/actions/MarkSegmentToUpgradeActionTest.java

processing/src/main/java/org/apache/druid/timeline/partition/LinearShardSpec.java

server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java

kfaraz

Leaving a partial review for the supporting changes.
Will review the core changes in CompactionTask and IndexerSQLMetadataStorageCoordinator shortly.

server/src/main/java/org/apache/druid/server/compaction/CompactionMode.java

...-tests/src/test/java/org/apache/druid/testing/embedded/compact/CompactionSupervisorTest.java

server/src/main/java/org/apache/druid/server/compaction/MostFragmentedIntervalFirstPolicy.java

kfaraz · 2026-02-27T11:54:19Z

server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java

+      case FULL_COMPACTION:
+        clientCompactionIntervalSpec = new ClientCompactionIntervalSpec(entry.getCompactionInterval(), null, null);
+        break;
+      case INCREMENTAL_COMPACTION:


As discussed earlier in the old PR, we should avoid supporting incremental compaction in the Coordinator-based CompactSegments duty to drive adoption of the Overlord-based compaction supervisors.
This will allow us to deprecate and eventually remove Coordinator-based compaction duty.

this method is used by both coordinator and overlord, coordinator would always use ALL_SEGMENTS compaction mode in line 266.

...ervice/src/main/java/org/apache/druid/indexing/compact/CompactionConfigBasedJobTemplate.java

kfaraz · 2026-02-27T12:18:18Z

...ervice/src/main/java/org/apache/druid/indexing/compact/CompactionConfigBasedJobTemplate.java

+          finalCandidate = CompactionCandidate.from(candidate.getUncompactedSegments(), null)
+                                              .withCurrentStatus(candidate.getCurrentStatus());


Why is this needed?

incremental-compaction-mode

682dd8b

github-actions bot added Area - Batch Ingestion Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Feb 26, 2026

github-advanced-security bot found potential problems Feb 26, 2026

View reviewed changes

cecemei requested a review from kfaraz February 26, 2026 03:37

cecemei marked this pull request as ready for review February 26, 2026 03:37

cecemei mentioned this pull request Feb 26, 2026

add incremental compaction support #18996

Draft

10 tasks

capistrant reviewed Feb 26, 2026

View reviewed changes

style

86ee6b8

cecemei added the Release Notes label Feb 27, 2026

pending

bdc28ac

kfaraz reviewed Feb 27, 2026

View reviewed changes

cecemei added 5 commits February 27, 2026 17:08

format

9a09e43

Merge branch 'master' into compact2.2

14294a2

build

3a0bd79

UncompactedInputSpec

3eac326

test

ba090ba

cecemei changed the title ~~add incremental compaction mode 2.2~~ add minor compaction for Overlord-based compaction supervisors using the MSQ engine Feb 28, 2026

cecemei added 5 commits February 27, 2026 18:43

uncompacted

64473a7

format

407e328

format2

df25187

format

0134e15

fix

c57b7f6

		@@ -282,18 +283,20 @@ private boolean startJobIfPendingAndReady(
		}

		// Check if the job is already running, completed or skipped

		finalCandidate = CompactionCandidate.from(candidate.getUncompactedSegments(), null)
		.withCurrentStatus(candidate.getCurrentStatus());

Conversation

cecemei commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Check notice

Uh oh!

capistrant left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

capistrant Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

cecemei Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

cecemei Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

capistrant Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kfaraz Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cecemei Feb 28, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kfaraz Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cecemei commented Feb 26, 2026 •

edited

Loading

kfaraz Feb 27, 2026 •

edited

Loading