Skip to content

[fix](compaction) Keep cumulative compaction input mergeable when max-score trim strands a single non-overlapping rowset#65470

Draft
Yukang-Lian wants to merge 2 commits into
apache:masterfrom
Yukang-Lian:codex/cumulative-compaction-max-score-trim
Draft

[fix](compaction) Keep cumulative compaction input mergeable when max-score trim strands a single non-overlapping rowset#65470
Yukang-Lian wants to merge 2 commits into
apache:masterfrom
Yukang-Lian:codex/cumulative-compaction-max-score-trim

Conversation

@Yukang-Lian

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: None

Related PR: #59268, #64954

Problem Summary:

Problem

Under memory pressure, cloud cumulative compaction reduces the maximum compaction score. A production-shaped MOW tablet can then contain a zero-segment, non-overlapping compaction output at the cumulative point, followed by large overlapping singleton rowsets whose individual scores exceed that reduced maximum.

The max-score back trim introduced by #59268 can remove every successor from the original selected set while retaining the non-overlapping head. The caller rejects that single rowset with CUMULATIVE_NO_SUITABLE_VERSION. Because no merge runs, the previous memory-limit status remains in effect, the same reduced maximum is reused, and the tablet can repeatedly hit the same no-suitable-version result while new versions accumulate.

The cloud scheduling loop also suppressed this status outside debug logging, which made the retry loop difficult to observe.

Root cause

The deferred max-score trim deliberately keeps at least one rowset, but it did not verify that the remaining input was mergeable. When the head was non-overlapping and its direct successor alone exceeded the remaining score budget, trimming could strand the head as an invalid single-rowset input.

Fix

  • In both cloud and local size-based cumulative compaction policies, remember the most recently removed rowset while trimming from the back.
  • If trimming leaves exactly one non-overlapping rowset, restore that direct successor and accept a one-off max-score overshoot. Restore both score and size accounting, and do not trim again.
  • Keep the cloud empty-rowset skip_trim behavior unchanged.
  • Leave cumulative-point and meta-service job handling unchanged. In the production layout, merging the head with its large successor produces an output large enough for the existing promotion logic to advance the cumulative point naturally.
  • Emit a rate-limited INFO log for cloud CUMULATIVE_NO_SUITABLE_VERSION submission failures, while preserving the existing debug-warning and BE_NO_SUITABLE_VERSION behavior.

This is a follow-up to #59268 and is independent of #64954.

Testing

  • sh run-be-ut.sh --run --filter='*CumulativeCompaction*': 74/74 passed on the final branch.
  • Covered a production-shaped non-overlapping head followed by oversized overlapping rowsets and empty/small rowsets; selection returns the head plus its direct successor and the merged output advances the cumulative point from 13 to 58.
  • Covered the negative branch where trimming really removes successors but the remaining head is overlapping, so no restoration occurs.
  • Existing cloud empty-rowset skip_trim coverage remains green.
  • Temporarily overlaid the two policy changes from [fix](compaction) Avoid repeatedly compacting large cumulative rowset #64954 (removing the promotion-size early return and guarding level trim with size() > 1), then reran the same suite: 74/74 passed. The overlay was removed and is not part of this PR.
  • build-support/clang-format.sh, build-support/check-format.sh, and git diff --check passed.

Release note

Prevent cumulative compaction from repeatedly selecting an unmergeable single non-overlapping rowset after max-score trimming, and add rate-limited visibility for repeated no-suitable-version retries.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes. Max-score trimming restores one successor only when needed to keep cumulative compaction input mergeable; repeated cloud no-suitable-version retries are now rate-limited at INFO level.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: apache#59268

Problem Summary: Max-score back trimming can leave one non-overlapping cumulative rowset, which callers cannot compact by itself and may repeatedly reject. Preserve mergeability by restoring the direct successor that was last removed, accepting a one-off score overshoot. Apply the same rule to cloud and local size-based policies and cover the production-shaped layout, the overlapping-singleton control case, the cloud skip-trim path, and natural cumulative-point promotion.

### Release note

Prevent cumulative compaction from repeatedly selecting an unmergeable single non-overlapping rowset after max-score trimming.

### Check List (For Author)

- Test: Unit Test
    - sh run-be-ut.sh --run --filter='*CumulativeCompaction*'
- Behavior changed: Yes, max-score trimming restores one successor only when required to keep the selected input mergeable.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#59268

Problem Summary: Cloud compaction scheduling suppresses cumulative no-suitable-version statuses, making repeated prepare failures invisible outside debug logs. Emit a rate-limited INFO log with tablet ID and status while retaining warning behavior under debug logging and preserving the existing base no-suitable-version handling.

### Release note

Add rate-limited visibility for repeated cloud cumulative compaction no-suitable-version retries.

### Check List (For Author)

- Test: Unit Test
    - sh run-be-ut.sh --run --filter='*CumulativeCompaction*'

- Behavior changed: Yes, cumulative no-suitable-version submissions now emit a rate-limited INFO log.

- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants