Skip to content

[Pipe] Support region-level realtime downgrading - #18441

Open
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:pipe-region-level-downgrading
Open

[Pipe] Support region-level realtime downgrading#18441
Caideyipi wants to merge 2 commits into
apache:masterfrom
Caideyipi:pipe-region-level-downgrading

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Description

Per-Pipe region-level downgrade ordering

  • Add source.realtime.region-level-downgrading and its legacy extractor.* alias, disabled by default. Each Hybrid source reads the option from its own Pipe parameters.
  • When enabled, serialize downgraded TsFiles at DataRegion level. A later TsFile cannot pass until the previous TsFile and all tablets generated from it have been committed.
  • Leave downgrade only after all downgraded TsFiles complete, and stop forwarding realtime tablets through USING_BOTH while region-level downgrade is active. This preserves ordering for equal-timestamp data with serialize-by-region and realtime-first=false.

Bounded latest-TsFile tail

  • Retain a bounded realtime tail only for the latest TsFile so flushes cannot continuously chase and prolong downgrade.
  • Promote the previous tail to TsFile transfer when a newer epoch arrives, while preserving events queued before downgrade.

TsFile lifecycle

  • Track generated-tablet completion for both processor conversion paths.
  • Keep the TsFile progress index until the final generated tablet commits.
  • Restrict raw-tablet commit hooks to progress-reporting or TsFile-derived events.

Verification

  • DataNode test compilation.
  • Node Commons and DataNode compilation with both English and Chinese locales.
  • DataNode and Node Commons Spotless checks.
  • PipeRealtimeExtractTest: 10 tests passed, including enabled/disabled Pipe isolation.
  • git diff --check.

This PR has:

  • been self-reviewed.
    • concurrent read and write
  • added comments explaining the intent of non-obvious logic.
  • added or updated unit tests to cover the new code paths.

Key changed/added classes (or packages if there are too many classes) in this PR
  • PipeRealtimeDataRegionHybridSource
  • PipeTsFileInsertionEvent
  • PipeRawTabletInsertionEvent
  • PipeEventCollector
  • PipeSourceConstant
  • PipeRealtimeExtractTest

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant