Skip to content

Pipe: Handle hybrid meta progress indexes - #18331

Merged
jt2594838 merged 5 commits into
apache:masterfrom
Caideyipi:fix/pipe-hybrid-meta-progress-index
Jul 31, 2026
Merged

jt2594838 merged 5 commits into
apache:masterfrom
Caideyipi:fix/pipe-hybrid-meta-progress-index

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Problem

IoTDBNonDataRegionSource assumed every non-minimum task progress was a MetaProgressIndex. Pipe task progress can also be merged into a HybridProgressIndex, and may be wrapped by a StateProgressIndex. This caused a ClassCastException while restoring a non-data-region source or calculating its remaining events during DataNode shutdown.

Changes

  • Extract the meta progress component from direct, hybrid, and state-wrapped progress indexes.
  • Use that component for source recovery and remaining-event accounting.
  • Conservatively fall back to snapshot recovery or full queue size when no meta component exists.
  • Add unit coverage for hybrid, state-wrapped hybrid, and hybrid-without-meta cases.

Verification

  • mvn spotless:apply -pl iotdb-core/node-commons
  • mvn test -pl iotdb-core/node-commons -Dtest=IoTDBNonDataRegionSourceTest

This PR has:

  • been self-reviewed.
  • added unit tests to cover new code paths.

Key changed/added classes
  • IoTDBNonDataRegionSource
  • IoTDBNonDataRegionSourceTest

…com/Caideyipi/iotdb into fix/pipe-hybrid-meta-progress-index

# Conflicts:
#	iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/source/IoTDBNonDataRegionSource.java
Comment thread iotdb-client/client-go Resolved
@jt2594838
jt2594838 merged commit 6559aa5 into apache:master Jul 31, 2026
41 checks passed
jt2594838 pushed a commit that referenced this pull request Aug 4, 2026
* Pipe: Handle hybrid meta progress indexes

* Pipe: Handle hybrid meta progress indexes

* Fix getter

* Pipe: Delegate progress index extraction to subclasses
VGalaxies pushed a commit to VGalaxies/iotdb that referenced this pull request Oct 26, 2026
* Pipe: Handle hybrid meta progress indexes

* Pipe: Handle hybrid meta progress indexes

* Fix getter

* Pipe: Delegate progress index extraction to subclasses
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants