Skip to content

Pipe: Never clear the exceptions for user information & Only record the last exception & Add suggested actions for exceptions - #16233

Draft
Caideyipi wants to merge 21 commits into
apache:masterfrom
Caideyipi:fix-e
Draft

Caideyipi wants to merge 21 commits into
apache:masterfrom
Caideyipi:fix-e

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Aug 22, 2025 •

Copy link
Copy Markdown
Collaborator

Description

As the title said.

Test

image

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

@Caideyipi
Caideyipi marked this pull request as draft August 27, 2025 02:36
@Caideyipi
Caideyipi marked this pull request as ready for review August 28, 2025 07:38
@Caideyipi Caideyipi changed the title Pipe: Never clear the exceptions for user information Pipe: Never clear the exceptions for user information & Only record the last exception & Add suggested actions for exceptions Aug 28, 2025
luoluoyuyu

This comment was marked as outdated.

@luoluoyuyu luoluoyuyu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Keeps only the latest pipe exception for display, avoids clearing it automatically, and adds suggested actions. trackException clears then adds a single entry; coordinator meta sync clears exceptions on the agent to avoid stale overwrites.

Please fix failing CI before merge. See inline notes.

// Here we still keep the map form to allow compatibility with legacy versions
exceptionMessages.clear();
exceptionMessages.add(exceptionMessage);
lastException.clear();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clear then add enforces a single last exception. If SHOW PIPES now exposes last_exception as a structured column, consider whether getExceptionMessage() (Set toString) should be updated or deprecated for user-facing output.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kept the existing exception serialization and restored the user-facing column names to ExceptionMessage / exception_message for compatibility. TPipeInfo.exceptionMessage and existing SHOW PIPES / information_schema clients therefore keep the same contract; this PR only adds the separate SuggestedAction column.

.getRuntimeMeta()
.getConsensusGroupId2TaskMetaMap()
.values()
.forEach(PipeTaskMeta::clearExceptionMessages);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearing exceptions when applying meta from the coordinator prevents old CN-reported errors from overwriting the true last error on the DataNode after restart. Matches the do-not-auto-clear behavior for operators.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. This behavior is intentionally preserved: coordinator-provided exceptions are cleared before applying the metadata so they cannot overwrite the DataNode's locally tracked latest exception.

@Caideyipi
Caideyipi marked this pull request as draft July 30, 2026 02:21
@Caideyipi

Copy link
Copy Markdown
Collaborator Author

Addressed the review and current deterministic CI failures in 0811b05:

  • preserved the existing Thrift exceptionMessage field and SHOW PIPES / information_schema ExceptionMessage / exception_message column names for client compatibility;
  • added only the new SuggestedAction / suggested_action column;
  • updated IoTDBDatabaseIT for the information-schema column list;
  • updated IoTDBPipePermissionIT for the shifted remaining-event-count column.

The related pipe/meta unit tests pass. A focused local IT build compiled all relevant modules, but the locally injected com.timecho.ao plugin prevents JDBC startup by calling the removed IoTDBConfig.isEnableWhiteList() method, so the IT itself could not execute in this environment. The latest GitHub dual-table-manual-basic run is green.

…pr-16233

# Conflicts:
#	integration-test/src/test/java/org/apache/iotdb/relational/it/schema/IoTDBDatabaseIT.java
#	iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/execution/config/sys/pipe/ShowPipeTask.java
#	iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/plan/execution/config/sys/pipe/ShowPipeTaskTest.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants