Skip to content

HBASE-30539 CreateTableProcedure ignores failed table descriptor write - #8877

Open
Umeshkumar9414 wants to merge 2 commits into
apache:masterfrom
Umeshkumar9414:HBASE-30539
Open

Umeshkumar9414 wants to merge 2 commits into
apache:masterfrom
Umeshkumar9414:HBASE-30539

Conversation

@Umeshkumar9414

@Umeshkumar9414 Umeshkumar9414 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

JIRA: https://issues.apache.org/jira/browse/HBASE-30539

Problem

FSTableDescriptors.writeTableDescriptor retries 10 times, logs each failure at DEBUG, then returns null. createTableDescriptorForTableDirectory turns that into false, which is the same value it returns for "an equal descriptor already exists". CreateTableProcedure.createFsLayout ignores the return value, so when HDFS refuses every .tableinfo write the procedure still moves on to CREATE_TABLE_ADD_TO_META and CREATE_TABLE_ASSIGN_REGIONS.

The table then exists in hbase:meta and in the master's descriptor cache, but not on the file system. The RegionServer has no descriptor cache, so every region open fails with Missing table descriptor for .... The child TransitRegionStateProcedure retries forever and the parent create procedure stays in ASSIGN_REGIONS holding the table lock.

Fix

writeTableDescriptor now throws IOException (last failure as the cause) after all attempts fail, and logs failed attempts at WARN. createFsLayout therefore fails the step with an IOException; because CREATE_TABLE_WRITE_FS_LAYOUT begins with the pre-clean of the table dir, the procedure's retry with backoff is idempotent. Once HDFS recovers, the same procedure completes.

createTableDescriptorForTableDirectory now returns false only when an equal descriptor already exists. The two callers that checked for null (updateTableDescriptor, tryUpdateAndGetMetaTableDescriptor) lose their dead checks. Other callers that ignored the result (CloneSnapshotProcedure, SnapshotManifest, MasterRegion, BackupUtils) now see the failure instead of continuing silently.

Tests

TestCreateTableProcedureDescriptorWriteFailure injects .tabledesc create failures through a DistributedFileSystem subclass:

  • testCreateTableDoesNotProceedWhenTableDescriptorWriteFails: every write fails. The create procedure keeps retrying and does not fail, finish, add rows to hbase:meta or start a child assign procedure. Once the injection is cleared, the same procedure completes and the region server can load the descriptor.
  • testTransientWriteFailuresAbsorbedByWriteRetries: 3 failures (fewer than the 10 in-method attempts) are absorbed by the retry loop and the create succeeds.
  • testProcedureRetriesAfterWriteAttemptsExhaustedThenSucceeds: 12 failures exhaust the first round, the procedure retries WRITE_FS_LAYOUT, then succeeds without ever adding the table to hbase:meta in between.

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>

FSTableDescriptors.writeTableDescriptor retried 10 times, logged every
failure at DEBUG and then returned null. CreateTableProcedure.createFsLayout
ignored the result, so a failed .tableinfo write still let the procedure go on
to ADD_TO_META and ASSIGN_REGIONS. Region servers then failed every open with
"Missing table descriptor", the TRSP retried forever and the create procedure
stayed in ASSIGN_REGIONS holding the table lock.

Make writeTableDescriptor throw IOException (with the last failure as cause)
once all attempts are exhausted, and log failed attempts at WARN. The create
procedure now stays in WRITE_FS_LAYOUT and retries, and the other callers that
ignored the result (clone snapshot, snapshot manifest, MasterRegion, backup)
no longer silently continue either. createTableDescriptorForTableDirectory
returns false only when an equal descriptor already exists.
TableName tableName = td.getTableName();
Path tableDir = CommonFSUtils.getTableDir(rootdir, tableName);
Path p = writeTableDescriptor(fs, td, tableDir, null);
if (p == null) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is dead code now as we are not returning null. we will throw exception.

}

@Test
public void testCreateTableDoesNotProceedWhenTableDescriptorWriteFails() throws Exception {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better also add javadoc to describe the details? Like the two methods below.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

…riptorWriteFails

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants