Repository navigation
HBASE-30539 CreateTableProcedure ignores failed table descriptor write - #8877
Open
Umeshkumar9414 wants to merge 2 commits into
Open
Umeshkumar9414 wants to merge 2 commits into
Umeshkumar9414 wants to merge 2 commits into
Conversation
FSTableDescriptors.writeTableDescriptor retried 10 times, logged every failure at DEBUG and then returned null. CreateTableProcedure.createFsLayout ignored the result, so a failed .tableinfo write still let the procedure go on to ADD_TO_META and ASSIGN_REGIONS. Region servers then failed every open with "Missing table descriptor", the TRSP retried forever and the create procedure stayed in ASSIGN_REGIONS holding the table lock. Make writeTableDescriptor throw IOException (with the last failure as cause) once all attempts are exhausted, and log failed attempts at WARN. The create procedure now stays in WRITE_FS_LAYOUT and retries, and the other callers that ignored the result (clone snapshot, snapshot manifest, MasterRegion, backup) no longer silently continue either. createTableDescriptorForTableDirectory returns false only when an equal descriptor already exists.
Umeshkumar9414
force-pushed
the
HBASE-30539
branch
from
November 13, 2026 10:34
372cd50 to
6a0444f
Compare
Umeshkumar9414
commented
Nov 13, 2026
| TableName tableName = td.getTableName(); | ||
| Path tableDir = CommonFSUtils.getTableDir(rootdir, tableName); | ||
| Path p = writeTableDescriptor(fs, td, tableDir, null); | ||
| if (p == null) { |
Contributor
Author
There was a problem hiding this comment.
it is dead code now as we are not returning null. we will throw exception.
This was referenced Nov 13, 2026
Apache9
reviewed
Nov 13, 2026
| } | ||
|
|
||
| @Test | ||
| public void testCreateTableDoesNotProceedWhenTableDescriptorWriteFails() throws Exception { |
Contributor
There was a problem hiding this comment.
Better also add javadoc to describe the details? Like the two methods below.
…riptorWriteFails Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA: https://issues.apache.org/jira/browse/HBASE-30539
Problem
FSTableDescriptors.writeTableDescriptorretries 10 times, logs each failure at DEBUG, then returnsnull.createTableDescriptorForTableDirectoryturns that intofalse, which is the same value it returns for "an equal descriptor already exists".CreateTableProcedure.createFsLayoutignores the return value, so when HDFS refuses every.tableinfowrite the procedure still moves on toCREATE_TABLE_ADD_TO_METAandCREATE_TABLE_ASSIGN_REGIONS.The table then exists in
hbase:metaand in the master's descriptor cache, but not on the file system. The RegionServer has no descriptor cache, so every region open fails withMissing table descriptor for .... The childTransitRegionStateProcedureretries forever and the parent create procedure stays inASSIGN_REGIONSholding the table lock.Fix
writeTableDescriptornow throwsIOException(last failure as the cause) after all attempts fail, and logs failed attempts at WARN.createFsLayouttherefore fails the step with anIOException; becauseCREATE_TABLE_WRITE_FS_LAYOUTbegins with the pre-clean of the table dir, the procedure's retry with backoff is idempotent. Once HDFS recovers, the same procedure completes.createTableDescriptorForTableDirectorynow returnsfalseonly when an equal descriptor already exists. The two callers that checked fornull(updateTableDescriptor,tryUpdateAndGetMetaTableDescriptor) lose their dead checks. Other callers that ignored the result (CloneSnapshotProcedure,SnapshotManifest,MasterRegion,BackupUtils) now see the failure instead of continuing silently.Tests
TestCreateTableProcedureDescriptorWriteFailureinjects.tabledesccreate failures through aDistributedFileSystemsubclass:testCreateTableDoesNotProceedWhenTableDescriptorWriteFails: every write fails. The create procedure keeps retrying and does not fail, finish, add rows tohbase:metaor start a child assign procedure. Once the injection is cleared, the same procedure completes and the region server can load the descriptor.testTransientWriteFailuresAbsorbedByWriteRetries: 3 failures (fewer than the 10 in-method attempts) are absorbed by the retry loop and the create succeeds.testProcedureRetriesAfterWriteAttemptsExhaustedThenSucceeds: 12 failures exhaust the first round, the procedure retriesWRITE_FS_LAYOUT, then succeeds without ever adding the table tohbase:metain between.Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>