Skip to content

SOLR-17704: Back ZkDistributedLock with Curator recipes - #5357

Draft
sarkaramrit1993 wants to merge 7 commits into
apache:mainfrom
sarkaramrit1993:SOLR-17704-curator-locks
Draft

sarkaramrit1993 wants to merge 7 commits into
apache:mainfrom
sarkaramrit1993:SOLR-17704-curator-locks

Conversation

@sarkaramrit1993

@sarkaramrit1993 sarkaramrit1993 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

https://issues.apache.org/jira/browse/SOLR-17704

Description

Upgrade impact: with the Overseer disabled, 10.x and 11 nodes don't see each other's API locks, so upgrading needs a full restart or paused Collection and ConfigSet API commands. See "Rolling Upgrades" in the 11 upgrade notes.

Solr carries its own ZooKeeper read/write lock. Curator already ships a well-used one, so this swaps it in. The same lock serves the Collection API and the ConfigSet API.

Solution

ZkDistributedLock now wraps Curator's InterProcessReadWriteLock. Nodes get created per level when the lock is acquired. The fixed order (collection, shard, replica) still prevents deadlock. Acquire and release must happen on one thread. DistributedLock.isAcquired is gone, and the root lock dir is now a container node.

Curator's release can throw where the old delete rarely did, so DistributedMultiLock.release now releases every lock and rethrows the first failure, and the ConfigSet runner logs a release failure instead of letting it hide the command's own error.

SOLR-18668 follows up by refusing these commands while an older major version node is live. SOLR-18669 tracks a ConfigSet CREATE lock-order deadlock that predates this change.

Tests

Lock tests run without isAcquired, plus new ones for lazy levels and interrupted waiters. DistributedMultiLockTest covers a release that fails part way. Curator cleans up after an interrupt asynchronously, so those tests poll. To check a lock is really blocked, the test helper waits until the acquiring thread parks in Curator's LockInternals.internalLockLoop. If Curator renames that method, the tests time out instead of passing.

@sarkaramrit1993
sarkaramrit1993 force-pushed the SOLR-17704-curator-locks branch from c3c43a7 to 6983ba2 Compare November 17, 2026 19:21

@HoustonPutman HoustonPutman left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I was going to do a review, but after looking at everything I think we are making progress removing our custom code (and using Curator recipes), but we haven't actually removed any of our classes. Let's see if we can improve the curator recipes such that we don't need nearly as much of our own code. Especially since this is a Solr 11 thing we have time to make the changes to Curator that we need.

Comment on lines +29 to +31

If the Overseer is disabled (cluster property `overseerEnabled` or `SOLR_CLOUD_OVERSEER_ENABLED` set to `false`), Collection and ConfigSet API locks now use a different ZooKeeper node format.
10.x and 11 nodes don't see each other's locks, so while both versions are running, two conflicting commands on the same collection or configset can run at the same time without an error.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes it sounds like the Collection and ConfigSet API locks use different ZK node format than each other. Make it more clear that its the versions that have different implementations

Suggested change
If the Overseer is disabled (cluster property `overseerEnabled` or `SOLR_CLOUD_OVERSEER_ENABLED` set to `false`), Collection and ConfigSet API locks now use a different ZooKeeper node format.
10.x and 11 nodes don't see each other's locks, so while both versions are running, two conflicting commands on the same collection or configset can run at the same time without an error.
If the Overseer is disabled (cluster property `overseerEnabled` or `SOLR_CLOUD_OVERSEER_ENABLED` set to `false`), Collection and ConfigSet API locks now use a different ZooKeeper node format than they did in Solr 10 and previous versions.
10.x and 11 nodes don't see each other's locks, so while both versions are running, two conflicting commands on the same collection or configset can run at the same time without an error.

Comment on lines +47 to +50
InterProcessReadWriteLock.ReadLock readLock = readWriteLock.readLock();
return new ZkDistributedLock.Owned(readLock, lockPath, readLock::getLockPath);
}
return new ZkDistributedLock.Mirrored(zkClient, lockPath, isWriteLock, lockIdToMirror);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add some else {} blocks here to make it a little easier to read.

if (StrUtils.isBlank(lockIdToMirror)) {
// Empty payload: otherwise Curator stores the local IP address in every lock node
InterProcessReadWriteLock readWriteLock =
new InterProcessReadWriteLock(zkClient.getCuratorFramework(), lockPath, new byte[0]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it doesn't hurt to put the solr node name there, just for future reference...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants