Repository navigation
Abort sync ironic on vault decryption failure - #2898
Merged
Merged
Conversation
There was a problem hiding this comment.
Hey - I've found 4 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="osism/tasks/openstack.py" line_range="263" />
<code_context>
+ skip_kernel_params=skip_kernel_params,
+ extra_kernel_params=extra_kernel_params,
+ )
+ except VaultDecryptError as exc:
+ osism_utils.push_task_output(
+ request_id,
</code_context>
<issue_to_address>
**Secrets appear unmasked**
When a secrets mapping contains an undecryptable value before a valid encrypted secret used in displayed parameters, `deep_decrypt()` stops at the first undecryptable value, so later valid secrets are not added to `secret_values`; the name-based fallback masks only `ironic_osism_*` parameters, leaving other matching displayed parameters, including kernel arguments, in plaintext.
Continue processing the mapping after an undecryptable value and collect every decryptable secret for masking.
Also at `osism/tasks/conductor/utils.py:75-77`.
</issue_to_address>
### Comment 2
<location path="osism/tasks/conductor/utils.py" line_range="75-77" />
<code_context>
+ try:
+ a[key] = vault.decrypt(value).decode().strip()
+ except Exception as exc:
+ raise VaultDecryptError(
+ f"Unable to decrypt '{key_path}': {exc}"
+ ) from exc
</code_context>
<issue_to_address>
**Failed deploy leaves node metadata changed**
When a deploy targets a node whose `instance_info` was cleared but is recoverable from `extra`, and FRR decryption fails while building its config drive, a node with cleared `instance_info` and saved metadata in `extra` is updated at `baremetal.py:420-422` before FRR decryption. When this new error is raised at `baremetal.py:518`, the config-drive handler marks the node failed and skips deployment, leaving Ironic changed despite the failed operation.
Decrypt and build the config drive before restoring `instance_info`, or roll back that update if config-drive preparation fails.
</issue_to_address>
### Comment 3
<location path="tests/unit/tasks/conductor/test_ironic_sync.py" line_range="770" />
<code_context>
+def test_sync_ironic_aborts_before_any_change_on_vault_error(sync_env):
+ sync_env.netbox.get_devices.return_value = [_make_device()]
+ sync_env.prepare.side_effect = ironic.VaultDecryptError(
+ "Unable to decrypt 'secrets.remote_board_password': no vault secrets"
</code_context>
<issue_to_address>
**Later-device changes go undetected**
When a vault error occurs while preparing a later device after an earlier device has prepared successfully, when an earlier device prepares successfully and a later one raises, both tests still pass against a per-device prepare-and-sync implementation: each supplies only one device and makes its preparation fail immediately, so neither exercises a successful device being changed before the later failure.
Use at least two devices, make preparation succeed for the first and fail for the second, and assert neither device is synced or otherwise changed.
Also at `tests/unit/tasks/conductor/test_ironic_sync.py:789`.
</issue_to_address>
### Comment 4
<location path="tests/unit/tasks/conductor/test_utils.py" line_range="312" />
<code_context>
- deep_decrypt(data, vault)
-
- assert data == {"b": "clear"}
+ with pytest.raises(VaultDecryptError, match="'secrets.outer.a'"):
+ deep_decrypt(data, vault, "secrets")
</code_context>
<issue_to_address>
**Wrong key paths pass the assertion**
When the path contains a wrong character in place of a dot, when `deep_decrypt()` formats the path with the wrong one-character separators, `test_deep_decrypt_dict_value_decrypt_failure_raises_with_path` still passes: the dots in `match="'secrets.outer.a'"` are regex wildcards, so a message containing `'secretsXouterXa'` matches.
Escape the dots or compare the exception message to the exact expected path.
</issue_to_address>Sourcery assessment
Approval pending. 4 findings to address first.
Blocking findings: osism/tasks/openstack.py:263, osism/tasks/conductor/utils.py:77, tests/unit/tasks/conductor/test_ironic_sync.py:770, tests/unit/tasks/conductor/test_utils.py:312
ideaship
marked this pull request as draft
November 16, 2026 14:19
ideaship
force-pushed
the
ironic-vault-decrypt-abort
branch
from
November 16, 2026 14:30
7801b28 to
17c1902
Compare
ideaship
marked this pull request as ready for review
November 16, 2026 14:49
deep_decrypt() silently dropped every dict value it could not decrypt and left undecryptable list elements as ciphertext. get_vault() only logs when the vault password is missing, so a missing or wrong password went unnoticed: sync ironic carried on, the node secrets lacked remote_board_username/remote_board_password, and _prepare_node_attributes() fell back to the hard-coded "admin" and "password". Because node updates send the full node attributes, those defaults replaced the working BMC credentials in driver_info whenever the node was created, --force was given, or any other attribute differed. deep_decrypt() now decrypts every value it can and then raises VaultDecryptError naming all keys that failed (e.g. 'secrets.remote_board_password'), for dicts and lists alike. Carrying on past a failure matters for callers that catch the error: they still get every value that could be decrypted, e.g. when some entries were encrypted with a different vault password. get_vault() stays lenient, since it is also called for devices without any encrypted values, and those must keep working without a vault password; its "Dropping encrypted entries" message is gone because nothing is dropped anymore. sync_ironic() now prepares the attributes of all devices before it deletes stale nodes or touches any node. On a decryption error it reports the device and key, states that no changes were made, and finishes with rc=1 so the CLI fails rather than printing a warning that scrolls past. Dry runs abort the same way, since their output would otherwise present the fallback credentials as the planned change. The other callers of deep_decrypt() now fail instead of continuing with secrets missing: the SONiC config generator already turns a per-device exception into a failed device and a non-zero rc, and baremetal deploy/dump stop before applying FRR parameters with missing secrets. The secret-masking helper in tasks/openstack.py is the exception: it only redacts kernel parameters for display, and it must keep masking ironic_osism_* parameters by name when their values cannot be decrypted. Its outer exception handler would skip that, so it catches VaultDecryptError around the decryption, logs a warning and carries on with the values that were decrypted. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
force-pushed
the
ironic-vault-decrypt-abort
branch
from
November 16, 2026 15:12
17c1902 to
64f6c85
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
osism sync ironicdid not stop when vault-encrypted secrets could not be decrypted.deep_decrypt()silently dropped dict values it failed to decrypt, andget_vault()only logs when the vault password is missing. With a missing or wrong vault password, the node secrets lackedremote_board_username/remote_board_password,_prepare_node_attributes()fell back to the hard-codedadmin/password, and since node updates send the full node attributes, these defaults overwrote working Redfish/IPMI credentials indriver_infowhenever a node was created,--forcewas given, or any other attribute differed. The only trace was an error line in the worker log.Changes
deep_decrypt()decrypts every value it can, then raisesVaultDecryptErrornaming all keys that failed (e.g.secrets.remote_board_password), instead of dropping dict values or leaving list elements as ciphertext.sync_ironic()prepares the attributes of all devices before deleting stale nodes or touching any node. On a decryption error it reports the device and the failing keys, states that no changes were made, and finishes with rc=1, so the CLI fails. Dry runs abort the same way.get_vault()stays lenient, because it is also called for devices without encrypted values, which must keep working without a vault password.tasks/openstack.pycatches the error and keeps maskingironic_osism_*kernel parameters by name, plus every value that could be decrypted, since it only redacts for display.Other
deep_decrypt()callers now fail instead of continuing with secrets missing: the SONiC config generator turns it into a failed device and a non-zero rc, andosism baremetal deploy/dumpstop before applying FRR parameters with missing secrets.Testing
deep_decrypt(), decryption of the remaining values after a failure, and masking when decryption fails.deep_decrypt()checked against a realansible-coreVaultLib: missing password and wrong password both raise with the key path; the right password decrypts dict and list values; with entries encrypted under two passwords, those matching the configured one are decrypted and the others are named in the error.🤖 Generated with Claude Code