Skip to content

Abort sync ironic on vault decryption failure - #2898

Merged
berendt merged 1 commit into
mainfrom
ironic-vault-decrypt-abort
Nov 17, 2026
Merged

berendt merged 1 commit into
mainfrom
ironic-vault-decrypt-abort

Conversation

@ideaship

@ideaship ideaship commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

osism sync ironic did not stop when vault-encrypted secrets could not be decrypted. deep_decrypt() silently dropped dict values it failed to decrypt, and get_vault() only logs when the vault password is missing. With a missing or wrong vault password, the node secrets lacked remote_board_username/remote_board_password, _prepare_node_attributes() fell back to the hard-coded admin/password, and since node updates send the full node attributes, these defaults overwrote working Redfish/IPMI credentials in driver_info whenever a node was created, --force was given, or any other attribute differed. The only trace was an error line in the worker log.

Changes

  • deep_decrypt() decrypts every value it can, then raises VaultDecryptError naming all keys that failed (e.g. secrets.remote_board_password), instead of dropping dict values or leaving list elements as ciphertext.
  • sync_ironic() prepares the attributes of all devices before deleting stale nodes or touching any node. On a decryption error it reports the device and the failing keys, states that no changes were made, and finishes with rc=1, so the CLI fails. Dry runs abort the same way.
  • get_vault() stays lenient, because it is also called for devices without encrypted values, which must keep working without a vault password.
  • The secret-masking helper in tasks/openstack.py catches the error and keeps masking ironic_osism_* kernel parameters by name, plus every value that could be decrypted, since it only redacts for display.

Other deep_decrypt() callers now fail instead of continuing with secrets missing: the SONiC config generator turns it into a failed device and a non-zero rc, and osism baremetal deploy/dump stop before applying FRR parameters with missing secrets.

Testing

  • New unit tests cover the sync abort (no deletions, no device sync even for devices prepared before the failing one, rc=1; also in dry-run mode), the raised error and key paths in deep_decrypt(), decryption of the remaining values after a failure, and masking when decryption fails.
  • deep_decrypt() checked against a real ansible-core VaultLib: missing password and wrong password both raise with the key path; the right password decrypts dict and list values; with entries encrypted under two passwords, those matching the configured one are decrypted and the others are named in the error.

🤖 Generated with Claude Code

@ideaship ideaship self-assigned this Nov 16, 2026
@ideaship
ideaship marked this pull request as ready for review November 16, 2026 14:04

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 4 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="osism/tasks/openstack.py" line_range="263" />
<code_context>
+                skip_kernel_params=skip_kernel_params,
+                extra_kernel_params=extra_kernel_params,
+            )
+        except VaultDecryptError as exc:
+            osism_utils.push_task_output(
+                request_id,
</code_context>
<issue_to_address>
**Secrets appear unmasked**

When a secrets mapping contains an undecryptable value before a valid encrypted secret used in displayed parameters, `deep_decrypt()` stops at the first undecryptable value, so later valid secrets are not added to `secret_values`; the name-based fallback masks only `ironic_osism_*` parameters, leaving other matching displayed parameters, including kernel arguments, in plaintext.

Continue processing the mapping after an undecryptable value and collect every decryptable secret for masking.

Also at `osism/tasks/conductor/utils.py:75-77`.
</issue_to_address>

### Comment 2
<location path="osism/tasks/conductor/utils.py" line_range="75-77" />
<code_context>
+            try:
+                a[key] = vault.decrypt(value).decode().strip()
+            except Exception as exc:
+                raise VaultDecryptError(
+                    f"Unable to decrypt '{key_path}': {exc}"
+                ) from exc


</code_context>
<issue_to_address>
**Failed deploy leaves node metadata changed**

When a deploy targets a node whose `instance_info` was cleared but is recoverable from `extra`, and FRR decryption fails while building its config drive, a node with cleared `instance_info` and saved metadata in `extra` is updated at `baremetal.py:420-422` before FRR decryption. When this new error is raised at `baremetal.py:518`, the config-drive handler marks the node failed and skips deployment, leaving Ironic changed despite the failed operation.

Decrypt and build the config drive before restoring `instance_info`, or roll back that update if config-drive preparation fails.
</issue_to_address>

### Comment 3
<location path="tests/unit/tasks/conductor/test_ironic_sync.py" line_range="770" />
<code_context>


+def test_sync_ironic_aborts_before_any_change_on_vault_error(sync_env):
+    sync_env.netbox.get_devices.return_value = [_make_device()]
+    sync_env.prepare.side_effect = ironic.VaultDecryptError(
+        "Unable to decrypt 'secrets.remote_board_password': no vault secrets"
</code_context>
<issue_to_address>
**Later-device changes go undetected**

When a vault error occurs while preparing a later device after an earlier device has prepared successfully, when an earlier device prepares successfully and a later one raises, both tests still pass against a per-device prepare-and-sync implementation: each supplies only one device and makes its preparation fail immediately, so neither exercises a successful device being changed before the later failure.

Use at least two devices, make preparation succeed for the first and fail for the second, and assert neither device is synced or otherwise changed.

Also at `tests/unit/tasks/conductor/test_ironic_sync.py:789`.
</issue_to_address>

### Comment 4
<location path="tests/unit/tasks/conductor/test_utils.py" line_range="312" />
<code_context>
-    deep_decrypt(data, vault)
-
-    assert data == {"b": "clear"}
+    with pytest.raises(VaultDecryptError, match="'secrets.outer.a'"):
+        deep_decrypt(data, vault, "secrets")

</code_context>
<issue_to_address>
**Wrong key paths pass the assertion**

When the path contains a wrong character in place of a dot, when `deep_decrypt()` formats the path with the wrong one-character separators, `test_deep_decrypt_dict_value_decrypt_failure_raises_with_path` still passes: the dots in `match="'secrets.outer.a'"` are regex wildcards, so a message containing `'secretsXouterXa'` matches.

Escape the dots or compare the exception message to the exact expected path.
</issue_to_address>

Sourcery assessment

Approval pending. 4 findings to address first.

Blocking findings: osism/tasks/openstack.py:263, osism/tasks/conductor/utils.py:77, tests/unit/tasks/conductor/test_ironic_sync.py:770, tests/unit/tasks/conductor/test_utils.py:312


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread osism/tasks/openstack.py Resolved
Comment thread osism/tasks/conductor/utils.py Resolved
@ideaship
ideaship marked this pull request as draft November 16, 2026 14:19
@ideaship
ideaship force-pushed the ironic-vault-decrypt-abort branch from 7801b28 to 17c1902 Compare November 16, 2026 14:30
@ideaship
ideaship marked this pull request as ready for review November 16, 2026 14:49
deep_decrypt() silently dropped every dict value it could not decrypt
and left undecryptable list elements as ciphertext. get_vault() only
logs when the vault password is missing, so a missing or wrong
password went unnoticed: sync ironic carried on, the node secrets
lacked remote_board_username/remote_board_password, and
_prepare_node_attributes() fell back to the hard-coded "admin" and
"password". Because node updates send the full node attributes, those
defaults replaced the working BMC credentials in driver_info whenever
the node was created, --force was given, or any other attribute
differed.

deep_decrypt() now decrypts every value it can and then raises
VaultDecryptError naming all keys that failed (e.g.
'secrets.remote_board_password'), for dicts and lists alike. Carrying
on past a failure matters for callers that catch the error: they still
get every value that could be decrypted, e.g. when some entries were
encrypted with a different vault password.
get_vault() stays lenient, since it is also called for devices without
any encrypted values, and those must keep working without a vault
password; its "Dropping encrypted entries" message is gone because
nothing is dropped anymore.

sync_ironic() now prepares the attributes of all devices before it
deletes stale nodes or touches any node. On a decryption error it
reports the device and key, states that no changes were made, and
finishes with rc=1 so the CLI fails rather than printing a warning
that scrolls past. Dry runs abort the same way, since their output
would otherwise present the fallback credentials as the planned
change.

The other callers of deep_decrypt() now fail instead of continuing
with secrets missing: the SONiC config generator already turns a
per-device exception into a failed device and a non-zero rc, and
baremetal deploy/dump stop before applying FRR parameters with
missing secrets.

The secret-masking helper in tasks/openstack.py is the exception: it
only redacts kernel parameters for display, and it must keep masking
ironic_osism_* parameters by name when their values cannot be
decrypted. Its outer exception handler would skip that, so it catches
VaultDecryptError around the decryption, logs a warning and carries
on with the values that were decrypted.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Roger Luethi <luethi@osism.tech>
@ideaship
ideaship force-pushed the ironic-vault-decrypt-abort branch from 17c1902 to 64f6c85 Compare November 16, 2026 15:12
@berendt
berendt merged commit 2990861 into main Nov 17, 2026
3 checks passed
@berendt
berendt deleted the ironic-vault-decrypt-abort branch November 17, 2026 15:55
@github-project-automation github-project-automation Bot moved this from New to Done in Human Board Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants