[bhouse-nexthop]

Description

Since #13101 (multiple CD-ROMs per VM), attachIso on main refuses an ISO that is already attached, and any ISO beyond the number of CD-ROMs, one by default. KubernetesClusterActionWorker.attachIsoKubernetesVMs relied on the attach replacing whatever ISO a node had. So a node left holding the binaries ISO by an operation that didn't finish, for example because the management server stopped, made every later upgrade, scale-up or start fail before it started, with "The specified ISO is already attached to this Instance."

Fixes #14630

For each node, attachIsoKubernetesVMs now reads the VM again and:

• leaves the binaries ISO in place if it is already attached, in the primary CD-ROM or in another one;
• otherwise detaches the ISO in the primary CD-ROM, if any, before attaching the binaries ISO, as the attach used to replace it.

4.22 is not affected, so this is for main only.

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

New unit tests in KubernetesClusterActionWorkerIsoTest:

• a node with no ISO gets the binaries ISO attached, and nothing is detached;
• a node that already has the binaries ISO, in the primary CD-ROM or in another one, is left as it is;
• a node with another ISO has it detached, then the binaries ISO attached.

Without the fix, the last three fail. mvn -pl plugins/integrations/kubernetes-service -am test -Dtest='KubernetesClusterActionWorker*Test' passes.

On a test environment running main, an upgrade failed as described in #14630 while both nodes still held the binaries ISO, and ran once the ISO was detached from both by hand. This PR itself has not been run against a live cluster.

How did you try to break this feature and the system with this change?

• A node with no ISO takes the same path as before.
• If the detach fails, the operation fails as an attach failure did before.
• If the attach fails after the detach, the node is left with no ISO, and the operation fails as before.
• detachIsoKubernetesVMs is unchanged.

[codecov]

Codecov Report

❌ Patch coverage is 87.50000% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 20.78%. Comparing base (a95eb59) to head (e9f5d4e).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...r/actionworkers/KubernetesClusterActionWorker.java	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #14637   +/-   ##
=========================================
  Coverage     20.78%   20.78%           
- Complexity    21927    21932    +5     
=========================================
  Files          6496     6496           
  Lines        591657   591665    +8     
  Branches      72592    72595    +3     
=========================================
+ Hits         122955   122967   +12     
+ Misses       455356   455345   -11     
- Partials      13346    13353    +7     

Flag	Coverage Δ
uitests	3.88% <ø> (ø)
unittests	22.09% <87.50%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[bhouse-nexthop]

Test report: CKS upgrade with the binaries ISO already attached to a node

Summary: with the upgrade's binaries ISO already attached to the control node, as an operation that didn't finish would leave it, upgradeKubernetesCluster no longer fails at the attach step. It leaves the ISO in place on that node, attaches it to the other node, upgrades both, and detaches it from both at the end.

Environment

• A CloudStack 24.0 (main) build with this PR, alongside some of my other open PRs, including CKS: deploy nodes of a cluster created before 4.21 from the default template #14634 and CKS: record a node's Kubernetes version, not the kubectl output #14635 in the same plugin. The PR is for main only.
• Three KVM hosts, Ceph RBD primary storage, one management server.
• Cluster B: 1 control node and 1 worker, Kubernetes v1.33.1, isolated network. It was created on 4.20 and upgraded to 24.0.
• Upgrade target: v1.33.13, from an ISO built with main's create-kubernetes-binaries-iso.sh.

Method

1. attachIso of the v1.33.13 binaries ISO to the control node. listVirtualMachines then showed that ISO on the control node and no ISO anywhere else.
2. upgradeKubernetesCluster to v1.33.13.
3. Check the log, the cluster, the nodes and which VMs hold an ISO.

Results

• The upgrade succeeded in 334 s: Running -> Upgrading -> Running.
• The attach step, trimmed:

  20:30:56,899 INFO  KubernetesClusterUpgradeWorker  Upgrading Kubernetes cluster: cluster B
  20:30:56,984 INFO  KubernetesClusterUpgradeWorker  Binaries ISO is already attached to VM: ... control-1 ...
  20:30:57,260 INFO  KubernetesClusterUpgradeWorker  Attached binaries ISO for VM: ... worker-1 ...
  20:30:57,261 INFO  KubernetesClusterUpgradeWorker  Upgrading node control-1 ...
  20:35:14,855 INFO  KubernetesClusterUpgradeWorker  Successfully upgraded node on VM ... control-1 ...
  20:36:26,576 INFO  KubernetesClusterUpgradeWorker  Successfully upgraded node on VM ... worker-1 ...
  20:36:28,844 INFO  KubernetesClusterUpgradeWorker  Detached Kubernetes binaries from VM: ... control-1 ...
  20:36:29,755 INFO  KubernetesClusterUpgradeWorker  Detached Kubernetes binaries from VM: ... worker-1 ...
  

• The log has no The specified ISO is already attached to this Instance, which is the stock failure in CKS (main): an upgrade, scale or start fails if a node still has an ISO attached #14630.
• kubectl get nodes afterwards: both nodes Ready, on v1.33.13. Every pod was Running or Completed.
• No VM held an ISO afterwards.

Result: pass. A node already holding the binaries ISO no longer blocks the upgrade. The ISO is kept on that node for the upgrade, and detached from every node at the end, as for a node that had none.

Not covered

• A node holding a different ISO, which the PR detaches before attaching the binaries ISO. That path is covered by the unit tests only.
• The same case in a scale-up or a start. A scale-up attaches the ISO only to the new nodes, so a leftover ISO on an existing node doesn't reach this code there.
• A leftover ISO in a second CD-ROM: the lab VMs have one CD-ROM.