Repository navigation
CKS: an upgrade records each node's Kubernetes version as raw kubectl output, with a leading v and a trailing newline #14629
Description
problem
After a CKS cluster is upgraded, each upgraded node's Kubernetes version (kubernetes_cluster_vm_map.kubernetes_node_version, shown as kubernetesnodeversion in listKubernetesClusters) is stored as the raw output of an SSH command: v1.33.1\n, with a v and a trailing newline. Nodes created by the cluster's create or scale paths store 1.33.1.
upgraded node: 76312E33332E310A ("v1.33.1\n")
scaled-up node: 312E33332E31 ("1.33.1")
Clients that print the field get a stray line break, and the two forms don't compare equal.
Cause: KubernetesClusterUtil.clusterNodeVersionMatches(...) (the SSH variant, used by KubernetesClusterUpgradeWorker) saves clusterVersionMatchesAndValue.second(), which is the whole output of
sudo /opt/bin/kubectl get nodes | awk '{if ($1 == "<node>") print $5}'
That includes the newline, and anything the command wrote to stderr, such as a sudo warning.
versions
4.21.0.0 and later (since #9102), including 4.22 and main.
The steps to reproduce the bug
- Create a CKS cluster at one Kubernetes version.
- Upgrade it to a newer version with
upgradeKubernetesCluster. listKubernetesClusters id=<cluster>and look atkubernetesnodeversionfor each VM, orSELECT HEX(kubernetes_node_version) FROM kubernetes_cluster_vm_map WHERE cluster_id = <id>.- Upgraded nodes have
v<version>\n; nodes added by a later scale-up have<version>.
What to do about it?
Once the node is found at the target version, record that version (the semantic version of the upgrade), the same form the create path records.
Activity
github-actions commentedon Nov 10, 2026
Warning
threat detection engine error
The threat detection engine encountered an error and could not complete analysis. This is a tooling failure, not a security finding.
What happened
The threat detection engine failed to produce results.
Review the workflow run logs for details.
🎯 Triage report
After a CKS upgrade, each upgraded node's Kubernetes version is stored as raw
kubectloutput, so it includes a leadingvand a trailing newline (for examplev1.33.1\n). Nodes added by create or scale store1.33.1. Clients that print the field see a stray line break, and the two forms do not compare equal.📊 Assessment
KubernetesClusterUtil🔗 Similar issues
💡 Notes and suggestions
kubernetesnodeversioninlistKubernetesClusters, orkubernetes_node_versioninkubernetes_cluster_vm_map.sudowarning) can end up in the same field.Add this agentic workflow to your repo
To install this agentic workflow, run