CKS: a cluster created before 4.21 can't be scaled up after the upgrade (NullPointerException, template is null) #14628

problem

A CKS cluster created before 4.21 can't be scaled up after CloudStack is upgraded to 4.21 or later. scaleKubernetesCluster with a larger size fails within seconds, with no VM created:

Unexpected exception while executing org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd
java.lang.NullPointerException: Cannot invoke "com.cloud.template.VirtualMachineTemplate.getUniqueName()" because "template" is null
  at com.cloud.vm.VirtualMachineManagerImpl.isBlankInstanceDefaultTemplate(VirtualMachineManagerImpl.java)
  at com.cloud.vm.VirtualMachineManagerImpl.isBlankInstance(VirtualMachineManagerImpl.java)
  at com.cloud.vm.UserVmManagerImpl.createAdvancedVirtualMachine(UserVmManagerImpl.java)
  at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.createKubernetesNode(KubernetesClusterResourceModifierActionWorker.java)
  at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.provisionKubernetesClusterNodeVms(KubernetesClusterResourceModifierActionWorker.java)
  at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterScaleWorker.scaleUpKubernetesClusterSize(KubernetesClusterScaleWorker.java)

The cluster goes back to Running.

Cause:

• CKS Enhancements #9102 (4.21) added control_node_template_id, worker_node_template_id and etcd_node_template_id to kubernetes_cluster. The upgrade adds the columns but doesn't fill them in, so they are NULL for every cluster created before 4.21.
• KubernetesClusterActionWorker.init() looks the node templates up from those columns only (templateDao.findById(kubernetesCluster.getWorkerNodeTemplateId())), which gives null.
• createKubernetesNode passes that null template to createAdvancedVirtualMachine.
• Before 4.21, nodes were deployed from the template CKS picks by default (clusterTemplate), and the owner was given launch permission for it during the operation. isDefaultTemplateUsed(), which decides about that launch permission, also returns false for these clusters, since none of the three columns equals template_id.

versions

4.21.0.0 and later, including 4.22 and main, for clusters created on 4.20 or earlier.

The steps to reproduce the bug

1. On 4.20, create a CKS cluster and let it start.
2. Upgrade CloudStack to 4.22.
3. Scale the cluster up by one worker (scaleKubernetesCluster id=<cluster> size=<current + 1>).
4. The job fails with the NullPointerException above. The cluster's control_node_template_id, worker_node_template_id and etcd_node_template_id are NULL.

Setting control_node_template_id and worker_node_template_id to the cluster's template_id in the database works around it.

What to do about it?

When a node type has no template set, deploy its nodes from the default CKS template, as before 4.21, and count the cluster as using the default template so the owner gets launch permission for it. Filling the columns in during the upgrade would be a schema change; the code fallback also covers clusters that were already upgraded.