You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have deployed K8s cluster in my CS 4.21.0.0 environment and I am trying to Autoscale the same to add a worker node and I am seeing below message in management server logs.
2025-11-05 16:07:17,543 INFO [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-34:[ctx-228e86cd, job-1437, ctx-43619b99]) (logid:56268dc3) Scaling Kubernetes cluster KubernetesCluster {"id":71,"name":"CL8S","uuid":"da626491-7457-4106-a165-de0d41d2417a"}
2025-11-05 16:07:17,553 WARN [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-34:[ctx-228e86cd, job-1437, ctx-43619b99]) (logid:56268dc3) Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!
2025-11-05 16:07:17,553 INFO [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-34:[ctx-228e86cd, job-1437, ctx-43619b99]) (logid:56268dc3) Revoking launch permission for systemVM template
2025-11-05 16:07:17,648 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-34:[ctx-228e86cd, job-1437]) (logid:56268dc3) Remove job-1437 from job monitoring
I am having a hypevisor as KVM which is running on Ubuntu 24.04.3.
Can anyone help figure out why I am facing this problem and whether KVM is really not supported for CKS Autoscale Feature ? if not, then which hypevisor is supported for Autoscale ?
I can't imagine why scaling k8s (by adding nodes right?) would not work on KVM. That sounds ridiculous. Now, resource scaling of existing nodes, maybe. I am actually bummed because I upgraded 4.19 to 4.21 not realizing it was a new type of release (non-LTS) , and it had broken kubernetes management but hey, so far everything else works great so it just needs some k8s fixes.
I1106 02:53:19.477668 1 client.go:169] NewAPIRequest API request URL:http://142.79.253.232:8080/client/api?apiKey=***&command=scaleKubernetesCluster&id=da626491-7457-4106-a165-de0d41d2417a&response=json&size=3&signature=***
I1106 02:53:19.477890 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"ad1efd21-9a52-4226-9c1d-15396820bba5", APIVersion:"v1", ResourceVersion:"205055", FieldPath:""}): type: 'Normal' reason: 'ScaledUpGroup' Scale-up: setting group da626491-7457-4106-a165-de0d41d2417a size to 3 instead of 2 (max: 4)
I1106 02:53:19.556058 1 client.go:175] NewAPIRequest response status code:200
I1106 02:53:19.863209 1 reflector.go:255] Listing and watching *v1beta1.PodDisruptionBudget from /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309
W1106 02:53:19.865022 1 reflector.go:324] /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309: failed to list *v1beta1.PodDisruptionBudget: the server could not find the requested resource
E1106 02:53:19.865056 1 reflector.go:138] /home/djumani/lab/autoscaler/cluster-autoscaler/utils/kubernetes/listers.go:309: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: the server could not find the requested resource
I1106 02:53:29.557052 1 client.go:169] NewAPIRequest API request URL:http://142.79.253.232:8080/client/api?apiKey=***&command=queryAsyncJobResult&jobid=acee7bf9-c308-4929-87e5-42f74652d5eb&response=json&signature=***
I1106 02:53:29.597851 1 client.go:175] NewAPIRequest response status code:200
E1106 02:53:29.598077 1 client.go:118] API failed for job acee7bf9-c308-4929-87e5-42f74652d5eb : map[account:admin accountid:9b2313ee-b4ae-11f0-914b-bc2411c796cd cmd:org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd completed:2025-11-06T08:23:19+0530 created:2025-11-06T08:23:19+0530 domainid:abbdc016-b4ad-11f0-914b-bc2411c796cd domainpath:ROOT jobid:acee7bf9-c308-4929-87e5-42f74652d5eb jobprocstatus:0 jobresult:map[errorcode:530 errortext:Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!] jobresultcode:530 jobresulttype:object jobstatus:2 userid:80acdbf0-fe5c-4ed1-a5ea-4d8e9fe7307a]
W1106 02:53:29.598146 1 clusterstate.go:277] Disabling scale-up for node group da626491-7457-4106-a165-de0d41d2417a until 2025-11-06 03:23:19.477324222 +0000 UTC m=+64348.574537733; errorClass=Other; errorCode=cloudProviderError
I1106 02:53:29.598178 1 metrics.go:380] Function scaleUp took 10.120850343s to complete
I1106 02:53:29.598196 1 eventing_scale_up_processor.go:47] Skipping event processing for unschedulable pods since there is a ScaleUp attempt this loop
E1106 02:53:29.598212 1 static_autoscaler.go:459] Failed to scale up: failed to increase node group size: Unable to scale cluster : API failed for job acee7bf9-c308-4929-87e5-42f74652d5eb : map[account:admin accountid:9b2313ee-b4ae-11f0-914b-bc2411c796cd cmd:org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd completed:2025-11-06T08:23:19+0530 created:2025-11-06T08:23:19+0530 domainid:abbdc016-b4ad-11f0-914b-bc2411c796cd domainpath:ROOT jobid:acee7bf9-c308-4929-87e5-42f74652d5eb jobprocstatus:0 jobresult:map[errorcode:530 errortext:Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!] jobresultcode:530 jobresulttype:object jobstatus:2 userid:80acdbf0-fe5c-4ed1-a5ea-4d8e9fe7307a]
I1106 02:53:29.598232 1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"kube-system", Name:"cluster-autoscaler-status", UID:"ad1efd21-9a52-4226-9c1d-15396820bba5", APIVersion:"v1", ResourceVersion:"205055", FieldPath:""}): type: 'Warning' reason: 'FailedToScaleUpGroup' Scale-up failed for group da626491-7457-4106-a165-de0d41d2417a: Unable to scale cluster : API failed for job acee7bf9-c308-4929-87e5-42f74652d5eb : map[account:admin accountid:9b2313ee-b4ae-11f0-914b-bc2411c796cd cmd:org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd completed:2025-11-06T08:23:19+0530 created:2025-11-06T08:23:19+0530 domainid:abbdc016-b4ad-11f0-914b-bc2411c796cd domainpath:ROOT jobid:acee7bf9-c308-4929-87e5-42f74652d5eb jobprocstatus:0 jobresult:map[errorcode:530 errortext:Scaling Kubernetes cluster : CL8S failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!] jobresultcode:530 jobresulttype:object jobstatus:2 userid:80acdbf0-fe5c-4ed1-a5ea-4d8e9fe7307a]
I1106 02:53:29.613991 1 metrics.go:380] Function main took 10.194459637s to complete
I1106 02:53:30.318148 1 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.StatefulSet total 8 items received
I1106 02:53:36.016083 1 reflector.go:255] Listing and watching *v1beta1.CSIStorageCapacity from k8s.io/client-go/informers/factory.go:134
W1106 02:53:36.018008 1 reflector.go:324] k8s.io/client-go/informers/factory.go:134: failed to list *v1beta1.CSIStorageCapacity: the server could not find the requested resource
E1106 02:53:36.018094 1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CSIStorageCapacity: failed to list *v1beta1.CSIStorageCapacity: the server could not find the requested resource
I1106 02:53:36.329876 1 reflector.go:536] k8s.io/client-go/informers/factory.go:134: Watch close - *v1.PodDisruptionBudget total 11 items received
I1106 02:53:39.614541 1 static_autoscaler.go:235] Starting main loop
I1106 02:53:39.614764 1 client.go:169] NewAPIRequest API request URL:http://142.79.253.232:8080/client/api?apiKey=***&command=listKubernetesClusters&id=da626491-7457-4106-a165-de0d41d2417a&response=json&signature=***
I1106 02:53:39.679225 1 client.go:175] NewAPIRequest response status code:200
@weizhouapache that check was for scaling the cluster with a new offering. It shouldn't affect scaling for node count unless something changed recently. Scaling running VM may not be working on KVM at the time of #3680
@weizhouapache that check was for scaling the cluster with a new offering. It shouldn't affect scaling for node count unless something changed recently. Scaling running VM may not be working on KVM at the time of #3680
I have implemented a fix in KubernetesClusterScaleWorker.java (lines 358-389) to address issues with cluster scaling. Updated code is as follows.
privatevoidscaleKubernetesClusterOffering(KubernetesClusterNodeTypenodeType, ServiceOfferingserviceOffering,
booleanupdateNodeOffering, booleanupdateClusterOffering)
throwsCloudRuntimeException {
// Do not validate for clusters in Created state (no VMs exist yet to upgrade).// Validation will be done later only if we are actually upgrading existing VMs.List<KubernetesCluster.State> scalingStates =
List.of(KubernetesCluster.State.Scaling, KubernetesCluster.State.ScalingStoppedCluster);
if (!scalingStates.contains(kubernetesCluster.getState())) {
stateTransitTo(kubernetesCluster.getId(), KubernetesCluster.Event.ScaleUpRequested);
}
if (KubernetesCluster.State.Created.equals(originalState)) {
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering,
updateNodeOffering, updateClusterOffering);
return;
}
finallongsize = getNodeCountForType(nodeType, kubernetesCluster);
ListvmList =
kubernetesClusterVmMapDao.listByClusterIdAndVmType(kubernetesCluster.getId(), nodeType);
finallongtobeScaledVMCount = Math.min(vmList.size(), size);
// Only validate hypervisor compatibility when upgrading existing VMs.// Horizontal scaling (adding new nodes) on KVM must not be blocked here.if (tobeScaledVMCount > 0) {
validateKubernetesClusterScaleOfferingParameters();
}
for (longi = 0; i < tobeScaledVMCount; i++) {
KubernetesClusterVmMapVOvmMapVO = vmList.get((int) i);
UserVmVOuserVM = userVmDao.findById(vmMapVO.getVmId());
booleanresult = false;
try {
result = userVmManager.upgradeVirtualMachine(userVM.getId(), serviceOffering.getId(),
newHashMap<String, String>());
} catch (RuntimeException | ResourceUnavailableException | ManagementServerException
| VirtualMachineMigrationExceptione) {
logTransitStateAndThrow(Level.ERROR,
String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s due to %s",
kubernetesCluster.getName(), userVM.getDisplayName(), e.getMessage()),
kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed, e);
}
if (!result) {
logTransitStateAndThrow(Level.WARN,
String.format("Scaling Kubernetes cluster : %s failed, unable to scale cluster VM : %s",
kubernetesCluster.getName(), userVM.getDisplayName()),
kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
if (System.currentTimeMillis() > scaleTimeoutTime) {
logTransitStateAndThrow(Level.WARN,
String.format("Scaling Kubernetes cluster : %s failed, scaling action timed out",
kubernetesCluster.getName()),
kubernetesCluster.getId(), KubernetesCluster.Event.OperationFailed);
}
}
kubernetesCluster = updateKubernetesClusterEntryForNodeType(null, nodeType, serviceOffering,
updateNodeOffering, updateClusterOffering);
}
Status:
Confirmed: Scale-up and scale-down are working correctly on KVM hypervisors.
Pending: Testing on other hypervisors is still required.
Testing Instructions
If you would like to test this fix, I have attached a patched cloudstack-4.22.0.0.jar file. You can apply it to your CloudStack Management Server using the following steps:
Stop the CloudStack management service.
Backup your existing JAR file: cp /usr/share/cloudstack-management/lib/cloudstack-4.22.0.0.jar /usr/share/cloudstack-management/lib/cloudstack-4.22.0.0.jar.bak
Replace the existing JAR file with the provided patched version.
This is the reason you are hitting the exception
Scaling Kubernetes cluster : test failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!"
This is the reason you are hitting the exception Scaling Kubernetes cluster : test failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!"
@kiranchavala
I totally agree with you that scaling vm with fixed offering is not supported in KVM.
however, when scale a K8S, scaling often means the provision of new VMs.
I am not sure if the inconsistent service offerings (e.g. user specify the service offering when scale Kubernetes cluster) of existing K8S nodes should block the scaling. IMHO, it should apply only on the new VMs.
@kiranchavala I totally agree with you that scaling vm with fixed offering is not supported in KVM.
however, when scale a K8S, scaling often means the provision of new VMs. I am not sure if the inconsistent service offerings (e.g. user specify the service offering when scale Kubernetes cluster) of existing K8S nodes should block the scaling. IMHO, it should apply only on the new VMs.
@weizhouapache I believe this behaviour has been historic. If one attempts to change the offering while changing the cluster size, it changes the offering on the pre-existing nodes as well (not just the new nodes that get added) - Tested this on 4.20.2
When implementing #12975, I forgot to remove this constraint in the Kubernetes cluster scaling workflow that prevented changing cluster compute offerings while they were running. I've just opened the PR #13226 to address that.
autoscaling of Kubernetes clusters on KVM fails when more than a single service offering is used in a cluster.
The cluster-autoscaler calls ScaleKubernetesClusterCmd with only a new clustersize (no offering). createNodeTypeToServiceOfferingMap() injects a single global DEFAULT, and scaleCluster() then compares every node type's real VM offering against that global default instead of against that node type's own stored offering (control_node_service_offering_id / worker_node_service_offering_id).
This leads to a code path were scaleKubernetesClusterOffering() -> validateKubernetesClusterScaleOfferingParameters() are called, subsequently triggering the hypervisor type check which then fails and leads to the observed error.
publicbooleanscaleCluster() throwsCloudRuntimeException {
...
for (KubernetesClusterNodeTypenodeType : Arrays.asList(CONTROL, ETCD, WORKER)) {
...
// This is were things go wrongbooleanisNodeOfferingScalingNeeded = isServiceOfferingScalingNeededForNodeType(existingServiceOffering, scalingServiceOffering);
...
if (isNodeOfferingScalingNeeded && clusterSizeScalingNeeded) {
// With more than one service offering we end up in this branchif (newVMRequired > 0) {
scaleKubernetesClusterOffering(nodeType, scalingServiceOffering, updateNodeOffering, updateClusterOffering);
scaleKubernetesClusterSize(nodeType);
} else {
scaleKubernetesClusterSize(nodeType);
scaleKubernetesClusterOffering(nodeType, scalingServiceOffering, updateNodeOffering, updateClusterOffering);
}
} elseif (isNodeOfferingScalingNeeded) {
scaleKubernetesClusterOffering(nodeType, scalingServiceOffering, updateNodeOffering, updateClusterOffering);
} elseif (clusterSizeScalingNeeded) {
scaleKubernetesClusterSize(nodeType); // We should end up here instead!
}
}
...
}
Activity
Jayd603 commentedon Nov 5, 2025
I can't imagine why scaling k8s (by adding nodes right?) would not work on KVM. That sounds ridiculous. Now, resource scaling of existing nodes, maybe. I am actually bummed because I upgraded 4.19 to 4.21 not realizing it was a new type of release (non-LTS) , and it had broken kubernetes management but hey, so far everything else works great so it just needs some k8s fixes.
kiranchavala commentedon Nov 6, 2025
@amitmvyas
Did you enable autoscaling on a kubernetes cluster which is in running state
Is the autoscaler pod running in the kubernetes cluster , can you provide the logs from the autoscaler pod
Example
k logs -f cluster-autoscaler-6d469c6c6-x6f9m -n kube-system
amitmvyas commentedon Nov 6, 2025
Tried with CMK as well
This is the logs for cluster-autoscaler.
I have attached entire logs as well.
cluster-autoscaler-744b9d65b-zzw2p.log
weizhouapache commentedon Nov 6, 2025
the whole block was introduced in the original PR #3680
cloudstack/plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java
Lines 316 to 324 in dbda673
cc @shwstppr
do you remember why KVM is not supported ?
shwstppr commentedon Nov 6, 2025
@weizhouapache that check was for scaling the cluster with a new offering. It shouldn't affect scaling for node count unless something changed recently. Scaling running VM may not be working on KVM at the time of #3680
weizhouapache commentedon Nov 6, 2025
thanks @shwstppr , makes sense.
so the issue could be related to #9102. cc @nvazquez @Pearl1594
chandrakantrai commentedon Jan 8, 2026
I have implemented a fix in KubernetesClusterScaleWorker.java (lines 358-389) to address issues with cluster scaling. Updated code is as follows.
Status:
Confirmed: Scale-up and scale-down are working correctly on KVM hypervisors.
Pending: Testing on other hypervisors is still required.
Testing Instructions
If you would like to test this fix, I have attached a patched cloudstack-4.22.0.0.jar file. You can apply it to your CloudStack Management Server using the following steps:
cp /usr/share/cloudstack-management/lib/cloudstack-4.22.0.0.jar /usr/share/cloudstack-management/lib/cloudstack-4.22.0.0.jar.bakLink for Patched jar file
https://github.com/chandrakantrai/cloudstack4.22.0-Patch/blob/main/cloudstack-4.22.0.0.jar
@weizhouapache, thank you for pointing toward the relevant code section.
sureshanaparti commentedon Feb 2, 2026
Hi @chandrakantrai can you raise PR (against 4.22) with these changes? cc @weizhouapache
kiranchavala commentedon Feb 2, 2026
@chandrakantrai Currently the issue is not reproducible on 4.22 release
The API call with the size parameter is working fine
cmk scale kubernetescluster id=<> size=2
dynamic change of service offering of a running cks cluster or vm is not supported for kvm hypervisor
https://docs.cloudstack.apache.org/en/4.22.0.0/adminguide/virtual_machines.html#cpu-and-memory-scaling-for-running-instances
This is the reason you are hitting the exception
Scaling Kubernetes cluster : test failed, scaling Kubernetes cluster with running VMs on hypervisor KVM is not supported!"
cc @weizhouapache @sureshanaparti
weizhouapache commentedon Feb 2, 2026
@kiranchavala
I totally agree with you that scaling vm with fixed offering is not supported in KVM.
however, when scale a K8S, scaling often means the provision of new VMs.
I am not sure if the inconsistent service offerings (e.g. user specify the service offering when scale Kubernetes cluster) of existing K8S nodes should block the scaling. IMHO, it should apply only on the new VMs.
kiranchavala commentedon Feb 2, 2026
Thanks @weizhouapache
There is an existing improvement request that I logged to support Custom compute offerings for CKS deployments
#11545
@abh1sar @sureshanaparti @rajujith Can we consider for the upcoming releases ?
Pearl1594 commentedon Mar 9, 2026
@weizhouapache I believe this behaviour has been historic. If one attempts to change the offering while changing the cluster size, it changes the offering on the pre-existing nodes as well (not just the new nodes that get added) - Tested this on 4.20.2
Fix K8s autoscale on KVM and use worker node offering for capacity pl…
shwstppr commentedon May 18, 2026
Just mentioning here as I read issue could be with scaling with fixed offering on KVM, #12975
kiranchavala commentedon May 18, 2026
Thanks @shwstppr
I think we can add a note in the documentation of the kubernetes page
Cc @sudo87 @Pearl1594
bernardodemarco commentedon May 22, 2026
Hello, all
When implementing #12975, I forgot to remove this constraint in the Kubernetes cluster scaling workflow that prevented changing cluster compute offerings while they were running. I've just opened the PR #13226 to address that.
fdegner commentedon Jul 10, 2026
Good morning,
autoscaling of Kubernetes clusters on KVM fails when more than a single service offering is used in a cluster.
The cluster-autoscaler calls
ScaleKubernetesClusterCmdwith only a newclustersize(no offering).createNodeTypeToServiceOfferingMap()injects a single globalDEFAULT, andscaleCluster()then compares every node type's real VM offering against that global default instead of against that node type's own stored offering (control_node_service_offering_id/worker_node_service_offering_id).This leads to a code path were
scaleKubernetesClusterOffering() -> validateKubernetesClusterScaleOfferingParameters()are called, subsequently triggering the hypervisor type check which then fails and leads to the observed error.plugins/integrations/kubernetes-service/src/main/java/com/cloud/kubernetes/cluster/actionworkers/KubernetesClusterScaleWorker.java