Skip to content

CKS cluster doesn't come up if "cloud.kubernetes.cluster.experimental.features.enabled" setting is enabled #11596

Open
Bug
Open
CKS cluster doesn't come up if "cloud.kubernetes.cluster.experimental.features.enabled" setting is enabled#11596
Bug
Labels
component:ckstype:bug
Milestone
4.20.4

Description

@kiranchavala
kiranchavala
opened on Sep 8, 2025

problem

CKS cluster doesn't come up if "cloud.kubernetes.cluster.experimental.features.enabled" setting is enabled

versions

ACS 4.20.1, 4.21

The steps to reproduce the bug

  1. Enable the global setting "cloud.kubernetes.cluster.experimental.features.enabled" setting

  2. Launch a cks cluster, enable private registry and provide your docker hub credentials

Image

dockerregistryurl= "https://index.docker.io/v1/"
dockerregistryusername= <>
dockerregistrypassword=<>

  1. CKS cluster is struck in starting state

  2. On logging into control node found the following errors

root@example-cluster-control-19927ba5b6e:/opt/bin# crictl images
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/opt/bin/crictl.yaml"
WARN[0000] Image connect using default endpoints: [unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///run/containerd/containerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///run/crio/crio.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/crio/crio.sock: connect: no such file or directory"
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: no such file or directory"
FATA[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/cri-dockerd.sock: connect: no such file or directory"

kublet logs


Sep 08 05:18:03 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 08 05:18:03 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 08 05:18:13 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 41.
Sep 08 05:18:13 example-cluster-control-19927ba5b6e systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 08 05:18:13 example-cluster-control-19927ba5b6e systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 08 05:18:14 example-cluster-control-19927ba5b6e kubelet[31660]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's -->
Sep 08 05:18:14 example-cluster-control-19927ba5b6e kubelet[31660]: E0908 05:18:14.034670   31660 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubele>
Sep 08 05:18:14 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 08 05:18:14 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 08 05:18:24 example-cluster-control-19927ba5b6e systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 42.
Sep 08 05:18:24 example-cluster-control-19927ba5b6e systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 08 05:18:24 example-cluster-control-19927ba5b6e systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 08 05:18:24 example-cluster-control-19927ba5b6e kubelet[32393]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's -->
Sep 08 05:18:24 example-cluster-control-19927ba5b6e kubelet[32393]: E0908 05:18:24.237711   32393 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubele>

The file

"/etc/containerd/config.toml" is not populated correctly

What to do about it?

Cloudstack should support private container registry

Activity

  1. kiranchavala
    added
    type:bug
    component:cks
    on Sep 8, 2025
  2. DaanHoogland
    added this to Apache CloudStack BugFest - Issueson Sep 8, 2025
  3. DaanHoogland
    added the
    Bug
    issue type on Sep 8, 2025
  4. DaanHoogland
    added this to the 4.20.2 milestone on Sep 8, 2025

  5. weizhouapache commented on Sep 8, 2025

    @weizhouapache
    weizhouapache
    on Sep 8, 2025
    Member

    @kiranchavala
    can you share the kubelet config in /var/lib/kubelet ?

  6. kiranchavala commented on Sep 9, 2025

    @kiranchavala
    kiranchavala
    on Sep 9, 2025
    MemberAuthor

    @weizhouapache there is no file /var/lib/kubelet and didn't generate

    
Sep 09 05:51:19 test-private-control-1992cfb0b17 kubelet[50328]: E0909 05:51:19.086110   50328 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Sep 09 05:51:19 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 05:51:19 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 09 05:51:29 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 67.
Sep 09 05:51:29 test-private-control-1992cfb0b17 systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 09 05:51:29 test-private-control-1992cfb0b17 systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 09 05:51:29 test-private-control-1992cfb0b17 kubelet[51068]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 05:51:29 test-private-control-1992cfb0b17 kubelet[51068]: E0909 05:51:29.237105   51068 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Sep 09 05:51:29 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 05:51:29 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Sep 09 05:51:39 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 68.
Sep 09 05:51:39 test-private-control-1992cfb0b17 systemd[1]: Stopped kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 09 05:51:39 test-private-control-1992cfb0b17 systemd[1]: Started kubelet.service - kubelet: The Kubernetes Node Agent.
Sep 09 05:51:39 test-private-control-1992cfb0b17 kubelet[51779]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 05:51:39 test-private-control-1992cfb0b17 kubelet[51779]: E0909 05:51:39.373116   51779 run.go:72] "command failed" err="failed to load kubelet config file, path: /var/lib/kubelet/config.yaml, error: failed to load Kubelet config file /var/lib/kubelet/config.yaml, error failed to read kubelet config file \"/var/lib/kubelet/config.yaml\", error: open /var/lib/kubelet/config.yaml: no such file or directory"
Sep 09 05:51:39 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 05:51:39 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Failed with result 'exit-code'


    
root@test-private-control-1992cfb0b17:/var/lib/containerd# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: activating (auto-restart) (Result: exit-code) since Tue 2025-09-09 05:50:58 UTC; 5s ago
       Docs: http://kubernetes.io/docs/
    Process: 48881 ExecStart=/opt/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code>
   Main PID: 48881 (code=exited, status=1/FAILURE)
        CPU: 99ms

Sep 09 05:50:58 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 05:50:58 test-private-control-1992cfb0b17 systemd[1]: kubelet.service: Failed with result 'exit-code'.
  7. DaanHoogland
    moved this to Todo in Apache CloudStack BugFest - Issueson Sep 18, 2025
  8. weizhouapache
    modified the milestones: 4.20.2, 4.20.3 on Oct 23, 2025
  9. abh1sar
    modified the milestones: 4.20.3, 4.20.4 on Feb 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    component:ckstype:bug

    Type

    Bug

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions