The cloudstack-agent can not work 4.18.1.0

[clustspace]

ISSUE TYPE

• Bug Report

COMPONENT NAME

• 4.18.1.0

CLOUDSTACK VERSION

1. Cloudstack management (fresh install)

• OS: Ubuntu 22.04
• Cloudstack-management: 4.18.1.0

2. Cloudstack agent (fresh install)

• OS: Ubuntu 22.04
• Cloudstack-agent: 4.18.1.0

SUMMARY

After installing cloudstack-agent it won't start.

Nov 12 22:25:13 dg42a-csrv-3 systemd[1]: Stopped CloudStack Agent.
░░ Subject: A stop job for unit cloudstack-agent.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit cloudstack-agent.service has finished.
░░
░░ The job identifier is 28655 and the job result is done.
Nov 12 22:25:13 dg42a-csrv-3 systemd[1]: cloudstack-agent.service: Consumed 2.244s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit cloudstack-agent.service completed and consumed the indicated resources.
Nov 12 22:25:13 dg42a-csrv-3 systemd[1]: Started CloudStack Agent.
░░ Subject: A start job for unit cloudstack-agent.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit cloudstack-agent.service has finished successfully.
░░
░░ The job identifier is 28655.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: log4j:WARN No appenders could be found for logger (com.cloud.agent.AgentShell).
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: log4j:WARN Please initialize the log4j system properly.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: INFO  [cloud.agent.AgentShell] (main:) (logid:) Agent started
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: INFO  [cloud.agent.AgentShell] (main:) (logid:) Implementation Version is 4.18.1.0
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: INFO  [cloud.agent.AgentShell] (main:) (logid:) agent.properties found at /etc/cloudstack/agent/agent.properties
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: SLF4J: Defaulting to no-operation (NOP) logger implementation
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Nov 12 22:25:14 dg42a-csrv-3 java[11473]: Unable to find the guid
Nov 12 22:25:14 dg42a-csrv-3 systemd[1]: cloudstack-agent.service: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit cloudstack-agent.service has successfully entered the 'dead' state.
Nov 12 22:25:14 dg42a-csrv-3 systemd[1]: cloudstack-agent.service: Consumed 2.239s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit cloudstack-agent.service completed and consumed the indicated resources.

Error log from cloudstack-management server

2023-11-13 00:34:31,480 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp1278852808-536:ctx-a43770e7 ctx-30b97ea5) (logid:4785c123) Exception:
com.cloud.exception.DiscoveryException: Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:823)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy200.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:782)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:603)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:110)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        ... 59 more
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        ... 60 more
2023-11-13 00:34:31,482 INFO  [c.c.a.ApiServer] (qtp1278852808-536:ctx-a43770e7 ctx-30b97ea5) (logid:4785c123) Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[rajujith]

@clustspace could you verify that the KVM host has the required dependencies installed like libvirt ,KVM etc? If thats good, is this KVM added to Cloudstack? try adding it.

[clustspace]

kvm --version
QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.15)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

libvirt-daemon-system is already the newest version (8.0.0-1ubuntu7.7).

virsh # version
Compiled against library: libvirt 8.0.0
Using library: libvirt 8.0.0
Using API: QEMU 8.0.0
Running hypervisor: QEMU 6.2.0

[weizhouapache]

@clustspace
this could be caused by some configurations (e.g. network) on the kvm host.
can you upload

• /var/log/cloudstack/agent/setup.log on kvm host
• /var/log/cloudstack/management/management-server.log on the management server during the process of AddHostCmd

[clustspace]

@weizhouapache there is no setup.log on kvm host

Log from management-server.log in first post

2023-11-13 00:34:31,480 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp1278852808-536:ctx-a43770e7 ctx-30b97ea5) (logid:4785c123) Exception:
com.cloud.exception.DiscoveryException: **Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host:** 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].

[weizhouapache]

@weizhouapache there is no setup.log on kvm host

Log from management-server.log in first post

2023-11-13 00:34:31,480 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp1278852808-536:ctx-a43770e7 ctx-30b97ea5) (logid:4785c123) Exception:
com.cloud.exception.DiscoveryException: **Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host:** 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].

@clustspace
can you enable DEBUG log level on management server and retry to add the host, it will give more details ?

sed -i "s/INFO/DEBUG/g" /etc/cloudstack/management/log4j-cloud.xml

You need to restart management server to make it effective.

[clustspace]

@weizhouapache same error

2023-11-13 12:19:34,915 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34
2023-11-13 12:19:34,915 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34
com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy199.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:782)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:603)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:110)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
2023-11-13 12:19:34,918 DEBUG [c.c.r.ResourceManagerImpl] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b) Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy199.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:782)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:603)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:110)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        ... 60 more
2023-11-13 12:19:34,920 INFO  [c.c.u.e.CSExceptionErrorCode] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
2023-11-13 12:19:34,921 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b) Exception:
com.cloud.exception.DiscoveryException: Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:823)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy199.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:782)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:603)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:110)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:649)
        ... 59 more
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)
        ... 60 more
2023-11-13 12:19:34,924 INFO  [c.c.a.ApiServer] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b) Could not add host at [http://10.64.11.34] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 10.64.11.34 - Failed to setup keystore on the KVM host: 10.64.11.34].
2023-11-13 12:19:34,925 DEBUG [c.c.a.ApiServlet] (qtp1278852808-20:ctx-3a025a3a ctx-8b551f7c) (logid:02bf9c8b) ===END===  185.122.140.1 -- POST  command=addHost&response=json
2023-11-13 12:19:35,132 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-e0f3971a) (logid:f2604127) Resetting hosts suitable for reconnect
2023-11-13 12:19:35,135 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-e0f3971a) (logid:f2604127) Completed resetting hosts suitable for reconnect
2023-11-13 12:19:35,135 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-e0f3971a) (logid:f2604127) Acquiring hosts for clusters already owned by this management server
2023-11-13 12:19:35,137 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-e0f3971a) (logid:f2604127) Completed acquiring hosts for clusters already owned by this management server
2023-11-13 12:19:35,137 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-e0f3971a) (logid:f2604127) Acquiring hosts for clusters not owned by any management server
2023-11-13 12:19:35,138 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-e0f3971a) (logid:f2604127) Completed acquiring hosts for clusters not owned by any management server
2023-11-13 12:19:37,354 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-658f94c5) (logid:9f33176f) HA health check task is running...
2023-11-13 12:19:40,490 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-eaafffc6) (logid:f40a1484) Enabled non-edge zones available for scan: 1
2023-11-13 12:19:40,494 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-eaafffc6) (logid:f40a1484) Zone 1 has no host available which is enabled and in Up state
2023-11-13 12:19:40,494 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-eaafffc6) (logid:f40a1484) Zone [1] is not ready to launch secondary storage VM.
2023-11-13 12:19:40,590 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-4e5a015e) (logid:2ce1a8da) Skip capacity scan as there is no Primary Storage in 'Up' state
2023-11-13 12:19:41,356 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-7597a4b3) (logid:a8227727) HA health check task is running...
2023-11-13 12:19:45,359 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-6fedacea) (logid:2d831233) HA health check task is running...
2023-11-13 12:19:49,361 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-9e6e5d21) (logid:8bb8ea01) HA health check task is running...

First of all, we can't start cloudstack-agent on host server

[tampler]

Having exactly the same error on the same setup. ACS 4.18.1.0 on Ubuntu 22.04.3.

For me this happens for a non-root user (admin). Here's my setup for a visudo list:

Defaults:cloud !requiretty
Defaults:cloudstack !requiretty
cloudstack ALL=NOPASSWD: /usr/bin/cloudstack-setup-agent

Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.0.20                                                           
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)                                                  
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)                                                                
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:816)          

PS: Seems like an agent miscofing on my side. That's what I see in the agent log:

Caused by: java.io.IOException: SSL Handshake failed while connecting to host: acs port: 8250
        at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)

[tampler]

Looks like #6716 is back

[weizhouapache]

Looks like #6716 is back

@tampler
It looks they are different.
Although the error logs show that there are exception "Failed to setup keystore", the root cause seems not to be related to keystore, but others like network configurations, privileges, etc.

It would be good to grant the non-root user stronger privileges, if you do not use root when add a host.

[tampler]

There are few caveats in installing KVM on Ubuntu 22.04. Those are not yet covered in the project documentation.
One is an ability for an agent host to register itself within the mgmt host, which is done somewhat counter intuitively. To resolve your issue, do this:

1. Update libvirt configs, following the doc
2. On the agent host: sudo nano /etc/cloudstack/agent/agent.properties and set proper values for UUID, host and bridges
3. Copy cloudstack generated ssh key to the authorized_keys on a mgmt host:

cat id_rsa.pub.cloud >> /root/.ssh/authorized_keys

4. Restart libvirt and agent and make sure they started properly:

sudo systemctl restart libvirtd.service
sudo systemctl restart cloudstack-agent.service

5. If everything is ok, switch to the mgmt service UI config. When you get to adding the host, specify for root user and click the SSH Public Key Access to login with the autogenerated key.

NOTE: You'll probably face another errors, which you would report here or open another issue 😏😏😏

[tampler]

That's a config issue for Ubuntu 22.04. Related to #8218

[weizhouapache]

There are few caveats in installing KVM on Ubuntu 22.04. Those are not yet covered in the project documentation. One is an ability for an agent host to register itself within the mgmt host, which is done somewhat counter intuitively. To resolve your issue, do this:

1. Update libvirt configs, following the doc
2. On the agent host: sudo nano /etc/cloudstack/agent/agent.properties and set proper values for UUID, host and bridges
3. Copy cloudstack generated ssh key to the authorized_keys on a mgmt host:

cat id_rsa.pub.cloud >> /root/.ssh/authorized_keys

4. Restart libvirt and agent and make sure they started properly:

sudo systemctl restart libvirtd.service
sudo systemctl restart cloudstack-agent.service

5. If everything is ok, switch to the mgmt service UI config. When you get to adding the host, specify for root user and click the SSH Public Key Access to login with the autogenerated key.

NOTE: You'll probably face another errors, which you would report here or open another issue 😏😏😏

yes @tampler
If you do not want to use root password, ssh key is a good alternative. it just works.

If you use a non-root user, /etc/sudoers should be configured. I suspect the following configuration is correct for non-root users.

cloudstack ALL=NOPASSWD: /usr/bin/cloudstack-setup-agent
Defaults:cloudstack !requiretty

https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html#install-and-configure-the-agent