error in cloudstack agent

[arpanbht]

i have setup a wireguard vpn between two of my networks and it is working 192.168.1.0/24 network can connect to 192.168.2.0/24

and i have setup a mgmt server on 192.168.1.102 and i have added a host from .1 network and it worked but while adding a .2 network host , i created a separate pod with a .2 network config but in the agent logs i am getting this handshake error

user@pc-2:~$ sudo tail -f /var/log/cloudstack/agent/agent.log
        at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:477)
        at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:454)
        at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:438)
        at com.cloud.agent.AgentShell.start(AgentShell.java:539)
        at com.cloud.agent.AgentShell.main(AgentShell.java:569)
Caused by: java.io.IOException: SSL Handshake failed while connecting to host: 192.168.1.102:8250
        at com.cloud.utils.nio.NioClient.init(NioClient.java:75)
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:109)
        ... 6 more

2026-03-31 13:08:51,112 INFO  [cloud.agent.Agent] (main:[]) (logid:) Connecting to host: 192.168.1.102
2026-03-31 13:08:51,112 INFO  [utils.nio.NioClient] (main:[]) (logid:) Connecting to 192.168.1.102:8250
2026-03-31 13:08:51,115 INFO  [utils.nio.NioClient] (main:[]) (logid:) Connected to 192.168.1.102:8250
2026-03-31 13:08:51,115 INFO  [utils.nio.Link] (main:[]) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2026-03-31 13:08:51,116 WARN  [utils.nio.Link] (main:[]) (logid:) Failed to load keystore, using trust all manager
2026-03-31 13:08:51,138 ERROR [utils.nio.Link] (main:[]) (logid:) SSL error caught during unwrap data: Received fatal alert: bad_certificate, for local address=/192.168.2.101:44202, remote address=/192.168.1.102:8250. The client may have invalid ca-certificates.
2026-03-31 13:08:51,138 ERROR [utils.nio.NioClient] (main:[]) (logid:) IOException while connecting to 192.168.1.102:8250 java.io.IOException: SSL Handshake failed while connecting to host: 192.168.1.102:8250
        at com.cloud.utils.nio.NioClient.init(NioClient.java:75)
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:109)
        at com.cloud.agent.Agent.start(Agent.java:338)
        at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:477)
        at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:454)
        at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:438)
        at com.cloud.agent.AgentShell.start(AgentShell.java:539)
        at com.cloud.agent.AgentShell.main(AgentShell.java:569)

2026-03-31 13:08:51,138 ERROR [utils.nio.NioClient] (main:[]) (logid:) Unable to initialize the threads. java.io.IOException: SSL Handshake failed while connecting to host: 192.168.1.102:8250
        at com.cloud.utils.nio.NioClient.init(NioClient.java:75)
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:109)
        at com.cloud.agent.Agent.start(Agent.java:338)
        at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:477)
        at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:454)
        at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:438)
        at com.cloud.agent.AgentShell.start(AgentShell.java:539)
        at com.cloud.agent.AgentShell.main(AgentShell.java:569)

2026-03-31 13:08:51,138 INFO  [utils.nio.NioClient] (main:[]) (logid:) NioClient connection closed
2026-03-31 13:08:51,138 INFO  [cloud.agent.Agent] (main:[]) (logid:) Attempted to connect to the server, but received an unexpected exception, trying again... com.cloud.utils.exception.NioConnectionException: SSL Handshake failed while connecting to host: 192.168.1.102:8250
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:115)
        at com.cloud.agent.Agent.start(Agent.java:338)
        at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:477)
        at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:454)
        at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:438)
        at com.cloud.agent.AgentShell.start(AgentShell.java:539)
        at com.cloud.agent.AgentShell.main(AgentShell.java:569)
Caused by: java.io.IOException: SSL Handshake failed while connecting to host: 192.168.1.102:8250
        at com.cloud.utils.nio.NioClient.init(NioClient.java:75)
        at com.cloud.utils.nio.NioConnection.start(NioConnection.java:109)
        ... 6 more

but i can easily ping from this host or even connect to the mgmmt server on 8250

user@pc-2:~$ nc -vz 192.168.1.102 8250
Connection to 192.168.1.102 8250 port [tcp/*] succeeded!
user@pc-2:~$ ping 192.168.1.102
PING 192.168.1.102 (192.168.1.102) 56(84) bytes of data.
64 bytes from 192.168.1.102: icmp_seq=1 ttl=62 time=1.81 ms
64 bytes from 192.168.1.102: icmp_seq=2 ttl=62 time=1.58 ms
^C
--- 192.168.1.102 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 1.583/1.695/1.807/0.112 ms

[sureshanaparti]

@arpanbht Can you set auth strictness global setting 'ca.plugin.root.auth.strictness' to false and restart the KVM agent. Once connected, for secure connections you can

• Provision the host certificates
• Set global setting 'ca.plugin.root.auth.strictness' back to true
• Restart KVM agent

[arpanbht]

I have tried doing that at first setting it false causing the host to go into unsecure state but after i created another zone inside that zone I am able to add that host and its state is up and running.

Now these are the current situation:

1. ca.plugin.root.auth.strictness set to false
2. kvm agent is added to another zone.
3. system vms are getting created on zone 2 and the state is up and running.
4. able to access the consoles of those system vms both cpvm and ssvm
5. able to create user vms and we can access it.

Now i need help about migrating these vms over the zone because during any failure we need to migrate it to somewhere in the running system.

[sureshanaparti]

@arpanbht I think, you cannot migrate them over the zone. You can disable zone 2, and destroy both cpvm and ssvm to re-create them on any enabled zones and then enable zone 2.

[NikhilGade07]

CloudStack Agent SSL Handshake Failure Across WireGuard VPN
This is a certificate trust issue, not a network connectivity problem. The TCP connection works fine (as confirmed by nc), but the CloudStack agent on the .2 network host is failing the SSL/TLS handshake because it doesn't trust the management server's certificate.

The key log line tells the whole story:

Failed to load keystore, using trust all manager
...
Received fatal alert: bad certificate — The client may have invalid ca-certificates.
Root Cause
When you added the .1 network host, Cloud Stack's management server provisioned that agent with its SSL certificates. The .2 network host is in a separate pod with a fresh agent installation — it's missing the Cloud Stack management server's CA certificate in its keystore (/etc/cloudstack/agent/).

Fix
Option 1 — Copy certificates from a working .1 host (quickest)
On your working .1 host, grab the agent keystore:
bashls /etc/cloudstack/agent/

Look for cloud.jks or similar keystore files

Copy cloud.jks (and any .cer / .crt files) to the .2 host at the same path, then restart the agent:
bashsudo systemctl restart cloudstack-agent
Option 2 — Re-run agent setup via management server
On the .2 host, clear the existing keystore and let CloudStack re-provision it:
bashsudo rm -f /etc/cloudstack/agent/cloud.jks
sudo cloudstack-setup-agent

Point it to 192.168.1.102 when prompted

Option 3 — Manual certificate import
Export the mgmt server cert and import it into the agent's truststore:
bash# On management server
keytool -export -alias cloud -keystore /etc/cloudstack/management/cloud.jks
-file cloud-mgmt.cer -storepass vmops

Copy cloud-mgmt.cer to the .2 host, then:

sudo keytool -import -alias cloud -file cloud-mgmt.cer
-keystore /etc/cloudstack/agent/cloud.jks -storepass vmops -noprompt

sudo systemctl restart cloudstack-agent