OOBM Issue during KVM HA

[muthukrishnang1100]

Hi, I have an issue. I am trying to Test KVM HA test on my 3 host cloudstack. When I enable the OOBM on any one of the host 5 to minutes later the error comes. and after that node not able to login dashboard. Please anyone face this isuue if please what i do for this issue? and also after i disable and restart the Cloudstack management it will works fine no error.

Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:147)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
... 30 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:356)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:141)
... 32 more
2025-12-30 12:51:49,231 ERROR [c.c.s.StatsCollector] (StatsCollector-1:ctx-2c2139dd) (logid:e41447a0) Error trying to monitor autoscaling
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:433)
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:362)
at com.cloud.utils.db.GenericDaoBase.listIncludingRemovedBy(GenericDaoBase.java:960)
at com.cloud.utils.db.GenericDaoBase.listBy(GenericDaoBase.java:939)
at com.cloud.cluster.dao.ManagementServerHostDaoImpl.findOneInUpState(ManagementServerHostDaoImpl.java:270)
at jdk.internal.reflect.GeneratedMethodAccessor284.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy161.findOneInUpState(Unknown Source)
at com.cloud.server.StatsCollector$AutoScaleMonitor.runInContext(StatsCollector.java:1766)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:147)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
... 30 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:356)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:141)
... 32 more
2025-12-30 12:51:49,240 ERROR [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-8d4322d8) (logid:dcafa96b) Fail to complete the CheckRouterTask!
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:433)
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:362)

[ingox]

@muthukrishnang1100 Your error message from the log does not directly belong to the KVM HA setup. The CloudStack management server does not get a connection to the database. Please check your database server and connection limits.

[muthukrishnang1100]

but, this error only occurs when i enable the OOBM on the host, so why it is comes only enable the OOBM? and also this error occurs when enable OOBM on host-1 error comes on host-1 or sometimes hots-2, 3.2025-12-31 12:45:37,062 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-2:null) (logid:7aa7e2e6) Process standard output for command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.10.117 -p 623 -U ADMIN -P ***** chassis power status]: [Chassis Power is on
].
2025-12-31 12:45:37,062 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-2:null) (logid:7aa7e2e6) Process standard error output command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.10.117 -p 623 -U ADMIN -P ***** chassis power status]: [Loading IANA PEN Registry...
Using best available cipher suite 3

Running Get VSO Capabilities my_addr 0x20, transit 0, target 0x20
Invalid completion code received: Invalid command
Discovered IPMB address 0x0
].
2025-12-31 12:45:37,062 DEBUG [o.a.c.o.d.i.IpmitoolOutOfBandManagementDriver] (pool-1-thread-2:null) (logid:7aa7e2e6) The command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.10.117 -p **** -U ADMIN -P **** chassis power status] was successful and got the result [Chassis Power is on].
2025-12-31 12:45:39,070 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-5:ctx-e9676f5b) (logid:a4534a37) HA health check task is running...
2025-12-31 12:45:41,710 DEBUG [c.c.a.ApiServlet] (qtp341748265-310:ctx-fcb0e415) (logid:5e23a71d) ===START=== 10.1.12.54 -- GET command=readyForShutdown&response=json&sessionkey=5cpt1hqcFjsg3EJGa1t8rtgrwnQ
2025-12-31 12:45:41,710 DEBUG [c.c.a.ApiServlet] (qtp341748265-310:ctx-fcb0e415) (logid:5e23a71d) Two factor authentication is already verified for the user 2, so skipping

after, this error comes we cant accessible the dashboard of that node

[ingox]

@muthukrishnang1100 you may check this:
while dashboard becomes inaccessible:

• check high CPU on management server
• are there many IPMI processes running? ps -ef | grep ipmitool | wc -l

under regular circumstances check:

• run several times: time ipmitool -I lanplus -R 1 -N 3 -H 192.168.10.117 -p 623 -U ADMIN -P '*****' chassis power status

What latency do you have between management and your hypervisor?

[muthukrishnang1100]

root@test-1:# ps -ef | grep ipmitool | wc -l
1
root@test-1:# time ipmitool -I lanplus -R 1 -N 3 -H 192.168.10.117 -p 623 -U ADMIN -P '******' chassis power status
Chassis Power is on

real 0m0.055s
user 0m0.036s
sys 0m0.006s
root@test-1:~# ping -c 10 192.168.10.117
PING 192.168.10.117 (192.168.10.117) 56(84) bytes of data.
64 bytes from 192.168.10.117: icmp_seq=1 ttl=63 time=0.513 ms
64 bytes from 192.168.10.117: icmp_seq=2 ttl=63 time=0.463 ms
64 bytes from 192.168.10.117: icmp_seq=3 ttl=63 time=0.445 ms
64 bytes from 192.168.10.117: icmp_seq=4 ttl=63 time=0.470 ms
64 bytes from 192.168.10.117: icmp_seq=5 ttl=63 time=0.459 ms
64 bytes from 192.168.10.117: icmp_seq=6 ttl=63 time=0.452 ms
64 bytes from 192.168.10.117: icmp_seq=7 ttl=63 time=0.431 ms
64 bytes from 192.168.10.117: icmp_seq=8 ttl=63 time=0.433 ms
64 bytes from 192.168.10.117: icmp_seq=9 ttl=63 time=0.465 ms
64 bytes from 192.168.10.117: icmp_seq=10 ttl=63 time=0.455 ms

--- 192.168.10.117 ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9232ms
rtt min/avg/max/mdev = 0.431/0.458/0.513/0.021 ms
root@test-1:~#

I checked all was fine but still that error comes when i enable the OOBM

[ingox]

@muthukrishnang1100 Okay, then let's have a look at your DB. Please check the following values on your DB. Before and after you enable OOBM.

SHOW VARIABLES LIKE 'max_connections';
SHOW GLOBAL STATUS LIKE 'Threads_connected';
SHOW GLOBAL STATUS LIKE 'Max_used_connections';

[muthukrishnang1100]

i hjave another issue same error is comming sometimes after i resart the cloudstack management it will work. why it bhappends again and again sometimes? I attached the document of the error.

muthukrishnan - Fri Jan 02 2026 15_39_31 GMT+0530 (India Standard Time).txt

[muthukrishnang1100]

see i checked these

what i do for these issues both i shared first issue is OOBM and second issue is i am not do nothing not enable OOBM but it will db error

[ingox]

@muthukrishnang1100 you're having connection issues to your DB. OOBM is just the trigger.
Does the screenshot shows the connections after management restart or while you having issues? Please paste both.

Check for "abandoned connection" messages in your management log.

Also check and paste this while you having the issue:
SHOW FULL PROCESSLIST;
SHOW ENGINE INNODB STATUS\G

You also may want to try these in your /etc/cloudstack/management/db.properties
db.pool.maxTotal=100
db.pool.maxIdle=20
db.pool.minIdle=10
db.pool.maxWaitMillis=60000

[muthukrishnang1100]

MariaDB [(none)]> SHOW FULL PROCESSLIST;
+------+-------------+--------------+-------+---------+-------+-------------------------+-----------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+------+-------------+--------------+-------+---------+-------+-------------------------+-----------------------+----------+
| 1 | system user | | NULL | Sleep | 18150 | wsrep aborter idle | NULL | 0.000 |
| 2 | system user | | NULL | Sleep | 4 | wsrep applier committed | NULL | 0.000 |
| 5 | system user | | NULL | Sleep | 1 | wsrep applier committed | NULL | 0.000 |
| 6 | system user | | NULL | Sleep | 4 | wsrep applier committed | NULL | 0.000 |
| 7 | system user | | NULL | Sleep | 1 | wsrep applier committed | NULL | 0.000 |
| 8 | system user | | NULL | Sleep | 3 | wsrep applier committed | NULL | 0.000 |
| 9 | system user | | NULL | Sleep | 2 | wsrep applier committed | NULL | 0.000 |
| 11 | system user | | NULL | Sleep | 4 | wsrep applier committed | NULL | 0.000 |
| 13 | system user | | NULL | Sleep | 2 | wsrep applier committed | NULL | 0.000 |
| 3587 | cloud | test-1:50386 | cloud | Sleep | 16 | | NULL | 0.000 |
| 7906 | cloud | test-1:43694 | cloud | Sleep | 0 | | NULL | 0.000 |
| 8928 | cloud | test-1:51266 | cloud | Sleep | 0 | | NULL | 0.000 |
| 9081 | cloud | test-1:37516 | cloud | Sleep | 6 | | NULL | 0.000 |
| 9116 | root | localhost | NULL | Query | 0 | starting | SHOW FULL PROCESSLIST | 0.000 |
+------+-------------+--------------+-------+---------+-------+-------------------------+-----------------------+----------+
14 rows in set (0.000 sec)

MariaDB [(none)]> SHOW ENGINE INNODB STATUS\G
*************************** 1. row ***************************
Type: InnoDB
Name:
Status:

2026-01-02 19:17:54 0x757124d186c0 INNODB MONITOR OUTPUT

Per second averages calculated from the last 32 seconds

BACKGROUND THREAD

srv_master_thread loops: 0 srv_active, 0 srv_shutdown, 18019 srv_idle
srv_master_thread log flush and writes: 18019

SEMAPHORES

---

TRANSACTIONS

Trx id counter 2846999
Purge done for trx's n:o < 2846997 undo n:o < 0 state: running
History list length 1
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION (0x75710ccaa480), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x75710cca9980), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x75710cca4c80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x75710cca8e80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x75710cca8380), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x75710cca7880), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x75710cca6d80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x75710cca6280), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x75710cca5780), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x75710cca4180), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x75710cca3680), not started
0 lock struct(s), heap size 1128, 0 row lock(s)
---TRANSACTION (0x75710cca2b80), not started
0 lock struct(s), heap size 1128, 0 row lock(s)

FILE I/O

Pending flushes (fsync): 0
2876 OS file reads, 0 OS file writes, 10793 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 1.00 fsyncs/s

LOG

Log sequence number 1037264888
Log flushed up to 1037264346
Pages flushed up to 1011307633
Last checkpoint at 1011307617

BUFFER POOL AND MEMORY

Total large memory allocated 167772160
Dictionary memory allocated 1421112
Buffer pool size 8112
Free buffers 5223
Database pages 2889
Old database pages 1070
Modified db pages 801
Percent of dirty pages(LRU & free pages): 9.873
Max dirty pages percent: 90.000
Pending reads 0
Pending writes: LRU 0, flush list 0
Pages made young 4, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2546, created 343, written 0
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2889, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]

ROW OPERATIONS

0 read views open inside InnoDB
state: sleeping

END OF INNODB MONITOR OUTPUT

1 row in set (0.001 sec)

---

You also may want to try these in your /etc/cloudstack/management/db.properties
db.pool.maxTotal=100
db.pool.maxIdle=20
db.pool.minIdle=10
db.pool.maxWaitMillis=60000

But this part is not default right? we need add manually because for my setup is it is not includes.

[muthukrishnang1100]

when i enable OOBM on first host it will affect the third host error like this

2026-01-03 08:26:15,452 DEBUG [c.c.v.VirtualMachineManagerImpl] (Vm-Operations-Cleanup-1:ctx-541bbd5f) (logid:45ace2f5) VM Operation Thread Running
2026-01-03 08:26:15,481 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-ecbf1dfd) (logid:c5c0bab1) VM scheduler.poll is being called at 2026-01-03 08:26:00 GMT
2026-01-03 08:26:15,492 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-ecbf1dfd) (logid:c5c0bab1) Got 0 scheduled jobs to be executed at 2026-01-03 08:26:00 GMT
2026-01-03 08:26:15,497 INFO [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-ecbf1dfd) (logid:c5c0bab1) Cleaned up 0 VM scheduled job entries
2026-01-03 08:26:15,738 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-1:null) (logid:f750332a) Preparing command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.10.117 -p 623 -U ADMIN -P ***** chassis power status] to execute.
2026-01-03 08:26:15,739 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-1:null) (logid:f750332a) Submitting command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.10.117 -p 623 -U ADMIN -P ***** chassis power status].
2026-01-03 08:26:15,739 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-1:null) (logid:f750332a) Waiting for a response from command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.10.117 -p 623 -U ADMIN -P ***** chassis power status]. Defined timeout: [60].
2026-01-03 08:26:15,802 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-1:null) (logid:f750332a) Process standard output for command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.10.117 -p 623 -U ADMIN -P ***** chassis power status]: [Chassis Power is on
].
2026-01-03 08:26:15,802 DEBUG [o.a.c.u.p.ProcessRunner] (pool-1-thread-1:null) (logid:f750332a) Process standard error output command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.10.117 -p 623 -U ADMIN -P ***** chassis power status]: [Loading IANA PEN Registry...
Using best available cipher suite 3

Running Get VSO Capabilities my_addr 0x20, transit 0, target 0x20
Invalid completion code received: Invalid command
Discovered IPMB address 0x0
].
2026-01-03 08:28:15,834 DEBUG [o.a.c.o.d.i.IpmitoolOutOfBandManagementDriver] (pool-1-thread-2:null) (logid:05dbe44f) The command [/usr/bin/ipmitool -I lanplus -R 1 -v -H 192.168.10.117 -p **** -U ADMIN -P **** chassis power status] was successful and got the result [Chassis Power is on].
2026-01-03 08:28:16,790 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-4:ctx-67852447) (logid:e29c6d96) HA health check task is running...
2026-01-03 08:28:18,528 DEBUG [c.c

After wait some minutes the DB error comes on the same third host

2026-01-03 08:29:51,056 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp341748265-313:ctx-9e12f4f5 ctx-fc18c527) (logid:1e5886ee) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-01-03 08:29:51,057 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp341748265-22:ctx-db55deee ctx-762f2a41) (logid:93061ece) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-01-03 08:29:51,066 DEBUG [c.c.a.ApiServlet] (qtp341748265-313:ctx-9e12f4f5 ctx-fc18c527) (logid:1e5886ee) ===END=== 10.1.12.54 -- GET page=1&pagesize=100&details=min&listall=true&showicon=true&command=listProjects&response=json&sessionkey=Id__9JN7wYWDV2te4E-9s5S5HXo
2026-01-03 08:29:51,084 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1058)
at com.cloud.utils.db.GenericDaoBase.lockRow(GenericDaoBase.java:1035)
at com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:981)
at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy97.findById(Unknown Source)
at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.commandCanBypassHostMaintenance(CloudStackPrimaryDataStoreDriverImpl.java:227)
at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.deleteAsync(CloudStackPrimaryDataStoreDriverImpl.java:238)
at org.apache.cloudstack.storage.volume.VolumeServiceImpl.expungeVolumeAsync(VolumeServiceImpl.java:440)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.cleanupVolumes(VolumeOrchestrator.java:1286)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy241.cleanupVolumes(Unknown Source)
at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:667)
at com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:600)
at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:2462)
at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:3357)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy186.destroyVm(Unknown Source)
at org.apache.cloudstack.api.command.user.vm.DestroyVMCmd.execute(DestroyVMCmd.java:137)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

The last packet successfully received from the server was 1 milliseconds ago. The last packet sent successfully to the server was 1 milliseconds ago.
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:175)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at com.mysql.cj.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:2022)
... 69 more
Caused by: com.mysql.cj.exceptions.CJCommunicationsException: Communications link failure

The last packet successfully received from the server was 1 milliseconds ago. The last packet sent successfully to the server was 1 milliseconds ago.
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
... 81 more
Caused by: java.net.SocketException: Connection reset
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at com.mysql.cj.protocol.ReadAheadInputStream.fill(ReadAheadInputStream.java:107)
... 87 more
2026-01-03 08:29:51,091 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Complete async job-1578, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"DB Exception on: null"}
2026-01-03 08:29:51,092 DEBUG [c.c.a.ApiServlet] (qtp341748265-328:ctx-386dbf87 ctx-14af538e) (logid:0cd05586) ===END=== 10.1.12.54 -- GET listall=true&details=group,nics,secgrp,tmpl,servoff,diskoff,iso,volume,affgrp,backoff&isvnf=false&page=1&pagesize=20&showIcon=true&command=listVirtualMachines&response=json&sessionkey=Id__9JN7wYWDV2te4E-9s5S5HXo
2026-01-03 08:29:51,094 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Publish async job-1578 complete on message bus
2026-01-03 08:29:51,094 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Wake up jobs related to job-1578
2026-01-03 08:29:51,094 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Update db status for job-1578
2026-01-03 08:29:51,096 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Wake up jobs joined with job-1578 and disjoin all subjobs created from job- 1578
2026-01-03 08:29:51,102 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-4:ctx-9a3395d4 job-1578) (logid:473a9a29) Done executing org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin for job-1578

after first host error like this

2026-01-03 08:41:02,803 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentManager-Handler-10:null) (logid:) Seq 3-6899233154154889263: MgmtId 200525778825546: Resp: Routing to peer
2026-01-03 08:41:02,803 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 3-6899233154154889263: No more commands found
2026-01-03 08:41:02,907 WARN [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-7:null) (logid:) Caught:
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:864)
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:819)
at com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:1490)
at com.cloud.vm.dao.ConsoleProxyDaoImpl.update(ConsoleProxyDaoImpl.java:138)
at jdk.internal.reflect.GeneratedMethodAccessor282.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy191.update(Unknown Source)
at com.cloud.consoleproxy.ConsoleProxyManagerImpl.updateConsoleProxyStatus(ConsoleProxyManagerImpl.java:1655)
at com.cloud.consoleproxy.ConsoleProxyManagerImpl$VmBasedAgentHook.onLoadReport(ConsoleProxyManagerImpl.java:282)
at com.cloud.consoleproxy.ConsoleProxyListener.processControlCommand(ConsoleProxyListener.java:56)
at com.cloud.agent.manager.AgentManagerImpl.handleControlCommand(AgentManagerImpl.java:307)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.processRequest(AgentManagerImpl.java:1408)
at com.cloud.agent.manager.AgentManagerImpl$AgentHandler.doTask(AgentManagerImpl.java:1499)
at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:709)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:147)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
... 30 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:356)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:141)
... 32 more
2026-01-03 08:41:02,908 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-7:null) (logid:) SeqA 10-10207: Sending Seq 10-10207: { Ans: , MgmtId: 270790104347868, via: 10, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.Answer":{"result":"false","details":"DB Exception on: null","wait":"0","bypassHostMaintenance":"false"}}] }
2026-01-03 08:41:02,910 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-14:null) (logid:) SeqA 10-10214: Processing Seq 10-10214: { Cmd , MgmtId: -1, via: 10, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":"2","_loadInfo":"{
"connections": [],
"removedSessions": []
}","wait":"0","bypassHostMaintenance":"false"}}] }
2026-01-03 08:41:04,810 ERROR [c.c.s.StatsCollector] (StatsCollector-3:ctx-d4726101) (logid:85fb8547) Error trying to retrieve VM stats
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:433)
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:362)
at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:351)
at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:1411)
at jdk.internal.reflect.GeneratedMethodAccessor101.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy96.search(Unknown Source)
at com.cloud.server.StatsCollector$VmStatsCollector.runInContext(StatsCollector.java:1193)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:147)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
... 29 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:356)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:141)

After this i cant able to access the browser dasboard for host 1

and also this error shows for 2nd and 3rd host

2026-01-03 08:42:47,318 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-59cb42e3) (logid:1448cbc8) AutoScaling Monitor is running...
2026-01-03 08:42:47,322 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-59cb42e3) (logid:1448cbc8) Skipping AutoScaling Monitor
2026-01-03 08:42:47,445 DEBUG [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:ctx-0dc9cd57) (logid:7ed4936e) Starting to remove all volume_stats rows older than [Fri Jan 02 20:42:47 UTC 2026].
2026-01-03 08:42:47,449 INFO [c.c.s.d.VolumeStatsDaoImpl] (StatsCollector-6:ctx-0dc9cd57) (logid:7ed4936e) Removed a total of [0] volume_stats rows older than [Fri Jan 02 20:42:47 UTC 2026].
2026-01-03 08:42:47,516 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c6510625) (logid:9d8ba108) Detected management node left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:47,516 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c6510625) (logid:9d8ba108) Trying to connect to 192.168.170.25
2026-01-03 08:42:47,517 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c6510625) (logid:9d8ba108) Management node 1 is detected inactive by timestamp but is pingable
2026-01-03 08:42:47,934 DEBUG [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:ctx-878cdf24) (logid:9ab5fc97) Starting to remove all vm_stats rows older than [Fri Jan 02 20:42:47 UTC 2026].
2026-01-03 08:42:47,957 INFO [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:ctx-878cdf24) (logid:9ab5fc97) Removed a total of [4] vm_stats rows older than [Fri Jan 02 20:42:47 UTC 2026].
2026-01-03 08:42:48,265 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-4:ctx-747221a1) (logid:6609a8eb) HA health check task is running...
2026-01-03 08:42:49,016 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-1d71274d) (logid:ea11cb26) Detected management node left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:49,016 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-1d71274d) (logid:ea11cb26) Trying to connect to 192.168.170.25
2026-01-03 08:42:49,016 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-1d71274d) (logid:ea11cb26) Management node 1 is detected inactive by timestamp but is pingable
2026-01-03 08:42:50,377 DEBUG [c.c.a.ApiServlet] (qtp341748265-330:ctx-ff157751) (logid:bb22092b) ===START=== 10.1.12.54 -- GET command=readyForShutdown&response=json&sessionkey=HUEhk8e7DRkUnItZrBXJj8wcyPY
2026-01-03 08:42:50,377 DEBUG [c.c.a.ApiServlet] (qtp341748265-330:ctx-ff157751) (logid:bb22092b) Two factor authentication is already verified for the user 2, so skipping
2026-01-03 08:42:50,442 DEBUG [c.c.a.ApiServer] (qtp341748265-330:ctx-ff157751 ctx-107a9660) (logid:bb22092b) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"373ce4b1-e2fc-11f0-99f1-f6483bfa74dc"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2026-01-03 08:42:50,459 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp341748265-330:ctx-ff157751 ctx-107a9660) (logid:bb22092b) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-01-03 08:42:50,461 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp341748265-330:ctx-ff157751 ctx-107a9660) (logid:bb22092b) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-01-03 08:42:50,468 DEBUG [c.c.a.ApiServlet] (qtp341748265-330:ctx-ff157751 ctx-107a9660) (logid:bb22092b) ===END=== 10.1.12.54 -- GET command=readyForShutdown&response=json&sessionkey=HUEhk8e7DRkUnItZrBXJj8wcyPY
2026-01-03 08:42:50,517 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c00bc642) (logid:691dfd91) Detected management node left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:50,517 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c00bc642) (logid:691dfd91) Trying to connect to 192.168.170.25
2026-01-03 08:42:50,518 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-c00bc642) (logid:691dfd91) Management node 1 is detected inactive by timestamp but is pingable
2026-01-03 08:42:51,028 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-fc33bca6) (logid:6a1d468e) Enabled non-edge zones available for scan: 3
2026-01-03 08:42:51,031 DEBUG [o.a.c.s.SecondaryStorageManagerImpl] (secstorage-1:ctx-fc33bca6) (logid:6a1d468e) Zone [3] is not ready to launch secondary storage VM.
2026-01-03 08:42:51,238 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-6f970a3d) (logid:f1022f8c) Enabled non-edge zones available for scan: 3
2026-01-03 08:42:51,241 DEBUG [c.c.c.ConsoleProxyManagerImpl] (consoleproxy-1:ctx-6f970a3d) (logid:f1022f8c) Zone 3 is not ready to launch console proxy yet
2026-01-03 08:42:52,017 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-a950b9e6) (logid:1307e641) Detected management node left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:52,017 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-a950b9e6) (logid:1307e641) Trying to connect to 192.168.170.25
2026-01-03 08:42:52,018 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-a950b9e6) (logid:1307e641) Management node 1 is detected inactive by timestamp but is pingable
2026-01-03 08:42:52,315 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-4:ctx-9faba4d1) (logid:b77f852d) HA health check task is running...
2026-01-03 08:42:53,517 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-d870df33) (logid:78abf199) Detected management node left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:53,517 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-d870df33) (logid:78abf199) Trying to connect to 192.168.170.25
2026-01-03 08:42:53,517 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-d870df33) (logid:78abf199) Management node 1 is detected inactive by timestamp but is pingable
2026-01-03 08:42:55,016 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-61681d00) (logid:9cab442a) Detected management node left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:55,016 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-61681d00) (logid:9cab442a) Trying to connect to 192.168.170.25
2026-01-03 08:42:55,016 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-61681d00) (logid:9cab442a) Management node 1 is detected inactive by timestamp but is pingable
2026-01-03 08:42:55,379 DEBUG [c.c.a.ApiServlet] (qtp341748265-23:ctx-71bde22b) (logid:736b2752) ===START=== 10.1.12.54 -- GET command=readyForShutdown&response=json&sessionkey=HUEhk8e7DRkUnItZrBXJj8wcyPY
2026-01-03 08:42:55,379 DEBUG [c.c.a.ApiServlet] (qtp341748265-23:ctx-71bde22b) (logid:736b2752) Two factor authentication is already verified for the user 2, so skipping
2026-01-03 08:42:55,412 DEBUG [c.c.a.ApiServer] (qtp341748265-23:ctx-71bde22b ctx-6d4332bd) (logid:736b2752) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"373ce4b1-e2fc-11f0-99f1-f6483bfa74dc"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2026-01-03 08:42:55,429 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp341748265-23:ctx-71bde22b ctx-6d4332bd) (logid:736b2752) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-01-03 08:42:55,431 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp341748265-23:ctx-71bde22b ctx-6d4332bd) (logid:736b2752) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-01-03 08:42:55,438 DEBUG [c.c.a.ApiServlet] (qtp341748265-23:ctx-71bde22b ctx-6d4332bd) (logid:736b2752) ===END=== 10.1.12.54 -- GET command=readyForShutdown&response=json&sessionkey=HUEhk8e7DRkUnItZrBXJj8wcyPY
2026-01-03 08:42:56,371 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-4:ctx-f757e80f) (logid:50d360f6) HA health check task is running...
2026-01-03 08:42:56,517 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-53e4ff9e) (logid:cb381e1c) Detected management node left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:56,517 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-53e4ff9e) (logid:cb381e1c) Trying to connect to 192.168.170.25
2026-01-03 08:42:56,517 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-53e4ff9e) (logid:cb381e1c) Management node 1 is detected inactive by timestamp but is pingable
2026-01-03 08:42:58,017 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-077362ee) (logid:75659ed9) Detected management node left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:58,017 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-077362ee) (logid:75659ed9) Trying to connect to 192.168.170.25
2026-01-03 08:42:58,018 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-077362ee) (logid:75659ed9) Management node 1 is detected inactive by timestamp but is pingable
2026-01-03 08:42:59,518 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-985e3a16) (logid:5bea8e90) Detected management node left, id:1, nodeIP:192.168.170.25
2026-01-03 08:42:59,518 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-985e3a16) (logid:5bea8e90) Trying to connect to 192.168.170.25
2026-01-03 08:42:59,518 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-985e3a16) (logid:5bea8e90) Management node 1 is detected inactive by timestamp but is pingable
2026-01-03 08:43:00,428 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-4:ctx-9b2ad256) (logid:d5b4b84f) HA health check task is running...
2026-01-03 08:43:01,016 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9290a1cc) (logid:e7200c92) Detected management node left, id:1, nodeIP:192.168.170.25
2026-01-03 08:43:01,016 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9290a1cc) (logid:e7200c92) Trying to connect to 192.168.170.25
2026-01-03 08:43:01,017 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-9290a1cc) (logid:e7200c92) Management node 1 is detected inactive by timestamp but is pingable
2026-01-03 08:43:02,367 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-f7f04c55) (logid:b1bb1740) Found 0 routers to update status.
2026-01-03 08:43:02,376 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-f7f04c55) (logid:b1bb1740) Found 1 VPC's to update Redundant State.
2026-01-03 08:43:02,380 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-f7f04c55) (logid:b1bb1740) Found 0 networks to update RvR status.
2026-01-03 08:43:02,386 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RedundantRouterStatusMonitor-5:ctx-a441b866) (logid:3b5c05ec) Skip router pair (r-11-VM,r-12-VM) due to not belong to this mgmt server
2026-01-03 08:43:02,512 DEBUG [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-72950dce) (logid:a71762c1) Detected management node left, id:1, nodeIP:192.168.170.25
2026-01-03 08:43:02,512 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-72950dce) (logid:a71762c1) Trying to connect to 192.168.170.25
2026-01-03 08:43:02,513 INFO [c.c.c.ClusterManagerImpl] (Cluster-Heartbeat-1:ctx-72950dce) (logid:a71762c1) Manag

[ingox]

@muthukrishnang1100 Thank you for the details. Please check this:

• Is MariaDB on the same host as CloudStack mgmt, or remote? Any VIP/HAProxy in between?
• What are the values of wait_timeout and interactive_timeout on MariaDB?
  SHOW VARIABLES LIKE 'wait_timeout';
  SHOW VARIABLES LIKE 'interactive_timeout';
• Are you running a Galera cluster?

[muthukrishnang1100]

yes, I am using this for testing purpose so, currently i am 3 servers has mariadb galera, haproxy and keepalived. And also managament server and agent is same servers. Yes, i used VIP.

---

MariaDB [(none)]> SHOW VARIABLES LIKE 'wait_timeout';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| wait_timeout | 28800 |
+---------------+-------+
1 row in set (0.001 sec)

MariaDB [(none)]> SHOW VARIABLES LIKE 'interactive_timeout';
+---------------------+-------+
| Variable_name | Value |
+---------------------+-------+
| interactive_timeout | 28800 |
+---------------------+-------+
1 row in set (0.001 sec)

MariaDB [(none)]>

Yes, I am running galera cluster.

[ingox]

So it's a question of the right Galera + HAProxy + Keepalived VIP setup.
Do the following steps:

• add this to /etc/cloudstack/management/db.properties
  db.pool.maxTotal=100
  db.pool.maxIdle=20
  db.pool.minIdle=10
  db.pool.maxWaitMillis=60000
  db.pool.testOnBorrow=true
  db.pool.validationQuery=SELECT 1
  db.pool.validationQueryTimeout=5
  db.pool.testWhileIdle=true
  db.pool.timeBetweenEvictionRunsMillis=30000
• restart management server(s)
• configure this:
  sysctl -w net.ipv4.tcp_keepalive_time=60
  sysctl -w net.ipv4.tcp_keepalive_intvl=10
  sysctl -w net.ipv4.tcp_keepalive_probes=6
  Persist in /etc/sysctl.conf (or /etc/sysctl.d/99-cloudstack.conf).

If you still having issues try without the VIP.

[muthukrishnang1100]

Yes, I tried all the things but still we get an same error.
But this is my major issue now

Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:147)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
... 30 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:356)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:141)
... 32 more
2025-12-30 12:51:49,231 ERROR [c.c.s.StatsCollector] (StatsCollector-1:ctx-2c2139dd) (logid:e41447a0) Error trying to monitor autoscaling
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:433)
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:362)
at com.cloud.utils.db.GenericDaoBase.listIncludingRemovedBy(GenericDaoBase.java:960)
at com.cloud.utils.db.GenericDaoBase.listBy(GenericDaoBase.java:939)
at com.cloud.cluster.dao.ManagementServerHostDaoImpl.findOneInUpState(ManagementServerHostDaoImpl.java:270)
at jdk.internal.reflect.GeneratedMethodAccessor284.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy161.findOneInUpState(Unknown Source)
at com.cloud.server.StatsCollector$AutoScaleMonitor.runInContext(StatsCollector.java:1766)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:147)
at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:557)
at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:468)
... 30 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:446)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:356)
at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:141)
... 32 more
2025-12-30 12:51:49,240 ERROR [c.c.n.r.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:ctx-8d4322d8) (logid:dcafa96b) Fail to complete the CheckRouterTask!
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:433)
at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:362)

why this error comes sometimes again and again 1 or 2 days how to solve this permanantly what is exact problem for this.

And also i found this for web some others really this is the cause?

The Exact Problem: "The Silent Kill"
The error Timeout waiting for idle object happens because of a 3-way conflict:

CloudStack Pool: Keeps connections open in a "pool" to save time. It expects these to stay alive for hours.

HAProxy (The Culprit): By default, HAProxy often has a timeout client and timeout server of 50 seconds. If a connection is idle for 51 seconds, HAProxy kills the "pipe" but doesn't tell CloudStack.

The Error: When CloudStack finally tries to use that connection, it finds a "dead pipe." It then tries to get a new connection from the pool, but because so many connections are now in this "broken" state, the pool becomes exhausted, leading to the java.util.NoSuchElementException: Timeout waiting for idle object.

Please check and give me exact solution

[ingox]

Can you confirm you still facing the issue without VIP/HAProxy layer?

Please paste your HAProxy config. Especially DB listener/backend section.

Provide the logs around the event.
journalctl -u haproxy -u keepalived --since "3 days ago" | egrep -i "reload|restart|master|backup|FAULT|VIP|down|up|server"

[muthukrishnang1100]

yes, still i am facing that issue.

muthukrishnan - Tue Jan 06 2026 17_44_28 GMT+0530 (India Standard Time).txt

root@mgmt1:~# cat /etc/haproxy/haproxy.cfg
global
log /dev/log local0
maxconn 50000
daemon

defaults
log global
mode tcp
option tcplog
timeout connect 5s
timeout client 50s
timeout server 50s

frontend mysql_frontend
bind 192.168.30.16:3306
mode tcp
default_backend galera_backend

backend galera_backend
mode tcp
balance roundrobin
option mysql-check user haproxy
default-server fall 3 rise 2 inter 2000
server prod-node01 192.168.30.2:3307 check
server prod-node02 192.168.30.11:3307 check
server prod-node03 192.168.30.15:3307 check

listen stats
bind *:9999
mode http
stats enable
stats uri /status
stats realm Haproxy\ Statistics
stats auth xxxx:xxxxxx

frontend cloudstack_ui_8080
bind *:80
mode http
option forwardfor
option httplog
default_backend cloudstack_ui_nodes

backend cloudstack_ui_nodes
mode http
option httpchk OPTIONS /client
balance source
server prod-node01 192.168.30.2:8080 maxconn 32 check inter 5000
server prod-node02 192.168.30.11:8080 maxconn 32 check inter 5000
server prod-node03 192.168.30.15:8080 maxconn 32 check inter 5000

---

[ingox]

The root cause is HAProxy TCP timeouts (50 seconds) applied to the MariaDB VIP. CloudStack keeps long-lived pooled DB connections; HAProxy closes idle TCP sessions after 50 seconds, causing stale/broken pooled connections. Eventually CloudStack’s DB pool becomes exhausted and throws Timeout waiting for idle object.

Update your proxy config and reload the haproxy:
frontend mysql_frontend
bind 192.168.30.16:3306
mode tcp
option tcplog
timeout client 1h
default_backend galera_backend

backend galera_backend
mode tcp
balance roundrobin
option tcpka
timeout server 1h
timeout connect 5s
option mysql-check user haproxy
default-server fall 3 rise 2 inter 2000
server prod-node01 192.168.30.2:3307 check
server prod-node02 192.168.30.11:3307 check
server prod-node03 192.168.30.15:3307 check

Update your /etc/cloudstack/management/db.properties to:
db.pool.testOnBorrow=true
db.pool.validationQuery=SELECT 1
db.pool.testWhileIdle=true

[muthukrishnang1100]

Yes, I applied these changes till now no problem or error like before. I will update you if any problem comes later.
But, still my HA is not working. Before, when i enable the OOBM on host it error comes that before i shared, after I configured these above changes now that error not occurs when i enable the OOBM. But, When i shutdoen my host directly on the IPMI, HA is not working.
all was good but HA is not working when I enable OOBMthe host HA state is Available. After, I shutdown the one host HA state is Suspect. After HA state is Fenced. But, inside that host instances are not migrated or restarted the other host more than 30 minutes still same
Please what i do for these?

see the test-3 host. what i do for work this HA because later we use production design so i am asked and also why it takes more than 30 to 40 minutes?

Please find the issue and solve me

[muthukrishnang1100]

root@test-2:~# grep -i "Restart VM" /var/log/cloudstack/management/management-server.log
2026-01-07 05:24:08,803 DEBUG [c.c.h.HighAvailabilityManagerImpl] (pool-3-thread-1:null) (logid:6aa4ee6f) Notifying HA Mgr of to restart vm 45-r-45-VM
2026-01-07 05:24:08,824 DEBUG [c.c.h.HighAvailabilityManagerImpl] (pool-3-thread-1:null) (logid:6aa4ee6f) Notifying HA Mgr of to restart vm 53-i-2-53-VM
2026-01-07 05:24:08,846 DEBUG [c.c.h.HighAvailabilityManagerImpl] (pool-3-thread-1:null) (logid:6aa4ee6f) Notifying HA Mgr of to restart vm 54-i-2-54-VM

root@test-1:# grep -i "Restart VM" /var/log/cloudstack/management/management-server.log
2026-01-07 06:27:50,718 WARN [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-7e914ac3 work-149) (logid:cdfe61be) Unable to restart VM instance {"id":54,"instanceName":"i-2-54-VM","state":"Stopped","type":"User","uuid":"aca1cc40-066c-4659-b768-b307f3f9bce7"} due to Unable to create a deployment for VM instance {"id":54,"instanceName":"i-2-54-VM","state":"Starting","type":"User","uuid":"aca1cc40-066c-4659-b768-b307f3f9bce7"}
2026-01-07 06:27:50,722 WARN [c.c.a.AlertManagerImpl] (HA-Worker-4:ctx-7e914ac3 work-149) (logid:cdfe61be) alertType=[8] dataCenterId=[3] podId=[3] clusterId=[null] message=[Unable to restart VM-07 which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod].
2026-01-07 06:27:50,727 WARN [c.c.a.AlertManagerImpl] (HA-Worker-4:ctx-7e914ac3 work-149) (logid:cdfe61be) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Unable to restart VM-07 which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod] and content [Insufficient capacity to restart VM, name: VM-07, id: 54 which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod].
2026-01-07 06:27:50,774 WARN [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-3:ctx-cb5fcf36 work-146) (logid:4d03732c) Unable to restart VM instance {"id":45,"instanceName":"r-45-VM","state":"Stopped","type":"DomainRouter","uuid":"a2c09b9d-85c7-459b-bd75-a1051fa5284f"} due to Unable to create a deployment for VM instance {"id":45,"instanceName":"r-45-VM","state":"Starting","type":"DomainRouter","uuid":"a2c09b9d-85c7-459b-bd75-a1051fa5284f"}
2026-01-07 06:27:50,786 WARN [c.c.a.AlertManagerImpl] (HA-Worker-3:ctx-cb5fcf36 work-146) (logid:4d03732c) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Unable to restart r-45-VM which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod] and content [Insufficient capacity to restart VM, name: r-45-VM, id: 45 which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod].
root@test-1:#

see in 2nd host it shows, but nothing happends what i do?

[ingox]

Your VMs are not restarted due to insufficient capacity:

Unable to restart VM ... due to Unable to create a deployment
...
Insufficient capacity to restart VM

Check resources and tagging.

[muthukrishnang1100]

Yes, i checked but all have more than enough. If you want i share the capacity

see and also i have just 1 vms for each host

[ingox]

Can you manually start the VM from the "failed" host on one of the others?

What servers are your system VMs are running on, especially the virtual router?

[muthukrishnang1100]

Yes, i tried to manually restart the failed host but, shows error

root@test-1:# grep -i "Restart VM" /var/log/cloudstack/management/management-server.log
2026-01-07 06:27:50,718 WARN [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-4:ctx-7e914ac3 work-149) (logid:cdfe61be) Unable to restart VM instance {"id":54,"instanceName":"i-2-54-VM","state":"Stopped","type":"User","uuid":"aca1cc40-066c-4659-b768-b307f3f9bce7"} due to Unable to create a deployment for VM instance {"id":54,"instanceName":"i-2-54-VM","state":"Starting","type":"User","uuid":"aca1cc40-066c-4659-b768-b307f3f9bce7"}
2026-01-07 06:27:50,722 WARN [c.c.a.AlertManagerImpl] (HA-Worker-4:ctx-7e914ac3 work-149) (logid:cdfe61be) alertType=[8] dataCenterId=[3] podId=[3] clusterId=[null] message=[Unable to restart VM-07 which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod].
2026-01-07 06:27:50,727 WARN [c.c.a.AlertManagerImpl] (HA-Worker-4:ctx-7e914ac3 work-149) (logid:cdfe61be) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Unable to restart VM-07 which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod] and content [Insufficient capacity to restart VM, name: VM-07, id: 54 which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod].
2026-01-07 06:27:50,774 WARN [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-3:ctx-cb5fcf36 work-146) (logid:4d03732c) Unable to restart VM instance {"id":45,"instanceName":"r-45-VM","state":"Stopped","type":"DomainRouter","uuid":"a2c09b9d-85c7-459b-bd75-a1051fa5284f"} due to Unable to create a deployment for VM instance {"id":45,"instanceName":"r-45-VM","state":"Starting","type":"DomainRouter","uuid":"a2c09b9d-85c7-459b-bd75-a1051fa5284f"}
2026-01-07 06:27:50,786 WARN [c.c.a.AlertManagerImpl] (HA-Worker-3:ctx-cb5fcf36 work-146) (logid:4d03732c) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Unable to restart r-45-VM which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod] and content [Insufficient capacity to restart VM, name: r-45-VM, id: 45 which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod].
2026-01-07 07:02:16,162 DEBUG [c.c.h.HighAvailabilityManagerImpl] (pool-4-thread-1:null) (logid:290274db) Notifying HA Mgr of to restart vm 45-r-45-VM
2026-01-07 07:02:16,175 DEBUG [c.c.h.HighAvailabilityManagerImpl] (pool-4-thread-1:null) (logid:290274db) Notifying HA Mgr of to restart vm 54-i-2-54-VM
2026-01-07 11:16:30,942 WARN [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-3:ctx-494e74cd work-166) (logid:10998aae) Unable to restart VM instance {"id":45,"instanceName":"r-45-VM","state":"Stopped","type":"DomainRouter","uuid":"a2c09b9d-85c7-459b-bd75-a1051fa5284f"} due to Unable to create a deployment for VM instance {"id":45,"instanceName":"r-45-VM","state":"Starting","type":"DomainRouter","uuid":"a2c09b9d-85c7-459b-bd75-a1051fa5284f"}
2026-01-07 11:16:30,952 WARN [c.c.a.AlertManagerImpl] (HA-Worker-3:ctx-494e74cd work-166) (logid:10998aae) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Unable to restart r-45-VM which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod] and content [Insufficient capacity to restart VM, name: r-45-VM, id: 45 which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod].
2026-01-07 11:55:33,903 WARN [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-1:ctx-2e27fa35 work-169) (logid:8fdc4e5e) Unable to restart VM instance {"id":54,"instanceName":"i-2-54-VM","state":"Stopped","type":"User","uuid":"aca1cc40-066c-4659-b768-b307f3f9bce7"} due to Unable to create a deployment for VM instance {"id":54,"instanceName":"i-2-54-VM","state":"Starting","type":"User","uuid":"aca1cc40-066c-4659-b768-b307f3f9bce7"}
2026-01-07 11:55:33,907 WARN [c.c.a.AlertManagerImpl] (HA-Worker-1:ctx-2e27fa35 work-169) (logid:8fdc4e5e) alertType=[8] dataCenterId=[3] podId=[3] clusterId=[null] message=[Unable to restart VM-07 which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod].
2026-01-07 11:55:33,915 WARN [c.c.a.AlertManagerImpl] (HA-Worker-1:ctx-2e27fa35 work-169) (logid:8fdc4e5e) No recipients set in global setting 'alert.email.addresses', skipping sending alert with subject [Unable to restart VM-07 which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod] and content [Insufficient capacity to restart VM, name: VM-07, id: 54 which was running on host name: test-3(id:6), availability zone: Xaasio-Zone, pod: Xaasio-Pod].
root@test-1:#

but, it comes error like this something is blocked. but< HA is working VM only still not auto started and also if itried to manually started error.

The system VMs are running on the First host

My virtual routers are running on 1st and 2nd hosts

[prashanthr2]

@muthukrishnang1100 Share full log files from management servers ( /var/log/cloudstack/management/management-server.log) to understand why the manual start is failing. The snippet you shared is not very useful to identify the issue