Cloudstack Agent Java IO exception

[midhunpjos]

problem

Network Actions,VM Deployments,router health checks are failing.
On cloudstack agent logs below is printed

2025-02-06 09:14:09,343 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-4:[]) (logid:64df14bf) Exception [Cannot run program "/usr/share/
cloudstack-common/scripts/network/domr/router_proxy.sh": error=0, Failed to exec spawn helper: pid: 605941, exit value: 1] occurred when attempting to run command [/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh filesystem_writable_check.py 169.254.156.203 ]. java.io.IOException: Cannot run prog
ram "/usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh": error=0, Failed to exec spawn helper: pid: 605941, exit value: 1
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
at com.cloud.utils.script.Script.execute(Script.java:232)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:553)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeInVR(LibvirtComputingResource.java:541)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.checkRouterFileSystem(VirtualRoutingResource.java:419)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.execute(VirtualRoutingResource.java:391)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeQueryCommand(VirtualRoutingResource.java:227)
at com.cloud.agent.resource.virtualnetwork.VirtualRoutingResource.executeRequest(VirtualRoutingResource.java:129)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:35)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtNetworkElementCommandWrapper.execute(LibvirtNetworkElementCommandWrapper.java:29)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1945)
at com.cloud.agent.Agent.processRequest(Agent.java:686)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1109)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 605941, exit value: 1
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.(ProcessImpl.java:314)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
... 13 more

2025-02-06 09:14:46,697 WARN [kvm.storage.MultipathSCSIAdapterBase] (MultipathMapCleanupJob:[]) (logid:) Exception [Cannot run program "/usr/share/cloudstac
k-common/scripts/storage/multipath/cleanStaleMaps.sh": error=0, Failed to exec spawn helper: pid: 605943, exit value: 1] occurred when attempting to run command [/usr/share/cloudstack-common/scripts/storage/multipath/cleanStaleMaps.sh ]. java.io.IOException: Cannot run program "/usr/share/cloudstack-common/script
s/storage/multipath/cleanStaleMaps.sh": error=0, Failed to exec spawn helper: pid: 605943, exit value: 1
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1073)
at com.cloud.utils.script.Script.execute(Script.java:232)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase.runScript(MultipathSCSIAdapterBase.java:382)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase.cleanupStaleMaps(MultipathSCSIAdapterBase.java:572)
at com.cloud.hypervisor.kvm.storage.MultipathSCSIAdapterBase$1.run(MultipathSCSIAdapterBase.java:122)
at java.base/java.util.TimerThread.mainLoop(Timer.java:566)
at java.base/java.util.TimerThread.run(Timer.java:516)
Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 605943, exit value: 1
at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
at java.base/java.lang.ProcessImpl.(ProcessImpl.java:314)
at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:244)
at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1110)
... 7 more

versions

Cloudstack 4.20 + KVM
CEPH as Primary Storage

The steps to reproduce the bug

1.Deploy Instance, add firewall rules for guest networks are failing
2.
3.
...

What to do about it?

Restarting Cloudstack agent resolved the issue

[DaanHoogland]

@midhunpjos , it looks like the agent is not allowed to write on disk. If so only restarting the agent would not solve the issue on NFS. I do not know what the behaviour on Ceph is. Can you reliably reproduce this?

[midhunpjos]

@DaanHoogland It happened to all Hosts in a cluster and just restarting agent everything was back to normal in all hosts.