Timeout snapshot

[Lauta11]

problem

When creating snapshots of large volumes (approximately larger than 200 GB), the task fails to complete successfully. The snapshot command reaches a timeout of 3600 seconds, which is not enough time to copy the entire disk.

We attempted to modify various timeout-related parameters in CloudStack (including those for snapshot and asynchronous tasks), but none of them appear to extend or affect this timeout limit.

It is unclear whether this is a bug or a missing feature that should allow increasing the timeout duration for snapshot operations.

The corresponding logs are attached for further analysis.

Management:

2025-10-21 02:01:57,863 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-93846872) (logid:f246f83a) Snapshot [455b227d-797f-45e6-aa03-1dc7609ac030] for volume [{"name":"ROOT-392","uuid":"8508204f-24ba-454a-8bd1-f56b274088ae"}] can be executed.
2025-10-21 02:01:59,126 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-93846872) (logid:f246f83a) Scheduling snapshot [455b227d-797f-45e6-aa03-1dc7609ac030] for volume [{"name":"ROOT-392","uuid":"8508204f-24ba-454a-8bd1-f56b274088ae"}] at [2025-10-21 05:00:00 GMT].
2025-10-21 02:01:59,179 DEBUG [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-93846872) (logid:f246f83a) Scheduled snapshot [455b227d-797f-45e6-aa03-1dc7609ac030] for volume [{"name":"ROOT-392","uuid":"8508204f-24ba-454a-8bd1-f56b274088ae"}] as job [d0e089bb-d2cb-4e81-8fb4-5e6a23eee556].
2025-10-21 02:02:01,607 DEBUG [o.a.c.s.s.StorPoolSnapshotStrategy] (Work-Job-Executor-28:ctx-a246b76c job-33232/job-33233 ctx-6f646c29) (logid:d0e089bb) StorpoolSnapshotStrategy.canHandle: snapshot=server18063_ROOT-392_20251021050158, uuid=70decb5c-6f4d-405a-b7e1-f2eb2ace4de0, op=TAKE



2025-10-21 03:02:01,993 DEBUG [o.a.c.s.s.SnapshotServiceImpl] (Work-Job-Executor-28:ctx-a246b76c job-33232/job-33233 ctx-6f646c29) (logid:d0e089bb) create snapshot server18063_ROOT-392_20251021050158 failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:59, com.cloud.exception.OperationTimedoutException: Commands 545217029888601412 to Host 59 timed out after 3600


Host:


2025-10-21 03:06:26,176 WARN  [utils.script.Script] (Script-10:null) (logid:) Interrupting script.
2025-10-21 03:06:26,176 WARN  [utils.script.Script] (agentRequest-Handler-2:null) (logid:d0e089bb) Process [3934352] for command [qemu-img convert -O qcow2 -U --imag
e-opts driver=qcow2,file.filename=/mnt/4c524bab-25a3-3db5-a665-d37159b81f11/11806790-0ad0-4a4a-b389-2f7ab41b4e87 /mnt/4c524bab-25a3-3db5-a665-d37159b81f11/snapshots/
3ae09c67-c072-48b4-a8a7-e3f2dbaf2687 ] timed out. Output is [].
2025-10-21 03:06:43,053 ERROR [kvm.storage.KVMStorageProcessor] (agentRequest-Handler-2:null) (logid:d0e089bb) Failed take snapshot for volume [volumeTO[uuid=8508204
f-24ba-454a-8bd1-f56b274088ae|path=11806790-0ad0-4a4a-b389-2f7ab41b4e87|datastore=PrimaryDataStoreTO[uuid=4c524bab-25a3-3db5-a665-d37159b81f11|name=Pool-01|id=18|poo
ltype=NetworkFilesystem]]], in VM [i-52-392-VM], due to [Failed to convert volumeTO[uuid=8508204f-24ba-454a-8bd1-f56b274088ae|path=11806790-0ad0-4a4a-b389-2f7ab41b4e
87|datastore=PrimaryDataStoreTO[uuid=4c524bab-25a3-3db5-a665-d37159b81f11|name=Pool-01|id=18|pooltype=NetworkFilesystem]] snapshot of volume [KVMPhysicalDisk {"dispN
ame":null,"format":"qcow2","name":"11806790-0ad0-4a4a-b389-2f7ab41b4e87","path":"\/mnt\/4c524bab-25a3-3db5-a665-d37159b81f11\/11806790-0ad0-4a4a-b389-2f7ab41b4e87","
pool":{"uuid":"4c524bab-25a3-3db5-a665-d37159b81f11","path":"\/mnt\/4c524bab-25a3-3db5-a665-d37159b81f11"},"size":122363461632,"virtualSize":161061273600,"vmName":nu
ll}] to [/mnt/4c524bab-25a3-3db5-a665-d37159b81f11/snapshots/3ae09c67-c072-48b4-a8a7-e3f2dbaf2687] due to [timeout].].
com.cloud.utils.exception.CloudRuntimeException: Failed to convert volumeTO[uuid=8508204f-24ba-454a-8bd1-f56b274088ae|path=11806790-0ad0-4a4a-b389-2f7ab41b4e87|datas
tore=PrimaryDataStoreTO[uuid=4c524bab-25a3-3db5-a665-d37159b81f11|name=Pool-01|id=18|pooltype=NetworkFilesystem]] snapshot of volume [KVMPhysicalDisk {"dispName":null,"format":"qcow2","name":"11806790-0ad0-4a4a-b389-2f7ab41b4e87","path":"\/mnt\/4c524bab-25a3-3db5-a665-d37159b81f11\/11806790-0ad0-4a4a-b389-2f7ab41b4e87","pool":{"uuid":"4c524bab-25a3-3db5-a665-d37159b81f11","path":"\/mnt\/4c524bab-25a3-3db5-a665-d37159b81f11"},"size":122363461632,"virtualSize":161061273600,"vmName":null}] to [/mnt/4c524bab-25a3-3db5-a665-d37159b81f11/snapshots/3ae09c67-c072-48b4-a8a7-e3f2dbaf2687] due to [timeout].
        at com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.validateConvertResult(KVMStorageProcessor.java:1915)
        at com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.createSnapshot(KVMStorageProcessor.java:1790)
        at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:140)
        at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:66)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStorageSubSystemCommandWrapper.java:36)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStorageSubSystemCommandWrapper.execute(LibvirtStorageSubSystemCommandWrapper.java:30)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1930)
        at com.cloud.agent.Agent.processRequest(Agent.java:683)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

### versions

Cloudstack v4.20 / KVM
S.O ubuntu 24


### The steps to reproduce the bug

1. Start a snapshot task for a volume > 200GB
2. Wait one hour
3. Error


### What to do about it?

Complete configuration to be able to modify this time or fix a bug if necessary.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[sureshanaparti]

@Lauta11 what is the value set for config 'kvm.incremental.snapshot'? and what is the underlying storage used for the volumes?

[erikbocks]

Hello, @Lauta11

When the CreateObjectCommand is sent to the Agent for the snapshot creation, no value is set for its timeout. When the Agent receives the command, as no timeout was informed, the fallback timeout is defined for it (3600 seconds). I am currently finishing a PR that will add a configurable timeout for volume snapshots, and soon I will open it.

[Lauta11]

@Lauta11 what is the value set for config 'kvm.incremental.snapshot'? and what is the underlying storage used for the volumes?

Hi @sureshanaparti , thanks for you response. The kvm.incremental.snapshot option isn't enabled or listed. We're not using incremental snapshots anyway.
The underlying storage used: 6 disk x 4TB WD-GOLD drives in RAID 10

Hello, @Lauta11

When the CreateObjectCommand is sent to the Agent for the snapshot creation, no value is set for its timeout. When the Agent receives the command, as no timeout was informed, the fallback timeout is defined for it (3600 seconds). I am currently finishing a PR that will add a configurable timeout for volume snapshots, and soon I will open it.

Hi @erikbocks , thanks for you response.
I'll wait for the PR.

[sureshanaparti]

Hi @Lauta11 sorry, 'kvm.incremental.snapshot' config was introduced in 4.21. For your case, I think, you can configure timeout in the global setting 'commands.timeout' (for 3 hrs timeout, set to CreateObjectCommand=10800).

[sureshanaparti]

Hello, @Lauta11

When the CreateObjectCommand is sent to the Agent for the snapshot creation, no value is set for its timeout. When the Agent receives the command, as no timeout was informed, the fallback timeout is defined for it (3600 seconds). I am currently finishing a PR that will add a configurable timeout for volume snapshots, and soon I will open it.

@erikbocks I think, no changes required if the granular commands timeout setting (#9659) addresses the timeout update for the CreateObjectCommand.

[Lauta11]

Hi @Lauta11 sorry, 'kvm.incremental.snapshot' config was introduced in 4.21. For your case, I think, you can configure timeout in the global setting 'commands.timeout' (for 3 hrs timeout, set to CreateObjectCommand=10800).

Hello @sureshanaparti a thousand apologies, I have cloudstack version 4.19.2, I thought I had updated, in this version the global configuration 'commands.timeout' does not appear, is this from 4.20 onwards?

[sureshanaparti]

Hi @Lauta11 sorry, 'kvm.incremental.snapshot' config was introduced in 4.21. For your case, I think, you can configure timeout in the global setting 'commands.timeout' (for 3 hrs timeout, set to CreateObjectCommand=10800).

Hello @sureshanaparti a thousand apologies, I have cloudstack version 4.19.2, I thought I had updated, in this version the global configuration 'commands.timeout' does not appear, is this from 4.20 onwards?

@Lauta11 the config 'commands.timeout' was introduced in 4.20.1.