Fail to do a Live Migration with storage via migrateVirtualMachineWithVolume API.

[fuchanghao]

ISSUE TYPE

Bug Report

COMPONENT NAME

API

CLOUDSTACK VERSION

CloudStack 4.14.0 LTS
CloudStack Monkey 6.0.0

CONFIGURATION

http://docs.cloudstack.apache.org/en/4.14.0.0/adminguide/virtual_machines.html#moving-vms-between-hosts-manual-live-migration

OS / ENVIRONMENT

CentOS 7.8 (2003) (2 agent)

VM:
ubuntu server 18.04 LTS (KVM)

SUMMARY

I Step from the docs: Moving VMs Between Hosts (Manual Live Migration)
http://docs.cloudstack.apache.org/en/4.14.0.0/adminguide/virtual_machines.html#moving-vms-between-hosts-manual-live-migration

It's says

(KVM) If the VM’s storage has to be migrated along with the VM, from a mounted NFS storage pool to a cluster-wide mounted NFS storage pool, then the ‘migrateVirtualMachineWithVolume’ API has to be used. There is no UI integration for this feature.

But I still fails.

STEPS TO REPRODUCE

1.install the CloudStack Monkey setup password, apikey, secretkey, and do sync.

2.use the command it says

migrate virtualmachinewithvolume virtualmachineid=<virtual machine uuid> hostid=<destination host uuid> migrateto[i].volume=<virtual machine volume number i uuid> migrateto[i].pool=<destination storage pool uuid for volume number i>

I do a test to Migrate a running ubuntu VM from Host 1&Storage 1 to Host 2&Storage 2.

Here's my info:

My ubuntu VM UUID : 1adf846d-2410-421f-9232-7148b68fe7cf
My ubuntu VM Volume UUID: 79aeeed9-0e34-4415-ad68-a515477b96e4
My ubuntu VM Volume device ID: 0

Host 1: 7d4c2d6c-efce-4e5b-a278-78d0ec874314
Host 2: 727c425d-fda1-426f-ae61-24c4af6653c9
Primary Storage 1: c556c89c-5f10-3e2d-bb9d-d131acfab5d7
Primary Storage 2: 9a4d6bde-e0f1-31e7-8a68-388c3923ef05

the command I wrote:

migrate virtualmachinewithvolume virtualmachineid=1adf846d-2410-421f-9232-7148b68fe7cf hostid=727c425d-fda1-426f-ae61-24c4af6653c9 migrateto[0].volume=79aeeed9-0e34-4415-ad68-a515477b96e4 migrateto[0].pool=9a4d6bde-e0f1-31e7-8a68-388c3923ef05

EXPECTED RESULTS

It moves vm volumes form storage 1 to storage 2.

ACTUAL RESULTS

{
  "accountid": "ebaef574-af1c-11ea-b783-000c296f76e6",
  "cmd": "org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd",
  "completed": "2020-06-28T16:15:22+0800",
  "created": "2020-06-28T16:15:20+0800",
  "jobid": "83a2f2ab-d953-4a7a-98c4-f447f657fe3d",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "Failed to migrated vm VM[User|i-2-8-VM] along with its volumes. "
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "ebb0cb1d-af1c-11ea-b783-000c296f76e6"
}
🙈 Error: async API failed for job 83a2f2ab-d953-4a7a-98c4-f447f657fe3d

I already tested migrating storage with VM power off and changing the hosts with VM running Via UI.
Both worked fine.

I don't know it's that I wrote the wrong command or I missed setting something?
I can't find the error details from the errorcode 530.

[yadvr]

@fuchanghao are the hosts within the same cluster and are both storages in zone-wide scope, or cluster-scope (but within same cluster then?)

[fuchanghao]

@fuchanghao are the hosts within the same cluster and are both storages in zone-wide scope, or cluster-scope (but within same cluster then?)

Hosts & Storages all in the same cluster and same zone, both storages scope are "CLUSTER".

Host 1&Primary Storage 1 on machine 1 and Host 2&Primary Storage 2 on machine 2.

[yadvr]

ping @GabrielBrascher @wido any thoughts?

[GabrielBrascher]

@rhtyd @fuchanghao I have never tried such a migration.

I know that the command works fine when migrating from local storage on KVM host A to local storage on KVM host B. Example of the command to migrate from one host to another: migrate virtualmachinewithvolume virtualmachineid=<VM-uuid> hostid=<target-host-B-uuid>.

However, I did not test migrating among two KVM hosts and different NFS storages at the same time. I would need to set a test scenario to check this.

@fuchanghao could you please share the log on the management node related to the error stack?

[fuchanghao]

@rhtyd @fuchanghao I have never tried such a migration.

I know that the command works fine when migrating from local storage on KVM host A to local storage on KVM host B. Example of the command to migrate from one host to another: migrate virtualmachinewithvolume virtualmachineid=<VM-uuid> hostid=<target-host-B-uuid>.

However, I did not test migrating among two KVM hosts and different NFS storages at the same time. I would need to set a test scenario to check this.

@fuchanghao could you please share the log on the management node related to the error stack?

I have tried oVirt in a simular way these days.
And oVirt can migrate storage with VM on.(At least works with 2 NFS) And I can migrate storage with Web-UI on oVirt.

[yadvr]

With @Spaceman1984 cc @DaanHoogland @borisstoyanov - triaging error against 4.14.0.0 and Darrin will explore a fix. Thanks.

[andrijapanicsb]

To the best of my knowlegde, this has never worked with KVM in general, except in a 2 cases:

• what @GabrielBrascher shared, and is a special case from local storage to local storage
• from Ceph/NFS to SolidFire specifically

For both to happen, the qemu must support live storage migration, which means using Ubuntu hosts (they do support it), or CentOS 7 with qemu-kvm-ev packages from the oVirt repo (RedHat removed support for live storage migration after CentOS 6, so in CentOS 7 that doesn't work with default qemu-kvm, nor with the qemu-kvm-ev from the special SIG repo) - thank you redhat for revoking a feature for $$$ reasons.

[DaanHoogland]

ok on no objections I'll remove the 4.15 milestone on this (given that @Spaceman1984 is dealing with a better error reporting than NPE, in some cases)

[nvazquez]

@fuchanghao can you post the error you get from the management server logs and/or the KVM agent?

[andrijapanicsb]

Read my last comment - only CEPH/NFS --> SolidFire worked
localstorage: Also VM with ROOT volume only, can be migrated to another host (with local storage)

Other cases (NFS to NFS, or NFS to local, or NFS to Ceph etc, etc, etc) - were never implemented, but there is a framework for this in case someone has time/will/$$$ to do it.