VM operation fail after storage drs in VMware datastore cluster

[mib1185]

ISSUE TYPE

• Bug Report

COMPONENT NAME

VMware

CLOUDSTACK VERSION

4.15.1.0-rc

CONFIGURATION

• Hypervisor: VMware, 2x VMware 6.7.0 ESXi hosts
• datastore cluster:

id	name	uuid	pool_type	port	data_center_id	pod_id	cluster_id	used_bytes	capacity_bytes	host_address	user_info	path	created	removed	update_time	status	storage_provider_name	scope	hypervisor	managed	capacity_iops	parent
59	iosctx01_1504_fp01_ns_0393	39c0a3b5-203c-3ee3-8703-4af20d640eac	PreSetup	0	1	1	3	4695523328	1099243192320	iosvvct1.esx-tst.os.itelligence.de	NULL	/DC_Salzenforst/iosctx01_1504_fp01_ns_0393	"2021-05-27 14:07:57"	NULL	"2021-06-09 07:10:18"	Up	DefaultPrimary	CLUSTER	NULL	0	NULL	63
63	DatastoreCluster_CS	9cd3cb24-a43a-338d-a61b-d7fb0abf3556	DatastoreCluster	0	1	1	3	626064883712	2198486384640	iosvvct1.esx-tst.os.itelligence.de	NULL	/DC_Salzenforst/DatastoreCluster_CS	"2021-06-08 15:51:05"	NULL	"2021-06-09 07:10:18"	Up	DefaultPrimary	CLUSTER	NULL	0	NULL	0
64	iosctx01_1504_fp01_ns_0394	b4f65ef4-8ee3-3776-a14a-d209c498204b	PreSetup	0	1	1	3	621369360384	1099243192320	iosvvct1.esx-tst.os.itelligence.de	NULL	/DC_Salzenforst/iosctx01_1504_fp01_ns_0394	"2021-06-08 15:51:05"	NULL	"2021-06-09 07:10:18"	Up	DefaultPrimary	CLUSTER	NULL	0	NULL	63

OS / ENVIRONMENT

SUMMARY

After storage DRS (volumes were migrated to different datastore in same datastore cluster) volume swere not found during next stop/start of vm (maybe related to #4674).

STEPS TO REPRODUCE

1. ingest VM online with only root volume and afterwards create new volume via CloudStack and attach to ingested vm
2. simulate storage DRS recommendation run (migrate volumes to other datastore in same datastore-cluster on vCenter)
3. stop instance
4. start instance

volumes on DB after each step

id	account_id	domain_id	pool_id	last_pool_id	instance_id	device_id	name	uuid	size	folder	path	pod_id	data_center_id	iscsi_name	host_ip	volume_type	pool_type	disk_offering_id	template_id	first_snapshot_backup_uuid	recreatable	created	attached	updated	removed	state	chain_info	update_count	disk_type	vm_snapshot_chain_size	iso_id	display_volume	format	min_iops	max_iops	hv_ss_reserve	provisioning_type
5329	10	2	64	NULL	1727	0	drs-test-02_2	42734603-a177-409b-92ac-ae15ca38d92a	10737418240	NULL	drs-test-02_2	NULL	1	NULL	NULL	ROOT	NULL	182	214	NULL	0	"2021-06-09 07:29:10"	NULL	NULL	NULL	Ready	"{"diskDeviceBusName":"scsi0:0","diskChain":["[iosctx01_1504_fp01_ns_0394] drs-test-02/drs-test-02_2.vmdk"]}"	0	NULL	NULL	214	1	OVA	NULL	NULL	NULL	fat
5330	10	2	64	55	1727	1	drs-test-02-data-01	265333cd-3676-4e2e-bf70-b42b5f8e1ee5	5368709120	/DC_Salzenforst/iosctx01_1504_fp01_ns_0394	drs-test-02	1	1	NULL	NULL	DATADISK	NULL	54	NULL	NULL	0	"2021-06-09 07:42:01"	"2021-06-09 07:42:15"	"2021-06-09 07:44:10"	NULL	Ready		6	NULL	NULL	NULL	1	OVA	NULL	NULL	0	fat

id	account_id	domain_id	pool_id	last_pool_id	instance_id	device_id	name	uuid	size	folder	path	pod_id	data_center_id	iscsi_name	host_ip	volume_type	pool_type	disk_offering_id	template_id	first_snapshot_backup_uuid	recreatable	created	attached	updated	removed	state	chain_info	update_count	disk_type	vm_snapshot_chain_size	iso_id	display_volume	format	min_iops	max_iops	hv_ss_reserve	provisioning_type
5329	10	2	64	NULL	1727	0	drs-test-02_2	42734603-a177-409b-92ac-ae15ca38d92a	10737418240	NULL	drs-test-02_2	NULL	1	NULL	NULL	ROOT	NULL	182	214	NULL	0	"2021-06-09 07:29:10"	NULL	NULL	NULL	Ready	"{"diskDeviceBusName":"scsi0:0","diskChain":["[iosctx01_1504_fp01_ns_0394] drs-test-02/drs-test-02_2.vmdk"]}"	0	NULL	NULL	214	1	OVA	NULL	NULL	NULL	fat
5330	10	2	64	55	1727	1	drs-test-02-data-01	265333cd-3676-4e2e-bf70-b42b5f8e1ee5	5368709120	/DC_Salzenforst/iosctx01_1504_fp01_ns_0394	drs-test-02	1	1	NULL	NULL	DATADISK	NULL	54	NULL	NULL	0	"2021-06-09 07:42:01"	"2021-06-09 07:42:15"	"2021-06-09 07:44:10"	NULL	Ready		6	NULL	NULL	NULL	1	OVA	NULL	NULL	0	fat

id	account_id	domain_id	pool_id	last_pool_id	instance_id	device_id	name	uuid	size	folder	path	pod_id	data_center_id	iscsi_name	host_ip	volume_type	pool_type	disk_offering_id	template_id	first_snapshot_backup_uuid	recreatable	created	attached	updated	removed	state	chain_info	update_count	disk_type	vm_snapshot_chain_size	iso_id	display_volume	format	min_iops	max_iops	hv_ss_reserve	provisioning_type
5329	10	2	64	NULL	1727	0	drs-test-02_2	42734603-a177-409b-92ac-ae15ca38d92a	10737418240	NULL	drs-test-02_2	NULL	1	NULL	NULL	ROOT	NULL	182	214	NULL	0	"2021-06-09 07:29:10"	NULL	NULL	NULL	Ready	"{"diskDeviceBusName":"scsi0:0","diskChain":["[iosctx01_1504_fp01_ns_0394] drs-test-02/drs-test-02_2.vmdk"]}"	0	NULL	NULL	214	1	OVA	NULL	NULL	NULL	fat
5330	10	2	64	55	1727	1	drs-test-02-data-01	265333cd-3676-4e2e-bf70-b42b5f8e1ee5	5368709120	/DC_Salzenforst/iosctx01_1504_fp01_ns_0394	drs-test-02	1	1	NULL	NULL	DATADISK	NULL	54	NULL	NULL	0	"2021-06-09 07:42:01"	"2021-06-09 07:42:15"	"2021-06-09 07:44:10"	NULL	Ready		6	NULL	NULL	NULL	1	OVA	NULL	NULL	0	fat

id	account_id	domain_id	pool_id	last_pool_id	instance_id	device_id	name	uuid	size	folder	path	pod_id	data_center_id	iscsi_name	host_ip	volume_type	pool_type	disk_offering_id	template_id	first_snapshot_backup_uuid	recreatable	created	attached	updated	removed	state	chain_info	update_count	disk_type	vm_snapshot_chain_size	iso_id	display_volume	format	min_iops	max_iops	hv_ss_reserve	provisioning_type
5329	10	2	64	NULL	1727	0	drs-test-02_2	42734603-a177-409b-92ac-ae15ca38d92a	10737418240	NULL	drs-test-02_2	NULL	1	NULL	NULL	ROOT	NULL	182	214	NULL	0	"2021-06-09 07:29:10"	NULL	NULL	NULL	Ready	"{"diskDeviceBusName":"scsi0:0","diskChain":["[iosctx01_1504_fp01_ns_0394] drs-test-02/drs-test-02_2.vmdk"]}"	0	NULL	NULL	214	1	OVA	NULL	NULL	NULL	fat
5330	10	2	64	55	1727	1	drs-test-02-data-01	265333cd-3676-4e2e-bf70-b42b5f8e1ee5	5368709120	/DC_Salzenforst/iosctx01_1504_fp01_ns_0394	drs-test-02	1	1	NULL	NULL	DATADISK	NULL	54	NULL	NULL	0	"2021-06-09 07:42:01"	"2021-06-09 07:42:15"	"2021-06-09 07:44:10"	NULL	Ready		6	NULL	NULL	NULL	1	OVA	NULL	NULL	0	fat

EXPECTED RESULTS

instance is booted properly without any errors

ACTUAL RESULTS

vm cannot be started, since volumes were not found anymore (_better say, were expected to be on its old location_)

[yadvr]

@mib1185 I could be wrong but generally it's not a good idea for storage DRS to be enabled as well as CloudStack (two sets of control planes doing operations). Did this use to work for you in older versions.
cc @harikrishna-patnala @nvazquez @sureshanaparti @andrijapanicsb @borisstoyanov @vladimirpetrov

[mib1185]

@rhtyd storage DRS is one of the main reason, why to use datastore clusters, those this should work absolutely transparent to CloudStack. Honestly there were already issues with that functionality -> #4674
In addition - as I know - CloudStack does not control load balancing of data between datastores inside a datastore cluster 🤔
This could not work on older version, since support for datastore cluster was introduced with 4.15 (see #4248).

[harikrishna-patnala]

@mib1185 may I know if this is happening only with ingested VMs or also on VMs deployed directly in CloudStack?

[mib1185]

@harikrishna-patnala issue also ocurs with cloudstack deployed instances - tested like follows

• deploy new instance with only root volume
• create and attach new data volume to instance
• migrate new data volume to same datastore-cluster as root-volume is located, since it was created on another available primary storage (maybe noteworthy that this was also the case when tested with ingested vm)
• perform storage DRS on vCenter
• stop instance
• start instance -> fails

[andrijapanicsb]

@harikrishna-patnala I don't recall we ever supported "STORAGE DRS" vs the regular "DRS" (which just live migrates VMs)

@mib1185 can you test Storage DRS with other "regular" datastore type (i.e. NFS or iSCSI etc) - I'm not aware we support this, but I might be wrong.

cc @rhtyd ^^^

[harikrishna-patnala]

@andrijapanicsb for specifically datastore clusters if the volumes are migrated within the datastore cluster, the management server syncs the volume state (not for the other datastore types)

[mib1185]

@andrijapanicsb I'm sorry, but we do not have iSCSI nor NFS datastores available on our test lab (even not in our productive environment). Nevertheless, I guess it is independent on used child-datastore-technology, since Storage DRS is part of datastore cluster logic on top of the child-datastores.

[andrijapanicsb]

@mib1185 NFS is so easy to add (on any Linux box, for testing purposes), but @harikrishna-patnala has just advised that this should work, if I read his reply correctly, so I will back up to my cave now and slowly disappear :)

[harikrishna-patnala]

@mib1185 how did you simulate DRS on data volume ? I tried to move it manually on vCenter but failed with lock issues since it is attached to the VM
On the other hand, I tried these steps and it is successful to me.

1. Created a VM with only root disk
2. Attached a data disk to VM (data disk on another primary storage)
3. Migrated the data disk to Datastore cluster
4. Simulate storage DRS on data disk. I manipulated the chain_info and pool_id of the data disk in 'volumes' table.
5. Stop instance
6. Start instance => succeeded

I also observed chain_info is missing in the "volumes" table entries that you posted. Can you observe the chain_info column of the "volumes" table and let us know

[mib1185]

Simulating storage DRS is very simples as you would only drag&drop the vm from one datastore to the other:

There was no manipulating of any db entries done by us.

Yeah, I also noticed that chain_info is empty for data volume.
Meanwhile I observed, that chain_info got lost (also folder and path is changed) right some moments after migrating data volume to data store cluster:

• "volumes" after additional data volumes is created (on different primary storage) and attached

id	account_id	domain_id	pool_id	last_pool_id	instance_id	device_id	name	uuid	size	folder	path	pod_id	data_center_id	iscsi_name	host_ip	volume_type	pool_type	disk_offering_id	template_id	first_snapshot_backup_uuid	recreatable	created	attached	updated	removed	state	chain_info	update_count	disk_type	vm_snapshot_chain_size	iso_id	display_volume	format	min_iops	max_iops	hv_ss_reserve	provisioning_type
5333	10	2	59	NULL	1729	0	ROOT-1729	43e29773-d630-4a59-8a22-c5aa3fd92fe9	17179869184	NULL	ROOT-1729_2	NULL	1	NULL	NULL	ROOT	NULL	182	223	NULL	0	"2021-06-09 10:57:26"	NULL	"2021-06-09 10:57:54"	NULL	Ready	"{"diskDeviceBusName":"scsi0:0","diskChain":["[iosctx01_1504_fp01_ns_0393] i-10-1729-VM/ROOT-1729_2.vmdk"]}"	2	NULL	NULL	NULL	1	OVA	NULL	NULL	0	fat
5334	10	2	55	NULL	1729	1	drs-test-04-data-01	91b72972-2442-4abb-b921-47a1fdff848b	10737418240	NULL	c15a6e7da9374f3aad71804b16b7fafe	NULL	1	NULL	NULL	DATADISK	NULL	54	NULL	NULL	0	"2021-06-09 10:58:42"	"2021-06-09 10:59:05"	"2021-06-09 10:59:05"	NULL	Ready	"{"diskDeviceBusName":"scsi0:1","diskChain":["[iosctx01_1504_fp01_ns_0396] i-10-1729-VM/c15a6e7da9374f3aad71804b16b7fafe.vmdk"]}"	4	NULL	NULL	NULL	1	OVA	NULL	NULL	0	fat

• "volumes" after additional data volumes is migrated to datastore cluster

id	account_id	domain_id	pool_id	last_pool_id	instance_id	device_id	name	uuid	size	folder	path	pod_id	data_center_id	iscsi_name	host_ip	volume_type	pool_type	disk_offering_id	template_id	first_snapshot_backup_uuid	recreatable	created	attached	updated	removed	state	chain_info	update_count	disk_type	vm_snapshot_chain_size	iso_id	display_volume	format	min_iops	max_iops	hv_ss_reserve	provisioning_type
5333	10	2	59	NULL	1729	0	ROOT-1729	43e29773-d630-4a59-8a22-c5aa3fd92fe9	17179869184	NULL	ROOT-1729_2	NULL	1	NULL	NULL	ROOT	NULL	182	223	NULL	0	"2021-06-09 10:57:26"	NULL	"2021-06-09 10:57:54"	NULL	Ready	"{"diskDeviceBusName":"scsi0:0","diskChain":["[iosctx01_1504_fp01_ns_0393] i-10-1729-VM/ROOT-1729_2.vmdk"]}"	2	NULL	NULL	NULL	1	OVA	NULL	NULL	0	fat
5334	10	2	64	55	1729	1	drs-test-04-data-01	91b72972-2442-4abb-b921-47a1fdff848b	10737418240	/DC_Salzenforst/iosctx01_1504_fp01_ns_0394	i-10-1729-VM	1	1	NULL	NULL	DATADISK	NULL	54	NULL	NULL	0	"2021-06-09 10:58:42"	"2021-06-09 10:59:05"	"2021-06-09 11:00:17"	NULL	Ready		6	NULL	NULL	NULL	1	OVA	NULL	NULL	0	fat

[harikrishna-patnala]

I suspect the chain_info being empty is the reason for failure, can you please try this. Before applying storage DRS on the data disk, can you detach and attach the data disk to the VM to reflect the chain info and then apply storage DRS. That should work.

By the way, I tried the same, migrating the data disk from nfs primary storage to datastore cluster but chain_info is not empty in my case. I'll try it again.

[mib1185]

By the way, I tried the same, migrating the data disk from nfs primary storage to datastore cluster but chain_info is not empty in my case. I'll try it again.

Please find attached the log file from management server for migration job execution, where afterwardy chain_info got lost:
MigrateVolumeCmdByAdmin.log