StorageClass Missing with CSI Driver Enabled

[KuasarCloud]

Hey team,

I'm deploying a Kubernetes 1.33.1 on ACS 4.22 with KVM. The cluster works but there is not StorageClass associated

➜  ~ kubectl get storageclass
No resources found
➜  ~ kubectl get storageclasses
No resources found 

When checking on the CSI pods, everything looks good except for this one in this containers - external-provisioner, external-attacher, external-resizer, csi-snapshotter

I0119 22:10:35.407041       1 leaderelection.go:255] failed to acquire lease kube-system/external-snapshotter-leader-csi-cloudstack-apache-org
I0119 22:10:55.752645       1 leaderelection.go:354] lock is held by cloudstack-csi-controller-85f5b85cdb-srlf9 and has not yet expired
I0119 22:10:55.752662       1 leaderelection.go:255] failed to acquire lease kube-system/external-snapshotter-leader-csi-cloudstack-apache-org
I0119 22:11:17.591492       1 leaderelection.go:354] lock is held by cloudstack-csi-controller-85f5b85cdb-srlf9 and has not yet expired
I0119 22:11:17.591550       1 leaderelection.go:255] failed to acquire lease kube-system/external-snapshotter-leader-csi-cloudstack-apache-org
I0119 22:11:38.516006       1 leaderelection.go:354] lock is held by cloudstack-csi-controller-85f5b85cdb-srlf9 and has not yet expired
I0119 22:11:38.516020       1 leaderelection.go:255] failed to acquire lease kube-system/external-snapshotter-leader-csi-cloudstack-apache-org

[Pearl1594]

Hi @KuasarCloud, you will need to explicitly create the resources like StorageClass, PVC, etc as one would need to specify the disk offering they'd want to use when when creating the storage class.
The logs you've pasted are normal, this is because the other cloudstack-csi-controller is the elected as the leader.

[KuasarCloud]

Thanks! after creating the StorageClass and I try to create a pvc I'm getting these errors.

Not sure why it's attempting with these users ids

kubectl logs -f -l app.kubernetes.io/name=cloudstack-csi-controller

I0322 19:10:00.451891       1 volumes.go:80] "CloudStack API call" command="ListVolumes" params={"name":"pvc-9a52f9ca-7964-4a99-9b27-facea24f3150"}

E0322 19:10:00.460866       1 driver.go:96] "GRPC method failed" err="rpc error: code = Internal desc = CloudStack error: CloudStack API error 432 (CSExceptionErrorCode: 9999): The API [listVolumes] does not exist or is not available for the account for user id [0b9a025e-4d80-44bc-9efc-463c5c66ca0a]." method="/csi.v1.Controller/CreateVolume"

I0322 19:10:01.465483       1 volumes.go:80] "CloudStack API call" command="ListVolumes" params={"name":"pvc-9a52f9ca-7964-4a99-9b27-facea24f3150"}

E0322 19:10:01.473147       1 driver.go:96] "GRPC method failed" err="rpc error: code = Internal desc = CloudStack error: CloudStack API error 432 (CSExceptionErrorCode: 9999): The API [listVolumes] does not exist or is not available for the account for user id [4cab7da1-fa30-4158-9147-e8005a93f366]." method="/csi.v1.Controller/CreateVolume"

I0322 19:10:03.476614       1 volumes.go:80] "CloudStack API call" command="ListVolumes" params={"name":"pvc-9a52f9ca-7964-4a99-9b27-facea24f3150"}

E0322 19:10:03.484031       1 driver.go:96] "GRPC method failed" err="rpc error: code = Internal desc = CloudStack error: CloudStack API error 432 (CSExceptionErrorCode: 9999): The API [listVolumes] does not exist or is not available for the account for user id [41358fa6-415c-4815-ad91-824c0a5653ad]." method="/csi.v1.Controller/CreateVolume"

I0322 19:10:07.487026       1 volumes.go:80] "CloudStack API call" command="ListVolumes" params={"name":"pvc-9a52f9ca-7964-4a99-9b27-facea24f3150"}

E0322 19:10:07.493864       1 driver.go:96] "GRPC method failed" err="rpc error: code = Internal desc = CloudStack error: CloudStack API error 432 (CSExceptionErrorCode: 9999): The API [listVolumes] does not exist or is not available for the account for user id [0991e1a0-0513-410b-9b82-0d40842ded0b]." method="/csi.v1.Controller/CreateVolume"

[chunkyen]

You need to use a service account in cloudstack with the listvolumes permission added to the CKS cluster as secrets.

[KuasarCloud]

Thanks @chunkyen for replying. Not sure what do you mean by create a service account in Cloudstack, any doc I can follow please?

[chunkyen]

I supposed you are using CKS? Are you using the default admin account to provision the cluster? If so, the secret should already contain the default service account used for CKS, which is admin-kubeadmin.

I am not sure why the CSI is trying to use different accounts when I look at your logs.
Can you double check what is in Kube-System - Secrets - cloudstack-secret ?

[KuasarCloud]

Hey @chunkyen

Yes, confirmed — this is a CKS cluster deployed under a specific project using a project-scoped user (not the default admin account). The cloudstack-secret in kube-system is provisioned automatically during cluster deployment and reflects the credentials of that project user, which is why the CSI driver is picking up a different account than admin-kubeadmin.

That would explain the behavior you're seeing in the logs.

[KuasarCloud]

@Pearl1594 any comments? thanks!

[Pearl1594]

@KuasarCloud Unfortunately Im not hitting this issue. I created a domain, and created an account/ user with User role in that domain. I then logged in as that user, created a project and in that project, created a CKS cluster with csi enabled, and was able to do so successfully.

Just to help invstigate what's happening, can you check what role does the user have, and does that role have the necessary permissions?

[KuasarCloud]

Thanks @Pearl1594 for responding

I created the CKS with k8s version 1.33, using the same way you mentioned above and then created the StorageClass

cat 0-storageclass.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cloudstack-custom
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.cloudstack.apache.org
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false
parameters:
  csi.cloudstack.apache.org/disk-offering-id: ea0a115a-0781-4230-bb4f-7b8c67fb9f36

➜  ~ kubectl apply -f 0-storageclass.yaml
storageclass.storage.k8s.io/cloudstack-custom created

➜  ~ kubectl get sc
NAME                          PROVISIONER                 RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
cloudstack-custom (default)   csi.cloudstack.apache.org   Delete          WaitForFirstConsumer   false                  10s


➜  ~ kubectl apply -f pvc.yaml
persistentvolumeclaim/example-pvc created
➜  ~ kubectl get pvc
NAME          STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS        VOLUMEATTRIBUTESCLASS   AGE
example-pvc   Pending                                      cloudstack-custom   <unset>                 6s
➜  ~ kubectl apply -f pod.yaml
pod/example-pod created

E0408 21:57:23.031870       1 driver.go:96] "GRPC method failed" err="rpc error: code = Internal desc = CloudStack error: CloudStack API error 432 (CSExceptionErrorCode: 9999): The API [listVolumes] does not exist or is not available for the account for user id [26b7ae4d-6279-4932-aa7e-fd7c27b246c1]." method="/csi.v1.Controller/CreateVolume"
I0408 21:57:27.034407       1 volumes.go:80] "CloudStack API call" command="ListVolumes" params={"name":"pvc-1bfe1ea0-8ab0-4edd-b4d3-3b0891499c7e"}
E0408 21:57:27.041125       1 driver.go:96] "GRPC method failed" err="rpc error: code = Internal desc = CloudStack error: CloudStack API error 432 (CSExceptionErrorCode: 9999): The API [listVolumes] does not exist or is not available for the account for user id [1acb3d55-8f0c-4fa8-a8eb-a3302c40e27c]." method="/csi.v1.Controller/CreateVolume"
I0408 21:57:35.044025       1 volumes.go:80] "CloudStack API call" command="ListVolumes" params={"name":"pvc-1bfe1ea0-8ab0-4edd-b4d3-3b0891499c7e"}
E0408 21:57:35.050280       1 driver.go:96] "GRPC method failed" err="rpc error: code = Internal desc = CloudStack error: CloudStack API error 432 (CSExceptionErrorCode: 9999): The API [listVolumes] does not exist or is not available for the account for user id [be722077-cdbc-4a56-8286-52a9268dcb40]." method="/csi.v1.Controller/CreateVolume"
^C%

[chunkyen]

Hi like what @Pearl1594 suggested, please check the role assignment for your project service account used for CKS CSI.