I am using the following link to setup my k3s setup with Truenas. I am only interested in NFS. I will not be doing ISCSI. After I applied the "helm upgrade --install --create-namespace --values truenas_nfs.yaml --namespace storage nfs democratic-csi/democratic-csi" I checked the pods and the controller is in a crashloopback state. The nodes are running. When I run the describe, I get the output below. Any suggestion?
Simple guide to use Kubernetes cluster with TrueNAS Scale over API - fenio/k8s-truenas
github.com
truenas_nfs.yaml
```
csiDriver:
name: "nfs"
storageClasses:
- name: nfs
defaultClass: false
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true
parameters:
fsType: nfs
detachedVolumesFromSnapshots: "false"
mountOptions:
- noatime
- nfsvers=3
secrets:
provisioner-secret:
controller-publish-secret:
node-stage-secret:
node-publish-secret:
controller-expand-secret:
volumeSnapshotClasses:
- name: nfs
parameters:
detachedSnapshots: "true"
driver:
config:
driver: freenas-api-nfs
instance_id:
httpConnection:
protocol: http
host: truenas.zbs.local
port: 80
apiKey: 2-ye8bjhvXrXQ63TekqTGVP5gCd6qW2hzJtC678A4gh6oCkCmeETt0M1fsWOUDgeav
allowInsecure: true
zfs:
datasetParentName: /mnt/Pool01/kubernetes_nfs
detachedSnapshotsDatasetParentName: /mnt/Pool01/kubernetes_nfs
datasetEnableQuotas: true
datasetEnableReservation: false
datasetPermissionsMode: "0777"
datasetPermissionsUser: 0
datasetPermissionsGroup: 0
nfs:
shareHost: truenas.zbs.local
shareAlldirs: false
shareAllowedHosts: []
shareAllowedNetworks: []
shareMaprootUser: root
shareMaprootGroup: root
shareMapallUser: ""
shareMapallGroup: "
```
kubectl describe po pod/nfs-democratic-csi-controller-68b8764d79-hcskx -n storage
Name: nfs-democratic-csi-controller-68b8764d79-hcskx
Namespace: storage
Priority: 0
Service Account: nfs-democratic-csi-controller-sa
Node: k3s-worker03.zbs.local/10.36.130.111
Start Time: Mon, 27 Nov 2023 18:56:53 -0500
Labels: app.kubernetes.io/component=controller-linux
app.kubernetes.io/csi-role=controller
app.kubernetes.io/instance=nfs
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=democratic-csi
pod-template-hash=68b8764d79
Annotations: checksum/configmap: 5b721ef46a66e2d07b598b06cf21ecddadfc20ca79bed5da6e0646fa18ad05fe
checksum/secret: e203b85b75fe92d33479fe0358327a8178cc5eeb2bde106f51c2f44a3f7935a0
Status: Running
IP: 10.42.4.8
IPs:
IP: 10.42.4.8
Controlled By: ReplicaSet/nfs-democratic-csi-controller-68b8764d79
Containers:
external-attacher:
Container ID: containerd://6bf2727fbb48f53c7aea4164e41da93c30745074a4f251a1da1f139812c541db
Image: registry.k8s.io/sig-storage/csi-attacher:v4.3.0
Image ID: registry.k8s.io/sig-storage/csi-attacher@sha256:4eb73137b66381b7b5dfd4d21d460f4b4095347ab6ed4626e0199c29d8d021af
Port: <none>
Host Port: <none>
Args:
--v=5
--leader-election
--leader-election-namespace=storage
--timeout=90s
--worker-threads=10
--csi-address=/csi-data/csi.sock
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 27 Nov 2023 18:57:24 -0500
Finished: Mon, 27 Nov 2023 18:57:25 -0500
Ready: False
Restart Count: 2
Environment: <none>
Mounts:
/csi-data from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jktkn (ro)
external-provisioner:
Container ID: containerd://c9b1156926ac6e1d9602eb5e1b046a74cfd7a47f0b7609450c229ea84cd843c0
Image: registry.k8s.io/sig-storage/csi-provisioner:v3.5.0
Image ID: registry.k8s.io/sig-storage/csi-provisioner@sha256:d078dc174323407e8cc6f0f9abd4efaac5db27838f1564d0253d5e3233e3f17f
Port: <none>
Host Port: <none>
Args:
--v=5
--leader-election
--leader-election-namespace=storage
--timeout=90s
--worker-threads=10
--extra-create-metadata
--csi-address=/csi-data/csi.sock
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 27 Nov 2023 18:57:25 -0500
Finished: Mon, 27 Nov 2023 18:57:25 -0500
Ready: False
Restart Count: 2
Environment:
NODE_NAME: (v1:spec.nodeName)
NAMESPACE: storage (v1:metadata.namespace)
POD_NAME: nfs-democratic-csi-controller-68b8764d79-hcskx (v1:metadata.name)
Mounts:
/csi-data from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jktkn (ro)
external-resizer:
Container ID: containerd://5d586bca5cf22a97526bb52bb61913825db66cf1a2ea7d5fa02989253815cead
Image: registry.k8s.io/sig-storage/csi-resizer:v1.8.0
Image ID: registry.k8s.io/sig-storage/csi-resizer@sha256:2e2b44393539d744a55b9370b346e8ebd95a77573064f3f9a8caf18c22f4d0d0
Port: <none>
Host Port: <none>
Args:
--v=5
--leader-election
--leader-election-namespace=storage
--timeout=90s
--workers=10
--csi-address=/csi-data/csi.sock
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Mon, 27 Nov 2023 18:57:25 -0500
Finished: Mon, 27 Nov 2023 18:57:25 -0500
Ready: False
Restart Count: 2
Environment:
NODE_NAME: (v1:spec.nodeName)
NAMESPACE: storage (v1:metadata.namespace)
POD_NAME: nfs-democratic-csi-controller-68b8764d79-hcskx (v1:metadata.name)
Mounts:
/csi-data from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jktkn (ro)
external-snapshotter:
Container ID: containerd://5f9d012b064dc3deae0ff9c04471af43ae7a19d74653d251173a46b2c2e1dce2
Image: registry.k8s.io/sig-storage/csi-snapshotter:v6.2.2
Image ID: registry.k8s.io/sig-storage/csi-snapshotter@sha256:becc53e25b96573f61f7469923a92fb3e9d3a3781732159954ce0d9da07233a2
Port: <none>
Host Port: <none>
Args:
--v=5
--leader-election
--leader-election-namespace=storage
--timeout=90s
--worker-threads=10
--csi-address=/csi-data/csi.sock
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 27 Nov 2023 18:57:25 -0500
Finished: Mon, 27 Nov 2023 18:57:25 -0500
Ready: False
Restart Count: 2
Environment:
NODE_NAME: (v1:spec.nodeName)
NAMESPACE: storage (v1:metadata.namespace)
POD_NAME: nfs-democratic-csi-controller-68b8764d79-hcskx (v1:metadata.name)
Mounts:
/csi-data from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jktkn (ro)
csi-driver:
Container ID: containerd://27d4b670c84ce5a4876f9977117238e0504bf94de1d9dc667199a23eee0ed9a6
Image: docker.io/democraticcsi/democratic-csi:latest
Image ID: docker.io/democraticcsi/democratic-csi@sha256:1b526bb4e1634ce84534a7a51b038efa91335767c625dcec62cdc42a744e2bcd
Port: <none>
Host Port: <none>
Args:
--csi-version=1.5.0
--csi-name=nfs
--driver-config-file=/config/driver-config-file.yaml
--log-level=info
--csi-mode=controller
--server-socket=/csi-data/csi.sock.internal
State: Running
Started: Mon, 27 Nov 2023 18:57:07 -0500
Ready: True
Restart Count: 0
Liveness: exec [bin/liveness-probe --csi-version=1.5.0 --csi-address=/csi-data/csi.sock.internal] delay=10s timeout=15s period=60s #success=1 #failure=3
Environment:
NODE_EXTRA_CA_CERTS: /tmp/certs/extra-ca-certs.crt
Mounts:
/config from config (rw)
/csi-data from socket-dir (rw)
/tmp/certs from extra-ca-certs (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jktkn (ro)
csi-proxy:
Container ID: containerd://76cb7416d15ca9a09c74ba9172b8dc58af214510de68205622608c181250cb48
Image: docker.io/democraticcsi/csi-grpc-proxy:v0.5.3
Image ID: docker.io/democraticcsi/csi-grpc-proxy@sha256:4d65ca1cf17d941a8df668b8fe2f1c0cfa512c8b0dbef3ff89a4cd405e076923
Port: <none>
Host Port: <none>
State: Running
Started: Mon, 27 Nov 2023 18:57:07 -0500
Ready: True
Restart Count: 0
Environment:
BIND_TO: unix:///csi-data/csi.sock
PROXY_TO: unix:///csi-data/csi.sock.internal
Mounts:
/csi-data from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jktkn (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
socket-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
config:
Type: Secret (a volume populated by a Secret)
SecretName: nfs-democratic-csi-driver-config
Optional: false
extra-ca-certs:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: nfs-democratic-csi
Optional: false
kube-api-access-jktkn:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 60s default-scheduler Successfully assigned storage/nfs-democratic-csi-controller-68b8764d79-hcskx to k3s-worker03.zbs.local
Normal Pulling 59s kubelet Pulling image "registry.k8s.io/sig-storage/csi-attacher:v4.3.0"
Normal Pulled 57s kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-attacher:v4.3.0" in 2.872173535s (2.87220083s including waiting)
Normal Created 57s kubelet Created container external-attacher
Normal Started 56s kubelet Started container external-attacher
Normal Pulling 56s kubelet Pulling image "registry.k8s.io/sig-storage/csi-provisioner:v3.5.0"
Normal Pulled 53s kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-provisioner:v3.5.0" in 3.063151953s (3.063161993s including waiting)
Normal Created 53s kubelet Created container external-provisioner
Normal Started 53s kubelet Started container external-provisioner
Normal Pulling 53s kubelet Pulling image "registry.k8s.io/sig-storage/csi-resizer:v1.8.0"
Normal Started 50s kubelet Started container external-resizer
Normal Pulling 50s kubelet Pulling image "registry.k8s.io/sig-storage/csi-snapshotter:v6.2.2"
Normal Pulled 50s kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-resizer:v1.8.0" in 3.018921829s (3.018935868s including waiting)
Normal Pulling 47s kubelet Pulling image "docker.io/democraticcsi/democratic-csi:latest"
Normal Pulled 47s kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-snapshotter:v6.2.2" in 2.857660298s (2.857676928s including waiting)
Normal Created 47s kubelet Created container external-snapshotter
Normal Started 47s kubelet Started container external-snapshotter
Normal Pulled 47s kubelet Successfully pulled image "docker.io/democraticcsi/democratic-csi:latest" in 271.233783ms (271.253468ms including waiting)
Normal Created 47s kubelet Created container csi-driver
Normal Started 46s kubelet Started container csi-driver
Normal Pulled 46s kubelet Container image "docker.io/democraticcsi/csi-grpc-proxy:v0.5.3" already present on machine
Normal Created 46s kubelet Created container csi-proxy
Normal Started 46s kubelet Started container csi-proxy
Normal Created 43s (x2 over 50s) kubelet Created container external-resizer
Normal Pulled 43s kubelet Container image "registry.k8s.io/sig-storage/csi-resizer:v1.8.0" already present on machine
