Knowledge Base
linbit.com Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Toggle Dark/Light/Auto mode Back to homepage
  1. Knowledge Base Topics
  2. /
  3. LINSTOR
  4. /
  5. Kubernetes
  6. /
  7. Read Write Many (RWX) for LINSTOR Volumes

Read Write Many (RWX) for LINSTOR Volumes

LINSTOR® creates block devices that are accessible only from a single pod: Read Write Once (RWO). However, it’s possible to create an NFS pod that shares a LINSTOR volume with many pods, indirectly enabling RWX support.

You need to have setup a LINSTOR storageClass in Kubernetes that the NFS server pod can use for its persistent storage. Verify that you set the appropriate storageClass name in the following persistent volume (PV) definition that will be used by the NFS server pod (the example below uses linstor-csi-lvm-thin-r3). Also, set the size of the NFS server’s volume accordingly:

cat << EOF > nfs-server-pv.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-pv-provisioning-demo
  labels:
    demo: nfs-pv-provisioning
spec:
  accessModes: [ ReadWriteOnce ]
  resources:
    requests:
      storage: 4Gi
  # some existing LINSTOR storageClass
  storageClassName: linstor-csi-lvm-thin-r3
EOF
kubectl create -f nfs-server-pv.yaml

Then, create the nfs-server pods (controlled by a replicationController):

cat << EOF > nfs-server-rc.yaml
apiVersion: v1
kind: ReplicationController
metadata:
  name: nfs-server
spec:
  replicas: 1
  selector:
    role: nfs-server
  template:
    metadata:
      labels:
        role: nfs-server
    spec:
      containers:
      - name: nfs-server
        image: k8s.gcr.io/volume-nfs:0.8
        ports:
          - name: nfs
            containerPort: 2049
          - name: mountd
            containerPort: 20048
          - name: rpcbind
            containerPort: 111
        securityContext:
          privileged: true
        volumeMounts:
          - mountPath: /exports
            name: mypvc
      volumes:
        - name: mypvc
          persistentVolumeClaim:
            claimName: nfs-pv-provisioning-demo
EOF
kubectl create -f nfs-server-rc.yaml

Create the service for the NFS server, then get and set the ClusterIP for the service in an environment variable:

cat << EOF > nfs-server-service.yaml
kind: Service
apiVersion: v1
metadata:
  name: nfs-server
spec:
  ports:
    - name: nfs
      port: 2049
    - name: mountd
      port: 20048
    - name: rpcbind
      port: 111
  selector:
    role: nfs-server
EOF
kubectl create -f nfs-server-service.yaml

NFSIP=$(kubectl describe services nfs-server | grep ^IP\: | awk '{print $2}')
echo $NFSIP

You now have an NFS server running in your cluster exporting a file system backed by LINSTOR. You could mount this NFS share manually within your application’s pods, but it’s more likely that you’d want to consume this share as a PV. Using a PV creates an indirection to the shared file system, and you won’t have to hard code the NFS server’s IP into your pod.

Create the PV and persistent volume claim (PVC) for your applications to consume the shared NFS file system. If you’re copy and pasting these commands exactly as they’re written, the $NFSIP environment variable should be evaluated and placed into the definition when the file is created. If you’re not following along like that, just be sure to replace $NFSIP with the correct value for your cluster. Also, the size of the storage here can be set to anything less than what the NFS server’s volume was set to:

cat << EOF > nfs-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: nfs
spec:
  capacity:
    storage: 1Mi
  accessModes:
    - ReadWriteMany
  nfs:
    server: $NFSIP
    path: /
  mountOptions:
    - nfsvers=4.2
EOF
kubectl create -f nfs-pv.yaml

cat << EOF > nfs-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs
spec:
  accessModes:
    - ReadWriteMany
  storageClassName:
  resources:
    requests:
      storage: 1Mi
EOF
kubectl create -f nfs-pvc.yaml

You can now consume the nfs PVC from many pods. Also, you can use the same NFS server pod or PV to host more than one share, but you’ll have to create directories for each share in the NFS server’s /exports/ directory, and a separate PV or PVC for indirection of each.

A simple way to test the RWX PV and PVC created above would be to create a pod with two (or more) containers updating the NFS share or PVC, while another pod with two (or more) containers reads data from the NFS share or PVC.

Test backend updating the NFS share or PVC with dummy data (hostname and date):

cat << EOF > nfs-busybox-backend.yaml
# This mounts the nfs volume claim into /mnt and continuously
# overwrites /mnt/index.html with the time and hostname of the pod.
apiVersion: v1
kind: ReplicationController
metadata:
  name: nfs-busybox
spec:
  replicas: 2
  selector:
    name: nfs-busybox
  template:
    metadata:
      labels:
        name: nfs-busybox
    spec:
      containers:
      - image: busybox
        command:
          - sh
          - -c
          - 'while true; do date > /mnt/index.html; hostname >> /mnt/index.html; sleep $(($RANDOM % 5 + 5)); done'
        imagePullPolicy: IfNotPresent
        name: busybox
        volumeMounts:
          # name must match the volume name below
          - name: nfs
            mountPath: /mnt
      volumes:
      - name: nfs
        persistentVolumeClaim:
          claimName: nfs
EOF
kubectl create -f nfs-busybox-backend.yaml

Test front-end pods and service for reading the dummy data from the NFS share or PVC:

cat << EOF> nfs-nginx-frontend.yaml
# This pod mounts the nfs volume claim into /usr/share/nginx/html and
# serves a simple web page.
apiVersion: v1
kind: ReplicationController
metadata:
  name: nfs-web
spec:
  replicas: 2
  selector:
    role: web-frontend
  template:
    metadata:
      labels:
        role: web-frontend
    spec:
      containers:
      - name: web
        image: nginx
        ports:
          - name: web
            containerPort: 80
        volumeMounts:
            # name must match the volume name below
            - name: nfs
              mountPath: /usr/share/nginx/html
      volumes:
      - name: nfs
        persistentVolumeClaim:
          claimName: nfs
EOF
kubectl create -f nfs-nginx-frontend.yaml

cat << EOF > nfs-frontend-service.yaml
kind: Service
apiVersion: v1
metadata:
  name: nfs-web
spec:
  ports:
    - port: 80
  selector:
    role: web-frontend
EOF
kubectl create -f nfs-frontend-service.yaml

You can now, from a pod or a Kubernetes cluster node, access the front end to see the updates or changes:

WWWIP=$(kubectl describe services nfs-web | grep ^IP\: | awk '{print $2}')
echo $WWWIP
watch -d -n10 curl http://$WWWIP

Reviewed 2021/12/08 – MDK