Using LINSTOR in Linode Kubernetes Engine (LKE) for DRBD Replicated Persistent Storage

The LINSTOR Operator for Kubernetes currently is capable of compiling DRBD® on RHEL, Ubuntu, and SLES based Kubernetes workers with the proper build dependencies installed. However, LKE uses Debian Bullseye based Kubernetes workers, which the LINSTOR Operator does not have an appropriate KernelModuleInjectionImage for at the time of writing (Dec. 2022). Therefore, the DRBD kernel module must be compiled and installed directly on the LKE worker nodes before deploying the LINSTOR Operator, in addition to LINSTOR®’s generic prerequisites.

Prerequisites and Considerations for Deploying LINSTOR into LKE

Attaching Linode Volume(s) to Each LKE Worker

A generic requirement for deploying LINSTOR into Kubernetes is having an empty block device for LINSTOR to use as a storage pool for the volumes that LINSTOR will provision. You can attach many volumes to each worker. These volumes should remain unformatted and unused, or else LINSTOR will refuse to use them.

In the image below you will see an example of a Linode with an additional unused Linode volume attached.

Screenshot of unused LINSTOR Linode volume attached to a
Linode

This volume attaches to the Linode as /dev/sdb which will be defined as a storage pool for LINSTOR in the Helm deployment step. Each node in your LKE cluster should be configured with its own additional Linode volume.

Building and Installing DRBD on the LKE Workers

SSH into each of your LKE workers and install the necessary packages for compiling and installing the DRBD kernel module:

# apt install linux-headers-$(uname -r) build-essential

Then, download the DRBD 9 kernel module source tar file of your choice from https://pkg.linbit.com/. At the time of writing, the latest currently recommended DRBD kernel module is version 9.1.12. Update the version number that is set in the BASH variable command that follows if a newer 9.1.x module has become available:

# DRBD_VERS=9.1.12
# curl -LO https://pkg.linbit.com//downloads/drbd/9/drbd-$DRBD_VERS.tar.gz
# tar xvf drbd-$DRBD_VERS.tar.gz
# cd drbd-$DRBD_VERS/
# make && make install
# modprobe drbd

You can verify that the correct DRBD kernel module has been loaded by running the following command and verifying the first line that reads, version: 9.x.x, matches the DRBD kernel module version you intended to build:

# cat /proc/drbd
version: 9.1.12 (api:2/proto:86-121)
GIT-hash: 86ec2326fef3aede9f4d46f52bfd35aac4d5eb7e build by root@lke83867-128388-639a05ada20c, 2022-12-14 20:31:50
Transports (api:18):

Disable the DRBD User Mode Helper on the LKE Workers

Finally, we will disable the DRBD user mode helper. This feature of the DRBD module enables running user configured commands on changes in DRBD state. When using DRBD within containers, it could confuse programs, such as the default drbdadm, that expect to know about all configured DRBD resources. To prevent any issues, you have to set the DRBD module parameter usermode_helper to disabled on each LKE worker by using the following commands:

# echo -n disabled > /sys/module/drbd/parameters/usermode_helper
# echo options drbd usermode_helper=disabled > /etc/modprobe.d/drbd.conf

Verify that the commands above succeeded and that the usermode_helper is now set to disabled and not drbdadm:

# cat /sys/module/drbd/parameters/usermode_helper
disabled

Deployment

With the prerequisites taken care of, all that’s left is to deploy the LINSTOR Operator. Deployment of the LINSTOR Operator follows the instructions found in the LINSTOR User’s Guide except that the following Helm options must be used because you have compiled and installed DRBD from source code:

operator:
  satelliteSet:
    kernelModuleInjectionImage: drbd.io/amd64/drbd9-jammy
    kernelModuleInjectionMode: DepsOnly

For completeness, the following commands include Helm options used to deploy LINSTOR into LKE where each LKE worker has a single additional Linode volume (/dev/sdb) attached. This additional volume will be used by LINSTOR as a thin LVM backed storage pool named thin-lvm:

# helm repo add linstor https://charts.linstor.io
# helm repo update


# kubectl create namespace linstor
# kubectl create secret docker-registry drbdiocred -n linstor \
    --docker-server=drbd.io --docker-username=$LBUSER \
    --docker-email=$LBEMAIL --docker-password=$LBPASS

# cat << EOF > linstor-op-vals.yaml
operator:
  controller:
    dbConnectionURL: k8s
  satelliteSet:
    storagePools:
      lvmThinPools:
      - name: lvm-thin
        thinVolume: thinpool
        volumeGroup:
        devicePaths:
        - /dev/sdb
    kernelModuleInjectionImage: drbd.io/amd64/drbd9-jammy
    kernelModuleInjectionMode: DepsOnly
etcd:
  enabled: false
stork:
  enabled: false
csi:
  enableTopology: true
EOF

# helm install -n linstor -f ./linstor-op-vals.yaml linstor-op linstor/linstor

Next, you can continue to configure LINSTOR’s storage classes in Kubernetes as outlined in the LINSTOR User’s Guide.

Written on 12/14/2022 - MDK

Reviewed on 12/15/2022 - MAT