kubernetes persistence volume issues

sulfred
sulfred Member Posts: 20 ✭
in General Discussions #1

Hi all,


I am using k8s to deploy solace pubsub +. While setting the persistence volumes, I hit a permission issue and here are some error messages:

Starting PubSub+ Software Event Broker Container: Sun Jan 15 05:04:27 UTC 2023
Setting umask to 022
Sun Jan 15 05:04:27 UTC 2023 INFO: startup-broker.sh-Node ordinal: 0
Sun Jan 15 05:04:27 UTC 2023 INFO: startup-broker.sh-Waiting for management API to become available
SolOS Version: soltr_10.2.1.32
<returnInfo><errorInfo>management host is not responding</errorInfo></returnInfo>
Sun Jan 15 05:04:27 UTC 2023 INFO: startup-broker.sh-Waited 0 seconds, Management API not yet accessible
ERROR: Required directory /var/lib/solace is not writable by current user
ERROR: Unable to create required directory /var/lib/solace/diagnostics: [Errno 13] Permission denied: '/var/lib/solace/diagnostics'
ERROR: Required directory /var/lib/solace/diagnostics does not exist
ERROR: Unable to create required directory /var/lib/solace/spool: [Errno 13] Permission denied: '/var/lib/solace/spool'
ERROR: Required directory /var/lib/solace/spool does not exist
ERROR: Unable to create required directory /var/lib/solace/spool-cache: [Errno 13] Permission denied: '/var/lib/solace/spool-cache'
ERROR: Required directory /var/lib/solace/spool-cache does not exist
ERROR: Unable to create required directory /var/lib/solace/spool-cache-backup: [Errno 13] Permission denied: '/var/lib/solace/spool-cache-backup'
ERROR: Required directory /var/lib/solace/spool-cache-backup does not exist
ERROR: Unable to create required directory /var/lib/solace/spool-cache-backup/image: [Errno 13] Permission denied: '/var/lib/solace/spool-cache-backup'
ERROR: Required directory /var/lib/solace/spool-cache-backup/image does not exist


I am using:

  • ubuntu 20.04
  • kubectl version
$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:58:30Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:49:09Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/amd64"}
  • minikube 1.28.0
  • helm 3.10.3

I install solace with commands:

$ helm repo add solace https://solaceproducts.github.io/pubsubplus-kubernetes-quickstart/helm-charts/
$ helm pull solace/pubsubplus --version 3.2.0 --untar ture
# prepare persistent storage
$ kubectl apply -f ./pubsubplus/custom_config/dev/storage.yaml
# start solace pub sub
$ helm install solace-mb ./pubsubplus/ --values ./pubsubplus/custom_config/dev/values.customer.yaml

values.customer.yaml

solace:
  usernameAdminPassword: adminpw

storage:
  persistent: true
  customVolumeMount: |
    persistentVolumeClaim:
      claimName: solace-volume
  useStorageClass: standard
  useStorageGroup: true

storage.yaml

# storage request
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: solace-volume
spec:
  storageClassName: solace-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi
---
# storage implement
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage
spec:
  storageClassName: solace-storage
  capacity:
    storage: 30Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/volumes/solaceVolume/"
    type: DirectoryOrCreate

From minikube vm, I see that the volume is created successfully and data are put to there by solace.

$ cd /mnt/volumes/solaceVolume/
$ ls -hal
total 0
drwxr-xr-x 8 root root 160 Jan 15 04:18 .
drwxr-xr-x 3 root root  60 Jan 15 04:18 ..
drwxr-xr-x 2 root root  40 Jan 15 04:18 adb
drwxr-xr-x 2 root root  40 Jan 15 04:18 diags
drwxr-xr-x 3 root root  60 Jan 15 04:18 internalSpool
drwxr-xr-x 2 root root  40 Jan 15 04:18 jail
drwxr-xr-x 2 root root  40 Jan 15 04:18 softAdb
drwxr-xr-x 2 root root  40 Jan 15 04:18 var


It seems that there is some configuration missing. I would like to ask if anyone has idea with this?


Thanks

Tagged:

Best Answer

  • pkondrat
    pkondrat Member, Employee Posts: 27 Solace Employee
    #2 Answer ✓

    Hi @sulfred,

    The issue is with the hostPath in your persistent volume. There are some limitations when using the hostPath. From the Kubernetes docs (Volumes | Kubernetes):

    • "The files or directories created on the underlying hosts are only writable by root. You either need to run your process as root in a privileged Container or modify the file permissions on the host to be able to write to a hostPath volume."

    You can see that the directories that the broker created are owned by root and only writeable by root. The processes in the container by default run as user 1000001. You can open up the permissions to allow group write or change the ownership to 1000001. The other option (and probably easiest) is to use Minikube's default storage class.

    Best Regards,

    Paul

Answers

Categories

This Month's Leaders

This Week's Leaders