kubernetes persistence volume issues

General Discussions
1

Hi all,

I am using k8s to deploy solace pubsub +. While setting the persistence volumes, I hit a permission issue and here are some error messages:
Starting PubSub+ Software Event Broker Container: Sun Jan 15 05:04:27 UTC 2023
Setting umask to 022
Sun Jan 15 05:04:27 UTC 2023 INFO: startup-broker.sh-Node ordinal: 0
Sun Jan 15 05:04:27 UTC 2023 INFO: startup-broker.sh-Waiting for management API to become available
SolOS Version: soltr_10.2.1.32
management host is not responding
Sun Jan 15 05:04:27 UTC 2023 INFO: startup-broker.sh-Waited 0 seconds, Management API not yet accessible
ERROR: Required directory /var/lib/solace is not writable by current user
ERROR: Unable to create required directory /var/lib/solace/diagnostics: [Errno 13] Permission denied: ‘/var/lib/solace/diagnostics’
ERROR: Required directory /var/lib/solace/diagnostics does not exist
ERROR: Unable to create required directory /var/lib/solace/spool: [Errno 13] Permission denied: ‘/var/lib/solace/spool’
ERROR: Required directory /var/lib/solace/spool does not exist
ERROR: Unable to create required directory /var/lib/solace/spool-cache: [Errno 13] Permission denied: ‘/var/lib/solace/spool-cache’
ERROR: Required directory /var/lib/solace/spool-cache does not exist
ERROR: Unable to create required directory /var/lib/solace/spool-cache-backup: [Errno 13] Permission denied: ‘/var/lib/solace/spool-cache-backup’
ERROR: Required directory /var/lib/solace/spool-cache-backup does not exist
ERROR: Unable to create required directory /var/lib/solace/spool-cache-backup/image: [Errno 13] Permission denied: ‘/var/lib/solace/spool-cache-backup’
ERROR: Required directory /var/lib/solace/spool-cache-backup/image does not exist

I am using:
ubuntu 20.04 kubectl version > $ kubectl version

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:“1”, Minor:“26”, GitVersion:“v1.26.0”, GitCommit:“b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d”, GitTreeState:“clean”, BuildDate:“2022-12-08T19:58:30Z”, GoVersion:“go1.19.4”, Compiler:“gc”, Platform:“linux/amd64”}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:“1”, Minor:“25”, GitVersion:“v1.25.3”, GitCommit:“434bfd82814af038ad94d62ebe59b133fcb50506”, GitTreeState:“clean”, BuildDate:“2022-10-12T10:49:09Z”, GoVersion:“go1.19.2”, Compiler:“gc”, Platform:“linux/amd64”}
minikube 1.28.0 helm 3.10.3 > I install solace with commands:
$ helm repo add solace https://solaceproducts.github.io/pubsubplus-kubernetes-quickstart/helm-charts/
$ helm pull solace/pubsubplus --version 3.2.0 --untar ture

prepare persistent storage

$ kubectl apply -f ./pubsubplus/custom_config/dev/storage.yaml

start solace pub sub

$ helm install solace-mb ./pubsubplus/ --values ./pubsubplus/custom_config/dev/values.customer.yaml
values.customer.yaml
solace:
  usernameAdminPassword: adminpw
storage:
  persistent: true
  customVolumeMount: |
    persistentVolumeClaim:
      claimName: solace-volume
  useStorageClass: standard
  useStorageGroup: true
storage.yaml

storage request

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: solace-volume
spec:
  storageClassName: solace-storage
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

storage implement

apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage
spec:
  storageClassName: solace-storage
  capacity:
    storage: 30Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: “/mnt/volumes/solaceVolume/”
    type: DirectoryOrCreate
From minikube vm, I see that the volume is created successfully and data are put to there by solace.
$ cd /mnt/volumes/solaceVolume/
$ ls -hal
total 0
drwxr-xr-x 8 root root 160 Jan 15 04:18 .
drwxr-xr-x 3 root root  60 Jan 15 04:18 …
drwxr-xr-x 2 root root  40 Jan 15 04:18 adb
drwxr-xr-x 2 root root  40 Jan 15 04:18 diags
drwxr-xr-x 3 root root  60 Jan 15 04:18 internalSpool
drwxr-xr-x 2 root root  40 Jan 15 04:18 jail
drwxr-xr-x 2 root root  40 Jan 15 04:18 softAdb
drwxr-xr-x 2 root root  40 Jan 15 04:18 var

It seems that there is some configuration missing. I would like to ask if anyone has idea with this?

Thanks

2

Hi @sulfred , how you doing? I am asking around for someone with Kubernetes experience to take a look at this ticket. But just read through it more carefully and have some questions:
In the very first code block, you seem to be getting permission errors for writing to /var/lib/solace/ but your yaml config file is referencing /mnt/volumes/solaceVolume/ … is this intentional, or a copy-paste error from a different configuration?
I’m assuming you’re not running this as root, due to permission errors. Have you tried (if you can) running as root?
And if running as non-root, there’s some new documentation here you might want to take a look at? Rootless Containers
What version of PubSub+ broker are you installing? I assume it’s the most recent?
Anyhow, I’ll ask around internally if someone can take a look at this.

3

Hi @sulfred ,
The issue is with the hostPath in your persistent volume. There are some limitations when using the hostPath. From the Kubernetes docs ( Volumes | Kubernetes ):
“The files or directories created on the underlying hosts are only writable by root. You either need to run your process as root in a  privileged Container  or modify the file permissions on the host to be able to write to a  hostPath  volume.” You can see that the directories that the broker created are owned by root and only writeable by root. The processes in the container by default run as user 1000001. You can open up the permissions to allow group write or change the ownership to 1000001. The other option (and probably easiest) is to use Minikube’s default storage class.
Best Regards,
Paul

4

Thanks @Aaron , @pkondrat for the information.
I will have a look to use the default storage class instead of hostPath . This minikube setting is just for testing so there is no urgency for this issue. I will try your suggestions and read the document first.

Thanks

5

Hi @pkondrat ,
How does this works when kuberenets is running in local windows? as securityContext.runAsGroup  - not possible on Windows as there is no GID support and securityContext.privileged  - Windows doesn’t support privileged containers. I am facing the same problem in windows. Below kuberenets docs I am referring Windows containers in Kubernetes | Kubernetes Create a Windows HostProcess Pod | Kubernetes Configure RunAsUserName for Windows pods and containers | Kubernetes This is how I have created the pv and pvc

image.png
image.png415×544 72.2 KB

6

Hi @prashantk2000 ,
What Kunbernetes distribution are you using on your Windows machine? I wrote a codelab a couple of years ago where I used Docker for Windows on Windows 10 PC to deploy an instance of PubSub+.
Here is the link:
Windows 10 Helm / Kubernetes Development Environment Setup (solace.dev)
Maybe it will help.
Best Regards,
Paul