Spooled Messages gets deleted after the broker container restarts in kubernetes
Hi,
We are deploying the pubsubplus broker in kubernetes cluster with helm chart. We did it using the reference doc - https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/pubsubplus/README.md
We did set up volume in aws efs, below is the pv and pvc.
And we are providing custom volume mount in values, please see below for values file detail.
Brokers configs gets backed up on volume and restores with new container however we are loosing the messages only after restart of container. we already tried both legacy and new storage group behavior using the flag useStorageGroup: true/false however in both ways we are loosing spooled messages.
Can someone please guide here? May be I am missing any configuration.
Comments
-
Hello Prashant -
The Solace PS+ brokers should be configured with dedicated block storage for their message spools, and Kubernetes needs to configure this storage as persistent. The minimum recommended spec for the storage is 20MB/s to support guaranteed messaging and the HA feature. In an HA configuration, each node has it's own dedicated storage.
Some questions for you:
1 Are you losing all messages with queue references or only some?
2 Were the messages on the spool published marked as persistent?
3 Are you configured for HA deployment of 3 nodes, or a single, non-HA broker?
I noticed two settings in your config that look odd to me...
customVolumeMount: |
This one is followed by a pipe character and some indented lines for setting a persistent volume claim. Note the comment above indicating that using this value will cause other values to be ignored. Let's try without the custom volume mount and use the standard config values instead.
storage.slow=true
This should be set to false, as you want the best performing block storage possible above the minimum recommended.
The other values you shared look reasonable and correct.
Karl
0 -
Hi @Karl Ossoinig ,
Thank you for your time. Please see my comments below.
1 Are you losing all messages with queue references or only some? - [Loosing all messages]
2 Were the messages on the spool published marked as persistent? [ Yes those are persistent messages ]
3 Are you configured for HA deployment of 3 nodes, or a single, non-HA broker? - [single, non-HA broker]
customVolumeMount: | - pipe character is required as mentioned in reference values and it is for multiline string and if we remove it, it throws error.
For storage we are using AWS EFS, which is similar to NFS so if we keep storage.slow= false It won't work and throws error at the time of broker starts -
Unable to fallocate file /usr/sw/internalSpool/softAdb/.diskTest to size 4096: Errno(95) Operation not supported
So, I believe it should be storage.slow=true. Does it will impact message spooling only? as I see there is no issue with broker config, and it gets loaded with new container.
Please suggest here.
0 -
Hello @prashantk2000 - Thank you for the additional information.
Based on the behavior and your elected storage type, it is likely the problem is due to delays when write command from the broker are persisted. In general, PS+ requires block storage because it needs to do fast direct writes frequently to keep state correct.
There are specific storage recommendations for various deployment environments. You can find the Solace recommendation for AWS storage options HERE.
To read more about how PS+ broker works with storage and how to configure, check out these links:
https://docs.solace.com/Software-Broker/Configuring-Storage.htm
1 -
Thanks @Karl Ossoinig ,
I tried couple of things, we provisioned our AWS -EFS storage to minimum 20MBps write capability, however it didn't work as storage.slow flag has to be true in case of NFS, which I believe causing not to restore the messages after new container comes online.
We are thinking of 2 approaches now,
a. The one you suggested, block storage in kubernetes node.
b. Using S3 bucket (as most of the application in our cluster uses either AWS-EFS or S3 bucket), however I didn't find any solution that uses helm chart for deployment and S3 bucket as storage device, If you think this is feasible with PS+ broker, please share some idea here?
Thanks
0 -
Hello @prashantk2000
Unfortunately, AWS S3 is an object storage mechanism, and does not allow direct access to volume blocks for write operations. This is not a recommended configuration, and is unlikely to work for you.
Here is a thorough writeup on storage from another vendor independent from Solace that explains the differences between object and block storage: https://bluexp.netapp.com/blog/block-storage-vs-object-storage-cloud
Again, the reason Solace provides minimum required guidance is that the design of PubSub+ broker requires block storage with 20Mb+ writes in order to function. But I am curious to know if S3 works for you...
0 -
Thank you @Karl Ossoinig for your guidance, we did setup EBS for our standalone PS+ and everything looks good. :)
0 -
That's great news @prashantk2000 - Thank you for sharing your results!
0
