🎄 Happy Holidays! 🥳
Most of Solace is closed December 24–January 1 so our employees can spend time with their families. We will re-open Thursday, January 2, 2024. Please expect slower response times during this period and open a support ticket for anything needing immediate assistance.
Happy Holidays!
Please note: most of Solace is closed December 25–January 2, and will re-open Tuesday, January 3, 2023.
Ways to handle a stopped queue consumer?
In a scenario where a consumer application reads data from a queue and inserts into some database, you could run into an issue where the database becomes available and so the queue starts filling up. If the outage lasts even a few hours and your data rate is high enough you could run into a spool problem pretty quickly.
Has anyone developed any creative solutions for such as scenario to prevent data loss, assuming you don't want to impact publishers? Maybe spinning up an on-demand consumer that dumps the data somewhere with more storage (e.g. SQS)? It'd be nice to have a strategy like this in place in case of an emergency.
Comments
-
Hi @petegehrman , Where do I store when database Iam trying to store messages is unresponsive ? Thats a tough question :-)
General patterns we see among Solace customers are:
0. Fix the downstream system :-)
1. Monitor the queue depth and increase the quota. In this time and age where broker supports TB of spool, this is an increasingly popular option.
2. Define Dead Message Queue so the pending messages are unspooled from the application queue into DMQ. This is esp useful with non-exclusive queues where a single hung consumer can impact other consumers.
3. DMQs can now be processed by other special consumers - like you suggested can write to other storage devices / files. In most cases though, we see them log these messages.We have seen customers use databases to store long living messages. However this is becoming an anti-pattern these days. In a realtime event driven use-cases, events/messages loose their value as time goes by. Also using different queuing system (eg: SQS) to store overflow messages from one queuing system (eg: Solace) doesn't seem very attractive to me.
0 -
Thanks for the input. I agree that moving messages to another queue technology is not ideal - it's basically just an external DMQ and adds complexity. Increasing the quota probably makes the most sense to me in this situation, since it should be short-term. However, we're using a Solace Cloud service installed into our own EKS cluster that we manage, and the max configurable spool size is 500GB. Since we manage the underlying hardware/storage in this setup, is there a way to increase the Message VPN's max spool? It's disabled in the UI.
0 -
The spool size dependents on the tier selected at service creation time. Pl see this link.
https://docs.solace.com/Solace-Cloud/Configure-Message-Spools.htm0
