Ways to handle a stopped queue consumer?

petegehrman
petegehrman Member Posts: 43 ✭✭
in General Discussions #1

In a scenario where a consumer application reads data from a queue and inserts into some database, you could run into an issue where the database becomes available and so the queue starts filling up. If the outage lasts even a few hours and your data rate is high enough you could run into a spool problem pretty quickly.

Has anyone developed any creative solutions for such as scenario to prevent data loss, assuming you don't want to impact publishers? Maybe spinning up an on-demand consumer that dumps the data somewhere with more storage (e.g. SQS)? It'd be nice to have a strategy like this in place in case of an emergency.

Comments

  • nram
    nram Member, Employee Posts: 80 Solace Employee
    edited April 2021 #2

    Hi @petegehrman , Where do I store when database Iam trying to store messages is unresponsive ? Thats a tough question :-)

    General patterns we see among Solace customers are:
    0. Fix the downstream system :-)
    1. Monitor the queue depth and increase the quota. In this time and age where broker supports TB of spool, this is an increasingly popular option.
    2. Define Dead Message Queue so the pending messages are unspooled from the application queue into DMQ. This is esp useful with non-exclusive queues where a single hung consumer can impact other consumers.
    3. DMQs can now be processed by other special consumers - like you suggested can write to other storage devices / files. In most cases though, we see them log these messages.

    We have seen customers use databases to store long living messages. However this is becoming an anti-pattern these days. In a realtime event driven use-cases, events/messages loose their value as time goes by. Also using different queuing system (eg: SQS) to store overflow messages from one queuing system (eg: Solace) doesn't seem very attractive to me.

  • petegehrman
    petegehrman Member Posts: 43 ✭✭
    #3

    Thanks for the input. I agree that moving messages to another queue technology is not ideal - it's basically just an external DMQ and adds complexity. Increasing the quota probably makes the most sense to me in this situation, since it should be short-term. However, we're using a Solace Cloud service installed into our own EKS cluster that we manage, and the max configurable spool size is 500GB. Since we manage the underlying hardware/storage in this setup, is there a way to increase the Message VPN's max spool? It's disabled in the UI.

  • nram
    nram Member, Employee Posts: 80 Solace Employee
    #4

    The spool size dependents on the tier selected at service creation time. Pl see this link.
    https://docs.solace.com/Solace-Cloud/Configure-Message-Spools.htm

Categories

This Month's Leaders

This Week's Leaders