Microservices - approaches to multiple container instance -> single event consumption

CloudGod
CloudGod Member Posts: 24 ✭✭
in General Discussions #1

Hi all,
I would like to ask your opinion on a microservice use-case, and what would be the best approach to tackle it.
The Context
Let's say you have 10 microservices, each one implementing a given bounded-context, as per DDD.
Each microservice is constantly publishing its own events so that the others who subscribe to this consume them. So far, so good... pretty standard stuff.

Let's also assume that we have some microservices that generate and consume a lot of events and are also strongly involved in synchronous API calls (REST-based).

Because the load on these microservices is not uniform throughout the day, they start running on 1 container instance at the beginning of the day, scales to 4 or 5 container instances by midday, and scale back to 1 and around 7 PM.

The underlying data layer for this microservice is a single DB instance serving all container instances.

The Challenge
Each instance of these microservices that is spawned subscribes to a set of topics that the other microservices will publish.

However, we want that any given event that microservice A subscribes is only processed one time by one of its pool of container instances. So, if microservice A is running 4 container instances and Event X is subscribed to, each Event X is to be processed by just one of the instances, not all 4.

Possible solutions that we believe may address this:

  • Use Direct-To-One (DTO)?
  • Use Topic-To-Queue mapping?

What other possibilities are out there?

Cheers
Jose

Tagged:

Comments

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #2

    Hi @CloudGod,
    Great question! Solace has two different options for this; essentially implementing a "Consumer Group" pattern which allows events to be spread across 1 to many instances of the same microservice. The choice is made based on the quality of service that you need. If you are using Guaranteed/Persistent messaging (you can't lose a message) then you'd have your consumers bind to a Non-Exclusive Durable queue which in turn has topic subscriptions for the events of interested. If you are using Direct Messaging then you would use our shared subscriptions capabilities.

    This blog from @Aaron covers the topic in more detail: https://solace.com/blog/consumer-groups-consumer-scaling-solace/

    Hope that helps!

  • CloudGod
    CloudGod Member Posts: 24 ✭✭
    #3

    Thanks @marc
    It does seem to solve the use case at hand.
    So, now I'm wondering in which cases would we actually use DTO?
    Cheers
    José

  • marc
    marc Member, Administrator, Moderator, Employee Posts: 959 admin
    #4

    Easy answer, Don't use it :)
    It was a legacy feature and has been supplanted by shared subscriptions.

    We added this as a recommendation to the docs as well. If you found somewhere else that was still pushing the use of DTO please let me know and we'll get it updated 👍

Categories

This Month's Leaders

This Week's Leaders