Spring cloud stream | Scaling

akg17

I just wanted to understand couple of things here..

1. If i have 10 consumer configured then for each queue i see 10 consumer/subscription, that's going to be a lot no doubt because i have 50 queue and there will be like 50 *10 = 500 consumers and there are certain infra limitation within the organisation so i will reduce it to 5 consumer, just wanted to know does solace have any way to set the consumer range like in JMS we set with solace, Also how does solace utilizes the connection ? solace shares these connections in case one queue has more load than another or they remain ideal if no use.

2. When we create 2 instances of the apps again its going to be more load on one queue as consumers will get double how do we have any way to set the concurrency at binding level instead default like spring cloud stream. so i can set the concurrency higher for required bindings and low where the traffic is low.

marc

Hi @akg17,

Great questions. As you mentioned, there are essentially two ways to scale when using Spring Cloud Stream with Solace. I'll quickly define them for others that come across this thread.
1. The first way is vertical scaling, which would be using the concurrency configuration provided by the framework. This controls the number of threads used for processing events in your microservice.
2. The second way is horizontal scaling, which would be deploying more instances of your microservice.

Now to answer your questions:

If i have 10 consumer configured then for each queue i see 10 consumer/subscription, that's going to be a lot no doubt because i have 50 queue and there will be like 50 *10 = 500 consumers and there are certain infra limitation within the organisation so i will reduce it to 5 consumer, just wanted to know does solace have any way to set the consumer range like in JMS we set with solace, Also how does solace utilizes the connection ? solace shares these connections in case one queue has more load than another or they remain ideal if no use.

• There is not currently a way to set a range of consumers like you can do in Spring JMS. Note the concurrency setting is provided by the framework and I believe it just takes an Integer.
• The Solace Spring Cloud Stream binder uses the Solace SMF protocol and actually only creates one Session per Microservice and all of the consumers are essentially Flows within that session. So if you have concurrency of 5 you would be using 1 session and 5 flows. Note that the session is what makes the TCP connection so you'd only have 1 TCP connection between your app and the broker. I highly recommend checking out the "SMF Process Flow" section on this great docs page: Overview: How Apps Interact with PubSub+ Messaging Components to get a better understanding of how this all fits together.

When we create 2 instances of the apps again its going to be more load on one queue as consumers will get double how do we have any way to set the concurrency at binding level instead default like spring cloud stream. so i can set the concurrency higher for required bindings and low where the traffic is low.

Concurrency can be set on a per binding basis in your app config, for example in the config below the sink-in-0 binding has a concurrency of 2 and the marc-in-0 has a concurrency of 5.

spring:
  cloud:
    function:
      definition: sink;marc 
    stream:
      bindings:
        sink-in-0:
          destination: my/test/topic/sink/* 
          group: aws
          consumer:
            concurrency: 2
        marc-in-0:
          destination: marc/>
          group: local
          consumer:
            concurrency: 5

Hope that helps!

akg17

@marc curious to know that if we specify consumer and dont specify group name what happens does an anonymous consumer group is created?
i currently get an error when i spcify the group name
Caused by: com.solacesystems.jcsmp.AccessDeniedException: Permission Not Allowed - Queue 'TestQueue' - Topic 'TestQueue' , without group name things work but then concurrency wont work as its not allowed more than one for an anonymous group, does specify group name try to make any changes to broker or something

marc

Hi @akg17,

The concurrency configuration option is not supported in certain scenarios such as this. More info here, but essentially without a group you are using an exclusive non-durable queue that only allows for one consumer and that exception is expected.

Checkout section 3 of the Spring Cloud Stream - Beyond the Basics codelab to read about the supported communication models w/ the Solace Spring Cloud Stream binder.

Hope that helps!