How can Message Eliding applied on Market data use case?

Newbie Wong
Newbie Wong Member Posts: 2 ✭
in Connectors & Integrations #1

we would like to migrate our application to use solace for the market data updates
I have read the document and find it very interesting as it provides Message Eliding.
Also, it states that it is suitable for streaming market data to clients that do not need every update.

let's say we have 10000 stocks need to publish if there are updates and the 3 clients only want the message rate of 5 updates/seconds,
does it mean I need to create 1 topic per stock which ends up of 10000 topics in total?
It seems to be quite a lot.

I am not sure if I understand this correctly or there is any other best method to deal with that.
I am very appreciated if someone can shed some light on it

Thanks

Comments

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 634 admin
    #2

    Hey Newbie Wong... thanks for the question. So in Solace, messages that are to be elided are tracked on a topic-by-topic basis by the broker. The publisher would be publishing updates for these 10,000 stocks as messages on 10,000 different topics... that part you got right, but there's nothing to "create"... the publisher just sets the topic of the message every time it publishes a new message, based on the stock symbol for that message. E.g. stocks/update/v1/amx

    Now, on the consumer side, it could receive data for all 10,000 stocks by simply using a single wildcard subscription: stocks/update/v1/* (for example). You don't need to subscribe to all 10,000 individual topics separately. (although some applications choose to).

    Now for the eliding part: this is specified as either a) the rate at which you wish to receive messages per-topic; or b) the minimum inter-message delay. E.g. 500ms delay, means I can get at most 2 msg/s. Now, this is PER-TOPIC, so if you have all 10,000 topics being published on a lot, then your consumer could expect to get around 20,000 msg/s (max 2 updates per second per topic). If you set the delay to 2000ms, then you'd expect a max message rate of 5,000 msg/s. Does that make sense?

    Different apps can have different eliding rates... you just need to create different client-profiles, and different client-usernames that point to those different profiles.

    Finally, note that the publisher has to turn on a flag per-message to say that it is eligible to be elided by the broker.

    Hopefully that helps?

  • Newbie Wong
    Newbie Wong Member Posts: 2 ✭
    #3

    Hi Aron, thanks for the reply. have a bit of understanding now.
    So you said sets the topic of the message, e.g.
    message.Destination = ContextFactory.Instance.createTopic(xxxx) ; // xxxx = stocks/update/v1/amx or stocks/update/v1/abc
    In the case above, I thought we need to create stocks/update/v1/amx and stocks/update/v1/abc in solace appliance first?
    otherwise, I think we may use ISession.createTemporaryTopic() , Do I need any special right in my account to do it?
    so if we set 10000 topics, it does not sound a lot?
    Thanks

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 634 admin
    #4

    Hi again @Newbie Wong ! No, you don't have to setup or create all these topics on the Solace appliance. You just publish on them. For comparison: think of an IP router... you do not need to configure all the destination IP addresses that you want to send to... your network card just sends its packets with the address, and the router does comparison and mapping and forwarding when it receives the packets. Same concept.

    In Solace, your publisher can publish on a billion different topics, without any configuration done on the broker. Only the subscribers need to specify their subscription, which can use wildcards.

    Check this video for more description: https://youtu.be/PP1nNlgERQI

Categories

This Month's Leaders

This Week's Leaders