Queue with random order?

klauszi
klauszi Member Posts: 3 ✭
edited March 3 in PubSub+ Event Broker #1
We aim to have a queue without order. The motivation behind this is that younger events update older events, and the updating process is relatively 'expensive'. However, if younger events arrive first, older events are simply discarded.

How can we enable this?
Tagged:

Answers

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 636 admin
    #2

    Hi @klauszi, welcome to the Community!

    That's a wacky idea! haha that goes against what almost all "queueing" systems try to prevent.

    Do you care if you lose messages?

  • klauszi
    klauszi Member Posts: 3 ✭
    edited March 4 #3

    Yes haha, it really contradicts the principle of queues. But it would be cool if there was such a possibility. However, messages must not be lost :( We do need that property of queues already...

    We've also experimented with prioritization. If the sender assigns a random priority from 0 to 9, then younger events could pass older events, but then it would be semi-randomized. Additionally, the sender would have to take care of that, which also bloats the code…

  • allmhhuran
    allmhhuran Member Posts: 47 ✭✭✭
    edited March 13 #4

    In principle you can't ever get complete knowledge about whether a younger event is "about to arrive" at the message destination, because "about to arrive" encompasses multiple possibilities:

    1. The destination has buffered multiple events associated with the same entity
    2. You are using guaranteed messages and the broker infrastructure has multiple events in a subscriber's queue associated with the same entity
    3. The publisher is currently in the process of publishing a new message for an entity, while the broker queue, or the subscriber, already has on older event for that entity.

    The only one you can control is (1), so controlling this should happen on the subscriber side.

    In my opinion the best solution is to have the subscriber receive messages into a buffer. You then release messages in batches. A batch would be released when, say, 500 messages are buffered, or some amount of time elapses, whichever comes first (tweak these numbers for your use case).

    You then deduplicate the batch in the subscriber processing code, keeping only the most recent message for each entity (and immediately acknowledging all of the older messages without processing them if you are doing guaranteed delivery). A subset of the batch, containing only the most recent messages, then proceeds through to normal message processing.

Categories

This Month's Leaders

This Week's Leaders