Integration patterns: your thoughts on this use case
Hi all,
Any thoughts on this use case? Can we somehow implement this using Solace without a stateful database component?🙂
Use case
- An application sends sales orders to a Boomi API. The destination is an ERP system.
- The application can cancel the sales order within 15 minutes. Don't ask me why, but this is the requirement. The application sends the same sales order again with the status cancelled.
Thoughts
- Consequence: the sales order cannot be processed immediately by Boomi and sent to the ERP system.
- Solace features I'm aware of: TTL (often used), delayed delivery (not used yet)
- Any 'm thinking of temporarily storing the message in the queue and somehow delaying the consumption of messages for a while so they can be processed together in the same batch in Boomi.
Requirements
- The solution must be stateless.
- Temporary queuing messages using Solace is OK. Storing the sales order in a database will work, but is not stateless and is therefore not the preferred solution.
- Guaranteed delivery is a must-have. Therefore, the Boomi part must be implemented with a Solace LISTEN operation instead of a GET operation.
- LISTEN: the event is consumed after a successful Boomi process execution
- GET: the event is consumed immediately by the Boomi process but can be lost when the process crashes.
Best Answer
-
@ChristianHoltfurth We have implemented V1 in a slightly different way. Thanks for the input so far; that was very useful.
- Queue A: contains sales order created messages using TTL of 15 minutes before they are moved to the dead letter queue (see queue C)
- Queue B: contains sales order cancelled messages that are processed by the listener process
- Queue C: contains all "approved" sales orders that are processed by the scheduled process
Boomi processes
- Scheduled process: consumes sales order lines using a TTL of 15 minutes. Delayed delivery does not work if using process 2
- Listener process: this process listens to queue B and consumes the related sales order line from queue A using a selector
I'm on holiday for the next three weeks. I keep you posted if we decide to make any changes.
1
Answers
-
Hi @sjaak ,
Interesting design problem. I had a discussion with my colleague @TomF and here is what we would recommend you to do to implement this with Boomi and Solace:
- Define separate topics for your orders and order cancellations. It can be something like
order/created/<id>/<meta-data-field1>/<meta-data-field2> and order/cancelled/<id>/<meta-data-field1>/<meta-data-field2>
- Create a queue for your orders that has a delayed delivery of 15 mins configured and a matching subscription for your orders.
- Create a queue for your order cancellations (without a delay) with a matching subscription for your order cancellations.
- Publish your orders to the order created topic.
- Publish your order cancellations to the order cancellation topic + your original order id in some user properties
In Boomi:
- Have a Listen adapter read messages of the order queue and extract the order ID. (remember, all these messages will come with a delay of 15 minutes).
- Use a GET adapter and set the Selector property to match only messages with your received order id to check whether there are any cancellations for this order on the cancellation queue.
- If a cancellation arrived in the meantime that matches your order, both messages can cancel each other out (ack both and don't produce an order to the downstream system).
- If no cancellation was found, order can be processed/forwarded to downstream system (ack incoming order to remove it from the queue once the message send has completed).
Some additional features you might want to consider:
- To avoid late cancellations potentially building up on the queue (cancellations that arrive after 15 minutes and once the order was forwarded/processed), add a TTL/expiry to the cancellation queue to discard any cancellations that were not retrieved by your Boomi flow as part of the processing.
- Not sure if Boomi supports this, but you could also make this flow transactional and "commit" the consumption of the order and potentially cancellation message + the sending/forwarding of the resulting order to the downstream ERP system in one step (if all within Solace).
Additional documentation:
https://help.boomi.com/bundle/connectors/page/int-Solace_PubSub_operation.html
https://docs.solace.com/Messaging/Guaranteed-Msg/Delayed-Delivery.htm
3 -
Interesting solution..! Nice one, @ChristianHoltfurth. Keep the state in the broker. Does it work?
Too bad the ERP system doesn't have some kind of order state machine, where it would accept a new order (the "created"), and set its status to pending or something. Then a 15 minute trigger on that pending order to become submitted if the cancel hasn't arrived in time.
Also (re: topics) doesn't "cancelled" have two "L"s? 😉
1 -
Thanks @Aaron , fixed the typo.
Curious to hear back from @sjaak if he accepts the answer.
I must admit that I was at first scratching my head about keeping the Boomi flow stateless and having the state managed in the broker without a cache or external database.
But this seems like a solution that should fit the requirements.
0 -
I'm reading it now @ChristianHoltfurth 😀
1 -
@ChristianHoltfurth We have implemented V1 in a slightly different way. Thanks for the input so far; that was very useful.
- Queue A: contains sales order created messages using TTL of 15 minutes before they are moved to the dead letter queue (see queue C)
- Queue B: contains sales order cancelled messages that are processed by the listener process
- Queue C: contains all "approved" sales orders that are processed by the scheduled process
Boomi processes
- Scheduled process: consumes sales order lines using a TTL of 15 minutes. Delayed delivery does not work if using process 2
- Listener process: this process listens to queue B and consumes the related sales order line from queue A using a selector
I'm on holiday for the next three weeks. I keep you posted if we decide to make any changes.
1
