Transactional applications in event driven approach

shanmuganand

I understand the benefits of event driven approach which is like decoupling the applications and making the integration seamless with more reliability but there is a bit of delay in as well with event driven more on the side for the call backs and so in case of an ecommerce application which has payment transaction involved can we consider event driven in the critical steps on Payment authorisation

himanshu

Hi @shanmuganand,

You are on the right track. Asynchronous messaging has its advantages like you mentioned but they don't fulfill requirements for all usecases. In most usecases, your applications can have deferred execution and tolerate eventual consistency so that as soon as an event is fired, downstream applications can process and store that event at their own pace and be consistent eventually.

However, like you mentioned, there are some usecases such as payment authorization where you need instant validation (synchronous reply) that the payment was processed. It is not acceptable for your system to eventually process the authorization at some later time. It has to be processed synchronously before any downstream processes such as auditing, logging, or analytics can be kicked off. In such usecases, you have to use synchronous messaging.

Practically, your system will be a hybrid of RESTful synchronous communication and asynchronous communication. With Solace PubSub+ event broker, you can use REST for request/reply messaging and use pub/sub messaging for asynchronous communication. Here is an example of what that hybrid architecture would look like:

And here is a blog post I wrote about how that works.

Additionally, Sumeet Puri from Solace, presented (video) this exact usecase, at API Days Conference in Singapore, of payment authorization and using both sync and async messaging via Solace. I highly recommend checking it out.

I would be curious to hear what you are working on and how we can collaborate further. Payments is a very trendy usecase these days with a lot of regulators around the world working on implementing real-time gross settlements (RTGS) systems.

Hope this helped,
Himanshu

Aaron

Hi @shanmuganand ... I'm just going through old threads while I was on vacation and wanted to respond to this.

First off... you don't have to choose REST-over-HTTP if you require synchronous request-reply behaviour... there are a TON of Solace / messaging distributed applications out there that use Solace for request-reply. Our APIs even have a Requestor helper object that blocks waiting for a response, hiding all the threading and Correlation ID matching. Examples for Java:

https://docs.solace.com/API-Developer-Online-Ref-Documentation/java/com/solacesystems/jcsmp/Requestor.html
https://github.com/SolaceSamples/solace-samples-java-jcsmp/blob/master/src/main/java/com/solace/samples/jcsmp/patterns/DirectRequestorBlocking.java

Of course, many payment use cases have multiple steps/hops between the payment/client gateway and the backend core banking system. From the external customer view, it may be request-reply, but once it lands on your edge system, it could kick off a number of either synchronous or async interactions (like Himanshu said above). Depending on how you manage state within your system, it can certainly be more efficient to to rethink processes as event-driven rather than synchronous/blocking processes where you need a THREAD for every single request (for each hop) while it is blocking and waiting for its response.

One other big advantage to using Solace (or other) brokers for request-reply is that you can scale up/down by simply adding more consumers... you don't need to have a load-balancer sitting in front of a bank of web servers. And as well, you can use the fact that requests are published as topics, and so you could build "listener" applications (e.g. audit, logging, fraud, etc.) subscribed to the same request topics, but not actively responding to them... the broker will give those consumers a copy "for free", as sending one message to multiple consumers is fairly cheap.

And you raised this concern:

there is a bit of delay in as well with event driven more on the side for the call backs

I'm guessing you mean that because the message has to go Publisher -> Broker -> Consumer, it's not as fast as REST send -> REST receive, which goes direct in your network. (except via a load balancer). Correct? Well, Solace has its roots and DNA in investment banking trading platforms, and we've optimized the data path for low latency. In fact, the bandwidth/overhead of a REST header is pretty significant vs. always-on Solace connection.

What is your latency budget for the overall request-reply flow? What kind of latency are you looking for, for each hop? < 10ms? < 1ms? Our software broker is generally around < 1ms from API send to API receive. The hardware broker is < 100µs (microseconds!). Generally, latency isn't the problem, at least for Solace. Streaming platforms like Kafka experience higher latency due to their disk/log-based systems and poll-based consumer APIs.

Anyhow, sorry for the long post! Hope that helps.

shanmuganand

Thanks to @himanshu and @aaron for such detail explanations, definitely helps.

Just to detail out on the delay aspect I had mentioned in the previous question,

My understanding: An event is just a notification that is broadcasted to different system to consume and act, example "cart created" , "order updated" - with some basic identification indicator(cart Id or order Id) of the cart and order respectively. Now I know different systems can implement the same event in different approaches/architecture. But for an event to be lightweight/minimal and to be broadcasted to all the different consumers, the approach was to keep it simple.

Now if the consumer is interested in knowing more about the event then further calls are initiated by the consumer to the produced who published the event with the identifier to get the details of it.

So my delay aspect here is the back and forth calls by the consumer after getting the event notification which can be a concern for any time boxed transaction isn't it ?

do you suggest me any other approach otherwise ?

Aaron

Ah! This is a good question. Essentially: notification msg vs. full-state transfer. Both can be correct in some instances, both have drawbacks.

E.g. super bandwidth constrained situations, or whenever the majority of recipients don't want all the details, or perhaps some security issues where different consumers (when they make their follow-up request) might get different payloads.

Also, sometimes, you might have a "chain" of processes that might do things like decorating the messages with additional details, and will publish the outbound message to a new topic (e.g. some kind of credit lookup process that adds additional account into to an original message)... reading from topic bank/payments/req/abc123 and publishing out to bank/payments/valid/abc123. This is common in event-driven microservice architectures... each publishing to the next topic in the chain. And because Solace topics are dynamic and hierarchical, you can include other information that might be useful for routing and filtering, such as country code, store ID, type of request, etc.

I personally think that including the full state (or most of it) with each message is more desirable, so that apps don't have to do that "go back and get more info" request. But if 95% of your apps are happy with just the notification event, then it makes sense to save bandwidth.

Here's a talk from the recent EDA Summit that I recalled discussed these approaches: https://youtu.be/GzL-Zo0AZ0Y?t=1045

Hope my rambling thoughts help..?

himanshu

I agree with @Aaron. If there are no bandwidth issues, it is worth adding the additional details either in the topic hierarchy or the payload itself so the downstream application(s) doesn't have to do additional lookups.

I have read few articles and watched some sessions where they focus on events being just notifications to keep them simple and lightweight but that might also be due to a lot of other brokers not supporting larger payload sizes. Solace PubSub+ broker has very high payload size limits so that's not really an issue.

shanmuganand

Thanks @Aaron and @himanshu , the EDA summit video explains the various approaches of event driven which is what I was looking for, I will get back to you more on this after validating the options.