Topic which gives whole state followed by updates

radekm

Hi,
I'm trying to create an app where each client initially gets whole state and then listens to state updates.
I would like to use Solace topic to distribute state updates to clients. But I also need to distribute whole state to each client immediately after it subscribes - how is this done in Solace?

Is there some proven solution to this problem?

Thanks

TomF

Hi @radekm , yes, we have a proven solution for just such a use case: PubSub+ Cache to store the state in the cache.

Typically you'd use an ingress plug-in (Cache has a plugin architecture, you write the plugin yourself to accommodate your data model) to read messages on the topic, read the current cached value and combine the two to create the new latest state. On subscription, the cache value (latest state) is read and then subsequent values flow over the subscription. The cache API takes care of ordering so you can be sure you're getting the latest cached state value when the cache request was made, followed by all the updates that occurred after the cached value was generated.

An alternative is the "event sourcing" approach, where you'd have a queue subscribed to your topic. On start up, the application reads from the queue, processing the old updates to generate the current state. The application can optionally choose to emit state snapshots to reduce the amount of startup processing required. Got a new application and so don't have a queue full of updates? No problem, simply create a new queue, invoke replay and your state is there waiting for you to construct.

marc

To follow on to @TomF's answer another option might be to use a Last Value Queue (LVQ) to store the snapshot of the latest state you want a client to have at start-up. Then when an app starts up it can browse the message from the queue (not consume! this way it stays on the queue for other clients) and then the app would subscribe to a different topic to receive state updates.

swenhelge

@TomF

Typically you'd use an ingress plug-in (Cache has a plugin architecture, you write the plugin yourself to accommodate your data model) to read messages on the topic, read the current cached value and combine the two to create the new latest state.

The ingress plugin seems to be relevant to very specific use cases?
The configurable behaviour of the cache is to store all messages received on a topic up to a configurable maximum capacity and for a specific time to live.
It seems to me that basic use cases like providing a last value cache or the events received within the last 60s to bootstrapping client applications can be achieved via configuration only?

TomF

@marc said:
To follow on to @TomF's answer another option might be to use a Last Value Queue (LVQ) to store the snapshot of the latest state you want a client to have at start-up. Then when an app starts up it can browse the message from the queue (not consume! this way it stays on the queue for other clients) and then the app would subscribe to a different topic to receive state updates.

This approach assumes the last message holds the complete state - if the last message is a state update, you'd need to have a service to roll up that update into the latest state and publish it on a different topic to the LVQ. This is a perfectly valid approach and I've seen it used. However it's also the approach used by the cache plugin, without a separate topic and LVQ.

swenhelge

OK, I overlooked "whole state" in the question.
I see two options and I think in these cases one will often have to use the ingress plugin approach as @TomF described above.
Assuming we are dealing with an event sequence that consists of periodic full updates and more frequent delta updates. For example an industrial sensor may emit 200 updates a second and transmitting the full sensor data set in each event wouldn't be feasible, so the events emitted will be one full update followed by 199 increments per second.

First option - the ingress plugin - provides the full state to the client. It's centralised logic - i.e. the client doesn't necessarily need to be aware/capable of tracking full and incremental updates.

Second option - if the full state is transmitted **periodically and reliably ** then caching for a time period greater than the full update periodicity should give a client all the data required to calculate the current full state. If the client implements the logic anyway to process the live event stream then caching events for a suitable period of time may be sufficient.