Data integrity verification

chaudharys
chaudharys Member Posts: 25 ✭
in General Discussions #1

Hi,
How does data integrity verification happen in the Solace C API? I could not find anything in the docs for it. Is there any good checksum API that we can try?

Best Answer

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 636 admin
    #2 Answer ✓

    Hi @chaudharys . Interesting question/use-case. Note that Solace uses TCP, so at the network level, there's all sorts of checksums and error detection already present. So if you're concerned about the network corrupting the data, you shouldn't worry about that.

    But yeah, if you wanted to publish a series of messages, and then publish a "end flag" or "checksum message" or similar message at the end, that should work just fine, as Solace guarantees order across topics (unless you're doing round-robin consuming like with Shared Subscriptions or Non-Exclusive Queue). I once wrote a statistics publishing app, that would get a big dump/page of data, and it would publish a "START_PAGE" message, then each individual row was a message, then an "END_PAGE" message. This would allow consumers to know when to expect a blob, and could react to that if required. However, some apps didn't care about start/stop, so they didn't subscribe to those topics, just the data ones.

    Hope that helps..?

Answers

  • Tamimi
    Tamimi Member, Administrator, Employee Posts: 538 admin
    #3

    Hey @chaudharys, if I get your question correctly, you cannot do any data/schema validation of your payload with the Solace API. The broker basically takes whatever data format from the input and passes it to the output. For any Data integrity verification of the payload of your events, you will have to implement that in a "data integrity microservice" that subscribes to the topic and republishes. What is your use-case?

  • chaudharys
    chaudharys Member Posts: 25 ✭
    #4

    Hi @Tamimi
    We have one publisher publishing about 50-100 messages, each message containing an array of serialized protobufs. We want to try single and multiple topics, and single and multiple subscribers, receiving messages. I was looking for a way in which we can pass a check sum value as the last message, calculate the check sum at subscribers and compare it with the value they receive.

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 636 admin
    #5 Answer ✓

    Hi @chaudharys . Interesting question/use-case. Note that Solace uses TCP, so at the network level, there's all sorts of checksums and error detection already present. So if you're concerned about the network corrupting the data, you shouldn't worry about that.

    But yeah, if you wanted to publish a series of messages, and then publish a "end flag" or "checksum message" or similar message at the end, that should work just fine, as Solace guarantees order across topics (unless you're doing round-robin consuming like with Shared Subscriptions or Non-Exclusive Queue). I once wrote a statistics publishing app, that would get a big dump/page of data, and it would publish a "START_PAGE" message, then each individual row was a message, then an "END_PAGE" message. This would allow consumers to know when to expect a blob, and could react to that if required. However, some apps didn't care about start/stop, so they didn't subscribe to those topics, just the data ones.

    Hope that helps..?

Categories

This Month's Leaders

This Week's Leaders