Clarity needed on some core concepts
Q1) when i publish to a topic created at run time in code (using jms ) where does the message go since topic is part of message payload. The message arrives at NAB (first point of entry) , moves to TRB (topic routing blade), is it stored at some temporary storage to see if there is consumer who matches its destination?What if that consumer is not live at that moment.
Q2) i am sending hundreds of millions of events to solace , does solace throttle those events , stores them at temporary place and then pass on to broker?
Q3) so in a scenario , i have jndi of connection factory , queue etc and a topic is already subscribed to queue, but the jms api doesn't allow making message producer without destination. so i call session.createProducer(destination)(destination can be queue/topic). what happens now if i give the same names that already exists for those queue and topic.
q4) does solace use a connection pool like mongo, what if don't close the connection from client , will it auto close those connections or is it client responsibility. what is the time before it auto closes?
@marc @Aaron
Comments
-
Hi @harry, I see you've tagged this post for @aaron and @marc but I can't help myself, so here goes:
Q1) You're talking about NAB and TRB, so I assume you're asking about the appliance architecture. The general architecture of the software and hardware is the same, although obviously the hardware components don't apply in the software case. I like your question, which if I can re-phrase is "since in PubSub+ the topic of a message is entirely dynamic and set at message send, how is the message stored before it is matched to its final destination?" In the hardware, it is stored in the NAB, specifically in the Ingress Data Buffers. Note that the message is never sent as a whole to the TRB. The NAB parses the message and strips off destination information as a data structure. This metadata structure only is sent to the TRB. Part of the meta-data sent is whether the message is DIRECT, NON-PERSISTENT or PERSISTENT quality-of-service. If the message is DIRECT, the TRB checks if there are any subscribers (queues or client applications), and if there are none the message is discarded and statistics counters incremented (ingress discard, no subscription match). If the message is NON-PERSISTENT or PERSISTENT, the TRB checks whether there are any valid destinations, and if none checks settings such as queue discard settings and message-vpn subscription match discard settings to determine if a negative acknowledgement should be sent to the producer client.
Q2) You mention "Solace" and "Broker" as two different concepts. The PubSub+ Appliance is an event broker, so there is no real concept of "Solace" passing a message to "the broker" - it's all one thing. The Solace architecture is heavily tuned to avoid back pressuring producers, so that event bursts can be absorbed.
Q3) I'm not sure I understand exactly what you're asking here, but I think it relates to some confusion about how JMS concepts match on to PubSub+. If you are using JNDI lookup, the JNDI topic or queue name is looked up and the physical topic or queue name returned. It is this name that the producer then uses to send the message. If you use the physical name rather than the JNDI name, JNDI will attempt to find a match for that physical name - and probably fail, so you'll get an exception.
Q4) No, the broker does not really have the concept of a connection pool. We have scaling tiers which are somewhat analogous. So, if you like, setting the scaling tier to 100 sets the size of the "connection pool" to 100. It's best to keep connections open - a common problem if people opening a connection, sending a message, then closing the connection and waiting to open it again to send another message. Don't! Keep the connection open. Connection to the broker is managed by and is the responsibility of the client, the broker will only close the connection if it detects a network or application problem and heartbeat like activity stops.1 -
Hey @TomF thanks for clearing these. follow back questions..
Q1, so in case of burst the principle would be same right? each event is an independent entity i presume , so they all will be sent to
NAB and then to TRB (just like u mentioned , stripping , meta data , checking consumers etc).? i would really like to understand this "heavily tuned" mechanism if possible.
Q2) so the reason i wanted to understand connection pools is i am sending messages in distributed architecture(SPARK). so i have partitions (logical piece of code ) may or may not sharing Jvm's depending on which node they run. So if i don't have to explicitly close connections i can increase my parallelism by making a single connection(using singleton) per jvm and inccrease number of partitions in one JVM . how ,much is this heartbeat time after which broker closes the connection. and by reading your explanation i think it's best to keep the connection open.0 -
@harry:
Q1. Yes. In the case of message burst nothing changes in the architecture - we don't have special burst handling buffers or anything like that, because the architecture was designed to deal with just such a scenario. Each event is separately handled - we don't resort to batch handling or anything like that. In fact, we can't - each event can have a completely different topic so they have to be treated individually. A further detail is that the architecture is designed to support fan-out: a single message in may be destined for many subscribers. That means the key resource is actually the egress buffers rather than the ingress buffers. What almost always happens is that message consumers end up being the bottleneck, and we start intelligently dropping messages or connections if consumers can't keep up and we start running out of buffer space.
Q2. In general I'd say you're right - if you have a long running stateful process, such as a JVM, have that open the connection and the short lived jobs running in that JVM use the same connection. There are some performance considerations, but let's cross that bridge when we come to it.2
