Solace SEMP error: "Unable to allocate a semp session"

Aaron

Have you ever seen this HTTP error? This occurs when too many processes are trying to access the Solace broker using SEMP. Or, perhaps a small number of processes/applications are polling the Solace broker too quickly. This often happens in larger distributed environments, where multiple monitoring frameworks are in place. This error indicates that the internal SEMP engine used for responding to remote commands is overdriven. Too many SEMP requests! Slow down!

Usually this occurs due to monitoring. I.e. show commands. Not always, but almost always. Show commands (vs. config commands) can generate a lot of response data, which puts load on the SEMP engine.

One way to avoid this is to centralize your SEMP polling. If you have multiple processes asking for the exact same data, that's dumb. Poll SEMP once, and use pub/sub or similar to send the monitoring data to multiple downstream applications. This is what the StatsPump does, if you're familiar with that product.

Otherwise, to help you figure out who/what/where is causing the Solace SEMP engine to be overdriven, here is my TIP OF THE WEEK™. You can configure some additional logging to see all of the commands. Solace by default reports all configuration commands to the command.log, stored in directory /usr/sw/jail/logs. But the following will show you how to enable all logging (including the "shows") to goto the command log. First, login to CLI, then:

enable
config
logging command all mode all-cmds
show logging command
show log command

Rather than looking at the command log in CLI, it's easier to simply tail -f the command log from the shell, and look for usernames and IP addresses that are doing all the polling. I often use grep to slice out one minute's worth of data, and then count the number of lines (e.g. grep "2021-01-20T15:01" command.log | wc -l). You might be surprised to see 10, 20, 30 requests per second sometimes!! No wonder SEMP is complaining.