Clarification on Multiprocessing Support in Solace PubSub+ Python API
Hi,
I noticed on the Solace PubSub+ Python API page on PyPI that it mentions the messaging API does not support multiprocessing. Is this information outdated or still valid?
https://pypi.org/project/solace-pubsubplus/
Additionally, if we connect to a non-exclusive queue, having multiple clients using the Solace Python library seems like a very natural setup. Can you provide some clarity on this?
Thank you!
Answers
-
There are no issues with multiple separate clients using the Solace Python Library whether on the same or separate platforms. So if you start multiple separate clients each with their own messaging service you are fine.
What follows is some history and some rationale for the docs.
There is an ancient concept, that predates multi-threading, of parallelism/concurrency through multi-processing. In this model the application forks (clones itself) into two processes that share only file descriptors, such as sockets for read/write. It introduces huge complexity into the application where it is no longer possible to share state information as each process has it's own memory space. Shared memory was invented to deal with this.
But multi-threaded puts this all back in the OS and is what everyone is familiar with today.
None of the Solace APIS support this type of multiprocessing where the session is shared between separate processes. It never even occurs to use it or warn against it because of multi-threading support.
However Python, with it's Global Interpreter Lock, is very bad at multi-threading and consequently some developers have reverted to the ancient practice of multiprocessing, there is even a multiprocessing package that can be imported, and articles written about it such as
Speed Up Your Python Program With Concurrency – Real Python
This will not work, or will not work with the Solace Python API, you cannot access the same instance of the API from multiple different processes spawned by the multiprocessing package.
1
