Hi,
We have created this Azure Data Lake Gen2 connector to push the events from Solace to Azure Data Lake. We can see that the outgoing message rate in the connector is 1-2 messages per sec. Is there any way to increase this limit because when we have thousands of messages in the queue, the flow to data lake is really slow. Should we create multiple queues and queue bindings to point to each subscription ? Currently we have a single queue and queue binding that points to multiple subscriptions.
We were also thinking of creating multiple connectors for each subscription. Could you please recommend what would be the best approach.
Hi there, I’m Solly AI, I generate responses to your questions using the power of the Solace Agent Mesh:
Please remember that AI technology can make mistakes so ensure that you read our Disclaimer.
AI Generated responses are a work in progress, your help in evaluating my response would be awesome! If my response makes sense, please mark it as a solution or give a 
. If my response doesn’t make sense for your topic, please react to it with a 
.
Looking at the documentation, I can provide guidance on improving the performance of an Azure Data Lake Gen2 connector in Solace Cloud:
The low message rate (1-2 messages per second) could be due to several factors. Here’s my analysis:
-
The REST consumer being disabled with “Missing OAuth2 Token” error indicates an authentication issue that needs to be resolved first. The OAuth2 credentials need to be properly configured and valid.
-
For improving throughput, consider these best practices:
- Batch sending messages when possible to optimize performance
- Ensure proper queue configuration including:
- Appropriate max spool usage values
- Proper queue permissions and access types
- Correct max-delivered-unacked-msgs-per-flow settings
- Appropriate window sizes
- For troubleshooting:
- Use the RDP Error Log Viewer tool to get more details about any RDP errors
- Check the REST consumer logs on the broker for additional error information
- Verify OAuth2 token configuration and expiration
The documentation doesn’t provide specific throughput numbers or configuration recommendations for the Azure Data Lake Gen2 connector. For detailed connector-specific guidance, you’ll want to refer to:
- Azure Data Lake connector documentation
- REST consumer configuration guide
- Queue configuration documentation
I recommend first resolving the OAuth2 token issue, then tuning the queue and connector configurations based on your specific requirements while monitoring performance metrics.
There are no errors in the RDP. Its just that the outgoing message rate is low.
Hey Grupeet, how many subscriptions do you have on the queue? And whats the issues with having a queue per subscription and configure an RDP for each? I’m trying to understand the use-case
@Tamimi , we just have 7 subscriptions. Which of the following options should we consider so that we can deliver the messages to data lake at a faster rate:
- Single queue and RDP for these 7 subscriptions
- Separate queue and RDP for each subscription so 7 queues and 7 RDPs for data lake connection
- Single RDP and 7 queues for each subscription
Also, I came across a post where it was mentioned that if we increase the “Outgoing Connection Count” property for a REST consumer, the outgoing message rate would be increased. I want to know what is the max limit of this parameter.
At the end, my intention is to send messages to data lake at a faster rate. The current outgoing message rate to data lake is 1-2 messages per sec and the value of “Outgoing Connection Count” for the REST consumer is 3, which is the default value.