Solace with Couchbase

Maulikd82

Hi,
I'm looking forward to developing a Solace - Couchbase connector for both Source/Sink types. My primary goal to post here is to have some guide/ideas on how do we monitor/configure the connector from Solace Console? Current Solace - Kafka connect is exposed at Kafka's side (uses Kafka Config) and ELK is looking at syslogs so both ain't helpful for me.

[Deleted User]

@Maulikd82 that is fantastic! Let us know what we can do to help support you in this!

amackenzie

@Maulikd82 it's great that you want to create a connector for PubSub+ to/from Couchbase.
As you have observed, we have several examples of connectors available both written by Solace (for example, the Kafka Connect connectors) and written by others (for instance, our integration partners such as Dell Boomi have added Solace-specific configuration to their JMS connectors).

These connectors are always in the context (config/monitoring as you point out) of the integration framework/tool because there is no such framework directly with PubSub+ at this time. This may change at some point, but not imminently enough to help your project.

So your connector would end up being a standalone piece of software that uses one of our APIs to act as a source or sink client to our broker. For this you have a number of choices. You can pick from the language you wish to use such as Java, C, JS/Node, .Net and whether you wish to using SMF (the Solace PubSub+ wireline), an open API like JMS or an open protocol like MQTT or even REST depending on your requirements.

If you choose to go the JMS route, there are examples of JMS connectors (albeit for specific frameworks like StreamSets Data Collector) in open source that may help you get a running start.
Let me know if you want to discuss further.

Maulikd82

Thank you Mackenzie for the detailed response. It solves the connector configuration aspect and for now will build a stand-alone connector. If it's okay and not too much to ask, I'll definitely need some guidance on initial designing and most likely final verification of the connector, from the best practices point-of-view.

Design thought process:
1. Utilising Java APIs (sol-jcsmp - 10.x) as they seem to have all different Messaging APIs for PubSub+.
2. As CB (Couchbase)'s, preferred type is JSON, will be utilising TextMessages on PubSub+.
3. For connectivity, using PubSub+ Cloud.
4. Using Topic for configuration as Connector can be used as an add-on to existing data flow or the data flowing to connector can be used for any other application. Dedicated queue for connector seems bit of an over-kill as it'll add PointToPoint to reference.
5.
Please provide your view on the initial thoughts.

Queries:
1. As CB relies heavily on a document key, which field shall be the best way to share the key from Solace? Kafka has Message Key for each and looking for something similar in PubSub+.

Maulikd82

Version 1.0 is out - https://github.com/maulikdoshi82/pubsub-couchbase-connect.
Please provide your comments.

amackenzie

I will try and take a look at this soon and provide some feedback.
Great initiative @Maulikd82