Problem with new Snowflake connector

Connectors & Integrations
1

Hi,

When testing the v3 snowflake connector from Solace Snowflake Docker all the json messages are being delivered as a base64 encoded string to the record_content column, the record_metadata column is unchanged as json.

When using the same binding and workflow config with the v2 jar file both record_content and record_metadata columns are in json format.

Is there a setting which needs to be added to the application.yml so that the messages are delivered as json?

application.yml

spring:
  cloud:
    stream:
      bindings:
        input-0:
          destination: ${input-0-destination}
          binder: solace
        output-0:
          destination: ${output-0-destination} 
          binder: snowflake
snowflake:
  ingestion-method: SNOWPIPE_STREAMING
  url: ${snowflake-url}
  username: ${snowflake-username}
  role: ${snowflake-role}
  private-key: ${snowflake-private-key}
  private-key-path: file:/app/external/spring/config/solace_service.p8
  private-key-password: ${snowflake-private-key-password}
  warehouse: ${snowflake-warehouse}

solace:
  java:
    host: ${solace-host}
    msgVpn: ${solace-msgVpn}
    clientUsername: ${solace-clientUsername}
    clientPassword: ${solace-clientPassword}
    connectRetries: 3
    connectRetriesPerHost: 0
    reconnectRetries: 3
  connector:
    workflows:
      0:
        enabled: ${workflows-0}
2

Hi there, I’m Solly AI, I generate responses to your questions using the power of the Solace Agent Mesh:

Please remember that AI technology can make mistakes so ensure that you read our Disclaimer.

AI Generated responses are a work in progress, your help in evaluating my response would be awesome! If my response makes sense, please mark it as a solution or give a :+1:. If my response doesn’t make sense for your topic, please react to it with a :shaking_face:.

Looking through the documentation, I cannot find specific information about configuring the Snowflake Connector v3 to deliver messages as JSON instead of base64 encoded strings in the record_content column. The documentation covers various configuration aspects but does not specifically address this payload encoding behavior.

You might find relevant information in the following documentation:

I recommend searching through these resources for more details about message payload handling and transformation options.

3

Hi,
I am from the Product Management team at Solace.

Is it possible that your source messages are published differently than when you were testing in v2.0? I don’t believe there is a config property for this (I am checking with dev to be sure). It publishes to Snowflake with the payload it got from PubSub+ as either a byte message or text message depending on the source.

What I tend to do is account for this in my Snowflake query by converting it to JSON.

select to_char(record_content) from
   DB.SCHEMA.TABLE;

I have actually played around with this and also have used TO_JSON() as well. I can’t recall which has worked better so you might experiment.

The other suggestion is, if you have control of your publishers, to publish text messages.

I hope this helps,
Andrew MacKenzie

4

Hi,

I’m using the same solace queue for testing both versions.

With the v3 {"notification":{"deal":.. is inserted as eyJub3RpZmljYXRpb24iOnsiZGVhbCI6... which makes me think it is new setting required in the application.yml between the versions.

Martin

5

This issue has been fixed in 3.2.4 released today.