Spring cloud stream Solace binder prevents application from running when Solace is down

siavashsoleymani

Hello everyone,

I am currently working on an application that consumes messages from Solace and serves users with HTTP endpoints. However, when Solace is down, the Solace binder prevents the application from running until Solace is reachable again. This behavior is problematic for our business as we want to continue serving our clients through HTTP even if we are unable to consume messages from Solace.

I have investigated this issue for hours and have not been able to find a workaround. As this is crucial for our business, I would appreciate any guidance on how to achieve this goal.

marc

Thanks for troubleshooting this @siavashsoleymani. I think you are correct in that the @PostConstruct failing is preventing the bean itself from being created.

I have opened a github issue here to track this: https://github.com/SolaceProducts/solace-spring-cloud/issues/223

If you are a customer with support please open a support ticket as that is always useful in getting issues like this prioritized.

marc

Hi @siavashsoleymani,

Can you share more about what causes the situation? Does this occur when the app has been running successfully and then it loses connection to the Event Broker, or if the app can't connect to the Event Broker when it first starts up?

siavashsoleymani

Hi @marc thanks for your prompt reply, when the application is running there is no problem and it starts trying to reconnect to the broker while it is serving the HTTP clients, the problem is when you want to run a new instance or restart an instance while the broker is down then the application starts retrying and doesn't allow the HTTP server serve the clients until the connection is completed or the retry counts reach the limit (when the retry count is reached the application exits)

Aaron

I'm a bit confused..! From my understanding, you have an app of some sort that does two things:

• Using SCSt, connects to Solace and consumes messages from a queue?
• Provides an HTTP interface to your client applications to make queries? So this app is a webserver of some sort.

These seem like two separate interfaces? A "data feed" for your app, and a "client API' interface. Does the app require some sort of initialization when starting, reading data from Solace, before it can start providing this HTTP interface? If not, I don't know why they are tied-together, the Solace connection and the HTTP interface.

BTW, for proper production deployments of Solace, you'd always deploy in an HA-pair, so if a single broker was down or unreachable, the app would auto-connect to the backup broker.

siavashsoleymani

@Aaron your understanding is correct the application has a DB and it connects to Solace through SCS for updating and populating the DB, and at the same time it provides some HTTP endpoints for its users to read and see those data from the DB, the problem is SCS bean should be created successfully in the bean creation chain and apparently embedded application server bean is built after that and prevents the application from running, in our use-case it is crucial to start the application independently even when the broker is out of reach.

org.springframework.context.ApplicationContextException: Failed to start bean 'inputBindingLifecycle'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.solace.spring.cloud.stream.binder.config.SolaceMessageChannelBinderConfiguration': Invocation of init method failed; nested exception is com.solacesystems.jcsmp.InvalidPropertiesException: All hosts in the host list: 'tcps://example:55443' are not resolvable
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring-context-5.3.22.jar:5.3.22]
	at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[spring-context-5.3.22.jar:5.3.22]
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring-context-5.3.22.jar:5.3.22]
	at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring-context-5.3.22.jar:5.3.22]
	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring-context-5.3.22.jar:5.3.22]
	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935) ~[spring-context-5.3.22.jar:5.3.22]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) ~[spring-context-5.3.22.jar:5.3.22]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) ~[spring-boot-2.7.3.jar:2.7.3]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734) ~[spring-boot-2.7.3.jar:2.7.3]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) ~[spring-boot-2.7.3.jar:2.7.3]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-2.7.3.jar:2.7.3]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring-boot-2.7.3.jar:2.7.3]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) ~[spring-boot-2.7.3.jar:2.7.3]
	at com.ingka.solacescs.SolacescsApplicationKt.main(SolacescsApplication.kt:13) ~[main/:na]
Caused by: com.solacesystems.jcsmp.InvalidPropertiesException: All hosts in the host list: 'tcps://example:55443' are not resolvable
	at com.solacesystems.jcsmp.JCSMPFactory.validate(JCSMPFactory.java:410) ~[sol-jcsmp-10.16.0.jar:na]
	at com.solacesystems.jcsmp.JCSMPFactory.createSession(JCSMPFactory.java:164) ~[sol-jcsmp-10.16.0.jar:na]
	at com.solacesystems.jcsmp.JCSMPFactory.createSession(JCSMPFactory.java:122) ~[sol-jcsmp-10.16.0.jar:na]
	at com.solace.spring.cloud.stream.binder.config.SolaceMessageChannelBinderConfiguration.initSession(SolaceMessageChannelBinderConfiguration.java:57) ~[spring-cloud-stream-binder-solace-3.4.0.jar:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:389) ~[spring-beans-5.3.22.jar:5.3.22]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:333) ~[spring-beans-5.3.22.jar:5.3.22]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:157) ~[spring-beans-5.3.22.jar:5.3.22]
	... 37 common frames omitted


5 actionable tasks: 3 executed, 2 up-to-date


FAILURE: Build failed with an exception.

arih

I believe it is by design the SCSt bean requires the binding to be up - cause there's no point starting up if it won't be able to do its job related to the messages. I simply think you should not deviate from such behaviour :) I would again go back to one important point, the Solace broker should not be down anyway :)

And while I share Aaron's view that the two different interface may be separated, I'm thinking of a few ideas:

• it's quite easy to spin up a broker as part of CI/CD - so if this is meant for testing, you could 'embed' the step to spin up a new broker with the application specific config (username, ACL, queues, etc.) along, so you'll always have the broker on each start up sequence and it's within your control.
• is there something like a lazy-loading for SCSt bean?
• maybe instead of Queue consumers, you can switch to having just a REST API endpoint and let Solace push the messages to you via REST, we call this RDP feature - that way, just like HTTP-based interaction, there is no 'persistent' connection initiated and kept open like the binder.

just my two cents.

Ari

Aaron

I'm not a Spring expert, but I would think that there's someway in Spring to configure the Solace connection as a "nice to have" and not a "required to start" resource. No? Like, the app starts up and does its basic checks/initialization... it might have a Boolean var to show whether the Solace connection is up or not... but that doesn't need to be required for startup. Tries to connect to Solace in the background. And when a client polls your app via HTTP, it can either return its cached/startup values (marked as "old"), or if the Solace connection is live, then its up-to-date values (marked as "live"). ... or something like that?

Unless the app definitely needs the Solace connection to be up in order to be serving the correct data to the HTTP clients, then I think failing at startup is probably correct?

siavashsoleymani

Apparently, there is no option to make it optional as far as I checked, and exactly as Aaron mentioned if there was an option to make this bean lazy or make this behavior optional that would be nice, as I see in the code I think this option would be achievable by changing the behavior of the PostConstruct method in this class 'SolaceMessageChannelBinderConfiguration.class'

siavashsoleymani

Also, I tried to extend the related bean and change the behavior but there is a problem with the second constructor parameter, the error says the second parameter cannot be autowired but I can see the SCS solace binder producing bean (I think the problem is about bean creation order)

import com.solace.spring.cloud.stream.binder.SolaceMessageChannelBinder
import com.solace.spring.cloud.stream.binder.config.SolaceBinderHealthIndicatorConfiguration
import com.solace.spring.cloud.stream.binder.config.SolaceMessageChannelBinderConfiguration
import com.solace.spring.cloud.stream.binder.meter.SolaceMeterAccessor
import com.solace.spring.cloud.stream.binder.properties.SolaceExtendedBindingProperties
import com.solace.spring.cloud.stream.binder.provisioning.SolaceQueueProvisioner
import com.solace.spring.cloud.stream.binder.util.SolaceSessionEventHandler
import com.solacesystems.jcsmp.*
import com.solacesystems.jcsmp.impl.client.ClientInfoProvider
import org.springframework.boot.context.properties.EnableConfigurationProperties
import org.springframework.context.annotation.*
import org.springframework.lang.Nullable
import java.util.concurrent.CompletableFuture
import javax.annotation.PostConstruct

@Primary
@Configuration
@Import(SolaceBinderHealthIndicatorConfiguration::class)
@EnableConfigurationProperties(SolaceExtendedBindingProperties::class)
@Lazy
class SolaceMessageChannelBinderConfiguration(
    private val jcsmpProperties: JCSMPProperties,
    private val solaceExtendedBindingProperties: SolaceExtendedBindingProperties,
    private val solaceSessionEventHandler: SolaceSessionEventHandler,
) : SolaceMessageChannelBinderConfiguration(
    jcsmpProperties,
    solaceExtendedBindingProperties,
    solaceSessionEventHandler,
) {
    private var jcsmpSession: JCSMPSession? = null
    private var context: Context? = null

    @PostConstruct
    private fun initSession() {
        CompletableFuture.runAsync {
            val jcsmpProperties = jcsmpProperties.clone() as JCSMPProperties
            jcsmpProperties.setProperty(JCSMPProperties.CLIENT_INFO_PROVIDER, SolaceBinderClientInfoProvider())
            try {
                if (solaceSessionEventHandler != null) {
                    context = JCSMPFactory.onlyInstance().createContext(ContextProperties())
                    jcsmpSession =
                        JCSMPFactory.onlyInstance().createSession(jcsmpProperties, context, solaceSessionEventHandler)
                } else {
                    jcsmpSession = JCSMPFactory.onlyInstance().createSession(jcsmpProperties)
                }
                jcsmpSession!!.connect()
                solaceSessionEventHandler?.connected()
            } catch (e: Exception) {
                if (context != null) {
                    context!!.destroy()
                }
                throw e
            }
        }
    }

    @Bean
    fun solaceMessageChannelBinder(
        solaceQueueProvisioner: SolaceQueueProvisioner?,
        @Nullable solaceMeterAccessor: SolaceMeterAccessor?,
    ): SolaceMessageChannelBinder? {
        val binder = SolaceMessageChannelBinder(jcsmpSession, context, solaceQueueProvisioner)
        binder.setExtendedBindingProperties(solaceExtendedBindingProperties)
        binder.setSolaceMeterAccessor(solaceMeterAccessor)
        return binder
    }

    @Bean
    fun provisioningProvider(): SolaceQueueProvisioner? {
        return SolaceQueueProvisioner(jcsmpSession)
    }
}

internal class SolaceBinderClientInfoProvider : ClientInfoProvider() {
    override fun getSoftwareVersion(): String {
        return String.format("3.4.0 (%s)", super.getSoftwareVersion())
    }

    override fun getSoftwareDate(): String {
        return String.format("2022-10-20 19:33:36 (%s)", super.getSoftwareDate())
    }

    override fun getPlatform(): String {
        return this.getPlatform("Solace Spring Cloud Stream Binder (JCSMP SDK)")
    }
}

Error:

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2023-05-01 10:14:20.291 ERROR 57632 --- [           main] o.s.b.d.LoggingFailureAnalysisReporter   : 


***************************
APPLICATION FAILED TO START
***************************


Description:


Parameter 2 of constructor in com.ingka.solacescs.SolaceMessageChannelBinderConfiguration required a bean of type 'com.solace.spring.cloud.stream.binder.util.SolaceSessionEventHandler' that could not be found.




Action:


Consider defining a bean of type 'com.solace.spring.cloud.stream.binder.util.SolaceSessionEventHandler' in your configuration.

marc

Hi @siavashsoleymani,

Thanks for the information that this is only an issue on initial start-up of the app. In general I tend to agree with @arih that if your app can't start up gracefully you usually don't want it to start-up at all.

That said, if you have requirements to do so I'm not sure of the solution off the top of my head, but I'll ping someone over at Spring and see if they know.

marc

Hi @siavashsoleymani,

Okay - I was able to chat with them today. Long term I think there are probably some Solace binder enhancements that we can make to avoid this situation, but in the short term Spring allows you to choose the order that your classes start via Spring Boot autoconfiguration (remember at the end of the day you are creating a Spring Boot microservice). Can you try using AutoConfigureBefore and/or AutoConfigureAfter to tell your HTTP class to start up first? It's probably worth checking out the org.springframework.boot.autoconfigure package to see if there are some other options as well.

https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/autoconfigure/AutoConfigureBefore.html

https://docs.spring.io/spring-boot/docs/current/api/org/springframework/boot/autoconfigure/AutoConfigureAfter.html

Hope that helps!

Tamimi

Just chiming in here as well, stepping back and looking at your overall design for the service, you mentioned that this service does the following:

1. Spin up and connect to a Solace Broker
2. Consumes messages from a queue and updates a database
3. Also acts as an HTTP endpoint for other clients to read data from the database

I have a couple of questions, that maybe could trigger a re-design of the overall approach.

• Have you considered two separate applications? One that connects to the solace broker and updates the DB and another http server microservice that acts as the endpoint for clients to connect to?
• Is it possible to have the clients establish a session with the session broker and consume the data from there?
• Did the AutoConfigureBefore / AutoConfigureAfter approach work?

siavashsoleymani

Hi @marc many thanks for your follow up I tested it both with AutoConfigureBefore and AutoConfigureAfter but none of them works, I think the problem is because the bean failed to be initialized (because first-time connection happens in the @PostConstruct method of the SolaceMessageChannelBinderConfiguration class, which prevents the bean from creation and subsequently the spring context)

I tested it like this:

@AutoConfigureAfter(SolaceMessageChannelBinderConfiguration::class, SolaceServiceAutoConfiguration::class)
@SpringBootApplication
class SolacescsApplication

fun main(args: Array<String>) {
    runApplication<SolacescsApplication>(*args)
}

siavashsoleymani

hey @Tamimi

Your understanding of the use case and design is totally correct.

In regards to your questions:

1. Yes we have considered doing it but it doesn't worth it for us to do so at the moment
2. The problem is that if the broker is unreachable for any reason the application fails to startup so there is no time to do that
3. I have tried it and mentioned the code in the previous post to Marc and it doesn't work (or maybe I am doing something wrong)

marc

Thanks for troubleshooting this @siavashsoleymani. I think you are correct in that the @PostConstruct failing is preventing the bean itself from being created.

I have opened a github issue here to track this: https://github.com/SolaceProducts/solace-spring-cloud/issues/223

If you are a customer with support please open a support ticket as that is always useful in getting issues like this prioritized.