Error sending message - Direct buffer memory

ruplim

Hello
We are using ( Solace + Spring JMS + Spring Boot + Pivotal Cloud Foundry ) to read a message from a queue and post to Topic. We are facing issue when reading 1 message of size - 7MB and posting to a topic. Smaller message size are working fine.

Below error when sending to topic
Error Message : javax.jms.JMSException: Error sending message - internal error (Direct buffer memory)
Could you please help on the next steps.

Error Stack:
Caused by: org.springframework.jms.UncategorizedJmsException: Uncategorized exception occurred during JMS processing; nested exception is javax.jms.JMSException: Error sending message - internal error (Direct buffer memory)
at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:311) ~[spring-jms-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:185) ~[spring-jms-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:507) ~[spring-jms-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:584) ~[spring-jms-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:661) ~[spring-jms-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
at com.....inbound.service.impl.PublishServiceImpl.publishError(PublishServiceImpl.java:49) ~[classes/:na]
at com.....inbound.messaging.ErrorHandler.handleError(ErrorHandler.java:42) ~[classes/:na]
at com.....inbound.messaging.InboundListener.onMessage(InboundListener.java:37) ~[classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_222]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_222]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_222]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_222]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:170) ~[spring-messaging-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:120) ~[spring-messaging-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
at org.springframework.jms.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:114) ~[spring-jms-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
... 10 common frames omitted
OUT Caused by: javax.jms.JMSException: Error sending message - internal error (Direct buffer memory)
OUT at com.solacesystems.jms.impl.JCSMPExceptionMapper.get(JCSMPExceptionMapper.java:107) ~[sol-jms-10.3.1.jar!/:na]
OUT at com.solacesystems.jms.impl.Validator.createJMSException(Validator.java:578) ~[sol-jms-10.3.1.jar!/:na]
OUT at com.solacesystems.jms.SolMessageProducer.sendMessage(SolMessageProducer.java:387) ~[sol-jms-10.3.1.jar!/:na]
OUT at com.solacesystems.jms.SolMessageProducer.send(SolMessageProducer.java:182) ~[sol-jms-10.3.1.jar!/:na]
OUT at org.springframework.jms.connection.CachedMessageProducer.send(CachedMessageProducer.java:155) ~[spring-jms-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
OUT at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:631) ~[spring-jms-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
OUT at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:608) ~[spring-jms-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
OUT at org.springframework.jms.core.JmsTemplate.lambda$send$3(JmsTemplate.java:586) ~[spring-jms-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
OUT at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:504) ~[spring-jms-5.1.6.RELEASE.jar!/:5.1.6.RELEASE]
OUT ... 22 common frames omitted
OUT Caused by: java.lang.OutOfMemoryError: Direct buffer memory
OUT at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_222]
OUT at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123) ~[na:1.8.0_222]
OUT at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311) ~[na:1.8.0_222]
OUT at com.solacesystems.jcsmp.protocol.smf.SimpleSmfClient.enqueueForNetOutput(SimpleSmfClient.java:592) ~[sol-jcsmp-10.3.1.jar!/:na]
OUT at com.solacesystems.jcsmp.protocol.smf.SimpleSmfClient.doSmfSharedWrite(SimpleSmfClient.java:542) ~[sol-jcsmp-10.3.1.jar!/:na]
OUT at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel.sendPubMsgBuffer(TcpClientChannel.java:611) ~[sol-jcsmp-10.3.1.jar!/:na]
OUT at com.solacesystems.jcsmp.protocol.impl.TcpClientChannel.send(TcpClientChannel.java:570) ~[sol-jcsmp-10.3.1.jar!/:na]
OUT at com.solacesystems.jcsmp.impl.JCSMPXMLMessageProducer.sendMsgOnce(JCSMPXMLMessageProducer.java:1211) ~[sol-jcsmp-10.3.1.jar!/:na]
OUT at com.solacesystems.jcsmp.impl.JCSMPXMLMessageProducer.sendImpl(JCSMPXMLMessageProducer.java:1031) ~[sol-jcsmp-10.3.1.jar!/:na]
OUT at com.solacesystems.jcsmp.impl.JCSMPXMLMessageProducer.sendToDestination(JCSMPXMLMessageProducer.java:848) ~[sol-jcsmp-10.3.1.jar!/:na]
OUT at com.solacesystems.jcsmp.impl.JCSMPXMLMessageProducer.sendToDestination(JCSMPXMLMessageProducer.java:795) ~[sol-jcsmp-10.3.1.jar!/:na]
OUT at com.solacesystems.jcsmp.impl.JCSMPXMLMessageProducer.send(JCSMPXMLMessageProducer.java:615) ~[sol-jcsmp-10.3.1.jar!/:na]
OUT at com.solacesystems.jms.impl.JCSMPMessageProducer.send(JCSMPMessageProducer.java:26) ~[sol-jms-10.3.1.jar!/:na]
OUT at com.solacesystems.jms.impl.TransactedMessageProducerAdapter.send(TransactedMessageProducerAdapter.java:90) ~[sol-jms-10.3.1.jar!/:na]
OUT at com.solacesystems.jms.SolMessageProducer.sendMessage(SolMessageProducer.java:380) ~[sol-jms-10.3.1.jar!/:na]
OUT ... 28 common frames omitted

Aaron

Hi ruplim! That's an interesting sounding problem. I can see from the stack trace that there are issues in the underlying Solace JCSMP API during the send() call. When using the JCSMP API directly, there are some settings you can adjust when sending large messages (see: https://docs.solace.com/Solace-PubSub-Messaging-APIs/API-Developer-Guide/Java-API-Best-Practices.htm#Increase ), but I don't know if or how to adjust these when using Spring. So, to help with some debugging:

• This is happening when you are sending just a single 7 MB message? Only one??
• Since it doesn't happen with smaller sizes, at what point does it start happening? 4MB? 6MB?
• Did you try using: https://github.com/SolaceProducts/solace-java-spring-boot or https://github.com/SolaceProducts/solace-jms-spring-boot ?

Thanks!

ruplim

Hello @Aaron
This is happening when you are sending just a single 7 MB message? Only one??
Yes - when we push 1 single message of around 7MB. The app is deployed in Pivotal Cloud Foundry. Seems the MaxDirectMemotySize setting in PCF is only 10MB. We tried setting solace property PUB_USE_INTERMEDIATE_DIRECT_BUF to false but it did not work. Any other properties provided by solace we can look into? Changing the PCF memory setting might not be possible.
Since it doesn't happen with smaller sizes, at what point does it start happening? 4MB? 6MB?
Smaller size go through. Most probably till it hits the PCF MaxDirectMemory

marc

@Aaron, you can set Solace Java API properties via the Spring config file under solace.java.apiProperties as seen here if that helps?

Honestly I haven't messed with many of these JCSMP settings, but maybe setting PUB_USE_INTERMEDIATE_DIRECT_BUF to false could help to avoid having an intermediate buffer. @Aaron - thoughts and any ramifications of doing that?

aharonmu

So what is right way to go?
Disable it?
Or increase the buffer?