Consumer Application Server going down

udaysinghrawat
udaysinghrawat Member Posts: 2 ✭
in Connectors & Integrations #1

We are seeing below error in JVM Heap Dump and our Consumer Application servers going down everytime:

"Context_115_jms_1868_ConsumerDispatcher" #6387 daemon prio=5 os_prio=0 cpu=0.21ms elapsed=70319.77s tid=0x00007fda7031d800 nid=0x1e08 waiting on condition [0x00007fda26f16000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.11/Native Method)
- parking to wait for <0x0000000567a531b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.11/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.11/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.ArrayBlockingQueue.take(java.base@11.0.11/ArrayBlockingQueue.java:417)
at com.solacesystems.jcsmp.protocol.nio.impl.ConsumerNotificationDispatcher.eventLoop(ConsumerNotificationDispatcher.java:111)
at com.solacesystems.jcsmp.protocol.nio.impl.ConsumerNotificationDispatcher.run(ConsumerNotificationDispatcher.java:130)
at java.lang.Thread.run(java.base@11.0.11/Thread.java:829)

Locked ownable synchronizers:
- None

Comments

  • Ragnar
    Ragnar Member, Employee Posts: 67 Solace Employee
    #2

    Hi.

    Can you provide more details. The thread trace here does not have any errors in it. It is just a thread that is waiting for an event to be added to a queue. This is normal.

    What do you mean by "Consumer Application servers going down"? Do you seen an exception? Is there an exception back trace?

    Can you provide the full JVM dump so we can see all the threads, including any thread where there may be an error?

    What are the visible symptoms to you when the application fails?

    Thank you.
    Ragnar

  • udaysinghrawat
    udaysinghrawat Member Posts: 2 ✭
    #3

    Thanks Ragnar for commenting, Attaching the full Heap Dump, issue is Memory is going 100% and Consumer Application (Boomi) Server is going down/crashing.

  • Aaron
    Aaron Member, Administrator, Moderator, Employee Posts: 638 admin
    #4

    Hi @udaysinghrawat ... I don't want to go looking through your heap dump, but I have a feeling that it could be because your consuming application is receiving messages off Solace (via a queue or something), and then failing to process them fast enough, or not removing all references to them.

    I've had similar heap/crash issues with Java where I had one thread (the Solace API context thread) pulling messages off Solace and pushing into a LinkedBlockingQueue Java data structure. And then my own application threads (ExecutorService) pulling the messages off that LBQ and processing. Without some smarter logic (e.g. starting/stopping the Solace flow of messages, bounded LBQ, etc.) Java would just happily completely fill my heap with an unbounded LBQ.

    Please provide more details on what your app is doing... what are the message rates, is it processing them, republishing them? Obviously the more detail you give us, the easier it is to help you.

Categories

This Month's Leaders

This Week's Leaders