Defragmentation Failing: Unmovable Local Transaction

AllenW

I'm having trouble running defragmentation of a broker's message spool - which is currently at 90% and using up all the disk space.

Each time I run defragmentation it completes about 5% until it exits with the same Exit Condition:
```
Last Result:
Completed On: Oct 30 2023 07:29:21 UTC
Completion %: 5%
Exit Condition: Unmovable Local Transaction
Owned by Client Name: xxx-client-0001/82/#003f000
```
I have located the client/queue and forced a client disconnect as per Solace documentation: https://docs.solace.com/Messaging/Guaranteed-Msg/Defragmenting-Guaranteed-Messaging.htm
"In some cases committing or rolling back a transaction may not be possible if the transaction is not in a prepared state. In these cases, a client disconnect may be required to resolve the blocked transactions."

Are there any other approaches I can take to allow defrag to occur? Thanks

AllenW

Thanks Ron & Marc.

We were able to find a work-around to this issue. Thought I would mention here for posterity.

In the documentation there is an example of an exit condition 'Unmovable XA Transaction XID:XXX-XXXXXXX' which in that case, allows us to either roll-back or delete the transaction blocking the defragmentation using the XID.

But in this case where it was an 'Unmovable Local Transaction' where only the client is listed. Given the client was a producer we had to ask the application owners to temporarily disconnect the client from their end. We attempted to disconnect the client from the Solace side, but it would automatically reconnect instantly - as it was configured to do. Only after this we were able to execute defragmentation and clear disk space.

solace> enable
solace# admin
solace(admin)# system message-spool
solace(admin/system/message-spool)# defragment-spool-files start

While this resolved our disk space issue defragmentation was still unable to complete fully - with it stopping halfway through, this time with the exit code: 'Unmovable XA Transaction XID:XXX-XXXXXXX'. So, this required rolling-back the particular transaction causing the defrag to exit.

solace> enable
solace# admin
solace(admin)# message-spool message-vpn <vpn-name>
solace(admin/message-spool)# rollback-transaction xid <xid>

Then we were able to complete defrag.

This broker is using Solace Version 10.4.1.76

marc

Hi @AllenW,

Unfortunately I've never dealt with this so I won't be any help, but I wanted to chime in and make sure you've also opened a support ticket on this.

RonInglesby

Hi @AllenW

I will ask in our Slack if there is another approach rather than disconnecting the client. In the meantime can you please create a support ticket and email me the details (ron.inglesby@solace.com).

AllenW

Thanks Ron & Marc.

We were able to find a work-around to this issue. Thought I would mention here for posterity.

In the documentation there is an example of an exit condition 'Unmovable XA Transaction XID:XXX-XXXXXXX' which in that case, allows us to either roll-back or delete the transaction blocking the defragmentation using the XID.

But in this case where it was an 'Unmovable Local Transaction' where only the client is listed. Given the client was a producer we had to ask the application owners to temporarily disconnect the client from their end. We attempted to disconnect the client from the Solace side, but it would automatically reconnect instantly - as it was configured to do. Only after this we were able to execute defragmentation and clear disk space.

solace> enable
solace# admin
solace(admin)# system message-spool
solace(admin/system/message-spool)# defragment-spool-files start

While this resolved our disk space issue defragmentation was still unable to complete fully - with it stopping halfway through, this time with the exit code: 'Unmovable XA Transaction XID:XXX-XXXXXXX'. So, this required rolling-back the particular transaction causing the defrag to exit.

solace> enable
solace# admin
solace(admin)# message-spool message-vpn <vpn-name>
solace(admin/message-spool)# rollback-transaction xid <xid>

Then we were able to complete defrag.

This broker is using Solace Version 10.4.1.76

marc

Thank you for sharing @AllenW 🙌