HA Pair Issue: AD-NotReady Local ADB Key Invalid

AllenW

Seeing an issue with a HA Software Event Broker pair where the standby broker keeps rebooting.

Each time it reboots we see the broker go into AD-Standby state which is expected, but then goings to AD-NotReady

Event logs is outputting the following message:
backup-broker event: SYSTEM: SYSTEM_AD_MSG_SPOOL_CHG: - - Message spool on Backup Virtual Router operational state change from AD-Standby to AD-NotReady Local ADB Key Invalid

Config-Sync is in sync however seeing some invalid keys for 'show redundancy detail' output.

backup-broker> show redundancy detail

….

Activity Status:                                       Mate Active

Operational Status                                     Not Ready

Redundancy Config Status                               Enabled

Message Spool Status                                   Not Ready

SMRP Status                    Ready                   Ready

Db Build Status                Ready                   Ready

Db Sync Status                 Ready                   Ready

Internal Priority                                      Backup-Reconcile

Internal Activity Status                               Mate Active

Internal Redundancy State                              Bkup-NotReady
Message Spool Status:                                  Not Ready

Message Spool Config Status                            Enabled

VRID Config Status                                     Ready

ADB Status                                             Ready

Flash Module Status                                    Ready

Power Module Status                                    Ready

ADB Contents                                           Stale

Local Contents Key                                     Invalid

Mate Contents Key                                      XXX.XXX.XXX.XX:5581,6

Schema Match                                           No

Disk Status                                            Ready

Disk Contents                                          Invalid

Disk Key (Primary)                                     Invalid

Disk Key (Backup)                                      Invalid

ADB Datapath Status                                    Ready

Internal Redundancy State                              AD-NotReady

primary-broker> show redundancy detail

….

Priority Reported By Mate:     None

ADB Hello Protocol             None

VRRP

intf0                      None (-1)
Activity Status:               Local Active

Operational Status             Ready

Redundancy Config Status       Enabled

Message Spool Status           Ready

SMRP Status                    Ready

Db Build Status                Ready

Db Sync Status                 Ready

Internal Priority              Active

Internal Activity Status       Local Active

Internal Redundancy State      Pri-Active
Message Spool Status:          Ready

Message Spool Config Status    Enabled

VRID Config Status             Ready

ADB Status                     Ready

Flash Module Status            Ready

Power Module Status            Ready

ADB Contents                   Ready

Local Contents Key             XXX.XXX.XXX.XX:5580,6

Mate Contents Key              Invalid

Schema Match                   Yes

Disk Status                    Ready

Disk Contents                  Ready

Disk Key (Primary)             XXX.XXX.XXX.XX:5580,6

Disk Key (Backup)              XXX.XXX.XXX.XX:5580,6

ADB Datapath Status            Ready

Internal Redundancy State      AD-Active

Lock Owner                     primary-broker

It seems the backup-broker is still stuck in Backup-reconcile state, and continues to reboot.

Is this a DB issue or related to the ADB key itself?

Aaron

Is your Monitor node saying anything?

There's a "pre-shared key" that must be configured as part of your Redundancy Group configuration, and it must be the same for all 3 brokers in the HA triple. Perhaps that key got blanked or typed incorrectly..? https://docs.solace.com/Features/HA-Redundancy/Pre-Shared-Keys-SMB.htm