We have 7 node rabbitmq cluster and we are using sharding queues. We have shutdown all non prod rabbitmq nodes as part of VM patching. As it was non prod, we thought of shutting down all together instead of rolling patching. We couldn’t start the any of the rabbitmq post patching. All of nodes were failing to start with the below error:
2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> “Cannot declare a queue ‘~s’ on node ‘~s’: ~255p”,
2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> [“queue ‘sharding: shard.my.queue – rabbit@rabbit1’ in vhost ‘my_vhost'”
This is a known issue in some old versions of rabbitmq. rabbitmq node fails to come up with above error. Work around for this is to disable sharding policy for the problematic sharding queue. It can be done either from command line or from rabbitmq admin page. It requires to connect a running node. But as all the nodes were down, we could not disable sharding policy. Luckily, rabbitmq allows disabling sharding pulgin without connecting to active node (rabbitmq-plugins disable rabbitmq_sharding). We disabled plugin and able to bring up the cluster.
Author Profile
Latest entries
- August 23, 2023How we reduced SOA OSB provisioning from 4 days to 4 hours
- May 23, 2023rabbitmqRabbitmq Connection Error: javax.net.ssl.SSLHandshakeException: Invalid ECDH ServerKeyExchange signature
- March 24, 2023rcuSOA Suite 12.2.1.4 installation: Got exception when auto configuring the schema component(s) with data obtained from shadow table
- March 9, 2023rabbitmqNot able to start Rabbitmq Cluster: Cannot declare a queue ‘~s’ on node ‘~s’: ~255p