We have 7 node rabbitmq cluster and we are using sharding queues. We have shutdown all non prod rabbitmq nodes as part of VM patching. As it was non prod, we thought of shutting down all together instead of rolling patching. We couldn’t start the any of the rabbitmq post patching. All of nodes were failing to start with the below error:
2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> “Cannot declare a queue ‘~s’ on node ‘~s’: ~255p”,
2022-06-23 05:13:10.104845-07:00 [error] <0.647.0> [“queue ‘sharding: shard.my.queue – [email protected]’ in vhost ‘my_vhost'”
This is a known issue in some old versions of rabbitmq. rabbitmq node fails to come up with above error. Work around for this is to disable sharding policy for the problematic sharding queue. It can be done either from command line or from rabbitmq admin page. It requires to connect a running node. But as all the nodes were down, we could not disable sharding policy. Luckily, rabbitmq allows disabling sharding pulgin without connecting to active node (rabbitmq-plugins disable rabbitmq_sharding). We disabled plugin and able to bring up the cluster.
Author Profile
Latest entries
- 2023.05.23rabbitmqRabbitmq Connection Error: javax.net.ssl.SSLHandshakeException: Invalid ECDH ServerKeyExchange signature
- 2023.03.24rcuSOA Suite 12.2.1.4 installation: Got exception when auto configuring the schema component(s) with data obtained from shadow table
- 2023.03.09rabbitmqNot able to start Rabbitmq Cluster: Cannot declare a queue ‘~s’ on node ‘~s’: ~255p
- 2023.03.084lw commandsTelegraf agent not able to monitor zookeeper