Kafka Broker Refusing to Start – Timed out waiting for connection

Posted on by Ramesh Poonati

As part of regular maintenance, we shutdown Kafka. When we tried to start it, it failed with below error:

[2021-11-15 05:01:03,824] INFO Socket connection established to mykafka-a1.abc.com/10.0.95.100:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2021-11-15 05:01:03,824] INFO Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2021-11-15 05:01:03,889] INFO [ZooKeeperClient Kafka server] Closing. (kafka.zookeeper.ZooKeeperClient)
[2021-11-15 05:01:03,927] INFO Session: 0x0 closed (org.apache.zookeeper.ZooKeeper)
[2021-11-15 05:01:03,927] INFO EventThread shut down for session: 0x0 (org.apache.zookeeper.ClientCnxn)
[2021-11-15 05:01:03,930] INFO [ZooKeeperClient Kafka server] Closed. (kafka.zookeeper.ZooKeeperClient)
[2021-11-15 05:01:03,933] ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
        at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:258)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
        at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
        at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:254)
        at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:112)
        at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1826)
        at kafka.server.KafkaServer.createZkClient$1(KafkaServer.scala:364)
        at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:387)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:207)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
        at kafka.Kafka$.main(Kafka.scala:84)
        at kafka.Kafka.main(Kafka.scala)
[2021-11-15 05:01:03,934] INFO shutting down (kafka.server.KafkaServer)
[2021-11-15 05:01:03,940] INFO shut down completed (kafka.server.KafkaServer)
[2021-11-15 05:01:03,940] ERROR Exiting Kafka. (kafka.server.KafkaServerStartable)
[2021-11-15 05:01:03,942] INFO shutting down (kafka.server.KafkaServer)

Also, found below logs in Kafka server log:

[2021-11-15 10:15:17,328] INFO Resolved hostname: kafka-dev-app-a4.abc.com to address: kafka-dev-app-a4.abc.com/10.216.95.73 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-11-15 10:15:17,329] WARN Cannot open channel to 5 at election address kafka-dev-app-a5.abc.com/10.216.95.99:3888 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:558)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:534)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:454)
at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:435)
at java.lang.Thread.run(Thread.java:748)
[2021-11-15 10:15:17,332] INFO Resolved hostname: kafka-dev-app-a5.abc.com to address: kafka-dev-app-a5.abc.com/10.216.95.99 (org.apache.zookeeper.server.quorum.QuorumPeer)
[2021-11-15 10:15:18,036] INFO Accepted socket connection from /10.216.95.72:47602 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2021-11-15 10:15:18,036] WARN Exception causing close of session 0x0: ZooKeeperServer not running (org.apache.zookeeper.server.NIOServerCnxn)
[2021-11-15 10:15:18,036] INFO Closed socket connection for client /10.216.95.72:47602 (no session established for client) (org.apache.zookeeper.server.NIOServerCnxn)

We have multiple test envs. In most of the envs, we are using two node 3 node ZooKeeper cluster. But this cluster is configured to have 5 ZooKeeper nodes. Came to know about this after checking zookeeper.properties file. We are able to start brokers after starting all 5 ZooKeeper nodes.

Related Post