quick update: I added

broker.id=0

in meta.properties. so kafka runs with one replica, but if i increase the replica it fails by this error:

"org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:126)
	at kafka.zk.KafkaZkClient$CheckedEphemeral.getAfterNodeExists(KafkaZkClient.scala:2185)
	at kafka.zk.KafkaZkClient$CheckedEphemeral.create(KafkaZkClient.scala:2123)
	at kafka.zk.KafkaZkClient.checkedEphemeralCreate(KafkaZkClient.scala:2090)
	at kafka.zk.KafkaZkClient.registerBroker(KafkaZkClient.scala:102)
	at kafka.server.KafkaServer.startup(KafkaServer.scala:356)
	at kafka.Kafka$.main(Kafka.scala:115)
	at kafka.Kafka.main(Kafka.scala)"

Hi, please be patient with us as we’re supporting a large community! @dazzling-yak-93039 might be able to speak here.

Hi, where are you seeing these logs?

The logs are from Kafka container, which i deployed in GKE cluster! but i think i need to resolve the zookeeper (1 replica out of 3) being unschedulable

cc this week's OSS oncall @brainy-tent-14503. Have you taken a look at the health of your kafka deployment?

@brainy-tent-14503 - Hey David, would be great if i get some help here!

If you are manually setting the

broker.id

then you need to set different ids for each replica. Each broker replica needs a unique id, they cannot all be configured to be id 0. Normally you do not set the broker id and let it be generated. For more information refer to kafka documentation.

@brainy-tent-14503 - So setting the broker.id = 0 is causing this error ? "EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1293)" Doesn't it necessarily do anything with zk configuration?

The broker ids are stored in zookeeper, it is likely that error message is from the kafka broker recognizing an inconsistent state and shutting down.

Zookeeper config:

env:

- name: BITNAMI_DEBUG

value: "false"

- name: ZOO_DATA_LOG_DIR

value: ""

- name: ZOO_PORT_NUMBER

value: "2181"

- name: ZOO_TICK_TIME

value: "2000"

- name: ZOO_INIT_LIMIT

value: "10"

- name: ZOO_SYNC_LIMIT

value: "5"

- name: ZOO_PRE_ALLOC_SIZE

value: "65536"

- name: ZOO_SNAPCOUNT

value: "100000"

- name: ZOO_MAX_CLIENT_CNXNS

value: "60"

- name: ZOO_4LW_COMMANDS_WHITELIST

value: "srvr, mntr, ruok"

- name: ZOO_LISTEN_ALLIPS_ENABLED

value: "no"

- name: ZOO_AUTOPURGE_INTERVAL

value: "0"

- name: ZOO_AUTOPURGE_RETAIN_COUNT

value: "3"

- name: ZOO_MAX_SESSION_TIMEOUT

value: "40000"

- name: ZOO_SERVERS

value: <zoo_servers>

- name: ZOO_ENABLE_AUTH

value: "no"

- name: ZOO_ENABLE_QUORUM_AUTH

value: "no"

- name: ZOO_HEAP_SIZE

value: "1024"

- name: ZOO_LOG_LEVEL

value: "ERROR"

- name: ALLOW_ANONYMOUS_LOGIN

value: "yes"

- name: POD_NAME

valueFrom:

fieldRef:

apiVersion: v1

fieldPath: metadata.name

ports:

- name: client

containerPort: 2181

- name: follower

containerPort: 2888

- name: election

containerPort: 3888

Kafka config:

env:

- name: BITNAMI_DEBUG

value: "false"

- name: MY_POD_IP

valueFrom:

fieldRef:

fieldPath: status.podIP

- name: MY_POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

- name: KAFKA_CFG_ZOOKEEPER_CONNECT

value: release-name-zookeeper:2181

- name: KAFKA_INTER_BROKER_LISTENER_NAME

value: "INTERNAL"

- name: KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP

value: "INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT"

- name: KAFKA_CFG_LISTENERS

value: "INTERNAL://:9094,CLIENT://:9092"

- name: KAFKA_CFG_ADVERTISED_LISTENERS

value: "<listeners>"

- name: ALLOW_PLAINTEXT_LISTENER

value: "yes"

- name: KAFKA_ZOOKEEPER_PROTOCOL

value: PLAINTEXT

- name: KAFKA_VOLUME_DIR

value: "/bitnami/kafka"

- name: KAFKA_LOG_DIR

value: "/opt/bitnami/kafka/logs"

- name: KAFKA_CFG_DELETE_TOPIC_ENABLE

value: "false"

- name: KAFKA_CFG_AUTO_CREATE_TOPICS_ENABLE

value: "true"

- name: KAFKA_HEAP_OPTS

value: "-Xmx1024m -Xms1024m"

- name: KAFKA_CFG_LOG_FLUSH_INTERVAL_MESSAGES

value: "10000"

- name: KAFKA_CFG_LOG_FLUSH_INTERVAL_MS

value: "1000"

- name: KAFKA_CFG_LOG_RETENTION_BYTES

value: "1073741824"

- name: KAFKA_CFG_LOG_RETENTION_CHECK_INTERVAL_MS

value: "300000"

- name: KAFKA_CFG_LOG_RETENTION_HOURS

value: "168"

- name: KAFKA_CFG_MESSAGE_MAX_BYTES

value: "1000012"

- name: KAFKA_CFG_LOG_SEGMENT_BYTES

value: "1073741824"

- name: KAFKA_CFG_LOG_DIRS

value: "/bitnami/kafka/data"

- name: KAFKA_CFG_DEFAULT_REPLICATION_FACTOR

value: "1"

- name: KAFKA_CFG_OFFSETS_TOPIC_REPLICATION_FACTOR

value: "1"

- name: KAFKA_CFG_TRANSACTION_STATE_LOG_REPLICATION_FACTOR

value: "1"

- name: KAFKA_CFG_TRANSACTION_STATE_LOG_MIN_ISR

value: "1"

- name: KAFKA_CFG_NUM_IO_THREADS

value: "8"

- name: KAFKA_CFG_NUM_NETWORK_THREADS

value: "3"

- name: KAFKA_CFG_NUM_PARTITIONS

value: "1"

- name: KAFKA_CFG_NUM_RECOVERY_THREADS_PER_DATA_DIR

value: "1"

- name: KAFKA_CFG_SOCKET_RECEIVE_BUFFER_BYTES

value: "102400"

- name: KAFKA_CFG_SOCKET_REQUEST_MAX_BYTES

value: "104857600"

- name: KAFKA_CFG_SOCKET_SEND_BUFFER_BYTES

value: "102400"

- name: KAFKA_CFG_ZOOKEEPER_CONNECTION_TIMEOUT_MS

value: "60000"

- name: KAFKA_CFG_AUTHORIZER_CLASS_NAME

value: ""

- name: KAFKA_CFG_ALLOW_EVERYONE_IF_NO_ACL_FOUND

value: "true"

- name: KAFKA_CFG_SUPER_USERS

value: "User:admin"

- name: KAFKA_ENABLE_KRAFT

value: "false"

Can you please help me to find out why brokers are not getting assigned in kafka automatically?

The kafka containers are setup by the bitami chart here, this is an upstream project with documentation here (linked to the broker configuration section). There may be further guidance on configuring multiple brokers in their documentation.