Hi everyone 👋, I’m new to Pinot and currently running it in a Docker environment with docker-compose. My setup includes: • 1 Zookeeper • 1 Controller • 1 Broker • 1 Server Over time, I’m seeing strange behavior: • Instances randomly go into a dead state, and new ones start showing up. • Segments become lost/unavailable. • Servers become unavailable, brokers’ resource groups break. • Brokers still try to query the dead servers. • Controllers reference dead brokers. • I can’t delete these dead instances because they still exist in the ideal state for some tables. This makes the cluster unstable and unusable. 👉 My main question: How can I stabilize this single-node Pinot setup under docker-compose? Am I missing some config (e.g. Helix cleanup, persistence, or hostname settings) that would prevent dead instances and segment loss? I’ve attached my

docker-compose.yaml

below for context. Any pointers would be really helpful 🙏 version: '3.7' services: pinot-zookeeper: image: ${ZK_IMAGE zookeeper3.9.2} container_name: "pinot-zookeeper" restart: unless-stopped ports: - "2181:2181" environment: ZOOKEEPER_CLIENT_PORT: 2181 ZOOKEEPER_TICK_TIME: 2000 networks: - aquaduct-pinot-network healthcheck: test: ["CMD", "zkServer.sh", "status"] interval: 30s timeout: 10s retries: 5 start_period: 10s volumes: - zookeeper-data:/data - zookeeper-datalog:/datalog pinot-controller: image: ${PINOT_IMAGE apachepinot/pinot1.2.0} command: "StartController -zkAddress pinot-zookeeper:2181" container_name: "pinot-controller" hostname: pinot-controller restart: unless-stopped ports: - "9000:9000" environment: JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms20G -Xmx40G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-controller.log -Dpinot.controller.instance.auto.cleanup=true -Dpinot.controller.instance.cleanup.interval.seconds=300 -Dhelix.instance.auto.cleanup=true -Dhelix.instance.max.offline.time.before.cleanup=PT10M -Dpinot.hostname=pinot-controller" depends_on: pinot-zookeeper: condition: service_healthy networks: - aquaduct-pinot-network healthcheck: test: ["CMD-SHELL", "curl -f http://localhost:9000/health || exit 1"] interval: 30s timeout: 10s retries: 5 start_period: 10s volumes: - controller-data:/tmp/data/PinotController pinot-broker-1: image: ${PINOT_IMAGE apachepinot/pinot1.2.0} command: "StartBroker -zkAddress pinot-zookeeper:2181" container_name: "pinot-broker-1" hostname: pinot-broker-1 restart: unless-stopped ports: - "8099:8099" environment: JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms45G -Xmx45G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-broker.log -Dpinot.hostname=pinot-broker-1" depends_on: pinot-controller: condition: service_healthy networks: - aquaduct-pinot-network healthcheck: test: ["CMD-SHELL", "curl -f http://localhost:8099/health || exit 1"] interval: 30s timeout: 10s retries: 5 start_period: 10s volumes: - broker-1-data:/var/pinot/broker/data pinot-server-1: image: ${PINOT_IMAGE apachepinot/pinot1.2.0} command: "StartServer -zkAddress pinot-zookeeper:2181" container_name: "pinot-server-1" hostname: pinot-server-1 restart: unless-stopped ports: - "8098:8098" environment: JAVA_OPTS: "-Dplugins.dir=/opt/pinot/plugins -Xms45G -Xmx45G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -Xloggc:gc-pinot-server.log -Dpinot.hostname=pinot-server-1" depends_on: pinot-broker-1: condition: service_healthy networks: - aquaduct-pinot-network healthcheck : test: ["CMD-SHELL", "curl -f http://localhost:8097/health/readiness || exit 1"] interval: 30s timeout: 30s retries: 5 start_period: 30s volumes: - server-1-data:/tmp/data/pinotSegments networks: aquaduct-pinot-network: name: aquaduct-pinot-network driver: bridge volumes: zookeeper-data: zookeeper-datalog: controller-data: broker-1-data: server-1-data:

Any reason you are running on single node? What are the cpu/mem resources available on that node? Also, 45GB is too much for heap. Can you google capacity planning for Apache Pinot, there are two blogs you can read up.

Also try setting

<http://pinot.set.instance.id.to|pinot.set.instance.id.to>.hostname=true

(ref) to avoid the ip based instance names which leds to dead instances and new instance popping up eveytime the container restarts (why are they restarting is answered by Mayan above) and gets new ip. To fix the

I can't delete these dead instances because they still exist in the *ideal state* for some tables

issue you need to untag all dead instances and perform table rebalance so that current live instance gets assigned all the segments, if re-ingestion is easier I would suggest tearing down and recreating the setup to get rid of all dangling instances.

Thank you Mayank and Shounak, Now my instances are running with Hostnames instead of IP's and also my node had only 126 gb of RAM but allocated more than that , Now I am running my cluster using conf files rather than passing java opts , more over the reason I am doing all of this on a single node is because I have set max.rows.in.join to 9000000000, I know that pinot is not meant for running join queries but some of my older filnk jobs are running on some complex joins and if I switch to multinode cluster i am running into mailbox issues, I dont really know but what I inferred was maybe the data for joins was to much for intermidiate stage to pass on using the mailbox, I am shifting my jobs to trino so then I can make it multinode. I just had another doubt for read heavy operations does increasing the number of zookeeper help?