Hey Team,
We are using Pinot in prod for more tha...
# troubleshootingr
raghav
10/10/2025, 2:09 PMHey Team,
We are using Pinot in prod for more than 6 moths now. Suddenly we started facing issues when ingestion drops suddenly and recovers after some time. I checked the logs thoroughly and were able to find two errors in server. Controller logs looks fine.
We have 24 servers, 36 kafka partitions, 50 GB memory each, peak ingestion rate 1MM rps, segment size - 300MB.
Can anyone please help us understand this and mitigate this issue?
ERROR/WARN logs
Copy code
pinotServer.2025-10-10.9.log.gz:2025/10/10 13:22:34.067 ERROR [RealtimeSegmentDataManager_metric_numerical_agg_1H__16__182629__20251010T1321Z] [metric_numerical_agg_1H__16__182629__20251010T1321Z] Holding after response from Controller: {"buildTimeSec":-1,"isSplitCommitType":true,"streamPartitionMsgOffset":null,"status":"NOT_SENT"}
pinotServer.2025-10-10.9.log.gz:2025/10/10 13:22:52.653 ERROR [ServerSegmentCompletionProtocolHandler] [metric_numerical_agg_1H__28__180921__20251010T1322Z] Could not send request <http://pinot-controller-0.pinot-controller-headless.d3-pinot-cluster.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=172544503662&instance=Server_pinot-server-1.pinot-server-headless.d3-pinot-cluster.svc.cluster.local_8098&name=metric_numerical_agg_1H__28__180921__20251010T1322Z&rowCount=696146&memoryUsedBytes=338498296>
2025/10/10 13:14:41.871 WARN [AppInfoParser] [HelixTaskExecutor-message_handle_thread_5] Error registering AppInfo mbean
javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=metric_numerical_agg_1H_REALTIME-D3NumericalSketchPartitioned-28
at java.management/com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:322)raghav
10/10/2025, 5:41 PMController error logs:
Copy code
2025/10/10 17:17:54.775 ERROR [BlockingSegmentCompletionFSM_metric_numerical_agg_1H__21__182593__20251010T1717Z] [grizzly-http-server-7] Caught exception while committing segment metadata for segment: metric_numerical_agg_1H__21__182593__20251010T1717Z
java.lang.IllegalStateException: Failed to persist segment ZK metadata for segment: metric_numerical_agg_1H__21__182593__20251010T1717Z of table: metric_numerical_agg_1H_REALTIME
at org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:836)
at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.persistSegmentZKMetadata(PinotLLCRealtimeSegmentManager.java:479)
at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.updateSegmentZKMetadataToDone(PinotLLCRealtimeSegmentManager.java:754)
at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadataInternal(PinotLLCRealtimeSegmentManager.java:656)
at org.apache.pinot.controller.helix.core.realtime.PinotLLCRealtimeSegmentManager.commitSegmentMetadata(PinotLLCRealtimeSegmentManager.java:627)
at org.apache.pinot.controller.helix.core.realtime.BlockingSegmentCompletionFSM.commitSegmentMetadata(BlockingSegmentCompletionFSM.java:846)
at org.apache.pinot.controller.helix.core.realtime.BlockingSegmentCompletionFSM.commitSegment(BlockingSegmentCompletionFSM.java:831)
at org.apache.pinot.controller.helix.core.realtime.BlockingSegmentCompletionFSM.segmentCommitEnd(BlockingSegmentCompletionFSM.java:390)
at org.apache.pinot.controller.helix.core.realtime.SegmentCompletionManager.segmentCommitEnd(SegmentCompletionManager.java:322)
at org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlers.segmentCommitEndWithMetadata(LLCSegmentCompletionHandlers.java:303)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356)
at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:190)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515)
at java.base/java.lang.Thread.run(Thread.java:1583)