https://pinot.apache.org/ logo
Join Slack
Powered by
We’re currently on Pinot 0.6 and we’re getting a t...
# troubleshooting
j
We’re currently on Pinot 0.6 and we’re getting a ton of warnings when ingesting a new PInot Upsert table. I haven’t seen this particular error before. Is there something I should be looking for?
Copy code
2021/08/18 00:35:45.093 WARN [LLRealtimeSegmentDataManager_enriched_station_orders_v1_14_rt_upsert_v2_1__7__213__20210818T0025Z] [enriched_customer_orders_v1_15_1_upsert__6__26__20210817T0050Z] Commit failed with response {"offset":-1,"streamPartitionMsgOffset":null,"status":"FAILED","isSplitCommitType":false,"buildTimeSec":-1}
2021/08/18 00:35:46.258 WARN [LLRealtimeSegmentDataManager_enriched_station_orders_v1_14_rt_upsert_v2_1__7__213__20210818T0025Z] [enriched_station_orders_v1_14_rt_upsert_v2_1__0__219__20210816T2149Z] Commit failed with response {"offset":-1,"streamPartitionMsgOffset":null,"status":"FAILED","isSplitCommitType":false,"buildTimeSec":-1}
2021/08/18 00:35:46.311 WARN [LLRealtimeSegmentDataManager_enriched_station_orders_v1_14_rt_upsert_v2_1__7__213__20210818T0025Z] [enriched_station_orders_v1_14_rt_upsert_v2_1__10__218__20210817T0010Z] Commit failed with response {"offset":-1,"streamPartitionMsgOffset":null,"status":"FAILED","isSplitCommitType":false,"buildTimeSec":-1}
enriched_station_orders_v1_15_1_upsert is a new table, but it seems to be impacting older tables as well.
s
can u look for the segment name in the controller logs?
j
Hrm….
Copy code
2021/08/18 00:53:29.740 ERROR [LLCSegmentCompletionHandlers] [grizzly-http-server-4] Caught exception while committing segment: enriched_customer_orders_v1_15_1_upsert__5__16__20210817T0128Z from instance: Server_pinot-upsert-server-zonal-2.pinot-upsert-server-headless.pinot-dev.svc.cluster.local_8098
java.io.IOException: No space left on device
	at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[?:?]
	at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62) ~[?:?]
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113) ~[?:?]
	at sun.nio.ch.IOUtil.write(IOUtil.java:79) ~[?:?]
	at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:280) ~[?:?]
	at java.nio.channels.Channels.writeFullyImpl(Channels.java:74) ~[?:?]
	at java.nio.channels.Channels.writeFully(Channels.java:97) ~[?:?]
	at java.nio.channels.Channels$1.write(Channels.java:172) ~[?:?]
	at java.io.InputStream.transferTo(InputStream.java:705) ~[?:?]
	at java.nio.file.Files.copy(Files.java:3078) ~[?:?]
	at org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlers.extractSegmentFromFormToLocalTempFile(LLCSegmentCompletionHandlers.java:445) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlers.segmentCommit(LLCSegmentCompletionHandlers.java:289) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at jdk.internal.reflect.GeneratedMethodAccessor330.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:679) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:353) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) ~[pinot-all-0.6.0-jar-with-dependencies.jar:0.6.0-8f7da20c24e77e292cdf89cef7f1aa653c9a35f3]
	at java.lang.Thread.run(Thread.java:834) [?:?]
is this out of disk space on the controller or on a server?
s
out of space on the controller. As segments complete, they are deposited on the controller. I think you may be running out of temporary space, or (depending on how you have your segment store configured) segment storage space
I think it is local temporary storage space
on the controller.
j
okay, thanks for the pointer, I’ll dig in more there.
got any advice on scaling a cluster for Pinot Upsert vs Offline tables? Is there more reliance on the controllers for Upsert?
Open in Slack
PreviousNext
Powered by