https://pinot.apache.org/ logo
Join Slack
Powered by
Hello, our pinot controller pod is reaching maximu...
# general
h
Hello, our pinot controller pod is reaching maximum volume(as shown in the chart below). The space got filled up quickly even after we doubled the pvc size. The are two main error in controller pod:
Copy code
1, 2024/05/22 14:17:06.780 ERROR [CompletionServiceHelper] [grizzly-http-server-1] Server: Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098 returned error: 404
Copy code
2, 2024/05/22 14:17:09.749 ERROR [LLCSegmentCompletionHandlers] [grizzly-http-server-1] Caught exception while uploading segment: bridge_d1e034ddd9404fde9a23ce3dc43f2b05__0__382__20240522T1202Z from instance: Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098
java.nio.file.FileSystemException: /var/pinot/controller/data/pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local_9000/fileUploadTemp/bridge_d1e034ddd9404fde9a23ce3dc43f2b05__0__382__20240522T1202Z.f51be2da-a3d3-48bf-83d3-9947b63942d9 -> /var/pinot/controller/data/bridge_d1e034ddd9404fde9a23ce3dc43f2b05/bridge_d1e034ddd9404fde9a23ce3dc43f2b05__0__382__20240522T1202Z.tmp.ffeea1e0-b0f2-481f-9d95-80258b022afc: No space left on device
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
	at sun.nio.fs.UnixCopyFile.copyFile(UnixCopyFile.java:258) ~[?:?]
	at sun.nio.fs.UnixCopyFile.copy(UnixCopyFile.java:603) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.copy(UnixFileSystemProvider.java:258) ~[?:?]
	at java.nio.file.Files.copy(Files.java:1295) ~[?:?]
	at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:850) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:756) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.spi.filesystem.LocalPinotFS.copy(LocalPinotFS.java:208) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.spi.filesystem.LocalPinotFS.copyFromLocalFile(LocalPinotFS.java:146) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.spi.filesystem.NoClosePinotFS.copyFromLocalFile(NoClosePinotFS.java:110) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.apache.pinot.controller.api.resources.LLCSegmentCompletionHandlers.segmentUpload(LLCSegmentCompletionHandlers.java:372) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at jdk.internal.reflect.GeneratedMethodAccessor168.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
	at java.lang.Thread.run(Thread.java:829) [?:?]
For the server pod, there's no error, but there is a repeating failed message
Copy code
2024/05/22 14:12:45.355 INFO [LLRealtimeSegmentDataManager_AppbuilderEvent__0__371__20240522T1202Z] [bridge_d1e034ddd9404fde9a23ce3dc43f2b05__0__382__20240522T1202Z] Controller response {"offset":-1,"isSplitCommitType":false,"status":"FAILED","streamPartitionMsgOffset":null,"buildTimeSec":-1} for <http://pinot-controller-0.pinot-controller-headless.pinot-quickstart.svc.cluster.local:9000/segmentUpload?segmentSizeBytes=542135590&buildTimeMillis=14999&streamPartitionMsgOffset=429404709&instance=Server_pinot-server-0.pinot-server-headless.pinot-quickstart.svc.cluster.local_8098&offset=-1&name=bridge_d1e034ddd9404fde9a23ce3dc43f2b05__0__382__20240522T1202Z&rowCount=10000&memoryUsedBytes=75812005>
x
Do you have deep store configured? If not then all the segments are pushed to controller, which might blow the disk. If so you can use s3 for deep store then controller disk pressure should get resolved
h
@Xiang Fu added configs based on Use S3 as Deep Storage for Pinot , but controller pod got an error. Any idea on how to fix it? controller pod log:
Copy code
2024/05/29 14:08:39.399 INFO [ZooKeeper] [main] Session: 0x100acc8d60e0124 closed
2024/05/29 14:08:39.399 INFO [BaseControllerStarter] [main] Initializing PinotFSFactory
2024/05/29 14:08:39.400 INFO [ClientCnxn] [main-EventThread] EventThread shut down for session: 0x100acc8d60e0124
2024/05/29 14:08:39.403 INFO [PinotFSFactory] [main] Did not find any fs classes in the configuration
2024/05/29 14:08:39.404 INFO [PinotFSFactory] [main] Got scheme s3, initializing class org.apache.pinot.plugin.filesystem.S3PinotFS
2024/05/29 14:08:39.404 INFO [PinotFSFactory] [main] Initializing PinotFS for scheme s3, classname org.apache.pinot.plugin.filesystem.S3PinotFS
2024/05/29 14:08:40.072 INFO [BaseControllerStarter] [main] Initializing ControllerFilePathProvider
2024/05/29 14:08:40.075 INFO [ControllerFilePathProvider] [main] Data directory: <s3://pinot-ds-dev/deeps-tore/>
2024/05/29 14:08:40.594 ERROR [StartServiceManagerCommand] [main] Failed to start a Pinot [CONTROLLER] at 6.859 since launch
java.lang.RuntimeException: Caught exception while initializing ControllerFilePathProvider
        at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:569) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:380) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:328) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:118) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:87) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:251) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:250) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:196) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:187) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202) [pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
Caused by: org.apache.pinot.controller.api.resources.InvalidControllerConfigException: Caught exception while initializing file upload path provider
        at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:107) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:567) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        ... 20 more
Caused by: java.lang.IllegalStateException: Local temporary directory is not configured, cannot use remote data directory: <s3://pinot-ds-dev/deeps-tore/> as local temporary directory
        at org.apache.pinot.shaded.com.google.common.base.Preconditions.checkState(Preconditions.java:518) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:81) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:567) ~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
        ... 20 more
2024/05/29 14:08:40.604 INFO [PinotServiceManager] [Thread-42] Shutting down Pinot Service Manager with all running Pinot instances...
2024/05/29 14:08:40.604 INFO [PinotServiceManager] [Thread-42] Shutting down Pinot Service Manager admin application...
2024/05/29 14:08:40.605 INFO [PinotServiceManager] [Thread-42] Deregistering service status handler
configmap pinot-controller-config:
Copy code
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  pinot-controller.conf: |-
    controller.helix.cluster.name=pinot-quickstart
    controller.port=9000
    controller.data.dir=<s3://pinot-ds-dev/deeps-tore/>
    controller.zk.str=pinot-zookeeper:2181
    pinot.set.instance.id.to.hostname=true
    controller.task.scheduler.enabled=true

    pinot.controller.storage.factory.s3.accessKey=**************
    pinot.controller.storage.factory.s3.secretKey=**************
    pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
    pinot.controller.storage.factory.s3.region=us-east-1
    pinot.controller.segment.fetcher.protocols=file,http,s3
    pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
kind: ConfigMap
metadata:
  annotations:
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: pinot
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: pinot
  creationTimestamp: "2024-03-07T18:05:27Z"
  labels:
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
  name: pinot-controller-config
  namespace: pinot
  resourceVersion: "173789930"
  uid: 6cc01c18-4d4b-437f-8162-8084ed84f062
x
Local temporary directory is not configured, cannot use remote data directory: <s3://pinot-ds-dev/deeps-tore/> as local temporary
Copy code
controller.data.dir=<s3://my.bucket/pinot-data/pinot-s3-example/controller-data>
controller.local.temp.dir=/tmp/pinot-tmp-data/
controller.zk.str=localhost:2181
controller.host=127.0.0.1
controller.port=9000
controller.helix.cluster.name=pinot-s3-example
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=us-west-2

pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
missing this config : 
controller.local.temp.dir=/tmp/pinot-tmp-data/
🍷 1
h
Thank you @Xiang Fu , we have successfully configured configmaps for the controller and the server. We are now working on migrating segments inside server pod into S3. The copy process is easy, but the segment formats are different in server pod(folder) and S3 bucket(file). Even old segments are in S3 now, this format mismatch causes bad segments inside pinot. Is there any method/tool to do the segment migration from server pod to S3?
x
segments on s3 are targz’ed file for the server segment directory
Open in Slack
PreviousNext
Powered by