:wave: <@UDQU92KBK> Just set this up <https://docs...
# troubleshooting
j
👋 @Mayank Just set this up https://docs.pinot.apache.org/operators/operating-pinot/decoupling-controller-from-the-data-path for realtime usecase. What config controls how often segments backup to S3?
m
You can check this doc out. It talks about segment flush threshold. When the threshold meets, that's when the segment is committed to S3: https://dev.startree.ai/docs/pinot/recipes/configuring-segment-threshold
j
Thanks @Mayank! What is the best way to test the S3 sync? Things I have tried 1. Verified I can upload to S3 using
aws s3 cp
from the realtime server 2. Tried executing this command
POST /tables/{tableName}/forceCommit Force commit the current consuming segments
m
If 2 worked, that is good
j
2 did not work
What's confusing is there is nothing in the logs
I think the issue is that API command uses the controller rather than servers
m
There will always be something in the logs
Which API command are you referring to? This feature is used almost all the time, so if it is not working it likely points to setup issue.
j
OK that makes sense. So you are correct. I am actually able to see successful upload in the logs
Copy code
Copy /mnt/pinot/server/index/spark_metrics_REALTIME/spark_metrics__0__6__20250321T1957Z.tar.gz from local to dev/pinot/realtime-data/spark_metrics/spark_metrics__0__6__20250321T1957Z08eccf4d-bd8e-41aa-b302-9bfaa4c41759
However when I look in S3, nothing is there
Screenshot 2025-03-21 at 4.04.04 PM.png
m
In that cases there was an AWS credential problem. Was that the problem here too?
j
There is no credential error
Copy code
cat /mnt/pinot/controller.conf
controller.data.dir=s3://dev/pinot/controller-data
controller.local.temp.dir=/tmp/pinot/controller-data/
controller.zk.str=zk.com:2181
pinot.set.instance.id.to.hostname=true
controller.port=9000
controller.helix.cluster.name=pinot-ec2-dev
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=us-east-1

pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher

controller.retention.frequencyPeriod=6h
controller.retentionManager.initialDelayInSeconds=300
controller.deleted.segments.retentionInDays=7

controller.task.scheduler.enabled=true
controller.task.frequencyPeriod=6h

controller.allow.hlc.tables=false
controller.enable.split.commit=true
controller.realtime.segment.deepStoreUploadRetryEnabled=true
Copy code
cat /mnt/pinot/realtime-server.conf
pinot.server.netty.port=8098
pinot.server.adminapi.port=8097
pinot.server.instance.dataDir=/mnt/pinot/server/index
pinot.server.instance.segmentTarDir=/mnt/pinot/server/segmentTars
server.helix.cluster.name=pinot-ec2-dev
server.zk.str=zk.com:2181
pinot.set.instance.id.to.hostname=true

pinot.server.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.server.storage.factory.s3.region=us-east-1
pinot.server.segment.fetcher.protocols=file,http,s3
pinot.server.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
pinot.server.query.executor.num.groups.limit=5000000
pinot.server.query.executor.min.server.group.trim.size=-1
pinot.server.query.executor.groupby.trim.threshold=5000000

pinot.server.instance.segment.store.uri=s3://dev/pinot/realtime-data
It looks like the data is going to
s3://dev/pinot/controller-data
so I am going to delete that line from the config
That raises null pointer exception
Copy code
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:602) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:396) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:344) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:118) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:87) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:252) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:305) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:251) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:195) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:188) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]: Caused by: org.apache.pinot.controller.api.resources.InvalidControllerConfigException: Caught exception while initializing file upload path provider
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:107) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:600) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         ... 20 more
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]: Caused by: java.lang.NullPointerException
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at java.net.URI$Parser.parse(URI.java:3104) ~[?:?]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at java.net.URI.<init>(URI.java:600) ~[?:?]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.common.utils.URIUtils.getUri(URIUtils.java:41) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:67) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         at org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:600) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
Mar 21 15:41:50 pinot-controller-internal-ec2-dev-iad-csxi start-controller.sh[939983]:         ... 20 more