https://pinot.apache.org/ logo
Join Slack
Powered by
Hello team, having some difficulties with the `pin...
# troubleshooting
t
Hello team, having some difficulties with the 
pinot-minion-statless
pod in kubernetes, seems to be getting evicted after starting a Realtime to Offline Segment Job, with the following error: 
The node was low on resource: ephemeral-storage. Container minion-stateless was using 5774600Ki, which exceeds its request of 0.
 I am using S3 as a Deep Store and have the following pinot minion config:
Copy code
pinot.minion.port=9514
dataDir=/var/pinot/minion/data
pinot.set.instance.id.to.hostname=true
pinot.minion.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.minion.storage.factory.s3.region=us-east-1
pinot.minion.segment.fetcher.protocols=file,http,s3
pinot.minion.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
Should my dataDir also be an S3 directory? I do not see any information about this in the S3 Deep Storage Documentation page (only for the config data dir). https://docs.pinot.apache.org/users/tutorials/use-s3-as-deep-store-for-pinot It does not look like the helm chart supplies any volume to the stateless minion, so it doesn’t seem like I am able to just increase the volume size for the pod
x
Can you try to give some fixed resources for the pod? Keep request and limit same for some cpu & memory
The log says the k8s exhausted the node resource then got killed
m
Also, minion doesn’t need S3 or deep-storage to be attached to it.
t
This is the exception that is being generated
Copy code
Caught exception while executing task: Task_RealtimeToOfflineSegmentsTask_a4a57f23-cf3f-4404-9e14-21be3da44d2c_1666375132046_0
java.io.IOException: No space left on device
	at java.io.FileOutputStream.writeBytes(Native Method) ~[?:?]
	at java.io.FileOutputStream.write(FileOutputStream.java:354) ~[?:?]
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1310) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:978) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1282) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.commons.io.IOUtils.copy(IOUtils.java:953) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.common.utils.TarGzCompressionUtils.untarWithRateLimiter(TarGzCompressionUtils.java:202) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.common.utils.TarGzCompressionUtils.untar(TarGzCompressionUtils.java:148) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.common.utils.TarGzCompressionUtils.untar(TarGzCompressionUtils.java:138) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:155) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.plugin.minion.tasks.BaseMultipleSegmentsConversionExecutor.executeTask(BaseMultipleSegmentsConversionExecutor.java:62) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.runInternal(TaskFactoryRegistry.java:113) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.pinot.minion.taskfactory.TaskFactoryRegistry$1.run(TaskFactoryRegistry.java:89) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.helix.task.TaskRunner.run(TaskRunner.java:75) [pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
It seems to me like where ever this pod is storing the data is running out of space, could this be another issue on top of the insufficient requests for mem
I seem to be good for memory now but I am still getting throttled on the storage usage
Copy code
Caused by: java.io.IOException: Cannot delete file: /var/pinot/minion/data/RealtimeToOfflineSegmentsTask
	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1344) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:324) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	... 6 more
Caused by: java.nio.file.NoSuchFileException: /var/pinot/minion/data/RealtimeToOfflineSegmentsTask/tmp-e089562e-7b6b-469b-be90-02bdbd9bf340/tarredSegmentFile_45
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]
	at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:149) ~[?:?]
	at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
	at java.nio.file.Files.readAttributes(Files.java:1764) ~[?:?]
	at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219) ~[?:?]
	at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) ~[?:?]
	at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373) ~[?:?]
	at java.nio.file.Files.walkFileTree(Files.java:2761) ~[?:?]
	at java.nio.file.Files.walkFileTree(Files.java:2797) ~[?:?]
	at org.apache.commons.io.file.PathUtils.visitFileTree(PathUtils.java:971) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.commons.io.file.PathUtils.deleteDirectory(PathUtils.java:434) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.commons.io.file.PathUtils.delete(PathUtils.java:391) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.commons.io.FileUtils.forceDelete(FileUtils.java:1341) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	at org.apache.commons.io.FileUtils.cleanDirectory(FileUtils.java:324) ~[pinot-all-0.11.0-jar-with-dependencies.jar:0.11.0-1b4d6b6b0a27422c1552ea1a936ad145056f7033]
	... 6 more
Pinot minion stopped
Pinot [MINION] Instance [Minion_pinot-minion-stateless-57fc575b5b-p92sf_9514] is Stopped...
Shutting down Pinot Service Manager admin application...
Deregistering service status handler
does the realtime to offline segment task attempt to download all the real time segments pending? If so, I have over 300gb of realtime segments and my nodes only have about 10gb/per node of free ephemeral storage currently.
x
oh, so if you have a lot of data, then yes, you need to mount a bigger enough disk volume for that
You can try the minion-statefulset
that will be simpler to mount a big enough disk
t
are there docs on configuring the statefulset? I only see stateless in the helm chart
x
you can enable this stateful minion 😛 and disable the stateless minions
🌟 1
Open in Slack
PreviousNext
Powered by