yes thats how it should be right?

if so, then you can change pvc to pv and mount it, but need to ensure that your pv is allowed of RWX(READ WRITE Many) mode

yes rwx is allowed, but is mounting same pv to all replicas possible through statefulset?

yes

just change mounting from pvc to pv

and ensure that you pv is already created

typically NFS is RWX

another thing for pinot controller is that Pinot support s3/azure datalake/gcs/hdfs already

so technically if you segments are there, you can configure and use that

ya i wanted the datalake sample configs but they r not yet updated on docs

here

thanks a lot! I was looking at https://docs.pinot.apache.org/plugins/pinot-file-system 😅

Getting this error during segment completion while trying 0.4.0 release (works fine with 0.3.0)

2020/06/15 10:37:08.770 ERROR [LLRealtimeSegmentDataManager_jeepusermap__2__0__20200615T0848Z] [jeepusermap__2__0__20200615T0848Z] Could not build segment

<http://java.io|java.io>.IOException: Unable to delete directory /var/pinot/server/data/index/jeepusermap_REALTIME/_tmp/tmp-jeepusermap__2__0__20200615T0848Z-1592217321638/tmp-1592217323364--1219958100746923301.

at <http://org.apache.commons.io|org.apache.commons.io>.FileUtils.deleteDirectory(FileUtils.java:1541) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-c152f18f273042668a6d91d5d15c7ca018f31b4a]

at <http://org.apache.commons.io|org.apache.commons.io>.FileUtils.moveDirectory(FileUtils.java:2756) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-c152f18f273042668a6d91d5d15c7ca018f31b4a]

at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.handlePostCreation(SegmentIndexCreationDriverImpl.java:230) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-c152f18f273042668a6d91d5d15c7ca018f31b4a]

at org.apache.pinot.core.segment.creator.impl.SegmentIndexCreationDriverImpl.build(SegmentIndexCreationDriverImpl.java:199) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-c152f18f273042668a6d91d5d15c7ca018f31b4a]

at org.apache.pinot.core.realtime.converter.RealtimeSegmentConverter.build(RealtimeSegmentConverter.java:141) ~[pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-c152f18f273042668a6d91d5d15c7ca018f31b4a]

at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentInternal(LLRealtimeSegmentDataManager.java:720) [pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-c152f18f273042668a6d91d5d15c7ca018f31b4a]

at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.buildSegmentForCommit(LLRealtimeSegmentDataManager.java:671) [pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-c152f18f273042668a6d91d5d15c7ca018f31b4a]

at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:588) [pinot-all-0.4.0-SNAPSHOT-jar-with-dependencies.jar:0.4.0-SNAPSHOT-c152f18f273042668a6d91d5d15c7ca018f31b4a]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]

@Shounak Kulkarni

/var/pinot/server/data/index/jeepusermap_REALTIME/_tmp/tmp-jeepusermap__2__0__20200615T0848Z-1592217321638/tmp-1592217323364--1219958100746923301

Does that directory exist?

and any permission issue

Yes directory exists. And if v go to azure portal, and delete it is able to delete the directory.

We noticed that the kubernetes chart for the server has a readiness probe. The server goes into a crash loop because it cannot receive traffic: localhost:8097/health - doesn't even work when running curl on the pod. When the readiness probe is removed the server starts up and the curl works.

Is the readiness probe necessary? I was able to reproduce this behavior 2x

(in staging of course 🙂 )

What is this with latest version?

@Laxman Ch also faced the same issue

With 0.3.0

We will be upgrading in a few days

Hmm 0.3.0 worked fine

It works fine without the readiness probe

Strange, if readiness probe is enabled even local traffic is not permitted, i.e. curl localhost:8097/health fails

hmm, so this endpoint is not working even if you execute

curl localhost:8097/health

inside a pinot server container?

Yep, but when I remove the readiness probe (and keep the liveness probe) it works