Also is there an easy way to just sink a Spark dat...
# generaln
Noah Prince
11/05/2020, 4:52 PMAlso is there an easy way to just sink a Spark dataframe to pinot segments?
Noah Prince
11/05/2020, 5:03 PMLooks like I’d just need to manually use SegmentIndexCreationDriverImpl with a foreachpartition or something. Then manually do the copying that
SparkSegmentGenerationJobRunnerNoah Prince
11/05/2020, 5:03 PMWould be useful to have this feature since you can read from whatever custom spark readers you have, repartition as necessary, etc.
j
Jackie
11/05/2020, 6:28 PM@Xiang Fu Thoughts?
x
Xiang Fu
11/05/2020, 6:29 PM
This is a useful feature
Xiang Fu
11/05/2020, 6:29 PM
I can help review the PR
Xiang Fu
11/05/2020, 6:30 PM
and let me know if you have any question on the segment gen task
n
Noah Prince
11/12/2020, 8:30 PMhttps://github.com/apache/incubator-pinot/pull/5787/files
looks like someone already did this 😄
Noah Prince
11/12/2020, 8:31 PMAh dang it’s read only. That sucks.
x
Xiang Fu
11/12/2020, 10:49 PM
yes, it’s reading from Pinot to spark, the sink is not yet there