Hi, I'm testing writing to a Pinot offline table u...
# troubleshooting
u
Hi, I'm testing writing to a Pinot offline table using the Spark connector. When I run my code like this:
Copy code
df.write()
.format("pinot")
.option("controller", "localhost:9000")
.option("table", "transcript")
.mode(SaveMode.Append)
.save("/tmp/pinot-segments")
The segment files are created correctly in the
/tmp/pinot-segments
directory, but they are not uploaded to the actual Pinot cluster. Does the Spark Pinot connector not support writing directly to Pinot? It seems like it only creates the files locally. Thanks.
m
@balci could you help with this?
you need to call commit to push
b
Hi @전이섭, Spark connectors write functionality only writes Pinot segments to a desired directory, but doesn’t ‘push’ them to Pinot. At Uber, we use it in conjunction with a simple minion task which automatically ‘pulls in’ the generated segments. Unfortunately that is not open source but it should be relatively simple to scaffold one if you want such ‘pull based’ ingestion. Alternatively, you can use controller API’s inside your spark application to push it final output. Hope this helps