https://pinot.apache.org/ logo
Join Slack
Powered by
Hey folks, excited to try out Pinot. I’m having is...
# troubleshooting
a
Hey folks, excited to try out Pinot. I’m having issues understanding how I can ingest data (specifically parquet data). Trying out a POC where I’m bringing up a vanilla pinot cluster up (based on these docs). I’m trying to ingest a single parquet file using the ingestFromFile API.
The CURL command that I’m using is
Copy code
curl -X POST -F file=@dt=2021-06-28/LATEST.parquet \
  -H "Content-Type: multipart/form-data" \
  "<http://localhost:9000/ingestFromFile?tableNameWithType=table_name_OFFLINE&batchConfigMapStr=%7B%22dataFormat%22%3A%22parquet%22%7D>."
I’m noticing the following null pointer exception.
Copy code
Caught exception when ingesting file into table: table_name_OFFLINE. null
java.lang.NullPointerException: null
	at org.apache.pinot.controller.util.FileIngestionHelper.buildSegmentAndPush(FileIngestionHelper.java:109) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
	at org.apache.pinot.controller.api.resources.PinotIngestionRestletResource.ingestData(PinotIngestionRestletResource.java:200) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
	at org.apache.pinot.controller.api.resources.PinotIngestionRestletResource.ingestFromFile(PinotIngestionRestletResource.java:129) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.9.3-jar-with-dependencies.jar:0.9.3-e23f213cf0d16b1e9e086174d734a4db868542cb]
Cross checked what folks out there are doing. Here’s an example that someone had written around batch ingestion of data locally (but in JSON format). Tried switching out the config for parquet instead of JSON - but still seeing some errors around fetching table schema in JSON..
Copy code
java.lang.RuntimeException: Failed to decode Pinot schema from json string
Looks like the job is expecting JSON back, but it’s getting HTML from the controller.
k
can you share the schema and table config
a
So, I was able to unblock myself from the above issue by consulting the swagger UI. Currently the issue that I seem to be running into is that the segment generation is failing.
Copy code
java.lang.RuntimeException: Caught exception during running - org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner
This stems from the error below.
Copy code
Caused by: java.io.IOException: Failed to copy full contents from /input/path to /output/path
May be this is a permissions issue?
Huh
Copy code
java.lang.IllegalStateException: Input path {} does not exist. [/var/folders/mz/frzjrqzx0gb8g6hn2pzqs4k80000gn/T/pinot-4ac6efdb-23a4-4ec3-a920-ae97a66e419c/input/LATEST.parquet]
m
Does the path exist?
a
The part till the parquet file exists. The file itself doesn’t. May be it would be useful for me to summarize the input formats.. Have a folder with bunch of sub folders inside. Each folder then has a file called LATEST.parquet..
I tried changing the permissions of the folder to try and see if that might be the issue. I dont seem to be able to do that for some reason. May be the error above is related to the fact that the individual files in the input are all of the same name?
@Xiang Fu could the issue that you are discussing in this thread also be affecting me?
Hmm, so it does seem to be connected to the fact that all of the files are named LATEST.parquet 🤷‍♂️
I imported just one directory and got a different error - relating to the way I had setup my schema.
x
Not that issue, that was the spark-submit issue or you want to run a standalone ingestion job
a
I’m using the 
LaunchDataIngestionJob
script. Doesn’t look like the errors are related. This issue is because the name of the individual files is the same.
x
Right, so how your input data directory look like?
a
My hypothesis about the inputs being the issue was true. I’ve a bunch of folders under the root folder that I’m trying to import. Each folder has 1 file in it - called LATEST.parquet. I renamed them all to something different - and then I was able to ingest all of the files at one go.
m
Interesting, the segment name is picked based on table name and time column in the ingestion job, can you file a GH issue?
x
If would be helpful if you can share the ingestionJobSpec file as well. I suspect there are some input data file copy which causes the file override issue
Open in Slack
PreviousNext
Powered by