Hi all, I have an offline table and trying to inge...
# troubleshooting
y
Hi all, I have an offline table and trying to ingest some parquet files via spark. I am trying to configure
SegmentCreationAndUriPush
because the spark runtime does not have access to Pinot’s deep store location. I need this job to let Pinot controller know that there is a new segment in S3 bucket, and let it pull from S3. Pinot controller has access to S3 location. But this is the error that I see;
Copy code
Got error status code: 500 (Internal Server Error) with reason: "Could not find paramName tableName in path or query params of the API
Is Uri push job works at all for anyone?
The back story of this question is, we were using
SegmentCreationAndMetadataPush
, it was working with smaller segments but when segments become larger than 5 MB, it started to become an issue. Simply we started to see following in the logs
Copy code
Got error status code: 413 (Request Entity Too Large)
Then I start to investigate what are other options, and it looks like UriPush is the other recommended way for production deployments.
x
you can try uri push. It requires controller to download segments then load metadata. We are using segment metadata push internally so it should work. Do you have more stack trace?
n
@Xiang Fu I am also facing similar issue when tried with
SegmentCreationAndUriPush
Please find the error below and help me resolve this
Copy code
25/07/04 10:27:10 INFO HttpClient: Sending request: https://prod-dp-pinot-controller.xyz.in/v2/segments to controller: pinot-controller-2.pinot-controller-headless.dataplatform.svc.cluster.local, version: Unknown
25/07/04 10:27:10 WARN SegmentPushUtils: Caught temporary exception while pushing table: poc_ems_attributes segment uri: gs://bucket-name/pinot-segments/poc_ems_attributes/poc_ems_attributes_OFFLINE_metadata_push_poc_0.tar.gz to https://prod-dp-pinot-controller.xyz.in, will retry
org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Could not find paramName tableName in path or query params of the API: http://prod-dp-pinot-controller.xyz.in:80/v2/segments" while sending request: https://prod-dp-pinot-controller.xyz.in/v2/segments to controller: pinot-controller-2.pinot-controller-headless.dataplatform.svc.cluster.local, version: Unknown
	at org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:448) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.common.utils.FileUploadDownloadClient.sendSegmentUri(FileUploadDownloadClient.java:981) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.segment.local.utils.SegmentPushUtils.lambda$sendSegmentUris$1(SegmentPushUtils.java:234) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.segment.local.utils.SegmentPushUtils.sendSegmentUris(SegmentPushUtils.java:231) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.segment.local.utils.SegmentPushUtils.sendSegmentUris(SegmentPushUtils.java:115) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentUriPushJobRunner.run(SparkSegmentUriPushJobRunner.java:109) [pinot-batch-ingestion-spark-3.2-1.0.0-shaded.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:150) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:126) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:130) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.main(LaunchDataIngestionJobCommand.java:162) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
	at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:738) [spark-yarn_2.12-3.5.0.jar:3.5.0]
x
which pinot-server version are you using? your client is pinot 1.0.0, which might be old
n
yeah, we are using 1.0.0
x
what's your pinot controller/server versions, those versions are likely higher than your client version.
n
No all of them are v1.0.0
x
ok, then can you share the controller logs when you are pushing the segments