Yusuf Külah
03/26/2025, 1:28 PMSegmentCreationAndUriPush because the spark runtime does not have access to Pinot’s deep store location. I need this job to let Pinot controller know that there is a new segment in S3 bucket, and let it pull from S3. Pinot controller has access to S3 location.
But this is the error that I see;
Got error status code: 500 (Internal Server Error) with reason: "Could not find paramName tableName in path or query params of the API
Is Uri push job works at all for anyone?Yusuf Külah
03/26/2025, 1:37 PMSegmentCreationAndMetadataPush , it was working with smaller segments but when segments become larger than 5 MB, it started to become an issue. Simply we started to see following in the logs
Got error status code: 413 (Request Entity Too Large)
Then I start to investigate what are other options, and it looks like UriPush is the other recommended way for production deployments.Xiang Fu
Nithish
07/04/2025, 10:36 AMSegmentCreationAndUriPush
Please find the error below and help me resolve this
25/07/04 10:27:10 INFO HttpClient: Sending request: https://prod-dp-pinot-controller.xyz.in/v2/segments to controller: pinot-controller-2.pinot-controller-headless.dataplatform.svc.cluster.local, version: Unknown
25/07/04 10:27:10 WARN SegmentPushUtils: Caught temporary exception while pushing table: poc_ems_attributes segment uri: gs://bucket-name/pinot-segments/poc_ems_attributes/poc_ems_attributes_OFFLINE_metadata_push_poc_0.tar.gz to https://prod-dp-pinot-controller.xyz.in, will retry
org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 500 (Internal Server Error) with reason: "Could not find paramName tableName in path or query params of the API: http://prod-dp-pinot-controller.xyz.in:80/v2/segments" while sending request: https://prod-dp-pinot-controller.xyz.in/v2/segments to controller: pinot-controller-2.pinot-controller-headless.dataplatform.svc.cluster.local, version: Unknown
at org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:448) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.pinot.common.utils.FileUploadDownloadClient.sendSegmentUri(FileUploadDownloadClient.java:981) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.pinot.segment.local.utils.SegmentPushUtils.lambda$sendSegmentUris$1(SegmentPushUtils.java:234) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.pinot.spi.utils.retry.BaseRetryPolicy.attempt(BaseRetryPolicy.java:50) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.pinot.segment.local.utils.SegmentPushUtils.sendSegmentUris(SegmentPushUtils.java:231) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.pinot.segment.local.utils.SegmentPushUtils.sendSegmentUris(SegmentPushUtils.java:115) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.pinot.plugin.ingestion.batch.spark3.SparkSegmentUriPushJobRunner.run(SparkSegmentUriPushJobRunner.java:109) [pinot-batch-ingestion-spark-3.2-1.0.0-shaded.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.kickoffIngestionJob(IngestionJobLauncher.java:150) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.pinot.spi.ingestion.batch.IngestionJobLauncher.runIngestionJob(IngestionJobLauncher.java:126) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.execute(LaunchDataIngestionJobCommand.java:130) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.pinot.tools.Command.call(Command.java:33) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.pinot.tools.Command.call(Command.java:29) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at picocli.CommandLine.executeUserObject(CommandLine.java:1953) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at picocli.CommandLine.access$1300(CommandLine.java:145) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at picocli.CommandLine.execute(CommandLine.java:2078) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at org.apache.pinot.tools.admin.command.LaunchDataIngestionJobCommand.main(LaunchDataIngestionJobCommand.java:162) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:738) [spark-yarn_2.12-3.5.0.jar:3.5.0]Xiang Fu
Nithish
07/06/2025, 6:15 PMXiang Fu
Nithish
07/07/2025, 3:24 AMXiang Fu