Hi All, we're running ingestion and Push Job Spec (doc) is configured like

pushJobSpec:
  pushParallelism: 20
  pushAttempts: 2
  segmentUriPrefix: "<s3://bucket-foo>"
  segmentUriSuffix : ""

And got this error message:

2022/05/11 142850.531 ERROR [BaseTableDataManager] [HelixTaskExecutor-message_handle_thread] Attempts exceeded when downloading segment: foo_OFFLINE_2022-05-03_2022-05-03_11 for table: foo_OFFLINE from: s3://bucket-foo/data/output/foo_OFFLINE_2022-05-03_2022-05-03_11.tar.gznull to: /tmp/pinot-tmp/server/index/foo_OFFLINE/tmp/tmp-foo_OFFLINE_2022-05-03_2022-05-03_11-b2f3a97c-9c14-4b4c-9874-fb028597a237/foo_OFFLINE_2022-05-03_2022-05-03_11.tar.gz ava:72) [pinot-all-0.10.0-SNAPSHOT-jar-with-dependencies.jar:0.10.0-SNAPSHOT-f

It adds 'null' at end of file's URI. Any idea how to resolve? Thanks in advance.

based on the exception msg, the wrong uri is actually from SegmentZKMetadata.getDownloadUrl(), so the wrong URI was set there when pushing the segment to Pinot. looks like if either prefix or suffix is empty, those three parts are simply stitched together (code, but still I’d assume suffix should be an empty string instead of null, perhaps a bug somewhere)

return URI.create(String.format("%s%s%s", prefix, fileURI.getRawPath(), suffix));

perhaps can give a quick try to simply leave both prefix and suffix as empty to get into this branch and see if it works as expected. You can confirm by checking the downloadUrl set in the segment metadata in ZK.

We solved it removing segmentUriPrefix and segmentUriSuffix from pushJobSpec -- thank you!

@Xiaobing Perhaps we need to document this or fix this?

yeah, will try to reproduce and patch the

null

suffix issue and refine the doc a bit

Thanks

I was able to reproduce the null issue. The segmentUriPrefix/Suffix fields are checked when doing segmentUriPush or segmentMetadataPush. But only when omitting

segmentUriSuffix

field, I could see

null

got appended to the uri. In fact, setting an empty string there was fine. So I wonder if you might accidentally omit the config when pushing the segments. I’ll patch the util method to avoid accidental null.