Apache Pinot

Hi, we noticed that the download url for all segments in our upsert tables is null, but the segments exist in deepstore. Has anyone ever seen that issue before? Thinking to manually update all the segment metadata in zk since they are realtime segments. lmk if that is how you resolve this.

Hmm, this seems quite odd. Personally, I haven’t seen this. Are there any errors in the logs during segment commit?

I will search for it, since we're using peer download should we look in the server or the controller?

also, we rebalanced the upsert tables after scaling up a tenant, not sure if that had something to do with it.

thanks for responding <@UDQU92KBK>! I will search the logs and see what I can find

Seeing messages like this:
```	2022/02/20 02:28:50.884 WARN [PinotFSSegmentUploader] [enriched_station_orders_v1_16_2_upsert__4__81__20220219T0227Z] Failed to upload file /var/pinot/server/data/index/enriched_station_orders_v1_16_2_upsert_REALTIME/enriched_station_orders_v1_16_2_upsert__4__81__20220219T0227Z.tar.gz of segment enriched_station_orders_v1_16_2_upsert__4__81__20220219T0227Z for table java.util.concurrent.TimeoutException ```

It seems that all the upsert table segments have that error, but we have realtime tables that do not have the error

Is there a way to force pinot to retry the upload the segments from the servers on the upsert tenant?

I see the timeout is hardcoded, I can try to increase it - but same cluster has servers which upload all segments within 10 seconds:
```PinotFSSegmentUploader.DEFAULT_SEGMENT_UPLOAD_TIMEOUT_MILLIS```


I extrapolated the segment upload time from some of our metrics - maybe gcs is slow for us, it's only 225mb per segment so it shouldn't take 10 seconds :man-shrugging:

if it times out and fails to upload, then no url is expected?

But <@UUZ3KPZ8T> Elon mentioned that the segment was indeed pushed to deepstore (or at least exists there).

well, it's possible that the file uploaded but fail to receive response from client side

also, i think the segment upload flow is the same for non-upsert tables, so i dont see anything special with upsert

Thanks! I'm trying with an increased timeout

FYI, <@UUZ3KPZ8T> <@UDQU92KBK> <@UDRJ7G85T> - I increased the timeout in the Pinot fs segment uploader and verified that we no longer get those timeouts and null segment url's

Would it be worth it to add a config? I think running in the cloud adds some latency for us.

lmk, I would be happy to do it if no one is already working on it

Hey <@UPK7Y0X0A> yeah go ahead and file the PR (or issue). If anyone has concern (hope not), we can discuss there.

sorry to bug you about something else: I posted about setting batch message mode, whenever you have some time