https://pinot.apache.org/ logo
#general
Title
# general
f

francoisa

04/08/2022, 9:17 AM
Hi getting deeper in my purge process dev many thanks to the pinot team for the awesome support and code quality ๐Ÿ˜‰ . Is there any way to upload a realtime segment ? Like purging it and refreshing it ? I do not really inderstand why this is not possible if the segment is not consuming ๐Ÿ˜• And Iโ€™m a bit surprised of the API getting an OFFLINE table for the uploadProcess instead my original type REALTIME
m

Mark Needham

04/08/2022, 9:59 AM
at the moment you can't replace a real-time segment, but I think someone might be working on it. @User will have more info.
f

francoisa

04/08/2022, 10:02 AM
I can see itโ€™s replacing my REALTIME to OFFLINE for the table name but donโ€™t know where ๐Ÿ˜•
m

Mark Needham

04/08/2022, 10:03 AM
at the moment you can only replace offline segments, real-time ones won't work
not sure why it'd be renaming your table
i thought it threw an error indicating that you can't replace real-time segments
f

francoisa

04/08/2022, 12:00 PM
Not really it complains about segment non existing (Still in realTime Table )`org.apache.pinot.common.exception.HttpErrorStatusException: Got error status code: 410 (Gone) with reason: "Cannot refresh non-existing segment, aborted uploading segment: actions__0__0__20220405T0951Z of table: actions_OFFLINE" while sending request: http://127.0.0.1:9000/segments?enableParallelPushProtection=true&tableName=actions`
ok I will switch my logic to get the segment in OFFLINE table an then purge them ๐Ÿ˜‰
m

Mark Needham

04/08/2022, 12:02 PM
There's a simple example showing one way to do that here --> https://dev.startree.ai/docs/pinot/recipes/backfill
f

francoisa

04/08/2022, 12:02 PM
Yes already use it ๐Ÿ˜‰
m

Mark Needham

04/08/2022, 12:02 PM
ah cool
s

Sid

05/18/2023, 7:38 AM
@Mark Needham the segment backfill task, can it be done as an adhoc task in minion. I try to avoid using the jobspecs.yaml as it will require me to run it at the instance where pinot runs.