Hi, I am exploring the possibility of using apache spark to move the segments from realtime table to offline table. What job type can I use in the Ingestion job spec to achieve this ? Has anyone achieved this , if so it would be helpful if you could point me to a doc/wiki

I believe there are cases for that. @User may have more details

Ideally this would be done using the RealtimeToOffline minion task

@User I'm aware of this doc , but this mostly talks about using Minion. I was exploring the possibility of solely using spark to move the segments from realtime to offline table.

Okay. It was not clear to me that you are exploring spark only option. This managed offline flow is the one I recently implemented in my project.

@User maybe you want to get some insights from @User for the pinot managed flow

already provided more info on that to @User on another channel and directly too

Thanks for your help Laxman