Hi all,
Data ingestion using scheduled minion tasks for offline tables is not working for us due to
https://github.com/apache/pinot/issues/11761
This has put all ingestion pipelines to halt for us.
I have tried playing with pinot admin cli and tested LaunchDataIngestionJob using pinot-admin.sh
We are now planning to remove minion servers from deployment and outsource ingestion jobs to separate k8s cron jobs using pinot-admin.sh
In these periodic cronjobs, we will fetch files from S3, create segments and push those segments to controller api.
This will also decouple ingestion jobs from pinot cluster.
Are we thinking right? Will this approach scale in production?