Hi all, Data ingestion using scheduled minion tas...
# general
p
Hi all, Data ingestion using scheduled minion tasks for offline tables is not working for us due to https://github.com/apache/pinot/issues/11761 This has put all ingestion pipelines to halt for us. I have tried playing with pinot admin cli and tested LaunchDataIngestionJob using pinot-admin.sh We are now planning to remove minion servers from deployment and outsource ingestion jobs to separate k8s cron jobs using pinot-admin.sh In these periodic cronjobs, we will fetch files from S3, create segments and push those segments to controller api. This will also decouple ingestion jobs from pinot cluster. Are we thinking right? Will this approach scale in production?