Hello I am trying to migrate from Druid 24 to Druid 29 to leverage MSQ for ingestion. Our pipeline are mainly batch pipelines which ingest yesterday's data by running the index_parallel jobs next day. In each of our job, we leverage Tuple sketch, as part of using MSQ for the same last year, we contributed to add sql function for the same. There is a large performance gap between MSQ JOB AND Index parallel job. For Index parallel it takes 40-60 mins to ingest 1 day of data around (100gb) - MM is used. But when I use MSQ it took 4 hours to ingest only 1hour of data, index parallel takes 7- 10 mins to do the same task. MSQ I am running on Indexers as compared to MM. Can I get some help where should I debug this.

How many Tasks are you running with MSQ.

I tried with 30/60 tasks

Very surprising. Could you please share the controller task report

I will fetch it and share, will it be available in console or I need to check my my Infra team

It would be available on druid console

Ok

Go to tasks-> search for the task id, go to the report section.

What are we looking in the logs.

it’d be good to see which stage consumes the most time. Are you ingesting a large amount of segments via INSERT?

Per day data is 100 gb, roughly 4-5gb per hour, 1000 partitions files

is it INSERT or REPLACE, and what are the number of segments you are ingesting?

INSERT

hmm, task reports/logs would help in this case. If there are a large number of segments, then INSERT has bad perf, but that doesn’t seem to be a problem in your case

I will get the logs for this sooner.