This message was deleted.

Hi @Vignesh Shetty have you tried using MSQ ingestion? In most cases, it is quicker than the native batch ingestion. Doc: https://druid.apache.org/docs/latest/multi-stage-query/

No, Ravin. We will try this if the current insertion process doesn't go well as expected.

12bn rows is a lot, however 72hours seem excessive even for that. Perhaps the cluster is underpowered and requires more resources to process the data. MSQ is another alternative that you can try out.

Tried MSQ as well, Process is taking a lot of time. Could you please suggest infra required for 20bn row count? @Ravin Trivedi, @Laksh Singla. Please find our current Druid Cluster nodes and instance type 1. druid-master-arm-node m6gd.large & 100G 2. druid-master-arm-node m6gd.large & 100G 3. druid-master-arm-node m6gd.large & 100G 4. druid-query-arm-node c6gd.4xlarge & 100G 5. druid-data-arm-node r6gd.4xlarge & 1TB 6. druid-data-arm-node r6gd.4xlarge & 1TB

It is difficult to suggest infra without knowing more about the dataset - number of columns, how are you partitioning, transforming, etc. Please note that the broker plans the MSQ ingestion and uses number of task slots configured on the middle manager (data node). For 20bn rows, 2

r6gd.4xlarge

seem less than required for ingestion to be performant. General rule is: more tasks on middle manager --> better handling of ingestion throughput --> lesser time for ingestion to complete. May I know how much time is the MSQ ingestion taking with the above infra?