Hey guys! Do we have any experts on batch ingestion here? I am currently trying to optimize my loading time of a roungly ~400GiB of daily data into my system; I am using regular batch ingestion on 20 concurrent tasks running on an two

i3en.6xlarge

instances, but the loading takes >10h; I am looking for a way to better utilize these resources and make those ingestions faster. From what I see in logs, we are getting pretty quickly thru

partial_dimension_cardinality

steps, but

partial_index_generate

are taking hours, filling up completely our memory usage and spilling a lot of data to disk at ingestion. What would be some obvious (or non-obvious) improvements we could implement to make this process more robust? Aside from spinning more instances, which I don't think will solve the problem (?)

Have you considered using MSQ?

I am OK with MSQ, although we have had some issues with too many Jetty connections opening up while ingesting with MSQ, although.. how does it help speed up the process?

few ideas: 1- is memory indeed the issue - or is it cause of low cpu utilization (did you check what it is?)? each

i3en.6xlarge

has

24 cores

, so it should have

23

processing slots by default I think. 2- what do you partition by? maybe you can do random instead of by column (which would hurt query), or pre-order the data and use

assume grouped

flag