Hi all, A few weeks back, I wrote regarding SQL-Base-Ingestion generating unbalanced shuffle segments (Lost the thread 😞 ). I added a cluster key that helped. And I tried Druid 29.0.0. I'm faced with the same issue again! Didn't specify the cluster by, so, it clustered by all the group by columns . And Stage 0

groupByPreShuffle

generated very large shuffles, the average is 188k, but I have one with 8.6M Ideas? Thanks

Didn’t specify the cluster by, so, it clustered by all the group by columns .

The columns in the preShuffle are the dimensions in the

GROUP BY

clause. The nomenclature is a misonmer. The segments should be sized correctly (assuming you haven’t used a LIMIT BY clause, which in itself is another unfixed bug)

The final segments are good! no issue there

The final segments are good! no issue there

thanks for confirming!

Processing the shuffle is the slow part! It is also very unbalanced

Hmm, upon thinking, that might be the case since the output of the

groupByPreShuffleFactory

should have been evenly partitioned 🤔

I think I narrow it down to the use of : DS_TUPLE_DOUBLES

Also, running native ingestion, works fine!

Difficult to compare the two

Do you mean native ingestion is faster than MSQ based ingestion?

Yes, way faster

can you share the comparative times please. feels weird

98mn MSQ vs 21mn

by any chance were you running an INSERT query with MSQ? And how many segments did the job generate at the end

Using replace into

weird 🤔 Thanks for providing info 🙂

Any idea how to debug this more?

you can check the logs, and see where the most time is getting consumed in the MSQ task.

In the task log, all I see is the following, over and over:

2024-03-19 13:02:07,851 [ServiceClientFactory-3] level: INFO  category: org.apache.druid.rpc.ServiceClientImpl - message: Service [query-b6b09176-9b11-4c3a-bcb1-474bc258a035] request [POST <http://x.x.x.x:8100/druid/worker/v1/chat/query-b6b09176-9b11-4c3a-bcb1-474bc258a035/counters/query-b6b09176-9b11-4c3a-bcb1-474bc258a035-worker0_0>] encountered exception on attempt #1; retrying in 100 ms (org.jboss.netty.channel.ChannelException: Channel disconnected)
2024-03-19 13:02:07,961 [ServiceClientFactory-0] level: INFO  category: org.apache.druid.rpc.ServiceClientImpl - message: Service [query-b6b09176-9b11-4c3a-bcb1-474bc258a035] request [POST <http://x.x.x.x:8100/druid/worker/v1/chat/query-b6b09176-9b11-4c3a-bcb1-474bc258a035/counters/query-b6b09176-9b11-4c3a-bcb1-474bc258a035-worker0_0>] completed.

I think the difference b/w the payload’s cool. The different partitions can happen theoretically, if the weights of the keys is different (though this varied result doesn’t seem correct, it can happen).

No