This message was deleted.

You will not see data from an

index_parallel

job until it completes

Thanks Kyle! How much time would you guess that amount of data take to complete being inserted into datasource? How can I estimate?

Very difficult to estimate this. It can be affected by the organisation of the source files, any secondary partitioning chosen, number of dimensions, cardinality of dimensions… too much to give a simple answer

Okay, is there any chance where I can see the Progress or so? like how much data left to be consumed or so?

not with a native index_parallel - if you run with MSQ you get some progress bars / rate etc.

To echo Kyle, MSQ is superior to index_parallel in pretty much every way. One specific improvement is that it gives you much better feedback of how stuff is going and how long it will take. You can convert an index_parallel to a SQL (MSQ) with just a handful of clicks: https://druid.apache.org/docs/latest/tutorials/tutorial-msq-convert-spec.html

Thank you so much! Will have a look on that one! @Kyle Hoondert @Vadim

Hmm - What version are you on? Might be time to talk to your team about upgrading. If I have it right, I think you’re on something like the Imply equivalent of Druid

0.21

@Kyle Hoondert We have Imply version 2022.01.6 LTS

Yes - this is too old to have MSQ - you need to upgrade

okay, thank you @Kyle Hoondert!