Hi, I have a bunch of files (10000+, 50 MB each) t...
# generals
Shivam Choudhary
08/27/2025, 1:00 AMHi, I have a bunch of files (10000+, 50 MB each) that my system writes into S3 every minute. What would be the recommendation to have near real time ingestion for this ? I can't use Kafka because my files are big. Should I go with batch ingestion every 5 minutes or so ?
j
Jae Calanog
08/27/2025, 3:25 AMYes. Batch would be good outside of Kafka. Because of the frequency and size, you would need to make sure you sized your system to have sufficient nodes for ingestion, query access and data retention.
Given your stated values, this looks like a large deployment. So ideally test on sample data to get a feel on how much to scale.
b
Bibek Sahoo
08/27/2025, 3:44 AMMSQ would be a good option to use here: https://druid.apache.org/docs/latest/multi-stage-query/examples
s
Shivam Choudhary
08/27/2025, 3:50 AMCan I schedule MSQs as well ? Sample request page 404s https://druid.apache.org/docs/latest/multi-stage-query/api-reference/sql-ingestion-api#sample-request
b
Bibek Sahoo
08/27/2025, 3:51 AMScheduling is not possible within druid but you can use some external tool like airflow or something similar
Bibek Sahoo
08/27/2025, 3:52 AMBibek Sahoo
08/27/2025, 3:52 AMsomething similar
s
Shivam Choudhary
08/27/2025, 3:56 AMMakes sense, can use some ETL to trigger that. Hopefully if I scale correctly 2 TB/5min should be possible.
b
Bibek Sahoo
08/27/2025, 3:57 AMNot sure on that 😞
g
Gian Merlino
08/27/2025, 10:34 AMJust wanted to chime in saying that it is possible in principle to push that data into Kafka, it doesn't really matter how big the files are since they'd be broken up into messages anyway. Assuming each file is composed of multiple rows. So, if you wanted to go that route, I wouldn't be too scared of it
Gian Merlino
08/27/2025, 10:41 AMAs for batch ingest, 2 TB / 5 min may be doable, the main limit you run into is the maximum number of workers per job is 1000 (see https://druid.apache.org/docs/latest/multi-stage-query/reference#limits)
This means each worker has to load 2 GB in 5 min. Whether this is doable will depend on how complex of processing you need to do, so I would give it a try and try to keep the processing as simple as possible
Gian Merlino
08/27/2025, 10:41 AMBtw, the correct doc page is https://druid.apache.org/docs/latest/api-reference/sql-ingestion-api#sample-request. Where did you see the broken link?