Hello! I am ingesting json in batch from s3 and have noticed that even though I have my segments set to 1 month, every ingest creates a new segment. Is there any way to have new data import into already existing segments granted they are within the same month?

Hello, segments in Druid are immutable and cannot be appended to. The process used to combine multiple segments as you are describing is called compaction. https://druid.apache.org/docs/latest/data-management/compaction At a simplified level: • ingestion creates 1 segment • an append (not overwrite) ingestion adds 1 new segment so now we have 2 segments • compaction combines the segments to create 1 new combined segment

How do people typically handle more micro batches? Do they just write them to a Kafka topic and ingest them via stream which appears to have a way around this? Also thanks for the quick reply!

Yes, streaming through Kafka would definitely help with the micro-batch scenario and is how this is typically handled

Awesome thanks!