Does MSQ replace always create tombstones? Is there a way to get rid of them if they don't have eternity timestamps?

I'm guessing you have (and want) dropExisting set to true? I'd expect kill jobs or autokill would eventually clean those up - or are you seeing differently?

Looks like that doc is for native batch. I'm using msq and my query is of format

REPLACE INTO xx OVERWRITE ALL WITH

Does the kill tasks remove tombstones even if they don't have timestamp as eternity?

You're right about that doc page, sorry. I would expect/hope that kill tasks would clean them up once they're no longer needed and durationToRetain is passed.

There's a separate coordinator duty that takes care of cleaning up eternity tombstone segments: https://druid.apache.org/docs/latest/design/coordinator/#clean-up-non-overshadowed-eternity-tombstone-segments Relevant metric:

segment/unneededEternityTombstone/count

@Abhishek Balaji Radhakrishnan What about removing non-eternity tombstones? I run replace tasks everyday and tombstone segments are piling up.

The above coordinator duty only cleans up used eternity tombstone segments because they don't honor the provided segment granularities and can cause segment allocation errors if appended to. That's why they are cleaned up separately. For non-eternity used tombstones, the granularity of the tombstone segments should match with what's provided in the replace queries, so they can be appended to like any other segment and hence we decided not to remove them at the time. Otoh, if tombstones are fully overshadowed and thus unused, they would become eligible for deletion by kill tasks and be removed regardless of the eternity/non-eternity endpoint. The presence of tombstones itself shouldn't be causing any issues. Are you seeing anything other than them simply piling up?

Thanks for the clarification! Main concern is the pile up especially if each tombstones has an entry in the druid_segments table.