This message was deleted.

Hi Sai, Your taskDuration is set to 2 hrs, so you are potentially accumulating intermediatePersist files which are stored on local disk, up until the time a segment is built. Dropping

intermediateHandoffPeriod

to a smaller interval will at least guarantee that a segment is built and published (and the local persist files deleted) more frequently than once every two hours. Not sure if it is relevant to this specific problem ... but you have a large

maxRowsInMemory

and

intermediatePersistPeriod

... this controls how long you leave the data in the first stage of "real-time segment" life, i.e. the initial row buffer, before persisting data (locally) to disk. These numbers are very high, the defaults are 150k for

maxRowsInMemory

and PT10M for

intermediatePersistPeriod

. Leaving data in the row buffer is generally a JVM heap issue, not a local disk storage issue, so not sure if setting these back to the defaults would help ... but I think it's worth a try. Can you also tell us how many ingestion tasks are in this Supervisor, how many segments-per-task are being created per two hour interval, and if you have other Supervisors running on the MM that are streaming in substantial volumes of data? Thanks. John