I know it's running again because the other tasks created now are failed, and it has already processed 3M rows, that was the same amount of rows that the previous job had (I usually have only ~ 200r/s), so for a job with 8min, I would have 100k at max

Maybe this is similar to a bug (or feature?) that when you have replica > 1, then the second job instead of having the cancelled status, it finishes with FAILED, but inside the log (peon) I can see the "Task was cancelled" in the logs, followed by "Task completed with status "status" : "SUCCESS" but another exception is throw after the task finishes, the jvm is already going down (logged by parent-monitor-0) then some task may still be trying to write the merge segments and fail and this may make the task status dirty

I don't think there's anything sensitive here: https://gist.github.com/renatocron/c86c263329f6d22202ded616acfcb4f6#file-peon-replica-failed-txt < almost full peon log

hmm, this is as well not expected to me, I din't know that the replica task would still write the data to deep storage