This message was deleted.
# generals
Slackbot
04/13/2023, 11:36 AMThis message was deleted.
a
Amatya Avadhanula
04/13/2023, 11:43 AMCould you please share the overlord logs during this time as well? Also, what is the configured task duration?
Amatya Avadhanula
04/13/2023, 11:45 AMIf data is persisted to deep storage, it means that segments were successfully committed and that the offsets are stored in the metadata irrespective of the status of the supervisor / tasks
n
Niranjan Sridhara
04/13/2023, 11:50 AM"taskDuration": "PT3600S"
Niranjan Sridhara
04/13/2023, 11:51 AMI have attached the coordinator log. Because in docker we are having only the container for coordinator but not for the overlord
a
Amatya Avadhanula
04/13/2023, 11:53 AMThe streaming ingestion supervisor runs on the overlord. Do you mean to say that you're running the overlord within the coordinator using
druid.coordinator.asOverlord.enabled=trueAmatya Avadhanula
04/13/2023, 11:54 AMAlso, the logs seem to start at 11:37. It'd help to have the task logs and overlord logs from the same time period
n
Niranjan Sridhara
04/13/2023, 11:57 AMit is using the server timings.. so it is showing the 11:37
a
Amatya Avadhanula
04/13/2023, 11:59 AMYes, I meant that the task log you attached is from 10:39 whereas the coordinator log is at 11:37. Could you please share logs for both the task and overlord during the same time window?
n
Niranjan Sridhara
04/13/2023, 12:00 PMok..
Niranjan Sridhara
04/13/2023, 12:14 PMoverlord-coordinator_log.txt
a
Amatya Avadhanula
04/13/2023, 12:54 PMStrange, it appears that the overlord shut down the task but it's not clear why.
You may want to also see the error message within the task status for taskId;
index_kafka_kttmss_3578bdd37efd9dc_bllnefeln
Niranjan Sridhara
04/13/2023, 1:42 PMlog.txt is the complete log for the same task..
a
Amatya Avadhanula
04/13/2023, 1:48 PMI'm referring to the status. You can view it using the web-console
n
Niranjan Sridhara
04/14/2023, 7:15 AMThis is what I am getting..
a
Amatya Avadhanula
04/14/2023, 7:21 AMCode 137 usually means an OOM. Could you please try reducing maxRowsInMemory for this supervisor? (Maybe to 100000)
Amatya Avadhanula
04/14/2023, 7:24 AMPlease refer to https://druid.apache.org/docs/latest/operations/basic-cluster-tuning.html and also go through your MM logs for more info if the above step doesn't help
n
Niranjan Sridhara
04/14/2023, 7:31 AMok.. let me try that..
and one more this.. as I mentioned earlier this deployment is on AWS ECS.. but I did a docker deployment on EC2 (t2.large), there everything is working fine. For ECS I have given 8vCPUs and 16 GB RAM, t2.large is 2vCPU and 8 GB RAM...