Daniel Khoa Le
07/05/2023, 10:13 AMhigh CPU Usage of the EC2 instance hosting my Airbyte but I cannot determine the root cause.
➡️ Context:
• I used a t2.large to host only Airbyte.
• I have several Airbyte jobs running every 15 minutes, several others on daily basic.
• I observed that cpu.user and cpu.stolen were constantly at 45-50% each, adding up to ~100% octavia shock (screenshot 1).
➡️ My quick solution:
• I tried to restart all the running Airbyte containers (docker-compose down and up again).
➡️ After I restarted the Airbyte containers:
• The CPU.user dropped to an avg. of ~ 5%, compared to ~50% before.
• CPU.stolen was gone.
• For testing, I manually triggered some Airbyte jobs one by one and also together, these scenario caused some spikes, up to 100% in case of CDC sync in the CPU usage (screenshot 2 & 3) but nothing was unusual there octavia rolling eyes.
• One step I missed was that I should have ran docker stats before restart them. Now I don't know the CPU usage of the Airbyte containers before the restart.
❓ Questions
• Do you think this issue is specific to Airbyte? I read about why cpu.stolen could be high and it does not make sense in my case. The cpu.stolen is complete gone now (after the restart) while my Airbyte workload remains intact octavia thinking.
• Should I implement some periodic restart for Airbyte?
Thanks for your help.
Daniel