This message was deleted.
# report-bugss
Slackbot
07/11/2023, 9:55 PMThis message was deleted.
g
Greg
07/11/2023, 9:55 PMtaking a look now!
Greg
07/11/2023, 9:55 PMwe are
Greg
07/11/2023, 9:56 PMdigging in
Greg
07/11/2023, 9:59 PMJust updated status page + 2x’d our server pods temporarily
b
Ben Jaffe
07/11/2023, 10:01 PMThanks Greg! Do y'all have alerting when API latency spikes?
g
Greg
07/11/2023, 10:03 PMWe do, cc @Brendan Ngo, but unfortunately datadog metrics from AWS are delayed. So in this case we got paged minutes after you posted here
🫣 1
Greg
07/11/2023, 10:03 PMWe have faster metrics from our server APM, but those are sometimes inaccurate for latency specifically - we prefer to measure from load balancer
Greg
07/11/2023, 10:03 PMso our graphs had the spike, but our alert was delayed
b
Ben Jaffe
07/11/2023, 10:04 PMgotcha <shakes fist at datadog>
g
Greg
07/11/2023, 10:04 PMlol
Greg
07/11/2023, 10:04 PMlatency is coming down, debugging root cause now
Greg
07/11/2023, 10:04 PMthanks for flagging!
👍 1
b
Ben Jaffe
07/11/2023, 10:04 PMdogs are supposed to be low-latency
g
Greg
07/11/2023, 10:07 PMLooks like a stray web app (specific user browser tab) ddos’d us momentarily. Excuse me while we tighten up our load balancer rate limits lol
😂 1
b
Ben Jaffe
07/11/2023, 10:10 PMBTW, the bold orange on the notif is a bit extreme in dark mode, though I think it's pretty.
g
Greg
07/11/2023, 10:10 PMcc @Xiulung Choy
x
Xiulung Choy
07/13/2023, 9:45 PMHave the banner colors on my radar! Thanks for reporting