https://linen.dev logo
Outlook Triggers not firing: URGENT
# 🖥│support-forum
c
Some users are not seeing triggers for the last 2 hours despite trigger status showing active and account connection appearing healthy https://dashboard.composio.dev/pingassistant/default/triggers/ti_AlEQSGA-kTUv I do not see this as part of the outage scope
User confirmed with screenshots inbox message reception
The last trigger was a manual rerun from my end
We do have a list of users that may have had chronic webhook reception failrues for the last ~24 hours if that is useful information. But that problem has been solved
u
Hi @Chad Holmes, we are checking this.
c
thank you @Uday
Am I reporting an outage here? What can I do
Our users need to know what is happening
u
@Chad Holmes we had a couple of outages in the past ~24hrs which might've caused the trigger issues. Digging into it now. You can checkout our status page [here](https://status.composio.dev/).
c
I don’t see an active status related to triggers
Just the slack one and the request timeouts but those appear as resolved
All status appear resolved on the status page
Users continue without triggers and the status page continues to show all issues resolved
u
@Chad Holmes, I understand it is frustrating but we really appreciate the patience.
I'll share the details asap, thank you🙏
Hi Chad, thanks for the report — confirmed on our side. Quick recap: What happened: Between 14:59 and 16:02 UTC today (~1h 3m), our webhook ingestion layer hit a runtime fault and 502'd inbound Outlook events from Microsoft Graph. Your subscriptions, connected accounts, and trigger configs were never affected — the failure was upstream of your triggers, which is why the dashboard showed everything as active. Status: Faulty instances were replaced; ingestion recovered at 16:02 UTC. Outlook triggers have been firing normally for the last hour. Could you confirm the same?
c
There is no outlook activity across our entire project for the last 4 hours which matches the 14:59 start point
we have not seen recovery
"active" triggers continue to have no activity
The outlook trigger logs in our project reflect this as well. There continues to be no activity
j
Experiencing the same. Last Outlook trigger was at 7:48 AM PST this morning. Have not had any since.
@Uday
@Chad Holmes same still for you?
c
We really need insight on the issue here we are going on 7-8 hours no status, no tracking of the issue
It seems this is not important to the Composio team yet our production outlook users are experiencing complete blackout
j
@Uday we need an update on this ASAP
c
Going on 9 hours of 0 outlook triggers. Still not even an acknowledgement of it being down or any insight on resolution timeline @Uday
u
@Chad Holmes, @jeff, I understand that this is frustrating, I've escalated this with the team. We are investigating the issue!
c
Why hasn't this been being investigated already for the last 9 hours?
Why is there no status report? How can this be considered production software to any degree?
We lost an entire business day of data
j
Can we please track this issue more officially? @Uday
I imagine Outlook is one of Composio's most popular connections, this needs to be tracked and we need some sort of ETA
c
Can we have leadership involved here? @Uday I can't believe this hasn't been identified as an outage yet and there is no tracking of progress
Once again, I understand outages. But the lack of communication, transparency, and urgency is really affecting us and our business right now
b
Hey Chad, I’m really sorry for the issues you’re facing here. I want to assure you that our team is actively investigating this with urgency. I’ll share an update with you as soon as we have more information on the issue
c
any ETA? Did the team just begin investigation?
j
Any updates? @Uday
c
Guys we are at 12 hours of complete outage for outlook triggers @Samvit @Uday Can we please get some clarity?
been 3 hours since the last information still not outage reported on the status page
I need to know if Composio will solve this problem. Otherwise we are launching our solution off of composio overnight. It has been prepared only because we have no other choice. I can't have mutli-day downtime for email ingestion. Just not an option
a
@Chad Holmes really sorry for the disruption 🙏. Outlook triggers should be back up in the next 1–2 hours. Once we're stable, we'll share a detailed RCA on what caused this and what we're doing to prevent it. I'll keep you posted here.
c
Thanks @Abhishek Just want to reiterate that having an outage is not the problem and my bigger problem here was the lack of reporting, transparency and urgency from the team Will be looking forward to the report and the fix
j
I'm seeing triggers coming in
will missed triggers since the outage still fire or do we lose those permanently?
we definitely need a post-mortem on this
a
@Chad Holmes @jeff quick update - we've implemented a fix and are currently testing and monitoring it internally. Since this was a meaningful change, we're giving it a 3-hour buffer before rolling it out to prod, though it could go live sooner. Will keep you posted
c
Thanks @Abhishek we saw some recovery but again are seeing a pause in delivery I will be up through the night awaiting the fix
b
Hi Chad, jeff - Karan here (CTO/co-founder). Triggers are back up and we're monitoring closely. Wanted to reach out directly. To be straight with you: today's communication and status-page handling were not acceptable. You shouldn't have had to push us for updates on a 12-hour outage - that's on us, not just the outage itself. jeff, on your question - events that failed at the ingestion layer during the panic windows are unfortunately not recoverable. We'll quantify this in the RCA. You'll have a detailed post-mortem within 48 hours: root cause, why our comms failed, and what we're shipping to prevent recurrence. I'll stay on this personally. Genuinely sorry for the disruption.
c
Thanks @Karan looking forward to the continued communication. Confirming that outlook traffic appears normal now
k
Sent the RCA on DM. We are also doing a public PM - Will link it here.
2 Views