We're no longer seeing errors from the PagerDuty endpoint and have confirmed escalations are polling successfully, meaning escalation messages (created in response to /inc escalate) are working once more.
We've followed up with PagerDuty to understand what caused this, and improved our retrying logic to make us more robust to future flakiness.
No components marked as affected
Resolved
We're no longer seeing errors from the PagerDuty endpoint and have confirmed escalations are polling successfully, meaning escalation messages (created in response to /inc escalate) are working once more.
We've followed up with PagerDuty to understand what caused this, and improved our retrying logic to make us more robust to future flakiness.
Investigating
We're seeing a specific PagerDuty endpoint that we use to fetch escalation log entries fail, which is preventing us from updating escalations with recent data.
Creating escalations via /inc escalate is still working, and PagerDuty auto-create is unaffected. But the escalation message presented to the channel after /inc escalate may show stale data, even while the escalation has been ack'd.