incident.io

Delayed updating of escalation status when using PagerDuty
Resolved

We're no longer seeing errors from the PagerDuty endpoint and have confirmed escalations are polling successfully, meaning escalation messages (created in response to /inc escalate) are working once more.

We've followed up with PagerDuty to understand what caused this, and improved our retrying logic to make us more robust to future flakiness.

Sat, Dec 17, 2022, 07:35 PM
(2 years ago)
·
Affected components

No components marked as affected

Updates

Resolved

We're no longer seeing errors from the PagerDuty endpoint and have confirmed escalations are polling successfully, meaning escalation messages (created in response to /inc escalate) are working once more.

We've followed up with PagerDuty to understand what caused this, and improved our retrying logic to make us more robust to future flakiness.

Sat, Dec 17, 2022, 07:35 PM

Investigating

We're seeing a specific PagerDuty endpoint that we use to fetch escalation log entries fail, which is preventing us from updating escalations with recent data.

Creating escalations via /inc escalate is still working, and PagerDuty auto-create is unaffected. But the escalation message presented to the channel after /inc escalate may show stale data, even while the escalation has been ack'd.

Sat, Dec 17, 2022, 06:55 PM(40 minutes earlier)