Resolved
Escalations are being processed normally and have remained stable since service was restored. Thank you for your patience.
Monitoring
We have now sent one notification to each escalation path affected by the incident this morning, with a link to review any expired escalations in more detail. We’ve chosen to roll these up, to avoid users receiving a large amount of stale notifications at once.
For now, we'll continue to monitor very closely, whilst the AWS incident is ongoing.
If you require further support, please reach out to us on Slack, or via support@incident.io
Monitoring
We have reviewed all delayed notifications from the incident this morning, and will shortly be sending one notification to each affected escalation path, with a link to review any expired escalations in more detail
We’ve chosen to roll these up, to avoid users receiving a large amount of stale notifications at once.
We'll confirm once this has been done. If you require further support, please reach out to us on Slack, or via support@incident.io
Monitoring
Escalations are continuing to process as normal, in real-time.
We're continuing to work on processing delayed notifications, with further updates to follow.
Monitoring
Escalations are now sending as normal, in realtime and we are seeing platform recovery across the board.
For previously delayed notifications, we are continuing to work on ensuring they are all delivered as intended, though there may be a further delay of up to 30m.
Once we've confirmed full recovery, we'll send a further update.
Identified
We're still seeing severe delays with sending escalations. We're monitoring the upstream AWS outage closely and exploring multiple mitigation options to work around affected upstream providers. Further updates to follow.
Identified
We've identified the source of the delayed escalation notifications. Several upstream vendors are currently being affected by a currently ongoing and widespread AWS outage. Our team are continuing to mitigate and restore normal notification service levels.
More details here: https://health.aws.amazon.com/health/status
We'll provide updates every 30m going forward.
Investigating
We are investigating an issue with delays on escalation notifications - our engineers are investigating and will update shortly.