Write-up published
Resolved
Google have confirmed that the issue is resolved, and we haven't seen any errors since our last update.
Monitoring
We're no longer seeing errors from Google Cloud APIs, and all our product features have recovered.
We're continuing to monitor the situation and will update if anything changes.
Summary of impact
Some Android push-notifications were failing
Insights were not available
The Google Meet integration was unavailable
Small numbers of temporary errors taking actions in Slack and the dashboard (which would work on a retry)
Investigating
We're continuing to see intermittent issues with a subset of the product due to ongoing Google Cloud API failures.
To reiterate, the following are NOT affected
Alert ingestion
Paging
Status Pages
Incident declaration
A subset of features that depend heavily on Google services directly are currently degraded or broken, including:
Android push notifications
Insights
Google Meet (Scribe calls)
We're continuing to monitoring the situation and will update here when we know more.
To all the hundreds of customers currently fighting the fires, we're right here with you. You've got this ❤️
Investigating
We're experiencing intermittent issues due to Google Cloud API failures. This is not currently affecting declaring incidents or any other critical incident or on-call features or status pages. Any features that depend on Google services are currently degraded or broken.
This includes Android push notifications for on-call, Insights (which relies on BigQuery), and incident calls powered by Google Meet.
We're continuing to monitoring the situation and will update here when we know more.