Filo connection issues and high response time
Incident Report for Filo
Postmortem

Some Filo shards were frozen due to not being able to reconnect to the Discord API. As there was no developer connected, a restart could not be requested, thus increasing the downtime of our service.

Posted Aug 09, 2020 - 09:29 CEST

Resolved
Discord update: We've received clarification from Google that the issue is a scheduled job from some other customer that is causing disruption. The next scheduled run is not for a few more hours, so this won't recur for now.

Separately, Discord engineering is powering through a migration of part of our infrastructure to a different class of machines that will not share any resources with the other customer, and this will give us more certainty that Discord should be safe from further impact.

We are working to get this migration done by the next scheduled job time, and anticipate that we should be able to do so. Therefore, since we do not anticipate further impact, we are going to close this statuspage. If anything does happen, we will of course open a new incident and let you know.
Posted Aug 09, 2020 - 01:25 CEST
Monitoring
Discord update: We apologize for the impact earlier. In the past few days, we've been working with Google to understand why our servers are undergoing random performance degradation. Google has identified a "noisy neighbor" situation, but their engineering teams don't have any solutions available to use at this time.

While Google investigates on their end, Discord engineers are working to mitigate the issue by doing some internal defensive changes to our infrastructure to reduce the severity if this happens again. However, until the root cause is addressed there is a chance this issue will recur. In these instances we will work quickly to restore service.
Posted Aug 09, 2020 - 00:06 CEST
Update
Discord update: Things are looking better but we're continuing to monitor.

Discord issue: https://status.discord.com/incidents/bnv0wbddzz2x
Posted Aug 08, 2020 - 22:34 CEST
Investigating
We are currently investigating this issue.
Posted Aug 08, 2020 - 22:00 CEST
This incident affected: Filo (Shard #0, Shard #1, Shard #2, Shard #3, Shard #4).