Enlarge / Today’s global Facebook and Facebook-owned-services outage appears to be the result of a flubbed BGP configuration change pushed by a Facebook engineer this morning. (credit: Sean Gladwell via Getty Images / Jim Salter)
Facebook—and apparently all the major services Facebook owns—are down today. We first noticed the problem at about 11:30 am Eastern time, when some Facebook links stopped working. Investigating a bit further showed major DNS failures at Facebook:
So, @facebook‘s DNS is broken this morning…
TL;DR: Google anycast DNS returns SERVFAIL for Facebook queries; querying https://t.co/0BDgaIHmlr directly times out. pic.twitter.com/3GHJ3mW0P0
— Jim Salter (@jrssnet) October 4, 2021
The problem goes deeper than Facebook’s obvious DNS failures, though. Facebook-owned Instagram was also down, and its DNS services—which are hosted on Amazon rather than being internal to Facebook’s own network—were functional. Instagram and WhatsApp were reachable but showed HTTP 503 (no server is available for the request) failures instead, an indication that while DNS worked and the services’ load balancers were reachable, the application servers that should be feeding the load balancers were not.
A bit later, Cloudflare VP Dane Knecht reported that all BGP routes for Facebook had been pulled. With no BGP routes into Facebook’s network, Facebook’s own DNS servers would be unreachable—as would the missing application servers for Facebook-owned Instagram, WhatsApp, and Oculus VR.
