Facebook, WhatApp and Instagram down....



 
"... faulty configuration change." Uh huh. A good story that explains well, you know the rest.
 
Ah ha. "... routine maintenance on a data center." Which story will they stick with? Stay tuned.
 
Last edited by a moderator:
On a serious note, with the "facebook whistleblower" Frances Haugen (for backround, there's a pretty good WSJ profile piece) coming out with her expose - and (count them) eight (!) very serious SEC complaints - it truly is a weird confluence of events. Her "60 Minutes" interview is remarkably tame considering the possible outcomes here. By all accounts Haugen's Senate Commerce Committee testimony (there seems to be a C-Span record of it) and the complaints are worth familiarizing oneself with.

Over the years I've of course only had indirect exposure to facebook myself, long read articles, data science (some of it recently hampered by facebook supplying researchers with faulty data over at least a year) and getting to understand all kinds of state actor affiliated information operations. Coupled with a smallish circle of tech billionaires with some esoteric (to put it mildly) beliefs, the mysterious ways in which facebook data has been scraped by or otherwise ended up with all kinds of operatives, there just seems to be a lot that has yet to be uncovered and properly understood.

The internet hasn't only been compartmentalized by systems of governance (Roskomnadzor's efforts coupled with troll farms, "the Great Firewall of China") but by just a few platforms being allowed to funnel more than half (I think around 70%) of all web traffic. (The lines are blurring here as well as Zuckerberg fancies facebook as a kind of a state, but that's another matter.) Anyway, for the better part of a decade it has required a conscious effort to maintain even some meaningful diversity in one's online life. It just shouldn't be this way.
 
Last edited:
Nothing nefarious, just a simple cockup. This kind of thing happens ALL THE TIME in IT support.

BGP is a dynamic protocol. There is no single database of internet routes maintained by a central authority - each router tells the next router it is connected to about the networks it hosts and the networks it knows about. They don't necessarily know the final destination - just that traffic to network 10.11.12.13 goes to router x, and that router knows the next hop of the journey. This means each router doesn't need to know the details of how to reach every computer on the internet. Being dynamic, if a particular bit of cable gets cut, other routes dynamically kick in.

Incidentally, its possible to set up your router to announce to the internet that YOU own Facebook's networks as BGP is a very insecure system, designed in a more trusting age where there were no bad actors. If you can convince your peers (it helps if you are a trusted ISP) you can hijack the traffic intended for Facebook and send it to your own network instead. This has happened a few times in the past, usually accidentally when someone makes a typo in their router configuration. That's what some people initially thought mmight have happened here.

However, if I am the network guy at Facebook, and I accidentally make a change to BGP that effectively says "stop telling the internet about all of Facebook's networks', all the routers connected to Facebook will dutifully delete all the routes that lead to Facebook. Nobody will be able to get there. The system did what you told it to do. That's essentially what happened here.

In a simple network where I have one building where I work, my computer is connected to the internal network, and the router is in the same internal network, such a mistake is annoying but not catastrophic. I can simply log back into the router via the local network, reverse the change I made, and then its fixed, probably in minutes.

In a big company like Facebook, I might not even be in the same country as the router I just broke. I might be working from home, relying on a remote connection into Facebook's network which is now gone since I disconnected all the networks. Once the change is made, there might be no way for my computer to reach the router to fix it.

The outage as described however appears to be the fault of bad design.

If you can afford it,
and I think Facebook really can, you should always have an out of band method to access critical infrastructure remotely if needed for this exact situation. Out of band means that it doesn't rely on the device working correctly to get to the device.

You can buy such systems off the shelf easily enough.

Hell, an old laptop in the datacentre with a simple 4G network adapter and serial cables attached to the console ports of the core routers is perfectly doable as an emergency access system - I've done this myself in the past.

Now, with a network as complicated as Facebooks, maybe they did have out-of-band management, but it took them a long time to figure out how they broke it and how to fix it.

Or maybe their 'emergency system' only worked from the office but due to COVID everyone was at home and that risk was never realized until now.

I hope this will result in some much-needed better design at Facebook. Not going to hold my breath though.
 
Last edited:
Facebook are denying that staff had to use angle grinders to get into their rooms......

There is also the issue of people working from home having moved out of the state..........makes it a bit harder for them to come in....
 
Nothing nefarious, just a simple cockup. This kind of thing happens ALL THE TIME in IT support.
While that may be true, when's the last time Facebook went down to that degree? The timing is damn convenient for some. (Pandora Papers dropped the same day.) "Look over here!"
 
The great news in that event is that, despite all the hype and superlatives used to qualify social media, the real working world carried on as if nothing happened ;)
 
Incidentally, its possible to set up your router to announce to the internet that YOU own Facebook's networks as BGP is a very insecure system, designed in a more trusting age where there were no bad actors. If you can convince your peers (it helps if you are a trusted ISP) you can hijack the traffic intended for Facebook and send it to your own network instead. This has happened a few times in the past, usually accidentally when someone makes a typo in their router configuration. That's what some people initially thought mmight have happened here.

I'm sure innocuous explanations are way more likely and didn't intend to play against those odds here. One doesn't have to go back many years to find well publicized examples of BGP hassles, the ones that I remembered off the top of my head are instances where China Telecom and Rostelecom have routed much of "western" traffic through their networks (facebook, Google and Amazon included). These of course stood out because ulterior motives couldn't be ruled out and there have been repeat incidents.

A good example where safeguards and a certain transparency of operations should be required.
 

The leak started at 21:13 UTC when MainOne Cable Company, a small ISP in Lagos, Nigeria, suddenly updated tables in the Internet’s global routing system to improperly declare that its autonomous system 37282 was the proper path to reach 212 IP prefixes belonging to Google. Within minutes, China Telecom improperly accepted the route and announced it worldwide. The move by China Telecom, aka AS4809, in turn caused Russia-based Transtelecom, aka AS20485, and other large service providers to also follow the route.
According to BGPmon on Twitter, the redirections came in five distinct waves over a 74-minute period. The redirected IP ranges transmitted some of Google's most sensitive communications, including the company's corporate WAN infrastructure and the Google VPN.
 
Some (many?) social media users felt cut off from their friends. When asked why they didn't use the telephone feature on their toys, they cited telephone phobia - at least, according to the rumour I've heard ☹️
 
This is all such nonsense. Facebook - and others - had no emergency plan? Nonsense. Working from home? Nonsense. If you have a cell phone and a few phone numbers and a plan then none of this. Ever.

"... routes traffic through China." Happens to me all the time. I forget my key card and I'm locked out. My router is located in Ubombistan (just outside of Tajikistan... or maybe some other stan).

Cover stories followed by more cover stories. They should watch the Keystone cops movies.
 
Last edited by a moderator:
 

No wonder the United States government is after Facebook and others. In an eternal attempt to create monopoly situations in-country, various "too wealthy for words" entities are buying up the smaller fish and hoping *nobody* notices. I have been watching the behind-the-scenes situation in the printing industry for years. The same in electronic games.

In this case, it's "log in with Facebook" or one of the Big Three or You GET NOTHING!!!! As in, NOTHING !!!!!!

If George Orwell were here, he'd tell you that "The Gate Keepers have to be knocked down a few notches or you will get more of this in future."
 
Some (many?) social media users felt cut off from their friends. When asked why they didn't use the telephone feature on their toys, they cited telephone phobia - at least, according to the rumour I've heard ☹️

I red that "people re-discovered SMS" during the blackout.
...
Hmmmm, what ?
...
Is SMS obsolete ?
 
No Social Media! Millennials overdosing on kale, Starbucks, various hair dye colors, Sponge Bob (or whatever is the latest?) and cell phone to face removal treatments.
 
Some (many?) social media users felt cut off from their friends. When asked why they didn't use the telephone feature on their toys, they cited telephone phobia - at least, according to the rumour I've heard ☹️

I red that "people re-discovered SMS" during the blackout.
...
Hmmmm, what ?
...
Is SMS obsolete ?
All the cool kids use Whatsapp for messaging. Old farts like me and my wife use Facebook Messenger. Grandpas use SMS :)
 
No Social Media! Millennials overdosing on kale, Starbucks, various hair dye colors, Sponge Bob (or whatever is the latest?) and cell phone to face removal treatments.

'face removal treatments'? That sounds... a bit disturbing.
 
One might say it's convenient Facebook had this happen. Especially for certain political forces at this time.
But most likely it's an accident rather than conspiratorial design.
 
Some (many?) social media users felt cut off from their friends. When asked why they didn't use the telephone feature on their toys, they cited telephone phobia - at least, according to the rumour I've heard ☹️

I red that "people re-discovered SMS" during the blackout.
...
Hmmmm, what ?
...
Is SMS obsolete ?
All the cool kids use Whatsapp for messaging. Old farts like me and my wife use Facebook Messenger. Grandpas use SMS :)

And this is completely silly, when you think about it - because SMS are not dependant from a freakkin' app or even the internet. Only the mobile network. Which never collapses except on December 31, midnight.
So as a 39 years old grandpa I feel smarter than you youngsters.

My 78 year old mom is 100% removed from the Internet, and considering her, hmmm, entrenched opinions, this is probably a blessing. She is blissfully unaware of all the craziness related to the Internet.
 
Last edited by a moderator:
All the cool kids use Whatsapp for messaging. Old farts like me and my wife use Facebook Messenger. Grandpas use SMS :)

As far as I can decipher trends (ha) Signal, Slack and Discord are some of the services to be on. Whatsapp showed promise but from my POV, y'know, Facebook. Real "influencer" pros are of course scouring the landscape to register themselves on the newest and most obscure social media/messaging environments in the hopes of those becoming successful and thus gaining "OG trendsetter" points/status. The real added value (apart from increased data security and privacy in some cases) is usually meager to nonexistent; seems mostly to be about where the herd is stampeding to be data mined (or is it "milked" in this case). I mean, big data as currency is where the innovation is, as far as end user experience goes I struggle to make meaningful distinctions between current structures and erstwhile methods such as IRCs, mailing lists, Classmates, MySpace, et.al (apart from the capacity and volume of the net itself growing meanwhile).
 
I just can't believe they chose such a name, for an app. DISCORD - O RLY ?
Then again, Tik tok is rather silly name - at least it quite appropriate for the stupid thing.
And Facebook souds like fessebouc: exactly ramsbottom.
 

Facebook has apologised after again reporting problems with its services, days after a major outage hit WhatsApp, Instagram and Facebook for hours.
The company said that a "configuration change" had impacted users globally.
It added that the incident was not related to the outage that saw its products taken offline for over six hours earlier this week.
Its Facebook, Instagram, Messenger and Workplace products had been affected, it said.
 

Similar threads

Back
Top Bottom