Facebook, WhatApp and Instagram down....

Grey Havoc

ACCESS: USAP
Senior Member
Joined
Oct 9, 2009
Messages
16,866
Reaction score
6,481


 

edwest2

ACCESS: Top Secret
Joined
Nov 9, 2007
Messages
2,158
Reaction score
1,362
"... faulty configuration change." Uh huh. A good story that explains well, you know the rest.
 

edwest2

ACCESS: Top Secret
Joined
Nov 9, 2007
Messages
2,158
Reaction score
1,362
Ah ha. "... routine maintenance on a data center." Which story will they stick with? Stay tuned.
 
Last edited:

UpForce

ACCESS: Secret
Joined
Feb 5, 2011
Messages
267
Reaction score
188
On a serious note, with the "facebook whistleblower" Frances Haugen (for backround, there's a pretty good WSJ profile piece) coming out with her expose - and (count them) eight (!) very serious SEC complaints - it truly is a weird confluence of events. Her "60 Minutes" interview is remarkably tame considering the possible outcomes here. By all accounts Haugen's Senate Commerce Committee testimony (there seems to be a C-Span record of it) and the complaints are worth familiarizing oneself with.

Over the years I've of course only had indirect exposure to facebook myself, long read articles, data science (some of it recently hampered by facebook supplying researchers with faulty data over at least a year) and getting to understand all kinds of state actor affiliated information operations. Coupled with a smallish circle of tech billionaires with some esoteric (to put it mildly) beliefs, the mysterious ways in which facebook data has been scraped by or otherwise ended up with all kinds of operatives, there just seems to be a lot that has yet to be uncovered and properly understood.

The internet hasn't only been compartmentalized by systems of governance (Roskomnadzor's efforts coupled with troll farms, "the Great Firewall of China") but by just a few platforms being allowed to funnel more than half (I think around 70%) of all web traffic. (The lines are blurring here as well as Zuckerberg fancies facebook as a kind of a state, but that's another matter.) Anyway, for the better part of a decade it has required a conscious effort to maintain even some meaningful diversity in one's online life. It just shouldn't be this way.
 
Last edited:

overscan (PaulMM)

Administrator
Staff member
Joined
Dec 27, 2005
Messages
13,605
Reaction score
7,638
Nothing nefarious, just a simple cockup. This kind of thing happens ALL THE TIME in IT support.

BGP is a dynamic protocol. There is no single database of internet routes maintained by a central authority - each router tells the next router it is connected to about the networks it hosts and the networks it knows about. They don't necessarily know the final destination - just that traffic to network 10.11.12.13 goes to router x, and that router knows the next hop of the journey. This means each router doesn't need to know the details of how to reach every computer on the internet. Being dynamic, if a particular bit of cable gets cut, other routes dynamically kick in.

Incidentally, its possible to set up your router to announce to the internet that YOU own Facebook's networks as BGP is a very insecure system, designed in a more trusting age where there were no bad actors. If you can convince your peers (it helps if you are a trusted ISP) you can hijack the traffic intended for Facebook and send it to your own network instead. This has happened a few times in the past, usually accidentally when someone makes a typo in their router configuration. That's what some people initially thought mmight have happened here.

However, if I am the network guy at Facebook, and I accidentally make a change to BGP that effectively says "stop telling the internet about all of Facebook's networks', all the routers connected to Facebook will dutifully delete all the routes that lead to Facebook. Nobody will be able to get there. The system did what you told it to do. That's essentially what happened here.

In a simple network where I have one building where I work, my computer is connected to the internal network, and the router is in the same internal network, such a mistake is annoying but not catastrophic. I can simply log back into the router via the local network, reverse the change I made, and then its fixed, probably in minutes.

In a big company like Facebook, I might not even be in the same country as the router I just broke. I might be working from home, relying on a remote connection into Facebook's network which is now gone since I disconnected all the networks. Once the change is made, there might be no way for my computer to reach the router to fix it.

The outage as described however appears to be the fault of bad design.

If you can afford it,
and I think Facebook really can, you should always have an out of band method to access critical infrastructure remotely if needed for this exact situation. Out of band means that it doesn't rely on the device working correctly to get to the device.

You can buy such systems off the shelf easily enough.

Hell, an old laptop in the datacentre with a simple 4G network adapter and serial cables attached to the console ports of the core routers is perfectly doable as an emergency access system - I've done this myself in the past.

Now, with a network as complicated as Facebooks, maybe they did have out-of-band management, but it took them a long time to figure out how they broke it and how to fix it.

Or maybe their 'emergency system' only worked from the office but due to COVID everyone was at home and that risk was never realized until now.

I hope this will result in some much-needed better design at Facebook. Not going to hold my breath though.
 
Last edited:

PMN1

ACCESS: Top Secret
Senior Member
Joined
Jun 4, 2006
Messages
904
Reaction score
474
Facebook are denying that staff had to use angle grinders to get into their rooms......

There is also the issue of people working from home having moved out of the state..........makes it a bit harder for them to come in....
 

sferrin

ACCESS: USAP
Senior Member
Joined
Jun 3, 2011
Messages
14,614
Reaction score
3,539
Nothing nefarious, just a simple cockup. This kind of thing happens ALL THE TIME in IT support.
While that may be true, when's the last time Facebook went down to that degree? The timing is damn convenient for some. (Pandora Papers dropped the same day.) "Look over here!"
 

TomcatViP

Hellcat
Joined
Feb 12, 2017
Messages
4,733
Reaction score
4,203
The great news in that event is that, despite all the hype and superlatives used to qualify social media, the real working world carried on as if nothing happened ;)
 

UpForce

ACCESS: Secret
Joined
Feb 5, 2011
Messages
267
Reaction score
188
Incidentally, its possible to set up your router to announce to the internet that YOU own Facebook's networks as BGP is a very insecure system, designed in a more trusting age where there were no bad actors. If you can convince your peers (it helps if you are a trusted ISP) you can hijack the traffic intended for Facebook and send it to your own network instead. This has happened a few times in the past, usually accidentally when someone makes a typo in their router configuration. That's what some people initially thought mmight have happened here.

I'm sure innocuous explanations are way more likely and didn't intend to play against those odds here. One doesn't have to go back many years to find well publicized examples of BGP hassles, the ones that I remembered off the top of my head are instances where China Telecom and Rostelecom have routed much of "western" traffic through their networks (facebook, Google and Amazon included). These of course stood out because ulterior motives couldn't be ruled out and there have been repeat incidents.

A good example where safeguards and a certain transparency of operations should be required.
 

overscan (PaulMM)

Administrator
Staff member
Joined
Dec 27, 2005
Messages
13,605
Reaction score
7,638

The leak started at 21:13 UTC when MainOne Cable Company, a small ISP in Lagos, Nigeria, suddenly updated tables in the Internet’s global routing system to improperly declare that its autonomous system 37282 was the proper path to reach 212 IP prefixes belonging to Google. Within minutes, China Telecom improperly accepted the route and announced it worldwide. The move by China Telecom, aka AS4809, in turn caused Russia-based Transtelecom, aka AS20485, and other large service providers to also follow the route.
According to BGPmon on Twitter, the redirections came in five distinct waves over a 74-minute period. The redirected IP ranges transmitted some of Google's most sensitive communications, including the company's corporate WAN infrastructure and the Google VPN.
 

Arjen

It's turtles all the way down
Senior Member
Joined
Nov 6, 2010
Messages
2,949
Reaction score
968
Some (many?) social media users felt cut off from their friends. When asked why they didn't use the telephone feature on their toys, they cited telephone phobia - at least, according to the rumour I've heard ☹️
 

edwest2

ACCESS: Top Secret
Joined
Nov 9, 2007
Messages
2,158
Reaction score
1,362
This is all such nonsense. Facebook - and others - had no emergency plan? Nonsense. Working from home? Nonsense. If you have a cell phone and a few phone numbers and a plan then none of this. Ever.

"... routes traffic through China." Happens to me all the time. I forget my key card and I'm locked out. My router is located in Ubombistan (just outside of Tajikistan... or maybe some other stan).

Cover stories followed by more cover stories. They should watch the Keystone cops movies.
 
Last edited:

Grey Havoc

ACCESS: USAP
Senior Member
Joined
Oct 9, 2009
Messages
16,866
Reaction score
6,481
 

edwest2

ACCESS: Top Secret
Joined
Nov 9, 2007
Messages
2,158
Reaction score
1,362

No wonder the United States government is after Facebook and others. In an eternal attempt to create monopoly situations in-country, various "too wealthy for words" entities are buying up the smaller fish and hoping *nobody* notices. I have been watching the behind-the-scenes situation in the printing industry for years. The same in electronic games.

In this case, it's "log in with Facebook" or one of the Big Three or You GET NOTHING!!!! As in, NOTHING !!!!!!

If George Orwell were here, he'd tell you that "The Gate Keepers have to be knocked down a few notches or you will get more of this in future."
 

AnthonyPanza33

ACCESS: Restricted
Joined
Sep 2, 2021
Messages
1
Reaction score
0
It was just awful, but at least I had nothing to distract me that night and I could work in peace.
 

Archibald

ACCESS: Above Top Secret
Senior Member
Joined
Jun 6, 2006
Messages
6,947
Reaction score
5,824
Some (many?) social media users felt cut off from their friends. When asked why they didn't use the telephone feature on their toys, they cited telephone phobia - at least, according to the rumour I've heard ☹️

I red that "people re-discovered SMS" during the blackout.
...
Hmmmm, what ?
...
Is SMS obsolete ?
 

Hydroman

ACCESS: Confidential
Joined
Jan 28, 2008
Messages
78
Reaction score
137
No Social Media! Millennials overdosing on kale, Starbucks, various hair dye colors, Sponge Bob (or whatever is the latest?) and cell phone to face removal treatments.
 

overscan (PaulMM)

Administrator
Staff member
Joined
Dec 27, 2005
Messages
13,605
Reaction score
7,638
Some (many?) social media users felt cut off from their friends. When asked why they didn't use the telephone feature on their toys, they cited telephone phobia - at least, according to the rumour I've heard ☹️

I red that "people re-discovered SMS" during the blackout.
...
Hmmmm, what ?
...
Is SMS obsolete ?
All the cool kids use Whatsapp for messaging. Old farts like me and my wife use Facebook Messenger. Grandpas use SMS :)
 

edwest2

ACCESS: Top Secret
Joined
Nov 9, 2007
Messages
2,158
Reaction score
1,362
No Social Media! Millennials overdosing on kale, Starbucks, various hair dye colors, Sponge Bob (or whatever is the latest?) and cell phone to face removal treatments.

'face removal treatments'? That sounds... a bit disturbing.
 

zen

ACCESS: Top Secret
Senior Member
Joined
Jul 15, 2007
Messages
2,733
Reaction score
1,450
One might say it's convenient Facebook had this happen. Especially for certain political forces at this time.
But most likely it's an accident rather than conspiratorial design.
 

Archibald

ACCESS: Above Top Secret
Senior Member
Joined
Jun 6, 2006
Messages
6,947
Reaction score
5,824
Some (many?) social media users felt cut off from their friends. When asked why they didn't use the telephone feature on their toys, they cited telephone phobia - at least, according to the rumour I've heard ☹️

I red that "people re-discovered SMS" during the blackout.
...
Hmmmm, what ?
...
Is SMS obsolete ?
All the cool kids use Whatsapp for messaging. Old farts like me and my wife use Facebook Messenger. Grandpas use SMS :)

And this is completely silly, when you think about it - because SMS are not dependant from a freakkin' app or even the internet. Only the mobile network. Which never collapses except on December 31, midnight.
So as a 39 years old grandpa I feel smarter than you youngsters.

My 78 year old mom is 100% removed from the Internet, and considering her, hmmm, entrenched opinions, this is probably a blessing. She is blissfully unaware of all the craziness related to the Internet.
 
Last edited by a moderator:

UpForce

ACCESS: Secret
Joined
Feb 5, 2011
Messages
267
Reaction score
188
All the cool kids use Whatsapp for messaging. Old farts like me and my wife use Facebook Messenger. Grandpas use SMS :)

As far as I can decipher trends (ha) Signal, Slack and Discord are some of the services to be on. Whatsapp showed promise but from my POV, y'know, Facebook. Real "influencer" pros are of course scouring the landscape to register themselves on the newest and most obscure social media/messaging environments in the hopes of those becoming successful and thus gaining "OG trendsetter" points/status. The real added value (apart from increased data security and privacy in some cases) is usually meager to nonexistent; seems mostly to be about where the herd is stampeding to be data mined (or is it "milked" in this case). I mean, big data as currency is where the innovation is, as far as end user experience goes I struggle to make meaningful distinctions between current structures and erstwhile methods such as IRCs, mailing lists, Classmates, MySpace, et.al (apart from the capacity and volume of the net itself growing meanwhile).
 

Archibald

ACCESS: Above Top Secret
Senior Member
Joined
Jun 6, 2006
Messages
6,947
Reaction score
5,824
I just can't believe they chose such a name, for an app. DISCORD - O RLY ?
Then again, Tik tok is rather silly name - at least it quite appropriate for the stupid thing.
And Facebook souds like fessebouc: exactly ramsbottom.
 

Grey Havoc

ACCESS: USAP
Senior Member
Joined
Oct 9, 2009
Messages
16,866
Reaction score
6,481

Facebook has apologised after again reporting problems with its services, days after a major outage hit WhatsApp, Instagram and Facebook for hours.
The company said that a "configuration change" had impacted users globally.
It added that the incident was not related to the outage that saw its products taken offline for over six hours earlier this week.
Its Facebook, Instagram, Messenger and Workplace products had been affected, it said.
 

Similar threads

Top