• If you enjoy the forum please consider supporting it by signing up for a NES Membership  The benefits pay for the membership many times over.

FB and IG down worldwide

Last edited:
My favorite comment overheard on Howie:

"Now do Twitter!"​

(Might have been Taylor; I fergit).

1633369707370-png.529225


Is it possible to overdose on schadenfreude? I think I might [devil]
<Insert Archer meme here>

About 20 years ago a cousin of mine singlehandedly took down Redmond (Microsoft). She was in a new cube and she was tasked with stress testing NT. The telecom guys mislabeled the plugs. She first realized there was a issue when Bill Gates (and 20 others) walked in her office. ...
I know a guy who took out the building Ethernet at DEC /LJ (Littleton).
He had a known shorted patch cord sitting on his desk
that he used to test the device driver on his own machine.
Some idiot in another cube needed a patch cord,
so he just helped himself to it and brought down the floor for hours.
[rofl2]

Ramenporn also stated that it wasn't an attack, but a mistaken configuration change made via a web interface.

[laugh]
life-in-hell-mistakes-were-made.jpg
 
Not being tech savvy, but reading about all those hackers that hold schools/hospitals/municipalities hostage for Bitcoin and such...

Why aren't places like FB/Google/ IG/Amazon/MySpace/Napster hacked day-in and day-out 24/7 ?
Lol google real time cyber threat map... you don't think they try

Fb probably gets attempted 35k times a second
 
My guesses:
Either a significant security breach was discovered
or
Some new algorithm was doing something which was supposed to be discreet a little too obvious for their liking and they needed to slam the emergency stop button before it got noticed
My bet. Database migration fail.
Been there in prod. Have the Scars.

Facebook is a pioneer in cache layer utilization. It's cool in that it means things are "instant" in the region of the world where they start, but trickle in to the "core" gradually. Also, it means that if someone facebook posts a night mare in progress in City Y, it can be deleted via a cache-clear in seconds before it goes past perminaintly.

I'm betting that trickery fought them and they looped. IE, pushed a new DC up, but didn't modify the BGP tokens properly to make it its a new space.

With the size of the DB. (I think Facebook useses vitess.io like youtube now)... A "roll back" of something simple could take half a day; easy.
 
My bet. Database migration fail.
Been there in prod. Have the Scars.

Facebook is a pioneer in cache layer utilization. It's cool in that it means things are "instant" in the region of the world where they start, but trickle in to the "core" gradually. Also, it means that if someone facebook posts a night mare in progress in City Y, it can be deleted via a cache-clear in seconds before it goes past perminaintly.

I'm betting that trickery fought them and they looped. IE, pushed a new DC up, but didn't modify the BGP tokens properly to make it its a new space.

With the size of the DB. (I think Facebook useses vitess.io like youtube now)... A "roll back" of something simple could take half a day; easy.
This, but it's obvious a contingency problem caught them off guard.... [rofl]
 
Back
Top Bottom