On Friday, as we were running around the hospital where we work trying to get every computer working again, we were following the work-around to rename the Crowdstrike folder under C:\Windows\system32\drivers to “bad-CrowdStrike”.
When my coworker was typing the rename command, instead of typing “cro TAB”, he started typing “clo TAB”. He’d ask me why it wasn’t finding it, and I’d point out the typo.
I started saying, it’s not “CloudStrike”, it’s “CrowdStrike”.
By the end of the day, we were both a little loopy. I started typing “CloudStrike”, and cursing him out for screwing with my head. By the end of the day I wasn’t sure what it was either.
CloudStrike
CrownStrike
ClownStrike
It occurred to us that CrowdStrike is an absolutely terrible name. It sounds like a terrorist attack. Of course, it felt like one on Friday.
Yeah, I’m usually a big stickler for making sure I’m saying something right, but that name was tongue twistering me from the first time I tried to say it out loud. And we don’t even use them and weren’t hit in any way lol
It occurred to us that CrowdStrike is an absolutely terrible name. It sounds like a terrorist attack. Of course, it felt like one on Friday.
When I first heard about what was going on, I assumed that “CrowdStrike” was not the name of the software/company, but rather some sort of advanced DDOS-like attack where they used systems they’d previously hacked and had them all do the same thing at once to another target.
Doesn’t Microsoft allow crowdstrike to make updates? Being such a critical part of the OS it’s up to Microsoft to ensure their procedures are robust and being followed.
How do you implement that? How is it feasible that Microsoft tests all the third party drivers?
Don’t get me wrong I believe Microsoft is partly to blame for this problem as well but for making it so hard for system admins to go around the system and solve things (as compared to Linux where you can do anything). I think sys admins would have solved this much faster if they were using Linux systems
I was just probing your argument because I guessed it was the typical nonsense of Microsoft bad, Linux good, without a good explanation
I think if it’s going on every windows computer windows should have a process in place to prevent what happened from happening. Windows are for profit, they have the money to do it right but they got greedy. A staggered rollout would have prevented most of it and is a very simple thing to require.
Also if it’s going on every windows computer or most I wouldn’t consider that a third party anymore even if that’s how they keep liabilities at arms length
It’s not, its just popular. Its not windows job to police what software you choose to run on it.
However Windows does actually have an optional certification program called WHQL for kernal level drivers. Getting this certification lets updates get posted via windows’ internal updater. It checks the driver calls apis correctly and doesn’t misbehave with interrupt handling among other tests. Crowdstrike driver did pass this, and in fact there was no bug with the driver, the bug was with the configuration file. The configuration file updates about once an hour (and it really needs to do that), and does so outside the windows update process, making windows powerless to control its rollout. whql certification takes a few days to run and configuration files aren’t really in scope.
Same thing would happen on Linux if someone wrote a bad kernel module and integrated it into the OS. In fact, Crowdstrike did have a similar problem a few months ago on Linux.
I’m no fan of Microsoft, but this isn’t their fault.
That is true. The issue is that because there are so many permission escalation issues in windows, that many anti malware products must run as kernel drivers.
Crowdstrike released bad code into prod without giving it some hours of testing in local machines or whatever. Incredible fuckup, inimaginable. But, let’s not take blame out of Microsoft, if a driver is faulty the system should be resilient enough no to crap the bed on login. At least enough for IT to be able to remotely access the system and fix it. The manual work the IT world has had to do because it’s lost remote access to workstations is insane.
To be fair, kernel level access by third party software is kind of frowned upon in the Linux world. Ask any desktop Linux user how they feel about NVIDIA (the only third party kernel code an average Linux user will install) and their drivers randomly causing strange issues on their systems up to and including kernel panics compared to the experience on AMD where the driver is open and built into the kernel itself. For security software that needs low level visibility, there is eBPF, direct kernel level access isn’t needed (though I believe CrowdStrike uses it, and thay actually did CrowdStrike Debian and Rocky Linux systems some time back).
MacOS blocked the majority of kernel extensions a few years ago as well.
Windows is the only OS where it has been designed in a way where kernel level access is the rule rather than the exception. So design flaws are at least partially at fault here.
Heard from someone else (so take it with a grain of salt) that CrowdStrike and/or similar companies threatened Microsoft with an antitrust suit when Microsoft tried to force them to use an API instead of working directly with the kernel.
The opinion of Linux desktop users (or any users really) do not count in the enterprise world. Somehow, if management bought in on the Crowdstrike rootkit bandwagon, you’ll see it on corporate hardware. It doesn’t matter if it’s a bad plan; it doesn’t matter if it gives an American company a backdoor to all you infrastructure; if the CISO decides everyone gets it, everyone get it.
The only thing you can really do as a lowly employee is keep any such device away from any personal info or network as if it’s infected by malware (which I would argue is exactly what it is).
Basically, crowdstrike wrote bad code that run as a driver, windows doesn’t like bad code in their drivers. Kernel level code is generally expected to run properly. crowdstrike’s kernel level code was really bad. Embarrassingly bad.
If the host creates a playlist and everyone can add their favorite song to the playlist, the host won’t be blamed if you add “erika”. People rightfully think you are an ignorant weirdo or a bad person, not the host.
Except that the playlists are super complex and there is no way to make sure. Like building an engine and having to make sure that no 3rd party accessory will break it. Like the parented “sand injector”.
The problem with CrowdStrike’s solution is that they got csagent.sys driver signed by WHQL, and the driver will download p-code from the internet and execute it. This allows them to push out changes without waiting for Microsoft approval.
The biggest problem occurs when you don’t sanitize your inputs and someone accidentally uploads a blank file padded with zeroes. The driver dereferences a null value, and crashes your system. Hard.
I don’t want to argue with you and I admit that my phrasing wasn’t ideal but I assumed that it was obvious that i was talking about everything that would be executed on the machine. Apparently it wasn’t.
When I heard “CrowdStrike” took down operating systems everywhere, I thought it was the name of a virus or a group of hackers. I’m not the only one hearing an inherent villainy in that name, right?
Oh, if you worked at a company that uses them (which is a lot of companies), you’d definitely be familiar with them as they hog up a ton of fucking CPU/disk. I basically had an entire CPU core dedicated to running their bullshit.
I dunno how blind I have to be to miss this for all this time, but the air vents they have on the pit wall are the same air vents you get on a Mercedes-Benz.
Something I heard about recently is that it’s unnervingly common for the stock prices of unknown but really important companies like these to shoot up following an outage because it reveals to stock investors how mich of a monopoly it has in an area.
Are you not in the US? Equifax is a credit bureau and if you’ve never heard of them, you never needed credit or you’re not from the US.
The other three, I’ve only heard of Ashley Madison because they had a very aggressive ad campaign before ad blockers became ubiquitous. One could say it was ads like theirs that made ad blocking a requirement.
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !programmerhumor@lemmy.ml
Post funny things about programming here! (Or just rant about your favourite programming language.)
Rules:
Posts must be relevant to programming, programmers, or computer science.
No NSFW content.
Jokes must be in good taste. No hate speech, bigotry, etc.
On Friday, as we were running around the hospital where we work trying to get every computer working again, we were following the work-around to rename the Crowdstrike folder under C:\Windows\system32\drivers to “bad-CrowdStrike”.
When my coworker was typing the rename command, instead of typing “cro TAB”, he started typing “clo TAB”. He’d ask me why it wasn’t finding it, and I’d point out the typo.
I started saying, it’s not “CloudStrike”, it’s “CrowdStrike”.
By the end of the day, we were both a little loopy. I started typing “CloudStrike”, and cursing him out for screwing with my head. By the end of the day I wasn’t sure what it was either.
CloudStrike
CrownStrike
ClownStrike
It occurred to us that CrowdStrike is an absolutely terrible name. It sounds like a terrorist attack. Of course, it felt like one on Friday.
Yeah, I’m usually a big stickler for making sure I’m saying something right, but that name was tongue twistering me from the first time I tried to say it out loud. And we don’t even use them and weren’t hit in any way lol
When I first heard about what was going on, I assumed that “CrowdStrike” was not the name of the software/company, but rather some sort of advanced DDOS-like attack where they used systems they’d previously hacked and had them all do the same thing at once to another target.
As we’ve discovered, both can be true.
A fitting rename after such a pathetic and catastrophic failure, that’s for sure.
Yeah, let’s not key them get away with it. Clownstrike forever on
CloudStrike
CrownStrike
ClownStrike
ClownStrife
Ah yes, the guy from FFVII
CloudStrife
CrowdStroke
CrowdStricken
Now that’s a good code word!
They struck the whole crowd for sure
So why is this considered a crowdstrike issue and not a Microsoft fuckup?
Can you explain why you think this is a Microsoft issue?
Doesn’t Microsoft allow crowdstrike to make updates? Being such a critical part of the OS it’s up to Microsoft to ensure their procedures are robust and being followed.
How do you implement that? How is it feasible that Microsoft tests all the third party drivers?
Don’t get me wrong I believe Microsoft is partly to blame for this problem as well but for making it so hard for system admins to go around the system and solve things (as compared to Linux where you can do anything). I think sys admins would have solved this much faster if they were using Linux systems
I was just probing your argument because I guessed it was the typical nonsense of Microsoft bad, Linux good, without a good explanation
I think if it’s going on every windows computer windows should have a process in place to prevent what happened from happening. Windows are for profit, they have the money to do it right but they got greedy. A staggered rollout would have prevented most of it and is a very simple thing to require. Also if it’s going on every windows computer or most I wouldn’t consider that a third party anymore even if that’s how they keep liabilities at arms length
It’s not, its just popular. Its not windows job to police what software you choose to run on it.
However Windows does actually have an optional certification program called WHQL for kernal level drivers. Getting this certification lets updates get posted via windows’ internal updater. It checks the driver calls apis correctly and doesn’t misbehave with interrupt handling among other tests. Crowdstrike driver did pass this, and in fact there was no bug with the driver, the bug was with the configuration file. The configuration file updates about once an hour (and it really needs to do that), and does so outside the windows update process, making windows powerless to control its rollout. whql certification takes a few days to run and configuration files aren’t really in scope.
Thanks for the info, i didn’t know that and understand it a bit better now.
Same thing would happen on Linux if someone wrote a bad kernel module and integrated it into the OS. In fact, Crowdstrike did have a similar problem a few months ago on Linux.
I’m no fan of Microsoft, but this isn’t their fault.
An OS should not have to require a 3rd party driver for security.
Microsoft should be writing that driver as an OS component. Drivers should be restricted for taking to hardware.
I thought only people who subscribed to CrowdStrike’s services had that driver installed.
That is true. The issue is that because there are so many permission escalation issues in windows, that many anti malware products must run as kernel drivers.
Windows: exists
Crowdstrike: stabs
You: why would Microsoft stab themselves?
Windows: exists
Crowdstrike: exists
Windows: open belly, right here!
Crowdstrike: stabs
Crowdstrike released bad code into prod without giving it some hours of testing in local machines or whatever. Incredible fuckup, inimaginable. But, let’s not take blame out of Microsoft, if a driver is faulty the system should be resilient enough no to crap the bed on login. At least enough for IT to be able to remotely access the system and fix it. The manual work the IT world has had to do because it’s lost remote access to workstations is insane.
Nelson Muntz: Stop striking yourself!
To be fair, kernel level access by third party software is kind of frowned upon in the Linux world. Ask any desktop Linux user how they feel about NVIDIA (the only third party kernel code an average Linux user will install) and their drivers randomly causing strange issues on their systems up to and including kernel panics compared to the experience on AMD where the driver is open and built into the kernel itself. For security software that needs low level visibility, there is eBPF, direct kernel level access isn’t needed (though I believe CrowdStrike uses it, and thay actually did CrowdStrike Debian and Rocky Linux systems some time back).
MacOS blocked the majority of kernel extensions a few years ago as well.
Windows is the only OS where it has been designed in a way where kernel level access is the rule rather than the exception. So design flaws are at least partially at fault here.
Heard from someone else (so take it with a grain of salt) that CrowdStrike and/or similar companies threatened Microsoft with an antitrust suit when Microsoft tried to force them to use an API instead of working directly with the kernel.
I’m so glad i got rid of my nvidia card. Having to reinstall the divers and kernel-headers every time my kernel updated was getting old.
The opinion of Linux desktop users (or any users really) do not count in the enterprise world. Somehow, if management bought in on the Crowdstrike rootkit bandwagon, you’ll see it on corporate hardware. It doesn’t matter if it’s a bad plan; it doesn’t matter if it gives an American company a backdoor to all you infrastructure; if the CISO decides everyone gets it, everyone get it.
The only thing you can really do as a lowly employee is keep any such device away from any personal info or network as if it’s infected by malware (which I would argue is exactly what it is).
deleted by creator
Basically, crowdstrike wrote bad code that run as a driver, windows doesn’t like bad code in their drivers. Kernel level code is generally expected to run properly. crowdstrike’s kernel level code was really bad. Embarrassingly bad.
If the host creates a playlist and everyone can add their favorite song to the playlist, the host won’t be blamed if you add “erika”. People rightfully think you are an ignorant weirdo or a bad person, not the host.
OTOH, if you build a playlist manager for playlists everyone can add to, you make sure nothing anyone adds will break it…
Except that the playlists are super complex and there is no way to make sure. Like building an engine and having to make sure that no 3rd party accessory will break it. Like the parented “sand injector”.
Well do you want to have Microsoft approving EVERY driver for windows? Rip 3rd party open source drivers for retro hardware
What do you think WHQL is?
The problem with CrowdStrike’s solution is that they got csagent.sys driver signed by WHQL, and the driver will download p-code from the internet and execute it. This allows them to push out changes without waiting for Microsoft approval.
The biggest problem occurs when you don’t sanitize your inputs and someone accidentally uploads a blank file padded with zeroes. The driver dereferences a null value, and crashes your system. Hard.
I don’t want to argue with you and I admit that my phrasing wasn’t ideal but I assumed that it was obvious that i was talking about everything that would be executed on the machine. Apparently it wasn’t.
Ahh. Approving every piece of software would make them… Apple.
You did say “driver”, and Microsoft typically approves every single driver on the majority of PCs.
They were legally not allowed to as part of an agreement to not be s monopoly and allow competition.
funny thing is I, and probably most people, had never even heard that there was something called “CrowdStrike” until Friday of last week
When I heard “CrowdStrike” took down operating systems everywhere, I thought it was the name of a virus or a group of hackers. I’m not the only one hearing an inherent villainy in that name, right?
Oh, if you worked at a company that uses them (which is a lot of companies), you’d definitely be familiar with them as they hog up a ton of fucking CPU/disk. I basically had an entire CPU core dedicated to running their bullshit.
I’m a Formula One fan. The Mercedes team are sponsored by them. You see their logo every time you see an on board shot of the cars.
I had no idea until this weekend.
Over/Under on the amount of time before that logo gets removed?
End of season, at least, or whenever the livery/sponsorship contract runs out. Mercedes isn’t liable to care too much - it’s just a livery.
Well you say that but their pit wall on Friday said otherwise
I dunno how blind I have to be to miss this for all this time, but the air vents they have on the pit wall are the same air vents you get on a Mercedes-Benz.
It makes perfect sense, but it’s rather amusing 😂
I knew of falcond as the service that makes my work mac run slow.
Unfortunately, having a mac meant i didn’t get friday off unlike most of the rest of the company
Same for me with Solar Winds, Equifax, SVB and Ashley Madison.
Weird to think that some kind of major catastrophe in the future could again be caused by some company that exists right now, but am unaware of.
Something I heard about recently is that it’s unnervingly common for the stock prices of unknown but really important companies like these to shoot up following an outage because it reveals to stock investors how mich of a monopoly it has in an area.
Are you not in the US? Equifax is a credit bureau and if you’ve never heard of them, you never needed credit or you’re not from the US.
The other three, I’ve only heard of Ashley Madison because they had a very aggressive ad campaign before ad blockers became ubiquitous. One could say it was ads like theirs that made ad blocking a requirement.
Yeah, I’m not in the US. The first time I heard about them was when they shit the bed.
Linux with eBPF:
Yes. I’m no security expert, but ebpf always seemed a bit weird to me. But in the end how much different is it from kernel drivers?
Shush, this is an opportunity for people to dump on Microsoft, if you take it from them they’ll turn on you.