A Brief Note on the CrowdStrike Outage and "Tech Influencer" YouTube
I fear things are heading in a bad direction.
We have ended up in a questionable state with "low-level tech influencer YouTube", if that’s the proper term for it. On the CrowdStrike exploit alone, I saw "explanation" videos that erroneously claimed:
Windows applications run in "ring 1" on x641,
The crash was caused by zeroes in a config file2,
The page fault was caused by accessing a field off a null pointer3,
User mode applications call kernel APIs by "raising an exception and waiting" for a kernel thread to “come along and look” at the arguments4,
Two of these are obviously false. The other two are probably false, but at the very least, you would need to provide your own crash analysis clearly showing them to be true in order to claim that they were in an “explainer” video, since the publicly available information we have suggests they aren’t.
Of the “low-level” videos I saw on the CrowdStrike crash, I don't think I saw a single one that was able to accurately represent the facts as they were known at the time the video was recorded. Most also prominently featured blatant errors about basic Windows system details. These errors were so rudimentary that I trivially spotted them myself, despite never having written a single line of kernel-mode code in my life.
Bizarrely, some seemingly pulled-out-of-thin-air mistakes appeared in multiple videos. The “ring 1” error was declared in two separate videos that I saw, even though they were on separate channels and did not reference each other. Best case, this is because it is somehow an easy mistake to make, and both people made it independently. Worst case, one of the two watched the other’s video and, not knowing anything about how x64 works, recorded their own video pretending to be an expert and regurgitated the same errors.
This is a bad trend. People should not be posting “explanation” videos on YouTube if they have only a vague understanding of the thing they're “explaining”.
You will notice that I did not put up a video on this topic. The reasons for that are hopefully obvious:
I have never worked on Windows kernel drivers, so I do not know the subtle details that you wouldn’t get from just reading the x64 architecture manual.
I don’t have access to the CrowdStrike driver binaries, data files, or crash dumps, so I can’t reproduce the crash and inspect the disassembly myself.
To me, those are the minimum two requirements for an “explainer” video on this topic. If you can’t claim both of those, you have no business posting an “explanation” at the current time. You need to wait until a complete, verified, technical analysis is posted somewhere, with all the details, so we know what actually happened. At that point, if you’re actually a knowledgeable low-level programmer (which a lot of these people don’t seem to be, but that’s a separate issue), there’s obviously value in trying to represent those details in a more approachable way to a wide audience.
I myself am willing to occasionally put up an “explainer” video on something I don’t work on if — and this is the important part — there is thorough, existing work on the subject that is highly technical and could benefit from a more accessible presentation. For example, a while back I did a video on the GoFetch attack where I attempted to do just that5. But I only felt comfortable taking a shot at that because we had an entire technical paper (with prior related papers) covering in minute detail exactly what was going on. Even then, I started the video with a big disclaimer explaining that I’m not a security researcher and was only covering what they disclosed in the paper.
I realize people like to hold themselves out as experts, even though they clearly aren’t, in order to try to build an audience and have a popular YouTube channel. But honestly, as I’ve said before, you aren’t going to make much money that way. Is a thousand dollars really so important to you that you’re willing to do widespread damage to the programming world by spreading erroneous information?
Plus, there’s other kinds of content you can make when there’s a hot topic like this that you don’t actually understand. You could have a guest on who does know something about kernel driver development, and have them explain it. You could make an “article reaction” video where you just bag on CrowdStrike for being terrible at deployment, which they very obviously are, and very much deserve.
There’s plenty of content you can make and put out in the world that can get clicks and views that doesn’t spread erroneous technical details in the guise of an “expert” “explainer” video6. If you feel like you have to capitalize on a particular technical news event, do one of these other types of videos instead. A few tech influencer folks did exactly this, and they should be commended for that! They avoided making the situation worse.
They run in ring 3.
Nobody has traced any part of the issue to zeroes in the config file. Zeroed files are a common occurrence that can have many causes, including if the file was being written during a crash, so it is highly questionable to claim that it was a root cause without further tracing. When CrowdStrike finally did post a breakdown, they confirmed that the zeroed files were likely caused by, not the cause of, the crash.
Nobody provided proof that a null pointer was to blame for the crash. Disassembly of the crash site showed a check for null just before the access, so it is equally plausible that the origination of the bad pointer was reused — meaning uninitialized — memory, not a field offset from a zero-valued pointer.
I'm not sure what exactly they meant by this. Charitably I will assume they meant "raise an interrupt", but even then, using an interrupt to call a kernel routine did not traditionally do a cross-core interrupt, so the core would not “wait” for anything to “come along”. It would jump directly to the interrupt handler and execute the kernel code immediately. Furthermore, Windows no longer uses INT to jump to kernel routines anyway, it uses SYSCALL because it is a more efficient way to call ring 0 code from ring 3 than using INT.
And even then, I had to read the original paper several times, read several references from the paper, and record the video two separate times to get the details right. Low-level code is complicated, and it’s very difficult to understand everything about code you didn’t develop yourself.
Yes, I’m running out of scare quotes at this point. This topic needed a lot of them.
This is such an obvious and important issue that it unironically influences what technology I use since these online noise generators (youtubers/bloggers) make it much harder to find valuable content.
In my personal experience it became much harder in the past few years to find useful in-depth information about rust/vim/linux (my "default" stack of choice) since these became more mainstream (you'd think that's good news..). The rate at which useful information is generated is much lower than the rate at which noise is created, and the noise is more popular for obvious reasons, making finding good information increasingly hard with time (since more views equals more visibility). Most of the good content I find (on youtube, which admittedly isn't the best place) always has less than 5000 views, and very often less than 500.
I've been thinking about this ever since you released that video explaining the exploit of the Apple's M-series chips. A lot of supposed experts on YouTube were making 10-15 minutes videos and not saying much more than "it's a problem in the hardware" while sometimes trying give brief explanations on some parts of modern CPUs. Then I watched your hour video where every 10 minutes you were saying "I am not an expert" while explaining in detail how the CPU parts interact with one another and end up manifesting the exploit. The difference between you and them is night and day! I know that those types of videos (like yours) probably don't get much views compared to the mainstream pop-science-y ones. This is the only explanation that I can give myself on why people are not investing more time and effort into making detailed in-depth explanations of these subjects -- "I can make, but no one would watch it".
All of this is to say -- thank you! You seem to have much harder job that pays back much less and I am glad that there are people like you that can pass on the in-depth knowledge they've acquired through laborious research and years of experience.