😦 The Real CrowdStrike Flaw Was In Deployment

-
By now, most of you are familiar with the story of CrowdStrike and the resulting worldwide outages. The initial, and angry, responses were along the lines of "Where are the devs? Where was QA?"

As it turned out, the actual patch, or software code, was fine. The problem was in the config file that governs the behavior of the patch.  Being flawed, it triggered the Microsoft Windows' blue screen of death. In a variation of "Who Watches The Watchers," the config file was indeed tested, but CrowdStrike's testing system was unknowingly broken.  The config file passed when it shouldn't have.

Nothing is perfect all the time, but what would have mitigated the fallout would have been a gradual or staggered rollout of the config file instead of the apparent "big-bang" release to all machines at once. The staggered rollout is an old and reliable technique. I can think of 2 reasons why it might have been forgotten or abandoned: inexperience or over-confidence.

When I worked at the NYSE, config file releases were, at minimum, on a 3 day schedule, no matter how trivial the change.  It meant more vigilance and operations monitoring over 3 days, but the culture aimed for 99.9% uptime. Sure, we had our periods of inexperience (technology changed rapidly), and we had characters with grand hubris (what company doesn't?).  What likely saved us was an attitude summarized by a project manager I knew there. He shared the time a developer fixed a bug and declared, "It was easy. Just one line of code."

The project manager knew it would be neither.

 






Comments

Post a Comment

Popular posts from this blog

MR2 Check Engine

-
Image
Shown here is the 1991 Toyota MR2. Thanks to the mechanical skills of a family enthusiast, it purrs, growls, and turbo whistles like a dream.   My hardware ability doesn't go much further than oil changes. Rather, my expertise is in software, and I wrote an Android App (iOS version is in progress) to diagnose the flashes of the check engine light. This second generation MR2 predates the OBD2 connector, a standard used to diagnose engine, transmission, and emissions problems quickly and easily. Instead, we have a flashing check engine light which needs some deciphering, followed by a lookup in the service manual. Basically, you count flashes to arrive at fault code number. A short pause (~1.5 seconds) indicates the start of the next digit. A medium pause (~2.5 seconds) indicates start of the next fault code. And a long pause (~4.5 seconds) means the sequence has ended and will repeat. The video below corresponds to the codes 41 and 51. Using the the MR2 Check app and selecting the 3
Read more

Bookshelf: UNIX A History and a Memoir

-
Image
  A man page will give you, in black and white, specific information about UNIX. Brian Kernighan's new book adds color and depth about UNIX you won't find anywhere else. I'm not a gray beard (yet), but I imagine they would plow through the book knowingly, but at the same time, learn a few things. Those new to UNIX -- recent college grads or those experimenting with Raspberry Pi -- would be better off starting with Chapter 8 which describes more relatable events and the descendants of UNIX, including Minux and Linux. It's when they start to ask questions like "Where does UNIX come from?" and "Why is Gamora grep?" that they should start from the beginning: Bell Labs, 1969. Kernighan opens with a birds-eye-view of Murray Hill New Jersey, zooms in on the buildings, the offices, the significance of Center 1127, and offers a brief look at some of the key players there. As I progressed through the book, the two factors that made UNIX not just possible, bu
Read more

Bookshelf Classic: The C Programming Language

-
Image
My first job had me programming in Microsoft BASIC for the IBM PC (DOS).  BASIC worked well enough, but its limitations were clear.  The language was interpreted and therefore slow.  More importantly, it wasn't a modern structured language, and instead, relied on line numbers and the GOTO statement.  Anyone who has read Dijkstra knew  GOTO was a bad thing . Having learned a structured language in college ( PL/I ), using BASIC felt unnatural.  When a C compiler became available for the PC, I saw a chance to improve and modernize our software.  The problem was selling the idea -- a problem made harder because I wasn't fluent in C. "It would be a staffing problem.  Not many people know C, but we can find a lot of programmers who know BASIC," noted one manager. The argument was strong as my knowledge of C was weak.  But I knew that C, by design, was a small language and thus easy to learn.  "It has about 30 keywords," I proffered to another manager. U
Read more