What do Prince Philip and Continuous Delivery have in common? In this month’s article, Mike Farahbakhshian takes on the cargo cult of DevOps. Suggested drink pairing: In honor of Pride Month, the Off the Wall.
Royal Pains
Poor Prince Philip. Things never seem to go the way the Duke of Edinburgh plans. He has a long history of putting his foot in his mouth.
His own grandson, Prince Harry, disregarded his advice not to marry Meghan Markle. One can only imagine his reaction to the conspiracy theory that Meghan Markle allegedly used a surrogate because Prince Harry is allegedly infertile. (Seriously, if you go down any of my linked-article rabbit holes, this is the weirdest conspiracy theory to fall into.) The Duke has an especially embarrassing track record of comments with Indigenous peoples of his own Commonwealth. A few examples:
- 1984: When accepting a figurine from a woman during a visit to Kenya he asked: “You are a woman, aren’t you?”
- 1994: “Aren’t most of you descended from pirates?” he asked an islander in the Cayman Islands.
- 1998: The Duke asked a British student who had been trekking in Papua New Guinea: “You managed not to get eaten then?”
- 2002: To Australian Aborigines during a visit to Australia with the Queen he asked: “Do you still throw spears at each other?”
Despite this very bad PR with the Indigenous demographic, there is an island in the South Pacific named Tanna in Vanatau where Prince Philip is considered a divine figure. The islanders believe he will one day – if they are good and righteous — live among them. The Duke is aware of the Prince Philip Movement and has in fact sent signed photographs to his worshippers. In return, the islanders believe that Prince Philip’s magic allowed a black man to become President of the United States and to find Osama bin Laden. Pretty impressive for somebody who has no mind-to-mouth filter.
The Prince Philip movement is an example of a cargo cult. Another analogous religion, also based in Tanna, Vanatau is the John Frum movement. John Frum is considered a divine figure, likely based on a soldier, or soldiers (“John from America”), stationed in the Pacific Theater during World War II. As part of the worship of John Frum, the natives of Tanna will paint themselves in the colors of the American flag, hoist it, and perform military marches every February 15th for John Frum Day. John, who appears in the form of an American GI, is a guardian of the true and pure ways, called kastom (“custom”). Don’t worry, Squids, you haven’t been left out; there’s a Navy flavored cult called the Tom Navy Cult. John Frum and Tom Navy, much like Prince Philip, are prophesied to eventually return and bring an era of prosperity and salvation where everyone will adhere to kastom.
Hazardous Cargo?
These cults, known as cargo cults, have been studied for decades by anthropologists and sociologists. I’m fascinated by cargo cults for three reasons:
First, cargo cults reflect a cultural value of sharing over force projection: In Melanesian, Polynesian and Micronesian cultures, there is a “Big-Man” social system. In a Big-Man system, the highest-status person is the one with the most gifts and largesse to give (“cargo”), versus the one that imposes the most rules. This even applies if the gift cannot be moved, such as Rai Stones. (In the case of Rai Stones, an oral history of transfer is sufficient to prove ownership, just like blockchain.) Cargo cults developed as a way to return to the Big-Man system (kastom), and encourage support of the community. Bernie Sanders jokes aside, I think that valuing sharing and generosity as much as, or perhaps more than, ruthless ladder-climbing is something we need more of in our society. Gordon Gekko may have famously said “Greed is Good,” but in small, insular communities like islands, where blood feuds can devastate populations, no one has the luxury of greed.
Second, cargo cults represent magical thinking, which I covered in an earlier article. One of the best books on the matter is The Golden Bough (public domain link) by Sir James George Frazer. Specifically, they represent sympathetic magic. By using symbolic imitations of objects like rifles and flags, and of actions such as rifle drills and landing planes on fake airstrips, there is an intent to bring prosperity (“cargo”) back.
Finally, cargo cults emerge as a result of crisis. Under conditions of social stress in small, tight-knit communities such as an island, a charismatic leader inspires the movement. Usually, the leader has a prophetic vision of the future. The future promises that by returning to simpler, traditional ways, where you say what you mean and do the right thing, an ancient perfect state (“mana”) will return. The charismatic prophet proposes a dismantling of the old social hierarchy and rituals, and removing ego to support the health of the society (“kastom.”) Magical thinking is performed, imitative rituals to bring back prosperity are conducted.
We can be fascinated by these cargo cults – or, perhaps, if you are Prince Philip, be venerated by them while inadvertently patronizing them – but I wouldn’t laugh at them. This is because cargo cult thinking permeates many aspects of our personal and professional lives. Namely, the way the Federal Health IT community practices DevOps is a cargo cult. Think about it:
- Crisis: IT has become complex. Gone are the days you had a workgroup server that handled all your needs, and maybe a web or email server for external communication. Everything’s scaled now, and because the first CIOs were CTOs and the first CTOs were usually CFOs, every activity got distilled to a line item. This has permeated Federal culture, with budget line items for Operations and Maintenance (O&M) and Development, Modernization, and Engineering (DME.)
- Charismatic leaders: The evangelists of DevOps, Continuous Deployment, Human Centered Design and Product based Focus are exciting, engaging, and peripatetic. They don’t deliver presentations, they emulate TED talks.
- Prophetic Future of the Ancient Perfect State: In the future, everything will be in the cloud, eternally young and invulnerable to power losses and hardware failures. Yet it will be like the past, when everything was as simple as deploying something to a workgroup server. You wouldn’t need a hundred meetings and a thousand checklists, because if it didn’t work, you’d just tweak it.
If you were born when this song came out, you’ll be old enough to vote in the 2020 election. Congratulations, now get off my lawn. - Repeat the Rituals: Even if you don’t need them or know what they mean. Automated build and deployment pipeline. Put it in the cloud. Infrastructure as code. Containerize and orchestrate. Automated testing. Microservices. Dark Launch. Rinse and repeat.
But do the rituals get you to your ancient perfect state? What good is automated testing if it generates gigabytes of logs no one reads? Do poorly maintained microservices trump an impeccably maintained and documented monolithic service? Why can’t DevOps work on-prem? (Spoiler: It can.) How is Infrastructure as Code any different than distributed configuration tools that have been around since the 1990s?
Culture Shift Without a Clutch
It’s easy to adopt the trappings of DevOps (or Agile, or Human Centered Design, or philosophy du jour.) A few open-plan bullpens with video Kanban boards and endless snacks: You’re just like Google now! Hey, you use Docker and Kubernetes! You must be a DevOps shop! For the Federal Health IT community, that’s how we end up with “scrum-fall” models. Here are a few reasons why:
Your kastom isn’t your mission: The concept of iterative work has been around since the Deming model, which you probably know as the Plan-Do-Check-Act cycle. DevOps just stretches out Do quite a bit. But does your organization practice what it preaches when it comes to iterative feedback? Do you use your lessons learned in the right way? When certain watermark metrics are used as a goal, rather than a canary in a coal mine to indicate intervention is needed, then retrospectives and lessons learned become pro forma bullshit exercises. Here’s an example: If an audit is coming, and your service desk is directed to hurry up and close tickets in a timely fashion so they won’t get “dinged” on the audit, that’s the wrong thing to do. If tickets are open too long, it may be a sign you need more service desk personnel. A few can burn the midnight oil so your department doesn’t get flagged, but that burns out your resources. Sometimes, lackluster numbers are a good thing: It teaches you where to reallocate resources. Pain and hunger feel bad, but they’re signs you need to take it easy or eat something. They are not to be minimized at all costs: That’s how people end up killing themselves. Teachable moment: Learn from your lessons learned. That means taking the bitter pill.
Know your cultural starting point: This dovetails with the last point. Some cultures aren’t going to change as easily and no amount of brown bag lunches or bloviation from the top will speed that process. This is especially important when modernizing legacy IT infrastructure with aging development communities (think MUMPS and COBOL). This is especially true in the Federal Health IT community, which combine the two cultures the most resistant to change: Government and Health. This means knowing how much of your charismatic whiz-bang flash will resonate and how much will deter. At the Veterans Affairs Advance Planning Brief to Industry (APBI), I heard a lot of the VA DevOps folks use military-inspired terms like “Joint Operations Center” and “Project Special Forces” to refer to product deployment. Over and over. I am sure this sounds great among the young, hip, DevOps folks to indicate mission focus and action, but I don’t think this resonated as well with a room full of service-disabled Veterans, many of whom are extremely disabled and have seen the horrors of war. I was sitting next to a former Colonel and battalion commander whose eyes rolled so hard every time he heard one of these phrases that I wondered if I needed to put a spoon in his mouth. I think that turned a lot of good people off a good message because of less than stellar word choice for messaging. Knowing your audience is key – will this lesson learned be internalized?
Know the Federal culture: A big problem with the change agents is a general ignorance of the Federal Acquisition Rules (FAR). There are some things that are very difficult to circumvent. The division between O&M and DME dollars is one, since that comes from Congressional mandate. The proscription on developing requirements and working to those requirements is another. This is the biggest reason why most Federal IT is stuck in a project model and not a product model. Yet why? Here’s some context that might help: The “thou shalt not write and work to requirements” is of paramount importance when working on durable goods like battleships. No one wants a vendor who cuts corners on a weapon of war, or who fleeces the Government during a time of war. Do Digital Service type websites suffer from that problem? Probably not, although in an era of cyberwar, who knows? Website development, and certain apps are better suited to a collaborative development model which blurs lines between O&M and development, because the end product is worth more than meeting certain contractual relationships. Yet the Government is, at its core, the ultimate arbiter of life and death within its domain. The Government’s ultimate missions of war and peace is above all else. That’s where the boundary-laden, slightly adversarial Federal contracting relationship originated. It’s reinforced by the fact that Congress appropriates budgets and Congress hasn’t shifted to an Agile model. Maybe it is time for a paradigm shift. However, that paradigm shift must come from an understanding of the FAR and Congressional appropriations cycle, not a handwaving dismissal of it. If the kastom needs to change, it must change from within. Write your Congressperson.
Make Signal-Driven Decisions, not Noise-Driven Decisions: DevOps tools provide a lot of data to make data-driven decisions. I guarantee you that 80-90% of that data is ignored. It’s easy to become numb to reading build and test logs and glean insights from them. What good is a highly verbose system or application log if no one ever reads it? I’m guilty of this – I have an empty server that sends me IDS reports every day and I haven’t read one in years. I am sure I have been compromised a thousand times over but I’m just too busy to read two to three thousand lines of text every morning. This is matter of maximizing the signal-to-noise ratio in all that data. It’s also a matter of chunking that data. Problems detected early and often are problems that don’t propagate to later in the build pipeline.
Cargo Comes From Within
These are some things I suggest to help your DevOps implementation achieve more results, and not have you go through the motions.
First, I suggest that every DevOps embed automation engineers into every feature scrum team. Don’t have a separate automation team or group. Get the Dev and the Ops side of DevOps really rubbing elbows, so important data can be shared holistically and not ignored.
I strongly encourage call-and-response activities during your scrums and scrum-of-scrums to make sure people don’t zone out while they wait for their turn to speak. If people aren’t using the scrums to know what everyone is doing, then it’s pro forma bullshit and a waste of time.
I think it’s key to incentivize your developers to write more and better unit tests that meet acceptance criteria. Test Driven Design and Design for Testability are buzzwords I see thrown around in proposals a lot. Encourage developers to use this with positive reinforcement like bug bounties, or team building activities when a build is defect free. Free beers for every minute you shave off your deployment time! This helps build team cohesion and a stronger, more effective pipeline.
Finally, I think it’s important to keep your cadence and not try to struggle to meet deadlines. Failure is important as a warning light. It lets you know you need more developers (or automation engineers, or testers, or …) Sprinting, pun intended, to meet a deadline is a way of covering up a systemic issue with personal sacrifice. If the Federal Health IT community wants to really embrace a product-based model, they need to know how many people and what skillsets it will take.
And that’s it. Have a happy proposal season and may you win all your bids (unless they’re our incumbent contracts, in which case go take a hike). One final note: Often, I close out my column with a music video. This month, in celebration of Pride Month and the 50th anniversary of the Stonewall Riots, and to offset the insensitivity of Prince Philip, I close out this article with Miike Snow’s Genghis Khan. I may know nothing about being LGBTQ, but I know a lot about wanting to be a Bond villain. (It was my first career choice before my guidance counselor told me to go into IT.) Based on that I suspect everyone has something they can learn from this video.
Something to learn for everyone, flashy automation, attempts at world domination, and a beat you can march to: Now isn’t that the real spirit of DevOps?
