In this month’s article. Mike Farahbakhshian shows how ancient superstitions live once more, as massive distributed systems like NoSQL and IoT evolve beyond data collection into distributed decision making. See how controversial anthropology and dystopian space operas show us how to envision this brave new world and why. Estimated reading time: 9 minutes. Suggested drink pairing: Garnacha or Priorat
“Any sufficiently advanced technology is indistinguishable from magic.” – Arthur C. Clarke
In the grim future of the year 40,000, the galaxy is in a state of endless war between species. Humanity is lashed together in a vast interstellar Imperium that is corrupt and fraying at every seam. It is uncertain if their leader, the God-Emperor of Humanity, is dead, or comatose. One of his last acts was to ban artificial intelligence, and thus all technology has been delegated to the cybernetically enhanced Tech-Priests of Mars, the Adeptus Mechanicus.
These priests forge and repair all technology, but its understanding has been lost to time. They invoke “machine spirits” before using every tool, lest the “machine spirit” become enraged and cause a malfunction of a weapon or spaceship. Are the spirits real or mere superstition? Are they fragments of a contraband artificial intelligence or something far more alien and sinister? No one knows.
Okay, so this isn’t real: this is from the sci-fi schlock-fest that is Warhammer 40,000. If you haven’t experienced it, imagine if Alejandro Jodorowsky actually delivered his failed attempt at a Dune movie… after a bender with Salvador Dali. Yet within this pile of dung is a piercing insight into human psychology and an equally shocking revelation into the future of Big Data.
The revelation? We must placate the Machine Spirits.
Why? Distributed logic and the intertwining of software and data. The era of Big Data is over. The era of Big Logic has begun.
*crickets*
Okay, I’ll explain this in more detail. Let’s begin this wild ride not 40,000 years in the future, but 40,000 years in the past.
Household Gods: Animism, Magic, and the Delegation of Control
Between 1890 and 1915, Sir James George Frazer wrote a controversial book called The Golden Bough: A Study of Magic and Religion. It is an exhaustive study of superstitions, rituals, and beliefs among a variety of peoples: Stone Age hunter-gatherer tribes, Bronze age city-states, and civilizations of antiquity.
The thesis is that society progresses in a uniform fashion: tribes of people initially believe in an animistic world, in which everything is imbued with a spirit. As society becomes more hierarchical, certain tasks are delegated to shamans, and finally religious clerics. Yet certain atavisms of the earlier belief systems remain. The tamest example:[i]: The term pontiff, which comes from an Archaic Latin word for “bridge,” refers to an ancient Roman belief that the spirits of bridges required placation by a religious caste to keep the bridge sound.
The key takeaway: the world of early humans was decentralized and individual people had very little control of events. A tree branch might fall, and with no scientific understanding of what would cause it, the data (a tree branch was attached to the tree at moment X, and is on the ground at moment Y) is conflated with the control (something made the tree branch fall). If an apple falls off a tree and hits you on the head, and you have no idea of the science behind it, you’d rightfully conclude that the tree intentionally dropped the apple. Why: Self-defense? A gift? A warning? That is up to interpretation, and the origin of many superstitions.
Early human language reflects this obsession with animacy and intent[ii], and a few relics survive in modern English: the verbs look versus see, or listen versus hear. Fans of historical linguistics — all three of you poor fellows — will know that there were two Proto-Indo-European words for “fire[iii]” and “water[iv],” one describing the material and one describing the material in action.
As technology advanced to allow people more control of their environment, society in turn became more hierarchical and controlled. While each individual person had more power over his or her environment, they in turn were ruled by family, community, and state. Belief systems reflect this: the Roman household gods of the pantry (the panes) and the household (lares) were subordinate to the state gods (Jupiter, Juno and Vesta). For much of Japanese History, the Shinto spirits (kami) were subordinate to the state and Emperor.
“Spirits” Are Just Big Data with an Agenda
The conclusion we can reach here is that when humans lived in a world where events occurred and they had no control over any of them, they conflated the data of the event with the action or intent. This conflation was envisioned in the form of an animate spirit. As society advanced, the paradigm changed: spirits were replaced by more centralized and hierarchical belief systems. These systems reflected our understanding of science, our use of technology and our more centralized and hierarchical societies.
So what’s changed? Well, three things are coming together in a perfect storm.
- For the first time in human history, technology is increasing but control is DE-centralizing. Big Data paradigms like MapReduce, operate on the expectation that no part of the whole could possibly, have a working understanding of the entire problem. The Internet of Things (IoT) is equally decentralized, especially as peer-to-peer meshes such as blockchain are leveraged to ensure reporting of metrics, in real time, asynchronously.
- Data is merging with control logic once again. This time, it isn’t due to a lack of understanding of the world. Rather, this time, it’s by Moreover, this isn’t a new concept! MUMPS based systems are prescient in their use of a NoSQL based data storage paradigm that allows for immediate module-based insights. Systems like VistA are a natural fit for migration to a modern NoSQL implementation like Hadoop; people have suggested this for some time. The only limitation is the poor coding standards.
- Finally, machine learning is coming into its own, allowing a distributed system like IoT to be used more than simply reporting of metrics. Each node in an IoT mesh can leverage the distributed network and the NoSQL data storage to perform independent data processing. In a sense, this is very similar to how a living organism works. The brain performs general command and control, but individual organs, cells and mitochondria have their own level of autonomy. More importantly, unlike the poor caveman making guesses as to the intent of the tree, we know the artificial intelligence is animate and has agency. We programmed it that way.
The implications of this are enormous. Instead of Fitbit-style devices reporting health metrics, these health devices will be empowered to perform certain autonomous activities based on localized analysis of biometric data. Moreover, as these devices interact, contextual data can be used to block certain actions. If I were kidnapped and forced to stand before a retinal-scanning ATM, my cellphone’s heart rate and blood pressure monitor might be able to intuit that I am in distress and block the interaction. Perhaps, based on the fact that I’m in distress and at an ATM, both the ATM and the phone will discreetly call the police and track my location. The ability to make contextual decisions for identity and access management (IAM) means an additional level of security considerations.
Medical devices as “smart” IoT devices and EHRs will be able to make contextually aware decisions such as when and where to push narcotics or begin a defibrillation process, or what family of medicines to prescribe based on allergy history. As the machine learning refines itself and quality increases, it’s entirely plausible that the human oversight will become more and more cursory and automated in turn.
In a situation like that, how far off are we from being Tech-Priests praying to machine spirits?
Preventing a Grim Future
Not all sci-fi futures are hopelessly dystopian.[v] What, then, is the secret to happiness? Here’s what we know:
- Data and logic are conflated into “processing nodes” – this time by design.
- Nodes are distributed – all elements of the network have some level of animacy and agency.
- There is no way a human brain can holistically understand the entire system at a glance.
We must leverage the power of machines on their own turf. The IoT and most Big Data design patterns are massive, fragmented, federal systems where each subcomponent or subsystem has some level of autonomy and agency. Business processes emerge in the moment, influenced by the data the nodes are processing. These processes must be understood, but will require brute force dividing and conquering of the problem. In other words, we need machine learning, tools like Apache Spark, to break down and reverse engineer business processes. Morever, we need this machine learning to run in real time, constantly, essentially “introspecting” the system at a pace that no human brain can meet. The spirits are alive and must be placated, but if we train our machines properly, they will be able to act as a real-time, up-to-date reference repository for business processes, taxonomies, coding standards, etc. In other words, an system that continually baselines itself is one that will operate in an orderly, predictable manner and less likely to have emergent examples of chaotic behavior.
Here’s a good test case for an AI that will solve two problems at once. Right now, with VistA, the MUMPS codebase is a mess. Variable scoping and naming conventions are not enforced, nor are there any reserved words in the language. Moreover, operators may be contracted, which means that two chunks of code that look different may actually be identical. As such, VistA Modules such as Lab, Rad, and Surgery may either look incredibly different but function the same, or look the same but function incredibly differently. There’s no way for a human to quickly resolve this.
Instead, I propose we have a machine learning tool walk through every module and analyze the existing MUMPS codebase in VistA. With “fresh eyes,” it can reverse engineer what each module is trying to do and re-create a consistent business process and taxonomy. Once re-created, the AI will auto-assemble this documentation into the continually monitored baseline, and output code to migrate VistA to Hadoop or another modern technology. The inconsistent naming conventions and forgotten business processes will be easily reverse engineered by an AI. Moreover, having the AI’s first goal be to baseline and continually document the system will allow humans an understanding and oversight over the system. Even though the system will be decentralized and have logic and data conflated, we will know this is an act of science and not spirituality because we can always refer to the continually groomed CONOPS and documentation repository.
Plus, maybe – just maybe — we will teach our fledgling AI a little bit about what it’s going to be like to live in a world full of messy, inconsistent humans.
Who knows, maybe in the distant future of the year 40,000, the machines will be trying to placate us.
[i] Frazer shocked the Victorian world by pointing out that all societies practiced human sacrifice, and posited that ancient rituals such as animal sacrifice and modern ones like the Eucharist were a modern evolution of this.
[ii] For anyone who wants to see me rant all day on the nature of split ergativity and the active-stative nature of Pre-PIE, feel free to contact me directly.
[iii] Fire: *paewr- “fire (substance),” whence Hittite pahhur, Armenian hur, Greek pur/pyro- Sanskrit pu, English fire; versus *egni- “fire (active),” whence Latin ignis, Russian ogni, Sanskrit agnih.
[iv] Water: wodor- “water (substance),” whence Hittite watar, Latin unda “wave,” Greek hudor/hydro-, Norse vatn, Russian voda, and of course English water; versus *ap-, “water (active),” whence Hittite ak-wanzi “he drinks,” Latin aqua, Old English ea, Sanskrit apam, Persian ab.
[v] Iain M. Banks’ Culture series is a great example. Start with Use of Weapons.
