Cognitive instabilities under contention, friction, and the fog-of-war challenge the AI revolution. Issue 3 (2nd July 2020) - Wallace, Rodrick
Abstract : The asymptotic limit theorems of information theory, and related formalism from punctuated equilibrium evolutionary theory, allow exploration of the dynamics of real-time, multi-component, hierarchical, cognitive enterprises – institutional, machine, or man/machine hybrid – under constraints of time, material resources, internal communication, situational awareness, and adversarial intent. Surprisingly direct argument shows such enterprises can almost always be confounded by an opponent with comparatively limited resources, provided it has sufficient situational understanding, adaptive capacity, and resolve. Consequently, there can be no free lunch for AI combat systems, driverless cars on intelligent roads, or other such entities, tasked with the control of real-time critical processes. Intelligent adversaries, resource limitations, "design flaws", inattentional blindness, command stupidity, coevolutionary dynamics, and the like confront all real-time, multi-component cognitive processes. AI systems will not be spared simply by being labelled "high tech" in marketing campaigns.