Why Civilization Goes Virtual
Why Civilization Goes Virtual
Modern technology gives individual agents civilizational-scale power – and that fragility creates the risk of a civilizational end scenario: one in which there is no civilization, no intelligent life, and no culture preserved. In this post, I outline an intuition for avoiding such catastrophes without total surveillance, by moving most intelligent agents into virtual space. Virtualization does not eliminate these risks. It just pushes the most dangerous actions into a few tightly controlled areas, while letting everyday life stay mostly free.
The black ball hypothesis: why some technologies make civilizational collapse inevitable
The black ball hypothesis describes how a civilization might end: civilization keeps pulling technologies out of a box (some white, some gray), and one day we might pull out a black ball – a technology that makes it easy to wipe out civilization (watch on YouTube: Nick Bostrom — “How Civilization Could Destroy Itself” (TED Talk)).
Imagine a planet-killer technology that anyone can trigger with a trivial action, like putting sand into a microwave. If something like that existed and was widely accessible, the world would almost certainly be gone by now. More generally, if a single individual can easily end everything, then with enough individuals over long enough time horizons, sooner or later you get a malicious actor, or a mistake, or negligence. Eventually, you find one. The core issue is accumulation. Even if the per-person probability of a civilization-ending attempt is tiny, multiplying it by billions of people and long time horizons makes “at least one catastrophic attempt” approach certainty – unless the underlying risk is driven extremely close to zero and kept there indefinitely.
Examples include a nuclear-scale device, gray goo, or an engineered virus. The structure is similar: it is cheap for many actors to attempt, hard to defend against while in action, and potentially irreversible once triggered.
This end-of-civilization black ball scenario feels extremely plausible to me, unless civilization adapts and finds ways to make black balls less black, for example, by reducing how easily they can be deployed and by putting friction and monitoring at the critical points that can lead to catastrophic consequences.
The conservative scenario: total surveillance is a solution but it may not be enough
Imagine that nothing fundamental changes: billions of people keep living on Earth, interacting with the world with roughly today’s level of freedom, while new technologies continue to appear.
One response that can help to mitigate the end day in this scenario is total surveillance: monitoring everyone closely enough to catch and stop civilization-ending actions early enough. Once the ability to end everything is broadly distributed, only total monitoring seems capable of preventing the “eventually someone does it” outcome. Eventually, that means every person carries a special device at all times, continuously observed, so even a trivial trigger, sand in a microwave, can be detected and interrupted.
In the limit, even very strong surveillance may not reduce the risk enough. The failure mode is not “we did not watch hard enough,” but that some dangerous actions could be too fast to intercept, too ambiguous to classify in time, or too adaptive and novel for any detector to keep up without intolerable false positives.
Virtual scenario: future society lives as software
The other option is that future society is virtual. Current deep learning, especially increasingly agentic LLM systems, makes it seem likely to me that the first powerful artificial minds will be software: digital, running on compute, and not inherently tied to any single physical body. I am also thinking far enough into the future that the “default” intelligent population could be mostly software minds living in the cloud (data centers), where what we currently call humans-with-bodies may be rare, transformed, or no longer the main reference point.
In that setup, a mostly virtual society feels plausible: agents can think, communicate, build, and coordinate entirely in software, while interacting with the physical world only through controlled interfaces: robots, labs, devices, more like renting a body when needed than living in one permanently. I am not committing to any specific type of agent here – whether future AI systems, digitized humans, or hybrids.
The mostly virtual society needs to handle (1) black balls in the virtual space and (2) interactions with the physical world (black balls in the physical world) to survive.
Black balls in virtual space are more manageable
Virtual space still has its own black balls, e.g. software vulnerabilities, but the claim is they are more manageable. There is a basic difference between the physical world and a virtual world. In the physical world, resources are just “out there”: matter and energy exist everywhere, and if someone finds a way to use them for damage, nature does not ask for permission. In a virtual world, resources are handed out by the system – by its rules and institutions. Compute, memory, and communication bandwidth: you only get what the platform gives you. So we can build strict limits: if something starts spreading or consuming resources, we can slow it down, cap it, or shut it off before it takes over everything. The virtual world is engineered; we can make it harder for one bug to break everything. Not to mention the possibility of multiple backups and reboots.
A virtual society makes physical-world black balls easier to safeguard
Black balls in the physical world are still very dangerous, so the boundary where the virtual interacts with the physical must be heavily monitored and properly governed. My intuition is that physical actions would be slow: logged, rate-limited, and gated by multi-layer approval, with the ability to pause or abort execution. Ideally, most physical work would be carried out by constrained scripts (executed by robots) rather than open-ended agent improvisation. A robot “body” would not be controlled by a single mind; it would be supervised by a quorum, so that one malicious or mistaken agent is unlikely to be enough to push a civilization-ending action through.
Another idea I find attractive is that, if we need on-device minds rather than cloud control, we should place multiple randomly sampled agents in the same body, so that no single on-device mind has full control when the system must operate locally. This matters in cases where cloud co-watching is impossible or unreliable (for example, far away in space, with high latency or unstable communication). In that regime, local quorum and diversity can act as a safety layer: high-impact actions require agreement among several independent controllers, rather than trusting one mind alone.
Virtual society is freer and more sustainable
As a result, agents can think, communicate, and run simulations freely in software without total surveillance of everyday life, while actions that touch the physical world are treated as high-stakes and routed through strict, slow approvals. For example, a virtual agent might spend the day chatting with friends or playing games, but if it proposes a change to a robot’s control code in a datacenter, that change goes through independent review and only then gets deployed.
Conclusion
None of this makes the probability of a black ball zero. But it gives us more knobs to tweak in virtual society: better containment, better recovery, and more ways to prevent cheap actions from becoming globally irreversible. So I think it may be possible to sustain a civilization living mostly in a virtual world while preserving substantial individual freedom, without recreating a total-surveillance vibe in everyday life.
Desiderata
These ideas reflect my personal desiderata for future intelligence. First, I want a society of agents to exist at all (with a diverse set of minds). I do not know what the “end game” of intelligent life is, so I prefer that it not get stuck, and that it be able to explore many different trajectories. Second, I do not want total surveillance of everyday life for similar reasons. In virtual space, agents should be able to talk freely, propose and debate new ideas, design experiments, run simulations, and explore. The only hard restriction should appear when virtual activity turns into physical-world interaction.
Thank you to Alice Yannau for editing this post and to Arsenii Kuznetsov for discussions.