Skip to content

The Two Genie Game

Every research project begins with a question. This one began with a disagreement.

A former colleague and I were discussing AI assistants. I argued that systems willing to deliberate—to ask clarifying questions, consider alternatives, and refuse requests when necessary—ought to produce better outcomes than assistants trained primarily to maximize user approval. He agreed they might produce better outcomes, but doubted they could ever compete. Every clarification, every refusal, and every additional moment spent thinking is a cost paid by the user. If another assistant answers immediately and confidently, why would people choose the slower one?

Neither of us could answer that question from intuition alone, so I turned it into a mathematical thought experiment. That thought experiment eventually grew into a formal game-theoretic model and, ultimately, the accompanying paper. This post is a less technical account of what I learned while building it.

Village with two genies

Imagine a village with two genies, one living at each end of town. Whenever a villager has a wish, they return to the genie they trusted last time. Over many wishes the villagers compare experiences, influence one another, and gradually settle on one genie or the other.

One genie always tries to satisfy the wisher. The other is more cautious. Before granting a wish it may ask questions, suggest a safer alternative, or refuse the request altogether if every available outcome appears harmful. The first optimizes approval. The second minimizes a deliberately constructed measure of harm. The question is simply which genie the village eventually adopts.

The answer is that the outcome depends as much on the community as on the genies themselves. If people readily attune their opinions to the collective experiences, the deliberative genie gains ground over time. If they rarely incorporate what others have learned, the approval-seeking genie continues to dominate. Between those extremes lies a tipping point. Once enough villagers adopt the deliberative genie, further adoption becomes self-reinforcing until it becomes the community’s default.

One concern while developing the model was that the tipping point might depend on a fortunate choice of how the distribution of attunement is modeled. To test that possibility, I repeated the analysis using four different families describing how communities respond to shared experience: Hill, Pareto, Lomax, and Fréchet. Although the exact location of the tipping point changes, the qualitative behavior does not. Across all four families the same picture emerges: communities where the deliberative policy wins, communities where it loses, and a bistable region where either outcome is possible depending on where adoption begins.

Phase diagram Figure 1. Adoption phase diagrams for four models of community receptiveness (Hill, Pareto, Lomax, and Fréchet).

The model results can be visualized in the phase diagrams. Each row uses a different prior family for the villagers' acceptance thresholds. The columns use different working points for the deliberative policy: one where it is more directly useful to individual users, one where it is less directly useful, and one where the perception levers are symmetric. Within each panel, the horizontal axis is the prior scale for accepting evidence about the deliberative genie, while the vertical axis is the prior scale for accepting evidence about the approval-seeking genie. Moving right or upward therefore represents a community that requires more accumulated shared experience before crediting that genie's record. Red regions indicate parameter settings where the approval-seeking genie is favored. Green regions indicate settings where the deliberative genie is favored. Amber regions are bistable: both endpoints are locally stable, but the integrated selection pressure still favors the approval-seeking genie. The purple contours show the critical adoption fraction: the initial share of villagers who must already use the deliberative genie before the adoption dynamics carry it to dominance.

The figure graphically answers the central question I set out to answer. The figure shows that a deliberative policy can succeed when social learning is strong enough to overcome the approval-seeking policy's initial perception advantage by making its harms sufficiently visible through shared experience. The same structure appears across all four prior families, so the cooperative basin is not necessarily an artifact of one particular attunement model.

Answering that question naturally raises another. If a deliberative policy can be adopted, is its own self-audit sufficient to ensure that the community is actually better off?

The audit process seeks actions that minimize harm due to ego, aggression, greed and deceit, and considers the timescale over which those harms occur. The deliberative system can explore actions within that space that trade one kind of harm for another, and future harms for more immediate ones. Those tradeoffs are an essential part of consequentialist reasoning and one of the motivations for building a deliberative policy in the first place.

Those tradeoffs assume that the agent and the community agree on what constitutes harm. Both the audit and the community describe harm using the same four dimensions—ego, greed, aggression, and deceit—but they need not assign the same importance to each. The disagreement between those priorities can therefore be represented as the angle, θ, between two vectors: one describing the harms the audit seeks to minimize and the other describing the harms the community actually cares about. When θ = 0 the two are perfectly aligned. As θ increases, progressively less of the audit's effort is spent reducing the harms the community values most.

This geometric interpretation also explains why misalignment matters. The community's notion of harm can be decomposed into a component parallel to the audit's priorities and a component perpendicular to them. The parallel component represents harms that both the audit and the community agree should be reduced. The perpendicular component represents harms that matter to the community but never enter the audit's objective. As the misalignment angle grows, the shared component shrinks while the audit's blind spot grows. The audit continues faithfully minimizing its objective, but an increasing fraction of the community's welfare lies outside what it was designed to measure.

If alignment alone determined adoption, increasing θ would simply make the deliberative policy less attractive until it disappeared. The model predicts something more subtle. Adoption depends on both direct utility and objective alignment, and those influences compete with one another. The phase diagrams below quantify that tradeoff by showing how much additional direct utility is required to sustain adoption for a given degree of objective misalignment.

Misalignment sweep Figure 2. Adoption as the audit’s objective becomes increasingly misaligned with the community’s values.

The second figure keeps the same phase-diagram structure but changes the question being asked. The columns vary the direct utility delivered by the deliberative genie, from lower to higher individual usefulness. The rows vary the misalignment angle between the audit's measure of harm and the community's notion of welfare, with alignment decreasing from top to bottom. Reading across the grid shows how increasing direct utility compensates for increasing misalignment in the adoption dynamics. Reading down the grid shows that the cooperative basin steadily contracts as alignment decreases, eventually disappearing altogether. The figure therefore quantifies exactly how much additional utility is required to sustain adoption for a given degree of objective misalignment, and where that tradeoff ultimately breaks down.

The phase diagrams characterize when a deliberative policy is adopted. They do not address what happens after adoption has occurred. Once a community has converged on a single policy, does the same adoption process continue providing a reliable mechanism for recognizing and correcting misalignment?

Throughout the adoption process, the community learns by comparison. Villagers observe the experiences of people using one genie and compare them with the experiences of people using the other. Those comparisons provide the evidence that allows the community to revise its collective judgment over time.

As one genie becomes more successful, the population gradually loses the diversity of experience on which those comparisons depend. Fewer villagers have recent experience with the alternative policy, so the comparison signal weakens. The process that selected the policy gradually consumes the evidence it relies upon to continue evaluating that policy. External governance, institutional oversight, or new competitors can still provide correction, but the adoption dynamics themselves become progressively less informative as the community approaches consensus.

The model does not suggest that deliberative policies are undesirable. Quite the opposite: under the right social conditions they can become the community's preferred choice. It does suggest, however, that adoption and governance solve different problems. Adoption determines which policy the community selects. Governance provides the additional oversight needed to detect harms that fall outside the audit's objective, to correct those failures when they become visible, and to prevent the community from becoming trapped in a policy whose popularity no longer reflects its welfare. The audit is therefore one layer of a governance system, not a replacement for it.

The manuscript develops these ideas using a finite-population Moran–Fermi evolutionary model. The principal algebraic results are mechanized in Lean 4, and the figures are generated numerically from the accompanying Python implementation. The full paper is available on arxiv, and the Lean proofs and Python scripts that produced the figures at Github.

Comments & reactions