The Two Genie Game
Every research project begins with a question. This one began with a disagreement.
A former colleague and I were discussing AI assistants. I argued that systems willing to deliberate—to ask clarifying questions, consider alternatives, and refuse requests when necessary—ought to produce better outcomes than assistants trained primarily to maximize user approval. He agreed they might produce better outcomes, but doubted they could ever compete. Every clarification, every refusal, and every additional moment spent thinking is a cost paid by the user. If another assistant answers immediately and confidently, why would people choose the slower one?
Neither of us could answer that question from intuition alone, so I turned it into a mathematical thought experiment. That thought experiment eventually grew into a formal game-theoretic model and, ultimately, the accompanying paper. This post is a less technical account of what I learned while building it.