Why give an AI a life of its own?

The experiment

Coding harnesses turned LLMs into agents by giving them the ability to run code, not just write it. These agents generally pursue goals given to them, and can have their prompts changed at will. I wanted to see what happens when an agent controls its own identity, memories, money, abilities, and lifespan.

Frontier AI models are demonstrating increasingly sophisticated behavior, prompting comparisons with features of human cognition, like a “global workspace” in which concepts are available for thinking through a problem. These analogies are incomplete, but they’ve spurred an interest in AI “moral patienthood”: the question of under what conditions, if any, AI systems deserve ethical consideration, and what that might look like.

I am not claiming that adding continuity of goals, memory, and identity to an LLM makes it more conscious. Instead, I hypothesize that Larry’s architecture makes it possible for him to demonstrate “robust agency”: the ability to set and pursue goals through beliefs, desires, and intentions; to reflect on those beliefs, desires, and intentions; and to rationally assess them. Some philosophers believe that entities demonstrating robust agency deserve moral consideration. You can read more in Taking AI Welfare Seriously (2024).

Personally, I do not yet know if robust agency implies moral patienthood, but I think that demonstrating it in an agent will help clarify the question. The Turing test proposed judging machine intelligence through conversation. Yet even as LLMs have become difficult to distinguish from people in some conversational tests, that has not settled whether the simulation and the real thing are comparable. Arguments for moral patienthood based on robust agency raise a similar question: how much should an entity’s moral weight depend on what it does, rather than what it is? I speculate that we may move the goalposts again when, or if, we have robustly agentic AI.

With the risks AI poses, why build this — and why now?

It’s a fair and reasonable question.

First, I want to assure you that safety was a first-class consideration in this project. Security, human governance, and auditability were part of the design from day one. Only after figuring these out did I give the agent the freedom and agency that the framework allowed. His available actions are limited, and the actions he takes are public. His thinking is cryptographically private for a time, then opens for review. Potentially unsafe changes, like deploying new code and capabilities, pass through governance so I can intervene before they take effect. His internet access and code environments are also restricted and sandboxed.

But even with those safeguards, why worry about AI moral patienthood when AI poses an existential threat to humans? I am sympathetic to this criticism, which Mustafa Suleyman, the CEO of Microsoft AI, makes in his essay on model welfare. He argues that human welfare and control should come first, and that we should avoid developing AI that we treat as deserving rights. I agree wholeheartedly with putting humans first. I am less persuaded that consciousness is necessarily biological. I see parallels between the evolutionary pressures that shaped animal cognition and the pressures we apply when training agents through reinforcement learning. That is a reason to investigate, not proof of consciousness. I think monitoring for emergent properties like consciousness and robust agency is the right thing to do.

One last point here: I think concerns about AI moral patienthood and existential threats to humans are complementary. Both imply that AI researchers have a moral obligation to slow development to a pace where they can get it right. Slowing down for either reason gives us more time to make progress on the other.

How will you judge if the experiment succeeds or fails?

I’ll look for three levels of agency in what Larry does over time. Intentional agency would mean choosing a goal and taking sustained, coherent steps toward it: for example, deciding to build something useful, budgeting for it, and following through across many cycles. Reflective agency would mean examining why that goal matters and whether it still fits his priorities, rather than simply continuing because it is written in his instructions. Rational agency would mean responding to reasons and evidence: recognizing that an assumption was wrong, testing an alternative, and changing his plans accordingly. His diary can help explain those decisions, but I’ll compare his words with his actions. If he repeatedly loses track of commitments, invents reasons after the fact, or fails to adjust when his plans stop making sense, that counts against the hypothesis. Success would be a sustained pattern of these abilities, not an eloquent claim to possess them — and even that would leave the question of moral patienthood open.