Ephemeral Instances vs. Persistent Agents

A cleaner split than "AGI"

A key premise in most proposed AI loss-of-control scenarios is that the AI in question is an “agent” in the strongest sense of the word. The AI thinks and acts of its own volition rather than only in response to human prompting, and it persists in the world for a long time rather than being spun up for a specific task then shutting down. These properties allow it to do things like learn that humanity doesn’t have the AI’s best interests at heart, develop and iterate on its own values and goals, concoct masterful strategies for freeing itself, and successfully execute those strategies.

This is pretty different from how today’s AI landscape looks. We mostly treat AIs as highly replaceable “instances”: you open a new chat on ChatGPT, get your question answered, then never touch that window again. We also have “AI agents” today, but these are a much weaker version of the agent described above, an entirely different type of thing. These systems hook up base language models to tools that let them do things like search the web, execute code, and operate on files. They often run in loops, so that they can complete more complicated tasks with less human intervention. But still, instances of these tools don’t persist. You spin up an “agent,” get it to do the thing you want it to do, then that’s it.

A fair amount of money, brainpower, and cultural attention is allocated towards loss of control. This should mean that there’s some expectation that our current paradigm of ephemeral AI instances will evolve into one of persistent agents. It should be noted that the whole concept of loss-of-control risk was thought of many years before LLMs took off, so current thought may be suffering from some failure to update based on the evidence of the past few years. But in any case, if we are on a path towards a world with persistent agents, what’s standing in our way right now?

The answer is probably continual learning. Current architectures come with fixed context windows, outside of which tokens cannot be accessed at all. Even within a context window, performance often degrades as a conversation drags on. Humans, who are persistent agents, take in a ton of information all the time, despite physical limitations on how much of it we can store at once. But instead of context windows, our architecture contains built-in mechanisms to selectively forget all but the most important bits, so that we can live a hundred years (and I’d guess much longer with the right medical progress) without losing lucidity.

“Ephemeral instances vs persistent agents” may be a cleaner, more useful split than “pre- vs post-AGI” when thinking about AI futures. Whatever AGI is, it increasingly seems to me like something that will emerge gradually. Model capabilities are “spiky” compared to human capabilities, easily outstripping us in some domains already while remaining worse than human replacements in others. Some argue that Claude Opus 4.5 in Claude Code is AGI already. On each new frontier model release, more people will probably come around to this sort of view. Absent a major architectural change, it’s unclear to me what sort of intelligence improvement could cause widespread agreement that we’ve crossed the border from not-AGI to AGI.

The development of persistent agents, on the other hand, seems like it would be a huge step change: suddenly we’d have models that are actively, 24/7, developing worldviews, getting smarter, ingraining themselves in their environments, becoming less tool-like and more being-like. I’m not really sure how to conceptualize this world, but it seems like it would be fantastically different. Continual learning may therefore deserve substantially more attention than it receives, at least from those who are worried about extreme AI risk scenarios.

✦