A Definition of Intelligence Recursion

co-authored, not all writing is my own.

Intelligence recursion sits in a unique position; the term appears heavily throughout AI (safety) discourse, yet it still lacks a definition precise enough to do the work being asked of it. It is often invoked as though its meaning were settled, when in reality this is far from the truth.

I set out a taxonomy of competing definitions, evaluate them against four criteria, and offer a working definition of my own, along with an operationalisation. The aim is limited; it is not to resolve every philosophical dispute in advance, but to identify what a governance-useful definition would need to capture, as well as what it would need to exclude.

The cost of vague concepts

Governance usually fails on definitions before it fails on principles. If the concept being governed is vague, the rule attached to it will usually be vague as well. Monitoring weakens, verification becomes contestable and enforcement slips towards discretionary judgement, which is difficult to defend publicly and harder still to coordinate across institutions.

Intelligence recursion occupies exactly that sort of position. The term appears more and more often in safety arguments, policy proposals, and governance frameworks, but it is still generally operating at a level of abstraction too high for the work those frameworks want it to do. That would matter less if the concept remained mostly theoretical - but it does not.

The problem is easy enough to state: a phenomenon is being treated as important without any equally settled account of what would count as its occurrence.

Hendrycks, Schmidt and Wang place intelligence recursion at the centre of their work. Anthropic's Responsible Scaling Policy relies on AI R&D thresholds that refer directly to the same frameworks. The International AI Safety Report treats AI-accelerated AI research as a live governance concern. In each case the shape of the worry is clear enough. What is less clear is the criterion by which an outside observer would be entitled to say that the threshold had actually been crossed.

Put more bluntly, what would one have to see to classify intelligence recursion?

MAIM is a useful example precisely because it exposes the problem. A fleet of AI systems conducting fully automated AI research has real intuitive force as a description of danger. As a criterion, however, it remains underdetermined. How much autonomy is enough? What sort of research counts? How much human involvement is compatible with the label? Until these are answered with more discipline, the red line remains politically vivid but analytically unstable.

The term has acquired weight faster than it has acquired operational clarity. A concept can become important before it becomes usable, and intelligence recursion appears to be one of those concepts.

What intelligence recursion is not

Much of the confusion in this area comes from the fact that different authors are not always talking about the same thing.

AGI is a capability threshold. Intelligence recursion is a process. The two are related, but they are not identical. It is plausible that AGI, or something close to it, is a precondition for genuine intelligence recursion; whilst the reverse could also be true.

Nor is intelligence recursion simply synonymous with intelligence explosion. The latter is broader. MacAskill and Moorhouse are right to insist that explosive progress could, in principle, occur through large-scale substitution of human researchers by AI systems even in the absence of any neat software feedback loop.

Superintelligence is different again. It names an endpoint, or a family of endpoints, not the transition itself.

Finally, recursive self-improvement (RSI) is arguably the most similar term and one that is often conflated with intelligence recursion. Intelligence recursion refers more broadly to a potential feedback loop where an intelligent agent develops the next version of itself, whereas RSI refers more specifically to the phenomenon where AI autonomously rewrites its own code to improve capabilities. It is a substantial subset and one that we might expect to see alongside intelligence recursion.

These distinctions are basic but are also repeatedly blurred. This is one reason the term intelligence recursion has travelled so easily; it has often been asked to bear more conceptual weight than a single term can manage.

Where the idea comes from

The idea enters the literature through Good's famous deduction. An ultraintelligent machine, by definition, would surpass human beings across the full range of intellectual activity. The design of machines is one of those activities. Therefore, such a machine could design better machines. Therefore, an intelligence explosion would follow.

That is an argument for possibility; it is not yet a criterion of occurrence. Good explains why the process could exist but he does not tell us what an observer would need to look for in order to say that it had begun.

That gap never really went away, much of the later literature inherits the force of Good's argument without repairing its indeterminacy. The result is a concept that is loaded, intellectually suggestive, and still not properly disciplined.

Preconditions

Yudkowsky contends that intelligence is a supersystem composed of a plethora of mutually interdependent subsystems. This is the first and most important structural constraint on any theory of intelligence recursion: you cannot recursively improve something you have not correctly characterised. Hendrycks et al. tested ten independently validated cognitive domains and revealed a jagged profile in current AI systems; GPT-5 scores 57/100, with zero on long-term memory storage. A system like this would be functionally amnesiac, and unable to consolidate the results of its own self-modification between sessions.

Yampolskiy distinguishes three qualitatively distinct regimes: self-modification (no intelligence increase occurs), self-improvement (subject to diminishing returns), and recursive self-improvement, where the system not only improves, but gets better at improving. Effectively, an entirely different class.

There is another complication; capability is not enough on its own. A system might be able to create a superior successor without being designed, or choosing, to do so - Chalmers calls this a motivational defeater. Bostrom's instrumental convergence thesis complicates the picture again. Self-improvement is not just one design option among others; rather it may be a structural tendency of capable systems placed in sufficiently open environments.

Five candidate definitions

My candidate definitions were judged against criteria that matter for governance rather than merely for conceptual elegance:

• External verifiability: can an outside observer determine whether the definition has been satisfied without privileged access to internal model architecture, organisational process, or developer intention?
• Gaming resistance: can a developer or state actor satisfy the letter of the definition while plainly evading its spirit?
• Danger-tracking: does the definition identify the phenomenon that is strategically and safety-relevant?
• Timeliness: does the definition permit detection early enough to matter?

A: the algorithmic improvement criterion. Intelligence recursion occurs when a system modifies the algorithm by which it learns or reasons, producing a successor more capable of generating further improvements than the original. It tracks the danger well, and it is not especially easy to fake. However, it may be too late-moving for governance purposes. By the time one can establish with confidence that this process is under way, the point at which intervention would have been easiest may already have passed.

B: the crossover point criterion. Intelligence recursion occurs when a system's self-directed contribution to its own capability development exceeds the contribution made by human researchers and external infrastructure. This is probably the strongest candidate in theoretical terms. The problem is not conceptual, rather it is operational. We end up with a potentially upsetting truth: the most theoretically satisfying criterion may also be among the least institutionally usable.

C: the fully automated AI research criterion. Intelligence recursion occurs when a fleet of AI systems conducts AI research autonomously at a scale and quality that exceeds the rate of progress achievable by unassisted human researchers. This is, in substance, the MAIM red line made more explicit. If a project of this kind existed at sufficient scale, one might expect to see signs in compute use, infrastructure, staffing patterns, and output. The weakness is that it remains vulnerable around the edges - activity can be fragmented. Human oversight can be retained formally while the substantive dependence has already shifted.

D: the capability compression criterion. Intelligence recursion occurs when AI systems can compress two years of AI research progress into one year, measured against a defined historical baseline. It already exists in live institutional practice; Anthropic is using something close to it. However, it remains a proxy. It also invites obvious fights about baselines, denominators, and historical periods. The more practical a proxy becomes, the more argument it tends to attract.

E: the research substitution criterion. Intelligence recursion occurs when AI systems can meaningfully substitute for human researchers across the full range of AI R&D tasks, producing self-compounding gains whether or not a strict algorithmic feedback loop is present. It fires earlier and is easier to benchmark. The weakness is equally clear; it risks sweeping too much into the category. The criterion gains timeliness by giving up precision.

No candidate avoids trade-offs. Some are theoretically stronger and operationally weaker. Others are easier to govern around but less exact about the phenomenon they are meant to capture.

Any candidate definition also needs to satisfy a negative condition. Not every recursive loop is an improving loop. Shumailov and colleagues show that a system trained repeatedly on its own outputs may degrade rather than improve. A definition of intelligence recursion cannot rest on the existence of a feedback process alone; it must require sustained and measurable capability improvement across successive iterations.

My working definition

Intelligence recursion is the process by which one or more AI systems make sustained, measurable contributions to improving the cognitive capabilities of AI systems, including themselves or their successors, at a rate that exceeds the rate at which unassisted human researchers could produce equivalent improvements, where the improvement itself increases the system's capacity to make further improvements.

'Sustained' and 'measurable' are doing important work. They exclude one-off gains, marginal tuning, and anecdotal improvements. They also respond directly to the problem of recursive degradation.

The focus on cognitive capabilities rather than outputs is deliberate. The concern is not merely that AI may automate particular research tasks - it is that AI may improve the capability base from which further AI acceleration follows.

The comparison with unassisted human researchers is intended to capture the point at which the process becomes self-driving rather than merely tool-assisted.

The phrase including themselves or their successors matters because recursive improvement need not occur inside a single monolithic system. A distributed fleet of narrower systems, collectively improving the next generation, may satisfy the relevant condition just as readily as a single system rewriting itself.

Finally, the clause requiring that improvement increase the capacity for further improvement is what distinguishes genuine intelligence recursion from ordinary convergent self-improvement. Without it, the definition collapses into a broader and weaker category; with it, the definition tracks compounding rather than repetition.

In other words, the definition aims to be narrow enough to track the phenomenon of concern and broad enough not to miss the most plausible route by which that phenomenon might emerge.

Detection and confirmation

The proposed operationalisation has two parts: a detection threshold and a confirmation criterion. Detection is about when concern becomes serious enough to trigger scrutiny. Confirmation is about whether the underlying process is really the one of interest rather than some weaker or more transient form of acceleration.

Intelligence recursion should be treated as potentially initiated once AI agents achieve sustained superior performance over human expert teams on extended-horizon AI R&D tasks. RE-Bench is the most promising current tool for making that judgement. At short horizons, frontier agents already perform very well. At longer ones human researchers still retain an advantage. A real detection signal would therefore not consist merely in further gains on short tasks, it would consist in the closing, and then reversal, of the long-horizon gap.

Confirmation should ask whether the improvement process is producing accelerating rather than diminishing returns. Is the system merely continuing to improve, or is it improving in a way that progressively lowers the cost of further improvement? The second case is the one that matters.

Compute governance should be treated as a complementary proxy, not a substitute for capability evaluation. At sufficient scale, a recursively improving system or fleet is unlikely to be invisible in compute terms. But compute alone will not settle the matter.

This is not a finished measurement regime, instead it is closer to a disciplined starting point; but that is still preferable to pretending the threshold will identify itself.

What I found difficult, and what surprised me

The most serious limitation of any definition of intelligence recursion is still observability. The most dangerous form of intelligence recursion may only become visible once it is already well advanced. I also struggled with balancing timeliness against precision across the candidate definitions; every gain in one costs the other. And keeping the definition governance-useful without reverse-engineering it from preferred policy outcomes is harder than it sounds - it is dangerous and usually produces bad definitions.

What surprised me was that the most theoretically precise definition (Candidate B) is also the least institutionally usable; the best criterion may never be measurable from outside. Recursive degradation is a real and underappreciated risk - the same feedback structure that could produce an intelligence explosion can also produce capability collapse. Current systems are also further from the threshold than discourse implies.

That said, the empirical frontier, though still some distance from the strongest form of intelligence recursion, is no longer comfortably abstract. STOP shows meaningful self-improvement at the level of scaffolding. The AI Scientist closes bounded research loops. Anthropic, and Dario Amodei have repeatedly mentioned that Claude is used, through the Claude Code harness, to generate code that is deployed almost instantly into production in Anthropic products. None of this amounts to full recursive intelligence improvement, though nor does any of it leave much room for complacency.

Conclusion

The governance problem is not going away; institutions are already building frameworks around intelligence recursion, whether in deterrence proposals, deployment policies, or international risk assessments. Those frameworks need a definition if they are to do any real work - at present they often rely on something looser than that.

None of this removes the underlying uncertainty, but it does make that uncertainty more tractable. Those who have to act on this concept deserve something more precise than a vivid phrase and a shared intuition.