The missing quadrant in responses to AI

Announcing STABL: Societal Transformation and AI Bottlenecks Lab

We are starting a non-profit research organization: STABL.

Adapting to advanced AI will require dramatic institutional change and broad societal resilience. It will also require evidence about AI progress and impacts — the prospects of recursive self-improvement (RSI), the pace of AI diffusion, and the bottlenecks that constrain how AI capabilities translate into societal outcomes.

Four theses define our worldview:

• We consider advanced AI a transformative general-purpose technology with the potential for broad societal benefits.

• Yet we recognize that the pace of societal transformation will be bottlenecked by the speed of human and organizational change more than progress in capabilities.

• Our institutions — the established ways we organize knowledge, work, and governance — are an important source of stability during this turbulent period, but they also need serious reforms to keep serving their purpose.

• We favor a broad portfolio of defenses to manage AI risks, of which alignment is only one part. Resilience, the most neglected component today, must become central.

Another way to state our worldview is to place it on a 2x2 chart. The vertical axis depicts taking AI’s transformative potential seriously. The horizontal axis depicts taking downstream bottlenecks to AI’s impacts seriously. At the top left is technological determinism: the view that superintelligence will unilaterally transform society by obliterating downstream bottlenecks. Many AI CEOs express this view, such as when predicting that a century’s worth of progress will be compressed into the next few years.

This view often ignores the organizational, regulatory, and societal bottlenecks to such a transformation; in this view, such questions are moot because superintelligence is all powerful. Surprisingly, some of the AI industry’s strongest critics share this worldview, focusing on slowing AI capability improvements as the main way to keep society safe from AI’s dangers.

The opposite view, at the bottom right of the chart, treats existing organizations, processes, professions, and governance as fixed points, and insists that technology be molded to fit them. People holding this view often work on limits on AI adoption, and protecting existing structures from disruption. They are good at identifying and leveraging the bottlenecks that blunt AI’s impact, such as regulation, professional norms, organizational inertia, and political pushback.

Slowing the pace of change can buy valuable time to shape technology’s impacts. But when preserving the status quo becomes a goal unto itself, it stands in the way of technology-enabled societal progress. (As an example, consider the resistance to self-driving cars despite their being safer than humans, or bans on AI-enabled legal advice).

STABL belongs in the top-right quadrant. We take frontier AI seriously as a transformative technology, yet consider the role of institutions — both new and existing — to be important in helping society navigate the transition.

For example, how should we understand and improve AI’s impacts on medicine? Many technological determinists claim superintelligence will rapidly cure nearly all diseases. Someone with the opposite view might respond that the bottleneck is not drug discovery, which AI may well accelerate, but clinical trials, which take years because of regulations and safety risks.

This response correctly identifies the bottlenecks to AI’s impacts. But this is often seen as the end of the conversation. In our view, understanding the bottlenecks to AI’s impacts is only the starting point of analysis. The next question is whether drug testing can be made faster and more efficient in order to meaningfully materialize AI’s benefits. For example, what are the barriers to in-silico testing? How can we rethink regulatory institutions and processes? Unfortunately, neither extreme pays close attention to these critical questions.

To be clear, STABL isn’t the only organization in the top-right quadrant; much of our work builds on other organizations and thinkers who take both AI’s potential and its downstream bottlenecks seriously. But compared to the other quadrants, there is a sparsity of thinking in this quadrant, which has led to polarization among the communities that take either the technology or the bottlenecks seriously. Our ultimate goal is to contribute to a rich intellectual agenda for this quadrant, create the space for new organizations, and collaborate with existing organizations in this and other quadrants.

To follow our work, subscribe to the STABL newsletter:

Our research agenda

The top-right quadrant leads us to many interesting research directions. On one hand, we expect AI capabilities to continue improving quickly, which motivates our empirical research on the pace of progress and RSI. We expect highly capable AI models to continue to be widely available, which motivates our analysis and advocacy for risk mitigation beyond aligning AI models and non-proliferation.

On the other hand, we consider institutions important for shepherding the transition to a world where advanced AI is broadly diffused. This leads to our interest in understanding the downstream outcomes of advanced AI adoption. We plan to measure specific benefits and harms, study the disruption of labor markets and beyond, propose ways to improve societal resilience and revitalize institutions.

These interests lead to four key research directions, which we detail below. While other organizations have worked on similar questions, we expect our research to be different in how we formulate these questions and how we go about answering them, informed by our worldview and priors about AI progress.

1. How should we measure progress towards recursive self-improvement?

Many forecasts of explosive AI progress rest on recursive self-improvement (RSI): AI automating AI research itself. We take RSI seriously precisely because our model of progress differs from the one underlying most RSI forecasts, and we want to empirically test claims about RSI to get early warnings of AI’s potential impacts. But how should we measure progress towards RSI?

Most RSI evaluations measure AI’s ability to complete tasks where success is easily verifiable. Here, AI systems have made rapid progress. But we think there is a growing chasm between AI performance on verifiable and open-ended tasks.

Through CRUX, we plan to run frequent open-ended evaluations to probe the upper bounds of AI capabilities on long, messy tasks, including whether AI agents can conduct open-ended AI research. Our first results suggest they cannot yet do so; expert reviewers unambiguously rejected the papers produced by frontier agents. We plan to repeat these evaluations regularly with improved models and scaffolds.

More broadly, the conversation about RSI is dominated by viewpoints in the top-left quadrant. Those most worried about superintelligence were ahead of the curve in recognizing the importance of studying RSI carefully. But communities and organizations have also self-selected into this work, leading to a lack of intellectual diversity. When the literature relies on the same (contested) assumptions about AI progress, it is prone to the same failures in its conclusions.

A more diverse ecosystem of researchers and organizations, including those with different priors about AI progress, could test whether these assumptions hold, for example by:

• Comparing causal models of AI progress. Rather than assuming a model and making predictions about the world based on that model, evaluate different causal models against one another. For example, does performance on verifiable tasks straightforwardly lead to real-world performance improvements?

• Studying how data bottlenecks shape recursive improvement. Does AI progress on tasks where it is straightforward to collect high-quality data translate to tasks where it isn’t? Understanding when improvements carry over across domains would help distinguish the hypothesis that RSI leads to broad superintelligence from alternatives in which a lack of real-world data limits progress in some domains.

• Disaggregating different dimensions of progress. Most benchmarks of AI progress rely on automated verification, which also makes them more suitable for reinforcement-learning training. Moving beyond automatic benchmarking could help us understand progress in non-verifiable tasks. Such disaggregation could help us understand which aspects of AI progress are amenable to recursive improvements and which ones aren’t.

As the importance of RSI-style dynamics becomes clearer for understanding AI progress, we hope there is a vibrant intellectual ecosystem of researchers and organizations evaluating RSI seriously while not necessarily treating its default outcome as superintelligence.

2. How can society defend against the risks of advanced AI?

AI safety has historically been focused on models themselves, and alignment has been the field’s technical grand challenge. The safety community’s response to open models, where alignment can be undone, has been to push for non-proliferation policies. But over time, it has become clear that advanced AI will be broadly available, including to people who might want to misuse it. So our responses to AI risks need to expand beyond alignment and non-proliferation.

AI safety will require three additional responses. First, we need to invest in the ability to control, monitor, and secure AI systems to prevent unintended actions even when AI systems are misaligned. Second, we need to improve downstream defenses, such as the cybersecurity of critical infrastructure to protect the attack surfaces where AI’s harms materialize. Third, we need to boost societal resilience to reduce the severity of harm when our defenses fail.

We think resilience is the most neglected approach. While the AI community has in principle emphasized resilience, in practice, the main emphasis remains on alignment and non-proliferation. We often hear the reaction that resilience sounds good in theory but no one knows what it means concretely. One of STABL’s goals will be to articulate an actionable resilience agenda. We hypothesize that for resilience to be effective, it must have the following properties:

• Domain-specific: We won’t solve resilience in a general way. We must understand and improve the resilience of specific sectors against specific threats. For example, what is the cost of securing a state’s hospital system against cyberattacks? Who are the actors best prepared to deploy these defenses? What is the most viable plan of action for implementing cyberdefenses at a large scale?

• High ambition: The investment needed in serious attempts at resilience (such as pandemic preparedness and large-scale cyber-defense) dwarfs our current investments. We will propose scalable, ambitious responses commensurate with the magnitude of the threat.

• Worst-case planning: Despite wide concern about catastrophic AI risk, there is little work on response and recovery plans for specific AI disaster scenarios, up to and including civilizational collapse. Improving resilience requires staring down the barrel of potential AI catastrophes and developing response plans.

3. How can we steer the economic impacts of advanced AI?

Even though advanced AI has been widely adopted by customers, workers, and firms, we don’t yet have a clear accounting of either the benefits or the costs of this adoption. While there is abundant evidence of workplace adoption, the returns on that investment have yet to materialize as broad productivity gains. And while many have raised concerns about imminent and massive job loss, the aggregate unemployment rate continues to be largely unchanged in the United States and other countries.

These trends are consistent with our view that the bottlenecks which govern economic transformation are largely beyond the scope of improving capabilities, and can’t directly be understood by AI capability evaluations. As a result, to better understand AI’s impacts, these bottlenecks themselves require dedicated study. On the other hand, we think the transformation that is already occurring as a result of AI might be obscured through the lens of traditional economic indicators.

These views motivate our work on steering AI towards augmentation, building prototypes for improving AI’s relationships with workers, and understanding how AI impacts the nature of work, beyond aggregate measures like employment statistics. We expect a lot of our work in the area to be conducted in collaboration with the growing set of organizations focused on the economic impacts of advanced AI. A few directions we are interested in:

• The dominant approach in companies developing and adopting AI is to try to automate costly human labor with comparatively cheaper AI. We expect some progress from this approach, much as how automating agriculture led to growth. But automation without structural transformation is limited in the gains it can deliver. We aim to shift focus from scaling today’s economy to the discovery of new ideas, goods, and industries that will yield broader prosperity. Specifically, we want to steer AI development and deployment to focus more on augmentation, which we believe is more economically promising.

• Augmentation also respects human contribution and agency, whereas automation prioritizes the firm’s objective to maximize productivity. To more directly account for the desires of workers, we will build prototypes for workers to better navigate constantly-evolving AI capabilities and identify ways to use AI that improve their job satisfaction.

• Established economic indicators like productivity and employment statistics are critical inputs for understanding and responding to the economic shocks produced by advanced AI. But overrelying on these indicators can mask important consequences. Even if the number of software engineers did not change much because of coding agents, the profession is dramatically different. The concerns of undergraduates about what they should study and what job opportunities will be available upon graduation also warrant attention.

• Finally, we expect significant economic impact to arise through personal use of AI outside the workplace, such as when people get tax advice from AI. This could generate surplus for consumers while simultaneously reducing demand for labor.

4. How should specific professions and sectors respond to AI?

Neither capability benchmarks nor exposure metrics determine how advanced AI might affect specific jobs, industries, and institutions, and how they should adapt to it. There is a need for examining different occupations and institutions in context. In recent work, we have conducted deep dives into AI’s impacts on science, legal services, and software engineering, and plan to continue analyzing the ongoing transformation and likely future of other kinds of knowledge work.

We aim to develop principles and policies that keep knowledge professionals and democratic institutions in control while also enabling productivity gains, transforming workflows and processes, removing unnecessary barriers to entry, and improving state capacity. We will propose ways to enable healthy competition between incumbents and AI-enabled upstarts — at the level of organizational forms, not just individual organizations. At the same time, we recognize that there are many cases where frictions in institutions are desirable. We aim to find a middle ground that welcomes beneficial reforms while avoiding the complete displacement of existing structures.

What’s next for STABL?

STABL is actively fundraising and hiring across all four workstreams. We are thrilled to have an advisory board comprising leading experts on AI’s societal impacts: Erik Brynjolfsson, Michael R. Strain, Audrey Tang, Helen Toner, and Robb Willer, as well as a great group of researchers and collaborators.

As with the previous work that the three of us have done, STABL will produce four main types of outputs.

• Conceptual work and public writing. AI discourse is polarized. We aim to improve it through conceptual work that identifies points of disagreement and empirical work that resolves them. We have written an award-winning book (AI Snake Oil), essays in the Wall Street Journal and WIRED, and a newsletter with over 90,000 readers.

• Empirical research. We plan to collect evidence about AI progress and its impacts on society. Our open-ended evaluations of RSI through the CRUX project have been among the first to evaluate AI progress on open-ended AI research tasks. Our agent benchmarks have been adopted by leading AI companies, and the Foundation Model Transparency Index has set the standard for responsible disclosures by AI companies. Our findings have been covered in New York Times, the Financial Times, the Atlantic, and others.

• Evidence synthesis. We plan to synthesize evidence of AI’s impacts across domains. Our work on open foundation models laid the groundwork for understanding their societal impacts, and we have analyzed AI’s effects on science, law, and software engineering.

• Policy guidance. We provide independent expertise to inform AI policy. Our work has directly shaped US and EU AI governance on transparency, third-party evaluations, and openness. We have contributed to both versions of the International AI Safety Report. And we have helped define and actualize the vision of evidence-based AI policy.

To follow our work, subscribe to our newsletter. We’d love to hear about other organizations in the top-right quadrant. We know that many people share our thinking, and what is missing is a systematic research program, an ecosystem of organization- and movement-building, and funding to sustain and amplify the work. We hope to play a role in building this intellectual project.