Unpredictable Patterns #124: Regulating orchestration

Preserving human judgment, abandoning the ontology of "models" and tracking emerging orchestration patterns

Dear readers!

Over the summer we will mostly focus on exploring ideas in a shorter format. The idea is to introduce an idea, examine the landscape around it and then suggest a few questions at the end that could be interesting to pursue. These lighter summer notes will hopefully be inspiring comments! If you are looking for a good podcast for your summer listening I had the pleasure recently of joining the Foresight Institute for a discussion of AI-policy. Have a listen here.

Orchestration is emerging as a term of art in describing one key aspect of multi-agent artificial intelligence. It does not have a single, agreed-upon, crisp definition, but it means something along the lines of “the careful choice and organization of different agents to accomplish a task”. Picking the right agents, the right data for any retrieval augmentation etc can all be a part of what orchestration refers to - and so can the construction of multi-agent hybrid teams with both human and artificial agents working together.

The term could also be quite interesting for policy makers looking for a target for regulatory models: it offers a general enough metaphor and specific ways of intervening that should be attractive to analyze in greater detail. Just as Anthropic suggested that we could have constitutional AI - we should think about the orchestration of AI as a regulatory challenge.

Orchestration is configuration of agents in different roles: suggesting, for example, that any sufficiently complex agent system has a committee of supervisory agents fulfilling certain requirements would be a way to seek to guarantee robustness and resilience in such systems. As such it is hardly alien or new to legal thinking: requiring that a corporation has a board, auditors and certain accountabilities is a simple example of regulating how agents are orchestrated in an organization.

Roman contract law provides the clearest early example. A single agent—the contracting party—could execute agreements efficiently, but Roman law mandated seven witnesses for wills and multiple witnesses for major contracts. These witness-agents created orchestrated validation systems where each agent could play a specific role: observing, remembering, and later testifying to the contract's authenticity.

Regulating orchestration is different from regulating technology. When we regulate technology we try to ensure that every agent is safe-by-design - focusing on orchestration we try to make sure that the outcome of agentic interaction always is safe, even if there may be flaws in individual agents. When we regulate orchestration, we are not even regulating systems - but the way interactions and relationships in a system should be set up. The system may be designed in many different ways, but the end result should be a set of functions that flow from the way that the system is orchestrated. Orchestration requires thinking about the roles of different agents in achieving a task - just like we think about the role of instrumentalists in playing a musical piece.

One does not replace the other, but complements it.

Orchestration can be regulated in many different ways, and in specific sectors of the economy orchestration standards may well emerge in organizations focused on best practices. In medicine, for example, the orchestration of different agents - artificial and human - will be a key focus for safe medical care. This will go beyond simple rules like “a human in the loop” and include the use of agents that monitor compliance in patients, report medical status etc — to agents that constantly revise and review treatment options based on the latest science etc. Some of these orchestrations will be deemed basic, and so important that they will be codified in different ways.

The same is likely to be true for military use cases. Simple orchestration rules have always existed in the military - the simplest being dyadic orchestration of nuclear launches on submarines, for example — where agency is split between two different agents.

In many ways orchestration is just the same thing - but with more complex allocations of agency across a network of different agents. Orchestration is also closely related to division of labor - but with that added component: it is a division of agency as well. And so we will look to find optimal orchestration patterns that we can work with.

Any closer study of how we use AI today reveals interesting, emerging such orchestration patterns. One of the key patterns is simply the use of one model to crictize or validate the output of another model — here many have understood naturally that critic-orchestrations help reduce hallucinations and creates a check on models.

Others ensure that when they ask advice they set up two different models to give advice for and against a decision. This model is also old: it is the pro and con-model that has been used in decision design for a very long time. It is an interesting one for agent systems - because it preserves human judgment.

We should take a moment and think about this, and note that human judgment is likely to be a key property that we want to preserve in any multi-agent decision architecture. One of the best ways of doing this is to ensure that models constantly argue different sides, and that they do so competently - shifting roles so that no favoritism can occur.

The right orchestrations may even strengthen human judgment overall - ensuring that we are asked to make active choices far more often than we usually do. The use of AI in education is a prime example of where this model should be implemented: if we can teach children the skills associated with good judgment, their autonomy and ability to work with cognitive technologies is likely to be much greater than if we just give them single agent recommendations or tutoring.

Good judgment is likely to be a key skill in the future - and the way we orchestrate agentic artificial intelligence can either maximize or eliminate it.

Regulating orchestration is going to require that we conduct research into what orchestration patterns are most effective and provide the greatest promise of resilience and robustness. Is it a single auditor model? Or 100? What are the best orchestration patterns for different decisions? As we research this we should also learn lessons of diversity and complexity, and seek to maximize diversity in the orchestration of different tasks as well — human as well as artificial diversity where we look to find the best possible combinations.

Our regulatory focus so far has been on models - and this approach is running into severe limitations. Models are no longer monolithic things - they are orchestrated networks of tools, capabilities, cognition - and the idea that you can test a model to ensure safe outcomes is woefully inappropriate to the kind of systems we now see emerging. Shifting from models to orchestration gives us a different set of legal concepts to work with, and is likely to help figuring out new regulatory challenges.

Thanks for reading!

Nicklas