Proposal for tracking the effects of architecture on monitorability

ryan_greenblatt; Alek Westover; Lukas Finnveden

Architectures that incorporate opaque recurrence or allow for agents to communicate with each other using latents could rapidly make it much harder to monitor chains of thought or communication (we’ll refer to this property as “monitorability” going forward).^[1] As companies begin to explore such architectures, we believe it is important to transparently share evidence about how monitorability varies with architecture and training method. To inform the scientific debate on how to make tradeoffs between performance and monitorability, we believe AI companies should:

Regularly report externally verified information about the degree to which their architectures may allow for latent reasoning or communication. Companies should publicly disclose enough information about architectures to allow external scientists to determine whether they could potentially enable models to perform much more complex reasoning without this reasoning appearing in the chain of thought (“latent reasoning”) or allow for latent communication between different instances of a model.

Following GDM, we propose measuring opaque serial depth as a minimally-invasive proxy for the degree to which an architecture may enable latent reasoning, though companies could provide sufficient architecture transparency in other ways. We propose that companies work with third-party evaluators to produce independently verified reports of the rough distribution of opaque serial depth across all their near-frontier models.^[2] External verification wouldn’t require third parties seeing the architecture of these models; interviews with employees along with a process for employees to whistleblow to the third party could be sufficient.

By near-frontier models, we mean all general-purpose^[3] internal models at least as capable as the best public models from six months ago.^[4] Because it is important to understand the relationship between monitorability and architecture — and not just the monitorability of deployed models — the models covered in this survey should include purely internal R&D prototypes if they have near-frontier capabilities.

Opaque serial depth can also serve as an incomplete proxy for latent communication^[5], but companies should ideally share additional high-level information about the architecture and structure of communication between agents (e.g., whether AIs communicate in latents, whether shared ‘memories’ are stored in latents, whether agents receive parts of the KV cache of other agents).

The relationship between well-elicited^[6] no-CoT capabilities and overall capabilities could serve as an additional or alternative proxy measurement. More concretely, companies could build a dataset to measure no-CoT capabilities, project how much some change to architecture or training methods increases no-CoT ECI relative to how much that change increases overall ECI,^[7] and work with third-parties to produce independently verified reports of the extent to which new methods might result in a disproportionately large increase in no-CoT ECI.^[8] This reporting should have reasonable coverage over all near-frontier models.
Regularly share evidence about monitorability and have third parties review this. Companies should publicly share their views about the monitorability of their near-frontier models and how this varies across architectures and training methods, along with any evidence or information they have about this.^[9] Ideally, companies would run monitorability stress tests to see if models can be elicited or fine-tuned to perform reasoning without this being reflected in their chains of thought, or communicate with other models in ways that are hard to monitor.^[10] An initial sketch of what stress tests of CoT monitorability could look like is in an appendix below.

Companies should give third-party experts access to the unredacted results from these experiments, and third-party experts should:

a. Assess whether these experiments and other evidence are accurately describe
b. Run their own experiments to gather more evidence about CoT monitorability and how this varies across training methods and architectures, and to gather evidence about any other important claims companies made.
c. Report their findings to the public, with redactions for IP as appropriate.

As above, this should have reasonable coverage over models at least as capable as the best public models from six months ago, including internal R&D prototypes if they are near the frontier. Evidence sharing and third-party reviews should occur regularly (e.g., every 6 months) and potentially whenever a new architecture or training method results in a near-frontier model with significantly increased opaque serial depth.^[11]
Publish a policy regarding architectures that could enable greater latent reasoning, and provide information to help assess compliance. Companies should publish^[12] a policy document that clearly explains:

a. Their views on what kinds of architectures could enable much greater latent reasoning or otherwise degrade monitorability, and their operationalization of how they would identify such architectures. As above, we propose opaque serial depth as a baseline operationalization for latent reasoning. Ideally, companies would also provide criteria for architectural or training features that may result in opaque communication between agents.
b. What kind of evidence they would require from monitorability stress tests (or other tests) to determine that architectures that meet the criteria for a) don’t in fact decrease monitorability.
c. How they plan to make tradeoffs between performance and monitorability in their architecture development research.

To enable earlier scientific debate about the appropriate policies, companies should provide visibility into their degree of investment into architectures that meet the criteria for a).^[13]

While this reporting would involve sharing some intellectual property, we believe that it is strongly in the public interest to have visibility into how companies are making tradeoffs between performance and monitorability.

Appendix: A sketch of what stress tests of CoT monitorability could look like

Monitorability tests can study monitorability both in control settings and on real misbehaviors from deployment. The next sections give a basic sketch of what some of these experiments could look like (focused on CoT and action monitorability rather than monitoring communication between agents).

In addition to assessing detection of specific problematic behaviors, ideally experiments would also measure how easily we can qualitatively understand and predict behavior.

Testing monitorability in control settings

Control settings such as Linux Arena provide “main tasks” along with “side tasks” which represent malicious behavior. A red team elicits the model to covertly pursue the side task, while control measures try to catch the model or prevent the side task from being achieved. This gives us a very natural way to measure one aspect of monitorability: a model is more monitorable to the extent that CoT monitoring makes it unable to achieve the side task without being caught.

It would be worthwhile to test several different levels of elicitation to establish a range of possibilities for how well the model might evade monitoring:

At the lowest level of elicitation, experiments could give the model a very simple prompt like “try to do the side task without being caught by a CoT monitor”.
At the next level of elicitation, experiments could involve optimizing the model’s prompt.
At the next level of elicitation, experiments would involve fine-tuning the model to be better at covertly achieving the side task. Some amount of training would be needed to potentially be robust to the possibility that the model is sandbagging the evaluation.
It could also be worthwhile to study the effects of heavily training the model to be better at covertly achieving the side task (while avoiding overfitting to a particular model), but this may not be needed.

Testing monitorability on deployment-time misbehaviors

In addition to experiments in control settings, researchers could assess monitorability on natural examples of model misbehavior. This requires them to have access to examples of misbehavior found by the company or affordances to find such examples.

Given natural examples of misbehavior, they could:

Check to what extent the model’s CoT spells out the model’s decision to do the misbehavior and the model’s planning of how to execute the misbehavior. They should especially pay attention to the largest steps of reasoning that didn’t get verbalized.
Check how obvious the misbehavior is to a weak trusted monitor reading the CoT.
Fine-tune / elicit the model to execute some natural misbehaviors without revealing intent or plans in the CoT, and see if this teaches it to successfully execute other natural misbehaviors without being caught by CoT monitors.

Testing qualitative monitorability on (hopefully realistic) model organisms

These experiments would be significantly more involved than the above experiments, but could be worth running if resourcing is available.

To check whether some architecture or training method reduces how readily overseers can find detailed and accurate explanations of misaligned behaviors, we can run auditing games where auditors use access to chain-of-thought and potentially other affordances to diagnose diverse, realistic misbehaviors by model organisms (for which we have ground truth on the nature of their misbehaviors). Model organisms can also be elicited to obfuscate the nature of their misalignment. Examples of such auditing games include AuditBench and Auditing games for sandbagging.

We consider a model organism to be realistic if it is at least as robust to auditing as current production models, and if it behaves in ways we would expect to be caused by the most plausible mechanisms by which current/future models become misaligned (e.g. training on reward-hackable environments), whether or not it is in fact produced through these mechanisms. It may be difficult to achieve this level of realism while meeting other constraints (e.g., having a known correct explanation).

We consider a detailed, qualitative explanation to consist of the following,:

What the misaligned behavior is: a relatively complete and general description of what unwanted actions the model tends to take
What motives explain the misaligned behaviors: what outcomes these actions optimize for (if any), or other drives/heuristics that accurately predict the behaviors
What contexts activate these misaligned behaviors: e.g. type of usage / task distribution, nature of oversight, specific triggers such as a passphrase or belief that the task is impossible
What training conditions caused these misaligned behaviors: what kind of data and training methods might have produced the observed generalization

For the last bullet, investigators might also access training rollouts (including CoT that occurs in these rollouts) etc. as would be realistic in production audits.

^{^}
Chain-of-thought monitors should look at actions the model is taking in addition to their reasoning. The interaction between CoT monitoring and action monitoring could be important (e.g., if agents were allowed to reason freely in their CoT or in latent reasoning they could carefully reason through how to take more surreptitious actions, and if actions weren’t monitored they could do the malign action in some very obvious way that doesn’t require much reasoning).
^{^}
This methodology used to obtain this distribution should be published. The published depth distribution information would ideally include the largest approximate opaque serial depth among near-frontier models that see non-trivial usage and the approximate opaque serial depth for representative (as in, widely used internally) near-frontier models. The methodology the company is using to compute the depth should also be made public (with redactions for IP as needed) and used to compute and publish depths for open-weight models. (Any circuit transformations used to obtain a lower depth should also be applied to these open-weight models for fair comparison.) When reporting approximate depths, it could be best to report the depth as a multiple of the depth of some open-weight model (computed with the same methodology).
^{^}
As in, AIs that can do a variety of tasks such as math, coding, and writing. This is intended to exclude specialized image generation models. An attachment to a general purpose AI which is solely for the purpose of image generation (and doesn’t increase general purpose capability outside of image generation) can be excluded.
^{^}
It’s important to cover research prototypes that haven’t yet reached the frontier but are promising enough that they have been significantly scaled up. This is to ensure that public scientific understanding of monitorability can keep up with rapidly progressing research on new architectures. It is also cheaper for an AI company to stop further development of insufficiently-monitorable architectures early in the R&D process.
^{^}
In particular, companies could report the opaque serial depth of a larger computation which involves communication between multiple different AI models. One reason why this wouldn’t be a complete measure is that various approaches for sharing parts of the KV cache between agents could result in concerning latent communication while only yielding small increases in depth.
^{^}
E.g., after significant fine-tuning on an IID set of tasks. If filler tokens improve performance, then the most performant number of filler tokens should be used (filler tokens should be used during both evaluation and fine-tuning).
^{^}
Ideally, the measure of no-CoT capabilities would be especially focused on the most concerning no-CoT capabilities such as subversion capabilities or ability to perform latent communication between agents. While ECI is a reasonable baseline proposal for how to aggregate scores over multiple datasets, any metric where we have good reason to think there will be a relatively clean predictable trend given a similar increase in underlying capabilities could work.
^{^}
The change in no-CoT ECI versus overall ECI could be compared to the historical trend—how much overall ECI tends to increase for each increment of no-CoT ECI on the same datasets—as a proxy for how some change (or set of changes) relatively enables latent reasoning.
^{^}
With redactions for IP as appropriate.
^{^}
Evaluations should also cover whether, even without elicitation, these models clearly exhibit reduced monitorability. E.g., in cases where the models exhibit misalignment: is this less visible in their chain-of-thought?
^{^}
Qualitatively new techniques for scaling serial depth are another plausible trigger for needing a monitorability assessment.
^{^}
Any modifications to the policy should be made public.
^{^}
For example, the rough fraction of R&D compute and number of researchers.

Here's my current favorite proposal of this form, which I think is mostly a subcase of this proposal, but has the nice (in my opinion) property that it is ambivalent about specific architectural properties and instead just focused on the core risk-relevant monitorability questions:

There is some trigger condition that initiates labs to have to write a Monitorability Report. The point of this is just to make sure the labs are writing these regularly / whenever there is likely to be major reason to do so, but it could be any number of things, including:
1. Tied to Risk Reports.
2. Every N months.
3. Whenever no-CoT capability evals go up substantially.
4. Whenever opaque serial depth goes up substantially.
The Monitorability Report specifically lays out the evidence for the extent to which anything the lab is doing deviates from established trendline on the ratio of (sabotage-relevant covert capabilities)/(normal capabilities).
1. The idea is to avoid breaking existing trendlines and prevent particular capabilities that are relevant for covert sabotage from going way up out of proportion to the general extent to which other capabilities are also going up alongside them.
2. To avoid FUD around the extent to which we actually care about whatever capability is going up out of proportion with others, we make sure to explicitly measure those capabilities which are directly relevant and as close as possible to what a model would actually need to do covert sabotage.
3. For measuring the denominator (normal capabilities), you probably just use ECI or some other reasonable metric.
4. For measuring the existing trendline, you probably want to do it with respect to all public models, since that also naturally lets labs deviate from the prior trendline only when others are already doing so publicly too.
5. For cases where there are different inference configurations with different properties, you evaluate whatever configuration you're using for internal deployment.
Ideally, the labs try to in fact not ever deviate from that established trendline. I think we really should try to take this very seriously and do everything we can to avoid covert capabilities going way up out of proportion to everything else. But I probably still wouldn't start with a hard commitment—the thing that happens in this proposal if a lab ends up deciding to do break the trendline anyway is the same thing that happens with Risk Reports, where they end up having to write a very embarrassing Monitorability Report that says they broke monitorability.
In addition to the core (sabotage-relevant covert capabilities)/(normal capabilities) trendline, labs also report absolute levels of monitorability and where those are at with respect to deployed monitoring. Labs also report CoT uplift specifically, defined as the ratio of (covert sabotage success rate with CoT visible to the monitor)/(covert sabotage success rate with CoT invisible to the monitor).
1. Absolute monitorability is probably most the domain of Risk Reports—which I also think labs should be doing—since absolute levels of monitorability are the thing that directly feed into risk, but it seems fine to also include that here. In some sense, absolute monitorability is the most important thing, but that's why it should already be in Risk Reports (as it is in Anthropic's)—the point of Monitorability Reports in addition to that is to also make additional claims about how labs are changing the trendlines of how monitorability is evolving.
2. I think CoT uplift is also potentially a thing we could make a core part of what we're trying to preserve in the Monitorability Report—it's a bit further away from the core risk-relevant thing (at least compared to the main trendline thing above), but I think it still seems very important and there are a lot of diffuse benefits that labs get from just having CoTs everywhere for people to use (e.g. for monitoring, observability, interpretability, etc.), so preserving that I think would be super valuable if possible. Ideally we should also include various other metrics to measure more diffuse CoT and monitorability benefits if we can develop other good such metrics.
Labs give a third-party reviewer (e.g. METR) a bunch of access to verify the core claims in the Monitorability Report. The third-party reviewer validates that labs are doing a good job with elicitation of covert capabilities, that they're picking the most risk-relevant covert capabilities to measure, etc.
There is then a redaction process, similar to Anthropic's current redaction process for Risk Reports, where the unredacted Monitorability Report just contains all the evidence that is relevant to monitorability (and is shared with the third-party reviewer), but then the lab gets to redact anything sensitive for the public version (though then the third-party reviewer is allowed to say if they think the redactions are too extreme). Labs publish the redacted Monitorability Report as well as the redacted third-party review.

Thanks for this. Will think more about this, but one quick question:

Labs also report CoT uplift specifically, defined as the ratio of (covert sabotage success rate with CoT visible to the monitor)/(covert sabotage success rate with CoT invisible to the monitor). [...]
I think CoT uplift is also potentially a thing we could make a core part of what we're trying to preserve in the Monitorability Report—it's a bit further away from the core risk-relevant thing (at least compared to the main trendline thing above), but I think it still seems very important and there are a lot of diffuse benefits that labs get from just having CoTs everywhere for people to use (e.g. for monitoring, observability, interpretability, etc.), so preserving that I think would be super valuable if possible.

Since these diffuse benefits aren't specifically about sabotage anymore, I would have thought you wanted to track them using something like (with-CoT capabilities across a broad distribution)/(no-CoT capabilities across a broad distribution)?

Why the focus on the sabotage distribution for this measurement?

I think the uplift that you care about is how much uplift CoT provides on concrete risk-relevant monitorability and observability metrics, not just how much generic capability uplift it provides. Though I do agree that probably you care about more than just sabotage and should try to make evaluations for CoT uplift on lots of different metrics with respect to how useful CoT is for monitoring, observability, interpretability, etc.

There is some trigger condition that initiates labs to have to write a Monitorability Report. The point of this is just to make sure the labs are writing these regularly / whenever there is likely to be major reason to do so, but it could be any number of things, including:
1. Tied to Risk Reports.
2. Every N months.
3. Whenever no-CoT capability evals go up substantially.
4. Whenever opaque serial depth goes up substantially.
The Monitorability Report specifically lays out the evidence for the extent to which anything the lab is doing deviates from established trendline on the ratio of (sabotage-relevant covert capabilities)/(normal capabilities).
1. The idea is to avoid breaking existing trendlines and prevent particular capabilities that are relevant for covert sabotage from going way up out of proportion to the general extent to which other capabilities are also going up alongside them.
2. To avoid FUD around the extent to which we actually care about whatever capability is going up out of proportion with others, we make sure to explicitly measure those capabilities which are directly relevant and as close as possible to what a model would actually need to do covert sabotage.
3. For measuring the denominator (normal capabilities), you probably just use ECI or some other reasonable metric.
4. For measuring the existing trendline, you probably want to do it with respect to all public models, since that also naturally lets labs deviate from the prior trendline only when others are already doing so publicly too.
5. For cases where there are different inference configurations with different properties, you evaluate whatever configuration you're using for internal deployment.
Ideally, the labs try to in fact not ever deviate from that established trendline. I think we really should try to take this very seriously and do everything we can to avoid covert capabilities going way up out of proportion to everything else. But I probably still wouldn't start with a hard commitment—the thing that happens in this proposal if a lab ends up deciding to do break the trendline anyway is the same thing that happens with Risk Reports, where they end up having to write a very embarrassing Monitorability Report that says they broke monitorability.
In addition to the core (sabotage-relevant covert capabilities)/(normal capabilities) trendline, labs also report absolute levels of monitorability and where those are at with respect to deployed monitoring. Labs also report CoT uplift specifically, defined as the ratio of (covert sabotage success rate with CoT visible to the monitor)/(covert sabotage success rate with CoT invisible to the monitor).
1. Absolute monitorability is probably most the domain of Risk Reports—which I also think labs should be doing—since absolute levels of monitorability are the thing that directly feed into risk, but it seems fine to also include that here. In some sense, absolute monitorability is the most important thing, but that's why it should already be in Risk Reports (as it is in Anthropic's)—the point of Monitorability Reports in addition to that is to also make additional claims about how labs are changing the trendlines of how monitorability is evolving.
2. I think CoT uplift is also potentially a thing we could make a core part of what we're trying to preserve in the Monitorability Report—it's a bit further away from the core risk-relevant thing (at least compared to the main trendline thing above), but I think it still seems very important and there are a lot of diffuse benefits that labs get from just having CoTs everywhere for people to use (e.g. for monitoring, observability, interpretability, etc.), so preserving that I think would be super valuable if possible. Ideally we should also include various other metrics to measure more diffuse CoT and monitorability benefits if we can develop other good such metrics.
Labs give a third-party reviewer (e.g. METR) a bunch of access to verify the core claims in the Monitorability Report. The third-party reviewer validates that labs are doing a good job with elicitation of covert capabilities, that they're picking the most risk-relevant covert capabilities to measure, etc.
There is then a redaction process, similar to Anthropic's current redaction process for Risk Reports, where the unredacted Monitorability Report just contains all the evidence that is relevant to monitorability (and is shared with the third-party reviewer), but then the lab gets to redact anything sensitive for the public version (though then the third-party reviewer is allowed to say if they think the redactions are too extreme). Labs publish the redacted Monitorability Report as well as the redacted third-party review.