This post was written as part of the Stanford Existential Risks Initiative ML Alignment Theory Scholars (MATS) program. thanks to Evan Hubinger for insightful discussion.

Introduction

Interpretability tools allow us to understand the internal mechanism and knowledge of a model. This post discusses methods through which interpretability could reduce x-risk from advanced AI. Most of these ideas are from Neel Nanda’s post on the same topic, my contribution comes from elaborating on them further. I have split these into a handful of different categories. These are not disjoint, but the framing helped with the analysis. The last section includes several reasons why interpretability might not be impactful that apply, for the most part, across the board. The conclusion discusses what this analysis implies for how we should approach interpretability research today.

In evaluating a method of impact, I have tried to analyse the following:

Mechanism & Scale: What is the mechanism of impact? how valuable would this be?
External considerations: what things outside of our interpretability tools need to happen in order for this impact to be realised and how likely are they?
Side effects: what possibility is there of a negative impact on x-risk?
Requirements: what do we require of our interpretability tools, as measured by:
- Understanding: what properties of our model do we need to be able to understand?
- Competitiveness: how cost-effective do our tools need to be?
- Performance impact: to what extent can our tools impact the final model performance?
- Reliability: how reliable must our tools be?
- Typology: what types of interpretability tools can we use?

With regards to understanding, it is useful to split this into two classes: comprehensive tools, which let you evaluate properties of the full model, and partial tools, which let you evaluate properties of only a part of the model. This distinction is the same as worst-case transparency introduced by Evan. Both of these define a spectrum based on what the class of properties that you can evaluate is. I will refer to the size of this class as the ‘strength’ of the tools.

With regards to reliability, this captures whether the tools work across different models, as well as under gaming attempts from deceptive models or overfitting attempts from SGD.

When referring to type, I have in mind the transparency trichotomy. Specifically, we can ...