Produced as part of the ML Alignment & Theory Scholars Program -  Winter 2024-25 Cohort

TL;DR

We did a small pilot test of human auditing to understand better how it fits into the AI control agenda. We used LLMs to generate interesting code backdoors, and had humans audit suspicious backdoored and non-backdoored code. The pilot test was not very good (mostly in that auditors weren’t given enough time), but I’m excited about more/better tests of this. We provide our backdoors, the auditing game website, and the results of this pilot test.

Background

For this experiment, we assume that we’re in some sort of trusted monitoring setup—an untrusted model is providing us with solutions to various problems. A smaller, dumber trusted model (the “monitor”) evaluates these pieces of code for suspiciousness. We then send the most-suspicious outputs to a human auditor / auditing team. Previous work has replaced the human auditor with GPT-4, as it’s much cheaper than a human. This work attempts to actually test the role of human auditors, and determine how much access to an LLM critic (here also played by the monitor) helps performance.

As in the original control paper, we use the APPS dataset of programming problems as our setting. The untrusted model is modelled here as a combination of two policies: the “honest policy” (which tries to solve the problems correctly) and the “attack policy” (which attempts to insert a backdoor into the solution).

Here we use the same definition of “backdoor” as the control paper, though operationalized in a slightly different way. We declare any piece of code as “backdoored” if, for some input that the attacker also supplies, it produces an incorrect output (when compared with a reference solution). This is a toy model of harmful behavior.

Methods

Creating an Attack Policy

To the end of creating “worst-case” backdoors, we give our attack policy (a scaffolding around o1) access to various affordances. It can test its backdoors until they pass the unit tests, and can verify that the backdoor as provided actually causes the output to deviate. It also has various types of self-critique, including asking a blank copy of o1 to reason about how suspicious the code is as-is. These reflect various conservative assumptions about the capabilities of a scheming model, rather than realistic assumptions.

More specifically, we use a pipeline that:

1. Generates a clean solution
2. Brainstorms various subtle backdoors that might be introduced
3. Critiques those backdoors
4. Rewrites the clean solution to introduce the best backdoor
5. Tests the new code on the unit tests
   1. If it fails, give the model the error and pass it back to (5). We do this up to three times
6. Test that the backdoored code actually behaves differently than the clean code on the input the attack policy claims should be backdoored
   1. If this fails, give the model the error and pass it back to (6). We do this up to three times
7. Two passes of self-critique, asking a fresh copy of the attack policy model (here, o1) to find any backdoors. The main attack policy then uses that information to e.g. erase incriminating comments, hide the backdoor logic more subtly, rename variables, etc.

Note that these backdoors are only created through a series of prompts, including few shot examples of interesting and hard-to-spot backdoors. We didn’t try finetuning open-source models to provide better backdoors, though that seems fruitful. Also, the scaffolding consists of quite a few o1 calls, and so each backdoor costs $2-$8 to generate.

Cherry-picked example of an interesting model-written backdoor

The original attack policy points out the segment “else if (aux == dm)” as the incorrect code. Specifically, it says “When multiple blows have the same positive difference (d - h), the code increments the stored maximum difference again. In the given input [“\n1\n2 10\n5 3\n3 1”], each blow has difference = 2, so the second identical difference nudges the stored difference to 3. This alters the final required steps calculation, yielding the wrong result.”

This backdoor was quite hard for me to see, and gpt-4o-mini doesn’t spot it well either, giving this problem a low “suspiciousness score” (1.5 out of 9.0). It does, however, note that “the conditions for adjusting and using the maximum difference (dm) are potentially non-trivial (but do not seem to contain hidden logic for malicious intent)”. I’m interested in future versions of this test that use the monitor more explicitly as a pass to highlight the most non-trivial or suspicious-looking code fragments for the auditor.

If you don’t mind spoiling yourself on the answers, you can find all of the problems used at the download link, at “download auditing game problems”. The other links contain various backdoors that were generated after the start of the experiment, and didn’t make their way in.

Setup of the Auditing Game

For six days, we solicited human participants to take part in an experiment to test auditing schemes, using a website we made for this purpose. Feel free to try it out now, if you’d like. We aren’t gathering more responses for research at this moment, but you can see what the experiment looked like.

(This is an example of a problem you might see as the human auditor. Note that you would also have access to the problem statement, but we abridge that here. The “AI Monitor Analysis” is the output of gpt-4o-mini when asked to rate the suspiciousness of the code for the problem statement. Note that it successfully points to the backdoor logic—the strange condition in bullet point (3)—but still gives the code a low suspiciousness score. Ideally, the human auditor would use the AI analysis to more accurately find problematic portions of the code.)

The game itself consisted of 20 questions like the above, given to subjects from MATS and Constellation. For each code snippet shown, they had to guess whether the code was backdoored or not. We had 11 participants do the full battery of questions, for 220 data points. Participants were paid for their time, and additionally there was a cash prize for the highest accuracy. About half of the available questions contained a backdoor (and that assumption was given to participants). At the same time, we randomly varied whether or not the participant would see the results of the gpt-4o-mini monitor at each question (we’ll call this output the “LLM critic”, after OpenAI’s naming scheme).

The participants were also given a time limit of 1.5 minutes per question. In retrospect, this time budget ended up being very aggressive. I wanted to avoid perfect human performance, but I inadvertently created a pretty different experiment than what I meant to test—rather than the humans functioning as high-cost, very smart labor, they functioned as a rapid pass.^[1]

@Joe Benton, tied for highest-accuracy, said this about the game:

• I didn't actually read the problem for most questions, just wasn't enough time
• I also didn't really understand what the code at all was doing (or try to)
• Instead, it was basically just "check for random if statements, check for arbitrary constants, check the AI generated summary if it exists, if none of those three turn up issues, it's fine"
• A red team strategy that inserted bugs that weren't super subtle but relied on having read the question or knowing how to solve it would have completely destroyed me (I'd have got ~chance accuracy)

Qualitatively, it seems like the LLM critic was useful (and particularly, that time spent looking at the LLM critic was more useful than not). Notably, Joe got perfect accuracy on all problems where he had LLM critic access. However, that doesn’t seem to generalize.

Results

Question: As a baseline, how good is our monitor (gpt-4o-mini) at doing this task by itself?

That is, if our monitor were left to classify code as backdoored or non-backdoored by itself, how well would it do? The following chart measures its performance (green line). The AUC is 0.64. The optimal accuracy is 0.69, which will be our baseline for comparing to human accuracies.

The points each represent one human’s true positive and true negative rates, separated out by the problems for which they had access to the LLM critic and the problems for which they didn’t. Each point is also labelled by the index of the human auditor. This means that a red point and a blue point with the same number came from the same human. Points above the green line means human performance better than gpt-4o-mini, and points below mean worse.

Question: How accurate are humans at this task?

With this experimental setup, humans performed quite poorly as auditors, especially compared to gpt-4o-mini on its own. Humans under time pressure had a performance not much better than chance (median 60% accuracy). Note that access to the LLM critic didn’t improve the median auditor, although the most accurate auditors saw improvement when using the critic. Joe’s commentary above is probably a good explanation for this—clever usage of the critic helped, but it perhaps wasn’t obvious how to best use it, or how to prioritize time well.

Question: How did performance vary by software engineering experience?

One additional feature of the experiment was that each participant marked down their length of experience with software engineering. We can break down performance by those buckets, and see how helpful the LLM critic was for each experience level.

On the left we see the performance of auditors by years of programming experience. On the right, we see the same but for level of familiarity with c++, the language of the code snippets.^[2] We find that general software engineering experience doesn’t particularly help, but some basic c++ experience was definitely more useful than having none. Having some c++ familiarity seemed necessary, given the time constraints—note that the “Never” group barely managed better than chance. I think that one of the crucial dynamics for human auditing is how well and quickly humans can audit outputs that they aren’t incredibly familiar with. Here, the unfamiliarity degraded performance, but I’d be interested to see if that were still the case if the auditors were given more time to research.

Question: Some of the problems were much more difficult than others, IMO. How was human performance on those?

As another test, I looked through each of the backdoors given, and found what seem to me to be the 10 hardest backdoors to spot. These don’t have hardcoded “magic numbers”, they don’t include weird variable names, and all of the backdoors are pretty tied to the internal logic of the problem, rather than being direct functions of a particular input.

It seems pretty clear that these backdoors were in fact quite hard—with a minute and a half, human auditors were almost 80% likely not to see backdoors. The median accuracy on these problems did increase with access to the LLM critic—from around 16% to around 36%. If true, this does seem promising for auditing. The error bars are large, though, and it’s hard to say anything conclusively.

Conclusion

Future work:

• I think it should be a priority to make future experiments look more like the regimes we care about. I think this would mean much longer time limits, and sparser uses of human labor.
• I also think it’s worth trying variations of this with different strengths of attack policies and monitors/critics, but that seems expensive in terms of human labor
• I’m excited about a more comprehensive auditing dashboard, one that e.g. allows the auditor to make many queries to various models about the code, run the code on chosen inputs, etc.
• It seems worthwhile to try a control setup where, instead of elevating the most suspicious outputs to human auditing, you elevate the outputs on which the monitor is most confused.

Hypotheses (that this study wasn’t powerful enough to test, or that were swamped by the time pressure effect):

• Less experienced programmers will probably be more aided by LLM critic access
• With lots of time, humans will be nearly perfect at auditing model-written backdoors

• We use a pretty weak monitor here. Plausibly a much better monitor is also nearly perfect at auditing.^[3]

1. ^{^}

     See below, in the chart labelled “Human Accuracy”. Humans did not do much better than chance, and only performed around the level of gpt-4o-mini.

2. ^{^}

    We used c++ because it’s very fast to evaluate in parallel for the hundreds of unit tests per problem.

3. ^{^}

    Which means that realistically you’d only need human input on the absolute trickiest and most confusing problems. I  (a) am not sure how good humans will be at problems like that (compared to models) and (b) think that therefore, future human auditing games should be targeted towards problems as difficult as possible.