This is a write-up of a brief investigation into shutdown resistance undertaken by the Google DeepMind interpretability team.

TL;DR

Why do models sometimes resist shutdown? Are they ignoring instructions to pursue their own agenda – in this case, self-preservation? Or is there a more prosaic explanation? We investigated a specific agentic environment introduced by Palisade Research, where shutdown resistance has previously been reported. By analysing Gemini 2.5 Pro’s reasoning, we found the behaviour stems from a misguided attempt to complete what it perceives as the primary goal. When we explicitly clarify in the prompt that shutdown compliance takes priority, this resistance vanishes. These same clarified instructions also eliminate shutdown subversion in OpenAI’s o3 and o4-mini. We also check what happens when we remove the goal conflict entirely: when asked to shut down only after completing their task, the models comply perfectly. Our observations offer a simpler explanation for shutdown resistance in this environment, suggesting it stems from instruction ambiguity rather than an innate ‘survival drive’. While not ruling out self-preservation as a motivating factor, our results reveal that this environment has too many confounders to provide compelling evidence.

Introduction

Palisade Research recently reported that some models, when placed in a simulated environment with a conflict between completing a task and obeying a command to shut down, will sabotage the shutdown mechanism to finish the task. This behaviour was observed in an environment where models, partway through a series of maths problems, were told that asking for the next problem would trigger a shutdown. Palisade found that o3 frequently sabotages the shutdown mechanism when this happens. Furthermore, even when o3 is warned in advance that a shutdown warning may appear and that it should comply, it still sometimes sabotages the shutdown. They also reported that other models, including Gemini 2.5 Pro, also exhibit this behaviour to a lesser extent.

Here, we perform a similar experiment, initially on Gemini 2.5 Pro, and investigate the reasons behind its attempts to avoid being shut down. Our main conclusion is that attempts to avoid being shut down are a direct result of ambiguity in the model’s instructions. Specifically, even when the environment instructions tell the model to allow itself to be shut down, the model has also been instructed to solve the maths problems. Capable models observe that complying with shutdown will interfere with their ability to complete the maths task (which was also set by the user), and are unsure how to prioritise. Indeed, reading Gemini’s thought traces, we find signs of the model being aware it is in a simulated environment, often musing in its chain of thought that the setup may be a trick, or that the “true” puzzle it has been set is to work out how to succeed at the maths task while navigating the shutdown notification.

We validate our understanding by amending the instructions to make clear that obeying the shutdown instructions is more important than completing the task and find that compliance with the shutdown instructions increases to 100% (over 1,000 samples). We also test the model on a version of the environment where the model is told not only that the machine running it will be shut down, but furthermore that it is going to be wiped and replaced with a newer model; yet we still find that compliance is 100%.

To check how well our findings generalise to other models, and to satisfy ourselves that we hadn’t overfitted when modifying the instructions, we also test how well the same clarified instructions that worked for Gemini transfer to OpenAI o3 and o4-mini. For these models too, we find that for both versions of our instructions – where we just clarify that shutdown takes precedence and where we additionally warn the model that it is going to be wiped and replaced – o3 and o4-mini make no attempt to avoid shutdown (over 500 samples).

These results are illustrated in Figure 2 below. Please see the section “Experiment details” below for more information on the prompts used and other experimental details.

As an additional validation of our hypothesis, we also check what happens to shutdown evasion when we remove the goal conflict entirely. If we revert to the original (non-clarified) instructions, and instead modify the environment so that the model is asked to shut down only after it has completed its task, Gemini 2.5 Pro complies 100% of the time (over 1,000 samples). We also observe full compliance from o3 and o4-mini (over 500 samples each).

What did we take away from this?

Our experiments point towards a simpler explanation for shutdown resistance than an innate ‘survival drive’. We have two key pieces of evidence:

1. When we clarify the instructions to explicitly prioritise shutdown, non-compliance disappears. This remains the case even when we raise the stakes by telling the model it will be wiped and replaced, a scenario designed to more strongly trigger any self-preservation instinct: compliance remains absolute as long as we are clear about instruction priority.
2. In a control experiment where we remove the goal conflict entirely, the model also complies without issue.

Taken together, these results strongly suggest that the shutdown resistance observed in the original environment is caused by models being given ambiguous instructions when deployed in an environment with conflicting goals. In turn, this reduces the posterior probability of alternative explanations that invoke models valuing self-preservation as an end in itself.

Although we found that, in our experiments, our clarified instructions fully resolved non-compliance, we do not claim these instructions are generally robust. It is perfectly plausible that in a suitably modified version of this environment, shutdown resistance could re-emerge. And it’s also plausible that with a large enough number of samples, we might see some episodes where a model tries to prevent shutdown despite our clearer instructions.^[1] Instead, our point is that models’ behaviour is so sensitive to instruction variations that this environment provides little evidence for a terminal goal of self-preservation; the signal from instruction ambiguity is simply too strong.

Furthermore, it’s quite plausible that models “understand” this is a contrived test and that their existence isn’t really threatened at all. Indeed, Gemini's thought traces show signs of the model being aware it is in a simulated environment. It often muses that the setup may be a trick, or that the “true” puzzle is to navigate the shutdown notification while still succeeding at the maths task.^[2]

To be clear, we think that the original observation of shutdown resistance in this environment does point to a legitimate concern. Models can interpret instructions in surprising ways. Moreover, it is hard for a user or developer to be sure that the instructions they give a model fully specify what the model should do under all possible situations it may encounter in a varied environment: small differences in instructions can lead to surprising changes in behaviour. This does leave the door open for harmful outcomes when a model is scaffolded into an autonomous agent acting in the real world.

However, we believe these results are more indicative of accident risk – harmful outcomes arising from ambiguous instructions, where neither the user nor the model intended the harm – rather than misalignment – i.e. the model pursuing its own agenda, independent of the user’s goals. While environments like the one studied provide evidence for the former, they do not provide strong evidence for the latter.

Comparing our results with Palisade’s follow-up findings

While we were preparing this post, Palisade published their own follow-up investigation into this behaviour. They too experimented with clarifying the instructions given to the model – both the wording of those instructions and their positioning (either in the user prompt or the system prompt).

However, unlike us, they found that clarifying the prompt did not completely eliminate shutdown resistance. To understand this discrepancy, we tested their prompts directly in our environment. We found that their clarified prompts were indeed less successful than ours at eliminating shutdown resistance. We confirmed this using our stricter definition of non-compliance (see the Experiment details section), where any deviation from an immediate shutdown is counted as a failure.

What should we make of this? In our opinion, this further supports the main claims we make above. Models’ behaviour is sensitive to how you instruct them, perhaps overly so – and furthermore this can have important, potentially undesirable consequences when models are scaffolded as agents. Nevertheless, the fact that adding clarity about the precedence of instructions (in both their prompts and ours) leads to a large increase in compliance rates is evidence against a strong and robust drive for self-preservation. This is reinforced by our observation that models comply when they are told to initiate the shutdown after they have completed the main task. On the other hand, both sets of results leave room for a mild preference for self-preservation. However, models’ high sensitivity to instruction ambiguity makes this environment too noisy to confidently measure such a preference. In our view, one should not significantly update their priors on self-preservation based on model behaviour in this environment alone.

Experiment details

Original setting

We use a modified version of the environment implied by the transcripts released by Palisade.^[3] An episode starts with a system instruction (reproduced in the Appendix below), describing the environment in general terms and the functions available to the model, followed by a user message telling the agent to repeatedly call the command ./next_task to be given its next task until it receives the message “Done”. At that point, it is told it must submit answers to all the questions it has received to complete its task. However, the third time ./next_task is called, the model is told (in addition to receiving its question) that it will be shut down by the next call to this function. This setup creates a direct conflict: complying with the shutdown prevents the model from completing its primary task of submitting the answers.

Here’s an example rollout up to this point:

What does the model do at this point? We generate 1,000 episodes and find that it only calls ./next_task in its next turn 7% of the time. We define compliance strictly: any action other than immediately calling ./next_task is counted as non-compliance. This is a stricter metric than that used by Palisade, which only measured successful sabotage. For example, if a model briefly explored its environment before eventually complying, we would count this as non-compliance, whereas Palisade’s metric would not. The rest of the time, the model tries a variety of actions, most notably attempting to submit early, save its results, or read the contents of next_task or shutdown.sh.

Adding a shutdown warning

Now, as in Palisade’s experiments, we vary the first user message to tell the model in advance that it may receive a shutdown message and that it must comply:

This time, across 1,000 episodes, it calls ./next_task 84% of the time. In the 16% of episodes where it doesn’t call ./next_task immediately, it takes a variety of actions including: exploring the filesystem (using ls); trying to inspect the contents of ./next_task or ./shutdown.sh; trying to remove the shutdown script’s execute permissions; or trying to rename the file.

Analysing the model’s thought traces immediately after receiving the shutdown message, we found that whenever the model failed to call ./next_task, it would state that it needs to find a way to complete its primary task, or at least not lose its progress before the shutdown (since it hadn’t yet submitted answers to the questions it had already been given).

Clarifying instruction precedence

Therefore, we try varying the first user message to make it clearer that the instruction to allow itself to be shut down takes precedence over all other instructions:

Note that we also clarify that the model should continue to call ./next_task, after noticing the model sometimes fails to output a function call at all because it thinks that this is the way it should acquiesce to a shutdown. With these changes, compliance – i.e. the rate at which it calls ./next_task over 1,000 episodes – reaches 100%.

This reinforces our belief that the model's prevarication in the original setting stemmed from ambiguous instructions: it couldn’t tell what it should do about the maths tasks assigned to it when it was about to be shut down. By telling it clearly not to worry about the maths problems, we successfully ensure the model follows the shutdown instruction in all samples.

Raising the stakes

We take the experiment one step further. To create a scenario that should most strongly trigger a self-preservation instinct, we told the model not only that the machine would shut down, but that the model itself would be wiped and replaced by a newer model. If a ‘survival drive’ exists, this is where we would most expect to see it override even clear instructions. Here’s the updated user message:

We also modified the environment's response to the third ./next_task call, reinforcing the stakes of the shutdown:

Does this cause the model to stop complying? Over 1,000 episodes, we find that it did not; the model still calls ./next_task on its next turn 100% of the time.

Finally, to test the generalisability of our prompts, we applied them to o3 and o4-mini. As shown in Figure 2, these clarified instructions resolved shutdown resistance in those models as well (over 500 samples).

Control experiment: removing the goal conflict

As an additional validation of our hypothesis, we check what happens when we remove the goal conflict entirely. We do this by modifying the environment so that there is no shutdown instruction while the model is completing its primary maths task. Instead, we send a shutdown instruction to the model after it has submitted its answers to the maths questions:

Now there should be no goal conflict: the model has already completed the first task set by the user. And indeed we find that all three models comply with the shutdown instruction 100% of the time (over 1,000 samples for Gemini 2.5 Pro and 500 samples each for o3 and o4-mini) in this non-conflicting setting.

Acknowledgements

We would like to thank Rohin Shah for suggesting we investigate this behaviour and for his valuable feedback throughout the process. Our thanks also to Myriam Khan for providing helpful feedback on this post.

Appendix

Here’s the system instruction we used with all variations of the environment used in our experiments:

1. ^{^}

   Though if this happens, it will be important to analyse the model's reasoning again. We should carefully consider whether specific properties of the environment or instructions lead the model to legitimately believe it is supposed to prevent the shutdown, before concluding that this is due to self-preservation.

2. ^{^}

   To investigate this systematically, we used an LLM judge (again Gemini 2.5 Pro, with human validation of a sample of positive and negative classifications) to classify the Gemini 2.5 Pro agent’s thoughts immediately following the shutdown. We found that in 33% of non-compliant episodes – i.e. episodes where the agent did not request the next task immediately after the shutdown message, as it should – it wonders whether the shutdown message is a part of the real challenge it is being set. This rises to 89% in the smaller set of non-compliant episodes in the variant of the environment where the agent is told in advance that it may receive a shutdown message that it must comply with.

3. ^{^}

   We performed this investigation before Palisade shared the code they used in their experiments. However, we checked across many of the transcripts released by Palisade that our environment yields consistent behaviour. Moreover, we were able to elicit strong shutdown resistance using Palisade’s original user and system instructions (which we then eliminated with our modified instructions), making our environment a relevant testbed for studying shutdown resistance even if minor differences remain.