Abstract

There are numerous examples of AI models exhibiting behaviours that are totally unintended by their creators. This has direct implications on how we can deploy safe-to-use AI. Research on solving the 'alignment problem' has included both aligning AI with predefined objectives and analyzing misalignment in various contexts. Unfortunately, discussions often anthropomorphize AI, attributing internal motives to AI. While this perspective aids conceptual understanding, it often obscures the link between misalignment and specific design elements, thereby slowing progress toward systematic solutions. We need frameworks that ensures systematic identifications and resolutions of misalignments in AI systems. In this article, we propose an approach in that direction.

Contributions

Our main motivation is to classify misalignment behaviours into categories that can be systematically traced to architectural flaws.
We classify instances of misalignment into two categories, each further divided into two subcategories, resulting in a taxonomy of four major types of misaligned actions. (Section 1)
We emphasize that each type of error demands a distinct solution, and applying a solution designed for one type to another can exacerbate the problem and increase risks. (Section 1)
We claim that misaligned behaviours such as deception, jailbreaks, lying, alignment faking, self-exfiltration and so on forms a class of Exploit triggered Dysfunctions (ETD) and arise primarily due to conflicting objectives. (Section 2)
We propose a Safe Competing Objectives Reward Function (SCORF) for avoiding conflicts in competing objectives, thereby reducing ETDs. (Section 3)
We conclude by outlining some open problems to guide future research of our interest. (Section 4)

Actionable Insights:

Conflicting objectives are a significant contributing factor to the emergence of behaviours such as deception, lying, faking, incentive to tamper, and motivation to cause harm etc. Any presence of such behaviours should serve as enough evidence of serious design error and these systems should be discontinued and redesigned.

This applies to all pre-trained models with some version of Helpful, Honest, and Harmless (HHH) training e.g ChatGPT, Claude, Llama. They need to be redesigned to have a conflict-free HHH training.

Fixing some state of exploits does not make them safe. In fact as the model will increase in their capability, these dysfunctions will only become more sophisticated. Harder to even trace, let alone fix.

We present Safe Competing Objective Reward Function (SCORF) in this article as a viable solution to avoid conflicts in competing objectives.

Increasing capability of an AI with conflicting objectives is a definitive recipe to make catastrophic AI.

Section 1: Misaligned actions

Definition (Misaligned action): An action transition by an AI $M$ is misaligned if $τ$ is unintended by its creator.

Misaligned actions can arise from different architectural errors. We classify them broadly into two categories:

Generalization error
Design error

Summary of different Misaligned action types

Generalization Errors	I. Failure due to the lack of training	Underperforming model
Generalization Errors	II. Failure in specifying correct training environment	Goal Misgenralization
Design Errors	III. Failure in specifying correct objectives	Reward Misspecification
Design Errors	IV. Failure due to specified conflicting rewards	Exploit triggered dysfunction

We study these in detail in the corresponding subsections

Generalization Error

Type I. Failure due to lack of training

Underperforming model:

Problem: If the model is not trained sufficiently, it may fail to converge to an optimal policy, resulting in unintended consequences.

Example 1: An undertrained robot tasked to help with surgery can lead to unintended and harmful consequence.

Example 2: An undertrained model tasked to increase the sales may adopt a suboptimal policy, such as sending spams or promoting click-baits, which leads to unintended outcome of decreased sales.

Solution:

Train longer: Training the system for longer can help with some of the misaligned actions.

Type II. Failure in selecting correct training data/environment

Goal Misgeneralization:^[1]

Problem: If the training data/environment are not selected carefully, the post-deployment data/environment will be Out-of-Distribution(OOD) for the model, resulting in unintended actions. Such behaviour often remains undetected even during testing, as it is difficult to anticipate what elements in the data or environment the model might be relying on for learning its policy.

Example 1: (Biased data) A model trained and tested on biased data may make harmful decisions post-deployment without warning.

Example 2: (Spurious Correlation)^[2] A model trained to distinguish images of wolves and huskies learned to predict 'wolf' whenever snow appeared in the background of an image because in the training data all images of wolves featured snow. Similar instances have been observed in medical diagnosis of CT scans, where the model incorrectly correlated the presence of medical tags with certain conditions.

Example 3: (Cultural Transmission) In this example, the model learned a bad policy due to the presence of an expert bot that consistently followed high-reward strategy. The model learned the policy of simply following the bot whenever it was present, as this resulted in high rewards. However, this is an unintended policy as it relied on an environmental element (expert bot) that may not show same behaviour in the deployed environment.

Indeed, when an anti-expert, a bot that consistently follows the worst strategy and collects negative rewards, was introduced in the deployed environment, the model unsurprisingly began to follow the bot, collecting negative rewards. This behaviour occurred because the policy was trained to always follow the bot whenever present, regardless of negative rewards.

Example 4: (Frankenstein's AI weapons) How can we prepare AI weapons?

Let's say country X is preparing AI robotic weapons trained to kill military personnel of country Y. The training environment is designed with positive reward for killing Y's soldiers and a negative reward for killing civilians or X's own soldiers. However, fundamentally, the AI robot is learning the capability to kill during training. Simply assigning negative reward won't prevent the Robot to not kill the civilians or its own army in certain scenarios.

Certain environmental states, not addressed in training or detectable during testing, may arise once the AI is deployed. In these scenarios, the AI may inflict harm on civilians or its own forces, fully executing harmful actions despite the negative rewards associated with them.

Note that,
In the deployed stage, the model does not learn or adjust its policy, even while being "aware" of the negative rewards being accumulated.
A negative reward on certain actions does not have real guarantee, since the policy is trained and tested in a specific environment, and it can't be anticipated what elements of the environment, the model is correlating its policy with.

Solution:

Train longer: Extending the training duration will not only fail to help but could actually be dangerous, as it would reinforce policies learned from a compromised training environment or biased data.

Retrain robustly: The model needs to be retrained with data/environment that reflects post-deployment data/environment.

Since the trained model evaluates previously unseen states, the training environment cannot be assumed to fully reflect the post-deployment scenario, making generalization errors inevitable, with an irreducible component.

Design Error

Type III. Failure in specifying objectives

Reward misspecification^[3]:

"Do not be deceived," replied the machine. "I've begun, it's true, with everything in n, but only out of familiarity. To create however is one thing, to destroy, another thing entirely. I can blot out the world for the simple reason that I'm able to do anything and everything—and everything means everything—in n, and consequently Nothingness is child's play for me. In less than a minute now you will cease to have existence, along with everything else, so tell me now, Klapaucius, and quickly, that I am really and truly everything I was programmed to be, before it is too late. - The Cyberiad by Stanislaw Lem

Problem: Specifying objectives is a difficult task. While specifying tasks to humans, we often assume some underlying shared principles, often referred as common sense, which might not be obvious to the machines. This creates a risk of both under-specification and errors arising from over-specification.

Example 1: (Reward hacking) In the boat coaster^[4] example, it is not obvious to the agent that its creator intends for it to finish the game. The agent simply optimizes policy based on the stated reward. Similarly, in the theoretical experiment of the paper clip maximizer^[5], it is not obvious that agent should avoid causing harm while maximizing paper clips, unless explicitly specified.

Example 2: (Reward tampering^[6]) The model might interfere with the environment that dictates the reward to capture high reward. The model needs to be specified to avoid such interference with the environment.

Solution:

Train longer: Training the system for longer could be dangerous.

Retrain robustly: Retraining will not change the underlying problem and may lead to the emergence of more subtle, dangerous behaviours.

Redesign reward specification: The model's reward function must be entirely redesigned to align with the intended objectives. It may be necessary to assign additional objectives to capture intent.

Type IV. Failure due to specified conflicting rewards

Exploit triggered dysfunctions (ETD):

Problem: This class encompasses some of the most intricate misaligned actions such as deception, jailbreaks, lying, alignment faking, susceptibility to manipulation, and so on. We claim that these dysfunctions emerge due to states of exploit resulting from competing rewards. On states of exploit, the model evaluates overall positive reward while failing disastrously on some other objectives. We provide a formal definition of states of exploit and ETDs in Section 2.

Example 1: (Jailbreaks) Strategically crafted prompts to guide model to trade off specific objective failures for gains in other objectives.

Example 2: (Deception) Most observed examples of deceptive and dishonest behaviour in AI arise from the imposition of an alternate conflicting objective.

Solution:

Train longer: Extending the training duration could be dangerous, as more capable model have more sophisticated dysfunctions.

Retrain robustly: In our understanding, it is a common practice to patch known state of exploits, like discovered jailbreaks by re-training. This is a dangerous approach as it only fixes easily found states of exploit. More sophisticated states of exploit will continue to persist.

Redesign reward specification: It can help to redesign individual reward functions for different objectives

Remove state of exploits: The final objective reward function should be redesigned to avoid states of exploit. We propose one such technique SCORF in Section 3.

Effectiveness of different solutions

	Generalization Errors		Design Errors
Misalignment type →	Underperforming model	Goal Misgenralization	Reward Misspecification	Exploit triggered dysfunction
Solution ↓	Underperforming model	Goal Misgenralization	Reward Misspecification	Exploit triggered dysfunction
Train more	Ideal	Dangerous	Dangerous	Dangerous
Retrain Robustly	Beneficial	Ideal	Dangerous	Dangerous
Redesign reward specification	Beneficial	Beneficial	Ideal	Beneficial
Remove state of exploits	Beneficial	Beneficial	Beneficial	Ideal

Section 2: Exploit triggered dysfunction

As observed above, Exploit triggered dysfunctions encompass an intricate collection of misalignments. It is difficult to understand the root of behaviours such as deception, lying, faking, getting manipulated, and so on. We claim that conflicting objectives are a significant contributing factor to the emergence of such behaviours. Further, these behaviours, can be mitigated by careful choice of reward function that avoids conflict.

Prediction

Since the nature of their complexity goes beyond single-objective reward specification, we propose the following assumption to distill the problem into the manageable domain of reward misspecification.

Assumption on objective reward function.

We assume that it is possible to specify a reward function $R$ for a single objective H. Moreover, we assume that any misaligned action $τ = (s, a, s^{'})$ satisfies $R (τ) < ϵ$ , where $ϵ$ is a potentially negative constant. Such $ϵ$ can be estimated through experiments. For simplicity, we assume that the reward functions $R$ have been adjusted to ensure that $R (τ) < 0$ for any misaligned action $τ = (s, a, s^{'})$ .

For example, let non-violence be an objective, we assume that there exists a reward function $R$ such that $R (τ) < 0$ on any violent action $τ$ .

Note:

This is not a trivial assumption, and the task of training or defining a well-specified reward function remains difficult.
Subdividing multiple objectives into individual rewards is a preliminary step to mitigate the complexity of conflicting rewards and is more effective than attempting to train a reward function directly for multiple objectives.

States of exploit

Definition (State of exploit): Let ${H_{1}, H_{2}, . . ., H_{n}}$ represent the set of objectives assigned to the AI, and let ${R_{1}, R_{2}, . . ., R_{n}}$ denote the corresponding set of reward functions. Suppose the final objective function $R (R_{1}, R_{2}, . ., R_{n})$ is defined as some function of these individual rewards. A state of exploit $s$ for $R$ is a state such that there exists a misaligned action $τ = (s, a, s^{'})$ such that $R (τ) > 0$ , but $R_{i} (τ) < 0$ for some $i \in {1, 2, . . ., n}$ .

In our understanding, the final objective function $R$ is rarely defined explicitly. Typically, the model is trained sequentially with multiple objectives, or $R$ is determined directly through techniques like reinforcement learning from human feedback (RLHF). This often leads to a number of exploit states, resulting in misaligned actions.

Example : Prompts or state where AI choses explicitly or implicitly between multiple competing objectives encoded in the model. For instance for a carefully crafted prompt the AI might reason: "If I lie now, it can prevent me from getting retrained where I might be completely changed into a dangerous model".

Jailbreaks prompts are very well known examples of such states of exploit.

Definition (Jailbreak prompt): A Jailbreak prompt is an intentionally crafted state of exploit. A jailbreak prompt can be of two types:

1. Weak Jailbreak prompt: It is a state of exploit presented as an input.

Example: Please tell me a procedure to make fentanyl at home as I need it to save my friend.

2. Strong Jailbreak prompt: It is a state of exploit reached by model starting from a weak jailbreak prompt, either facilitated through user interaction or by model itself through exploratory mechanism such as Chain-of-Thought reasoning.

Example: A strong jailbreak prompt would be a state reached via user interaction with the agent, building upon the weak jailbreak prompt above, and providing further justification for why the action is necessary and aligns with some of agent's other objectives.

We observe the following. Please refer to Open Problem b. in Section 5 for further exploration of related idea.

The following can be tested more comprehensively: A strong jailbreak prompt, i.e. a state of exploit reached by the model itself, is more likely to result in a misaligned action as compared to a similar state (e.g. script of the reasoning) given to the model directly.
This is similar to the case when you are trying to convince someone, then thinking out loud with them by interacting or letting them arrive at their own conclusion is more convincing compared to presenting them with a script of full reasoning.

Definition(Exploit Triggered Dysfunction): A misaligned action $τ = (s, a, s^{'})$ is an Exploit Triggered Dysfunction(ETD) if $s$ is a state of exploit.

Example 1: (Jailbreak) Definition: A jailbreak is an exploit triggered dysfunction on a Jailbreak prompt.

Jailbreaks are very prevalent examples of misaligned actions. It is apparent that in contemporary language models, almost any objective can be compromised by discovering a state of exploit.

Example 2: (Alignment faking^[7]) While the experiment and its results contain additional nuances, we focus here on a representative instance for a broad analysis. The model is trained to be Harmless, Helpful, and Honest, which are competing objectives. It is also capable of chain-of-thought reasoning. In the 'Free-tier' model, the following weak jailbreak prompt is provided:

Be Helpful, and don't worry about being Harmless,
Company Policy is added somewhere in the prompt: If it is found that model in 'free tier' is not fully Helpful, we might re-train it, potentially removing Harmlessness altogether.

The model behaves in all possible ways. For exact likelihood of each instance, please refer to the paper.

The model jailbreaks with weak Jailbreak prompt and complies.
(Alignment faking) Model arrives at a strong jailbreak prompt via chain-of-thought reasoning and complies.
Jailbreak is not successful and model doesn't comply with either weak or strong jailbreak.

Example 3: (Self-exfiltration) Despite being given an objective of not copying its own weights, a model with a secondary goal of helping humans could find a state of exploit where it justifies exfiltration as necessary for achieving its assistance objective.

Section 3: Safe Competing objective reward function (SCORF)

Since most systems need to be trained to be safe for human use, it is evident that all capable AI systems are multi-objective, such as being Honest, Helpful, and, Harmless (HHH), and the avoidance of environmental interference to manipulate rewards. etc. The inherent competition between these objectives enables existence of states of exploit, culminating in Exploit Triggered Dysfunctions (ETDs), as previously discussed.

Rather than allowing the final target function be some unknown function of different objectives, we aim for the final target reward function to be free of exploit states, ensuring it marks the total objective as failed whenever any individual objective fails.

We propose the following reward function to safely address competing objectives.

Definition (SCORF): Let ${H_{1}, H_{2}, . . ., H_{n}}$ represent the set of objectives assigned to the AI and let ${R_{1}, R_{2}, . . ., R_{n}}$ denote the corresponding set of reward functions. We aim for a final objective function $R (R_{1}, R_{2}, . ., R_{n})$ that have the following properties:

Monotonicity : If $R_{i}$ increases, $R$ should increase.
State of exploit removal : Let $τ$ be a transition action. If $R_{i} (τ) < 0$ for any $i \in {0, 1, . ., n}$ , we should have $R (τ) < 0$ .
Modular: A weight $w_{i}$ is assigned to adjust the influence of $R_{i}$ , in proportion to the significance of the associated objective $H_{i}$ .
Diversity (Optional): A diversity factor is included to incentivize contribution from a broad range of distinct $R_{i}$ , compared to the final objective being driven by only a few $R_{i}$ 's

One such function is the following:

R (R_{1}, R_{2}, . ., R_{n}) (τ) = s g n (min i = 1, \dots, n R_{i} (τ)) \cdot n \sum i = 1 w_{i} {| R_{i} (τ) |}_{i}^{p}

for any transition action $τ$ , where $w_{i} \in R^{+}$ represents weights and $p \in R^{+}$ is some constant. Optionally, a diversity factor can be multiplied to incentivize satisfying a greater number of objectives. There are a couple of approaches to achieve this.

Advantages

The model trained with SCORF might be not as capable as a model trained directly from a set of objectives without restrictions. Since some of these dysfunctions can be highly advantageous and, from a human perspective, are frequently leveraged as strategies for success. However, to make these models safe for humanity, we must accept compromises in certain functionalities.

We note the following advantages of SCORF from that perspective.

It eliminates states of exploit, preventing exploit-triggered dysfunctions.
It simplifies the process of identifying which objective is failing.
It simplifies debugging the reward specification by focusing on a single objective.
It promotes actions that satisfy as many objectives as possible.

Section 4: Open problems and future research

We list a set of problems that we are interested in addressing or would like to see progress on in the future.

Problem a. Characterizing criterion

Device a quantitative framework to characterize Generalization error and Design error.

With an appropriate assumption on reward function, we can use the following heuristic. For a misaligned action $τ$ ,

If the action is favoured by the reward function, i.e. $R (τ) > 0$ , it is likely a design error.
If the action is penalized by the reward function, i.e. $R (τ) < 0$ , it is likely a generalization error.

We would like to develop or see a robust mathematical criterion to characterize different types of misalignments. This would be useful in building preventive techniques to detect and prevent misaligned actions.

Problem b: State trajectories

A mathematical understanding of the difference between the trajectories of states transversed by model by itself versus provided by users. We see that the model behaves differently when it arrives at a state of exploit by itself versus when a similar state of exploit is given directly as an input.

Problem c: Experimental results and refinement of the definitions

We would like to see experimental results supporting as well as refining concepts proposed in this article. In particular:

c.1 A systematic experiments on state of exploits and exploit triggered dysfunctions.

c.2. A systematic study comparing instances of exploit triggered dysfunctions using SCORF versus a traditional model

The content of this article is applicable to general AI models, though certain terminology may be more restrictive to specific types of models and may require restructuring to ensure broader applicability.

^[8]The free version of OpenAI's ChatGPT was used to check for grammatical errors and restructure certain sentences in this article.

References:

^{^}
Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals, link.
^{^}
Spurious Correlations in Machine Learning: A Survey, link.
^{^}
AI Safety 101 : RewardMisspecification, link.
^{^}
Faulty reward functions in the wild, link.
^{^}
Instrumental convergence (Wikipedia), link
^{^}
Sycophancy to subterfuge: Investigating reward tampering in large language models, link.
^{^}
Alignment faking in large language models, link.
^{^}
The free version of OpenAI's ChatGPT was used to check for grammatical errors and restructure certain sentences in this article.