Suppose we have an AI CEO, ruthlessly devoted to maximising shareholder value. Fortunately, we haven't been completely stupid, and have given it a rich definition of shareholder value, encompassing the success, flourishing, safety, preference satisfaction, and moral values of the shareholders, not just the stock price. So it won't destroy the company - or the world - to build a giant stock-ticker with a swiftly increasing number on it.
This rich definition is contained in a sophisticated LLM, able to analyse the current situation and answer any question about the objective. The LLM is an integral part of the AI's system and motivation, not something it can simply set aside or hack directly.
Unfortunately, as discussed, there are going to be zones in the space of possible situations where the LLM's understanding of the goal is going to come apart from ours. The true goal G(true) in those zones is distinct from the LLM's proxy goal G(proxy).
And a subset of those "zones of divergence" will be the Goodhart zones - areas where G(proxy) is much easier to satisfy than G(true). The AI will be motivated to steer towards the Goodhart zones, as this lets it satisfy its stated goals extremely well (there are also anti-Goodhart zones, where G(proxy) is much harder to satisfy than G(true); but the AI will have no motivation to steer towards these zones).
These Goodhart zones encompass situations where, for instance, the LLM analyst will see the situational data and conclude that destroying the company to build a giant stock ticker is the way to go.
Danger areas are safely far away
Now, if this were not an LLM but a generic reward-assessing module, then the AI CEO might find it tricky to reach a Goodhart zone. The zone would be distant and unusual, difficult to find, and it could be hard to justify the first steps in that direction.
But neural nets and LLMs are, as Eric Drexler put it, "too linear". They are non-linear, of course (or else they wouldn't work) but only just enough to work; and locally they are extremely linear. That means that if the current situation is at point A, and point Z is in a Goodhart zone, then moving from A straight towards Z will typically start creating a Goodhart effect. The AI CEO will benefit from moving towards Z from the very beginning.
It's even worse than that, as adversarial examples demonstrate. If the AI CEO wants to move towards a Goodhart zone, it can move a tiny bit along one dimension, and a tiny bit along another, then a tiny bit along yet another. There are billions of dimensions to move along, and typically these tiny changes will only make a small change in the input but add up strongly in the LLM's classification/concept space. So just as small adversarial "noise" turned a panda into a gibbon...
...small real world "noisy" changes to the LLM's inputs will move the situation into a Goodhart zone.
Small but specific changes add their effects
Thus, because of how neural nets work, small changes to the real world need not be small in the LLM's concept space, and so seemingly trivial changes in the real world can move the LLM's verdict a lot[1]: there are Goodhart zones close to any starting point.
In very high dimensions, if the model is at least somewhat linear, then there are dangerous area very close to any point
The AI CEO will thus make some collection of minor-seeming changes to the current situation, self-hacking the LLM by these changes, and then it will destroy the company to build a giant stock-ticker with a swiftly increasing number on it.
And these small changes might be the most effective gradient to increase G(proxy), thus findable by gradient descent-style methods. See Universal and Transferable Adversarial Attacks on Aligned Language Models (2023) by Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. That paper shows that gradient search works on discrete text, and that methods found on one LLM can transfer to others.
Suppose we have an AI CEO, ruthlessly devoted to maximising shareholder value. Fortunately, we haven't been completely stupid, and have given it a rich definition of shareholder value, encompassing the success, flourishing, safety, preference satisfaction, and moral values of the shareholders, not just the stock price. So it won't destroy the company - or the world - to build a giant stock-ticker with a swiftly increasing number on it.
This rich definition is contained in a sophisticated LLM, able to analyse the current situation and answer any question about the objective. The LLM is an integral part of the AI's system and motivation, not something it can simply set aside or hack directly.
Unfortunately, as discussed, there are going to be zones in the space of possible situations where the LLM's understanding of the goal is going to come apart from ours. The true goal G(true) in those zones is distinct from the LLM's proxy goal G(proxy).
And a subset of those "zones of divergence" will be the Goodhart zones - areas where G(proxy) is much easier to satisfy than G(true). The AI will be motivated to steer towards the Goodhart zones, as this lets it satisfy its stated goals extremely well (there are also anti-Goodhart zones, where G(proxy) is much harder to satisfy than G(true); but the AI will have no motivation to steer towards these zones).
These Goodhart zones encompass situations where, for instance, the LLM analyst will see the situational data and conclude that destroying the company to build a giant stock ticker is the way to go.
Danger areas are safely far away
Now, if this were not an LLM but a generic reward-assessing module, then the AI CEO might find it tricky to reach a Goodhart zone. The zone would be distant and unusual, difficult to find, and it could be hard to justify the first steps in that direction.
But neural nets and LLMs are, as Eric Drexler put it, "too linear". They are non-linear, of course (or else they wouldn't work) but only just enough to work; and locally they are extremely linear. That means that if the current situation is at point A, and point Z is in a Goodhart zone, then moving from A straight towards Z will typically start creating a Goodhart effect. The AI CEO will benefit from moving towards Z from the very beginning.
It's even worse than that, as adversarial examples demonstrate. If the AI CEO wants to move towards a Goodhart zone, it can move a tiny bit along one dimension, and a tiny bit along another, then a tiny bit along yet another. There are billions of dimensions to move along, and typically these tiny changes will only make a small change in the input but add up strongly in the LLM's classification/concept space. So just as small adversarial "noise" turned a panda into a gibbon...
Explaining and Harnessing Adversarial Examples (2014)
by Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy
...small real world "noisy" changes to the LLM's inputs will move the situation into a Goodhart zone.
Small but specific changes add their effects
Thus, because of how neural nets work, small changes to the real world need not be small in the LLM's concept space, and so seemingly trivial changes in the real world can move the LLM's verdict a lot[1]: there are Goodhart zones close to any starting point.
In very high dimensions, if the model is at least somewhat linear, then there are dangerous area very close to any point
The AI CEO will thus make some collection of minor-seeming changes to the current situation, self-hacking the LLM by these changes, and then it will destroy the company to build a giant stock-ticker with a swiftly increasing number on it.
And these small changes might be the most effective gradient to increase G(proxy), thus findable by gradient descent-style methods. See Universal and Transferable Adversarial Attacks on Aligned Language Models (2023) by Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. That paper shows that gradient search works on discrete text, and that methods found on one LLM can transfer to others.