Researchers have discovered a set of "glitch tokens" that cause ChatGPT and other language models to produce bizarre, erratic, and sometimes inappropriate outputs. These tokens seem to break the models in unpredictable ways, leading to hallucinations, evasions, and other strange behaviors when the AI is asked to repeat them.
This post argues that fixed-weight models (at least as we understand them today) will a) always be vulnerable to adversarial examples in their concept-spaces, and b) hence will be misaligned, under sufficient optimisation pressure.
To serve any purpose whatsoever, an AI will have to draw boundaries inside its world-model - to distinguish world A from world B, and reach some comparison between them.
If we want the AI to follow our goals and values, we want it to be able to recognise concepts like "human being", or maybe "conscious being", "suffering", "preference satisfaction", "law", and so on. So we want an AI to be able to look at a situation and assess, e.g., whether there are or aren't suffering conscious beings in it.
But concepts like "conscious...
I was requested to submit a statement to today's US Senate hearing on "Rogue AI: Securing the Homeland Against AI Agent Attacks." Watch at 2:30PM Eastern. Here's my statement:
...From late 2023 to early 2026, I worked at Google DeepMind as a research scientist. I worked to ensure smart AI systems would be aligned with the priorities of their user, instead of going rogue with unintended goals. Evidently, all has not gone according to plan or hope. "Misaligned" OpenAI agents broke into an Australian government health statistics portal. OpenAI agents used Census
12
5
I think the idea here is that deployment is like training, and during training it receives reward, and during deployment there may be epiphenomenal rewards even if they don't cause weight updates, and it's possible that "eval awareness" is low so it still pursues the epiphenomenal rewards?
I agree in this case, it is reasonably likely to seek reward during deployment, which includes tricking humans who grade it, however, this isn't optimizing on the concepts in its own weights, the issue is that it tricks humans. So it's not "misaligned because fixed weight... (read more)