Suppose we have a capable and potentially scheming model, and before we deploy it, we want some evidence that it won’t do anything catastrophically dangerous once we deploy it. A common approach is to use black-box alignment evaluations. However, alignment evaluations are only reassuring to the extent that the model can't reliably[1] distinguish the deployment distribution from the evaluation distribution, as it is otherwise difficult to rule out the possibility of alignment faking.
There are many approaches one could use to try to make evaluations appear more realistic: you can try to create realistic environments (e.g. Petri, WebArena, OSWorld); use data from past deployments (e.g. OpenAI, SAD); and spoof tool-call responses (e.g. ToolEmu).
However, the core difference between an alignment evaluation and a...
As various people have written about before, AIs that have long-term memory might pose additional risks (most notably, LLM AGI will have memory, and memory changes alignment by Seth Herd). Even if an AI is aligned or only occasionally scheming at the start of a deployment, the AI might become a consistent and coherent behavioral schemer via updates to its long-term memories.
In this post, I’ll spell out the version of the threat model that I’m most concerned about, including some novel arguments for its plausibility, and describe some promising strategies for mitigating this risk. While I think some plausible mitigations are reasonably cheap and could be effective at reducing the risk from coherent scheming arising via this mechanism, research here will likely be substantially more productive in the future...
if the selection of memories happens in clear rounds where memories are evaluated according to rewards, then I don't see clear structural differences between training and deployment.
Isn't another structural difference that in deployment, this round structure may be happening many times in parallel, leading to increased variance in deployed models and increased chance that one of the models is misaligned?
E.g. say we have some sort of continual learning where deployed models specialize over the timeline of weeks-months to different task distributions. I thi...
Decision theory is back in fashion (defining fashion as "one good post on a good EA blog"). Bentham's Bulldog (BB) has published a case against FDT (functional decision theory), contrasting rationalist enthusiasm with academic scepticism: "Academic decision theorists don't like the theory. The number of academic decision theorists who adopt it could be counted on one hand by someone missing four of their fingers."
I am, just barely, a published academic decision theorist, so you can keep a small finger to count me too. My position is that, though FDT may have problems with its definitions and under-definedness, we can build defined variants that achieve what FDT attempted to.
I want to do two things in this post. First, sketch a "pragmatic" version of FDT designed to sidestep the...
It depends on what we apply the word "rational" to
People tend to use "rational" it to defend their preferred position, so I find discussions often degenerate into fruitless semantic debates.
The context is MIRI's twist on Axelrod's Prisoner's Dilemma tournament. Axelrod's competitors were programs, facing each other in an iterated Prisoner's Dilemma. MIRI's tournament is a one-shot Prisoner's Dilemma, but the programs get to read their opponent's code. Or, rather, a description of the behavior of the code in Gödel-Löb provability logic, which turns out to be enough to determine their behavior in the setup.
One fun result, right in the beginning of the paper, is about a program, FairBot, whose behavior is specified by "I'll cooperate with you if you (provably) cooperate with me". Despite the appearance of circularity, FairBot cooperates with itself. The proof involves Löb's theorem, so we call this Löbian cooperation.
Andrew Critch has suggested another way of proving self-cooperation. Instead of Löb's theorem, we use what he calls "Payor's lemma". It...
Man, okay I'm now wondering if the following works for a PrudentBot implementation? Calling our PrudentBot
In symbols,
This is derived from the limit of an algorithm that searches for increasingly long proofs that we can obtain the opponent's cooperation one way or another, with some budget for looking harder for a (defect, cooperate) proof s...
Thank you for the flag! Yeah, I do think the posts Deployment Awareness Matters More Than Evaluation Awareness and If This Were a Test, How Much Would It Cost? are relevant here. [1]
To give me two cents on this:
Agreed. Though I would frame this more as "taxonomy of the most important underlying differences between evals and deployments + discussion of when these differences translate into a reliable signal". An attempt to gesture at some... (read more)