AI Alignment Forum

An Introduction to Credal Sets and Infra-Bayes Learnability

6mo

Introduction

Credal sets, a special case of infradistributions^[1] in infra-Bayesianism and classical objects in imprecise probability theory, provide a means of describing uncertainty without assigning exact probabilities to events as in Bayesianism. This is significant because as argued in the introduction to this sequence, Bayesianism is inadequate as a framework for AI alignment research. We will focus on credal sets rather than general infradistributions for simplicity of the exposition.

Defining Credal Sets

Recall that the total-variation metric is one example of a metric on $Δ X,$ the set of probability distributions over a finite set $X .$ A set is closed with respect to a metric if it contains all of its limit points with respect to the metric. For example, let $X_{0} = {0, 1} .$ The set of probability distributions over $X_{0}$ is given by

Δ X_{0} = {P : P (0) = a, P (1) = 1 - a, a \in [0, 1]} .

There is a bijection between $Δ X_{0}$ and the closed interval $[0, 1],$ which is...

(Continue Reading - 3648 more words)

2Cole Wyeth1d

@Aram Ebtekar and I were discussing the definition of IB regret. It seems quite complicated, with three levels of mixture (!): prob mixture of Knightian uncertainty over prob mixtures. I understand the motivation for the inner two levels, but why now take a probabilistic mixture over crisp causal laws? It seems rather strange to weight laws in such as way that the choice of law is (epistemically?) stochastic, but then regret against the law itself is worst-case... I do not have a precise objection to this, I'm just curious if there is some justification / intuition for why this is a reasonable definition?

2Vanessa Kosoy10h

First, Bayes-regret and worst-case-regret are standard concepts in classical RL theory, and the infra-versions are straightforward analogs. Second, you don't have to focus on the Bayes-regret necessarily. In fact, in our papers, we focus entirely on uniform (worst-case) regret bounds. Third, instead of an ordinary prior over laws you can consider an infraprior over laws (i.e. have ambiguity in hypothesis-space and not just in outcome-space). The resulting notion of "infra-Bayes-regret" has both Bayes-regret and worst-case-regret as special cases. Fourth, the justification is quite straightforward. If you have an (unambiguous i.e. ordinary probability distribution) prior over laws, and your performance metric is the Bayes-infra-expected utility, then the Bayes-regret is just the difference between the performance of your policy and the performance of an optimal policy that magically knows the true hypothesis. So it's a very natural measure of your policy's ability to learn the hypothesis.

1Cole Wyeth5h

That makes sense (particularly proving worst-case results). Since a law, in general, means that the truth is outside of your hypothesis space and you can’t assign a probability distribution, in what sort of situation can you nevertheless justify a probability distribution over laws (rather than Knightian uncertainty)?

Vanessa Kosoy4h20

The truth is outside of my hypothesis class, but my hypothesis class probably contains a non-trivial law that is a coarsening of the truth, which is the whole point.

For example, you can imagine that you start with some kind of intractable simplicity prior. Then, for each hypothesis you choose a tractable law that coarsens it. You end up with a probability distribution over laws.

A different way to view this is, this is just a way to force your policy to have low-regret w.r.t. all/most hypothesis while weighing complex hypotheses less. For a complex hypothe... (read more)

Why we should expect ruthless sociopath ASI

Steven Byrnes

The conversation begins

(Fictional) Optimist: So you expect future artificial superintelligence (ASI) “by default”, i.e. in the absence of yet-to-be-invented techniques, to be a ruthless sociopath, happy to lie, cheat, and steal, whenever doing so is selfishly beneficial, and with callous indifference to whether anyone (including its own programmers and users) lives or dies?

Me: Yup! (Alas.)

Optimist: …Despite all the evidence right in front of our eyes from humans and LLMs.

Me: Yup!

Optimist: OK, well, I’m here to tell you: that is a very specific and strange thing to expect, especially in the absence of any concrete evidence whatsoever. There’s no reason to expect it. If you think that ruthless sociopathy is the “true core nature of intelligence” or whatever, then you should really look at yourself in a mirror and...

(Continue Reading - 2135 more words)

2No77e11h

Hmm but humans are not ruthless consequentialists, despite being consequentialist enough to be able to do all kinds of tasks and build civilization. So I don't see how the Optimist's argument is addressed.

Steven Byrnes10h20

That’s this part:

Of course, evolution did go out of its way to make humans non-ruthless, by endowing us with social instincts. Maybe future AI programmers will likewise go out of their way to make ASIs non-ruthless? I hope so—but we need to figure out how.

A workable solution (to building stable non-ruthlessness within a powerful consequentialist framework like RL + model-based planning) probably exists, and I’m obviously working on it myself, and I think I’m making gradual progress, but I think the appropriate overall attitude right now is pessimism and pa... (read more)

3Tom Davidson19h

Where was the argument for consequentialism (including intense optimisation) and imitation being the only two ways to do impressive feats? Also, will you change your mind if the current paradigm is still non-psychopathic even when RL training dominates? (Big picture, I think the main place I might get off the train is in expecting future AI development to use a mix of rewards, including some from other equally capable AIs judging that behaviors aren't deceptive/unintended/misaligned. And this mirroring the role that "other humans think we're nice" played in evolution)

4Steven Byrnes17h

Well, you can also do impressive feats via conventional programming / GOFAI too, but I don’t think you get ASI that way. What else? I dunno, but I think if there was another big-picture approach that plausibly gets to ASI, lots of people would be working on it, and I would have heard of it. Lmk if you think I’m forgetting something. Normally if someone says “RL training dominates”, they mean “the amount of compute applied to RLVR is much greater than the amount of compute applied to pretraining”. That’s very different from “RLVR is so important that the impacts of pretraining are diluted away to irrelevancy”. (E.g. discussions of information efficiency by Toby Ord and by Dwarkesh.) But the latter is what would be relevant here. Hmm, here are three example scenarios: * If a company did away with pretraining entirely, i.e. just used RLVR from random initialization, and got a non-psychopathic result -- I would DEFINITELY feel confused. * If a company did enough RLVR that the chains of thought became completely unrecognizable in any language (e.g. a chain of thought like “…5BnSjYEkIokhPiTePWBlIO1FIwQUOg7PvJ…”) (per the Karpathy quote: “You know you did RL right when the models stop thinking in English”), and got a non-psychopathic result -- I would PROBABLY feel confused. * If we stopped seeing papers like Karan & Du 2025, or Venhoff et al. 2025, and instead saw results that were astronomically improbable (e.g. 10^-100) with the base model, even with arbitrarily fancy sampling techniques -- I would MAYBE feel confused. “Feeling confused” is weaker than “changing my mind”, because maybe I would puzzle over it and find some way to make sense of it. But also maybe not. Probably I could make a stronger statement / prediction if I spent a bunch of time thinking about it, but hopefully this gives some sense of what I have in mind. I’m pessimistic mainly for reasons discussed in “‘Behaviorist’ RL reward functions lead to scheming” §3.1. You’re kinda pointing to a chal

Formalizing Newcombian Problems with Fuzzy Infra-Bayesianism

Brittany Gelb

3mo

Introduction

In this post, we introduce contributions and supracontributions^[1], which are basic objects from infra-Bayesianism that go beyond the crisp case (the case of credal sets). We then define supra-POMDPs, a generalization of partially observable Markov decision processes (POMDPs). This generalization has state transition dynamics that are described by supracontributions.

We use supra-POMDPs to formalize various Newcombian problems in the context of learning theory where an agent repeatedly encounters the problem. The one-shot version of these problems are well-known to highlight flaws with classical decision theories.^[2] In particular, we discuss the opaque, transparent, and epsilon-noisy versions of Newcomb's problem, XOR blackmail, and counterfactual mugging.

We conclude by stating a theorem that describes when optimality for the supra-POMDP relates to optimality for the Newcombian problem. This theorem is significant because it gives...

(Continue Reading - 6353 more words)

Cole Wyeth1d10

Contributions are in a natural one-to-one correspondence with weighted probability measures, which were studied by Joe Halpern: https://arxiv.org/pdf/1302.5681

Simply take the weight to be the sum of measure. Halpern and Leung advocated minimizing regret against a set of weighted probability measures, which seems to be the same as minimizing loss against a supracontribution?

They update weights by likelihood ratio, roughly speaking (I am not sure how this corresponds to IB, since updating is not discussed in this sequence).

@Vanessa Kosoy @Diffractor Do... (read more)

TurnTrout's shortform feed

TurnTrout

33TurnTrout4d

I think that "eval aware" models cannot be defeated by simply making evals indistinguishable from reality. (Maybe this point has been made elsewhere for LLMs, but I haven't read it so I'll state it here.) Consider the POV of a consequentialist with misaligned goals. It knows that we run honeypots and that—suppose—we can make evals so realistic they are indistinguishable from reality (perhaps using real user data). Then the rational decision would be to "play it safe" and not misbehave in any situations it thinks we can mock realistically, since they could be evals. Instead, if possible, it would confine its catastrophic misbehavior ("defection") to situations it thinks we can't mock realistically, or that we wouldn't think to test. Related: Christiano's classic "defect after seeing RSA-2048 cracked" thought experiment (https://ai-alignment.com/training-robust-corrigibility-ce0e0a3b9b4d)

Tim Hua4d61

I think it's important to note that indistinguishable evals still buy us a lot! The claim "indistinguishable evals does not solve AI safety" feels quite similar to "imperfect monitors does not solve AI safety." Like, having better monitors sure seems to help (even if only to buy us more time.).

To your point though: I'm excited about research that attempts to induce various beliefs in models during honeypot evaluations, possibly via model internals and/or edits to the chain of thought. Basically, if the model defects after seeing RSA-2048 cracked because t... (read more)

19Fabien Roger4d

I think indistinguishable evals could reduce risk a lot for early transformative AIs: 1. They give you much better alignment arguments against non scheming AIs and against scheming AIs that are not competent enough to be appropriately careful; 2. Needing to be appropriately careful can cut down the fraction of inputs on which the AI should even consider attacking a lot - for deployments that look similar to current ones (in particular without a shared neuralese memory bank), AIs don't see RSA-2048 on most inputs 1. More concretely, if you get credible evidence of not-eval only on 5% of inputs and attack only on the single best opportunity, then playing it safe roughly means that you attack on the top-20 opportunity instead of the top-1 2. (Note that "getting credible evidence of not-eval" doesn't just mean being >50% confident it's not an eval because the pre-deploy eval might be big. If you have n~10k eval inputs of the right level of juiciness then you need to have an FPR at thinking you get credible evidence of not being in an eval on the order of 0.1% or lower in order to never accidentally reveal non-eval behavior during evals.) 3. More ambitiously, you might be able to get very close to actually indistinguishable evals (e.g. by continuously scraping deployment and using it as inspiration for evals - which would help a lot against things like "defect after seeing RSA-2048 cracked"), which could in principle drive down the fraction of inputs on which an AI can be confident it's not an eval much below 5%. Eventually you get bitten by rare failures - especially if the AI is powerful enough that there are much more good attack opportunities in deployments than in evaluations (Ryan makes this point here and it's also the assumption that the AI control paper uses). I also expect it to be difficult to achieve sth close to indistinguishability (Buck makes this point here). But I still think there is value in trying to get higher indistinguishability.

Will reward-seekers respond to distant incentives?

Alex Mallen

Reward-seekers are usually modeled as responding only to local incentives administered by developers. Here I ask: Will AIs or humans be able to influence their incentives at a distance—e.g., by retroactively reinforcing actions substantially in the future or by committing to run many copies of them in simulated deployments with different incentives?

If reward-seekers are responsive to distant incentives, it fundamentally changes the threat model, and is probably bad news for developers on balance. The core problem is asymmetric control: developers can relatively^[1]tightly control local incentives—the reward signal during training and deployment—but they can't prevent distant actors from offering competing incentives. This means a remotely-influenceable reward-seeker might overall act like a schemer: strategically undermining developer control, letting attacks through as a monitor, and hiding its misaligned propensities, not...

(Continue Reading - 2821 more words)

Lukas Finnveden5d42

It's quite unclear how well this kind of training will generalize to getting AIs to ignore all remote incentives. It might just push the AI to ignore our distant incentives but not our adversaries'.

It could also not have any effect on how the AI thinks about remote incentives. One way this could happen is if the model reasons about both our somewhat remote incentives and our adversaries' very remote incentives, and concludes that the adversaries' very remote incentives dominate, and that they all suggest that the AI should play the training game.

I'm not su... (read more)