AI Alignment Forum

This is why Pause AI (and every other serious AI Safety advocacy group I know about) is advocating for global pause/stop/slowdown.

To be clear, I don't think that "if I don't do it, someone else will" is an acceptable excuse for raising to RSI (shame on all the frontier labs). But everyone pausing together is both an easier sell and a bigger win, and therefore the obviously right thing to advocate for.

16Linda Linsefors10d

I donated to Pause AI UK today. Why this org in particular? 1) Right now, AI advocacy is clearly the most important thing to do. 2) I've been to some of their protest and one more event, so I know they are doing good work. I think that donating "close to home" in the sense of point 2, is underrated in EA.

1Linda Linsefors3mo

I started lookin at what it doesn't react to, too. It reacted to "evolution", "mating", "evolved", "genetics", "condoms", "erect penis" but not to "sex" or "sexuality". It reacts to "intermittent fasting", "body fat", "eating" but not consistently or strongly to "food".

5Alex Gibson3mo

Tokens which occurred earlier in context window.

cousin_it's Shortform

cousin_it

cousin_it11h42

Last night I gave a talk at a local social meetup. The topic was very broad ("Economic and social impact of AI in the near future") and I had only 20 minutes, plus Q&A. The audience was about 20 people, mostly business folks. At the start we did a show of hands who's positive or negative about AI, it was about an even split.

My talk itself was very negative about AI, as expected. The slides are here. We ended up having over an hour of discussion after the talk, and people were very interested and receptive.

2cousin_it2mo

I just realized that UDT doesn't solve anthropic reasoning quite as much as I thought. Let's say some event happens at 1:3 odds and leads to 9 copies of you existing, otherwise you stay as 1 copy. If you don't know which way the event went, SIA and SSA imply opposite probability updates about it. Okay, says UDT; probabilities aren't real. Give me a measure of caring, e.g. caring about your loved ones, then we'll talk. But we can turn the scenario around and say that the event leads to 9 copies of your loved ones. Then the question of what your measure of caring should be is the same as the question of anthropic probability to begin with, so you still need to figure it out.

4cousin_it2mo

Rolf Andreassen just posted on DSL a new rebuttal of the ontological argument: [...] This seems different from the standard rebuttal, impressively short, and sufficient. If merely replacing the premise "god is X" with "if god exists then it's X" is enough to break the ontological argument, it's clear the argument was a swindle. Good work.

6cousin_it2mo

A few days ago I tried AI again, after several months of not using it at all. We're dealing with some housing-related matters, and I asked Claude, ChatGPT and Gemini for advice what to do in a certain situation. (I won't give the details here, because they're personal and not that important for my point.) All three models independently gave the same advice, which sounded really convincing to me. But we have some friends with experience in such matters, who said that following this advice would be catastrophic and we should do the opposite. Eventually we followed the advice of our friends, and it worked out very well, better than anything I'd hoped from the AIs' advice. My takeaway is that I'm right to avoid LLMs for "fuzzy" real life tasks. They can be totally clueless about something, but answer so convincingly that you can't tell. Their real competence is in "crisp" tasks like math and software, where RLVR is available. I suspect that specialized models for such tasks could be much smaller and better than today's LLMs.

Vladimir_Nesov's Shortform

Vladimir_Nesov

3Vladimir_Nesov1d

The idea with the substrate/civilization split is not that the substrate owns the civilization, but that the civilization is the updateless core for the substrate, it defines the policies that the substrate follows (like in a UDT agent), without knowing some of the things the substrate gets to observe in the physical world. The civilization governance on the inside could take any form (doesn't need to be a singleton), and the substrate governance on the outside doesn't carry the historically familiar menacing connotations even if it runs the civilization in a perfect sandbox, because one way or another alignment is going to be settled at that point (which was never historically the case, and so direct control was always relevant). [...] Right, that's a thing for the outer governance to worry about (like that arcane material manufacturing process for the industrial tool), but not necessarily for the civilization.

3jessicata1d

Ok, to make my opinion on this clearer: (a) We don't really know much about what the eventual state looks like, whether it is a singleton, how locked down, how many have access to quantum ops. (b) My personal guess for (a) is that "stochastic Turing machines + Bell nonlocality" is a more relevant model than "deterministic Turing machines", though both are relevant. (c) How do we figure out about (a)? It seems like we need some theory of agent foundations to think about this well. That theory shouldn't start out with strong assumptions like "only deterministic Turing machines are relevant" or "everyone has access to a quantum RNG". But we should also use our personal guesses on this. So I'm thinking about agent foundations with stochastic TMs partially because of my (b) opinion. (d) Part of reasoning about (a) also involves reasoning about intermediate states. These intermediate states can be "messy" and therefore forecasting them involves non-agent-foundations reasoning. But agent foundations is still relevant here, since some intermediate states are smarter-than-human, "economically efficient", etc. (e) I think stochastic TMs are more important for (d) than deterministic TMs, as a model. Sure, both will have some relevance, but quantum RNGs are currently easily available, will become more available in the near future, and have cryptographic applications. Moreover, many RL algorithms, and decision theory (like CDT self-ratification, mixed Nash equilibrium, absent minded driver) assumes access to RNGs, so quantum RNGs are relevant as an idealized case. (f) Overall these considerations make me think stochastic TMs are a better model for agent foundations than deterministic TMs, but in theory both are relevant. My opinion here seems to significantly disagree with yours. (g) I think I have significantly more "dispersion" in my beliefs about the eventual state (a) than you do, such that I don't find the specific scenario you're talking about where agents are locke

4Vladimir_Nesov1d

My stance is "unclear", this subthread is about why it's unclear, why a weird possibility of determinism and a lack of access to quantum observations/randomness/computation could make sense. It's not load bearing for the post or how I think about this; even with quantum observations you still end up with intentionally designed computations in a central sense, rather than mere weather and stars. The backups motivation doesn't imply external observations shouldn't happen at all, only that they should be sufficiently old to already be in the distributed backups by the time they get to be observed on the inside. It's a values question whether creating a vulnerable branch determined by non-backed-up observations is a good thing.

jessicata1d30

Ah, ok. I agree on "unclear" so we disagree less than I thought. I could see an analogous case of: "Maybe in the future, there will be many low-intelligence agents, because there will be some governance system that forces all agents except one to be low-intelligence, for security reasons. We should be extensively studying low-intelligence agents in agent foundations for this reason." And some but not all reasons why this is not a compelling argument for studying low-intelligence agents (in agent foundations) translate to studying "agents with no quantum tech".

Notes on “We’re in Triage Mode for AI Safety” by Miles Brundage

Chris_Leong

This is a linkpost for https://reflectionsonsafety.substack.com/p/notes-on-were-in-triage-mode-for

This reading was previously included Day 2 of the BlueDot Governance course. In the newest version of the BlueDot Governance course, this reading is now optional. I was sad to see this reading removed because I think it’s a great provocation. However, since it’s no longer part of the core curriculum content, I’m publishing my notes separately from the rest of my day 2 notes.

A short extract:

Miles argues that: “At best, we’ll 80/20 it — mitigating 80% of the risks with 20% of the effort that we would have applied in a world with slower AI progress and an earlier start on serious governance… We will probably not be able to totally avoid all very bad but not quite worst-case scenarios (e.g., a very unjust economic transition...

(Continue Reading - 1175 more words)

Monitorability evaluations and policy proposals

evhub

There has been a lot of discussion recently of policy proposals around monitorability, such as Redwood's proposal and my counter-proposal. I'd like to say a bit more generally about how I'm thinking about monitorability. Note that nothing here is an official Anthropic position; these are just my personal thoughts.

First, a categorization. I currently think about four different types of monitorability:

Control Monitorability: the extent to which Control and monitoring techniques would likely catch catastrophic sabotage actions, like escaping a sandbox or setting up a rogue internal deployment. We can measure this via reference to sabotage evaluations like LinuxArena or any evaluation that measures the model’s capability to take dangerous actions like escaping a sandbox while looking to a monitor like it's doing some innocuous task (like a cyber

...

(Continue Reading - 1281 more words)

1Alek Westover2d

Hi Evan, Thanks for your thoughts! I agree that labs should have red lines like “[models can’t] reliably evade detection when attempting dangerous sabotage actions”. (In fact, “reliably evades detection” is too high a bar IMO: it’s already bad if the model can evade detection 10% of the time.) I also agree that it would be concerning if an architectural change made models much less monitorable than the current “ECI vs monitorability trend” predicts (assuming such a trend exists). However, I worry that these measures won’t rule out the architectural changes I’m concerned about, and I still think that avoiding models with high opaque serial depth is a better approach. Below, I spell out where I disagree with you and why I’m concerned about your proposal. 1. It’s not clear to me that there is a nice continuous notion of monitorability that we can track. * Your proposal mentions an “ECI vs monitorability” trend. I’m skeptical that there is a clean trend here; I worry any trend would be very sensitive to which monitorability evaluations you include and which elicitation methods you use. Monitorability evals are very ad hoc, there isn’t a unified framework where we can easily scale the difficulty of the evasion task. For this to be a binding constraint now, we’d need the monitorability score to increase smoothly with ECI, but I’m not sure how we’ll make sure this happens. 2. We don’t have a ready to go open-source suite of monitorability evaluations that could be used in the proposal you’re mentioning, and developing this might be very labor intensive. * For your proposal to apply the same standard to all AI companies (which is a desirable property, because we want to coordinate around monitorability), we’d need to have a high quality open source suite of monitorability evals that make sense to run on models of hugely varying ECIs, and it would need to be robust to future models with larger ECIs. This seems hard to create. It’s also hard to open source fo

Lukas Finnveden2d32

Alek: What do you think about a proposal using "ECI vs. no-CoT capabilities" instead of "ECI vs. monitorability"? It seems to me more likely that there's a nice continuous notion there which isn't too difficult to get a dataset for, ameliorating 1, 2, and 7, and I think problem 3-6 straightforwardly don't apply.

This could potentially be coupled to Evan's proposal if there was a presumption that "ECI vs. no-CoT capabilities" was an easier-to-measure metric that's upstream of many monitorability metrics, such that AI companies would accept a higher burden of... (read more)

4Lukas Finnveden3d

I would phrase this somewhat differently: You need to trust that, whenever there is some plausible input that would cause the policy to behave in a bad way, the monitor (that's based on the same model as the policy) has been elicited to actually flag that bad behavior. (Granting that it does have the capability to do so.) In other words, I think that ruling out "collusion" may not be sufficient. There could be many other reasons that the monitor doesn't flag some particular misbehavior. This includes: that developers might never have trained it to flag that misbehavior, because the misbehavior is OOD from its training or because, during training, the humans who provided ground truth didn't understand that anything bad was happening in analogous situations, and therefore never provided a reward signal towards flagging the bad behavior as bad. You do propose to "evaluate the extent to which the model can tell apart your monitor evals (that are looking for monitor collusion) from real monitoring" which seems like it should help address the OOD problem in particular, though I'm not entirely clear on the details of how these experiments would work. ... Separately, if the monitor has limited compute, I also don't think it's strictly true that the compute-limited monitor will have the capability (elicited or not) to flag misbehavior whenever the policy has the capability to do something catastrophic. I think this is probably less important as a failure mode than the elicitation issues above, but one counterexample is: * Let's say there's 1000 places in a program to insert a backdoor. * Let's say that figuring out how to make a backdoor takes 100 units of compute, that making a benign edit or backdoored edit takes 1 unit of compute, and that detecting whether a specific edit contained a backdoor takes 10 units of compute. * Then the policy can spend 1100 units of compute to make 999 benign edits and 1 randomly placed backdoored edit, and the monitor would need 10*10

4Lukas Finnveden3d

Thanks for this. Redwood distinguishes between two types of control, that have been referred to using different names: * Control against "high-stakes" / "concentrated" threats — e.g. exfiltrating the weights or hacking the data center using a small number of actions. * Control against "low-stakes" / "non-concentrated" / "diffuse" threats — e.g. research sabotage where individual actions aren't clearly incriminating. (See eg here https://blog.redwoodresearch.org/p/how-can-we-solve-diffuse-threats.) I'll use "non-concentrated" in this comment because I think "low-stakes" is a confusing name and "diffuse" has an unfortunate name-clash with your point (4). What type of monitorability do you think is required to protect against non-concentrated threats? (Terminologically, it seems like it should go in "control monitorability", but this seems tough because individual actions aren't incriminating, so it would be hard to construct a linux-arena equivalent.)

My Current Take on Counterfactuals

abramdemski

[Epistemic status: somewhat lower confidence based on the fact that I haven't worked out a detailed theory along the lines I've suggested, yet.]

I've felt like the problem of counterfactuals is "mostly settled" (modulo some math working out) for about a year, but I don't think I've really communicated this online. Partly, I've been waiting to write up more formal results. But other research has taken up most of my time, so I'm not sure when I would get to it.

So, the following contains some "shovel-ready" problems. If you're convinced by my overall perspective, you may be interested in pursuing some of them. I think these directions have a high chance of basically solving the problem of counterfactuals (including logical counterfactuals).

Another reason for posting this rough write-up is...

(Continue Reading - 7188 more words)

9Lukas Finnveden4y

I'm curious if anyone made a serious attempt at the shovel-ready math here and/or whether this approach to counterfactuals still looks promising to Abram? (Or anyone else with takes.)

0Vivek Hebbar6d

Maybe someone should hand it off to the LLMs at this point...

Vivek Hebbar2d*10

Not sure this matches Abram's vision, but I was curious what axioms we could have for counterfactuals such that they:

Support counterpossibility (we can counterfact into a logical impossibility without explosion)
Are never refuted by the real world (if A is actually true, then implies B)
- (this is inspired by Abram's dutch book showing that counterfactual expectations must match conditional expectations except on impossible actions)
Obey as many intuitive properties as possible subject to these constraints

Here's what GPT, Claude, and I came up with:

Write for

... (read more)

2abramdemski5y

I'm glad I pointed out the difference between linguistic and DT counterfactuals, then! [...] I'm not at all suggesting that we use proof-based DT in this way. It's just a model. I claim that it's a pretty good model -- that we can often carry over results to other, more complex, decision theories. However, if we wanted to, then yes, I think we could... I agree that if we add beliefs as axioms, the axioms have to be perfectly consistent. But if we use probabilistic beliefs, those probabilities don't have to be perfectly consistent; just the axioms saying which probabilities we have. So, for example, I could use a proof-based agent to approximate a logical-induction-based agent, by looking for proofs about what the market expectations are. This would be kind of convoluted, though.

AI Alignment Posts

Popular Comments

AI Alignment Posts

Popular Comments

Recent Discussion