AI ALIGNMENT FORUM
AF

All of rotatingpaguro's Comments + Replies

When is Goodhart catastrophic?
rotatingpaguro
20

I wasn't saying you made all those assumption, I was trying to imagine an empirical scenario to get your assumptions, and the first thing to come to my mind produced even stricter ones.

I do realize now that I messed up my comment when I wrote

in practice reduces just to the part "have a story for why inductive bias and/or non-independence work in your favor", because I currently think Normality + additivity + independence are bad assumptions, and I see that as almost a null advice.

Here there should not be Normality, just additivity and independence, in the ... (read more)

Reply
2Drake Thomas
An example of the sort of strengthening I wouldn't be surprised to see is something like "If V is not too badly behaved in the following ways, and for all v∈R we have [some light-tailedness condition] on the conditional distribution (X|V=v), then catastrophic Goodhart doesn't happen." This seems relaxed enough that you could actually encounter it in practice.
When is Goodhart catastrophic?
rotatingpaguro
10

This model makes two really strong assumptions: that optimization is like conditioning, and that  and  are independent.

[...]

There's also a sort of implicit assumption in even using a framing that thinks about things as ; the world might be better thought of as naturally containing  tuples (with  our proxy measurement), and  could be a sort of unnatural construction that doesn't make sense to single out in the real world. (We do think this framing is relatively natural, but won't get int

... (read more)
Reply
2Drake Thomas
I'm not sure what you mean formally by these assumptions, but I don't think we're making all of them. Certainly we aren't assuming things are normally distributed - the post is in large part about how things change when we stop assuming normality! I also don't think we're making any assumptions with respect to additivity; X=U−V is more of a notational or definitional choice, though as we've noted in the post it's a framing that one could think doesn't carve reality at the joints. (Perhaps you meant something different by additivity, though - feel free to clarify if I've misunderstood.) Independence is absolutely a strong assumption here, and I'm interested in further explorations of how things play out in different non-independent regimes - in particular we'd be excited about theorems that could classify these dynamics under a moderately large space of non-independent distributions. But I do expect that there are pretty similar-looking results where the independence assumption is substantially relaxed. If that's false, that would be interesting! Late edit: Just a note that Thomas has now published a new post in the sequence addressing things from a non-independence POV.
Connectomics seems great from an AI x-risk perspective
rotatingpaguro
41

I have a vague impression that I am not crazy to hope for whole primate-brain connectomes in the 2020s and whole human-brain connectomes in the 2030s, if all goes well.

After reading the post "Whole Brain Emulation: No Progress on C. elegans After 10 Years" I was left with the general impression that this stuff is very difficult; but I don't know the details, and that post talks about simulation given a connectome, not getting a connectome, which maybe then is easier even for a huge primate brain, I guess? And I don't know what probability you mean with "no... (read more)

Reply
4Steve Byrnes
Me neither. I’m not close enough to the technical details to know. I did run that particular sentence by a guy who’s much more involved in the field before I published, and he said it was a good sentence, but only because “not crazy to hope for X” is a pretty weak claim. Yeah, the C. elegans connectome has been known for a very long time. The thing that’s hard for C. elegans is going from the connectome to WBE. As crazy as it sounds, I think that there are ways in which a human WBE is easier than a C. elegans WBE. I talk about that to some extent here.