Double descent is a puzzling phenomenon in machine learning where increasing model size/training time/data can initially hurt performance, but then improve it. Evan Hubinger explains the concept, reviews prior work, and discusses implications for AI alignment and understanding inductive biases.
As discussed in Intro to Brain-Like-AGI Safety, I’m working on the technical alignment problem for a hypothetical future “brain-like AGI”, with a particular focus on treating human innate social and moral drives as a possible jumping-off point for our technical alignment approach.
After all, if it’s possible for humans to do stuff that ultimately leads to a good future, then it’s probably also possible for sufficiently human-like AGIs to do stuff that ultimately leads to a good future. Or if it’s not possible for humans to do stuff that ultimately leads to a good future, then we’re screwed no matter what. But assuming it’s possible, the “sufficiently human-like AGIs” would certainly need to have good prosocial motivations. What code do we write that would...
(Part 2 of the CAST sequence)
As a reminder, here’s how I’ve been defining “corrigible” when introducing the concept: an agent is corrigible when it robustly acts opposite of the trope of "be careful what you wish for" by cautiously reflecting on itself as a flawed tool and focusing on empowering the principal to fix its flaws and mistakes.
This definition is vague, imprecise, and hides a lot of nuance. What do we mean by “flaws,” for example? Even the parts that may seem most solid, such as the notion of there being a principal and an agent, may seem philosophically confused to a sufficiently advanced mind. We’ll get into trying to precisely formalize corrigibility later on, but part of the point of corrigibility is to work even when it’s only...
Here's an emergent downside that I failed to include:
Prince wants to test Cora's hacking ability, so he sets her up in an automated test environment where she is told to break into a target machine and "capture the flag" therein. Cora explores the digital environment and finds that the virtual machine in question appears to be unresponsive, and speculates that Prince made an error when configuring the environment. Cora also notices that there is a pathway by which she might hack the VM that is responsible for ...
I'm not convinced by the argument that a lot of these black-box evals would be ecologically valid (without significant effort to make them so). I think models could plausibly see through you saying it will be monitored closely/reviewed by humans, and potentially end up not acting as if that is true; or, including that statement may trigger it to believe its alignment may be being tested.
For the former, see the example here: https://www.lesswrong.com/posts/LhXW8ziwnn7Dd8edm/a-toy-environment-for-exploring-reasoning-about-reward#The_model_is_increasingly_ins...
I was prompted to think about why standard human checks-and-balances against violent takeover would fail for AIs given that they work for humans. Here's some notes on that.
The analogy to human society isn’t that reassuring. Wars, coups, and other violations of property rights are common in history. I think recent history features more than one civil war and coup per country per century. These are driven by technological changes, economic changes, political negotiations, that are ultimately strongly related to cognitive work. During takeoff, I expect more t...
Many thanks to Peter Barnett, my alpha interlocutor for the first version of the proof presented, and draft reader.
Analogies are often drawn between natural selection and gradient descent (and other training procedures for parametric algorithms). It is important to understand to what extent these are useful and applicable analogies.
Here, under some modest (but ultimately approximating) simplifying assumptions, natural selection is found to be mathematically equivalent to an implementation of stochastic gradient descent.
The simplifying assumptions are presented first, followed by a proof of equivalence. Finally, a first attempt is made to consider the effect of relaxing each of these assumptions and what departure it causes from the equivalence, and an alternative set of assumptions which retrieves a similar equivalence is presented.
It is essential to understand...
Cool post, thank you! I'm very surprised that this natural selection model of SGD has only 1 mutant offspring. I would have sent many mutations in all directions and then made the selection distribution really sharp based on fitness. Which is effectively zeroth-order optimization, which approximates gradient descent on a smoothed function.
---
> where mutation (xt+1−xt)∼Pm.
I think this is a typo and ought to be (x'_t - x_t) ~ Pm
So the distribution over xt+1 given xt is
P(xt+1|xt)=αsPm(xt+1−xt)Ps(f(xt+1)−f(x))
This is another typo, and the expression on the ...
I propose the Long Self-Correction[1] as an alternative name/idea/concept to AI Pause and Long Reflection.
Problem with AI Pause: Pause until when, and for what purpose? Presumably to make AI (that we'll build later) safer, but the deeper problem is that humans aren't safe, and can't safely serve as builders, overseers, or alignment targets for powerful AIs.
Problem with Long Reflection: It seems to imply that the main problem with humans is that we just haven't had enough time to think, that reflection is the main thing we need to do more of, and then we can get on with building powerful AIs or other technologies. Or that if we build aligned AIs that sincerely help us think a lot more, or do the thinking for us, then things will...
I read your post, and I had thoughts about it. I made a vocal about it and asked fable to improve it.
I agree "Self-Correction" is a better name than "Long Reflection", though the post doesn't say why. Here is my reason: "Reflection" suggests the fix is more thinking. "Correction" admits the fix is changing what we are. That's the right framing.
But I disagree with most of the list. I think it mixes three different kinds of "flaws", and they call for very different responses.
1. The metaethics flaws are based on a framing I reject.
"Not having a workable moral...
I think we do want the AGI to have long-term goals, just not to have exclusively long-term goals, at least in the §6.2.1 approach. See my old 2021 post Consequentialism & corrigibility. Ag... (read more)