All of Rafael Harth's Comments + Replies

I really don't think this is a reasonable measure for ability to do long term tasks, but I don't have the time or energy to fight this battle, so I'll just register my prediction that this paper is not going to age well.

Instead of "have LLMs generated novel insights", how about "have LLMs demonstrated the ability to identify which views about a non-formal topic make more or less sense?" This question seems easier to operationalize and I suspect points at a highly related ability.

As someone who expects LLMs to be a dead end, I nonetheless think this post makes a valid point and does so using reasonable and easy to understand arguments. I voted +1.

I don't find this framing compelling. Particularly wrt to this part:

Obedience — AI that obeys the intention of a human user can be asked to help build unsafe AGI, such as by serving as a coding assistant. (Note: this used to be considered extremely sci-fi, and now it's standard practice.)

I grant the point that an AI that does what the user wants can still be dangerous (in fact it could outright destroy the world). But I'd describe that situation as "we successfully aligned AI and things went wrong anyway" rather than "we failed to align AI". I grant t... (read more)

The post defending the claim is Reward is not the optimization target. Iirc, TurnTrout has described it as one of his most important posts on LW.

I know he's talking about alignment, and I'm criticizing that extremely strong claim. This is the main thing I wanted to criticize in my comment! I think the reasoning he presents is not much supported by his publicly available arguments.

Ok, I don't disagree with this. I certainly didn't develop a gears-level understanding of why [building a brain-like thing with gradient descent on giant matrices] is doomed after reading the 2021 conversations. But that doesn't seem very informative either way; I didn't spend that much time trying to grok his arguments.

I also don't really get your position. You say that,

[Eliezer] confidently dismisses ANNs

but you haven't shown this!

• In Surface Analogies and Deep Causes, I read him as saying that neural networks don't automatically yield intelligence just because they share surface similarities with the brain. This is clearly true; at the very least, using token-prediction (which is a task for which (a) lots of training data exist and (b) lots of competence in many different domains is helpful) is a second requirement. If you take the network of GPT-4 and trained it

This document doesn't look to me like something a lot of people would try to write. Maybe it was one of the most important things to write, but not obviously so. Among the steps (1) get the idea to write out all reasons for pessimism, (2) resolve to try, (3) not give up halfway through, and (4) be capable, I would not guess that 4 is the strongest filter.

Yes, but I didn't mean to ask whether it's relevant, I meant to ask whether it's accurate. Does the output of language models, in fact, feel like this? Seemed like something relevant to ask you since you've seen lots of text completions.

And if it does, what is the reason for not having long timelines? If neural networks only solved the easy part of the problem, that implies that they're a much smaller step toward AGI than many argued recently.

Do you think this describes language models?

I think what you get is a person talking with no inhibitions whatsoever. Language models don’t match that.

What do you picture a language model with no inhibitions to look like? Because if I try to imagine it, then "something that outputs reasonable sounding text until sooner or later it fails hard" seems to be a decent fit. Of course haven't thought much about the generator/assessor distinction.

I mean, surely "inhibitions" of the language model don't map onto human inhibitions, right? Like, a language model without the assessor module (or a much worse a... (read more)

(Extremely speculative comment, please tell me if this is nonsense.)

If it makes sense to differentiate the "Thought Generator" and "Thought Assessor" as two separate modules, is it possible to draw a parallel to language models, which seem to have strong ability to generate sentences, but lack the ability to assess if they are good?

My first reaction to this is "obviously not since the architecture is completely different, so why would they map onto each other?", but a possible answer could be "well if the brain has them as separate modules, it could mean t... (read more)

Thanks! I agree it's an error, of course. I've changed the section, do you think it's accurate now?)

No; it was just that something about how the post explained it made me think that it wasn't #1.

I don't completely get this.

Let's call the short term predictor (in the long term predictor circuit) $P$, so if $P$ tries to predict [what $P$ predicts in 0.3s], then the correct prediction would be to immediately predict the output at whatever point in the future the process terminates (the next ground truth injection?). In particular, $P$ would always predict the same until the ground truth comes in. But if I understand correctly, this is not what's going on.

So second try: is $P$ really still only trying to predict 0.3s into the future, making it less of a "long t... (read more)

Author here. One thing I think I've done wrong in the post is to equate black-box-search-in-large-parametrized-space with all of machine learning. I've now added this paragraph at the end of chapter 1:

Admittedly, the inner alignment model is not maximally general. In this post, we've looked at black box search, where we have a parametrized model and do SGD to update the parameters. This describes most of what Machine Learning is up to in 2020, but it does not describe what the field did pre-2000 and, in the event of a paradigm shift similar to the deep l

I strongly believe that (1) well-being is objective, (2) well-being is quantifiable, and (3) Open Individualism is true (i.e., the concept of identity isn't well-defined, and you're subjectively no less continuous with the future self if any other person than your own future self).

If (1-3) are all true, then utilitronium is the optimal outcome for everyone even if they're entirely selfish. Furthermore, I expect an AGI to figure this out, and to the extent that it's aligned, it should communicate that if it's asked. (I don't think an AGI will therefore deci... (read more)

I don't have any reason why this couldn't happen. My position is something like "morality is real, probably precisely quantifiable; seems plausible that in the scenario of humans with autonomy and aligned AI, this could lead to an asymmetry where more people tend toward utilitronium over time". (Hence why I replied, you didn't seem to consider that possibility.) I could make up some mechanisms for this, but probably you don't need me for that. Also seems plausible that this doesn't happen. If it doesn't happen, maybe the people who get to decide what happe... (read more)

This comment seems to be consistent with the assumption that the outcome 1 year after the singularity is locked in forever. But the future we're discussing here is one where humans retain autonomy (?), and in that case, they're allowed to change their mind over time, especially if humanity has access to a superintelligent aligned AI. I think a future where we begin with highly suboptimal personal utopias and gradually transition into utilitronium is among the more plausible outcomes. Compared with other outcomes where Not Everyone Dies, anyway. Your credence may differ if you're a moral relativist.

1: To me, it made it more entertaining and thus easier to read. (No idea about non-anecdotal data, would also be interested.)

3: Also no data; I strongly suspect the metric is generally good because... actually I think it's just because the people I find worth listening to are overwhelmingly not condescending. This post seems highly usual in several ways.

Is Humbali right that generic uncertainty about maybe being wrong, without other extra premises, should increase the entropy of one's probability distribution over AGI, thereby moving out its median further away in time?

My answer to this is that

The total absence of obvious output of this kind from the rest of the "AI safety" field even in 2020 causes me to regard them as having less actual ability to think in even a shallowly adversarial security mindset, than I associate with savvier science fiction authors. Go read fantasy novels about demons and telepathy, if you want a better appreciation of the convergent incentives of agents facing mindreaders than the "AI safety" field outside myself is currently giving you.

While this this may be a fair criticism, I feel like someone ought to point out... (read more)

Yeah, it's fixed now. Thanks for pointing it out.

Survey on model updates from reading this post. Figuring out to what extent this post has led people to update may inform whether future discussions are valuable.

Results: (just posting them here, doesn't really need its own post)

One of my updates from reading this is that Rapid vs. Gradual takeoff seems like an even more important variable for many people's model than I had assumed. Making this debate less one-sided might thus be super valuable even if writing up arguments is costly.

Thanks. It's because directupload often has server issues. I was supposed to rehost all images from my posts to a more reliable host, but apparently forgot this one. I'll fix it in a couple of hours.

I think the Go example really gets to the heart of why I think Debate doesn't cut it.

Your comment is an argument against using Debate to settle moral questions. However, what if Debate is trained on Physics and/or math questions, with the eventual goal of asking "what is a provably secure alignment proposal?"

In the ball-attached-to-a-pole example, the honest debater has assigned probabilities that are indistinguishable from what you would do if you knew noting except that the claim is false. (I.e., assign probabilities that doubt each component equally.) I'm curious how difficult it is to find the flaw in this argument structure. Have you done anything like showing these transcripts to other experts and seeing if they will be able to answer it?

If I had to summarize this finding in one sentence, it would be "it seems like an expert can generally find a set of ... (read more)

Fantastic sequence! Certainly, for anyone other than you, the deconfusion/time investment ratio of reading this is excellent. You really succeeded in making the core insights accessible. I'd even say it compares favorably to the recommended sequences in the Alignment Forum in that regard.

I've never read the "Towards a new Impact Measure" post, but I assume doing so is redundant now since this sequence is the 'updated' version.

(This sequence inspired me to re-read Reinforcement Learning: An Introduction, hence the break.)

I realize that impact measures always lead to a tradeoff between safety and performance competitiveness. But setting $R_{\text{AUX}}:=R$ seems to sacrifice quite a lot of performance. Is this real or am I missing something?

Namely, whenever there's an action $a$ which doesn't change the state and leads to 1 reward, and a sequence $a_1,...,a_n$ of actions such that $a_n$ has $m$ reward with $m>n$ (and all $a_{<n}$ have 0 reward), then it's conceivable that $R_{\text{AUP-1}}$ would c... (read more)

Many thanks for taking the time to find errors.

I've fixed #1-#3. Arguments about the universal prior are definitely not something I want to get into with this post, so for #2 I've just made a vague statement that misalignment can arise for other reasons and linked to Paul's post.

I'm hesitant to change #4 before I fully understand why.

I'm not exactly sure what you're trying to say here. The way I would describe this is that internalization requires an expensive duplication where the objective is represented separately from the world model despite the world

An early punchline in this sequence was "Impact is a thing that depends on the goals of agents; it's not about objective changes in the world." At that point, I thought "well, in that case, impact measures require agents to learn those goals, which means it requires value learning." Looking back at the sequence now, I realize that the "How agents impact each other" part of the sequence was primarily about explaining why we don't need to do that and the previous post was declaring victory on that front, but it took me seeing the formalism here to really get... (read more)

I was initially writing a comment about how AUP${}_{\text{conceptual}}$ doesn't seem to work in every case because there are actions that are catastrophic without raising its power (such as killing someone), but then I checked the post again and realized that it disincentivizes changes of power in both directions. This rules out the failure modes I had in mind. (It wouldn't press a button that blows up the earth...)

It does seem that AUP${}_{\text{conceptual}}$ will make it so an agent doesn't want to be shut off, though. If it's shut off, its power goes way down (to zero if... (read more)

The technical appendix felt like it was more difficult than previous posts, but I had the advantage of having tried to read the paper from the preceding post yesterday and managed to reconstruct the graph & gamma correctly.

The early part is slightly confusing, though. I thought AU is a thing that belongs to the goal of an agent, but the picture made it look as if it's part of the object ("how fertile is the soil?"). Is the idea here that the soil-AU is slang for "AU of goal 'plant stuff here'"?

I did interpret the firs... (read more)

Thoughts after reading and thinking about this post

The thing that's bugging me here is that Power and Instrumental convergence seem to be almost the same.

In particular, it seems like Power asks [a state]: "how good are you across all policies" and Instrumental Convergence asks: "for how many policies are you the best?". In an analogy to tournaments where policies are players, power cares about the average performance of a player across all tournaments, and instrumental convergence about how many first places that player got. In tha... (read more)

Thoughts I have at this point in the sequence

• This style is extremely nice and pleasant and fun to read. I saw that the first post was like that months ago; I didn't expect the entire sequence to be like that. I recall what you said about being unable to type without feeling pain. Did this not extend to handwriting?
• The message so far seems clearly true in the sense that measuring impact by something that isn't ethical stuff is a bad idea, and making that case is probably really good.
• I do have the suspicion that quantifying impact properly is impossible with

In addition, current RL is episodic, so we should only expect that RL agents are goal-directed over the current episode and not in the long-term.

Is this true? Since ML generally doesn't choose an algorithm directly but runs a search over a parameter space, it seems speculative to assume that the resulting model, if it is a mesa-optimizer and goal-directed, only cares about its episode. If it learned that optimizing for X is good for reward, it seems at least conceivable that it won't understand that it shouldn't care about instances of X that appear in future episodes.

I might be confused here, but it seems to me that it's easy to interpret the arguments in this post as evidence in the wrong direction.

I see the following three questions as relevant:

1. How much sets human brains apart from other brains?

2. How much does the thing that humans have and animals don't matter?

3. How much does better architecture matter for AI?

Questions #2 and #3 seem positively correlated – if the thing that humans have is important, it's evidence that architectural changes matter a lot. However, holding #2 constant, #1 an... (read more)

Ex 5 (fixed version)

Let $D$ denote the triangle. For each $n\in N$, construct a 2-d simplex $S_n$ with $(n+1{)}^2$ nodes in $D$, where the color of a point corresponds to the place in the disk that $f$ carries that point to, then choose $x_n$ to be a point within a trichromatic triangle in the graph. Then $(x_n{)}_{n\in N}$ is a bounded sequence having a limit point $x^{\ast }$. Let $t$ be the center of the disc; suppose that $f\left(x^{\ast }\right)\ne t$. Then there is at least one region of the disc that $f\left(x^{\ast }\right)$ doesn't touch. Let $ϵ$ be the distance to the furthest side, that is, let $ϵ=\text{max}\left\{d\right(f\left(x^{\ast }\right),G),d(f\left(x^{\ast }\right),R),d(f$... (read more)

I'm late, but I'm quite proud of this proof for #4:

Call the large triangle a graph and the $n^2$ triangles simply triangles. First, note that for any size, there is a graph where the top node is colored red, the remaining nodes on the right diagonal are colored green, and all nodes not on the right diagonal are colored blue. This graph meets the conditions, and has exactly one trichromatic triangle, namely the one at the top (no other triangle contains a red node). It is trivial to see that this graph can be changed into an arbitrary graph by re-col... (read more)

Ex 1

Let $V=\{v_0,...,v_n\}$ and $e_k:=(v_{k-1},v_k)$. Given an edge $e$, let ${\varphi }_e$ denote the map that maps the color of the left to that of the right node. Given a $k\in \{1,...,n\}$, let ${\varphi }_k={\varphi }_{e_k}\circ ...\circ {\varphi }_{e_1}$. Let $b$ denote the color blue and $g$ the color green. Let $c\left(k\right)$ be 1 if edge $e_k$ is bichromatic, and 0 otherwise. Then we need to show that ${\varphi }_n\left(b\right)=g⟹\left({\sum }_{k=1}^{n}c\right(k\left)\right)\text{is odd}$. We'll show $\left({\sum }_{k=1}^{n}c\right(k\left)\right)\text{is even}⟺{\varphi }_n\left(b\right)=b$, which is a striclty stronger statement than the contrapositive.

For $n=1$, the LHS is equivalent to $c\left({\varphi }_1\right)=1$, and indeed $\varphi \left(e_1\right)\left(b\right)$ equals $g$ if $e_1$ is bichromatic, and $b$ o... (read more)

Ex 4

Given a computable function $\varphi :S\mapsto \text{Comp}(S,\{T,F\left\}\right)$, define a function $f:S\mapsto \{T,F\}$ by the rule $f\left(s\right)=\{\begin{array}{cc}T& \varphi \left(s\right)\left(s\right)=F\\ F& \varphi \left(s\right)\left(s\right)=T\end{array}$. Then $f$ is computable, however, $f\notin \varphi \left(S\right)$ because for $s\in S$, we have that $\varphi \left(s\right)\left(s\right)=F⟹f\left(s\right)=T⟹\varphi \left(s\right)\ne f$ and $\varphi \left(s\right)\left(s\right)=T⟹f\left(s\right)=F⟹\varphi \left(s\right)\ne f$.

Ex 5:

We show the contrapositive: given a function halt, we construct a surjective function $f$ from $S$ to $\text{Comp}(S,\{T,F\left\}\right)$ as follows: enumerate all turing machines, such that each corresponds to a string. Given a $t\in S$, if $t$ does not decode to a turing machine, set $f\left(t\right)\equiv F$. If it does, let $\left[t\right]$ denote that turning machine. Let $f$... (read more)

Ex 1

Exercise 1: Let $f:S\mapsto P\left(S\right)$ and let $A=\{x\in S|x\notin f(x\left)\right\}$. Suppose that $A\in f\left(S\right)$, then let $x\in S$ be an element such that $f\left(x\right)=A$. Then $x\notin f\left(x\right)=A⟹x\in A$ by definition, and $x\in A=f\left(x\right)⟹x\notin A$. So $x\notin A⟺x\in A$, a contradiction. Hence $A\notin f\left(S\right)$, so that $f$ is not surjective.

Ex 2

Exercise 2: Since $T$ is nonempty, it contains at least one element $t_0$. Let $h:T\mapsto T$ be a function without a fixed point, then $t_0\ne h\left(t_0\right)=:t_1$, so that $t_0$ and $t_1$ are two different elements in $T$ (this is the only thing we shall use the function $h$ for).

Let $\Psi :S\mapsto T^S$ for $S$ nonempty. Suppose by contradiction that $\Psi$ is surject... (read more)