All of Nisan's Comments + Replies

On 2018-04-09, OpenAI said^[1]:

OpenAI’s mission is to ensure that artificial general intelligence (AGI) [...] benefits all of humanity.

In contrast, in 2023, OpenAI said^[2]:

[...] OpenAI’s mission: to build artificial general intelligence (AGI) that is safe and benefits all of humanity.

The subject of this post appears in the "Did you know..." section of Wikipedia's front page(archived) right now.

I'm saying "transformers" every time I am tempted to write "LLMs" because many modern LLMs also do image processing, so the term "LLM" is not quite right.

"Transformer"'s not quite right either because you can train a transformer on a narrow task. How about foundation model: "models (e.g., BERT, DALL-E, GPT-3) that are trained on broad data at scale and are adaptable to a wide range of downstream tasks".

I agree 100%. It would be interesting to explore how the term "AGI" has evolved, maybe starting with Goertzel and Pennachin 2007 who define it as:

a software program that can solve a variety of complex problems in a variety of different domains, and that controls itself autonomously, with its own thoughts, worries, feelings, strengths, weaknesses and predispositions

On the other hand, Stuart Russell testified that AGI means

machines that match or exceed human capabilities in every relevant dimension

so the experts seem to disagree. (On the other hand, ... (read more)

I wonder why Gemini used RLHF instead of Direct Preference Optimization (DPO). DPO was written up 6 months ago; it's simpler and apparently more compute-efficient than RLHF.

• Is the Gemini org structure so sclerotic that it couldn't switch to a more efficient training algorithm partway through a project?
• Is DPO inferior to RLHF in some way? Lower quality, less efficient, more sensitive to hyperparameters?
• Maybe they did use DPO, even though they claimed it was RLHF in their technical report?

Thanks! For convex sets of distributions: If you weaken the definition of fixed point to $g^C\left(S\right)\le S$, then the set $\{S\in CX:g^C(S)=S\}$ has a least element which really is a least fixed point.

Conception is a startup trying to do in vitro gametogenesis for humans!

CFAR used to have an awesome class called "Be specific!" that was mostly about concreteness. Exercises included:

• Rationalist taboo
• A group version of rationalist taboo where an instructor holds an everyday object and asks the class to describe it in concrete terms.
• The Monday-Tuesday game
• A role-playing game where the instructor plays a management consultant whose advice is impressive-sounding but contentless bullshit, and where the class has to force the consultant to be specific and concrete enough to be either wrong or trivial.
• People were encouraged t

Yep, I skimmed it by looking at the colorful plots that look like Ising models and reading the captions. Those are always fun.

No, I just took a look. The spin glass stuff looks interesting!

I think you're saying $A_{\omega +1}:=[\Delta A_{\omega }\to \Delta A_0]$, right? In that case, since $A_0$ embeds into $A_{\omega }$, we'd have $A_{\omega +1}$ embedding into $A_{\omega }$. So not really a step up.

If you want to play ordinal games, you could drop the requirement that agents are computable / Scott-continuous. Then you get the whole ordinal hierarchy. But then we aren't guaranteed equilibria in games between agents of the same order.

I suppose you could have a hybrid approach: Order $\omega +1$ is allowed to be discontinuous in its order-$\omega$ beliefs, but higher orders have to be continuous? Maybe that would get you to $\omega 2$.... (read more)

I apologize, I shouldn't have leapt to that conclusion.

it legitimately takes the whole 4 years after that to develop real AGI that ends the world. FINE. SO WHAT. EVERYONE STILL DIES.

By Gricean implicature, "everyone still dies" is relevant to the post's thesis. Which implies that the post's thesis is that humanity will not go extinct. But the post is about the rate of AI progress, not human extinction.

This seems like a bucket error, where "will takeoff be fast or slow?" and "will AI cause human extinction?" are put in the same bucket.

The central hypothesis of "takeoff speeds" is that at the time of serious AGI being developed, it is perfectly anti-Thielian in that it is devoid of secrets

No, the slow takeoff model just precludes there being one big secret that unlocks both 30%/year growth and dyson spheres. It's totally compatible with a bunch of medium-sized $1B secrets that different actors discover, adding up to hyperbolic economic growth in the years leading up to "rising out of the atmosphere".

Rounding off the slow takeoff hypothesis to "lots and lots of little innovations addin... (read more)

I feel excited about this framework! Several thoughts:

I especially like the metathreat hierarchy. It makes sense because if you completely curry it, each agent sees the foe's action, policy, metapolicy, etc., which are all generically independent pieces of information. But it gets weird when an agent sees an action that's not compatible with the foe's policy.

You hinted briefly at using hemicontinuous maps of sets instead of or in addition to probability distributions, and I think that's a big part of what makes this framework exciting. Maybe if one takes a... (read more)

See also this comment from 2013 that has the computable version of NicerBot.

Or maybe it means we train the professional in the principles and heuristics that the bot knows. The question is if we can compress the bot's knowledge into, say, a 1-year training program for professionals.

There are reasons to be optimistic: We can discard information that isn't knowledge (lossy compression). And we can teach the professional in human concepts (lossless compression).

This sounds like a great goal, if you mean "know" in a lazy sense; I'm imagining a question-answering system that will correctly explain any game, move, position, or principle as the bot understands it. I don't believe I could know all at once everything that a good bot knows about go. That's too much knowledge.

Red-penning is a general problem-solving method that's kinda similar to this research methodology.

I'd believe the claim if I thought that alignment was easy enough that AI products that pass internal product review and which don't immediately trigger lawsuits would be aligned enough to not end the world through alignment failure. But I don't think that's the case, unfortunately.

It seems like we'll have to put special effort into both single/single alignment and multi/single "alignment", because the free market might not give it to us.

I'd like more discussion of the claim that alignment research is unhelpful-at-best for existential safety because of it accelerating deployment. It seems to me that alignment research has a couple paths to positive impact which might balance the risk:

1. Tech companies will be incentivized to deploy AI with slipshod alignment, which might then take actions that no one wants and which pose existential risk. (Concretely, I'm thinking of out with a whimper and out with a bang scenarios.) But the existence of better alignment techniques might legitimize governa

In this case humans are doing the job of transferring from $D$ to $D^{\ast }$, and the training algorithm just has to generalize from a representative sample of $D^{\ast }$ to the test set.

Thanks for the references! I now know that I'm interested specifically in cooperative game theory, and I see that Shoham & Leyton-Brown has a chapter on "coalitional game theory", so I'll take a look.

A proof of the lemma $V(a,b^{\ast })\le v^{\ast }$:

$$V(a,b^{\ast })=V(a,{\mathrm{argmin}}_b\underset{a^{\prime }}{max}V(a^{\prime },b\left)\right)$$

$$\le \underset{a^{\prime }}{max}V(a^{\prime },{\mathrm{argmin}}_b\underset{a^{\prime }}{max}V(a^{\prime },b\left)\right)$$

$$=\underset{b}{min}\underset{a^{\prime }}{max}V(a^{\prime },b)$$

$$=v^{\ast }$$

Ah, ok. When you said "obedience" I imagined too little agency — an agent that wouldn't stop to ask clarifying questions. But I think we're on the same page regarding the flavor of the objective.

Might not intent alignment (doing what a human wants it to do, being helpful) be a better target than obedience (doing what a human told it to do)?

My takeaway from this is that if we're doing policy selection in an environment that contains predictors, instead of applying the counterfactual belief that the predictor is always right, we can assume that we get rewarded if the predictor is wrong, and then take maximin.

How would you handle Agent Simulates Predictor? Is that what TRL is for?

The observation can provide all sorts of information about the universe, including whether exploration occurs. The exact set of possible observations depends on the decision problem.

$E$ and $O$ can have any relationship, but the most interesting case is when one can infer $E$ from $O$ with certainty.

Thanks, I made this change to the post.

Yeah, I think the fact that Elo only models the macrostate makes this an imperfect analogy. I think a better analogy would involve a hybrid model, which assigns a probability to a chess game based on whether each move is plausible (using a policy network), and whether the higher-rated player won.

I don't think the distinction between near-exact and nonexact models is essential here. I bet we could introduce extra entropy into the short-term gas model and the rollout would still be superior for predicting the microstate than the Boltzmann distribution.

The sum isn't over $i$, though, it's over all possible tuples of length $n-1$. Any ideas for how to make that more clear?

I'm having trouble following this step of the proof of Theorem 4: "Obviously, the first conditional probability is 1". Since the COD isn't necessarily reflective, couldn't the conditional be anything?

The linchpin discovery is probably February 2016.

Ok. I think that's the way I should have written it, then.

Oh, interesting. Would your interpretation be different if the guess occurred well after the coinflip (but before we get to see the coinflip)?

What predictions can we get out of this model? If humans use counterfactual reasoning to initialize MCMC, does that imply that humans' implicit world models don't match their explicit counterfactual reasoning?

I agree exploration is a hack. I think exploration vs. other sources of non-dogmatism is orthogonal to the question of counterfactuals, so I'm happy to rely on exploration for now.

"Programmatically Interpretable Reinforcement Learning" (Verma et al.) seems related. It would be great to see modular, understandable glosses of neural networks.

This doesn't quite work. The theorem and examples only work if you maximize the unconditional mutual information, $H(X;Y)$, not $H(X;Y|A)$. And the choice of $X$ is doing a lot of work — it's not enough to make it "sufficiently rich".