i.e. if each forecaster $w\in W$ has an first-order belief $f\left(w\right)\in B\left(S\right)$, and $w\in B\left(S\right)$ is your second-order belief about which forecaster is correct, then $\left(w{⊳}_{WS}f\right)\in B\left(S\right)$ should be your first-order belief about the election.

I think there might be a typo here. Did you instead mean to write: "$w\in B\left(W\right)$" for the second order beliefs about the forecasters?

We aren’t offering these criteria as necessary for “knowledge”—we could imagine a breaker proposing a counterexample where all of these properties are satisfied but where intuitively M didn’t really know that A′ was a better answer. In that case the builder will try to make a convincing argument to that effect.

Bolded should be sufficient.

In fact, I'm pretty sure that's how humans work most of the time. We use the general-intelligence machinery to "steer" ourselves at a high level, and most of the time, we operate on autopilot.

Yeah, I agree with this. But I don't think the human system aggregates into any kind of coherent total optimiser. Humans don't have an objective function (not even approximately?).

A human is not well modelled as a wrapper mind; do you disagree?

Thus, any greedy optimization algorithm would convergently shape its agent to not only pursue $R$, but to maximize for $R$'s pursuit — at the expense of everything else.

Conditional on:

1. Such a system being reachable/accessible to our local/greedy optimisation process
2. Such a system being actually performant according to the selection metric of our optimisation process 

I'm pretty sceptical of #2. I'm sceptical that systems that perform inference via direct optimisation over their outputs are competitive in rich/complex environments. 

Such o... (read more)

Some Nuance on Learned Optimisation in the Real World

I think mesa-optimisers should not be thought of as learned optimisers, but systems that employ optimisation/search as part of their inference process.

The simplest case is that pure optimisation during inference is computationally intractable in rich environments (e.g. the real world), so systems (e.g. humans) operating in the real world, do not perform inference solely by directly optimising over outputs.

Rather optimisation is employed sometimes as one part of their inference strategy. That is systems o... (read more)

GPTs are not Imitators, nor Simulators, but Predictors.

I think an issue is that GPT is used to mean two things:

1. A predictive model whose output is a probability distribution over token space given its prompt and context
2. Any particular techniques/strategies for sampling from the predictive model to generate responses/completions for a given prompt.

[See the Appendix]

The latter kind of GPT, is what I think is rightly called a "Simulator".

From @janus' Simulators (italicised by me):

I use the generic term “simulator” to refer to models trained with pre

What do you think MIRI is currently doing wrong/what should they change about their approach/general strategy?

To be clear, I enjoyed the post and am looking forward to this sequence. A point of disagreement though:

One feasible-seeming approach is "accelerating alignment," which involves leveraging AI as it is developed to help solve the challenging problems of alignment. This is not a novel idea, as it's related to previously suggested concepts such as seed AI, nanny AI, and iterated amplification and distillation (IDA).

I disagree that using AI to accelerate alignment research is particularly load bearing for the development of a practical alignment craf... (read more)

I disagree that intelligence and rationality are more fundamental than physics; the territory itself is physics, and that is all that is really there. Everything else (including the body of our phone knowledge) are models for navigating that territory.

Turing formalised computation and established the limits of computation given certain assumptions. However, those limits only apply as long as the assumptions are true. Turing did not prove that no mechanical system is superior to a Universal Turing Machine, and weird physics may enable super Turing computati