All of Mateusz Bagiński's Comments + Replies

If you think the primary bottleneck to dangerous ASI is not that, but rather something else, then what do you think it is?

So far in this thread I was mostly talking from the perspective of my model(/steelman?) of Abram's argument.

I think the primary bottleneck to dangerous ASI is the ability to develop coherent and correct understandings of arbitrary complex domains and systems

I mostly agree with this.

Still, this doesn't^[1] rule out the possibility of getting an AI that understands (is superintelligent in?) one complex domain (specifically here, whate... (read more)

Right, so one possibility is that you are doing something that is “speeding up the development of AIS-helpful capabilities” by 1 day, but you are also simultaneously speeding up “dangerous capabilities” by 1 day, because they are the same thing.

TBC, I was thinking about something like: "speed up the development of AIS-helpful capabilities by 3 days, at the cost of speeding up the development of dangerous capabilities by 1 day".

I think Abram is saying the following:

• Currently, AIs are lacking capabilities that would meaningfully speed up AI Safety research.
• At some point, they are gonna get those capabilities.
• However, by default, they are gonna get those AI Safety-helpful capabilities roughly at the same time as other, dangerous capabilities (or at least, not meaningfully earlier).
  • In which case, we're not going to have much time to use the AI Safety-helpful capabilities to speed up AI Safety research sufficiently for us to be ready for those dangerous capabilities.
• Therefore, it mak

recently^[1].

empty footnote

I wonder how the following behavioral patterns fit into Shard Theory

• Many mammalian species have strong default aversion to young of their own species. They (including females) deliberately avoid contact with the young and can even be infanticidal. Physiological events associated with pregnancy (mostly hormones) rewires the mother's brain such that when she gives birth, she immediately takes care of the young, grooms them etc., something she has never done before. Do you think this can be explained by the rewiring of her reward circuit such that she finds s

Standardized communication protocols

Language is the most obvious example, but there's plenty of others. E.g. taking different parts of the body as subsystems communicating with each other, one neurotransmitter/hormone often has very similar effects in many parts of the body.

In software, different processes can communicate with each other by passing messages having some well-defined format. When you're sending an API request, you usually have a good idea of what shape the response is going to take and if the request fails, it should fail in a predictable wa... (read more)

I don't have a specific example right now but some things that come to mind:

• Both utility functions ultimately depend in some way on a subset of background conditions, i.e. the world state
• The world state influences the utility functions through latent variables in the agents' world models, to which they are inputs.
• $U_A$ changes only when $M_A$ (A's world model) changes which is ultimately caused by new observations, i.e. changes in the world state (let's assume that both A and B perceive the world quite accurately).
• If whenever $U_A$ changes $U_B$ doesn't decrease,

Can't you restate the second one as the relationship between two utility functions $U_A$ and $U_B$ such that increasing one (holding background conditions constant) is guaranteed not to decrease the other? I.e. their respective derivatives are always non-negative for every background condition.

$$\frac{\partial U_A}{\partial U_B}\ge 0\wedge \frac{\partial U_B}{\partial U_A}\ge 0$$

Re second try: what would make a high-level operationalisation of that sort helpful? (operationalize the helpfulness of an operationalisation)