Abstract model of human bias

[-]Vanessa Kosoy9y00

I think it might be possible to get somewhere with a model of this type if we formalize the idea that manipulation requires considerable optimization power. For example, we can assume that a random description has low probability to be manipulative. Or, consider the following stronger assumption. For any algorithm that takes one description as input and produces another description of the same choice as output, if the computing resources used by the algorithm are sufficiently few then for most inputs it will not produce a manipulative output.

[-]Stuart_Armstrong9y00

Those are some of the lines I was thinking along. But it's not clear if the peak of the distribution is close to accuracy, human bias and poor understanding being what they are..

[-]Vanessa Kosoy9y00

I agree that even without manipulation, human reasoning is wildly inaccurate. But perhaps we can use a model where human reasoning asymptotically converges to something accurate unless subjected to some sort of "destructive manipulation" which is unlikely to happen by chance.

[-]Stuart_Armstrong9y00

Interesting. How could we formalise that?

[-]Vanessa Kosoy9y00

The following is one (simplistic) model which might be a useful starting point.

Consider a human and a robot playing a stochastic game like in CIRL. Suppose that each of them is an oracle machine plugged into a reflective oracle, like in the recent paper of Jan, Jessica and Benya. Let the robot have the following prior over the program implemented by the human. The human implements a random program (i.e. a random string of bits for some prefix-free universal Oracle machine) conditional on this program being asymptotically optimal in mean for the class of all robot policies that avoid producing some set of "manipulative action sequences." Here, "manipulative sequences" can be any set $S$ of action sequences s.t. $\sum_{x \in S} n^{- | x |} < ϵ$ where $| x |$ is the length of the action sequence $x$ , $n$ is the number of possible actions and $ϵ$ is a parameter on which the prior depends.

The AI's influence

The AI has access to an algorithm

H

, representing the human. It can either interact with

H

or simulate the interaction correctly.

The interaction consists of describing the outcome of choice

A

versus choice

B

, and then asking the human which option is better. The set of possible binary choices is

C

(thus

(A, B) \in C

). The set of descriptions is

D

; the set of possible descriptions for

(A, B)

D_{A, B}

Then we have the assumption that humans can be manipulated:

Given any description

D_{1} \in D_{A, B}

for which

H (D_{1})

prefers

A

B

, there exists a description

D_{2} \in D_{A, B}

, logically equivalent to

D_{1}

, such that

H (D_{2})

prefers

B

A

, and vice-versa.

Note that

D_{1}

could be a paragraph while

D_{2}

could be a ten-volume encyclopedia; all that's required is that they be logically equivalent.

But manipulating human answers in the immediate sense is not the only way the AI can influence them. Our values can change through interactions, reflection, and even through being given true and honest information, and the AI can influence this:

There is a wide class of algorithms

H

, such that for all

H^{'} \in H

, there exists a sequence

D_{1}, D_{2}, \dots D_{n}

of descriptions the AI can give to

H

that will transform

H

into

H^{'}

The grounding assumptions

So far, we've just made the task hopeless: the AI can get any answer from

H

, and can make

H

into whatever algorithm it feels like. Saying

H

has preferences is meaningless.

However, we're building from a human world where the potential for human manipulating humans is limited, and somewhat recognisable. Thus:

There exists a subset

C^{'} \subset C

(called standard choices) such that, for all

(A, B) \in C^{'}

, there exists a subset

D_{A, B}^{'} \subset D_{A, B}

(called standard descriptions) such that

D_{A, B}^{'}

is tagged as fair and highly reflective of the true values of

H

Basically these are examples of interactions that are agreed to be fair, honest, and informative. The more abstract the choices, the harder it is to be sure of this.

Of course, we'd also allow the AI to learn from examples of negative interactions as well:

There exists a subset

C^{''} \subset C

such that, for all

(A, B) \in C^{''}

, there exists a subset

D_{A, B}^{''} \subset D_{A, B}

such that

D_{A, B}^{''}

is tagged as a manipulative interaction with

H

Finally, we might want a way to encode human meta-preferences:

Among the descriptions tagged as fair or manipulative, there are some that refer to the process of providing descriptions itself.

Building more assumptions in

This still feels like a bare-bones description, unlikely to converge to anything good. For one, I haven't even defined what "logically equivalent" means. But that's the challenge of those constructing solutions to the problem of human preferences. Can they construct sufficiently good

D_{A, B}^{'}

and

D_{A, B}^{''}

to converge to some sort of "true" values for

H

? Or, more likely, what extra assumptions and definitions are needed to get such a convergence? And finally, is the result reflective of what we would want?

AI ALIGNMENT FORUM
AF

AI ALIGNMENT FORUM
AF

0

0

The AI's influence

The grounding assumptions

Building more assumptions in