All of winstonne's Comments + Replies

Hi! Long time lurker, first time commenter. You have written a great piece here. This is a topic that has fascinated me for a while and I appreciate what you've laid out.

I'm wondering if there's a base assumption on the whole intelligence vs values/beliefs/goals question that needs to be questioned.

sufficiently complex goals may only be well-represented by sufficiently intelligent agents

This statement points to my question. There's necessarily a positive correlation between internal complexity and intelligence right?. So, in order for intelligence to incre... (read more)

Hmm, Looks like I should add an examples section and more background on what I mean related to freedom. What you are describing sounds like a traffic system that values ergodic efficiency of it's managed network and you are showing a way that a participant can have very non-ergodic results. It sounds like that is more of an engineering problem than what I'm imagining.

Examples off the top of my head of what I mean with respect to loss of freedom resulting from a powerful agent's value system include things like:

  • paperclip maximizer terraforming the earth pre
... (read more)