Nate Soares moderates a long conversation between Richard Ngo and Eliezer Yudkowsky on AI alignment. The two discuss topics like "consequentialism" as a necessary part of strong intelligence, the difficulty of alignment, and potential pivotal acts to address existential risk from advanced AI.
Linkpost for https://arxiv.org/html/2611.14454v1
Our PIBBSS fellow Damini Kusum wrote an excellent paper on prediction with reflective oracle Solomonoff Induction (rOSI).
Reflective oracles are best known for their game-theoretic applications, where they are often dismissed because "why would all players choose the same reflective oracle?"
It turns out that reflective oracles can also be used to construct a version of Solomonoff induction (called rOSI) with very nice sequence prediction properties - roughly speaking, it can learn to extract computable structure from (partly) uncomputable sequences. In fact, its properties are quite reminiscent of logical induction, but this paper is much shorter and more accessible. In this context, there is only one "player" to choose an oracle, which is just a fixed-point (though you can sort of consider the set of hypotheses entertained by that player as subplayers who share the oracle).
I think this paper is the clearest exposition of a very neat but previously obscure idea in agent foundations: self-reflection can overcome unrealizability.
This is a transcription of Eliezer Yudkowsky responding to Paul Christiano's Takeoff Speeds live on Sep. 14, followed by a conversation between Eliezer and Paul. This discussion took place after Eliezer's conversation with Richard Ngo.
Color key:
| Chat by Paul and Eliezer | Other chat | Inline comments |
[Yudkowsky][10:14] (Nov. 22 follow-up comment) (This was in response to an earlier request by Richard Ngo that I respond to Paul on Takeoff Speeds.) |
[Yudkowsky][16:52] maybe I'll try liveblogging some https://sideways-view.com/2018/02/24/takeoff-speeds/ here in the meanwhile |
[Yudkowsky][16:57]
![]() It seems to me to be... |