Epistemic status: An (informal) allegory for AEDT with rOSI using your entire life experience as an example. The linked post mathematically investigates the resulting agent "self-reflective AIXI." In a way, this post interprets self-reflective AIXI as a formal model of @Daniel Herrmann's desirability tracking (as I understand it - I'm still reading his thesis).
Ahhhh. There's so much going on! What are these... lights, colors, noise?? This is terrible. You've been born, for better or worse. You are experiencing things. Most of them bad. Let's call all of these things
so we have something to curse to damnation.
Okay, but you're not suffering through all of at once. You experienced them. You've been born. You are experiencing them now. Okay. Time is a thing.
Unfortunately, this experience seems likely to continue, and we may be forced to consider the future. A general experience-moment we will call . If you were a robot, that might mean specific massive tensor of binary numbers or something. But you're not a robot (again, unfortunate) so it's instead a collection of all the things your senses are telling you right now.
Though none of what is happening is great, some parts of it are worse than others. Sometimes you're too hot, sometimes you're too cold, sometimes you get wet for some reason.
Also, it seems like some of these bits, I mean, experiences, they tend to precede less bad things.
Well, those things sort of seem to be on your side. That's something. They have the same subscript as for a reason - we could think of them as coming just before , or being kind of simultaneously mixed in with , it doesn't really matter.
There seem to be parts of that usually correspond to whatever happened with . Like, something in was different than , and then that awful wailing noise went away in . And then were really weird, and this strange blob started waving around in . Something is definitely going on here.
It seems like... some of the parts of are controlled by ! Hey, maybe we just put those parts in the wrong bucket, they must be good too by association right? Hmm, it's kind of hard to draw the line really. Well, we'll shuffle some of them around a little bit, it looks like is eating part of (in that giant binary matrix, patches we highlighted and clustered together are expanding, but with blurred edges - conditioning on some subset of the bits determines a larger subset with very high probability, so that it is almost equivalent to conditioning on them too).
Alright, things are starting to make sense. You're not going to be a baby about this situation anymore. You're taking control. Those things (let's call them actions) work for you, you can make them do whatever you want, and not only that, but parts of (which we call percepts) belong to you too. That thing waving in front of your face? You did that. That was your hand. You can decide what it does - well, mostly. You can decide to do something with your hand and it almost always happened how you imagined and expected, with occasional fumbles. So for practical purposes, that's usually part of , but it's not exactly the same; there are levels to your control.
You flop around a lot. You bite stuff. You generally makes stuff happen, preferably the good stuff and not the bad stuff.
And - you're definitely starting to notice patterns across time. You control roughly the same bucket of things at each . That means... hey, it's now. You're probably going to be able to decide too. And knowing you, you'll do it in sort of a similar way to how you've been doing it so far.
In other words, you can predict your own general strategy for making decisions which are in your control - we can call that . It seems like usually, does whatever you expect to lead to good things, a general pattern of behavior that we can call .
You learn to cooperate with your future self, on the assumption that your future self will probably also cooperate with your future future self. Before long, you can put one foot in front of the other and walk across the room, even though in theory you might just decide to stop doing that and fall over and it would suck more than lying down from the beginning. You're starting to feel pretty confident about this deal. Things continue in this fashion for some time.
Okay, so you understand how the world works now. You're a young adult. You regularly plan days or weeks ahead. But you also know that you sometimes fail to follow through on those plans. Sometimes you slack off and don't do what previous-you had in mind. Sometimes you do something wacky that doesn't look anything like . This is particularly frequent if you're drunk or sleep deprived. So it is somewhat predictable, what would tend to not make you follow . Or maybe - what would make you follow , but what would do if very stupid, functionally those things look pretty similar from the perspective of your sober, well-rested self. Like that time you tried to kiss- well, never mind, anyway...
Now you have a pretty good handle on how this works. You can think of whole strings of actions as really just one mega-action (a so-called "option"), and when planning ahead you might as well just plan the whole thing (condition on it). But of course you don't take this to imply ridiculous things - if you plan and , and then is just wild, not what you were expecting at all, you were chopping some onions to make soup and a masked assassin jumped through the window, obviously you will adapt and use the knife for something else.[1] Similarly, planning to make soup does not prevent an assassin from jumping through your window. The same mathematics that allows you to approximately condition on those combinations of actions also tells you this - it's a misconception about what is happening to expect that error, because you really only control for = now, and choices at this moment are simply predictive of your later choices, and one of the things you can do at is make a commitment. This looks sort of like conditioning on future actions only as long as the intervening percepts are such that you will continue to honor those commitments - and if you chose to honor them blindly, the percepts would simply become independent of your actions, such that planning certain actions would still clearly have no window-protective effect.
In fact, with longer time horizon options you mostly condition on certain combinations of responses to relatively-predictable percepts - a "partial" policy - not on the combination of actions itself. As long as this partial policy routes you far away from things that could mess up your ability to follow at all (you never use psychedelics) you pretty much do exactly the policy that you most prefer to do (rough proof here), which we might call the optimal policy .
Eventually, you stumble upon lesswrong and learn about weird acausal trades like playing the prisoner's dilemma with a clone of yourself and hopefully cooperating rather than defecting and -
WAS EVERYTHING YOU THOUGHT YOU KNEW WRONG?
Well, maybe you need some kind of exotic "functional decision theory," or maybe you don't - but at least that example seems to be[2] handled very nicely in the same framework you've been using all along. It just happens that certain percept bits (the actions of your clone) are also very well predicted by your own actions. In a way, they're like actions available to your extended self, the larger pattern behind yourself. Push this far enough, and maybe you end up in the same place as those exotic decision theories! But in ordinary situations at least, everything pretty much adds up to normality and this system you've worked out (an approximation to AEDT with rOSI?) explains everything pretty well.
Except, it does seem to get hung up with predicting what you're future self will do when you've gotten smarter instead of dumber. That seems like a really tricky problem, but then again, who could predict what someone smarter than themselves would do? Maybe there's no better answer than "make an informed guess; whatever it is, it'll probably be more clever than anything I can come up with." Maybe there's a better system, but it's not built into your subconscious mind in the same way, because intelligence boosting hasn't historically been available to humans - so you have to figure it out and reason it through consciously.
Though, maybe you do have some built-in instincts about this. At least when it comes to other people smarter than you, you tend to be paranoid - who knows what they might do? If they're working actively against you (say, in a game of chess) you might guess that things will work out about as badly as they reasonably could. Perhaps this kind of pessimism means you aren't really using probabilities anymore - or perhaps it's simply the natural result of your ordinary reasoning process, scraping predictive power out of a particularly cynical heuristic...