AI ALIGNMENT FORUM
AF

All of Eigil Rischel's Comments + Replies

Clarifying the Agent-Like Structure Problem
Eigil Fjeldgren Rischel30

I mean, "is a large part of the state space" is basically what "high entropy" means!

For case 3, I think the right way to rule out this counterexample is the probabilistic criterion discussed by John - the vast majority of initial states for your computer don't include a zero-day exploit and a script to automatically deploy it. The only way to make this likely is to include you programming your computer in the picture, and of course you do have a world model (without which you could not have programmed your computer)

Reply
3Alex Flint
But the vast majority of initial states for a lump of carbon/oxygen/hydrogen/nitrogen atoms do not include a person programming a computer with the intention of taking over the internet. Shouldn't you apply the same logic there that you apply to the case of a computer? In fact a single zero day exploit is certainly much simpler than a full human, so aprior it's more likely for a computer with a zero day exploit to form from the void than for a computer with a competent human intent on taking over the internet to form from the void.
davidad's Shortform
Eigil Fjeldgren Rischel10

Ha, I was just about to write this post. To add something, I think you can justify the uniform measure on bounded intervals of reals (for illustration purposes, say ) by the following argument: "Measuring a real number " is obviously simply impossible if interpreted literally, containing an infinite amount of data. Instead this is supposed to be some sort of idealization of a situation where you can observe "as many bits as you want" of the binary expansion of the number (choosing another base gives the same measure). If you now apply the princ... (read more)

Reply
2davidad (David A. Dalrymple)
Given any particular admissible representation of a topological space, I do agree you can generate a Borel probability measure by pushing forward the Haar measure of the digit-string space ΣN (considered as a countable product of ω copies of Σ, considered as a group with the modular-arithmetic structure of Z/|Σ|) along the representation. This construction is studied in detail in (Mislove, 2015). But, actually, the representation itself (in this case, the Cantor map) smuggles in Lebesgue measure, because each digit happens to cut the target space "in half" according to Lebesgue measure. If I postcompose, say, x↦√x after the Cantor map, that is also an admissible representation of [0,1], but it no longer induces Lebesgue measure. This works for any continuous bijection, so any absolutely continuous probability measure on [0,1] can be induced by such a representation. In fact, this is why the inverse-CDF algorithm for drawing samples from arbitrary distributions, given only uniform random bits, works. That being said, you can apply this to non-compact spaces. I could get a probability measure on R via a decimal representation, where, say, the number of leading zeros encodes the exponent in unary and the rest is the mantissa. [Edit: I didn't think this through all the way, and it can only represent real numbers ≥1. No big obstacle; post-compose x↦log(x−1).] The reason there doesn't seem to be a "correct" way to do so is that, because there's no Haar probability measure on non-compact spaces (at least, usually?), there's no digit representation that happens to cut up the space "evenly" according to such a canonical probability measure.
Biextensional Equivalence
Eigil Fjeldgren Rischel20

But then shouldn't there be a natural biextensional equivalence ? Suppose , and denote . Then the map is clear enough, it's simply the quotient map. But there's not a unique map - any section of the quotient map will do, and it doesn't seem we can make this choice naturally.

I think maybe the subcategory of just "agent-extensional" frames is reflective, and then the subcategory of "environment-extensional" frames is coreflective. And there's a canonical (i.e natural) zig-zag

Reply
1Scott Garrabrant
You might be right, I am not sure. It looks to me like it satisfies the definition on wikipedia, which does not require that the morphism rB is unique, only that it exists.
Biextensional Equivalence
Eigil Fjeldgren Rischel10

Does the biextensional collapse satisfy a universal property? There doesn't seem to be an obvious map either or (in each case one of the arrows is going the wrong way), but maybe there's some other way to make it universal?

Reply
1Scott Garrabrant
I think the right way to think about biextensional collapse categorically is as a reflector.