AI ALIGNMENT FORUM
AF

All of Dacyn's Comments + Replies

Contra Common Knowledge
David Simmons20

The infinite autoresponse example seems like it would be solved in practice by rational ignorance: after some sufficiently small number of autoresponses (say 5) people would not want to explicitly reason about the policy implications of the specific number of autoresponses they saw, so "5+ autoresponses" would be a single category for decisionmaking purposes. In that case the induction argument fails and "both people go to the place specified in the message as long as they observe 5+ autoresponses" is a Nash equilibrium.

Of course, this assumes people haven... (read more)

Reply
2Abram Demski
I agree that something in this direction could work, and plausibly captures something about how humans reason. However, I don't feel satisfied. I would want to see the idea developed as part of a larger framework of bounded rationality. UDT gives us a version of "never be harmed by information" which is really nice, as far as it goes. In the cases which UDT helps with, we don't need to do anything tricky, where we carefully decide which information to look at -- UDT simply isn't harmed by the information, so we can think about everything from a unified perspective without hiding things from ourselves. Unfortunately, as I've outlined in the appendix, UDT doesn't help very much in this case. We could say that UDT guarantees that there's no need for "rational ignorance" when it comes to observations (ie, no need to avoid observations), but fails to capture the "rational ignorance" of grouping events together into more course-grained events (eg "5+ auto responses"). So if we had something like "UDT but for course-graining in addition to observations", that would be really nice. Some way to deal with things such that you never wish you'd course-grained things. Whereas the approach of actually course-graining things, seems a bit doomed to fragility and arbitrariness. It seems like you have to specify some procedure for figuring out when you'd want to course-grain. For example, maybe you start with only one event, and iteratively decide how to add details, splitting the one event into more events. But I feel pessimistic about this. I feel similarly pessimistic about the reverse, starting with a completely fine-grained model and iteratively grouping things together.  Fortunately, the induction argument involves both agents following along with the whole argument. If one agent doubts that the other thinks in this way, this can sort of stabilize things. It's similar to the price-undercutting dynamic, where you want to charge slightly less than competitors, not as little
How promising are legal avenues to restrict AI training data?
David Simmons1-1

-"For example, I could imagine laws requiring anyone scraping the internet to ensure that they are not collecting data from people who have denied consent to have their data scraped."

In practice this is already the case, anyone who doesn't want their data scraped can put up a robots.txt file saying so, and I imagine big companies like OpenAI respect robots.txt. I guess there could be advantages in making it a legal rule but I don't think it matters too much.

Reply
Counterfactuals, thick and thin
David Simmons20

Sure, in that case there is a 0% counterfactual chance of heads, your words aren't going to flip the coin.

Reply
2Nisan
Ok. I think that's the way I should have written it, then.
Counterfactuals, thick and thin
David Simmons30

The question "how would the coin have landed if I had guessed tails?" seems to me like a reasonably well-defined physical question about how accurately you can flip a coin without having the result be affected by random noise such as someone saying "heads" or "tails" (as well as quantum fluctuations). It's not clear to me what the answer to this question is, though I would guess that the coin's counterfactual probability of landing heads is somewhere strictly between 0% and 50%.

Reply
2Nisan
Oh, interesting. Would your interpretation be different if the guess occurred well after the coinflip (but before we get to see the coinflip)?
Repeated (and improved) Sleeping Beauty problem
David Simmons20

I'm confused, isn't the "objective probability" of heads 1/2 because that is the probability of heads in the definition of the setup? The halver versus thirder debate is about subjective probability, not objective probability, as far as I can tell. I'm not sure why you are mentioning objective probability at all, it does not appear to be relevant. (Though it is also possible that I do not know what you mean by "objective probability".)

Reply
Repeated (and improved) Sleeping Beauty problem
David Simmons10

This argument seems to depend on the fact that Sleeping Beauty is not actually copied, but just dissociated from her past self and so that from her perspective it seems like she is copied. If you deal with actual copies then it is not clear what is the sensible way for them to all pass around a science journal to record their experiences, or all keep their own science journals, or all keep their own but then recombine somehow, or whatever. Though if this thought experiment gives you SIA intuitions on the Sleeping Beauty problem then maybe those intuitions will still carry over to other scenarios.

Reply
0The Ancient Geek
This statement of the problem concedes that SB is calculating subjective probability. It should be obvious that subjective probabilities can diverge from each and objective probability -- that is what subjective means. It seems to me that the SB paradox is only a paradox if y ou try to do justice to objective and subjective probability in the same calculation.
Probability is fake, frequency is real
David Simmons10

I don't know what you mean by "should be allowed to put whatever prior I want". I mean, I guess nobody will stop you. But if your beliefs are well approximated by a particular prior, then pretending that they are approximated by a different prior is going to cause a mismatch between your beliefs and your beliefs about your beliefs.

[Nitpick: The Kelly criterion assumes not only that you will be confronted with a large number of similar bets, but also that you have some base level of risk-aversion (concave utility function) that repeated bets ... (read more)

Reply
2Linda Linsefors
I agree that "want" is not the correct word exactly. What I mean by prior is an agents actual a priori beliefs, so by definition there will be no mis-match there. I am not trying to say that you choose your prior exactly. What I am gesturing at is that no prior is wrong, as long as it does not assign zero probability to the true outcome. And I think that much of the confusion in atrophic situation comes from trying to solve an under-constrained system.
Beyond Astronomical Waste
David Simmons10

MUH doesn't imply the existence of halting oracles. Indeed, the Computable Universe Hypothesis is supposed to be an extension of the Mathematical Universe Hypothesis, but CUH says that halting oracles do not exist.

Reply
2Wei Dai
There may be several confusions happening here. First I've been using MUH to mean "ultimate ensemble theory" (i.e., the idea that the Level IV multiverse of all mathematical structures exists), because Wikipedia says MUH is "also known as the ultimate ensemble theory". But Tegmark currently defines MUH as "Our external physical reality is a mathematical structure" which seems to be talking just about our particular universe and not saying that all mathematical structures exist. Second if by "MUH doesn’t imply the existence of halting oracles" you mean that MUH doesn't necessarily imply the existence of halting oracles in our universe, then I agree. What I meant in the OP is that the ultimate ensemble theory implies that universes containing halting oracles exist in the Level IV multiverse. Hopefully that clarifies things?
Formal Open Problem in Decision Theory
David Simmons10

I will have to think more about the issue of continuity vs uniform continuity. I suppose my last remaining argument would be the fact that Bishop--Bridges' classic book on constructive analysis uses uniform continuity on bounded sets rather than continuity, which suggests that it is probably better for constructive analysis at least. But maybe they did not analyze the issue carefully enough, or maybe the relevant issues here are for some reason different.

To fix the argument that every locally compact Polish space admits a proper metric, let be as before

... (read more)
Reply
0Alex Mennen
Hm, perhaps I should figure out what the significance of uniform continuity on bounded sets is in constructive analysis before dismissing it, even though I don't see the appeal myself, since constructive analysis is not a field I know much about, but could potentially be relevant here. f is the reciprocal of what it was before, but yes, this looks good. I am happy with this proof.
Formal Open Problem in Decision Theory
David Simmons10

I don't know why my comment doesn't have a reply button. Maybe it is related to the fact that my comment shows up as "deleted" when I am not logged in.

Sorry, I seem to be getting a little lazy with these proofs. Hopefully I haven't missed anything this time.

New proof: ... We can extract a subsequence such that if and , then for all , and for all and , either (A) and or (B) and . By extracting a further subseque

... (read more)
Reply
0Alex Mennen
It appears that comments from new users are collapsed by default, and cannot be replied to without a "Like". These seem like bad features. Your proof that there's no uniformly continuous on bounded sets function f:X×X→[0,1] admitting all uniformly continuous on bounded sets functions X→[0,1] as fibers looks correct now. It also looks like it can be easily adapted to show that there is no uniformly continuous f:X×X→[0,1] admitting all uniformly continuous functions X→[0,1] as fibers. Come to think of it, your proof works for arbitrary metric spaces X, not just complete separable metric spaces, though those are nicer. I see what you mean now about uniform continuity giving you an algorithm, but I still don't think that's specific to uniform continuity in an important way. After all, if you have an algorithm for computing images of points in the countable dense set, and a computable "local modulus of continuity" in the sense of a computable function h:X×[0,∞)→[0,∞) with h(x,0)=0 and d(x,y)<r⟹|f(y)−f(x)|<h(x,r), then f is computable, and this does not require f to be uniformly continuous. Although I suppose you could object that this is a bit circular, in that I'm assuming the "local modulus of continuity" is computable only in the standard sense, which does not require uniform continuity. I'm not sure why you would allow singularities at some points (presumably a uniformly discrete set, or something like that) while still insisting on uniform continuity elsewhere. It still seems to me that the arguments for uniform continuity rather than continuity all point to wanting uniform continuity entirely, rather than some sense of local uniform continuity in most places. Thanks for pointing out the error in my definition of fn; I've fixed it. In your argument that locally compact Polish spaces can be given metrics with respect to which they are proper, it isn't true that d′ is necessarily a proper metric. For instance, consider a countably infinite set with d(x,y)=1 for x
Formal Open Problem in Decision Theory
David Simmons10

I am going to take some license with your question because I think you are asking the wrong question. Arbitrary topological spaces and abstract continuity are rarely the right notions in real-world situations. Rather, uniform continuity on bounded sets usually better corresponds to the intuitive notion of "a small change in input produces a small change in output".

Thus, suppose that is a complete separable metric space and that is uniformly continuous on bounded sets. Then we can show that there exists a function which is uniformly

... (read more)
Reply
1Alex Mennen
g can be a fiber of f, since for each n, xn and yn could be distance greater than n from the basepoint. Example: let X:={xn,yn|n∈N}, with d(xn,xm)=d(yn,ym)=d(xn,ym)=2|n−m| for n≠m and d(xn,yn)=6−n. Let x0 be the basepoint (so that (x0,x0) is the point you were calling "0"). Let g(xn):=0, g(yn):=1, f(z,w):=g(z), and hn(r):=3nr. I also don't see how to even construct the function g, or, relatedly, what you mean by "geometrically nicely", but I guess it doesn't matter. Also, I'm not convinced that metric spaces with uniform continuity on bounded subsets is a better framework than topological spaces with continuity.