All of Jiro's Comments + Replies

The Halting problem is a worst case result. Most agents aren't maximally ambiguous about whether or not they halt. And those that are,  well then it depends what the rules are for agents that don't halt.  There are set ups where each agent is using an nonphysically large but finite amount of compute. There was a paper I saw somewhere a while ago where both agents were doing a brute force proof search for the statement "if I cooperate, then they cooperate" and cooperating if they found a proof. (Ie searching all proofs containing <10^100 symbols)

In this case, we have (by assumption) an output of the program, so we just look at the cases where the program gives that output.

Certainly. But even if the results are not useful and can't be generalized to other situations, it's probably possible to replicate it, in a way that's slightly different from running the same program with a different seed. (E.g. you could run the same algorithm on a different environment that was constructed to be the kind of environment that algorithm could solve.) So this wouldn't work as a test to distinguish between useful results and non-useful results. Relevant recent Andrew Gelman blog post

So I don't really like the terminology of "p-hacking" and "replication", and I'm going to use different terminology here that I find more precise and accurate; I don't know how much I need to explain so you should ask if any particular term is unclear. Indeed, because I don't think it's the relevant concept for ML research. I'm more concerned about the garden of forking paths. (I also strongly recommend Andrew Gelman's blog, which has shaped my opinions on this topic a lot.) Almost all ML experiments will replicate, for some definition of "replicate" -- if you run the same code, even with different random seeds, you will usually get the same results. (I would guess it is fairly rare for researchers to overfit to particular seeds, though it can happen.) In research involving humans, usually some experimental details will vary, just by accident, and that variation leads to a natural "robustness check" that gives us a tiny bit of information about how externally valid the result is. We might not get even that with ML. It's also worth noting that even outside of ML, what does or doesn't count as a "replication" varies widely, so I prefer not to use the term at all, and instead talk about how a separate experiment gives us information about the validity and generalization properties of a particular theory. Hopefully it's a bit clearer now. Given what I've said in this comment, I might now restate the sentence as "I expect that the preregistered experiments will not be sufficiently different from the experiments the researchers ran themselves. So, I expect the results to be of limited use in informing me about the external validity of the experiment and the generalization properties of their theory."