When people say “the heart’s purpose is to pump blood” or “a pencil’s function is to write”, what does that mean physically? What are “purpose” or “function”, not merely in intuitive terms, but in terms of math and physics? That’s the core question of what philosophers call teleology - the study of “telos”, i.e. purpose or function or goal.
This post is about a particular way of approaching conceptual/philosophical questions, especially for finding “True Names” - i.e. mathematical operationalizations of concepts which are sufficiently robust to hold up under optimization pressure. We’re going to apply the method to teleology as an example. We’ll outline the general approach in abstract later; for now, try to pay attention to the sequence of questions we ask in the context of teleology.
Cognition
We start from the subjective view: set aside (temporarily) the question of what “purpose” or “function” mean physically. Instead, first ask what it means for me to view a heart as “having the purpose of pumping blood”, or ascribe the “function of writing” to a pencil. What does it mean to model things as having purpose or function?
Proposed answer: when I ascribe purpose or function to something, I model it as having been optimized (in the sense usuallyusedon LessWrong) to do something. That’s basically the standard answer among philosophers, modulo expressing the idea in terms of the LessWrong notion of optimization.
(From there, philosophers typically ask about “original teleology” - i.e. a hammer has been optimized by a human, and the human has itself been optimized by evolution, but where does that chain ground out? What optimization process was not itself produced by another optimization process? And then the obvious answer is “evolution”, and philosophers debate whether all teleology grounds out in evolution-like phenomena. But we’re going to go in a different direction, and ask entirely different questions.)
Convergence
Next: I notice that there’s an awful lot of convergence in what things different people model as having been optimized, and what different people model things as having been optimized for. Notably, this convergence occurs even when people don’t actually know about the optimization process - for instance, humans correctly guessed millenia ago that living organisms had been heavily optimized somehow, even though those humans were totally wrong about what process optimized all those organisms; they thought it was some human-like-but-more-capable designer, and only later figured out evolution.
Our everyday experience implies that there is some property of e.g. a heron such that many different people can look at the heron, convergently realize that the heron has been optimized for something, and even converge to some degree on which things the heron (or the parts of the heron) have been optimized for - for instance, that the heron’s heart has been optimized to pump blood. (Not necessarily perfect convergence, not necessarily everyone, but any convergence beyond random chance is a surprise to be explained if we’re starting from a subjective account.) Crucially, it’s a property of the heron, and maybe of the heron’s immediate surroundings, not of the heron’s whole ancestral environment - because people can convergently figure out that the heron has been optimized just by observing the heron in its usual habitat.
So now we arrive at the second big question: what are the patterns out in the world which different people convergently recognize as hallmarks of having-been-optimized? What is it about herons, for instance, which makes it clear that they’ve been optimized, even before we know all the details of the optimization process?
Candidate answer (underspecified and not high confidence, but it will serve for an example): the system has lots of parts which are all in unusual/improbable states, but all “in a consistent direction” in some sense. So it looks like all the parts were pushed away from what’s statistically typical, in “the same way”.[1]
Ideally, we could operationalize that intuitive answer in a way which would make convergence provable; it has the right flavor for a natural latent style convergence argument.
Corroboration
Imagine, now, that we have a full mathematical operationalization of “parts which are all in unusual/improbable states, but all ‘in a consistent direction’”. Imagine also that we are able to prove convergence. What else would we want from this operationalization of teleology?
Well, I look at a heron, I notice that it has a bunch of parts which are all in unusual/improbable states, but all ‘in a consistent direction’ - i.e. all its parts are in whatever unusual configurations they need to be in for the heron to survive; random configurations would not do that. I conclude that the heron has been optimized. Insofar as my intuition picks up on “parts which are all in unusual/improbable states, but all ‘in a consistent direction’” and interprets that pattern as a hallmark of optimization, and my intuition is correct… then it should be a derivable/provable fact about the external world that “parts which are all in unusual/improbable states, but all ‘in a consistent direction’” occur approximately if-and-only-if a system has been optimized.
More generally: insofar as we have some intuitions about how teleology works, we should be able to prove that our operationalization/characterization indeed works that way. (Or, insofar as the operationalization doesn’t work the way we intuitively expect, we should be able to propagate the counterexamples back to our intuitions and conclude that our intuitions were wrong or required additional assumptions, as opposed to the operationalization being wrong.)
Cognition -> Convergence -> Corroboration
Let’s go back over the teleology example, with an emphasis on what questions we’re asking and why.
We start with questions about my cognition:
…first ask what it means for me to view a heart as “having the purpose of pumping blood”, or ascribe the “function of writing” to a pencil. What does it mean to model things as having purpose or function?
Two things to emphasize: first, these are questions about my cognition (or, more generally, one person’s cognition); the answers may or may not generalize to other people. Second, they are questions about my cognition; they’re not asking about how the external world “actually is” (at least not directly).
Some nice things about starting from questions about my cognition:
I can get lots of relevant data by introspecting.
I can get lots of relevant data from my background models of cognition, and leverage abstract models of cognition (like e.g. Bayesianism or relaxation-based search) to formulate my understanding mathematically.
The downside is that introspection is notoriously biased and error-prone, and this is all not-very-legible and hard to test/prove. That’s fine for now; (some) legible falsifiability will enter in the next steps.
From cognition, we move on to questions about convergence:
Next: I notice that there’s an awful lot of convergence in what things different people model as having been optimized, and what different people model things as having been optimized for. [...]
The standard answer of interest, which generalizes well beyond teleology, is: people pick up on the same patterns in the environment, and convergently model/interpret them in similar ways. Then the generalizable question is: what are those patterns? Or, in the context of teleology:
… what are the patterns out in the world which different people convergently recognize as hallmarks of having-been-optimized? What is it about herons, for instance, which makes it clear that they’ve been optimized, even before we know all the details of the optimization process?
At this point, we start to have space for falsifiable predictions and/or mathematical proof: if we have a candidate pattern, then we should be able to demonstrate/prove that it is, in fact, convergently recognized (in some reasonable sense, under some reasonable conditions) by many minds. Such a proof is where a natural latent style argument would typically come in (though of course there may be other ways to establish convergence).
Once convergence is established, we know that we’ve characterized some convergently-recognized pattern. The last step is that it’s the convergently-recognized pattern we’re looking for. For instance, maybe dogs are a convergently-recognized pattern in our environment, and having-been-optimized is also a convergently-recognized pattern in our environment. If we’ve established that “parts which are all in unusual/improbable states, but all ‘in a consistent direction’” is a convergently-recognized pattern in our environment, how do we argue that that pattern is the-thing-humans-call-“teleology”, as opposed to the-thing-humans-call-“dogs”?
Well, we show that the pattern has some of the other properties we expect of teleology.
More generally, this is the corroboration step. We want to prove/demonstrate some further consequences of the pattern identified in the previous step (including how it interfaces with other patterns we think we’ve identified), in order to make sure it’s the pattern we intended to find, as opposed to some other convergently-recognized pattern. This is where all your standard math (and maybe science) would come in.
Cognition -> Convergence -> Corroboration. That’s the pipeline.
Examples are Confusing, Let’s Make it Really Abstract!
The Cognition -> Convergence -> Corroboration Algorithm:
Cognition: Guess at a Cognitive Operationalization.
Cognitive Model Selection: Choose a framework to model your own mind. Bayesianism is one often-fruitful place to start.
Cognitive Operationalization: Within that framework, operationalize the intuition itself. E.g. within my bayesian world model, what is a “dog.” Within my bayesian world model, what is “purpose” or what is going on in my bayesian world model when I ascribe “purpose” to some part of it?
Convergence: Guess at a Pattern + Prove Pattern is Convergent.
Operationalization: (Somehow) use the Cognitive Operationalization to intuit an Environmental Operationalization of the concept in terms of the external world, possibly using methods which include investigating instances which many people point to and agree is “the thing.” If you fail, return to step 1.
Convergability: Check that your operationalization is in fact a candidate member of “patterns which minds tend to convergently recognize” (which we usually operationalize using natural latents). If you fail, return to step 2.a
Corroboration: Derive further properties about the candidate external pattern (operationalization) and check if those further properties are consistent with the original intuitive concept. If they aren’t, return to step 2.
Upon failure sending you back to step 1, three things could be wrong. Use magic to figure out which it is:
Cognitive Model is fine, Intuition is fine, Cognitive Operationalization needs to update. -> Update the Cognitive Operationalization and return to step 2.
Cognitive Model is fine, Intuition needs to update. -> Update the intuition and return to step 1.b.
Intuition is fine, Cognitive Model needs to update. -> Update the cognitive model and return to step 1.b.
Also, obviously, if you’re caught in a loop (like, e.g., failing step 3 and going back to step 2.a over and over, jump back a bit further, e.g. step 1.)
When is the Cognition -> Convergence -> Corroboration Pipeline Useful?
The central use case is:
There’s some concept…
which different humans can successfully communicate about well enough that basically-the-same concept seems to show up in their different heads (as evidenced by e.g. systematically pointing at stuff and calling it the same thing)…
and we want a robust mathematical operationalization of that concept (i.e. a “True Name”).
Most topics studied in philosophy are in-scope. Most (but importantly not all) “deconfusion” work is in-scope.
Beyond just a useful process to follow for such use-cases, we’ve also found the Cognition -> Convergence -> Corroboration structure useful for organizing thoughts/arguments: it’s useful to explicitly distinguish a cognitive characterization from a convergent pattern characterization from a consequence. For instance, we’ve often found it useful to explain some problem we’re thinking about as “What are the patterns/structures in the world which people convergently recognize as X?”.
Some use-cases for which this pipeline is probably not the right tool:
Operationalizing some concept which humans are not able to communicate about at all. (You know which I’m talking about. (He’s lying, you don’t.))
Doing math or engineering with concepts which have already been operationalized.
Punditry, publishing ML papers, and other use cases for which being robustly correct or understanding what one is doing is not a particularly central objective.
If you want to see more examples where we apply this methodology, check out the Tools post, the recent Corrigibility post, and (less explicitly) the Interoperable Semantics post.
Thank you to Steve Petersen and Ramana Kumar for our discussions of teleology; it was in those discussions that the example in this post bubbled around in my head.
If “unusual/improbable” still sounds too subjective, then you can think of operationalizing it in the Solomonoff/Kolmogorov sense, i.e. in terms of compressibility using a simple Turing machine.
Opening Example: Teleology
When people say “the heart’s purpose is to pump blood” or “a pencil’s function is to write”, what does that mean physically? What are “purpose” or “function”, not merely in intuitive terms, but in terms of math and physics? That’s the core question of what philosophers call teleology - the study of “telos”, i.e. purpose or function or goal.
This post is about a particular way of approaching conceptual/philosophical questions, especially for finding “True Names” - i.e. mathematical operationalizations of concepts which are sufficiently robust to hold up under optimization pressure. We’re going to apply the method to teleology as an example. We’ll outline the general approach in abstract later; for now, try to pay attention to the sequence of questions we ask in the context of teleology.
Cognition
We start from the subjective view: set aside (temporarily) the question of what “purpose” or “function” mean physically. Instead, first ask what it means for me to view a heart as “having the purpose of pumping blood”, or ascribe the “function of writing” to a pencil. What does it mean to model things as having purpose or function?
Proposed answer: when I ascribe purpose or function to something, I model it as having been optimized (in the sense usually used on LessWrong) to do something. That’s basically the standard answer among philosophers, modulo expressing the idea in terms of the LessWrong notion of optimization.
(From there, philosophers typically ask about “original teleology” - i.e. a hammer has been optimized by a human, and the human has itself been optimized by evolution, but where does that chain ground out? What optimization process was not itself produced by another optimization process? And then the obvious answer is “evolution”, and philosophers debate whether all teleology grounds out in evolution-like phenomena. But we’re going to go in a different direction, and ask entirely different questions.)
Convergence
Next: I notice that there’s an awful lot of convergence in what things different people model as having been optimized, and what different people model things as having been optimized for. Notably, this convergence occurs even when people don’t actually know about the optimization process - for instance, humans correctly guessed millenia ago that living organisms had been heavily optimized somehow, even though those humans were totally wrong about what process optimized all those organisms; they thought it was some human-like-but-more-capable designer, and only later figured out evolution.
Why the convergence?
Our everyday experience implies that there is some property of e.g. a heron such that many different people can look at the heron, convergently realize that the heron has been optimized for something, and even converge to some degree on which things the heron (or the parts of the heron) have been optimized for - for instance, that the heron’s heart has been optimized to pump blood. (Not necessarily perfect convergence, not necessarily everyone, but any convergence beyond random chance is a surprise to be explained if we’re starting from a subjective account.) Crucially, it’s a property of the heron, and maybe of the heron’s immediate surroundings, not of the heron’s whole ancestral environment - because people can convergently figure out that the heron has been optimized just by observing the heron in its usual habitat.
So now we arrive at the second big question: what are the patterns out in the world which different people convergently recognize as hallmarks of having-been-optimized? What is it about herons, for instance, which makes it clear that they’ve been optimized, even before we know all the details of the optimization process?
Candidate answer (underspecified and not high confidence, but it will serve for an example): the system has lots of parts which are all in unusual/improbable states, but all “in a consistent direction” in some sense. So it looks like all the parts were pushed away from what’s statistically typical, in “the same way”.[1]
Ideally, we could operationalize that intuitive answer in a way which would make convergence provable; it has the right flavor for a natural latent style convergence argument.
Corroboration
Imagine, now, that we have a full mathematical operationalization of “parts which are all in unusual/improbable states, but all ‘in a consistent direction’”. Imagine also that we are able to prove convergence. What else would we want from this operationalization of teleology?
Well, I look at a heron, I notice that it has a bunch of parts which are all in unusual/improbable states, but all ‘in a consistent direction’ - i.e. all its parts are in whatever unusual configurations they need to be in for the heron to survive; random configurations would not do that. I conclude that the heron has been optimized. Insofar as my intuition picks up on “parts which are all in unusual/improbable states, but all ‘in a consistent direction’” and interprets that pattern as a hallmark of optimization, and my intuition is correct… then it should be a derivable/provable fact about the external world that “parts which are all in unusual/improbable states, but all ‘in a consistent direction’” occur approximately if-and-only-if a system has been optimized.
More generally: insofar as we have some intuitions about how teleology works, we should be able to prove that our operationalization/characterization indeed works that way. (Or, insofar as the operationalization doesn’t work the way we intuitively expect, we should be able to propagate the counterexamples back to our intuitions and conclude that our intuitions were wrong or required additional assumptions, as opposed to the operationalization being wrong.)
Cognition -> Convergence -> Corroboration
Let’s go back over the teleology example, with an emphasis on what questions we’re asking and why.
We start with questions about my cognition:
Two things to emphasize: first, these are questions about my cognition (or, more generally, one person’s cognition); the answers may or may not generalize to other people. Second, they are questions about my cognition; they’re not asking about how the external world “actually is” (at least not directly).
Some nice things about starting from questions about my cognition:
The downside is that introspection is notoriously biased and error-prone, and this is all not-very-legible and hard to test/prove. That’s fine for now; (some) legible falsifiability will enter in the next steps.
From cognition, we move on to questions about convergence:
The standard answer of interest, which generalizes well beyond teleology, is: people pick up on the same patterns in the environment, and convergently model/interpret them in similar ways. Then the generalizable question is: what are those patterns? Or, in the context of teleology:
At this point, we start to have space for falsifiable predictions and/or mathematical proof: if we have a candidate pattern, then we should be able to demonstrate/prove that it is, in fact, convergently recognized (in some reasonable sense, under some reasonable conditions) by many minds. Such a proof is where a natural latent style argument would typically come in (though of course there may be other ways to establish convergence).
Once convergence is established, we know that we’ve characterized some convergently-recognized pattern. The last step is that it’s the convergently-recognized pattern we’re looking for. For instance, maybe dogs are a convergently-recognized pattern in our environment, and having-been-optimized is also a convergently-recognized pattern in our environment. If we’ve established that “parts which are all in unusual/improbable states, but all ‘in a consistent direction’” is a convergently-recognized pattern in our environment, how do we argue that that pattern is the-thing-humans-call-“teleology”, as opposed to the-thing-humans-call-“dogs”?
Well, we show that the pattern has some of the other properties we expect of teleology.
More generally, this is the corroboration step. We want to prove/demonstrate some further consequences of the pattern identified in the previous step (including how it interfaces with other patterns we think we’ve identified), in order to make sure it’s the pattern we intended to find, as opposed to some other convergently-recognized pattern. This is where all your standard math (and maybe science) would come in.
Cognition -> Convergence -> Corroboration. That’s the pipeline.
Examples are Confusing, Let’s Make it Really Abstract!
The Cognition -> Convergence -> Corroboration Algorithm:
Upon failure sending you back to step 1, three things could be wrong. Use magic to figure out which it is:
Also, obviously, if you’re caught in a loop (like, e.g., failing step 3 and going back to step 2.a over and over, jump back a bit further, e.g. step 1.)
When is the Cognition -> Convergence -> Corroboration Pipeline Useful?
The central use case is:
Most topics studied in philosophy are in-scope. Most (but importantly not all) “deconfusion” work is in-scope.
Beyond just a useful process to follow for such use-cases, we’ve also found the Cognition -> Convergence -> Corroboration structure useful for organizing thoughts/arguments: it’s useful to explicitly distinguish a cognitive characterization from a convergent pattern characterization from a consequence. For instance, we’ve often found it useful to explain some problem we’re thinking about as “What are the patterns/structures in the world which people convergently recognize as X?”.
Some use-cases for which this pipeline is probably not the right tool:
If you want to see more examples where we apply this methodology, check out the Tools post, the recent Corrigibility post, and (less explicitly) the Interoperable Semantics post.
Thank you to Steve Petersen and Ramana Kumar for our discussions of teleology; it was in those discussions that the example in this post bubbled around in my head.
If “unusual/improbable” still sounds too subjective, then you can think of operationalizing it in the Solomonoff/Kolmogorov sense, i.e. in terms of compressibility using a simple Turing machine.