Relevant powerful agents will be highly optimized

The probability that an agent that is cognitively powerful enough to be relevant to existential outcomes, will have been subject to strong, general optimization pressures. Two (disjunctive) supporting arguments are that, one, pragmatically accessible paths to producing cognitively powerful agents tend to invoke strong and general optimization pressures, and two, that cognitively powerful agents would be expected to apply strong and general optimization pressures to themselves.

An example of a scenario that negates RelevantPowerfulAgentsHighlyOptimized is KnownAlgorithmNonrecursiveIntelligence, where a cognitively powerful intelligence is produced by pouring lots of computing power into known algorithms, and this intelligence is then somehow prohibited from self-modification and the creation of environmental subagents.

Whether a cognitively powerful agent will in fact have been sufficiently optimized depends on the disjunction of:

• PowerfulAgentAlreadyOptimized. Filtering for agents which are powerful enough to be relevant to advanced agent scenarios, will in practice leave mainly agents that have already been subjected to large amounts of cognitive optimization from some internal or external source.
• PowerfulAgentSelfOptimizes. Agents that reach a threshold of power sufficient to be relevant to advanced agent scenarios will afterwards optimize themselves both heavily and generally.

Ending up with a scenario along the lines of KnownAlgorithmNonrecursiveIntelligence requires defeating both of the above conditions simultaneously. The second condition seems more difficult and to require more Corrigibility or CapabilityControl features than the first.