I understand the security mindset (from the ordinary paranoia post) as: "What are the unexamined assumptions of your security systems which merely stem from investing or adapting a given model?". The vulnerability comes from the model. The problem is the "unknowable unknowns". In addition to the Cryptographer and the Cryptography skeptic, I would add the NSA Quantum computing engineer. Concretisation and operationalisation of these problems may have implicit assumptions that could be system wide catastrophic.
I don't have clear ways of better articulating this back from analogy to Paul's concretisations of a proposed AI system. I'm not sure there's no disanalogy here. However it could be something like "We have this effective model of a proposed AI system. What are useful concretisations in which the AI system would fail?". The security mindset question would be something like "What representations in the 'UV-complete' theory of this AI system would lead to catastrophic failure modes?"
I understand the security mindset (from the ordinary paranoia post) as: "What are the unexamined assumptions of your security systems which merely stem from investing or adapting a given model?". The vulnerability comes from the model. The problem is the "unknowable unknowns". In addition to the Cryptographer and the Cryptography skeptic, I would add the NSA Quantum computing engineer. Concretisation and operationalisation of these problems may have implicit assumptions that could be system wide catastrophic.
I don't have clear ways of better articulating this back from analogy to Paul's concretisations of a proposed AI system. I'm not sure there's no disanalogy here. However it could be something like "We have this effective model of a proposed AI system. What are useful concretisations in which the AI system would fail?". The security mindset question would be something like "What representations in the 'UV-complete' theory of this AI system would lead to catastrophic failure modes?"
I'm probably missing something here though.