I'd be concerned that our instincts toward vengeance in particular would result in extremely poor outcomes if you give humans near-unlimited power (which is mostly granted by being put in charge of an otherwise-sovereign AGI); one potential example is the AGI-controller sending a murderer to an artificial, semi-eternal version of Hell as punishment for his crimes. I believe there's a Black Mirror episode exploring this. In a hypothetical AGI good outcome, this cannot occur.
The idea of a committee of ordinary humans, ems, and semi-aligned AI whi... (read more)
I'd be concerned that our instincts toward vengeance in particular would result in extremely poor outcomes if you give humans near-unlimited power (which is mostly granted by being put in charge of an otherwise-sovereign AGI); one potential example is the AGI-controller sending a murderer to an artificial, semi-eternal version of Hell as punishment for his crimes. I believe there's a Black Mirror episode exploring this. In a hypothetical AGI good outcome, this cannot occur.
The idea of a committee of ordinary humans, ems, and semi-aligned AI whi... (read more)