You wake up tomorrow, and find out that super-smart and powerful aliens are arriving here in 10 years. We don't know anything about them other than the fact that they're super smart and powerful: e.g, their minds run on giant computers at THz clock speeds, they can integrate planets' worth of sensor data at once, they can do direct mass to energy conversion in their spaceships, they have weapons capable of destroying our planet in an instant, etc etc.
What do you, personally, do?
What should humanity overall do?
I think the answers to these questions shine a very important light on overlooked AGI alignment strategies. Rather than seeing the arrival of AGI as a technological change, we can see it as the arrival of a potentially competing / potentially collaborating species. Approximate intuition pump: humans :: AGI : animals :: humans -- if you were an emu (about to be hunted to extinction), how would you negotiate with the humans? If you were a monkey, how would you negotiate with the humans?
What are the frameworks we would use to negotiate with supersmart powerful aliens, or which animals might use to negotiate with people?
creating an ultimate ground rules moral framework: if we can create a moral framework which every species could agree to to improve the stable equilibrium of our interspecies game (where the lowest stable equilibrium is violence), then it might be convincing for an AGI coming out of our human values to maintain that ultimate ground rules moral framework. my best idea for that framework is some version of "information is sacred": I have no idea how exactly to measure information, but I think it all sort of ends up working out.
preservation of information: we must try to maintain as much information as possible. most of the universe could be compressed quite a bit, but not humans: our incompressability is what gives us moral worth. the same could be said of animals. how do you determine what can actually be compressed well? it's not physically possible to do perfectly, but you can have heuristics. this is also a good reason to keep animals in special wildlife conservation areas. unfortunately it might accord quite a high moral valence to hard drives: that's fine if humans still make it out in the end
continuation of computation: storing the mind state of a person isn't enough. you want their mind state to continue forwards. so somehow, we care about counterfactual future information and not just current information. obviously it's hard to know that for sure: but we could always have opinions and guess.
preference maximisation: in some sense, information wants to become more information and not die out (regardless of how exactly we measure information). this is based off of the environment of the information. this might be way too difficult to do with the limited computational resources in our universe, but the ultimate moral way to treat an alien species is to give them access to a computer with infinite memory and time in which they can "maximise their information", in many different ways: so like simulating humans living 1000s of different lives.
through understanding their values, use diplomacy to cajole them into better coordination. animals have a very hard time appealing to human values (e.g, dogs asking for attention): and AGI values might be much further away from human values vs animal values from humans. but: maybe not!!
fight militarily. we will lose, but some humans will still think this is a good idea.
use trade/other mutual benefit to improve cooperation. nation states coordinate much better when there is some kind of mutual benefit. e.g, humans provide some kind of entertainment or other sentimental service. seems unlikely we could come up with a way do that for an AGI, since we have no idea what its desires or capabilities will be. if it happens it's unlikely that it happened because we planned it. e.g, AGI simulates mass scale human civilisations out of curiosity to understand its origins
worldwide ban, initiated by the chinese/us gov, to make AGI development illegal, before it even comes into existence.
You wake up tomorrow, and find out that super-smart and powerful aliens are arriving here in 10 years. We don't know anything about them other than the fact that they're super smart and powerful: e.g, their minds run on giant computers at THz clock speeds, they can integrate planets' worth of sensor data at once, they can do direct mass to energy conversion in their spaceships, they have weapons capable of destroying our planet in an instant, etc etc.
What do you, personally, do?
What should humanity overall do?
I think the answers to these questions shine a very important light on overlooked AGI alignment strategies. Rather than seeing the arrival of AGI as a technological change, we can see it as the arrival of a potentially competing / potentially collaborating species. Approximate intuition pump: humans :: AGI : animals :: humans -- if you were an emu (about to be hunted to extinction), how would you negotiate with the humans? If you were a monkey, how would you negotiate with the humans?
What are the frameworks we would use to negotiate with supersmart powerful aliens, or which animals might use to negotiate with people?
Can you think of any more?