An important caveat is that many non-AI systems have humans in the loop somewhere that can intervene if they don't like what the automated system is doing. Some examples:
Much of the concern about AI systems is when they lack support for these kind of interventions, whether it be because they are too fast, too complex, or can outsmart the would-be intervening human trying to correct what they see as an error.
Much of the concern about AI systems is when they lack support for these kind of interventions, whether it be because they are too fast, too complex, or can outsmart the would-be intervening human trying to correct what they see as an error.
All of these possible causes for the lack of support are valid. I would like to add one more: when the humans that could provide this kind of support don't care about providing it or have incentives against providing it. For example, I could report a bug in some system, but this would cost me time and only benefit people I don't know, so I will happily ignore it :-).
I am grateful to Cara Selvarajah for numerous discussions regarding this post.
Events in the world can be divided into those with mostly pre-determined outcomes and those that look more like a choice. For example, suppose you ask me to tell you a number at random. It will likely seem that I can plausibly give a number of different answers. On the other hand, if we take a random number generator and fix its random seed, there is no choice involved --- its output is pre-determined. Note that this distinction is less about whether the decision is actually pre-determined and more about how things seem to an observer. In practice, an observer will have limited information about the situation and limited ability to predict the situation's outcome. In particular, the above distinction makes sense even in a fully deterministic universe, so we can proceed without this post without resolving the philosophical discussions about determinism vs non-determinism.
On the "pre-determined vs choice" axis, it is sensible to view events involving humans as choices. Indeed, from an observer's point of view, humans are nearly always physically capable of causing more than a single outcome. However, upon closer inspection, some of these outcomes might be *effectively pre-determined_. For example, if a cash machine asks for my favourite number after inserting a credit card --- sure, I can pick anything, but it's rather more likely that I will type in my PIN. Similarly, if coal was a hundred times more cost-effective source of energy than solar, developing countries will probably build coal power plants.
To be more rigorous, we should view this concept as a continuum between "having a choice" and "pre-determined for all practical purposes". However, even the binary version seems quite useful.
A similar situation arises with the distinction between automated and non-automated services. A math course on Coursera.org is automated while a math tutoring session is not. And, as with pre-determined outcomes, many non-automated services are nevertheless effectively automated. Effectively-automated services often consist of several effectively pre-determined outcomes chained together. For example, the cashier in a supermarket is not a robot, but for most practical purposes, they might as well be. Likewise, for most considerations, we can view a McDonald's restaurant as being automated. Typically, a service is effectively automated when the person providing it has some explicit instructions they are supposed to follow. However, there is also that guy who used to teach me linear algebra by rewriting some other guy's textbook on the whiteboard while half asleep and complaining about how boring it is. In other words, effective automation can also arise as a consequence of less explicit incentives (such as minimizing the preparation time and presumably maximizing time spent playing computer games the day before).
We claim that a number of potential problems with AI can also appear in effectively automated systems. To see this, note first that Reframing Superintelligence proposes that highly impactful AI can also take the form of a system that consists of narrow-purpose AI services. Such AI systems could be dangerous for various reasons. Some of the dangers are only present with artificial systems --- for example, thanks to their ability to create copies cheaply or to work much faster than humans. However, at least according to our intuitions, many potential issues are caused by the system's outputs being pre-determined (or automated). For example, we might be worried that a robot instructed to "keep walking straight" will happily fall off a cliff since it cannot choose to do otherwise.
Now, the importance of the problems caused by automation is debatable. However, to the extent that we are worried about automation, we believe that we should also worry about systems of effectively automated services. (Indeed, humans seem quite happy to keep walking if you give them a few dollars, or likes on facebook, for every step towards the cliff's edge.)
The good news is that, to the extent that automated and effectively automated systems behave similarly, some of the same techniques can be used to analyze them. To give some nearer-term examples: when studying the efficiency of Uber's taxi service, it does not make a lot of difference whether the cars are self-driving or not. Similarly, questions like bias in court decisions make sense independently of whether the judge is an AI algorithm or a human. More abstractly, some of the same issues that are present in the current world can also arise in systems with a higher degree of automation. (This could be the case with, for example, increasing inequality, economic incentives causing global warming, delegation and coordination problems.)
The overlap between solutions to problems with automated and effectively automated systems is somewhat smaller. In one direction, some solutions that apply to AI might not work for humans (e.g., reverting a faulty program to an earlier state) or be unethical (e.g., attaching a shutdown button to a robot's forehead). Similarly, humans are capable of some things that are beyond (current) AI: for example, they adapt faster to new environments and they can decide to go against the instructions if they deem it important enough. However, some solutions apply to both settings equally. For example, by holding orchestra auditions behind a screen, we mitigate the effects of various biases of both human and robot members of the hiring committee.
So, there is a similarity between automated and effectively automated systems. However, how can we tell whether a system will behave as if it was automated? One way to think of effective automation is that deviations from the prescribed behaviour are possible, but only at some costs. If the costs are high, individual humans will be very reluctant to deviate from their instructions. Moreover, this will often not suffice to alter the overall course of the system, since doing so would require the coordination of multiple people. In other words, the degree to which we should expect an effectively-automated system to behave like an automated system depends on the stakes, the costs associated with deviating, and the coordination required to alter the outcome.
To sum up, we argue that some human-implemented services and institutions are "effectively automated", which causes overlap between topics studied in AI safety and problems that do not involve (only) AI. This suggests that we could study automated systems, which are easier to formalize and transfer the obtained insights into non-fully-automated systems. Conversely, examples of present-day systems could provide inspiration for the study of automated systems.
For completeness, we conclude by giving two objections to the above argument. First, the argument suggests that understanding of automated systems could transfer to systems that include human institutions, companies, etc. However, to get any value out of this, we would first need to understand at least the automated systems --- and we currently don't. Second, the argument is somewhat weakened by the fact that merely knowing that something is effectively automated does not give us a description of the "as-if-automated" behaviour, or the costs caused by deviating from it. So studying effectively-automated systems is likely to be at least somewhat more difficult than studying the actually-automated ones.