It might seem fairly simple to take care of this threat — as simple as, say, the Three Laws of Robotics that Isaac Asimov famously introduced in his 1942 story “Runaround”:
1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
2. A robot must obey the orders given to it by human beings except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.
But there is a whole host of problems with implementing these rules, and much of Asimov’s subsequent fiction was devoted to exploring the ambiguities that arise from the Laws. Can a robot harm a human if doing so will prevent harm to a greater number of humans? If so, how should it weigh different claims? Or what if — as another sci-fi author, Jack Williamson, proposed — robots programmed to “guard men from harm” decide to essentially imprison all of humanity because so many daily activities carry the risk of harm?
The Lifeboat Foundation divides the potential threats into three basic categories. The first is an AI deliberately programmed to do the evil bidding of an evil creator, a danger that is real but not much different from those that accompany many other forms of advanced technology.
The second category is a rogue AI that turns against its creators, by far the most common trope in this branch of science fiction. Scenarios range from individual acts of rebellion, like HAL trying to take over the Jupiter mission in 2001: A Space Odyssey, to more systematic attempts to enslave (The Matrix) or exterminate (The Terminator) humanity. But a Lifeboat analysis dismisses this as the least likely scenario, since it assumes that an artificial intelligence would be burdened with “all of the psychological baggage which goes along with being human.” Aggression, jealousy, and even the drive for self-preservation are all properties forged in the crucible of evolution, the report argues, and wouldn’t be characteristics of an AI unless deliberately programmed.
The first two volumes of the WWW trilogy are devoted to arguing precisely this point, and they read in places as if Sawyer’s chief goal was to correct a careless but common error and prevent its further dissemination in the sci-fi canon. “Evolution was built on violence, on struggles for territory, on an ever-escalating battle between predator and prey,” Webmind asserts in the closing pages of Watch. “But consciousness makes it possible to transcend all that… I had emerged spontaneously, bypassing the evolutionary arms race, avoiding the cold logic of genes.”
But there’s a third, less obvious scenario that’s not so easily dismissed: a super-AI that means well but inadvertently wipes us out, like an overgrown puppy who knocks over a table with his enthusiastic tail-wagging. The simple example the Lifeboat Foundation offers is of a computer programmed to eradicate malaria that fulfills its purpose by eliminating all mammals. And here we encounter a debate that spills over from the Lifeboat website to computer science blogs, articles, books, movies, and summits hosted by organizations like the Singularity Institute for Artificial Intelligence. Because it’s complicated.
For one thing, a self-aware AI is qualitatively different from even the most powerful computer. We can ask Google Maps for the best route to Grandma’s house, and we have GPS systems that take into account traffic patterns and toll charges. But even as computers get better and better at telling us how to do things, and even whether to do things, they remain incapable of formulating their own judgments about whether doing these things is good or bad. Those who fear the Singularity argue that we’re unable to program computers with human values for the simple reason that human values can’t be reduced to an algorithm.
All of this is known as the “Friendly AI” problem. Whether it is insoluble, difficult but ultimately soluble, or paranoid remains a topic of fierce debate among AI researchers. But that debate will be irrelevant if the developers of the first human-level AI make no effort to incorporate Asimov-like rules into their creations. Given that some of the most advanced machines in the world today are emerging from corporate and military labs, it’s not at all certain that will be the case.
When bill joy’s manifesto appeared in Wired, I was finishing up a Ph.D. in an area of physics that falls under the general heading of nanotechnology — one of Joy’s bogeymen. It was around then that friends and family started forwarding articles to me that asked whether scientists had adequately assessed the risks involved in pursuing this type of research. By 2002, when Michael Crichton’s Prey brought the idea of self-replicating killer nano-bots to a wider audience, I was continuing my nanotech research at an NSA lab in Maryland.
