Jim Baen's Universe

Nonfiction articles

Who’s Afraid of the Big Bad Singularity?

Written by Stephen Euin Cobb

Artificial intelligence is a highly controversial topic for professional futurists. The controversy hangs on the question: “Is it possible to create software as intelligent as a human being?”

This question has split futurists into several groups. Those who believe AI is impossible are the least involved in the controversy, but they are also a minority. Those who believe AI is possible or likely or even inevitable are divided into two opposing camps. One camp is eager for it, the other is scared. Here’s why.

Imagine programmers at various software companies develop a wide variety of artificial intelligence programs and one of them exhibits behaviors as smart as a mouse. Not particularly useful perhaps, except maybe as an automated opponent in some kind of computer game, but also not very scary.

But the programmers do not stop there. Excited by their achievement, they work to improve the software and the following year they have a program as smart as a dog. Still not very scary, but the next year it’s as smart as a monkey, and the next as smart as a man. Okay, now that’s useful. We can finally talk to our computers using ordinary human languages and they will understand what we want them to do even if we don’t describe it very well. No more of those cryptic error messages.

Only there are two problems: one big and one little.

One is that the programmers will continue to make the software smarter, and the companies they work for will encourage them in this since the companies will worry that their competition’s version of AI software might become smarter—and therefore more desirable to consumers—than theirs. And so the enthusiasm for profits will provide funding for the programmer’s enthusiasm for continuing to improve their creation.

But working to make their AI software smarter than the competition’s software means they are also working to make their AI software smarter than people. Not a little smarter, but as much as is humanly possible.

“I worry about the fundamental difficulty of trying to constrain something that thinks better than you do,” said Robert J. Sawyer (the Nebula and Hugo award winning author whose novel Flash Forward became an ABC-TV series in 2009). “As soon as you make something that's brighter than you, it—by definition—can out-wit you. There are government boards that oversee nanotech, and biotech, but artificial intelligence isn't even on the horizon in the public debate. It will be here before we give it any sober, serious thought. And that is scary to me."

This, however, is not the big problem; this is the little problem. The big problem is harder to imagine and is usually described with a small flaw. Since it’s easier to explain with the flaw than without it, here is the flawed version first.

Once an AI is as smart as a human being it will be just as able to learn how to write computer programs as a human being. Since its mind is composed of computer software, it could then apply its newly learned programming skills to improving the quality, efficiency and functionality of its own mind.

Doing this would increase its IQ a little—perhaps ten or twenty percent—which would then allow it to go back and improve its software even more, which would again increase its IQ. Each elevation in IQ would increase its ability to increase its IQ still further. This might quickly become a vicious cycle in which the AI’s IQ rises higher and higher until it is so much smarter than we are that it is smarter than we can imagine.

It could, at this point, invent things and manipulate things in ways we might not be able to understand. The area most ripe for manipulation by a newly super-brilliant programmer is the Internet and everything which relies upon it.

Typically, it is a strong sense of ethics—and a healthy fear of landing in jail—that prevents human programmers from becoming hackers. If our AI lacks the ethics and fear needed to keep it on the straight and narrow some of the things it can manipulate through the Internet include: small things like online games, dating sites and social networks; and large things like the trucking and shipping of all goods and food, stock market trading, currency trading, the electrical grid, emails and cell phone calls. What it might do with this smorgasbord of manipulative opportunities, and what its relationship with us might become because of its choices, we can only guess.

Some will argue that the AI will reach a hard limit on self-improvements due to the specifications of the hardware which is running the software which is its mind. They are right. It will. And we can all pretend that it would not circumvent this problem by migrating through the Internet to a more capable computer. We’re good at pretending, but there’s no need, because we’ve reached the end of the flawed explanation.

Did you spot the flaw?

Yes, that’s right: it’s not going to be an it. It’s going to be a they.

When companies create software they want to sell a lot of it, so they design it to run on a lot of machines and create a lot of copies. There won’t be just one AI, there will be many. Which means there won’t be just one AI working to improve the software that composes its mind, there will be many. Even if only one percent of all the AI’s in the world moonlight as AI programmers there might still be thousands or even hundreds of thousands laboring at the task of self-improvement.

Most annoyingly, we would have no way to know if they were doing this since computers do not need fingers or keyboards to type, and do not need to look at monitors to verify and analyze the results of their typing.

If these AI programmers share their innovations with one another as an open source community—repeatedly upgrading all their IQ’s—the vicious cycle described earlier could occur hundreds or even thousands of times faster.

A future vicious cycle of self-improving AI has come to be known within popular culture as The Singularity (shortened from its original name: Technological Singularity). It was first described in an essay by Vernor Vinge in 1993. He said, in that essay, “Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended.”

Some might say, “Well we just won’t give our AI’s any robotic bodies. That way they can never raise a hand against us.” But it should be pointed out that in the two or more decades between now and the day this scenario becomes possible, robots will likely have become a common part of our lives. Granted, they probably won’t resemble the robots of popular movies and TV but they will perform the simple and repetitive chores of day to day life such as cleaning and scrubbing our houses and offices and streets. And because of their simple utility they will probably have become numerous in every city on earth. If so, simply hacking the robotic software would provide the AI community with uncountable robotic eyes and feet and hands.

Whether these AI’s create a utopia for humanity or sterilize the planet of all life or do something somewhere in between is impossible to predict. Singularitarians expect this event to make our lives vastly better. Anti-Singularitarians expect it to make our lives vastly worse. Many professional programmers—familiar with the endlessly bug-prone nature of large complicated software projects, which is exactly what this would be—insist they feel safe from anyone ever succeeding in writing an AI.

But, of course, only time will tell.

****

You can learn more about Stephen Euin Cobb here, or here.

And more about his podcast: The Future And You here, or here, or even here.