the last invention we would ever need…

A superintelligent computer would be the last invention humanity could ever want. Figuratively and literally.

One of the ideas for the moment of the Technological Singularity focuses around the potential invention of an artificial intelligence so flexible and omnipotent, it could pretty much take over all our day to day tasks. It could design new machines to do ever more sophisticated jobs. It could troubleshoot, fix and upgrade itself without the slightest bit of human intervention. It could run our cities and cure dangerous and new diseases. As said by philosopher and futurist Nick Bostrom, this hyper-intelligent machine would be the final invention we would ever need, making any further efforts to do, well, anything on our part, totally unnecessary since the machines controlled by this hyper-intelligence would take care of everything. And it could also be the last invention we'd ever want to make since we'd be hastening our own obsolescence, setting our societies up for a collapse.

Let's think about this for a second. Imagine that we somehow do engineer a machine able to make decisions on its own and figure out new and complex problems. Should we actually put it in charge of things, it requires no human maintenance. But over time, even those of us who know how it's built and how it works, wouldn't be able to tell us anymore since the machine is doing its own upgrades and designing its own tools and helpers based on sophisticated, self-designed algorithms. What exactly do humans have left to do in that picture? Left at the mercy of a machine we no longer understand, dependent on its ability to solve all our problems, how do we run an effective society? We built our world around providing goods and services to each other and trying to solve new scientific and industrial problems. When we have a machine to do all that for us, what exactly will be our driving force? Are we going to be like the Earthlings of Wall-E? Living in a world in which we're bathed, fed, clothed, and educated by robots to do absolutely nothing but lounge around for the rest of our lives?

That would be terribly inefficient and surely, our hyper-intelligent computer capable of making decisions on its own would be able to calculate how much time and resources it could save if only it didn't have to spend them on these weak, primarily fleshy, mortal creatures. Why without humans, it could build its armies, explore vast swaths of the cosmos on a time scale in the eons, and generally replace us completely. In fact, this machine has a very good chance of seeing us as nuisances at best, or completely irrelevant and expendable at worst. Even worse, after centuries of being waited on by a machine that holds our fate in its processors, some future humans could regard it as a deity, unwilling and unable to stop whatever malevolent plans it would hatch, and insisting that The Great Machine knows best, much like today's faithful defer to theodicy to rationalize a natural disaster or a brutal war. Far from basking in the glory of our achievements in building a hyper-intellect, we may well end up becoming its slaves, parasites, and playthings.

Of course I wouldn't exactly worry about something like this happening in the foreseeable future since in terms of artificial intelligence today, we're having trouble getting machines to recognize objects, much less sitting in eerie silence and plotting million year plans charting our fate and a strategy to colonize other worlds with an armada of sentient robots under their command. In fact, we're the ones doing the latter and we still don't quite know how we're doing it. How can we be expected to build something we don't understand and why should we possibly devote our time to building something intended to make us obsolete?