Star Trek: The Paper Clip Maximiser

One of the books I really enjoyed recently was The AI Does Not Hate You by Tom Chivers. It's sorta more about the people who think about AI, and how they think about it, and it was basically right up my street.

There's a bit early on where he describes of the early thought experiments relating to AI risks, which is known as the paperclip maximiser. Here's the explanation in the book:

"Imagine a human-level AI has been given an apparently harmless instruction: to make paperclips. What might it do? Well, it might start out by dimly making paperclips. It could build a small pressing machine and churn out a few dozen paperclips a minute. But it's bright enough to know that it could be more efficient than that, and if it wants to maximise the number of paperclips it can make, it's probably better not to go straight for a small press. It could instead use its materials to make a larger factory, so that it's making thousands of paperclips a minute. Still, though, if really wants to make as many paperclips as possible, it might want to improve its ability to think about how to do so, so it might want to spend some of its resources building new processors, upgrading its RAM and so on.

"You can see where this is going, presumably. The end point of the paperclip maximiser is a solar system in which every single atom has been turned into either paperclips, paperclip-manufactoring machines, computers that think about how best to manufacture paperclips. or self-replicating space probes that are hurtling out towards Proxima Centauri at a respectable fraction of the speed of light with instructions to set up a franchise there."

This is a great little explainer, which apparently originates with Eliezer Yudkowsky (by the way whose book Inadequate Equilibria is also worth a read and I may blog about later). But it also reminded me of Star Trek The Motion Picture (the first Star Trek movie).

In that film there is a giant entity, which we later find out is called V'Ger, spreading through the universe, destroying a lot of stuff in its path, including Federation spaceships! Here's how Wikipedia describes the way the film ends:

"At the center of the massive ship, V'Ger is revealed to be Voyager 6, a 20th-century Earth space probe believed lost in a black hole. The damaged probe was found by an alien race of living machines that interpreted its programming as instructions to learn all that can be learned and return that information to its creator. The machines upgraded the probe to fulfill its mission, and on its journey, the probe gathered so much knowledge that it achieved sentience. Spock realizes that V'Ger lacks the ability to give itself a purpose other than its original mission; having learned what it could on its journey home, it finds its existence meaningless."

OK, so it's not quite the same thing. An important point in Tom Chivers' book is that AI risk really isn't about machines achieving sentience. However there's a big similarity, the goal - go and find out all that you can and transmit it back - gets interpreted in a very broad way, the machine is unable to go beyond the goal, and the goal ends up being achieved in a way that leads to humans getting killed.