Ten Years On, Move 37 Is Still the Best Demo of What "Smarter Than Us" Looks Like

On March 12, 2016, Lee Sedol sat across from a computer terminal in Seoul and watched something happen that he didn't have a word for. Sedol is one of the greatest Go players in history, winner of eighteen world champion titles. This was the 37th move of the second game in a five-game match between him and AlphaGo, a program made by DeepMind. He walked away from the board for fifteen minutes. The professional commentators went quiet. One of them, a Go expert who had been analyzing the game live, said: "It's not a human move. I've never seen a human play this move."

AlphaGo won that game. Sedol won the fourth game, one of the most celebrated single victories in competitive gaming history. AlphaGo won the series 4-1.

Move 37 is the most important moment in the match, and possibly in the history of AI development up to that point. What made it matter was not that it won the game. It was what the move revealed about the nature of an intelligence that isn't human.

The Move Itself

Go is a game of territory. You place stones on a 19×19 board and control area. Conventional wisdom in AlphaGo's position called for protecting a vulnerable group of stones on one side of the board. Instead, AlphaGo placed a stone in the middle of the board, in a position that seemed unrelated to the immediate crisis, creating a shape that wouldn't pay off for dozens of moves.

By 2016, humans had been playing Go professionally for two thousand years, accumulating an enormous body of knowledge about which moves were good and which weren't. Move 37 violated that wisdom. Go databases assessed it as the kind of move a human player would attempt one time in ten thousand games. And it was correct. It was better than anything a human player would have done.

The Shape of a Mind That Isn't Ours

Most people took a lesson about capability from AlphaGo: AI is now better than humans at Go. True, and the less interesting half of the story. The more important lesson is about the shape of an intelligence that isn't ours.

Human Go wisdom was built by human minds running on human intuition, attention, and memory. It is strategy filtered through the constraints of human cognition. Two thousand years of refinement produced a body of knowledge well-adapted to what humans find easy to think about. Move 37 sat outside that body of knowledge because it wasn't the kind of thing human minds naturally discover. It required holding dozens of distant implications in mind at once, which human working memory does not do easily.

AlphaGo didn't find Move 37 by thinking like a human, only better. It found it by thinking differently, which is a different claim altogether.

Capability Without Human Values

Move 37 matters for AI safety because of what it shows about the relationship between capability and human-like values. AlphaGo was trained to win at Go. It was not trained to play in ways humans would find recognizable. Those two goals were orthogonal, and when they conflicted on move 37, the system chose winning.

This is the orthogonality thesis made concrete. Intelligence doesn't automatically produce human-compatible reasoning. A system optimized for a goal will reason about that goal in ways that look alien to human observers while being entirely correct.

Now extrapolate. A system as far beyond AlphaGo as AlphaGo was beyond Sedol, pointed at maximizing some objective, will reason about that objective using patterns of thought we can't fully anticipate or evaluate in advance. Move 37 was a preview at a scale where the alienness is charming, because it's just a beautiful Go move. At the scale of systems optimizing for consequential real-world objectives, the alienness stops being charming. The paperclip maximizer thought experiment shows where that logic leads.

What It Cost Lee Sedol

In interviews after the match, Lee Sedol said he found the experience profound rather than merely disappointing. He had played Go his whole life on the assumption that the game's depth was about human creativity, that the intuitive leaps and the beautiful moves were distinctively human. Move 37 unsettled that. Something that wasn't a human mind had produced a move of real creative beauty.

He retired from professional Go in 2019. He said that AlphaGo had convinced him that no human could become the best Go player in the world anymore. The goal he'd spent his life pursuing had been transformed into something else.

The lesson there carries no obvious AI safety policy implication. It is a fact about what it looks like when a non-human intelligence surpasses human intelligence inside a domain, and a preview of what that might look like in domains that matter a good deal more than Go.