Eric Zhang, former VP of AI at 1X Technologies and senior research scientist at Google DeepMind Robotics, rebuilt AlphaGo from scratch during his sabbatical. Zhang achieved comparable performance to the original DeepMind system using just $7,000 in compute, representing a 40x improvement in efficiency.

The conversation explores how AlphaGo works from first principles, covering Go rules, Monte Carlo Tree Search (MCTS), neural network architectures, and self-play training. Zhang explains how MCTS provides a superior learning signal compared to standard reinforcement learning by relabeling every action with improved alternatives.

The discussion extends to broader implications for AI research, including test-time compute scaling, automated research workflows, and connections to modern LLM training. Zhang demonstrates how neural networks can compress seemingly intractable search problems into efficient forward passes.

Go Rules and Game Mechanics for AI

Go involves placing black and white stones to control territory, with capture occurring when stones are surrounded on all four neighbors, creating 'dead stones' that lose oxygen

Tromp-Taylor rules provide unambiguous computer scoring, unlike human play which relies on consensus about dead stones and territory control

Game complexity reaches 361^300 possible states on 19x19 board with ~300 move games, far exceeding atoms in the universe

Monte Carlo Tree Search Architecture

MCTS uses four-step process: selection (PUCT criteria), expansion (adding leaf nodes), evaluation (neural network value estimation), and backup (propagating values upward)

PUCT formula combines exploitation (Q values) with exploration (visit count ratios): argmax[Q(s,a) + c_puct * P(a) * sqrt(n)/(1+n_a)]

Tree structure stores visit counts, mean action values, action probabilities, and children references for each node

Search creates sparse trees focusing on promising paths rather than exhaustive exploration, with most leaves having low visit counts

Neural Network Value and Policy Functions

Value network predicts win probability from board state, replacing need to play games to completion for evaluation

Policy network outputs probability distribution over legal moves, guiding search toward promising actions

ResNets outperform Transformers in low-data regimes due to local convolution bias, though Transformers excel with global feature aggregation

Architecture choice matters less than training setup; both ResNets and Transformers can achieve strong performance with proper tuning

Self-Play Training and Policy Improvement

MCTS acts as improvement operator, distilling search results into neural network to start at higher baseline for next iteration

Training objective combines policy loss (imitating MCTS distribution) and value loss (predicting game outcomes) using supervised learning

Each action gets supervision signal from MCTS search, avoiding high-variance policy gradients that only learn from trajectory endpoints

Expert initialization provides crucial warm start; random play on smaller boards (9x9) can bootstrap value functions for transfer learning

Compute Efficiency and Modern Implementation

KataGo achieved 40x compute reduction over AlphaGo Zero through architectural improvements and training optimizations

Modern GPUs enable simplified infrastructure: synchronous training replaces complex distributed asynchronous systems with replay buffers

Many KataGo tricks become unnecessary with strong initialization; best response training against existing bots provides efficient starting point

Test-time compute scaling allows trading inference compute for training compute, as demonstrated in Andy Jones' 2021 scaling laws paper

Connections to LLM Training and Broader AI

Standard RL suffers from 'sucking supervision through a straw' - learning only from trajectory endpoints with high variance

MCTS provides dense supervision by improving every action, contrasting with LLM RL that reinforces entire sequences based on final outcomes

Neural networks compress intractable search into forward passes, suggesting NP-hard problems may be more tractable than worst-case analysis implies

Chaos theory connection: while exact future states are unpredictable, macro properties (like game outcomes) can be learned and predicted reliably

Automated Research and Future Directions

LLM coding assistants excel at hyperparameter optimization and experiment execution but struggle with lateral thinking and track selection

Current models can't effectively choose next experiments or step back from unproductive research directions without human guidance

Go provides excellent testbed for automated research: quick verification loops, rich sub-problems, and clear outer loop metrics

Research taste involves knowing when bitter lesson scaling applies versus when algorithmic innovations matter in current compute regimes