This essay explores the fundamental tension between short AI timelines and current reinforcement learning approaches. The author argues that if we're truly close to human-like learners, then the massive investment in training models for specific skills through RL environments becomes pointless.

The discussion covers why current models require extensive pre-training for tasks that humans learn on-the-job, examines the economics of AI deployment, and questions whether we're scaling the right capabilities. The author predicts continual learning will be the key breakthrough but expects it to develop gradually over 5-10 years rather than as a sudden game-changing moment.

The RL Training Paradox: Billions Spent on Skills Humans Learn Naturally

Labs are building entire supply chains of RL environments to teach models web browsing, Excel, and other software skills that humans learn quickly on the job

"Either these models will soon learn on the job in a self-directed way, which will make all this freebanking pointless, or they won't, which means that AGI is not imminent"

Baron Millage noted that frontier model improvements reflect not just scale and research, but "the billions of dollars that are paid" for specialized training

Robotics Reveals the Learning Gap

"In some fundamental sense, robotics is an algorithms problem, not a hardware or a data problem" - humans can teleoperate current hardware with minimal training

The fact that robotics remains largely unsolved despite hardware advances proves we lack true human-like learners

The "automated Ilia" argument - building superhuman AI researchers through current RL methods - feels like "losing money on every sale but making it up in volume"

The Economic Reality Check: Trillions vs Current Revenue

Knowledge workers globally earn "tens of trillions of dollars a year in wages" while labs are "orders of magnitude off this figure"

True human-level AI would diffuse incredibly quickly - able to "read your entire Slack and drive within minutes" and instantly share skills between instances

A biologist's skepticism about AI automating microscope work illustrates the gap: "Human workers are valuable precisely because we don't need to build in the schleppy training loops for every single small part of their job"

"Every day, you have to do 100 things that require judgment, situational awareness, and skills and context that are learned on the job"

The Goalpost Problem and Scaling Reality

"Models keep getting more impressive at the rate that the short timelines people predict, but more useful at the rate that the long timelines people predict"

Toby Board's analysis of O-series benchmarks suggests "we need something like a million" times more compute for significant improvements

By 2030, models will likely earn "hundreds of billions of dollars in revenue a year" but won't have "automated all knowledge work"

Continual Learning: The Gradual Path Forward

The future may involve "continual learning agents who are all going out and they're doing different jobs" and bringing learnings back to a "HiveMind model"

Continual learning will progress like in-context learning - GPT-3 was titled Language Models Are Few Shot Learners but we're still improving context capabilities

"Human level on-the-job learning may take another five to ten years to iron out" rather than appearing as a sudden breakthrough

Competition remains fierce between model companies despite supposed advantages - "every month or so, the big three model companies will rotate around the podium"