← all weeks
MLn CLUB · WEEK 3

Tiny Recursive Models

Reading

About this week

This paper introduces Tiny Recursive Models (TRM), which drastically simplifies previous hierarchical reasoning approaches while achieving better performance on puzzle tasks like Sudoku and ARC-AGI. Using just a single 2-layer network with 7M parameters, TRM outperforms much larger language models by recursively refining both its current answer and internal reasoning state.

The work challenges conventional scaling wisdom by showing that smaller networks with deeper recursion can outperform larger models, offering a parameter-efficient approach to complex reasoning tasks.

Discussion at 20:00, (optional) quiet reading from 19:00.

Handwritten notes — the Tiny Recursive Model loop: nested for-loops over iterations and a fast/slow inner net refining answer z from input x
The recursive refinement loop — input / fast / slow, refining the answer z.
Handwritten notes — a from-scratch tour of MLPs, CNNs with kernels, RNNs, and autoregressive token generation
From MLPs and CNNs to RNNs and autoregressive generation.
Handwritten notes — residual connections f(x)+x, f(x)=Ax, a ResNet note, and hierarchical stacking
Residual connections and hierarchical reasoning.
Handwritten notes — formal languages: an alphabet Sigma, a language L as a subset of Sigma*, Thue-style rewriting, and code vs data
Formal languages — alphabets, L ⊆ Σ*, and code-as-data.