Research notes · Retrospective (June 2024 interview)
From the Jul 20, 2026 daily brief
François Chollet, creator of the Keras framework, once gave the claim "large models memorize rather than reason" a testable probe: models can solve a Caesar cipher — the beginner's encryption that shifts every letter forward by n places — but mostly only when n is 3 or 5, the values most common in online tutorials; move it to 9 and they fail. If a model were genuinely synthesizing the solution on the spot, the shift size would not affect difficulty at all (Dwarkesh Podcast, Jun 2024). The tension when set beside today's lead 1: the 2024 diagnosis was "no on-the-fly synthesis of algorithms," and 2026 has produced a perfect-score formal-proof claim. In fairness, the tasks differ in kind — the probe tests instant synthesis inside the model's parameters, while the IMO system is a multi-agent assembly wrapped around a Lean toolchain — and June 2024 predates the reasoning-model era. The probe's current status has no public retest; we log it as pending.
Just an email address, unsubscribe anytime. This is the only thing we ask of you.
The current mainstay of post-training (the stage after base training where a model is taug…
In the same AMD assessment, SemiAnalysis reports firsthand engineering observations: 2.5 e…
The Jacobian conjecture, posed in 1939, is a famous problem in algebraic geometry. It says…
Our July 21 Research Notes covered this empirical study (gains from optimizing an agent pi…