AI & Computer Science · AI Papers
ARC-AGI-3 and Time Horizons (ARC Prize and Thomas Kwa): What Do AI Tests Measure?
LISTEN TO THIS EPISODE

ARC Prize’s ARC-AGI-3 report and Thomas Kwa’s METR note examine different ways of measuring AI capability.
ARC-AGI-3 asks systems to explore unfamiliar interactive environments, infer their rules and goals, and act efficiently.
METR’s time horizon relates task success at a specified reliability to the time human professionals need for those tasks.
Kwa explains uncertainty, task selection, human baselines, reliability, and the distance between benchmark results and real work.
Read together, the texts raise a measurement question: what would improvement on one test, without improvement on the other, tell us?
Read about the original source: ARC-AGI-3 and the Limits of Time Horizon
Additional source: Clarifying limitations of time horizon
Part of Marginalia: books, papers, and ideas for learning and debate.