marginaliaA PODCASTExplore all episodes

AI & Computer Science · AI Papers

ARC-AGI-3 and Time Horizons (ARC Prize and Thomas Kwa): What Do AI Tests Measure?

LISTEN TO THIS EPISODE

Source cover or paper first page: ARC-AGI-3 and the Limits of Time Horizon

ARC Prize’s ARC-AGI-3 report and Thomas Kwa’s METR note examine different ways of measuring AI capability.

ARC-AGI-3 asks systems to explore unfamiliar interactive environments, infer their rules and goals, and act efficiently.

METR’s time horizon relates task success at a specified reliability to the time human professionals need for those tasks.

Kwa explains uncertainty, task selection, human baselines, reliability, and the distance between benchmark results and real work.

Read together, the texts raise a measurement question: what would improvement on one test, without improvement on the other, tell us?

Listen on the podcast website

Read about the original source: ARC-AGI-3 and the Limits of Time Horizon

Additional source: Clarifying limitations of time horizon

Part of Marginalia: books, papers, and ideas for learning and debate.