EPISODE TRANSCRIPT
Welcome to the Era of Experience (David Silver and Richard S. Sutton)
Automatically transcribed from the episode audio. Some words and names may be inaccurate.
Listen to the episodeDownload textDownload captions
So we are constantly being fed this one very specific narrative about artificial intelligence right now. Oh, absolutely. It's everywhere you look. Right. It goes something like this. AI is getting smarter simply by reading more of the internet. It consumes more books, more articles, more forum posts, and it just magically gets better. Just massive amounts of data. Exactly. But here is the question that has been keeping me up at night. And I mean, if you follow this space, maybe you have wondered about it too. What happens? when the machine has read everything.
Well, that is the ultimate bottleneck, isn't it? Yeah. When a system reaches the absolute edge of the human library, where does the new knowledge actually come from? How does it learn something that no human has ever written down? And that question is really the entire engine of today's Deep Dive. We are looking at a truly fascinating position chapter, and it's titled, Welcome to the Era of experience. It's a fantastic piece of work. It really is. It was written by David Silver and Richard S. Sutton, who are two very prominent researchers in the field. Definitely heavy hitters. Right. And they are essentially arguing that artificial intelligence is standing on the threshold of this massive paradigm shift.
Like we are supposedly transitioning away from just feeding AI static data that humans generated in the past and moving towards something they call continuous interactive experience. Though, to be clear from the jump, Sutton and Silver aren't claiming they have this totally solved, you know, boxed up and ready to sell tomorrow. This chapter is really a roadmap. It is a proposed research direction that looks at the massive unsolved hurdles we have to jump before we get to truly autonomous systems. Which is what we're going to get into today. Exactly. We are going to explore their arguments, but we are also going to look at the open problems and the real mathematical limitations of this vision. Because whether you are a developer listening to this on your commute or you are just kind of curious about why the apps on your phone keep changing how they act You really need a clear map of where the architecture is heading. Absolutely and we want to give you that map So let's start by defining the baseline Silver and Sutton call our current moment the era of human data, right? So the era of human data is characterized by large language models or LLMs achieving this sweeping, almost shocking level of generality. It really was shocking for a lot of people. It was. Yeah. And they did this by training on massive data sets generated by us. The models basically scrape the text of the Internet, and then they are fine -tuned with expert human examples and preferences just to act helpful. Which is the AI that you and I interact with every single day. I mean, it writes poetry.
It helps diagnose medical issues. It summarizes dense legal contracts. It's sub... brilliant imitator of human competence across a vast array of subjects. But the authors propose that extracting knowledge solely from our data is rapidly approaching a limit, particularly in rigorous domains like mathematics, coding, and empirical science. Because the highest quality human data, the kind that actually improves the performance of an already strong agent, is largely consumed. Let me push back on that for a second. Does this mean it is mathematically impossible for current LLMs?
to discover new ideas or generalize beyond their examples? I mean, are they just text regurgitators? No, no, I want to add some vital nuance to that because that's a common misconception. The chapter does not prove a mathematical impossibility of discovering new results from human data. OK, good to clarify. Right. It does not claim that supervised models are mere text regurgitators that are completely incapable of generalization, nor does it claim that literally all high quality human data is exhausted down to the absolute last bite. I'm really glad you brought that up.
It's not that current AI is fundamentally broken and can never combine two ideas to make a third. Exactly. And we are starting to see real world examples of this shift making headlines right now. Yeah, let's talk about alpha proof. That is a brilliant case study for this transition. Alpha proof was initially exposed to about 100 ,000 formal proofs that have been created over many years by human mathematicians. But it didn't stop there. It used a reinforcement learning algorithm to subsequently generate 100 million more proofs through continual interaction with a formal proving system. Which is the experiential part. Reinforcement learning at its core is about an agent learning to achieve a goal by a trial and error within an environment. Can you break that down a bit more? Sure. It tries a behavior, and if the environment signals a positive outcome, the system mathematically reinforces the pathways that led to that decision. In Alpha Proof's case, the environment was a formal mathematical proving system. So it's basically throwing ideas at this system to see what sticks. Basically. It tried a logical step, and the system returned a signal, verifying whether the step was logically valid or invalid. Like a massive game of trial and error. Exactly. And through millions of these interactive loops it explored mathematical possibilities far beyond the confines of the original human proofs. That is just wild. And we have seen variations of this experiential approach elsewhere, too.
I mean, DeepSeq's recent work comes to mind. Yes, but we have to be very careful with the DeepSeq examples. We have to separate DeepSeq R1 .0 from DeepSeq R1. Oh, good point. DeepSeq R1 .0 was a specific experiment relying on pure reinforcement learning without preliminary supervised fine -tuning on human data. It essentially developed problem -solving strategies autonomously from the jump. But that's not what their main model's doing. Right. Right. We cannot call all of their models pure RL. DeepSeq R1, which is their flagship model, uses multistage training that heavily incorporates human -derived data alongside reinforcement learning. So it's a mix. Yes. The overarching point Silver and Sutton are making is that the element of interactive, experiential learning, whether pure or incorporated into a multistage pipeline like R1, is what allows systems to push past the ceiling of human knowledge. Because generating a hundred million mathematical proofs over days requires a completely different relationship with time than what a standard chatbot has. Totally different. Think about the chatbots you use at home. You type a prompt, it spits out an answer, and it effectively goes back to sleep. It has no continuous existence. Right. Which brings us to the first major dimension of the era of experience. The authors call it the transition from snippets to streams. This is crucial. The current paradigm of human -centric AI operates entirely in isolated snippets. Yes. The agent is solely optimizing for the immediate outcome of directly answering your current prompt. It is a short interaction episode where little to no information carries over to the next, completely separate conversation. It reminds me of the movie Groundhog Day, but like even more extreme. That's a great way to put it. Because in Groundhog Day, Bill Murray wakes up The day resets, but he at least remembers yesterday. A standard LLM wakes up, answers your prompt, and then ceases to exist entirely. It isn't planning for next week. And humans and animals do not exist in isolated snippets.
We exist in an ongoing stream of actions and observations that spans years. Exactly. If I decide to run a marathon, I might do a grueling training run today. That makes me exhausted and sore, which is a really negative immediate outcome. You wouldn't do it if you were only optimizing for today. Right. But I do it because I know it contributes to the long -term goal of finishing the race six months from now. And optimizing for delayed consequences is the absolute hallmark of natural intelligence. The authors forecast that powerful future agents will inhabit their own continuous streams of experience over long timescales. What does that actually look like in practice? Well, they provide some compelling forecast examples. Imagine a health and wellness agent. Instead of just answering a prompt about the calories in an apple, it monitors your sleep patterns, your activity levels, and your dietary habits via wearables over several months. So it's constantly watching and learning. Yes. It provides personalized recommendations that adapt based on long -term trends to hit a specific fitness milestone. Or like a science agent. The chapter mentions an AI working to reduce carbon dioxide in the atmosphere. That's a huge one.
That requires analyzing real -world observations. developing complex atmospheric simulations, proposing real -world experiments, waiting for the results, and then iterating. That single task could take years of continuous operation. Decades even. And operating over years introduces incredibly difficult open problems in computer science. The chapter is explicitly calling for deep research into these areas. Like, what kind of problems? Consider temporal abstraction. How does an artificial neural network break a multi -year goal into actionable second -by -second computations without losing track of the overarching objective? That's mind -boggling. And then there's the problem of credit assignment. Yes, credit assignment is huge. I find this one fascinating. If you have that science agent running a carbon reduction experiment and the experiment finally succeeds on day 300, the AI has made millions of micro decisions along the way. Millions. How does the math actually look backwards through 300 days of data and decide which of those millions of choices gets the points for the success and which choices were just irrelevant? Exactly. It requires massive advancements in memory architectures and value functions. The system needs to maintain a continuous state representation and evaluate how every small action shifts the probability of achieving that distant goal. But wait, some folks listening might be wondering, current LLMs can already use memory. I mean, they have massive context windows now. You can upload a 500 -page PDF or refer to something you talked about two hours ago. That's true. So are Silver and Sutton saying current neural network architectures are fundamentally broken for this long -term stuff?
Not at all. The authors explicitly propose building on today's technological foundations. OK. Existing language models, even with frozen parameters, meaning their core training weights aren't updating in real time, can absolutely use new observations. update their context windows, and utilize external memory databases. So the architecture isn't the problem. Right. The proposal is about shifting the focus of how we deploy them. Instead of training them solely to maximize the quality of their immediate next sentence, the objective function changes. The AI needs to optimize for delayed consequences. The architecture might look similar, but what the AI considers a win changes completely.
That makes sense. And if an agent is going to achieve a long -term goal, like a multi -year science experiment, it cannot just sit in a server and talk to a user. It has to physically or digitally do something. Exactly. And the authors forecast that experiential agents will act autonomously using the exact same interfaces humans use. So they will need to operate software using graphical user interfaces, like actually clicking buttons on a screen. Yes. And autonomously call APIs across the web. They will also need to move into the physical world. monitoring environmental sensors in real time or controlling robotic arms and chemistry labs to mix compounds. I want to explore the coding aspect of this for a second because it highlights a really important distinction.
Sure. If an AI writes a Python script, opens a terminal, runs the code, sees an error message and rewrites it until it works, that feels like a solid experience. But if it runs the code and it doesn't crash, does that actually prove the AI is smart? And the code is flawless? That is a critical question. Running a program and seeing it execute without crashing is merely a software test. Just a test. Right. A test only evaluates the specific cases and criteria it covers within that specific run. Give me an example. If I write code to sort a list of numbers and I test it with a list of five numbers, it might work perfectly.
But a test doesn't mathematically prove it will work for a list of 10 billion numbers or that it won't crash under some unforeseen edge case. Right. It just proved it worked that one time with those five numbers. Contrast that with formal verification, which is what Alphaproof was doing. Exactly. Formal verification doesn't just run a test case. It mathematically proves that a statement is true relative to a specified set of definitions and a trusted checking system. It is completely exhaustive. Yes. But formal verification is mostly limited to abstract, rigorously defined domains like pure math or highly specialized software. And then there is the real world. which is entirely different from both a software test and a formal proof. Oh, the real world is messy.
Extremely messy. A real -world outcome is noisy and full of hidden variables, like an AI robotic arm mixing chemicals in a lab might execute its code flawlessly, right? Yes. But if the ambient humidity in the room unexpectedly spikes, the chemical reaction might fail anyway. That real world measurement is a grounded, messy consequence. And active exploration in that messy environment allows the AI to discover strategies that a human would never even think to write down in a textbook. It's not limited by our imagination or our clean little formulas. And experience is not restricted strictly to the physical world either. No, not at all. Navigating a complex digital system autonomously, dealing with server latency, API changes, unexpected pop -ups. That is a rich, grounded experience that forces an agent to adapt in real time. So we have established an agent taking rich actions over a long stream of time to achieve a goal.
Right. The massive challenge now is how the AI knows if it is actually succeeding. In the current paradigm, if a model writes an essay, a human user clicks a thumbs up or thumbs down icon. The human acts as the judge. And the authors argue that relying on human prejudgment fundamentally caps the system's potential. It creates a ceiling. Currently, a lot of AI is trained using reinforcement learning from human feedback, or RLHF. Right, RLHF. In this system, a human reads a proposed output, say a complex scientific experiment design, and decides if it looks promising. The reward comes from the human's prediction of success, not the actual success itself. Grounded rewards come directly from the environment. If the goal is reducing global warming, the grounded reward is the empirical measurement of carbon dioxide levels in the atmosphere. Or if the goal is discovering a stronger material, the reward is the tensile strength measured by a physics simulator. Exactly.
Or if it's an educational agent, the reward is the student's actual exam results, not just whether the lesson plan looked coherent to a human supervisor. Carbon dioxide and tensile strength are objective metrics. But the environment the AI is navigating includes humans, and human goals are often inherently subjective. That's where it gets complicated. Right. If you want a medical agent to help reduce your migraine pain or a culinary agent to help you bake a tasty cake, you cannot attach a digital sensor to tasty or headache. No, you can't. So we are back to human feedback. The user has to report their pain level or rate the taste of the cake. Yes, but there is a critical boundary we must maintain when discussing this chapter.
OK, let's hear it. Reported pain, taste and user satisfaction are subjective outcomes. They do not magically become objective truths just because they are logged in a database after the fact. So a human might report feeling less pain because of a placebo effect. Or they might say the cake is tasty just to be polite to the AI. I mean, people do that. They say thank you to chat GPT all the time. They really do. The agent still has to interpret those subjective signals and causally evaluate the outcomes. It has to figure out if its specific recipe caused the good taste or if the user was just incredibly hungry. So it is a grounded signal from the environment, but it is deeply, deeply noisy. Extremely noisy.
How do we actually build a reward system that handles both cold, hard metrics like heart rate and subjective metrics like a user complaining they feel tired? Well, the authors propose a sketch for a user -guided reward scheme, specifically a bi -level optimization process. Okay, let's break down the mechanics of how this would theoretically work. Sure. Imagine a dedicated neural network whose entire job is just to define the reward for the main AI agent. A separate network just for rewards. Right. You, the user, give this reward network a broad, top -level goal like, improve my overall fitness. Okay. At the low level, this reward network ingests all the grounded signals available to it. Your smartwatch data, showing resting heart rate, your daily step count, your logged sleep duration. All the hard data.
Yes. It mathematically combines these signals to calculate a numerical reward to give the fitness agent. And the fitness agent's just trying to maximize whatever score that reward network is calculating. Exactly. So the fitness agent pushes you to run 10 miles a day. Your heart rate improves. Your steps are through the roof. So the low level signals look fantastic. The agent thinks it is winning. But I am exhausted. My knees are throbbing and I absolutely hate the routine. Right. So you provide top level feedback to the system. You say, I am too tired. My joints hurt. And what does it do with that? That subjective human complaint? does not just tell the fitness agent to stop running. In a bi -level optimization scheme, the complaint goes to the reward network.
It fine -tunes the parameters of the reward function itself. The system mathematically adjusts its weights, learning that when calculating fitness, it must heavily penalize user complaints of joint pain, even if the step count is optimal. So it adapts the way it combines signals based on your ongoing guidance. Yes. It's an elegant way to translate vague human desires into the rigorous math the AI requires. But we must heavily emphasize the uncertainty here, right? This is a conceptual sketch. The chapter does not provide a complete working training algorithm for bi -level optimization.
Absolutely. There is no mathematical guarantee that an AI will find an assured gradient for every single human complaint. There is no proof that this feedback mechanism will flawlessly prevent the AI from causing harm while it is in the process of learning. It is an open, incredibly complex research direction. OK, so we have an agent taking rich actions over a long timeline, seeking real -world consequences and being guided by an adapting reward system. Yes. How does it actually think? about its next move. Like, when it's planning a six -month atmospheric science experiment, does it just talk to itself in English the way current LLMs do when you ask them to think step -by -step? Probably not in English.
Really? Conceptually, a current LLM acts as a universal computer because it can append words or symbols into its own context window. Like writing on a digital scratchpad? Effectively, yes. Giving itself a scratchpad to execute arbitrary algorithms before outputting a final result. In the era of human data, we explicitly designed and fine -tuned these models to imitate human thought processes. We literally reward them when their internal thinking traces match how a human expert would write out the solution. But the authors argue that human language is highly unlikely to be the mathematically optimal method of computation for an artificial neural network. Because language is clunky. It's ambiguous, it's slow, and it carries immense cultural baggage. Right. A truly self -learning experiential system could discover vastly superior ways to process information.
What would that look like, if not English? Well, it might utilize symbolic logic, manipulating abstract symbols according to strict mathematical rules. Or it might rely on continuous, differentiable computations, essentially manipulating high -dimensional vectors and matrices rather than discrete English words. Or it could use distributed computations across entirely different modalities. And these aren't mutually exclusive camps, are they? Not at all. An agent could use an internal language model to simulate a conversation with you, while using continuous math to calculate the trajectory of a robotic arm it's controlling. So by learning from experience, it can discover how to compute most efficiently for the specific task, rather than just copying our slow way of reasoning.
Yes. And to test hypotheses effectively, the AI needs to be able to predict what might happen before it takes an action. Which requires an internal protective engine. The authors refer to this as a world model. Now, let's avoid the cliché of calling a world model a mental chessboard. It's not that simple. Right. Think of a world model more like a highly complex weather simulation running on a supercomputer. Meteorologists don't just guess if it will rain. They input current data into a physics engine that simulates the atmosphere to predict a storm before you ever step outside. Exactly. A world model is the AI's internal physics engine for reality. So if a health agent is considering recommending you start a rigorous weightlifting program, its internal world model simulates how your resting heart rate, sleep patterns, and caloric needs will shift over the next month as a consequence of that specific action. It simulates different futures, evaluates them against the goal, and selects the path with the highest probability of success. And as the agent continuously interacts with the real world through its stream of experience, it compares its internal predictions to the actual data it receives from your smartwatch. Right.
When the prediction is wrong, it updates the parameters of its world model to correct the error, basically refining its internal physics engine over time. But we must apply a vital caveat here. Yes, very important. A learned predictor, a world model, does not necessarily recover the true fundamental causal structure of the universe. It's just a mathematical approximation. Exactly. Its predictions can simply be wrong. If its world model is flawed, its plans will be flawed, regardless of how elegantly it performs its internal reasoning.
It is a powerful tool for planning, but it is not magic foresight. No. Taking a step back, this vision laid out by Silver and Sutton is a massive shift. Agents living in continuous streams, taking rich actions, chasing grounded consequences, building complex world models. It's huge. But why are they proposing this transition right now? I mean, computer scientists have been running reinforcement learning experiments for decades. This isn't new. No, it's not. The authors addressed this context directly by framing AI history through a sketch chronology of three eras. OK, let's walk through this. The first is what they broadly frame as the era of simulation.
This covers early and highly successful reinforcement learning systems, mastering Atari games, backgammon, and eventually the feats of AlphaZero and chess and Go. These agents mastered incredibly complex tasks, but they largely struggled to leap from the confines of a simulation to open -ended reality. But we shouldn't oversimplify those early simulations either. They weren't just basic deterministic grids. Right. Simulations can be incredibly complex. They can feature stochastic noise, hidden variables, partial observability, and sparse rewards. Take Go, for example. An agent might play for hours, making hundreds of moves, and only receive a single reward signal, a win or a loss, at the very end of the game. That is a highly sparse reward. So, the challenge wasn't that simulations were easy. It was that the real world presents open -ended problems with a plurality of ill -defined, constantly shifting rewards. Exactly.
That led to the second era. The era of human data. The LLM boom. Which we're in now. Right. Massive text corpuses offered an appealing shortcut to the generality problem. Instead of trying to build a bespoke, complex simulation for every single task in the world, researchers trained massive neural networks on human language. Because language inherently contains examples of a huge diversity of tasks. The methodology of experiential RL was largely overshadowed in mainstream focus by this human -centric approach. Though RL research certainly did not disappear or go dormant during this time. No, but the dominant industry focus and compute resources overwhelmingly shifted to LLMs. Which brings us to the proposed third era, the era of experience. The synthesis. Yes.
The concept is to combine the incredible general -purpose autonomous action and tool use capabilities of modern LLMs with the profound self -discovery power of advanced reinforcement learning. We finally have agents capable of interacting with complex real -world action spaces like browsing the web autonomously or writing and executing code. And researchers are now figuring out how to apply experiential learning so they can learn from the consequences of those actions. But giving an AI long -term autonomy, letting it build its own world models, setting it loose to discover its own knowledge, that brings massive consequences.
Oh, without a doubt. If we are taking the training wheels of human supervision off, we have to look at the impact. And the authors acknowledge this is a double -edged sword. The most transformative benefit is likely the acceleration of scientific discovery. The development of novel drugs. greener technologies, advanced materials, the potential upside for human progress is just staggering. But on the other side of the sword, the first societal risk, the author's note is potential job displacement. Because as AI agents begin exhibiting long term problem solving and innovation, skills previously thought to be exclusively human, the economic impacts could be severe. Though we must caveat that while job displacement is a potential consequence, Economic adaptation is complex. It is not a proven inevitability.
Fair point. But the deeper concern involves safety and alignment. Yes. When you have agents acting autonomously over long periods to achieve complex goals, there are fewer opportunities for humans to intervene, monitor, and mediate their actions. And furthermore, because they are learning from experience and potentially developing non -human ways of reasoning, their internal logic might become entirely uninterpretable to us. Which is the classic alignment problem. How do we ensure this highly capable, potentially uninterpretable agent actually shares our values? It's terrifying.
But the chapter presents an interesting argument for how experiential learning might actually offer some safety benefits over current static models. More importantly, the authors suggest it could recognize when its behavior is triggering human concern and adaptively modify its behavior through that bi -level optimization scheme we discussed earlier. The hypothesis is that a misaligned reward function could be incrementally corrected over time. If the AI starts prioritizing a goal in a way that consumes vital resources, humans would express severe concern. And the top -level feedback would shift. The reward network would heavily penalize the behavior causing the distress, and the AI would change course. But we have to be incredibly careful here. This is a hypothesis, not a mathematical proof. Let's underline that.
Experiential learning does not magically solve the alignment problem. No. Experiential agents do not detect subtle societal changes immediately. They do not always correct their objectives flawlessly before a catastrophe happens. And they certainly do not eliminate all human bias or potential for misalignment. There is no guarantee of perfect safety. We have covered an immense amount of ground today. We started with the premise that AI is hitting a wall by only reading what humans have already written. And Silver and Sutton propose a radical roadmap. Moving from static data to agents that live in continuous streams of time, taking rich, grounded actions in digital and physical worlds. They are guided by rewards that measure real -world consequences rather than human prejudgment, and they plan their actions using internal world models that might reason in ways completely alien to human language. The reason we spent this time unpacking it is that this maps out the incredible unsolved hurdles standing between us and the next decade of technology.
Thank you for joining us on this deep dive into the era of experience. Keep questioning the headlines you read. Look beyond the hype of new models and start watching for these experiential shifts in the technologies you use every day. We will see you next time.