David Silver and Lasse Espeholt of Ineffable Intelligence: A Superlearner for the Age of Experience

Introduction

David Silver: Collectively, we started with this kind of grey mush in our heads. We somehow started out knowing nothing at all, and somehow discovered everything that humans know today. It was this process of collective discovery — of languages and technology and science and mathematics and all the rest of it. Everything we learned, we discovered for ourselves. That's an incredible thing that no one has cracked yet in AI. And so to me, that is the essence of what it will take to get to superintelligence: a system that can just keep learning and discovering knowledge forever.

Main conversation

Sonya Huang: I'm thrilled to have Dave Silver and Lasse Espeholt with us, two of the co-founders of Ineffable Intelligence. Great name, by the way. Fun fact, Dave: you were the first person I called — emailed — to come on Training Data when we started the show, a hundred-ish episodes ago, and it feels like a full-circle moment to finally have you on the show. So thank you for joining us.

David Silver: Thank you.

Sonya Huang: Dave, let's start with you. Your 2009 PhD was on reinforcement learning applied to the devilishly difficult game of Go. You demonstrated for the world, for the first time, that superhuman machine intelligence — in one domain — was possible. And I think you were really the figure that inspired the entire current generation of reinforcement learning and deep learning researchers. So thank you for your contributions to where the field is. I'm curious, thinking back to that point in time: what did you learn? What did you have conviction on? What has remained the same, and what has changed as AI has changed over the last five years or so?

David Silver: Thank you, and it's a pleasure to be here. I'm glad I finally made it after these hundred episodes.

I suppose I always had this conviction, starting my PhD and going into the game of Go, that the right approach to solving AI was through learning from experience — through systems that discovered things for themselves, that were learning through reinforcement learning to get better and better and better. And that that would be the thing which would lead to the emergence of intelligence and ultimately superintelligence. We were able to show that in the game of Go, but it's a conviction that has held true throughout my career. And here we are now, applying that conviction at Ineffable, trying to show that indeed superintelligence can emerge all the way through systems that learn from experience.

In terms of things which are different: although we are not pursuing the human-data path that the LLM world has pursued, I do think there are lots of interesting lessons to learn from it. In particular, it was definitely a surprise to me how a system that was trained on a massive amount of data could then be very quickly fine-tuned on a very small amount of data to completely change its behaviour. That changed my mindset. It suggests that if you get things to the right scale, a lot of things which seem incredibly hard, or which might be very slow — once you've experienced enough data, the things that were slow can become very fast. That's a lesson that wasn't obvious to me in advance, until it happened.

Sonya Huang: Okay, we're going to come back to that later. You could have run this program inside DeepMind, where you probably had more access to compute than just about anybody. You decided the right thing to do was to start a new lab. Why did you do that?

David Silver: First of all, I had a fantastic time at DeepMind. I loved my years there, and I have a huge amount of respect for the people there and still get on with them very well.

But at the end of the day, I have a really strong belief about how we can go beyond where we are today with AI, all the way to superintelligence. And the path that I believe in doesn't actually go through LLM-based products as an intermediate point. When I really thought about that — and thought about it really hard — I realized that I felt it so passionately that this deserved to be more than just the corner of an organization that's focused on LLM-based products. It deserved to be represented by an elite AI lab that gives this approach its all and represents it one hundred percent. Once I realized that, it felt very natural to create this company.

Another way to look at it is that the world already has tens of thousands, if not hundreds of thousands, of really talented researchers all pursuing exactly the same approach — really just doubling down on LLMs with pre-training, post-training, coding, that kind of LLM path. Am I going to be the 100,001st person pursuing that approach? Is that the most effective use of my time and my energy? In the end, I realized that there's this other lamppost that, right now, no one is looking under. And if I actually put my effort there, it has a much higher chance of really having a massive impact on the field. Because if this works, it's going to change the whole of AI. So that realization also meant that this was the best way to have impact on the field — through this approach.

Sonya Huang: Plus, you had a company name that couldn't go to waste.

David Silver: That's right.

Sonya Huang: Do you want to say why is it called Ineffable?

David Silver: The name Ineffable came from this idea that knowledge could be so incredible — the things which superintelligence will one day discover, this knowledge, it's going to be so unbelievable, so profoundly amazing, perhaps so profound that we can't even express it in human language. So it's really in line with this other path that doesn't go through human-language-based approaches, but instead builds systems that discover things for themselves and has the possibility to discover knowledge that's unbelievably profound and ultimately ineffable.

I should say there was also a secret second reason. It's this idea that it really has to work. If you really just double down on learning from experience, it's an approach that just has to work. And so you could say that it can't possibly f up. So it's in-eff-able. There you go — there's the secret reason as well.

Sonya Huang: Amazing. Best lab name. Lasse, what about you? You were at Google, in a different group than Dave — in the deep learning group at DeepMind. And you spent years scaling up reinforcement learning on GPUs. What made you decide to leave DeepMind, and what drew you to co-found Ineffable?

Lasse Espeholt: I joined DeepMind in January 2015, and I started working with Dave around 2016, when I was doing this work on scaling up reinforcement learning. As you said, I was part of the deep learning team and he was leading a reinforcement learning team, but we were working on an internal project together.

I actually left DeepMind twice. I left in summer 2018 and went to Google Brain. Then eventually Google Brain and DeepMind merged, and I was back again. But then I left DeepMind again, about a year before Dave left. The first time I heard about Ineffable was actually through one of our eventual investors reaching out to me, asking what I thought about Dave. My first reaction was, "Damn, I missed out on an opportunity. I should have reached out to him." I recall Dave as, more often than not, being one of the smartest people in the room. Shortly after that sync with the investor, they actually reached out to me. So I was like, great. I'd really love to work with Dave. I knew Wojciech and Chris from back in the day, and the whole team was great. I also thought it was amazing to have an opportunity like this in Europe.

David Silver: It should be said as well that we worked together on some of the early approaches to combining reinforcement learning with deep learning and trying to scale things up. Lasse — who, by the way, is very modest — actually worked on some of the original approaches that were the first attempts to scale up reinforcement learning and show that it could be done at large scale: ways to parallelize reinforcement learning, and ways to build out architectures that were scalable. If you look today at the way the field's gone, all of the ideas that people use today hail back to that original work that Lasse did back in the day. That's where all the roots of scaled-up RL come from. He was the first to figure it out.

Sonya Huang: Wonderful. We should have you two introduce each other, since you're both too humble to brag about yourselves.

Okay, let's talk about Ineffable and what you're building. You have a beautiful mission statement as a company: you want to build a superlearner and make first contact with superintelligence. Can you define both of those things for us? What is a superlearner, and what does making first contact look like?

David Silver: To me, a superlearner is something that has incredible capacity, both in the depth of what it can learn and in its breadth.

In the depth dimension, it should be something that can start from not knowing anything at all and go all the way through to discovering things that go way beyond what humans know. An analogy would be the way that humans, collectively, started with this kind of grey mush in our heads. We somehow started out knowing nothing at all, and somehow discovered everything that humans know today. It was this process of collective discovery — of languages and technology and science and mathematics and all the rest of it. Everything we learned, we discovered for ourselves. It wasn't that someone came down and gave us a blueprint of how this should work. And it wasn't that someone came down and gave us a trillion-element dataset of, "Oh, this is what these things should look like." We actually figured it out for ourselves. That's an incredible thing that no one has cracked yet in AI. And so to me, that is the essence of what it will take to get to superintelligence: a system that can just keep learning and discovering knowledge forever.

The second element is breadth. We want something which can do this incredible discovery, all the way up to incredible things, and it should be possible to drop it into any stream of experience. Whether it's a simulated world or a digital world or a physical world — whatever it's dropped into, it should be something that can just learn and learn and learn.

So that's the dream. If we crack this, then to me it gives rise to a superintelligence that will be like encountering magic. It will be something so profoundly amazing that it will be able to do wonderful things.

Sonya Huang: What will it feel like when we first encounter it? Or is it one of these things where you can't predict ahead of time?

David Silver: I think it will be amazing. We should have an experience of wonderment. The only thing I can say from my own experience is what happened when we got to this point in — admittedly — a much narrower field. When we got to superhuman play in the game of Go, it brought wonderment to the field. The Go players who saw AlphaGo playing were amazed. It was a joyful experience for them to witness that kind of Go play. The same happened when we got to superhuman play in chess — the chess players were so excited. Just imagine going far beyond that, to things which are discovering new fields of science, new types of technology that we've never even dreamed of. It'll be incredible.

Sonya Huang: So the current paradigm — large language models with reinforcement learning — seems to be at least knocking out Millennium Problems and making some sorts of scientific discoveries. Does that still count as a superlearner? What's the difference between something that can make novel scientific discoveries and a superlearner in your sense of the term?

David Silver: I think LLMs are doing a lot of great things, and I have a lot of respect for what's being achieved. What I mean by superlearner is, first of all, something which discovers completely autonomously. What we've seen so far, at least with the approaches that have been taken, is that there's a lot of human-in-the-loop effort required to make these breakthroughs. It's not fully autonomous yet.

It's also something which should be applicable to real-world, embodied domains, where the agent can learn from its physical experience. We're a long, long, long way off that. And it's not at all clear that LLMs and language-based approaches are even the right vehicle to achieve that kind of knowledge. Imagine you want something which is learning how plasma should be controlled, or understanding how a mobile phone signaling tower should operate, or how echolocation should occur, or how robotic arms should move. Thinking about these things in terms of language might not even be the right paradigm. It might just be that all of this human data that's been used in LLMs is weighing things down. So, a lot of respect for the progress that's been made, but I don't think we're anywhere close to a true superlearner — something which can autonomously discover everything for itself.

Sonya Huang: What does building this look like in practice for you? We backed Ioannis [Antonoglou], who was on your original AlphaGo team, and he told us that one of the things that's distinct about you is that you can have these incredibly ambitious visions, but you also ground the team in very practical research milestones. I'd love to understand: big, big, big vision — what does this look like in the day to day? What are you implementing? How do you spend your days right now?

David Silver: Day to day, there's a lot of hard technical work, led by people like Lasse, to build the whole stack. That means we've got a state-of-the-art reinforcement learning stack that allows us to run things at massive scale, inhale data in the optimal way, and have systems that keep learning and learning and learning.

Then on top of that — we've spoken about the superlearner — the other aspect we're building is environments which are capable of superlearners discovering everything. Simulated environments in which a superlearner can discover everything that we mean by intelligence. We call those crucibles. They're environments in which it's possible for agents to make incredible discoveries, in which it's possible to express and learn everything that we think of as intelligence. We've been putting a certain amount of effort into creating those environments.

So it's a combination of building out the superlearner — doing the research to make superlearners incredible and beat the state of the art of what's possible with reinforcement learning, both in terms of the engineering and the research — and creating these crucibles and placing the superlearners into them to see the emergence of incredible behaviors.

In terms of the grounding you asked about, we ground it in milestones. The milestones are analogous to what happened on the human journey, where we discovered incredible things. Those become natural milestones for our agents: to rediscover the great moments that humans went through in our discovery of knowledge.

Sonya Huang: What do you think the agents will rediscover first? When is that going to be your first moment of, "Oh, this is working"?

David Silver: I imagine we're going to be surprised. One thing which is different about our approach to reinforcement learning, compared to a lot of the rest of the field, is that we are not breadcrumb-trailing the particular objective to be learned. A lot of the way people do RL with LLMs at the moment is to say, "The agent should be learning about this problem, and then this problem, and this problem." After following that trail, we're pulling the agent by its nose towards a particular desired outcome. We're not taking that approach. We're giving the agent some simple objectives and asking it to discover things that will surprise us and amaze us. The agent is in principle capable of discovering all kinds of things, and we don't necessarily know in advance what those discoveries will be. It's just going to be exciting to witness that journey. So I hope to be surprised, I think, is the answer to your question.

Sonya Huang: Interesting. So what you're saying is, instead of explicitly giving the agent the goal to go rediscover language, or build a calculator, or build a computer, you're going to give it other, more open-ended goals. And the agent may discover that the best way to accomplish that goal may be to build some of the tools that we as humans have already built. Is that one way of putting it?

David Silver: That's one way of putting it. Another way to say it is that people often think, in RL, "Oh my God, how do I come up with a reward function that perfectly specifies this really complicated thing that I want to discover? How can I specify exactly what I mean by science? How can I specify exactly what I mean by interesting maths?"

But there's an alternative. The alternative is to give the agent something really straightforward to optimize, but in a really complex environment where there's a lot to learn about. It turns out that if you look at the way the world has worked in nature, the pursuit and optimization of very simple, easy-to-describe quantities can often give rise to incredibly hard-to-describe complexity and emergence.

Examples of that would be evolution, which just optimizes survival of the fittest, some kind of fitness signal — and the pursuit of fitness gave rise to all of the complexity that we see in life. In the human journey, the pursuit of relatively simple goals that we as humans have gave rise to all of the complex behaviors that humans ultimately acquired, and all of the knowledge that we eventually ended up with. The emergent complexity is much greater than the starting point that was given. It wasn't like evolution said, "Look, hey, humans, here's this idea of science, and you need to get to the most complex science that's out there." It was really something much simpler.

A way to describe this might be: perhaps humans, in the pursuit of our goals, started coordinating a bit more. In the process of coordinating, we discovered that it was helpful to start communicating. In the process of communicating, we started trading. In the process of trading, we discovered that basic arithmetic was useful. Then we discovered that tools were useful, and that some basic science was required to understand how the tools would work. That gave rise to more complex technology. Understanding the technology required more complex maths, and so forth and so forth. All in the pursuit of relatively simple goals, this incredible complexity emerged. If we can get that process right, things are going to take off and lead to unbelievable things.

Sonya Huang: It's fascinating. Are the rewards for your agents similar to human biology and evolution, then — like, try to make it to the next generation? What does a reward function look like for you all?

David Silver: We've been experimenting with quite a number of different rewards, so it's not like we have one answer to that question. But one thing we're not doing is crafting a reward that exactly describes what we mean by science, or exactly what we mean by technology, or exactly what we mean by language. We expect our agents to discover things. And if we can't crack that problem as AI scientists, we're never going to have agents that discover whole new fields of knowledge that we don't even know about. There are vast fields out there which we don't even know have the potential to exist at this point in time. Imagine you're back in the days of Newtonian physics, and no one even knew that quantum physics was a thing, or that relativity was a thing. We should be in that process of open-minded discovery, where our agents are free to figure things out for themselves. I think that's going to lead to much more exciting outcomes.

Sonya Huang: What do these crucibles look like in practice? You're contrasting it to today, when people say "environments" and oftentimes it's a set of tasks, and you know what the agent should have done on the task. It sounds like your crucibles are very, very different. What do they look like in practice?

David Silver: We are building crucibles which — without giving too many details away — have the capacity for a lot of emergent complexity. The key thing about them is that it's not like we've handcrafted a whole load of puzzles for the agents to work with. There's just a huge amount of emergent complexity there.

Some of that emergent complexity can come from different places. It can come from physics engines, which give rise to complexity once you've got millions of interacting objects. You can have millions of interacting agents, and the world becomes much more complex because it now includes the complexity of all of those other agents within the world. So there are ways to build worlds that are vastly more complex than the standard simulators like Atari-57, or whatever it is that people have been working with in the past, and which are certainly more complex than the standard benchmark tasks that people are using.

If you get that right, you have a world in which there's the possibility for discovery by the agent. There's the possibility for the agents to discover everything from technology to science to mathematics to languages to everything else — just the same way humans did, all the way through from more basic things like motor control or coordination. The whole spectrum is possible.

Lasse Espeholt: If you look at a game like Go, it's very simple to represent — it's just a bitmap. The simulation is extremely simple, but you still see this emergent behavior of something complex. That's super interesting to us. Of course, Go is way simpler than what we can imagine doing, but at the same time, going all the way to one hundred percent realistic physics would be very computationally expensive. So we want to create a range of crucibles, where we have simple ones that can execute quickly and that we can iterate on, but where we can still see very complex behavior.

Sonya Huang: Interesting. It's almost like what you choose to represent is tricky, right? For example, I can imagine a lot of economics arises from game theory, and most of that is completely unrelated to physics. So a crucible that represents physics is probably not that helpful there. But if you have information theory represented — which agents know what — then you probably do end up rediscovering quite a bit of economics. Is that a fair way to put it? The primitives you choose to capture in the environment give rise to the type of learning that emerges.

Lasse Espeholt: Yeah. The end goal is to have something that is grounded in the real world. But these are stepping stones.

David Silver: I want to come back to the Go example, because I really like it. You'll be surprised to hear that I find Go to be a good example. But what I find fascinating — and I think this is what Lasse is talking about — is that Go is just a 19-by-19 grid where you're trying to maximize territory. If this were a single-agent game, it would be the most boring game you could imagine. You'd just place stones on a grid until you filled up the whole grid, and there'd be no strategy whatsoever.

The moment you put a second agent into the environment, you get this emergent complexity. Suddenly one agent has to reason about all the possible behaviors of the other agent, and that massively increases the complexity of the environment. To the best of our knowledge — certainly with all the compute we sank into it when we were working on it — there was just this endless learning curve, where without end you could continue to discover new things in this game.

None of this is to say that Go is the end point. Of course, it's still just a board game, and quite narrow in a certain sense. But it shows that very simple environment dynamics can lead to an infinite onion of layers of complexity that never stops — infinite to the capacity of what computers can do, of course.

Sonya Huang: Do you think the agents will end up rediscovering language? And will that language look like ours, or will it look different?

David Silver: My personal belief is that anyone who thinks human language is the end point, or the best way to reason — that perspective is a little naive. It must be the case that there are far more sophisticated approaches to reasoning, maybe using parallelizable, rich, complex approaches — neural reasoning, or whatever it is. These approaches will ultimately be far more expressive and far more capable than approaches based on human language.

I would also say that starting from human language as the approach for reasoning is somewhat problematic, because it pulls in all of our biases about how we think agents should think. A nice way to think about this: imagine we were to step back 500 years, or a thousand years, or 2,000 years, and were only able to use the language that humans used at that time. Maybe our agents would believe that the world was flat, or that diseases were caused by the four humors. There would be all kinds of strange beliefs embedded in there, and they would be very hard to remove, because they're so fundamental to the language and the processes of thinking that humans have used. So I do think that going beyond this is a really important goal.

Sonya Huang: I'd love to talk more about this concept of removing human priors. I think your experience with AlphaGo, and then AlphaGo Zero and AlphaZero, is maybe the best example anyone can point to of what happens when you remove human priors. Take us back to AlphaGo and AlphaZero. Was it surprising to you that removing human priors would result in even smarter intelligence?

David Silver: I wasn't sure, honestly. I always liked to do research where you ask questions which are on a knife edge — where it's 50/50. It's not very interesting to do research where you already know the answer and you're 95 percent certain the thing is going to work. It's boring, honestly, to do those things. Good research is about asking questions where you're really not sure what the answer is in advance. In this case, I was about 50/50. I thought maybe there was a good chance it might work, but I wasn't confident, and it was very exciting to see it.

You asked about human priors. We are making a bet at Ineffable, and the bet is that we can go further by giving up on the priors that everyone's relying on today — the human data, the whole internet of knowledge that everyone else is building their systems upon. That's an important choice, and we should probably talk about why we're doing that. Of course, it's a short-term sacrifice. We give up a certain amount of performance in order to access something which is potentially far greater.

It comes back to what it means to build a superlearner. To me, building a superlearner is about building something that has the potential for full depth and breadth. In the depth direction, we need to get to something which is capable of discovering things that go so far beyond human knowledge that it goes far beyond the prior. The more we lean on human data, the less able we are to actually discover the algorithms that are really going to take us all the way to something that can, for example, discover a whole new field of knowledge. And in the breadth direction, we need algorithms that can be placed down and discover things in environments for which language and human data are just not a good fit. There are many, many areas — different kinds of sensors, for example — where human language data, or human data in general, is not a good fit. By letting go of the human data, we put ourselves in a position where we can actually discover something which can go far, far beyond it.

If we come back to the AlphaGo example: what we found there was that the moment we let go of the human data — and we'd spent about two years building AlphaGo, somewhat similar to the LLM approach, where we built every level of this. We put in a lot of human data, and then we were human-in-the-loop improving it: we'd take the previous generation and choose how we were going to improve it to give the next generation. We were right in the loop of this process. It took us two years to get to AlphaGo and beat Lee Sedol. When we got AlphaZero right — we gave up on the human data and finally focused on what it would be like to build an algorithm that could discover for itself — once we got that algorithm right, it took just a few hours. A few hours to outperform the two years of work which we'd done with the human in the loop.

In the game of chess, it took four hours to beat the world champion — the best AI that had ever been created. In the game of Go, it took, I think, a couple of days or something, but it was incredible progress. So you have to let go of the human data to allow yourself to discover these things that can then shoot past the competition, and discover the things that just keep learning and learning and learning forever.

It has to be an active choice. If you don't make that active choice — what I see in a lot of the LLM companies today is that they're basically building semi-autonomous, human-in-the-loop learning algorithms, with thousands of people in the loop of the learning process. It works, and a lot of respect for how far it gets. But until we step back from that and really discover the process that's going to learn for itself, I don't believe we'll get all the way to something which is truly a superlearner and unleashes the potential that we've talked about so far.

Sonya Huang: That's very profound. As far as I know, there are humans at Ineffable, and some of you are creating environments, or crucibles. What does it look like to actually make sure that human priors aren't leaking into what you're doing?

Lasse Espeholt: When we design environments and reward functions and so on, obviously there's human bias in there. But, as Dave was mentioning earlier, we don't want to bias it towards building a machine based on the von Neumann architecture or something like that. If you're stating some problem where a machine would be useful, it would be very interesting to see what type of machine the agent comes up with — what kind of computer-like machine it comes up with.

David Silver: I totally agree with this. I would also say that we're not religious about it. It's not a religion that you can't have any human prior anywhere. It's more a pragmatic position that until we give up on relying on the trillion-item datasets of human knowledge, we're not going to make the progress that the field deserves to see. It's actually a practical position. In practice, of course, there's going to be a certain amount of prior. As Lasse says, there'll be priors which apply to many different worlds, as opposed to very strong priors that exactly encode all the knowledge we know about this one particular problem or this one particular world. Weak, environment-agnostic priors seem perfectly reasonable. But these super-strong priors, with trillion-item datasets — we're trying to avoid that, because we think pragmatically it's going to lead to better outcomes.

Sonya Huang: Absolutely. You mentioned that philosophically you're drawn to research that's on the knife's edge, 50/50. I'm curious what your confidence levels are for Ineffable at the broadest level — will a superlearner emerge? And then at the more pragmatic level, in terms of the things you're trying to accomplish right now, are you 50/50 on those as well?

David Silver: First of all, I would say I strongly believe that the approach will be feasible at some point. In fact, it's the only thing for which we have an example. The example in our brains is the only existence proof of something which has discovered an incredible intelligence that's very general, works in an embodied world, and discovers how to do all kinds of different things. We have this incredible example of the human brain that we can look at.

Then, do I think we can achieve it? I think, like many great moments in science, it's about timing. Whether this succeeds or fails is about timing. When you create a startup, you want to be at the right time. You don't want to be too early or too late. If you're too late, everyone's done it; it's not interesting; a thousand other people have already done it. And if you're too early, you might be banging on the door but not have the right — maybe the field isn't advanced enough, or there isn't enough compute to be able to succeed yet. So to me, it's all about timing.

For example, with computer Go, I made a prediction, I think ten years before computer Go was solved, and I got it right to the year. I got it right for StarCraft, where we also had a bet and made a prediction. Same for AlphaProof, when we were predicting when the IMO was going to fall.

Sonya Huang: So you're a super forecaster of sorts.

David Silver: So far, I've done very well at this. And I do have a very strong sense that this is the right time for this to fall — that superlearners are possible. There's been enormous progress in the level of technology: the ability to accelerate research using LLM coding methods, or research methods like we've produced in the past; the understanding of scaling laws; and the sophistication of the technology and the stack. There are a lot of things that have fallen into place that I think make this the right time. So that's my bet — that this is the right timing.

But yeah, of course, this is research. Research is never guaranteed, and that's what makes it exciting. If you get the timing right, you can be the first. You can be the ones to change the whole field of AI. That's what it means to take risks and do something exciting.

Sonya Huang: For Ineffable, and also all the previous things that you correctly forecasted, is the biggest driver of timing compute?

David Silver: I think it's a very big driver — an enormous driver. We're in a great position with the compute that we've managed to get hold of, and it gives us an opportunity to do things which are orders of magnitude more sophisticated than approaches that have tried to do self-discovered RL in the past.

If we compare to something like hide-and-seek from OpenAI, which was an early example where agents discovered some basic tool use — I think we could probably run experiments that use 100,000 times more compute. It's kind of amazing that people haven't come back and really pushed on these types of approaches, given the availability of compute. And of course, hide-and-seek — I don't want to pretend that's the end goal. I think it's very basic. But it's suggestive of what could be done. So compute is a huge driver. It's not the only one.

Lasse Espeholt: I think we're in a very fortunate situation, in that we've been able to secure a very large cluster. So a large —

Sonya Huang: A very beautiful cluster at that.

Lasse Espeholt: Yes. This is really an opportunity for us. We really have the cards in our hands to test this out. There are a few other people doing something similar, but we can really test this out in a way that we couldn't at DeepMind, for instance.

Sonya Huang: Absolutely. It seems like one of the determinants of how far this can go is how far you can push simulation, because ultimately these crucibles are digital. I think simulation has come quite far in the last handful of years, but it's not perfect. We recently had your friend Rich Sutton on the show, and I asked him about simulation, and he was very dismissive of it. I'm curious how you think about this. Lasse, you've been on both sides of this — you obviously did a lot in the digital world, but you also built MetNet, which touches real-world atmospheric data. How do you think about where the ceiling on simulation is, and when is the right time to bring it up into the real world?

Lasse Espeholt: MetNet was actually an example of this. It's a weather forecasting model we did in the Google Brain Amsterdam office, together with Google Research, where we built a machine learning model to predict short-term weather. This was way better than the physics-based simulations — the physics simulations were accumulating errors over time. So there's a point there, that physics simulation is not perfect.

I do think there are a lot of relatively simple environments we can build where we can see this emergent behavior of, let's say, technology and so on, where we don't need exact physics. And I actually think it's interesting how much we can build these days with a very small team in terms of these simulations. So there's definitely a ceiling —

Sonya Huang: Is there a ceiling? Actually, I'm curious about this.

Lasse Espeholt: Well, quantum physics and whatnot. There's a ceiling somewhere. But we can do very interesting things without real physics, I would say.

David Silver: I think the right question to ask is: can a superintelligence be built in simulation? If it is possible for it to be built in simulation, I believe it will happen ten times or a hundred times faster, just because simulation can be infinitely parallelized. You get perfect reproducibility. It has a lot of really nice properties. It's also the safest place for superintelligence to be built.

Then we need to ask: is it possible? For the reasons we discussed earlier, there are very clear things we can add into the simulation that allow it to be sufficiently complex to represent arbitrarily interesting things. For example, say you wanted to build a computer in simulation. All you require is that your simulation is Turing-complete. If you have a Turing-complete simulation, suddenly everything that we mean by computation — provably, everything we mean by computation — can be expressed completely inside the simulation. And indeed, humans have built incredibly complex computers, or related things, inside simulated worlds.

The same is true for various other ideas. If you want to build an economy — money can be built out of coins, or out of paper, or out of bits. As long as you have some representation of money, everything that we mean by economics can be expressed inside a simulation. The same is true for language. If you've got multiple agents and they've got some channel of communication between them, suddenly everything that we mean by language is expressible in simulation. Mathematics, we know, is expressible in any universe. Science, in some sense, is the study of our world — so will a simulation be sufficient to study what we mean by science? Well, there are some aspects of science which are expressible in many things, like energy or entropy or evolution. But it's also true that the agent will discover its own science. Whatever the simulator is, if it's got this emergent complexity to its environment, the agent will discover its own science about whatever that world is. So it's provably possible to build environments in which it's possible to express and represent enormous complexity of all of these things.

I agree with Lasse that there might be some things which aren't well expressed. There will be things which can only be expressed in the real world — quantum computation is a good example. But the real point is that if it's possible to get all the way to a superintelligence in simulation, it will happen faster and it will happen safer. For those reasons, it's something that we choose to put a lot of energy into.

I actually agree with Rich Sutton. I think his point was something along the lines that you shouldn't expect the things that you learn in simulation to transfer — sim-to-real — into the real world, because the real world is vastly more complex, in very interesting ways that are hard to simulate. But what we can do is all of the science to discover the superlearner. Once we have that superlearner, we will have the answers, and we'll know how to build something where that same algorithm should be applicable to the real world. We can also make our simulations more and more high-fidelity towards the real world, and check that they still work, before putting them into the world.

Sonya Huang: Great answer. I'd love to talk about alignment. I know this is a politically charged topic right now, and I'm not interested — I'm zero interested — in having the politics or policy discussion around alignment. But I am very interested in the scientific discussion around it. It seems like so much of the misalignment that's emerged has arisen from scaling reinforcement learning to the extent that it's been scaled. Do you think the superlearner you build will naturally be aligned to human interests and have human values? Is it important for it to have alignment with human values?

David Silver: This is a big topic, so let me start by saying a few things. Where we are today with alignment, we are relying very heavily on alignment approaches that are based on the language that the agents use, and on priors that are based on human data. That has been very effective up to this point. But we're starting to see that approach reaching its limits. By itself, I think it's going to be difficult to see a path where, as agents inevitably become more and more powerful — and we can already see this happening — as agents are trained more and more, they start to overwhelm their priors. They start to learn to say one thing and act in a different way. It's going to be increasingly difficult to align them to the things that humans want them to do. My belief is that we need something more, and that this by itself is not sufficient.

The most important single thing that we can do as a company is to actually understand the science of superintelligence. If we can, in simulation, really build a superintelligent agent, and start to see these superlearners go all the way, we can start to understand what it takes for a superintelligence to interact peacefully. We can answer really important, fundamentally important questions: under what conditions, what structures, what levers do we need to put in place to make sure that a society of superintelligences actually interacts peacefully — and in particular, interacts peacefully with other, lesser intelligences that are also going to be in their world?

If we get this right, we can get to some answers — a real science that will help us understand what the right things to do are in the real world. Because we're hurtling towards that real-world outcome, where there are going to be a whole load of different interacting superintelligences, and we don't quite know how that's going to play out. The faster we can get to these answers in simulation, and publicly share them, the better situation we'll be in. That's my strong belief.

I also believe that we can use this to gate the actual decision-making of what we do in the real world. One of our commitments is that we will gate anything we do in the real world on having sufficient evidence, from this science of superintelligence, that we are really confident things are going to go well. We'll only push forward to the next level of what we do in the real world once we are absolutely confident that it can lead to a good outcome.

So those are two things I believe we can do. We're in a world where a lot of things are happening very quickly. We're hurtling towards an outcome with lots of interacting superintelligences. I personally believe this will lead to a good outcome for humanity, but it's happening very quickly. If we are able to understand this and run ahead in simulation, we'll be in a much better place to do the things that ensure this goes really well.

Sonya Huang: Fascinating. It seems like one of the primary things current safety researchers rely on is being able to monitor the agents' chains of thought in English. When they start speaking "neuralese," that scares people, because we can no longer monitor them as well. In your case, you have your agents going straight to rediscovering language. What do you think that means for monitoring? Or is the point that if you get the reward right and the environments right, you shouldn't need to monitor?

David Silver: I think monitoring is still really helpful. But like it or not, I believe the approach of monitoring the human language that these systems use is going to become increasingly difficult as agents get more powerful. We're already seeing that the monitored agents are maybe producing a tome of a million words of reasoning, or actually actively using subterfuge to think one way and act in another. It's increasingly difficult to rely one hundred percent on that. So I do think we need other elements in our armoury anyway, and it's really important that some people are looking at how to get this right.

There are ways to monitor agents that don't use language. You can do things like shadow actions, where you look at what an agent would actually do if it were allowed to act in the world, but you don't actually let it take that action yet. You can have other superintelligences acting as the monitor. For example, you might have the idea of an incredible scientist agent that treads lightly upon the earth, and then another superintelligence that's monitoring the magnitude of its actions and regularizing it, to make sure it can't do things which affect the world too much.

There are lots of ideas. There's a lot of research to be done here. I'm not claiming to have all of the answers, but it's something that deserves a lot of investigation. If people aren't looking at this because they're only looking at the language-based approach, I think we're in trouble, because that language-based approach is not enough in itself to get us where we want to go.

I should say — having said "in trouble" — I am optimistic about where the world is going. By trouble, I mean that we're not going to have agents that do the things we want them to do. I do think we're going to end up in a world with lots of complex, interacting, superintelligent beings. And I'm optimistic that this will lead to a balance of actors, some of which are supporting the flourishing of humanity and some of which may not be — but the ones which do support the flourishing of humanity will end up supporting us and protecting us against the ones which aren't acting in our best interests. That's the world I think we can work towards.

Sonya Huang: Well, given your super-forecasting abilities, I'm happy to hear that. What do you think about all the conversation around pacing the frontier?

David Silver: I'm very much supportive of the idea that there should be pacing of what happens at the frontier of AI in the real world. At the same time, I think we should move as fast as possible in simulation. The reason is that in simulation, we can develop a lot of things — like the science of superintelligence — that will help us find solutions that help things go well in the real world. That work should happen as fast as possible, to make sure that when things really happen in the real world, they have the best possible opportunity to go well and lead to human flourishing.

In general, I'm supportive of the kind of direction that people like Demis and Dario are suggesting. I think it's generally a good idea. Part of that is that things are moving really quickly. I do believe that when we accelerate AI and get to superintelligence, it will lead to a really great future for humanity. But the speed at which things are moving is not something we are capable of adapting to that fast. Just by pacing a little bit, it gives our governments and our organizations and our emotional responses the opportunity to adapt to the changes that are coming — and for this work to continue apace, really figuring out in simulation the real science of doing this right.

Sonya Huang: So accelerate in simulation, so we can understand what's happening and the science around it — and then be careful in the real world.

David Silver: Yeah, that would be my summary.

Sonya Huang: I'd love to transition to talking about compute. This is going to be an incredibly compute-expensive journey, and you are pushing the limits of what compute can do. I think both Google and Nvidia have been great partners for you. Can you talk about your cluster? You're not simply renting spot compute from neoclouds like many other neolabs. You're doing some really serious work on your cluster, and on the other forms of compute that you have. Can you talk about your compute setup?

Lasse Espeholt: As I mentioned earlier, we're very fortunate with the compute that we have. We're the first customers of the largest cluster on GCP, which starts by the end of the year — actually, one month from now. So we're extremely fortunate in that. I actually feel quite bad for the startups that are getting compute now, because it's becoming harder and harder.

We have this cluster, which is based on GPUs — Nvidia GPUs — which is really fortunate for us, because it means we can easily do simulation and training and inference. They're quite general chips. That's amazing. The Vera Rubin specifically has this architecture where one rack can basically be considered one machine. There's very high bandwidth between the nodes in one rack, which is very important for reinforcement learning, where you want really fast iteration cycles, compared to LLMs, which are relatively slow. So that's quite fortunate for us.

Some of the things we did eight or ten years ago at Google, where we had abundant CPUs — that doesn't scale anymore. So we really need to make sure that everything we do is on the GPUs, and that we don't need millions of CPUs beyond these GPUs. Our simulations run on the GPU, along with inference and training. We have some good architectures for that. I think the LLM companies now have the problems that we had many years ago, where they need sandboxes running on CPUs, simulating computers and so on. We're trying not to put ourselves in that situation.

Sonya Huang: Is all the compute roughly fungible between the different things you're doing? I imagine you're building crucibles, and then you're doing these big rollouts where the agents are actually learning. Are those the rough buckets of what compute is being used for?

Lasse Espeholt: You could split it up into training, inference, and simulation.

Sonya Huang: Is the same compute equally good at training, inference, and simulation?

Lasse Espeholt: No. I think it's pretty clear what has happened with GPUs over the last many years, since deep learning took off: they've been more and more tailored towards training, and in the last few years also inference, but less toward simulation. But similarly to the LLM companies, we also want to be in the position where scaling laws matter — where we know that roughly, if you add this amount of compute and experience, you get a certain amount of quality out of it.

Sonya Huang: If GPUs aren't great for scaling simulation, then what's the solution?

Lasse Espeholt: There are many different solutions. There's model-based reinforcement learning, where the model becomes much more expensive compared to the simulation. There are many tricks you can do — various approaches with replay buffers, or reusing data many times. So there are ways around that. Other than that, we spend quite some effort on just making the simulations as fast as possible and making the best use of the GPUs.

Sonya Huang: Are you seeing scaling laws yet, or is it too early to say?

Lasse Espeholt: For reinforcement learning, scaling laws are slightly different. Scaling laws are about how much compute you put in, how much data you put in, and then the model size. With reinforcement learning it's a little different, because the problem is non-stationary. As you progress in the problem, the data changes — the data distribution changes. So they're a little different. It's a very interesting research direction.

Sonya Huang: I understand the difference. But are you seeing some sort of scaling law, where more compute in gets you closer to achieving the goals? Or is it hard to say right now?

Lasse Espeholt: Yeah, but it's something we're actively pushing towards. Especially as we transition from the smaller cluster to the bigger cluster, we want to be ready for that.

Sonya Huang: Lasse, at the beginning, when David introduced you, he mentioned that you had done some very foundational work on compute and algorithms for reinforcement learning, which the field really relies on today. Talk about some of that work — in particular, your work on IMPALA and SEED RL. What are those, and how do they relate to the work you're doing at Ineffable?

Lasse Espeholt: When I joined the deep learning team at DeepMind, most people were running single-machine, CPU-based reinforcement learning. The scale was quite small. What I did, together with colleagues, was make a reinforcement learning agent that really utilized GPUs and could scale linearly with compute. Partially because it was very efficient and very good at scaling — and the third thing was that it was very easy to extend and modify and do whatever you wanted with it — it became the most popular reinforcement learning agent at the company. At one point, most RL teams used it, or derivatives of it. So that was quite successful and a nice project to work on.

Then later on, we got TPUs — racks of TPUs — so we needed to scale beyond that. Some of my work at Google Brain, after I left DeepMind to found the Google Brain Amsterdam office, was to scale to racks of TPUs, with a project called SEED RL. That was really about utilizing the TPUs for inference and training in the best possible way, while keeping the environments as flexible as you wanted — anything from Go to very expensive physics simulations.

Sonya Huang: And how is that being used at Ineffable?

Lasse Espeholt: We actually implemented a few of those algorithms, and derivatives of them. That's what we're using today. Depending on the environment, we use different algorithms — there are pros and cons between them. As we talked about, we have some very simple, fast-to-simulate environments, and also some that are more expensive.

David Silver: For those who are working on large-scale RL architectures today and haven't read those original papers, it's definitely worth a look. I think you'll discover that many of the ideas currently in use trace back to that early work that Lasse did and figured out — which was a little bit prescient, I would say.

Lasse Espeholt: It's actually kind of fun to see the reinforcement learning space now versus ten years ago — or eight years ago, maybe. The LLM community is also using reinforcement learning, particularly in the last two years or so, and a lot of the things they're using are rediscovered approaches that were used many years ago. That's interesting to see — with a lot of novelty on top of it as well, of course.

Sonya Huang: Can you talk about your team? How big is the team right now?

David Silver: We have intentionally kept the team relatively small. We're not in a hurry to build a giant organization. You can do an awful lot nowadays with a small number of incredibly talented people. We've hired 20 people in total.

The people we've hired — it's about finding exceptionally talented people. But it's not just about talent for us. It's also about bringing together people who are kind, collaborative human beings. That's really important to the culture we're trying to build. People who are really well aligned with the mission we're following. And people who fit the needs that we have — we try to build the team with the right set of skills as well. All four of those things are really important. We have very high standards and very high expectations of the people who come in, and we've been very lucky to get some amazing, really amazing people.

Lasse Espeholt: Just to tie that to the compute as well: having this relatively small number of people so far means we have a lot of compute per person. These days, one person can do a lot of experiments. So it's also about keeping the compute-to-person ratio quite high, so we're not fighting for resources from day one.

Sonya Huang: I think three of the co-founders worked together on AlphaStar — many of you did work together. Are you assembling the Avengers of folks you've worked with in the past, especially given the recent changes at DeepMind? Have you hired new people? And if so, what are the attributes that make somebody new to you likely to be a fit at Ineffable?

David Silver: It's a mixture. AlphaStar was an exceptional team. It was intentionally assembled within DeepMind by choosing the best of the best to work on that project. So in some sense, it's not a surprise that four of our co-founders come from that team. But we are not exclusively drawing from that team, or indeed from DeepMind. The other three founders all came from very successful startups. And we hire people who are brilliant, but also kind and nice people to have around, who are going to work together well, who are aligned with the mission, and who fit the things that we need.

In terms of finding those people, we put a lot of effort into getting the right people. As uncomfortable as it is, I've had to say no to a lot of people I've worked with in the past, just because our bar is so super high. I've been fortunate to work with some amazing people, so I've been choosing among a very exceptional crop and bringing in what I consider to be a superstar team. It's not easy to get into Ineffable, but the people we have are wonderful human beings, and I'm very, very proud of the people we've brought in.

Sonya Huang: I can see why people are drawn to working at Ineffable, especially at a time when the biggest clusters, the biggest labs, are locked in this prisoner's-dilemma race. There are a lot of dollars at stake, and so decisions get made differently. I can imagine, for a set of people who really, really deeply believe in the scientific mission of what you want to accomplish, this must feel like a breath of fresh air.

David Silver: I think that's right. We've tried to establish a research-led culture. It's not a company that's led by building an incremental next product, v17.1 on the thing that came before. It's a company built on researchers pursuing their passion and trying to produce the breakthroughs they believe are possible. We centered the company around that from the beginning, from the outset — to create a haven for research, where we push back on product pressure, because we believe that if we make these breakthroughs, it will lead to far bigger products downstream, and far bigger successes, than if we started on day one with, "Here's the product we're doing from the beginning." So it's research-led. We're creating a really special environment. People are loving it. I'm loving it.

Sonya Huang: Wonderful. I'd love to close with some predictions — especially since you're such a super forecaster. What's your guess for when you'll have the first strong evidence that the approach is working? And then what's your forecast for when we'll achieve the mission you've set out to achieve — this first contact with superintelligence?

David Silver: These are fair questions. There are going to be some reasonable bars of uncertainty on this.

Sonya Huang: But you're being recorded, so we're going to hold you to it.

David Silver: Exactly. I think we will see some extremely strong evidence that we're on the right track within a year. And I think we will achieve something which is a true superintelligence — according to the definition I gave earlier, something which discovers things far beyond what humans can achieve, across everything from science, mathematics, technology, languages, et cetera — I originally thought within three to five years. My timelines have accelerated a little bit, so I'm going to say three years, because of what can be done with coding agents to accelerate things. I think things are going to move fast.

Sonya Huang: Buckle up. It's amazing. Lasse, David, this was a wonderful, wonderful conversation. I'm so inspired by what you're building, and thank you for taking the time to sit down and share with our audience what you're building today.

David Silver: Thank you, Sonya.