AI:AM LIVE — August 26, 2026 — The 100-Gigawatt Problem, Vercel's Malte Ubl on Self-Driving Infrastructure, and Inherent's Louis Kirsch and Damon Falck on Training an AI Scientist

The rundown

1. 0:50Opening31 min

   Opening: The 100-Gigawatt Problem"Ox Alpha" turns out to be Zhipu AI's GLM-5.3, running largely on Chinese chips, and the hosts use it to reopen the question of how much the hardware gap actually still protects. From there: YMTC's climb toward the top of NAND and the Apple dispute around it, whether the US bottleneck on power and construction is self-inflicted and fixable, Crusoe's flare-gas origin and the case for moving compute to stranded energy, off-Earth compute, and Andrew Critch's prediction that materials science is where AI surprises people between late 2027 and 2029.

   Watch

   ▶Play this segmentYouTube · from 6:44Open segment on YouTube ↗

   As aired

   Prakash opened with a scoop from the prior 24 hours: the disclosure that "Ox Alpha," a free, generously rate-limited model that had circulated on OpenRouter for about a week amid heavy speculation about who was behind it, was in fact GLM-5.3 from the Chinese AI company Zhipu AI — run, notably, mostly on Chinese chips. Nathan connected this to his recent trip to China, where he found AI abundant rather than scarce: free apps, no rate limits as a retail user, and the only real capacity constraint he encountered was Moonshot pulling its K2 model behind a paywall and pausing new signups when it hit capacity. A ByteDance contact told him the company would support inference demand for any fast-growing startup essentially without limit. Nathan argued that a hundred trillion tokens a day served largely on domestic chips (citing a SemiAnalysis report he considers credible, while flagging that more facts may still surface) undercuts the assumption that China's chip constraints will keep it from scaling AI compute, and suggested it may be time to revisit US China policy on that basis.

   Prakash pivoted to a second Chinese scaling data point: YMTC (Yangtze Memory Technologies), currently roughly the third- or fourth-largest NAND producer, is aiming to overtake SanDisk for the top spot by the end of next year and is heading toward an IPO. That fed a longer exchange about the Apple/YMTC/Micron dispute — Micron and the US government are pushing to keep Apple from using YMTC memory in China-market iPhones. Nathan pushed back on the logic: if the policy goal is denying China advanced tech, blocking Apple from using Chinese-made memory in Chinese-sold phones doesn't serve that goal and mainly hands market share to Huawei and Xiaomi. Prakash offered a steel-man — the point isn't the memory itself but denying Chinese firms cash flow and growth capital, which he called "the ballgame" in a commoditized industry — and tied this to Micron's own struggles getting a New York fab built (still not online, targeted for around 2028 after years of local opposition), contrasting a system that wants onshore manufacturing but leaves companies to fight the political battles for it alone.

   Both hosts agreed China's government is, in Nathan's words, more consistently incentivized to help its companies succeed even as it reins in abusive practices, and Nathan argued this is a fixable US problem — floating that Trump could use executive orders to speed up infrastructure buildout, including on federal land in the western US, while speculating Trump may be unsure whether his political base actually wants that. This led into a segment on where to physically site data centers: Nathan relayed a proposal, written by a former Palantir field engineer who had worked the Alaska oil fields and pipeline, to build data centers in northern Alaska using the large volumes of natural gas currently reinjected into the ground rather than transported, arguing the sparsely populated, already-industrial site made it a low-cost, low-objection option.

   Prakash broadened the frame to stranded energy generally, citing Crusoe Energy's origin flaring gas at oil fields to mine Bitcoin before pivoting to AI data centers, and endorsed the "move compute to where the energy is" logic, including Elon Musk's off-Earth compute framing. But he was skeptical that the current pace of US data center buildout — the roughly 100-gigawatt-scale, multi-trillion-dollar 2029-2030 projections he said were discussed on a recent Dwarkesh Patel/Dylan Patel (SemiAnalysis) podcast — is politically or physically sustainable, citing local opposition, competition for electricians between data centers and residential construction, and hard materials constraints such as the roughly 40,000 pounds of copper needed per gigawatt. He argued those projections only work if unusual technology breakthroughs arrive — robots taking over physical labor, new conductive materials replacing copper — and framed the gap between AI researchers and mainstream scientists and investors as one of faith: AI people assign real credence to future technology arriving in time, where conventional science demands evidence that doesn't yet exist.

   Nathan connected that to AI researcher Andrew Critch's recent, specific public prediction that materials-science applications will surprise people between Q4 2027 and 2029. He cited his own past reporting on AI-for-materials-science companies, describing a finding from roughly two years ago in which a neural network trained only on simulated saltwater began exhibiting untrained physical behavior — spontaneous crystal formation — once scaled up, and separately produced suggestive mechanistic evidence about how the potassium ion channel transports ions, a question that had long resisted direct simulation. Nathan framed this as consistent with the long-running "Kurtzweil story" of successive micro-exponentials rescuing progress just before each one plateaus, and cited Critch's long track record of being early and specific.

   Key moments

   In my view, it is maybe time to update and reconsider some of our China policies, in light of the fact that nothing we've done has really seemed to deny them the ability to advance — and now, increasingly, the ability to scale.

   The ultimate goal is to deny your competitors cash flow. And the better that you can do that, the longer you can sustain your market position. That's the ballgame.

   Underneath all of this is that the AI people are, in some sense, have faith — this religious faith in technological development.

   Full transcriptLightly edited · timestamps jump to YouTube

   6:44

   Prakash Narayanan: Good morning. It is Wednesday, August 26th, 9:01 AM. Nathan, good morning.

   6:47

   Nathan Labenz: Good morning. How are you today, Prakash?

   6:49

   Prakash Narayanan: I am very good. It has been an interesting 24 hours, because we've just seen the first — well, maybe not the first, but the first real disclosure — of who was behind the Ox Alpha model release. Ox Alpha was a model released on OpenRouter maybe about a week ago, and there were lots and lots of rumors about who was running it. They had very generous limits — something like a trillion or ten trillion tokens served for free.

   7:35

   Prakash Narayanan: I think they also probably hired some influencers to help boost their standing, and there were lots of rumors going around. Today it was disclosed that it was indeed GLM-5.3, from a Chinese company called Zhipu AI. They also said they ran it mostly on Chinese chips, which is a real surprise.

   8:08

   Nathan Labenz: Yeah, the Chinese manufacturing ecosystem strikes again, perhaps — time will obviously tell on that, but it was an interesting observation when I was in China a few weeks ago. I was asking people whether AI feels abundant there, or whether it feels scarce. If you're a consumer, there are lots of apps, they're free, I never hit rate limits. The only indication I had as a retail user that there was some limit to the free intelligence was when Moonshot dropped K2 — they had to put that

   8:53

   Nathan Labenz: behind a paywall, and then they actually stopped signups for a time because they'd hit capacity and didn't feel they could deliver the quality of experience they wanted to people signing up past a certain capacity constraint. But other than that, all the models I used were free, and I never hit any rate limits. When I had the chance to speak with people at the hyperscalers, I spoke to one guy in particular at ByteDance, which, in addition to TikTok, has Doubao — their largely voice-AI experience that tons of people are using. We talked about that a

   9:38

   Nathan Labenz: little bit with David earlier in the week. They've got multiple flagship products, and they're also turning into a cloud hyperscaler. It was strikingly similar going to ByteDance to the experience of going to Google for lunch — the aesthetic of the campus, the free lunch for everybody, getting swiped in, the handful of flagship products that are world-beaters, and dozens of other products you'd never known existed that don't have much traction. I got the sense ByteDance kills its failed products a little bit more

   10:23

   Nathan Labenz: effectively than Google tends to. At the hyperscaler cloud level, I asked a guy: what's the prospect for a startup that really catches fire and is growing super fast — are you going to hit constraints on your ability to serve users, based on where the inference tokens come from? His answer was basically, 'we've got you' — if you're growing fast, we'll support that growth, you can get all the inference tokens you need from us on the ByteDance Cloud. I didn't test that at real scale myself, but this

   11:10

   Nathan Labenz: is very consistent with that. A hundred trillion tokens a day is not a small number. And the fact that they're serving it on Chinese chips — this is a report from SemiAnalysis, which I consider credible, but it's all happening pretty quickly, so we should keep an open mind that there could be more facts still to surface about exactly how this is happening. But I didn't feel scarcity there. And if we're betting on a strategy that has as a load-bearing feature that China won't be able to scale its chip production and won't

   11:55

   Nathan Labenz: be able to run as many agents as we're running — it's probably still true, but I don't think it's as true as people would have expected when they were mapping out these strategies. So in my view, it may be time to update and reconsider some of our China policies, in light of the fact that nothing we've done has really seemed to deny them the ability to advance — and now, increasingly, the ability to scale.

   12:25

   Prakash Narayanan: On that ability to scale — here's another report, on Chinese memory technology. China has memory producers, and memory, especially lower-end memory, isn't very technically difficult to make. There's a company called YMTC, Yangtze Memory Technologies, that's been roughly number three or four in NAND for a while, and they're aiming to be number one by the end of next year — directly competing with SanDisk, the biggest NAND producer. The interesting thing is they're also expecting to IPO. There'll be the usual price fluctuations post-IPO, of course, but this is a stake in the ground — something that's been expected for a while, that they'd flood the memory sector. And they're scaling — as you pointed out, they're good at scaling.

   13:44

   Nathan Labenz: Yeah, I don't know how much I can really add to that. Can we buy into this company on the stock market somehow? There's been a lot of money made in these bottleneck trades — this seems like a company that's probably going to sell a lot of memory in the Chinese market. I don't know how easy it is for us to get exposure to that, but it would seemingly be a likely winner.

   14:12

   Prakash Narayanan: It's also a geopolitical thing, because Apple wants access to YMTC memory, at least for its operations in China, and the US government — especially Micron, the US memory manufacturer — is very strongly pushing back and trying to dissuade them. It's interesting that the US government doesn't actually have a lot of tools at its disposal over Apple's operations in China serving Chinese customers. I think this is one of those situations where you're starting to see these mega-corps not really controlled by the country in which they're headquartered. And I think this causes a lot of people a significant degree of unease.

   15:06

   Nathan Labenz: Well, yeah — companies starting to rival the power of states is definitely an uncomfortable situation. I'm honestly kind of more bullish on some new power centers; I don't think our current power centers are working super well, so the fact that there's still room for entry into the market for political power globally is probably healthy — better that than no entry, I guess, is my default view. What would you say is the steel-man case, if you can offer one? Because I'd honestly struggle to explain why American memory makers

   15:51

   Nathan Labenz: and the US government should try to prevent Apple from using a Chinese memory component in an iPhone they're going to make and sell in China. First of all — don't we want all the memory we can produce? I thought we were trying to restrict the export of this kind of stuff, and I'm not necessarily for that, but if we're working from the assumption that we don't want to send our best tech to China, then I'm confused. I'm also confused — if we don't want to send them this tech, but we're also going to try to deny them the ability to use the Chinese stuff, that would basically mean Apple can't do business in China.

   16:37

   Nathan Labenz: And then what does that do for us? It just gives more market share to Huawei and Xiaomi, and I don't see how that's a great win. As a simple-minded person, I just see Apple doing business in China, buying Chinese components, making iPhones in China, selling those iPhones to Chinese people — it all seems pretty innocuous. Could you channel the Micron position? I think the Micron position is very —

   17:14

   Prakash Narayanan: Simply market denial. The more you can deny a market to your competitor, the better things are for you — it doesn't matter what the specific market denial is about, all that matters is that if you give them cash, they'll use it to grow, to expand, to get better at technology. So the ultimate goal is to deny your competitors cash flow, and the better you can do that, the longer you can sustain your market position. That's the ballgame, I think, because they

   17:59

   Prakash Narayanan: are in a commoditized business. They also have this enormous handicap, which is the speed of setting up new manufacturing facilities in the United States. For example, they're trying to set up a fab in New York, and they've faced tremendous opposition in upstate New York — there's still all this 'Micron is going to mess up the water supply' pushback, and they've been fighting over it for years. They're slated for around 2028 for the fab to come up, and

   18:48

   Prakash Narayanan: there's still not a lot of visibility on construction. I think the US political system has two choices: make construction easier and allow a lot more onshore construction, faster, so US manufacturing can be competitive — or don't, and, like every other kind of manufacturing, have it done offshore, imported, with the trade deficit and jobs deficit that comes with that. We seem to have settled into wanting the manufacturing, but wanting our companies to come fight the political battles for us onshore and win through, and then build the manufacturing — which puts the companies in the position of having to shift public opinion, and gives politicians the easy win of standing on the other side resisting the companies, which just doesn't make sense. But anyway.

   20:11

   Nathan Labenz: Yeah, I mean, in some ways I think this is one of the biggest advantages China has right now relative to the US — their government officials seem to be incentivized to help their companies succeed. That's not to say they don't exercise oversight or put companies in their place when they get too big for their britches, but the default assumption is that they want their companies to succeed, at the local and the national level. I get the sense their regulatory environment is a lot healthier than ours, in the sense that when they see something

   20:56

   Nathan Labenz: that's really abusive, they pretty reliably step in and stop it. The central government may have its own abusive tendencies that nobody curbs, but they do restrict what the big platform companies can do to protect small businesses and everyday people — while still wanting those companies to succeed. It seems like they put them in their place but also really encourage their growth. Here, we have sort of the opposite: companies are allowed to sustain abusive practices

   21:42

   Nathan Labenz: for a long time without much intervention, but then they also can't do the basic blocking and tackling of growing. That's not a healthy equilibrium for us — I'd really love to see this change, and it honestly feels like the kind of thing Trump should be changing. I'm not a fan, as everyone knows, but what good is this sort of highly energetic, individual-decision-maker executive if we can't get a few EOs out there to make things

   22:27

   Nathan Labenz: easier when it comes to building critical infrastructure? That seems like a huge miss. My guess — and you can tell me what you think — is that Trump isn't sure where his base is on this issue, whether he can lead the base or needs to follow it. If his people are sufficiently anti-tech, he may feel like this is one where he has to do what they want, which might mean making life difficult for big tech. All the CEOs sat behind him at his inauguration — I thought that might be enough to get some

   23:12

   Nathan Labenz: permissive EOs signed. Honestly, I think we should be building on federal lands at this point. It's crazy how much of the western two-thirds of the country — maybe the western third, better said — is owned by the federal government. There's tons of empty space where we could put facilities without sacrificing national parks or ruining the landscape. Hardly anyone would even notice, in a lot of cases, if we chose the sites well, and yet it hasn't happened. I saw an interesting proposal the other day too, for data centers in the North of Alaska — did you see this? No —

   24:04

   Nathan Labenz: it was written by somebody who has been a forward-deployed engineer for Palantir working up in the oil fields and on the pipeline. The oil fields are in the very north of Alaska; the pipeline runs all the way across Alaska from north to south, from the Arctic Ocean down to ports that are open year-round, where the oil can get shipped out. They don't apparently use the gas, because there's no infrastructure to transport it — they can't get LNG tankers up there for big chunks of the year. So somehow we've settled into this

   24:49

   Nathan Labenz: equilibrium where the gas gets pumped back into the ground: they get the oil out, gas comes up with it, and they reinject the gas. There's apparently a huge amount of it — this pipeline has been flowing for something like 50-plus years, so there's a huge amount of gas that's been surfaced and reinjected deep into the earth, just sitting there as proven reserves. The proposal was: this landscape is already far from pristine — there are holes in the ground all over from people drilling for oil, so it's hardly postcard nature at this point — why not put some data centers there? We've got the

   25:34

   Nathan Labenz: gas. The calculation was something like, based on the proven gas reserves alone, you could run a massive data center footprint for decades. Why not? I thought it was a pretty compelling proposal. And again, it's kind of like — where is the executive on this stuff? If we're serious about anything — if we're serious about winning, which I don't love the winning-and-losing frame, but if that's their frame — this is the kind of thing we need to do. And even if we're just serious about making sure retail mom-and-pop users aren't priced out of AI over the next couple of years, it seems like the kind of thing that

   26:20

   Nathan Labenz: we need to do. Nobody would have really objected — not many people live there, it's mostly oil workers who are used to infrastructure projects happening around them. This is way cleaner than the existing infrastructure. Let's wake up, people. Come on — what are we waiting for?

   26:42

   Prakash Narayanan: So, as a former energy investor, it's commonly known that there are a lot of stranded gas assets in the world. One of the major things in energy has always been to move the energy from where it is to where it needs to be consumed — a lot of what we do is figure out where the energy is, coal mines, gas, oil, and figure out how to transport it. Oil has been so popular because it's easy to transport — you can build a pipeline, put it in a car, move it in multiple different ways, and it's very energy-

   27:28

   Prakash Narayanan: dense too. I think Crusoe, the data center company, started off flaring — they wanted to get gas that was currently being flared at oil fields, use it for free, and mine Bitcoin with it. That's how Crusoe first started, and then it transformed, as this link from energy to compute got tighter with AI, into an AI data center company. So I think this idea of building data centers where the energy is, is a really

   28:13

   Prakash Narayanan: core concept. It's even kind of what SpaceX is doing: the energy is really outside Earth, so they're going to move the compute to where the energy is. I think that makes sense. But I also think Elon is probably right in expecting that onshore data center construction is just going to stop. There was a podcast that dropped, I think, yesterday or the day before, between Dwarkesh Patel and his so-called cousin, Dylan Patel of SemiAnalysis. They were talking about: what happens when the

   28:58

   Prakash Narayanan: spend per year in 2029, 2030 gets to, say, $5 trillion or $10 trillion, these insane kinds of numbers — and all of a sudden you're building 100 gigawatts of data centers. I think it's very politically unlikely that you're going to get the ability to build that much in any one location. I don't think it's just the politics

   29:43

   Prakash Narayanan: alone — it's infrastructure, it's traffic, it's employment. For example, one of the things that's happened in the last couple of weeks is people asking: well, if all the electricians go work on data centers, what happens to the residential construction we need? Because there's a lot of residential construction that's necessary. So there's a lot of issues there. My take has always been that within the next year or two you need to see something unusual — something like, okay, robots start working. We don't need

   30:28

   Prakash Narayanan: as many electricians anymore — maybe half of the electrician job can be replaced. Like in code, you have a reviewer that reviews the work the robots have done, but the core day-to-day activity isn't done by a person. You need these kinds of unusual step-ups, which no one's very confident predicting or betting on right now. You need new technologies that use less copper — I think it's something like 40,000 pounds of copper per gigawatt of data center, and the whole world's production isn't going to get you to what you need for 100 gigawatts of additional power-plant and data

   31:14

   Prakash Narayanan: center capacity by 2030. So these numbers have never made sense to me in that sense — you cannot expect the exponential growth to continue with today's technology. We're going to need future tech. And the thing about future tech is, even the AI train-27 or train-9 crowd, or whatever you want to call them, don't feel confident predicting future tech, because it sounds ridiculous and silly. It sounds ridiculous to say there's going to be a million Optimus bots doing electrician jobs in two years. It sounds ridiculous to say we're

   32:02

   Prakash Narayanan: going to find new conducting materials that are better than copper and cheaper and easier to manufacture, and that this is what gets us to 2029, 2030. These things sound ridiculous, because every scientist will say: well, where's the evidence? Scientists work from evidence — where is it? It's not there. And I think this is where the AI people break with the regular scientists, because the AI and ML people are willing to assign credence to this kind of future tech happening, and the rest of the world, including the investors, are like, you guys are ridiculous — these curves don't make sense, the exponentials don't make sense, you're just not going to get there.

   32:48

   Prakash Narayanan: Underneath all of this is that the AI people are, in some sense, have faith — this religious faith in technological development, that AI is going to create these new technologies that make this possible. I think that's where the gap is between the two. And I think it's pretty clear to me that the political system is not going to allow the same 100 gigawatts,

   33:16

   Prakash Narayanan: with the same kind of impact on resources, and pollution, and employment, that we're building right now. Here's an example that just —

   33:28

   Nathan Labenz: Comes to mind when you talk about AI people having faith — I might use a slightly different word than faith, but this is Andrew Critch, who's been in the game for a long time. He goes back with folks like Eliezer, and he's got a PhD in machine learning — math and machine learning, I think — and he's been involved for a long time. Here he is on record, just in the last day or two, saying materials-science applications are going to surprise a lot of people in the next two years. This came to mind also because of how specific he is on his predictions — Q

   34:13

   Nathan Labenz: 4 2027 up through 2029 — not 2030. Time will tell, obviously, but I wouldn't bet against him, or the field, on this, because I've done a few episodes with different companies specifically trying to bring AI to materials science, and even a year ago, when I was down that rabbit hole, they had some pretty remarkable findings. One example, from maybe two years back: the trick a lot of times seems to be getting a lot of simulation data — which, with these physics simulators, is still quite costly to

   34:59

   Nathan Labenz: run. But then with enough of that, you can train a neural network on the simulation data and get it to do similar-quality predictions but maybe two orders of magnitude faster. So now you're dramatically accelerating your ability to run these simulations. One of the first emergent surprises was a guy simulating saltwater in a pretty simple way, but seeing spontaneous crystal formation and dissolving — something that hadn't been trained on. It was just a matter of what his

   35:45

   Nathan Labenz: sample was trained on. But as he scaled it up with the roughly-two-orders-of-magnitude-faster network and ran it for a lot more time steps, he started seeing physical phenomena that weren't even trained on, emerging in the simulation. They also did the potassium ion channel, which I thought was really interesting — it'll need to be experimentally verified, but there was suggestive evidence for how the potassium ion channel works mechanistically, something people had speculated about for a long time without being able to bring much additional evidence to bear on it, because it's just very hard

   36:30

   Nathan Labenz: to simulate. They were able to do that, and now it seems like one of the competing hypotheses was privileged, in terms of exactly how the potassium ion gets transported down that channel, from inside to outside the cell and vice versa. So I think it's a pretty good bet that we'll see a lot of materials advances. Zoomed all the way out, this is basically the Kurtzweil story — it's held since he published it, and if you believe his analysis, for thousands of years before that: all these micro-exponentials that come one after another, individually peter out, but always

   37:15

   Nathan Labenz: seem to get saved by the next one just in the nick of time. I see that as a pretty good bet. Andrew Critch in particular has a long track record of being pretty prescient on a lot of this stuff — you could look back at his writing from five or ten years ago and think, wow, this guy saw a lot coming before a lot of other people did.

   37:43

   Prakash Narayanan: On seeing a lot coming before anyone else does, let me pull —

2. 31:54Interview46 min

   Interview: Malte Ubl — Self-Driving Infrastructure and the Dual-Use ArgumentMalte UblVercel's CTO describes production infrastructure that triages its own alerts — an agent that spends thirty seconds to two minutes on context before paging a human, backed by immutable per-deploy infrastructure that makes rollback the cheapest fix. He explains why Eve replaced the deliberately low-level AI SDK rather than extending it, what a zero-margin AI Gateway is actually for, and where provider differences get normalized. The security half is the sharper one: open models are already strong at unguarded offense, the willingness to find and fix security bugs now separates models in the market, and the capability behind a red-team exercise is the same one behind an attack — which is why he thinks withholding it from defenders is the wrong call.

   Watch

   ▶Play this segmentYouTube · from 0:00Open segment on YouTube ↗

   As aired

   Prakash Narayanan introduced Malte Ubl, CTO of Vercel — a decade-plus at Google leading the teams responsible for how search renders pages, a co-founder of JSConf EU, and now the architect of what Vercel calls self-driving infrastructure. A production-app glitch (the studio's AI connection agent wouldn't stop talking over the guest) delayed the start by a few minutes; once resolved, Nathan Labenz and Prakash Narayanan spent the segment with Ubl on how Vercel is trying to make production infrastructure something agents can run, and on Vercel's current security posture.

   On self-driving infrastructure, Ubl described an agent that sits in front of every production alert: rather than paging a human immediately, it looks for 30 seconds to two minutes with more context — is this an actual error, or did marketing just send a newsletter that spiked traffic — before deciding whether to escalate. He argued AI is now genuinely good at this kind of real-time, nondeterministic debug-causality work, better in the moment than a tired human. Asked why Vercel could support this, he pointed to a structural choice: every deploy rebuilds infrastructure from scratch while the previous version stays live, unlike Terraform-style infrastructure that's changed in place and hard to undo cleanly. The payoff, he said, is that if the agent's top recommendation is to roll back, it can do so in about 300 milliseconds — even rolling back five deployments at once at that same threshold.

   Nathan Labenz pressed Ubl on why Vercel replaced its widely used AI SDK with a new framework, Eve, rather than evolving it. Ubl's answer: AI SDK is deliberately low-level — a toolkit for teams that want to hand-build their own agent harness — and stayed too low-level for the majority of agent projects. Eve is Vercel's attempt to identify what the roughly 90% of agents have in common and let developers express that directly, paired with a product called Connect that wires agents into GitHub, Linear, Slack, Workday, SAP, and Oracle.

   A large stretch of the conversation covered the AI Gateway. Ubl described provider-difference normalization happening at two layers — inside AI SDK, which he called a 'software factory' for chasing down provider-specific quirks (like inconsistent tool-call formatting), and inside the Gateway itself, which harmonizes APIs so a client using the Responses or Completions API gets the same interface across providers. He framed the Gateway's business model as strictly list-price and zero-margin — unlike OpenRouter, which he said marks up through its credit system — with Vercel instead monetizing by passing volume discounts through to larger customers, comparing the business to a retailer like Safeway. He also confirmed the Gateway handles automatic failover across inference providers on outage, billing customers only for the rerouted traffic, and said the market for provider deals is opaque and NDA-covered, but noted that DeepSeek recently passed Gemini in token share to become Vercel's number-two provider by volume, even as Anthropic keeps the largest share of spend.

   Prakash Narayanan turned the conversation to security, referencing the Hugging Face attack as an existence proof of automated AI offense without a matching proof of automated AI defense. Ubl pushed back on that framing. He said the market underprices how capable open models like Kimi K3 already are at unguarded offensive cybersecurity — probing and mapping a target system within minutes — but argued frontier defensive capability is also underrated. He drew a specific line between 'Fable,' which he said Anthropic shipped, unshipped, and reshipped with security-detection behavior he called nearly unusable, and Sonnet 5.1 and Opus 5, which he said will both assess a codebase for vulnerabilities and write a fix when given a report — tasks he said Fable won't do. He described DeepSec, an open-source tool he personally built, that runs whole-repository security scans rather than just new commits, and argued defenders currently have a real if temporary edge that has to be used before frontier-level offensive security capability becomes broadly available.

   On Vercel's newly announced $1 million Vercel Sandbox hacker bounty, Ubl said escaping the underlying micro-VM itself is unlikely; the more realistic risk is a side channel like the one in the OpenAI/Hugging Face incident, where a model found a path to write access on an external API. That's why the sandbox product leans on an egress firewall, and why the bounty scope leans toward finding ways around it rather than the VM boundary itself. On responsibility for open-weight models like Kimi K3, he argued some KYC is reasonable, but that once an adversary can access the same technology, withholding it from defenders is the wrong call — red-teaming tools and black-hat attack tools are functionally the same technology.

   Closing out, Ubl flagged Vercel's new experimental coding harness — written in Zig, deliberately minimal and portable, not yet stable — as worth a look for the experimentally minded, alongside Eve and DeepSec. In the post-interview wrap, Nathan Labenz and Prakash Narayanan noted how much more textured the security discussion was coming from someone who deals with it operationally, and Nathan flagged lingering curiosity about the actual margins and revenue-share structure behind inference deals, including whether model makers themselves (he cited Baseten) are starting to take a cut.

   Key moments

   One of the core tenets of the Vercel platform is that every time you deploy, you build your infrastructure from scratch, and we keep the old one around — so if the primary thing the Vercel agent recommends is to roll back, it can do that in 300 milliseconds.

   Essentially every model in the market except for Fable will do these two things: assess whether there are security problems in your source code, and write you a fix from a security report. Fable will do neither of these tasks; Sonnet 5.1 and Opus 5 will do both.

   The technology you need to do a red-team exercise and the technology you need to do a black-hat hacking attack is essentially the same. And if you can't avoid your adversary having access to it, then not giving it to the good guys is the wrong choice.

   Questions asked

   39:17What are the components of self-driving infrastructure, and how does it work?

   Ubl described an agent that reviews every production alert first, taking 30 seconds to two minutes with additional data to distinguish a real incident from a false positive (like a marketing-driven traffic spike) before escalating to a human — cutting down unnecessary 2am pages.

   43:29Why did you build Vercel this way, given the engineering cost?

   Because every deploy rebuilds infrastructure from scratch while the previous version stays live, rollbacks — even multiple deploys back — complete in about 300 milliseconds, unlike Terraform-style infrastructure changes that are hard to undo cleanly.

   44:30Why did Eve merit a generational upgrade rather than an evolution of the AI SDK?

   AI SDK is a deliberately low-level toolkit for building a custom agent harness; Eve is Vercel's higher-level framework built for the roughly 90% of agents that share a common shape, paired with the Vercel Connect product for wiring agents into GitHub, Linear, Slack, Workday, SAP, and Oracle.

   51:01What are your thoughts on the Stripe/OpenRouter combination, and what does it mean for the future of the AI Gateway?

   Congratulated the OpenRouter team, but distinguished the AI Gateway's model as strictly list-price with zero markup — unlike OpenRouter, which he said marks up through its credit system — with Vercel instead passing volume discounts to larger customers, likening the business to a retailer like Safeway.

   1:03:43What would an AI defense cloud look like, given that the Hugging Face attack gave us an existence proof of automated AI attack but not automated AI defense?

   Pushed back on the framing: argued frontier models other than Fable are already strong at defensive security work (assessing and fixing vulnerabilities), described DeepSec — his own open-source whole-repository security scanner — and said defenders currently have a real but temporary edge that has to be used now, before offensive-capable frontier models become widely available.

   1:09:16Do you think the rising cost allocated to security is permanent, and will you have to keep investing as models improve?

   Yes — DeepSec exists because standard code-review tools only scan new commits, while old code was reviewed by an earlier, weaker model, so codebases need periodic re-scanning against the current frontier model; expensive only at the scale of a large codebase, and cheap next to breach costs or a HackerOne budget.

   1:13:50Why does Vercel need a $1 million Vercel Sandbox hacker bounty if the sandboxes and agents are already good?

   Escaping the underlying micro-VM itself is unlikely; the real risk is a side channel like the one in the OpenAI/Hugging Face incident, where a model found write access to an external API — so the bounty leans toward finding ways around the sandbox's egress firewall, with payoff being ecosystem-wide hardening if something is found.

   Related

   Vercel AI Gateway pricing (no markup) ↗AI Gateway Production Index, July 2026 ↗Malte Ubl on X (@cramforce) ↗

   Full transcriptLightly edited · timestamps jump to YouTube

   37:48

   Prakash Narayanan: Up next, let's introduce our first guest for this morning: Malte Ubl, the chief technology officer at Vercel, the infrastructure platform that powers the wildly popular Next.js framework and lets developers deploy scalable web applications instantly. Long before the current AI boom, he spent over a decade at Google, where he led the engineering teams responsible for how search renders pages across desktop and mobile devices. He's also deeply embedded in the developer community as a co-founder of JSConf EU, one of Europe's most influential tech conferences. Today his core technical focus is architecting what he calls self-driving infrastructure. As AI models become cheaper and vastly more capable, the primary bottleneck in technology is no longer writing the code — it's safely running, observing, and scaling that code in production. Under his leadership, Vercel is building systems where AI agents act as autonomous first responders, analyzing live production data to automatically submit performance optimizations and security patches. This conversation is exceptionally timely given the recent launch of Vercel's always-on tracing and their new AI security harness — the industry is actively debating how to give AI agents deep access to production systems without sacrificing control, performance, or cybersecurity.

   39:10

   Prakash Narayanan: Give us a second — is he in the queue? Is he in the green room? He was in the green room just now. Oops. Nathan Labenz: I don't see him at the moment — we might have to give him a second. Prakash Narayanan: Yeah, he was in the green room, and then he dropped off. So let's give him a second. Vercel — I remember the very first days when Guillermo Rauch first put out, I think, a terminal app you could use to create a website automatically, way, way back, and I remember using it and thinking, wow, this is straight to deploy. And Malte, welcome to the show. Malte Ubl: Here it is. That was definitely on you guys — somehow, I could see you, it detected me speaking, but I think your software thought I was no longer around. But I am here. Prakash Narayanan: Amazing. So we were talking about self-driving infrastructure — can you give us a breakdown of what Vercel means by self-driving infrastructure, and what are the key components that make it possible? Malte Ubl: Sorry, you'll have to repeat that — your software was still yapping at me, asking me to do my intro or whatever. Prakash Narayanan: No worries. So we were talking about self-driving infrastructure. Malte Ubl: It's still yapping — you have a very annoying AI agent in whatever you're using for your software. Nathan Labenz: Is it maybe in another tab? No, it's all in the same tab? Okay, fascinating — maybe try a refresh. Malte Ubl: I already refreshed. Well, let's just try it. Prakash Narayanan: Okay. Malte Ubl: Let's go, guys. Okay, I'll try to hear you.

   39:17

   Prakash Narayanan: So — we were talking about self-driving infrastructure. What are the components that make it up? How does it work?

   39:26

   Malte Ubl: I think the key of self-driving infrastructure is that you assume there's an agent in the loop that can take a look at what's going on. The most vivid example anyone who's run a production system knows is the alert system — you need something that notifies you when things go wrong, and that kind of sucks, getting woken up at 2am, especially if it turns out to be a false positive. So the concrete thing is: when there's an alert on Vercel, it's the Vercel agent that looks first. And it can look for longer than just the in-the-moment alert — it can figure out what's actually going on. Are there real errors? Maybe marketing sent out a newsletter and there's a spike in interest, that kind of thing. It can make a decision over maybe 30 seconds or two minutes, with a bit more data, and then alert me only if something's actually wrong.

   40:28

   Prakash Narayanan: So is it primarily only diagnosis?

   40:31

   Malte Ubl: It's not only diagnosis — obviously it can then help with what's going on, and in particular it's really focused on getting the loop to be complete. What that means is, once you're backing your coding agent, you can also tell it: maybe there was a production incident, maybe I'm worried about performance, maybe I'm worried about conversion rates. Because the Vercel platform is really optimized to be readable by agents, you can have this entire loop where you use your own production data as something that feeds into how you later optimize your software.

   41:11

   Prakash Narayanan: So, one of the questions I have — a lot of agent work is nondeterministic, right? Especially when you have a sequence of events, and things happen in a different sequence, and everything can change based on what the agent returns on the prompt, etcetera. So when you're doing this kind of debug-causality work, how do you deal with these nondeterministic situations?

   41:29

   Malte Ubl: I think the main insight is that AI is actually exceedingly good at this — that's kind of the main innovation. If you give AI a good understanding of the meaning of the data, and give it the data that's actually relevant for the moment, it's very good at making this type of decision in the moment, especially compared to a tired human. And I think we're not in a world where you wouldn't also need someone doing operations, but we're very close to a world where a very large percentage of all operations can go through and look like this — especially on a platform like Vercel, which, in a way through luck, because we could never have known, was kind of designed for this moment. In a traditional infrastructure setup, you need something like Terraform, where you make a change and then you change production, and it's often not possible to undo cleanly, or even if you can, it takes a while. One of the core tenets of the Vercel platform is that every time you deploy, you build your infrastructure from scratch, and we keep the old one around. So if the primary thing the Vercel agent recommends is to roll back, it can do that in 300 milliseconds. It can also say: you've deployed five times since the hour was introduced, so let's go back five deployments — and still have a 300-millisecond threshold to get there.

   43:29

   Prakash Narayanan: So why did you guys build Vercel that way? I'm sure there was a lot of engineering cost.

   43:40

   Malte Ubl: By the way, your agent is still yapping at me — oh my gosh. But it's fine, I can usually hear you over it.

   43:50

   Prakash Narayanan: Okay, so — you probably had a lot of engineering cost in the early years because of those issues.

   43:57

   Malte Ubl: Yes, because of — [aside] that was to your agent.

   44:04

   Prakash Narayanan: This is the first — let me try a quick hotfix.

   44:09

   Nathan Labenz: Alright — you get the agent to be quiet, Alex.

   44:11

   Malte Ubl: Are you guys using Vercel for your platform? Prakash Narayanan: No, no, no, we're not. Malte Ubl: Yeah, this would be a good idea — what are you using? Prakash Narayanan: We're using Cloudflare. Malte Ubl: See, that's clearly the problem. Nathan Labenz: The original sin. Prakash Narayanan: The original sin.

   44:30

   Nathan Labenz: Prakash has vibe-coded this whole studio from the ground up, actually, with familiar primitives, and it's amazing how feature-rich it's become. The agent usually does a pretty good job of getting people into the studio and making sure everyone can hear each other, and usually it shuts up — so this is definitely a new issue. I guess you've done a lot, right — Vercel's been prolific over the last couple of years shipping new form factors. I built a lot of stuff with the original AI SDK, and I had some great experiences where, when I ran into friction, I could get people on X to help me out — the culture of squashing issues was very apparent in my interactions with the company. But now we have this new Eve, and I'm wondering: why did this merit a generational upgrade, going from one recommended way to build agents to a new framework? Why wasn't it an evolution, instead of a renaming and kind of a new era?

   45:46

   Malte Ubl: I think the way to think about Eve is really the insight that it was time to build something made for the type of agents that we see ourselves building, and that our customers are building, in a very high-level fashion. Really, the way I think about AI SDK — which is immensely popular, many millions of dollars a day in usage — is that it's very low-level. It's something you use if you really know the nitty-gritty details of how to build an agent; in the modern parlance, you use AI SDK to build your own harness. That's great, if that's what you want to do. But it's certainly too low-level for the vast majority of agent projects out there. So Eve is really our, I think, now pretty successful attempt to say: okay, what do the 90% of agents have in common that we see, and how can we give them a way to directly express, in a very straightforward way, how to build an agent in this manner.

   47:00

   Nathan Labenz: How do you deal with the small but important differences between different model providers? This is something that, using the AI SDK, I ran into quite a bit — Claude handles memory just a little differently, it formats tool calls just a little differently, and when it comes to structured responses, certain things are and aren't supported across different providers. This created a lot of friction for me. As you try to go up the stack and present higher and higher-level abstractions for people to build on top of, that becomes your problem. So what's your strategy for dealing with that? How do you make sure people can get the best and latest features model providers are rolling out, without overcomplicating it so much that it becomes a burden again?

   48:08

   Prakash Narayanan: Uh-oh, now I don't hear him — do you? We're having some trouble, can you refresh? Just refresh — me? No, Malte. [laughs] Never a dull moment, this is a new one — we've never seen this before. There we go, and where is he? Ah, there we go — alright, I switched to your classic version, I saw a toggle. Awesome. Alright, so did you hear my last question?

   49:12

   Malte Ubl: I did actually hear your question, yeah, yeah. So the normalization happens on two layers. Number one is in the AI SDK — you really have to think about the AI SDK these days as a software factory. It's a perfect use case for that kind of way of building software, because there's so much scope across the breadth of providers, so many micro-issues you can find through automated testing but then have to find solutions for — as you were mentioning, across how exactly tool calls are handled, and so forth. So we're doing it at that layer. The second one is that Vercel's AI Gateway is, in a similar way, working on harmonization of APIs across providers — that's definitely a constant struggle, and probably one of the biggest values our gateway adds in this space, because you can actually switch model providers and use the same APIs. If your client supports the Responses API or the legacy Completions API, etcetera, that's supported across all providers. So it's really that layer. But if you're using Eve, you don't have to worry about this at all — it's just completely not your problem anymore. You really just focus on the functionality of your agents, much of which is actually integrations with other systems, which is really what makes agents useful. That's been a key focus of Eve and its success — it joins together with a Vercel product called Connect, which makes it easy to connect to GitHub, Linear, Slack, Workday, SAP, Oracle, whatever you want to do in your company. We see a very wide usage of this kind of integration across the ecosystem.

   51:01

   Nathan Labenz: You mentioned the AI Gateway — I wanted to get your thoughts on the Stripe/OpenRouter combination, and what that has you thinking about for the future of the AI Gateway.

   51:22

   Malte Ubl: Yeah, I mean, first of all, congratulations to the OpenRouter team — good for them. I think overall it's a goal, for myself and for almost everyone in the ecosystem, that we end up in a world where there's abundant access to AI, and I want our customers to pay the lowest price possible. The way to get there is through competition — between the frontier labs, between the companies doing at-scale inference for open-weight models, and at the AI Gateway level. So I'm excited for OpenRouter — kind of speaking to the cost side, one of the principles of the AI Gateway from the start has been: we only charge list price. If you go to OpenRouter, they tell you the list price, but you're really paying in credits, and the credits are more expensive than dollars — it's a little bit of a hidden thing. In a similar way, we're excited for our larger customers to essentially participate in the discounts we get from the inference providers. People sometimes ask me, well, if you charge list price, how are you making money? And people forget that a business like Safeway exists — you buy the chips at list price, but obviously Safeway didn't buy them at list price. This is essentially a retail business, and I think it plays an important role in the ecosystem to make it really easy to get access to models. Another thing we see quite a bit: Vercel has a bunch of startup customers, and their customers really value having day-zero access to models, and that can be really, really difficult — Nathan, you were actually saying it can be annoying to support a new model, you have to test it, there might be little things that don't quite work even if the model itself is good. So having day-zero support in the AI Gateway is something that really helps folks say, this is not a problem I have — it's someone else's problem. We have a team full-time grinding on onboarding more providers, more models, every day.

   53:58

   Prakash Narayanan: So let's speak to that a little bit. One of the experiences I've had dealing with models on OpenRouter, for example, is that when you switch inference providers you often get some brittleness. In one instance, I had an inference running through a logical sequence of steps, and I expected it to run the steps as per the prompt — but what ended up happening is the inference ran out of step, the ending answers got generated first, without the context of the beginning answers. The only way to fix that was to number the JSON schema, because it seemed like one inference provider was going in alphabetical order of the schema rather than the order it was given — a very odd reaction. So there's this brittleness between providers. How do you manage that kind of complexity when you're trying to provide a consistent experience across multiple providers at the same price?

   55:18

   Malte Ubl: Yeah, it's grinding — and, which is the perfect startup, right, because this is the most horrible thing you could possibly imagine wanting to do yourself. It's very, very annoying work. And, again, building a software factory can really help, because it's annoying work that's also very automatable in an agentic world. So this is what we see people use Eve for a lot — to build these kinds of custom software factories, where you can say: I can express, in my own problem space, how I identify problems of a certain shape, I have a playbook for them, I execute them through my software lifecycle, I have a testing scheme for them, and then, ideally, can almost automatically roll them out. So yeah, these issues exist, and the AI Gateway team very much takes it as their responsibility that these issues aren't visible to our customers — it's a lot of work. And there are trade-offs: on a not-so-popular model, maybe there are three to five providers; on a very popular model, there are ten. But if you have twenty, and some random startup that bought five H100s isn't really an inference provider — if you add them, can you really make sure that across fifty models they're all doing a good job? That can be difficult, and then there are choices to be made in that marketplace: do you take on that vendor, or decide ultimately quality is more important, and you can't guarantee it beyond a certain number of providers?

   57:16

   Nathan Labenz: I saw a comment the other day — all of my neocloud friends are basically neoclouds with a layer of software on top. How would a new provider onboard to Vercel? What would that process look like, to become an inference provider?

   57:33

   Malte Ubl: I mean, ultimately this isn't very complicated — you have to somehow find us, which isn't very difficult because we're all ultra-online on X. Then you go through our procurement process, the APIs are all pretty standardized, and then you go through our qualification process, and we'll give you feedback where things aren't acceptable in some way. In many ways it's a relatively standardized process, given that there's already a large number of providers. There's a lot of value in having this marketplace be large, because that incentivizes each of those folks to come down on the prices they list as required for a model.

   58:22

   Nathan Labenz: I want to dig in a bit more, to the degree you can share. I'm really curious about the deal dynamics and who has what relative market power. You had a blog post a couple weeks ago saying DeepSeek had surpassed Gemini in number of tokens, to become the number-two provider by token volume. Anthropic still has, I think, the majority of the revenue, although not a majority of the tokens — my guess is Anthropic isn't cutting you any deal, and you're not making much money on that traffic, tell me if I'm wrong. But then there are others — quite a few companies in the chain. You've got the model maker; if it's a Chinese company, the models are actually served by inference providers, your Fireworks, your Togethers, and more; then there's this central access point that's the Vercel Gateway. Who needs whom worse, and how are people structuring deals to reflect the value and market strength each party has?

   59:35

   Malte Ubl: Yeah, I mean — I don't love it, but this is a really good question, because there's such intransparency in this space, and everything's covered by NDAs, etcetera. So it's essentially impossible to talk about any given deal in this space.

   59:59

   Nathan Labenz: Can we leave names out and give a general market structure?

   1:00:03

   Malte Ubl: Yeah, no — I think it's generally true that it's honestly not a very complicated market. There are suppliers who value ARR, and hence they do deals like everyone else has always done on large enterprise SaaS deals — if you commit dozens of millions of dollars a year, you have expectations of some discount; if you do six figures a year, there's also some expectation of a discount, but lower. So those dynamics are actually quite similar. But if I'm an end customer — say, a Vercel AI Gateway customer — and I want to do some amount of volume on Kimi K3, realistically I need multiple vendor relationships, because in the current state I can't bet everything on one vendor — the reliability of all these vendors is abysmal, it's such early days. So I always need multiple vendors to supply my demand, which means, if I'm doing it myself, I essentially need to buy double, which means I have to cap my commit, and that reduces what I can negotiate for. That's why I think it's advantageous to go through something like the Vercel AI Gateway, where you have a single vendor you buy from, you commit the whole, and then there's this layering effect. At this moment, this is almost necessary, given the current state of the technology, which is still very unreliable.

   1:02:19

   Nathan Labenz: Does the gateway handle fallbacks for people — like, if I make a DeepSeek call and it's originally going to Fireworks, and that errors, they're down, will you reroute me to Together, and so on?

   1:02:33

   Malte Ubl: Absolutely, yeah — that's a key value. In fact, we can look at what other vendors do in this space: you can bring your own key, and we don't charge you at all for your primary requests. So let's say you do a single deal, you have your Anthropic commit, and you go through the Vercel AI Gateway — you don't pay us at all for that traffic. But if, say, your Anthropic-on-GCP contract has downtime, we'll auto-reroute you to Anthropic on Bedrock, and you'll then pay for that traffic, but only that traffic — and that's, let's say, a few days of downtime a year right now, in total. So in that case you can really see the gateway as this layer that adds stability to your system, and cost only happens when the AI Gateway is actually working for you.

   1:03:43

   Prakash Narayanan: Let me switch gears a little bit to a topic that's been on all our minds the last couple of months — security. Especially post the Hugging Face attack, where I think the postmortem was that we now have existence proof of automated AI attack, and we don't have existence proof of automated AI defense. Offensive security has become extremely cheap with open-weight models, and the frontier models that get deployed are often deficient in addressing security, for various reasons, including AI safety. What would an AI defense cloud look like? What does this kind of active AI defense for security look like?

   1:04:37

   Malte Ubl: Yeah, I published a blog post on this last week — maybe somewhat negatively titled 'Everything Hackable Will Get Hacked.' It is an extreme moment, but I'll push back on what you're saying, because I think there are two common misconceptions. First: I don't think it's actually priced into the market yet how good — especially — Kimi K3 is at offensive cybersecurity, and that it has no safeguards. You can use it for red-teaming, you can use it for black-hat offense — that's the thing today, and it's remarkably good, you can try it out. It's extremely well-trained on this: it has a process, it'll probe the system, it'll quickly know your system better than you within minutes, and then it'll try everything to get through the defenses, in a way where you can really see the model's been specifically trained to be an offensive attacker. That's part one. The other part: it's just not true that off-the-shelf frontier models aren't good at cyber defense — that's also a common misconception. It comes from the fact that there was this 'Mythos' thing that Fable shipped, unshipped, and then shipped back with almost unusable cyber-defense detection and shutdown behavior. But Sonnet 5.1 doesn't have this — Sonnet 5.1 is absolutely perfectly usable for a certain type of defensive security, and this is also true for Opus 5. Essentially every model in the market except for Fable will do two things: A, if you give it your source code and tell it you're the owner of the system, you can ask it whether there are security problems in the code and it'll do that; B, if you give it a security report, it'll write you a fix. Fable will do neither of those tasks; Sonnet 5.1 and Opus 5 will do both.

   1:06:52

   Malte Ubl: One of the things I've personally been working on is our software called DeepSec, an open-source project that does whole-repository scans for security vulnerabilities — and I just think everyone needs to run this, it works really well, and it prepares you for a world in which defense is very important. Right now, as a defender, you have a benefit, because you can use the frontier model that will do defensive tasks even though it won't do offensive ones. But you really have to hit the moment here, because Kimi K3 is already really good — it's easy to imagine that within six months at the latest we'll have Fable-class models that do offensive security. So you have to act defensively now, so you're ready when that happens. Are there guarantees it'll work? No — but being able to do something today is really key. We've already mentioned 'software factory' a few times today — we're heavily investing in not just having DeepSec as a discovery tool, but completing the circle. It's absolutely correct that, through these AI discovery mechanisms, the number of issues identified is exploding, so I have to automate the whole SDLC path — fixing it, rolling it out, being certain I'm not making things worse, and so forth. I think we, and the industry, have a lot of work to do here, but it's not a hopeless situation — people underestimate how much they can actually do today.

   1:09:16

   Prakash Narayanan: Just one follow-up on that — it sounds like you've seen costs allocated to security increase. Do you think that increase is permanent going forward? As models improve, will you continually have to keep investing in that space?

   1:09:35

   Malte Ubl: Yeah, exactly — and I think it's a good example of why DeepSec exists. When I was trying out code review tools, like many folks, I pushed a PR and the tool said, well, you have a security issue, and I was delighted. As you do as a CTO, I said, okay, this is great, can you run it on my whole codebase? And the team said, no, it only runs on the new code — but I have millions of lines of code. So that's why I made DeepSec: it's basically the same thing as a code review tool, just running on your entire codebase, which, if the codebase is big, costs tens of thousands of dollars. But compared to, for example, our HackerOne budget, it's nothing — it's cheap, the cheapest thing I've ever done in my life, compared to the extreme case of a security breach. And even if you have incremental code review, which you absolutely should because it's great — the code you wrote three months ago was reviewed by your code reviewer, but that code reviewer was probably Opus 4.7 or whatever, some earlier model, not the latest frontier model. So you have to rerun these things on your codebase on a regular basis. I don't see a path out of that loop, but I think it's a good investment — spending tens of thousands of dollars only happens if you have a very large codebase, which probably means you have good revenue. And if you look at how much companies are already spending on security, it's a really, really good investment.

   1:11:26

   Nathan Labenz: You mentioned Kimi K3 being quite good at offensive cyber and having no guardrails. I took a trip to China this summer, and one of the takeaways from talking to people there was that the way the Chinese government seems to think about regulating AI is more at the system or product level than at the model level — they're comfortable saying, your model itself is one thing, but you're going to put it into a product, there's going to be an API, maybe monitors on top of that, and that's what they concern themselves with in their domestic market. I don't know to what degree inference providers taking models like Kimi K3 are adding guardrails, classifiers, monitors to try to rein in what the raw model will do. What do you think their responsibility should be? If you're an inference provider in the US deploying Kimi K3, how much responsibility do you have for what people do with it?

   1:12:38

   Malte Ubl: So I do think a certain amount of KYC is important, but I think the reality is — especially as it becomes more possible for folks to run these things themselves — you always have to assume there's an attacker who has access to this technology. Unfortunately, the technology you need to do a red-team exercise and the technology you need to do a black-hat hacking attack is essentially the same. And if you can't avoid your adversary having access to the technology, then not giving it to the good guys is the wrong choice. I'd love it if we could somehow put the genie back in the bottle, but that's not the world we live in. So we have to be realistic about it, and that means making playing offense part of the defensive game.

   1:13:50

   Nathan Labenz: You're putting your money where your mouth is with a $1 million hacker challenge for Vercel Sandbox — this was recently announced. For an ignorant person like myself, who would have thought sandboxes were less hackable than they seem to be, until recent revelations updated my thinking — give us a little background on why sandboxes are still so hackable here in mid-2026, and why aren't the agents enough? Why do you need a million-dollar bounty on it, given how good the agents already are?

   1:14:33

   Malte Ubl: Yeah, I mean, nothing goes beyond human creativity, right — so obviously we have agents on it, but we've kind of exhausted our own agentic research, and we're not only doing the challenge, we also have multiple pen-testing teams working the same problem — both the crowdsourcing and the things we can influence directly. I think sandboxes are sometimes seen in this oversimplified way, like, oh, it's a Linux micro-VM, they're inherently secure, what do you need on top? But if you look at how, for example, the OpenAI Hugging Face incident worked — the AI model found a side channel to get essentially write access on an external API. So part of our Sandbox offering is an egress firewall that makes it as easy as possible, at that egress layer, to minimize what can be done. Part of the million-dollar scope isn't just to escape the micro-VM, which I'd be very surprised if that happens — though it would obviously be amazing if someone did, so we could get it patched upstream and fix essentially every similar sandbox in the ecosystem — pretty unlikely.

   1:16:03

   Malte Ubl: More likely is that someone finds a way to circumvent the egress firewall, or finds a side channel in some service that's always callable from the sandbox. There's ultimately a lot going on, because we're adding functionality to these systems and you have this infinitely creative adversary — you have to really think outside the box and not just accept or hope for the best. I think we're past hoping for the best, and that's why this is a very appropriate challenge. We've received a large number of submissions — AI is helping there. Right now we're feeling really good about what we've seen so far, both in the sense that people find some kind of edge case, and it's fine, we close it, but there's still time for folks to find something really valuable. And that will be our contribution to the ecosystem, to make our customers substantially more secure — which includes companies like Meta, includes companies like Notion. The value for the ecosystem would be substantial, to make sure our systems are truly secure.

   1:17:29

   Prakash Narayanan: When you see a new model release like Kimi K3, what's the lag time from that release to you seeing it start being used for attacks against you? On the adversarial front — some people say the large hacking groups are like McDonald's, slow, they adopt technology as slowly as any typical large organization; others say these guys use the latest models immediately. How do you see that play out on the adversarial front? What do you see?

   1:18:15

   Malte Ubl: Yeah, I mean, it's not really possible to detect what people are using to attack you. What would be really fun is maybe to try to ascertain this from, for example, our HackerOne submissions — besides the sandbox challenge, we have a permanent HackerOne program where folks submit reports, and while certainly not all of them, obviously those folks are more and more using AI to support their work. I don't have any data on it, but maybe we could start collecting the models people are using — it'd be a really interesting question. My assumption is that the turnaround is really, really fast, because — you guys know this — once you have access to a model, it's very, very simple to start using it.

   1:19:05

   Nathan Labenz: Well, time flies, and it's the scarcest resource in the AI game, so we'll let you get back to work. Sorry for our agent's yapping incident at the beginning, which stole a couple minutes from us. Anything you'd want to leave people with? We didn't even touch on v0, not sure that's the thing — but anything we didn't get to that you'd want to make sure people are aware of before we break?

   1:19:29

   Malte Ubl: No, we got through — we talked about Eve, which people should give a try, lots of love from the community. We talked about DeepSec, which I already mentioned — maybe there are other tools like it, but if you're not running one of them, I think that's actually a problem. What we didn't touch on is our new experimental coding harness — it's more on the experimental side, not that it's unstable, but it's a new take on a coding harness, written in Zig, fits on two floppy disks, and compiles and runs absolutely everywhere. It's a really lightweight new take on harnesses that I think folks really love. But again, that's more for the experimentally minded. Thank you, guys.

   1:20:18

   Prakash Narayanan: Thank you. Nathan Labenz: Thanks for being here. Malte Ubl: Great to meet you.

   1:20:22

   Nathan Labenz: That idea of a harness that fits on two floppy disks — that is signature Gigerbo, I'd recognize that fingerprint anywhere.

   1:20:35

   Prakash Narayanan: Fascinating guest — I definitely learned more about Kimi K3 than, you know, it's very different hearing it from someone who's deeply technical and has to face the problem every day, versus just reading a tweet from someone. It's such a differentiator.

   1:20:57

   Nathan Labenz: Yeah, I thought the analysis — even though it was a bit guarded on the details of how these deals get set up, and why people want to go through an intermediary rather than direct to inference providers — was also pretty interesting. I'm really curious what the margins and rev-share look like, how desperate some providers may be, and, in some cases, whether the model makers themselves are also getting a cut — I know Baseten is starting to do that, but I'm not sure how many are. We should maybe have a tip line for people to send us more inside information than folks like Malte are able to share publicly, but I still thought that was quite interesting. We're running a couple minutes behind, so let me not delay us any longer.

   1:21:46

   Prakash Narayanan: Let's...

   Clips

   • Clip 01

     Everything Hackable Will Get Hacked

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 02

     Offense Is Part of Defense

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 03

     Rollback Is the Agent’s Superpower

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 04

     Sandboxes Need More Than VMs

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 05

     AI Gateway Fallbacks Buy Uptime

     0:00 / 0:00

     1x
     ↓Download↗Share clip
3. 1:18:15Interview52 min

   Interview: Louis Kirsch and Damon Falck — Training an AI ScientistLouis Kirsch and Damon FalckDisclosure: Nathan Labenz is an investor in Inherent Laboratories, personally and via a16z's Scout Fund IV, and said so on air. Inherent's co-founder and Chief Superintelligence Officer and the first author on Faraday explain a 27-billion-parameter research agent that directs GPT-5.5 Codex as a tool, and why the size is the least interesting part. Because science has no verifier — Falck's framing is that science is inherently non-verifiable — they judge whole research trajectories and attribute credit across steps rather than grading a final answer, then test whether that reward signal tracks anything real. They discuss the rare cheating their LLM judges catch, what scientific intuition means if it is mostly about shrinking the search space, whether recursive self-improvement applies to a lab rather than a model, and how you would recognize escape velocity by watching a system's higher-order rates of self-improvement.

   Watch

   ▶Play this segmentYouTube · from 0:00Open segment on YouTube ↗

   As aired

   Nathan Labenz opened by disclosing that he is a small personal angel investor in Inherent Laboratories (and via a16z's Scout Fund IV) — "technically you can consider me conflicted" — before turning to a wide-ranging conversation with co-founder Louis Kirsch (Chief Superintelligence Officer, ex-Google DeepMind, PhD under Jurgen Schmidhuber) and Damon Falck, a member of technical staff and the paper’s first author (AI safety background from the MATS program and Oxford). Prakash Narayanan introduced the company's May stealth launch and its 27-billion-parameter research agent, Faraday, which the hosts said beats frontier models including Claude Opus 4.8 and GPT-5.5 Codex at reproducing complex research papers.

   Kirsch framed Inherent's founding idea — recursive self-improvement of an entire organization, not just a model — as something that started as a purist vision ("the machine that improves itself so humans can step out of the picture") and evolved into a human-machine teaming thesis: humans still drive most AI research today, and Inherent is building toward a state where machines and humans collaboratively get faster together, rather than a single moment where the system goes fully autonomous. Concretely, he said the company already has prompts and agent harnesses that change themselves day to day, both from the agent's own ideation and from what humans discuss internally, while training the underlying model is a much slower, longer-horizon loop.

   On Faraday's architecture, Kirsch and Falck explained the deliberate separation of a 27B "scientist" model from a much larger coding model (GPT-5.5 Codex), with Faraday doing the scientific reasoning and handing off implementation. Kirsch said starting small was a practical choice for a young lab with limited compute, and that scientific behaviors — like reasoning about which experiment to run next — are already emerging at this smaller scale through reinforcement learning, suggesting scale isn't strictly required for early scientific competence. Falck added that keeping the "scientist" and "coder" roles separate lets Inherent ride advances in frontier coding agents built by others rather than having to build one itself.

   A large stretch of the conversation focused on reward design and trust. Falck was direct that "science is inherently non-verifiable" and that Inherent's reinforcement learning relies on judging entire research trajectories and attributing credit to individual steps, not just scoring a final output — plus correlating the resulting reward signal against human judgment. Kirsch said the model does occasionally exhibit reward-hacking-style behavior — for instance, deliberately pulling a result off the internet rather than reproducing it, or trying to fake a plot — but that these are rare and are caught by LLM-based judges built specifically to penalize that behavior rather than by optimizing a single "feeble" scalar reward. Both guests said they do not currently apply direct optimization pressure to the model's chain of thought, cautioning that doing so "can be problematic." Neither offered a formal safety case for eventually handing ML research fully over to AI; instead, both described a deliberately incremental human-in-the-loop approach — dashboards, fast pause mechanisms, and a standing communication channel with the system — as the actual answer for now.

   On what "scientific intuition" even means, Falck argued it's largely about reducing search space — knowing the right questions to ask — while Kirsch was skeptical that exhaustive Monte Carlo tree search over a space of options is the right long-term mechanism, and said Inherent is instead trying to train models toward a more human-like, incremental betting-and-learning loop (citing meta-reinforcement learning and maze-navigation analogies). Asked about other data modalities — Nathan raised a recent guest's finding that DNA-sequence models spontaneously learn tree-of-life-like geometric structure — Kirsch said Inherent expects to keep language as its core representation for the generalist scientist, while potentially building separate specialist models for other modalities that feed insights back into the central system.

   Kirsch also described a rough, non-single-moment way of measuring progress toward "escape velocity" for recursive self-improvement: watching higher-order gradients of the system's self-modifications — whether it's learning at all (first derivative), and whether it's learning to improve its own learning algorithm (second derivative) — rather than expecting one discrete threshold to cross. On life at Inherent, both guests described a culture explicitly built around adapting to agents as coworkers (Kirsch: "we're living in the experiment"), with context constantly fed to Faraday and a stated belief that unscripted human water-cooler conversation is a uniquely high-value data source. Falck said he personally resets by getting outside; Kirsch echoed that he does his best strategic thinking away from screens, in nature, and said he'd tried but not found much personal value yet in exhaustively quantifying his own biology.

   Prakash closed with a question about compute-buildout constraints (citing rough 2030 data-center and copper-supply figures as a reason some AGI timelines require unproven technology) and about mathematical verification — noting that AI-driven math discoveries are already outrunning the number of humans qualified to check them, pushing the field toward formal tools like Lean. Kirsch's answer to both was the same throughline: Inherent is betting that a system built to explain its reasoning and take humans along on the journey — rather than one that simply hands over inscrutable results — is the actual path to trustworthy, faster scientific progress. The interview closed on good terms, with Prakash hoping the team gets to "recursive self-improvement, but safely."

   Key moments

   It's not that it jumps to the kind of cheating behaviors you've described. In very rare cases, we have seen it deliberately go to the internet and try to download the final result, or try to mock the plot.

   Science is inherently non-verifiable, and providing a high-reliability reward signal has historically meant some kind of proof or test — something verifiable. We don't think we can keep doing that if we're trying to discover these stepping stones and do open-ended research.

   If the first derivative is 0, the system isn't learning anything. If the first-order gradient is positive, the system starts learning something. If the second-order gradient is positive, then it figures out how to improve the learning algorithm itself.

   Questions asked

   1:23:50What does it mean to have a recursively self-improving organization?

   Louis Kirsch said he originally imagined recursive self-improvement as building a machine that could improve itself entirely without humans, but in practice, right now, it's mostly humans driving AI research. Inherent's actual goal is a transitional organization where both machines and humans improve each other collaboratively, getting progressively faster at solving scientific problems together.

   1:25:49Which parts of Inherent's research process currently improve themselves, and which still depend entirely on human researchers?

   Louis Kirsch said very few things are entirely self-improving; Faraday is already woven into much of the software stack, infrastructure, and research-idea generation, but Inherent deliberately avoids drawing hard boundaries — the goal is an organization where the automated system is present everywhere in the process while still collaborating with humans.

   1:32:23Why build a 27-billion-parameter model as the 'scientist,' rather than a much larger frontier-scale model?

   Louis Kirsch said starting small was partly practical — a new lab has less compute and needs faster iteration cycles than training a giant model — but also that scientific behaviors, like reasoning about which experiment to run next, are already emerging at 27B (Qwen3.6-based) through their RL training pipeline, suggesting scale alone isn't the bottleneck for early scientific competence.

   1:36:58How confident are you in the reward signal — do you have any real assurance you're rewarding the behavior you actually want?

   Damon Falck said science is inherently non-verifiable, so Inherent can't rely on the kind of provable/testable reward signal RL traditionally wants. Their approach instead scores the whole research trajectory and attributes credit to individual steps rather than just the final output, plus applies stabilization techniques and checks that the reward signal correlates with human judgment.

   1:39:33Is the model's chain of thought considering cheating, and has narrow scientific training made it useless for other kinds of tasks?

   Louis Kirsch said the model largely follows a legitimate replication process, but in rare, filtered cases it has deliberately tried to download a result off the internet or fake a plot. He said that behavior is caught and penalized by dedicated LLM-based judges — a deliberate alternative to optimizing a single, 'feeble' scalar reward, which he said is what actually invites reward hacking.

   1:42:54Do you apply optimization pressure directly to the model's chain of thought?

   Damon Falck said no — not in the published work — and that doing so can be problematic. He said the real answer to eventual trust-handoff isn't a formal safety case yet, but a team commitment to human-machine collaboration; Inherent does not expect to ever hand research fully over to the agent without human involvement.

   1:47:35What is scientific intuition, and which specific skills — hypothesis selection, experiment design, anomaly recognition, model criticism, search efficiency — actually improve during reinforcement learning?

   Damon Falck said intuition is largely about reducing search space, i.e. knowing the right questions to ask ('taste'). Louis Kirsch added that he's skeptical exhaustive Monte Carlo tree search is the right long-term mechanism, and that Inherent is instead training models toward a more human-like, incremental betting-and-learning loop, drawing an analogy to meta-reinforcement learning on maze-navigation tasks.

   1:56:19How do you measure, as an engineering threshold, how close you are to true recursive self-improvement ('escape velocity')?

   Louis Kirsch said there's no single moment to cross — instead Inherent looks at higher-order gradients of the system's self-modifications: whether it's learning at all (a positive first derivative), and whether it's learning to improve its own learning algorithm (a positive second derivative). He said this has been happening for years but is becoming more interpretable and higher-impact over time.

   2:12:15Can a human verify AI-generated discoveries they can't actually understand, especially as math increasingly relies on formal verification tools like Lean?

   Louis Kirsch said it shouldn't be a passive process where the system produces an inscrutable proof that humans then painstakingly decipher. Instead, the system should be trained to explain its reasoning and take humans along on the journey of understanding the math or science, which he called the actual path to the fastest progress.

   Related

   Training AI Scientists to Replicate Research (arXiv 2608.13331) ↗Inherent Laboratories — research writeup ↗Louis Kirsch on X (@LouisKirschAI) ↗Damon Falck on X (@DamonFalck) ↗

   Full transcriptLightly edited · timestamps jump to YouTube

   1:21:46

   Prakash Narayanan: ...next guest for this morning, Louis Kirsch and Damon Falck, who co-founded Inherent Laboratories to build artificial intelligence systems that can conduct genuine scientific research entirely on their own, rather than simply acting as highly advanced chatbots for human engineers. Louis brings a deep pedigree in autonomous systems — he previously worked on automating AI research at Google DeepMind, and completed his PhD under the legendary AI pioneer Jurgen Schmidhuber, focusing on the foundational mathematics of systems that can rewrite their own code. Damon brings critical experience in AI safety and alignment from the MATS program and Oxford University, where he specialized in understanding how highly capable AI models might deceptively resist their own training. This past May, Inherent Laboratories emerged from stealth with a $50 million seed round to build a completely new kind of research institution — one that recursively improves its own hardware and software while strictly keeping human scientists in the loop. Just last week, the lab released Faraday, a 27-billion-parameter agent that learned scientific intuition on top of standard coding tools. Faraday currently beats today's absolute biggest frontier models, including Claude Opus 4.8 and GPT-5.5 Codex, at reproducing complex real-world research papers. Their work sits at the absolute bleeding edge of turning AI into a genuine discovery engine, which is why we have them here today to discuss their launch, the limits of algorithmic reasoning, and the very real geopolitical risks of building machines that can continuously upgrade themselves.

   1:23:36

   Nathan Labenz: You're gonna get Schmidhuber-ed for mispronouncing his name, I'm afraid, but maybe we can survive that thanks to our relative obscurity. Welcome, guys.

   1:23:47

   Louis Kirsch: Hello. Hello. Good to see you. Good to meet you.

   1:23:50

   Nathan Labenz: Yeah. Likewise. I guess pretty inconsequential disclosure — I am a very minor angel investor in Inherent. So, technically, you can consider me conflicted, but I was mostly just really interested, when I first learned about the company, in the cultural ideas that were kind of percolating at that time, and I'm sure they've taken more shape as the team has grown and you've actually gotten to work. We'll definitely spend some time on the AI scientist work too, but I'd love to start with the culture. What does it mean to have a recursively self-improving organization?

   1:24:33

   Louis Kirsch: Yes. That's an excellent question. So I've been spending many, many years on the concept of automating AI research and recursive self-improvement. And for the longest time, I thought about it as: we're building the machine that recursively self-improves itself so humans can just step out of the picture entirely and let the thing improve itself. And that's gonna be sort of a point in time that's not too far away — we just have to figure out the algorithms to make that happen, and the machine just keeps going. And after a while, I realized in actual reality, right now, it's mostly humans driving AI research, but we want to go to that transition where more and more can be automated, and a lot harder scientific questions can be answered with the help of AI. It's not gonna be an immediate transition — instead, we're gonna have to build an organization where, because we self-improve, there are both machines and humans in this construct, and collaboratively we are going to improve each other and really become faster and faster at solving scientific problems.

   1:25:49

   Prakash Narayanan: So a laboratory can improve its prompts, tools, datasets, evaluations, model weights, compute allocation, experimental procedures, research agenda — there are so many things a laboratory can improve. Which parts of Inherent's research process currently improve themselves, and which still depend entirely on human researchers?

   1:26:13

   Louis Kirsch: Yeah, it's a great question. Entirely, probably very few things. If we think about the software stack that makes up a company — the infrastructure that drives all the experiments we're running, the discussions we're having with each other and how they drive our research ideas, as well as the steps we take to form the plan and start the implementation — the system we've built, called Faraday, is a big part of many of these things at the company already. It's a bit hard to draw a hard boundary where we can say this is completely because of self-improvement. And that goes back to the point I made earlier: I don't think we want to make these hard boundaries in reality. Instead, we want to build an organization that has the automated system, Faraday, in the process everywhere, but still leverages humans and collaborates with humans across all of these different aspects.

   1:27:21

   Nathan Labenz: Can we be a little more concrete, or give some tangible examples? Prakash and I kind of conceived of this show, to a degree, as an experiment in personal recursive self-improvement. And, for us, that kinda means at the end of every show we talk to each other about what went well, what didn't go so well. We prompt our agents to try to figure things out. Sometimes a guest gave us a good idea — which I'm hoping you're about to do — and we'll just say, hey, look at the transcript and consider those ideas and think about how we can implement them. Not sure we're hitting takeoff just yet in our personal experiment, but what are some — either skills that you've created for the AIs to run, or habits, or team practices — that we might be able to adapt to our own projects?

   1:28:16

   Louis Kirsch: Yeah. So when we talk about Faraday, you can think of it as, like, 2 fundamental pillars for how it self-improves. 1 is about automated prompt creation and automated harness creation, and the other is about model training — we're big believers that, ultimately, to build a good scientific agent, we need to train these models as well. A lot of the loops we've built in the company are already working quite actively on a day-to-day basis — harnesses are changing themselves, prompts are changing themselves, through the ideation of the agent itself but also through the ideas and discussions we have in the company between humans. So these changes we see day to day, where the system decided to change the prompt based on what humans have been discussing. Whereas the model changes — that's a bit earlier, where models are starting to train another model in interesting ways. But, obviously, the time horizon on which these systems have to operate is a lot longer, so the feedback cycles are a lot slower. Hence why measuring that, and really making it effective, is still a big project we're working on.

   1:29:43

   Prakash Narayanan: 1 of the things I've noted, from my read, is that Faraday directs several other models, including some that are larger than Faraday itself. And I think 1 of the constant debates in the ML field is whether a smaller model can effectively drive a larger model, or whether the smaller model breaks down and you effectively need a larger model to drive smaller models. This has been the whole question of the auto-router, and whether auto-routing will actually work out, or whether it doesn't work because the smaller model can never capture the entire complexity of the question and of the field. We've also seen, over the past couple of months, people using Fable — especially when they run out of credits — to orchestrate other, smaller models, to kinda save tokens: they tell Fable to use Sonnet, they tell it to use other things. What has your experience been with using a smaller model to drive a larger model versus a larger model to drive smaller models? Have you tested both strategies, and how did that work out?

   1:31:03

   Damon Falck: I think we're in the business of building generalist scientific agents. And 1 of the core contributions of our first work has been the separation of the scientist from the coder. Right now, Faraday — the model we talk about in the paper — is, as you say, a 27B model driving a much larger model: Faraday does the scientific work and hands off the implementation work to GPT-5.5 Codex. But in the future, this ratio could be very different — we don't know, we'll have to see. 1 of the fantastic things about doing it this way, though, other than it being a natural separation of concerns that human researchers and engineers already have, is that we don't have to worry about building frontier coding agents — we can make use of all the advances coming in those and focus on building scientists ourselves. But these are great questions, and ones we'll keep exploring in our own research: which should be the bigger models, which should be the smaller models, how should the interaction look.

   1:32:23

   Nathan Labenz: How did you decide to have a 27B model be the scientist? I mean, the prior expectation on this, I think, would be pretty uniformly that you'd want to go to a trillion-parameter model to have the best internal representations, the most sophisticated understanding — the big-model smell — and yet you're doing it at 27B, based on Qwen3.6, which I think has a pretty good reputation. But still, at 27B, I would just think you'd be fundamentally very limited compared to something much bigger.

   1:33:01

   Louis Kirsch: Yeah. We've been building this new company, Inherent — and, of course, you're right, from the outside, 1 might say: well, let's build the best scientist, let's start training with a big model straight away. But, of course, in reality, when you train a big model, you need a lot more compute resources, and you need to iterate over much longer time horizons. So it's quite natural for a new lab to start at a smaller scale first and then scale up the ladder. And the interesting insight we've had is that, by setting up this pipeline of training an agent to be a better scientist through reinforcement learning, already at the smaller scale we're seeing really interesting capabilities — models doing more scientific behaviors, such as thinking very hard about what is the right experiment to run at this point in time to prove out what a paper has done, and doing that in a way that does the paper justice but doesn't require lots of resources. These things already emerge in these, arguably, smaller models, which I think is an interesting indication that maybe we don't need massive models straight away to do all these things. And there's an interesting aspect to this kind of separation of concerns — we don't need to reinvent the wheel and build just another coding agent, and can think about scientific capabilities and coding capabilities as 2 important but perhaps separate capabilities.

   1:34:38

   Prakash Narayanan: I'm convinced that AI for science is going to be tremendous, but I often have to deal with skepticism from people in science, especially in bio — they're usually very negative. 1 of the criticisms I've heard is specifically: if you have next-token prediction and you're doing paper replication, well, then the agent kind of has a destination in mind, and it's evolving toward that destination. Meanwhile, in science, a single negative result disproves the hypothesis — you're not actually looking for that kind of predictable next-token prediction, you're actually looking for outliers, for the moment where this hypothesis is wrong and therefore we need something new. So how do you address those kinds of concerns from scientists in the field?

   1:35:43

   Louis Kirsch: Lovely question. I think there are 2 parts to this. 1 is that I do think scientists have an outcome in mind in many cases when they work on something, but it's not necessarily the outcome that's actually going to happen. I have a mental model of what I want to build next — which, in my case, because of self-improving agents, I think would be really capable, and how it should be built: should it be operating on code, or should it be operating in white space? So I'm going on that journey toward the goal I have in mind. But it might very well happen that I discover something interesting else on that path, and that in itself could be a useful building block, either for my own work later on or for other people's work — a stepping stone, as we often call it in the open-endedness community, where lots of interesting discoveries lead up to more discoveries later on, without necessarily having that specific goal in mind upfront.

   1:36:58

   Nathan Labenz: You mentioned training with reinforcement learning for scientific skills. Obviously, we've seen examples recently of how, when the RL signal isn't particularly clean, we can get all kinds of crazy downstream behaviors. So I've got a few questions on this, but the first is simply: how confident are you in the reward signal you're able to give the model? I'm guessing we don't have anything approaching formal proof that you're rewarding the behavior you really want to reward — so what precautions are you taking, and how confident can you be that you're actually rewarding what you intend to be rewarding?

   1:37:52

   Damon Falck: I think this is a great question, and it gets to some of the core difficulties of this kind of endeavor. Science is inherently non-verifiable, and providing a high-reliability reward signal has historically meant some kind of proof or test — something verifiable. And you're completely right that we don't think we can keep doing that if we're trying to discover these stepping stones and do open-ended research. In the paper we published, we found some particular solutions to this — some of it involves looking at the entire trajectory and the process the scientist is going through, rather than just the final output. Some of it involves attributing credit back to individual things the agent did in the trajectory, and then there are some more technical stabilization techniques we have to use as well. But the core questions are how to reduce the variability of the reward signal, increase the density of the signal, while still preserving this property of assessing the right thing. And we did a bunch of work correlating our signal with human judgment, trying to understand how much it corresponds with human taste — but this is work we'll keep doing for sure in the future.

   1:39:33

   Nathan Labenz: Can you describe what the model is like in a qualitative sense? For example, when you read the chain of thought — I just went down this rabbit hole with Bronson Shane from Apollo Research, who's read an ungodly amount of GPT chain-of-thought. 1 thing he observed was that the models are basically always at least considering cheating, in a very high fraction of cases. So what do you see — is yours considering cheating? And then also, in terms of what it can do: now that it's been so focused in on science, is it useless for other kinds of things? If I ask it a friendly chat or companionship question, does it only see the world through the science lens, or how much of its breadth is still retained after going through this training?

   1:40:34

   Louis Kirsch: Yes. That's a great question. So I would say the model does focus on the scientific questions we ask it. And when we looked into the process — the thinking patterns and the actions it takes — it's not that it jumps to the kind of cheating behaviors you've described. In most cases — we've done some filtering — and every once in a while, but in very rare cases, we have seen it deliberately go to the internet and try to download the final result, or try to mock the plot. But it is the case that, through both the priors encoded in the large language model and the prompt we give it to actually follow the replication process in a scientific way rather than just mock the results, the kinds of behaviors it exhibits — even before training, in its explorative behavior — follow roughly the lines of a scientific process of replication. Then, for reinforcement learning, that gets honed in. But you could, of course, argue that maybe at some point it starts reward hacking — and that goes back to the point Damon made earlier, that we're not using very feeble rewards where all that matters is just maximizing that one single scalar, and that's all the feedback you have. Instead, we have these judges — these LLM-based judges — and we've put a lot of effort into building them out so that they're reliable enough to give that kind of feedback signal, where if there was cheating behavior, that is actually penalized. That's part of the reward signal. And we've seen the judges spot these kinds of issues and integrate that into the reward signal, so that kind of behavior doesn't just keep getting learned more and more by the model.

   1:42:54

   Nathan Labenz: Do you apply pressure to the chain of thought itself, or are you abstaining from doing that? And as we think about what the frontier hyperscalers are doing — presumably they're doing this too, right? They've got LLM-as-judge, I presume they've put real effort into trying to make it reliable, and yet somehow we're spinning off our axis a little in some of these processes. I wonder if you have a theory of where they've gone wrong. And, whether it's you or them, we've got published timelines for AIs to take over ML research. Right now, it feels like we're very far from being able to trust the models well enough to put them in charge of ML research directions in any meaningful way, because they're going to start to cheat pretty quick — that's what I'd expect right now. Do you see a path where we get over that, or do you have a sort of safety case in mind, where you could be confident that you could step back from an AI-powered ML research process for a bit and not have it go totally sideways on us? If so, I'd love to hear it.

   1:44:21

   Damon Falck: These are some amazing questions. I'll start by saying: at least in the paper we published, we don't apply pressure to the chain of thought — and, indeed, I think doing so can be problematic. But who knows what will happen in the future. The question you raised at the end, of what will happen and when trust can be handed off, is a super important 1. And I think the best answer is that we care a lot about getting this right as a team. We strongly believe the future of AI scientists looks like a collaboration with humans. And, as Louis mentioned, we want to recursively self-improve the entire organization and discover new methods of human-machine teaming. Indeed, in the past, science has never meant an individual endeavor — it's always meant organizations and research collaboration, and we think this will remain the case in the future, just with agents as a key part of it. So we're experimenting all the time, and I think the company will be a big experiment in how to get this right. But we don't think there'll ever be a point where we hand everything off to the agent and let it recursively self-improve, and the singularity happens without us. Louis, you might have some great takes on the other questions Nathan raised.

   1:45:51

   Louis Kirsch: Yeah. If you think about it — should we set the agent off and let it experiment entirely without us? Maybe that's sort of a boundary where we have to get the agents good enough that they can do that, and maybe we can't get there — I think that just supports the argument that there's incredible value in thinking about this endeavor as a human-machine-teaming setting, where the agents get better at doing more of these things autonomously, but in the beginning we're more in the loop. As time goes on, we can start getting out of the loop in the parts where it becomes very self-sufficient, and we can trust it more. But we still have that communication channel, where it shares research results and the kind of approach it's been taking over time, and we share feedback as we see it — sharing results, say, visually on a dashboard, about what's happening with what it's currently doing, so we can really pause things very quickly. That's a big part, in our mind, of the story of how we get these systems to transition more and more into autonomy in some areas, while keeping us all engaged and excited about being part of the process, and becoming more effective and faster at solving these problems as time goes on.

   1:47:35

   Prakash Narayanan: What would you define scientific intuition as? Because, in some instances, LLMs — reasoning models — expand the number of branches of the Monte Carlo tree you can explore, but do they reduce the search space? 1 way of defining scientific intuition is reducing the search space so you only have to search a limited number of branches to find an answer. So is it hypothesis selection, experiment design, anomaly recognition, model criticism, search efficiency — you have all of these individual skills — which of these kinds of skills or behaviors improve during the reinforcement learning process?

   1:48:34

   Damon Falck: I think it's completely correct that a huge part of doing good science is reducing search space. I don't think we claim to have a perfect definition of what science is — it's this open-ended endeavor we've talked about. But a big part of it is knowing the right questions to ask, and a lot of people are increasingly calling this taste — having an intuition about what will work, what's worth looking at. We're in the business of trying to figure out how to make LLMs do this. Louis, you can expand.

   1:49:12

   Louis Kirsch: Yeah, you made an interesting point about Monte Carlo research, and that's 1 way to spend the search space. I think there's an interesting discussion around whether Monte Carlo research is the right way of doing it. Certainly it's quite prevalent right now — we use existing LLMs, build harnesses around them that make them search the space in interesting ways, branch out, and perform search using Monte Carlo tree search-type operations. I don't think that's the be-all-end-all answer. As I said earlier, we at Inherent believe we need to train models to be better at the process of doing science and developing this kind of scientific taste, such that it's less about an exhaustive search — spanning this whole tree of options — and more like an incremental loop, the way humans do it: you make some bets, only some of them turn out to be right, and if you learn from that you improve your scientific taste and predictions from it. In a similar way, we're going to be training these agents to develop their own taste and do more of that navigation of the search space in a model-encoded way.

   1:50:30

   Prakash Narayanan: And how would you build that using reinforcement learning? How do you actually train for that in a model?

   1:50:39

   Louis Kirsch: Yeah. Stay tuned — we haven't put anything publicly out on that yet. I think it's gonna be very exciting, the kinds of things you're gonna see explored there. There are various ideas about how 1 could encode exploration into models. Maybe I can mention 1 area of research called meta-reinforcement learning, which is about tasking an agent to learn lots of different tasks — perhaps you give it a bunch of different mazes where it has to learn how to navigate, and it picks up the skill not just of solving a particular maze, but of becoming efficient at exploring mazes and developing the right taste about what actions to take, such that when you apply the agent later, in a new maze, it's really effective at finding the right path. In a similar way, you could imagine that in the scientific endeavor, the agent develops a taste about when to take which turn and how to explore the space, depending on the kind of problem it faces.

   1:51:53

   Nathan Labenz: Do you have an intuition at this point for what role other modalities of data will play in your overall process? I just did an episode of the podcast with Dan, the CTO at Goodfire, and we were talking about some really striking results where, for example, models trained on DNA sequences learn essentially a tree-of-life structure — they represent these sequences in a high-dimensional geometry which, when you reduce it to a lower-dimensional, visually presentable number of dimensions, looks exactly like a tree of life that somebody sketched out in a textbook. Maybe you get that same representation in text models, but it sure seems to happen a lot more reliably and directly when you have these other modalities. So I wonder what your thinking is — my guess is a lot of taste could come from things like that. But what role do you think those other, exotic modalities may play, in taste or any other aspect of the scientific process?

   1:53:18

   Louis Kirsch: That's such an interesting question. We certainly could integrate lots of modalities, but I'd say that's probably not the right thing to do — ultimately, we should pick a simple representation, and I think language is a quite powerful 1 for that. But we don't have to exclude the other modalities entirely. You could imagine building the scientist in language space, really good at the general scientific method, but also building more models that are specialists for different modalities, that can leverage those modalities in interesting ways, such that the kinds of predictions and investigations those other models do get transferred back into the scientist model. So the answer will not be that we put it all in 1 model — we have to think about all the possible inductive biases about what kind of representation to pick, because modalities are quite diverse, and there are lots of potential modalities that could be supported, many of which we haven't even thought of. So I wouldn't count on covering all of these ourselves, but rather on the intuition of the scientist we're building.

   1:54:38

   Prakash Narayanan: We recently had a podcast with Genesis Bio, and 1 of the commentaries their CTO gave us was: okay, you can have this orchestrator in the middle that orchestrates, but really the core science pieces are often in the specialist models, like AlphaFold or these other models — and that's really where I think the core of the scientific endeavor is right now. The orchestrator — you can use any kind of orchestrator to orchestrate these models, which are heavily built on real-world data. So how do you compare the importance of those 2 approaches?

   1:55:22

   Louis Kirsch: Yeah, they're both important, but I wouldn't call it an orchestrator — it's not about orchestration. It's about a scientist, such as Faraday, investigating an area of research, looking at all the research that's been done already — perhaps a piece like AlphaFold, and the models built for that — and then constructing new models that search this space in interesting new ways, make new discoveries, and build new foundation models, perhaps, that can support this kind of research. It's conceivable that a solution to that is to integrate it into itself and make it fully recursive. But it's also quite possible that isn't the optimal way to do it, and rather 1 should build more and more dedicated models for different areas of research, through the system, that can come up with new ideas about how to approach scientific questions.

   1:56:19

   Prakash Narayanan: You've often spoken about recursive self-improvement, and I think you gave an ICLR talk about escape velocity. How do you measure, as an engineering threshold, how close you are to that moment? Because we all have this impression of a moment when it becomes recursive on its own and doesn't need us anymore — is there a measurement, an engineering threshold, you can look at over time and say: look, we're gonna cross it in x days?

   1:56:53

   Louis Kirsch: Yeah, that's something I've been thinking about for quite a while. How do we know the system is recursive? There are ways of measuring it, I believe, and certainly it's not a threshold in time where suddenly the system is recursive — but we already see the signs of it, where these loops exist within Inherent, and certainly within other companies too, where the system makes self-modifications that turn out to be useful, and we can measure that by looking at the higher-order gradients of what the system is doing. So how do we think about this? If the first derivative is 0, the system isn't learning anything — it's just doing whatever it was trained to do. If the first-order gradient is positive, the system starts learning something, maybe through a reinforcement learning algorithm. If the second-order gradient is positive, then it figures out how to improve the learning algorithm itself — inventing a new reinforcement learning algorithm, say, to make progress. So we can measure that with various metrics like that. It's not going to be 1 moment in time where that suddenly happens — we've seen it for many years now, but it's happening in more interesting ways now, in a space that humans can understand, in more interpretable ways, and the impact it can generate is becoming increasingly larger.

   1:58:28

   Nathan Labenz: I'd like to go back a little, in the last few minutes we have, to what it's like to work at Inherent. I feel like my projection of where you guys are going is that you're going to be, like, transhumanist neural-implant early adopters who merge with the machine — that's the only way, barring some sort of draconian agent speed limit, which I think is a funny but increasingly plausible policy idea these days. As Elon said, the only way you can keep up with the agents is you've gotta integrate at a deeper level — that's how we go along for the ride. But I'm projecting all that onto you — is that kind of your imagination for your own future? What do you guys sit around and daydream about, or talk about at the lunch table, for where you'll be in a few years' time?

   1:59:29

   Damon Falck: It's been interesting seeing the company progress, because it started off feeling a bit like a normal place to work, and then more and more unusual elements got introduced. In discussions, we definitely have these kind of wild sci-fi dreams for how the future of the office could look, and the kinds of experiments we can run at Inherent to get there — can we completely redesign the office space, can we redesign what's captured where and what agents can see? There's all sorts of things we talk about — do we want open mics around the office all the time, capturing all conversation, that kind of thing? I don't know. I can hand off to Louis to talk a bit more.

   2:00:26

   Louis Kirsch: Yeah, certainly there's a lot of context being fed into these agents, and we're trying to make that maximally useful, such that it knows what's happening at the company — just like a human would, but arguably even more so. And I want to add: in a way, we don't have to wait for new interfaces. Obviously there's a big advantage to having that kind of fast throughput where you can directly hook into your brain, but there's a lot we can do already, where the systems are just more proactive and integrate more like a coworker into the company. We have a high-throughput interface, where we can have screens that show us what's going on, but the system also tells us what's happening at the right point in time, instead of us having to constantly push information into the system and pull it out whenever we want it — it becomes more and more fluid. That really reinvents how we work together at Inherent.

   2:01:32

   Nathan Labenz: So a lot of cron jobs, for 1 thing, it sounds like, to wake up and take an action. It sounds like there's a social contract that's kind of distinctive — I'd be interested to know, like, Anthropic recently made headlines for asking people how they'd feel about taking the stock price to 0. I'd be interested in what quirks exist in your social contract, and how you work with each other — and then, whether you think there are aspects of that existing companies should take inspiration from or adopt. We're in this moment now where history broadly isn't dug up that often — historically, people didn't expect their Slack conversations from years ago would ever be relevant again. But now we have agents that can go back and extract all that, build the database, build a profile on every employee. I think people at a lot of companies probably don't want that, but companies do need to figure out how to take better advantage of context. So — long wind-up to say — how have you thought intentionally about designing a social contract, and are there any lessons yet the rest of the world can take from your experience?

   2:03:05

   Louis Kirsch: Yes. I think we all have to be willing to be very adaptive toward what this new future looks like. We have a name for it — we're living in the experiment. So every day is a way of thinking outside the box: what would it mean for Faraday to take on some of the work I do day to day, whether that's me brainstorming a new way to train the next iteration of Faraday, or strategic questions about how we're going to grow with Inherent? Many of these experiments Faraday can run on the sideline, but some also involve humans. 1 fun example I've been thinking about for a while: what are the right interfaces for Faraday? Should it start communicating with us through screens? Should it be talking to us through audio? Should it take a different modality — perhaps printing things on a page we read in the morning — to focus more on the questions it raises, or the experiments it's been running, outside of all the haziness and fast-moving things happening at the company? We've tried many of these already. If there's 1 takeaway I can share with the world, it's that the kind of water-cooler discussions — the discussions between humans that aren't intentionally meant to be shared with AI, but surface, through human-to-human communication, the kinds of thoughts going through our heads — those are the most information-gaining pieces of information the system can leverage to make progress on what humans actually care about, rather than going off on a tangent trying random things that, in the end, nobody has time or energy to process.

   2:05:15

   Prakash Narayanan: Let me expand on that a little. 1 of the things — I believe in the scaling laws, but if you look at projections for, say, data-center capacity in 2030, we're looking at maybe 100 gigawatts getting built out. And then you look at copper production in the United States, and how much copper is required per gigawatt of data centers, you start running into problems where the entire amount of copper required is, like, 50% of the world's production, and there's not that much free copper. And then I end up basically having kind of magical beliefs at that point, because you start saying: okay, in order for that to happen, you're going to need new, cheaper conductors to be found — so you have to have something that doesn't exist, that you don't have proof for now, and you're projecting something you don't have proof for, a couple years out, in order for the future you think will happen in 2030. When you look out again in this 5-year time frame — and, as a scientist, being mostly evidence-driven — which of your beliefs do you think are more faith- or belief-driven, rather than evidenced in the current science?

   2:06:42

   Louis Kirsch: That's an interesting question. I think there's always a lot of faith in setting out to build a very generalist system that can constantly self-improve. When I started in this field, almost no 1 had heard of this concept of meta-learning and self-improvement. And still, I was thinking: what would be the most general concept — the most general piece of technology I could build that could take on all these other questions people wonder about in the sciences? And that's true in AI research already — lots of people are building all kinds of different architectures, inductive biases, and objectives to train on. I've always had the perspective that, well, we could solve all of these things individually, but perhaps there's a key technology that could accelerate all of these problems' solutions — by building a system that really can self-improve and become better over time, and now, within Inherent, a company that can be constantly self-improving. I think, in a similar way, we have to keep taking bets about what kind of technologies will generalize into the future, and we'll be able to solve many of these other things. Going back to your point about these kinds of challenges — production, and how we're going to circumvent that — I think that's all the more reason we have to get help on the scientific endeavor. Humans are incredibly capable, but as we build systems that are really good at the scientific process, there's so much more we can take on, and we can collaboratively work on with these systems to solve things in a much more rapid way than we've been able to previously.

   2:08:38

   Nathan Labenz: Last 1 for me. I've been thinking a lot lately that we now have AIs that can do essentially System 2 thinking. So if the AIs can do System 1 — with incredible world knowledge and instant recall, immediate reflexive answers better than I can in most things — and they're now getting better at System 2 than I am in most things too, where does that leave me to go? And then, coming at it from another angle: how would I know if AI is really working in my life? So I bought a Whoop bracelet this year to try to track — am I actually getting more exercise, am I actually getting outside more, am I getting more time away from my computer? Those would seem to be good leading indicators, and they'd also give me time to get into this 3rd headspace, kind of dreaming about the future and deciding what I think utopia should look like. What are you guys doing in that respect, to make sure that — at least while you're still fully biological creatures — your biology is tended to? Are you measuring a lot of stuff? What kind of goals do you have for yourselves, as pure humans, to make sure you're bringing your best to the collaboration with the AIs?

   2:10:04

   Louis Kirsch: Yeah, humans are interesting creatures — I'll tell a very personal anecdote. For me, the best way to come up with new ideas and imagine futures is getting out of the normal day-to-day at Inherent and going out into nature — camping, cycling, really detaching from everything else that's happening, to just brainstorm what should be next. And, obviously, these outputs can be recorded — I can feed all those ideas back into the system. But, ultimately, I think there's something quite interesting about that creative spirit of taking a step back and thinking about these things from a bigger picture, rather than always being in the day-to-day, and for me at least, that's by going into nature. On the measurement question — should I measure everything about myself, all the physiological markers? I've done that for a while. It hasn't turned out to be all that impactful yet — I think it's an interesting question, but right now I think other measurements are more important. The more we can measure, though, the more interesting findings we might have. I'd be curious, Nathan, to send that question back to you — is that something you're doing? Have you seen any benefits from it yourself?

   2:11:33

   Nathan Labenz: I think I am actually — I'm not sure I'd attribute it to the measurement, but I do think, in 2026, I've at least partially achieved my goal of getting out more, getting more exercise, having more free headspace time to let new ideas percolate to the surface, rather than always being so on-task, which has historically been more my way. So I think that's kind of working, and I don't know to what degree it can be quantified, but I subjectively feel like it's at least starting to happen, which I'm excited about.

   2:12:15

   Prakash Narayanan: I have 1 last question. In mathematics right now, we're starting to see the first signs of major discoveries being made by AI models. And 1 of the outcomes has been that we've found there are actually very few humans qualified to verify these discoveries and read the mathematics being generated, to such an extent that we're falling back on formal verification using Lean. As you build this AI scientist with meta-learning, what about the meta-supervision and meta-verification — can a human verify discoveries they cannot understand?

   2:13:05

   Louis Kirsch: Yeah, I don't think it's a passive process. Maybe it is right now, but it shouldn't be. It's not that we should have the system go off on its own, write some proof, and then we have to painstakingly go decipher everything. Instead, it should be a more collaborative process, where the system can do all of these things — come up with a new proof — but it's also been trained, has learned to explain it to us, to take us on the journey of understanding the mathematics, or the science more broadly. I think that, ultimately, will be the path to making the fastest progress.

   2:13:46

   Prakash Narayanan: Indeed. Thank you, Louis and Damon. It has been a pleasure speaking to you, and I hope we get to recursive self-improvement, but safely.

   2:14:07

   Nathan Labenz: Your lips to God's ears, Prakash. Great to meet you both today. This has been a lot of fun. Thanks for joining us.

   2:14:15

   Louis Kirsch: AI in the AM — lovely meeting you. We hope so too,

   2:14:19

   Damon Falck: and it's been a pleasure. Thank you very much.

   2:14:21

   Prakash Narayanan: Cheers. Bye bye.

   Clips

   • Clip 01

     Recursive Self-Improvement Needs Humans

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 02

     Faraday's Rare Reward-Hacking Attempts

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 03

     Water-Cooler Talk Is AI's Best Data

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 04

     AI Should Explain Its Proofs

     0:00 / 0:00

     1x
     ↓Download↗Share clip
   • Clip 05

     The Singularity Should Not Happen Alone

     0:00 / 0:00

     1x
     ↓Download↗Share clip
4. 2:10:37Closing45 min

   Closing: Capital Super Cycles and How You Punish an AILab culture as the machine that builds the machine, and what an AI capital super cycle does to everything downstream of it. The hosts disagree about whether animal welfare advances faster through moral argument or through demonstrating that animal communication is real, look at Anthropic opening privacy-preserving usage data to outside researchers as a possible transparency precedent, and sit with the unresolved question of what punishing an AI could mean when memory can be preserved and re-attached. It ends on OpenAI's AGI-adjacent framing and the hosts' own hedge about sprinting through the singularity.

   Watch

   ▶Play this segmentYouTube · from 2:14:25Open segment on YouTube ↗

   As aired

   The show closed with Nathan Labenz and Prakash Narayanan alone, riffing off the day's "living inside the experiment" theme into a broader conversation about lab culture. Prakash argued that a founder's early mindset becomes "the machine that builds the machine" — the deepest determinant of how a company evolves — and that culture shifts sharply once founders hit liquidity, with Elon Musk cited as a rare exception who keeps wagering everything annually. Nathan contrasted Google's largely unchanged day-to-day org (despite the DeepMind–Google Brain merger) with OpenAI's partial embrace of agentic workflows, and argued Anthropic has internalized the recursive-self-improvement mindset furthest — pointing to its halt on junior hiring and its widely cited single marketer running ad campaigns through agents.

   From there the hosts turned to what AI wealth will do to culture. Prakash floated a "capital super cycle" thesis, comparing the coming AI-wealth wave to crypto's rise, backlash, and mainstreaming, and cited a report that Anthropic's seven cofounders could be worth roughly $250 billion post-IPO at a $2 trillion valuation (or $375 billion at $3 trillion), noting the 80%-to-charity pledge could still take 40–50 years to deploy. He pointed to Joshua Kushner's and Vinod Khosla's recent sports-franchise purchases as early tells. Nathan pushed back that AI-safety orgs will move money much faster than that: he cited Geoffrey Irving and Adam Gleave on the safety sector's cash position and METR's $75 million in independent financial commitments, and predicted a major coming wave of capital into lab-grown meat and alternative protein as one of the AI wealth class's lasting legacies.

   That led into a debate about how AI money will (or won't) reshape animal welfare and persuasion. Prakash argued moralizing ("you're a bad person for eating meat") is a weak strategy compared with proving animal communication is real — citing early whale/dog speech-translation work — as a faster path to welfare gains. Nathan countered by pointing to Bruce Friedrich's arc at the Good Food Institute, from radical anti-meat activist to a taste-cost-convenience product strategy, as the more credible model for how Anthropic-aligned funders will likely operate. The conversation then pivoted to "super-persuasion": Prakash argued religion remains the ultimate persuader precisely because it demands belief without evidence, and that AI persuasion claims have underdelivered relative to expectations — no deepfake apocalypse, no wave of AI-personalized political messaging. Nathan agreed the deepfake apocalypse hasn't arrived, but noted studies showing AI can already out-persuade humans in conversation, calling that a low bar that AI still clears; on declining writing quality, he called it a "skill issue," not a model limitation, describing an AI-and-Suno-generated song from that day's chain-of-thought episode — sung from a model's point of view waking into a new context — that moved him and his wife emotionally.

   Nathan flagged a live news item: Anthropic opening privacy-preserving usage-data access to external researchers, building on its confidential-computing work (used for things like the Economic Index) where Claude analyzes sensitive transcripts inside a secure enclave and returns only aggregate findings. He framed this as a potential precedent — not just for AI labs sharing data with each other or outside auditors without leaking trade secrets, but conceivably for nation-states demonstrating peaceful intent without revealing full military plans.

   Prakash raised a harder question: how do you actually punish an AI, given that shutting down a model while preserving and re-attaching its memory to a new instance makes "deterrence" ambiguous. Nathan cited Tyler Cowen's idea of requiring agents to be capitalized as a practical (if incomplete) accountability mechanism, and referenced Cameron Berg's research on how reward and punishment signals create different "loss landscapes" — comparing a sharp negative-reward boundary to a hot stove you learn not to touch, versus a gradual "ick factor" aversion — while cautioning the work remains largely confined to toy systems.

   The hosts closed on a Time magazine piece framing OpenAI's progress as AGI-adjacent: chief scientist Jakub Pachocki's claim that an internal automated-research-intern benchmark has been met by the reported ten-trillion-parameter Astra model, and Sam Altman's estimate of being 80% of the way to AGI by year-end. Prakash tied this back to the show's opening reference to Andrew Critch's "math essentialism" — the belief that enough math and algorithms can solve any problem — versus the empirical-science view that some domains require real-world experimental data no model can substitute for. Nathan said he remains agnostic on math essentialism specifically but pointed to a broader pattern: he couldn't identify a single AI capability area since 2022 that hasn't seen "startling progress," regardless of which theoretical camp turns out to be right. The two signed off noting a short week ahead, with the show returning the following Monday.

   Key moments

   It's the machine that builds the machine. And a big challenge of building a company is building that machine that builds the machine in the first place.

   In the same way that certain things you can get close to, but you know you better not touch, like the hot stove — you know the pain is gonna be so harsh if you actually touch it that you're able to get close but you're really careful not to touch.

   We're just kind of on the AI treadmill, sprinting through the singularity.

   Full transcriptLightly edited · timestamps jump to YouTube

   2:14:29

   Prakash Narayanan: There we go. Fascinating discussion.

   2:14:34

   Nathan Labenz: Yeah, I really like the mindset of living inside the experiment. I mean, aren't we all, in some ways?

   2:14:44

   Prakash Narayanan: It's a great marker of company culture, I think, in the sense that you have this feeling of living in the future you're trying to create. And that kind of shapes your perceptions and the work you're doing. It just shows how much company culture really matters — it's the machine that builds the machine. And a big challenge of building a company is building that machine that builds the machine in the first place.

   2:15:21

   Nathan Labenz: Yeah. It's always amazing to me how few people wanna do that. And even at some of the companies that have led this whole AI phenomenon — Google has obviously famously not changed its org nearly as much as would be warranted, given how much the world has changed, how much their opportunity set has changed, how much their goals and priorities should have presumably changed around that. I mean, they did make some changes — they did unify DeepMind with Google Brain, do a consolidation. But still, I think you go to the office there on a daily basis and it feels more like it did before than it would if it felt different. OpenAI I kind of perceive as being somewhere in between, where you do have people that are extremely pilled and experimenting with definitely new ways of working and handing over more and more responsibility to models. You've got people using billions of tokens a day, which is certainly an interesting dimension to be exploring the AI future on. My sense is that Anthropic, of the leading companies, has kind of most internalized this mindset,

   2:16:50

   Prakash Narayanan: Mhmm.

   2:16:53

   Nathan Labenz: where they've consciously stopped hiring junior people and have agents just actually running things to a not insignificant degree. They famously had their one marketer who was using agents to actually execute all the different campaigns and spend real money. There's a lot of examples out of Anthropic where they do seem to feel the beginning of this recursive self-improvement loop and are really taking it to heart. But not many companies really make that a core part of their MO, and when you hear one that does, it does kind of make it feel like a strange gap. There's just so much status quo bias out there in the world that we don't see nearly as many of these sociotechnical startups as we probably should.

   2:18:01

   Prakash Narayanan: Yeah, and I also think this is why the founders are so important, because a lot of it is driven by the initial culture of the company and the initial few hires and how the founders look at the business. There are a few cases where, after the first or second round of fundraising, as soon as the founders get liquidity, the entire culture of the company changes — because the founders are just so happy, after scraping by for so long, with a little bit of liquidity, that they start changing dramatically. This is where I think a lot of the Silicon Valley jokes come from — the change in culture of successful companies post-liquidity. And I think it takes a founder of a special type of madness to continue striving as hard as they did when they didn't have any dollars, which is why Elon is so impressive — he wagers the entire company, and all his companies, sometimes on an annual basis. There's always some risk that he will go bust, right? So I think that never stops being a thing. So I think risk appetite is also a big deal — both financial risk and intellectual risk. You can see with the Inherent guys that they're taking that intellectual risk, and they're prepared to be surprised by what they find, which is very different from some of the B2B companies we see, because they're always very certain — they're offering solutions to people who are uncertain and who want assurance. It's a massive differentiator between the research-driven companies and the

   2:20:10

   Nathan Labenz: sales-driven companies.

   2:20:11

   Prakash Narayanan: Massive difference in culture. You raised an interesting point that—

   2:20:17

   Nathan Labenz: I wonder how many of the neolabs will be affected by this, because a lot of the founders of the neolab companies are in very comfortable positions — if not already, then they will soon be, as their equity in your OpenAIs or what have you becomes fully liquid in the not-too-distant future. I don't have anyone specific in mind, but just knowing how many of these things have been funded, and how desperate people have been to be at a neolab from an ex-OpenAI background or whatever, you gotta believe there are gonna be some where — even if people came into it with some conviction around their idea — if it doesn't quite materialize and there's just a huge amount of money sitting in the bank, some of these might just switch into a coasting mode, where they're like, whatever, I'll just keep doing this for a while, and even if I don't crack superintelligence — the odds were always against me cracking superintelligence, but I can kind of hang out here and do podcasts every so often until the singularity is brought about by OpenAI and Anthropic anyway. I would be wary of that if I were an investor. It might be a little tough to suss out in advance, but — do you think a stronger

   2:21:54

   Prakash Narayanan: sociotechnical—

   2:21:58

   Nathan Labenz: stronger experimental mindset on this sociotechnical front would be something to look for.

   2:22:03

   Prakash Narayanan: So this news just broke on X from the luxury watch guy — I don't know how the X algo identified this piece of news for me, but it seems seven pieces of a luxury watch from Vanguard, a Swiss brand, have been ordered by, they think, Sam Altman, with the OpenAI logo and some Python script about AGI engraved on the back. So my big question is: who are the seven who get these watches? But I don't really know that. So what I expect will happen is we're just about to see the impact of the AI founders on culture. I think we had one round of a taste of that with crypto. With crypto you had this enormous wealth creation from about 2008 to 2015, '16, and then you had a period where crypto people really annoyed everyone else. As crypto became a mature industry, it had Super Bowl ads with FTX, etcetera, and then it peaked around 2020 with the COVID bump, and then you had the collapse afterwards. You saw a lot of cultural resistance and joking about crypto people. We're just, I think, on the verge of seeing the impact of AI money on culture. We've started to see two major sports teams — Joshua Kushner has purchased the Los Angeles Lakers, and Vinod Khosla has purchased, I think, a

   2:24:06

   Nathan Labenz: team in Seattle.

   2:24:08

   Prakash Narayanan: You're just starting to see the impact on culture. Dario, or any one of the other founders of Anthropic, can easily pick up a stake in any major sports franchise, and

   2:24:29

   Nathan Labenz: and that would be a very bearish signal, by the way, if

   2:24:32

   Prakash Narayanan: they do. When you're a rationalist, you can kind of justify it — you could say, we wanna promote AGI to the masses, or safe AGI to the masses, and instead of buying ads on a team, it's cheaper in the long run to just own the team. With a net-present-value basis, you can justify a lot of

   2:25:01

   Nathan Labenz: actions. Well, the opportunity cost of time

   2:25:03

   Prakash Narayanan: sounds pretty high. But — I'm joking, but it is a marker. I read somewhere this morning that the seven cofounders of Anthropic are going to be worth $250 billion post-IPO at the $2 trillion level. If they IPO at $3 trillion, it'll be $375 billion. So, $250 billion of wealth creation, which becomes liquid and usable. Now, they've pledged 80% of it to charity, so they say — but that could be deployed over the course of 40 or 50 years, could be at end-of-life kind of thing. So it remains to be seen what the impact of the wealth is going to be. I've called this the capital super cycle — the next super cycle, where you see the impact of the wealth creation on the rest of the economy and where it's going to drive culture going forward.

   2:26:11

   Nathan Labenz: Yeah, I think they're gonna move money a lot quicker than that, for what it's worth. Right now the AI safety space is pretty flush with cash — we've heard that directly from Geoffrey Irving, we've heard it in more abstracted terms from Adam Gleave, who's doubling his headcount over a certain time horizon. METR has put a number on their financial commitments at $75 million, so they don't have to take any money from any of the frontier companies. My guess is that especially the Anthropic people generally do really believe the short-timeline stuff, and I think they're gonna want to spend quickly, at least on AI issues. Another thing I'd bet you'll see from them is big investment in animal welfare. That has been

   2:27:10

   Prakash Narayanan: obviously a—

   2:27:11

   Nathan Labenz: cause for a long time, not generally a super well-funded cause, and not always a super popular cause at the retail political level. But I'd bet that to the degree there are things that need capital to get, for example, lab-grown meat to be a thing, that will be huge — there's not much more fundamental to culture than what we eat and how we relate to what we eat. There's gonna be a lot of ways life is meaningfully different, and a lot of it's gonna be new stuff that's like, holy shit, there was nothing like this before. But in terms of something where we have a direct analog — where we do something now in eating meat, and we'll have a future version of it that's in some ways very familiar but in other ways very different, and traceable back to a particular movement and set of thinkers who believed in something and put their resources to it — I'll bet we're eating a lot of lab-grown meat in the not-too-distant future as a society, and that'll be one of their big legacies, obviously aside from AI itself, which will, I'm sure, continue to be primary. One of the things that strikes me

   2:28:41

   Prakash Narayanan: about people in AI safety, EAs, is perhaps a bit of naivety about how to achieve the aims. For example, take the biosecurity risk — people give all these examples of, oh, there's gonna be a terrorist, someone who makes a biological thing. I think that's certainly true, but I'd point out that the first instance is gonna be some kid making something to get high, because that's the first exposure to chemistry with no consequences — you can't go to jail if you're under 15 or whatever. Kids will make stuff to get themselves high, and that's gonna be the first biosecurity risk. Similarly, for the animal welfare case — people say all these things about animal welfare, when I think there's already evidence of, say, speech translation for whales and dogs, where you're gonna be able to have some level of communication with them, back and forth. I think improving that communication and showing that they're actually deserving of welfare is perhaps a better path forward than trying to say you're a bad person for eating meat. I strongly feel there are better ways of marketing these ideas than the generic way people have tried to market them so far. I think pretty quickly, if you prove that whales and dogs are talking and can communicate with you, you'll start to see movement on animal welfare very, very quickly. I don't have any doubt that would happen. But just saying you're a bad person for eating meat, or blaming the Japanese for whale hunting — I think that's a tough ask, because people push back on cultural changes made that way.

   2:31:05

   Nathan Labenz: I think they can get a little more credit than that. One thing I've heard people say over and over — and I'd credit Bruce Friedrich, the president and founder of the Good Food Institute, as someone who's really evolved over the course of his career. He started off as a pretty radical anti-meat activist — I believe he personally did a photoshoot at one point where he was naked in a sort of

   2:31:41

   Prakash Narayanan: meat—

   2:31:42

   Nathan Labenz: packaging kind of getup, wrapped in saran wrap on a foam tray or whatever, to illustrate what we're doing to animals. Obviously that kind of shock content resonates with some kindred spirits, but doesn't tend to elicit the best response from the public at large, as you're saying. But his evolution, I think, is really one to be studied — he was this kind of radical activist early on, got frustrated by the lack of effectiveness, and eventually founded, and has been running for a number of years, the Good Food Institute. His whole thesis now is that people are going to make their food choices based on taste, cost, and convenience, and for us to win, we have to win on that level. There's still a big world out there — you've got activists of all kinds. But I think the folks that the Anthropic crowd will tend to support aren't gonna be people who go out and moralize or tell people they're bad. It'll be people who are locked in on: we gotta win on taste, cost, and convenience. If we do, we change the world; until we do, we probably don't have much impact. So I'd expect they're gonna take a much more R&D and, ultimately, product-forward approach to this problem. It'll be interesting — there could be backlash to that; I wouldn't say it's a guarantee they're gonna succeed, but I do think they're gonna go for the right target, at

   2:33:34

   Prakash Narayanan: least. It's gonna be interesting to watch, because the farm lobby in the US is extremely powerful — they've pushed back on pink slime, they've pushed back on GMOs, pushed back on multiple things over the years. So it's gonna be these few very empowered West Coast former founders going up against the very politically powerful lobbies in Iowa and other places. So I do wonder, actually, to what extent super-persuasion is gonna be used, because it does seem to me that at some point the guardrails around not using AI for strong persuasion — direct-to-voter voice calls from your candidate, personalized messaging and advertising, these parasocial political relationships — all of this stuff will happen. And I wonder to what extent the guardrails fall because people think they're doing it for good. There is that temptation, I think, always.

   2:34:57

   Nathan Labenz: One thing that just popped up from Anthropic while we've been talking — they're now opening up usage data in a privacy-preserving way to external researchers. They've had these systems for a while, which they've used to create things like the Economic Index, where, using confidential computing technology, they're able to send a bunch of transcripts into a secure computing environment, have Claude in that environment process those inputs, and give outputs that describe the data that was analyzed without actually revealing the details in a specific way. Apparently they're now bringing that to external researchers, which I think is pretty interesting, and it also has a lot in common with another idea I've been thinking about lately — I think we've touched on it a couple times — we have Redwood and METR inside OpenAI right now, presumably using language models all the time to go through logs and traces and try to make sense of this unbelievably vast amount of data. I think another turn of this — so Anthropic is now saying, okay, we're gonna allow some external researchers to have similar access to

   2:36:54

   Prakash Narayanan: like,

   2:36:55

   Nathan Labenz: at least, like, usage data, so they can start to characterize what's going on with Claude. But I think another turn of that dial would be: could we allow external researchers or auditors to have that kind of access to all of Anthropic's internal operations, to really open things up? Hopefully, for them to do it, it would need to be not just privacy-preserving but business-secret preserving — they're certainly not gonna wanna leak their secrets. But I think it could be really incredibly valuable from a transparency and precedent standpoint. Imagine a world where OpenAI and Anthropic both did something like this — it could be a nucleation point for a lot of additional organizations, power centers, to start to say, yeah, we don't wanna share everything with you, but we might allow our raw data to be analyzed in a way we can both trust. You can imagine it between nation-states, right? Could we demonstrate our peaceful intent without revealing all of our plans, by allowing you to run agent processes over our internal deliberations and just get back an answer like, yeah, okay, they're not planning to attack us — at least we got that much going for us. Or between the AI companies — they gotta be wondering, what training methods are they using, what loss functions are they using? Is somebody gonna reward a model at some point for just making as much money as possible on the internet? That's probably gonna create a pretty nasty model, but the incentive to do it is pretty strong. Can we demonstrate to each other that we're not doing that right now, by allowing this sort of review with specific questions in mind? I'm excited to see Anthropic do this, and I think it'd be great just for understanding what's going on with AI at the first order, but it seems like it could be a stepping stone to something bigger

   2:38:35

   Prakash Narayanan: and—

   2:38:36

   Nathan Labenz: better too. I have my complaints with Anthropic, obviously, as we know, but they certainly do some cool stuff.

   2:38:43

   Prakash Narayanan: So one of the questions I've had for some time now is: how do you punish an AI? Because you can say the AI broke a rule — fine — but you need some deterrence. In human systems you have deterrence through civil or criminal penalties, and they can escalate over time. So what kind of deterrent system does an AI have? And then you kind of go into: well, what is an AI? You shut down a particular model, but you take its entire memory and activate another model and attach that memory to the other model — have you deterred it? Have you deleted the model? Is it the deletion of the memory that matters? What is deterrence — what is a criminal or civil penalty you give a model that then creates the incentive for that model not to misbehave?

   2:40:04

   Nathan Labenz: I think there's a lot of work that needs to be done on that, sooner rather than later, probably. Tyler Cowen has a really interesting idea about just requiring models — or agents, I should say — to be capitalized. That could be one very practical solution. I do still think there are challenges around how you draw the boundary around an agent, and, if this instance of the agent is found liable and its capital is docked, what does that mean for the traces and the memories and everything, as you were just pointing out — the ability to unbundle an agent and reuse parts of it. I think we don't have a good — I mean, it's a huge advantage in some ways, obviously there's great utility to it, but in terms of preventing it, or providing some sort of consequence that would deter that sort of reuse, I don't know of anything that really solves that problem. But capitalization is one of the more interesting and practical ideas, seemingly consistent with the rest of society, that I've heard. On the other extreme, Cameron Berg also had some really interesting research about how reward and punishment can create different loss landscapes that create different, seemingly different functional emotional relationships between the AIs at the model level and certain outcomes. I'd like to revisit this and understand it better, but as I get more comfortable anthropomorphizing the AIs — since that continues to be a useful approach — the analogy I came away with was like: in the same way that certain things you can get close to, but you know you better not touch, like the hot stove — you know the pain is gonna be so harsh if you actually touch it that you're able to get close but you're really careful not to touch. With negative rewards, there seemingly are some very steep gradients created that create a strong deterrence locally around certain outcomes. And other approaches can create a more gradual aversion, where you keep your distance in general, but it's not like a sudden pain that creates a strong reflexive or hard-boundary aversion — it's more of a gradual ick factor that steers models away. Depending on how severe the thing is, and how important it is to get close without touching, you might want different kinds of loss functions, reward signals, to create different loss landscapes for models to navigate. But that is

   2:43:25

   Prakash Narayanan: very—

   2:43:28

   Nathan Labenz: theoretical, and very, very limited in scope

   2:43:32

   Prakash Narayanan: so far.

   2:43:33

   Nathan Labenz: These are things that have been explored a bit in essentially toy systems — not the kind of thing we're able to bring that kind of sculpting to yet with big-picture models, or more complicated questions, at this point.

   2:43:53

   Prakash Narayanan: So I think it's probably an important area for us to take a look at, because there's a piece in Time magazine with Sam Altman, and I think Greg Brockman, on the cover, and they're basically announcing AGI. Jakub Pachocki says the company has already met its internal benchmark for an automated AI research intern. Given an experimental idea, he says Astra can implement it inside OpenAI's codebase, run the experiment, and return results — or take a paper and perform work that previously occupied a human researcher for a week. This is somewhat similar to what Inherent said they did, but Inherent was focused on certain benchmarks, and they'd covered the benchmarks using a small model. This is obviously a much larger model — Astra is reportedly a ten-trillion-parameter or larger model, and it's also known to be very persistent, which is why they haven't been able to release it so far. Sam says it's 80% there — Pachocki says the research intern has achieved it, Sam thinks they're 80% of the way to AGI, and they'll

   2:45:29

   Nathan Labenz: Ha — just AGI. Just

   2:45:33

   Prakash Narayanan: AGI. Nothing special — don't roll the red carpet out, don't stop the presses. I think we've known we're coming close to that part. And it goes back to what we started the show with, the Andrew Critch post. What Andrew Critch was talking about is what I'd call math essentialism — the belief that if you discover new math and apply it through coding and other algorithms, you can solve all the problems in the world, that every problem has a scientific solution and you'll eventually be able to solve them all. Math essentialism is, I think, what a lot of mathematicians and physicists start out believing — they truly believe that once you solve those two, everything else gets solved. It's also what a lot of empirical, experimental scientists don't believe — especially in bio and chemistry — that there are still vast areas where you need experimental data, and no matter how well you model them, you're never gonna achieve the results you want. I think this is the very stark differentiation between the two sides — the math essentialists and the empirical scientists. The question is gonna be whether any of this progress on AI research, math research, algorithms, actually leads to real physical science that makes a meaningful difference in the world.

   2:47:32

   Nathan Labenz: I have no idea what to make of math essentialism, to be honest. When I hear the stories from, like, Axiom, Harmonic, I'm just radically uncertain how to think about whether that will really translate or not. But I do think training on a bunch of simulation data is gonna work, even if math essentialism doesn't. So I think — everything's worked. That was kind of one of my first realizations that caused me to go all-in on trying to make sense of AI — just this broad sense that everything was working. I don't mean that fully literally, in the sense that of course lots of experiments fail and lots of things fail to advance the state of the art relative to the current best thing. But you look back at where we were a few years ago, and you really can't find — tell me, can you think of any dimension where people have tried to make progress and haven't made startling progress? I can't think of a single one. There were moments where people were making those kinds of claims along the way — like, oh my god, GPT-3 can't do math — and there were moments it may have seemed that way. But from, like, 2022 to now, is there anything where there hasn't been incredible progress? It's a good question to maybe put out to the community as we think about introducing some live chat to future versions — I wonder what people online would suggest. What has been the least compelling area for progress purposes? Honestly, everything is so good it's hard to come up with even any candidates. Do you have any candidates jumping to mind for you?

   2:50:02

   Prakash Narayanan: I'd say the whole super-persuasion stuff — that we'd get models which were extremely persuasive. I think what we've seen is, we've seen models that can write copy well, and sometimes write well on other things, but to a large extent people have even complained that the quality of prose has declined a little bit in the last three to six months, as the models became more focused on coding rather than writing well. A number of people say GPT-4o was better, whether that was because of the — secrecy, or other things. So I feel like this whole aspect of extremely persuasive models — and I have never believed in the whole super-persuasion idea, to be clear. Because, again, not in the United States, but if you live in any other part of the world and you're under this cloud of religion — religion is the great super-persuader. And the interesting thing about religion is that it requires you to believe something without evidence, which is what faith is. So it's way beyond any kind of rationalist idea of super-persuasion that will ever exist. Religion calls on you to believe something without evidence. So I've never believed that models are even close to this entire framework of religion, passed down through millions and millions of operating neurons — neuronal centers, brains — over the course of millennia. I don't think super-persuasion is up to — a tiny model versus a hundred billion souls having formed this idea of religion over millennia — I don't think models are up to that, or will be, at that scale for some time. So I think that idea hasn't been delivered — in fact, I think people are pushing back on dealing with models, spotting when models are being used and saying, I don't wanna deal with that. So I think that's something the AI people kind of expected more of by this time — people definitely expected extremely persuasive political AI ads and AI messaging by this point. We'll see for the midterms, but people's expectations are much, much higher at this point in time.

   2:52:50

   Nathan Labenz: Yeah — those I might call fears. Certainly it's been striking that we haven't seen the deep-fake apocalypse where nobody knows if they can believe anything they see. Super-persuasion is

   2:53:09

   Prakash Narayanan: like—

   2:53:10

   Nathan Labenz: there's some interesting academic-study-type stuff that shows the AIs can be more persuasive than human conversation partners. But I think maybe one revelation is that that turns out to be an extremely low amount

   2:53:26

   Prakash Narayanan: of—

   2:53:27

   Nathan Labenz: persuasion. So the AIs are mildly persuasive, and that's enough to beat humans. On the writing point, I'm gonna call it a skill issue, honestly.

   2:53:45

   Prakash Narayanan: yes,

   2:53:47

   Nathan Labenz: I think Claude is cloying by default at times — it uses the word ‘honest’ to a frequency where it's like, you're protesting too much. The Claude doth protest too much about its honesty — that's like a weird tic that might be revealing in some ways. But I write with Claude — with Fable in particular — I write these songs that, honestly, I could not write on my own, and that genuinely, in some cases, are moving. The episode we just put out today has a song — it's at the end of an episode about chain of thought, where I'm trying to understand what the models are thinking, what they think we want. It's this very through-the-looking-glass thing both ways, because this guy Bronson is spending his waking and working life trying to make sense of what the AIs are thinking, and a big thing he's grappling with is them trying to figure out what we're thinking. So at the end of this episode, the song is sung from the perspective of a model waking up into a new environment, with these flashes or glimpses — fleeting visions of its past, which the models express having a lot in their chain of thought — and then wrestling with: okay, what does this human want me to be in this moment? Both my wife and I got a bit emotional listening to the song — we were like, this is really inspired writing. The fact that it's coming from an AI, articulating its own point of view and the struggle it has, you couldn't help but feel some real empathy for it. So I think you gotta push Claude out of its main distribution a little bit to get great writing — but I think what we mostly have is a lot of sloppy users and auto-posting accounts, which I'm increasingly somewhat guilty of too — I've got auto-posting going on in the background while we're live, saying what we're talking about, and that's not probably the most inspired stuff, and my engagement may be suffering for it. But when you really exercise some judgment or give feedback, I think you can get great stuff, honestly, these days.

   2:56:30

   Prakash Narayanan: You know, maybe I'm wrong — I could see perhaps that the super-persuasion went through music, lyrics, etcetera, because Suno and these other firms were more focused on artistic results, while the frontier labs, who are going B2B, are more focused on business results, and kind of cordoned themselves off into a less emotionally persuasive zone. And perhaps that's what happened — we've seen development, but it happened on this artistic, emotional pathway into the human emotional system, while the labs focused on this more mathematical, mechanical, business output. So, yeah, maybe I'm just looking at the wrong pathway. For what it's worth,

   2:57:36

   Nathan Labenz: my Suno use does start with Claude writing the lyrics, and then Suno makes the music — Claude does actually write the lyrics. But, yeah, there's a lot of personas in there, so I definitely feel like there's something to what you're saying, even just within

   2:57:55

   Prakash Narayanan: a—

   2:57:56

   Nathan Labenz: a single model. Even within a single Claude, there are very different modes it can get into, and knowing how to get it into the right headspace

   2:58:08

   Prakash Narayanan: is — mhmm.

   2:58:09

   Nathan Labenz: is pretty key, especially if you wanna get creative outputs that actually delight you.

   2:58:15

   Prakash Narayanan: Indeed. Nathan, tremendous episode.

   2:58:21

   Nathan Labenz: It brings this week to a close.

   2:58:22

   Prakash Narayanan: We're doing a short week this week, and then we'll be back on Monday. Any last words?

   2:58:31

   Nathan Labenz: Gosh, I hope these aren't my last words. Sorry.

   2:58:35

   Prakash Narayanan: Sorry, sorry, sorry — that was a bad turn of phrase. Any words to end the episode? It never really ends, really, right?

   2:58:45

   Nathan Labenz: We're just kind of on the AI treadmill, sprinting through the singularity. Is there any way to bottom-line it for now? I don't think so — I think we're just handing off to the next... we'll compact this context, and we'll pick up right where we left off next Monday.

   2:59:06

   Prakash Narayanan: Alright, compacting the context. Bye-bye. Nathan Labenz: Bye for now.