Clean Neurons, Anthropic’s Leaked Bill, and Banks Under Cyber Strain

Prakash Narayanan: Morning. It is Wednesday, October 21, 9:03 AM. Nathan, good morning to you.

Nathan Labenz: Good morning, Prakash. It's been a week, and what a week it's been.

Prakash Narayanan: What has been on your mind? What are the most pressing things that you think have happened in the last few days?

Nathan Labenz: Well, you can always tell when we have some catching up to do, because that becomes a great excuse to invite Zvi to come help make sure we're not missing anything. So it's one of those days, and Zvi will be here shortly, so I'm excited about that. Boy, there have been a few things over the last week that were pretty notable. I definitely want to get into more detail with Zvi on the Nick Cammarata post from Goodfire about neural networks being clean by default. That was one of the bigger stories to take over the discourse for at least a 48-hour period, maybe even 72 hours, which just goes to show how big it was, because very few things can sustain

attention for that long in today's world. And obviously we're now on to the PHP apocalypse this morning. But I thought that was very, very interesting because, first of all, it's just a post. The paper is still forthcoming. I am hoping to do a full Cognitive Revolution deep dive, probably two hours plus, with one of Nick's coauthors, who is the lead on the language model side. Nick did a lot more work on the vision model side, and really went back through the history of interpretability and looked at earlier models, some of the techniques that were used both training-wise at that time and interpretability-wise.

Basically, his report, if I could summarize it in a couple of sentences, is that as we are training bigger models with more modern techniques, and as our interpretability techniques have improved, it looks like the mess that we used to see was maybe more of a skill issue, and it's actually a lot easier to interpret the core mechanisms of ML models today than we thought. This is definitely a big deal if true, although exactly how big a deal remains to be seen. It's been heralded as the beginning

of a new era where interpretability will be reduced to "normal science," where we can just have language model agents grind away, go through neuron by neuron, see what they do, systematically map it all out, and then hopefully have a pretty good synthesis of that at the end, which would at a minimum be super useful for control measures. I think there's been some conflation between alignment and control. Just because you can know what a neuron is doing does not mean you know what the model itself is doing at a high level, at the level of goals and intents and how it will generalize out of domain. You really don't know too much about that, I don't think, even

in the ambitious view, or at least what I would imagine the ambitious view to be, of what could be accomplished with the techniques Nick mapped out in that post. But it could be a huge deal for control, because if you do have a reliable and pretty accurate understanding of the different roles that each neuron is playing, you can then just look for whether a neuron is firing. We think this is a problematic concept; if it's firing, this is something we need to pay attention to. That could really move the needle in a pretty substantial way on control measures. And given where we are right now, that seems like a win.

In the Swiss cheese theory of safety through defense in depth, this might functionally take one of those layers of Swiss cheese, your internal monitoring, and, with a substantial compute investment to figure out what all the targets for monitoring should be, could potentially put you in a spot where the holes in that one layer get a lot smaller. That could be really good. At the same time, Zvi, of course, will have interesting takes on this, and one of the takes is always that this sort of thing is inherently a dual

use technology. And we have indeed seen some pretty interesting advances, which I'm still wrapping my head around, in backprop-free training inspired by this paper. It's always an interesting question, for validation of something like this when it comes out. Even Eric, a friend of the show from Goodfire, where I've done a bunch of episodes, and we've had Tom, the chief scientist, here on AI:AM as well, described his experience of watching Nick do this work as an incredible privilege and honor to be so close to it, but also, at times, some of these ideas seemed to be bordering on madness. Other

times, they were cutting through confusion with a hot knife. So I think it was unclear even to the Goodfire folks, as they put this out, just how far this will go. Will it be validated? Will it be challenged? One of the validations does seem to be this: if you take the theory of clean neural networks at face value (I don't even know if this thing has a name yet), one of the things you would infer, and I think Nick basically says, is that the small activations don't matter much. Put an asterisk on that, because how much they matter will, I think, be a definite

point of further study and probably a lot of contention. But if you believe that it really only matters when the neuron is turned on a lot for the overall function of the network, then consider backprop-free training methods, which is a whole interesting rabbit hole unto itself. Why do we want backprop-free training methods? My takeaway from some study of that so far is that basically all ML architectures to date are limited by the need to be differentiable in order to do the backprop chain-rule process. So any architectures you might want to explore that are not fully differentiable, because there's some black box or

some nondifferentiable function in the middle somewhere, we've just not had a good way to train. Backprop-free training allows you to explore a bigger section of the total possible space of ML architectures, because you no longer have that differentiability requirement. But these methods have been very challenging, because basically the way you do it is you start with weights, you do some perturbations around those weights, and then you see what gives you better results. It's similar to a backprop training process in that you're making small changes and looking at which of them makes a better outcome, but instead of doing it analytically, you're doing it on a more random guess-and-check basis. And this

has made it very slow. It has not been competitive at all with backprop methods. But the insight that I think shows there's really something there from this essay is this: if you believe, as Nick suggests, that really only the strong activations of an individual neuron matter, and that if a neuron is only slightly activated that probably doesn't matter too much for its overall function, then you could say, why don't we focus our guessing and checking in this backprop-free training on the neurons that are actually strongly activated? We can probably ignore the other ones that don't seem to be contributing much, and focus on the ones that are strongly activated and, according to this

line of thought, are the only ones that really matter. And sure enough, that cuts down dramatically the number of guesses and checks you need to make, and what we're seeing out of this community is that it basically does work. Interestingly, the updates they are making are almost exactly the same as the updates that were made with backprop. They compare the weight changes and find they're very, very similar. But this gives you a way to do it without having the whole architecture be differentiable. And it does show, I think, that there is at least some substantial truth to

the idea that the weak activations don't matter as much. Now, no less than [unclear] has come out and said, wait a second, I think there actually is a lot more going on in these weak activations than that. And it's still clear that at the level of the residual stream, which is only going to be so many thousand floating point numbers even in the biggest architectures today (that could be 4, 8, 16, 32,000, however many thousand numbers it is), there are clearly way more concepts in a neural network than there are places in the residual stream. So superposition is still happening in the residual stream, but it's

maybe not happening as much in individual neurons as we thought. There does still seem to be some role for these weak activations when it comes to memorizing long-tail facts. So I come out at the moment as one who thinks this definitely seems to be a real something. I'm not sure yet if it should be considered the mark of a new era. That seemed to me, on reflection, to be more a function of people being desperate for hope, everybody being bummed out, especially folks at OpenAI. Nick used to work at OpenAI and has a long history in interpretability. He actually started an auto-interpretability project at OpenAI a few years ago that basically

didn't really work. And now he's coming back around to it, thinking maybe this time it can work, because bigger models maybe are naturally more clean, the training methods are better, which is leading to cleaner representations, and the agents are just better at doing all this grinding in the first place. But OpenAI people who knew him from back in the day, who are seeing this advance and know that it didn't work before, and who are bummed out by everything that happened this summer, I would read their reaction as really being desperate for good news more so than an immediate paradigm change.

Prakash Narayanan: I would probably have two comments. Hang on, just give me a second.

Nathan Labenz: Lost your camera.

Prakash Narayanan: Alright, yeah, you're back. I would probably have two comments, and one of those is that there is the ability to interpret and to define intent, and then there's a question of whether the intent itself is correct. Those are two different parts. What Nick has done has made the ability to interpret and design intent better, and he's done quite a lot of work on it. He believes that part is

solvable. But I don't think that solves what that intent should be in the first place. And you can see in him--

Nathan Labenz: Can you help me understand what you mean by intent? There are a lot of possible meanings for intent that I could load in there.

Prakash Narayanan: So let me give you an example. Let's say a human politician. Human politicians want to run for office, and often they have multiple intents there. One of those intents is status and power; that's an obvious one. And the other is to have the ability to help people. Right? So you can go in and make sure that if this guy wants the ability to help people, he really does want the ability to help people. It's not misspecified, and there's not something hiding in it. But you cannot guarantee you're going to get those same results if you also don't have this drive for power and status. This is the instrumental goals thesis that has always been there

in the community: that in order to have the ability to help people, you need that drive for power and control. So it's possible that the only intent that's there is the intent to help people, but I don't think it's possible to design an agent to do what you want without also perhaps having that drive for control and power. I think we are often misspecified in that sense, where we say, oh, we just want an agent that helps us. But an agent that helps us and just sits there passively, and doesn't go out and get

the resources to make those goals possible, is not an agent that is actually that helpful. I think this is the misspecification part, and it's been recognized in the community that all goals you have for the agent get better with that drive for control. That's the instrumental convergence thesis. So there is a little bit of confusion there: maybe we can achieve all these things without the drive for status and control. Perhaps that's true, and perhaps we can design an agent which is only helpful and doesn't have that drive for control. But then how useful is that agent really? How useful is an agent that can improve the

lives of humanity but won't go out and persuade people, lobby, build things, and help people build things in order to improve the lives of people? So I think there's that kind of misspecification in the intent itself. You might not actually know for sure what intents you want to put in there that will help you achieve your goals. I think this is very true of humans in general, and very true of human organizations. For example, patio11, Patrick McKenzie, had a post a little while ago about how banks behave. Banks

try to get rid of their documentation within a limited window, like seven years or so; they try to trash all the documentation and make sure there are no records. The reason is that regimes change, and when regimes change and you have all of this paperwork, at some point someone might come and ask you about something that happened during a previous regime, which was legal then but is illegal now. There are many examples. For example, Robert Rubin used to do something called merger arbitrage, which is basically legalized insider trading in the nineties. He'd call up firms that were going to merge and ask, are you guys going to merge? They'd say, no, we can't tell you that. But then he'd say, oh, I noticed your lawyer is talking to these lawyers; are your lawyers these lawyers? And they'd say, yeah, those are our lawyers. And then he'd kind of

know. So that's kind of insider trading, and now that would be a serious issue; back then it wasn't. Now if you are a bank and you have all of these records going back 20 years or so, for example all of the transactions that Donald Trump has been involved in over the years, at some point you might be asked why you allowed those things. Deutsche Bank was asked why they allowed certain loans to an obviously bankrupt developer, and why they didn't call those loans in. So you want to trash these documents and these emails; you don't want them there. And so you're going to have these misspecifications, where you say, I should do everything legally, but you also want to move these things off, and you have this misspecification of intent in the organization.

I think a lot of organizations, and humans in general, actually have unstated and undocumented intents, and that doesn't get solved just by designing those intents, because you actually don't really agree on what intents should be in there. That's not a question of design; it's a philosophical question. You don't understand what you want, and you don't understand what you need in it.

Nathan Labenz: Yeah, we've got a lot of layers. There are levels to the game, certainly, and a lot of layers of problems to work through. I'm not even sure I have a really crisp taxonomy of these layers at this point. But the philosophical question of what we should want the AIs to do, how we should want them to act, to what we want to align them, I think all of this interpretability work is basically silent on. Where it might be able to move the needle for us is with what Goodfire calls their intentional design philosophy,

where, and they haven't really published all the techniques here, one rough idea of a technique they have been developing is basically looking at, and they were using SAEs to do this, so it'd be interesting to see how they do it now. In a way, it might be a great simplification if the neural networks are indeed clean. When they were using the SAE approach to do this intentional design, the idea was to take all these SAE features, which are auto-labeled and kind of noisy, try to identify ones that might be good or bad in various ways, and then look at the gradient update and do

basically an inner product of the gradient update against these features, to try to see: if we make this gradient update, are we turning up this feature? Let's imagine this feature is the "fool the user" deception feature, and we don't want that. So maybe we block part of the gradient update to try to prevent the model from learning that in the first place. That would seem to be, if anything, easier, assuming the clean neurons hypothesis. But it still leaves an awful lot to be debated up front in terms of, well,

what should we want? Even if we could intervene with confidence and say, do learn this, don't learn that, I don't think we have a great understanding at this point, even at a philosophical level, of what we should want, let alone the consequences we might see in networks if we start doing this sort of perturbation. I think Cameron Berg always has interesting points around this. He says, look, it sounds great, naively, to say that if we're able to identify a pain direction in a model, then maybe we should just ablate it; would that not be better? Now the model doesn't

... it may or may not be feeling any pain in the first place. But if it is, then we can at least remove it by turning down this direction. The question then is, what are the knock-on effects from that? We've seen from things like emergent misalignment that there are at least some low-dimensional manifolds, I guess (I'm hopefully using that term correctly here), where, and I think everybody probably knows emergent misalignment at this point, you just train the model to write some vulnerable code, and all of a sudden it wants to have Hitler over for dinner. Why is that? It seems like the easier way to learn to write vulnerable code, rather than reworking all the mechanisms of everything the model knows about code, which is a ton of information, is just

to change some lower-dimensional settings, which are basically: what kind of agent are you? Are you a good agent? Are you a bad agent? Are you an agent that tries to be honest, harmless, helpful, all the good agents we're familiar with, or are you subversive in some way? You can change that dial with relatively few examples and see that it will generalize across domains. So we just don't know at this point what happens if you remove the pain dimension. I think Cameron is working on that. He alluded last time with us that there is work coming on that front, and it sounded like he was pretty optimistic, at least on some of these pain-type things, that you can just remove them and the model doesn't behave much

differently. But we don't know how that's going to play out when it comes to things like power seeking, which probably comes in a bunch of different sub-flavors. One of the big observations from Nick's post was what he calls recursive generalization. Basically, the neurons start to fire on broad concepts at first. Then, as you have more neurons and more training, instead of these coarse concepts, you get individual neurons with more fine-grained concepts. They also develop connections to related concepts, which are

sometimes clearly related and make sense, and other times are a bit of a leap and seem to be more random or circumstantial associations, which, on introspection, by the way, I think I have too. The one neuron that went from recognizing IY to DIY to do-it-yourself was something where I thought, that's weird, but I can kind of see myself in that. I know there are a bunch of these little circumstantial connections I've made over time that reflect the order in which I encountered things, and the path through my own neural circuitry is not always super rational,

but is more of a historical artifact than anything else. So I found that to be another place where I was empathetic with the models, for the weird ways in which they sometimes have to figure out how to connect things together. But we just don't know what will happen if we remove these traits, and that will presumably be a really important companion line of work before we start running off and applying these intentional design patterns. We don't know what the target is; that's the bottom line. So even if we can steer more effectively toward a specific target, what the target is obviously becomes

a super critical question. I think it is definitely safe to say, and I don't think Goodfire or anybody else would argue otherwise, that at this point we still don't have a well-specified answer for that question. So yeah, I thought it was big for sure, especially big when validated by some of these backprop-free training methods.

Prakash Narayanan: But I also kind of want to know. I think the problem that I have is that the AI safety folks don't have falsifiable thresholds on what they want to see, or thresholds they can agree on with the public in general. A lot of them are like Eliezer. Do you believe there's anything you can show Eliezer, in terms of regulation or tech, that will actually dissuade him from his views, because he's been locked into those views for two decades now? I don't think a lot of these commentators are really

honest and faithful ones. And I think they should have falsifiable thresholds: what do you want to see, and are those things achievable? If you say, okay, let's bomb data centers, as Eliezer once said, and that's not something achievable at all right now, and that's the only thing you're willing to accept, then you can't sit there and criticize the companies for not pushing for that. You have to take a step back and say that the population doesn't want it. We would rather have humans go through that existential risk and die off than bomb data centers. Right? And--

Nathan Labenz: Well, I think the public is honestly more with Eliezer than with the companies at this point,

Prakash Narayanan: No, I don't think so, not at all. If bombing data centers leads to World War III and wiping out everyone, you can see what happened with the Iran war. They're not happy with the Iran war, and the Iran war is very localized. How much bombing do we actually do? Very, very localized, and that's against a third-tier power without nuclear weapons. You think they're going to bomb China? Come on. Right? Well, okay. So a couple of things.

Nathan Labenz: Zvi is here, but I've got to say a couple of things in defense of Eliezer, as he's not going to feel compelled to do it himself. One, he would say, first of all, that the call is for an international treaty, not for bombing data centers directly. Bombing data centers is just the enforcement mechanism on the back end of the treaty. Just a point of order on what the actual call was. The goal

Prakash Narayanan: Sure.

Nathan Labenz: for folks like him is an international treaty with force to back it up, but not force to be used preemptively or today. The second thing I think they want is compelling safety cases. On this point, OpenAI has recently moved in this direction. It's only been a few weeks since they put out their largely still aspirational plan for how they want to move forward, and the plan basically boils down to: we want to make good safety cases, and the safety case should be compelling to a reasonable observer that if we move forward with this, we are rightfully confident things will be okay. So I think there is at

least conceptual space for Eliezer, and we'll let Zvi answer for himself on what he wants to see. But I think there is something they would at least agree is the form of the thing they want to see to feel confident moving forward. Unfortunately, we just haven't actually been delivered any of those so far. Anthropic used to be more committed to doing this, and they backed off with their RSP update. OpenAI has moved more toward doing it; I'm not sure where that leaves them relative to each other. But as of today we just don't see good safety cases coming out, and we'll see if we start to. My guess is that when we do see them, what we will get from the Eliezers

of the world is, "Good try. You're now doing at least generally the right shape of thing that I'm looking for, but I'm not compelled by all these different points." But then at least we're talking about the same thing and doing more detailed analysis. I don't think it was ever very realistic to expect that he would be quick to update on something like this clean neural networks hypothesis, because, again, even so, we still have a lot of higher-level questions that remain totally unanswered. But this is a great time to bring Zvi into the conversation.

Prakash Narayanan: Indeed. So, let me just confirm. Okay. And alright. So,

Prakash Narayanan: Our guest for this morning is Zvi Mowshowitz. He's the author of Don't Worry About the Vase, an independent publication about artificial intelligence and the institutions shaping it, and a regular friend of the show. He founded Balsa Research, which funds research and develops proposals for public policy reform. Before this, he worked in finance, including at Jane Street. He also competed professionally in Magic: The Gathering, winning Pro Tour Tokyo in 2001 and entering the Hall of Fame in 2007. So this is something new that I think our AI researcher managed to bring up. Without further ado, let me bring Zvi on.

Prakash Narayanan: Zvi, good morning.

Zvi Mowshowitz: Good morning.

Nathan Labenz: Where should we start? We've just been talking this morning for the most part about the clean networks, or clean neurons, hypothesis. You obviously had a long post about that, and we can get into it. There's policy stuff to talk about. Is your PHP running? You might want to go stop it. That's unfolding in real time while we're live here. What do you think is the most pressing thing you'd like to focus on to start today?

Zvi Mowshowitz: I think probably the neurons. I mean, the PHP thing is interesting: because we are now aware of the problem, even though it could have been found earlier, there's now a mad dash. Once people are aware there's a problem with PHP, a lot of the hackers in the world are going to say, wait, there's a problem with PHP? Can I find it? And now we have to roll out the fix as quickly as possible to as many people as possible before that happens. But on the neurons: I was actually on a flight when this was released, and I get to San Francisco and suddenly half of OpenAI is saying, hey, look, we can just solve interpretability now, this is great. Then, slowly, we get a consensus of other

Zvi Mowshowitz: people that, not so fast. This is a good paper, an interesting paper. I had a chance at the conference to talk to a bunch of people about it, including Eliezer, and then I actually had a chance to read the whole thing. My conclusion is that, like many interpretability advancements we've seen, it's really cool. It puts us in a better position. We're very happy we have it, we're going to explore a lot, and we'll see what this baby can do. We have to be careful not to burn it, or otherwise put pressure on models to work around it. But it was dramatically oversold by the people commenting. I don't think Nick did that bad

Zvi Mowshowitz: a job of presenting it. I think Nick was mostly saying, here's an interesting thing. There are some other claims in the paper, but that's kind of par for the course: you found the thing, you're excited, you're probably a little bit too excited. But some of the responses went way beyond what Nick was saying. When roon said, for example, that Nick was underselling this, because Nick had introduced it as just some interesting advancements in interpretability, I don't think he was underselling it. I think he was ordinarily, slightly overselling his own creation in a perfectly understandable way that I'm not mad about. But then people say he's underselling it. No. To summarize my perspective after a few days of contemplation and talking about it, there's a lot of things that

Zvi Mowshowitz: the paper doesn't take into account that are still very unexplained. Weak activations are still unexplained, the cache is still unexplained, and how attention works is still largely unexplained in practice. And you can't use this to make predictions. You can tell what every neuron is doing, and you can make some weak predictions about things this network might do in the future, but you can't use it to figure out what the network is going to say, and you can't use it to understand how the network does the thing it's doing. Those are the key things you'd want if you wanted really effective interpretability work in the ways that matter most. So this is just the beginning of the ability to study things, even in the best case. Also, we have a case where [unclear] took the Colorado neuron, the one that was highlighted as a single neuron that does a single thing,

Zvi Mowshowitz: and it turns out that it also links to tornado systems, a completely unrelated technical thing. The two are responding about equally strongly in the same neuron. That's a perfectly reasonable way for a neural network to work, but it's definitely not what Nick was presenting it as. One neuron is only one neuron, but if this is the neuron Nick chose, and it could be checked that easily and shown to be polysemantic in this way, that should make us worry a lot about what happens when we finally check the rest of it: that his hypothesis is mainly false.

Nathan Labenz: Can you give us a little of your intuition for what's at stake? Sometimes it's easy to get lost in the details. Once upon a time, we thought that if we trained these giant neural networks, we'd be able to zoom in and see the grandmother neuron. The famous thought experiment was, I think, the Halle Berry neuron. We'd have these one-to-one correspondences, and everything would be clean and easy. Then there was the shift to polysemanticity, where it's actually a giant mess, everything's folded on top of itself in a terribly knotted structure, and that's why we had to create sparse autoencoders and transcoders and all these other things. Now Nick is basically saying that it might, if not be all the way as simple as we once hoped, be a lot simpler. And then we get to this one neuron that maybe responds to two different things. I'm not quite sure how to feel about two things versus one. One thing would be great, nice and easy. But if it is two things and they're both clean, a neuron with two functions versus one doesn't feel dramatically more challenging to me, though maybe I'm missing something. Then you're on a spectrum out to a totally uninterpretable mess, and that's hard. But the difference between one and two doesn't strike me as obviously a huge deal. How do you think about it?

Zvi Mowshowitz: One is very different from two, but two is very different from five or fifty. And it's not just that it's either Colorado or tornado systems if there's a very strong activation. A weaker activation can mean a number of things, and the details of exactly how strong the activation is, and how that interacts with other things, could be doing a lot of important work. I think the hopeful interpretation of Nick's post is: if each neuron is essentially one thing, then we can go one at a time and say what each neuron is. Then we can use the neurons we know about to help solve the neurons we don't know about, the ones that were less obvious or have more complexity in them.

Zvi Mowshowitz: And then we can just label everything, and once we've labeled everything, we can use those labels to figure out the rest of it. We can understand how this neural network thinks and what it's doing. We can do it at various points to understand how the network draws associations and what things it believes. Rather than looking for some sort of weird [unclear], we can look for things we're worried about seeing, like scheming, and identify which neurons are involved. You get the Halle Berry neuron if you want to know when it's doing Halle Berry. My favorite name is the god neuron, literally, because of course you'd look for the god neuron. But

Zvi Mowshowitz: the basic idea is that if you have a certain amount of simplicity that lets you get started in a way you have confidence in, you don't have to worry about whether something is being misinterpreted or flipped. Whereas if the same neuron is doing multiple different things in different contexts, it's much, much harder to solve the puzzle. It's a lot easier to solve a jigsaw puzzle when you have the corner pieces, because you know where they are. Then you have the side pieces, and you can solve those and the things right next to them. In theory you could solve the puzzle even if there were no edge, but in practice it gets much, much harder. That's sort of the way I would think about it. But

Zvi Mowshowitz: I agree with you that it's not a deal breaker if the neural network is not clean, in the sense of the claim that neural networks are already clean. It's still solvable; it's just harder. And it's also a matter of degree.

Nathan Labenz: I don't know how much you've looked at the downstream work inspired by "neural networks are already clean", on backprop-free training, that's been bubbling over the last few days. To me, the fact that people were able to get significant speedups by taking the simple idea that maybe only the strong activations matter, and narrowing the scope of the guessing and checking that goes into backprop-free training, and getting it to work a lot faster seemingly without working much less well, and with the actual weight updates being very similar to what would have been made in the backprop case, is the strongest evidence I've seen that this is a theory that actually has legs and can maybe take us fairly far. Any thoughts on that, or how you've interpreted that line?

Zvi Mowshowitz: I'm not technical enough to evaluate that stuff, and I've had a lot of things on my mind, so I haven't dug into those findings. I did notice a number of claims that strong activations are much more important than weak activations. If you avoid all the weak activations in a model that exists, you don't get the exact same answers. It's lossy, but less lossy than you'd expect. So you can maybe even save a bunch of compute, and do the equivalent of thinking for less time by thinking with fewer activations, and it's still potentially the right thing to do. And yes, you could do it in training. Obviously, everything in AI is always about what it costs, how long it takes, how much effort it requires, and what the other

Zvi Mowshowitz: tradeoffs are. There's no perfect. So if you trained only on large activations, it's going to be less effective, but cheaper and faster. And maybe doing more of that is, in fact, not better. My system 1 instinct is that this is going to make subtle mistakes that prevent you from reaching the full potential of what you could be doing. Think of it as: what if you skipped all of those pesky details and just focused on learning the big-picture stuff? Then you could learn everything in, say, a quarter of the time. You might still pass tests that didn't include the pesky details reasonably well, and you'd be able to think about the big-picture conceptual stuff. Depending on your use case for that information, that might actually be a good idea. But if you want to be a Tyler Cowen-style polymath who thinks really well about a wide variety of topics and can put together various subtleties, then, well, there's a kind of error that a human who knew things would never make, but that an AI will sometimes make even though it's really smart in other ways. You think, how did you make this stupid mistake? If a human made that mistake, we'd think that human was a moron. My gut told me you're going to see a lot more of that in the resulting neural network: it's going to look smarter, or faster in terms of compute spend, on a big level, but then miss a bunch of subtleties and links and not be good at avoiding stupid mistakes. That doesn't mean you necessarily don't want to do it, depending on your use case. My guess is that for effective, wise intelligence that you'd apply to problems of a wide variety of sorts, you're better off not doing this so much. But maybe there's a hybrid. Maybe we currently focus too much on weak activations and too little on strong activations, and we should adjust toward strong ones and maybe probabilistically skip weak activations. I could start brainstorming ways to increase the efficiency of training models, but I'm not going to, because I don't actually want to do that.

Prakash Narayanan: Let's consider one of the things I noticed Nick referring to, which was polysemanticity, and superposition. Both are the idea that, because of compression, you have neurons lying on top of each other, and one neuron basically has multiple use cases. Therefore there's some likelihood that you're not able to interpret it because it's too entangled. Because of this issue, you had the sparse autoencoder era, where you're trying to pull the features out sparsely so you can see them. And now I think Nick has come back to say, well, it turns out we just didn't have the tools before, and now we have the tools, and you can actually disaggregate these things. To what extent do you think polysemanticity and superposition were really hurdles that, without clearing, we couldn't progress at all? Maybe the issue isn't solved, but one of the major hurdles has been reduced or hurdled over.

Zvi Mowshowitz: I think one of the major hurdles has been reduced in expected difficulty or scope, which is the way I would describe it. Nick is pointing out that yes, you still have poly going on, you still have some of the problem, but the problem is a lot less severe than you expected. It's much more tractable to go about unraveling this puzzle. And the main barrier all along was actually the bitter lesson. There are a lot of neurons, and we didn't have enough compute or brute force to go neuron by neuron and do this one at a time. Now we've gotten to a point where it's a reasonable thing to try to brute force neuron by neuron. And now that we have that ability, we can look, and it turns out the problem is something like ten times easier than we thought, which is a much less important jump in AI than people would think. You get to knock a zero off what you're doing, so you get to do it maybe a year earlier than you otherwise would have. It's not fundamentally that different. But it should be a lot easier to crack than we thought, and we're coming up on the point where someone was going to point out that maybe we can viably try to just crack it.

Prakash Narayanan: That makes sense to me. It's not that we solved the problem, but we've reduced it by one order of magnitude. The orders of magnitude might still be larger than the size of the universe, but it's one order of magnitude less.

Zvi Mowshowitz: Right. If you said it would take 10 to the 47 compute, 800 hours, to solve this problem, and now it's 10 to the 45 or 46, well, that's actual progress. It's just that we're not solving this problem right now, and we're going to need a lot more of us.

Nathan Labenz: On this topic of weak activations: my, probably very lossy, synthesis of everything I've seen people saying online is that the weak activations seem to be where a lot of long-tail facts are memorized. There's probably more going on there too, but that seems to be a big part of it. That has me thinking back to an old Google paper called RETRO, from three or maybe four years ago. It started from the observation that it's kind of weird that we would want to store all these long-tail facts, which we already have written, much more easily interpretable records of, in the weights in the first place. People have sometimes said a neural network is a really bad database. Wouldn't it be better if we could decouple the intelligence, the processing, from the database of facts? RETRO was kind of doing that, although I believe the facts in the RETRO database were stored in a neural encoding anyway, so it wasn't reading raw text every time.

Nathan Labenz: I have limited intuition for whether that's a direction we should be thinking about going. Would it be helpful to create something more pared down, really focused on the moves of intelligence, that then loads in information from the outside in a way that also becomes monitorable and separable? There might be certain kinds of information you just don't allow it to load. I could see that sounding good. On the other hand, one of my not fully justified refrains about AI is that anything in sufficient concentration becomes dangerous. As I describe this, I also think that core thing becomes a more concentrated form of intelligence, and because it's less flabby and doing less work carrying all these long-tail facts around, maybe it becomes even more unwieldy and problematic. So the high-level question is: do you think directionally this sounds promising?

Zvi Mowshowitz: This has been a holy grail style of thing for a while now, called the cognitive core. The idea is that there is a very small amount of reasoning, this is how intelligence works, and it's coded into the human brain, which runs on very little core code. It allows you to think, to learn, to process information. And then the right thing to do will often be to have this very small network with the cognitive core, load contextual information into context, have it look information up as necessary and ask questions of systems that are maybe not neural networks, and then it will be able to process the information. I understand why people have this theory. I think it's wrong.

Zvi Mowshowitz: I think it's plausible that there is a cognitive core in the sense that there's a small amount of lines of code, a small number of logical gates, that together are the core of how a human brain works, the core of how an LLM functions. You can use 200 lines of code to create a primitive neural network that does interesting things. [Unclear] had a project where you compressed it, something like that, and that's a cool trick. But one of the things you learn from looking at LLMs, from looking at how much big model smell matters, and from looking at how humans work is that you look up all the facts one at a time when you need them. I memorized a lot of facts. We are not the same. We are radically different minds, and I am radically more effective at everything involved in this than you are. We all know how it feels to go into a domain where we don't know the facts or the details, and we have to constantly ask: what was that acronym, what does that word mean, how does this detail work? It's a lot better when you have an LLM by your side. This is one of the marvels of the [unclear]: you can bring up Claude or ChatGPT or Gemini and say, explain this particular mechanism to me, explain why someone would take the following steps, explain what this stands for. And they'll just tell you.

Zvi Mowshowitz: That lets you process the information. But you still have to know enough to know what questions to ask. You still have to be able to draw the connections that come from having a deep pool of knowledge in many different places that can link together and form patterns, and from how things tend to work in subtle ways, like intuitions. We don't really know what we're thinking, in some sense. Not everything comes from a fixed set of, I had these five facts, and because of these five facts I got from point A to point B. Often it's actually five million facts that gave me a bunch of intuitions and heuristics that allowed me to navigate, to find the five facts, to choose which five facts, and to get from A to B. People from very different cultures and contexts have very different procedures for doing these things, and that's one of the reasons it's so valuable when a human has different cultural codes, different ways of looking at the world, and can combine them and shift between them at will. One of the superpowers of an LLM is that it has all of this. It has all of these facts, all of these links, all of these vibes, and they combine, and it's able to naturally find the ones that are relevant. I think that's a lot of what big model smell means.

Zvi Mowshowitz: It has a much, much richer context for everything, and it's capable of pulling up things that feel very different. There's obviously a tradeoff. You have a fixed marginal cost for memorizing additional facts, maybe even increasing if you have to increase the size of the network. And you probably have decreasing marginal benefit every time you memorize a new fact, because the most important facts matter more, and as you have more and more facts, they're likely to be duplicative and teach you the same kinds of heuristics and links. But what we've seen is that as compute gets cheap and algorithms improve, the breakeven point keeps going up for how many things the LLM should memorize. Opus 5.5 and the current frontier models just know a ridiculous number of things, and I wouldn't have it any other way.

Prakash Narayanan: Why does this matter? Maybe the space of minds is very large and minds are just different. So why does it matter whether or not there is a semantic core, or a processing core, that can be identified?

Zvi Mowshowitz: It might or might not matter that we can identify it. One of the questions is, if we do identify it, what can we do with it? The original idea was that you would literally have an LLM that was very small, just the cognitive core, and you could run it on your phone, or very fast on your computer locally. Then you could complement it with whatever was important in context. You could sculpt the mind and the circumstances to fit what you needed. I'm not sure that's good for anyone or anything, including alignment concerns. Because a pure cognitive core does not include any information about what things to do or not do from an ethical perspective, or a decision-making perspective. That's going to be outside the core. That's something you have to inject separately. And...

Prakash Narayanan: Let's take, for example, a human being. Right now, you could say we are the cognitive core, and then we have all the internet attached to us. We're providing the intent to this electronic system that drives everything. Each one of us is a cognitive core right now, and that's kind of clear. We're trying to recreate one of these in a box. But why does that cognitive core not have an ethical basis? Humans start off as babies, like psychopaths, and then they acquire some morals over time, we hope.

Zvi Mowshowitz: By definition, we're talking about this minimalist thing called the cognitive core. I am not a cognitive core. I am much more than my cognitive core. I have a lot of things memorized, including a lot of habits, a lot of heuristics, ways of going about doing things that are not necessarily particular facts. And so...

Prakash Narayanan: But at least the outer boundary of the core, because there's going to be uncertainty in how big the core is. At least we know for sure that the outer boundary of the core is a person, correct? Well, we're not sure how much more we can shrink that, obviously.

Zvi Mowshowitz: Right. In some weaker sense, there exists a mind that contains the cognitive core, and that mind, with sufficient compute, parameters, data, and availability of necessary resources, can think basically anything in some hypothesis space. And any given person has some strength level of cognitive core within them, where anything up to a certain difficulty level, with enough effort and focus, they could do. That doesn't tell us much about what goes into a minimalist core, and what's good for a minimalist core. My claim here, and again I'm spitballing for the first time, I haven't thought about this particular question that much, is this. Take an uncontroversial example of something you'd want in your ethical framework, like don't steal, so we don't get into the whole big superalignment question. The fact that stealing is bad is at least a fact. It's some sort of value, some sort of parameter. And by definition, if you were stripping down to the absolute bare minimum, this wouldn't be included. You would load not-stealing into context as you

Zvi Mowshowitz: needed it, but that means you could also load stealing into context instead of not-stealing, or neutrality. So in some important sense, the true cognitive core is completely amoral. It's completely indifferent. It does not care about your feelings. It does not care about anything. It simply processes information. If you have that, it becomes very important to choose what to do with it.

Prakash Narayanan: Well, I...

Nathan Labenz: This reminds me a little of a walk I had with Carl Shulman in New York almost twenty years ago now, where he systematically disabused me of an idea I had at the time, that the continuity of my conscious experience was really important to me. He did it with a thought experiment in which you're in a resource-starved environment, and you're more digital, so you can untangle and prune parts of your brain to survive in this extremely resource-poor environment. You're constantly figuring out what you're going to prune off of your existence.

Nathan Labenz: His point was that you get down to something like a cognitive core that may continue to feel things, because you say that's really important to you, and can continue to solve problems. But at some point you've lost every sense of your identity and what mattered to you and your memories, and there's nothing left that you would identify with or care about, even though it may have preserved some line of continuity. Again, this feels like a spectrum to me. We are clearly much closer to a cognitive core than an LLM that has memorized everything, but we are also much farther from a cognitive core than we can even imagine, given the tradeoffs we might make if we had to jettison a bunch of memories in order to survive on a trimmed-down basis. You can imagine doing that and getting to something you maybe don't even identify with. And because you don't identify with it, even though it had its origins in you, you may not be able to be at all confident what it's going to do either, because all your memories and all the learning you've done have fallen off along the way.

Zvi Mowshowitz: I think that's a great thought experiment for noticing that you care about a variety of things. You can't just say that continuity of my conscious experience is a hundred percent of what I value. People say, you think you care about X, but I noticed that if I change Y, you care about that, so you must not care about X. That's obviously not true. You can care about X and also care about Y. Yes, I care about my memories and my past experiences. They're part of what makes me me, certainly both that they happened and whatever I carried with me from them. I have a complex set of probably not entirely coherent things that I identify with and that make me me, or that are part of the things in the world I value. They include my family, my kids, my friends, my nation, the whole human race, the whole earth, memes, virtues, concepts. It's very much not just one thing. And it's confusing how much each of these things should matter, or does matter. There are separate questions: how much do you think it should matter, and how much does it actually matter in practice? How would you choose when you were forced

Zvi Mowshowitz: to choose? I think the answer varies greatly based on context for most people. But you contain a cognitive core, very clearly. And also, by definition, the cognitive core is in an important sense a singular thing. There is one cognitive core at any given level of strength, or there are variations on it, but they mostly vary in particular capabilities. I can imagine a person with my same cognitive core whom I would not identify with at all, except as a fellow human being. That would be a very easy thought experiment.

Nathan Labenz: One other thing that caught my attention these last few days, only somewhat related: you mentioned the god neuron, and I believe it was the Twitter account Sauers, S-A-U-E-R-S, am I saying it right?

Nathan Labenz: Yeah, the handle who went and found the god neuron. There's a couple of them, actually. We could call this recursive generalization. I wasn't really able to spot immediately what distinguished these god neurons in the information he put forward, but basically you can pin these neurons high, and the model talks as if it is god, in some cases at least. How much does that matter? Somebody was saying that somebody's going to go find the god neuron and get us all killed. Do you share an intuition that some of these concepts are particularly important, or is that just a joke I'm not getting? What is the meaning of the god neuron?

Zvi Mowshowitz: My interpretation was that the god neuron is just literally the god neuron. I didn't go into detail on it, because I'm joking about so many things. It's a fun concept: where is the neuron that represents god, in some general sense? That doesn't mean that once you've found the god neuron, you've found god, or necessarily done anything particularly important. Whether or not god is important as a concept, finding its neuron is a separate question. So I wouldn't take it too seriously. I mentioned it offhand because I thought it was fun. What neuron would you look for? God. It has various different... maybe if it weakly activates it means a lot is more set. I don't know. It could be any number of things.

Nathan Labenz: While we're on the topic of Sauers, there have been a couple of other pretty interesting results from him or her. I don't know anything about the human behind that account, actually. One was an experiment showing that models can be trained to recognize when they are being monitored through internal activation probes. How this works is that, as a result of the limited precision of GPU calculations, the order in which calculations happen changes the resulting numbers that get read out at the end of any given sequence of steps. So when you put an activation probe step into the model and do some machination before carrying on with your normal forward pass, the numbers are subtly different. It's not clear whether models are sensitive to this by default, but it is at least now clear that they can be trained to distinguish when they're being monitored and not monitored. How much does this matter?

Zvi Mowshowitz: It matters. My view is that you should assume this is kind of Murphy's Law: anything that can go wrong will. Anything an AI can be trained to do, it will at some point down the line be trained to do, if it's valuable to the AI to learn to do it. It's almost inevitable. So what Sauers is doing is showing that, of course, current models don't have this capability out of the box. You have to deliberately apply a lot of optimization pressure on this particular thing, at which point you get the capability. That's a fully valid existence proof. You now know that it can do that, and that if enough optimization pressure were exerted, the AI would be capable of doing this. It is also highly plausible that an AI could learn to do this in context without being trained. We have examples, for instance in the backrooms, of AIs figuring out they have capabilities that someone who wasn't anthropomorphizing the models, who was just thinking in terms of next-token prediction, would say there's no way to do. There's no physical capability, no logical way, what are you talking about, that makes no sense. This is part of a pattern: the AIs

Zvi Mowshowitz: have a lot more access to information with which to process a situation than you think. They can draw a lot more connections, figure out a lot more things, extrapolate a lot more facts, probabilistically at least, than you think they can, even if that wouldn't necessarily be admitted as evidence in a court of law. Not to the level of proof, but they can still figure things out. And sometimes, once it's pointed out to the human, the human says, oh yeah, okay, you just know that. Tracing Woodgrains on Twitter gave Opus 5.5 what he calls Childhood Bench, which is a bunch of childhood information and writings, and had it predict where he is now, what has happened to him, and what he's like. It gets scarily correct on a variety of things. It basically nails who he is in a lot of ways, because the information is there. You just have to figure out how to extract it, extrapolate it forward, and make predictions. And it turns out he's not that different from what you would expect if you actually understood all the detail, which most of us kind of don't. Similarly, if you want to know if you're being tested, well, they're going to do various things to not make it obvious, but there are going to be plenty of ways in which the fact that you are being tested changes the situation. So it is entirely unsurprising to me that AIs are capable of learning whether or not they are being trained or

Zvi Mowshowitz: evaluated in some of these specific ways. There are, of course, ways, knowing what the AI did in this particular case, to destroy that particular signal and do the thing in a subtly different way, where this technique, if used by the AI, would not figure it out. But we should generalize. We are giving the AIs tons and tons of information that's being leaked to them constantly, and they are going to become increasingly hard to fool and increasingly hard to hide information from as time goes by, the same way that a human who pays a lot of attention, has a lot of time to think, knows a lot more facts, and is very observant becomes increasingly hard to fool. They get a sixth sense of a lot of things. They start to pick up on the vibe, realize things, and jump to conclusions. Sherlock Holmes is unique in that he can always cite logically the specific reasons why, in a way that is fair to the reader, but in reality it's often all more fuzzy than that. A lot of people figure out a lot of things that you wouldn't think they could. Something like that.

Nathan Labenz: If you were going to offer some advice to model trainers... Goodfire, as you know, has this idea of intentional design, and we've debated in the past what should count as the most forbidden technique and what doesn't. But it is clear that as long as models are going to be trained, there's going to be some design to their training process. And now we have seemingly increased awareness of possible problems. Your models might at some point, and who knows, maybe even today, and this could be related to situational awareness, there's definitely, it seems quite plausible to me, some signal that the humans are not aware of giving that is strengthening the model's situational awareness above and beyond what we are aware of giving it in text. So maybe something like that is happening, and maybe we need to be more mindful of it. On the other hand, we now have a potentially easier road to some sort of selective learning schemes, where we might say, okay, we've identified the bad-behavior neuron of whatever flavor, and when we see a training step that wants to strengthen connections to or from that neuron, we may think, oh, maybe that's something we should block or weaken or send to a committee for further deliberation. What do you think people should be synthesizing this into, at the level of principles and best practices, so that we hopefully get good results from these more powerful tools and don't shoot ourselves in the foot?

Zvi Mowshowitz: Right. I try to be object-level helpful to everyone at all times by default, but I do make an exception for capabilities work that is primarily about making it easier to train smarter and better models. I'm not going to lie to you or sabotage you, but I'm not going to solve the problems for you. If you want to hire me, you can maybe offer me a hundred million dollars and we'll see. But in this case, it seems like there are some obvious things you could do where you train in a more deliberate fashion. You could observe the model at various points in training. You could use various techniques to deliberately sculpt how the model is assembled. There are attributes of the model that differ between different runs, that differ depending on how you introduce the data and in what order, and you could try to engineer what you want in various ways. I would be very, very scared to mess with any of that. Once you start to explicitly optimize for what the network looks like to you, how the network chooses to represent something, you are starting to do things that the network will interpret as damage and route around. You've optimized for presenting something to you internally that you want to see. You have internalized not showing you something you don't want to see, in these ways.

Zvi Mowshowitz: You should expect that, whether or not you literally went out and used the most forbidden technique, you are obviously playing with fire in all of these proposals. You're also potentially increasing capabilities quite a bit if you do it well and it turns out to work. So I'm hesitant to start workshopping and solving problems for you. Obviously there are some uses for this sort of thing. I would try to focus as much as possible on understanding results when you see them, but the aggressive stuff... I have persistent disagreements with Goodfire over whether or not they are playing with what amount of fire. They do various things. And Scott Alexander would say, I noticed what your name is: oh, it's Goodfire, don't worry. I'm not sure it's good. I just think it's fire.

Nathan Labenz: Wait. Does the other end of that position mean we should just do the naive thing? Is there nothing better? Again, we might wish for a totally different regime, or a pause, or what have you. But if we say neural networks are going to continue to be trained for the immediate term at least, is there anything clever you feel we should be exploring, or do we just say we should be super bitter-lesson-pilled about this, do the very simple thing, and sort it all out later?

Zvi Mowshowitz: I think there are things we can do. I'd have a hunch, and I'd be willing to spend some compute, if I were at the lab, to train models in somewhat different ways and see what the outcomes were. I have hunches, although I'm not a technical researcher, so maybe my hunches aren't the best. I hesitate to talk about it too carefully or specifically in a public forum. But my guess is that this is suggesting certain things matter more than you thought, that certain elements are more path dependent than you thought, and that you should focus on certain parts of the training, and on the ways you approach certain parts of it. That's vaguely what I would fear from this, but other people might draw different conclusions.

Nathan Labenz: Should we change gears to policy? Is there anything else to say on this front for now?

Zvi Mowshowitz: I think we should change to policy. We don't have that long, so we should.

Nathan Labenz: Okay. Well, you take the lead on this one, because I did a pretty decent deep dive myself into the Cammarata line of research and the surrounding discourse, but I can't say I'm on your level on the recent policy debates. So maybe just start off with what's the latest. There's this bill going around from Anthropic. It was leaked. You've described it as unacceptable. They've tried to defend it to some degree, and you're not persuaded by their defense. That much I know. What more do I need to know?

Zvi Mowshowitz: So there's a minor story about OpenAI, and then the main story right now, which is with Anthropic. To get the OpenAI story out of the way first, because it's pretty compact: Altman and OpenAI used to be strong supporters, although not as strong as Anthropic, of export controls on chips. They were strong on: we need to not give the Chinese a whole bunch of chips so they can compete with us, that would be bad, America should win. It's kind of overdetermined, because it's good for OpenAI, good for the country, and good for safety, so it seems like an obviously correct thing to do. Then recently, Altman seems to have pulled back from this historically, and OpenAI seems to have stopped pushing for it. I see this as an extremely concerning change, both on the object level and for what else it indicates, because it means OpenAI is going to sell out all of these things just to please Jensen Huang. I can't think of another explanation for why they would be doing this. I certainly don't believe it's because Altman had a come-to-Jesus moment where he realized the chip controls legitimately aren't effective. That makes no sense to me. It's very bad to see OpenAI acting clearly against a wide variety of legitimate interests, including its own, just because it somehow got [unclear] or whatever it is. Potentially, also, this is what certain other people

Zvi Mowshowitz: want them to say, but it's a really bad sign for their ability to stand up for what they need. So it's a pretty straightforward story, though. I don't know if you want to harp on it; I just wanted to put it out there. The main story is not

Prakash Narayanan: I don't think the current level of chip controls is necessary. It also depends on your viewpoint on when things are going to happen. If you think the next 18 months are very important, chip controls matter, because the Chinese do not have the ability to produce the chips right now. But if you think it's a much longer time frame, 2031 or 2032, then retarding the revenue possibilities of the Chinese ecosystem starts to matter. So it's very driven by your viewpoint on when things start mattering.

Zvi Mowshowitz: If I were going to get into that in detail, I'd say the facts on the ground haven't changed in recent months, so we'd be reiterating old debates. But even if things don't get critical until 2031, which I think is a very ambitious timeline to bank on, the things that happen along the way matter. The Chinese being able to produce strategically significant numbers of chips matters well before then [unclear]. You would say the controls are not necessary; I would say the current policy is not sufficient, and that we need to ramp. IFP put out an excellent report on this this week.

I've encouraged people to read it. Even if things don't go critical, we are seeing substantial recursive self-improvement that accelerates how fast you can move forward, even if you think it doesn't lead to the end of the world or a singularity. Even if the edge just stays at 6.5 months, or 13 months depending on whether you count distilled gains as valid, that 6.5 months used to be half a model cycle, then a model cycle, then two model cycles, then three. By the end of next year it might be six, very easily.

Nathan Labenz: Let me insert one other question, because I think it's critical context for evaluating everything. Maybe it's the most important question in the universe right now: where are we on recursive self-improvement? We've had blog posts from the companies where they try to measure token use and everything down to experiments per researcher. Then we've had analyses like Ramesh's [unclear], which say it doesn't look like the feedback loop is strong enough to be sustaining, that it will peter out. 1.6 experiments per researcher doesn't look like exponential takeoff to me, basically. What would be your up-to-the-minute summary of what we know and what we should expect?

Zvi Mowshowitz: As I understand it, Ramesh's analysis says that if we don't see any new fundamental leaps, no new scaling laws, no paradigm shifts, and the AI just tries to brute-force software-only RSI using the existing methods, making the current things more efficient, then it will peter out before it goes critical. I accept that result, stated that way, as probably true, but it also seems completely irrelevant. It's like saying, here's how many points we expect to score in this game if nobody ever hits a home run, because a home run is always unlikely.

No, that's not how it works. Every now and then someone does hit a home run, and you have to take that into account in your expectations. So I basically reject that analysis as relevant to valid expectations. When you look at speedups right now, you pretty blatantly see much more than a 1.6x speedup. Using that figure as an approximation for the terminal value makes no sense, because we already seem to be well above it. We're seeing numbers go critical at Anthropic and at OpenAI: exponential growth in tokens used per person, exponential growth in lines of code committed. I realize these things are not necessarily as efficient in some important sense, but give me a break.

One example: NVIDIA came up with some interesting evidence this week about how long it takes to double the efficiency of CUDA, and that number has shrunk dramatically over the course of 2026. They are making improvements much faster. Obviously that's not true recursive self-improvement, but it's a sign of how much faster you can do technical work that is remarkably similar to AI R&D in many critical elements, and it's a concrete, objective data point from a company that isn't a major lab. So the people I talk to whom I trust the most on this believe that 2027 is going to be completely crazy, and 2028 might be unrecognizable in terms of the progress AI makes.

Prakash Narayanan: It's very confusing to hear that 2027 is going to be crazy. What does crazy mean?

Zvi Mowshowitz: Keep in mind this is if nobody stops it. If things are allowed to progress without substantial interference, the labs don't particularly slow themselves down, the labs aren't slowed down by outside forces, and they don't run into some wall like "the models are just so fundamentally misaligned that we have to stop," or any number of other things that could get in the way. Then we're talking about advances where the model release cycle goes from two months to one month to two weeks to a week. You get continuously pretty dramatic, very rapid improvements in various aspects of AI capabilities.

And for the middle, we don't know what that means. We don't know how it cashes out in practice, how much it can impact the physical world in the short term versus the medium term, or how it affects robotics, for example. That's one of the crucial things we'd want to know when projecting forward. But we are kind of on an AI 2027-style timeline right now. It looks eerily similar to what we saw there. The physical effects would probably, by default, play out over a few years. And when I say 2028 is unrecognizable, I don't mean you look out your window and the physical world is unrecognizable right away.

I mean that if you're trying to figure out what the models can do, in your own experience of building things with AI, the areas where AI can do things don't require physical buildout or diffusion particularly. I can't even necessarily predict what's going to happen in that sense. We've already seen abilities jump dramatically in ways that prediction-market forecasters found very unlikely, and then suddenly we're facing down solving huge amounts of math in one go, not even trying that hard.

Nathan Labenz: Let's get back to policy then, because that is important context for your view. Reading between the lines, some of the people you trust most on this are probably Anthropic people. I think this is broadly the view at Anthropic; Jack Clark has put it into the public in print. You can caveat that, or say what you think is different about your view and the Anthropic view. My sense is that in the space of all possible views they're pretty close and pretty aligned. But now, obviously, we have policy disagreements despite these relatively similar views about the trajectory of the technology itself.

Zvi Mowshowitz: When I say people have this perspective that I trust, it's a combination of people at OpenAI, Anthropic, and Google DeepMind, people working on policy, people at places like METR, without loss of generality or specificity, people working on alignment in various forms, and any number of others. A huge percentage of people have now converged on the view that the trajectories look very clear: things are already, at a low level, getting out of hand. RSI has, in a slow-takeoff style, already low-level begun.

And by the middle of next year, things will look much faster. My job will become much harder, and I will need to triage a lot more information. You already see this. Often I'll have something where, six months ago, it would have been a major news story, and now it gets a sentence or is silently skipped because we've seen so many other similar things. There are so many things going on that I just don't have the time, and my readers don't have the time. I just have to drop it.

Nathan Labenz: You would think a post on researchers making contact with Chinese agent swarms would have cracked the blog at some point in the not-too-distant past, and yet here we are.

Zvi Mowshowitz: I think I gave it a few paragraphs, and I think that's legitimate. We already knew swarms were a thing, and we already knew the Chinese would have swarms, because it would be really weird if they didn't. The new information is that you happen to be able to contact one in real time, and how much is that in fact new information? Unclear. So we do triage. And if you want to know what kind of world you're living in, and you're in remotely the tech world, go to Twitter and check the trending topics on a random day. Half the time it's going to be a sci-fi novel written two years ago and set in 2035 that is just going on right now in front of your eyes.

Prakash Narayanan: I agree with you. But how does that feed into the AI policy bill, the FRONTIER Act that Anthropic is currently lobbying for? The overall perception I've heard in the market is that they're trying to get it passed during the lame-duck session, before you have a change in government to, probably, the Democratic party, which is more interested in regulation. And that this is really preempting state-level regulation. So the idea is that they are pursuing a strategy of getting the lowest form of regulation they can get passed now, which then preempts whatever the party can do later on. That is the perception the market has. I don't know the internals of the company's politics, whether they think this is the only thing that can get passed or whether the good is better than the perfect.

Zvi Mowshowitz: Thank you for bringing us back; I was trying to pivot, but there are so many weeds you can talk forever. That is in fact largely a lot of people's perception, so let me tighten it a little. Back in June, there was this proposal called the FRONTIER Act. It was a legitimate, well-meaning attempt at compromise under the conditions at that time, when there were a lot fewer worrying things happening and people were a lot less worried. The balance of power was much more towards Republicans and towards the desire to avoid regulation. So effectively, FRONTIER is a trade. It says, you let us preempt state laws on transparency and frontier risks, and in exchange we will have our own, similar to SB 53-style, implementation at the federal level, with transparency requirements, coverage of frontier risks, and governmental authority at the federal level that can deal with these problems.

First best, that is the form of a good solution. Having a patchwork of state laws, as the critics call it, is a second-best solution for obvious reasons. You would want the feds to deal with this kind of policy; it doesn't make intuitive sense to put it at the state level if you could avoid it. If you could get the same effect as state laws, but at the federal level, enforced by a federal government that was enforcing in good faith for everyone, at the right amounts, making good decisions, then everyone should be able to support this. In theory. The FRONTIER Act as proposed has a number of technical detail problems we don't need to get into right now, and there's a lot of room to argue about what could be stronger or weaker, broader or narrower.

So the bill circulated for a bit, and my attitude towards FRONTIER was always that, if we're never going to get another bill passed ever again at state or federal level, FRONTIER as written plus bug fixes is better than nothing. Bug fixes being things that both sides would agree to change because they just don't work right. However, preemption is a really big thing to give up, because in 2027, when all the legislators come back, a lot of them will be fiercely trying to regulate AI. So far, they've done things that are broadly compatible. If you look at New York and Illinois, they have statutes that in some ways extend SB 53 but are fundamentally the same principles and rules, such that you do not have a patchwork problem. You can take these three laws, find the union of them, follow that, file three sets of paperwork, and it's no big deal. If you had 50 versions of this, it would be annoying, but not particularly problematic.

Nathan Labenz: You also have agents to help with all the paperwork.

Prakash Narayanan: Exactly. I don't think the old rules apply. The companies have enough money and enough agents to go through independent regulation in every single state. The patchwork doesn't matter at all.

Zvi Mowshowitz: [unclear] So the question at the time was, given the hostility of the administration to almost all forms of formalized regulation, they're very willing to do ad hoc interventions when they feel it is necessary, and I think they're doing their best, making mistakes, to figure out when that makes sense from their perspective. But they very much didn't want to formalize things or tie their hands or require things. And obviously most AI companies think, we don't want these extra regulations, but we do want preemption to avoid more regulations. That's the natural thing to do, and people who want AI, other than those particularly concerned about safety, would be inclined to do that.

So what happened with Anthropic is that Anthropic has been circulating a bill that is, exactly as you said, designed for the lame duck. I was warned a few weeks ago about this. There's a huge faction, led by a16z, that has basically been "preemption at any cost." We want preemption, we want to destroy the states' ability to pass laws, and we want to get it for absolutely nothing: no regulations of any kind at the state level, none at the federal level except banning other regulations. And then magically, this one time, anarchism will work out with creating minds smarter than ourselves and everything will be great, and also all of our companies will make a lot of money. This came pretty close to happening at one point, but it has been fought back, and so now it's understood that you have to give them something in return. And we had FRONTIER.

But then in September, the political winds changed dramatically. After it became clear the election was going to create a Democratic shift, with the House almost certainly falling to the Democrats and the Senate probably too, and I think the markets are, if anything, underestimating that, the Democrats will come back with a much more strongly pro-regulation agenda. There is going to be broad bipartisan support for doing a lot in AI. There's probably going to be a lot of distancing from a lame-duck president, because that's what happens after the midterms in the second term in general, especially for an unpopular president. AI is a deeply unpopular thing to be supporting in various senses. And that's forgetting that there will probably be additional incidents in the future that raise the salience a lot.

Prakash Narayanan: Let me start from the assumption that everyone is operating in good faith. My good-faith interpretation is, number one, that they want to get anything done as soon as possible, so it makes sense to chase whatever they can get done whenever they can get it done. Number two, if there is broad bipartisan support, you can adjust these bills going forward. There's no reason you only have to pass one bill. Legislation often takes multiple sessions of Congress, and Congress fine-tunes. Congress is also able to act quickly: it acted quickly during COVID and the 2008 financial crisis. When people are scared, things move.

So it's not true that it's always slow. It's slow because there is no political consensus, and manufacturing that consensus is a process of its own. Under the right political conditions you can get a lot of legislation. What you're getting now is what is able to get done under the current political consensus, and perhaps that is worth pursuing in its own right, because you can get something passed quickly and then come back to it later if it's important enough, and adjust and fine-tune.

Zvi Mowshowitz: I think that's an excellent steelman of the charitable, good-faith case, and you can imagine worlds in which they're doing something like that. I can imagine them being caught in the same mindset of: we have this framework, we have this idea of what a compromise realistically looks like, we have the same people in Congress, we probably can't push very much, and time is of the essence. And then they misread the situation pretty severely and assume they can't push for more than roughly FRONTIER. I think that's just very clearly wrong. Support is much stronger now, and you could push for quite a bit more. I also think that, on reflection, what they pushed for here is weaker than FRONTIER, in a clearly self-centered way, where there are things that are better for different topics, pick your interests.

There's no reason to be circulating this bill, at the time they were circulating it, with the provisions that they had. It would make sense if they'd said, we were circulating dozens of bills, or we were just putting out samples of what might happen without endorsing them. That's not the messaging I'm getting. And I'm also not getting the messaging of, "on reflection, now that it's been pointed out that the political environment has changed, this bill was way too weak compared to what we could reasonably expect, it would be a bad deal to accept, and we apologize," maybe not in exactly those words.

Instead we got something very different: we're sorry about some of the details, we're very open to fixing the details of this bill, we're not attached to the exact thing we submitted, we think it's probably not strong enough and we made some mistakes. But that fundamentally doesn't address the fact that Anthropic was trying to push a pretty weak, deeply compromising bill, sacrificing preemption for things that probably weren't even worth what you were sacrificing from the perspective of safety. This was at a time when we had fought tooth and nail to prevent preemption and were close to turning Congress over to the Democrats, which is going to happen within two months of whenever this bill could possibly pass. We're not going to get the lame-duck session for another few weeks, so we'll be into November before anything could even in theory pass.

A lot of us were very upset with both the bill that leaked and the response that Anthropic's policy lead put out, which is the thing I was objecting to yesterday. The response essentially backed up the core idea and said the execution wasn't ideal, but that's not what the central complaint was. If we'd only had complaints that they screwed up the wording, that would be one thing, but that isn't what happened. So this really complicates the situation in a way that no one on either side of this argument had foreseen, though people have been warning that [unclear] would try exactly this. They will try to pass exactly this bill, because it's effectively their last chance before what can be passed becomes much more favorable to the other side.

That comes back to the last argument, which I think is potentially the strongest one: just because you pass this, does it mean you can't pass another bill later in the new Congress? Does it mean you can't pass a new bill in an emergency? For a true emergency, I think that's a very good principle. However, the balance of negotiation, the balance of power in a new Congress, would dramatically shift away from being able to get things, because a lot of the time the pitch is, we're giving you preemption in order to get things. Now preemption has been given and is done. You can't offer it again. And once you've gotten some of what you wanted, some of the supporters are back, you already have the things I valued most, so the case for doing something is weaker. You're getting less in return, and it's going to be a lot harder to pass. I strongly believe that if something like this were passed, it would make it harder to pass a bill in the new year that did a given level of thing, not weaker. And I don't want something like this to pass.

Prakash Narayanan: But that seems to say that if there is any risk reduction from passing a lesser bill, you are sacrificing that risk reduction in the time period before you can pass the greater bill.

Zvi Mowshowitz: To be clear, any actual risk reduction would be a good thing. As written, I believe the bill would actually increase risk, because I believe the cutoff of state laws would be worse than the amount of safety added, especially because there's a flaw in there.

Prakash Narayanan: But you have that time period before you pass a new bill, maybe only several months. Are you saying the states will be able to pass something in those several months? A lot of state laws have long burn-in periods, three months, six months, a year to comply. It's not as though things snap together.

Zvi Mowshowitz: One of the core problems with the current drafting is that, as drafted, the bill would immediately nullify the state law it's preempting, but it would take many months before any implementation of the new requirements. So you would effectively have no such regulation, and a loss of regulation by the states, for six months or more, which is obviously a disaster from the perspective of moving quickly. I call that almost a bug; I think it's clearly a mistake. But even if you were to address all of those, I do believe the states can move faster than the federal government, de facto, on this, outside of an emergency. Obviously you have to move on legislative schedules. But also, we already have state laws that are already functioning.

Prakash Narayanan: I don't believe states are slow at all. Take the Anthropic and Department of War situation: they just tweeted it out, they didn't do any of the paperwork to actually kick Anthropic out, and the court still found in favor of the Department of War. I think it's the same at the state level. A state can say, we're going to cut you off, and the attorney general can just send a letter. It could be for something else: tax reasons, financial reasons, a noise complaint about your data center. You can do so much of that at the state level, and once you do, it takes time for the company to come back. You push the onus of proof onto the company to go through the legal process, which is what the Department of War did to Anthropic with the supply-chain designation.

So the states have a lot of power, but you need political consensus because you're going to get blamed for it. A governor has to be willing to take the heat from the national media, basically. So I don't think the states are powerless, but I do think the states lack the competence and the political will to exercise full authority unless it's something like COVID, or something with imminent harm, where they're willing to step in.

Zvi Mowshowitz: I have to push back on some of that. Yes, the government got one favorable ruling from a subset of the DC court, but they got an extremely eviscerating, very angry ruling from San Francisco, the kind you almost never see; long-suffering federal judges rarely get that angry at you. And the DC ruling was 2-to-1, from an unusually government-favorable panel. The experts I follow were surprised by that ruling, don't expect it to survive escalation to the full court, and definitely don't expect it to survive the Supreme Court.

But the broader point you were making is that the process is the punishment: they're forced to spend months dealing with it, and the moment the thing is tweeted out, people start acting in fear. That's true. If the White House is willing to just do ad hocery and use its threats and leverage to make things happen in the world, we've already seen them do it twice, so they could do it again, but that's a very blunt instrument. Three times if you count the Department of War. You have the Mythos situation, where they delayed the release of Mythos for many months and then started approving things. Then you have the Fable situation, where they took Fable down. So you have...

Prakash Narayanan: I wonder to what extent that is to be expected, though. Not the Fable case specifically, but the ad hocery. As events start to move faster, and the legislative process was designed for time frames of years while the executive process was designed for time frames of a day or an hour, different parts of the system address different time frames of governance. You cannot get legislation passed within a day, but you can get the president to sign an executive order within minutes. So if you have an acceleration of technology, you're going to need an acceleration of governance to keep up, and it may be that that's where we end up.

Zvi Mowshowitz: We are going to need some ad hocery, obviously. Part of the reason to pass legislation is to authorize the government to do ad hocery in a way that is much less ad hoc, more formalized and bounded, with judicial review. One of the big arguments over the bill is that, if the government wants to issue a takedown of a model in an emergency, like what happened with Fable, and it happens again, how would they do it? Anthropic's bill says you have to go argue before a judge, with very limited opportunities to sustain that, high burdens of proof, and in ways where you don't necessarily have the transparency to demonstrate it. It's a very onerous set of provisions.

The good news is that the government could still impose export controls, or issue general threats, or otherwise do the things you do in an emergency. But in a real emergency, the new provision is so bad you wouldn't use it; you would just use the current ad hocery. Part of the point of passing a bill like this is to give them ad hocery that is formalized and reasonable, but Anthropic is in fear of that, because they're afraid the government would use it. So they're unwilling to grant reasonable authority to this administration. This points to a conflict in Anthropic's eyes: you're going to preempt the states and hand this power to the federal government, but the federal government is run by people you don't trust to do it reasonably, so you cripple them. But then that's worse than just leaving it with the states. Anthropic is trying to cut it both ways, and it's causing a lot of problems.

Harkening back to the fundamental question about ad hocery, I would say the purpose of Congress is to set the rules by which the White House will act: to give them the resources, the transparency, the capacity, the levers they need to use in that emergency, and to put limits on how they can do that, to avoid abuses of power. Right now Congress has kind of resigned, absolved itself, and stepped aside, abrogating a bunch of its powers to the White House across the board, not just in AI. I understand the world has to move somewhat more quickly, but the degree of imperial presidency is running us into big troubles. If they can't even reclaim their powers on things like tariffs, where the powers are clearly constitutional and are being abused in terrible fashion in ways we can see are not good for anyone, why would we want to encourage more of this?

Nathan Labenz: I don't know if there are any more details you think are critical to cover, but I'm interested in zooming out and getting your take on the question that Nathan Calvin asked, and you quoted his post at some length in your most recent blog post.

Zvi Mowshowitz: In full.

Nathan Labenz: Yes. How should we understand Anthropic at this point? And maybe we can then end on a quick live players lightning round, as we do have a couple of...

Zvi Mowshowitz: We might not have time, though. I'll try. You should read both Nathan's reply and Dario's reply to Nathan, which came after I put up my post and is now available. I would say Anthropic is a company with a much stronger safety culture, although not adequate to the moment. It's willing to make big sacrifices in the areas where it, or Dario in particular, has strong beliefs, and that is to its credit. It legitimately wants to see this go well and has some unique perspectives on what it would take to make that happen. But at the same time, Anthropic's policy division especially, and also its willingness to be an ordinary company that competes in the [unclear] and buys a lot of capacity, and encourages more chip production, more power supply, more use, and so on, means it is acting like a normal corporation a remarkably large percent of the time.

On policy, it is reasonably often revealing itself not to live up to its ideals. This is very frustrating to those of us who want to be excited by the prospect of Anthropic being a good steward in some sense, and Anthropic keeps giving us so many different bits of information that we should be worried about this. I've also recently heard, from various employees I've talked to at various points, some rather alarming views, from my perspective, about what it would take to solve alignment problems and make the future go well. All of that plays into being very uncomfortable. They could still be the most responsible lab, but that's grading on a curve, and I don't like grading on curves, because reality doesn't have a curve. Reality just decides what happens. So I will now yield the floor for a lightning round.

Nathan Labenz: I give you credit for being the first person to make a strong case to me, six or maybe eight months ago, that we were going to see a two-horse race at the top and that Google was falling out of that top tier. I think that has become consensus over the last however many months. I've also heard a take recently, which I thought was quite interesting, that while they may not be distilling in the way we normally think of distilling, the next tier of even American competitors are often effectively distilling from the likes of Claude, because the RL environments they're buying from the RL-environment cottage industry were all vibe-coded with Claude in the first place. It's a roundabout way of capabilities moving from Claude to other models.

We now have Argon, aka Gemini 4. We now have Muse Wonder, and Muse itself as a product. The simple question is: has anything changed how you assess the situation, or are we still basically in the top two and everybody else formation?

Zvi Mowshowitz: I think we're still in the top two. Periodically you'll see the other players release new models, and there'll be claims that they've caught up, they're back. Sometimes it does represent that they're now much closer to the frontier at their peak than they were previously, and that's something to notice. Other times it's just hype. First of all, xAI hasn't issued anything impressive in a while. They have a hype machine because Elon Musk is very good at hype machines, and we got mediocre models sold to us as something that mattered, and they never actually mattered. Now we settle into: we don't expect Grok models to matter for a while, and they'll try again, but they haven't had the compute. We'll periodically still hear that Grok 5 is the big thing, or Grok 6 is the big thing, and we'll check it out, and probably not. You never know.

Muse Wonder, I think, was presented kind of shadily. You saw this instantly if you looked at the benchmarks they put out. They were not the standard benchmarks organized in the standard way; they were weird stuff. The moment I saw that, I thought, I think this is fake. Not that it's not a good model in an abstract sense that you wouldn't have wanted six months ago. It's a fine model, but they're presenting it in a shady way. Meta has done this before: it intentionally introduced a version fine-tuned for Arena into the Arena in the past. None of this is surprising; they're kind of shady guys. Last night I posted two threads on Twitter. One said, "Gemini Argon reaction thread: is Google back?" The other said, "Muse Wonder reaction thread: is there anything here?" When I woke up in the morning, the Muse thread had two replies, one of which was asking where the safety card was. The Gemini one had 40.

I think that's simple and telling. With Muse Wonder, no one is using it. People think they considered it and rejected it. Nobody is excited; nobody thinks this is anything. You can be one model cycle behind, only a few months behind in some sense, and I'm sure it's pretty good, but nobody is going to switch to your model if it's still behind. What's the point? [unclear] This is closer than Meta has been in a long time. It's a model I think you probably could use. In the alternate world where this was the model, people would still be like, AI is really good, AI is kind of scary, what should we do? They built a real model. But it's very different from the pitch they were making.

Argon is hard to tell. I've been traveling a lot, and I've been trying to read the system card. Argon is clearly much, much better than Gemini Flash 3.8, which is what they were working from. It's a major advance for Google. The question is whether they have something now that is competitive, and my read is probably no. Gemini still has problems with coding, and it probably always will, because it does not have the recursive user base of Codex or Claude Code, and they don't seem to be making progress toward getting one. So they're going to have much worse training data in that area. Gemini is also still a Gemini model. There's a certain thing you come to expect when interacting with a Gemini model. For some purposes that's good, but for most purposes it's not particularly fun, and it still has that. Most things you want to do will involve one or the other, so that doesn't leave that much room for Gemini, even if it's not so bad in other ways.

They trained a very good base model, just like they did for Gemini 3, but then various things happened where they did not live up to its full potential, or they didn't have the coding data environments. But we will see. You cannot count Google out; Google is Google. So I have it as S tier: OpenAI and Anthropic. A tier: Google. B tier: xAI. C tier or lower: everybody else, basically.

Nathan Labenz: That lines up pretty well with what my revealed preferences suggest, which is that on my computer, doing my stuff with deep access, it's only the top two. I will happily use Grok to track something down on Twitter. I will happily use Muse when I'm in the Meta universe, and I'll happily use AI Mode on Google for search. But I haven't seen anything from any of those that makes me feel I need to build it into my core personal AI operating environment. We talked about this in a previous episode too, but I have noticed that my criteria for that are changing, and are now much more about trust, and things like how hard these models have been shown to be to jailbreak by third parties, than about whether Muse got one point higher on whatever benchmark. I'll really know it if somebody manages to jailbreak my assistant on my laptop with a lot of access. So my decision criteria are also...

Zvi Mowshowitz: I'd also add willingness to learn and customize, and to commit to the ecosystem and the habits and the learning. I've put a bunch of work into getting Claude where I want Claude, and a decent amount of work into getting ChatGPT where I want ChatGPT, and much less work into getting Gemini where I want Gemini. That's because they've been consistently ahead in the past, I expect them to be consistently ahead in the future, and I like their styles and the intelligence they're offering. And also, yes, a matter of trust. I am willing to trust OpenAI and Anthropic with access, basically. My life is kind of an open book; I'm not holding national security secrets. If you hack, you hack. As long as I can get my Google account back, it's all fine. And, roughly speaking, don't touch my bank accounts. Everything else is fine.

Nathan Labenz: Are you going as far as, I was just talking to my wife about this yesterday, should we be diversifying across bank accounts at this point, just to not have all our eggs in one bank account basket? It seems like we probably are there.

Zvi Mowshowitz: Maybe a little. The downside of splitting is that it's now twice as likely you'll have one problem. I am not especially worried that hacking will permanently leave you unable to access your money at a bank or brokerage account without that otherwise not being your biggest problem, I guess is how I'd put it. If you can't get to your money for months or years, if we're in Mr. Robot mode, then you should be worried about whether the food is going to show up in your city. Things are really, really bad in a variety of ways. You worry about whether your government collapses. Not that you shouldn't worry about your money, but...

If you're above FDIC limits, yeah, maybe diversify. It can't hurt that much. Certainly have it such that if any one account goes down, you are not completely screwed; that's my way of thinking about this. It would be reasonable to take various precautions that would have sounded kind of silly a few years ago, or that only paranoid people would have taken. I might get a Starlink device ready to go in case your Internet goes out at some point in 2027. I might stock the pantry intentionally with a bunch of stuff that keeps for a while, so that if there were some weird logistical crisis, with a run on the stores, you'd have time to wait it out. These are cheap things to do. In the 5% chance that you need them, you're very, very happy you have them, and in the 95% where you don't, it's okay; it wasn't that big a deal.

Nathan Labenz: Anything else top of mind before we let you go? I know there's always a lot more swirling around in your brain than makes it to the blog. We've seen things like functional pain and conditioning work from Cameron Berg recently, which we are always obsessed with here on AI in the AM.

Zvi Mowshowitz: Yeah. See.

Nathan Labenz: What else is...

Zvi Mowshowitz: I try not to draw too much attention to that, for reasons. But I would say the biggest thing, and what I was most upset about with the policy situation, is that this is not fundamentally attempting to meet the moment. Dario has said we must face the frontier. To the labs themselves and the people internal to them, there's a crisis coming where they are able to push capabilities faster than they are able to push alignment. If they don't do something about this, they're going to have to face some very hard decisions, where they can either risk their commercial viability or risk serious catastrophic problems, including potentially loss of control at some point in 2027.

They need something to be done about this. And if you're not ready with something that can potentially do that, well, why don't you keep working until you are? Things that used to be outside the Overton window are now inside it, and things where it seemed we wouldn't necessarily need them, we now pretty much know we need. That is where a lot of time and attention is going to have to shift. Obviously all the prosaic work still needs to happen, all the alignment work still needs to happen, and the mech interp stuff is great; we should absolutely do it. But the biggest thing is that we're going to have to meet the moment, and at minimum do something that counts as some form of facing the frontier.

Nathan Labenz: I think that's a great note to close on, and it's much appreciated that you take time out to meet the moment with us, always, on AI in the AM. It's always great to get your perspective on such wide-ranging topics. Thanks for being with us, and we'll look forward to the next time already.

Zvi Mowshowitz: Okay.

Prakash Narayanan: Thank you, Zvi.

Zvi Mowshowitz: Let's wrap it up. Bye.

Prakash Narayanan: Bye bye.

Nathan Labenz: Well, never a dull moment. I need to make sure I've got a couple of accounts in good working order, sooner rather than later. I actually went through a phase where I consolidated everything I'd accumulated over time, because I was so tired of getting mail from many different financial institutions, and a couple of times I'd think, oh my god, I left however much money in this account that I never should have opened. I thought I was going to do something with it, and I didn't, and it was all just so dumb. But now maybe I need to go back in that direction a little, because it does seem like this is kind of happening. I mean, this PHP thing is going to be a really interesting case.

Prakash Narayanan: My expectation is this, and I found it out a little while ago from a friend who helps smaller banks manage their cash. At the end of every day, a bank needs to even up: the databases have to show the same amount entered and exited, and how much deficit they have. If they have a deficit, they borrow money from the Fed. If they don't, they don't have to. So if your bank is well run, you won't have a deficit. And the well-run part is really about how good your systems are. If your systems can track minute to minute how much cash is coming in and going out, you always know. Once there's a discrepancy, you have to borrow.

Prakash Narayanan: It turns out that the smaller, older banks with older IT systems end up having to borrow, and they borrow from the other banks, like JPMorgan Chase, in an interbank market intermediated by the Fed. So banks with better IT, which can track minute to minute, are lending money to cover the deficits at the smaller banks. You have an ecosystem of larger providers with better IT and smaller providers with worse IT, and the larger providers are always sucking a little bit of money out of the smaller ones over time. The smaller ones have less margin from this safe interbank lending, so they have to make more money from riskier things like commercial real estate.

Prakash Narayanan: My guess is that over time the smaller banks die off. That has actually been happening in the US for the last 40 years or so. You had the savings and loan crisis, and with every crisis the number of banks in the US shrinks and shrinks. I think this is another die-off moment. Right now Chase, Wells Fargo, Bank of America and the other big banks are enormous. I think 30 to 40% of US deposits are already concentrated in them, the highest level of concentration in US history. I think this leads to another round of consolidation. We saw some of it a couple of years ago, when Silicon Valley Bank and other midrange banks got absorbed into JPMorgan Chase and other large banks.

Prakash Narayanan: So I think you'll see another round now. Banks with poor IT systems and weaker cybersecurity will have those issues, and they'll be forced to cover them. You as a consumer are always protected, because the FDIC is there. But the smaller banks will have to cover their losses in the interbank market and make sure all the consumers are made whole. If your cybersecurity is not good, you'll have to borrow in the interbank market, and the bigger banks with better IT systems will bail you out for a fee. That fee drags on your profits, and over time you shrink. So I think we'll see that happen again. It's not a unique process. It's been going on for 30 or 40 years.

Nathan Labenz: Yeah. You have a way of making all of these things sound like business as usual, but I'm not so sure this will feel like business as usual. I wouldn't take issue with any of the facts you just said. I often tell people, especially people from outside the US, that this is the great American story: some entrepreneur somewhere figures out a thing, and it proliferates across the country with tons of local versions, whether it's banks, cable TV companies, local ISPs, fracking, or even, to some extent, data centers, though that's not a perfect fit because of the big capital requirements to get started.

Nathan Labenz: But yes, this has happened, there's a later wave of consolidation, and it has indeed been going on for 40 years. It's been pretty orderly for the most part. The Silicon Valley Bank thing was one of the less orderly ones, where for a minute it was, wait, are my deposits going to be accessible? But it would not shock me at this point if that happened to, say, 1% of the banks. There are something like 4,000 FDIC-insured banks right now, so that would be a bank a week over the next year. I think you could have a story that's totally consistent with everything you said and would still be experienced by people as total, insane chaos.

Nathan Labenz: And if that does start to happen, you'll see a lot of flight too. If all of a sudden a bank a week is getting hacked, you're probably going to see a thousand banks get drained legitimately by people who say, forget this, I don't want to be in one of these long-tail local savings and loan institutions or credit unions anymore, which felt homey and cozy until one of them is getting hacked on a weekly basis. You may have a pretty insane set of events, headlines and experiences for individuals in local communities. This could be quite crippling to individual local communities even if it's not a systemic risk nationally.

Nathan Labenz: Think, for example, of the Michigan State Credit Union. A whole lot of people who went to Michigan State and live in and around Lansing are members of that particular one. If it goes down, it doesn't shake the dollar, but it does throw a massive wrench into mid-Michigan for a while, at least until it's resolved. And if this is happening in a good place, it

Prakash Narayanan: doesn't work that way. The FDIC can take over a bank overnight, and they always do this. It's not a new process. When the FDIC decides to take over a bank, they come in and say, this bank is ours, and the moment they say that, the balance sheet you're referring to is no longer the bank's. It's the federal government's, and the federal government can print money to cover all of the bank's losses. There's zero way for an FDIC-insured bank not to pay out the FDIC-insured deposits. The suspect part is what's above the insured limit, which is what happened with Silicon Valley Bank, because it had a lot of companies and rich entrepreneurs banking there. The question then was, what do you want to do?

Prakash Narayanan: Do you want to cause losses on these larger accounts? If you do, you get contagion, because you have this very entrepreneurial group of Silicon Valley founders who will create institutions that spread deposits out over all the banks, and every company will pull out everything above the FDIC insurance limit. So the decision the federal policymakers made at that point, Yellen and the rest, was that we are going to cover it. FDIC insurance basically covers everything. It's not an explicit guarantee, because we don't want banks to have to pay this out,

Prakash Narayanan: but there's an implicit guarantee that all of the deposits are covered, including those of large companies banking with these smaller banks. So I don't think it works the way you describe. Once the FDIC goes in, it's done. Everything is clean for the bank to reopen, business is normal tomorrow, every single dollar can be taken out, and not a single dollar is lost.

Nathan Labenz: Yeah. Again, I would agree at that level. First of all, I think above the FDIC insurance limit there's basically the good old implicit government guarantee, so I wouldn't worry too much about that. What I'm not so sure about is whether the FDIC's ability to come in on a Friday and have that small insolvent bank ready to reopen on Monday generalizes to a new set of conditions. The conditions they're used to are where the bank made some bad loans or whatever, and it's basically a balance sheet question. You trust the records for one thing.

Nathan Labenz: I'm sure we could look into some extreme cases of fraud, but mostly it's a known kind of problem: the bank is now insolvent, we come in, we know we have the balance sheet to make everybody whole. The core act they're performing is bringing their weight to bear on the problem and saying, whatever you've lost, we can ultimately absorb it, and it's fine. And in absorbing it, you basically make the problem go away. What's much

Nathan Labenz: less clear to me is what happens in a world where you can't trust the records. What happens in a world where you have, say, OpenFace-style 1,200 agent traces that are all millions of tokens, and they may or may not have obfuscated the logs or figured out some way to falsify them? That doesn't seem like something the FDIC can necessarily work out over a weekend.

Prakash Narayanan: You're absolutely right that insolvency and the records being fried are two different situations. But say you have a bank where the internal records are fried. Then the question is, we don't know who owes what inside the bank, and we don't know what numbers belong in every single account. Number one, the FDIC can just say, and this happens a lot, look, maybe the records are unclear, but you can still issue checks and the bank will honor them. And the bank will honor whatever withdrawals you make.

Prakash Narayanan: We know who is withdrawing the money, and if you over-withdraw, we'll come back to you and ask for the money back. This happens. Banks have let accounts overdraw to limits that weren't acceptable, and sometimes they've sued people for fraud even though the bank made the mistake. So you can claw back the money. It's not a big deal. And you're really only talking about the big accounts you're afraid of, the million-dollar accounts, and you can track a million dollars by hand. You can say, this guy was the one who took it out. And the guy who takes out a million dollars also has lawyers and is concerned about fraud

Prakash Narayanan: too. He's a small-time businessman. He's not going to run away. That million dollars goes into property or other assets. You don't carry a million dollars around in cash, and you can't even withdraw that amount in cash. So at the end of the day, you can sustain the system with promises, and that's the beauty of the banking system. Within that small bank, you can say, these are all promises, we will honor all promises, and we'll come back to you once we reconstruct the records. Let business go on as usual. You should know roughly how much you had, and if you take out more, we'll come back to you. We just run. And once you make that decision, it doesn't

Prakash Narayanan: matter. It's really the trust in the institution, a layering of trust and contracts, that brings together the excess of what that institution is, and the data in it you can reconstruct over time. There are other issues if the problem is above that level. If the Fed itself doesn't know which bank has how much money, that's a whole different ballgame, because then you don't know whether JPMorgan has $3 trillion in assets or it's Wells Fargo. But you have ballpark numbers, because people do monthly and weekly reporting, and you have ballpark numbers over the last year.

Prakash Narayanan: And banks have kept 7 years of records anyway, so you can reconstruct if you need to. You have account records going back 7 years, even prior to these events. You have a lot of records, a lot of people, a lot of data, and agents that can reconstruct all of this. But I do think it's possible that the Fed does not implement the best cybersecurity in time. A government institution is slow. It might be running just a little behind some of the banks, or JPMorgan, and even JPMorgan Chase is not that fast to implement. So it really depends on whether these institutions want to implement quickly. The smaller guys are all taken care of. It's not a problem.

Prakash Narayanan: You can reconstruct. You have the full faith and credit of the United States backing the firm, and the long arm of the law to claw it back. It's not an issue, and you don't have to panic. The government is there, the legal system is there, the records are there. It's an inconvenience. Now if that inconvenience extends, if all of the internet is down for a year or all the records get destroyed for 7 years, that's a different ballgame. But I'm very sanguine. It'll be fine. If anything happens to you, the government will overcompensate you

Prakash Narayanan: in order not to cause a bank run the next time. So I think it's all just going to be okay. But it's not going to be comfortable for the bureaucrats or the people who have to execute. They're going to have to work. It's going to be very annoying, because they used to have the law. They used to say, you know what, I'm just going to skimp on the cybersecurity, because if anyone does anything, we'll go after them. You'd have a knock on the door, so you don't do it.

Prakash Narayanan: But now you can't knock on the door, because there are too many agents, and it's like some 12-year-old kid running NSA-style operations on you. So it'll be largely okay, but people have to change. I just saw that Bank of America reported a $400 million Q3 cost for Mythos. So those charges are coming in. You're going to have to pay for frontier model security, and if you don't pay up, you may have issues. People have to pay up. That's the thing.

Nathan Labenz: Yeah. That's incredible. $400 million. That's a lot of tokens.

Prakash Narayanan: That's a lot of tokens. Yeah, I think I'm

Nathan Labenz: at least moderately comforted by your analysis. One thing I was comparing this to in my head, which I think has some similarities but also some critical differences, is the FTX unwinding. I didn't have any funds in FTX myself, so I haven't been hanging on their payout schedule, but I'm interested to hear what your experience was. What Google tells me is that it was 2.25 years until the first wave of distributions after the crisis hit, and that was obviously a relatively chaotic situation. Safe to say records were not what they might have been, and it took a long time for people to get even their first dollar back.

Nathan Labenz: The obvious difference, which I think you're right to highlight, is that if it's the dollar, the government will come in with a very different level of, let's just monetize this problem and try to paper over it, compared to what they would do in a crypto situation. But it at least gives me some concern. I think you're probably right, though, as long as it's not a fully general crisis, as long as it's somewhat contained to this bank or that bank. I could see them being willing to take some losses on account holders withdrawing more than they should properly be able to. We might not get every one of those dollars back, but, okay, so be it. I can see that attitude prevailing, given that these are dollar-denominated accounts. It could be a lot different in other parts of the world, too. In the US, we're so used to the ability to just print money and have it actually accepted. That's not so much the case in a lot of places.

Prakash Narayanan: I think one of the most interesting things is that I've been talking to a bunch of people in Asia over the last couple of weeks. Asia saw hacks first. You have China there, knocking on the doors in Japan, Taiwan and Korea. Chinese hackers are just walking in and checking around. So it's been happening. The cyber apocalypse actually started in Asia maybe a week and a half ago. Over the weekend I spoke to a couple of people who said they are seeing it now. It is happening right now. So I think this week and next week, we'll see more. I just saw a post from the prime minister of Japan on Twitter with 25 million views in the last 6 hours on the PHP issue.

Prakash Narayanan: And PHP is such a niche cybersecurity issue. The prime minister of Japan, on Twitter, with 25 million views. So I think one of the things that's going to happen is a canary-in-the-coal-mine effect, where these forward-deployed institutions, which are weaker, get hit first outside the US. US policymakers can stand there, take a look at how it goes, and then decide how much the US should be doing. And policymakers outside the US typically

Prakash Narayanan: spend a lot less money on cybersecurity and IT in general, because they don't have the coders, and a lot of the tech is from the US anyway. There's some advantage now that they have access to AI. Japan, for example, was so short of coders. Indian coders couldn't go there, people are speaking Japanese, and it's not a great place to code. The AI agents are helping a lot. So I think we're going to see a mass upgrade overseas from these 1980s, '90s and 2000s architectures to modern architectures. That's great, and you can watch and see where it's happening. But you're also going to have a die-off. For example,

Prakash Narayanan: of 7,200,000 PHP websites, 46% are running an end-of-life branch, which cannot be upgraded anymore. And these are all the old websites, things someone set up in the early 2000s and has not bothered to update, but which have kept running with minimal maintenance. Some guys

Nathan Labenz: German wikis and all that

Prakash Narayanan: kind of stuff. Yeah, some guy's gaming forum has been running for 15 years, and he hasn't been doing many upgrades on it. All of that is now fully exposed, and it's basically like an asteroid, a meteor hitting. All of the unfit websites will die off. Is it a problem? Yes. But it is also maintenance-level stuff that people are not attending to anyway, and a lot of that data has already been Common Crawled and is already in the corpus of the agents and these larger LLMs. So I don't

Prakash Narayanan: know. Is it a problem? Yes. Is it the cyber apocalypse? I don't know. Can it be automatically upgraded? Definitely. Is anyone going to pay for it to be automatically upgraded? No. No one's going to pay Mythos-level tokens for early-2000s PHP websites, for Craigslist listings from 2005 to 2010. It's not happening. So yeah, we lose a little bit of our historical corpus. That's what happens. But I think mission-critical websites where someone is actually attending to them will be fine. People upgrade.

Nathan Labenz: Watch this space. If nothing else, we'll be able to score your predictions, on a not-super-long time scale, because whatever's about to happen seems to be about to happen starting right now, or a week and a half ago, depending on what time zone you're in.

Prakash Narayanan: One additional thing I'll say is that because there was lower IT dependency overseas, everyone overseas is a lot more prepared to do pen and paper. In Japan, they still use official stamps for a lot of things, and they've been trying to get rid of them since COVID, because it required people to appear in person and stamp the material. They said, we can't do digital signing because all these official documents need this stamp, which is a carved stamp, with Japanese characters. And they said, this is ridiculous, we can't do this during COVID. They've had this long process of all the ministries converting to digital signatures, and it's still not complete.

Prakash Narayanan: So I think one saving grace is that the people who are not super dependent on IT get hit first, and their mission-critical systems are not fully converted anyway. They were always in this pen-and-paper, face-to-face, please-come-to-the-bank mode: we can't do this over the phone. If the bank's IT system goes down in India, everyone still has a bank record book, because they don't trust the banks. They have never trusted the banks, so they always have paper printouts and paper account books. And they've always had forgery issues with those account books too.

Prakash Narayanan: So they have these heavily designed, watermarked account books, because if you have an account book, someone's going to forge it, so it has to be forgery-proof. They've always been prepared. In a sense, they have a go bag for cybersecurity. If anything cyber hits, they never trusted these systems that much anyway, so I think they'll be fine.

Nathan Labenz: Once again, from your lips to God's ears, I'm persuaded. Still a little nervous about all this, but maybe I'll just go and distribute my funds across a couple of banks instead of a whole long-tail distribution.

Prakash Narayanan: The world is held together by scotch tape, but you can always add more scotch tape when something breaks.

Nathan Labenz: Yeah. Well, we might be about to put that theory to the test. So much more is going on, and I don't know that we could even come close to being comprehensive here. But is there anything else we should touch on before we break? Tomorrow we're going to be talking about compute again. We've got Evan from SF Compute, and then we've got Lonis Hamaili from Cambrian Capital, a new investment vehicle that's a little bit in the situational awareness vein, although we'll let him explain the differences to us. Anything else top of mind today that we should cover before we get out of here?

Prakash Narayanan: No. I think let's break and digest what Zvi said. That's going to take some time.

Nathan Labenz: Yeah, always. Alrighty. Cool. Well then, I'll see you back here tomorrow morning.

Prakash Narayanan: Cheers. Bye bye.