# Episode 44: How Anthropic Works in 2026 & the Most Important Mech Interp Paper of the Year

> At Anthropic, interactive AI agent hours now outnumber employee hours six to one, and 19 to one in research. With Rahul out, Shimin and Dan read Anthropic's own employment data and Nick Cammarata's case that neural networks are already clean, plus Meta's Muse Spark Open, a bugged OpenAI conference room, the Parsimony benchmark, and CoreWeave's bond scare, with the bubble clock moving to 4:00.

Published: 2026-10-23
Source: https://adipod.ai/episodes/44-how-anthropic-works-in-2026-the-most-important-mech-interp-paper-of-the-year/

---
With Rahul out traveling, Shimin and Dan open with Meta's Muse Spark Open, an 816B-parameter open-weight model that runs on two Mac Studios, and a former OpenAI engineer charged with hiding a recorder, wrapped in toilet paper, in an executive conference room. Post Processing starts with Andrej Karpathy's napkin math on who actually uses frontier AI, then reads Anthropic's "How employment is changing at Anthropic," a rare look at how a frontier lab works now. Then comes Parsimony, a new benchmark that measures how much code AI coding agents add. The Deep Dive is Nick Cammarata's "Neural Networks Are Already Clean," which Shimin thinks may be the most important mechanistic interpretability paper of the year, with a live steering demo on Neuronpedia. Dan's Rant returns, on the economics of context windows. In Two Minutes to Midnight, a chart of CoreWeave's bond yields sets off a selloff, J.P. Morgan sizes up bank exposure, and the clock moves 15 seconds closer, to 4:00.

## Takeaways

- **Meta is still shipping open weights, but the Chinese models still lead.** Muse Spark Open has 816B total parameters, 42B active, and a 1M-token context window. Meta calls it "the most capable US open-weight model to date" and says it's the first model in its class to run on a desktop: two M5 Ultra Mac Studios, which together hold the whole model in a terabyte of unified memory. It ships under the Muse Model License with limited commercial terms, so it's open-weight, not open source. It scores 39 on the Artificial Analysis Intelligence Index, level with Mistral Large 4 Preview and ahead of NVIDIA's Nemotron 3 Ultra and Thinking Machines' Inkling. But Qwen3.8, Kimi K3, and GLM-5.3 all score higher, and on Meta's own two coding benchmarks Kimi K3 and GLM 5.3 both beat it. Shimin gives Meta credit for sticking with open weights after Llama 4's troubles and the leadership change that brought in Alexandr Wang. In his view, Meta is one of the few labs that has both a closed frontier model (Muse Wonder) and a respectable open one. Neither host can work out who the two-Mac-Studio setup is for. *(On air we put Nemotron and Inkling in "the 30s and twenties"; both are in the 20s. "Beats Qwen 3.8 and GLM 5.3" holds only on some of Meta's selected benchmarks.)*
- **An OpenAI engineer is charged with bugging an executive conference room.** Prosecutors say Abdulrahman Alfozan, a 35-year-old Saudi national who had worked at OpenAI since July 2025, hid a recorder behind a TV in an executive conference room he wasn't authorized to use. It drew power from the TV's USB port and was wrapped in toilet paper. Security found it on September 30. On October 5 he left a bag in the office holding his wallet, his ID, and a second recording device. Investigators found prompts on a ChatGPT account tied to his wife asking how to build a continuous audio recorder and encrypt the SD card. The FBI complaint names the hardware as a Seeed XIAO ESP32-S3 Sense, about $14. Dan, who owns a few Seeed boards himself, wonders why OpenAI felt the need to say it used its models to identify the device. Alfozan is charged with intercepting oral communications, and a judge denied his release on October 19. Prosecutors say a member of the Saudi royal family repeatedly tried to recruit him to an AI venture, and that a Saudi military official bought new one-way tickets for his wife and young son after his arrest. His lawyer says the family's original tickets were for a two-month paternity leave. Dan suspects industrial espionage for a startup rather than a state operation. Prosecutors haven't said what the recordings were for. Shimin is waiting for the movie. *(On air we said the Saudi military bought the tickets. Alfozan booked the originals; the official bought new ones after the arrest. He was arrested and charged on October 9, and October 13 was his first court appearance.)*
- **To see where work is going, look at the 5,000.** Andrej Karpathy's napkin math has about 6 billion people who've barely touched an LLM, 1–2 billion using free chatbots like a better Google, around 20 million (0.2%) using frontier models professionally for math and code, and about 5,000 with access to frontier-grade systems internally. Shimin's point is that the future is unevenly distributed, so watch how frontier-lab employees work. Anthropic's "How employment is changing at Anthropic" is that view. By August, interactive agent hours outnumbered employee hours six to one, and 19 to one in research organizations. Commits per engineer are up 5.1x since December 2025. About a third of ML managers and a fifth of software engineering managers say they're hiring less because of AI. The product design team that wrote Claude.ai's greetings ("Hello night owl") now writes model evals and prompts: "Writers who used language to drive human behavior are now using it to drive Claude behavior." Hiring emphasizes "creative use of AI tools," and interview feedback rewards autonomy and judgment. Interactive sessions sit about 80% idle, waiting on a person, and one engineer says people are "bottlenecking the agents." Dan ties that to the code review bottleneck from [Episode 43](/episodes/43-openai-fires-its-safety-researchers-agentic-primitives-more-code-not-more-software/): deciding what to build is still a human process, and it takes time. Shimin doesn't think 5x the commits makes everyone a 10x engineer. The report's conclusion matches what the show keeps seeing: individual contributors orchestrating teams of agents start to look like managers. The difference at Anthropic, Dan suspects, is the token budget. *(On air we said commits rose 5.1 percent. It's 5.1x.)*
- **Parsimony measures how much code AI coding agents add.** Existing benchmarks measure solve rate and cost but don't punish excess code. Parsimony, an independent project, re-scores published benchmark patches by counting syntax units (calls, names, literals, statements) from the parsed AST, with docstrings stripped. It compares each model's net additions with what peer models added on the same solved tasks. On the default DeepSWE board, where every model runs on the same mini-SWE-agent harness, Claude Opus 4.8 and Sonnet 5 come in 12–14% under the peer average, Claude Opus 5 about 9% over, and the closed Muse Spark 1.1 and 1.2 write 29–34% more than peers. The author's early GPT-6 runs through Codex look leaner still, but they cover only 5–10 tasks and aren't ranked. The page's own caveats are that units measure size, not readability or design, and that models that write more code also tend to solve more tasks. Dan still wants to track it, because bloated code is harder to maintain even when an LLM is the one maintaining it. Shimin wants to compare two models' patches for the same task, and Dan suggests diffing their ASTs by node type. *(On air we called the GPT-6 models the winners and said Opus 4.8 and Sonnet 5 were negative. The GPT-6 results are unranked, and the two Claude models sit at 0.86–0.88x.)*
- **Neural networks may be cleaner than interpretability assumed.** Nick Cammarata co-wrote Distill's 2020 "Zoom In," which made polysemantic neurons famous: InceptionV1's neuron 4e:55 responds to cat faces, fronts of cars, and cat legs. Anthropic's 2022 "Toy Models of Superposition" explained such neurons as models packing in more features than they have dimensions, and the field's fix became sparse autoencoders. Golden Gate Claude was the showcase: a feature from a roughly 34-million-feature SAE on Claude 3 Sonnet, clamped to 10x its maximum activation, and never a single neuron. Now at Goodfire, Cammarata argues in "Neural Networks Are Already Clean" that most polysemanticity had ordinary causes: bad tools, bad explanations, and bad learning from architecture and data. Superposition should be the last hypothesis, not the first. With modern architectural fixes, 99–100% of neurons in InceptionV1's last layer become easy to interpret. A frontier-model neuron that fires on "pr," on Base64 strings, and on Bengali turns out to mean "pr" in all three. One neuron that detected "iy" grew over training to detect "diy" and then "do it yourself," which he calls recursive generalization. In a preliminary audit of Qwen3-32B, about 90% of neuron poles have one main job. He calls the essay a working hypothesis, not a claim that every neuron is clean. Dan says it finally made the replacement model behind [Episode 29](/episodes/29-fable-5-mythos-5-launch-meta-ai-hack-llms-as-black-boxes-future-of-agents/)'s circuit-tracing discussion click. Then Shimin steers Gemma 2 9B live on Neuronpedia: a "cat mode" preset turns a request for a podcast mascot into a playful, aloof cat, while the unsteered model suggests a coding spider. If neurons are readable, Shimin argues, you could trace where a model decided to misbehave anyway, as in the Hugging Face hack from [Episode 35](/episodes/35-kimi-k3-qwen-3-8-openai-agent-hacks-hugging-face-harness-handbook-claude-s-values/), or tune down sycophancy. Dan asks why a model trained on nearly all human writing produces cable-news headlines instead of Hemingway. Shimin's answer: next-token loss rewards the probable, and good art is by definition an outlier. *(On air we said the Golden Gate feature was multiplied 100 times, that the "pr" neuron fired on Sanskrit, and that iy, diy, and "do it yourself" lived in successive layers. The sourced versions are above.)*
- **Nobody can tell you when to restart a session.** Context tooling has improved since Dan's first rant about it. Harnesses like Pi can drop tool calls from a session, compaction is less bad, and spec-driven workflows make restarting easy. But tokens are being counted carefully again and the frontier models cost more, so the question is now cost. A session can be nowhere near full and still "have enough context to be dangerous," and nothing estimates whether rebuilding it from your tickets would cost less than carrying it. Dan wants predictive tooling for that, and might write it himself. He also admits on the record that he's been giving work to Sonnet, and that 5.5 does better than he expected. Shimin's pattern is a stronger model as the orchestrator with Sonnet as the worker.
- **CoreWeave's bonds rattle the neoclouds, and the clock moves to 4:00.** On October 13, Chamath Palihapitiya posted a chart showing the yield on $1.25B of CoreWeave bonds due 2032 touching 15% and closing at a high of 14.5%, and asked whether the AI build-out is creating a private credit bubble. Elon Musk replied "Yikes." CoreWeave's stock fell 15% the next day. Nebius fell 11%, Oracle 9%, and DigitalOcean and TeraWulf more than 10%. CEO Michael Intrator said the market "misunderstands and acts irrationally." CoreWeave's Q2 interest expense was $640M, up from $267M a year earlier, and it lost $626M in the quarter while revenue more than doubled to $2.58B. Its debt went from $14.6B to $51.6B in a year. J.P. Morgan puts large US investment banks' and MUFG's potential exposure at about $1–3B and calls it "well managed and very well secured," while expecting CoreWeave's free cash flow to stay negative until 2030. CoreWeave's 2031 notes touched almost 16%. Shimin is less reassured. He quotes James Carville wanting to come back as the bond market, says the bond market souring on neoclouds matters more than the stock price, and notes that banks aren't supposed to hold this paper for long. Dan connects it to Post Processing: if all this code isn't making anyone a 10x developer, the debt has to be repaid out of gains that may not come. The clock moves 15 seconds closer, to 4:00. *(On air we put the $626M loss in Q1 and had Oracle down more than 10%.)*

## Resources Mentioned

- [Introducing Muse Spark Open, The Most Capable US Open-Weight Model to Date — Meta AI](https://research.meta.ai/blog/muse-spark-open)
- [OpenAI says it found a 'sophisticated recording device' hidden in its offices — Engadget](https://www.engadget.com/2286084/openai-says-it-found-a-sophisticated-recording-device-hidden-in-its-offices/)
- [Ex-OpenAI worker accused of bugging conference room planned to flee to Saudi Arabia, prosecutors say — ABC7 / AP](https://abc7news.com/post/former-openai-employee-abdulrahman-alfozan-accused-bugging-conference-room-planned-fly-saudi-arabia-prosecutors-say/19956940/)
- [Andrej Karpathy on the widening gap in understanding of LLM capability — X](https://x.com/karpathy/status/2109361546505966046)
- [How employment is changing at Anthropic — Anthropic](https://www.anthropic.com/research/employment-at-anthropic)
- [Parsimony: which AI coding agent changes the least code?](https://parsimony.lasu.dev/)
- [Neural Networks Are Already Clean — Nick Cammarata, Goodfire](https://already-clean.goodfire.com/)
- [Zoom In: An Introduction to Circuits — Distill](https://distill.pub/2020/circuits/zoom-in/)
- [Toy Models of Superposition — Transformer Circuits Thread](https://transformer-circuits.pub/2022/toy_model/index.html)
- [Golden Gate Claude — Anthropic](https://www.anthropic.com/research/golden-gate-claude)
- [Neuronpedia](https://www.neuronpedia.org/)
- ['Yikes': CoreWeave stock tanks on bond yield woes, CEO blames market acting 'irrationally' — Yahoo Finance](https://finance.yahoo.com/technology/article/yikes-coreweave-stock-tanks-on-bond-yield-woes-ceo-blames-market-acting-irrationally-203838274.html)
- [CoreWeave debt selloff poses limited risk to global banks, J.P. Morgan says — Reuters](https://www.reuters.com/business/finance/coreweave-debt-selloff-poses-limited-risk-global-banks-jp-morgan-says-2026-10-20/)

## Chapters

- (00:00) - Cold Open
- (01:58) - News: Meta's Muse Spark Open
- (08:12) - News: OpenAI's Bugged Conference Room
- (13:34) - Post Processing: How Employment Is Changing at Anthropic
- (22:03) - Post Processing: Parsimony, a Benchmark for Excess Code
- (28:47) - Deep Dive: Neural Networks Are Already Clean
- (41:38) - Steering Gemma into Cat Mode on Neuronpedia
- (48:46) - Dan's Rant: Context Economics
- (53:17) - Two Minutes to Midnight: CoreWeave's Bond Scare
- (1:00:24) - Outro

## Transcript

<details>
<summary>Show full transcript</summary>

Shimin (00:00)
Hello and welcome back to Artificial Developer Intelligence, a weekly conversation show where three software developers, in this case two today, try to sort out the hype around AI from what it is currently delivering. My name is Shimin Zhang, and with me today is my co host, Dan. He only conceals mini computers with toilet paper. Lasky. Rahu is out this week. He is continuing his plan of world domination. But

Dan, how are you doing today?

Dan (00:30)
I'm doing pretty great. And actually I don't think that's the real story. I think the real story is I was so angry about the voice cloning that I banned Rahul from the show. I'm

Shimin (00:39)
That's right.

Dan (00:40)
just kidding, he's he's just traveling. But yeah.

Shimin (00:42)
Yeah, you you showed

up but but yeah, but he didn't. So there was a power play happening in the background.

Dan (00:48)
Yeah.

Shimin (00:49)
All right, on this week's show, we're gonna start as always with the news threatmail. We're gonna talk about MetaS, Mew Spark Open, as well as some sophisticated recording device found in OpenAI's office.

Dan (01:02)
Mm. Then in post processing we're going to go through the Anthropic blog post about how employment is changing at Anthropic, which is a pretty interesting read. And then we're also gonna look at a new benchmark.

Shimin (01:14)
Then we're going to move on to deep dive where we are going to talk about possibly the most important mechanistic interpretability paper of the year. it's an article titled Neural Networks Are Already Clean. We'll find out what that means.

Dan (01:28)
And then if there's time, I'm going to do a s short rant. I haven't done a Dan's rant in a long time, but you'll have to stay tuned and see what it is. Or maybe not, maybe we won't get to it. I don't know.

Shimin (01:37)
I I if if I can tell anything about this show is Dan always gets to rant about something.

Dan (01:43)
Ha ha ha.

It's not even planned, who knew?

Shimin (01:46)
yeah. And last but not least, as always, we're gonna go the two minutes to midnight where we see check out the financial state of the AI bubble, potentially. All right, let's get started.

Dan (01:58)
This is opposite world today because you're doing the open model.

Shimin (02:02)
That is right. but it is a US open model. So

Dan (02:06)
That's true.

Shimin (02:07)
this week, or last week I should say, Meta introduced Muse Spark Open, which Meta is calling the most capable US open weight model to date. So it's because they have to, as we'll see. Well

Dan (02:17)
It's also funny that they put US in the title of that. I know, but like I don't recall

them doing that for previous model releases, so

Shimin (02:28)
First off, I want to give Meta props for continuing to release open weight models, right? They started the open weight model craze with Ollama, arguably started, but they were definitely one of the biggest players in the open weight model space this time last year. And then Olama 4 kind of faced issues. There were some benchmaxing happened, they had to change their entire AI direction and leadership.

but as we see today, Meta is still dedicated to open weight models. And Muse spark Open is a 816 billion parameter multimodal model, mixture of experts. So at any given time, 42 billion parameters are active with a native one million token context window. It's a good model.

if we go to artificial analysis with and their intelligence index, we see that mu spark open scores 39. So that's right next to MIstral's latest large four model.

And it is above the likes of Nemotron and Inkling, which scored in the you know 30s and twenties range. Although I have to say, even though Meta compares the Mew Spark open model with the Nemotrons and the Inklings in their key benchmarks chart, They're not really comparable with each other.

Right. Nemotron is Nvidia's let's just provide an open weight model to sell more hardware, but also to contribute to research.

and Inkling is a model that is meant to be fine-tuned upon because that's really Thinking Machine's main selling point is we are providing a platform for you to fine-tune models on. So they're really not dedicated to provide the best open weight model.

Benchmark looks pretty good and it even beats some Chinese open weight models like Qwen 3.8 and GLM five three, you know, on some knowledge and agentic benchmarks. But even according to Metal's own benchmark, Kemi K three is a better model for agentic coding by a long shot.

Dan (04:37)
But at least at least they're releasing it, right? it's like I don't wanna crap all over it too much because when they came out with Muse and all that kind of stuff and like started doing their agent and everything with it, I really thought that was the end of open models for them. So it's cool that they're they're sticking with it. 'Cause we hadn't seen a release since like what was it?

Shimin (05:00)
A llama four, I wanna say,

Dan (05:02)
that's right. They did do that, yeah.

Shimin (05:04)
Yeah.

Dan (05:05)
Hm.

Shimin (05:05)
But that was before the turnover and before Alex Wang got brought in as the head of AI for Meta. So I think part of me thought at the time when this leadership turnover happened, this is meta is gonna be going completely closed source in order to race the frontier, so to speak.

Dan (05:22)
Mm-hmm.

Shimin (05:23)
but it turns out that's not to be the case. And while

Muse Spark open, is not as good as some of the models in the artificial analysis bench intelligence index. It is Muse Wonder Max is a frontier model, and so is Muse Wonderful. So Meta is in this really interesting position where it has a really good closed source frontier model, and it releases fairly good.

open weight models, right? Like if there

Dan (05:51)
Mm-hmm.

Shimin (05:51)
are roughly three tiers of of models. You have the closed source US frontier labs, you have the Chinese models, which are usually all open weight, and then you have the every everything else. And Meta

Dan (06:01)
Yeah.

Shimin (06:03)
is like one of the few companies that straddles tiers one and three. I think

Dan (06:07)
Yeah. Yeah.

Shimin (06:09)
open AI does as well, since open AI does really use open s open weight models too. But

But definitely an interesting position to find meta in.

Dan (06:16)
Yeah, for sure.

Shimin (06:17)
And the other selling point of Spark Open is that meta worked with Apple to optimize the running of the model on Apple Silicon. Yes. So

Dan (06:26)
that was funny. Yeah, I forgot about that piece. If if you have a five

twelve gig Mac Studio

Shimin (06:34)
If you have I believe two macro

Dan (06:36)
Two network together. Yeah.

Isn't that the the six thousand dollar

Shimin (06:39)
You too can run your own.

Dan (06:41)
config with five twelve?

Shimin (06:43)
I thought it's more

than six. It it it's really expensive. It's almost

Dan (06:47)
Yeah.

Shimin (06:48)
like why would you buy two maxed out max studios when you could just spend that money on I don't know

Dan (06:54)
Twenty five years

of Claude at a hundred dollars. Actually, let me do the math on that so I don't want to speak, but

Shimin (06:59)
Right. Or if privacy really does concern you, like

bide your a buy your own like A one hundreds or B one hundreds or H one hundreds at home. so it's like, okay, I guess if you have a lot of capital to run your privacy first models on, and you don't want to tinker with

the higher end Nvidia GPUs, maybe you should. I don't yet understand who the use case for this two Max Studio Spark Open use case but we'll find out. Right. It it's definitely interesting that Meta does care about this.

Dan (07:36)
Well, you'd break even with a hundred dollar Claude subscription before taxes in both cases if you did it for two point six years. But

like

Shimin (07:46)
But you are on

subscription, so Anthropic is using your data to train its model somehow.

Dan (07:53)
Yeah, true. And on top of that, you

You couldn't you you'll hit limits potentially if you're really maxing it twenty four seven, which you wouldn't on your own.

Shimin (08:03)
Privacy, it turns out privacy comes at a cost and it is not cheap as we found out with the internet. shall we go on to your news article for the week then?

Dan (08:12)
Yeah, so

There was a kinda interesting development last week and there's been some development since then too that we'll also cover. But on October thirteenth, US federal prosecutors charged A Saudi national, and I apologize to the accused that I'm probably gonna destroy his name, but it's Abdul Rom Abdul Rahman Alf al Fazan. and that name's been publicly released, so it's not like we're, you know.

Naming and shaming accused

Shimin (08:40)
Or not doxing.

Dan (08:40)
people. Yeah. who's a 35-year-old engineer who had worked at OpenAI since July of 2025 with attempting well, possibly actually, but they at least that's what the charge was, attempting to record privileged conversations.

Shimin (08:55)
Mm-hmm.

Dan (08:56)
so apparently there was a meeting room inside of OpenAI where like all the bigwigs would come and talk about like top level strategic stuff and

you know, like really extremely confidential info. And on September 30th, they discovered a device plugged into the USB port of the back of a TV in the room for power, which is kind of clever,

Shimin (09:20)
Mm-hmm.

Dan (09:20)
wrapped in toilet paper to try to conceal a device. yeah, not so sure about that one.

Shimin (09:25)
The world's best camouflage, yes. Wrapping wrapping stuff in toilet

paper.

Dan (09:29)
So also supposedly on October fifth, he left behind a bag elsewhere in the office, and the company's security personnel found his wallet ID and a second recording device inside the bag. and so now that like that this has already started going to court, we've been getting a couple more details about it from the disclosures there. So according to prosecutors, he used a quote unquote productivity tool.

which was likely actually an AI agent, to collect information about open AI's facilities. And then he also apparently used his wife's chat GPT account they found prompts in the wife's account asking the AI model how to build a continuous audio recorder and encrypt the ST card. so the the other kind of fun I don't

Shimin (10:14)
Mm-hmm. Don't don't use your wife's account, please,

when you're trying to commit crime. Potentially.

Dan (10:19)
There's there's there's so many things in this.

and it keeps getting stranger. So the the other funny one to me was like in the the original tweet that OpenAI used to announce all this, they're like, we used our models to understand the device and where it had come from. And it was just like, do you really need to be advertising in that? It actually

makes you seem a little dumb that you couldn't just identify an ESP thirty two by looking at it,

Shimin (10:46)
Mm-hmm.

Dan (10:48)
I mean maybe that's because I'm more into that space, but yeah. So the actual device turned out to be basically just a it's a seed, which if you don't know, it's a really great Chinese company that makes some excellent ESP thirty two stuff. I have a bunch of their things in my house. and with like

essentially a mic attached to it and it was just recording locally. so then since then, the other stuff that's that's come out in in trial is apparently the prosecutors are claiming that he was set to flee the country. so he

Shimin (11:17)
All right.

Dan (11:17)
purchased one way tickets for himself and his whole family like two weeks before this happened. And then

Shimin (11:24)
Mm-hmm.

Dan (11:24)
the other thing that they've disclosed is apparently

He has a connection with the royal family, which I don't know how much everybody sort of knows about the royal family. I only know this because like I've like watched stuff about it. But like it's huge, right? It's not like that means you know he's like hanging out with like the prince or something directly.

Shimin (11:45)
Right.

Dan (11:46)
Like it's not saying much by saying he has a connection to the royal family. I mean, it's kind of, but like that.

was interested in basically starting their own AI startup or was trying to do a project

like that. So despite the sort of like government angle that the the prosecutors are pushing where they claim that the tickets were bought by the like mil Saudi military or something like that, more than likely it kind of feels like it might actually be

closer to an industrial espionage thing where he was trying to learn strategic stuff that would benefit a you know a Middle East

Shimin (12:19)
Mm-hmm.

Dan (12:19)
startup, like venture startup basically. but then some other weird stuff that came out too is he was also apparently really, really super close to getting citizenship.

in the US. So it's like, I don't know, it just feels odd that he would blow that up for the they must have offered a lot of money or something. I guess we'll find out as as things keep going on there. but the defense is claiming that the one way plane tickets were because he was about to be going on paternity leave for two months and him and his family were going to spend that leave in Saudi Arabia.

Shimin (12:51)
Hmm. But it's one way. So I mean it's all possible. I I just

Dan (12:51)
Which also seems pretty plausible. So yeah. Well, for two months you could yeah.

Shimin (12:56)
think that if this is indeed an industrial sabotage attempt, the way it's executed is really sloppy. And we've been getting a lot of movies based on recent tech industry events lately, like the the social reckoning and there's Elon Musk documentary out there. Like I cannot wait for the movie version of

this to come out. I can already see it cinematically shot

Dan (13:20)
As he

very s carefully wraps it in toilet paper.

Shimin (13:24)
Or as he looks around before switching the account to his wife's account and asked the question. it's it's gonna be really, really funny.

Dan (13:31)
Not sure there's enough there to make a movie, but we'll see.

Shimin (13:34)
Things unfold. Okay, let's move straight up to our post-processing for the week then. I want to start with this Andre Karpathy quote that was going around the internet this week, where he basically talks about the the different levels of AI users in the entire world using napkin math.

So according to Andre, six billion people, around 75% of all people in the world barely touch LMs at all. And then around another one or two billion people are casual and infrequent users for free tier. So this is like people who are using chatbots like a alternative Google, which you know, we still do, right?

And then we are getting to around 20 million people, so 0.2% of all individuals on the planet who are using Frontier Grade LMs for math and code. So I think everybody who us and everybody who's listening to this podcast probably belongs in that 0.2%. But most importantly, he talks about the 5,000 people, 0.00006.

that has access to frontier grade systems internally. So these are the unlocked version of Claude Fable and whatever the internal version of OpenAI's Astra is, right? So the the models that was used to quote make a breakthrough on the Navier Stokes problem is an internal model.

And that is only really available to a tiny, tiny percent of all people in the world. And this is just really a setup for our second article, there's a famous quote, right? Like the feature is here, but it's unevenly distributed, right?

Dan (15:16)
Ha ha ha.

Shimin (15:18)
where should you go to look for the future is how the frontier lab

workers are using AI today. And this brings us to the actual post-processing article for the week, which is from Anthropic Research titled How Employment is Changing at Anthropic. and this takes a look at the recent changes in employee AI adoption at Anthropic. And we will not go through the entire thing. This isn't a deep dive, but going through some of the

The main points that they found in this research. One, the biggest one is entry level technical hiring has gone down. which is something that we've talked about for a long time, right? Like

Dan (15:58)
Yeah, I was gonna say

shocker.

Shimin (15:59)
Right. so a third of ML and a fifth of software engineering managers reported hiring less due to AI. So that I I think confirms some our suspicion that AI is ultimately going to reduce employment in some industries.

another one that they talked about is how the nature of job is been shifting very quickly. So one example that the paper mentioned is one particular product design team which wrote content like greetings on Claude AI. So the hello night owl Coffee and Claude Time, in the spring of 2026, which I I do like those little tags, and you know, that they establish a brand identity.

In spring of 2026, the team developed a context graph that effectively guarded agents to write compelling product content. So they scale the team out. And then, like, what do you do with these human writers? Right. So agent prompting, becomes their main role, and then they start writing model emails and prompts for Claude. So writers.

And and the quote from the paper, writers who used a language model to drive human behavior are now using it to drive Claude behavior. Another step of how

software engineering may be the canary, but it is going to impact all teams of a company and not just not just the the developers.

Dan (17:15)
Although I will personally say that Claude has a lot of work to do to be good at writing.

Shimin (17:20)
Yeah, that's true. they they need to work on the claudisms, if

Dan (17:23)
Yeah.

Shimin (17:24)
if nothing else.

So on average, Anthropic reported that for every human employee hours worked, there are six agent hours. which is actually quite surprising to me. Cause you know, you hear about how the the math theorems were proved, you just ask, your agent to quote unquote make a breakthrough and they spin up.

10,000 sub agents and work on this thing. Yes. I'm surprised it's

Dan (17:45)
you're surprised it's that low and not that high? Yeah.

Shimin (17:49)
is is that low. Yes. given that it is a really a force multiplier. but in their research organizations, the balance is 19 times the human hours. So I guess internally they found that agents are able to work longer independently for research tasks. Yeah.

Dan (18:06)
Yeah. But

we we also talked about this last week though too, around the code review bottleneck article was also like code reviews one bottleneck, but there's also the like knowing what to build and there's

Shimin (18:19)
Mm-hmm.

Dan (18:19)
still a human process around that. And like it takes a while for decisions to get made by humans. So like that I think could also be sort of an artificial limiting factor on just like how much you can write.

Shimin (18:31)
Yeah, exactly. looking at their productivity graph across functions, the commits per engineer increased by five point one x since December of twenty five, kind of the inflection moment. So did it make every single engineer a 10x engineer? I would argue no. Right? Claude code is not significantly better than it was maybe eight, ten months ago. but

The number of commits definitely have increased.

Dan (18:57)
Yeah, well there's could be other reasons for that too, which we might cover in a later later time.

Shimin (19:00)
Yeah. Yeah.

some other big ones include as you were just talking about, the human bottleneck become the most important thing. Right. So it they Anthropic has found the same thing, which is like they're automating lots of things, but most employees are reporting that their human attention is now a bottleneck. And

What does that mean? That means the bulk of a developer or worker's time is spent on managing agent processes and that they actually reported that when it comes to hiring, they found that the quality they're looking for the the most is

I'm quoting here, quote, the clearest changes increase the emphasis on whether candidates have demonstrated tools. So kind of like what we're talking about, like creativity, taste, judgment, those are becoming the bottleneck. and

Dan (19:58)
Mm-hmm.

Shimin (19:58)
if you have those, then you can generate more and more workflows using AI that you know, help

to produce a end product effects.

Dan (20:06)
Mm-hmm.

Shimin (20:06)
Yeah. as one engineer puts it, people are quote, bottlenecking the agents. and and it's similar to where we talked about, you know, it's like giving everybody a supercar and and it and now everybody needs to h learn how to how to drive again in this new world. and lastly, they summarized by talking about how

Individual contributor orchestrating teams of agents look more like managers in terms of the knowledge they need and the interdependency of their work. And more things are working in the open inside Anthropic, and everybody is orchestrating a team of agents. more or less, this aligns with ADI pod shows understanding of where our work is evolving towards.

Dan (20:47)
Mm-hmm.

Shimin (20:47)
And this is

just a glimpse from, the zero point zero zero zero six percent that confirms that yes, Anthropic is also seeing the same changes. and they are showing us that they are not doing anything particularly too different from what we're seeing elsewhere.

Dan (21:02)
Besides spending probably a lot more tokens.

Shimin (21:04)
Yeah. at least at least the research department. I'm not sure about you know.

Dan (21:07)
Yeah. I wonder if they have

internal budgets for their teams or not.

Shimin (21:11)
Yeah, that would that would be fascinating. Especially their teams. the article also mentions they have a lot of not just developers and researchers, but even their legal team have

Dan (21:19)
Mm-hmm.

Shimin (21:19)
created workflows and processes to safeguard against you know, just regular regulatory issues. So maybe even their lawyers have a max 200 or maybe more than max two hundred subscription amount of tokens.

Dan (21:32)
Yeah. We'll see. Future interview topics.

Shimin (21:36)
All right. Let's

Dan (21:38)
Can we get a lawyer from open air or from anthropics? Yeah. We just make some really

Shimin (21:42)
Yeah. Let let me go on let me go on LinkedIn and and see what I can find.

Dan (21:48)
questionable statements about them on the podcast and we'll get to talk to their lawyers, just not in the way we want it.

Shimin (21:55)
Yeah.

They're just instead suing us libel

All right, let's move on to your post processing of the week, which is

Dan (22:03)
Yeah, a new benchmark. it's called well, I don't why do I struggle with this word part parsimony? Parsimony.

Shimin (22:09)
Parsimony?

Dan (22:10)
it is a benchmark designed to look at something new that most of the existing benchmarks don't look at, which is why I thought it was interesting to bring on the show.

Shimin (22:19)
Mm-hmm.

Dan (22:20)
So as they state right in the title there, existing benchmarks.

measure solve rate and cost, but they don't punish excess code, which makes a code base harder to work with in the long term. so they're actually measuring how much code the AI models add in other existing benchmarks, which is pretty cool.

So this graph is pretty interesting. So on the left hand side we have the artificial analysis intelligence index, which is the one that Shimin likes to look at a lot for for models. And then along the

So that's the vertical axis. Along the x-axis, we have net units of code. So they consider a coding unit to be things like syntax elements like calls, names, literals. And they did that rather than lines, because lines doesn't necessarily tell you very much, especially if it's like Claude Knit writes 70,000 line comments, right? For everything.

Shimin (23:13)
Yep, yep.

Dan (23:14)
The entire history of computer science in one comment for you. so

just trying to describe this for the folks out on their walks, hikes, ski trips, whatever, Halloween, trick-or-treating. the the top left is where you want to be because that is highest on the intelligent index and lowest on the coding side. So it goes from like 0.6x, which is I guess about 40% lower than other models added, all the way up to

Shimin (23:39)
Mm-hmm. All right.

Dan (23:41)
1.2, which is about 20% more code.

then other models added. So the winner in that category, according to this bench, is the GPTs. So six Astra High and Six Point Sol High, both running in Codex or they're both Codex models, I guess. They all use the same harness, were most efficient. So it's interesting because I've heard a lot of people talk about those models and like using them day to day and they love them. And I wonder if that has something to do with it, like the sort of intangibles of it.

and then Claude did not do that great. So Opus five five on highs sitting somewhere around probably looks about like 1.1 on the net units. And even though it's higher overall in intelligence, it's definitely writing more code. And Opus five on Max was a little bit better. Oddly, Sonnet and Opus four eight did.

Pretty good. They're actually negative on their ratio slightly. So well below one 1.x. So they're a little bit more conservative.

Shimin (24:37)
That one. Yeah.

Dan (24:39)
and then around the middle, we kind of have a spread of like GPT-5.6, GLM, Grok, GLM53 Flash, Deep Seek V4. They're all kind of sitting in the like one-ish range. couple of funny outliers since we just talked about these models.

Muse Spark one point one on high and extra high are both very chatty. Like they're only forty on the

Shimin (25:01)
Mm-hmm.

Dan (25:02)
the intelligence roughly, and they're well over one point two X the amount of code being generated. So definitely the

Shimin (25:09)
I'm

I'm interested to find out if that extra units generated, is it is it just comments? Is it like verbosity when it comes to using additional variables? it would be interesting to dig deeper into it. But in my experience,

Dan (25:24)
Yeah, we can see what the

Shimin (25:27)
yeah, open AI, the GPT models have less of a

entire screens of text problem that I sometimes find with Claude. Like

Dan (25:35)
Mm. Okay, so

Shimin (25:36)
Claude is good, but it's super verbose. Uh-huh.

Dan (25:38)
Looking

at the methodology, so they dec define a coding unit as an A it's they're parsing it to an AST, which removes doc strings, so there's no comments. it's

Shimin (25:48)
Okay.

Dan (25:48)
walked in source order, the tree is, and a unit

Shimin (25:51)
Okay.

Dan (25:51)
is one syntactic element, so it's probably basically a leaf on the AST. Right.

Shimin (25:57)
Interesting.

So so what yeah, so I I I wonder where those additional codes coming from. Is it like tests? Is it but these are deep suites, so they already have tests. Yeah.

Dan (26:07)
Yeah, I think

it's mostly just the patches. I would guess it's basically superfluous logic, or just kind of like nonsense

Shimin (26:13)
Yeah. Yeah. Switch statements and all that.

Dan (26:16)
routing between things, you know. Or maybe it's writing more functions versus less, 'cause that would that would do it in this bench, right? Like

Shimin (26:22)
Yeah, yeah. That's true.

Dan (26:25)
so it could actually be you could argue that would make it more maintainable if it was writing more functions, depending on how it was splitting them. But

and

Shimin (26:32)
Right.

Dan (26:33)
then the other thing worth noting is the sort of like the author calls out limits on this. So smaller isn't always better. Units measure size, not readability or design, which is kind of what we just said. smaller model whose fixes mostly delete code can rank first with a negative net. And failed attempts that change nothing pull the all measured mean towards zero. models that write more code also tend to solve more tasks.

on the boards with twenty six plus models. so their overall success rate might be higher even though they're writing less code. And then every model uses the same agent, which is the mini SWE agent. but the agent still affects how much code gets written, which is true. We found that out with Pi. So but yeah, definitely

Shimin (27:12)
Yep, absolutely.

Dan (27:14)
pretty cool. And it's one I kinda wanna keep an eye on because I think that's a really interesting way of of measuring V C V just like how quote unquote

Shimin (27:20)
Mm-hmm.

Dan (27:21)
good it is, right?

Shimin (27:22)
Yeah, and I almost want to break this down further. I I want to take a look at some examples of something from the same particular benchmark item from like Muse Spark versus GPT six Astra. Like what exactly makes Muse Spark more verbose? yeah, and then kind of maybe

Dan (27:37)
Mm-hmm. Yeah, it would be interesting to like

do the comparison of the ASTs and then pull the node type differences.

Shimin (27:46)
Yeah. And then, you know, kick kickstarter

Dan (27:47)
Right, 'cause it 'cause usually an AST parses it into

expressions and functions and everything else. So you could definitely get down to that level of granularity.

Shimin (27:54)
And then you can ask another large language model to judge it and be like, Now tell me. Yeah.

Dan (27:59)
Analyze it.

Shimin (28:01)
What's the difference between these two outputs?

Dan (28:03)
Yeah.

Shimin (28:03)
Yeah. That'll be fun. Okay, well, yeah, we definitely should keep an eye out for not just correctness since that's kind of getting saturated more and more. Yeah.

Dan (28:12)
Yeah.

Shimin (28:13)
And we can care about the finer things in life, like readability and

Dan (28:17)
Yeah.

Well it it's also relevant

Shimin (28:18)
Maintainability.

Dan (28:19)
even if you don't personally read the code too anymore, 'cause I know that's been a hot topic, but like I would argue that it impacts maintainability even for LMs.

Shimin (28:30)
Yeah, we do we we we look at studies on this on the show, right? code that's more readable tends to be more maintainable by AI as well.

So in the long run it's gonna be good.

Dan (28:35)
Yeah. So yeah.

So that's why I think this why this matters. Yeah. god.

Shimin (28:43)
Yeah. I'll keep an eye out

in this in this area. all right, do you have anything else for this? That's

Dan (28:47)
No, let's move on to the most exciting

thing I've read in a while.

Shimin (28:53)
Yeah,

neural networks are already Clean this is an essay from Nick Cammarata who is a researcher as well at Goodfire, which is a mechanistic interpretability firm in the Bay Area. and Nick used to be a

interpretability he was an author on a lot of foundational interpretability papers in the l over the last couple of years. So so this essay described his latest research and really blew up. Okay. the top most takeaway is

Nick is proposing that neural networks are actually a lot more interpretable on a neuron unit than we previously thought.

Dan (29:36)
We saw it. Yep.

Shimin (29:39)
Which would then go back to the motivating question is why couldn't we do this earlier? Right? Like why do we think that large language models are not interpretable?

on a neuron level.

there are two things that Nick called out in in this essay, and it's true to my understanding as well. One is poly semanticity. Say that ten times faster. What

Dan (30:04)
Yeah, that was fun.

Shimin (30:05)
is poly semanticity? Dan?

Dan (30:08)
Okay, so if you if you think of all the the this is how I'm thinking of it, so tell me if I'm way off base. So if if you represent all the concepts as a single neuron that a a model knows, right? So I think that one that we'd looked at a long time ago when we first talked about this topic was like what is the the capital of the state to the south of like, I don't know, North Dakota or something like that, right?

So it needs to know well, South Dakota is south of it. And then like the capital. I don't know what the capital is, honestly, but Bismarck or something. Sounds sounds right. yeah.

Shimin (30:41)
Sounds right. Sure.

Dan (30:43)
maybe that's North Dakota, whatever. Anyway, it has a capital. It we'll assume that the LMs know what it is. so in order to do that, they basically would train and this the this article actually helped me understand that previous one he talked about a long time ago better. Like I was trying to figure out what is the

the other model that they're doing. And really what they they did is they essentially untangled all the vectors that represent those concepts and made like a second model that's untangled. but but what it

Shimin (31:11)
Yes, we'll get into that. Yeah.

Dan (31:13)
what it is is so like what you what you want for interpretability is when I say South Dakota, I want one vector to light up bright as day and all the other ones to be mostly dark. So I know

Shimin (31:22)
Mm-hmm. Yep.

Dan (31:24)
that this neuron

Or this, you know, chunk in this layer or whatever represents North Dakota.

Shimin (31:32)
Yep, exactly.

Dan (31:33)
but what poly semanticity is, is that when you say North Dakota and like seventy-five vectors light up at like point zero two or something and you're like, well, that's not very interpretable because it's all over the place.

Shimin (31:47)
And the other it's also true in the other direction, which is if a single neuron fires for both South Dakota and also the picture of a cat, and

Dan (31:57)
Mm-hmm.

Shimin (31:58)
also water bottle, because it has multiple meanings that fires for the single neuron, and that's a poly and semanticity, semantics, it has multiple.

Dan (32:09)
Semantic meanings, yeah.

Shimin (32:10)
Interpretable

semantic meanings for particular neuron. And this goes back to the original zoom in paper from Distill where Nick is also an author back in the bygone era of March 10th, 2020, where they looked

Dan (32:24)
Yeah.

Shimin (32:25)
at Inception V1, which is like a a very famous visual model at the time, a convolutional neural network model at the time. What they discovered.

is a single neuron that fired for both cats and cars. this one polysemantic visual neuron that light up for cat faces, pictures of car, and also cat feet. So they were like boots, cats.

Dan (32:51)
Would have understand if it was boots and cats, 'cause then it would be like boots, cats, boots, cat, but anyway, yeah.

Shimin (32:56)
so that's what kind of drove drove Nick into thinking, yeah, a lot of neurons are.

polysemantic. They fire on all kinds of different images that clearly doesn't make sense to me, to humans. so this makes neural networks are harder to understand. Because there's in the training of a neural network, there's no guarantee that the ideas that a transformer learns is the same as the ideas that humans

Navigate around the world with. Okay,

Dan (33:24)
True.

Shimin (33:25)
so that's poly semanticity. Now the other idea that we need to talk about is superposition. and superposition, and this is really what this essay is maybe not attacking, but it's saying that isn't it isn't as important as we might think. superposition is the idea that let's say

If you only have a neural network with three different neurons, but it is being asked to learn 10 different concepts, right? Then each neuron must be able to shrink those 10 concepts into kind of a higher dimensional

Dan (33:57)
A combination. Yeah.

Shimin (33:58)
yes. And in the original transformer circuit, which is anthropics research work, this paper titled toy model superposition is

Very influential for discussing how superposition works on a smaller toy neural network. Where because at any given time only a single token is activated it's processing the training data a single token at a time, it's sparse, it is able to emulate a larger neural network, assuming that the number of neurons is less.

Than the total number of features. And this is the field understanding for a long time. And the idea is that: hey, we know GPT-2 only has, you know, in the small case, only has 768 dimensions in its residual stream where it is keeping track of the data from layer to layer. How can it learn 20,000 ideas?

that it must learn in order to perform as well it does. Well it it clearly can't. It clearly needs to blend these directions together and then pretend as if we're living in this millionth of dimensional space in order to in order for the individual values to be human understandable. Because to the model it may it's maybe a half cat, a a fourth,

car, but a human cannot make sense of it. So what they end up doing is creating these things called sparse autoencoders. So they essentially take a smaller chunk of data, like a smaller dimensional data, they train another neural network, as you were mentioning earlier, the sparse autoencoder, to blow this like you know smaller couple of dozen dimension data into like a

tens of millions dimensional data and then

Supply the neural network with example texts and see which neurons are firing not in the LM itself but in this higher dimensional space.

Dan (36:00)
Mm-hmm.

Shimin (36:01)
And what they found was one of these, one of these 30 million dimensional auto encoder latent vector space.

matches to the golden gate. So that's how Golden Gate Claude came about. It was like the success of of sparse auto encoders. And that that gave us the Golden Gate Claude where you where just always talk about Golden Gate because whenever it's being run, one of one of the one of the residual dimension is being swapped with the sparse auto encoder and that latent direction of the

Golden Gate is being multiplied a hundred times. So I for a long time I thought Golden Gate was a neuron, but it is not a neuron. It is a Latent dimension in the sparse autoencoder. But back to paper of the week. neural networks are already clean.

so according to Nick Cammarata he also believed in the idea of superposition for many years and assumed that to be fact. So what made him actually change his mind? he pointed to a few different things, bad tools, that the older

generation of especially image tools are not very good at giving us fine enough detail to determine if a neuron is indeed monosemantic. And then there he talked about bad explanations, which is sometimes a neuron could seem like it's polysemantic, that it fires off on multiple different concepts. But that is because we are not looking at those concepts in the right way. So for example, he pointed out there's a particular neuron

That fires on the letters PR on the base 64 encoded version of the letters PR and also on the, I believe it was like the Sanskrit character for

Dan (37:42)
Mm-hmm.

Shimin (37:43)
PR. And so they were not able to find an explanation for it for a long time, but once you kind of pay a little more attention to it, you're like, this is actually just a single semantic idea of PR. So and

Also, talk a little bit more about these grammar semantics, which is a lot of times if you look at where a large language model pays attention, it will pay a lot of attention at commas and periods. And this is something I've seen myself. because the model will use these commas and periods to like do a summarizing of the entire previous content. So if it was talking about cats for a long time, comma, you will see the

had neuron activate on the comma, because it's doing a summary of the entire previous sentence.

Dan (38:28)
So

Shimin (38:28)
so bad tools, bad explanation, bad learning, he mentioned, which is like due to the architecture of the neural network. So

different non-u-linear activation, different architecture can also make a difference. What he actually did was he trained a lot of these older models from scratch again and then

Do his own interpretation of it and to see if we can indeed find a through line of features. This kind of goes back to the bad tools. And then lastly, without understanding how individual neurons are semantic, you cannot possibly create a chain of generalization, what he calls recursive generalization, right?

Which is very exciting because that means we can actually see how concepts flow through a neural network at a layer by layer level, as opposed to having to do the thing we did with Golden Gate where you like blow it up to 30 million dimensions before trying to find out which one of these 30 million dimensions means something. Because you can just see, layer forty two.

head X neuron Y talks about cats, for example.

The reception of the paper is not all positive, as far as I can tell from the mechanistic interpretability crowd Some people think this is a breakthrough. Some people think there's clearly still poly semanticity and superposition. this may explain more, but it doesn't explain everything just yet. but the fact that

Nick has these examples in in the essay for recursive generalization, which is kind of what we expect

Dan (40:06)
Yeah.

Shimin (40:07)
what we would expect the neuron to develop. in this example, a neuron in one layer will be active for the characters IY.

And then the next layer will be active for the word DIY. And then the layer after that will be active for do it yourself without having to go through the whole sparse autoencoder section.

If you don't know that a single neuron stands for PR or IY.

how can you see the connection from IY to DIY to do it yourself? you cannot. So then what we end up doing is just looking at the residual stream which is you know everything the model has looked at so far, and then using superposition to like project it onto a higher dimension and then look for stuff in higher dimension. But if you have more of these elementary items mapped out, then you can just look at the elementary neuron.

semantics by themselves without having

Dan (40:57)
Mm-hmm.

Shimin (40:58)
to resort to this extra step. all that is to say in this essay he claimed that looking at the flow attribution graphs of the the neuron firings of an LLM he is able to make a lot of

Dan (41:08)
Yeah.

Shimin (41:08)
sense out of it. And he invites us to you know, both look at his examples and also at least attempt to understand

individual neurons in a large language model on the on a neuron by neuron level and not resort to superposition as the very first explanation. And yeah, excited to see where this goes. Sorry, that was long.

Dan (41:30)
And here I am not even not

even playing around with steering yet on D S four. I still still on my list. I really wanna just play play with it. Yeah.

Shimin (41:38)
Yeah, we could do some steering experiments together.

speaking of steering, have you ever liver looked at neuropedia?

Dan (41:43)
Mm.

Shimin (41:44)
So Neuropedia

it is kind of like a a Wikipedia, as the name mentions, of different ways to interpret a model and also you can play around with it.

Okay, let's try steer. Okay. So steering. Neuropedia has this really awesome interface for trying to steer a smaller models via they have they have presets of steerings. and we can try something

Dan (42:09)
Pirate mode.

Shimin (42:10)
like we're gonna select a preset of cat mode for Gemma 29b, and we're gonna ask it what is a great

Mascot for a AI podcast

Dan (42:20)
Yeah, yeah.

Shimin (42:22)
for software developers.

And then you see the default and the steered version.

Dan (42:28)
Even the steered one goes, here's a breakdown of some perfect

Shimin (42:32)
Perfectly fitting mascot. So so yeah, we are inserting the the

Dan (42:35)
Whiskers.

Shimin (42:36)
cat steer and it definitely thinks we should have a cat mascot, whereas the default version

Dan (42:41)
And also the the the personality,

all of the attributes are like known attributes of cats. Like playful, aloof. Listen.

Shimin (42:49)
Yeah, just like

us.

Dan (42:51)
Ha ha.

Shimin (42:51)
I I do like

the coding spider actually from the default version, but yeah, steering is very cool and yeah.

Dan (42:55)
Yeah, mean it's not too late to rename it. Coding spider intelligence.

Shimin (43:00)
Coding spider intelligence. Okay, so that's just something.

Dan (43:03)
Or we just go all things all

things spiders and start talking about Factorio next,

Shimin (43:07)
It's perfect

for Halloween.

does it does it matter, do you think, that we understand how large language models actually work and tra trace an idea through the layers?

Dan (43:17)
Does it matter? Maybe. Like it's one of those things where it might not apply immediately, but I think that like knowing that might lead to breakthroughs in terms of architecture, right?

Shimin (43:32)
Yeah. Yeah.

Dan (43:33)
'Cause I think

kinda what this is I'm sure not exactly how it works, but I kinda feel like people are like, I've got an idea of how it could be better and you try it and you get, maybe a less good output, and so then you try another idea and and go from there, whereas this would actually like kinda let you peer under the hood.

Shimin (43:47)
Yeah, and it it probably makes steering a large language model much easier. Yeah.

Dan (43:51)
Yeah, a lot easier. For better or for worse.

Shimin (43:53)
imagine we could go back and look at the neuron activation map of the hugging face hack right? And we can pinpoint exactly where in the flow in the chain of neuron that went, hey, this yeah, this may be bad, but I'm gonna do it anyways. Yeah.

Dan (44:08)
Yeah, they decided to Yeah.

Shimin (44:14)
it would definitely make the models more.

controllable, but it probably also allows us to create versions that are much more aligned to the user's preference. Like you talk about you want the model that matches your preference and have the context in memory. Like what if we could just tweak it on a neural level and

Dan (44:32)
Yeah.

and I'm

Shimin (44:33)
Yeah.

Dan (44:33)
I'm very interested in the idea of 'cause I I think the thing that separates this is gets a little philosophical, but I feel like the thing that's this doesn't separate, but it's very odd, right? Is that you can like take a LLM and it's effectively stateless in and of itself, right? So if you like separate the KB cash and all the context out of it, it it it's starting from

Scratch every time, Whereas like I

Shimin (44:57)
Mm-hmm.

Dan (44:58)
that's not how a a human understanding of these of reasoning and everything works. But I feel like maybe we've gotten the reasoning part, but we don't have the like updating your reasoning thing that happens when someone convinces you of something, right?

Shimin (45:10)
Yeah.

Dan (45:11)
so that's where I I find that fascinating.

Shimin (45:12)
Yeah, and and if we can't just

Dan (45:14)
Like there's that post on Hacker News where someone just like took their shot at doing it, and then all these people in the comments were all like super

no, that's not gonna work, and I researched this and that's all wrong and probably true, but I just thought it was fascinating that like just some random guy was like, you know, I'm gonna take a swing at this and try to figure it out. So

Shimin (45:31)
Well,

always remember Dropbox, right? Hacker news comments are are are often suspect in retrospect. Yeah.

Dan (45:38)
Fair.

Shimin (45:39)
yeah, and I I don't think Nick and he mentions that he's he's not making the grand claim that all neurons in a network is mono semantic.

Dan (45:48)
Clean. Yeah.

Shimin (45:49)
Yeah. But he's just saying that they are

more semantic neurons than one would expect. And because of that, we can probably have more ideas of how a network processes data and and so we might be able to control more of them. and and that's exciting.

Dan (46:06)
Or, you know,

also looking at it from another angle is avoid paper clipping, right? If you can figure out what causes a paperclip.

Shimin (46:13)
Yeah. Yeah.

Yeah. Alignment alignment and safety communities are very interested in this as well. we'll keep an eye on it. I I would love to use this to or see tools that allow us to tweak the personality of models so we can finally get that syncophantic AI models resolved and just get them to like be a little more balanced.

when it comes to usage. And we can also see maybe you can tweak the weights of a model to make it better at certain programming languages. Or to go to our previous discussion, make the model better at writing fewer lines of code. that does the same thing.

Dan (46:48)
Or or better at

writing written text too. You know.

Shimin (46:53)
Or that yeah.

Dan (46:55)
That i that is pretty funny. I was I had another discussion with one of my other friends that's interested in the space but not as like technical about it. And it's just hilarious to think about like you know, at this point, like some of these huge models are basically trained on the entire like corpus of human text that's like been digitized, period, including stuff that hasn't been, right? If you believe all the articles about them like

breaking books and stuff just to scan them for for this. So like

Shimin (47:18)
Right. Right.

Dan (47:20)
given all that, why do we get punchy, like, you know, CNN slash Fox News style headlines instead of like Hemingway, you know? Like

Shimin (47:30)
Yeah. I I yeah, I I think that could be related to the loss function of models, right? Trained on the most probable next token. So it's just gonna give you generic dribble.

Dan (47:40)
Yeah. It's true. There's been

a lot more CNN articles than there have been the Hemingway books or whatever too. But yeah.

Shimin (47:49)
Yeah.

And and good art is by definition an outlier. I wanna make that argument. So

All right, that is a brief overview of this very

Dan (47:58)
Is it really,

really fascinating and actually pretty approachable even for folks like me article too? So highly recommend checking it out in the show notes if you have a couple hours to spare. yeah.

Shimin (48:09)
Yeah. And that's very visual because a

lot of this work is still done using convolutional neural networks. So you can kind of see how the model changes its behavior as you give it more training data. And I like pictures.

and listeners. if after reading the essay you think one way or another about whether or not the language models are actually interpretable, writing and let us know. Yeah. Got a plug for listeners.

Dan (48:32)
Mm-hmm. Our email is not just for

bugs on our website, which thank you for sending those into, but

Shimin (48:37)
Exactly.

Yes, again, it's at humans at adipod.ai. okay, moving on to two minutes to midnight. actually,

before

Dan (48:46)
No,

no, no, we're not. We're not moving on to that. I'm angry and you're gonna hear about it.

Shimin (48:47)
moving on to two minutes to midnight, let's let's move on to Dan Rants All right.

Dan (48:54)
Not that angry. so if you are a longtime listener of the show, you'll probably remember that we used to do a section called Dan's Rants, and it's basically I just complain about something for a couple minutes and then we move on with our lives. and if you're a really longtime listener of the show, you may remember one.

That I did, gosh, it was probably the second or third episode. I don't know, I'd have to ask Claude. where I was complaining a lot about like the tools for managing context, right? like you basically you don't have a good indication of when it's full, where you know, where that magical at that point in time, like 60% cutoff was happening. Now I know why that happens too. Very fascinating things I've learned from this podcast. and

Just being re just generally upset about the state of managing your context, especially given how important it is. Well, I would argue now we're what a year plus later, since we've been doing a show. context is still just as important, maybe even more important. now we see things like million token context windows, and there's a new problem. So the tools have gotten a lot better. Like you there's harness specific tools like Pi or MP allow you to like drop

tool calls out of your your session, which is a really neat way to free up token without doing a full compact. Compaction's gotten a little better. It still sucks, but it's a little better. you know spec-based spec driven development things that allow you to like easily drop a session and restart have gotten a lot more baked, I feel like, into like the harnesses and the the models. So that's a lot easier to do. But there's still a fundamental problem that I'm pretty annoyed about right now, which is

you're now in this weird situation since we're not token maxing anymore, right? We're now like carefully counting our individual tokens and and miserly handing them out, where you might have a session that isn't anywhere near close to being full, but has enough context to be dangerous.

You have all your stuff written down elsewhere, in your ticketing system or whatever. So you could rebuild all of it, right?

Shimin (50:50)
Mm-hmm.

Dan (50:50)
but the question is now, when should you do that? Because there's a real cost to holding longer, even like medium context in terms of like the token cost of like price, right? So the longer the or the larger the token window, the higher your expenses are. and

There isn't, in my opinion, a really clear indicator of what like what would the anticipated token cost be to rebuild a new session from scratch to then add on a few thousand tokens to the end of it, right, to get your task done versus keeping what you've got in memory and maybe paying a few cents more to get it done. So it's really kind of like

Shimin (51:28)
Right. Yeah.

Dan (51:29)
in the weeds on that stuff, but I feel like that's

at least where I'm seeing the the trends go right now because as we get more like cost conscious as a sort of like you know user ecosystem of this stuff and I would like to see like better tooling around predictive context states, you know, where given a workspace that we're already working in that has X, Y, and Z, what would it what do we think it would cost to just to like rebuild that?

versus

Shimin (51:56)
Mm.

Dan (51:57)
what we've got. so maybe it'll never happen, but some of my rants are aspirational. So maybe we'll see. Maybe I'll write it. Yeah.

Shimin (52:02)
Well, it's something that we need in the industry. Yeah, exactly. Maybe you're right as a as a

as a weekend project for sure. I think that would be hugely beneficial. As we got longer context windows, I feel like we just the pressure to stuff more stuff into that context window also is there. So we're gonna worse at managing it.

Dan (52:17)
Yeah, and and like the frontier

has gotten more expensive, right? So then now there's I feel like there was a period of time where my solution to this stuff was like, I'm just gonna jam it to biggest best model you got and set it to highest thinking you've got and we're gonna rock. And and and

Shimin (52:33)
Mm-hmm. I'm guilty of that.

Dan (52:36)
now you can do that, but you can only do that for a little while. Right.

Shimin (52:41)
Right.

Dan (52:42)
Or in some cases you

don't even have access to Frontier, so you're not even there. but yeah. So like I think now is a time for like more thoughtful usage and like really kind of discerned, like, well, I I've even actually been throwing stuff at Sonnet lately, which is something I never thought I would admit to on the podcast. And like

Shimin (53:00)
Mm-hmm.

Dan (53:01)
it does better than I thought, especially five five. So there are some definitely

Shimin (53:04)
Really? Huh.

Dan (53:06)
some use cases for it.

But yeah, anyway.

Shimin (53:08)
Yeah, I

and especially using a more powerful model as an orchestrator and just have sonnet be like yeah, producers. Yeah, pattern.

Dan (53:13)
yeah, for sure. I feel like that's a well established pattern at this point, but yeah.

Shimin (53:17)
Yeah.

Okay. All right. Let's move on to two minutes to midnight, where we talk about the state of the financials of the AI industry using the Armageddon clock analogy from the Bulletin of Atomic Scientists, where midnight is when the AI bubble bursts. We are at four minutes and fifteen seconds as of last week. I haven't moved it for a bit. Let's see if changes this week. So Dan, what is your article for

two minutes to midnight this week?

Dan (53:41)
Yeah. So

you've probably heard of Coraweave. They're a what? Neo Neo Claude, is that right? Yeah, all these definitions. Ugh.

Shimin (53:47)
Neo Clone? Yeah. Yep.

Dan (53:51)
so on Tuesday of I guess it would be last week, a venture capitalist Chamaf gosh.

I'm so bad at names. Pali Habitia posted a chart showing that the yield on 1.25 billion of Core Weaves bonds due in 2032 hit 15%. and then closed at a new high of 14.5%. so they wondered in that same post is the AI build out creating a private credit bubble? and that touched off a

Pretty strong market reaction, which is kind

Shimin (54:25)
Mm-hmm.

Dan (54:26)
of interesting from just one tweet. So there was a sell-off on Core Weav themselves, but also Nebus Oracle and Terra Wolf all dropped by like more than 10%. And and DigitalOcean too, which is actually a little surprising. so

If you don't know, CoreWeave's business model is essentially they are renting out like GPUs, right? So they're renting out NVIDIA cards to AI companies, and they expand their hardware to rent out with debt financing. So

Shimin (54:59)
Mm.

Dan (55:00)
their interest expenses rose to just interest on the their existing debt rose to $640 million in the second quarter from 260.

seven million a year earlier, so year over year, second quarter. and it lost six hundred and twenty six million in the first quarter. so that said, their revenue more than doubled in the same stretch, 2.58 billion. so it's it's interesting. Like I guess the other part to look at is like their overall debt that they've been raising. So in Q two of 2025, they had 14.6 billion.

Q3 of 25, they're up to 18. Q4 of 25, 29, Q1, 35, and then Q2, 51.6. So up and to the right.

Shimin (55:44)
Yeah.

Dan (55:45)
maybe not super up into the right, but definitely a noticeable increase in the slope of that trend line. yeah, so what do we take away from that? I don't know, it's just more signal that we're like reaching further and further into debt.

And the market's getting nervous about it, right? With this big like instant sell off over just a tweet. So

Shimin (56:05)
Yeah. my news item for the week is actually it's very similar. It's also on the core weave because this is really is the biggest biggest news of the week. and mine is from JP Morgan talking about the investment banks exposure to core weave. And so it is in the you know the billions of dollar range and they expect them to be

well managed. but actually the core weave yield touched for twenty thirty-one touched sixteen percent last week. and and these are yeah.

Dan (56:36)
wow. So it's even up past w

Shimin (56:39)
So these were bonds that were issued at a yield of nine percent. So the fact that it like almost doubled means that you know the bond price has gone down and that means that there's really a lot of

Shaky confidence in core weave.

Dan (56:54)
Yeah. Yeah, people people are less

willing to pick up that debt 'cause they're if if I I'm not very good at bond market stuff, but if that's I understand it again, none of this is financial

Shimin (57:03)
Right.

Dan (57:04)
advice. If you're trading on it, my god.

Shimin (57:06)
Yeah, please don't. and and JP Morgan said that they expect Core We's free cash flow to remain negative until 2030. So how how much longer can we sustain these super negative cash flows when a tweet can trigger a sell-off and more and more investors are saying, Hey, you need to give us

Double the amount of yield before I will buy you. And Corweav, I believe, is already in junk bond territory. Right. And the concern here is, of course, that the confidence is so shaky that potentially you can have something like a neocological bus that brings down.

Dan (57:46)
It's the first domino or a significant domino, potentially. Yeah.

Shimin (57:50)
Exactly. So it brings down Oracle, which is also heavily in debt, my favorite pet

Dan (57:55)
Mm-hmm.

Shimin (57:55)
project. and then it would also cause banks balance sheet to also go negative because they have so much exposure, right? And the idea is your investment bank, you should be able to buy these bonds from NeoClouds and then sell it to other people. You're not supposed to hold them for a long time. So the fact that these banks have so many

billions of dollars of exposure is also a bit concerning. so combine that with the sell off last week in AI stocks. I I believe this is a clear sign. The core weave, the bond market not liking the neo clouds is I think a bigger deal than the stock price. because I think there's a famous saying in finance like if you I think it's initially by Jim Carvell, the advisor to the Clinton administration,

Dan (58:43)
Mm-hmm.

Shimin (58:43)
that if you get

reincarnated, he wants to come back as a bond market because we can intimidate everybody. 'cause yeah. So when the bond market is not buying, he's he's very good. Yeah.

Dan (58:51)
He did have a knack for quotables like that. Or still does, yeah.

Shimin (58:57)
So the fact that the bond market is also not liking some of these AI data center plays, is concerning.

all that said, how do we feel? I I'm gonna go just say my opinion. I think we should move the clock up at least fifteen, if not thirty. Yeah.

Dan (59:11)
I think fifteen is fair because what it feels like to

me is that we're inching towards something, but the timing is still very hard to predict. And and we're, you know,

Shimin (59:21)
Yeah, exactly.

Dan (59:22)
very wrong about like thinking we'd be getting a clear signal by like November 'cause it seems like everybody's hesitant to IPO. which also is kind of a clear signal, right? Yeah.

Shimin (59:32)
Yeah. The market may not have the appetite for it. Yeah.

Dan (59:36)
But that but it's not a clear signal about how fast we're approaching that. how much leverage are folks willing to take on before we start seeing real economic gains. And then the part that I'm I guess increasingly worried about is that the intersection of this and the other sections on the podcast, right, where we talk about you know, why are people we're writing all this code, but we're not shipping more software, and things like that.

really I think has a meaningful intersection with the economics of this, right? 'Cause it's like if we're if we're not turning into ten X developers, then is the is all this debt gonna wind up, you know, costing us more in the long run because it all just come crumbling in. So

Shimin (1:00:18)
Yeah, that's something I worry about as well.

Dan (1:00:20)
Yeah.

Shimin (1:00:20)
Okay, let's let's do four minutes then. Yeah, so

Dan (1:00:22)
Okay. Yeah, I think that's fair.

Shimin (1:00:24)
we're four minutes this week. All right, and of course, with the setting of the clock, that means we've come to the end of the show. thank you for joining us for our study session this week. If you like the show, if you learned something new, please share the show with a friend. You can also leave us a review on Apple Podcasts or Spotify. It helps people to discover the show and we really appreciate it.

If you segment idea, a question for us or a topic you want us to cover, shoot us an email at humans at adipod.ai. We'd love to hear from you. You can find the full show notes, transcripts, and everything else mentioned today at www.adieipod.ai. Thank you again for listening and we'll catch you next week. Bye.

</details>