Episode 47 · November 13, 2026

Claude Fable 5.2 Review, AI Accelerating AI Research & How Agent Memory Works

Claude Fable 5.2, Claude Mythos 5.2, Claude Fable 5.2 review, Fable 5.2 vs Opus 5.5, Claude Opus 5.5, Anthropic, FrontierCode 2.0, FrontierSWE v2, Remote Labor Index, Chartography, AutomationBench, AA-Briefcase, Artificial Analysis Intelligence Index, effort levels, Xhigh effort, LLM benchmarks, benchmaxxing, GPT-6 Astra, Gemini 4 Carbon, Gemini 4 Argon, Hamlet soliloquy benchmark, The Great Wave off Kanagawa, Hokusai, paper cutout animation, SVG animation, orders of effect, Claudisms, AI writing tics, Claude Code, settings.local.json, don't ask again, leaked secrets, secret scanning, pre-commit hooks, gitleaks, TruffleHog, Apiiro, Shmulik Cohen, Matan Giladi, Geoffrey Hinton, The Guardian, Three Mile Island, Chornobyl, Fermi 1, FDA for AI, AI regulation, Hugging Face, Slay the Spire 2, Godot, C#, decompiling, game mods, vibe coding, METR, Ajeya Cotra, Planned Obsolescence, AI R&D acceleration, recursive self-improvement, AECI, Epoch Capabilities Index, Responsible Scaling Policy, Anthropic S-1, Sam Altman, RAM Altman, Ram-Altman, Iain Kay, DRAM shortage, local inference, Cheeky Pint, John Collison, agent memory, The components of agent memory, Lance Martin, dreaming, sleep-time compute, Letta, memory notes, MEMORY.md, context injection, Claude Plays Pokemon, Pokémon Red, Sonnet 3.7, Fable 5, Anthropic IPO, ANT, IPO price range, Karen Kwok, Reuters Breakingviews, Greg Brockman, OpenAI IPO, Nvidia, Jensen Huang, Colette Kress, key-man risk, Goldman Sachs, tech sell-off, Two Minutes to Midnight, AI bubble, Shimin Zhang, Dan Lasky, Rahul Yadav, AI podcast

All three hosts are back. The news opens with a public service announcement: Apiiro found Claude Code’s “don’t ask again” approvals writing passwords and tokens into a settings file that developers then commit. Then Geoffrey Hinton tells the Guardian that AI’s Three Mile Island has already happened. Shimin reviews Claude Fable 5.2 against Anthropic’s own benchmark table and three house tests, including a Hamlet with unsettlingly human eyes and a paper-cutout Great Wave that Fable 5.2, GPT-6 Astra and Gemini each attempt. In Vibe and Tell, Dan has Opus 5.5 decompile Slay the Spire 2 and patch its touch bug. Post Processing reads Ajeya Cotra on METR’s report that AI is speeding up AI research at Anthropic, then the Ram-Altman script. Rahul’s deep dive covers Anthropic’s “The components of agent memory” and its Pokémon Red experiment. Two Minutes to Midnight covers Anthropic’s IPO price range, down from the figures floated when Episode 45 read the S-1, Greg Brockman on an OpenAI IPO, and the Nvidia CEO scandal, and the clock moves 15 seconds closer, to 2:45.

Takeaways

Resources Mentioned

Chapters

Transcript

Show full transcript

Shimin (00:00) Hello, and welcome back to Artificial Developer Intelligence, a weekly conversation show where three software developers chat about all things AI. My name is Shimin Zhang, and with me today are my co-hosts, Dan You’ll get his RAM stick from his code dead hand, Lasky. And Rahul, he’s been dreaming about Pokemon Red, Yadav.

Rahul, welcome back from your travels and the whatever you’ve been up to, uh conquering the world

Rahul Yadav (00:28) Thank you.

Shimin (00:29) out there.

Rahul Yadav (00:30) Yeah.

Dan (00:30) Yeah.

Rahul Yadav (00:30) I was just scaring people because it was Halloween and I decided to spend the whole month of October doing that instead of just doing it on a day. It was a good time. That’s how you make friends with your neighbors.

Dan (00:41) Whoa.

It’s scaring them to death.

Rahul Yadav (00:44) Mm.

Shimin (00:44) All right, what do we have on the show this week? we’re gonna start with the news, as always. We’re gonna talk about coding agents copying your secrets, as well as a AI godfather’s warning.

Dan (00:54) Hmm. Then we’re gonna do a model review of the newest hotness.

Shimin (00:59) Fable five point two.

Dan (01:00) No no no no no we aren’t you ruined it. You ruined it. Okay.

Shimin (01:05) and then uh Dan is going to do a vibe and tale about this sleigh the spiral? Hmm.

Dan (01:13) Then uh we’re gonna grudgingly do a post processing where we talk about AI dramatically accelerating AI research, which is pretty fun. And then this barely qualifies as post processing, but we’re still gonna do it, which is the tale of Ram Altman.

Shimin (01:27) then we’re gonna go into deep dive on components of agent memory.

Dan (01:32) And finally, we’re gonna close out, as per usual, with our two minutes to midnight segment where we talk about the finance side of AI through the lens of the atomic clock.

Shimin (01:42) All right, let’s get started.

Okay, first up in the news segment, we have a public service announcement really. This is coming from Apiiro Research, where they used AIT and probably also programmatically to uh scan public GitHub repositories for essentially secret files. And the headline number is they found 2539 real secrets in 1994 public.

GitHub repository. So these are secrets like database passwords, Claude keys, API tokens, SSH, root logins. And the the issue here is that if you have Claude code on permission mode, and Claude asks you to whether they can run a command line command with a secret key as a part of the command, and you press yes and don’t ask again, like I always do.

what Claude is gonna do

Dan (02:33) Yeah.

Shimin (02:34) is it’s gonna save it under dot Claude slash settings dot local.json. So everybody else can also uh run those commands without permission going forward. And because of that.

Dan (02:45) Well, if

you commit your dot claude and and push it, yeah. But most people do, so

Shimin (02:51) Right. and because of that, all those secrets got uploaded to public GitHub accounts. now this exploit is I guess a little unexpected because how frequently do you actually commit your dot Claude directory as a part of your repo?

Dan (03:06) Yeah, that’s true. I guess it depends on how you’ve got it structured. Um

But clearly folks do, ‘cause it’s happening, so

Shimin (03:13) Right. So the question here is I could see the argument both ways. I could see the argument that yes, you should, if you’re working on the repo, all that memory, all that learnings, all that directions should be committed so that any devs working on the repo have access to them. But again, some developers would like to have their own private settings private on their local machine. And I think we just haven’t had

A good system to determine what belongs in the public repo and what belongs individually on the developer’s machine. And that’s why we have these kind of issues.

Dan (03:42) Yeah, and it this isn’t really like a lot of things with AI, this isn’t new, right? I mean I I can think of at least two incidents that I’ve like personally been involved with. I didn’t do it, but like where someone accidentally committed a secret and we had to clean up after it. and like you know, yeah, I feel like it’s just good practice to have secret scanners in place on on your like

even if you just do a pre hook, it’s it’s so better than nothing, Pre commit hook.

Shimin (04:09) Yeah, and I’ve I’ve noticed

Rahul Yadav (04:09) The

Shimin (04:10) that Claude now actively searches for code that looks like secrets before it does a commit for you. so this may be a just like an older generation of Claude code issue as well.

Rahul Yadav (04:22) And I think for anyone who’s just using LLMs for code review, the new system one models might be useful too, because you can uh i instead go with a probability and then trigger more like high risk reviews if if something has some secrets that secret that they posted. So my guess is a lot of it could also be because people didn’t really then what you called up, people didn’t really have anything that was scanning it.

Shimin (04:49) Right. And of course no one’s reading the PRs anymore, right? if if they had

Rahul Yadav (04:52) Yeah.

Shimin (04:53) read it, you know, even skimmed it, they might be looking. Um so this issue,

Rahul Yadav (04:56) Yeah.

Dan (04:56) Yeah, yeah, yeah.

Shimin (04:59) they checked, they saw 104,000 copies of settings.local.json and all of them, something like a dozen has a real secret, and then uh another dozen in K history. So something like two percent of all repos with settings uh committed has this issue. So

Listeners, if you have either public or even private repos, right? rotate your keys, do a check. I’m sure Claude is really good at finding that for you. And protect yourself.

Dan (05:27) Protect yourself.

Shimin (05:29) speaking of protecting ourselves, Dan, the next article is brought to us by you.

Dan (05:35) A

grumpy old man. One grumpy old man to another. yeah. So yeah, I don’t think Geoffrey that grumpy. I’ll I’ll take all the grump in this. so yeah, Geoffrey Hinton did a interview with the Guardian, or at least was quoted by the Guardian a couple times. And he has a kind of a dire warning for us that, you know, frankly, we’ve talked about.

a a fair amount on the podcast, but it’s interesting to hear it coming from him, which is he thinks that within a decade and possibly sooner, um, humanity will know whether or not it can manage the risks that AI is bringing to the table or and whether or not we will get sort of that bimodal outcome that we’ve talked about before, right? It’s either radical abundance or we fail to manage the risks. And what was interesting is like his description of failing to manage it was

Not necessarily that paper clipping or something like that. It’s that we would have a huge gap between rich or poor, large cul cultural problems, massive unemployment, or societal rifts that are unreparable. So it’s intro kind of an interesting take on it. but there’s some other pretty kind of fun bits in here. He he he takes a couple digs specifically at leaders in the space.

So

Shimin (06:50) Mm.

Dan (06:50) he he said that he explicitly doesn’t trust Mark Zuckerberg, Elon Musk, or Sam Altman. And he said they understand the risks and they don’t give a shit. And that’s a direct quote. They’d rather

Rahul Yadav (07:02) Yeah.

Dan (07:03) have the money. That’s my belief. yeah. So Geoffrey Hinton coming coming in hot.

Shimin (07:09) Yeah, and of course Geoffrey Hinton was a early researcher. He like created one of the earlier versions of Neural Network. that’s why he is considered a godfather of the whole thing. But I also just want to say just because he helped create an earlier generation of the technology does not necessarily mean he has better insights than anyone else, right? Um as

Dan (07:30) No, it’s just

Shimin (07:31) much as I agree with him on on on all that, yeah.

Dan (07:33) just,

mildly famous and is using it

Shimin (07:34) Yeah.

Dan (07:35) as a platform. So why not? but the other part he said that I do kind of agree with as well is that he said that he thinks within the next like year or so we’re gonna have what he called like a three mile island event. So for for those of you that don’t know the history of like nuke stuff, Three Mile Island was actually like a pretty mild

nuclear incident that happened in the the 70s. And but people treated it as like this big wake up call about like regulation and stuff like that. So it was kind of like when the industry got revitalized. As someone that’s like studied that area a lot, it’s funny to me because Three Mile Island gets all the like public attention, but actually almost the entire city of Detroit was destroyed in a far more serious incident that no one knows about. So it’s like

You know, whatever. But

Shimin (08:25) Was that also nuclear?

Dan (08:27) yeah, there was the Enrico Fermi test reactor um that

Shimin (08:32) Yeah.

Dan (08:32) was like I think a little bit south of Detroit proper, and it they had a a partial meltdown and it was almost super serious. Where like they had these zirconium cones that’s random things I know. It’s like zirconium cones that were like part of the cooling system, and one of them got crumpled, and so that prevented like

cooling from happening and

Shimin (08:52) Mm.

Dan (08:53) yeah, it was like a whole thing. Like we almost lost Detroit.

Shimin (08:56) Wow.

Dan (08:57) but you know, everyone talks about through mom. Anyway.

Shimin (09:00) Then with a hot take, three mile island,

Dan (09:01) Whoa.

Shimin (09:02) not really something we should worry about. Okay. Um yeah, I I

Dan (09:07) I’m just saying, of all

the incidents, it’s funny to me that that one got the attention. It’d be like equivalent of I don’t know, if like the hugging face incident was was everything everyone talked about. Meanwhile like a swarm of agents had like dismantled Wikipedia and nobody noticed or something like that, you know? Yeah.

Rahul Yadav (09:24) Sol solar winds or something. Yeah.

Dan (09:27) Like

But yeah, so anyway he’s he’s also predicting basically we’re gonna see some incident like that, that’ll be a warning shot and he’s interested to see what world leaders r will do when that happens. will they be smart

Shimin (09:37) Right. oh

Dan (09:38) or not?

Shimin (09:39) What’s interesting is of course, like the public is already fairly against AI in data centers, at least in the United States. so it feels like the propaganda, the kind of social feeling part of AI has already reached that Chernobyl three mile island level. it’s just that the governmental leaderships who have not wanted to control it or uh impose policy upon it um in any

Dan (10:04) Uh-huh.

Shimin (10:04) way. And

And of course, Hinton also said he looks forward to having a FDA like agency that both is able to pace the frontier but also have an idea of of how dangerous exactly these AI models are. Um we’ll see if it happens.

Dan (10:18) Yeah. Which is which

is kinda necessary in my opinion because because of the conflict of interest we talked about previously, it’s like I think the safety researchers are great, but they’re also getting employed by safety, you know. So

Rahul Yadav (10:32) Yeah, self

regulation only goes so far.

Dan (10:35) Yeah.

Shimin (10:36) Okay, shall we move on to model talk?

Dan (10:38) If

we have to. Do we have to?

Shimin (10:40) I don’t think we have to. Well, this week, Anthropic um released Claude Fable five two and Mythos Five Two. their latest iteration of the Fable and Mythos level models. of course, Fable is for general public access, and Mythos are only available to certain security and bio partners of anthropic.

Guys, I’m gonna just start off saying, you know, sometimes we think the show has a bit of an anthropic bias, right? No one’s ever accused of that on purpose. But

Dan (11:11) Yeah.

Shimin (11:12) like if we have an anthropic bias, uh the following sentence I’m gonna say

Dan (11:16) They write really good blog posts, okay?

Shimin (11:18) it is not gonna help with that. because Fable 5.2 is my favorite new AI model. It is

Really freaking good. And I’m gonna

Dan (11:29) How did how could you afford it?

Shimin (11:31) back this up. Well, with my subscription, of course. Um they do. Yeah.

Dan (11:34) wait it they allowed it for subscription now? I guess I missed the memo on that one. Or is it limited

time only?

Rahul Yadav (11:40) Just today, I think. Richard? I just saw it today. okay.

Shimin (11:44) Yesterday and today at least. Yeah, I don’t know how long it’s

gonna be available for all the Max and Ultra quote unquote users for, but I had a chance to play around with it over the last two days or so. And well, first let’s do some model rundown. tests Fable 5.2 and Mythos 5.2 are slightly cheaper than Fable 5.1.

5.2 is at $8 per million input tokens and 40 per million output, uh, whereas 5.1 was $10 per million input and $50 per million output. quite pricey. according to Anthropic, the 5.2 models are also to generate output 30% faster than than 5.1.

When it comes to the benchmarks that Anthropic listed, we can kind of go through this a little side by side. One of the things that always puzzle me is Opus five point five seems to do a much better job than Fable five point one on a lot of the benchmarks. And you can probably see it on this chart right here. where on the Frontier Code two point benchmark, Opus five five able was able to do

Be 45.5% successful, while Fable 5.1 is only 32.4% successful on X height effort. By the way, now these models are being run on max effort. Now even Anthropic itself can afford to run these benchmarks at max level.

Rahul Yadav (13:08) Ha ha ha. uh

Dan (13:08) Ha ha.

Shimin (13:09) and Fable 5.2, uh uh

Dan (13:11) Or it would just never end

‘cause it thinks for twenty seven days.

Shimin (13:15) That could also be possible. Um mm-hmm. Yep.

Rahul Yadav (13:17) So Shimin, the

some of them say X high effort, but then Opus five five doesn’t say what the effort level was. Gemini four says it was run at medium effort. Which seems like to me you just run it at a lower effort to make the numbers look a certain way.

Dan (13:36) Juice a little bit, yeah.

Shimin (13:35) That is true. Yeah.

Rahul Yadav (13:38) Yeah, either like go all medium or all lower, like pick the highest on everything, but it it seems like

The the missing little text there is also saying something.

Shimin (13:47) Agreed and even uh not taking that into account, right? Fable 5.2 only scored 49.1%. So it’s like a less than four percent jump on the Frontier Code 2.0 benchmark between Opus 5.5 and Fable 5.2. That is not a huge jump. That is probably not worth the extra cost that you will have to pay for the Fable 5.2 model, right? So

So that’s one of the first things I think we should talk about when we talk about Fable 5.2, that it is not a coding specific model. If you look at some of the coding specific benchmarks, you know, Frontier Co, we already mentioned Frontier Suite, even automation bench, cyber scenario bench. In all these cases, Fable five two does not have a 10% lead over Opus.

It’s actually a moderate size incremental improvement. I’m not sure it’s warrants the actual cost. However, on the more general purpose tasks, like automation bench for business workflow, knowledge workflow, which is artificial analysis briefcase one point one, uh, we d we do see a much higher jump. although here too, it’s interesting that Opus five five actually has a higher ELO than Fable five point two, which I’ll

in that sense, it’s nice that Anthropic maybe isn’t gaming the benchmark entirely, right? They are showing certain benchmarks where f Fable isn’t doing as well as Opus. but for the remote labor index benchmark, Fable five point two has a twenty percent jump over Opus. I think like that is where. Yeah. Fable is only

Rahul Yadav (15:20) Opus didn’t have high effort, man. uh Opus needs another chance

Dan (15:22) Ha ha ha.

Rahul Yadav (15:25) to prove itself is all I’m saying.

Shimin (15:28) Yeah,

Fable is only at high effort itself as well. And lastly, we

Rahul Yadav (15:32) Yeah.

Shimin (15:33) see a big jump when it comes to visual chart recognition at the chartography benchmark where there’s uh a 10 point jump as opposed to Opus Five Five. when it comes to the actual task-based workflows that this blog post showcased.

I found the coolest one for me was the robotic arms demo where it showed Fable using a robot arm to uh do a a series of tasks that all seem to be rather strong. possibly because it has much better visual recognition. Okay.

All that said, we looked at the artificial analysis intelligence benchmark five last week. And here we see that Fable five two has about a six point jump over Opus five five. If you if that benchmark means anything to you, right? Like it’s it’s a significant step up, but it probably doesn’t warrant, you know, like an upgrade to Fable six per se.

So let’s look at a couple of our ADI pod custom benchmarks. And this is where really when I say it is my new favorite model, this is where it comes in.

Dan (16:41) I can’t wait.

Shimin (16:42) When

it comes to the orders of effect of AI and its impact on society benchmark, the first thing that really jumped at me is the output of Fable 5.2 are much more succinct than Opus 5.5. these same

Rahul Yadav (16:59) Ha ha ha.

Shimin (17:00) benchmark, much shorter paragraphs, similar content, and it ran fairly fast. So if

If you hate Claudism and the AI slop Yeah, this

Dan (17:09) That just means they gamed your benchmark already.

Rahul Yadav (17:13) Yeah.

Shimin (17:13) is already in the training data. Yes. Um

Rahul Yadav (17:16) I

so I have started and I need to put this in my just the personal system prompt to Claude. I have started telling it to be less verbose in its responses and that’s why I left ‘cause like that’s the one of the things that you found great about Fable Five too is it doesn’t drown you in a wall of text and yeah.

Dan (17:36) Although

even as you were just scrolling down this artifact, for those of you that are listening, I immediately just without even trying or wanting to spotted at least two claudisms that made me cry a little bit.

Shimin (17:50) Well the

Claude claudism is significantly less, perhaps. there are no em dashes on the page that I can see.

Dan (17:56) That was never one that I cared

about or bothered me. It it was the

Rahul Yadav (18:00) It it’s more just

h how the the the sentences, how it like the

Dan (18:04) Yeah, can you s

can you scroll back up?

Sorry, he’s he’s screen sharing for those of you on the pod. He goes, where is the the good one? Three facts do most of the work. That’s it. I don’t have to say anything else. Yep. Okay, marketing man.

Rahul Yadav (18:12) Three f I I was just gonna point that out. Three three facts do most of the work.

Shimin (18:14) That that’s fair. That’s fair.

I didn’t see any low bearing arguments in this in this entire new version, so that was nice. Sentence structure, yes.

Rahul Yadav (18:25) uh It’s all in the substrip.

Dan (18:26) Mm, but there’s definitely some facts doing some work. So yeah. Which would’ve been load bearing.

Shimin (18:32) Yes.

Dan (18:34) Just dial down the punch.

Shimin (18:35) Yeah, so that’s you know point A of what I like about the latest fable, that it is concise. And as for the quality of the prediction, it’s good. It definitely didn’t get lazy and start talking about, you know, what does it mean to be human? It gave out concrete effects, even fourth order and above, with like a sufficient caveats about how these are things that you know really aren’t possible to predict.

but next let’s move on to my favorite benchmarks, the uh

Rahul Yadav (19:05) yeah.

Shimin (19:06) Shakespeare soliloquy.

Shimin (19:07) All right. So here we have Astra’s version of the Hamlet soliloquy. We saw this a couple of weeks ago. Um

We see that it’s got pretty good, nice animation, but janky hands basically, right?

Rahul Yadav (19:19) Yeah.

Shimin (19:20) And then recall last time we have the Gemini 4 version of the same soliloquy, uh

Rahul Yadav (19:26) Yeah.

Shimin (19:27) where there’s some weird candle shadow effects. And overall, it’s got eyes that follows my cursor, which is pretty cool. And what’s a cool haircut. And

Rahul Yadav (19:36) And he’s got a cool haircut too. Like this uh yeah.

Shimin (19:40) both these artifact uses the native browser web API for its soliloquy.

And now I’m gonna show the Fable 5-2

Rahul Yadav (19:49) Opa, né?

Shimin (19:49) version of

Dan (19:50) Wow.

Shimin (19:51) the same soliloquy. what do you guys what do you guys notice about the face, first of all? Like I

Dan (19:57) The

eyes are extremely spooky looking. Like

Rahul Yadav (20:01) Yeah.

Dan (20:02) they’re like very human looking eyes.

Rahul Yadav (20:04) Th th

this guy’s I’m I don’t spend much time on the

Dan (20:07) Oh he got mad.

Rahul Yadav (20:08) internet, but I believe

Shimin (20:08) Yeah.

Rahul Yadav (20:09) he’s looks maxing over there. I think so.

Shimin (20:14) It’s it’s a Luxmax version of Hamlet. Um yeah. Yeah.

Rahul Yadav (20:17) Yeah.

Shimin (20:18) it’s got it’s got

Dan (20:18) There’s some like real

emotional expression happening on that face. That’s

Rahul Yadav (20:23) Uh-huh.

Dan (20:23) pretty cool.

Shimin (20:24) Yeah, it’s rather subtle, I wanna say. And of course, listeners, we’re gonna include this in the show notes as uh so you can check it out yourself with the comparisons.

Rahul Yadav (20:33) A and

it’s it’s zooming in on his face, isn’t it? Yeah.

Shimin (20:37) Mm-hmm. It it does

Dan (20:38) Slightly

Shimin (20:39) a camera zoom as the soliloquy reaches its peak.

Rahul Yadav (20:41) Okay.

Shimin (20:43) Um

Rahul Yadav (20:43) I hadn’t noticed that with the other ones.

Shimin (20:45) Yeah and

It also uses a small 82 million parameter model to do the audio. It is no longer using the web API for its audio. So the audio actually also sounds really human-like. uh

Dan (20:58) It chose

to do that, from its own

Shimin (21:01) Yep, it just decided to with the same prompt. the downside is this did take up more than one full context window worth of tokens. But I think the output is indisputable. Like it this it made really good assumptions and it worked on it for a long time. I think it took maybe 20 minutes to get this whole thing together. and the and the you know, for for

Prompt where I didn’t prompt anything about art style, I think it did a pretty good job of giving something that almost feels a little uncanny bally. Um, that

Dan (21:34) Yeah.

Rahul Yadav (21:35) Yeah.

Shimin (21:35) wasn’t an issue with the first two, right? So okay.

Rahul Yadav (21:39) What is it’s fascinating that you said it took a long time and it took twenty minutes to do it, which i if we just go back a couple of years ago, not even that long, you would have to pay someone to take actually a very long time and then

Shimin (21:57) Mm-hmm.

Rahul Yadav (21:58) a lot of back and forth to then get to this and so

Just like the pace of progress has been insane. We’re twenty minutes in and this output still feels like a a a long time. Um but it it’s just insane to see the world we’re living in.

Shimin (22:12) Yeah, it’s like the Lewis CK uh airplane joke, right? damn it, I want this lifelike Shakespeare’s soliloquy and I want it now. Uh

Rahul Yadav (22:20) Ha ha ha.

Dan (22:20) Sorry.

Shimin (22:21) And I’ve reached this point where I couldn’t spot any real serious, obvious flaws with the model, other than it looks a little looks maxed. so I’m a little worried that the benchmark got benchmaxed already uh

Rahul Yadav (22:33) Yeah.

Shimin (22:34) and saturated. So I was looking for something that is a little more difficult. And here I decided to ask again the frontier models to create a cutout animation from the

The Great Wave of

Rahul Yadav (22:48) Interesting.

Shimin (22:49) uh Kanagawa, you know, that very famous Blockwood painting, and do

Dan (22:53) Mm-hmm.

Shimin (22:54) a 3D animated rendering based on a public domain image of the wave. And so let’s take a look at the Gemini version. Again, Gemini is very into the artifact working with my cursor, which which I

Dan (23:07) Yeah.

Shimin (23:07) thought was kind of cool. well, how did you play it?

Dan (23:09) Yeah, it’s got some of

Shimin (23:10) And the prompt is to create essentially a cutout animation of the wave, showing the the the wave itself forming, cresting, and then falling. Okay. Kind of impressive. Yeah, that’s all right.

Dan (23:19) Mm-hmm.

Rahul Yadav (23:19) Yeah, is that right. Yeah, the

the the the whatever the splash just looks very odd. Look at that. It’s

Dan (23:28) Yeah,

as someone that did professional animation for a early chunk of my career, I find this a little bit offensive, but

Shimin (23:36) Okay, Dan is offended. So now the test is whether the Astra 6 version would also offend Dan.

Dan (23:43) Okay.

Shimin (23:44) All right, now we’re gonna we have on screen the Astra VI version where the wave itself got cut out and was rotated to show its its rising. And then it threw a bunch of other cutouts

Rahul Yadav (23:55) Yeah.

Shimin (23:56) as as the wave formed.

Dan (23:57) Yeah, they’re not gonna win any timing awards for this, but the the sequence

Shimin (24:02) Mm-hmm.

Dan (24:02) of the animation’s better.

Shimin (24:04) Yeah. And I have

Rahul Yadav (24:04) Yeah. The last one

is a bit funny. The top of the wave kinda splits apart right there and then you just see that.

Dan (24:11) Yeah. Yeah. uh

Rahul Yadav (24:16) Yeah.

Shimin (24:17) Yeah. Um and also the if you look at the boats themselves, the boats are not continuous. The bolts are actually

Rahul Yadav (24:23) Yeah.

Shimin (24:23) rendered entirely separately. Okay. So pretty good. Better than the Gemini version, but still no cigar. And now let’s take a look at the Fable five point two version of it.

And here too, we see the wave broke apart in a unnatural manner with some splashing.

Dan (24:38) how do I describe that?

Imagine that the wave is someone actually literally waving at you. And that’s what we got.

Rahul Yadav (24:45) Yeah.

Dan (24:47) The the tip of the foam of the wave

Rahul Yadav (24:48) They this way is less of

Dan (24:50) just comes out and goes like, Hey there!

Rahul Yadav (24:52) Yeah.

It i it’s like a ta-da and then it goes back down the same way

Dan (24:56) Yeah.

Rahul Yadav (24:57) it came out and doesn’t actually wave.

Shimin (25:01) So which

Dan (25:01) Mm.

Shimin (25:01) one was your guys’ favorite? We have three Frontier

Rahul Yadav (25:03) Oh.

Shimin (25:04) models on the same prompt.

Rahul Yadav (25:05) I yeah, I think OpenAI one was the I except for the little piece at the end is probably the the one I like the most.

Shimin (25:12) Okay. I I don’t agree with you two. but

Dan (25:16) You like this one the best?

Shimin (25:17) I like this one the best because I like that every single part was actually cut out from the image itself. The boats are not fading

Rahul Yadav (25:23) That is fair.

Shimin (25:24) in and out. And I also find just like the background wave before the the wave like was

Rahul Yadav (25:30) So

Shimin (25:31) uh rising to look better. Yeah.

Rahul Yadav (25:33) Yeah.

The initial part I like a lot.

Dan (25:34) Except for the part where

i if if we if I could ignore this is kinda like so I like I said, I worked in animation for a little bit. There was always a trick that we would well not always, but there was a trick that we learned by the end of doing this professionally, which was you always do something a little bit wrong on purpose so that the customer has something to

Shimin (25:52) Mm-hmm. uh

Dan (25:55) to look at and point out. yeah, you really sc oh I’m so sorry, we really messed it up. It it did that with the waving wave.

I if I could if I could just fix that, it would be pretty great, but

Shimin (26:07) Yeah. and of course these are um this is a coding task, right? We’re essentially asking you it’s a coding and a visual recognition task. the the proper way to deal with would be via a actual vision model. So lastly, ask Gemini to actually use its visual to generate a folk full on video, and that’s gonna be the best version of the bunch.

So yeah, I think all things considered, I find Fable five two to be to be my favorite current generation frontier model. but probably because I just have spent more time with it more than Astra and Argonne. so

Rahul Yadav (26:42) Carbon soon.

Shimin (26:43) Carbon. Yes, I have no access to carbon, of course. So that’s that’s the other thing.

Rahul Yadav (26:46) I just saw the

I saw the disabled thing saying you can’t have it just today. So

Shimin (26:52) Ha ha ha.

Rahul Yadav (26:52) h hopefully it’ll be enabled uh in the next few days.

Shimin (26:56) Yeah. Google just announces and then keeps it a secret for a couple of weeks. And by the time it’s released

Rahul Yadav (27:02) Yeah.

Shimin (27:02) for G A, nobody nobody remembers it anymore. Um

Rahul Yadav (27:05) Yeah.

It’s like we have these brand new seats on the plane now. Economy’s back there, by the way.

Shimin (27:10) That’s that.

Rahul Yadav (27:11) If you could please

Dan (27:11) Yeah.

Rahul Yadav (27:12) move along.

Dan (27:14) Yeah.

And while you’re busy looking at them, those new seats will

Rahul Yadav (27:19) Yeah.

Dan (27:19) Hack Wikipedia. I don’t know why I’m so

Rahul Yadav (27:21) Yeah.

Dan (27:22) obsessed with Wikipedia angle today, but

Rahul Yadav (27:24) Ha ha.

Shimin (27:25) okay, so that was my model review of Fable five point two. I really liked it. Dan and Rahul were less impressed, but listeners, check it out.

Dan (27:33) Well, I’ll have to use it and then I might be more impressed for other things.

Rahul Yadav (27:36) It’s almost

the holiday Shimin so we’re waiting for the next major version bump of all these things. Uh

Shimin (27:42) Uh yes.

Rahul Yadav (27:44) I I think the uh the minor bumps are to build hype for the IPO next week is what I would guess at.

Shimin (27:52) Yeah, and of course Chinese New Year’s is coming, so I’m looking forward to the new open weight models.

Rahul Yadav (27:55) yeah.

Dan (27:56) But what I will say that I did I did have some experience with is Opus five five and something I did recently with it. So transitioning to my stupid little vibe and tell. Kind of a hard act to follow, a whole bunch of like cool wave animations with just me talking about something dumb, but here we are.

Shimin (28:16) You can share your screen if you want to.

Dan (28:21) yeah.

Shimin (28:20) And we could be a gaming. We attack gaming.

Dan (28:23) god.

Rahul Yadav (28:23) A blank screen.

Dan (28:24) So

I guess maybe long, longtime listeners of the podcast know this about me that I have a horrific addiction to the Slay the Spire series of of card battlers. Don’t know why. It’s really out actually kinda out of character for me, gaming wise, to play a card battler, but here we are. And I I really liked playing Slay the Spire one on my iPad. I was just like enjoyed something about like dragging the cards and whatever.

Slate

the Spire 2 uh doesn’t run, I mean isn’t out yet for iPad. So this is a little embarrassing, but I actually bought a Windows tablet so I could play it with Touch. And it basically doesn’t work with Touch at all. And I’ve just kind of been like hoping someone would like fix it or come out with a mod or something, and no one’s done it. And I’m just like, you know what? I haven’t even like attempted to look at this myself. So

I was like, hey Claude, I play this game. It’s written in the Godot engine. It’s just open source. So maybe you could figure out something about it. Apparently, it’s C sharp and it’s actually like super easy to like decompile C sharp. So Claude goes, install these two tools for me. And I’m like, okay, I did. It decompiles the entire game, reads through all the source code of the game, decompiled, finds the bug, fixes it, and then

packages up into a mod that I can just install that basically patches the the touch bug. So it turned out they were basically using the entire viewpoint ports coordinates when touch actions occur instead of where the actual cursor is. so once they once it patched that it’s just like a coordinate bug. So once they patched it

I’m able to happily drag cards around all day on the screen. It’s fantastic.

Shimin (30:07) Did you commit your patch to GitHub and did you leave your dot claude directory in there?

Dan (30:11) I didn’t. I you know, I I did genuinely debate like if I should upload it to like Steam workshop or something because like there’s

Shimin (30:19) Right. Yeah.

Dan (30:19) a couple touch mods out there that claim to do this, but neither of them really worked for me. And one of them’s like a big full game overhaul. It does a bunch of other crap, and I’m like, I don’t want that. I just want touch to work, you know? So

Shimin (30:31) And it just worked, right? y did you look at the code base, the patch code at all? You just tried it? Nice.

Dan (30:35) I none of it. Yep. I just

I just ran it with mods enabled and it worked and I’m like, cool. And I copied my saves over from the non modded and that was it.

Shimin (30:48) Sometimes all you need to do is yolo mode. Yeah.

Dan (30:50) Yeah, I know. It was maybe twenty minutes tops to having a a working game, which is pretty cool. So

Shimin (30:57) It it goes back to what we just need to imagine to be able to do more. if you don’t like a game, just just patch it yourself. it’s it’s all in the open. But also I see why um Anthropic had instructed its agents to not reverse engineer games because apparently it’s very good at it. Yeah.

Dan (31:11) Yeah. I mean, you know,

it did decompile the entire source, so it wouldn’t have been a far stretch to be like, make this game entire game but with cows instead of uh knives or something. Yeah.

Shimin (31:20) Yeah. Add a new character, why not? Yeah.

All right. speaking of yoloing and just have Claude do everything for you, I think we’re gonna move on to post

processing, where our first post this week is brought to us by Rahul

Rahul Yadav (31:34) so Ajeya Cotra at Planned Obsolescence is summarizing a report that the METR people did about internal research at Antrophic and it potentially accelerating and I’ll spend a decent amount of time on that potentially piece. So I looked at the original report, it starts with this giant disclaimer about a lot of the things that

We wanted to talk about are redacted because um Entropic described the redactions as aimed at removing confidential information that could accelerate other AI developers. So a lot of the material things that they’re you know basing this off of is all redacted. So the best thing we’re already looking at is someone’s interpretation of the data they looked at. and then

Even that interpretation they’re saying sometimes had different you know conclusions that came out of it. And a lot of it is also based on talking to people and asking them if they feel like the the you know, they’re the amount of work they’re able to do and all these things have increased over time or not, have sped up over time. So I say this all at the front because take this as a

big mountain of salt and with the company planning to go public next week the i it’s relevant to start there. Um so yeah. Yeah.

Shimin (32:56) Yeah, and of Ajaya does work for METR right?

Rahul Yadav (33:00) And their colleagues did the research and so they they’re just summarizing the work that that their colleagues did weren’t directly involved.

So Anthropic, one of the things that they had in their S1 was that they may they they have this responsible scaling policy and then and that they may imminently meet that threshold that they’ve define um for the dramatically accelerated progress due to due to AI. And a bunch of things that they talk about are by the time they get to this, there should be all sorts of measures in place to be able to capture runaway AI.

Now the report says uh we might be within a year of AI that can just you know keep uh improving itself over and over again. Um

Dan (33:44) Like recursive self improvement or whatever.

Rahul Yadav (33:46) yeah, recursive self-improvement. but at the same time, as we all know, there’s no set safeguards that are in place other than the the self-policing that we were talking about earlier. Um

And the way they measure this is so Anthropic has its internal version of the Epoch Capabilities Index. And what what that index is is it takes all these different benchmarks across all all the different models and it creates one index out of it. And then Enthropic has their internal version of that that they use to measure the Claude Haiku versus Sonnet versus Opus and all these models that they work on. Um

Shimin (34:24) Mm-hmm.

Rahul Yadav (34:24) one of the

big takeaway from the report was that between October of last year and October of this year, the capability progress has doubled i in that time. And that doubling implies that uh and when you play that that down on how it impacts the researchers everyday work, it comes out to the researchers is would be able to do ten days of work in one day day’s time because of this

this doubling of progress and and how it multiplies in in their workflow. which is a huge improvement. And if that improvement feeds into more improvements, then it can lead to that recursive self improvement. There’s obviously like, you know, you could have some hard limits that you might run into in like how much you can improve something and all the all sorts of things like that. but according to what they can share, they don’t really see

a a limit to it so far.

Shimin (35:21) Yeah, Rahul, while you were away, Dan and I covered an article from Anthropic about how they work. And one of the things they noted was that their research department had nineteen agent hours per one human hours worked, and that’s like the highest percentage

Rahul Yadav (35:36) Yeah.

Shimin (35:38) of all of their departments. So this

Rahul Yadav (35:39) Yeah.

Shimin (35:40) seems to be aligned with that reading.

Rahul Yadav (35:41) Yeah. Nice. and so the the kind of the big takeaway is we’re getting to Enthropic is getting to what they say uh would be the RSI threshold and without the safeguards that they say that should be in place, those these would be things like protecting against state level

model way theft and all sorts of stuff like that. Um which will not specifically that, but something related to that we’ll touch on later. And then the the whole industry in general is unprepared. Um and there’s all these requirements around, you know, we should have monitoring. We were earlier mentioning how there’s a call for maybe an FDA for m these models and everything, alignment training.

all of that is close to non-existent today. And the the guess that everybody’s making is within the next year we would have this. So it’s pretty concerning if it’s true. At the same time, there’s so much unredacted data that no one can just do go and look at the actual data themselves and um come to the same conclusion. So take it with the that is.

Dan (36:52) A grain of redaction salt. Partially redacted salt.

Rahul Yadav (36:54) Yeah I redacted some

Dan (36:58) We can infer

Shimin (36:58) Um

Dan (36:58) that it’s shaped like a cube, but half of it’s painted black, so

Shimin (37:02) It it does feel like it generally agrees all with like how I’ve been feeling about all the various capability improvements that the models have been releasing. Like the the speed of new model release and that capability improvement has been feeling like it’s been accelerating, right? And and the agents are working longer and longer unattended. And that’s where alignment really counts.

one of the things that the blog post talks about is when it comes to research, you take away a lot of the low-hanging fruits first. And then as time goes on, scientific progress sometimes slows down because all the low-hanging fruits are are picked. And I wonder if we are also in that low-hanging fruit of AI accelerated scientific discovery stage. Like we are probably

Rahul Yadav (37:47) Yeah.

Shimin (37:47) also somewhere in that S curve, that second derivative S curve.

We just don’t know we where we are on that S curve and if yeah, how much room there is to go on that.

It we talked about recursive self improvement like months ago. It’s when we first talked about it, and now

Dan (38:02) Uh-huh.

Shimin (38:03) it seems like it’s upon us. Fascinating.

Rahul Yadav (38:06) The the skeptic in me always is like looks at the timing and i if if the

Dan (38:12) Yeah.

Rahul Yadav (38:13) if this if you have fable and all these models coming out, and you know like it could also be that I’m looking for something that’s not there. Um you have the new models coming out and you had the report saying we would be we’re very close to AI building itself within the next year.

And the week after that you’re saying we’re gonna go IPO. the you’re trying to tell a certain story. I I would be very surprised if some of the smartest marketing people are not there to like tell this story there as the company ramps up to raise billions of dollars at the at trillion dollar valuation. So

Dan (38:49) Mm-hmm.

Rahul Yadav (38:50) um th there’s that part of me that’s

a

Shimin (38:52) Yeah.

Good good cynical take. I can always kill on you for that. Dan, do you have anything to add to this? If not, we’ll move on to the next one.

Dan (38:59) Uh I do, but it’s redacted, so

Shimin (39:02) Okay.

Rahul Yadav (39:02) Yeah.

Shimin (39:03) Well.

Dan (39:03) We’re just gonna have to roll with that.

Shimin (39:05) Alright. Dan’s response is redacted. Yeah.

Rahul Yadav (39:05) The competitors might steal whatever Dan has to say, so he’s

just not gonna say it.

Shimin (39:11) Sounds good. on the much lighter note, let’s move on to a dance post processing for the week.

Dan (39:17) I get all the trolley ones this week. I guess it’s in character, but yeah.

Shimin (39:21) quickly.

Dan (39:21) Sam Altman was interviewed on a podcast, I think it was last week, along with I think it was one of the co-founders of Stripe that was on the podcast. And

Rahul Yadav (39:30) Yeah, cheeky pine, John Collison.

Shimin (39:30) Mm-hmm.

Dan (39:32) yeah. And he apparently uh said something along the lines of well, it’s appalling how consumers are allowed to have RAM because it’s a much better use of it to be in data centers.

concentrated in a data center so we can use it for inference essentially or compute. And uh boy, you know, like what a what an amazing

Rahul Yadav (39:53) Ha ha ha.

Dan (39:55) thing to say. Like you don’t get RAM. Sorry. So an enterprising individual named is it Lane or Ian K has created an open source batch script called Ram Altman.

It only runs on Linux. Sorry, Max U Mac users. but essentially what it does is after a little bit of configuration, he intentionally didn’t make it work out of the box. So you have to, you know, hook it up with a mail server and stuff like that. It will every time you have some idle RAM on your system, it will send an email to Sam Altman with various sort of cheeky messages. one of them go reads as, Hi Sam.

Please don’t be alarmed, but there is one point six gigs of RAM on my desk that isn’t in a data center. That’s fifty-six point two percent of what’s installed here. And as far as I can tell, it has no plans today. Hoarding responsibly, a concerned RAM owner. Yeah. But if you’d like to you know, just try it out without actually emailing Sam and you’re having me running Linux, you can just run the dry run and it’ll send you the message only, which is

Shimin (40:58) Ha ha.

Dan (40:59) fun. they they also note in the post, and it is pretty funny that the the project page is pretty good. So the first question: does this help with the DRAM shortage? No. does it work on Mac OS? No. It reads proc slash mem info, but on the other hand, your RAM is probably soldered on a Mac, which makes you the hardest target in the global allocation program.

Rahul Yadav (41:23) Yeah.

Shimin (41:24) yeah, why does this matter? I mean, Sam Altman’s comment was of course extremely tone deaf, right? Although

Dan (41:30) Yeah.

Shimin (41:31) probably technically true, right? RAM RAM sitting in a data center gets utilization of near a hundred percent is a better overall allocation of resources than everyone having empty RAM sitting their computers.

Dan (41:43) Well, the part that was I found personally like it’s very offended was that he was saying that it’s explicitly stupid to run personal inference. And I couldn’t disagree more with that because, like, quite frankly, there’s a reason why I use anthropic models over OpenAI, and that’s because they run ads.

Shimin (42:05) Mm-hmm.

Dan (42:07) It’s

that simple because I think that as soon as you involve advertising in something, your motives can easily become suspect, right? Look at what happened to Google. Like, you know, do you know evil? And then here we are. Like, does that still true today? Probably not. Like, um,

Rahul Yadav (42:22) Yeah.

Dan (42:24) so I’m not saying they’re doing anything like, you know, outright nefarious right now, but like, would I want to give them their data and then have them change their mind later? No. So

Shimin (42:31) Mm-hmm.

Dan (42:32) That’s why I want my own RAM, Sam. That’s why.

Shimin (42:35) and I find this to be really it’s such a great showcase of how the frontier labs or maybe one particular frontier labs feel about resource distribution, right? Like if that is how Sam Altman feels about RAM in your computers, how does he feel yeah, about capital? About yeah.

Dan (42:52) Yeah, how does he feel about weights and yeah or money? Yeah.

Shimin (42:59) About political power, about all the other stuff. Um

Rahul Yadav (43:02) The so I I asked Gemini to give its spin on similar, you know, like what are other things that we’re hoarding and would be better allocated? And first one is fresh potable water.

Dan (43:16) you’re definitely hoarding that, yeah.

Shimin (43:17) Ha ha ha. uh

Rahul Yadav (43:20) We we 90%

N people have high drinking water at enormous expense, only for ninety percent of it to be used for by households to flush toilets, irrigate decorative turf

Shimin (43:29) Mm.

Rahul Yadav (43:29) lawns, wash driveways and r and run half empty washing machines.

Dan (43:34) I I d I

agree with that for its own reasons, but I don’t think the corollary there is that all that fresh potable water should be used for data centers either.

Rahul Yadav (43:43) Exactly.

And yeah, it had like baseload electricity, arable LAN and caloric production, direct thermal thermal natural gas.

Shimin (43:54) Color production.

Rahul Yadav (43:55) Yeah, so it it it had a lot of analogies to that.

Dan (43:59) Yeah. Those greedy humans hoarding the things they need to live.

Rahul Yadav (44:02) Ha ha.

Shimin (44:03) all right.

Dan (44:03) Should we amend the

constitution to include Ram as a fundamental human right?

Rahul Yadav (44:06) Ha ha ha.

Shimin (44:09) w yeah. Wait wait till wait till AIs can also make donations. okay. On to our deep dive

Rahul Yadav (44:14) Why

Shimin (44:15) of the week. Rahu, what do you have for us?

Rahul Yadav (44:18) So staying as a anthropic not sponsored but almost

Dan (44:22) Fanboy.

Rahul Yadav (44:24) sponsored podcast over here. Um the yeah, the unofficial Yeah.

Dan (44:28) Or the unofficial podcast. It’s not our fault that their posts are so good.

Shimin (44:33) It’s unofficial,

unofficial, yeah.

Rahul Yadav (44:35) so they have this breakdown of what are the different components of agent memory. and one of the things I learned from this is apparently our brains have s different chemical signals when we want to st store a memory for later. Which I don’t know. I was just like, we remember stuff. Who knows how these things work? And um

The claim here is that our our brain uses some specific chemicals to mark more important memories versus less ones. So so anyways, with with that aside, because they they do talk about how like agent memory versus human memory and all that, and that I I was a new thing I learned. There are a few um layers to what goes into giving agents um a memory. And so

At the very bottom you start with the the little transcript of every single thing that’s going on. Um and you kind of tag those things on like what could be useful later to learn from. from that you extract different notes based on the things that you tagged. and you you you turn this these into the well-organized notes.

but they’re still you can think of it as a just a a stack of notes that you might be working with. and then you can start linking the different notes together, like a wiki style or something where you can create a representation of those notes. And then finally you can use that to, you know, when necessary, inject context into the conversations that Claude is having. And let’s say you, you know, talk to it about something, um

a a few weeks ago and it stored it as a memory and um now you’re talking about uh talking about similar things in a different chat but if it has that in its memory it can inject that context that it learned from before uh into that. And they they call this the the the process as dreaming which helps them kind of goes from taking that Ross stream all the way up to

Creating that in the context that they’ll later inject into um conversations in the future. I I was telling you guys before how I’ve been on a sci-fi kick, and one thing that I start seeing more and more is the idea of memory implants that sci-fi plays a lot with. So

Blade Runner has it. I I watched this movie Moon.

Sam Rockwell w was in it. That movie was great.

Shimin (46:58) Yeah, great movie.

Rahul Yadav (46:59) Yeah, he has memory implants. And so when I was reading this post, I’m like, would in the future rehab the, you know, sped up version of Claude where it it wakes up with memory implants instead of and th that’s kind of where this is going, right? Once you have the context,

Dan (47:15) Yeah.

Rahul Yadav (47:16) once you can just implant that into anything else and it will just act like

It always had those memories. Um

Shimin (47:21) Yeah. And

and the kicker is of course it is still markdown files, right? It’s just like you just go to your home directory in

Rahul Yadav (47:26) Yeah.

Shimin (47:29) your memory implant and and search for script for stuff.

Rahul Yadav (47:32) Yeah, it’s it’s marked down all the way down. the and so they try this on different things. they tested it on Pokemon Red, which is where my middle name came from this week.

Shimin (47:43) Mm-hmm.

Rahul Yadav (47:43) and so in

Shimin (47:45) Can I can I just say how as a elder millennial, a child in the nineties, how weird it is to come to to face a serious discussion of AI with Pokemon Red as as an example. did I know back then

Rahul Yadav (47:56) It is insane.

Shimin (47:57) when I was seven or eight years old playing Pokemon Red, one day AI is gonna use this to learn how to improve their memories better.

Rahul Yadav (48:03) Yeah, it’s crazy.

Dan (48:04) there’s lots of other

prior art, right? Starcraft was an early target.

Shimin (48:08) Yeah, that’s true.

Rahul Yadav (48:09) so they start with the Sonnet three point seven, which I honestly had forgotten was a thing. and they they what what Sonnet three seven did um in when it was playing Pokemon Red is it stored all sorts of details that really polluted the memory, but didn’t really help in the performance of the model in accomplishing uh you know playing the game.

And then they did the try the same thing with Fable Five. And between Sonnet three seven and Fable Five, the model itself, the harness, all that has gotten so much better that now it’s focusing on Fable Five focuses on the strategy part and tries to

Shimin (48:50) Mm.

Rahul Yadav (48:51) inf infer the rules of the game so that it can store that in its m memory so that later

it can use those when even if it ends up losing this time, it’s gonna use those to be able to play the game again um with that additional context and optimizing for kind of the global goals versus just what what’s at hand right now.

Shimin (49:11) So what I’m hearing

is Anthropic is saying that you gotta dream big to become the Pokemaster. Is that is that correct? Also, I just wanna

Rahul Yadav (49:16) Ha ha ha ha.

Dan (49:17) Ha ha ha.

Shimin (49:19) say there’s there’s no way they played Pokemon Red on an actual Game Boy event, right? Which means they’re probably running some version of this on an emulator somewhere. But they’re telling us

Rahul Yadav (49:26) Yeah. Emulator. Yeah.

Shimin (49:31) we cannot cut this uh

Dan (49:31) Watson.

Shimin (49:32) decompile our own games. This this seems like rank rank

Rahul Yadav (49:34) Ha ha ha.

Shimin (49:36) hypocrisy to me. Yeah.

Rahul Yadav (49:39) If you if you do it in the name of science and research,

Shimin (49:41) Ha ha.

Rahul Yadav (49:42) I th I think uh I’m I’m not a lawyer, but I think that should be fine. W we can try and make the case.

Dan (49:48) mean that’s how they justified

Shimin (49:48) Lawyer corner here.

Dan (49:50) stealing half the books ever written too, right? I mean

Rahul Yadav (49:53) Um

Dan (49:56) It’s all in the name of science.

Rahul Yadav (49:57) in the in the name of S human progress, man. one of the things that becomes uh apparent from this is um and you know you can kind of see this in in humans too, so it’s interesting how studying the models is is helping us introspect as well. If you inject too much memory

Context management is also becomes hard. And it’s similar to someone who gets dementia or something where they have stronger memories of the past, but not much context of the recent times. And I’m sure there are like some um specific you know, th there might be conditions out there where people do suffer from um the the memories of the past bear more weight than the the recent ones. And so the

i it’s like a tricky thing to manage where you want to make sure that the recent stuff gets relevance, but at the same time you want to inject things at the right time when uh they make sense and at the same time you have to make sure you you only have s so much of a context window to to maintain all this in. And so how do you uh continue continue to trade off between all those things.

Shimin (51:02) Yeah, how how do we deal with humans, right? we have a regulation system where things that gets used more gets remembered, but we don’t actually like introspect on it every evening to see, hey, I use the tying my shoes skill today. So let’s remember this for tomorrow. Like

Rahul Yadav (51:20) Yeah.

Shimin (51:21) um there’s gotta be other designs. and and

Probably a lot more iterations on how memory and dreaming works in LLMs going forward.

Rahul Yadav (51:29) Yeah. And it’s also interesting that every time we we recall someone, we we’re always slightly tweaking the details. I I think you guys have, you know,

Shimin (51:39) Mm-hmm.

Rahul Yadav (51:40) seen that in others and may maybe i in yourself too. And so as Claude and these agents do it, there would be the obviously the truth that that memory initially started with.

But as you use it over time, if it follows the same model

Shimin (51:55) Mm.

Rahul Yadav (51:56) that humans do too, where it’s rewriting its memory of the thing that happened over and over again, um, it’s gonna operate in all the weird ways we do, where the story

Shimin (52:05) Yeah.

Rahul Yadav (52:05) is end completely different than what actually happened.

Shimin (52:08) That’s how you end up getting kidnapped by aliens. I went to the bar and then I got kidnapped by aliens. And then I woke up. Hung over.

Rahul Yadav (52:10) Yeah.

Dan (52:15) But it’s it’s it’s also interesting

though that the same problems that I mean as you know, I’m like kind of fascinated by the idea of like why aren’t models updating their weights, right, during

Shimin (52:27) Mm-hmm.

Dan (52:27) during this. And it’s it like my very I don’t know layman’s understanding of it is that like it effectively causes model collapse if you try to do that because it rates all the

recent things highest. And then after

Rahul Yadav (52:42) Mm.

Dan (52:43) a while that that just causes it to start just like speaking gibberish instead of like cogent sentences. so it’s interesting that like some of the same problem space applies here even if the you know outcomes are a little different.

Shimin (52:57) Alright, thank you for that, Rahul. shall we move on to two minutes to midnight, where we take a look at the financial side of the AI industry using the analogy of the Armageddon clock from the Bulletin of Atomic Scientists. we are currently at three minutes, and of course midnight is when these supposed AI bubble bursts. I’m

gonna go.

Dan (53:20) We do we

want to talk about the huge stock sell off at all before we Yeah. It was starting, but then yeah.

Shimin (53:24) I think we mentioned it a little bit a little bit last week. so yes, there has been yeah.

Rahul Yadav (53:31) Was there one?

Dan (53:32) Yeah.

Shimin (53:32) There there was like I don’t know, six six percent, six to eight percent sell-off for a couple of days, and especially the day after election day. There was a major sell-off

Dan (53:42) Mm-hmm.

Shimin (53:42) of tech and all tech related stocks, but it has since came back a couple of percentages.

Dan (53:46) Uh

Shimin (53:47) So we we may or may not be in

Corrections territory. listeners, if you’re listening to this, check your stock portfolios. the state that may be out there. So, yes. And as a consequence of the broad tech sell-off, Anthropic’s IPO valuation is currently being adjusted. And this is coming from Reuters.com by Karen Kwok\

When we first talked about Andropics S one, I think they were looking for a one point seven five to two trillion dollar

Market cap and and this week has come out that they are lowering that market cap to be about one to one point four trillion dollars indeed. So six hundred two or trillion dollars in market cap just evaporated since uh

Dan (54:28) Mm.

Shimin (54:28) since we covered the S one. Not that long ago, like two weeks ago. I’m just gonna say that’s that’s a bear sign, guys.

Dan (54:36) Mm-hmm.

Shimin (54:37) Yeah. Dan, what do you got for us?

This week.

Dan (54:40) well, it’s kinda similar. Um on the other side of the fence we have open AI. apparently Greg Brockman was being interviewed, and someone directly asked him whether or not they felt that uh OpenAI would be IPO ready next year. And he said, quote unquote, I think so, yes. We’ve been very clear this year that we want to go public when we feel is the right time for us.

And I feel extremely good about the business. I feel like

Rahul Yadav (55:06) Hmm.

Dan (55:07) we’ve made so much progress this year and I’ve honestly never felt more confident in what we’re doing, Brockman said.

Yeah. So and then they they kinda go on to talk about something that I do actually sort of agree with, which is that like

OpenAI wasn’t really a serious player in the enterprise market a year ago, and now they kind of are.

Shimin (55:28) Mm-hmm.

Dan (55:29) and so there’s, you know, maybe some truth to that, but I’m sure it has nothing to do with the numbers.

Shimin (55:34) Yeah, or the fact that they um had some trouble getting the round that they were looking for.

Dan (55:39) Yeah, that’s the b the bridge.

Rahul Yadav (55:40) Yeah.

Dan (55:41) I haven’t we haven’t heard the end of that. I didn’t follow up on it. But yeah, they really were struggling in that bridge round. So

Shimin (55:47) and Rahul, of course, you are going last but definitely not least this week.

Rahul Yadav (55:53) boy.

I wish this was the least. so Jensen Huang has been having an affair with a person named Alice Huang who is an alleged Chinese

Dan (56:07) No relation.

Rahul Yadav (56:08) spy. No, sorry. Thank you, Dan. That that that yeah.

Shimin (56:10) No relations, yes. Huan is extremely

Dan (56:11) Ha ha ha ha.

Shimin (56:12) common uh Chinese last name, yes.

Rahul Yadav (56:14) No no relation. he put out a memo memo to the employees uh apologizing for his actions. the the where this Chinese spy thing leads to is their report that the this relationship may have helped facilitate seven and a half billion dollars in chip sales to China

Shimin (56:34) Mm-hmm.

Rahul Yadav (56:34) through a network of different companies and everything. And so that’s where, you know, the the whole thing gets pretty messy.

the the government is on it now, House is requesting uh doing subpoenas and stuff and requesting that he goes and testifies. Um and then apparently uh Marco Rubio back in May knew about it and had told Trump about it. that it he might have relationships with a sustain suspected Chinese spy and it it’s almost six months later now.

So here we are. Um and so this has caused a lot of panic because there is a key man dependency here. There is n no obvious successor to him. And the board uh of Nvidia is doing an investigation into him. and based off of all this news, I’m looking right now, and in the last five days, Nvidia is down seventeen plus per sev in stock. And so

that’s billions of dollars that the the company has lost and uh you know could potentially uh peop people are saying could potentially lose a lot more if the i if Huang is forced to step down because of this.

Dan (57:44) And let’s not

forget they’re doing a lot of the financing or guarantees. Yeah.

Rahul Yadav (57:48) Yes. And so

Shimin (57:47) I was just gonna say that then, yes.

Rahul Yadav (57:50) the all the debts, all these financing things, everything’s based on Nvidia having a high valuation and once that is down, everything starts a chain reaction that that could cause a lot of bubble bubble toil and trouble.

Dan (58:06) Yeah,

the the whole time we’ve been doing it’s what over a year now we’ve been doing this segment. I never thought I never in a million years thought it would be this domino in this or this

Rahul Yadav (58:17) De

Dan (58:18) the it’s like that, you know the XKCD for the open source thing where it’s like all the Django blocks and then there’s just

Shimin (58:22) Mm-hmm. uh

Dan (58:24) like the one tiny one and they’re like that one

Rahul Yadav (58:26) One little

Dan (58:26) unmaintained or you know, apparently one guy maintaining the one thing. Yeah.

Rahul Yadav (58:29) One

Shimin (58:30) Mm-hmm.

Dan (58:31) It’s kind of how I felt this would be in the the grand scheme.

Rahul Yadav (58:35) boy.

Dan (58:35) Thanks, but here we are.

Rahul Yadav (58:37) I agree. It would be insane if this is what causes it.

Shimin (58:40) So

Dan (58:40) Yeah.

Shimin (58:40) we got three bear news items this week. all that said, how do we feel?

Three minutes. I I’m just gonna throw it out there. we’ve been moving the clock forward thirty seconds at a time. I don’t quite feel like we need to do another thirty seconds, but at least fifteen ‘cause there’s been no silver lining in this whole thing.

Dan (58:59) Okay.

at least we’re not waffling lately. we had that time period where we were just oscillating for a while. Yeah.

Shimin (59:04) Yeah. We’re waiting and seeing. Yeah.

Now the wait is over.

Rahul Yadav (59:07) And next week the

Anthropic IPO will have well

Dan (59:09) That’s true.

Rahul Yadav (59:10) yeah, so we’ll know something.

Shimin (59:11) Maybe. All

right, let’s do two minutes and forty five seconds. How’s that?

Dan (59:16) Forty five. All

right.

Rahul Yadav (59:19) Yeah. That’s what you should really be asking Fable about, Shimin. Is

Shimin (59:23) Predict the market.

Rahul Yadav (59:24) G given yeah, given given all this news and two minutes to midnight, how do you feel?

Shimin (59:28) Yeah. Me me and me

and all the Wall Street Bets subreddit are doing that.

Dan (59:32) Uh go gold

Rahul Yadav (59:33) Ha ha ha.

Dan (59:34) Goldman just banned uh their desks from having uh Claude access. Kinda interesting. I don’t I

Rahul Yadav (59:41) Interesting.

Shimin (59:42) Yeah. Too many tools.

Dan (59:43) don’t think it’s because they were, you know, trying to use Claude for predicting. I suspect it has more to do with shipping buggy trading code, but

Rahul Yadav (59:51) Yeah. And high frequency trading you just need one and and and and you and you have another mic Yeah.

Dan (59:54) Yeah. there’s that decimal point. Oh oops.

Shimin (59:58) They were like, I’m not making enough money. Let me just code this plugin

and upload it to Steam and silo it into my training software. That would do it.

Rahul Yadav (1:00:05) Yeah. And then

Michael Lewis appears and then he’s like, I got a book’s worth of material right here and then uh

Dan (1:00:12) Yeah.

Rahul Yadav (1:00:13) he he he fires off another one.

Shimin (1:00:16) AI took

over Wall Street. Well, with the setting of the clock, uh, we’ve once again come to the end of the show. thank you for joining us for our study session and also Wall Street bets segment this week. Again, not

financial advice.

Rahul Yadav (1:00:28) Yeah.

Shimin (1:00:30) Do not trade based on anything we’ve said today. if you like the show, if you learned something new, please share the show with a friend. You can also leave us a review on Apple Podcasts or Spotify. It helps people to discover the show and we really appreciate it. If you have a segment idea, a question for us or a topic you want us to cover.

Shoot us an email at humans at adipot.ai. We’d love to hear from you. You can find the full show notes, transcripts, and everything else mentioned today at www.adipod.ai. Thank you again for listening and we’ll catch you on next week’s episode. Bye.