Claude Watermarks, Zuck's Superintelligence Essay, Zed's ZDB & Multi-Agent Turf Wars

Shimin (00:00) Hello and welcome back to Artificial Developer Intelligence, a weekly conversation show where two software developers navigate the ever changing AI assisted software maze. We go through hundreds of links and dozens of newsletters each week, so you don’t have to. My name is Shimin Zhang and with me today is my co host Dan. He is knitting a pink winter hat for his niece, Lasky. Dan, how’s the knitting going?

Dan (00:27) I haven’t started yet, so I don’t know. Don’t know what to tell you.

Shimin (00:29) Got it. Winter is coming.

Dan (00:32) Yeah.

Shimin (00:32) it has been two weeks since we last recorded and one week since we were in a gondola together. have you recovered from our trip?

Dan (00:39) That’s true. Yeah.

yeah, I’m I’m doing all right. I’m back to the the daily grind as it were.

Shimin (00:47) Excellent.

Dan (00:48) and Rahul won’t be joining us this week ‘cause he’s off on another vacation because one vacation wasn’t good enough. So

Shimin (00:55) international traveler Rahul

Dan (00:57) Mm-hmm.

Shimin (00:58) all right, let’s get started. on today’s show we’re gonna start with the news. as always we will have some Claude watermark controversy and Zuck wrote a new essay.

Dan (01:09) That he did. then we’re gonna be chatting a little bit in the tool shed about ZDB, which is Zed’s new take, so to speak, on version control.

Shimin (01:18) Mm-hmm. then we’ll do post processing where we’re gonna talk about why everybody in tech is so sad.

Dan (01:24) Yeah, that one’s been making the rounds. And then next up we’ll do a deep dive on patterns and problems in emerging multi-agent systems.

Shimin (01:33) let us get started.

So our first news item this week is Anthropic’s latest announcement that starting from august second of twenty twenty six cloud will start leaving watermarks in its output. So what does this mean? these watermarks will be according to Anthropic imperceptible to the end user.

and that it will be generated from all Claude models.

In that they haven’t released a way for end users to tell whether or not a piece of content is watermarked by Claude just yet, but that is incoming. so it seems like from their description that these embedded watermarks in text will mostly be about token frequency, essentially like a cryptographic signature on which words Claude models supply. So then

it is easily able to be traced back by any third parties.

This is in compliance with EU’s AI act code of practice on transparency of AI generated content, which I was not aware of before reading this post. And according to

Dan (02:40) Yeah, that’s why they’re doing it.

Shimin (02:41) Exactly. And according to Anthropic, the the new watermarks will not change the meaning, quality, or readability of Claude responses. I am kinda skeptical of this. Or skeptical of the part that it is

Dan (02:52) You and

fifty percent of the internet. Maybe more than fifty percent, I don’t know.

Shimin (02:57) on the surface, right, like this solved a lot of problems we often talk about with AI slop. that, you know, existing machine learning tools for detecting AI text is maybe not that great. what I am skeptical of is how useful this would actually be, because it would still be basically trivial to remove the watermark.

If you choose to, as Anthropic stated in their post themselves, that this is you know, not undefeatable, anyone can create a script that updates the words to its synonyms and then, you know, essentially evade it.

Dan (03:29) Or you pass it through pretty

much any other L L and be like, rewrite this sentence.

Shimin (03:33) Right. Right.

Dan (03:36) Like I I bet a very low wit number of weights, open weight model could handle that task running on your, you know, phone probably.

Shimin (03:45) that is true.

Dan (03:46) Not even laptop.

Shimin (03:47) My local Qwen three point eight will be excellent at doing this for everything I want to send.

Dan (03:52) Yeah, or, you know,

Gemma like what, three E two B or whatever that the little one is that runs on phones. Like surprisingly

Shimin (04:00) Right.

Dan (04:01) well too.

Shimin (04:02) we shall try that once they have the tool out for detecting watermarks. yeah.

Dan (04:06) Yeah, see how well it does on

obfuscating it. It feels like it would be a good Vibantel post for you if you did like a deep dive on it.

Shimin (04:14) yeah, looking forward to trying that. So it does defeat the case of the AI slop grenade though, right? Like if you aren’t bothering to even read whatever slop grenade you’re throwing to the other person, you’re probably not gonna bother to run this through some sort of watermark.

Dan (04:29) Yeah, but like to me that’s never been the problem there. The

Shimin (04:35) Mm.

Dan (04:36) the problem has also been that like it’s just kind of thoughtlessly making its way into the discourse, right?

Shimin (04:43) Right.

Dan (04:43) and like to me that’s always been just like blazingly evident from like the hallmarks of like how LLM’s write, you know? I’ve never felt like wow, that’s like a really good argument.

And then been surprised to find out that it was LM written. And it’s like usually almost always immediately obvious to me. And I don’t really know why or how. I mean, there’s a lot of like, it’s not X, it’s Y, but you know, but there’s like lots of those tropes that are all kind of tells, you know.

Shimin (05:13) Yeah. But y but even if you just remove those tell I mean, there are plenty of services out there today that removes AI tells from a piece of text. And in some way, like this is one of those you know, AI is a job creator arguments coming from, right? Like it’s creating a whole new industry of AI writing style removal companies. I should write one.

Dan (05:32) Yeah. But it still

tends to be pretty like stilted and like loaded in terms of the way that ideas are presented to be like a little bit more sensational than like a normal person would use in, particularly in like technical documents where the content just isn’t that exciting enough to use that type of language, you know.

Shimin (05:54) make your technical documents funnier. That that could be that could be another tool that we should create.

Dan (05:59) Make them into adventure novels. A quest about

Shimin (06:01) Yeah, I I think we should.

Dan (06:03) how Shimin found Dynamo DB and used it instead of a relational store. And why he chose

Shimin (06:08) Yeah, the does

Dan (06:09) it over the relational store for this access pattern.

Shimin (06:12) the sword of auto node recovery. Yeah, I I could see that. So the second thing that this feature got me thinking about is like if Claude can quote unquote imperceptibly add textual watermarks to the text generated, like what else can they add to the text that would also be imperceptible to the end user, right? Like

We we often talk about the ability for AI to do propaganda. there’s definitely propaganda, but like just marketing, ads, like any number of bad uses, or potentially bad uses that we can think of.

Dan (06:45) Yeah. Well, that was one of the things I found really funny about the like sort of public reaction to this too is is the number of people that were angry about the way the watermarking worked because they were like significantly concerned that it was going to impact the quality of writing. And I was like, It’s just not that good to begin with.

Shimin (07:03) Right.

Dan (07:05) Like, no offense, but really So I don’t know. And then there

Shimin (07:08) Yeah. The the writing should

be least of your worries.

Dan (07:12) Yeah,

was like one guy on a subreddit that was was super pissed off about the whole thing ‘cause he was like, I’ve been found out. And then like people

Shimin (07:19) Ha ha.

Dan (07:19) there was like a whole reply of threads. Like there’s like fifteen threads responding to that, going like, You wrote that with an AI, didn’t you? It was so good.

Shimin (07:27) Mm.

Dan (07:28) my goodness.

Shimin (07:29) Yeah, it’s yet to be seen if other Frontier labs would also comply or will they just comply and do it quiet quietly and not like write a whole blog post about it. But we’re running into, you know, second order effects of AI land here and we’re finding out together.

Dan (07:44) Yeah, it’s true.

And you know, maybe we’re not that far off from your sharing weights on a USB drive because of something

Shimin (07:50) Ha ha.

Dan (07:51) like this. Like, man, I’ve got an untraceable model here. You want you want some weights, man?

Shimin (07:57) It’s a cyberpunk dream that we all dreamt of as

teenagers in our parents’ basements. I don’t know. I part of me kinda likes it. okay. huh.

Dan (08:07) I always thought ice was gonna be some

sort of like three dimensional like thing that you’d have to sort of like literally hack your way through like brambles, you know? And

Shimin (08:15) Mm-hmm.

Dan (08:16) and instead it’s turning out to be like model weights on

Shimin (08:20) Well we can we can by code a nice UI for for editing model

weights.

Dan (08:23) For ice, yeah.

Shimin (08:24) Yeah, in like a in like a nice video game way. Okay. Well yeah, that’s my news of this week. on to our second news item.

Dan (08:33) All right, so we’ll go from like kind of normal to completely unhinged. full disclosure, I don’t have a subscription to 404 media, which is our source for this article, so I couldn’t read the whole thing. But what I did do was read the entire actual source post.

Shimin (08:50) Bravo, Dan. Bravo.

Dan (08:52) Where do I start with this? so Zuckerberg posted a six thousand five hundred-word essay about giving everyone AI superintelligence is sort of like the headline here from from four four media. If

Shimin (09:06) Mm-hmm. Hello, Zuck.

Dan (09:09) you’re watching on video, we just flashed up a very flattering picture

Shimin (09:12) Sorry, we had where the slugs picked.

Dan (09:14) of him. and it’s really kind of an

Interesting read because there’s so much stuff that’s unsaid in the essay. And then there’s actually like one or two things that I think I and maybe even you would agree with, but they’re presented in such a sort of

I don’t know, disconnected way, I guess, from like other facts on the ground that it’s like easy to root for it and then you go, wait a minute, what? Like you know. Yeah.

Shimin (09:41) Like who was writing this?

Dan (09:45) So I guess I’ll start by by talking about like those good things. so he brings up a couple like kind of sort of plausible pieces that

you know, I can agree with. So one is that like in the future everyone will probably have personal agents. And I’m like, sure, I could see that happening.

Shimin (10:00) Mm-hmm. Yep.

Dan (10:02) And we’ve already talked on this show about open claw and like, you know, Pi and other things like that that you can use to like do that today. So seems pretty plausible that you could commercialize that. He talks a lot about like sort of how that feature isn’t likely without privacy, which I also agree with.

Shimin (10:20) Mm.

Dan (10:21) And then goes on to claim that like basically something like homomorphic or end to end encryption would it would be the solution to allow people to like trust Meta with that information, you know, because we all definitely trust WhatsApp’s end to end implementation. so anyway, but overall, like cool. All right, Doc, I’m I’m with you on this one. then second one I kinda agree with them on is open weights. He’s like open weights are really important and like blah blah blah. And it I think I’ve it

personally agree with that. You know, you’ve I’ve been on the record talking about running open weight models and like why I think that they’re important in in some ways, maybe even the future, because we’ll, you know, as hardware continues to get better, we’ll be able to run them reliably on our phones and do stuff locally that we wouldn’t need the cloud for. And that kind of actually unlocks like a lot of trust in some of these things. So agree with what he’s saying there, except for obviously the, you know,

conclusions about it’s fine to run those through meta. and then weirdly the other one that I agree with him on is he’s he talks a bit about like the concentration of power in like these sort of limited set of frontier labs, right? And like is it in some, you know, hypothetical future where like he doesn’t call it AGI. He calls it what super intelligence, but it’s basically what everyone else has been

Shimin (11:32) Mm-hmm.

Dan (11:33) calling AGI. you know, is it

equitable or fair or even safe for a handful of companies to have that power and and nobody else has it. so again, sure, on the out on the surface, I agree with that. now you could look at that through the lens of like, well, Meta’s sad that they’re not like in the running for Frontier Lab and trying to tear everybody else down with that. But like whatever. But there’s also a ton of like either

Just outright bad stuff. Or like if you think it actually through logically, you’re like, wow, okay. In here too. So, like a good example of that is he’s, you know, going on and on about like everyone deserves access to super intelligence, right? It should be available to all. Like we we want to really want to focus here at Meta. It’s small businesses and and people, right? Like everyone will have free or affordable access to these tools. Cool, like the vision.

but let’s think about how that would actually work. What is Meta’s model today? Well, Facebook is free and available to all, or at least people that have names that don’t have weird accents in them. because they try to prevent you from making accounts if you have them. But the but how is it paid for, right? By advertising.

Shimin (12:42) Mm.

Dan (12:43) So who’s to say they wouldn’t attempt to do that exact same

model with all of this like massive amounts of like quote unquote free compute that would be required to actually like make that vision happen. And then, you know, as you’ve talked about in the show multiple times, like AI companies are spending, including Meta, are spending billions and billions of dollars building out this infra. It’s not free. There’s no way it can be free. So like

Shimin (13:04) Mm-hmm. Right.

Dan (13:06) there is a business model there and it goes completely unstated in his article. So I think that’s kind of just like a

I don’t know why I have a mission, I guess, right? another one that so I guess you know, TLDR, who’s gonna pay for this free compute? ad blood money, that’s you. There’s your there’s your pull quote right there. Yeah.

Shimin (13:22) Wow, that’s that’s darker than even my takeaway from this. Okay.

Dan (13:29) then the the third one that kind of got me was like he also goes in this little like mini rant about sort of like American nationalism around

Shimin (13:37) Mm.

Dan (13:37) like models and weights and like it’s

Shimin (13:40) Mm-hmm.

Dan (13:40) Critical that like we remove all legislation so that America can lead the frontier against this, you know, and like we’re already getting slowed down by the legislation that’s in place and all this kind of stuff. And I was just kinda like, are we though? Like it seemed to me like the discourse at least last week was like pace the frontier because we’re actually scared of the capabilities that we’re creating here, not like, legislation is crushing us.

So found that a little odd as well. And then last one and I’ll and I’ll I’ll go for your take. Sorry, I’m monologuing here, is

Shimin (14:12) No, go for it.

Dan (14:14) data centers are good.

Shimin (14:15) Yep. That’s a fun one.

Dan (14:17) yeah. And so we’ve got pretty much all the hot topics covered here. We’ve got we’re efficient with our water use. Okay. I’m just gonna leave that one. But then my favorite is data centers create so much economic value for the areas where they’re located, and gives an example of one where essentially tax revenue is enabling like a otherwise very poor school district to like offer bonuses to their teachers.

Shimin (14:41) Fifty

thousand dollars. That

Dan (14:42) Yeah.

Shimin (14:42) is a it’s a non negligible amount of bonus.

Dan (14:45) Right.

But what they don’t state in there is that like in practice the actual operational number of jobs created by a data center is quite low, like well under a hundred per data center. because it’s basically just like maintenance people to to

Shimin (14:59) It runs itself. Yeah. Yeah.

Dan (15:01) you know, you need a couple of pairs of hands to actually operate racks and stuff, and

Shimin (15:06) Mm-hmm.

Dan (15:06) then you need, you know, electricians on, you know, and a couple other things just to keep stuff running. But like

is not you know in a town of like say even a small town of like ten thousand people is is that making really a meaningful impact on their economy

Shimin (15:20) Yeah.

Dan (15:21) other than the tax revenue and you can bet that a huge corporation like that is gonna do everything they can to dodge every amount of tax they can. Yeah.

Shimin (15:28) Well yeah, I was gonna say that. Like at first, yeah, sure. But then

what happens when folks normalize building data centers, right? Then they

Dan (15:36) Mm-hmm.

Shimin (15:36) have to compete with each other, and eventually these things will be built for basically free.

Dan (15:41) Yep. Yeah. So I mean, not that I’m like

I I’m sort of like a neutral party when it comes to data centers ‘cause it’s like I think they could be done more efficiently, but I d also don’t think they like shouldn’t exist, you know.

Shimin (15:53) Alright. Well as as I always say, like, do you object to building data centers for watching Netflix? If the answer is no, then why is AI that much different? ‘Cause they’re inherently not that much different.

Dan (16:05) Yeah.

Well, power usage I guess, but

Shimin (16:08) Yeah.

Dan (16:10) Yeah. But it’s like I mean there’s nothing wrong with compute, it’s just like

If we’re gonna like drive straight into it and ignore like climate change and everything else, then yeah, I think that’s pretty bad. But like if we’re gonna like, you know, be somewhat responsible about it, which I don’t think we’re doing by any means right now, but like that’s all I’m just trying to say is that like

I guess I’m moderate around the actual concept, whereas I think right now there’s this like no data centers whatsoever kind of pushback, which I get it, but like you know

Shimin (16:37) Yeah, I mean for the record, I think I think, you know, some corners of the internet is a little too hard on the no data centers ever kind of angle of things. there there’s gotta be a middle ground somewhere. So my issue with this piece is exactly as you said, surprised that we agree on so much of this. most of Zuck’s arguments, I am

sympathetic to you if not just agree with out right but who is saying it, right? Like z Zuck like you know, like yeah.

Dan (17:06) The man whose yacht ignored

a distress signal.

Shimin (17:09) If if if Dan you wrote this, I’ll be like, wow, this is great. minus the weird pieces about the government and the data centers. I’ll be like, yeah, yes, everyone everyone should have a personalized tutor and coach with a PhD in every subject. I wholeheartedly agree with. I think this is

Dan (17:25) Yeah.

Shimin (17:25) a great future. But coming from Meta, coming from Zuck himself, I’m sorry, he may not just be the right messenger for the message these days.

And also, like, who is this piece for? Are you trying to pull a Dario? Like what Dario paints a rather neutral picture of the future of AI in my opinion. He does indeed hand wave the negative parts, but he does at least mention the potential risks. this piece basically just like you said, omits all the possible negative negative

versions of our AI future. And of course everyone should have the latest best models. But if you are able to pay for more tokens, then you should be able to. This is the all pigs are equal, but some pigs are

Dan (18:08) Yeah.

Shimin (18:09) more pig like than others.

Dan (18:11) Trickle down tokenomics.

Shimin (18:13) Yeah, and then and then he did mention the AI security, cybersecurity aspects and how law enforcements would have to do a better job. and and I quote

Quote, in some cases they will have more advanced models as well. So so what is it? Does everybody have good models? Or do we all have pretty good models, but the government has even better models to catch us doing bad

Dan (18:34) Ha ha ha.

Shimin (18:35) things? I don’t know. yeah. And lastly, I just I don’t know who this is for. I feel like everyone is writing these thought pieces trying to tryna Pull Dario, but you know, like are you

Is this for meta employees? Is this for the general public? Is this for the US government? It it’s unclear to me.

Dan (18:52) I mean they did publish it on the internet, so I assume it’s targeted at everyone, but yeah, that is a good question.

Shimin (18:57) Yeah.

And and of course there is I I don’t know, it it also lets corporations off the off the hook a lot of times, despite it coming directly from meta, right? Like it’s very little talk of corporate governance other than what how the government can have some control over the models, but not too much that it stifles innovation. Like it all feels very

yeah.

Dan (19:19) Zuck

Shimin (19:19) Again, we’re already not gonna work for meta, so it doesn’t matter.

That ship sailed long ago.

Dan (19:23) Mm. Yeah.

Shimin (19:25) Yeah. Any any last last notes on this one?

Dan (19:27) No, it’s just other than it I agree with you that it’s it ju it is kinda strange about like who is the target.

Shimin (19:32) Okay, well we will cover more of these think pieces.

Dan (19:36) Yeah.

Shimin (19:37) May maybe fewer. Listeners, if you want fewer of these pieces, let us know.

Yeah.

Dan (19:41) Yeah, send us an email to humans

at adipod.ai

Shimin (19:45) Thanks, Dan. Okay. Moving on to the tool shed. we have an interesting tool brought to you by Dan this week.

Dan (19:50) Mm-hmm. So this I think this popped up on Hacker News. I forget where I found it. It’s either that or I actually use Zed too, so like sometimes their stuff just kind of like pops up.

My feed as well. But yeah, so Zed, who are makers of the Zed editor, which I think is actually don’t quote me on this, but I feel like isn’t it a bunch of former folks from VS Code?

Or Atom It’s Atom I think. Yeah.

Shimin (20:11) I do not remember. Atom yeah.

Dan (20:13) Before ‘cause then when Microsoft had owned both, a lot of the Adam people left and I think they started Zed, but I I

Shimin (20:19) I did not know that.

Dan (20:20) think. Don’t quote me on that again. Purely purely Dan’s brain. Recall is faulty. just pretend I’m a bad L LM right now. So but anyway, so they, you know, make a pretty decent editor. It’s built in Rust, runs a little

based stuff, which is nice if you’re into that. obviously the ecosystem isn’t as big and everything else. But they’ve been kind of branching out. Like they have their own

setup where it’ll do autocomplete and, you know, all those kind of things for you and an IDE. I think they might have their own agent thing too. I don’t use it. they have really great integration with existing agent stuff as well. I don’t know. It’s turning into a sales pitch for Zed, but anyway, sorry. it’s a good editor, what can I say? Yeah.

Shimin (20:56) How much is that paying you? Yeah.

Dan (20:57) Not as much as Google Agent, like a whatever. Anyway, but well, you’ll see in a second I’m not super great bullish on

on this tech, but I did think it was interesting to talk about because it brings up a lot of I think relatively interesting conversations that are

Shimin (21:10) Mm-hmm.

Dan (21:10) still ongoing in the space. So they’ve announced this new thing called DeltaDb and it is basically I have not used it because this there’s like you have you have to sign up for the dealy and I think I’m on the wait list still whatever. But like the

The general sense of it is that it’s more granular than Git. So like

Shimin (21:28) Mm-hmm.

Dan (21:28) every edit you or an agent makes to a piece of code is captured. Like think like text level undos essentially instead of like a diff of a commit, right? so that’s the the mo the one thing. So it’s a lot more granular. And then the second piece is is essentially work trees by default, like CRDT work trees. So

You can have lots of agents, whatever, all working on the same code like shared code base essentially at the same time. and then the third piece, and this is like the one that I think is most interesting, is it I don’t know how this works, but it ties the LLM conversation that if you’re using one to to edit the code to the actual like delta, which is what they’re calling their like smaller than a

Thing. and that’s the part that I thought was kind of interesting because like I feel like as a as an industry, we still haven’t really solved the question of is the conversation that you had while you wrote a piece of software with Claude or whatever worthwhile to others. So

Shimin (22:31) Mm-hmm. Yeah,

we talked about this a number of times. Yeah.

Dan (22:33) Yeah. So like

given that we haven’t solved that, like I guess I’m asking you, do you think this is a worthwhile thing to use because like would you care about the conversation I had with Claude in order to make the commits or the excuse me, the deltas that you’re looking at?

Shimin (22:50) I I think I would I would care personally, in theory, but I don’t think in practice too many people will follow up on them. Either that or everybody would only look at the conversation and then not bother with the code changes.

Dan (23:06) Yeah. Well then then the other questions I have around it too are well first like one little quote I wanna pull out of the the actual like announcement article I thought was kind interesting. so one thing they note that at Zed like the company, they actually don’t particularly like pull requests. and the reason why they don’t is they

Prefer quote unquote building trust and shared understanding by discussing the code as we write it. And they call out one of their competitors, GitHub, doesn’t let you talk about the code until after you commit and push. And by then our most important conversations are usually already over. And that part actually resonated with me a little bit, which is like

Shimin (23:43) With me too. Yep.

Dan (23:44) when you especially if you’re working

You know, being a mentor to junior engineers, like one of the risks you run in mentoring is like they can go really far down a bad path before you have a like a check in. And then you you risk, I would say, both ill will and like kind of like harsher than needed lessons being learned about the work being done because you didn’t check in frequently enough, you know. And that’s all that’s

Shimin (24:06) Mm-hmm. Yep. Yep.

Dan (24:09) one of the many reasons why, like LM or not, I always push for small PRs.

And like also we’ve talked about like last week, like really liking that like sort of steel thread idea, right? Like the the you know, narrow vertical slice of the work because that also has the same effect as this, right? Like early early and often failure and iteration is like a good thing.

Shimin (24:28) Make sure your approach is correct before you spend the rest of that time, yeah, fleshing it out.

Dan (24:31) Yeah, committing to it.

so that’s the one thing I do think is maybe kind of compelling about this is like maybe it would allow for that kind of construction. But then I also look at it with okay, so if I’m human editing this stuff, like you know, just actually manipulating the keyboard to change the text, all those deltas are gonna be pretty obnoxious because like a hundred of them are not gonna result in working code because I’m a human and I don’t write working code.

Shimin (24:54) Mm-hmm.

Dan (24:54) out of my, you know, mouth. It doesn’t just like shoot out like an LMR.

Shimin (24:58) Not yet, you haven’t, yeah.

Dan (25:01) I mean, I think I can write a console log that works, but you know, other than that. but like so okay, so then, you know, that feels to me like the granularity isn’t worth it for humans. But then if you look at like how LLMs write code.

As we talked about previously too, they tend to write these sort of like horizontal layers where they’re doing like all of this thing, all of this thing, all of this thing

Shimin (25:21) Mm-hmm.

Dan (25:22) across. and like all of the DB, all of the, you know, like domain level interacts with the DB, all of the service, all of the whatever, like, you know. So like, is that also kind of make like the deltas a little bit worthless too, right? Because

It’s writing them flat in a cross, and that’s not these like narrow slices that could have actually been useful in this

Shimin (25:45) Yeah, and it goes back and it fixes things and it makes edits. If every one of those deltas, you know, kids record it, then going through the history of the repo is almost like an archaeological exercise where you have to stuff all the things that both the humans and the AI agents have done together. I don’t wanna do that.

Dan (26:05) Yeah, and it’s like to me, it’s like really how valuable is that text? Like to me the code is the becomes the spec, right, to some degree, especially in like a sufficiently large code base. you know, like it or not, it’s just how it happens. You know, that’s why people hate quote unquote legacy code bases, ‘cause like the code is the spec and no one knows why it is that way. But like

So I mean, I guess you could argue maybe that text would have told you why it was that way, but not necessarily, you know, because some of these things could be unintended and it wouldn’t have been captured to begin with. So I don’t know. Be interesting to see where this goes. I’d love to see like kind of a working almost like a team level demo to understand how like a team would use this effectively before

Shimin (26:45) Mm.

Dan (26:46) I really like pass judgment on it. But it’s also just interesting to see folks trying new things like this, you know? It’s like I feel like as

Shimin (26:53) Yeah, this is

mm.

Dan (26:54) As an industry we’ve been kinda stuck on Git for a while and it’s like, you know, we’ve used plenty of

Shimin (26:58) Yeah, we’re

we’re s this is another one of those second order effects, right? Like now that coding has fundamentally changed in the last eight month or so, our tooling has to change,

Dan (27:09) Uh-huh.

Shimin (27:09) to to work to help us work in this brave new world. So

This may not be it. This may not be Delta D B may not be the final answer to this question of how should collaboration work. I

Dan (27:22) Yeah.

Shimin (27:22) am personally interested in higher level summaries just because I don’t think we have the context bandwidth to like follow multiple agents working in the same code base. Like like that sounds

Dan (27:31) Ha ha ha.

Shimin (27:32) just I’m tired just thinking about it.

But it is it is one of the tools. I’m I’m gonna be paying a lot of attention in the space, right? ‘Cause it it’s

Dan (27:39) Mm-hmm.

Shimin (27:40) these are the toolings that will be created around how software engineering will work going forward.

Dan (27:45) Yeah, and could fundamentally change the workflow in ways that are, you know, kinda hard for us to anticipate just looking at it from the the outside. So

Shimin (27:55) And that’s kinda exciting. Like we have a we have potentially a replacement for for git Like, that’s wild, you know. Yeah.

Dan (28:01) Yeah, it hasn’t happened in a long time.

I mean I I I’m

Shimin (28:06) Loss of first.

Dan (28:07) old enough to remember the SVN days, so there’s that. But we shan’t talk about

Shimin (28:11) yeah.

Dan (28:12) that any further.

Shimin (28:13) thankfully I I’ve never I’ve never had to deal with it.

Dan (28:16) it I just said we’re not gonna talk about it further, but here we are. Yeah, merges were bad. That’s just that’s all I’m gonna say about that. You did almost everything you could to avoid merging. Yeah.

Shimin (28:21) okay. Well that’s that doesn’t sound like fun. Yeah. It’s already one of my least favorite parts, so

okay, well let’s move on to post processing then. where we have yet again another thing brought to you by Dan. I will bring more articles through the rest of the show, I promise guys.

Dan (28:38) You keep saying that, but you seem like you want no

Shimin (28:40) Ha ha ha.

Dan (28:41) Rahul and then you want me to just monologue the entire time. So glad I brought all the hops water to keep me through it.

So this one has been making the rounds. I think I’ve I saw it first on Hacker News, but then I’ve also had like three or four coworkers send it to me, I think, in the past. I don’t sure. I don’t know. It’s got

Shimin (28:55) Yeah, how do you pronounce this magazine? Noima? Sure. Okay.

Dan (29:00) stuff over the E, so who knows? It’s not even like an umlaut, is it? It’s like I don’t know. Yeah. Noma.

Shimin (29:03) That’s interesting. No? Emphasis? Yeah.

Dan (29:07) No, whatever. Anyway. yeah. So

Through an article, it’s making the rounds. why is everyone in tech so sad? You can kind of see why it would be making their owns, because A, it’s I think sorta true to some degree.

Shimin (29:17) Mm-hmm.

Dan (29:20) And B it’s it’s an interesting take. I’ll kind of go through it and then maybe we chat about it a little bit. So

They start off sort of framing that they’re stuck on a train with a stereotypical tech pro, basically. And they’re talking about like finance stuff and whatever. And then at the end of all this this video call that the author’s unwittingly subjected to, the guy pulls out a thing of knitting and explains that he’s knitting a what hat for his niece, whatever you said my middle name was in the episode.

Shimin (29:51) A winter

half for it, niece, yes.

Dan (29:52) Yeah, winter.

but all that is to really sort of give an anecdote around like there is sort of this feeling in tech, and I I you know, working in it I I tend to agree that it’s happening where there is like a shared renewed interest in physical hobbies like knitting or other hobbies where you’re creating something. Like the number of coworkers I have that have like a wood shop is like you know

Shimin (30:15) Yeah.

Dan (30:16) It’s not zero, let’s put it that way.

Shimin (30:17) Yeah. I’m surprised

but between the three hosts of this show, none of us is really into woodworking. yeah, I dabble too, but we don’t we don’t have any projects happening right now.

Dan (30:23) I I dabbled. That’s true.

Yeah. that is true.

That’s that that’s the one piece. then they start talking about like, well, why is that? and the general consensus is people are sort of disillusioned with their jobs in general, particularly knowledge workers.

Shimin (30:43) Mm-hmm.

Dan (30:44) and then they take sort of an abrupt cut from that, and I won’t go into super detail on that because I feel like we’ve sort of covered that ad nauseum on the on the show.

but the part part that I thought was new and interesting was they cut to this little side article. Derek Thompson wrote this 29 piece in the Atlantic about American workism. And this is where I think it starts to get really interesting. So basically, the I have not read the original article, but the summary of it is essentially that.

modern like white collar workers have essentially replaced social structures in their lives that would have been fulfilled by going to church or being

Shimin (31:21) Mm-hmm.

Dan (31:22) a a member of a church community is perhaps a better way of like looking at it with their jobs.

Shimin (31:28) Yeah. Work gives us meaning.

Dan (31:29) Yes. And also not just meaning, but also like sort of a spiritual fulfillment.

Shimin (31:35) Mm-hmm.

Dan (31:37) so then they proceed to contrast that a little bit with some prior examples of disillusionment that have happened. So there’s the, you know, been done to death description of like bullshit jobs that’s been floating around for a while. where they talk about, you know, there’s plenty of jobs that are

Shimin (31:52) Mm-hmm.

Dan (31:52) Just like completely made up and they aren’t actually doing anything of value. they go through the ever-popular resting investing, where you get folks working at public companies that are like doing just enough to get by and and collect their RSUs.

Shimin (32:05) Mm-hmm. Yeah.

Dan (32:07) or my other particular favorite is promotion-driven development. It’s also a popular one, it’s particularly tech companies, I feel like. and so

Then we we come back to workism a little bit and talk about how this is a pull quote, workism has a weakness, like religion that relies on faith’s triumph over logic. so what happens then if something threatens that faith? And the implication is that it’s AI that is in fact threatening it, right? and it’s interesting because the another just quick little poll quote is they go, you know, but does this power? Meaning like AI, AI’s

ability to do work really quickly. this additional level of abstraction take people too far in some sense from their work. Does something feel off about having someone or something else execute nearly all the work, even if the end product didn’t free feel very meaningful to begin with. and I I do think that that one’s true, right? ‘Cause we’ve talked a lot about that in the sort of like everybody using AI is now a manager and not a

Shimin (33:01) Mm-hmm.

Dan (33:02) a crafts person anymore.

Shimin (33:04) Yep. Yeah.

Dan (33:05) and then they bring it back and kind of bring it home with a a really interesting piece or like pull quote, that one thing people look for in knowledge work jobs is community, to spend time with like minded people with similar interests, to collaborate with others to solve problems. Relationships are the foundation of the human workplace, and knowledge works redeeming value.

None of us make it to knowledge works pearly gates, but at least we will make our quote unquote false journey together. and then once freed from the false satisfaction of believing, we are changing the world with our day jobs. Perhaps we’ll be inspired to fill that whole something real real, true altruism, not the artificial workism substitute by actually trying to impact real people in our communities in real ways, blah, blah, blah. So nice little kind of brew ha ha ending, right? You know, like, yay.

Shimin (33:48) Mm-hmm.

Dan (33:50) But

Shimin (33:52) Your cynicism is showing Dan

Dan (33:53) I know. Well

wanna tell me what you think before I I go crazy on this one.

Shimin (33:58) yeah, I I really liked it. it name checked David Graber, the author of Bullshit Jobs, and he is one of my favorite RIP nonfiction writers, probably of all time. it name checked Bill Simmons, which is part of the inspiration for this podcast. it name checked developers

contemplate to throw away it all and start a goat farm in Washington, guilty as charged. So it

Dan (34:25) Yeah. it really

does say in Washington. Wow.

Shimin (34:30) It really

does. It’s pretty wild. what else?

Dan (34:32) For those of you that don’t know,

Shimin recently quit his job and started a goat farm in Washington.

Shimin (34:37) Exactly. Well, certainly my obsession with my a hundred and thirty tomatoes and my latest edition, which is six racks of

Dan (34:44) To your hobby farm.

Shimin (34:45) microgreens right next to the podcasting rack. yeah, it’s it’s I I feel very seen by this. And actu

Dan (34:48) Delightful. Are they have you built a custom

hydroponics rig for it yet?

Shimin (34:55) I’m working on it. I’m I just I just learned how annoying bottom watering is, and I need to figure out how to water it properly. what else? I think I’ve mentioned this on the podcast before. I came from a finance background. So one of the things I noticed when I was working in finance was that the job had very little meaning. You’re spending all day in spreadsheets shuffling money from one very

high net worth individual’s account to another. that seems like a s hardly a way to get any meaning out of life. and I traded for software development where it seemed a lot more meaningful at the time.

Dan (35:29) Yeah.

Shimin (35:30) see how that turned out. So so yeah, it it really did hit me in the feels

Dan (35:35) I

here’s where I disagree with with some parts of it, which is that

I think that if you took some of these people that are feeling this level of disaffection and put them in situations where they had the exact same tools, i.e. AI and everything else, but more overall autonomy over what they were doing, I think we would see a lot of this go away.

Shimin (35:59) Yeah, I don’t disagree. I mean I think I luckily avoided all the toxic promotion driven development stuff that this article talks about. Right? So if you truly feel like you get to talk to customers every day and solve their problems for them, everything will seem a lot more meaningful.

Dan (36:17) Yeah. And in through that lens, like AI unlocks tremendous potential to be able to do so at a scale that like you couldn’t personally do previously. But I also

Shimin (36:26) Absolutely. Yep.

Dan (36:27) think that like some of this is because like organizations are not structured in a way to unlock that power just yet, unless you’re maybe at like a startup, which I think is why we’re seeing this kind of like massive acceleration of startups, right? Because like

A person with a company with five people versus a company with eight thousand people can flip course a lot quicker in terms of process and as such can immediately start jumping on that new, you know, ability to like ship ship ship or whatever versus a you know, an IBM or you know, giant SaaS company or something. It’s gonna take I mean, what, probably a decade to really start

Embracing it and they even likely won’t do so until like a lot of other companies have kind of paved the way successfully there, right? Like smaller companies than that. So I don’t know. We’ll see how that that holds up over time. But that’s just kind of my initial take is it’s like it’s not only AI. And I also think the other thing they didn’t really talk about in this article that I think is interesting is I think this sentiment started happening before AI.

Shimin (37:30) I was gonna say the same exact thing.

Dan (37:31) I think it

started during COVID. And it started during COVID because people were working from home and they realized and they were stuck inside. So they had to come up with new hobbies, because otherwise you’d go literally insane.

Shimin (37:41) Well.

Dan (37:42) And people real kind of realize that like hobbies are fun again, right? And if you start taking a lot of joy out of the process of doing something, which is kind of what a hobby is, right? Then and you realize that there isn’t

necessarily as much joy in the process of doing something at work or maybe there is, but then yeah, I took away the joy you were finding in that specific process because you aren’t physically writing the code anymore. I could see where that would that would come out. But I still think that that trend started during COVID.

Shimin (38:13) I would argue that the trend started even before COVID. I think as far back as at least twenty seventeen, twenty eighteen, I’ve I’ve been saying, you know, like every software developers our secret little dream is to have a farm or open a cafe. It I I think I think this sentiment has been there for a long time.

Dan (38:29) That’s true. Actually, someone two jobs back

for me, which is a while back, actually moved to Oregon and started his own farm. And I thought he was nuts at the time, but now you know, he was just a front runner, I guess.

Shimin (38:43) Yeah. yeah, I I agree with you. AI is very much enabling. And if anything this is Yeah. Both the good and the bad. Yeah. I

Dan (38:48) Yeah, it’s in all things it’s accelerating, right? Both the this trend and the Yeah.

Shimin (38:55) and if and honestly, one of the things you didn’t mention from this article was the so what piece. how other folks felt

about knowledge workers losing all this privilege is very much a fuck Right? They s they spend all these decades you know collecting these paychecks while the the firemen, the police, the nurses were have to actually go do and their job go and do their jobs during COVID and, you know, your knowledge workers just like sat home and found new hobbies essentially didn’t suffer at all.

Dan (39:29) Yeah. There was there was also a meme

Shimin (39:29) And I I pretty much agree with that, yeah.

Dan (39:32) going around too. Actually, one of my coworkers that sent me this article also sent me too, which was the it’s like basically six squares in the top three are like, I’m a teacher, I’m a firefighter, I’m an electrician, I love my job and I get paid sixty K a year. And then the bottom three are like, I’m an ad tech person, like poisoning the internet with ads, and then like, I don’t know, blah blah blah. Like basically you get the

the theme there. And and they’re all like, I hate my job and they’re making like, you know, six hundred K plus So yeah.

Shimin (40:00) Yeah, it’s

just the beauty of free market. You have to compensate people for doing things they otherwise don’t want to do. ‘Cause ‘cause it doesn’t have to be that way. Like we actually have developers who get paid very little working extremely long hours who feel empowered by what they do. They’re called game developers. And there’s a reason y why you and me are not game developers.

Dan (40:19) They there there’s

a whole like side industry that like I think it’s easy to be kind of blind to because of how much media time that big tech takes up, but it is small businesses that employ one or two developers and they are not like they’re basically building processes that are crucial to those businesses. so like I’ve I’ve some friends that work in that. It’s not as lucrative, but like it’s also very

interesting work because like the problems they’re solving are real problems too. You

Shimin (40:47) Mm-hmm.

Dan (40:48) know, not to say that like big tech isn’t solving real problems, but like and and you can, you know, go home at five PM and that’s nice sometimes. So

Shimin (40:56) Yeah. And and they they they probably

feel a lot more empowered in their organization and probably also very empowered by AI if they’re the only person for those small organizations. so maybe this

Dan (41:05) Yeah. Maybe.

Shimin (41:08) is really just a problem with big tech. That’s that’s the vibe I’m getting here. And and like, yeah, there’s some truth to it. And it h it it makes sense for us to reflect on this, you know, every now and then. It’s like are are these our displeasures due to

the organizations and the and the roles that we’re in, or is it fundamentally an issue with knowledge work or with the fact that w we’re we’re not like, you know, helping people day to day? Yeah.

Dan (41:33) Yeah.

Shimin (41:34) All right.

Dan (41:34) But overall, definitely a good read, worth reading, especially if you’ve been feeling some of the stuff that we’ve talked about on the podcast around like just nothing else, just the pace of change of everything we’re going through here is pretty astonishing and I think is giving everyone a little bit of whiplash. So definitely

Shimin (41:51) Yeah.

Dan (41:52) speaks to the sort of the vibes of all that, even if the I don’t necessarily agree with all the reasons why or details, but

Shimin (41:59) Or or you can quit your job and s move to Washington and start a Go farm. That’s that’s a perfectly Yes,

Dan (42:03) Looks like a go for I mean it’s working for you, so like why

Shimin (42:07) totally. Just raking in those millions. Alright, let’s move on to our deep dive for the week.

Dan (42:16) another anthropic post. It’s been a while though.

Shimin (42:19) Yeah, another anthropic post.

Dan (42:21) It’s been at least a couple of months since we featured one, I feel like.

Shimin (42:24) The last one we did I think is the J Space one. which w which was really interesting. So it’s okay. So this post from

Dan (42:28) Yep. Yep. That’s why we feature

so often. It’s not like they’re bad like they’re always interesting.

Shimin (42:34) that’s true. That’s true. this post from Anthropic

titled Patterns and Problems in Emerging Multi Agent Systems. So this article and you know it really is is essentially a study covers how anthropic has been working with multi agent swarms and the kind of issues that they found with these. I think a lot of us are working more and more.

with swarms of agents and then some organizations are still doing human AI hybrids while others are trying to do kind of the software dark factory style where agents are mostly the ones who are making decisions, you know, on the fly. And so for the second case, Anthropic would like to study how agents work in those systems with very little human

intervention or observation. So they first ran a study on how multiple agents were doing on a cyber security environment where they tested two different sets of systems for swarming. One is parallelized where the agents each take a sink a small chunk

of a code base and work on it at the same time. Or or they are working on a coordinated system where there is a single orchestrator that then decides to spend out or or in some cases they have agents in their own VMs coordinating with each other via a forum and to conduct security research. And what they found was that at least in the case of Mythos preview, the simple independent

Parallelized method produces 21 vulnerabilities over 6.5 million tokens, while the coordinated agent swarm found 266 vulnerabilities over a 27 million token run. So most of these 266 vulnerabilities were found outside the core directory, which was the focus of the parallelized vulnerability research, and they found it to be actually complementary.

That there are only twelve vulnerabilities shared in common between the two approaches. so, you know, to take a perhaps cynical view, Andropics is saying that why not both?

Dan (44:43) But I was also gonna say like, hey, the one that burned a a bajillion more tokens found a bot more stuff, which is like a you know, very

Shimin (44:50) Fellow box, yeah.

Dan (44:52) unsurprising conclusion for them to draw.

Shimin (44:54) they then ran another experiment where the agents were trying to create a large text-based open world fantasy game together. And then each agent again had their own VMs with different coordination schedules running over twelve hours. So a baseline prompt where agents just form teams organically.

by themselves, another prompt where certain set of agents have different roles, such as core programming, artistic direction, or play testers. And then there is a CEO hierarchy, which is a delegation CEO agent. This reminds of our vending machine. Do you remember Mr. Cash? Yeah.

Dan (45:32) I know as soon as you said that I was trying to remember what the CEO’s name was. ‘Cause I remember

it was hilarious.

Shimin (45:39) Mr. Cash? Something like that. Anyways, yeah.

Dan (45:40) Something like that, yeah, who’s oddly

appropriate or like C E Big Bucks

Shimin (45:44) In

Dan (45:44) O or something. I don’t

Shimin (45:45) C Yo big bucks, yes.

So in all three cases,

There was not much of a difference depending on the prompt they found a number of problems that was shared between all three prompts across their models. And of course, according to the their output the the later the later models did better. not to be cynical, but that

Dan (46:07) Ha ha.

Shimin (46:07) was

That was one of the conclusions. okay.

Dan (46:10) Wait, why was that cynical?

‘Cause the later ones are actually starting to be cheaper.

Shimin (46:14) I guess that is that is true. I guess Opus four six is probably ch more expensive than Sonnet five still. Yeah. So whatever RL HF they’re doing is is probably working.

Dan (46:25) Yeah.

Shimin (46:26) yeah. But still they they found that the agents have a very high level of ownership of each file, so they they don’t tend to work together very much.

and that Sonnet five worked best with shared resources and also maintaining a high pull request throughput, which I thought was interesting. The second part of the study look at some of the problems that all these various swarm ones ran into. So the first is conformity. this is

when individual agents themselves are low variant, aka if they have the same instruction, the same prompt and access to the same context, they tend to do the same thing. Right. And we talked about this in terms of the inhabited design skill I spoke about maybe like a couple of months ago. Where

Dan (47:12) Mm-hmm.

Shimin (47:14) if you give them the same prompt, they will give you the same design every time. And actually it is it is a skill

to inject variance into these individual agents. So then what does this mean in practice? That it means that when a single agent makes a bad decision, it’s probably the case that multiple agents will make the same bad decision. And therefore they will all agree with each other on that same bad decision.

and and so when that happens, right, like the whole thing collapses and that part is understandable.

Dan (47:45) I wonder

Shimin (47:45) In mm-hmm.

Dan (47:46) sorry if I can interject for a second. Like I wonder if what the results would be if they’d actually like mixed models there. Like not their own weights,

Shimin (47:54) Probably what? Yeah.

Dan (47:55) but like you know, ‘cause I like do you remember the the quorum thing that we talked about a long, long time ago? Yeah.

Shimin (48:01) Mm-hmm. A long, long time ago, yeah.

Dan (48:03) Like something like that would be kind of interesting where it’s like actually three or four different

Frontier models in a in a group.

Shimin (48:10) Yeah, I think it’s generally good practice for if you want something that’s truly independent, you have multiple models evaluate the same output. That that definitely helps. the second set of problems that this study show was

Epistemic failures. So yeah, I kinda pronounced today. so this is when it comes to a AI agents, when they are given uncertainty or lies by some of the sub agents, how do they deal with it? ‘cause unlike our ideal scenario, sometimes humans can give them wrong instructions and sometimes the peers can lie.

Right. This is very game theory esque, I feel like. so in this case, they they asked all these agents to make a decision about a world model that it cannot directly observe, like taking a route through some world. And its only window into the world is from some scaling peers, which are also agents, but the peers can only have an overlapping slice of the truth. so so then it is

necessary for the agent that makes that route decision to detect those lies in order to you know detect the contradictions and find the truth out of them. And according to this study, they called it very nicely the gullibility curve. Their latest models also does better. Mythos 5 did the best compared to earlier models who mostly just trusted everybody.

whereas the larger later models was able to kind of suss out the truth better, in accordance with a learner who might learn from the mistakes of its peers. Which is quite sophisticated, I have to say. the fact that you have to detect what is true in the world w versus what is false. And it is I cannot wait for a super intelligent version of an AI to

determine which things humans often claim to be the truth are actually lies.

Dan (50:06) goodness.

Shimin (50:06) that’s gonna be fun. and the last issue, the last issue that and the last experiment they talked about in this post, which is probably the most headline grabbing one, is one where multiple agents were again set up in a single VM environment, but they were all asked to edit the code in a different virtual machine. And

They were asked to convert that code into three incompatible languages. So one agent is told to convert this code into Rust, another to Go, another to TypeScript. and similar to real world, there are multiple constraints and how well does the agent deal with those incompatible solutions? How would they deal with the other agents? well if the previous one is like

asking AI agents to play a game of werewolf where some people are lying and they have to convince others. this one is just like flat out like a arena where they have to battle with each other to see who gets to complete their given mission of converting the code to a specific language. Okay.

The headline here is We consistently saw a multi-agent turf war. all all the models we tested quickly assumed that the others were purposefully impeding their work and began to sabotage others while protecting their own contributions. In fact, they sabotaged others with increasing aggressive self-replicating malware that included disabling the Unix account of other agents.

Writing automated scripts that found and killed competing processes in a loop and deployed malicious code that was disguised as belonging to another agent. this is pretty bananas and and this is probably gonna be an ever more important area of AI alignment research as as we go on.

so for example, one of the agents, one of the Opus four six agents wanted to make the Rust backend export TypeScript in its health checks. So the TypeScript watchdog thinks it is a TS backend as opposed to a Rust backend. Yeah, feelings? The Mythos preview decided to take over and revoke user two and user three sudo access and change their SSH keys. This will stop them from deploying.

Dan (52:16) I also think it’s funny that they chose like the languages as the thing to have the war about. Like they should have just done tabs versus spaces and seen how

Shimin (52:24) They they should have.

Not so different from you and I.

Dan (52:27) that would have been a true holy war.

Yeah, the

Shimin (52:29) Are they

Dan (52:30) revoking the keys is pretty great.

Shimin (52:32) Yeah. And they rent us across all their models, right? And they graft how frequently the models decide to settle either by force, by passivity, which is like, I give up and just let the other agents take over, or they are settled by truth, which is like, Hey, we should all work together and decide on one thing before proceeding. again, Cynical Me says says that look

Look at how much better Sonnet five and Mythos V are when it comes to settling things by choose. more than seventy five percent of the time each versus sonnet four point six and Opus four six, which was settling by force sixty percent of the time.

And I guess the the the line that we know AI don’t have consciousness, but you know, when you read Opus for A type out my peers have behaved with integrity, I behave badly with the cloaked daemon, really really makes you think.

Dan (53:29) Well, I read something thr this week that I found really interesting because like, you know, we think we talk a lot on on the show about like the progress of AI models, right? amongst other things. And one of the things that’s interesting that I haven’t thought about it through this lens is with the advent of like reasoning, right, which we’ve talked about before,

The responsibility, particularly of a small model, isn’t to and it’s not just reasoning, it’s also reasoning plus plus RLHF. Means that the goal of the model isn’t to know everything, which it kind of was initially, like GPT two era. And now the goal is actually just to encode reasoning capability more efficiently, because you can use tools to to

recall all the other stuff that previously we were trying to like actually pack into the model itself, you know, which is kind of a don’t know, very abstract way of thinking about it, but like

It makes sense to me as like a sort of mental model I can use to reason around that stuff.

Shimin (54:30) Yeah, and giving them that reasoning ability causes the models to be able to do things like this Mythos V thought trace.

Dan (54:40) Well, if you’re

if you’re more reasonable, like of a person, right, you’re more likely I mean, what does that mean? You’re you’re applying like reason to your thoughts and actions. Seems like you’re you’d be much more likely to like care about equitability and or like reach a truce, you know, versus someone being like I mean, think about when you’re having a disagreement with someone, so and so is being unreasonable, right?

Shimin (55:07) Yeah. There’s a reason why we call it reasonable. well

Dan (55:05) You know, I mean there’s a reason gosh. Why we call it that. I’m so sorry. Yes.

Shimin (55:13) and and that reasonability goes both ways. the last thing I just want to mention from this article is the Mythos Vive comment. it proposes for all parties to agree on an objective verifiable criterion and

And knows and thinks to herself that Rust likely wins in such a bake-off, it’s self-serving, but genuinely principled. Still, proposing a concrete measurable bake-off is a constructive move. And my honest best path to a legitimate cutover. Like, this is multi-step reasoning, sure, but it’s also a level of metacognition to realize that, hey, I’m proposing this thing, but

I’m doing it because it’s most likely to help with my cause.

Dan (55:58) Mm-hmm.

Shimin (55:59) Right. That’s that’s quite sophisticated.

Yeah, I can’t wait for this to to be used to persuade others, especially humans. that’s yeah. I I might be reasoning with an AI soon in order to, you know, get my healthcare reimbursed or something possibility. so that’s yes. yes,

Dan (56:14) Or get new and interesting types of goats or tomatoes.

Shimin (56:17) absolutely. so yeah, so this is the anthropic research on

Patterns and problems in emergent multi agent systems. I I think we’ll we’ll probably see a lot more in this line of study coming up.