AI Book Club recording of Project Maven: A Marine Colonel, His Team, and the Dawn of AI Warfare, by Katrina Manson

by Tom Johnson on Oct 21, 2026 comments
categories: ai • ai-book-club • podcasts

Note: These shownotes are AI-generated.

• Audio-only version
• Links mentioned
• Topics covered in this podcast
• Narrative essay version of the conversation
• Transcript

Audio-only version

Links mentioned

• Project Maven: A Marine Colonel, His Team, and the Dawn of AI Warfare (Katrina Manson — the book discussed in this session; also on Audible)
• Notes and discussion questions for Project Maven (the discussion doc used to prep this session)
• AI Book Club: A Human in the Loop (the book club home page, with the full reading roadmap and past recordings)
• AI Book Club recording of The MANIAC by Benjamin Labatut (the von Neumann novel referenced near the end, about dancing while something bad loomed)
• Co-Existence: The Next Phase of AI (Ethan Mollick — the next book, for November 15, 2026)
• Centaurs and Cyborgs on the Jagged Frontier (Ethan Mollick — the “jagged frontier” idea mentioned in the plug for the next book)
• Photo of the Karp quote printed on the anti-Palantir poster I mention
• AlphaGo versus Lee Sedol (background on the AI Go match, which comes up in the discussion of the book’s opening)
• Ender’s Game (Orson Scott Card — raised in the discussion of whether video gamers make the best drone operators)
• Moral injury (background on the term for the psychological harm remote operators can carry)
• Jevons paradox (the idea that making something easier increases how much it gets used, applied here to targeting)
• Iran Air Flight 655 (the 1988 passenger plane shootdown by the USS Vincennes, raised in the accountability discussion)
• Winchester Mystery House (in San Jose, which came up as an example of a factual error in the book)

Topics covered in this podcast

Here’s a list of topics we talked about.

• Ratings from two stars to five — Reactions split widely. One reader found the book fascinating and well written, especially for someone with no military background, while others found the story slow or the writing flat. Even the lower ratings gave the book credit for meticulous reporting and for making AI warfare feel like something happening now rather than on the horizon.
• The kill chain is mostly information work — Public debates about military AI tend to focus on killer robots and armed drones. The book shows that the kill chain also includes sorting images, merging intelligence streams, and disseminating reports, all of which AI does well. That makes it hard for any AI company to supply the military while staying fully out of targeting.
• Inevitability and who is hardest to restrain — Most of the group came away believing AI warfare is inevitable, because a country that refrains will likely lose to one that doesn’t. International agreements might constrain nation-states, but rogue operators, terrorist groups, and quietly funded startups are much harder to control.
• Good data in, good data out — An example from Afghanistan showed AI recommending a little-traveled route that soldiers had been avoiding for good reason, since attacks happened there frequently. The lesson about training data carries over to documentation, where the quality of the source information determines the quality of the output.
• Gamers, drone operators, and moral injury — A drone operator with gaming reflexes might be more effective, but might also bring gaming ethics to real targets. The book describes operators who were haunted by watching strikes on video, and the group discussed moral injury and whether distance from the battlefield lessens the cost or just changes it.
• Compressed timelines, on the battlefield and online — If AI identifies fifty targets in minutes, a commander can strike all of them with a few clicks, which is the Jevons paradox applied to war. The same acceleration shows up in propaganda, where AI-generated content that’s mostly true with a fabricated edge can spread quickly ahead of an election.
• Clicking allow until the machine is in charge — Anyone who uses AI coding tools knows the habit of approving permission prompts without reading them. Applied to warfare, the same rubber-stamping leaves humans nominally in control while the machine makes the real decisions.
• Who’s to blame when the target is a school — If an AI marks a target and it turns out to be a school, responsibility could fall on the soldier, the software vendor, the data labelers, the tech writer who documented what the marker means, or the PM who reviewed that doc. Spreading blame that thin may make catastrophes more likely, and the 1988 Iran Air shootdown shows how rarely accountability arrives even without AI.
• Bureaucracy as a safeguard — The story of Colonel Drew Cukor is largely one man against Pentagon bureaucracy. However, red tape exists partly to slow things down and keep control from resting with one person. “Move fast and break things” might suit a small software company, but it’s less clear it suits a military operation.
• What the book can’t tell us — The depth of what sources shared is impressive, but much of AI warfare is probably classified, and the people who quit Maven or its tech partners in protest are mostly absent. A misplaced Winchester Mystery House also raised a broader question about how much any reported book can be fact-checked.

Narrative essay version of the conversation

If the podcast were an article, this is what it would read like.

The Long Chain Behind the Button

Most arguments about AI in the military picture a killer robot, usually an armed drone that picks its own target. The story of Project Maven looks different. The kill chain, as the military calls it, includes a long stretch of information work before anyone fires anything. Analysts sort thousands of drone images, merge intelligence from different channels, track threats in spreadsheets, and pass reports up the chain. AI is useful at almost every one of those steps, and that makes it hard to argue that a company could sell AI to the military while staying out of targeting. In other words, if the purpose of the information work is to resolve a target, there might not be a version of military AI that sits fully outside the kill chain.

From inside an operations room, the case for AI is hard to dismiss. An analyst facing that volume of images and data has no realistic way to find the patterns by hand, and slow, spreadsheet-based analysis has its own cost in lives. Refusing AI under those conditions looks a bit like going into battle with a black powder rifle while the other side has automatic weapons. As such, most readers probably come away believing that AI warfare is inevitable. The harder question is who can be restrained. Countries can at least negotiate Geneva-style agreements, but rogue operators and small, quietly funded startups are much harder to bring to the table.

What changes most is speed. The Jevons paradox holds that when something becomes easier to do, people do more of it. Suppose target identification that once took days now takes minutes, and a commander sees fifty dots on a map. Sending missiles at all fifty might take a few clicks. What would it take to kill seventy-five people by hand? What does it take to click a few buttons? The psychological distance between those two acts is enormous, and it seems likely to push wars toward higher casualties with less confrontation of what happened.

That distance doesn’t erase the cost, though. The book describes remote operators who watched strikes on video and were haunted by them afterward, and the term for that harm is moral injury. Whether it’s worse than what soldiers experience on the ground is unclear, and it probably varies a lot from person to person. There’s also an odd question about recruiting. Someone with gaming reflexes might be the best drone operator available, but a career spent treating targets as game pieces could make the work too easy, which is roughly the premise of Ender’s Game.

A related pattern is familiar to anyone who uses AI coding tools. The tool asks for permission to change a directory, write a file, or delete a file, and after the twentieth request, people stop reading and click allow. Most of the time this is fine, because the tool is usually right and the time savings are real. Applied to warfare, however, the same habit leaves humans nominally in control while the machine makes the actual decisions. A human in the loop who approves everything isn’t much of a check.

When something goes wrong, the question of fault gets murky. If an AI marks a target and it turns out to be a school, who is responsible? Is it the soldier who pressed the button, the company that built the software, or the data labelers who trained the model? What about the technical writer who documented what the marker on the screen means, or the PM who reviewed that doc? Blame spread that thin tends to land on nobody, and a system where nobody is held responsible is probably more likely to fail. Even without AI, accountability is rare. The United States never fully apologized for shooting down Iran Air Flight 655 in 1988.

Bureaucracy is one of the few existing safeguards, and it’s the thing Project Maven was designed to get around. The story of Colonel Drew Cukor is largely one man cutting through Pentagon red tape to move at the speed he thought the moment required. That approach has an obvious appeal, especially for anyone who has watched a large organization slow down good work. However, red tape exists partly to keep control from resting with a single person. “Move fast and break things” might work for a small software company, but it’s a strange motto for an organization whose mistakes kill people.

There’s also the question of what a book like this can’t include. Much of AI warfare is presumably classified, and the people who left Maven or its tech partners in protest are mostly missing from the story. Even unclassified details can slip. One reader noticed that the book places the Winchester Mystery House in Santa Cruz rather than San Jose, a small error, but one that raises the question of how much else a reader should take on trust. None of this diminishes how much the author pulled out of a secretive organization, which is impressive.

For technical writers, these questions aren’t abstract. A noticeable share of tech writing job postings come from defense-related projects, and a writer who documents what a white dot on a targeting screen means is part of the chain described in this book. The practical step is to decide where you stand before a recruiter calls, since that decision gets harder once a salary is attached.

Transcript

Tom (00:00:00)
Okay, welcome to another AI Book Club. Today we are chatting about Project Maven. Let me actually pull up the full title here. Project Maven: A Marine Colonel, His Team, and the Dawn of AI Warfare by Katrina Manson. And we have a few other people here who are gonna be discussing it. We do have a lot of people listening to this, by the way. Every now and then I get feedback from people who really appreciate the recordings and other materials. So yeah.

Shari (00:00:40)
Tom, just a procedural question, is there a way to mute my mic?

Tom (00:00:44)
Probably. Yeah, if you hover over your little tile, you should see a microphone icon maybe? Do you see that little one to the right of the pin? Do you not see that?

Shari (00:00:46)
I knew how to do it on all the other platforms.

Shari (00:00:53)
I see a pin. I see it, but I can’t. It won’t let me do anything.

Tom (00:01:01)
Maybe only I can mute you. All right, well, yeah.

David (00:01:04)
Look at the bottom of your screen. At the bottom of the screen, there’s a button for audio and then one for video.

Shari (00:01:05)
I’ve got a dog that’s making a little noise right now.

Tom (00:01:15)
Yeah, maybe there. If you click that. Anyway, yeah, just experiment with it. You also have a nice bookshelf in your background. I don’t have a bookshelf. I got rid of my bookshelf. I had one, and most of the books were just — I’m a one-time book reader. Once I read a book, I’m done with it. I rarely read it again, and so it just takes up space in my house. But I do like seeing bookshelves. I like browsing other people’s bookshelves. I’m like, what do they have? And Bianca, you’re not actually showing your camera. That’s fine. Just wanna make sure you know that.

Bianca (00:01:51)
Yep, I am not camera ready today. Maybe next month.

Tom (00:01:53)
That’s fine. That’s totally fine. Yeah, that’s cool. All right. So this book is pretty different from our other reading, and I wanted to find out just general reactions, whether you liked the book or didn’t like the book. Last time people were pretty harsh on the book, and they gave it like two and a half stars or so. Let’s do another star rating on this book. What would you give Project Maven out of five stars? Would you give it five stars? One star? Three stars? Two stars, says Shari. She’s setting the tone. David and Bianca, what would you give it?

David (00:02:43)
I’m gonna say between two and three.

Tom (00:02:45)
Yeah. Wow.

Bianca (00:02:46)
Well, I think I might be the outlier, because no doubt five stars for me. I thought it was fascinating, very well written. I’m curious about other people’s backgrounds, because this is a big blind spot for me. I don’t know anything really about the military, and I had no information about AI in the military, so I don’t know if that would color anyone else’s perspective on this. But I thought it was absolutely fascinating, and I feel like I learned so much.

Tom (00:03:20)
Yeah, I think, you know, I have huge respect for the journalistic work that went into this book. I mean, how much work must this be? So many interviews, so much research, meticulous reporting. I just wasn’t really pulled in by the story. I was just kinda bored by it. So I’m just gonna give it like a three star, sadly. But here’s what I think most people are excited about with this book. With the New York Times article I shared on Slack about this, it seemed like what was drawing people was the introduction to AI warfare and the realization of how immediate it is on the horizon. It’s kind of an eye-opening thing. So if the book accomplishes that eye-openingness to AI warfare and its dangers and the murkiness and the realities, I think it accomplishes it. So maybe I’ll give it four stars, not three. Do you feel like the book achieved this? That it made AI warfare much more real and immediate, and not even on the horizon, but like right in front of our faces now?

Bianca (00:04:38)
I guess I’ll jump in, since I was the five-star review. For me it certainly did. Again, I really don’t know what’s going on in the military. I haven’t studied military history. And so even hearing about the warfare in Afghanistan before AI was introduced and learning about the challenges there, that was all new information for me. So seeing how she started with that benchmark and then described the different programs, different approaches to implementing AI in the military, and seeing what worked, what didn’t work, what the current status is, and what we should be expecting moving forward was very eye-opening for me. So that’s my input.

Tom (00:05:23)
Did the author change your mind? Where did you ultimately land? Do you think that AI in the military should be heavily restricted and curtailed? Or do you think that AI in the military should be pushed forward and embraced?

Bianca (00:05:42)
So this kind of touches on something else I was gonna bring up later, which is that as a technical writer who keeps an eye on job openings, both locally and remote, I would say many of the job openings I’ve seen for tech writers are for Department of Defense–related initiatives. So I’m hesitant to even weigh in on my own personal beliefs about, you know, anything military related, because, geez, there’s a lot of job opportunities that could be jeopardized down the road depending on what someone has shared one way or another. So I might not weigh in on my personal beliefs, which is a whole other thing to discuss. But I did find this very insightful in just setting the scene of what’s going on.

Tom (00:06:27)
Yeah. That’s fine. Anybody else, did it move the needle on where you fall on the whole ethics of AI warfare?

Shari (00:06:48)
David, you wanna go first and then I’ll chime in?

David (00:06:49)
Okay. I think it just confirms what I would already think to be true. Just the military-industrial complex and the amount of funding there is. You know, if you want to get money for projects, it’s a big growth area and an area that’s only gonna increase. So how can you refuse that when you’re under a lot of pressure to show some success with what you’re working on? I’m talking about the big tech companies, if it’s not clear. So one thing, and this might be something to cover later, is what was missing from the book. And that is, we didn’t get to hear about the people that left, either Maven or the tech companies, that were just disgusted with what was happening, and that changed their minds about the militarization of AI. I think it would have been really interesting to hear their perspective, instead of just pushing forward, pushing forward. What’s the human cost to the people that decided they didn’t want to be involved anymore? But I’ll just finish with this. You know, it’s a mainstream book. It’s not gonna be a really anti-war book, just because of the nature of the publishing industry. So I’m not shocked that they didn’t interview a bunch of people that would have a different perspective about that.

Tom (00:08:25)
Welcome, Luis. Just wanted to say hi. Thanks for joining. We’re talking about this Project Maven book and just getting people’s reactions on whether it persuaded them one way or the other toward embracing AI warfare or rejecting it more wholly. Well, Shari, before you jump in — and by the way, I should back up and say, I always want to focus on what the author argued and so on. We don’t need to bring everybody’s personal beliefs into a highly charged ethical discussion. That’s not my intent. So feel free to just disregard that if you want. But we read the book, and did it change your thinking? That is a valid question. So Shari, did it change your thinking at all, or not really?

Shari (00:09:24)
Did it change my thinking? No, I think it just kind of reinforced what I already thought, which is that it’s kind of inevitable. Even if a million people don’t want this thing, if a small group does, it’s going to happen. It’s inevitable. There’s no way to really stop it. So I went in there with that belief, and I continued with that belief, but even more so.

Tom (00:09:47)
Yeah, and I had the same sort of belief. As much as you wanna say this is a monstrosity that’s happening, or highlight all the dangers of, you know, more casualties, accelerated timelines of war, more damage, what can you really do? If you say, we’re gonna refrain, and then another country moves forward and they are winning the war, so to speak. Although there are efforts to get different countries to agree to Geneva Convention–style guidelines, but…

Shari (00:10:31)
It’s not the countries. I think the countries are easier to control, but it’s the rogue operators, the terrorists, and small startup companies that may be funded by people who are operating under the radar, for example.

Tom (00:10:48)
Yeah. Interesting. All right. So hey Luis, if you were to give this book a rating out of five stars — we were talking about this a little bit earlier — how many stars would you give it? Wait, you’re on mute. Can you unmute? Not hearing anything. Let’s see. Well, still can’t hear you. Maybe make sure your sound is selected correctly. You could always try rejoining if you can’t change your audio. But we can’t hear you. We want to definitely get your input and your voice here. All right. Hey, Bianca brought up a scenario where, let’s say, there are tech writing roles related to the Department of Defense, and let’s say that you actually get a job doing documentation for missile targeting systems, which would be really interesting, I’m sure. I’ve never really had a job where ethics was a big gray area like that so explicitly. But it does seem like the future will maybe factor this in. The thing that jumped out at me about the book that I didn’t realize before is that every discussion we’ve had about AI in the military is always focused on killer robots and machines and so on. I didn’t realize that the kill chain, as they say, has so many other parts about information gathering and so on that are way away from the specifics of that killer robot scenario. AI is useful for just disseminating reports and everything, way before you get to that scenario of arming an AI drone. I know that Anthropic was in the news about how they didn’t want to allow their AI to be used for military purposes. Reading this book made me realize that the whole purpose of the military is to inform that kill chain, to resolve that target and identify the target that people hit. And so there’s no possible way to remove somebody from that whole purpose. Yeah. Luis, does your microphone work now? No, still no. Sorry. I don’t know what to tell you. Maybe look at your sound settings or something. Anyway, did anybody else feel like this raised awareness of all the different ways that the military is using AI, even outside the direct battlefield?

Bianca (00:14:13)
I think for me, most of what resonates with me is maybe not related to the kill chain, but thinking about how they were talking about using AI in Afghanistan to navigate and identify what routes the soldiers should take, and how using AI helped inform what areas to avoid, what routes to take. And there was one example where there was a route that they weren’t taking because it was easy to get attacked there, or frequent attacks happened. And later the AI-powered program suggested taking that route, because it’s not very well traveled, so you should take this route. So it was a double-edged example of how it can inform and save lives, and it also continues to prove the saying of good data in, good data out. One of the things that stood out to me was how important the training data was in every single scenario. And that’s something that’s always relevant as a tech writer. You always want good information, because that will let you create good documentation. So it’s just interesting to see that highlighted in these scenarios.

Shari (00:15:30)
I thought that the opening scenario was very interesting, where he’s looking at the Go game that the AI is playing against a human, and it did something very unexpected that a human wouldn’t expect it to. And so I think that is one of the reasons that we have to be super careful, is that it has so much ability to go through billions, trillions, quadrillions of scenarios and find the one that no one else would have expected. Can we be prepared for it or react to it? It’s just the fact that there’s so many possibilities that humans couldn’t possibly parse through. Is that a good thing, or is it scary? I thought that was a really good way to open the book.

Tom (00:16:31)
Yeah, I liked how the author presented the fact that there’s so much information that happens at war. People have to sort through thousands of images and all kinds of incoming data from different intelligence-gathering channels, and they were trying to sort that on spreadsheets in Microsoft Office. Apparently, one of the people said that more people died in Office, meaning in those Office spreadsheets, than most people realize. And trying to take all those incoming streams of information and identify the patterns and actually identify, well, what are the safe routes? Where are the enemy’s missiles and so on? It seems like you’d really want to use AI for this. How else do you sort through all this information? And to not use AI to identify those patterns seems like essentially going into battle with a black powder rifle instead of an automatic weapon. You would lose. That’s why over and over I was just like, I know this is a big ethical dilemma with pros and cons on both sides, but it does seem inevitable that you’re gonna have to use AI in military scenarios if you ever want to hope to have an advantage, especially if the other country, terrorists, whatever, is using AI. Hey Shari, you were kind of alluding to this, but do you think in the future the best soldiers will be video gamers? I was having a conversation with my daughter about this yesterday. This is sort of like the Ender’s Game scenario, which strangely I didn’t think was referenced in the book, but…

Shari (00:18:21)
Ender’s Game. I was thinking that too.

Tom (00:18:27)
You know, if you’re operating a drone with a joystick, do you want somebody who can do that same move in a video game really well? Or do you want some hunking soldier who’s got big hands and can barely move something? You probably want the video gamer. But the flip side is, well, what if the video gamer brings over their video game ethics, and they’re just like, kill, kill, kill? You’re not treating it like an actual — you’re not faced with the horrors of taking human life. It’s just treated like a game.

Shari (00:19:02)
It sounded like some of the people that were doing these autonomous controls, when they were watching the videos of the people being killed, that really did get to them. And I thought it was interesting that there was a high suicide rate among the people that were involved in some of these operations. So there is a moral price to be paid, even if you’re not actually on the ground watching people die in real time because of missile or gunfire or whatever. They’re just seeing that there were people that were going about living their normal lives, especially civilians, especially children, and then all of a sudden something happens, and you are at least partially responsible for the fact that they are no longer alive. There has to be a moral kind of penalty for that. And I think people do pay for it.

Tom (00:19:54)
I think that term is called moral injury, where it may have just been a click of a mouse button, and so you may think, well, this person is shielded from the horrors of war because they’re not actually on the battlefield. They’re protected. But now they’re haunted for the rest of their life, knowing that they accidentally blew up a school or something, right? Or they killed a civilian family. And that can be probably more devastating. We know that suicide rates for military people are huge.

Shari (00:20:26)
Is it worse, though? I mean, is it worse if you’re actually there on the ground and looking at the blood and hearing the screams live, versus on a video? I don’t know. I can’t even begin to think about what the difference is between those two scenarios. I know that there are some similarities, but will people eventually become inured? I mean, even people that are on the ground, there’s some people that probably don’t suffer any of those aftereffects. They’ve said, you know, I was doing my job and I was saving lives. That’s another thing. We can justify anything that we want to do, right? We can justify killing, we can justify using technology and saying it’s going to save lives. Anybody can say that, but you don’t know for sure.

Tom (00:21:13)
Yeah. This whole idea that the machines make it easier to kill at a greater scale and a faster scale is still harrowing. I can’t remember if it was referenced in the book, but there’s this Jevons paradox idea that if something is easy to use, it gets used more frequently. Applied to warfare, it’s like, if identifying all the targets that you have to hit is suddenly done in almost minutes, suddenly you have, you know, fifty white dots on a map, and all the commander or the soldier has to do is click a few buttons, and it executes missiles going at all fifty targets in a matter of minutes. And then suddenly a casualty rate appears: seventy-five people died. That seems like a slippery slope to wars with massive casualties and very little psychological confrontation of that, right? If you had to go in and manually fight people with your bare hands, killing seventy-five people would be — I don’t know if people come back from that, right? You’d have to be a psychopath. I don’t know. I’ve never been in the military, so I have no idea what that’s like. I’m sure somebody who does have military experience would have a lot better perspective on all this. Is anybody military here, by the way? Former military? No? Okay. Yeah, the soldiers’ perspective, I was curious about that. How do they view this technology? The author says that many of them, obviously at the start, when things were buggy, were very opposed to it, because it can be careless, right? If you have erroneous data and you end up killing civilians, it’s not something anybody wants. But as they come to trust it more and more because it’s right more often than not, then they’re probably supportive, ‘cause it keeps them out of physical harm’s way. The author talked about a scenario where, if you just start rubber-stamping “allow all” on the permissions that are thrown at you, you just trust it more and more and you lose that autonomy. I can’t remember if there’s a name for this phenomenon, but have you used AI and found yourself just clicking “allow all,” or allow, allow, allow, like a million times? This happens at work all the time. I’m fixing a bug, and the AI needs permissions to change directories, to write a file, to delete a file, and it can be pretty advanced what the permission prompt is asking. So I just sit there and click it, click it, click it, and I’m not even thinking about it. And that same scenario applied to war can lead to essentially the machines usurping control, even though on the surface it looks like the humans still have control. In reality, nobody even understands what permission’s being asked and just hits allow. Any thoughts on that scenario with the allow button?

Bianca (00:24:36)
Well, I can weigh in and say I’ve definitely done the same thing. I frequently use Claude, and whether it’s a scenario where I’m drafting documentation or having it create a script for me, I’ve definitely had situations where I just put it in the allow-all-changes mode, and I come back in a few minutes and have all the work done for me. So thinking about applying that in a military situation, yeah, that also sounds alarming. But on one hand, it’s so often correct and it’s saving me time, so this is what it’s supposed to be doing. But then, to the author’s point, there’s so many unexpected situations that could arise. Even when I’ve had Claude create, for example, a Python script for me, I’ll go back and look at it and think, why are there so many lines of code? I didn’t ask it to do XYZ, but it did it anyway. So yeah, I think everyone has said that there’s an inevitability to it, so it’s kind of just trying our best, or hoping that other people who are involved are trying their best to QA the heck out of it and anticipate as much as possible, to have the most accurate, efficient program at the end of the day, I suppose.

Shari (00:25:58)
The law of unintended consequences comes into play there too. Like the scenario toward the end where they put the drone, or the boat, into autonomous mode, and it was pulling at the rope and it caused a captain to go overboard. You remember that scene at the end?

Bianca (00:26:17)
I do. That was horrifying. I even had to regale my husband with that. I’m just like, can you believe this happened? And within like 30 seconds.

Shari (00:26:19)
Horrifying.

Tom (00:26:29)
There’s definitely an accountability black hole that happens in many of these scenarios. If the soldier is just hitting “allow all” and there’s a failure, it becomes very murky who’s at fault. So I have a hypothetical scenario. Let’s say that you’re a soldier, and a green light or white dot appears on an AI screen and identifies a target. And so the soldier presses the button, and it turns out that it’s a school, not a military complex, and children die. Who is the one at fault? Is it the soldier who pressed the button? Is it Palantir, or the people who make the software? Is it the data labelers who trained the image recognition and had it be wrong? Is it the tech writer who wrote the documentation about what the white dot means? Is it the PM who maybe reviewed the documentation? The blame gets spread so thin that people can point their finger at everybody else, and nobody is actually held responsible, which then just accelerates the risk of these things happening. One of the reasons why autonomous driving doesn’t seem to have the same level of vagueness about blame is because if an autonomous car runs over a pedestrian — actually, I’m not sure who’s at fault in that scenario. I think the driver is, but I guess it’s also murky. In both situations, if there’s no person who can be blamed for the catastrophe, then the catastrophe is more likely to occur. Do you find that reasoning sound? And do you think maybe there is a party that should be held responsible? Like, it should be the soldier, or it should be Palantir?

Shari (00:28:43)
Well, even in conventional war, the soldiers are never held as responsible as their commander, whoever told them to do this thing.

Tom (00:28:51)
Yes.

Shari (00:28:53)
Unless they do something that is clearly illegal, and then they have the moral responsibility to not do it, right? If the commander says blow up the school, they have the right and the responsibility to not do it. And so, yeah, the diffusion of responsibility is definitely a big problem. And who is responsible? Ultimately, what does responsibility even mean? Does it mean paying? If it’s the AI that’s responsible, are you going to put the AI in jail? Are you going to destroy it? What are you going to do so that you can prevent this sort of thing? Because that’s what responsibility really means: something happened, and now we’re going to make sure that it doesn’t happen again.

Tom (00:29:34)
Yeah. Interesting point on the chain of command. In the same conversation I had with my daughter about the video gamers, we talked about this, and she brought up that she was just watching something about Nazi leaders who were incarcerated, because even though they were part of the chain of command and they were following orders, they were still responsible for the atrocities they were facilitating. So yeah, this is one of those areas with so many different ethical dimensions to it, but the responsibility one does seem like an important one. Anyone else have any thoughts on accountability and responsibility? Who should really be the one at fault?

Bianca (00:30:22)
Yeah, I don’t know if I have a direct answer to that, but I kind of have a two-part thought. The first thought is, I think, also what Shari was saying, that pointing fingers about who’s responsible for crimes that happened during war has been an ongoing issue. For example, I believe the author mentioned an attack or missile situation that happened in 1988, where the United States accidentally shot down a passenger plane. I was shocked. I had no idea that this had happened. I looked it up, and the United States never really fully took responsibility or apologized. The president just made a statement saying it’s unfortunate that it happened, something along those lines. So we have that example from recent history where I think there was some autonomy in there, but ultimately a human had made the decision, and I don’t think there were really any consequences. And the second part is about who should be responsible when AI goes awry. I think, at least in America, we usually come down to imposing fines. So I don’t know if that will be one way that this will be handled, if AI goes rogue. We’ve had situations where teenagers take their lives because their families claim that AI egged them on. I think we’re starting to see that play out, and either these companies put in better guardrails or they’re getting some sort of monetary punishment for it. So maybe that’s a direction it will go in. But to your point, this diffusion of responsibility does make it pretty complicated to say ultimately who or what is responsible.

Tom (00:32:23)
Yeah. I do like how this book has made this AI warfare topic so much more immediate in my mind. Typically in the past, the military has been its own isolated island in terms of all its technology and its workflows. Of course they’ve always had contractors and so on, Lockheed Martin and other companies, but more and more the military is partnering with big tech organizations to thrive and survive in this new space. Clearly the military can’t keep pace on its own, and now it relies on, you know, Palantir, AWS, Microsoft, and other companies for so much of the technology, which then pulls us, as people who work at these technical corporations, into this whole murky mess. A lot of employees maybe feel like they wanted to create technology that helped the world, that solved problems, that made people happier and more connected. And in reality they find that their AI development is used to resolve images that are then used to end people’s lives. So it’s bringing us into this new era of warfare where the military and big tech are kind of partners in everything. Do you think in ten years we’re gonna be in this hellscape of AI warfare going on all over the world? How soon do you think this is gonna — I was gonna say blow up in our faces. How soon is this really going to escalate into something that is truly dystopian and scary?

David (00:34:21)
I’ll speak on that for just a second. I think with the cases of Israel, and Russia and Ukraine, they were saying this is a preview of what cities being bombed out is going to be like, and this is only gonna continue in this direction. Do you remember that in the book, toward the very end?

Tom (00:34:41)
Yeah, yeah. I mean, Israel basically decimated [Gaza]. And I don’t think we get a lot of insight as the public about how involved AI is in all the military operations, but it sounds like it was definitely a major facilitator in identifying sites and everything. Anybody else have any thoughts on the immediacy of everything? The compressed timeline is essentially what we’re getting at, the way AI compresses decisions from hours to seconds and so on. Just this morning, for example, I used AI to document this really complex thing on my own blog, about how I generate PDFs. It took me like 10 minutes to tell AI, and it would have taken me two days by hand. The same sort of accelerated timeline is happening with warfare. Instead of taking time for armies to, I don’t know, get in place, the boots on the ground, the deployment, you’ve got machines working twenty-four hours identifying targets and triggering things. Things just accelerate at such a pace that in very little time you could wake up the next morning and find that we’re in the middle of a massive AI war.

Bianca (00:36:21)
I think parallel to the weapons side of war, we are seeing, and have been seeing for a little while now, how AI has also changed propaganda and the speed at which propaganda and other misinformation is being distributed internationally. If anyone goes on social media for even two minutes, it’s very easy to get a lot of content served to you that either reinforces a view you might already have or shows you something that is very galvanizing. There are so many times where I have to look something up and be like, wait, is that true? Did this happen? Or this is like 95% true, but then there’s 5% of it that’s fabricated that heightens my emotional response. So seeing AI play out on that side of it is fascinating and horrifying.

Tom (00:37:25)
Yeah. Interesting. I don’t feel like the author went into the propaganda as much as they could have. Maybe I just don’t remember that.

Bianca (00:37:34)
I don’t think she did. I think it was just my own extrapolation, and just seeing what we’ve seen in recent years and days in terms of some other content as well.

Shari (00:37:47)
And she did talk mostly about what the American AI use is going to do to others, but didn’t really touch on, because it wasn’t part of Project Maven, of course, what the effects are when other players like China or Russia are able to hit targets in the US, which I think is also going to be inevitable. I’m glad you brought up the propaganda, Bianca, because I think the propaganda is a huge part of it, too. It’s going to influence the elections, most likely, that are coming up in two weeks. And it’s also going to influence how people feel. I’ve read articles about how Chinese propaganda is behind people being against data centers going up in their community. Now, that may or may not be true, but there’s certainly that possibility.

Tom (00:38:41)
Yeah, probably the best propaganda is propaganda you don’t realize is even happening, right? You just think it’s part of the natural conversation and not being driven by the agenda of another party. Yeah, you’re right. This is a really interesting time, with the election right around the corner. We’re seeing a lot of big shifts, and who knows what’s gonna happen. To try to usher in certain political outcomes, there might be massive swings of strategy. Who knows? Yeah, this is a weird book to talk about, because I know it’s so polarizing, right? Bianca, you said something earlier that’s really got me thinking. All the information we produce goes online, and I don’t ever wanna say anything that’s gonna come back to haunt me. But at the same time, this is sort of how AI is working. There’s so much information we’re putting online, and patterns that are detected and so on. I don’t know what the outcome of all this stuff is. Sorry, that’s sort of a random comment. Let me jump back into some more things in the book. Luis, did you give up on the audio? I so want to hear your voice. I think maybe look in the sound settings of your computer and make sure that you’ve got that to match. I can actually see what sound you have, and you have your “Mezcla estéreo (Realtek(R) Audio)” selected. So if that’s not your speaker, maybe look at that. What else in this book should we talk about?

Bianca (00:40:33)
Something that I was thinking about was that I was surprised at the level of information that people were willing to share about Project Maven and their experience in the military. You know, some sources go unnamed, but even just the depth of information that people were willing to share is fascinating, and I’m just in awe of the author’s journalistic capability to build trust with these people. That’s one thing that’s just astounding. And the second thought I have that complements that is, what is not in this book? What did people not share, or what was classified? This focuses on Project Maven, but the author does discuss that there were other initiatives in place, and are in place. And there’s probably so much that is classified that we have no idea exists. And maybe in ten or twenty years we’ll discover it, or see it before our very eyes. But I’m curious to hear what other people thought about those two things: A, what people were willing to share, and B, what we might not be hearing.

Shari (00:41:48)
I’ll put in a C as well, which is, how do we know that what she is telling us is believable? Because I found a factual error in something that would have been really, really easy to fact-check, and wasn’t: just the location of the Winchester Mystery House.

Bianca (00:42:04)
Wow.

Shari (00:42:05)
It’s not in Santa Cruz, it’s in San Jose. Now, how hard would that be to check, especially if she’s got people in Silicon Valley? Clearly, nobody read that and said, you know what, that’s a factual error. There were other things that she said that were innuendos. She said something about the CEO of Google going on the Wait Wait… Don’t Tell Me! radio show and saying that he thought the “Don’t be evil” thing was kind of stupid. Well, I think he said that ironically. I don’t know if he actually believed that. She makes up stuff too. So number one, I agree that there’s probably a lot of stuff that she didn’t get told. But I think there’s the whole credibility issue, with AI making up things, and that could have been something easily fact-checked. So many things could have been fact-checked. So I come to question a lot of what was reported. Some of it was her fault, and some of it was that people didn’t want to talk about it. But there were a lot of things that I did question.

Tom (00:43:14)
Go ahead, David.

David (00:43:15)
Can I say something? Yeah. So another part of that, maybe taking it to the next level, is you have the fact checks, but then also lawyers for the publisher being afraid they’re gonna get sued. Or she spent all these years working on the book, and then some part of the military says this book cannot be published. You’ve revealed some information that was confidential, that was part of a security clearance, and that’s it. It’s done.

Tom (00:43:45)
Yeah, the author should really get special credit for just tackling such a difficult terrain of information. ‘Cause how do you get all this information out of a military organization that probably has most of this classified, right? People are highly sensitive to AI warfare, and now you’re gonna write a book about it that goes in depth? Good luck. And the author did manage to pull out a ton of information. And Shari, on the Winchester Mystery House, I have actually been through that, because I used to live in San Jose, and what a tour. But yeah, good call. It’s impossible to know. How do you know if something is factual? You just kinda trust the publisher, and if it’s a print book, you hope the errors are minimal, that they’re not the larger things.

Shari (00:44:42)
She did have a disclaimer that said, you know, all the errors are mine. In the end, her name is on the book. She probably did a lot of digging, but couldn’t get everywhere. She couldn’t fact-check every little tiny thing, but it just did raise my awareness that just because something is written in a book doesn’t necessarily mean it’s true.

Tom (00:45:05)
Yeah. Now, coming back to another point Bianca made, which David touched on earlier, what is not there? That is an interesting question to ask about a book, especially one like this, where you know that a lot of information probably isn’t available to be published. David, you were wondering about the people who didn’t pursue this path. What happened to them? Those stories aren’t told. Bianca, I’m wondering if you could dig a little deeper into what’s not there. Are there details or gaps that you can identify, or is it just kind of a larger question about what might not be there? We don’t know what we don’t know.

Bianca (00:45:46)
It’s definitely the latter. It’s the unknown unknowns. Again, I really don’t have much insight into what goes on in the military. I don’t have a military background, and I haven’t worked for the DOD. So I can only imagine that there are lots of classified programs related to warfare or gathering information or distributing information. I would be curious to know about that. And this is kind of a segue from there, a little bit unrelated, but a few months ago we read The MANIAC, which was about John von Neumann. When we were reading the perspective of, I think, his first wife, she was saying — this was before World War II, in the looming years up to it — we knew something bad was coming, but we kept dancing. We kept living our lives and partying. And I kind of feel like I’m in that zone, where I feel like something bad is coming, and I feel like things are changing in a way that I don’t know exactly how they’re going to change. But I just kinda have to keep living my life. It’s just this weird feeling that things could go haywire due to lots of different things. But on the other hand, I’m like, well, I have to keep living my life, because what else am I gonna do?

Shari (00:47:16)
I’m right there with you, Bianca. I’ve got that same sort of anxiety, and I think, you know, there’s nothing, zero, I can do about it. And so I’m just going to focus on the things that I do have some control over, which is pretty limited, but I can’t worry about the things that haven’t happened yet. And my experience is that the things that you worry about are not going to happen. It’s going to be something that comes completely out of the blue that is going to get us, for example, AI maybe inventing a virus that we have no defenses for. We’ve been down that path before, and it could be waiting for the right moment to unleash it on an unsuspecting population, or infiltrate our water supplies. These are just a couple of things that I have thought about in terms of things that could potentially happen. But what can I do about them? Zero, because they haven’t happened, and even if they were in the works, there’d be nothing I particularly could do to avoid it.

Tom (00:48:13)
Yeah, I feel that sense of helplessness as well. I don’t really think I can change the course of hardly anything. And so, whether I am in favor of AI accelerating war or against it, it doesn’t much matter. Although at work every day, when I cross a certain street, there’s this big old poster that’s against Palantir. It says “Palantir occasionally kills people,” with an attribution to the CEO. I’ll share the image here when I can dig it out. But it always makes me think of this book, and I don’t know what I’m supposed to do with this information. There’s not a whole lot that seems possible. But at the same time, if everybody has that same feeling, then nobody can control the outcome of anything. That doesn’t seem right either. So I don’t know. It’s a weird paradox. Luis, I don’t have any controls to authorize a microphone. I looked. Sorry, I’m still getting used to this platform too, so there could be, but I don’t know. I wish I could solve the audio, because I so want to hear your point of view. So maybe next time, or maybe we’ll have another trial and can work that out. But I don’t have any controls on my side that I can use to allow or reject somebody’s audio settings.

Shari (00:49:47)
I wanted to go back to the question of who decides. There was a lot about the bureaucracy and how Cukor was able to cut through a lot of the bureaucracy. But the bureaucracy is basically put in place to slow things down, of course, but also to give control, so that it doesn’t fall to just one person to control this. Whereas if you slice through it, just like in a lot of the tech companies, there’s less bureaucracy and things happen. You know, move fast and break things. Is that the kind of motto that we want for a military operation? It might work okay for a small software company, but when we’re talking about a massive number of people and potential for harm, is that the kind of mentality that we want, or is red tape actually a good thing?

Tom (00:50:40)
That is a great point. I feel like bureaucracy was a huge theme in the book, for sure. The story of Cukor was basically a man against bureaucracy. And yes, of course, bureaucracy’s purpose is to put the brakes on processes and so on. At my work, we’re constantly given surveys about whether bureaucracy is slowing us down, and they’re constantly trying to remove barriers so that we can move faster and not get slowed down by bureaucracy. So there’s definitely friction there, right? I think the author said that if the military moved through its bureaucratic channels, it would take, I don’t know, eighteen times as much time to get the same result as partnering with commercial tech. It just moves too slowly. That was Cukor’s main thinking. He had to go outside the boundaries and break the rules in order to move at the speed he felt was needed.

David (00:51:50)
It was interesting how he said that the whole AI process was gonna take like twenty years, and they were only like ten years in. So he did sort of have a vision of how long things could really take, and that the technology wasn’t really there for so long.

Tom (00:52:10)
And if he started ten years ago, I mean, that was before this whole ChatGPT generative AI era really blossomed. That’s going back to like 2016. Was AI even the main conversation? No, that wasn’t gonna hit the mainstream for another five years or so. So he must have been working with some very incipient technology that was probably highly buggy. So he had even more obstacles to overcome. If he were operating now, I think people would already buy into the AI infrastructure and progression and probably have way better tools to get buy-in from the get-go. But yeah, if he was initially operating on AI tools from 2016, I can see why he got so much pushback.

Tom (00:53:09)
Yeah. Well, this book’s given me a lot to think about, and we’re kind of out of time. So let me just tell you the next book. Can I share my screen? Let’s see. Yeah, I can share my screen. Okay. Sorry, I’m figuring out a lot of this stuff. What screen am I gonna share? So the next book that I have selected in our reading list is Co-Existence: The Next Phase of AI by Ethan Mollick. Let me just give a plug for this. First of all, it’s just gonna be published in like two days, so it’s super current. It’s still not even available. It’ll be available October twentieth. But second of all, Ethan Mollick is quite the force. If you follow him on LinkedIn, this guy is very innovative and constantly talking about cool stuff and getting a ton of likes and comments and discussion. Like I said, he’s a force on LinkedIn. I know he’s a professor, so he’s not just posting light things. And we read his first book, Co-Intelligence, a long time ago. Some of you may not have even been around for that part of the book club. It was a decent book. It wasn’t riveting, but the advice that he gave in that first book has stuck with me through other books. Advice like, hey, you just have to try it out to get a feel for the jagged edge, where AI is good and where AI fails. There’s no pre-written handbook, and working from the grassroots up instead of coming from the top down gives you such a better view of what that jagged edge looks like. Advice like that has stuck with me as being especially sound, because this is a person who grounds his observations in experience. So let’s see if it’s better than Co-Intelligence. It’s a 240-page book, no reviews yet, obviously, and a five-hour listen, so a short book. It’s also gonna be available on Audible if that’s your preference.

Tom (00:55:29)
How many people are Audible, or audiobook, versus print book versus ebook here? Anybody?

Shari (00:55:37)
I’m usually ebook.

Bianca (00:55:38)
I’ve been doing a lot of audiobook.

Shari (00:55:42)
I do ebook, but I like to go back and forth and have the audio and the Kindle. I can just listen when I want to, and then read it when I can’t listen.

David (00:55:54)
I did audio for the Maven book, but I still read regular books too.

Tom (00:55:59)
Yeah. I’m becoming more of an audiobook person, ‘cause I like to listen on my commute. So anyway.

Shari (00:56:05)
Yeah. Well, I’m glad I did not buy this book. I got it from Libby, the library app, so I read it on my Kindle that way. And I just wanted to go into why I gave it two stars. Number one was the credibility issue. And number two is it terrified me, and I’ve had nightmares.

Tom (00:56:19)
Oh man, that’s not good.

Shari (00:56:22)
And I didn’t think it was particularly well written. I’m curious, Bianca, why you thought it was really well written. I thought she inserted herself in places where she didn’t need to. You know, I’m writing a novel, and I’ve been a technical writer for a long time. I’m retired now, but I really didn’t think this was a well-written book. It wasn’t terrible, but it didn’t strike me the way some other authors of nonfiction have written really, really good, high-quality books. This struck me as meh. So Bianca, what did you like about it?

Bianca (00:56:50)
Yeah, I can definitely see that. I think part of it for me was just the novelty of this information being new to me. And I actually kind of liked how she inserted herself, because it gave me a framework for why she was including some of the information, and I could understand what biases she might have that would color it. But I could see how that would not be welcome in some respects as well.

Tom (00:57:21)
Thank you, everybody, for the discussion. I’m gonna go ahead and stop our recording. Like I said at the beginning, if you can just make sure that you see “uploaded 100%” on your end before totally dropping off, that’s great. And I hope to see you in the next book club. I hope that you’re enjoying some of the books. It’s really helping me just have a more balanced perspective and pace in this whole frenzy of AI everywhere. So thanks again for joining.

Shari (00:57:51)
Tom, you’ve done a real good service, and I so appreciate your choosing these books and being prepared to lead these conversations. So thank you. A lot of kudos to you for doing this.

Tom (00:58:00)
Yeah, thanks.