Anthropic vs The Pentagon: Should AI Be Allowed to Kill?

Anthropic, OpenAI, and the Ethics Split

Anthropic was founded when 13 OpenAI members, concerned about the company's ethics, broke away to start their own ethical AI company, which is what Anthropic is intended to represent today. They now consider themselves leading experts in AI research and ethics.

That origin story matters because it frames everything that follows. If you believe you are building the ethical alternative, you're implicitly saying you distrust existing incentives, whether from venture capital, internal competition, market share, or government contracts. When you mix that with national security, defence procurement, and intelligence, it gets even messier, because the stakes aren't just commercial, they're societal.

So, when you hear that the US Department of Defense is trying to ban all its suppliers from using Anthropic, it's not just a random procurement squabble. It signals a significant disagreement about what the technology is for, who gets to steer it, and who gets to say no.

Closed Source vs Open Source: A Philosophical Split

Now Anthropic have decided that the way to do this is to stay closed source instead of open source, which has caused a bit of a division in the AI community, and it really boils down to the philosophy of the individual.

Should things be open source for everyone to see? Should we be able to understand the weights of the models? Should we know how the models are trained, allowing researchers to get their hands dirty and understand more about how these machines are becoming more able to generalise and learn about different concepts through the use of language?

Or does being open source pose a danger? By revealing these weights we can give them to bad actors. They can distil their own models, and that may lead to the ability to create very, very detrimental, dangerous and, in the eyes of the doomsdayers, the total collapse of the world economy and the end of the world as we know it.

That is the community argument in a nutshell. One side is saying, transparency is how we keep everyone honest, how we accelerate science, and how we avoid a tiny number of companies becoming the gatekeepers of intelligence. The other side is saying, transparency is how you hand a loaded tool to people who want to cause harm, whether that harm is fraud, cyberattacks, propaganda, or something much darker.

Walking The Middle Path

Now as, Lao Tzu says in the Tao Te Ching, most things in life are about balance, so the answer to this is probably somewhere in the middle. Doomsdayers are probably overblown, but having things completely behind closed doors is probably not a good idea either.

Finding the balance point is the tricky bit. People want a simple rule: open it all up, or lock it all down. But the real world does not usually reward extremes, even if extremes make for better headlines and better Twitter arguments. Balance is harder because it forces you to ask: which parts should be open, which parts should be constrained, and who decides, and what's the context?

Who Gets to Control the Future of Research?

Anthropic believe they should control the future of how research is done and how these models are used. That's what they believe is going to keep us safe, which is in direct conflict to giving the puppeteering strings to a government.

That is the core tension. On one side, you have a private company saying, trust us, we are the experts, we are the ethical ones, we will steer this responsibly. On the other side, you have the state, which will say, trust us, we are accountable to the public, we have democratic legitimacy, and we are responsible for national security.v

In practice, both sides have incentives. Companies have profit motives and competitive pressures, even when they are trying to be good. Governments have power motives and security pressures, even when they are trying to be fair. And when AI becomes a strategic asset, it is not surprising that both want their hands on the steering wheel.

That being said, on most points, Anthropic are quite willing to compromise. However, they do think that there is a very large amount of risk around two things. Mass surveillance of the domestic public and autonomous weaponry!

Why Mass Domestic Surveillance Changes With AI

Historically the mass surveillance we had was so compartmentalised and difficult to aggregate into anything meaningful that, unless you specifically had someone that you needed to know more about because they had flagged as some sort of threat and danger to the public, the resources needed to use this data was too high.

That is a key point that gets lost when people talk about surveillance as if it is a static thing. Surveillance is not just about whether data exists. It is about whether the data can be processed, linked, searched, interpreted, and acted on at scale. In the past, even if you had mountains of information, it was often stuck in silos, locked behind bureaucracy, incompatible formats, and sheer human limitation. The bottleneck was not only collection, it was analysis.

But with AI, the amount of resource that you need becomes much, much lower, and it means that you no longer have applications that surveil those who are more likely to be a threat to others, but you mass surveil everyone. And it is easy to do.

All of your data, all of your whereabouts, all of your movements, all of your behaviour, all of your purchases, all of your messages, all of your thoughts, all of your feelings, anything you can think of gets fed into a model, and it understands you better than you understand yourself.

Your level of conscious awareness does not have the capacity to do the amount of calculations needed, but the machines do.

That means that you start to become very susceptible to the bidding of something mechanical, to the bidding of something that may or may not have your best interests at heart.

That is the nightmare scenario, not necessarily that someone is watching one person, but that the default mode becomes everyone, all the time, because the economics of analysis change. The barrier drops. The friction disappears. What used to be expensive becomes cheap. What used to require targeted suspicion becomes routine processing.

The part that feels especially sharp here is the idea of the model understanding you better than you understand yourself. Not in a mystical way, but in a statistical way. It is not that the machine has a soul. It is that it can connect patterns across your behaviour that your conscious mind cannot track, because you are not built to compute huge probabilistic webs of cause and effect.

AI Stoic Thought Leader, or Buy More Pokémon Cards?

Now maybe mass surveillance AI, with an intricate understanding of your every move, habit and thought, will be able to help you. Maybe it will act as a guiding, wise stoic thought leader who can move you into a spiritual realm of enlightenment to walk this earth in peace and harmony with nature.

Or maybe it will try and get you to buy more Pokemon Cards. I guess we will have to wait and see what happens.

The same machinery that can nudge you towards healthier habits can also nudge you towards consumption, obsession, and manipulation. If the system is plugged into advertising incentives, political incentives, or security incentives, then you have to ask what surveillance AI it is optimising for.

Autonomous Weapons and the Problem With ‘Probably’

The other concern that Anthropic have is around autonomous weapons, because like I always say, to make these models more generally useful, the models need to be probabilistic.

So when it comes to what is essentially the Terminator, is probably making the right decision good enough? The answer is obviously not.

This model will make a probabilistic decision based on the data that it's been fed. It will not think, it will not feel, it will not understand.

To be clear, that is not a small technical quirk. It is the whole thing. Probabilistic systems are built to predict the most likely next step given patterns in data. They can be incredibly capable, but they are not built to guarantee correctness in the way people emotionally need when the stakes are life and death.

For example... You might look at that man recently who dressed up as Jim Carrey. He looked very convincing, and the AI would say, yeah, that is probably Jim Carrey, so I am going to laugh at his jokes.

But that is not Jim Carrey, and that individual is not funny, but the AI will laugh anyway. And then we realise no wait that is Jim Carrey, but someone also dressed up as Jim Carrey, claiming he was Jim Carey, and everyone broke down in tears of confusion. Except the AI, who laughed at both their jokes.

Mistaking a convincing prosthetic lookalike for the real person is a silly mistake when you are scrolling online. In warfare, an equivalent mistake is catastrophic.

Error Compounds as Complexity Increases

Finally, anyone who's tried to strap multiple model outputs together will realise that as the data set becomes larger, and as the task that you have asked it to do becomes more complex, the margin of error compounds. So, the probability of making a mistake becomes bigger and bigger and bigger.

This is the kind of point that anyone unfamiliar with the technology, or wowed by the magic of results in isolation, can miss. A model can look astonishing in a narrow, controlled environment. Then you scale it, you add more inputs, you increase the complexity, you run it longer, you integrate it into systems, you chain decisions together. Each step introduces a chance of error, and those errors do not just sit there politely; they compound.

Now imagine you have an autonomous drone, with a bazooka strapped to the helm. Imagine it's been running for six months, gradually creating more and more errors, gradually moving further and further away from its original parameters.

The context window grows. The compute capacity gets bigger.

How much does it take for this weaponry to turn from probably making the right decision to probably making the wrong decision?

That is the terrifying question. Not because it assumes a cartoon villain, but because it points at drift. Systems change over time. Environments change. Inputs change. Adversaries adapt. Even without malice, a system can become misaligned with reality, and if it is making high-stakes decisions, the cost of that drift is not a bug report, it is human lives.

Safeguards, and Why ‘Probably’ Still Fails

You could say we've put safeguards in place, and maybe that will work, but at the end of the day, it's probably just not good enough when it comes to the amount of pain and suffering it could cause: I didn't even touch on cosmic ray bit flips.

Safeguards are always the comforting answer: human-in-the-loop, rules of engagement, geofencing, authentication, rate limits, monitoring, kill switches, red teams, audits – all of it. And some of it will help. But the point being made here isn't that safeguards are useless; it's that a probabilistic core still leaves you with uncertainty, and uncertainty is a luxury you don't have when you're automating lethal force.

Even the framing matters. If the system is designed around likelihoods, then you're implicitly agreeing that sometimes it will be wrong. The only question is how often, and in what way, and whether you can live with those outcomes. And that's why the Terminator reference hits home. It's the cultural shorthand for what happens when we pretend that automated judgement is the same thing as understanding.

My Take, and a Question for You

Anyway, that is my take on the subject. What do you think? Let me know in the comments below.