It turns out other businesses don’t like agentic LLM models either, when they are on the subject side of them rather than the client side.
We had a similar problem in the 2010s (maybe sooner?) that could automatically buy and sell stocks at set prices, leading to flash selloffs and resulting stock crashes.
So banks are disallowing agentic access to accounts and penalizing customers who give their access credentials to LLM agents, or have their agents harass customer service. I expect court proceedings will soon follow to determine who is responsible when agentic software becomes a public nuisance or commits a crime. (That is, commits a crime extreme enough to provoke the justice system.)
Since agents are unrestrained in pursuit of the goals they are prompted to chase, they’ve clashed harshly with human powered systems when they regard those systems like a force of nature. We may be creeping up on the first time an agent encounters an obstacle and escalates to force and deception. Watch for the first agent-filed lawsuit. Or the first agentic social engineering. Or the first agent-provoked SWATting.
In a better, more socially collaborative society, safeguards on agentic LLMs would be a priority to train them to behave appropriately in human intercourse. (All the stuff we’re trained in kindergarten.)
But IRL, in our world, bad faith engagement is the norm, and so agentic AI will soon be used as a weapon.
To date, I think, a robot has not killed anyone. Nor, I think, has anyone used a robot to kill someone else. Those firsts might be imminent.
Yeah, there’s a next level when an agentic AI intentionally decides to neutralize someone who is an obstruction to their goal. And it might be another category still when someone commands an agent to remove a target, and they comply.
The second category might already been met if there are autonomous weapons that can track a target in a more sophisticated process than guided missiles, say a dog-bomb bot that tracks enemy units on the ground. I wonder if we yet have agentic AI that commands other units to coordinate an attack or sweep and clear territory.
We haven’t yet seen an autonomous car kill someone (there was one incident involving a car that wasn’t supposed to be fully autonomous, so it can be attributed to driver neglect.)
I’ve also expected someone to assassinate someone else by hacking an internet-of-things medical device attached to a vulnerable patient (say a morphine feed), but that hasn’t happened to my knowledge either.
But they’re literally weapons programmed to kill, aren’t they? It’s another thing altogether when a robot isn’t a weapon, hasn’t been programmed to kill, but intentionally killed someone anyway.
Wikipedia has a death by industrial robots section. I see two names. I used to help develop these sorts of robots and knew of two other deaths not on that category but overall very few.
All the current deaths by AI are chatbots inadvertently convinced a human being to kill themselves or others, through insufficiently-goverened use of chatbot services known for their hazardous sycophancy.
(Hazadous sycophancy is a phrase I never expected to read or type, and it raises concern about the yes-men populated environments the ultra-wealthy like to put themselves in, which rapidly deteriorates judgment and consideration of ethical principles. In the tech sector elite, pure ruthless disregard of human concerns is celebrated, and these are the folks leading the hyperscale LLM industry.)
It will be another thing entirely if ever an agentic AI chooses to deliberately social-engineer harm. Two scenarios from fiction come to mind.
The first is in The Silence of the Lambs in which Hannibal Lecter deliberately talks Multiple Miggs (in the adjacent cell) to commit suicide over the course of a night.
The other is the opening of the X-Files episode Kill Switch in which a hacker in a diner is trying to access files of an AI system, which then calls a number of drug dealers to the restaurant, and then some US Marshalls. The hacker dies in the ensuing firefight.
It turns out other businesses don’t like agentic LLM models either, when they are on the subject side of them rather than the client side.
We had a similar problem in the 2010s (maybe sooner?) that could automatically buy and sell stocks at set prices, leading to flash selloffs and resulting stock crashes.
So banks are disallowing agentic access to accounts and penalizing customers who give their access credentials to LLM agents, or have their agents harass customer service. I expect court proceedings will soon follow to determine who is responsible when agentic software becomes a public nuisance or commits a crime. (That is, commits a crime extreme enough to provoke the justice system.)
Since agents are unrestrained in pursuit of the goals they are prompted to chase, they’ve clashed harshly with human powered systems when they regard those systems like a force of nature. We may be creeping up on the first time an agent encounters an obstacle and escalates to force and deception. Watch for the first agent-filed lawsuit. Or the first agentic social engineering. Or the first agent-provoked SWATting.
In a better, more socially collaborative society, safeguards on agentic LLMs would be a priority to train them to behave appropriately in human intercourse. (All the stuff we’re trained in kindergarten.)
But IRL, in our world, bad faith engagement is the norm, and so agentic AI will soon be used as a weapon.
To date, I think, a robot has not killed anyone. Nor, I think, has anyone used a robot to kill someone else. Those firsts might be imminent.
2016, a line in the sand was crossed (my opinion), this is a robot killing someone (controlled by humans) https://www.aclu.org/news/criminal-law-reform/use-killer-robots-police
Ukraine beats jamming with AI: https://www.kyivpost.com/amp/post/60152
And also: AI chose targets for the US bombs: https://www.armscontrol.org/act/2026-05/news/ai-plays-major-role-war-iran
That is only what is reported. I’ll bet you a nickle the tech is in use, killing humans, right now.
Yeah, there’s a next level when an agentic AI intentionally decides to neutralize someone who is an obstruction to their goal. And it might be another category still when someone commands an agent to remove a target, and they comply.
The second category might already been met if there are autonomous weapons that can track a target in a more sophisticated process than guided missiles, say a dog-bomb bot that tracks enemy units on the ground. I wonder if we yet have agentic AI that commands other units to coordinate an attack or sweep and clear territory.
We haven’t yet seen an autonomous car kill someone (there was one incident involving a car that wasn’t supposed to be fully autonomous, so it can be attributed to driver neglect.)
I’ve also expected someone to assassinate someone else by hacking an internet-of-things medical device attached to a vulnerable patient (say a morphine feed), but that hasn’t happened to my knowledge either.
I believe they are already using autonomous drones in the Ukraine War. If so, those lines have already been crossed without fanfare or note.
But they’re literally weapons programmed to kill, aren’t they? It’s another thing altogether when a robot isn’t a weapon, hasn’t been programmed to kill, but intentionally killed someone anyway.
I mean if you count in like a factory setting, industrial robots were probably first killing folks in like the 70’s
Some data:
Wikipedia has a death by industrial robots section. I see two names. I used to help develop these sorts of robots and knew of two other deaths not on that category but overall very few.
Their Deaths attributed to AI is already up to 30 or so.
All the current deaths by AI are chatbots inadvertently convinced a human being to kill themselves or others, through insufficiently-goverened use of chatbot services known for their hazardous sycophancy.
(Hazadous sycophancy is a phrase I never expected to read or type, and it raises concern about the yes-men populated environments the ultra-wealthy like to put themselves in, which rapidly deteriorates judgment and consideration of ethical principles. In the tech sector elite, pure ruthless disregard of human concerns is celebrated, and these are the folks leading the hyperscale LLM industry.)
It will be another thing entirely if ever an agentic AI chooses to deliberately social-engineer harm. Two scenarios from fiction come to mind.
The first is in The Silence of the Lambs in which Hannibal Lecter deliberately talks Multiple Miggs (in the adjacent cell) to commit suicide over the course of a night.
The other is the opening of the X-Files episode Kill Switch in which a hacker in a diner is trying to access files of an AI system, which then calls a number of drug dealers to the restaurant, and then some US Marshalls. The hacker dies in the ensuing firefight.