“The outputs of this opt-in vulnerability scanner will be fully model-generated, without human review or triage,” Anthropic explained. “This will enable faster and more frequent scanning, but means that it is possible reports will be incorrect or invalid.”
When I saw the headline, I was wondering about this specifically. This may make this service not super useful.
My experience with AI security reviews is that they’re fantastic at finding faults, but they always find a list of things to complain about. If there are no real/serious faults they’ll start finding things that kind of have the same shape as a security issue, but really aren’t if you dig into them. I’ve regularly had an LLM generate a list of 10-15 issues ranging in severity from “nits” to “critical” where none of them were actual issues.
Periodic reviews seem like they could get annoying really quickly, becoming more of a maintenance burden than a help.
AI security reviews is that they’re fantastic at finding faults, but they always find a list of things to complain about.
Same experience for me. At work, I always review the output because a lot of times it finds something that can be noteable but not relevant to the scope of what is being reviewed. Even with guardrails like instructions to not go beyond the stated scope, a human absolutely still needs to read and check the findings at the end.
Yeah. I did use an open weight model LLM to scan an app that I use and it came up with a critical severity issue and I checked on it first to see if I could reproduce the problem before ever submitting it to the developer as a problem.
Turns out it was actually a legitimate security problem that has now been fixed.
Yeah, there is some fatigue to be expected if you run this thing on the regular.
But the library ecosystem is changing constantly, and that might mean that while it doesn’t find a real issue at one point that there won’t be one some time in the future. So i would say: Scan it semi-regulary for issues and review what was found, but don’t go overboard. Somewhat similar to getting an MRT every year instead of once a decade.
When I saw the headline, I was wondering about this specifically. This may make this service not super useful.
My experience with AI security reviews is that they’re fantastic at finding faults, but they always find a list of things to complain about. If there are no real/serious faults they’ll start finding things that kind of have the same shape as a security issue, but really aren’t if you dig into them. I’ve regularly had an LLM generate a list of 10-15 issues ranging in severity from “nits” to “critical” where none of them were actual issues.
Periodic reviews seem like they could get annoying really quickly, becoming more of a maintenance burden than a help.
Same experience for me. At work, I always review the output because a lot of times it finds something that can be noteable but not relevant to the scope of what is being reviewed. Even with guardrails like instructions to not go beyond the stated scope, a human absolutely still needs to read and check the findings at the end.
Yeah. I did use an open weight model LLM to scan an app that I use and it came up with a critical severity issue and I checked on it first to see if I could reproduce the problem before ever submitting it to the developer as a problem.
Turns out it was actually a legitimate security problem that has now been fixed.
Yeah, there is some fatigue to be expected if you run this thing on the regular.
But the library ecosystem is changing constantly, and that might mean that while it doesn’t find a real issue at one point that there won’t be one some time in the future. So i would say: Scan it semi-regulary for issues and review what was found, but don’t go overboard. Somewhat similar to getting an MRT every year instead of once a decade.