Pentest, vulnerability scan, AI-assisted testing: what is the difference?

Anyone comparing offers for security testing gets three different things sold under similar names. The price difference quickly reaches a factor of ten, and so does the difference in insight – just not always in the same direction. This article sorts out the terms.

Vulnerability scan: breadth without judgement

A vulnerability scanner checks a system against a database of known problems. It notices that a library runs at version 2.3.1, looks it up in the CVE database and reports a hit.

That is valuable and should run automatically in every pipeline. It has a hard limit, though: the scanner does not know whether the vulnerable code is reachable at all. It does not know whether your permission model makes sense. And it does not know that a user with the role "guest" in your application can retrieve invoices from other tenants – because that check is not in any database.

Typical characteristics: minutes to hours of runtime, three-digit monthly cost, high false-positive rate, no context.

Penetration test: judgement without scale

A classic pentest is manual work. An analyst gets to know your product, understands the business model and looks specifically for ways to abuse it. She finds the pricing logic that can be tricked with a negative discount value. She finds the attack chain made of three individually harmless weaknesses.

This work cannot be automated because it requires understanding, not pattern matching. But it cannot be bought arbitrarily often either. With four-digit day rates and two to three weeks of effort you end up with sums that are reasonable once a year – not once a quarter.

Typical characteristics: weeks of runtime, four- to five-digit cost per run, low false-positive rate, deep contextual understanding.

AI-assisted testing: the attempt to combine both

This is where it gets confusing, because very different things are sold under the label. Two variants are worth distinguishing.

Variant A: a scanner with a language model in front. A model phrases scanner output in full sentences. That improves readability, not quality. Anyone who believes they see a pentest here is buying a scan at a pentest price.

Variant B: models as an analysis tool, humans as the deciding authority. The model reads code, follows data flows, forms hypotheses and tries attack variants – far faster and broader than a human could. The results are candidates, not findings. An analyst checks every one of them, discards the untenable ones and keeps working where a chain emerges.

The gain is not that the machine is smarter than the human. It is that it takes the first sixty percent of the work off their hands – the boring sixty percent. What drops as a result is the price per run. What becomes possible as a result is testing more often.

The question to ask in the sales conversation

"How many of the findings in the report did a human reproduce before you delivered it?"

The answer separates variant A from variant B more reliably than any product brochure. If it is "all of them", ask whether the discarded candidates are documented as well. Whoever verifies can prove it.

What makes sense when

Situation	Sensible approach
Ongoing protection of the supply chain	Scanner in the CI pipeline, automated
First assessment of a product	Penetration test or AI-assisted application test
Frequent testing with a high release cadence	AI-assisted testing with human verification
Evidence for ISO 27001, SOC 2 or NIS2	Test with documented methodology and retest
Testing the entire attack chain	Red teaming

The three approaches compete less than the offers suggest. The scanner belongs in the pipeline, the deep test in the annual plan, AI-assisted testing in between. It only becomes a problem when one of them is sold as another.