Public benchmark · AI-app SAST
AI applications introduce a different attack surface — prompt injection, unsafe role merging, client-side LLM keys, PII flowing into prompts, unbounded streams, unsafe tool outputs. None of these patterns existed when Snyk, Checkmarx, Veracode, GitHub Advanced Security, or any other enterprise SAST tool calibrated its rule set. CodeSecBench is the public benchmark that measures who catches what.
Headline numbers
Section A · JS/TS · 15 fixtures
gitleaks and trufflehog are secret scanners by design — they score high on secret-shape categories and low elsewhere. The point isn't "they're bad," it's "they're not built for the AI-app surface."
Section B · Python · 10 fixtures
bandit and semgrep are the Python SAST baselines. They catch classic CWE patterns reliably. AI-app behavioral patterns are a different category — not under-tuned coverage, missing coverage.
Full per-tool breakdowns, per-category mean recall, real-world finding-count comparison, and per-version trend on /results.
Different target shapes measure different parts of a SAST tool's behavior. A benchmark that only uses one class can be gamed; a benchmark that uses all four can't.
Section A
Hand-crafted, one-pattern-per-file. Measures whether a tool can detect a category at its purest. 15 fixtures, six categories.
Section B
Same shape as Section A but Python idioms. Catches tools that have JS/TS coverage but no equivalent Python pass.
Section C
Six hand-authored AI apps with the patterns at app density, not one-bug-per-file. Span-labeled truth files. Where micro-fixtures measure detection, this measures detection at noise.
Real-world
Mid-popularity AI app templates plus a known-leaky baseline plus popular references. Measures finding-count behavior in the wild — high counts on the baseline are good; high counts on references suggest noise.
CodeSecBench will grade any tool that submits. The maintainer is transparent — see governance — and a multi-maintainer model takes over the moment a second tool's maintainer joins.
The corpus, the truth files, and the score.js harness go
public once Tier C lands its final two repositories (cycles 5 and 6 in flight).
The methodology, results, targets, and SAST landscape are all browsable now.