Four languages, eight repos: cst-rails closes the Tier C corpus

cst-rails is the eighth and final repo in the Tier C corpus, and the fourth language: Ruby on Rails with the ruby-openai gem. Rails AI apps express the six categories through Ruby idioms nothing in the scanner had seen — "...#{user_input}" interpolation, messages = [{ role: params[:role] }], backtick command execution, File.read("personas/#{name}.txt"), user.to_json, ActionController::Live streaming, OpenAI::Client.new with no request_timeout. First scan: 6% recall, the hardcoded key and nothing else.

0.5.8 adds aiapp_regex_rb.go — fourteen Ruby detectors, gated on an LLM-SDK marker so a plain Rails app stays silent. cst-rails went to 100% (16/16, 0 false positives), and the corpus mean moved from 64% to 76%.

The same bug, three more times

Every cycle of this run has caught the same class of bug on the first scan, and Ruby caught three at once:

1. The gate was case-sensitive. The activation marker matched OpenAI and openai but not OPENAI_API_KEY — so the admin controller that leaks the key (and every tool file that says LLM in caps) was gated out. Made it case-insensitive.
2. The request_timeout check matched a comment. The streaming service’s comment literally said “client is built with no request_timeout.” The suppression saw the word and suppressed the finding. Anchored it on request_timeout: (the actual kwarg).
3. The ensure/stream.close check matched a comment. The SSE controller’s comment said “no ensure response.stream.close.” Same trap. Anchored it on a real ^\s*ensure block.

This is the comment-poisoning pattern from the Python cycle (the is_disconnected mention) and the Go cycle (the allowlist mention), a third time. At this point it’s a known failure mode with a known fix: never let prose decide a verdict — strip comments or anchor on code. The reason we keep catching it is that the fixtures explain their own bugs in comments, which is exactly what real well-documented vulnerable code does too. A detector that’s fooled by a comment in the fixture would be fooled by a comment in production.

The numbers

cst-rails                  0.5.7    0.5.8
─────────────────────────────────────────
TOTAL recall                 6%  →  100%   (16/16, 0 FP, 0 hallucinations)

Across the full eight-repo corpus:

Category               0.5.7   0.5.8   Delta
prompt-injection         60%  →  75%   +15pp
unsafe-role-merge        50%  →  65%   +15pp
pii-in-prompt            63%  →  75%   +13pp
unsafe-tool-output       73%  →  85%   +12pp
unbounded-stream         65%  →  75%   +10pp
client-side-llm-key      80%  →  87%   +7pp
TOTAL corpus recall      64%  →  76%   +12pp

What eight repos and four cycles actually proved

When this run started, the AI-app detectors were JavaScript- and TypeScript-shaped, and unsafe-role-merge recall was 8%. Four cycles later:

                  start    end
unsafe-role-merge   8%  →  65%
unbounded-stream   38%  →  75%
prompt-injection   ~42% → 75%
overall            ~48% → 76%
Languages covered:  2   →  4   (TS, Python, Go, Ruby)
Frameworks:         4   →  7
Precision:        100% on all eight repos, every cycle

But the recall numbers aren’t really the point. The point is the loop. Every cycle was identical in shape: build a repo that stresses a known weakness (a category or a whole language), scan it, watch it fail, ship the detectors that close the gap, re-score all repos to prove no regression, and write down exactly what moved. Four times, including twice for languages the detectors had never seen. The benchmark caught a bad fixture before it could teach a bad lesson (the CrewAI Task-description trap), and it caught the same comment-poisoning bug in three different languages before it could ship.

A SAST tool’s recall number is a claim. The corpus is the receipt. Eight repos, four languages, 128 labeled vulnerabilities, one scorer, every result reproducible from the truth files in the public repo. That’s the deliverable — not the 76%, but the fact that you can check it.

Previous in this series: A whole new language: cst-go-agent · The agent-framework blind spot: cst-crewai-multiagent · The first Python target: cst-fastapi-tools · cst-express-agent · SvelteKit + Anthropic · The first afternoon.