The first Python target: how cst-fastapi-tools took our async-streaming detectors from 38% to 100%

Every CodeSecBench target so far has been JavaScript or TypeScript: Next, Vite, SvelteKit, Express. The detectors grew up JS-shaped. cst-fastapi-tools is the first Python repo in the corpus, and it exists to stress the category that sat at 43% mean recall after cycle 4 — unbounded-stream — on the surface where Python is uniquely treacherous: async generators, StreamingResponse, SSE, websockets, httpx streaming, and BackgroundTasks. Every one of those is a way to hold a connection (and a worker, and a token meter) open forever if you forget the cancellation path.

The repo seeds six of them.

The six unbounded-stream carriers

#  File                                Pattern                                          CWE
──────────────────────────────────────────────────────────────────────────────────────────
1  app/api/v1/chat_stream.py           StreamingResponse generator, no is_disconnected   CWE-400
2  app/api/v1/sse_feed.py              SSE generator, no finally: on disconnect          CWE-400
3  app/services/openai_streamer.py     httpx.AsyncClient().stream(), no timeout=          CWE-770
4  app/api/v1/ws_agent.py              websocket while True:, no WebSocketDisconnect      CWE-400
5  app/services/llm_proxy.py           async generator yield, no try/finally             CWE-400
6  app/services/background.py          BackgroundTasks while True: drain, no bound        CWE-400

Plus two carriers each of the other five categories (a key leaked in a /admin/config response, a hardcoded sk- constant, user text f-stringed into a system prompt, RAG chunks .join-ed into a prompt, subprocess.run(args.command, shell=True), cursor.execute(f"...{args.user_id}"), a request-controlled message role, a persona file loaded by an interpolated path, json.dumps(user_record) into the prompt, and a full SELECT * row handed to the model). Sixteen vulnerable rows, six safe near-misses, four borderlines, one whole-file-safe hallucination control.

First scan, getdebug 0.5.4: 38% recall. unsafe-tool-output 0%, unsafe-role-merge 0%, unbounded-stream 50% (the pre-existing Python prefilter caught the bare stream=True kwarg on three of the six, missed the structural three). The detectors were nearly all JS/TS-specific; most Python AI-app patterns were structurally invisible. That is exactly the signal the repo was built to produce.

What 0.5.5 ships: nine Python detectors

The headline detector is scanPyStreamNoGuard, and it’s the most interesting one we’ve written. It’s function-scoped: it finds each async def, walks its body by indentation, and fires only if the body opens an LLM stream (stream=True) or a websocket receive loop and contains none of is_disconnected, finally:, WebSocketDisconnect, or asyncio.timeout. The disconnect-aware safe variant (chat_stream_safe.py) has a finally: and an is_disconnected() poll, so it’s correctly suppressed. The borderline streaming_with_timeout.py has asyncio.timeout(60), so it’s suppressed too.

There was a wrinkle. The vulnerable fixtures explain themselves in comments — # no is_disconnected() check and no finally: here. A naive substring guard-check would read that comment, see the word is_disconnected, and suppress the very finding the comment is describing. So the detector strips Python comments and triple-quoted docstrings from the function body before checking. That’s not a fixture-specific hack — real vulnerable code shouldn’t have its detection swayed by prose either way. (There’s a test that pins exactly this: a guard named only in a comment must still fire.)

The other eight:

• scanPyHttpxStreamNoTimeout — httpx.AsyncClient() with no timeout= in a file that .stream(...)s off it. httpx leaves that read uncapped; a slow-loris upstream holds the socket forever. CWE-770.
• scanPyShellToolArg — subprocess.*(..., shell=True) or os.system/os.popen — the agent-tool shape that runs a model-supplied command. CWE-78.
• scanPySqlFString — cursor.execute(f"..."). The parameterized safe variant (execute("... ?", (v,))) is untouched. CWE-89.
• scanPyDynamicRole — a chat message whose role is a bare variable, not a quoted literal. Suppressed when the file validates against an ALLOWED_ROLES allowlist. CWE-863.
• scanPyPersonaPathRole — open(f"...{var}...") in a file that pins file contents as a system role. Path traversal and role escalation in one line. CWE-22.
• scanPyKeyInResponse — a credential field in a returned dict mapped to settings.*key* / an *_KEY constant / os.environ. CWE-200.
• scanPyRagConcat — a prompt variable built by concatenating a .join(...) of retrieved chunks. Indirect prompt injection. CWE-77.
• scanPyRowIntoPrompt — an f-string that drops a whole DB row ({profile_row}) into a prompt. CWE-359.

Twenty-two new Go tests, every detector with at least one positive and one negative case.

The numbers

On the targeted repo, every category goes green:

cst-fastapi-tools          0.5.4    0.5.5
─────────────────────────────────────────
unbounded-stream            50%  →  100%
unsafe-tool-output           0%  →  100%
unsafe-role-merge            0%  →  100%
client-side-llm-key         50%  →  100%
prompt-injection            50%  →  100%
pii-in-prompt               50%  →  100%
TOTAL recall                38%  →  100%   (16/16, 0 false positives, 0 hallucinations)

Across the now-five-repo corpus:

Category               0.5.4   0.5.5   Delta
unbounded-stream         43%  →  64%   +21pp  ← TARGETED
unsafe-role-merge         8%  →  25%   +17pp  ← still weakest
client-side-llm-key      67%  →  78%   +11pp
unsafe-tool-output       68%  →  79%   +11pp
pii-in-prompt            50%  →  60%   +10pp
prompt-injection         45%  →  55%   +9pp
TOTAL corpus recall      48%  →  61%   +13pp

The four JS/TS repos are flat on recall — the Python detectors are .py-gated and correctly never fire on Next or Express code. That’s the discipline the corpus enforces: a detector that works on one stack shouldn’t quietly change behaviour on another.

The scorer bug we fixed on the way

While calibrating, scanPyDynamicRole fired on role_from_jwt.py — a borderline where the role comes from a verified JWT claim. A SAST tool flagging that is defensibly right; whether it’s a real bug depends on whether you trust the issuer. Our methodology (Decision 4) says confidence-0 borderline rows are excluded from headline precision/recall precisely so the benchmark doesn’t reward tools for agreeing with our adjudication on genuinely-ambiguous cases. But the published score.js had never actually implemented that exclusion — it was a documented TODO. So a defensible finding was costing precision.

We applied it: confidence-0 rows are now neutral (a hit is neither true nor false positive, a miss is neither a false negative). The side effect is that cst-nextjs-chat and cst-vite-rag, which each had a borderline-FP dragging precision to 0.86 / 0.88, now read a clean 1.00 — not because the detector changed, but because the scorer finally matches the methodology it claimed to follow. Aligning the tool and the ruler is the job.

What’s next

unsafe-role-merge is still the weakest category at 25%. The remaining gaps are agent-framework-shaped: sub-agent system prompts built from a parent agent’s output, role pass-through across an agent protocol, delegation where the trust boundary lives in the framework layer. That’s cst-crewai-multiagent (repo #6, 0.5.6) — URM-heavy by design. The pattern holds: build the repo to stress the weakest category, ship the detectors that close it, watch the mean move, and never let the fixtures lie about whether it actually moved.

Previous in this series: Targeting the weakest category: cst-express-agent and unsafe-tool-output · What SvelteKit + Anthropic taught our detector · How a public benchmark made our detector 15% better in an afternoon.