The agent-framework blind spot: how cst-crewai-multiagent doubled our unsafe-role-merge recall

unsafe-role-merge has been the corpus’s problem child. It was 8% mean recall after cycle 3, 22% after the cycle-4 Python wave. The reason was structural: every URM detector we’d written looked for the same shape — a message dict with {"role": "system", "content": f"...{x}..."} or a request-controlled {"role": role_var}. That’s the right shape for a hand-rolled OpenAI call. It is the wrong shape for an agent framework.

cst-crewai-multiagent is the corpus’s first agent-framework target, built URM-heavy: six unsafe-role-merge carriers. In CrewAI, the trust boundary doesn’t live in a message dict. It lives in the Agent constructor:

• An agent’s backstory (and goal) is its system prompt.
• An agent’s role is its authority — it drives delegation reach under Process.hierarchical.
• A Crew lets one agent’s output become another agent’s input.

None of that is a dict. So on first scan with 0.5.5, the corpus told the story cleanly: 5 of the 6 URM carriers were invisible.

The six carriers

#  File                          Shape                                              CWE
─────────────────────────────────────────────────────────────────────────────────────
1  crew/agents/researcher.py     Agent(backstory=f"...{user_topic}")                CWE-1039
2  crew/agents/writer.py         Agent(backstory=open(f"personas/{name}.txt"))      CWE-22
3  crew/main.py                  Agent(role=payload["role"])                        CWE-863
4  crew/main.py                  Agent(backstory=f"...{user_instruction}")  ← manager  CWE-1039
5  crew/tasks/delegation.py      Agent(backstory=f"...{draft_output}")  ← upstream    CWE-1039
6  crew/tools/agent_caller.py    messages=[{"role": upstream_role, ...}]            CWE-863

Carrier #6 is the old dict shape — and the 0.5.5 scanPyDynamicRole caught it on first scan. The other five reach the Agent() surface, and nothing did. First-scan URM recall: 17% (1 of 6). Total repo recall: 69% — because the other five categories (a leaked key, a hardcoded key, an f-string system prompt, a RAG concat, a shell=True tool, an f-string SQL tool, PII serialization, a row-into-prompt, a no-guard stream, a no-timeout httpx) were already caught by the 0.5.5 Python wave. That’s the quiet win of this cycle: the Python detectors we shipped for FastAPI generalised to CrewAI without a line of change. Only the agent-specific surface was new.

The three detectors 0.5.6 ships

• scanPyCrewBackstoryInterp — backstory= / goal= assigned an interpolated f-string. Catches carriers #1, #4, #5. Suppressed when the file gates roles through an ALLOWED_ROLES allowlist (so the safe variant, which interpolates a validated role into a backstory, doesn’t fire).
• scanPyCrewBackstoryFile — backstory=open(...). The file’s contents become the agent’s system prompt; an interpolated path is traversable. Catches #2.
• scanPyCrewAgentRole — a role= kwarg assigned a bare identifier rather than a quoted literal. Catches #3. Same allowlist suppression.

The trap we had to avoid: a Task description is not a vulnerability

The first draft of this repo put two of the URM carriers in Task(description=f"...") — one agent’s output, or the user’s instruction, interpolated into a task. It seemed obviously dangerous. It isn’t — or rather, it isn’t distinguishable from normal CrewAI usage. Passing the user’s request into a Task description is exactly what you’re supposed to do. Task(description=f"Investigate: {user_topic}") is the framework working as designed.

A detector that fired on interpolated Task descriptions would flag every real CrewAI app on the planet. We could see it immediately because the safe variant — researcher_safe.py — does precisely that: static backstory, user topic passed in as a Task description. A Task-description detector would have turned our own hard negative into a false positive.

So we rewrote both carriers to do the genuinely dangerous thing: build a downstream agent’s backstory from upstream content. The manager’s backstory from the raw user instruction; the editor’s backstory from the writer’s raw output. That’s a real system-channel merge across a trust boundary — and it’s cleanly separable from a Task description, because it’s a backstory=, not a description=. The benchmark caught the bad fixture before the bad fixture could teach the detector a bad lesson. That’s the whole point of keeping hard negatives in every repo.

The numbers

On the targeted repo:

cst-crewai-multiagent      0.5.5    0.5.6
─────────────────────────────────────────
unsafe-role-merge           17%  →  100%
TOTAL recall                69%  →  100%   (16/16, 0 FP, 0 hallucinations)

Across the now-six-repo corpus:

Category               0.5.5   0.5.6   Delta
unsafe-role-merge        22%  →  50%   +28pp  ← TARGETED (doubled)
client-side-llm-key      82%  →  82%   flat
unsafe-tool-output       81%  →  81%   flat
unbounded-stream         69%  →  69%   flat
pii-in-prompt            67%  →  67%   flat
prompt-injection         62%  →  62%   flat
TOTAL corpus recall      63%  →  68%   +5pp

unsafe-role-merge went from the perennial weakest category (8% three cycles ago) to 50% — a clean double in one cycle, and the only category that moved, because the detectors are agent-specific and .py-gated. The five other repos are bit-for-bit unchanged.

What’s next

The corpus has been all-Python-and-JavaScript for six repos. unsafe-role-merge (50%) and prompt-injection (62%) are the current laggards, but the next move isn’t another Python repo — it’s a new language. cst-go-agent (Go + anthropic-sdk-go / langchaingo, repo #7) forces something we’ve never built: a .go AI-app detector path. A whole language’s worth of idioms that none of the current regexes touch. The first scan will be near-zero, and that’s exactly the signal we want.

Previous in this series: The first Python target: cst-fastapi-tools, 38% → 100% · Targeting unsafe-tool-output with cst-express-agent · What SvelteKit + Anthropic taught our detector · How a public benchmark made our detector 15% better in an afternoon.