CodeSecBench

Governance

How CodeSecBench commits to neutrality — the trigger that graduates the project from solo to multi-maintainer, the rules that follow, and the open invitations.

This document declares the governance model CodeSecBench commits to. It applies the moment the project is anything more than getdebug’s internal harness — i.e., the moment any external tool maintainer or independent researcher wants a maintainer seat.

Status today

Solo maintenance, acknowledged. getdebug is currently the only maintainer of CodeSecBench and is one of the tools CodeSecBench grades. This is documented here, in /methodology, and on the getdebug.dev/bench compatibility page — not hidden behind branding. The conflict-of-interest reducer is the openness of the methodology + corpus + harness, all MIT-licensed and re-runnable.

What triggers graduation

The project adopts the multi-maintainer governance below the moment any one of the following is true:

  1. A maintainer of a tool currently in CodeSecBench (gitleaks, trufflehog, semgrep, snyk, CodeQL, getdebug, or any future addition) requests a maintainer seat. Acceptance is the default; refusal requires a public explanation.
  2. A maintainer of a new tool wants their tool included. They contribute a runner adapter (see src/runners/ in the harness repo) and the CodeSecBench maintainers review for fairness of default flags.
  3. An independent security researcher with a public track record commits to ≥10 hours/month of corpus expansion, label adjudication, or methodology review.

Graduation isn’t gated on getdebug’s preference; it’s gated on the trigger above. If the trigger fires and getdebug delays the move, that’s a documented failure of this governance doc and the third party should escalate.

Multi-maintainer governance (post-graduation)

When the project graduates, three rules apply:

1. Methodology changes that shift any scanner’s score require sign-off

A PR that changes anything in the methodology (scoring rules, the fixture corpus, the default flags for any runner, the artifact-grep oracle’s behavior, the same-finding overlap heuristic) requires sign-off from any maintainer of a tool whose score that PR would change.

The PR template asks the proposer to declare: “this change is expected to move scores for tools: X, Y, Z.” Maintainers from those tool teams have veto rights. If the declaration is wrong (a change moves another tool’s score) and the affected team objects post-merge, the change reverts pending re-review.

2. Corpus changes require quorum

Adding or removing repos, fixture categories, or labeled findings requires sign-off from two maintainers, at least one of whom is not affiliated with any scanner in CodeSecBench. The non-scanner maintainer’s role is to be the independent voice on what belongs in the corpus.

If no non-scanner maintainer exists yet, additions require sign-off from two scanner-team maintainers from different teams (so getdebug + a hypothetical gitleaks co-maintainer can both approve; getdebug solo cannot).

3. Disagreements over a finding’s label are public

Per-finding labels are not committed via a PR with single-reviewer merge. Each label PR is open for 7 days; any maintainer can register a counter-label with a rationale. If both labels remain at the end of 7 days, the finding goes into the “contested” set surfaced separately on /results. The benchmark never silently picks a side; contested findings are scored by both interpretations and the precision/recall numbers carry “low” and “high” bands.

This is the analog of academic adjudication for ground-truth datasets.

Money, hosting, and the conflict of doing this on getdebug’s dime

Right now, getdebug pays for:

Post-graduation, these are donated to whatever neutral entity hosts the project:

getdebug’s commitment: when graduation fires, the project leaves any getdebug-owned namespace, and getdebug does not retain veto rights beyond its slot in the multi-maintainer rules above.

Open invitations

If you maintain a security scanner — secrets, SAST, AI-app, supply-chain — and want a seat: open an issue against the corpus repo titled Maintainer seat request — <your tool> and link to your tool. We respond within 7 days.

If you’re a researcher who wants to contribute corpus labels: same — issue with Label contribution interest. No tool-affiliation required.

Initial maintainer roster

This document is itself open to revision via PR. Changes require the same multi-maintainer sign-off the methodology does. The earliest version of this doc has all three “(open)” slots empty by design — the gap is the point.