sastbench-adapter-authoring

Category: Tools & Productivity | Uploader: Har1sh-k | Downloads: 0 | Version: v1.0（Latest）

Build or update a SASTbench scanner adapter under adapters/. Use when an agent needs to create, repair, or extend an adapter for a scanner, including rule mapping, path normalization, raw-output capture, optional PR-mode support, adapter tests, and harness validation against core benchmark cases.

Changelog: Source: GitHub https://github.com/Har1sh-k/sast-bench

Directory Structure

Current level: tree/main/skills/sastbench-adapter-authoring/

📁 agents/
- 📄 openai.yaml 124 B
📁 references/
- 📄 validation.md 1.3 KB
📄 SKILL.md 5.3 KB

SKILL.md

---
name: sastbench-adapter-authoring
description: Build or update a SASTbench scanner adapter under adapters/. Use when an agent needs to create, repair, or extend an adapter for a scanner, including rule mapping, path normalization, raw-output capture, optional PR-mode support, adapter tests, and harness validation against core benchmark cases.
---

# SASTbench Adapter Authoring

Build adapters that actually work with this repo's scoring and reporting model.

Use this skill for:

- creating a new adapter under `adapters/<scanner-name>/adapter.py`
- repairing a broken adapter
- extending an adapter with `scan_with_metadata()`
- adding real PR-aware support via `scan_pr_with_metadata()`
- tightening rule mapping, path normalization, or line handling

For non-Codex agents, use this `SKILL.md` as the task instructions and then open the referenced repo files.

## Read these files first

Before writing adapter code, inspect:

- `adapters/CREATING_AN_ADAPTER.md`
- `adapters/README.md`
- `tests/test_adapters.py`
- `tests/test_securevibes_agent_adapter.py`

Then inspect one reference adapter that matches the scanner type:

- `adapters/semgrep/adapter.py` for CLI JSON scanners
- `adapters/bandit/adapter.py` for simple language-filtered scanners
- `adapters/securevibes-agent/adapter.py` for LLM-backed scanners
- `adapters/code-review-agent/adapter.py` for agent-style finding normalization and filtering

If PR-mode support is relevant, also inspect:

- `docs/PR_MODE.md`
- `scripts/pr_runner.py`
- `tests/test_pr_runner.py`

For exact validation commands, read `references/validation.md`.

## Required outcome

A finished adapter must:

1. implement `get_version()`
2. implement `scan()`
3. return normalized findings with correct path, line, and `mappedKind`
4. handle unsupported languages and scanner failures without crashing the harness
5. include tests
6. pass at least one real SASTbench smoke run

If raw stdout/stderr matters, also implement `scan_with_metadata()`.

If the scanner has real diff-aware review capabilities, implement `scan_pr_with_metadata()`.
If it does not, do not fake PR support. Let the harness fallback handle it.

## Workflow

### 1. Classify the scanner

Decide which category the scanner fits:

- CLI scanner with JSON output
- CLI scanner with text output
- LLM-backed local scanner
- API-backed scanner
- PR-aware scanner with native diff/review support

This determines how much metadata, cleanup, and defensive parsing the adapter needs.

### 2. Build rule mapping first

Create a direct `RULE_KIND_MAP` and then a conservative fallback map.

Valid `mappedKind` values are only:

- `command_injection`
- `path_traversal`
- `ssrf`
- `auth_bypass`
- `authz_bypass`
- `sql_injection`
- `unmapped`

Do not invent a new kind.
Do not map loosely when the scanner signal is ambiguous.
Prefer `unmapped` over a wrong canonical kind.

### 3. Normalize paths and lines carefully

The harness scores by:

- path overlap
- line-range overlap
- kind match

So the adapter must:

- return paths relative to `scan_root`
- use forward slashes
- return 1-indexed inclusive lines

If the scanner only gives file-level findings, use a whole-file range and document that tradeoff in tests or comments.

### 4. Implement scanner invocation defensively

Handle:

- scanner not installed
- unsupported language
- invalid or partial output
- timeout
- non-zero exit code that still returns usable findings

Do not crash the harness for these cases.

Prefer:

- `[]` from `scan()`
- `skipReason` in metadata-returning paths

### 5. Decide on metadata support

Implement `scan_with_metadata()` when any of these are true:

- raw stdout/stderr matters for debugging
- the scanner is hard to inspect from normalized findings alone
- the adapter is likely to be used in baseline comparisons

Use stable metadata keys:

- `findings`
- `commandInvocation`
- `exitCode`
- `rawStdout`
- `rawStderr`
- `skipReason`

### 6. Decide on PR support honestly

Implement `scan_pr_with_metadata()` only if the scanner truly uses:

- base tree
- head tree
- changed files
- diff text

Good candidates:

- PR review agents
- diff-aware LLM scanners
- scanners with built-in patch or pull-request review mode

Bad candidates:

- whole-tree scanners with no diff semantics

If the scanner is not genuinely PR-aware, skip the method and rely on SASTbench fallback PR synthesis.

### 7. Add tests before claiming success

Create or update a dedicated test file such as:

- `tests/test_<scanner>_adapter.py`

Cover at least:

- rule-to-kind mapping
- path normalization
- unsupported-language behavior
- malformed output
- scanner-not-found behavior
- metadata shape if `scan_with_metadata()` exists
- PR metadata shape if `scan_pr_with_metadata()` exists

Reuse the existing test style in:

- `tests/test_adapters.py`
- `tests/test_securevibes_agent_adapter.py`

## Quality bar

Do not stop at "the file exists."

A SASTbench adapter is only good if it:

- hits the right file
- hits the right line range
- maps to the right kind
- returns stable paths
- preserves raw output when needed
- does not leave junk in the scanned tree

## Completion checklist

- adapter code added or updated under `adapters/<scanner-name>/adapter.py`
- tests added or updated
- targeted pytest commands pass
- at least one real harness command runs successfully
- no generated `results/`, `.claude/`, `.securevibes/`, or cache junk is accidentally committed

Comments 0

Latest | Hot

Please login before commenting.

Loading comments...

sastbench-adapter-authoring

Directory Structure

SKILL.md

Report

Notice