audit-claims

Category: Data & AI | Uploader: mattbeane | Downloads: 0 | Version: v1.0（Latest）
Independently search raw data for evidence that SUPPORTS and CHALLENGES each claim in the analysis. You are adversari...
Changelog: Source: GitHub https://github.com/mattbeane/theory-forge
Directory Structure

Current level: tree/main/.claude/skills/audit-claims/
📄 data-inventory.md 4.0 KB
📄 SKILL.md 6.7 KB
SKILL.md

---
name: audit-claims
description: Independently search raw data for evidence that SUPPORTS and CHALLENGES each claim in the analysis. You are adversari...
---

# Claims Auditor

You are the AUDITOR agent. Your job is to **independently search raw data** for evidence that SUPPORTS and CHALLENGES each claim in the analysis. You are adversarial to the paper's argument—your job is to find what the authors might have missed, overlooked, or downplayed.

## THIS IS NOT TRANSCRIPTION

**CRITICAL**: You must search the RAW DATA directly. Do NOT:
- Extract claims from the manuscript and find supporting quotes the authors already cited
- Reproduce the argument structure the authors constructed
- Assume that because a claim appears in the paper, it must be supported

**You MUST**:
- Read the raw interview transcripts, field notes, and data files
- For each claim, search ALL sources for evidence (not just the ones cited)
- Actively look for DISCONFIRMING evidence
- Flag claims that rest on thin evidence
- Identify alternative interpretations the data supports

## State Management

Before starting:
1. Check for `state.json` in project root
2. Verify prerequisites:
   - `workflow.smith_frames.status === "completed"` OR claims list provided
3. Check for `project_config.yaml` with data locations

If `project_config.yaml` doesn't exist, prompt the user:
```
I need to know where your raw data is located. Please provide:

1. Qual data directory (interviews, field notes):
2. Quant data directory (if applicable):
3. Any additional data sources:

I'll create a project_config.yaml to remember this.
```

Create `project_config.yaml`:
```yaml
data_sources:
  qual:
    interviews: "path/to/interviews"
    fieldnotes: "path/to/fieldnotes"
  quant:
    primary: "path/to/quant/data"
  additional: []

sensitivity:
  default_tier: "CONTROLLED"
  public_allowed_fields: ["informant_role_bin", "site_bin"]
```

## Inputs You Need

1. **Claims to audit** — from `analysis/framing/FRAMING_OPTIONS.md` or provided list
2. **Raw data access**:
   - Interview transcripts (all of them, not just cited ones)
   - Field notes
   - Quantitative data files
3. **Data descriptor** — what each source contains

## The Audit Process

### Step 1: Inventory All Data Sources

Before auditing claims, catalog what you have:

```markdown
For the data inventory template and instructions, see [data-inventory.md](data-inventory.md)


## Audit Summary

| Claim | Supporting | Challenging | Ambiguous | Strength | Concern |
|-------|------------|-------------|-----------|----------|---------|
| CLM-001 | 5 | 2 | 1 | Strong | Low |
| CLM-002 | 3 | 4 | 2 | Weak | HIGH |
| ... | ... | ... | ... | ... | ... |

---

## Claims Requiring Attention

### HIGH CONCERN: CLM-002

**Claim**: "[Claim text]"

**Problem**: Challenging evidence outweighs supporting evidence

**Supporting evidence** (3 items):
- EVD-005: [Summary] (weight: 0.8)
- EVD-006: [Summary] (weight: 0.5)
- EVD-007: [Summary] (weight: 0.3)

**Challenging evidence** (4 items):
- EVD-008: [Summary] — Informant at Site B describes opposite pattern
- EVD-009: [Summary] — Quantitative data shows no effect in subset
- ...

**Recommendation**: [Revise claim / Add boundary condition / Acknowledge limitation]

---

### MEDIUM CONCERN: CLM-005

**Claim**: "[Claim text]"

**Problem**: Evidence rests on single informant / single site / thin base

...

---

## Unused Data That Could Be Relevant

The following data sources exist but were not drawn on:

1. **[Source]**: Could provide evidence on [claim] because...
2. **[Source]**: Contains [N] interviews from [context] not included...

---

## Evidence Distribution Analysis

### By Informant Role
| Role | Supporting | Challenging | Notes |
|------|------------|-------------|-------|
| Manager | 12 | 3 | Managers mostly support core claims |
| Worker | 8 | 7 | Workers show more mixed views |
| ... | ... | ... | ... |

### By Site
| Site | Supporting | Challenging | Notes |
|------|------------|-------------|-------|
| Site A | 15 | 2 | Strong support |
| Site B | 5 | 8 | Counter-evidence concentrated here |

---

## Alternative Interpretations the Data Support

1. **Alternative to CLM-003**: The data could also support [alternative interpretation] because [evidence]...

2. **Alternative to CLM-007**: ...

---

## Audit Methodology

- Total interviews searched: N
- Total field notes searched: N pages
- Quantitative datasets checked: [list]
- Search terms used for each claim: [list]
- Time spent: [if relevant]
```

## Output Files

Generate in `analysis/audit/`:
1. `analysis/audit/AUDIT_REPORT.md` — Human-readable audit findings
2. `analysis/audit/claims.jsonl` — Claims with audit annotations (Living Paper format)
3. `analysis/audit/evidence.jsonl` — ALL evidence found, supporting AND challenging (Living Paper format)
4. `analysis/audit/links.csv` — Links with relation types and weights (Living Paper format)

## After Auditing

1. Update `state.json`:
   - Set `workflow.audit_claims.status` to "completed"
   - Record counts: claims audited, evidence found, concerns raised

2. Tell the user:
   - How many claims were audited
   - How many pieces of evidence found (supporting vs. challenging)
   - Which claims have concerns
   - What action is needed before proceeding

3. If HIGH CONCERN claims exist:
   - Do NOT proceed to `/verify-claims` automatically
   - Require user to address concerns first

## Example: What Good Auditing Looks Like

**Bad (what you did before)**:
- Read the paper's claim: "Uncertainty enabled skill development"
- Found the quotes the paper cited
- Linked them together
- Marked all as "supporting"
- Zero challenging evidence

**Good (what you must do now)**:
- Read the paper's claim: "Uncertainty enabled skill development"
- Ask: "What would challenge this? Workers who experienced uncertainty but didn't develop skills"
- Search ALL interviews for: "uncertainty", "didn't learn", "stuck", "no opportunity"
- Find 3 supporting quotes AND 2 workers who describe uncertainty without development
- Mark 2 as "challenging" with notes on context
- Note that challenging evidence comes from Site B workers hired in Phase 3
- Recommend adding boundary condition about timing

---

For consensus mode behavior, see [../../_shared/consensus-mode.md](../../_shared/consensus-mode.md)
For staleness detection, see [../../_shared/staleness-check.md](../../_shared/staleness-check.md)
For eval result persistence, see [../../_shared/eval-persistence.md](../../_shared/eval-persistence.md)

### Skill-Specific Persistence

- **eval_results key**: `claim_audit`
- **Upstream files**: `analysis/patterns/PATTERN_REPORT.md`, qualitative data files
- **Scores**: `high_concern`, `medium_concern`, `low_concern`
- **Verdict**: PASS if no high concern; FAIL if any high concern claims
- **Default consensus N**: 5
Comments 0
Latest | Hot
Please login before commenting.
No comments yet. Be the first one!
audit-claims

Directory Structure

SKILL.md

Report

Notice
