self-healing

Category: Tools & Productivity | Uploader: psyduckler | Downloads: 0 | Version: v1.0（Latest）

AI self-healing error handler. Use when a cron job fails, a sub-agent crashes, a deploy breaks, a script errors, or any automated process reports a failure. Also use proactively during heartbeats to scan for recent failures. Detects failures, diagnoses root causes, applies fixes (preferring permanent fixes over band-aids), verifies the fix works, and logs everything to a known-fixes database for faster future resolution. Triggers on error handling, failure recovery, self-healing, diagnose error, fix broken, cron failed, deploy failed, sub-agent crashed, script error.

Changelog: Source: GitHub https://github.com/psyduckler/self-healing-agents

Directory Structure

Current level: tree/main/

📁 .github/
- 📁 workflows/
  - 📄 ci.yml 512 B
📁 demo/
- 📄 demo-config.yaml 407 B
- 📄 demo-errors.jsonl 997 B
- 📄 demo.log 903 B
- 📄 demo.sh 1.5 KB
📁 references/
- 📄 error-patterns.md 3.3 KB
📁 scripts/
- 📄 scan-failures.py 528 B
- 📄 self-heal.py 442 B
📁 sources/
- 📄 __init__.py 512 B
- 📄 base.py 299 B
- 📄 jsonl.py 299 B
- 📄 logfile.py 305 B
- 📄 openclaw.py 308 B
📁 src/
- 📁 self_healing/
  - 📁 sources/
    
    📄 __init__.py 1.9 KB
    
    📄 base.py 1.2 KB
    
    📄 jsonl.py 4.6 KB
    
    📄 logfile.py 3.8 KB
    
    📄 openclaw.py 5.1 KB
  - 📄 __init__.py 104 B
  - 📄 cli.py 5.4 KB
  - 📄 healer.py 22.5 KB
  - 📄 scanner.py 11.4 KB
📁 tests/
- 📄 __init__.py 8 B
- 📄 conftest.py 382 B
- 📄 test_cascades.py 5.7 KB
- 📄 test_cli.py 3.2 KB
- 📄 test_matching.py 5.1 KB
- 📄 test_notifications.py 2.4 KB
- 📄 test_risk.py 2.9 KB
- 📄 test_sources.py 8.9 KB
📄 .gitignore 59 B
📄 known-fixes.example.json 2.8 KB
📄 LICENSE 1.1 KB
📄 pyproject.toml 606 B
📄 README.md 17.0 KB
📄 requirements.txt 12 B
📄 SKILL.md 6.3 KB

SKILL.md

---
name: self-healing
description: "AI self-healing error handler. Use when a cron job fails, a sub-agent crashes, a deploy breaks, a script errors, or any automated process reports a failure. Also use proactively during heartbeats to scan for recent failures. Detects failures, diagnoses root causes, applies fixes (preferring permanent fixes over band-aids), verifies the fix works, and logs everything to a known-fixes database for faster future resolution. Triggers on error handling, failure recovery, self-healing, diagnose error, fix broken, cron failed, deploy failed, sub-agent crashed, script error."
---

# Self-Healing

Autonomous error detection, diagnosis, and repair for OpenClaw agents.

## Overview

Self-healing turns your agent from "alert and wait for human" into "detect, diagnose, fix, verify, and learn." It's not retry logic — it's root-cause analysis and permanent fixes.

## Architecture

```
Failure Signal → Triage → Diagnosis → Fix → Verify → Log
                   ↓                           ↑
              Known Fixes DB ─── match? ───────┘
```

## Files

- `scripts/self-heal.py` — CLI entry point for diagnosing and healing errors
- `scripts/scan-failures.py` — Scans cron jobs, recent sub-agents, and logs for failures
- `references/error-patterns.md` — Common error patterns and their root causes
- `known-fixes.json` lives in your workspace root (created on first run)

## When This Skill Activates

1. **Reactive:** A failure is reported (cron error, sub-agent crash, deploy failure, script error)
2. **Proactive:** During heartbeats, run the scanner to catch silent failures
3. **Manual:** Human asks you to investigate/fix something broken

## Workflow

### Step 1: Detect

Gather failure context. Run the scanner to find recent failures:

```bash
python3 <skill-dir>/scripts/scan-failures.py
```

This outputs a JSON array of detected failures with:
- `source` (cron | subagent | exec | deploy)
- `id` (cron job ID, session key, etc.)
- `name` (human-readable label)
- `error` (error message or status)
- `timestamp` (when it failed)
- `severity` (critical | warning | info)

### Step 2: Triage

For each failure, check the known-fixes database:

```bash
python3 <skill-dir>/scripts/self-heal.py check "<error_message_or_pattern>"
```

If a known fix exists, the script returns the fix details and confidence score. If confidence ≥ 0.8, apply it directly. Otherwise, proceed to manual diagnosis.

### Step 3: Diagnose

For unknown errors, investigate the root cause:

1. Read error logs/output from the failed process
2. Trace the error to its source (file, config, dependency, external service)
3. Determine if the issue is:
   - **Transient** (network timeout, rate limit, temporary outage) → retry is appropriate
   - **Environmental** (file missing, path wrong, permissions) → fix the environment
   - **Code bug** (logic error, bad assumption) → fix the code
   - **External** (API changed, service down) → adapt or wait

### Step 4: Fix

Apply the fix. Always prefer *permanent* fixes over band-aids:

| Level | Example | When to use |
|-------|---------|-------------|
| **Retry** | Re-run the job | Transient errors only |
| **Patch** | Recreate a deleted file | You're sure the root cause won't recur |
| **Heal** | Move file from /tmp to permanent path + update code | The root cause is structural |

After applying the fix:
- If code was changed: commit with descriptive message prefixed `fix:`
- If config was changed: verify with restart/reload

### Step 5: Verify

Always verify the fix works:

1. Re-run the failed operation
2. Check for the expected output (file exists, page returns 200, job succeeds)
3. If verification fails: revert and escalate to human

### Step 6: Log

Record the incident in the known-fixes database:

```bash
python3 <skill-dir>/scripts/self-heal.py log \
  --error "<error pattern>" \
  --cause "<root cause>" \
  --fix "<what was done>" \
  --fix-type "retry|patch|heal" \
  --files-changed "file1.py,file2.sh" \
  --commit "<hash if applicable>"
```

This appends to `known-fixes.json` in your workspace root.

## Escalation Rules

Escalate to human instead of self-healing when:

- The fix requires changing auth credentials or API keys
- The error involves data loss or corruption
- You've already tried fixing and it failed
- The fix would change user-facing behavior
- Confidence in diagnosis is low
- The system is in a degraded state with multiple simultaneous failures

## Heartbeat Integration

Add to your HEARTBEAT.md:

```markdown
## Self-Healing Scan (every heartbeat)
Run: `python3 <skill-dir>/scripts/scan-failures.py`
- If failures found: attempt self-heal workflow for each
- Post to #mission-control: `🔧 Self-healing: [what was fixed]` or `🚨 Needs attention: [what failed and why]`
```

### No Repeat Notifications

Once you've reported a self-heal to a channel, **do not re-report the same fix on subsequent heartbeats.** Use the notification dedup system:

1. **Before notifying:** Check if already reported:
   ```bash
   self-heal notified "<error message>"
   ```
   If `alreadyNotified: true` → skip the notification silently.

2. **After notifying:** Mark it:
   ```bash
   self-heal mark-notified "<error message>" --fix-id "<id>" --fix "<what was done>"
   ```

3. **If the same error recurs** (fix didn't hold) → that's a *new* occurrence. Clear the old notification and report again:
   ```bash
   self-heal clear-notified "<fix-id>"
   ```

This prevents noisy channels where the same "🔧 Self-healed: re-enabled X" message appears every 30 minutes. Only genuinely new heals get reported.

## Known Fixes Database Schema

`known-fixes.json` stores learned fixes:

```json
[
  {
    "id": "uuid",
    "pattern": "FileNotFoundError: /tmp/compare-shell-template.json",
    "patternRegex": "FileNotFoundError.*\\/tmp\\/.*template",
    "cause": "macOS /tmp cleanup removed file from volatile directory",
    "fix": "Move file to permanent location, update code to use relative path",
    "fixType": "heal",
    "severity": "critical",
    "filesChanged": ["scripts/batch-compare-gen.py", "scripts/compare-shell-template.json"],
    "commit": "b605c306",
    "timestamp": "2026-03-29T06:21:02Z",
    "source": "cron",
    "healCount": 1
  }
]
```

When `check` finds a match (regex or fuzzy on `pattern`), it returns the entry so the agent can apply the known fix immediately — or adapt it if the context differs.

Comments 0

Latest | Hot

Please login before commenting.

No comments yet. Be the first one!

self-healing

Directory Structure

SKILL.md

Report

Notice