My Project

Category: Research & Analysis | Uploader: warlockee | Downloads: 0 | Version: v1.0（Latest）

Orze is a filesystem-coordinated GPU experiment orchestrator. It runs the loop: **generate ideas → train → evaluate → learn → repeat**.

Changelog: Source: GitHub https://github.com/warlockee/orze

Directory Structure

Current level: tree/main/

📁 .claude/
- 📁 agent-memory/
  - 📁 asr-research-engineer/
    
    📄 MEMORY.md 148 B
    
    📄 project_8b_eval_results.md 1.3 KB
  - 📁 ui-monitor/
    
    📄 debugging.md 1.6 KB
    
    📄 MEMORY.md 1.3 KB
- 📁 agents/
  - 📄 orze-distributed-fs.md 11.6 KB
  - 📄 release-engineer.md 6.7 KB
  - 📄 ui-monitor.md 8.3 KB
📁 src/
- 📁 orze/
  - 📁 admin/
    
    📁 ui/
    
    📁 dist/
    
    📁 assets/
    
    📄 AlertsTab-BmcCy3L1.js 933 B
    
    📄 index-DemuRWa7.js 27.7 KB
    
    📄 index-l34gh9MV.css 18.8 KB
    
    📄 LeaderboardTab-DFlesFBc.js 1.8 KB
    
    📄 NodesTab-Bq2-2Duo.js 2.2 KB
    
    📄 OverviewTab-CbhUmCEl.js 4.5 KB
    
    📄 RunsTab-C8v0EOGi.js 3.9 KB
    
    📄 SettingsTab-BYqmfH5c.js 552 B
    
    📄 vendor-icons-C0JR74dJ.js 17.8 KB
    
    📄 vendor-motion-D282nMnh.js 1009 B
    
    📄 vendor-react-7wvOsoTm.js 131.5 KB
    
    📄 index.html 851 B
    
    📁 src/
    
    📄 AlertsTab.tsx 1.1 KB
    
    📄 api.ts 2.6 KB
    
    📄 App.tsx 5.7 KB
    
    📄 components.tsx 13.9 KB
    
    📄 hooks.ts 3.0 KB
    
    📄 IdeaDetailContext.tsx 800 B
    
    📄 IdeaDetailModal.tsx 5.1 KB
    
    📄 index.css 1.3 KB
    
    📄 LeaderboardTab.tsx 2.8 KB
    
    📄 main.tsx 237 B
    
    📄 NodesTab.tsx 2.8 KB
    
    📄 OverviewTab.tsx 6.0 KB
    
    📄 QueueTabView.tsx 8.4 KB
    
    📄 RunsTab.tsx 5.6 KB
    
    📄 SettingsTab.tsx 544 B
    
    📄 types.ts 3.5 KB
    
    📄 index.html 503 B
    
    📄 package-lock.json 92.5 KB
    
    📄 package.json 608 B
    
    📄 postcss.config.js 81 B
    
    📄 tailwind.config.ts 191 B
    
    📄 tsconfig.json 557 B
    
    📄 tsconfig.tsbuildinfo 354 B
    
    📄 vite.config.ts 472 B
    
    📄 __init__.py 0 B
    
    📄 mcp.py 13.8 KB
    
    📄 server.py 24.3 KB
  - 📁 agents/
    
    📄 __init__.py 0 B
    
    📄 archive_ideas.py 10.0 KB
    
    📄 hf_discover.py 4.5 KB
    
    📄 manual_notify.py 3.6 KB
    
    📄 orze_gc.py 19.0 KB
    
    📄 rebuild_lake.py 3.5 KB
    
    📄 train_idea.py 13.5 KB
  - 📁 core/
    
    📄 __init__.py 136 B
    
    📄 config.py 15.3 KB
    
    📄 fs.py 6.5 KB
    
    📄 ideas.py 5.7 KB
  - 📁 engine/
    
    📄 __init__.py 700 B
    
    📄 auto_ideas.py 5.6 KB
    
    📄 cluster.py 4.9 KB
    
    📄 config_dedup.py 3.0 KB
    
    📄 evaluator.py 12.3 KB
    
    📄 experiment_analysis.py 14.0 KB
    
    📄 failure.py 8.7 KB
    
    📄 failure_analysis.py 7.4 KB
    
    📄 family_guard.py 6.8 KB
    
    📄 gpu_slots.py 8.1 KB
    
    📄 guardrails.py 3.8 KB
    
    📄 health.py 9.2 KB
    
    📄 launcher.py 15.3 KB
    
    📄 lifecycle.py 14.2 KB
    
    📄 orchestrator.py 32.4 KB
    
    📄 phases.py 32.4 KB
    
    📄 process.py 6.5 KB
    
    📄 reporter.py 14.1 KB
    
    📄 retrospection.py 14.8 KB
    
    📄 roles.py 4.7 KB
    
    📄 scheduler.py 10.4 KB
    
    📄 sealed.py 4.8 KB
    
    📄 smart_suggestions.py 6.8 KB
    
    📄 smoke_test.py 5.3 KB
    
    📄 upgrade.py 7.6 KB
  - 📁 hardware/
    
    📄 __init__.py 69 B
    
    📄 gpu.py 3.5 KB
  - 📁 reporting/
    
    📄 __init__.py 175 B
    
    📄 leaderboard.py 24.4 KB
    
    📄 leaderboard_admin.py 6.2 KB
    
    📄 notifications.py 22.1 KB
    
    📄 state.py 10.1 KB
  - 📁 service/
    
    📄 __init__.py 0 B
    
    📄 install.py 8.1 KB
    
    📄 status.py 5.6 KB
    
    📄 watchdog.py 7.6 KB
  - 📁 skills/
    
    📄 __init__.py 0 B
    
    📄 core.skill.md 6.1 KB
    
    📄 loader.py 4.9 KB
    
    📄 ops.skill.md 4.6 KB
    
    📄 research.skill.md 2.3 KB
    
    📄 setup.skill.md 6.3 KB
  - 📄 __init__.py 46 B
  - 📄 AGENT.md 4.4 KB
  - 📄 cli.py 20.0 KB
  - 📄 cli_demo.py 7.1 KB
  - 📄 cli_pro.py 4.4 KB
  - 📄 cli_setup.py 25.2 KB
  - 📄 cli_star.py 1.4 KB
  - 📄 extensions.py 4.7 KB
  - 📄 idea_lake.py 22.5 KB
  - 📄 lifecycle.py 10.2 KB
  - 📄 RULES.md 20.7 KB
📁 tests/
- 📄 conftest.py 1.2 KB
- 📄 test_evolution.py 13.5 KB
- 📄 test_integration.py 14.4 KB
📄 .gitignore 393 B
📄 .mcp.json 79 B
📄 CLAUDE.md 2.3 KB
📄 LICENSE 10.5 KB
📄 orze.yaml.example 9.4 KB
📄 pyproject.toml 1.1 KB
📄 README.md 9.1 KB
📄 requirements.txt 12 B
📄 setup.sh 2.9 KB
📄 SKILL.md 11.5 KB

SKILL.md

# Orze — Operations & Extension Guide

Orze is a filesystem-coordinated GPU experiment orchestrator. It runs the loop: **generate ideas → train → evaluate → learn → repeat**.

## Mental Model

```
ideas.md → [claim via mkdir] → train on GPU → metrics.json → leaderboard
     ↑                                                           |
     └──── research agent reads results, appends new ideas ──────┘
```

- **ideas.md**: append-only experiment queue (markdown + YAML config blocks)
- **idea_lake.db**: SQLite archive (hot ideas consumed from ideas.md into DB)
- **results/{idea_id}/metrics.json**: the contract — `{"status": "COMPLETED", ...}`
- **Coordination**: atomic `mkdir` for claims, filesystem locks for roles — works across machines on shared storage (NFS/FSx/EFS)

## Common Operations

### Start / Stop
```bash
orze -c orze.yaml                  # start (auto-detect GPUs)
orze -c orze.yaml --gpus 0,1      # specific GPUs
orze --once                        # one cycle then exit
orze --stop                        # graceful stop (training detaches)
orze --disable                     # persistent disable (survives restarts)
orze --enable                      # re-enable
```

### Monitor
```bash
orze --admin                       # web UI at :8787
orze --report-only                 # regenerate leaderboard
orze --check                       # validate config, API keys, GPUs
cat results/status.json            # machine-readable status
cat results/report.md              # human-readable leaderboard
```

### Run a Single Role
```bash
orze --role-only research          # run research agent once
orze --role-only documenter        # any configured role
```

### Service (auto-restart)
```bash
orze service install -c orze.yaml  # install watchdog (crontab or systemd)
orze service status                # check health
orze service uninstall
```

### Sentinel Files (multi-machine control)
```bash
touch results/.orze_stop_all       # stop all nodes
touch results/.orze_disabled       # persistent disable
rm results/.orze_disabled          # re-enable
```

## Extending Orze

### Add a New Role (LLM Agent)
```yaml
# orze.yaml
roles:
  my_analyzer:
    mode: claude                   # or: script, research
    rules_file: ANALYZER_RULES.md  # prompt for Claude
    model: sonnet
    cooldown: 600                  # seconds between runs
    timeout: 300                   # max execution time
    allowed_tools: "Read,Glob,Grep,Bash"
```

Modes:
- `mode: claude` — spawns `claude -p <rules_file>` as subprocess
- `mode: research` — built-in multi-backend LLM agent (gemini/openai/anthropic/ollama)
- `mode: script` — any Python script: `script: my_agent.py`, `args: [...]`

Template vars in rules_file and args: `{ideas_file}`, `{results_dir}`, `{cycle}`, `{gpu_count}`, `{completed}`, `{queued}`, `{role_name}`.

### Add Pre/Post Scripts
```yaml
pre_script: check_features.py            # runs before training (exit 0 = proceed)
pre_args: ["--idea-id", "{idea_id}", "--gpu", "{gpu}"]

eval_script: evaluate.py                 # runs after successful training
eval_args: ["--idea-id", "{idea_id}", "--gpu", "{gpu}"]
eval_output: eval_report.json            # skip if exists

post_scripts:                            # run after eval
  - name: overlay
    script: generate_overlay.py
    args: ["--idea-id", "{idea_id}"]
    timeout: 1800
    output: overlay_done.json            # skip if exists
```

### Training Script Contract
Input (provided by orze):
```bash
CUDA_VISIBLE_DEVICES=N python train.py \
  --idea-id idea-001 --results-dir results \
  --ideas-md ideas.md --config base.yaml
```

Output (required):
```json
// results/{idea_id}/metrics.json
{"status": "COMPLETED", "test_accuracy": 0.92, "training_time": 142.5}
```

### Ideas Format
```markdown
## idea-001: My Experiment
- **Priority**: high
- **Category**: architecture
- **Parent**: none
- **Hypothesis**: Why this might work.

\```yaml
model:
  type: transformer
training:
  lr: 0.001
\```
```

### Report Columns from Eval Output
```yaml
report:
  columns:
    - key: auc
      source: "eval_report.json:metrics.auc_roc"
```

## Diagnosing Problems

| Symptom | Check | Fix |
|---------|-------|-----|
| No experiments running | `cat results/status.json` — check `free_gpus`, `queue_depth` | Add ideas to ideas.md, check `orze --check` |
| Training stuck | `results/{idea_id}/train_output.log` | Set `stall_minutes` in orze.yaml |
| Research agent not producing ideas | `results/_research_logs/cycle_*.log` | Check `rules_file` exists, API keys set |
| Disk full | `df -h` | Set `gc.enabled: true`, `min_disk_gb`, `cleanup.patterns` |
| Ideas duplicating | Config dedup cache at `results/_config_hashes.json` | Normal — dedup auto-skips duplicates |
| Eval not running | Check `eval_script` exists, `eval_output` not already present | Delete stale eval output to re-run |
| Multi-machine desync | `results/_host_*.json` heartbeats | Check shared filesystem mount, version compat |

## Architecture (source: `src/orze/`)

```
engine/
├── orchestrator.py       (679) — Orze class, __init__, run() recipe
├── phases.py             (600) — main loop phases (OrzePhaseMixin)
├── lifecycle.py          (390) — startup, shutdown, PID management
├── retrospection.py      (362) — signal detection + dispatch
├── experiment_analysis.py(375) — cross-experiment regression analysis
├── reporter.py           (338) — notifications + plateau detection
├── launcher.py           (324) — training subprocess launch
├── evaluator.py          (298) — eval script launch & monitoring
├── health.py             (263) — disk, stall, FS health monitoring
├── gpu_slots.py          (230) — VRAM-aware GPU scheduling
├── smart_suggestions.py  (200) — rule-based idea generation (Smart Suggestions)
├── auto_ideas.py         (180) — parameter variation generator
├── upgrade.py            (223) — auto-upgrade pipeline
├── scheduler.py          (273) — idea claiming, GPU scheduling
├── failure.py            (240) — failure tracking
├── failure_analysis.py   (210) — structured failure classification
├── family_guard.py       (180) — approach family taxonomy
├── sealed.py             (143) — sealed file integrity
├── process.py            (197) — process tracking dataclasses
├── cluster.py            (140) — multi-machine coordination
├── roles.py         (115) — role process health checks
├── config_dedup.py   (85) — config hash deduplication
└── retrospection.py  (67) — periodic analysis runner

core/
├── config.py     — config loading, validation
├── ideas.py      — ideas.md parsing, sweep expansion
└── fs.py         — atomic filesystem ops, locks

agents/
├── research.py   — multi-backend LLM idea generator
├── bug_fixer.py  — auto-diagnosis watchdog
├── train_idea.py — example training script
└── ...           — archive, GC, HF discover, notify

reporting/
├── leaderboard.py    — report generation
├── state.py          — state persistence, heartbeats
└── notifications.py  — telegram/slack/discord/webhook
```

Each module has a calling spec at the top — read it before reading the implementation.

## Admin API

`orze --admin` → http://localhost:8787

| Method | Endpoint | Returns |
|--------|----------|---------|
| GET | `/api/status` | Machine-readable pipeline status |
| GET | `/api/leaderboard` | Top models ranked by primary metric |
| GET | `/api/runs` | Active + recent runs (paginated) |
| GET | `/api/run/detail?idea_id=...` | Metrics for one run |
| GET | `/api/run/log?idea_id=...` | Tail of training log |
| GET | `/api/queue` | Filterable idea queue |
| GET | `/api/nodes` | Host heartbeats + GPU details |
| GET | `/api/alerts` | Active alerts |
| POST | `/api/ideas` | Add new idea |
| POST | `/api/actions/stop` | Stop all instances |
| POST | `/api/actions/kill` | Kill specific run |

## Notifications

Channels: telegram, slack, discord, webhook.

Events: `completed`, `failed`, `new_best`, `heartbeat`, `milestone`, `disk_warning`, `stall`, `shutdown`, `started`, `role_summary`, `upgrading`, `watchdog_restart`.

```yaml
notifications:
  enabled: true
  on: [completed, failed, new_best]
  channels:
    - type: telegram
      bot_token: "..."
      chat_id: "..."
```

## orze.yaml Quick Reference

```yaml
# Required
train_script: train.py
ideas_file: ideas.md

# Paths
base_config: configs/base.yaml
results_dir: results
python: /path/to/venv/bin/python3

# Execution
timeout: 21600
poll: 30
stall_minutes: 30
max_idea_failures: 3
max_fix_attempts: 2
min_disk_gb: 50
orphan_timeout_hours: 6
auto_upgrade: true

# Roles
roles:
  research:
    mode: claude
    rules_file: RESEARCH_RULES.md
    model: sonnet
    cooldown: 300
    timeout: 600
  code_evolution:               # requires orze-pro (triggered by retrospection on plateau)
    mode: claude
    model: opus
    timeout: 900
    triggered_by: retrospection
    allowed_tools: "Read,Write,Edit,Glob,Grep,Bash"
  meta_research:                # triggered by retrospection on family imbalance
    mode: script
    script: src/orze/agents/meta_research.py
    timeout: 300
    triggered_by: retrospection

# Sealed evaluation (metric integrity)
sealed_files: []                # e.g. ["eval.py", "data/test_set.json"]
metric_validation:
  reject_nan: true
  reject_inf: true
  min_value: {}                 # e.g. {accuracy: 0.0}
  max_value: {}                 # e.g. {accuracy: 1.0}

# Auto-evolution
evolution:
  enabled: false                # opt-in
  max_attempts_per_plateau: 2   # evolution attempts before pause

# Retrospection (periodic analysis + dispatch)
retrospection:
  enabled: false
  script: ""                    # optional custom analysis script
  interval: 50                  # trigger every N completions
  timeout: 120
  max_consecutive_family: 5     # family concentration threshold
  evolution_attempts_before_pause: 2
  dispatch:                     # signal → role mapping
    plateau: code_evolution
    family_imbalance: meta_research
    high_failure_rate: meta_research
    persistent_failure: pause

# GC
gc:
  enabled: true
  checkpoints_dir: checkpoints
  keep_top: 50

# Report
report:
  title: "My Project"
  primary_metric: accuracy
  sort: descending
  columns:
    - {key: "accuracy", label: "Acc", fmt: ".4f"}
```

### Approach Family Taxonomy

Ideas are tagged with an `approach_family` field for diversity tracking:
`architecture`, `training_config`, `data`, `infrastructure`, `optimization`, `regularization`, `ensemble`, `other`.

In `ideas.md`: `- **Approach Family**: training_config`

The research agent auto-assigns families. Retrospection detects when one family dominates and dispatches to `meta_research` to rebalance.

### Auto-Evolution Flow

```
Retrospection detects signal (plateau / failure rate / family concentration)
  → dispatch to evolution role (code_evolution or meta_research)
  → role makes changes (code patches or strategy adjustment)
  → generates new ideas exercising the changes
  → if evolution exhausted → escalate to pause
  → on metric improvement → reset attempt counters
```

### Structured Failure Analysis

Every failed experiment gets `results/{id}/failure_analysis.json`:
```json
{"category": "oom", "what": "CUDA out of memory", "why": "...", "lesson": "..."}
```
Categories: `oom`, `timeout`, `stall`, `crash`, `pre_script_error`, `eval_failure`, `config_error`, `sealed_violation`.

The research agent reads these to avoid repeating failure patterns.

Comments 0

Latest | Hot

Please login before commenting.

No comments yet. Be the first one!

My Project

Directory Structure

SKILL.md

Report

Notice