gui-agent

Category: Databases & Storage | Uploader: Fzkuji | Downloads: 0 | Version: v1.0（Latest）

Visually operate any application on macOS — native apps, browsers, or anything with a GUI window. Open apps, click buttons, type text, send messages, fill forms, navigate menus, all driven by visual UI detection. Use when the task requires interacting with what's on screen.

Changelog: Source: GitHub https://github.com/Fzkuji/GUIClaw

Directory Structure

Current level: Root

📁 actions/
- 📄 _actions.yaml 4.0 KB
📁 archive/
- 📄 SKILL.md.v1-903lines 36.5 KB
📁 assets/
- 📄 architecture.svg 16.3 KB
- 📄 banner.png 5.8 MB
- 📄 pipeline.png 115.9 KB
- 📄 pipeline.svg 7.5 KB
📁 docs/
- 📄 core.md 9.0 KB
- 📄 README_CN.md 11.4 KB
📁 scripts/
- 📄 agent.py 55.9 KB
- 📄 app_memory.py 48.3 KB
- 📄 gui_agent.py 41.8 KB
- 📄 setup.sh 3.2 KB
- 📄 template_match.py 20.5 KB
- 📄 ui_detector.py 19.8 KB
📁 skills/
- 📁 gui-act/
  - 📄 SKILL.md 3.1 KB
- 📁 gui-learn/
  - 📄 SKILL.md 3.2 KB
- 📁 gui-memory/
  - 📄 SKILL.md 4.4 KB
- 📁 gui-observe/
  - 📄 SKILL.md 2.1 KB
- 📁 gui-setup/
  - 📄 SKILL.md 2.0 KB
- 📁 gui-workflow/
  - 📄 SKILL.md 2.1 KB
📄 .gitignore 301 B
📄 LICENSE 1.0 KB
📄 README.md 15.9 KB
📄 requirements.txt 308 B
📄 SKILL.md 6.3 KB

SKILL.md

---
name: gui-agent
description: "Visually operate any application on macOS — native apps, browsers, or anything with a GUI window. Open apps, click buttons, type text, send messages, fill forms, navigate menus, all driven by visual UI detection. Use when the task requires interacting with what's on screen."
---

# GUI Agent Skill

You ARE the agent loop. Every GUI task follows this flow, in order:

```
INTENT MATCH → OBSERVE → ENSURE APP READY → ACT → VERIFY → SAVE WORKFLOW → REPORT
```

## Sub-Skills

Each step has detailed instructions in its own skill file:

| Step | Skill | When to read |
|------|-------|-------------|
| **Observe** | `skills/gui-observe/SKILL.md` | Before any action — screenshot, OCR, identify state |
| **Learn** | `skills/gui-learn/SKILL.md` | App not in memory, or match rate < 80% |
| **Act** | `skills/gui-act/SKILL.md` | Clicking, typing, sending messages, waiting for UI |
| **Memory** | `skills/gui-memory/SKILL.md` | Visual memory — profiles, components, pages, CRUD, cleanup |
| **Workflow** | `skills/gui-workflow/SKILL.md` | Intent matching, saving/replaying workflows, meta-workflows |
| **Setup** | `skills/gui-setup/SKILL.md` | First-time setup on a new machine |

Read the relevant sub-skill when you reach that step. You don't need to read all of them upfront.

## agent.py — Unified Entry Point

**All GUI operations go through `scripts/agent.py`.** Do not call `app_memory.py` or `gui_agent.py` directly.

```bash
source ~/gui-agent-env/bin/activate

# Core operations
python3 scripts/agent.py open --app AppName
python3 scripts/agent.py learn --app AppName
python3 scripts/agent.py detect --app AppName
python3 scripts/agent.py click --app AppName --component ButtonName
python3 scripts/agent.py list --app AppName
python3 scripts/agent.py read_screen --app AppName
python3 scripts/agent.py wait_for --app AppName --component X
python3 scripts/agent.py cleanup --app AppName
python3 scripts/agent.py navigate --url "https://example.com"
python3 scripts/agent.py workflows --app AppName
python3 scripts/agent.py all_workflows

# Messaging
python3 scripts/agent.py send_message --app WeChat --contact "小明" --message "明天见"
python3 scripts/agent.py read_messages --app WeChat --contact "小明"
```

agent.py automatically handles:
- New app → learn → plan
- Known app → eval (template match; ≥80% proceed, <80% re-learn)
- Error → re-learn + new plan
- Chinese aliases: 微信→WeChat, 浏览器→Chrome

## Execution Flow

### STEP -1: INTENT MATCHING
→ Details: `skills/gui-workflow/SKILL.md`

Match user request to saved workflows before doing anything. If matched, use workflow steps as plan. If not, proceed and save after success.

### STEP 0: OBSERVE
→ Details: `skills/gui-observe/SKILL.md`

Screenshot, identify current state. Record `session_status` for token reporting.

### STEP 1: ENSURE APP READY
→ Details: `skills/gui-learn/SKILL.md`

Check if app is in memory. If not → learn. If match rate < 80% → re-learn. This is YOUR responsibility — do not wait for the user.

### STEP 2: LEARN (when needed)
→ Details: `skills/gui-learn/SKILL.md`

Detect all components (YOLO + OCR), identify them, filter, save to memory. Privacy check: delete personal info.

### STEP 3: ACT
→ Details: `skills/gui-act/SKILL.md`

Execute clicks, typing, sending. Pre-verify before every click. Pre-verify contact before every message send.

### STEP 4: POST-ACTION VERIFY
→ Details: `skills/gui-act/SKILL.md`

Screenshot after every action. Did the expected change happen? If not → re-observe.

### STEP 5: SAVE WORKFLOW
→ Details: `skills/gui-workflow/SKILL.md`

Save successful multi-step sequences for future replay.

### STEP 6: REPORT

Every GUI task ends with a report:
```
⏱ 45.2s | 📊 +10k tokens (85k→95k) | 🔧 3 screenshots, 2 clicks, 1 learn
```
Compare `session_status` from STEP 0 vs now.

---

## Key Principles

1. **Vision-driven, no shortcuts** — every GUI interaction goes through the visual pipeline (screenshot → detect → match → click). Do not use system commands (`open <url>`, `osascript tell app to set URL`, CLI tools) to manipulate app state. Only allowed system calls: `activate` (bring window to front), `screencapture` (take screenshot), `cliclick` (click/type after visual detection provides coordinates).
2. **Memory first, detect second** — template match before YOLO+OCR
3. **Template > OCR > YOLO > LLM** — cheapest method first
4. **Relative coordinates** — all positions relative to window top-left
5. **Window-based, not screen-based** — capture and operate within target window only
6. **Paste > Type** for CJK text and special chars
7. **Learn incrementally** — save new components after each interaction
8. **Integer coordinates only** — cliclick requires integers
9. **Learn once, match forever** — UI positions are stable unless app updates

## Safety Rules

These exist because of real bugs:

1. **Verify before sending** — always OCR the chat header to confirm correct contact
2. **Stay within window bounds** — get bounds first, filter all OCR/clicks
3. **No wrong-app learning** — validate window bounds before auto_learn
4. **Reject tiny templates** — <30×30 pixels produce false matches
5. **LLM never provides coordinates** — detection tools provide coordinates, you decide what to click by name
6. **Never send screenshots to conversation** — internal detection only

## Memory System
→ Details: `skills/gui-memory/SKILL.md`

Visual memory stores app profiles, components, page fingerprints, workflows. See gui-memory for directory structure, profile schema, CRUD operations, and cleanup rules.

## File Structure

```
gui-agent/
├── SKILL.md              # This file (main orchestrator)
├── skills/               # Sub-skills (read on demand)
│   ├── gui-observe/SKILL.md
│   ├── gui-learn/SKILL.md
│   ├── gui-act/SKILL.md
│   ├── gui-memory/SKILL.md
│   ├── gui-workflow/SKILL.md
│   └── gui-setup/SKILL.md
├── scripts/              # Core scripts
│   ├── agent.py, ui_detector.py, app_memory.py, gui_agent.py, template_match.py
├── memory/               # Visual memory (gitignored)
│   ├── apps/<appname>/
│   └── meta_workflows/
├── actions/              # Atomic operations
├── docs/
└── README.md
```

Comments 0

Latest | Hot

Please login before commenting.

No comments yet. Be the first one!

gui-agent

Directory Structure

SKILL.md

Report

Notice