activity-monitor

Category: Tools & Productivity | Uploader: zylos-ai | Downloads: 0 | Version: v1.0（Latest）

Guardian service that monitors the active runtime agent's state and automatically restarts it if stopped. Use when checking agent liveness state or understanding the auto-restart mechanism.

Changelog: Source: GitHub https://github.com/zylos-ai/zylos-core

Directory Structure

Current level: skills/activity-monitor/

📁 scripts/
- 📁 __tests__/
  - 📄 daily-schedule.test.js 3.8 KB
  - 📄 heartbeat-config.test.js 912 B
  - 📄 heartbeat-engine.test.js 37.6 KB
  - 📄 usage-check-engine.test.js 1.7 KB
  - 📄 usage-check-init.test.js 1.1 KB
  - 📄 usage-codex-probe-runner.test.js 2.3 KB
  - 📄 usage-codex-rollout-reader.test.js 2.2 KB
  - 📄 usage-codex-status-parser.test.js 1.3 KB
  - 📄 usage-probe-lock.test.js 2.0 KB
  - 📄 usage-probe-parser.test.js 932 B
  - 📄 usage-probe-runner.test.js 1.0 KB
- 📄 activity-monitor.js 75.0 KB
- 📄 context-monitor.js 6.2 KB
- 📄 daily-schedule.js 1.8 KB
- 📄 heartbeat-config.js 608 B
- 📄 heartbeat-engine.js 18.0 KB
- 📄 hook-activity.js 4.9 KB
- 📄 proc-sampler.js 8.5 KB
- 📄 session-start-prompt.js 1.7 KB
- 📄 upgrade-check.js 5.3 KB
- 📄 usage-check-engine.js 984 B
- 📄 usage-check-init.js 447 B
- 📄 usage-codex-probe-runner.js 5.4 KB
- 📄 usage-codex-rollout-reader.js 4.0 KB
- 📄 usage-codex-status-parser.js 1.6 KB
- 📄 usage-probe-lock.js 2.0 KB
- 📄 usage-probe-parser.js 1.0 KB
- 📄 usage-probe-runner.js 4.4 KB
📄 package.json 146 B
📄 SKILL.md 9.1 KB

SKILL.md

---
name: activity-monitor
description: Guardian service that monitors the active runtime agent's state and automatically restarts it if stopped. Use when checking agent liveness state or understanding the auto-restart mechanism.
user-invocable: false
---

# Activity Monitor Skill

PM2 guardian service that monitors the active runtime agent's (Claude or Codex) activity state and ensures it's always running.

## When to Use

This is a **PM2 service** (not directly invoked by Claude). It runs continuously in the background.

## What It Does

1. **Activity Monitoring**: Tracks the agent's busy/idle state every second
2. **Status File**: Writes `~/zylos/activity-monitor/agent-status.json` with current state (busy/idle, idle_seconds, health)
3. **Guardian Mode**: Automatically restarts the agent if it stops or crashes
4. **Maintenance Awareness**: Waits for restart/upgrade scripts to complete before starting the agent
5. **Heartbeat Liveness Detection**: Periodically sends heartbeat probes via the C4 control queue to verify the agent is responsive, triggering recovery when probes fail
6. **Health Check**: Periodically enqueues system health checks (PM2, disk, memory) via the C4 control queue
7. **Daily Upgrade**: Enqueues a Claude Code upgrade via the C4 control queue at 5:00 AM local time daily (Claude runtime only)
8. **Context Monitoring**: Receives context usage data after every turn; triggers new-session handoff when usage exceeds 70%

## Status File Format

```json
{
  "state": "idle",
  "last_activity": 1738675200,
  "last_check": 1738675210,
  "last_check_human": "2026-02-04 20:30:10",
  "idle_seconds": 5,
  "inactive_seconds": 10,
  "source": "conv_file",
  "health": "ok"
}
```

- `state`: "busy" | "idle" | "stopped" | "offline"
- `idle_seconds`: Time since entering idle state (0 when busy)
- `source`: "conv_file" (reliable) | "tmux_activity" (fallback)
- `health`: "ok" | "recovering" | "down" — liveness health from the heartbeat engine

## PM2 Management

```bash
# Start
pm2 start ~/zylos/.claude/skills/activity-monitor/scripts/activity-monitor.js --name activity-monitor

# Restart
pm2 restart activity-monitor

# View logs
pm2 logs activity-monitor

# Check status
pm2 list
```

## Guardian Behavior

- **Detection**: Checks if the agent is running every second
- **Restart Delay**: Waits 5 seconds of continuous "not running" before restarting
- **Maintenance Wait**: Detects restart/upgrade scripts and waits for completion
- **Recovery Prompt**: Sends catch-up message via C4 after restart

## How It Works

1. **Monitor Loop** (every 1s):
   - Check if tmux session exists
   - Check if agent process is running
   - Detect activity (conversation file mtime for Claude; tmux activity for Codex)
   - Calculate idle/busy state
   - Write status to ~/zylos/activity-monitor/agent-status.json

2. **Guardian Logic**:
   - If agent not running for 5+ seconds → restart
   - Wait for maintenance scripts to complete
   - Send recovery prompt via C4 after successful restart

3. **Activity Detection**:
   - Claude primary: Conversation file modification time (reliable)
   - Codex / fallback: tmux window activity
   - Threshold: 3 seconds without activity = idle

## Heartbeat Liveness Detection

The heartbeat engine runs inside the activity monitor and uses the C4 control queue to verify Claude is actually responsive (not just process-alive).

### State Machine

```
          heartbeat interval elapsed
  ┌─────────────────────────────────────┐
  │                                     ▼
  │  ok ──[primary fails]──► ok (verify phase)
  │  ▲                              │
  │  │                     [verify fails]
  │  │                              ▼
  │  │                         recovering ──[recovery fails]──► recovering
  │  │                              │                               │
  │  │                    [ack received]              [max failures reached]
  │  │                              │                               │
  │  └──────────────────────────────┘                               ▼
  │                                                               down
  │                                                                 │
  │                                          [ack received after manual fix]
  └─────────────────────────────────────────────────────────────────┘
```

### Phases

| Phase | Trigger | On Success | On Failure |
|-------|---------|------------|------------|
| **Primary** | Heartbeat interval elapsed (default 30min) | Reset timer, stay `ok` | Enter verify phase |
| **Verify** | Primary probe failed | Return to `ok` | Kill tmux, enter `recovering` |
| **Recovery** | In `recovering` state, Claude restarted | Return to `ok`, notify pending channels | Kill tmux, retry (up to max) |
| **Down** | Max restart failures reached (default 3) | Return to `ok`, notify pending channels | Stay `down`, wait for manual fix |

### Ack Deadline

Each heartbeat probe is enqueued with an ack deadline. If Claude does not acknowledge the probe before the deadline expires, the control record transitions to `timeout` status and the engine treats it as a failure.

### Recovery Behavior

When health transitions back to `ok`, the engine reads `~/zylos/activity-monitor/pending-channels.jsonl` and sends a recovery notification to each recorded channel/endpoint via C4, then clears the file.

## Health Check

The activity monitor periodically enqueues system health checks via the C4 control queue.

- **Interval**: Every 6 hours (21600 seconds)
- **Persisted state**: `~/zylos/activity-monitor/health-check-state.json` (survives restarts)
- **Priority**: 3 (normal)
- **Gated by health**: Only enqueued when `health === 'ok'`
- **Gated by agent**: Only enqueued when the agent process is running

The health check control message instructs Claude to:
1. Check PM2 services via `pm2 jlist`
2. Check disk space via `df -h`
3. Check memory via `free -m`
4. If issues found, notify the most recent communication channel
5. Log results to `~/zylos/logs/health.log`
6. Acknowledge the control message

## Daily Upgrade

The activity monitor enqueues a daily Claude Code upgrade via the C4 control queue.

- **Schedule**: 5:00 AM local time (configured by `DAILY_UPGRADE_HOUR`)
- **Timezone**: Loaded from `~/zylos/.env` `TZ` field, falls back to `process.env.TZ`, then `UTC`
- **Persisted state**: `~/zylos/activity-monitor/daily-upgrade-state.json` (tracks last upgrade date)
- **Priority**: 3 (normal)
- **Gated by health**: Only enqueued when `health === 'ok'`
- **Gated by agent**: Only enqueued when the agent process is running (Claude runtime only)
- **Once per day**: Checks local date to avoid duplicate enqueues

The control message instructs Claude to use the `upgrade-claude` skill, which handles idle detection, `/exit`, upgrade, and automatic restart.

## Context Monitoring

Event-driven context monitoring via Claude Code's statusLine feature, replacing the old hourly polling.

- **Mechanism**: `context-monitor.js` runs as a statusLine command — Claude Code pipes context data to it via stdin after every turn (zero turn cost)
- **Status file**: Writes `~/zylos/activity-monitor/statusline.json` with full context data (used_percentage, remaining_percentage, cost, session_id)
- **Threshold**: Triggers new-session handoff when `used_percentage >= 70%`
- **Cooldown**: 10 minutes between triggers (state in `~/zylos/activity-monitor/context-monitor-state.json`)
- **Delivery**: Enqueues via C4 control queue with priority 1, no require-idle — ensures the trigger reaches Claude even during long tasks
- **Two-stage design**: The trigger message instructs Claude to start the new-session handoff flow; the actual `/clear` is gated by require-idle in the new-session skill's final step
- **Log**: `~/zylos/activity-monitor/context-monitor.log`

The `check-context` skill remains available for manual on-demand context checks.

## Daily Memory Commit

The activity monitor runs a daily git commit of the `memory/` directory.

- **Schedule**: 3:00 AM local time
- **Timezone**: Same as daily upgrade (from `~/zylos/.env` TZ)
- **Persisted state**: `~/zylos/activity-monitor/daily-memory-commit-state.json`
- **Script**: Calls `zylos-memory/scripts/daily-commit.js` directly (no C4, no Claude needed)
- **Once per day**: Date-based deduplication
- **Idempotent**: If no memory changes exist, the script skips the commit

## Timing Guarantees

Both daily tasks (upgrade, memory commit) use the same `DailySchedule` class:
- **Window**: Each target hour provides a ~3600s window; with ~1s loop interval it is virtually impossible to miss entirely
- **Dedup**: Date-based (`last_date === today`) ensures exactly-once per day, even with imprecise timing
- **Persistence**: State files survive activity monitor restarts

## Log File

Activity log: `~/zylos/activity-monitor/activity.log`
- Auto-truncates to 500 lines daily
- Logs state changes and guardian actions

Comments 0

Latest | Hot

Please login before commenting.

No comments yet. Be the first one!

activity-monitor

Directory Structure

SKILL.md

Report

Notice