obol-stack-dev

Category: 数据与AI | Uploader: ObolNetwork | Downloads: 0 | Version: v1.0（Latest）
Obol Stack development, testing, and LLM routing validation through LiteLLM. Use when developing, testing, or validating inference paths (Ollama, Anthropic, OpenAI) through the LiteLLM gateway, writing integration tests, or working with obol CLI wrappers.
Changelog: Source: GitHub https://github.com/ObolNetwork/obol-stack
Directory Structure

Current level: .agents/skills/obol-stack-dev/
📁 references/
- 📄 dev-environment.md 3.1 KB
- 📄 integration-testing.md 6.7 KB
- 📄 litellm-routing.md 6.1 KB
- 📄 obol-cli.md 5.7 KB
- 📄 overlay-generation.md 6.3 KB
- 📄 troubleshooting.md 6.1 KB
📄 SKILL.md 11.2 KB
SKILL.md

---
name: obol-stack-dev
description: Obol Stack development, testing, and LLM routing validation through LiteLLM. Use when developing, testing, or validating inference paths (Ollama, Anthropic, OpenAI) through the LiteLLM gateway, writing integration tests, or working with obol CLI wrappers.
metadata:
  version: "2.0.0"
  domain: infrastructure
  triggers: obol, litellm, openclaw, inference, integration test, model routing, smart routing, LLM proxy, provider setup
  role: specialist
  scope: development-and-testing
  output-format: code-and-commands
  related-skills: golang-pro, helm-chart-patterns
---

# Obol Stack Dev & LLM Routing Validation

Complete guide for developing, testing, and validating the Obol Stack's LLM routing through LiteLLM. Covers the dev environment, CLI wrappers, overlay generation, all 3 provider paths, and integration testing.

## When to Use This Skill

- Setting up the Obol Stack development environment
- Testing LLM inference through LiteLLM (Ollama, Anthropic, OpenAI)
- Writing or running integration tests for OpenClaw instances
- Debugging model routing issues (401s, 500s, provider misconfig)
- Understanding the 2-tier LLM architecture (LiteLLM gateway + per-instance config)
- Deploying and validating OpenClaw instances with different providers
- Working with the `obol` CLI wrappers (kubectl, helm, helmfile, k9s)

## Architecture Overview

The stack uses a **2-tier LLM routing** architecture:

```
Tier 2: Per-Instance                Tier 1: Cluster-Wide Gateway
(OpenClaw in openclaw-<id> ns)      (LiteLLM in llm ns)

+---------------------------+       +---------------------------+
| OpenClaw                  |       | LiteLLM (port 4000)       |
| model: openai/<model-id>  | ----> | Routes by model_list:     |
| api: openai-completions   |       |   ollama_chat/* -> Ollama  |
| baseUrl: litellm:4000/v1  |       |   claude-*  -> Anthropic   |
+---------------------------+       |   gpt-*     -> OpenAI      |
                                    +---------------------------+
                                          |       |       |
                                          v       v       v
                                       Ollama  Anthropic OpenAI
                                       (host)   (cloud)  (cloud)
```

**Key insight**: All traffic routes through LiteLLM regardless of provider. OpenClaw uses the `openai` provider slot (since LiteLLM speaks the OpenAI API protocol) with `openai-completions` API format. LiteLLM resolves the actual upstream provider by model name via its `model_list` config.

## Quick Reference

| Task | Reference |
|------|-----------|
| Dev environment setup | `references/dev-environment.md` |
| LLM routing architecture | `references/litellm-routing.md` |
| CLI wrappers and commands | `references/obol-cli.md` |
| Overlay generation (values-obol.yaml) | `references/overlay-generation.md` |
| Integration testing | `references/integration-testing.md` |
| Troubleshooting | `references/troubleshooting.md` |

## 3 Inference Paths (All Through LiteLLM)

| Path | Model Name | LiteLLM model_list | Example |
|------|-----------|-------------------|---------|
| **Ollama** (default) | `<model>` | `ollama_chat/<model>` → Ollama svc | `llama3.2:3b` |
| **Anthropic** (cloud) | `<claude-model>` | `<claude-model>` → Anthropic API | `claude-sonnet-4-5-20250929` |
| **OpenAI** (cloud) | `<gpt-model>` | `<gpt-model>` → OpenAI API | `gpt-4o` |

All 3 paths use the same OpenClaw config pattern:
- Provider slot: `openai` (LiteLLM is OpenAI-API-compatible)
- API: `openai-completions`
- Base URL: `http://litellm.llm.svc.cluster.local:4000/v1`
- API key: LiteLLM master key (`sk-obol-<cluster-id>`)

## Essential Commands

```bash
# --- Dev Environment ---
OBOL_DEVELOPMENT=true ./obolup.sh      # Bootstrap dev mode
go build -o .workspace/bin/obol ./cmd/obol  # Build binary

# --- Stack Lifecycle ---
obol stack init && obol stack up        # Start cluster
obol stack down                         # Stop (preserves data)
obol stack purge -f                     # Destroy everything

# --- Model Provider Setup (Tier 1: LiteLLM) ---
obol model setup --provider anthropic --api-key sk-ant-...
obol model setup --provider openai --api-key sk-proj-...
obol model setup --provider ollama      # Auto-discovers local models
obol model status                       # Show enabled providers

# --- OpenClaw Instance Management (Tier 2) ---
obol openclaw onboard --id my-agent     # Interactive deploy
obol openclaw sync <id>                 # Deploy/update instance
obol openclaw token <id>                # Get gateway Bearer token
obol openclaw list                      # Show all instances
obol openclaw delete --force <id>       # Remove instance
obol openclaw dashboard <id>            # Open web UI

# --- Debugging ---
obol kubectl get pods -n openclaw-<id>
obol kubectl logs -n openclaw-<id> -l app.kubernetes.io/instance=openclaw
obol kubectl port-forward -n openclaw-<id> svc/openclaw 18789:18789

# --- Testing ---
go test ./internal/openclaw/                                    # Unit tests
go test -tags integration -v -timeout 10m ./internal/openclaw/  # Integration tests
```

## OpenClaw Skills System

Skills are SKILL.md files (with optional scripts and references) that give the agent domain-specific capabilities. Delivered via host-path PVC injection to `/data/.openclaw/skills/` inside the pod.

### Default Embedded Skills

| Skill | Contents | Purpose |
|-------|----------|---------|
| `hello` | `SKILL.md` | Smoke test |
| `obol-blockchain` | `SKILL.md`, `scripts/rpc.py`, `references/` | Ethereum JSON-RPC, ERC-20, ENS via eRPC |
| `obol-k8s` | `SKILL.md`, `scripts/kube.py` | K8s cluster diagnostics via ServiceAccount API |
| `obol-dvt` | `SKILL.md`, `references/api-examples.md` | DVT monitoring via Obol API |

### Skills CLI

```bash
obol openclaw skills list                   # list installed skills
obol openclaw skills sync                   # re-inject embedded defaults
obol openclaw skills sync --from ./custom   # push custom skills
obol openclaw skills add <package>          # add via openclaw CLI in pod
obol openclaw skills remove <name>          # remove skill from pod
```

### Skills Delivery Flow

1. `stageDefaultSkills(deploymentDir)` — copies embedded skills to deployment dir
2. `injectSkillsToVolume(cfg, id, deploymentDir)` — copies to host PVC path (`$DATA_DIR/openclaw-<id>/openclaw-data/.openclaw/skills/`)
3. `doSync()` — helmfile sync; OpenClaw file watcher discovers skills on startup

### Skills Testing

```bash
# Unit tests (embedding + injection)
go test -v -run TestGetEmbeddedSkillNames ./internal/embed/
go test -v -run TestInjectSkillsToVolume ./internal/openclaw/

# Integration tests (requires running cluster)
go test -tags integration -v -run TestIntegration_Skills -timeout 10m ./internal/openclaw/

# In-pod smoke tests (piped via kubectl exec)
obol kubectl exec -i -n openclaw-<id> deploy/openclaw -c openclaw -- python3 - < tests/skills_smoke_test.py
```

## Key Source Files

| File | Purpose |
|------|---------|
| `internal/openclaw/openclaw.go` | `Onboard()`, `Sync()`, `Delete()`, `buildLiteLLMRoutedOverlay()`, `generateOverlayValues()`, `SyncOverlayModels()` |
| `internal/openclaw/import.go` | `DetectExistingConfig()`, `TranslateToOverlayYAML()` |
| `internal/openclaw/overlay_test.go` | Unit tests for overlay generation |
| `internal/openclaw/integration_test.go` | Full-cluster integration tests (build tag: `integration`) |
| `internal/model/model.go` | `ConfigureLiteLLM()` — patches LiteLLM ConfigMap + Secret + restart |
| `cmd/obol/model.go` | `obol model setup` CLI command (also syncs OpenClaw overlays) |
| `cmd/obol/openclaw.go` | `obol openclaw` CLI commands (including `skills` subcommands) |
| `internal/embed/infrastructure/base/templates/llm.yaml` | LiteLLM + Ollama Kubernetes resources |
| `internal/embed/skills/` | Embedded default skills |
| `internal/embed/embed.go` | `CopySkills()`, `GetEmbeddedSkillNames()` |

## Constraints

### MUST DO
- Always route through `obol` CLI verbs in tests (covers CLI + helmfile + helm chart)
- Use `obol openclaw token <id>` to get Bearer token before API calls
- Set `Authorization: Bearer <token>` on all `/v1/chat/completions` requests
- Use `obol model setup --provider <name> --api-key <key>` for cloud provider config
- Wait for pod readiness AND HTTP readiness before sending inference requests
- Clean up test instances with `obol openclaw delete --force <id>` (flag BEFORE arg)
- Set env vars for dev mode: `OBOL_DEVELOPMENT=true`, `OBOL_CONFIG_DIR`, `OBOL_BIN_DIR`, `OBOL_DATA_DIR`

### MUST NOT DO
- Call internal Go functions directly when testing the deployment path
- Skip the gateway token (causes 401 Unauthorized)
- Put `--force` flag after the argument in `obol openclaw delete` (urfave/cli v2 quirk)
- Assume TCP connectivity means HTTP is ready (port-forward warmup race)
- Use `app.kubernetes.io/instance=openclaw-<id>` for pod labels (Helm uses `openclaw`)
- Run multiple integration tests without cleaning up between them (pod sandbox errors)

## Sell-Side Monetize Lifecycle

### Architecture

The monetize subsystem enables pay-per-request access to local compute via x402:

```
ServiceOffer CR → monetize.py reconciliation → Middleware + HTTPRoute + pricing route
                                                       │
Client request ──► Traefik ──► x402-verifier (ForwardAuth) ──► backend (Ollama)
                                    │                              │
                               402 (no payment)              200 (valid payment)
                               Payment requirements          Inference response
```

### Three-Layer Integration

1. **monetize.py** (OpenClaw skill) — Creates Middleware, HTTPRoute, pricing ConfigMap route
2. **x402-verifier** (ForwardAuth) — Checks X-PAYMENT header against facilitator
3. **Traefik Gateway API** — Routes traffic; requires ClusterIP backends (not ExternalName)

### Testing the Monetize Flow

```bash
# Prerequisites
obol stack up && obol agent init

# Create offer
obol sell http qwen35 --model "qwen3.5:35b" --per-request "0.001" \
  --network "base-sepolia" --pay-to "0x<wallet>"

# Trigger reconciliation (or wait for heartbeat)
obol kubectl exec -n openclaw-obol-agent deploy/openclaw -c openclaw -- \
  python3 /data/.openclaw/skills/monetize/scripts/monetize.py process qwen35 --namespace llm

# Verify 402
curl -X POST http://obol.stack:8080/services/qwen35/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.5:35b","messages":[{"role":"user","content":"hi"}],"stream":false}'

# Run e2e test (with mock facilitator)
export OBOL_DEVELOPMENT=true OBOL_CONFIG_DIR=$(pwd)/.workspace/config OBOL_BIN_DIR=$(pwd)/.workspace/bin
go test -tags integration -v -run TestIntegration_PaymentGate_FullLifecycle -timeout 5m ./internal/x402/
```

### Known Gotchas

- **ExternalName services**: Traefik Gateway API rejects ExternalName as HTTPRoute backends → 500 after valid payment. Use ClusterIP+Endpoints.
- **Model pull timeout**: monetize.py checks `/api/tags` before `/api/pull` to avoid hanging on cached models.
- **Facilitator HTTPS**: URLs must be HTTPS except localhost, 127.0.0.1, host.k3d.internal, host.docker.internal.
- **ConfigMap propagation**: File watcher takes 60-120s. Force restart verifier for immediate effect.
Comments 0
Latest | Hot
Please login before commenting.
No comments yet. Be the first one!
obol-stack-dev

Directory Structure

SKILL.md

Report

Notice
