anysite-cli

分类: 数据库与数据存储 | 上传者: anysiteio | 下载: 0 | 版本: v1.0（最新）
Operate the anysite command-line tool for web data extraction, batch API processing, multi-source dataset pipelines with scheduling/transforms/exports, database operations, and LLM-powered data analysis. Use when users ask to collect data from LinkedIn, Instagram, Twitter, or any web source via CLI; create or run dataset pipelines; schedule automated collection; batch-process API calls; query collected data with SQL; load data into PostgreSQL or SQLite; analyze data with LLM (summarize, classify, enrich, match, deduplicate); or work with anysite commands. Triggers on anysite CLI usage, data collection, dataset creation, scraping, API batch calls, scheduling, database loading, or LLM analysis tasks.
更新日志: Source: GitHub https://github.com/anysiteio/anysite-cli
目录结构

当前层级: skills/anysite-cli/
📁 references/
- 📄 api-reference.md 6.7 KB
- 📄 dataset-guide.md 30.1 KB
- 📄 llm-reference.md 9.1 KB
📄 SKILL.md 23.5 KB
SKILL.md

---
name: anysite-cli
description: Operate the anysite command-line tool for web data extraction, batch API processing, multi-source dataset pipelines with scheduling/transforms/exports, database operations, and LLM-powered data analysis. Use when users ask to collect data from LinkedIn, Instagram, Twitter, or any web source via CLI; create or run dataset pipelines; schedule automated collection; batch-process API calls; query collected data with SQL; load data into PostgreSQL or SQLite; analyze data with LLM (summarize, classify, enrich, match, deduplicate); or work with anysite commands. Triggers on anysite CLI usage, data collection, dataset creation, scraping, API batch calls, scheduling, database loading, or LLM analysis tasks.
---

# Anysite CLI

Command-line tool for web data extraction, dataset pipelines, and database operations.

## Agent Planning Workflow

**BEFORE planning any data collection task, follow this sequence:**

1. **Discover available endpoints**
   ```bash
   anysite describe --search "<keyword>"   # Search by domain (linkedin, company, user, etc.)
   ```

2. **Select endpoints needed for the task** — identify which endpoints will provide the required data

3. **Inspect each selected endpoint**
   ```bash
   anysite describe /api/linkedin/company  # View input params and output fields
   ```

4. **Only then plan** — now you know the exact parameters, field names, and data structure to build your config or API calls

This prevents errors from wrong endpoint paths, missing required parameters, or incorrect field names in dependencies.

## Best Practices

1. **Use dataset pipelines for multi-step tasks**
   - If a task requires sequential API calls, LLM enrichment, or chained data processing — create a `dataset.yaml` config instead of running multiple ad-hoc commands
   - Dataset pipelines handle dependencies, incremental collection, and error recovery automatically
   - Even for "simple" tasks that grow in scope, a dataset config is easier to maintain
   - Benefits: run history, incremental sync, scheduling, notifications, DB loading

2. **Save data in Parquet format by default** — unless user requests another format or CSV/JSON fits better

3. **Prefer datasets over ad-hoc scripts** — one dataset.yaml replaces dozens of shell commands

## Quick Start Checklist

Before any data collection task:

```bash
# 1. Check CLI is available and see latest changes
anysite --version
anysite changelog --last 1 --json            # Check what's new in this version
# If not found: source .venv/bin/activate or pip install anysite-cli

# 2. Update schema cache (required for endpoint discovery)
anysite schema update

# 3. Verify API key
anysite config get api_key
# If not set: anysite config set api_key sk-xxxxx
```

After upgrading, run `anysite changelog --since <old_version> --json` to discover new features.

## Endpoint Discovery

**ALWAYS discover endpoints before writing API calls or dataset configs:**

```bash
anysite describe                          # List all endpoints
anysite describe --search "company"       # Search with dependency context
anysite describe /api/linkedin/company    # Full details: params, output, connections

```

Search returns matched endpoints plus upstream providers (who can supply input IDs) and downstream consumers (who can use output IDs). Use this to plan endpoint chains for dataset pipelines.

Input params show type, description, examples, and defaults. Array params show item structure:
```
Input parameters:
  * urn                            string      User URN, only fsd_profile urn type is allowed
                                               example: "urn:li:fsd_profile:ACoAABXy1234"
    count                          integer     Number of posts to return
                                               default: 20
    companies                      array[object{type,value}]  Company URNs
                                               example: [{"type": "company", "value": "14064608"}]
```

## Prerequisites

```bash
pip install "anysite-cli[data]"        # DuckDB + PyArrow for dataset commands
pip install "anysite-cli[llm]"         # LLM analysis (openai/anthropic)
pip install "anysite-cli[postgres]"    # PostgreSQL adapter
pip install "anysite-cli[clickhouse]"  # ClickHouse adapter

anysite config set api_key sk-xxxxx   # Configure API key
anysite schema update                  # Update schema cache
anysite llm setup                      # Interactive setup (human)
anysite llm setup --provider openai --api-key sk-xxx --no-test    # Non-interactive (agent)
anysite llm setup --provider anthropic --api-key-env ANTHROPIC_API_KEY --no-test
anysite db add pg --type postgres --host localhost --database mydb --user app --password secret
# Or via env var: anysite db add pg ... --password-env PGPASS
anysite db add ch --type clickhouse --host ch.example.com --port 8443 --database analytics --user app --password secret --ssl
```

## Authentication

```bash
anysite auth login                        # Interactive browser-based OAuth2 (human)
anysite auth login --force --no-browser   # Re-authenticate without confirmation (agent)
anysite auth status                       # Check current auth status
anysite auth status --json                # Machine-readable auth status
anysite auth logout                       # Interactive logout (human)
anysite auth logout --force               # Logout without confirmation (agent)
```

## Single API Call

```bash
anysite api /api/linkedin/user user=satyanadella
anysite api /api/linkedin/company company=anthropic --format table
anysite api /api/linkedin/search/users keywords="CTO" count=50 --format csv --output ctos.csv
anysite api /api/linkedin/user user=satyanadella --fields "name,headline,urn.value" -q | jq
```

### URN/Name Parameter Formats

Parameters like `location`, `current_companies`, `industry` accept two formats:

```bash
# Single name (text search) — resolves to URNs automatically
location="London"
current_companies="Microsoft"

# Multiple URNs (direct) — use JSON array in single quotes
'location=["urn:li:geo:101165590", "urn:li:geo:101282230"]'
'current_companies=["urn:li:company:1035", "urn:li:company:1441"]'
```

**Note:** List of names `["Microsoft", "Google"]` is NOT supported — use either one name OR multiple URNs.

## Batch Processing

```bash
anysite api /api/linkedin/user --from-file users.txt --input-key user \
  --parallel 5 --rate-limit "10/s" --on-error skip --progress --stats
```

## Dataset Pipeline (Multi-Source Collection)

For complex data collection with dependencies, LLM enrichment, scheduling — use dataset pipelines.

### Initialize

```bash
anysite dataset init my-dataset
# Creates my-dataset/dataset.yaml with template config
```

### Six Source Types

1. **Independent** — single API call with static `input`
2. **from_file** — batch calls iterating over input file values
3. **Dependent** — batch calls using values extracted from a parent source
4. **Union (type: union)** — combine records from multiple parent sources into one
5. **LLM (type: llm)** — process parent data through LLM without API calls
6. **SQL (type: sql)** — query a database connection or dataset Parquet files via DuckDB

### Comprehensive Dataset YAML Reference

```yaml
name: my-dataset                    # Dataset name (required)
description: Optional description   # Human-readable description

sources:
  # === TYPE 1: Independent source (single API call) ===
  - id: search_results              # Unique identifier (required)
    endpoint: /api/linkedin/search/users  # API endpoint (required for type: api)
    input:                          # Static API parameters
      keywords: "software engineer"
      count: 50
    parallel: 1                     # Concurrent requests: 1-10 (default: 1)
    rate_limit: "10/s"              # Rate limit: "N/s", "N/m", "N/h"
    on_error: stop                  # Error handling: stop | skip (default: stop)

  - id: search_extra                # Another search (can be combined with union)
    endpoint: /api/linkedin/search/users
    input: { keywords: "data engineer", count: 50 }

  # === TYPE 2: from_file source (batch from file) ===
  - id: companies
    endpoint: /api/linkedin/company
    from_file: companies.txt        # Input file: .txt (line per value), .csv, .jsonl
    file_field: company_slug        # CSV column name (for CSV files only)
    input_key: company              # API parameter to fill with each value
    parallel: 3

  # === TYPE 3: Dependent source (values from parent) ===
  - id: employees
    endpoint: /api/linkedin/company/employees
    dependency:
      from_source: companies        # Parent source ID (required)
      field: urn.value              # Dot-notation path to extract from parent records
      match_by: name                # Alternative: fuzzy match instead of exact field
      dedupe: true                  # Remove duplicate values (default: false)
    input_key: companies            # API parameter for extracted values
    input_template:                 # Transform values before API call
      companies:
        - type: company
          value: "{value}"          # {value} = extracted value placeholder
      count: 5
    refresh: auto                   # Incremental behavior: auto (default) | always | never
                                    # never = skip if data exists

  # === Shorthand for dependent sources ===
  # ${source.field} auto-expands to dependency + input_key:
  - id: profiles
    endpoint: /api/linkedin/user
    input:
      user: ${companies.urn.value}    # Equivalent to dependency + input_key above

  # === TYPE 4: Union source (combine multiple sources) ===
  - id: all_search_results
    type: union                     # Source type: api (default) | union | llm | sql
    sources: [search_results, search_extra]  # Parent source IDs to combine (required)
    dedupe_by: urn.value            # Optional: field path for deduplication (dot-notation)
    # NOTE: type: union cannot have endpoint, dependency, from_file, input_key, input
    # NOTE: all sources in the list must have the same endpoint (same data structure)
    # Records are annotated with _union_source = parent source ID

  # === TYPE 5: LLM source (process parent data without API) ===
  - id: employees_analyzed
    type: llm                       # Source type: api (default) | union | llm | sql
    dependency:
      from_source: employees
      field: name                   # Required by schema (not used for LLM sources)
    llm:                            # LLM enrichment steps (required for type: llm)
      - type: classify              # Step types: classify | enrich | summarize | generate
        categories: "developer,recruiter,executive"  # Comma-separated (omit for auto-detect)
        output_column: role_type    # Output column name (default: category)
        fields: [headline]          # Record fields to include in LLM prompt
      - type: enrich
        add:                        # Field specs (required for enrich)
          - "sentiment:positive/negative/neutral"   # Enum: value1/value2/value3
          - "language:string"                       # Types: string | number | integer | boolean
          - "quality_score:1-10"                    # Range: min-max
        fields: [headline, summary]
        temperature: 0.0            # LLM temperature: 0.0-1.0 (default: 0.0)
        provider: openai            # Provider override: openai | anthropic
        model: gpt-4o-mini          # Model override
      - type: summarize
        max_length: 50              # Max words (default: 100)
        output_column: bio
      - type: generate
        prompt: "Write pitch for {name}"  # Template with {field} placeholders (required)
        output_column: pitch
        temperature: 0.7            # Higher for creative text
    export:                         # Export destinations (runs after Parquet write)
      - type: file
        path: ./output/{{source}}-{{date}}.csv  # Templates: {{date}}, {{source}}, {{dataset}}
        format: csv                 # Format: json | jsonl | csv
    # NOTE: type: llm cannot have endpoint, from_file, input_key, input

  # === TYPE 6: SQL source (query a database) ===
  # Mode A: Named connection (query external database)
  - id: billing_users
    type: sql
    connection: billing            # Named connection from 'anysite db add'
    query: "SELECT name, email FROM subscriptions WHERE status = 'inactive'"
    # query_file: queries/inactive.sql   # Alternative: read SQL from file
    filter: ".email != null"       # All base fields work: filter, llm, transform, export, db_load

  # Mode B: Dataset views (no connection — query Parquet via DuckDB)
  # Each collected source becomes a view by its id (hyphens → underscores)
  - id: enriched_profiles
    type: sql
    query: |
      SELECT u.*, c.description as company_desc
      FROM user_profiles u
      LEFT JOIN company_profiles c ON u._input_value = c._input_value
    # Use for cross-source JOINs within the pipeline — no external DB needed
    # NOTE: type: sql cannot have endpoint, from_file, input_key, input, parallel, rate_limit, on_error

  # === OPTIONAL BLOCKS (any source type) ===
  # THREE-LEVEL FILTERING:
  # Level 1: source.filter — before LLM + Parquet (saves tokens, drops records entirely)
  # Level 2: transform.filter — before exports only (Parquet keeps all records)
  # Level 3: db_load.filter — before DB loading (Parquet keeps all records)
  # All use same syntax: .field op value, booleans (true/false), and/or
  - id: profiles
    endpoint: /api/linkedin/user
    dependency: { from_source: employees, field: internal_id.value }
    input_key: user
    filter: '.follower_count > 100'  # Level 1: early filter (before LLM + Parquet)
    transform:                      # Level 2: export filter (Parquet keeps all)
      filter: '.location != ""'     # Safe expression
      fields:                       # Field selection with dot-notation aliases
        - name
        - urn.value AS urn_id
        - headline
      add_columns:                  # Static columns to inject
        batch: "q1-2026"
    export:
      - type: file
        path: ./output/profiles.csv
        format: csv
      - type: webhook
        url: https://example.com/hook
        headers: { X-Token: abc }
    db_load:                        # Database loading config
      filter: '.active == true'     # Level 3: DB filter (only active to DB)
      table: people                 # Custom table name (default: source ID)
      key: urn.value                # Unique key for diff-based incremental sync
      sync: full                    # Sync mode: full (default) | append (no DELETE)
      fields:                       # Fields to load (default: all except _input_value)
        - name
        - urn.value AS urn_id
        - headline
      exclude: [_input_value, raw_html]  # Fields to skip

storage:
  format: parquet                   # Storage format (only parquet supported)
  path: ./data/                     # Storage directory (relative to dataset.yaml)

schedule:
  cron: "0 9 * * *"                 # Cron expression for scheduling

notifications:
  on_complete:
    - url: "https://hooks.slack.com/xxx"
      headers: { Authorization: "Bearer token" }
  on_failure:
    - url: "https://alerts.example.com/fail"
```

### type: union Source Details

The `type: union` source combines records from multiple parent sources:

- **Requires**: non-empty `sources` list (parent source IDs)
- **Optional**: `dedupe_by` field path for removing duplicates (supports dot-notation)
- **Cannot have**: `endpoint`, `dependency`, `from_file`, `input_key`, `input`
- **Validation**: all parent sources must have the same endpoint (same data structure)
- **Use case**: merge multiple search results before a single dependent source processes them

```yaml
sources:
  - id: search_cto
    endpoint: /api/linkedin/search/users
    input: { keywords: "CTO fintech", count: 50 }

  - id: search_vp
    endpoint: /api/linkedin/search/users
    input: { keywords: "VP Engineering", count: 50 }

  # Union combines all search results
  - id: all_candidates
    type: union
    sources: [search_cto, search_vp]
    dedupe_by: urn.value              # Remove duplicates by URN

  # Single dependent source processes all candidates
  - id: profiles
    endpoint: /api/linkedin/user
    dependency:
      from_source: all_candidates
      field: urn.value
    input_key: user
```

Records from union sources are annotated with `_union_source` (the parent source ID they came from).

### type: llm Source Details

The `type: llm` source processes existing parent data without making API calls:

- **Requires**: `dependency` (parent source) and non-empty `llm` list
- **Cannot have**: `endpoint`, `from_file`, `input_key`, `input`
- **Use case**: Run LLM enrichment on already-collected data, re-analyze with different prompts

```bash
# Collect only the LLM source (reads parent Parquet, applies LLM steps)
anysite dataset collect dataset.yaml --source employees_analyzed
```

### Validate & Collect Commands

```bash
anysite dataset validate dataset.yaml                    # Validate config (catches errors early)
anysite dataset collect dataset.yaml --dry-run          # Preview plan
anysite dataset collect dataset.yaml                     # Run collection
anysite dataset collect dataset.yaml --load-db pg       # Collect + auto-load to DB
anysite dataset collect dataset.yaml --incremental      # Skip already-collected inputs
anysite dataset collect dataset.yaml --source employees # Single source + dependencies
anysite dataset collect dataset.yaml --no-llm           # Skip LLM enrichment steps
anysite dataset collect dataset.yaml --limit 100        # Pilot run: max 100 inputs per source
```

### Incremental Collection (Resume Where You Left Off)

For `from_file` and `dependency` sources, anysite tracks collected input values in `metadata.json`. This enables resuming collection without re-fetching existing data.

**Workflow:**
```bash
# First run — collects all inputs
anysite dataset collect dataset.yaml

# Later: add new items to input file, run with --incremental
anysite dataset collect dataset.yaml --incremental
# → Only new items are collected, existing ones skipped

# Force full re-collection
anysite dataset reset-cursor dataset.yaml
anysite dataset collect dataset.yaml
```

**Per-source `refresh` option:**
```yaml
sources:
  - id: profiles
    refresh: auto      # (default) respects --incremental

  - id: posts
    refresh: always    # always re-collects, ignores --incremental
                       # use for time-sensitive data (feeds, activity)

  - id: companies
    refresh: never     # skip if data exists (any snapshot)
```

**Reset cursor:**
```bash
anysite dataset reset-cursor dataset.yaml                  # all sources
anysite dataset reset-cursor dataset.yaml --source posts   # specific source
```

### Query with DuckDB

```bash
anysite dataset query dataset.yaml --sql "SELECT * FROM companies LIMIT 10"
anysite dataset query dataset.yaml --source profiles --fields "name, urn.value AS id"
anysite dataset query dataset.yaml --source profiles --exclude "_input_value,_parent_source"
anysite dataset query dataset.yaml --interactive        # SQL shell
anysite dataset stats dataset.yaml --source companies
anysite dataset profile dataset.yaml
```

### Load into Database

```bash
anysite dataset load-db dataset.yaml -c pg --drop-existing  # Full load
anysite dataset load-db dataset.yaml -c pg                   # Incremental sync (when db_load.key set)
anysite dataset load-db dataset.yaml -c pg --snapshot 2026-01-15
```

### Compare Snapshots

```bash
anysite dataset diff dataset.yaml --source profiles --key urn.value
anysite dataset diff dataset.yaml --source profiles --key urn.value --from 2026-01-30 --to 2026-02-01
```

### Scheduling & History

```bash
anysite dataset schedule dataset.yaml --incremental --load-db pg  # Generate cron entry
anysite dataset history my-dataset
anysite dataset logs my-dataset --run 42
anysite dataset reset-cursor dataset.yaml                         # Clear incremental state
```

## Database Operations

```bash
# Connection management
anysite db add pg --type postgres --host localhost --database mydb --user app --password secret
anysite db add pg --type postgres --host localhost --database mydb --user app --password-env PGPASS
anysite db add ch --type clickhouse --host ch.example.com --port 8443 --database analytics --user app --password secret --ssl
anysite db add local --type sqlite --path ./data.db
anysite db add replica --type postgres --host replica.example.com --read-only
anysite db list
anysite db test pg

# Data operations
cat data.jsonl | anysite db insert pg --table users --stdin --auto-create
anysite db query pg --sql "SELECT * FROM users" --format table
anysite db query pg --sql "SELECT * FROM users" --format parquet --output users.parquet
anysite db query pg --sql "SELECT * FROM users" --format csv --output "reports/{{date}}/users.csv"

# Pipe API output to database
anysite api /api/linkedin/user user=satyanadella -q --format jsonl \
  | anysite db insert pg --table profiles --stdin --auto-create

# Database discovery (schema introspection, sample data, LLM descriptions)
anysite db discover mydb                              # Discover schema
anysite db discover mydb --with-llm                   # Add LLM table/column descriptions
anysite db discover mydb --tables users,posts         # Filter tables
anysite db discover mydb --exclude-tables _migrations

# View saved catalogs
anysite db catalog                       # List all catalogs
anysite db catalog mydb                  # Show full catalog
anysite db catalog mydb --table users    # Show specific table
anysite db catalog mydb --json           # JSON output for agents
```

Credentials: `--password` saves directly in `~/.anysite/connections.yaml`, `--password-env` references an env var. Direct value takes priority. LLM API keys follow the same pattern via `anysite llm setup`.

Supported databases: SQLite, PostgreSQL, ClickHouse. ClickHouse uses `clickhouse-connect` driver (HTTP protocol, port 8123 default, 8443 for HTTPS/SSL).

## LLM Analysis Commands

Analyze collected dataset records using LLM. Requires `anysite llm setup` first.

Non-interactive setup (for agents):
```bash
anysite llm setup --provider openai --api-key <key> --no-test
anysite llm setup --provider anthropic --api-key-env ANTHROPIC_API_KEY --model claude-sonnet-4-5-20250514 --no-test
```

```bash
anysite llm summarize dataset.yaml --source profiles --fields "name,headline" --format table
anysite llm classify dataset.yaml --source posts --categories "positive,negative,neutral"
anysite llm enrich dataset.yaml --source profiles \
  --add "seniority:junior/mid/senior" --add "is_technical:boolean"
anysite llm generate dataset.yaml --source profiles \
  --prompt "Write intro for {name} who works as {headline}" --temperature 0.7
anysite llm match dataset.yaml --source-a profiles --source-b companies --top-k 3
anysite llm deduplicate dataset.yaml --source profiles --key name --threshold 0.8
anysite llm cache-stats
anysite llm cache-clear
```


Use `anysite describe --search <keyword>` for more endpoints.

## Key Patterns

- **Output formats**: `--format json|jsonl|csv|table|parquet` (parquet requires `--output`)
- **Array params**: `keywords=a,b,c` auto-wraps as `["a","b","c"]`
- **Field selection**: `--fields "name,headline,urn.value"` (dot-notation for nested)
- **Error handling**: `--on-error stop|skip|retry`
- **Config priority**: CLI args > ENV vars > `~/.anysite/config.yaml` > defaults

## References

For detailed option tables and advanced configuration:
- [api-reference.md](references/api-reference.md) — all CLI options
- [llm-reference.md](references/llm-reference.md) — LLM provider config, cache, prompts
- [dataset-guide.md](references/dataset-guide.md) — full YAML schema, advanced features
登录后下载/点赞/收藏 ❤ 5 | ★ 0
anysite-cli

目录结构

SKILL.md

举报内容

提示
