download-script-dev

分类: 数据与AI | 上传者: uw-ssec | 下载: 0 | 版本: v1.0（最新）

当用户要求“开发下载脚本”、“调试数据下载”、“修复下载错误”、“创建数据管道模板”、“下载模板”、“GAIA 数据管道”、“从 S3 下载”、“访问 Zarr 存储”、“云数据访问”或提及特定数据源名称（例如“CONUS404”、“HRRR”、“WRF”、“PRISM”、“Stage IV”、“USGS”、“ORNL”、下载或处理数据时的“DEM”、“Synoptic”或“IRIS”。为 GAIA 项目中使用的水文气候数据下载脚本提供模板、配置验证和调试指南。

更新日志: Source: GitHub https://github.com/uw-ssec/rse-plugins

目录结构

当前层级: tree/main/community-plugins/gaia-data-downloader/skills/download-script-dev/

📁 references/
- 📁 sources/
  - 📄 conus404.md 1.1 KB
  - 📄 dem-srtm.md 653 B
  - 📄 hrrr.md 1.5 KB
  - 📄 iris.md 739 B
  - 📄 ornl-daymet.md 616 B
  - 📄 prism.md 859 B
  - 📄 stage-iv.md 943 B
  - 📄 synoptic.md 881 B
  - 📄 usgs.md 1.5 KB
  - 📄 wrf-cmip6.md 1.1 KB
- 📄 CONFIGURATION.md 7.8 KB
- 📄 DATA_SOURCES.md 990 B
- 📄 DOWNLOAD_PATTERNS.md 15.1 KB
📄 SKILL.md 8.1 KB

SKILL.md

---
name: download-script-dev
description: >
  This skill should be used when the user asks to "develop a download script",
  "debug data download", "fix download error", "create data pipeline template",
  "download template", "GAIA data pipeline", "download from S3", "access Zarr store",
  "cloud data access", or mentions specific data source names like "CONUS404", "HRRR",
  "WRF", "PRISM", "Stage IV", "USGS", "ORNL", "DEM", "Synoptic", or "IRIS" in the
  context of downloading or processing data. Provides templates, configuration
  validation, and debugging guidance for hydroclimatological data download scripts
  used in the GAIA project.
version: 2026-03-20
---

# Download Script Development Skill

## Overview

Assist in developing, refining, and debugging data download scripts for GAIA hydroclimatological data sources. This skill provides templates, configuration schemas, and troubleshooting guidance for building reproducible data pipelines across 10+ environmental data sources.

## Requirements

- **Python 3.9+** with `xarray`, `geopandas`, `rioxarray`
- **Source-specific libraries:** `herbie-data` (HRRR), `pyPRISMClimate` (PRISM), `obspy` (IRIS), `boto3` (WRF/S3), `elevation` (DEM), `s3fs` (CONUS404)
- **System dependencies:** `wgrib2` for HRRR (install via conda-forge, not pip)

## When to Use

- Developing a new download script for a GAIA data source
- Debugging an existing download script (timeouts, auth errors, CRS mismatches)
- Adapting a notebook pattern to a new use case or study area
- Validating download configuration parameters
- Understanding which access pattern or library to use for a data source

## Script Structure Pattern

All download scripts follow a CONFIG-at-top pattern separating parameters from logic:

```python
import xarray as xr
import geopandas as gpd
from concurrent.futures import ThreadPoolExecutor

# ============================================================
# Configuration — modify these parameters before running
# ============================================================
CONFIG = {
    "source": "SOURCE_NAME",
    "date_range": ("2024-01-01", "2024-01-31"),
    "variables": ["var1", "var2"],
    "aoi_path": "../data/GIS/boundary.json",
    "output_path": "../data/output.zarr",
    "output_format": "zarr",
    "max_workers": 8,
}
# ============================================================
# Download logic — generally no need to modify below this line
# ============================================================

def main():
    # 1. Load AOI
    aoi = gpd.read_file(CONFIG["aoi_path"])

    # 2. Download data (parallel)
    # 3. Combine datasets
    # 4. Spatial subset
    # 5. Derive variables (if needed)
    # 6. Save to output format
    # 7. Print QC summary
    pass

if __name__ == "__main__":
    main()
```

## Four Data Access Patterns

### 1. Direct HTTP Download (PRISM, Stage IV, DEM)

Simple URL-based fetching with `requests`. Use `ThreadPoolExecutor` for parallel downloads. Handle retries for network failures.

### 2. REST API Query (USGS, Synoptic)

Parameterized endpoints returning JSON or RDB format. Build URL query strings from CONFIG parameters. Parse response formats appropriately (RDB requires custom parsing).

### 3. Cloud Object Storage / S3 (CONUS404, HRRR, WRF-CMIP6)

Access via `s3fs`, `boto3`, or library wrappers. Supports partial reads and lazy loading with `xarray.open_zarr()`. Use anonymous/unsigned credentials for public buckets.

### 4. Specialized Libraries (Herbie for HRRR, pyPRISMClimate, obspy for IRIS)

Domain-specific wrappers that handle authentication, URL construction, and data parsing internally. Consult library documentation for parameter conventions.

## Spatial Subsetting Methods

Choose based on grid type:

| Grid Type | Method | When to Use |
|-----------|--------|-------------|
| Regular (lat/lon) | `ds.rio.clip(aoi.geometry)` | PRISM, Stage IV, DEM |
| Curvilinear (model) | `regionmask` | CONUS404, WRF-CMIP6 |
| Irregular (points) | `shapely.contains()` | USGS station data |

Ensure the AOI CRS matches the data CRS before subsetting. Model grids often use Lambert Conformal Conic — reproject the AOI with `aoi.to_crs(ds.rio.crs)`.

## Parallel Download Pattern

```python
from concurrent.futures import ThreadPoolExecutor, as_completed

def download_item(item):
    """Download a single item. Return path or dataset."""
    # ... download logic
    return result

with ThreadPoolExecutor(max_workers=CONFIG["max_workers"]) as executor:
    futures = {executor.submit(download_item, item): item for item in items}
    for i, future in enumerate(as_completed(futures), 1):
        result = future.result()
        print(f"Downloaded {i}/{len(items)}")
```

Worker count guidance: 4-8 for HTTP downloads, 8-16 for S3 reads, 2-4 for API endpoints with rate limits.

## Output Formats

| Format | When to Use | Trade-offs |
|--------|-------------|------------|
| **Zarr** (preferred) | Large gridded datasets, cloud workflows | Fast parallel I/O, chunked, no single-file limit |
| **NetCDF** | Sharing with traditional tools, small datasets | Widely supported, single-file, 2 GB limit (classic) |
| **CSV** | Tabular station data (USGS) | Human-readable, no spatial metadata |

## Common Issues and Debugging

### wgrib2 Not Found (HRRR)

`wgrib2` is a C binary, not pip-installable. Install via conda-forge: `conda install -c conda-forge wgrib2` or `pixi add wgrib2`. Verify with `shutil.which("wgrib2")`. Add a runtime guard in scripts:

```python
import shutil
assert shutil.which("wgrib2"), "wgrib2 not found. Install: conda install -c conda-forge wgrib2"
```

### S3 Authentication Errors (CONUS404, WRF)

Public buckets require anonymous access. For CONUS404: set `anon=True` in `s3fs.S3FileSystem()`. For WRF: use `botocore.UNSIGNED` config in boto3.

```python
# CONUS404
fs = s3fs.S3FileSystem(anon=True, client_kwargs={"endpoint_url": "https://usgs.osn.mghpcc.org"})

# WRF-CMIP6
from botocore import UNSIGNED
from botocore.config import Config
s3 = boto3.client("s3", config=Config(signature_version=UNSIGNED))
```

### CRS Mismatch During Spatial Subsetting

If `rio.clip()` raises a CRS error, reproject the AOI to match the dataset:

```python
aoi = aoi.to_crs(ds.rio.crs)  # reproject AOI to dataset CRS
ds_clipped = ds.rio.clip(aoi.geometry)
```

For datasets without CRS metadata, set it explicitly: `ds.rio.write_crs("EPSG:4326", inplace=True)`.

### Memory Issues with Large Datasets

Use chunked loading to avoid loading entire datasets into RAM:

```python
ds = xr.open_dataset(path, chunks={"time": 100})  # lazy loading
print(f"Dataset size: {ds.nbytes / 1e9:.1f} GB")   # check before computing
```

Process in temporal batches rather than loading the full dataset. Use `.compute()` only on subsets.

### Network Timeouts and Retries

Wrap downloads in retry logic with exponential backoff:

```python
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
session.mount("https://", HTTPAdapter(max_retries=Retry(total=3, backoff_factor=2)))
response = session.get(url, timeout=60)
```

For S3, boto3 has built-in retry configuration via `botocore.config.Config(retries={"max_attempts": 3})`.

### USGS RDB Format Parsing

USGS returns tab-separated RDB format with comment headers (`#`) and a data-type row below the column headers. Parse by skipping both:

```python
lines = [l for l in response.text.splitlines() if not l.startswith("#")]
df = pd.read_csv(io.StringIO("\n".join([lines[0]] + lines[2:])), sep="\t")
```

Note: USGS returns data in **local time zones**. Convert to UTC using station timezone metadata.

## Additional Resources

### Reference Files

For detailed data source documentation, code templates, and configuration schemas, consult:

- **`references/sources/`** — Per-source documentation files (e.g., `sources/hrrr.md`, `sources/conus404.md`): endpoints, response formats, authentication setup, and example API calls. Load only the source relevant to the current task.
- **`references/DOWNLOAD_PATTERNS.md`** — Complete code templates for each access pattern with full pipeline examples for HRRR, CONUS404, and USGS
- **`references/CONFIGURATION.md`** — Per-source parameter tables with types, defaults, and validation rules; size estimation formulas

登录后下载/点赞/收藏 ❤ 20 | ★ 0

download-script-dev

目录结构

SKILL.md

举报内容

提示