jimeng-browser-automation

Category: Content & Multimedia | Uploader: gxagxagx | Downloads: 0 | Version: v1.0（Latest）
Automate JiMeng (即梦AI at jimeng.jianying.com), Seedance video generation, and canvas projects in a real browser with Playwright. Use for Douyin QR login, JiMeng image or video generation, canvas project creation/opening/prompting, record-id based status checks, canceling queued video tasks, and downloading completed results.
Changelog: Source: GitHub https://github.com/gxagxagx/jimeng-browser-automation
Directory Structure

Current level: tree/main/
📁 agents/
- 📄 openai.yaml 791 B
📁 references/
- 📄 jimeng-flow.md 11.2 KB
📁 scripts/
- 📄 jimeng-browser.js 129.6 KB
📄 .gitignore 62 B
📄 LICENSE 1.0 KB
📄 package-lock.json 1.7 KB
📄 package.json 551 B
📄 README.md 7.3 KB
📄 SKILL.md 12.6 KB
SKILL.md

---
name: jimeng-browser-automation
description: Automate JiMeng (即梦AI at jimeng.jianying.com), Seedance video generation, and canvas projects in a real browser with Playwright. Use for Douyin QR login, JiMeng image or video generation, canvas project creation/opening/prompting, record-id based status checks, canceling queued video tasks, and downloading completed results.
---

# JiMeng Browser Automation

Use the bundled Playwright driver for JiMeng.

Default workflow:

1. Ensure login
2. Submit an image or video task
3. Track it by `record-id`
4. Cancel a queued video task if needed
5. Download the finished result

Always identify tasks by `record-id`, not by page order.

## Setup

Run from this skill directory:

```bash
npm install
npx playwright install chromium
```

## Hard rules

- Ensure login before any generate action.
- Use structured flags for structured choices: `--model`, `--reference-mode`, `--duration`, `--aspect`, `--resolution`, `--reference-file`, `--first-frame-file`, `--last-frame-file`.
- Keep the prompt focused on content: subject, scene, action, emotion, camera language, style.
- Do not hide option labels such as `9:16`, `1080P`, `高清 2K`, `Seedance 2.0 Fast`, `主体参考`, or `首尾帧` inside the prompt unless the user literally wants those words in the output.
- Exception: JiMeng mention tokens are part of the content binding, not normal option flags.
- Do not guess unavailable options. Use exact visible Chinese labels or omit the flag.
- Treat the entire `runtime/` directory as private local data. Do not commit it.

## Natural language to flags

- `竖屏` -> `--aspect 9:16`
- `横屏` -> usually `--aspect 16:9`
- `5 秒视频` -> `--duration 5s`
- `12 秒视频` -> `--duration 12s`
- `高清 2K` / `1080P` -> `--resolution ...`
- `用这张图做主体参考` -> `--reference-mode 主体参考 --reference-file ...`
- `首尾帧过渡` -> `--reference-mode 首尾帧 --first-frame-file ... --last-frame-file ...`

## Minimal interface

Use these commands for normal agent work.

### 1. Ensure login

```bash
node scripts/jimeng-browser.js login
```

- Run this first when the session may be stale.
- If login is required, the script saves a popup screenshot and a QR screenshot under `runtime/artifacts/`.
- Show the QR screenshot to the user and wait for a Douyin scan.
- The persistent browser profile lives under `runtime/jimeng-profile/`.

### 2. Open canvas home

```bash
node scripts/jimeng-browser.js open-tool --tool canvas
```

- The real canvas home route is `https://jimeng.jianying.com/ai-tool/assets-canvas`.
- This page lists recent projects and a `新建项目` entry point.

### 3. Create a canvas project

```bash
node scripts/jimeng-browser.js canvas-create-project --json true
```

- JiMeng opens the new project in a separate window.
- The command captures that new window and returns `projectId`, `projectUrl`, and `projectTitle`.

### 4. Open an existing canvas project

```bash
node scripts/jimeng-browser.js canvas-open-project --project-name "victor_测试" --json true
```

Optional:

- `--project-index` when multiple projects share the same visible name
- `--project-id`
- `--project-url`

Behavior:

- Clicking a recent project also opens a separate window.
- The command captures that new window and returns the resolved project URL.

### 5. Rename a canvas project

```bash
node scripts/jimeng-browser.js canvas-rename-project --project-id 123456 --name "my_project" --json true
```

Behavior:

- The current canvas page allows renaming by clicking the project name in the top-left header.
- The automation enters the inline title input, replaces the name, and confirms it.

### 6. Prompt inside a canvas project

```bash
node scripts/jimeng-browser.js canvas-prompt --project-id 123456 --kind image --prompt "一只白色柴犬坐在木桌上，白底，简洁插画风。" --json true
```

Optional:

- `--kind image|video|auto`
- `--reference-file /absolute/path[,/absolute/path2]`
- `--project-name`
- `--project-id`
- `--project-url`

Behavior:

- The automation first ensures the top-right `对话` drawer is open.
- It then uses the right-side conversation drawer input instead of the canvas bottom editor.
- The drawer exposes the same `图片生成 / 视频生成` tool choices as the normal generate page.
- The automation switches the drawer tool first, then reuses the normal image/video option logic.
- The prompt is sent as normal generate content. It is not wrapped with extra `生成一张图片：...` or `生成一个视频：...` prefixes.
- The command submits the prompt inside the project and now returns the underlying generation `recordId` when JiMeng surfaces it.
- Once `recordId` is available, the normal `record-status`, `download-record`, and `cancel-record` commands can be reused.
- The current observed project flow first enters an Agent planning phase before it calls JiMeng generation tools.
- If the project prompt includes JiMeng reference mentions such as `@图片1` or `@主体1`, the automation inserts real mention tags instead of plain text.

### 7. Generate image

```bash
node scripts/jimeng-browser.js generate --tool image --prompt "一只白色柴犬站在雨夜霓虹街头" --json true
```

Required:

- `--prompt`

Optional:

- `--model`
- `--aspect`
- `--resolution`
- `--reference-file /absolute/path.png`

Observed image options:

- Models: `图片5.0 Lite`, `图片4.6`, `图片 4.5`, `图片 4.1`, `图片 4.0`, `图片 3.1`, `图片 3.0`
- Aspect ratios: `智能`, `21:9`, `16:9`, `3:2`, `4:3`, `1:1`, `3:4`, `2:3`, `9:16`
- Resolutions: `高清 2K`, `超清 4K`

Behavior:

- Returns `recordId` on success.
- Supports text-only and text-plus-reference generation.
- `--reference-file` accepts one or more absolute image paths, comma-separated.
- One image task can produce up to `4` result images.
- Image tasks can partially fail. Treat them as finished with partial failure, not as a pure hard failure.

### 8. Generate video

```bash
node scripts/jimeng-browser.js generate --tool video --prompt "让一只小猫坐在窗边，看向窗外的雨夜" --json true
```

Required:

- `--prompt`

Optional:

- `--model`
- `--reference-mode`
- `--duration`
- `--aspect`
- `--resolution`
- `--reference-file /absolute/path`
- `--first-frame-file /absolute/path`
- `--last-frame-file /absolute/path`
- `--submit-retries`
- `--submit-retry-delay-ms`
- `--record-id-wait-ms`

Observed video options:

- Models: `Seedance 2.0 Fast`, `Seedance 2.0`, `视频 3.5 Pro`, `视频 3.0 Pro`, `视频 3.0 Fast`, `视频 3.0`
- Reference modes: `全能参考`, `首尾帧`, `智能多帧`, `主体参考`
- Durations currently seen: `4s` to `15s`
- Aspect ratios: `21:9`, `16:9`, `4:3`, `1:1`, `3:4`, `9:16`
- Resolutions seen: `720P`, `1080P`

Behavior:

- Usually returns `recordId` on success.
- Supports pure text, `全能参考`, `主体参考`, `首尾帧`, and `智能多帧`.
- One video task produces `1` MP4.
- For hot models, inspect `serverAccepted`, `historyRecordId`, `serverRet`, and `serverErrmsg`.
- JiMeng can accept a task before the history card appears. `serverAccepted=true` with no `recordId` is accepted, not a hard failure.
- Resolution availability is dynamic. Not every model, aspect ratio, and reference-mode combination exposes the same resolution.
- For `Seedance 2.0 Fast` and `Seedance 2.0`, prefer:

```bash
--submit-retries 2 --submit-retry-delay-ms 60000 --record-id-wait-ms 180000
```

### 9. Check task status

```bash
node scripts/jimeng-browser.js record-status --record-id <record-id> --json true
```

Useful fields:

- `status`
- `isComplete`
- `isFailed`
- `isCanceled`
- `canCancel`
- `progressPercent`
- `queuePosition`
- `queueTotal`
- `etaText`
- `historyRecordId`
- `failedCount`
- `partialFailure`
- `auditFailureType`
- `auditFailurePhase`
- `failureReason`

### 10. Cancel queued video task

```bash
node scripts/jimeng-browser.js cancel-record --record-id <record-id> --json true
```

- Supported for video tasks.
- Uses JiMeng's direct cancel API.
- Works even if the card is no longer visible in the current list.

### 11. Download result

```bash
node scripts/jimeng-browser.js download-record --record-id <record-id> --wait-complete true --json true
```

- Image records download up to `4` PNG files.
- Video records download `1` MP4.
- Older records can still be recovered through the history API fallback.
- For partial image records, download every successful image that exists and record the counts in `record-download.json`.
- Default wait timeout is `900000` ms. For hot queues, poll with `record-status` first or increase the timeout.

## Video reference-mode rules

Choose models from the reference mode first, not the other way around.

Current observed compatibility:

- `全能参考` -> `Seedance 2.0 Fast`, `Seedance 2.0`
- `首尾帧` -> `Seedance 2.0 Fast`, `Seedance 2.0`, `视频 3.5 Pro`, `视频 3.0 Pro`, `视频 3.0 Fast`, `视频 3.0`
- `智能多帧` -> `视频 3.0 Fast`, `视频 3.0`
- `主体参考` -> `视频 3.0`

Important:

- JiMeng can silently rewrite incompatible model/mode combinations.
- The automation re-reads the final visible model and mode after selection and stops on mismatch.

## Mention rules

Mention tokens are only for modes that bind uploaded materials through the editor.

- `全能参考`: use `@图片1`, `@图片2`, `@图片3` in upload order.
- `主体参考`: use `@主体`. If multiple subject images are uploaded, use local syntax `@主体1`, `@主体2` to choose the first and second visible `主体` options deterministically.
- `智能多帧`: do not rely on prompt mention tokens.
- `首尾帧`: do not rely on prompt mention tokens. Use `--first-frame-file` and `--last-frame-file`.

Important:

- Plain text `@图片1` is not enough. The automation must select the popup option so JiMeng inserts a real mention tag node.
- Verified live: `主体参考` submits reach JiMeng as structured reference nodes, not degraded plain text like `让@@`.
- The same mention insertion logic is also reused inside canvas project prompts when the active project editor supports uploaded-material mentions.

## Canvas project rules

- Canvas home is `assets-canvas`, not the normal generate route.
- `新建项目` opens a new canvas window at `/ai-tool/canvas/<projectId>?enter_from=create_new...`.
- Clicking a recent project opens a new canvas window at `/ai-tool/canvas/<projectId>?enter_from=assets...`.
- Clicking the project name in the top-left header enters inline rename mode.
- The preferred project generation entry is the top-right `对话` drawer.
- The drawer exposes `图片生成` and `视频生成`, and the resulting controls match the normal generate page closely enough to reuse the same option-selection logic.
- For now, treat project creation requests as prompt-driven:
  - image request -> `--kind image`
  - video request -> `--kind video`
- After submit, the project commonly enters `意图分析` / `任务规划` before the actual image/video generation step.

## Status model

Map JiMeng results into these coarse states:

- `queued`: `排队加速中`, `生成中`, `造梦中`, `智能创意中`
- `completed`: `已完成`
- `canceled`: `已取消`
- `failed`: any task with `isFailed=true`

Special case:

- Partial image tasks can be both complete and failed: `isComplete=true`, `isFailed=true`, `partialFailure=true`

Verified failure cases:

- Submit-time reject: `你输入的文字不符合平台规则，请修改后重试`
- Post-generation reject: `视频未通过审核，本次不消耗积分`
- Peak-hour reject: `因目前处于使用高峰期，暂时无法提交更多任务，请等待其他任务完成后再尝试提交～`
- User cancel: `你已取消生成，积分已返还`

## When to read references

Read [references/jimeng-flow.md](references/jimeng-flow.md) when you need:

- exact current option labels
- selector/debugging detail
- route and login observations
- canvas project window behavior
- failure and queue API shape
- mention behavior verification

## Maintenance only

These commands are useful for debugging site changes, but they are not part of the normal workflow:

- `node scripts/jimeng-browser.js snapshot --tool image|video`
- `node scripts/jimeng-browser.js open-tool --tool home|image|video|canvas`
- `node scripts/jimeng-browser.js canvas-create-project`
- `node scripts/jimeng-browser.js canvas-open-project --project-name "未命名项目"`
- `node scripts/jimeng-browser.js canvas-prompt --project-id 123456 --prompt "..."`
- `node scripts/jimeng-browser.js find-record --record-id <record-id>`
- `node scripts/jimeng-browser.js list-records`

## Bundled files

- `scripts/jimeng-browser.js`: browser driver and command entrypoint
- `references/jimeng-flow.md`: site observations and verified edge cases
- `runtime/tracked-records.jsonl`: local append-only task registry
Comments 0
Latest | Hot
Please login before commenting.
No comments yet. Be the first one!
jimeng-browser-automation

Directory Structure

SKILL.md

Report

Notice
