video-understanding

What this does

Turns a source video into an understanding index an agent (or a downstream stage) can read:

1. Scene detection — scenes.json (cut points, durations) + junk-scene filtering.
2. Frame extraction — sampled frames for the visual analysis.
3. ASR — asr_result.json (timestamped dialogue) via MiMo mimo-v2.5-asr.
4. Silence detection — silence_periods.json (quiet windows, has_speech flag).
5. VLM analysis — vlm_analysis.json (per-scene description, depth analysis, frame_facts).
6. Timeline fusion + brief — timeline_fusion.json, asr_writing_chunks.json, agent_narration_brief.md.

Stateless: reusable stages are skipped only when their output and provenance sidecar match the current source video plus output-affecting settings. --force recomputes.

Requirements

# ffmpeg: brew install ffmpeg | apt install ffmpeg | choco install ffmpeg
export MIMO_API_KEY=***          # one key drives ASR (mimo-v2.5-asr) + VLM (mimo-v2.5)

ASR uses MiMo mimo-v2.5-asr; pass --skip-asr to skip dialogue transcription. The full understanding run still requires MIMO_API_KEY for VLM scene analysis. Optional MiMo scene-chunk video understanding: --mimo-video-overview.

If work_dir/background_research.json exists (story research the agent did first, see references/research-guide.md), its synopsis and named characters are folded into the VLM context, so scene descriptions can name people and read scenes with plot knowledge. Combine with --context for a quick inline hint.

Running the scripts below — the scripts/… paths are relative to this skill's own directory (the folder containing this SKILL.md). Claude Code runs commands from there, so they work as written. If your harness runs commands from the project root instead (opencode / Codex / OpenClaw commonly do), prefix this skill's absolute directory — e.g. <skill-dir>/scripts/…, using the directory your harness reports when it loads the skill. The scripts self-locate from their own path, so once started by the correct path they resolve their sibling skills and assets regardless of the working directory.

Run

python3 scripts/understand.py <video> --work-dir <work_dir> \
  [--context "节目名/角色名"] [--scene-threshold 0.1] [--skip-asr] [--mimo-video-overview] [--force]

Output contract

| File | Content | |------|---------| | scenes.json | scene cut list (start/end/duration) | | asr_result.json | [{start, end, text}] timestamped transcript | | vlm_analysis.json | per-scene description / depth / frame_facts | | silence_periods.json | [{start, end, duration, has_speech}] quiet windows | | timeline_fusion.json | VLM + ASR + silence overlap, unified timeline | | asr_writing_chunks.json | ASR split at sentence boundaries, scene-aligned | | agent_narration_brief.md | the human/agent-facing writing brief (read this first) |

Downstream, video-script reads the brief + index to write narration.json.

References

• Background research before writing: references/research-guide.md (writes background_research.json).
• Output JSON shapes: references/data-schema.md.

What this skill does NOT do

• Does NOT write narration / 解说词 or score it — that is video-script.
• Does NOT cut, edit, voice, or render video.
• Does NOT invent plot the signal doesn't support — it emits a substrate warning when ASR/VLM are thin, rather than fabricating.
• Does NOT publish or schedule anything; it writes artifacts to work_dir and stops.