tao-generate-video-reasoning-annotations

Video Reasoning Annotation Pipeline Generate Chain-of-Thought training datasets from videos by producing multi-level captions, structured descriptions, and QA pairs (MCQ, binary, open-ended) with step-by-step reasoning traces. Domain-agnostic by default — customize prompts for any video domain. Purpose Transform raw videos into CoT Q&A training data for video understanding models. VLMs (e.g., Gemini, Qwen) act as "teacher" annotators: Steps 0–1 require the model to see the video (VLM calls); Steps 2–3 are text-to-text (cheaper LLM calls). Pipeline architecture Step 0: [Optional] Filter & classify videos → Keep domain-relevant, classify anomaly vs normal Step 1a: Global + dense captions → VLM: narrative summary + timestamped events Step 1b: Chunk captions → VLM: fixed-duration segment micro-captions Step 1c: [Optional, anomaly only] Highlight → LLM extracts anomaly timestamp, VLM captions clip Step 2: Description synthesis → LLM: synthesize captions into structured narrative Step 3: QA generation → LLM: MCQ, binary, open-ended with reasoning Step 4: Parse outputs → Per-task tao-vl-reason-v1.0 JSON files Show more Installs 586 Repository nvidia/skills GitHub Stars 1.9K First Seen Jun 8, 2026 Security Audits Gen Agent Trust Hub Pass Socket Pass Snyk Pass