tao-generate-image-grounding

Image Grounding Pipeline Turn (image, caption) pairs into per-image grounded annotations: cleaned captions, referring expressions with character spans, and pixel-space bounding boxes for each expression. A single VLM (Gemini or any OpenAI-compatible endpoint) handles both steps. Purpose Generate phrase-grounded training data for referring-expression and grounding models. The VLM acts as a "teacher" annotator: Step 0 extracts referring expressions from the caption while looking at the image; Step 1 returns one bbox set per expression for each image. Pipeline Architecture Step 0: Expression extraction → VLM cleans caption, extracts referring expressions + char spans Step 1: Phrase grounding → VLM returns pixel bboxes + scores per expression Steps are individually selectable via workflow.steps . Each step writes a per-sample checkpoint to step__*/.ckpt/.json and skips already-processed records on re-run. Set workflow.force_reprocess: true to ignore checkpoints and reprocess from scratch. Instructions Show more Installs 585 Repository nvidia/skills GitHub Stars 1.9K First Seen Jun 8, 2026 Security Audits Gen Agent Trust Hub Pass Socket Pass Snyk Fail