tao-generate-referring-expressions

Image Referring Expression Pipeline Generate referring-expression and grounding annotations from images with KITTI-format bounding box labels. A single VLM (Gemini or any OpenAI-compatible endpoint) runs four steps: per-object region descriptions, holistic image captions, grouped grounding expressions tied to bboxes, and an optional double-check verification pass. Purpose Transform (image, KITTI labels) pairs into a unified annotations.jsonl containing rich, grounded referring expressions. The VLM acts as a "teacher" annotator: Steps 0-1 see the image; Step 2 groups Step 0 outputs into grouping phrases with bbox lists; Step 3 (optional) re-examines those bboxes against the image and corrects mismatches. Pipeline Architecture Step 0: Region expression ──┐ ├──▶ Step 2: Grounding expression ──▶ [Step 3: Double check] Step 1: Image caption ──────┘ (optional) Show more Installs 584 Repository nvidia/skills GitHub Stars 1.9K First Seen Jun 8, 2026 Security Audits Gen Agent Trust Hub Pass Socket Pass Snyk Fail