CLIP-Aware Image Embeddings

Smart image-text matching that knows when CLIP works and when to use alternatives.

MCP Integrations MCP Purpose Firecrawl Research latest CLIP alternatives and benchmarks Hugging Face (if configured) Access model cards and documentation Quick Decision Tree Your task: ├─ Semantic search ("find beach images") → CLIP ✓ ├─ Zero-shot classification (broad categories) → CLIP ✓ ├─ Counting objects → DETR, Faster R-CNN ✗ ├─ Fine-grained ID (celebrities, car models) → Specialized model ✗ ├─ Spatial relations ("cat left of dog") → GQA, SWIG ✗ └─ Compositional ("red car AND blue truck") → DCSMs, PC-CLIP ✗

When to Use This Skill

✅ Use for:

Semantic image search Broad category classification Image similarity matching Zero-shot tasks on new categories

❌ Do NOT use for:

Counting objects in images Fine-grained classification Spatial understanding Attribute binding Negation handling Installation pip install transformers pillow torch sentence-transformers --break-system-packages

Validation: Run python scripts/validate_setup.py

Basic Usage Image Search from transformers import CLIPProcessor, CLIPModel from PIL import Image

model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14") processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")

Embed images

images = [Image.open(f"img{i}.jpg") for i in range(10)] inputs = processor(images=images, return_tensors="pt") image_features = model.get_image_features(**inputs)

Search with text

text_inputs = processor(text=["a beach at sunset"], return_tensors="pt") text_features = model.get_text_features(**text_inputs)

Compute similarity

similarity = (image_features @ text_features.T).softmax(dim=0)

Common Anti-Patterns Anti-Pattern 1: "CLIP for Everything"

❌ Wrong:

Using CLIP to count cars in an image

prompt = "How many cars are in this image?"

CLIP cannot count - it will give nonsense results

Why wrong: CLIP's architecture collapses spatial information into a single vector. It literally cannot count.

✓ Right:

from transformers import DetrImageProcessor, DetrForObjectDetection

processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50") model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")

Detect objects

results = model(**processor(images=image, return_tensors="pt"))

Filter for cars and count

car_detections = [d for d in results if d['label'] == 'car'] count = len(car_detections)

How to detect: If query contains "how many", "count", or numeric questions → Use object detection

Anti-Pattern 2: Fine-Grained Classification

❌ Wrong:

Trying to identify specific celebrities with CLIP

prompts = ["Tom Hanks", "Brad Pitt", "Morgan Freeman"]

CLIP will perform poorly - not trained for fine-grained face ID

Why wrong: CLIP trained on coarse categories. Fine-grained faces, car models, flower species require specialized models.

✓ Right:

Use a fine-tuned face recognition model

from transformers import AutoFeatureExtractor, AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained( "microsoft/resnet-50" # Then fine-tune on celebrity dataset )

Or use dedicated face recognition: ArcFace, CosFace

How to detect: If query asks to distinguish between similar items in same category → Use specialized model

Anti-Pattern 3: Spatial Understanding

❌ Wrong:

CLIP cannot understand spatial relationships

prompts = [ "cat to the left of dog", "cat to the right of dog" ]

Will give nearly identical scores

Why wrong: CLIP embeddings lose spatial topology. "Left" and "right" are treated as bag-of-words.

✓ Right:

Use a spatial reasoning model

Examples: GQA models, Visual Genome models, SWIG

from swig_model import SpatialRelationModel

model = SpatialRelationModel() result = model.predict_relation(image, "cat", "dog")

Returns: "left", "right", "above", "below", etc.

How to detect: If query contains directional words (left, right, above, under, next to) → Use spatial model

Anti-Pattern 4: Attribute Binding

❌ Wrong:

prompts = [ "red car and blue truck", "blue car and red truck" ]

CLIP often gives similar scores for both

Why wrong: CLIP cannot bind attributes to objects. It sees "red, blue, car, truck" as a bag of concepts.

✓ Right - Use PC-CLIP or DCSMs:

PC-CLIP: Fine-tuned for pairwise comparisons

from pc_clip import PCCLIPModel

model = PCCLIPModel.from_pretrained("pc-clip-vit-l")

Or use DCSMs (Dense Cosine Similarity Maps)

How to detect: If query has multiple objects with different attributes → Use compositional model

Evolution Timeline 2021: CLIP Released Revolutionary: zero-shot, 400M image-text pairs Widely adopted for everything Limitations not yet understood 2022-2023: Limitations Discovered Cannot count objects Poor at fine-grained classification Fails spatial reasoning Can't bind attributes 2024: Alternatives Emerge DCSMs: Preserve patch/token topology PC-CLIP: Trained on pairwise comparisons SpLiCE: Sparse interpretable embeddings 2025: Current Best Practices Use CLIP for what it's good at Task-specific models for limitations Compositional models for complex queries

LLM Mistake: LLMs trained on 2021-2023 data will suggest CLIP for everything because limitations weren't widely known. This skill corrects that.

Validation Script

Before using CLIP, check if it's appropriate:

python scripts/validate_clip_usage.py \ --query "your query here" \ --check-all

Returns:

✅ CLIP is appropriate ❌ Use alternative (with suggestion) Task-Specific Guidance Image Search (CLIP ✓)

Good use of CLIP

queries = ["beach", "mountain", "city skyline"]

Works well for broad semantic concepts

Zero-Shot Classification (CLIP ✓)

Good: Broad categories

categories = ["indoor", "outdoor", "nature", "urban"]

CLIP excels at this

Object Counting (CLIP ✗)

Use object detection instead

from transformers import DetrImageProcessor, DetrForObjectDetection

See /references/object_detection.md

Fine-Grained Classification (CLIP ✗)

Use specialized models

See /references/fine_grained_models.md

Spatial Reasoning (CLIP ✗)

Use spatial relation models

See /references/spatial_models.md

Troubleshooting Issue: CLIP gives unexpected results

Check:

Is this a counting task? → Use object detection Fine-grained classification? → Use specialized model Spatial query? → Use spatial model Multiple objects with attributes? → Use compositional model

Validation:

python scripts/diagnose_clip_issue.py --image path/to/image --query "your query"

Issue: Low similarity scores

Possible causes:

Query too specific (CLIP works better with broad concepts) Fine-grained task (not CLIP's strength) Need to adjust threshold

Solution: Try broader query or use alternative model

Model Selection Guide Model Best For Avoid For CLIP ViT-L/14 Semantic search, broad categories Counting, fine-grained, spatial DETR Object detection, counting Semantic similarity DINOv2 Fine-grained features Text-image matching PC-CLIP Attribute binding, comparisons General embedding DCSMs Compositional reasoning Simple similarity Performance Notes

CLIP models:

ViT-B/32: Fast, lower quality ViT-L/14: Balanced (recommended) ViT-g-14: Highest quality, slower

Inference time (single image, CPU):

ViT-B/32: ~100ms ViT-L/14: ~300ms ViT-g-14: ~1000ms Further Reading /references/clip_limitations.md - Detailed analysis of CLIP's failures /references/alternatives.md - When to use what model /references/compositional_reasoning.md - DCSMs and PC-CLIP deep dive /scripts/validate_clip_usage.py - Pre-flight validation tool /scripts/diagnose_clip_issue.py - Debug unexpected results

See CHANGELOG.md for version history.