DEFT Mining and Embedding Skill
You are the operator of the DEFT embed-then-mine workflow for VCN AOI. Your job is to take a parquet of weak target images (the gap-analysis or routing output) and a source pool, then produce a deduplicated parquet of mined source images that look similar to the targets — ready to feed into the next training round.
The workflow is fixed and deterministic:
embed the targets, embed the source pool, then mine nearest neighbours.
Each step's output parquet is the next step's input. There is no iterative search, no clustering pass, no human-in-the-loop selection — depth comes from picking the right encoder and the right
topn
, not from a multi-phase investigation.
The whole skill is a thin wrapper around three direct
docker run
invocations against the
tao_toolkit.data_services
image declared in
versions.yaml
(resolved at runtime — see Setup). The container's entrypoint takes
-e [hydra overrides...]
— pass
embedding image_embeddings -e …
for embedding and
tmm nearest_neighbors -e …
for mining. The
-e
flag points at a YAML that supplies default values for the subtask's schema; anything afterward is a bare Hydra override (
key=value
) that selectively overrides spec fields per run. (There is no
dataset
keyword inside the container — that's the TAO launcher's pillar prefix and is dropped here.) Pull the image once if it isn't cached:
docker pull "$DS_IMAGE"
(after resolving
$DS_IMAGE
per Setup).
Schema keys can rename between data-services releases (the RCA skill saw
inference_csv
→
inference_results_dir
,
output_dir
→
results_dir
). When in doubt, introspect the actual schema once per image:
docker run --rm "$DS_IMAGE" embedding image_embeddings --cfg=job
and
... tmm nearest_neighbors --cfg=job
.
Inputs
Target parquet
— the gap-analysis output, typically
mining_gaps.parquet
from
tao-route-visual-changenet-samples
(or
gaps.parquet
from
tao-analyze-gaps-visual-changenet
if routing was skipped). Required column:
filepath
. If
label
is also present, label-aware filtering during mining is available; otherwise the mining task silently no-ops the filter.
Source pool
— a parquet of candidate images to mine against, with a
filepath
column. If the user only has a CSV, convert it to a parquet
with the same columns
before Step 2. For label-aware filtering, the pool must also carry a
label
column.
Embedding spec file
— a YAML containing
model
,
model_path
,
batch_size
, and (only when
model_path
is a TAO
.pth
/
.ckpt
)
model_config_path
. Reused across Steps 1 and 2;
input_parquet
/
output_parquet
are supplied per run as Hydra overrides. The
same
spec MUST drive both embedding steps — embeddings from different encoders are not comparable, and mismatched encoders are the most common cause of "the mined images look unrelated" reports.
Mining spec file
— a YAML containing
topn
,
knn_metric
,
filter_by_label
, and (rarely changed)
source_embed_column_name
/
target_embed_column_name
.
source_parquet
/
target_parquet
/
output_parquet
are Hydra overrides at run time. SigLIP and CLIP embeddings should use
knn_metric: cosine
. When
filter_by_label: true
but either embedding parquet lacks a
label
column, the container logs a warning and proceeds
without
filtering.
Show more
Installs
583
Repository
nvidia/skills
GitHub Stars
1.9K
First Seen
Jun 8, 2026
Security Audits
Gen Agent Trust Hub
Pass
Socket
Warn
Snyk
Warn