TAO Inference Microservice
Instructions
To start an inference service:
Collect required inputs (Section 1) and resolve the container image (Section 2).
Build the job payload and inner command (Sections 3–4.1); use
references/code-templates.yaml
→
job_payload_builder
.
Read
skills/platform//SKILL.md
and start the container (Section 4.2).
Write the service registry and poll readiness (Section 4.3); use
references/code-templates.yaml
→
registry_write.
and
readiness_check
.
To send an inference request:
Resolve which service receives the request per Section 6.0 (by
job_id
, by
network_arch
, or by explicit user choice when multiple services run —
never silently default to
"latest"
when more than one service exists
), then read the endpoint from
references/code-templates.yaml
→
request.registry_read
with the resolved
job_id
.
Before building the request body, prompt the user for the vLLM-style sampling parameters (Section 6.1).
Present
max_tokens
,
top_p
,
temperature
(and any per-arch extras) with their defaults; let the user override or skip each one to accept the default. Never silently use defaults.
Build and send the body per Section 6.2; handle the response per Section 6.3.
To stop a service:
Read
references/code-templates.yaml
→
stop.registry_read
to resolve the job_id, read
skills/platform//SKILL.md
, then follow Section 5.
Show more
Installs
579
Repository
nvidia/skills
GitHub Stars
1.9K
First Seen
Jun 8, 2026
Security Audits
Gen Agent Trust Hub
Pass
Socket
Pass
Snyk
Warn