on Orin JetPack 7.2 / L4T r39+, use upstream vLLM 0.20+ (
vllm/vllm-openai:latest
); on older Orin, pick the NVIDIA-AI-IOT prebuilt vLLM container; on Thor, use upstream vLLM 0.20+ or validated native vLLM 0.20+, and use NVIDIA SGLang 26.01 (
nvcr.io/nvidia/sglang:26.01-py3
, SGLang 0.5.5.post2) when SGLang is requested. Set MAXN, make Hugging Face credentials/cache available, and launch an OpenAI-compatible server. Works for both LLMs and VLMs.
Purpose
Provide a Jetson-appropriate serving recipe for an LLM or VLM using vLLM or SGLang, including runtime path, launch command, endpoint, and verification step.
When to use
"Run / serve / host this model on a Jetson."
"Start a vLLM server I can hit from Open WebUI / my app."
After
jetson-inference-mem-tune
produced launch flags and the user wants to actually start the server.
For recipe-only questions, answer from this document without starting
containers. Run live pre-flight checks only when the user asks you to check this
device or execute the deployment.
Prerequisites
Show more
Installs
687
Repository
nvidia/skills
GitHub Stars
2.4K
First Seen
Jun 22, 2026
Security Audits
Gen Agent Trust Hub
Pass
Socket
Pass
Snyk
Warn