jetson-speculative-decoding

Jetson Speculative Decoding (vLLM) Speculative decoding lets a small "draft" model propose tokens that the target model verifies in a single forward pass, reducing per-token latency. On Jetson, the win/loss is dominated by VRAM headroom , not by the draft quality. This skill encodes the parts an LLM won't already know. Purpose Tune an existing Jetson vLLM deployment for faster token generation by appending the right --speculative-config and validating whether it improves single-stream decode speed. When to use TPOT/ITL is the bottleneck (TTFT is fine, output is just slow). Workload is single-stream or low-concurrency (≤2). Speculation usually loses at high concurrency. Jetson family is Thor or AGX Orin . Do not suggest EAGLE-3 on Orin Nano/NX — there is rarely enough VRAM headroom to host both target and draft, and you'll OOM at startup. When NOT to use High-concurrency serving (≥8): batched decode usually beats speculation; the draft model just steals VRAM. Models without a published EAGLE-3 head — do not train one ad-hoc as a "fix". After applying jetson-inference-mem-tune flags that already pushed --gpu-memory-utilization near the ceiling. Free at least ~2 GB first. Show more Installs 700 Repository nvidia/skills GitHub Stars 2.4K First Seen Jun 22, 2026 Security Audits Gen Agent Trust Hub Pass Socket Pass Snyk Pass