TensorRT-LLM
Inference & Compute · Free + paid
NVIDIA's LLM inference framework for NVIDIA GPUs with TensorRT acceleration
At a glance
- Made by
- NVIDIA
- Korean
- None (English and a few others only)
- API
- Yes
- Available on
- API, CLI
- Open source
- Apache 2.0
- GitHub
- 14,807 stars · NOASSERTION
What is TensorRT-LLM?
TensorRT LLM optimizes inference for LLMs and visual generation models with dedicated kernels for common operations, an efficient runtime, and a Python framework you can modify and extend directly. It provides an LLM API, the trtllm-serve server, and a benchmark CLI, with OpenAI-compatible client examples. It supports disaggregated serving, speculative decoding, KV cache compression and offloading, multi-LoRA, quantization, and more, and offers model deployment guides for Blackwell and Hopper.
- Disaggregated serving
- Speculative decoding
- KV cache compression and offloading
- Multi-LoRA adapter generation
- Visual generation model support (beta)
- trtllm-serve online serving (OpenAI client examples)
- trtllm-bench·trtllm-eval CLI
- Prometheus metrics
Who uses TensorRT-LLM and for what
- Deploying LLMs such as GPT-OSS and Qwen3 on Blackwell and Hopper GPUs
- Serving and benchmarking LLMs on Slurm clusters
Pricing
Open source (free)
Apache 2.0 오픈소스 LLM 추론 라이브러리(직접 설치), 요금 표기 없음
공식 페이지 확인 2026-10-14(산출 규칙 2026-10-15)
AI tools similar to TensorRT-LLM (alternatives)
TensorRT-LLM vs. alternatives at a glance
| AI | Pricing | Combos on the map | Connected tools | Free | Cheapest monthly | Korean |
|---|---|---|---|---|---|---|
| TensorRT-LLM | Free + paid | 0 | 5 | — | — | No |
| llama.cpp | Free + paid | 0 | 6 | — | — | — |
| ONNX Runtime | Free + paid | 0 | 6 | — | — | No |
| SGLang | Free + paid | 0 | 6 | — | — | No |
| Sol Engine | Free + paid | 0 | 6 | — | — | No |
0 of these 5 tools have a free plan.
Combo and connection counts are based on the relationships recorded on AI Galaxy Map (2026-10-26); they are not a quality ranking.
TensorRT-LLM on AI Galaxy Map
- One of 43 AI tools in the Inference & Compute field.
- Connected to 5 other tools — #16 in its field.
Frequently asked questions
Can I use TensorRT-LLM for free?
Open source (free). Prices can change, so check the official site before paying.
What AI can I use instead of TensorRT-LLM?
Tools that do the same job, such as llama.cpp, ONNX Runtime, SGLang, Sol Engine, are connected to it on the map.
Can I use TensorRT-LLM in Korean?
None (English and a few others only)
Sources (official pages)
Checked on the official pages on 2026-10-14. Prices and features can change, so check the official site again before paying.