#inference-server

2 posts

open-source-ai• 15 min

Text Generation Inference: Hugging Face's production-grade LLM inference server (Apache 2.0, 9k stars)

Deploying LLM inference at scale with continuous batching, tensor parallelism, and flash attention — Hugging Face's production-grade inference server.

Jun 30, 2026

#tgi#huggingface

open-source-ai• 15 min

Triton Inference Server: NVIDIA's production inference server (BSD-3-Clause, 10.8k stars)

Supporting multiple frameworks, GPU optimization, and dynamic batching — NVIDIA's production inference server.

Jun 30, 2026

#triton#nvidia