#quantization

2 posts

open-source-ai• 15 min

ExLlamaV2: The fastest inference engine for quantized Llama/HF models (MIT, 5k stars)

Achieving 140+ tok/s on a single RTX 4090 with 4-bit quantization — the fastest inference engine for quantized Llama/HF models.

#exllamav2#inference

open-source-ai• 15 min

llama.cpp: The definitive LLM inference engine (MIT, 75k stars)

The definitive LLM inference engine — running quantized models on CPU and GPU with state-of-the-art performance via GGUF format.

#llamacpp#inference-engine