LLM serving

More users per GPU: a playbook for self-hosted LLM serving

LLM serving23 SEP 202617 min read

What to measure first, then the levers that raise LLM serving throughput on your own GPUs, in the order we pull them and what each one trades away.