Topic
Latency
1 essay
Latency Budgets for LLM Features That Need to Feel Fast
Users feel LLM latency before they feel model quality. Here is how I set a per-feature budget, spend it on first token and streaming, and watch p50 and p95.
Topic
1 essay
Users feel LLM latency before they feel model quality. Here is how I set a per-feature budget, spend it on first token and streaming, and watch p50 and p95.