AI Glossary · Last reviewed August 2026

Latency

Hand-written by a real person. Reviewed against current practice in August 2026.

Definition

Time-to-first-token: the delay between sending a request and seeing the first word.

Why it matters

Latency is the delay between sending a prompt and seeing the first token of the response. High latency makes AI tools feel sluggish; low latency makes them feel instant. For real-time applications like coding assistants or chatbots, latency matters more than raw quality.

Understanding latency trade-offs helps you pick the right model size - smaller models respond faster but may sacrifice accuracy.

How it works

4 steps

STEP 01

Request is sent

Your prompt leaves the client and travels to the AI server over the network.

STEP 02

Processing queue

The server queues the request, loads the model if needed, and begins processing - this is compute latency.

STEP 03

First token generated

The model produces its first output token - the time from request to this point is Time to First Token (TTFT).

STEP 04

Streaming continues

Remaining tokens stream back to your screen, each adding a small inter-token latency.

Frequently asked questions

What is latency in AI?+

Latency is the amount of time an AI system takes to process a request and return a response.

Why is low latency important?+

Lower latency improves user experience by providing faster responses in real-time applications.

What factors increase AI latency?+

Large models, slow hardware, network delays, and inefficient processing can increase latency.

New to Latency?

See the tools that use it.

The fastest way to understand Latency is to see it inside real products. Browse hand-reviewed tools that put it to work, each one checked by a person before it was listed.

Browse hand-reviewed AI tools