AI Glossary · Last reviewed August 2026

Tokens

Hand-written by a real person. Reviewed against current practice in August 2026.

Definition

The chunks (usually word-pieces) that models read and produce.

Why it matters

Tokens are how AI models measure and charge for text. Every word you send and receive is split into tokens - roughly 1 token per word in English. Understanding tokens helps you estimate costs, stay within context limits, and write more efficient prompts.

When comparing AI tool pricing, token-based pricing means longer conversations and documents cost more. Knowing this helps you optimize usage and control expenses.

How it works

4 steps

STEP 01

Text is split into tokens

The tokenizer breaks input text into subword pieces. Common words become single tokens; rare words are split into multiple tokens.

STEP 02

Tokens are numbered

Each token is mapped to a unique ID from the model's vocabulary - typically 50,000 to 100,000 possible tokens.

STEP 03

Model processes token IDs

The model works entirely with these numeric IDs internally, predicting the next token ID in the sequence.

STEP 04

Tokens are decoded back to text

Output token IDs are converted back into readable text for the user.

Frequently asked questions

What are tokens in AI?+

Tokens are small units of text that AI models process when understanding and generating language.

Why do tokens matter in AI models?+

Token limits affect how much information a model can process and how responses are generated.

Are tokens the same as words?+

No. A token can be a word, part of a word, punctuation mark, or character depending on the language.

New to Tokens?

See the tools that use it.

The fastest way to understand Tokens is to see it inside real products. Browse hand-reviewed tools that put it to work, each one checked by a person before it was listed.

Browse hand-reviewed AI tools