What the model actually reads
You type text. The neural network never receives that text directly. A tokenizer first turns it into model-specific pieces and integer IDs.
Follow one request end to end
Use Next step to move through the path. The IDs and splits are deliberately made-up teaching values.
Visible text arrives
The API receives Reset password?. This is still human-readable text. The neural network has not processed it yet.
Click the pieces, inspect the IDs
Choose a preset, then click a colored token. These examples are illustrative toy tokenizers—not claims about a commercial model.
Reset password?
Notice that spaces, punctuation, fragments, or bytes may be represented as pieces. A token is not reliably a word.
"Reset"
The number identifies this piece only inside this toy vocabulary. A neighboring ID does not imply a neighboring meaning.
" password", the leading space belongs to the token piece. It was not discarded.A mechanical unit, not a meaning unit
A token is one entry in a particular tokenizer’s finite vocabulary, represented by an integer ID.
A word
One word can be several tokens. A token can also contain whitespace plus a whole word.
A character
One token can contain many characters. One visible character can also require several tokens.
A unit of meaning
Vocabulary construction favors useful recurring pieces, but every piece need not represent a clean concept.
Which statement is safest?
Why can round-tripping differ?
What are special or control tokens?
The same text can become different sequences
Related model versions may share a tokenizer, but compatibility must be checked rather than assumed.
[4821, 10237, 30]
[91, 882, 377, 2041, 17]
Token counts are not portable
A prompt counted with tokenizer A may use a different number of positions with tokenizer B.
Token IDs are not portable
ID 10237 has no universal meaning. It only identifies an entry inside the vocabulary that assigned it.
Vocabulary is width. Context is length.
How many token types could occupy one position?
At each generation step, the model produces scores over its available vocabulary.
How many token positions fit in this run?
Input, output, system instructions, role markers, tools, and provider formatting may all consume positions.
Make space for the answer
This calculator illustrates budgeting, not a universal provider formula. Move the sliders and watch the remaining capacity.
Cost
Providers commonly use input and output tokens as billing units, but rates and accounting differ.
Latency
More tokens generally mean more work. The relationship is not universally linear.
Output limits
500 output tokens does not mean 500 words, characters, or lines.
Where surprising splits appear
Tokenization stops. Numerical representation begins.
Segmentation + ID mapping
It produces discrete IDs. Numeric adjacency has no semantic promise: ID 4821 and ID 4822 need not have related meanings.
Embedding lookup
Each ID indexes a learned input vector. Later layers turn that starting vector into context-dependent internal representations.
Recognize now: position information
Deferred: embedding geometry
What belongs at each depth
The core model
- Text → pieces → IDs → embeddings
- Token ≠ reliable word or character
- Generation decoding ≠ tokenizer decoding
- Vocabulary width ≠ context length
- Tokenizer/model pairing matters
Engineering caveats
- Special/control tokens
- Bytes and normalization
- Inference caching
- Provider-specific accounting
- Edge-case splits
Later lessons
- Embedding geometry
- Sampling and temperature
- Attention and transformers
- Cache mechanics
- Building tokenizers
Capture questions while you read
Write anything that still feels hand-waved. This stays only in this browser unless you copy it.