Token

The smallest unit a language model processes. In English, ~0.75 words per token.

All terms