Tokenization (AI)

August 23, 2025 2 weeks ago 1 min read

Process of splitting text into tokens using schemes like BPE or WordPiece; affects vocabulary, context use, and cost.