GenAI – Tokens In LLM
Table Of Contents:
- What Are Tokens In LLM ?
- Definition Of Tokens.
- Examples Of Tokens.
- How Tokenization Works?
- Why Tokens Matter ?
- Tokens Vs Character Vs Words.
- Tokenization Techniques.
- Tokenization Tools.
(1) What Is Tokenization ?
(2) Definition Of Tokens.
(3) Examples Of Tokens.
(4) Why Tokens Matter ?
(5) Tokens Vs Character Vs Words
(6) What Is Token Id & How LLM Going To Use It ?
(7) For The Same Word Will I Get Same Token Id ?
(7) Different Tokenization Techniques.
- Word Level Tokenization.
- Character Level Tokenization.
- Sub-word Tokenization.(Most Popular)
- Byte Pair Encoding.
- WordPiece
- SentencePiece
- Unigram Language Model
- Byte Level BPE
- Tokenization with Special Tokens
(7) Word Level Tokenization
(8) Character Level Tokenization
(9) Subword Tokenization
(10) Byte Level BPE
(11) Tokenization with Special Tokens
(12) Tokenization Libraries
