-
GenAI – What Are Tokens In LLM ?
GenAI – Tokens In LLM Table Of Contents: What Are Tokens In LLM ? Definition Of Tokens. Examples Of Tokens. How Tokenization Works? Why Tokens Matter ? Tokens Vs Character Vs Words. Tokenization Techniques. Tokenization Tools. (1) What Is Tokenization ? (2) Definition Of Tokens. (3) Examples Of Tokens. (4) Why Tokens Matter ? (5) Tokens Vs Character Vs Words (6) What Is Token Id & How LLM Going To Use It ? (7) For The Same Word Will I Get Same Token Id ? (7) Different Tokenization Techniques. Word Level Tokenization. Character Level Tokenization. Sub-word Tokenization.(Most Popular) Byte Pair
-
GenAI – Recursive Chunking
GenAI – Recursive Chunking Table Of Contents: What Is Recursive Chunking ? How Recursive Chunking Works ? When To Use Recursive Chunking ? When Not To Use Recursive Chunking ? Advantages Of Recursive Chunking. Disadvantages Of Recursive Chunking. Examples Of Recursive Chunking. (1) What Is Recursive Chunking ? (2) How Is Recursive Chunking Works ? (3) When To Use Recursive Chunking ? (4) When Not To Use Recursive Chunking ? (5) Advantages Of Recursive Chunking. (6) Disadvantages Of Recursive Chunking. (7) Examples Of Recursive Chunking.
-
GenAI – Delimiter Based Chunking.
GenAI – Delimiter Based Chunking Table Of Contents: What Is Delimiter Based Chunking ? Examples Of Delimiters. When To Use Delimiter Based Chunking ? When Not To Use Delimiter Based Chunking ? Advantages Of Delimiter Based Chunking. Disadvantages Of Delimiter Based Chunking. Examples Of Delimiter Based Chunking. (1) What Is Delimiter Based Chunking ? (2) Examples Of Delimiter. (3) When To Use Delimiter Based Chunking ? (4) When Not To Use Delimiter Based Chunking ? (5) Advantages Of Delimiter Based Chunking . (6) Disadvantages Of Delimiter Based Chunking . (7) Examples Of Delimiter Based Chunking . Example 1: Chunking Markdown
-
GenAI – Sentence Based Chunking.
GenAI – Sentence Based Chunking Table Of Contents: What Is Sentence Based Chunking ? When To Use Sentence Based Chunking ? Advantages Of Sentence Based Chunking. Disadvantages Of Sentence Based Chunking. Examples Of Sentence Based Chunking. (1) What Is Sentence Based Chunking ? (2) When To Use Sentence Based Chunking ? (3) When Not To Use Sentence Based Chunking ? (4) Advantages Of Sentence Based Chunking . (5) Disadvantages Of Sentence Based Chunking . (6) Examples Of Sentence Based Chunking . Example 1: Using Sentence Chunks with Transformers from transformers import GPT2Tokenizer tokenizer = GPT2Tokenizer.from_pretrained("gpt2") sentences = [ "This is
-
GenAI – Semantic Based Chunking ?
GenAI – Semantic Chunking Table Of Contents: What Is Semantic Chunking ? When To Use Semantic Chunking ? Advantages Of Semantic Chunking. Disadvantages Of Semantic Chunking. Examples Of Semantic Chunking. Techniques Available For Semantic Chunking . (1) What Is Semantic Chunking ? (2) When To Use Semantic Chunking ? (3) How Semantic Chunking Works ? (4) Tools & Libraries for Semantic Chunking (5) Advantages Of Semantic Chunking. (6) Disadvantages Of Semantic Chunking. (6)Techniques Available For Semantic Chunking. (7) Examples Of Semantic Chunking. Example 1: Using LangChain’s RecursiveCharacterTextSplitter This splits the text based on semantic boundaries like paragraphs, sentences, etc. from
-
GenAI – Sliding Window Chunking.
GenAI – Sliding Window Chunking Table Of Contents: What Is Sliding Window Chunking ? When To Use Sliding Window Chunking ? Advantages Of Sliding Window Chunking. Disadvantages Of Sliding Window Chunking. Examples Of Sliding Window Chunking. (1) What Is Sliding Window Chunking ? (2) When To Use Sliding Window Chunking ? (3) Advantages Of Sliding Window Chunking . (4) Disadvantages Of Sliding Window Chunking . (5) When Not To Use Sliding Window Chunking ? (6) Examples Of Sliding Window Chunking .
-
GenAI – Fixed Size Chunking .
GenAI – Fixed Size Chunking Table Of Contents: What Is Fixed Size Chunking ? When To Use Fixed Size Chunking ? Advantages Of Fixed Size Chunking. Disadvantages Of Fixed Size Chunking. Examples Of Fixed Size Chunking. (1) What Is Fixed Size Chunking ? (2) When To Use Fixed Size Chunking ? (3) Advantages Of Fixed Size Chunking. (4) Disadvantages Of Fixed Size Chunking. (5) Examples Of Fixed Size Chunking. Example – 1: Document Embedding for Semantic Search Use case: Splitting a long article into 512-token chunks for embedding in a vector database. from transformers import GPT2Tokenizer tokenizer = GPT2Tokenizer.from_pretrained(“gpt2”) text
-
GenAI – What Is Chunking ?
GenAI – Chunking Table Of Contents: What Is Chunking ? Why Is Chunking Is Needed ? Different Chunking Techniques. Challenges Faced In Chunking. Best Practices In Chunking. (1) What Is Chunking ? (2) Why Chunking Is Needed ? (3) How Chunking Helps To Solve The Problem? (4) Advantages Of Chunking . (5) Disadvantages Of Chunking . (6) What Will Happen If We Exceed The Token Limit For The LLM ? (7) Different Types Of Chunking Techniques.
-
Python – What Is Pydantic Library ?
Python – What Is Pydentic Python Library ? Table Of Contents: What Is Pydentic Library? key Features Of Pydentic Library ? Common Use Cases Of Pydentic Library. Examples Of Pydentic Library. (1) What Is Pydentic Library? (2) Validate , Parse & Enforces Type . Validation: Parsing: Enforces Types: (3) Validation Example. from pydantic import BaseModel, EmailStr class User(BaseModel): id: int name: str age: int email: EmailStr Valid Input: valid_data = { "id": 101, "name": "Alice", "age": 28, "email": "[email protected]" } user = User(**valid_data) print(user) Invalid Input: Missing One Parameter valid_data = { "id": 101, "name": "Alice", "email": "[email protected]" } user
-
Data Science – How To Handle Data Imbalance ?
Data Science – How To Handle Data Imbalance? Table Of Contents: Resampling Techniques. Class Weighting. Algorithmic Approach. Evaluation Metric Adjustments. Data Augmentation. Anomaly Detection Approach. (1) Resampling Techniques Types Of Resampling Techniques: When To Use Which Sampling Techniques ? (2) Class Weight Balance Technique Python Implementation When To Use Class Weight Technique: (3) Algorithmic Approach Of Handling Class Imbalance. (4) Evaluation Metric Adjustments. Can We Use ROC – AUC for Imbalanced Class ? (5) Data Augmentation Techniques to Handle Class Imbalance Common Data Augmentation Techniques for Imbalanced Data
