-
GenAI – Recursive Depth First Search Algorithm.
GenAI – Recursive Depth First Search Algorithm Table Of Content: What Is Recursive Depth First Search Algorithm? Step By Step Working Of RDF Algorithm Applications Of RDF Algorithm. How DFS Helps In Tokenization. (1) What Is Recursive Depth First Search Algorithm ? (2) Step by step way of traversing through Trie tree using RDF algo using one example . (3) How DFS Algorithm Works On Trie Tree ? (4) How DFS Algorithm Helps In Tokenization ?
-
GenAI – Custom RAG Tokenizer.
GenAI – Custom RAG Tokenizer. Table Of Content: Custom RAG Coadding. (1) Python Code import logging import copy import datrie import math import os import re import string import sys from hanziconv import HanziConv from nltk import word_tokenize from nltk.stem import PorterStemmer, WordNetLemmatizer from api.utils.file_utils import get_project_base_directory class RagTokenizer: def key_(self, line): return str(line.lower().encode("utf-8"))[2:-1] def rkey_(self, line): return str(("DD" + (line[::-1].lower())).encode("utf-8"))[2:-1] def loadDict_(self, fnm): logging.info(f"[HUQIE]:Build trie from {fnm}") try: of = open(fnm, "r", encoding='utf-8') while True: line = of.readline() if not line: break line = re.sub(r"[rn]+", "", line) line = re.split(r"[ t]", line) k = self.key_(line[0]) F = int(math.log(float(line[1]) /
-
GenAI – What Is ‘Trie’ Prefix Tree ?
GenAI – What Is ‘Trie’ Prefix Tree ? Table Of Content: What Is Trie ? Key Properties Of Trie. Example Of Trie. What Can We Do With Trie? Why Not Just Use A List ? What Is Prefix Lookup? How ‘Trie’ Helps Text Tokenization ? (1) What Is ‘Trie’ ? (2) Key Properties Of ‘Trie’ . (3) How ‘Trie’ Works Internally ? (4) How “New York City” Will Get Inserted ? ['Ney York City', 'India', 'South Africa'] (5) What Happens If The Word Does Not Exist ? (6) How Trie Helps In Text Tokenization ? (7) How Trie will give
-
GenAI – Audio File Chunking Process
GenAI – Audio File Chunking Process Table Of Content: How To Chunk Audio Files ? (1) Reference Links https://github.com/infiniflow/ragflow/blob/main/rag/app/audio.py (2) How To Chunk Audio Files ? Imported File Links: https://github.com/infiniflow/ragflow/blob/main/api/db/__init__.py https://github.com/infiniflow/ragflow/blob/main/rag/nlp/rag_tokenizer.py https://github.com/infiniflow/ragflow/blob/main/api/db/services/llm_service.py https://github.com/infiniflow/ragflow/blob/main/rag/nlp/__init__.py import re from api.db import LLMType from rag.nlp import rag_tokenizer from api.db.services.llm_service import LLMBundle from rag.nlp import tokenize def chunk(filename, binary, tenant_id, lang, callback=None, **kwargs): doc = { "docnm_kwd": filename, "title_tks": rag_tokenizer.tokenize(re.sub(r".[a-zA-Z]+$", "", filename)) } doc["title_sm_tks"] = rag_tokenizer.fine_grained_tokenize(doc["title_tks"]) # is it English eng = lang.lower() == "english" # is_english(sections) try: callback(0.1, "USE Sequence2Txt LLM to transcription the audio") seq2txt_mdl = LLMBundle(tenant_id, LLMType.SPEECH2TEXT, lang=lang) ans = seq2txt_mdl.transcription(binary) callback(0.8,
-
Python – What Is Enum Classes ?
Python – What Is Enum Class ? Table Of Content: What Is Enum ? Why Do We Use Enum ? Example Of Enum Class ? What Does StrEnum Means ? Where Do Enums Shine ? Benefits Of Enum ? Summary. (1) What Is Enum ? (2) Why Do We Use Enum ? (4) Example Of Enum. (5) What Is StrEnum ? Example-1: Without StrEnum from enum import Enum class LLMType(Enum): CHAT = "chat" EMBEDDING = "embedding" SPEECH2TEXT = "speech2text" task_type = LLMType.SPEECH2TEXT print(task_type) LLMType.SPEECH2TEXT Example-2: With StrEnum from enum import StrEnum class LLMType(StrEnum): CHAT = "chat" EMBEDDING = "embedding" SPEECH2TEXT
-
GenAI – RagFlow Product Architecture
GenAI – RagaFlow Product Architecture Table Of Content: What Is RagFlow? Demo Of RagFlow. Key Features Of RagFlow. System Architecture. (1) What Is RagFlow? (2) Demo On RagFlow. (3) Key Features Of RagFlow. (4) System Architecture Of RagFlow. (5) RagFlow Tool Frontend (6) RagFlow Architecture Explanation (7) Function Of Each Layer In Details (8) Line Connection Significance
-
GenAI – RagaAI Catalyst Product Architecture
GenAI – RagaAI Catalyst Table Of Content: What Is RagaAI Catalyst? RagaAI Metrics Library. Synthetic Data Generation. Human Feedback & Annotations. On Premise Deployment. Fine Tuning. (0) Reference Links https://docs.raga.ai/ragaai-catalyst/ragaai-metric-library/rag-metrics/hallucination https://github.com/raga-ai-hub/RagaAI-Catalyst?tab=readme-ov-file#project-management (1) What Is RagaAI Catalyst ? (2) RAG Metrics (3) Chat Metrics (4) Text To SQL (5) Text Summarization (6) Information Extraction (7) Code Generation (9) Marketing Content Evaluation (10) Learning Management System (11) Guardrails Metrics (12) Vulnerability Scanner (13) Different Guardrails (14) Human Feedback & Annotation
-
GenAI – Query Aware Chunking.
-
GenAI – Adaptive Chunk Sizing
-
GenAI – Galileo’s Chunk Attribution & Utilization Metrics.
