-
EAP – RAG Pipeline Design
EAP – RAG Pipeline Design 1. A document event arrives └─ Via Google Pub/Sub, or HTTP POST `/invoke` 2. `main.py` starts the pipeline └─ Reads tenant, connector, bucket, file path, and event type └─ Ignores duplicate events already in progress 3. Parse documents └─ Reads files from Google Cloud Storage └─ Uses Document AI by default └─ Extracts text, tables, layout, and image details └─ Saves parsed output under: `{tenant}/{connector}/parsed` 4. Chunk documents └─ Splits parsed content into small meaningful sections └─ Keeps headings/metadata where possible └─ Saves chunks under: `{tenant}/{connector}/chunked` 5. Index chunks └─ Creates/uses a tenant collection: `col-{tenant_id}`
