EAP – RAG Pipeline Design

1. A document event arrives
   └─ Via Google Pub/Sub, or HTTP POST `/invoke`

2. `main.py` starts the pipeline
   └─ Reads tenant, connector, bucket, file path, and event type
   └─ Ignores duplicate events already in progress

3. Parse documents
   └─ Reads files from Google Cloud Storage
   └─ Uses Document AI by default
   └─ Extracts text, tables, layout, and image details
   └─ Saves parsed output under:
      `{tenant}/{connector}/parsed`

4. Chunk documents
   └─ Splits parsed content into small meaningful sections
   └─ Keeps headings/metadata where possible
   └─ Saves chunks under:
      `{tenant}/{connector}/chunked`

5. Index chunks
   └─ Creates/uses a tenant collection: `col-{tenant_id}`
   └─ Generates embeddings using `gemini-embedding-001`
   └─ Stores searchable chunks in Google Vector Search

6. Handle event action
   └─ `CREATE`: add document chunks
   └─ `UPDATE`: clear old collection data, then add new chunks
   └─ `DELETE`: remove collection data

7. Publish completion
   └─ Sends a Pub/Sub event indicating indexing completed

8. Search-ready
   └─ Retrieval can use semantic, text, or hybrid search
   └─ The current orchestrator also runs a sample hybrid query after indexing

Leave a Reply

Your email address will not be published. Required fields are marked *