EAP – RAG Pipeline Design
1. A document event arrives
└─ Via Google Pub/Sub, or HTTP POST `/invoke`
2. `main.py` starts the pipeline
└─ Reads tenant, connector, bucket, file path, and event type
└─ Ignores duplicate events already in progress
3. Parse documents
└─ Reads files from Google Cloud Storage
└─ Uses Document AI by default
└─ Extracts text, tables, layout, and image details
└─ Saves parsed output under:
`{tenant}/{connector}/parsed`
4. Chunk documents
└─ Splits parsed content into small meaningful sections
└─ Keeps headings/metadata where possible
└─ Saves chunks under:
`{tenant}/{connector}/chunked`
5. Index chunks
└─ Creates/uses a tenant collection: `col-{tenant_id}`
└─ Generates embeddings using `gemini-embedding-001`
└─ Stores searchable chunks in Google Vector Search
6. Handle event action
└─ `CREATE`: add document chunks
└─ `UPDATE`: clear old collection data, then add new chunks
└─ `DELETE`: remove collection data
7. Publish completion
└─ Sends a Pub/Sub event indicating indexing completed
8. Search-ready
└─ Retrieval can use semantic, text, or hybrid search
└─ The current orchestrator also runs a sample hybrid query after indexing
