EAP – Knowledge Base To RAG Workflow


EAP – Knowledge Base To RAG As A Service

                            USER
                              │
                              │ 1. Create Connector
                              ▼
                  ┌─────────────────────────┐
                  │     FastAPI API Service │
                  │                         │
                  │ Validate Request        │
                  │ Create Integration      │
                  │ Create Job              │
                  │ Save to Firestore       │
                  └───────────┬─────────────┘
                              │
                 2. Save Metadata
                              │
                              ▼
                 ┌─────────────────────────┐
                 │      Firestore          │
                 │─────────────────────────│
                 │ integrations            │
                 │ jobs                    │
                 │ tenants                 │
                 │ file_items              │
                 │ errors                  │
                 └───────────┬─────────────┘
                              │
                 3. Publish Event
                              │
                              ▼
                  ┌────────────────────────┐
                  │     GCP Pub/Sub        │
                  └───────────┬────────────┘
                              │
         API Returns Success  │
         Immediately (2 sec)  │
                              │
                              ▼
                  ┌────────────────────────┐
                  │     Worker Service     │
                  │────────────────────────│
                  │ Listen for Events      │
                  │ Read Firestore Config  │
                  │ Connect to Source      │
                  └───────────┬────────────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
        ▼                     ▼                     ▼
  Confluence API       SharePoint API      ServiceNow API
        │                     │                     │
        └─────────────────────┼─────────────────────┘
                              │
                              ▼
                    Download Documents
                              │
                              ▼
                   Chunk / Clean Content
                              │
                              ▼
                     Store in Cloud Storage
                              │
                              ▼
                 Generate Embeddings (RAG)
                              │
                              ▼
                    Store in Vector Database
                              │
                              ▼
                     AI Agent / Chatbot
                              │
                              ▼
                       Answer User Queries

Better Approach:

Firestore Collections:

Leave a Reply

Your email address will not be published. Required fields are marked *