AI Cloud Drive

Project Overview

The AI Cloud Drive is a document retrieval platform combining secure file storage with hybrid retrieval pipelines. Users upload technical documents to a personal drive and query them using context-grounded retrieval and cross-encoder reranking.

Core Capabilities

Secure Storage: Personal cloud drive with file management (upload, share, delete).
Semantic Search: Semantic search across all uploaded documents.
RAG Chatbot: Context-aware Q&A using Groq's Llama models.
Enterprise-Ready Auth: Google OAuth, Magic Links, and RBAC features.
Observability: Prometheus monitoring and comprehensive audit logging.

Visual Tour

User Registration & Authentication

Features: Email verification with tokenized links, password strength validation, resend verification functionality.

Main Dashboard (File Manager)

Capabilities: Drag-and-drop upload, real-time status polling (Processing → Indexed), file metadata display.

RAG Chat Interface

Features: Multi-document filtering, inline citations with [Source 1] references, context preview cards.

Query Auditor & Guardrail Validation

The system validates context grounding by preventing generation when retrieved context is insufficient, logging query telemetry for auditing.

Admin Dashboard

Admin Capabilities: User management, storage analytics, real-time audit logs, system health monitoring, chat history auditing.

System Architecture

                Client [Vanilla JS] --> Nginx Proxy

                  |

                  +--> API [FastAPI]

                        |

                        +--> Auth [Google OAuth]

                        +--> LLM Ops [Groq API]

                        +--> Task Queue [Redis]

                              |

                              +--> Worker [Celery]
                --> OCR/Index

                        |

                        +--> Data Layer

                              |--> PostgreSQL
                (Metadata)

                              |--> MinIO (Object
                Storage)

                              |--> ChromaDB
                (Vectors)

Component Details

3.1 Frontend (/frontend)

Tech Stack: Vanilla HTML5, CSS3, JavaScript (ES6+), served via Nginx.
Key Modules: app.js (Main logic), admin.js (Dashboard), auth-container (Dynamic auth flows).
Features: Drag-and-drop uploads, real-time chat with typing indicators, file sharing dashboard.

3.2 Backend (/backend)

Framework: FastAPI (Python).
key Routes: auth.py (JWT/Login), files.py (CRUD), query.py (RAG search).

3.3 RAG Architecture

The RAG engine is modular and designed for high precision:

Ingestion: Upload -> MinIO -> Celery Task -> OCR (Tesseract) -> Chunking -> Embedding (Sentence Transformers) -> ChromaDB. Processed and indexed 1,000+ document chunks with hybrid retrieval and reranking pipelines.
Retrieval Strategy: Vector Search (ChromaDB) + Cross-Encoder Re-ranking to re-score candidates for semantic relevance, optimizing retrieval quality, citation tracking, and context validation to improve retrieval reliability and context grounding.
Generation: Top results assembled into context -> Groq API (Llama 3 70B) for response.

Key Design Decisions

Hybrid Search/Re-ranking: Instead of relying solely on vector similarity, the system retrieves a larger set of candidates and uses a Cross-Encoder to re-rank them, significantly improving accuracy.
Async Processing: Computational heavy lifting (PDF parsing, embedding) is offloaded to Celery workers to keep the API responsive.
Privacy-First RAG: ChromaDB queries are strictly filtered by user_id, ensuring users can never search across another user's private documents.
Modular Auth: Supports passwordless (Magic Link), OAuth (Google), and standard flows simultaneously.

Security Features

JWT Authentication: Stateless, secure session management.
RBAC: Role-Based Access Control (Admin vs. Standard user).
Audit Logging: All sensitive actions (deletions, sharing) are recorded.
Secure File Serving: Files served via signed URLs or authenticated proxy endpoints.

Tech Stack

Links