> ## Documentation Index
> Fetch the complete documentation index at: https://docs.olis-ai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# API Architecture & Backend Guide

> Comprehensive guide to the Olis RAG API architecture, endpoints, and development

# Olis API Architecture & Backend Developer Guide

## Table of Contents

1. [Architecture Overview](#architecture-overview)
2. [System Components](#system-components)
3. [API Endpoints](#api-endpoints)
4. [Authentication & Authorization](#authentication--authorization)
5. [RAG Pipeline Deep Dive](#rag-pipeline-deep-dive)
6. [Configuration Reference](#configuration-reference)
7. [Development Guide](#development-guide)
8. [Performance & Monitoring](#performance--monitoring)
9. [Deployment](#deployment)
10. [Troubleshooting](#troubleshooting)

***

## Architecture Overview

### High-Level Architecture

The Olis API Server is a **FastAPI-based RAG (Retrieval-Augmented Generation) system** that provides intelligent question answering with document context.

```mermaid theme={null}
graph TB
    Client[Client Application] -->|HTTP/JSON| API[FastAPI Server]
    API --> Auth[JWT Auth Layer]
    Auth --> Endpoints[API Endpoints]

    Endpoints --> Intent[Intent Detection]
    Endpoints --> RAG[RAG Pipeline]
    Endpoints --> Ingest[Document Ingestion]

    RAG --> QP[Query Processor]
    RAG --> Retriever[Hybrid Retriever]
    RAG --> LLM[LLM Generator]

    Retriever --> Milvus[(Milvus Vector DB)]
    Retriever --> ES[(Elasticsearch BM25)]
    Retriever --> Reranker[Reranker]

    LLM --> Memory[Session/Thread Memory]
    Memory --> Redis[(Redis Cache)]

    Ingest --> Parser[Document Parser]
    Parser --> Embedder[Embedding Model]
    Embedder --> Milvus
    Embedder --> ES
```

### Technology Stack

| Component            | Technology               | Purpose                           |
| -------------------- | ------------------------ | --------------------------------- |
| **API Framework**    | FastAPI                  | High-performance async API server |
| **Vector Database**  | Milvus                   | Semantic search with embeddings   |
| **Full-Text Search** | Elasticsearch            | BM25 keyword search               |
| **Cache/Memory**     | Redis                    | Session state, thread memory      |
| **LLM Provider**     | OpenAI / Ollama          | Answer generation                 |
| **Embeddings**       | HuggingFace Transformers | Document & query embeddings       |
| **Reranker**         | BGE Reranker v2-m3       | Result re-ranking                 |
| **Authentication**   | JWT                      | Token-based auth                  |
| **Background Tasks** | FastAPI BackgroundTasks  | Async memory updates              |

***

## System Components

### 1. Application State (`AppResources`)

The API maintains application-wide state through the `AppResources` class:

```python theme={null}
class AppResources:
    retriever: object              # Hybrid retriever (Milvus + ES)
    generator: object              # LLM handler
    answer_service: AnswerService  # Core RAG service
    user_chains: Dict              # Per-user conversation chains
    session_memory: SessionMemory  # User session memory
    thread_memory: ThreadMemory    # Conversation thread memory
    ingester: DocumentIngester     # Document ingestion pipeline
    org_manager: OrgResourceManager # Multi-tenant org isolation
```

**Lifecycle**: Initialized in `lifespan()` context manager, available via dependency injection.

### 2. Multi-Tenant Organization Manager

**Hard isolation** per organization:

* Each org gets dedicated Milvus collection: `{base}_orgid`
* Each org gets dedicated ES index: `{base}_orgid`
* Resources cached per org to avoid reconnection overhead

```python theme={null}
# Example: Getting org-specific resources
org_res = resources.org_manager.get(org_id, init_ingester=True)
retriever = org_res["retriever"]
answer_service = org_res["answer_service"]
```

### 3. RAG Pipeline Components

#### a. Hybrid Retriever

* **Vector Search** (Milvus): Semantic similarity using embeddings
* **BM25 Search** (Elasticsearch): Keyword/lexical matching
* **Weighted Fusion**: Configurable weights (`WEIGHT_VECTOR`, `WEIGHT_BM25`)
* **Reranking**: Optional BGE reranker for precision

#### b. Query Processor

* Query decomposition into sub-queries (optional)
* Query expansion and reformulation
* Temporal filtering and date extraction

#### c. LLM Generator

* Supports OpenAI (`gpt-4o-mini`) and Ollama (local models)
* Configurable temperature, top-p, max tokens
* Structured output with citations

#### d. Memory System

* **Session Memory**: Long-term user context (10 mins TTL)
* **Thread Memory**: Conversation history per thread (5 mins TTL)
* Stored in Redis with automatic expiration

***

## API Endpoints

### Base URL

```
http://localhost:8002
```

### 1. Health Check

```http theme={null}
GET /healthz
```

**Response**:

```json theme={null}
{
  "status": "ok"
}
```

***

### 2. Intent Detection

Analyze user input to detect intent before RAG pipeline execution.

```http theme={null}
POST /query
```

**Headers**:

```
Authorization: Bearer <JWT_TOKEN>
Content-Type: application/json
```

**Request Body**:

```json theme={null}
{
  "text": "I need to review the compliance procedures before contacting the client",
  "user_info": {
    "user_name": "john@company.com",
    "real_name": "John Doe"
  }
}
```

**Response**:

```json theme={null}
{
  "intent": "compliance_check",
  "confidence": 0.95,
  "requires_rag": true,
  "input": "I need to review...",
  "user_info": {
    "user_name": "john@company.com",
    "real_name": "John Doe",
    "org_id": "uuid-here",
    "current_roles": ["analyst"],
    "current_groups": []
  }
}
```

**Use Case**: Pre-filter queries, route to appropriate handlers, avoid unnecessary RAG calls.

***

### 3. RAG Query/Prediction

Main endpoint for question answering with document retrieval.

```http theme={null}
POST /predict
```

**Headers**:

```
Authorization: Bearer <JWT_TOKEN>
Content-Type: application/json
```

**Request Body**:

```json theme={null}
{
  "query": "What are the security implementation challenges in Olis?",
  "user_info": {
    "user_name": "developer@olis.com",
    "real_name": "Dev User"
  },
  "thread_id": "optional-existing-thread-id"
}
```

**Response**:

```json theme={null}
{
  "answer": {
    "callout_answer": "The main security challenges in Olis include...",
    "positive_list": [
      "Implementing end-to-end encryption for sensitive data",
      "Establishing secure authentication flows"
    ],
    "negative_list": [],
    "info_list": [
      "Security audit scheduled for Q2",
      "RBAC implementation in progress"
    ],
    "keywords": ["security", "encryption", "authentication", "RBAC"]
  },
  "sources": {
    "doc_1": {
      "title": "Security Implementation Plan",
      "source": "internal-wiki",
      "page_content": "...",
      "metadata": { "owner_id": "admin@olis.com", "acl": "|user:_all_|" }
    }
  },
  "query": "What are the security implementation challenges in Olis?",
  "original_query": "What are the security implementation challenges in Olis?",
  "input": "What are the security implementation challenges in Olis?",
  "thread_id": "uuid-thread-id"
}
```

**Flow**:

1. Extract JWT, validate user
2. Create or retrieve thread ID
3. Normalize query text
4. Run RAG pipeline (see [RAG Pipeline Deep Dive](#rag-pipeline-deep-dive))
5. Filter results by RBAC
6. Update session memory (background task)
7. Return structured answer with citations

***

### 4. Document Upsert/Ingestion

Insert or update documents in the knowledge base.

```http theme={null}
POST /upsert
```

**Headers**:

```
Authorization: Bearer <JWT_TOKEN>
Content-Type: application/json
```

**Request Body**:

```json theme={null}
{
  "user_email": "uploader@olis.com",
  "documents": {
    "doc_id_1": {
      "title": "Product Requirements Document",
      "content": "Full document text here...",
      "metadata": {
        "source": "confluence",
        "created_at": "2026-01-15",
        "owner_id": "pm@olis.com",
        "acl": "|user:_all_|role:pm|role:engineering|"
      }
    },
    "doc_id_2": {
      "title": "API Design Spec",
      "content": "...",
      "metadata": { "source": "github", "acl": "|role:engineering|" }
    }
  }
}
```

**Response**:

```json theme={null}
{
  "status": "success",
  "user": "uploader@olis.com",
  "num_chunks": "42"
}
```

**Processing**:

1. Validate JWT and extract org\_id
2. Get org-specific ingester (creates if needed)
3. Chunk documents (default: 1000 chars, 150 overlap)
4. Generate embeddings (HuggingFace model)
5. Store in Milvus (vectors) and Elasticsearch (full-text)
6. Apply org\_id stamp for hard isolation

**Supported Document Fields**:

* `title`: Document title
* `content`: Full text content
* `metadata.source`: Source system (confluence, slack, github, etc.)
* `metadata.owner_id`: Document owner email
* `metadata.acl`: Access control list (pipe-delimited)
* `metadata.created_at`: ISO date string
* `metadata.*`: Any additional metadata

***

### 5. Test Prediction (No Auth Required)

Simplified prediction endpoint for testing (auth bypass).

```http theme={null}
POST /testpredict
```

**Request**: Same as `/predict`

**Response**: Same as `/predict` + `prediction_time` field

**⚠️ WARNING**: This endpoint bypasses JWT auth. Only enable in development!

***

## Authentication & Authorization

### JWT Authentication

**Token Format**: Bearer token in `Authorization` header

```
Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...
```

**Required JWT Claims**:

```json theme={null}
{
  "email": "user@company.com",
  "role": "engineer",
  "org_id": "uuid-org-id",
  "type": "access",
  "exp": 1738425600,
  "iat": 1738422000,
  "iss": "oauth-app",
  "jti": "unique-token-id"
}
```

**Configuration**:

```bash theme={null}
JWT_SECRET=your-secret-key-here
JWT_ISSUER=oauth-app
JWT_ALGORITHM=HS256
```

**Token Validation**:

* Signature verification with `JWT_SECRET`
* Expiration check (`exp`)
* Issuer validation (`iss`)
* Token type must be `access`

**Error Responses**:

* `401`: Missing/invalid/expired token
* `503`: JWT\_SECRET not configured

### Role-Based Access Control (RBAC)

Documents include ACL metadata for fine-grained access control.

**ACL Format** (pipe-delimited keyset):

```
|user:john@company.com|role:engineer|role:pm|group:product|deny:banned@company.com|
```

**Access Rules**:

1. **Deny always wins**: If `|deny:user@email.com|` present, access blocked
2. **Owner access**: If `owner_id` matches user email, granted
3. **Public access**: If `|user:_all_|` present, granted to all
4. **User-specific**: If `|user:email@company.com|` matches, granted
5. **Role-based**: If `|role:engineer|` and user has role, granted
6. **Group-based**: If `|group:product|` and user in group, granted
7. **Default deny**: If no ACL or no match, only owner can access

**Implementation**:

```python theme={null}
def _doc_allowed_for_user(doc: Any, user_info: Dict[str, Any]) -> bool:
    meta = doc.metadata
    user = user_info.get("user_name").lower()
    owner = meta.get("owner_id", "").lower()
    acl = meta.get("acl", "").lower()

    # Deny check
    if f"|deny:{user}|" in acl:
        return False

    # Owner check
    if owner == user:
        return True

    # Public check
    if "|user:_all_|" in acl:
        return True

    # User/role/group checks...
    # (See main.py:_doc_allowed_for_user for full implementation)
```

**Filtering**:

* Post-retrieval filtering: `_filter_docs_by_rbac(docs, user_info)`
* Applied after retrieval, before LLM generation

***

## RAG Pipeline Deep Dive

### Pipeline Execution Flow

```mermaid theme={null}
graph TB
    Start[Query Input] --> Normalize[Normalize Query]
    Normalize --> Retrieval[Hybrid Retrieval]

    Retrieval --> Vector[Milvus Vector Search]
    Retrieval --> BM25[Elasticsearch BM25]

    Vector --> Merge[Weighted Fusion]
    BM25 --> Merge

    Merge --> Rerank{Reranking Enabled?}
    Rerank -->|Yes| RerankerStep[BGE Reranker]
    Rerank -->|No| RBAC[RBAC Filtering]
    RerankerStep --> RBAC

    RBAC --> TopK[Select Top-K]
    TopK --> EmptyCheck{Docs Empty?}

    EmptyCheck -->|Yes + Skip LLM| FastPath[Return Empty Response]
    EmptyCheck -->|No| Context[Prepare Context]

    Context --> LLM[LLM Generation]
    LLM --> Parse[Parse Answer + Citations]
    Parse --> Prune[Prune Unused Sources]
    Prune --> Return[Return Response]

    FastPath --> End[Response]
    Return --> End
```

### Step-by-Step Breakdown

#### 1. Query Normalization

```python theme={null}
query_text = _normalize_query_value(input.query)
# Handles string or dict input, extracts query value
```

#### 2. Retrieval Phase

**Vector Retrieval (Milvus)**:

* Embed query using same model as documents
* Cosine similarity search in vector space
* Top-K results: `VECTOR_RETRIEVER_TOP_K` (default: 8)

**BM25 Retrieval (Elasticsearch)**:

* Lexical keyword matching
* TF-IDF scoring
* Top-K results: `BM25_RETRIEVER_TOP_K` (default: 8)

**Hybrid Fusion**:

```python theme={null}
# Weighted combination
score = (vector_score * WEIGHT_VECTOR) + (bm25_score * WEIGHT_BM25)
# Default weights: 0.5 / 0.5
```

**Performance Timing**:

```python theme={null}
retrieval_ms = int((time.monotonic() - t0) * 1000)
_perf_emit("retrieval_done", {"retrieval_ms": retrieval_ms, "docs_count": len(docs)})
```

#### 3. Reranking (Optional)

If `USE_RERANKER=true`:

* Model: `BAAI/bge-reranker-v2-m3`
* Re-score top results for precision
* Select top `RERANK_TOP_K` (default: 5)

#### 4. RBAC Filtering

```python theme={null}
docs = _filter_docs_by_rbac(docs, user_info)
# Remove docs user doesn't have permission to access
```

#### 5. Context Preparation

```python theme={null}
context = answer_service.prepare_context(
    inputs={
        "retrieved_docs": docs,
        "query": query_text,
        "original_query": query_text,
        "subqueries": [query_text],
    },
    user_info=user_info,
    chat_history=None,  # Added from thread memory if available
)
```

**Context Structure**:

* `context`: Concatenated document chunks with source markers
* `sources`: Dict of source documents with metadata
* `query`: Current query
* `original_query`: Original user query (before decomposition)
* `subqueries`: List of decomposed queries (if enabled)

#### 6. LLM Generation

```python theme={null}
raw_answer = answer_service.generate_answer_with_prepared_context(
    context=context,
    answer_prompt=AnswerWithKeywordsPrompt,
)
```

**Prompt Structure**:

```
You are Olis, an intelligent assistant that answers questions using provided documents.

Context:
[Document chunks with [Source: doc_1] markers]

Question: {query}

Instructions:
- Provide a comprehensive answer using the context
- Include specific examples and details
- Cite sources using [Source: doc_id] format
- Structure your answer with:
  - callout_answer: Main answer summary
  - positive_list: Specific points/facts supporting the answer
  - negative_list: Caveats/limitations
  - info_list: Additional relevant information
  - keywords: Key terms from the answer

Format your response as JSON.
```

**LLM Configuration**:

* Model: `RAG_GENERATOR_MODEL` (default: `gpt-4o-mini`)
* Temperature: `RAG_LLM_TEMP` (default: 0.0)
* Top-P: `RAG_LLM_TOP_P` (default: 0.2)
* Max tokens: Configured via `OLLAMA_NUM_PREDICT` or model default

#### 7. Post-Processing

```python theme={null}
# Flatten answer + keywords structure
payload = answer_service.flatten_ak(context)

# Coalesce citation keys ([Source: doc_1] -> sources dict)
payload = answer_service.coalesce_citation_keys(payload)

# Remove unused sources not cited in answer
payload = answer_service.prune_sources_by_citations(payload)

# Parse JSON string answers to objects
payload = _maybe_parse_answer(payload)
```

#### 8. Fast Path (Empty Context)

If `SKIP_LLM_ON_EMPTY_CONTEXT=true` and no docs retrieved:

```json theme={null}
{
  "answer": {
    "callout_answer": "No information could be found for the question based on the provided sources.",
    "positive_list": [],
    "negative_list": [],
    "info_list": [],
    "keywords": []
  },
  "sources": {},
  "query": "..."
}
```

Saves LLM cost and latency when no relevant docs found.

***

### Query Decomposition (Advanced)

If `USE_QUERY_DECOMPOSITION=true`:

1. **Decompose** complex query into `QUERY_DECOMPOSITION_N` sub-queries (default: 5)
2. **Retrieve** docs for each sub-query independently
3. **Select** top `SUBQUERY_TOP_K` docs per sub-query (default: 5)
4. **Merge** and deduplicate results
5. **Reason** over combined results (if `SUBQUERY_REASONING_ENABLED=true`)

**Example**:

```
Query: "What security challenges does Olis face and who is addressing them?"

Sub-queries:
1. "What are the security challenges in Olis?"
2. "Who is responsible for security in Olis?"
3. "What is the current security implementation status?"
4. "What security features are planned?"
5. "What security risks have been identified?"
```

***

## Configuration Reference

### Environment Variables

#### Database Configuration

| Variable            | Default                     | Description           |
| ------------------- | --------------------------- | --------------------- |
| `MILVUS_URI`        | `http://milvus:19530`       | Milvus connection URI |
| `MILVUS_COLLECTION` | `docs_collection`           | Base collection name  |
| `ES_URI`            | `http://elasticsearch:9200` | Elasticsearch URI     |
| `ES_INDEX`          | `docs_bm25`                 | Base index name       |

#### LLM Configuration

| Variable              | Default                  | Description                         |
| --------------------- | ------------------------ | ----------------------------------- |
| `RAG_LLM_PROVIDER`    | `openai`                 | LLM provider (`openai` or `ollama`) |
| `OLLAMA_BASE_URL`     | `http://localhost:11434` | Ollama server URL                   |
| `OLLAMA_MODEL`        | `llama3.1`               | Ollama model name                   |
| `OLLAMA_NUM_PREDICT`  | `null`                   | Max tokens to generate              |
| `OLLAMA_NUM_CTX`      | `null`                   | Context window size                 |
| `RAG_GENERATOR_MODEL` | `gpt-4o-mini`            | OpenAI model name                   |
| `RAG_LLM_TEMP`        | `0.0`                    | LLM temperature                     |
| `RAG_LLM_TOP_P`       | `0.2`                    | Top-p sampling                      |
| `OPENAI_API_KEY`      | -                        | OpenAI API key                      |

#### Retrieval Configuration

| Variable                    | Default | Description              |
| --------------------------- | ------- | ------------------------ |
| `VECTOR_RETRIEVER_TOP_K`    | `8`     | Vector search top-K      |
| `BM25_RETRIEVER_TOP_K`      | `8`     | BM25 search top-K        |
| `RETRIEVER_TOP_K`           | `5`     | Final top-K after fusion |
| `WEIGHT_VECTOR`             | `0.5`   | Vector search weight     |
| `WEIGHT_BM25`               | `0.5`   | BM25 search weight       |
| `USE_RERANKER`              | `false` | Enable reranking         |
| `RERANK_TOP_K`              | `5`     | Reranker top-K           |
| `RERANKER_SCORE_THRESHOLD`  | `0.5`   | Min reranker score       |
| `SKIP_LLM_ON_EMPTY_CONTEXT` | `true`  | Fast path when no docs   |

#### Query Processing

| Variable                     | Default | Description                |
| ---------------------------- | ------- | -------------------------- |
| `USE_QUERY_DECOMPOSITION`    | `true`  | Enable query decomposition |
| `QUERY_DECOMPOSITION_N`      | `5`     | Number of sub-queries      |
| `SUBQUERY_TOP_K`             | `5`     | Docs per sub-query         |
| `SUBQUERY_REASONING_ENABLED` | `true`  | Multi-hop reasoning        |

#### Memory Configuration

| Variable             | Default | Description           |
| -------------------- | ------- | --------------------- |
| `REDIS_URL`          | -       | Redis connection URL  |
| `REDIS_HOST`         | `redis` | Redis host            |
| `REDIS_PORT`         | `6379`  | Redis port            |
| `REDIS_DB`           | `0`     | Redis database number |
| `REDIS_PASSWORD`     | -       | Redis password        |
| `REDIS_SSL`          | `false` | Use SSL for Redis     |
| `SESSION_TIMEOUT`    | `600`   | Session TTL (seconds) |
| `MAX_SESSION_TOKENS` | `500`   | Max tokens in session |
| `THREAD_TIMEOUT`     | `300`   | Thread TTL (seconds)  |

#### Document Ingestion

| Variable             | Default                                            | Description                         |
| -------------------- | -------------------------------------------------- | ----------------------------------- |
| `CHUNK_SIZE`         | `1000`                                             | Chunk size (characters)             |
| `CHUNK_OVERLAP`      | `150`                                              | Overlap between chunks              |
| `USE_SEMANTIC_SPLIT` | `false`                                            | Semantic chunking                   |
| `USE_RAPTOR`         | `true`                                             | Enable RAPTOR hierarchical indexing |
| `EMBED_MODEL_NAME`   | `sentence-transformers/msmarco-distilbert-base-v4` | Embedding model                     |
| `EMBED_DIMENSION`    | `768`                                              | Embedding dimension                 |

#### Authentication

| Variable        | Default     | Description           |
| --------------- | ----------- | --------------------- |
| `JWT_SECRET`    | -           | JWT signing secret    |
| `JWT_ISSUER`    | `oauth-app` | JWT issuer claim      |
| `JWT_ALGORITHM` | `HS256`     | JWT signing algorithm |

#### Performance

| Variable                | Default | Description                  |
| ----------------------- | ------- | ---------------------------- |
| `PERF_LOG`              | `false` | Enable performance logging   |
| `MAX_RETRIEVAL_THREADS` | `10`    | Concurrent retrieval threads |
| `MAX_INGESTION_THREADS` | `10`    | Concurrent ingestion threads |

***

## Development Guide

### Local Setup

#### Prerequisites

* Python 3.10+
* Docker & Docker Compose
* Poetry (Python dependency manager)

#### 1. Clone Repository

```bash theme={null}
git clone <repo-url>
cd olis-monorepo/apps/api-server
```

#### 2. Install Dependencies

```bash theme={null}
poetry install
# or
pip install -r requirements.txt
```

#### 3. Start Infrastructure Services

```bash theme={null}
docker-compose up -d
# Starts: Milvus, Elasticsearch, Redis
```

#### 4. Set Environment Variables

```bash theme={null}
cp .env.example .env
# Edit .env with your configuration
```

**Minimal `.env`**:

```bash theme={null}
# Database
MILVUS_URI=http://localhost:19530
ES_URI=http://localhost:9200
REDIS_HOST=localhost
REDIS_PORT=6379

# LLM
RAG_LLM_PROVIDER=openai
OPENAI_API_KEY=sk-...
RAG_GENERATOR_MODEL=gpt-4o-mini

# Auth
JWT_SECRET=your-secret-key-change-in-production
JWT_ISSUER=olis-api
```

#### 5. Run Server

```bash theme={null}
# Development with hot reload
uvicorn src.api_server.main:app --reload --port 8002

# Production
gunicorn src.api_server.main:app -w 4 -k uvicorn.workers.UvicornWorker --bind 0.0.0.0:8002
```

#### 6. Verify

```bash theme={null}
curl http://localhost:8002/healthz
# Expected: {"status":"ok"}
```

***

### Testing

#### Integration Tests

```bash theme={null}
# Run integration tests (requires running server)
RUN_INTEGRATION_TESTS=1 pytest tests/integration_tests/test_fastapi.py -v
```

**Example Test**:

```python theme={null}
import requests

def test_predict_endpoint():
    url = "http://localhost:8002/predict"
    headers = {"Authorization": f"Bearer {get_test_token()}"}
    data = {
        "query": "What are the Olis features?",
        "user_info": {"user_name": "test@olis.com"}
    }
    response = requests.post(url, json=data, headers=headers)
    assert response.status_code == 200
    assert "answer" in response.json()
```

#### Unit Tests

```bash theme={null}
pytest tests/unit_tests/ -v
```

#### Load Testing

```bash theme={null}
# Using locust
locust -f tests/load_tests/locustfile.py --host=http://localhost:8002
```

***

### Debugging

#### Enable Debug Logging

```python theme={null}
# In main.py
logging.basicConfig(level=logging.DEBUG)
```

#### Performance Logging

```bash theme={null}
# Enable performance metrics
export PERF_LOG=true

# Server will emit:
# PERF {"event":"rag_start","ts":1738425600000,"request_id":"..."}
# PERF {"event":"retrieval_done","ts":1738425601000,"retrieval_ms":342,"docs_count":5}
# PERF {"event":"rag_end","ts":1738425603000,"total_ms":3124}
```

#### Agent Debugging

Custom debug logging is available via `_debug_log()`:

```python theme={null}
_debug_log(
    "main.py:lifespan",
    "retrieval_config",
    {"top_k": config.RETRIEVER_TOP_K, "use_reranker": config.USE_RERANKER},
    "H1"  # Hypothesis ID
)
```

Logs written to: `DEBUG_LOG_PATH` (default: `.cursor/debug.log`)

***

## Performance & Monitoring

### Performance Metrics

The RAG pipeline emits detailed performance metrics:

```json theme={null}
{
  "event": "rag_end",
  "request_id": "uuid",
  "query_hash": "abc123def4",
  "retrieval_ms": 342,
  "context_ms": 12,
  "llm_ms": 2456,
  "post_ms": 23,
  "total_ms": 2833,
  "docs_count": 5,
  "context_chars": 4532,
  "model": "gpt-4o-mini",
  "num_predict": 500,
  "num_ctx": 4096
}
```

**Key Metrics**:

* `retrieval_ms`: Time for hybrid retrieval
* `llm_ms`: Time for LLM generation
* `total_ms`: End-to-end latency
* `docs_count`: Number of retrieved documents
* `context_chars`: Context size sent to LLM

### Optimization Tips

#### 1. Tune Retrieval Parameters

```bash theme={null}
# Reduce top-K for faster retrieval
VECTOR_RETRIEVER_TOP_K=5
BM25_RETRIEVER_TOP_K=5
RETRIEVER_TOP_K=3

# Disable reranker if not needed
USE_RERANKER=false
```

#### 2. Use Fast Path

```bash theme={null}
# Skip LLM when no docs found
SKIP_LLM_ON_EMPTY_CONTEXT=true
```

#### 3. Optimize Context Size

```bash theme={null}
# Limit doc/context characters
MAX_DOC_CHARS=1000
MAX_CONTEXT_CHARS=4000
```

#### 4. Disable Query Decomposition

```bash theme={null}
# Faster but less comprehensive
USE_QUERY_DECOMPOSITION=false
```

#### 5. Use Ollama for Local LLM

```bash theme={null}
# Avoid OpenAI API latency
RAG_LLM_PROVIDER=ollama
OLLAMA_MODEL=llama3.1
OLLAMA_NUM_PREDICT=300  # Faster generation
```

### Caching Strategy

* **Session Memory**: 10-minute TTL, stores user context
* **Thread Memory**: 5-minute TTL, stores conversation history
* **Org Resources**: Cached indefinitely per org\_id

**Redis Memory Usage**:

* Session: \~10KB per user
* Thread: \~5KB per thread
* Estimate: 100 concurrent users = \~1.5MB

***

## Deployment

### Docker Deployment

#### 1. Build Image

```bash theme={null}
cd apps/api-server
docker build -t olis-api:latest .
```

#### 2. Run Container

```bash theme={null}
docker run -d \
  -p 8002:8002 \
  -e MILVUS_URI=http://milvus:19530 \
  -e ES_URI=http://elasticsearch:9200 \
  -e REDIS_HOST=redis \
  -e OPENAI_API_KEY=sk-... \
  -e JWT_SECRET=production-secret \
  --name olis-api \
  olis-api:latest
```

#### 3. Docker Compose

```yaml theme={null}
version: '3.8'
services:
  olis-api:
    build: .
    ports:
      - "8002:8002"
    environment:
      MILVUS_URI: http://milvus:19530
      ES_URI: http://elasticsearch:9200
      REDIS_HOST: redis
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      JWT_SECRET: ${JWT_SECRET}
    depends_on:
      - milvus
      - elasticsearch
      - redis

  milvus:
    image: milvusdb/milvus:latest
    ports:
      - "19530:19530"
    volumes:
      - milvus_data:/var/lib/milvus

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
    volumes:
      - es_data:/usr/share/elasticsearch/data

  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data

volumes:
  milvus_data:
  es_data:
  redis_data:
```

### Production Considerations

#### 1. Security

* **Never expose `/testpredict` in production** (bypasses auth)
* Use strong `JWT_SECRET` (32+ chars, random)
* Enable HTTPS/TLS for all connections
* Lock down CORS: `allow_origins=["https://yourdomain.com"]`

#### 2. Scaling

* **Horizontal Scaling**: Run multiple API instances behind load balancer
* **Stateless Design**: All state in Redis, safe to scale
* **Database Scaling**:
  * Milvus: Standalone → Cluster mode
  * Elasticsearch: Single node → Cluster
  * Redis: Single instance → Redis Cluster/Sentinel

#### 3. Monitoring

* **Health Checks**: `/healthz` endpoint for load balancer
* **Metrics**: Export performance logs to DataDog/Prometheus
* **Alerting**: Monitor `total_ms` > 5000ms, error rates

#### 4. Backup

* **Milvus**: Regular snapshots of `/var/lib/milvus`
* **Elasticsearch**: Snapshot and restore API
* **Redis**: RDB/AOF persistence enabled

***

## Troubleshooting

### Common Issues

#### 1. "JWT\_SECRET not configured"

**Symptom**: All API calls return 503

**Solution**:

```bash theme={null}
export JWT_SECRET=your-secret-key
# Restart server
```

#### 2. Milvus Connection Failed

**Symptom**: Server fails to start with Milvus error

**Check**:

```bash theme={null}
# Verify Milvus is running
curl http://localhost:19530/healthz

# Check logs
docker logs milvus-standalone
```

**Solution**:

```bash theme={null}
# Restart Milvus
docker-compose restart milvus

# Verify connection
export MILVUS_URI=http://localhost:19530
```

#### 3. Empty RAG Results

**Symptom**: All queries return empty answers

**Debug**:

```python theme={null}
# Check if documents exist
from pymilvus import connections, utility
connections.connect(uri="http://localhost:19530")
stats = utility.get_query_segment_info("docs_collection")
print(f"Docs in collection: {stats}")
```

**Solution**:

* Ingest documents via `/upsert`
* Check ACLs match user permissions
* Verify embeddings are generated correctly

#### 4. Slow Query Performance

**Symptom**: `total_ms` > 5000ms

**Profile**:

```bash theme={null}
export PERF_LOG=true
# Check which phase is slow:
# - retrieval_ms > 1000? Optimize Milvus/ES
# - llm_ms > 3000? Use smaller model or reduce context
# - context_ms > 500? Too many docs
```

**Optimize**:

```bash theme={null}
# Reduce retrieval size
RETRIEVER_TOP_K=3

# Disable query decomposition
USE_QUERY_DECOMPOSITION=false

# Use faster LLM
RAG_GENERATOR_MODEL=gpt-4o-mini
OLLAMA_NUM_PREDICT=300
```

#### 5. Redis Connection Errors

**Symptom**: "Error connecting to Redis"

**Check**:

```bash theme={null}
redis-cli -h localhost -p 6379 ping
# Expected: PONG
```

**Solution**:

```bash theme={null}
# Check Redis is running
docker ps | grep redis

# Verify connection params
echo $REDIS_HOST $REDIS_PORT

# Test connection
python -c "import redis; r=redis.Redis(host='localhost', port=6379); print(r.ping())"
```

***

## API Design Patterns

### Dependency Injection

FastAPI's dependency injection provides clean resource access:

```python theme={null}
@app.post("/predict")
async def predict(
    input: QueryInput,
    resources: AppResources = Depends(get_resources),
    current_user: Dict[str, Any] = Depends(get_current_user),
):
    # resources and current_user automatically injected
    pass
```

### Background Tasks

Expensive operations run asynchronously:

```python theme={null}
background_tasks.add_task(
    _update_memory_background,
    resources.session_memory,
    user_id,
    memory
)
# Response returns immediately, memory updated in background
```

### Lifespan Context Manager

Resources initialized once at startup, cleaned up at shutdown:

```python theme={null}
@asynccontextmanager
async def lifespan(app: FastAPI):
    # Startup: Initialize databases, LLM, etc.
    app.state.resources = AppResources(...)

    yield  # Server runs here

    # Shutdown: Close connections
    connections.disconnect("default")
```

***

## Advanced Topics

### Custom Prompts

Modify prompts in `backend/variables/prompts.py`:

```python theme={null}
AnswerWithKeywordsPrompt = """
You are Olis, an AI assistant that provides accurate answers based on provided context.

Context:
{context}

Question: {query}

Provide a JSON response with:
- callout_answer: Main answer (2-3 sentences)
- positive_list: Key points (list)
- negative_list: Caveats (list)
- info_list: Additional info (list)
- keywords: Relevant keywords (list)
"""
```

### Multi-Tenant Isolation

Each organization gets isolated resources:

```python theme={null}
# Org A: docs_collection_orgaid, docs_bm25_orgaid
# Org B: docs_collection_orgbid, docs_bm25_orgbid

# Hard isolation at database level
org_res = resources.org_manager.get(org_id)
retriever = org_res["retriever"]  # Only sees org's data
```

### Streaming Responses

For real-time LLM streaming:

```python theme={null}
from fastapi.responses import StreamingResponse

@app.post("/predict-stream")
async def predict_stream(input: QueryInput):
    async def generate():
        async for chunk in llm.astream(prompt):
            yield f"data: {chunk}\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")
```

***

## Additional Resources

* **FastAPI Docs**: [https://fastapi.tiangolo.com/](https://fastapi.tiangolo.com/)
* **Milvus Docs**: [https://milvus.io/docs](https://milvus.io/docs)
* **LangChain Docs**: [https://python.langchain.com/](https://python.langchain.com/)
* **OpenAI API**: [https://platform.openai.com/docs](https://platform.openai.com/docs)
* **Ollama**: [https://ollama.ai/](https://ollama.ai/)

***

## Support & Contact

For questions or issues:

* **GitHub Issues**: [olis-monorepo/issues](https://github.com/yourusername/olis-monorepo/issues)
* **Email**: [backend-team@olis.com](mailto:backend-team@olis.com)
* **Slack**: #olis-backend

***

**Last Updated**: 2026-02-10
**Version**: 1.0.0
**Author**: Olis Backend Team