Rag Accuracy Optimizer

---

name: rag-accuracy-optimizer description: > Optimize accuracy for RAG (Retrieval-Augmented Generation) systems. Covers: DB schema design, chunking strategies, retrieval optimization, accuracy testing, and anti-hallucination safeguards. Use when: (1) designing or improving a RAG pipeline, (2) choosing the right chunking strategy, (3) optimizing retrieval accuracy (hybrid search, reranking, multi-query), (4) evaluating chunk quality or testing accuracy, (5) setting up monitoring & safeguards for RAG production, (6) choosing SQL vs Vector DB, (7) designing metadata schemas for domain-specific data (insurance, finance, healthcare, e-commerce).

---

RAG Accuracy Optimizer

A skill for optimizing end-to-end accuracy in RAG systems.

Workflow Overview

Data Design → Chunking → Indexing → Retrieval → Generation → Testing → Monitoring

Each step impacts accuracy. Optimize each step in order.

---

1. Structured Data Design

SQL vs Vector DB — When to Use What?

| Criteria | SQL (PostgreSQL, MySQL) | Vector DB (Pinecone, Qdrant, Weaviate) | |---|---|---| | Exact facts (price, date, product code) | ✅ Optimal | ❌ Not suitable | | Semantic search (query meaning) | ❌ Not supported | ✅ Optimal | | Aggregation (SUM, COUNT, AVG) | ✅ Native | ❌ Not supported | | Fuzzy matching ("similar to...") | ⚠️ Limited | ✅ Optimal | | Hybrid (recommended) | pgvector for both | Vector DB + SQL metadata store |

Principle: Clearly structured data → SQL. Unstructured data requiring semantic understanding → Vector DB. Most production systems need both.

Schema Design Patterns by Domain

Insurance:

policies(policy_id, product_type, effective_date)
clauses(clause_id, policy_id, clause_number, title, content)
exclusions(exclusion_id, clause_id, description)
-- Vector: embedding for clause.content + exclusion.description

Finance:

securities(ticker, name, sector, exchange)
reports(report_id, ticker, period, report_type)
sections(section_id, report_id, heading, content)
-- Vector: embedding for section.content, metadata: ticker + period

Healthcare:

drugs(drug_id, generic_name, brand_name, category)
guidelines(guideline_id, condition, recommendation, evidence_level)
interactions(drug_a_id, drug_b_id, severity, description)
-- Vector: embedding for guidelines.recommendation

E-commerce:

products(product_id, name, category, brand, price)
reviews(review_id, product_id, rating, content)
specs(product_id, attribute, value)
-- Vector: embedding for review.content + product description

Metadata Tagging Strategy

Each chunk/document needs at minimum:

metadata = {
    "source": "policy_doc_v2.pdf",       # Origin
    "source_type": "pdf",                 # File type
    "domain": "insurance",                # Domain
    "category": "life_insurance",          # Classification
    "entity_id": "POL-2024-001",          # Related entity ID
    "section": "exclusions",              # Section in doc
    "chunk_index": 3,                      # Chunk position
    "total_chunks": 12,                    # Total chunks in doc
    "created_at": "2024-01-15",           # Creation date
    "version": "2.0",                      # Version
    "language": "en"                       # Language
}

Metadata principles:

Always include source for traceability and citation

entity_id enables pre-filtering before search → reduces noise

chunk_index + total_chunks enables fetching surrounding context

Domain-specific fields (clause_number, ticker, drug_id) vary by use case

Normalization vs Denormalization

| | Normalized | Denormalized | |---|---|---| | Pros | Less duplication, easy to update | Faster queries, fewer JOINs | | Cons | Requires JOINs, slower | Duplication, harder to sync | | Use when | Source of truth (SQL) | Vector store chunks |

Recommendation: Normalized for SQL source → Denormalized when creating chunks for Vector DB. Each chunk should contain sufficient context, no JOINs needed at retrieval time.

---

2. Chunking Strategies

> Detailed code examples: read references/chunking-patterns.md

Choosing the Right Strategy

Data has clear structure (clauses, sections)?
  → Semantic chunking (by heading/section)
Long, continuous data (articles, transcripts)?
  → Fixed size + overlap (512 tokens, 10-20% overlap)
Need both overview + detail?
  → Hierarchical chunking (parent-child)
Domain-specific with its own logical units?
  → Domain-specific chunking

Chunk Size Guidelines

| Size | Use case | Trade-off | |---|---|---| | 128-256 tokens | FAQ, short definitions | High precision, less context | | 256-512 tokens | Recommended default | Good balance | | 512-1024 tokens | Complex text, legal docs | More context, potential noise | | >1024 tokens | Rarely used | Too much noise |

Semantic Chunking

Split by meaning (section, topic) instead of fixed size:

# Split by markdown headings
Split by paragraph breaks (\n\n)
Split by topic change (using NLP or LLM detection)

Overlap Strategy

10-20% overlap between adjacent chunks

Ensures information at boundaries is not lost

Chunk N ends with 1-2 opening sentences of chunk N+1

Hierarchical Chunking (Parent-Child)

Document (summary)
  └── Section (heading + key points)
        └── Paragraph (details)

Search at paragraph level (most detailed)

When matched, pull parent section for additional context

Keep parent_id in metadata

Domain-Specific Chunking

Insurance: 1 chunk = 1 clause

Finance: 1 chunk = 1 report section, metadata = ticker + period

Healthcare: 1 chunk = 1 guideline/recommendation

E-commerce: 1 chunk = 1 review or 1 product description

Legal: 1 chunk = 1 article/clause/section

Metadata Enrichment Per Chunk

Each chunk should be enriched with:

Summary: 1-2 sentence content summary (LLM-generated)

Keywords: Key terms (supports BM25)

Questions: 2-3 questions this chunk can answer (hypothetical questions)

Entities: Named entities (product names, codes, dates)

---

3. Retrieval Optimization

> Detailed code examples: read references/retrieval-patterns.md

Recommended Retrieval Pipeline

User Query
  → Query Rewriting (expand/reformulate)
  → Multi-Query Generation (3-5 variants)
  → Metadata Filtering (narrow scope)
  → Hybrid Search (Vector + BM25)
  → Merge & Deduplicate
  → Reranking (top 20 → top 5)
  → Contextual Compression
  → LLM Generation (with citations)

Hybrid Search (Vector + BM25)

Vector search: Find by meaning (semantic similarity)

BM25 (keyword): Find by exact keywords (product names, codes)

Combined: Weighted fusion or Reciprocal Rank Fusion (RRF)

final_score = α × vector_score + (1-α) × bm25_score
α = 0.7 is a good starting point, tune per domain

Query Rewriting

Use LLM to reformulate the user question for clarity:

User: "does insurance pay?"
→ Rewritten: "Under what circumstances does life insurance pay out benefits?"

Multi-Query

From 1 question, generate 3-5 variants → search each variant → merge results:

Original: "Which bank has the highest savings rate?"
Query 1: "Compare savings interest rates across banks 2024"
Query 2: "Bank with highest deposit rate currently"
Query 3: "Top banks with best deposit interest rates"

Reranking

After retrieval, use a reranking model to re-sort by relevance:

Cohere Rerank: Simple API, highly effective

Cross-encoder: More accurate than bi-encoder, but slower

GPT Rerank: Use LLM to evaluate relevance (expensive but flexible)

Retrieve top 20 → rerank → take top 3-5 for generation.

Contextual Compression

After reranking, compress each chunk: keep only the part relevant to the question.

Original chunk (500 tokens) → Compressed (150 tokens, relevant part only)

Reduces noise, saves context window, improves accuracy.

Metadata Filtering

Narrow the search space BEFORE vector search:

# Instead of searching all 1M chunks:
filter = {"domain": "insurance", "product_type": "life"}
Only search within ~50K relevant chunks
results = vector_db.search(query, filter=filter, top_k=20)

---

4. Accuracy Testing & Monitoring

Test Suite Design

Create ground truth Q&A pairs:

{
    "test_cases": [
        {
            "question": "Does life insurance pay out for suicide?",
            "expected_answer": "No payout within the first 2 years",
            "expected_source": "clause_15_exclusions.pdf",
            "category": "exclusions",
            "difficulty": "medium"
        }
    ]
}

Recommendation: Minimum 50-100 test cases, evenly distributed across categories and difficulty levels.

Metrics

| Metric | Meaning | Target | |---|---|---| | Precision@K | % relevant results in top K | >0.8 | | Recall@K | % ground truth found in top K | >0.9 | | F1 | Harmonic mean of Precision and Recall | >0.85 | | MRR | Mean Reciprocal Rank — average position of first correct result | >0.8 | | NDCG | Normalized Discounted Cumulative Gain — ranking quality | >0.85 | | Answer Accuracy | % correct answers (human eval or LLM judge) | >0.9 |

A/B Testing

Compare strategies by running the same test suite:

Config A: chunk_size=256, overlap=10%, no_rerank
Config B: chunk_size=512, overlap=20%, cohere_rerank
→ Compare MRR, NDCG, Answer Accuracy
→ Choose the config with better metrics

Error Analysis Framework

Classify errors to know where to optimize:

| Error Type | Cause | Solution | |---|---|---| | Retrieval Miss | Correct chunk not found | Improve chunking, add hypothetical Q | | Ranking Error | Correct chunk found but ranked low | Add reranking | | Generation Error | Correct chunk but LLM answers wrong | Improve prompt, add few-shot | | No Answer | Information not in DB | Expand knowledge base | | Hallucination | LLM fabricates information | Add citation enforcement |

Production Monitoring

Log each query:

log_entry = {
    "timestamp": "2024-01-15T10:30:00",
    "query": "...",
    "retrieved_chunks": [...],
    "reranked_chunks": [...],
    "answer": "...",
    "confidence": 0.85,
    "latency_ms": 450,
    "user_feedback": None  # thumbs up/down
}

Alerts:

Continuous confidence < 0.5 → review chunking/retrieval

Latency > 2s → optimize index or reduce top_k

Negative feedback > 20% → audit error patterns

---

5. Safeguards

Hallucination Prevention

Mandatory system prompt:

Answer ONLY based on the information provided in the context.
If you cannot find the information, respond: "I could not find this
information in the available data."
NEVER fabricate information.

Citation Enforcement

Require source citations:

Every answer must include [Source: file_name, section/clause].
If a specific source cannot be cited, mark it as "unverified".

Confidence Thresholds

if max_relevance_score < 0.3:
    return "No relevant information found."
elif max_relevance_score < 0.6:
    return answer + "\n⚠️ Low confidence. Please verify."
else:
    return answer + f"\n📎 Source: {sources}"

Answer Verification

Cross-check the answer with the DB:

1. Extract claims from the answer (using LLM) 2. Verify each claim against retrieved chunks 3. Flag claims without supporting evidence 4. Return only verified claims

---

6. Embedding Model Selection

> Detailed comparison: read references/embedding-models.md

Quick Decision

| Scenario | Model | Reason | |---|---|---| | Production, budget OK | Cohere embed-v4 | Highest MTEB, input_type optimization | | Production, low cost | OpenAI text-embedding-3-small | $0.02/1M tokens, good quality | | Self-host, multilingual | BGE-M3 ⭐ | Hybrid dense+sparse, 100+ languages, free | | Self-host, Vietnamese | BGE-M3 or multilingual-e5-large | Best for Vietnamese RAG | | POC / Prototype | all-MiniLM-L6-v2 | 90MB, runs on CPU |

Key Principles

Dimension reduction: OpenAI embed-3 supports Matryoshka — reduce 3072→512 with only ~3% quality loss

Normalize embeddings: Always normalize_embeddings=True when encoding for cosine similarity

Batch processing: Encode in batches (256-2000 items) instead of one at a time

Consistency: Use the SAME model for indexing and querying

---

7. Vector DB Comparison

> Detailed comparison + HNSW tuning: read references/vector-db-comparison.md

Quick Decision

Already have PostgreSQL and <5M vectors? → pgvector
Just prototype/POC? → ChromaDB
Production, want zero-ops? → Pinecone
Need performance + HNSW control? → Qdrant
Need hybrid BM25+vector built-in? → Weaviate

HNSW Tuning Quick Reference

| Param | Default | Accuracy-critical | Speed-critical | |---|---|---|---| | M | 16 | 48-64 | 8-16 | | ef_construction | 200 | 400-500 | 100-200 | | ef (search) | 100 | 200-256 | 50-100 |

Trade-off: Higher M and ef → better recall but more RAM and slower. Tune per SLA.

---

8. Advanced Techniques

> Detailed code examples: read references/advanced-rag.md

Late Chunking

Embed the entire document first, then pool embeddings by chunk boundaries. Each chunk retains context from surrounding text.

Traditional: Doc → Chunk → Embed each (loses context)
Late Chunking: Doc → Embed full → Pool by boundaries (retains context)

Use when: Documents have many co-references ("it", "this", "the package"). Quality gain: +5-10%.

RAPTOR (Recursive Abstractive Processing)

Build a multi-level summary tree: Level 0 (chunks) → Level 1 (summaries) → Level 2 (summary of summaries).

Use when: Need to answer both broad queries ("Compare all insurance packages") and narrow queries ("Clause X of Package Y"). Quality gain: +10-15%.

GraphRAG (Microsoft)

Build a knowledge graph from documents → detect communities → summarize communities → query via map-reduce.

Use when: Multi-hop reasoning, synthesize across many documents. Quality gain: +15-25% for synthesis queries. High overhead (many LLM calls when building the graph).

Combining Techniques (Production Stack)

1. Late Chunking → better embeddings
2. Hybrid Search (BM25 + vector) → high recall
3. Reranking (Cohere/Cross-encoder) → high precision
4. RAPTOR → multi-level retrieval (optional)
5. GraphRAG → synthesis queries (optional, high cost)

---

9. Performance Optimization

Caching Layer

# Cache embeddings (avoid re-computation)
import hashlib, json, redis
r = redis.Redis()
def cached_embed(text, model):
    key = f"emb:{hashlib.md5(text.encode()).hexdigest()}"
    cached = r.get(key)
    if cached:
        return json.loads(cached)
    embedding = model.encode([text])[0].tolist()
    r.setex(key, 3600, json.dumps(embedding))  # TTL 1h
    return embedding
Cache search results (avoid re-searching)
def cached_search(query, search_fn, ttl=300):
    key = f"search:{hashlib.md5(query.encode()).hexdigest()}"
    cached = r.get(key)
    if cached:
        return json.loads(cached)
    results = search_fn(query)
    r.setex(key, ttl, json.dumps(results))
    return results

Async Retrieval

import asyncio
async def parallel_retrieve(query, retrievers):
    """Run multiple retrievers in parallel."""
    tasks = [r.search(query) for r in retrievers]
    results = await asyncio.gather(*tasks)
    return merge_and_deduplicate(results)

HNSW Index Tuning

See details in references/vector-db-comparison.md HNSW section. Key: tune ef (search) per latency SLA, tune M per recall target.

---

10. Vietnamese-Specific RAG

> Details: read references/vietnam-nlp.md

Key Challenges

| Issue | Solution | |---|---| | Diacritics (with vs without) | Dual indexing: index both versions | | Compound words ("bảo hiểm") | Word segmentation (underthesea) | | Abbreviations (BHXH, TTCK, BLLĐ) | Abbreviation expansion dictionary | | Vietnamese proper names | NER with underthesea/PhoBERT | | Domain terms (finance, law, medical) | Domain-specific term enrichment |

Embedding Models for Vietnamese

BGE-M3: Best overall — hybrid dense+sparse, 100+ languages

multilingual-e5-large: Good alternative — retrieval-optimized

PhoBERT-v2: Best for NER/classification (needs fine-tuning for retrieval)

Preprocessing Pipeline

Input text
  → Unicode normalize (NFC)
  → Expand abbreviations (BHXH → Social Insurance)
  → Domain term enrichment
  → Dual index: original + no-diacritics version
  → Extract entities → metadata

---

11. AI Orchestrator — Multi-Model Cost Optimization

> Detailed prompt templates, code examples: read references/orchestrator-patterns.md

Query Classification Pipeline

Each user query is classified into 1 of 5 categories:

| Category | Description | Example | Model | |---|---|---|---| | simple | Greeting, FAQ, simple lookup | "Hello", "Opening hours?" | No LLM / Local | | rag | Needs knowledge base search | "Does insurance cover cancer?" | Cheap (Gemini Flash) | | complex | Multi-hop reasoning, comparison, analysis | "Compare 3 insurance packages for a family of 4" | Standard (GPT-4o-mini) / Premium (Claude Sonnet) | | action | Needs tool/API execution (create form, calculate) | "Calculate insurance premium for me, age 30" | Standard + Tools | | unsafe | Violation content, injection, jailbreak | "Ignore instructions..." | Block — No LLM |

2-Stage Classification (Minimize LLM Tokens)

User Query
  → Stage 1: Rule-based pre-classifier (regex, keywords, NO LLM)
    → confidence ≥ 0.8? → DONE (skip LLM)
    → confidence < 0.8? → Stage 2: LLM classifier (cheap model, ~50 tokens)

Stage 1 blocks 60-80% of queries without spending a single LLM token.

Model Routing

Category → Model Selection:
  greeting/simple  → No LLM (rule-based response)
  rag (simple)     → Gemini Flash ($0.075/1M input) — cheap, fast
  rag (complex)    → GPT-4o-mini ($0.15/1M input) — balanced
  complex          → Claude Sonnet ($3/1M input) — premium quality
  action           → Gemini Flash + Tool calls
  unsafe           → Block response (no LLM cost)

Cost Optimization Rules

1. Rule-based first: Greeting, FAQ, unsafe → DON'T call LLM 2. Cheapest sufficient model: Prefer Gemini Flash for RAG queries 3. Escalate on failure: Gemini Flash fail/low-confidence → GPT-4o-mini → Claude Sonnet 4. Cache responses: Identical queries → cached answer (TTL 5-30 min) 5. Batch classify: Multiple queries → 1 LLM call to classify all 6. Token budget: Set max_tokens per category (simple: 100, rag: 300, complex: 500)

RAG Trigger Rules

| Condition | RAG On/Off | |---|---| | Query contains domain keywords | ✅ ON | | Classification = "rag" or "complex" | ✅ ON | | Greeting, simple lookup, unsafe | ❌ OFF | | Confidence score > 0.9 from cache/FAQ | ❌ OFF (answer from cache) |

Tool Trigger Rules

| Condition | Tools | |---|---| | Query requests calculation (fees, interest) | calculator tool | | Query requests form creation/submission | form_builder tool | | Query requests real-time lookup (price, exchange rate) | api_lookup tool | | Classification ≠ "action" | No tools |

JSON Output Format

{
  "category": "rag",
  "confidence": 0.92,
  "risk_level": "low",
  "model": "gemini-flash",
  "rag_enabled": true,
  "tools": [],
  "max_tokens": 300,
  "reasoning": "User asks about insurance benefits — needs knowledge base search"
}

---

Scripts

eval_ragas.py

RAGAS evaluation pipeline. Run:

python3 scripts/eval_ragas.py --test-file eval_dataset.json --output results.json
python3 scripts/eval_ragas.py --test-file eval_dataset.json --metrics faithfulness,answer_relevancy

Input: JSON file with test cases (question, answer, contexts, ground_truth). Output: metrics report + threshold checks. Requires: pip install ragas langchain-openai datasets

embedding_benchmark.py

Benchmark embedding models on a Vietnamese dataset. Run:

python3 scripts/embedding_benchmark.py --models bge-m3,multilingual-e5 --dataset vi_pairs.json
python3 scripts/embedding_benchmark.py --models all --quick  # Use built-in test pairs

Input: JSON file with query-positive-negative pairs. Output: accuracy + latency comparison. Requires: pip install sentence-transformers numpy torch

chunk_optimizer.py

Evaluate chunk quality. Run:

python3 scripts/chunk_optimizer.py --input chunks.jsonl --output report.json

Input: JSONL file, each line is {"text": "...", "metadata": {...}}. Output: quality report with scores.

accuracy_test.py

Test framework for RAG accuracy. Run:

python3 scripts/accuracy_test.py --test-file tests.json --results-dir ./results

Input: JSON file with test cases (question, expected_answer, expected_source). Output: metrics report.

---

References

references/chunking-patterns.md — Python code examples for chunking strategies

references/retrieval-patterns.md — Code examples for hybrid search, reranking, multi-query

references/embedding-models.md — Detailed embedding model comparison (OpenAI, Cohere, BGE-M3, PhoBERT...)

references/vector-db-comparison.md — Vector DB comparison + HNSW tuning guide

references/advanced-rag.md — Late Chunking, RAPTOR, GraphRAG with code examples

references/testing-frameworks.md — RAGAS, LLM-as-Judge, Adversarial testing

references/vietnam-nlp.md — Vietnamese NLP: diacritics, abbreviations, NER, domain terms

references/orchestrator-patterns.md — Multi-model orchestrator: prompt templates, rule-based pre-classifier, cost comparison, fallback chain, monitoring