Skip to content
Skjld Labs

RAG Architecture: How to Build Intelligent AI Systems with Your Own Data

Retrieval-Augmented Generation (RAG) lets you combine large language models with your own knowledge base. Learn the architecture behind modern enterprise AI solutions.

3 min read Laget med AI-hjelp
Teal Skjld Labs cover graphic with the text «RAG-arkitektur» (RAG architecture)

What is RAG and Why Do You Need It?

Retrieval-Augmented Generation (RAG) combines the power of large language models with your own knowledge base. Instead of relying solely on what the model learned during training, RAG systems retrieve relevant information from your documents and databases in real-time.

This solves several critical challenges:

  • Up-to-date information: LLMs have a knowledge cutoff. RAG provides access to current data.
  • Company-specific knowledge: Your products, processes, and guidelines.
  • Reduced hallucinations: Answers are based on actual documents, not the model's "imagination".
  • Source citations: Users can verify information against original documents.

Components of RAG Architecture

1. Document Processing

The first step is preparing your documents for search:

Chunking Strategies

Large documents must be split into smaller pieces (chunks). Common strategies:

  • Fixed-size chunking: Split the document into equal-sized pieces (e.g., 512 tokens)
  • Semantic chunking: Split based on meaningful boundaries (paragraphs, sections)
  • Sliding window: Overlapping chunks to preserve context
  • Recursive chunking: Start with large chunks, split as needed

Best practice: Experiment with chunk size. For most use cases, 256-1024 tokens work well, with 10-20% overlap.

Metadata Enrichment

Add metadata that improves search and filtering:

  • Document title and source
  • Creation date and last updated
  • Author and department
  • Document type and category
  • Access level and confidentiality

2. Embedding Models

Embeddings convert text to numerical vectors that capture semantic meaning:

Popular Embedding Models:

  • OpenAI text-embedding-3: High quality, easy to use
  • Cohere Embed: Good multilingual support
  • Voyage AI: Optimized for retrieval
  • Open-source (MTEB leaders): BGE, E5, GTE

Norwegian language: For best results in Norwegian, choose models trained on multilingual data. Always test on your own documents.

3. Vector Database

The vector database stores and searches embeddings efficiently:

Popular Options:

  • Pinecone: Fully managed, scalable
  • Weaviate: Open-source with hybrid search
  • Qdrant: High performance, good Rust implementation
  • Chroma: Lightweight, perfect for prototyping
  • pgvector: PostgreSQL extension for existing databases

Important Features to Consider:

  • Hybrid search (vector + keyword)
  • Metadata filtering
  • Scalability and performance
  • Hosting options (cloud vs. self-hosted)

4. Retrieval Strategies

How you retrieve relevant information dramatically affects quality:

Basic Similarity Search

Find chunks with highest cosine similarity to the query. Simple, but not always sufficient.

Hybrid Search

Combine vector search with traditional keyword search (BM25). Provides better results for specific terms and technical jargon.

Re-ranking

Use a more powerful model to re-rank the initial results. Cohere Rerank and similar services significantly improve precision.

Query Transformation

  • Query expansion: Expand the query with synonyms and related terms
  • HyDE: Generate a hypothetical answer, use it as search string
  • Multi-query: Generate multiple versions of the query, combine results

5. Generation with Context

Finally, send retrieved documents along with the user's question to the LLM:


System: You are a helpful assistant for Company Inc.
Answer only based on the information in the context below.
If you cannot find the answer, say so honestly.

Context:
{retrieved_documents}

User: {user_question}

Evaluation and Optimization

Retrieval Metrics

  • Recall@k: Proportion of relevant documents in top-k results
  • Precision@k: Proportion of top-k results that are relevant
  • MRR: Mean Reciprocal Rank: how quickly is the first relevant document found

End-to-End Evaluation

  • Answer quality: Are the answers correct and complete?
  • Groundedness: Are the answers actually based on retrieved documents?
  • Relevance: Does the system answer what the user actually asked?

Common Pitfalls

Chunks Too Large or Small

Too large chunks include irrelevant information. Too small loses context. Test systematically.

Ignoring Metadata

Metadata filtering can dramatically improve relevance. Filter by document type, date, department.

No Re-ranking

Embedding search alone often gives suboptimal results. Re-ranking is often the simplest improvement.

Poor Prompt Engineering

Clearly instruct the model to use the context and acknowledge uncertainty.

Conclusion

RAG architecture is the key to building AI systems that actually work in enterprise context. By combining LLMs' language understanding with your own knowledge base, you get the best of both worlds.

Skjld Labs has extensive experience designing and implementing RAG solutions for Norwegian businesses. Contact us to discuss how we can help you.

From article to practice

Want to put this to work?

AI agents for businesses. We build agents that take on recurring tasks in email, customer service, sales and finance, within limits we agree with you.

Read about AI agents for businesses Book a free call
Agents and automation

Read on.

Newsletter

AI in practice, one email a month.

What we've tested, one tip you can use, and what's coming next.

You'll get the Skjld Labs newsletter once a month, and can unsubscribe at any time. Privacy

First step

Thirty minutes. Free.

A no-obligation conversation about what you are facing. We will say so if AI is not the right answer. We are happy to meet you at Langes gate 8 in Sandefjord, or on video.