RAG and its Practical Implementation with LangChain, Bun, Ollama and Qdrant

Modern Large Language Models (LLMs) are impressive, but they have a major limitation: their knowledge is fixed in their weights, making it difficult to update and extend their knowledge. Retrieval-Augmented Generation (RAG) is an approach designed to address this problem. Introduced by Meta in 2020, it connects a language model to an external knowledge base (for example, a set of documents) so that it can incorporate up-to-date and specific information into its responses. In practice, for each question asked, the RAG system first extracts relevant content from its document base, then generates a response by combining this retrieved context with the linguistic capabilities of the LLM.

Note: The complete source code for the example project mentioned in this article is available on GitHub.

Article Outline

1. What is RAG and why use it?

   • Operating principle
   • Advantages over classical approaches
   • Concrete use cases

2. Architecture of a RAG system

   • Essential components
   • Data flow
   • Technology choices

3. Practical implementation with TypeScript

   • Project setup with Bun
   • LangChain integration
   • Ollama and Qdrant configuration

4. Code analysis and best practices

   • Document indexing
   • Semantic search
   • Response generation

5. Advantages of the technical stack

   • Bun performance vs Node.js
   • LangChain simplicity
   • Ollama flexibility
   • Qdrant scalability

6. Going further

   • Advanced optimizations
   • Evaluation and metrics
   • Technological alternatives

What is RAG and why use it?

Retrieval-Augmented Generation (RAG) literally means "generation augmented by retrieval." The idea is to separate knowledge from the model. Instead of trying to incorporate all information into the parameters of an LLM (through costly fine-tuning) or designing a classical model that would predict responses from data, we let the main model generate text and augment it with an intermediate step of information retrieval. A typical RAG pipeline works as follows:

1. User query – The user asks a question or provides a query in natural language (e.g., "What is class X used for in this project?").
2. Search for relevant documents – The system transforms this question into a vector representation (embedding) and then queries a vector database to retrieve documents or passages that are semantically most similar to the query. This identifies the relevant context (e.g., an excerpt from documentation, code, or an article corresponding to the question).
3. Context + question combination – The retrieved documents or excerpts are then provided as context to the language model. In practice, they are inserted into the LLM's prompt, typically via a system message or by prefixing the user's question with the text of the found documents.
4. Response generation – The language model (LLM) then generates a response based on both the question and the provided context. The response should contain information from the documents, formulated coherently thanks to the LLM's capabilities.

This process allows the model to rely on specific external knowledge at the time of generation, without having to permanently memorize it. This can be compared to a human who, faced with a question, would consult books or reference documents before answering: the LLM "searches its library" before speaking.

Concrete use cases for RAG

The RAG approach is particularly useful whenever a conversational assistant needs to handle an evolving or voluminous knowledge base. Here are some examples of concrete use cases where RAG excels compared to classical methods:

Documentary chatbots: An assistant powered by a company's technical documentation, capable of answering questions from developers or customers by drawing directly from manuals, internal knowledge bases, or even source code. For example, the model can be connected to API specifications or open-source project code to explain how a function works or the reason for a certain design.

Dynamic FAQs: In a customer support context, a RAG chatbot can answer common questions (FAQs) based on the latest policies or product data. If a policy (e.g., return conditions) changes, you only need to update the reference document and the bot will take it into account instantly, without requiring retraining. This results in always up-to-date FAQs, with the ability to provide the source of information to support the answer.

Legal assistants: An assistant can help lawyers or legal professionals by finding relevant passages in a database of laws, case law, or contracts for a given question, then formulating the answer in natural language. The model doesn't need to know the entire Civil Code by heart; it just needs to look up the appropriate articles. The same applies to a medical assistant, which could query databases of scientific publications or medical protocols to provide answers based on the latest clinical knowledge.

Programming assistant: This is the case of our example project – an assistant that knows the content of a code repository and can answer questions about this code (architecture, role of a module, potential bugs, etc.). Rather than training a specialized programming model, we use a generalist LLM augmented by searching for relevant code files in the repository.

Architecture of a RAG system

Essential components

A complete RAG system typically includes the following components:

1. Indexing and storage

   • Document processor (extraction, cleaning, chunking)
   • Embedding generator (transformation into vectors)
   • Vector database (storage and search)

2. Query pipeline

   • Query preprocessor
   • Semantic search engine
   • Prompt generator

3. Generation and post-processing

   • LLM interface
   • Response evaluator
   • Output formatter

Data flow

typescript

Technology choices

For our implementation, we've chosen a modern and performant stack:

• Bun: Ultra-fast JavaScript runtime, ideal for server applications
• TypeScript: Static typing for better maintainability
• LangChain: Framework for building LLM-based applications
• Ollama: Tool for running language models locally
• Qdrant: Performant and easy-to-deploy vector database

This combination offers an excellent balance between performance, ease of development, and flexibility.

Practical implementation with TypeScript

Project setup with Bun

Let's start by initializing our project:

bash

Basic configuration

typescript

Document indexing

Indexing is a crucial step in a RAG system. It involves transforming raw documents into appropriately sized chunks, then generating embeddings for each chunk.

typescript

Search and response generation

typescript

Simple user interface

typescript

Code analysis and best practices

Efficient chunking

Splitting documents into chunks is a critical step that directly influences the quality of results. Some best practices:

• Appropriate size: Chunks should be large enough to contain context, but not too large to remain relevant (typically between 500 and 1500 characters).
• Overlap: Overlap between chunks prevents losing context at boundaries.
• Semantic splitting: Ideally, splitting should respect the semantic structure of documents (paragraphs, functions, etc.).

Search optimization

The quality of semantic search is essential:

• Metadata filters: Use metadata (file type, date, author) to refine searches.
• Re-ranking: Apply a second level of filtering to improve relevance.
• Diversity: Ensure diversity in results to cover different aspects of the question.

Advanced prompting

Prompt construction is an art that strongly influences the quality of responses:

typescript

Advantages of the technical stack

Bun performance vs Node.js

Bun offers significant advantages for this type of application:

• Fast startup: Startup time up to 4x faster than Node.js
• Optimized execution: Superior execution performance, particularly for I/O operations
• Integrated bundler: Simplification of the development workflow

LangChain simplicity

LangChain greatly facilitates the development of LLM-based applications:

• Abstraction: Unified interface for different models and providers
• Reusable components: Ready-to-use chains, agents, and tools
• Established patterns: Reference implementations for common use cases

Ollama flexibility

Ollama allows running language models locally with great flexibility:

• Local models: No dependency on external APIs
• Privacy: Data remains on your infrastructure
• Customization: Possibility to adjust models according to your needs

Qdrant scalability

Qdrant is a modern vector database designed for semantic search:

• Performance: Optimized for fast similarity searches
• Filtering: Advanced filtering capabilities on metadata
• Flexible deployment: Usable in embedded mode or as a service

Going further

Advanced optimizations

• Hybrid search: Combine vector search and keyword search
• Hierarchical chunking: Use different levels of granularity for chunks
• Caching: Cache search results and frequent responses

Evaluation and metrics

To measure the quality of a RAG system:

• Relevance: Are the retrieved documents relevant to the question?
• Faithfulness: Is the answer faithful to the source documents?
• Usefulness: Does the answer effectively address the user's question?

Technological alternatives

• Frameworks: Haystack, LlamaIndex as alternatives to LangChain
• Vector databases: Pinecone, Weaviate, Milvus as alternatives to Qdrant
• Models: Different local models (Llama, Mistral) or APIs (OpenAI, Anthropic)

Conclusion

Retrieval-Augmented Generation represents a major advance in how we can leverage language models for specific use cases. By separating knowledge from the generation model, RAG enables the creation of AI assistants that are more accurate, more up-to-date, and more transparent.

Our implementation with TypeScript, Bun, LangChain, Ollama, and Qdrant demonstrates that it is now possible to build performant RAG systems with modern and accessible technologies. This approach paves the way for a new generation of AI assistants capable of reasoning on specific knowledge bases while maintaining the fluidity and coherence of large language models.

Feel free to explore the complete source code on GitHub and adapt it to your own use cases. RAG is an evolving technology, and there are numerous opportunities for innovation in this exciting field.

1// Simplified example of data flow in a RAG system
2async function processQuery(query: string): Promise<string> {
3  // 1. Transform the query into an embedding
4  const queryEmbedding = await embedder.embedQuery(query);
5  
6  // 2. Search for relevant documents
7  const relevantDocs = await vectorStore.similaritySearch(queryEmbedding);
8  
9  // 3. Build the prompt with context
10  const prompt = buildPromptWithContext(query, relevantDocs);
11  
12  // 4. Generate the response with the LLM
13  const response = await llm.generate(prompt);
14  
15  return response;
16}

1// src/config.ts
2export const CONFIG = {
3  // Ollama configuration
4  OLLAMA_BASE_URL: 'http://localhost:11434',
5  OLLAMA_MODEL_NAME: 'llama3',
6  
7  // Qdrant configuration
8  QDRANT_URL: 'http://localhost:6333',
9  QDRANT_COLLECTION_NAME: 'code_repository',
10  
11  // Indexing configuration
12  CHUNK_SIZE: 1000,
13  CHUNK_OVERLAP: 200,
14  
15  // Path to the code repository to index
16  REPOSITORY_PATH: './repository',
17};

1// src/indexer.ts
2import { DirectoryLoader } from 'langchain/document_loaders/fs/directory';
3import { RecursiveCharacterTextSplitter } from 'langchain/text_splitter';
4import { OllamaEmbeddings } from '@langchain/community/embeddings/ollama';
5import { QdrantClient } from 'qdrant-js';
6import { CONFIG } from './config';
7
8export async function indexRepository() {
9  // 1. Loading repository files
10  const loader = new DirectoryLoader(
11    CONFIG.REPOSITORY_PATH,
12    {
13      '.ts': (path) => new TextLoader(path),
14      '.js': (path) => new TextLoader(path),
15      '.tsx': (path) => new TextLoader(path),
16      '.jsx': (path) => new TextLoader(path),
17      '.md': (path) => new TextLoader(path),
18    }
19  );
20  
21  const docs = await loader.load();
22  console.log(`Loading of ${docs.length} files completed`);
23  
24  // 2. Splitting into chunks
25  const textSplitter = new RecursiveCharacterTextSplitter({
26    chunkSize: CONFIG.CHUNK_SIZE,
27    chunkOverlap: CONFIG.CHUNK_OVERLAP,
28  });
29  
30  const chunks = await textSplitter.splitDocuments(docs);
31  console.log(`Creation of ${chunks.length} chunks completed`);
32  
33  // 3. Generating embeddings
34  const embeddings = new OllamaEmbeddings({
35    baseUrl: CONFIG.OLLAMA_BASE_URL,
36    model: CONFIG.OLLAMA_MODEL_NAME,
37  });
38  
39  // 4. Storing in Qdrant
40  const qdrant = new QdrantClient({ url: CONFIG.QDRANT_URL });
41  
42  // Creating collection if it doesn't exist
43  try {
44    await qdrant.createCollection(CONFIG.QDRANT_COLLECTION_NAME, {
45      vectors: {
46        size: 1536, // Size of embedding vectors
47        distance: 'Cosine',
48      },
49    });
50  } catch (e) {
51    console.log('Collection already exists or error:', e.message);
52  }
53  
54  // Inserting documents with their embeddings
55  for (let i = 0; i < chunks.length; i += 100) {
56    const batch = chunks.slice(i, i + 100);
57    const batchEmbeddings = await embeddings.embedDocuments(
58      batch.map(chunk => chunk.pageContent)
59    );
60    
61    await qdrant.upsert(CONFIG.QDRANT_COLLECTION_NAME, {
62      points: batchEmbeddings.map((vector, idx) => ({
63        id: `${i + idx}`,
64        vector,
65        payload: {
66          text: batch[idx].pageContent,
67          metadata: batch[idx].metadata,
68        },
69      })),
70    });
71    
72    console.log(`Processing batch ${i}-${i + batch.length} completed`);
73  }
74  
75  console.log('Indexing successfully completed!');
76}

1// src/rag.ts
2import { OllamaEmbeddings } from '@langchain/community/embeddings/ollama';
3import { QdrantClient } from 'qdrant-js';
4import { Ollama } from '@langchain/community/llms/ollama';
5import { CONFIG } from './config';
6
7export async function generateResponse(query: string): Promise<string> {
8  // 1. Initializing components
9  const embeddings = new OllamaEmbeddings({
10    baseUrl: CONFIG.OLLAMA_BASE_URL,
11    model: CONFIG.OLLAMA_MODEL_NAME,
12  });
13  
14  const qdrant = new QdrantClient({ url: CONFIG.QDRANT_URL });
15  
16  const llm = new Ollama({
17    baseUrl: CONFIG.OLLAMA_BASE_URL,
18    model: CONFIG.OLLAMA_MODEL_NAME,
19  });
20  
21  // 2. Generating query embedding
22  const queryEmbedding = await embeddings.embedQuery(query);
23  
24  // 3. Searching for relevant documents
25  const searchResults = await qdrant.search(CONFIG.QDRANT_COLLECTION_NAME, {
26    vector: queryEmbedding,
27    limit: 5,
28  });
29  
30  const relevantDocs = searchResults.map(result => result.payload.text).join('\n\n');
31  
32  // 4. Building prompt with context
33  const prompt = `
34  You are an expert programming assistant who helps understand a project's source code.
35  Use the following context to answer the user's question.
36  If you can't find the information in the context, clearly say so.
37  
38  CONTEXT:
39  ${relevantDocs}
40  
41  QUESTION: ${query}
42  
43  ANSWER:
44  `;
45  
46  // 5. Generating the response
47  const response = await llm.call(prompt);
48  
49  return response;
50}

1// src/index.ts
2import { indexRepository } from './indexer';
3import { generateResponse } from './rag';
4
5async function main() {
6  // Argument for indexing or querying
7  const args = process.argv.slice(2);
8  
9  if (args[0] === 'index') {
10    console.log('Starting indexing...');
11    await indexRepository();
12  } else if (args[0] === 'query') {
13    const query = args.slice(1).join(' ');
14    if (!query) {
15      console.log('Usage: bun run src/index.ts query "Your question about the code"');
16      process.exit(1);
17    }
18    
19    console.log(`Question: ${query}`);
20    console.log('Generating response...');
21    const response = await generateResponse(query);
22    console.log('\nAnswer:');
23    console.log(response);
24  } else {
25    console.log('Usage: bun run src/index.ts [index|query "Your question"]');
26  }
27}
28
29main().catch(console.error);

1// Example of a more advanced prompt
2const prompt = `
3# ROLE
4You are an expert programming assistant specialized in source code analysis.
5
6# CONTEXT
7${relevantDocs}
8
9# INSTRUCTIONS
10- Answer the user's question based only on the provided context
11- If the context doesn't contain the necessary information, clearly indicate this
12- Cite relevant code excerpts in your answer
13- Explain the code in an educational manner
14- Suggest improvements if you identify any
15
16# QUESTION
17${query}
18`;

1# Project creation
2mkdir rag-code-assistant
3cd rag-code-assistant
4bun init
5
6# Dependencies installation
7bun add langchain @langchain/community @langchain/openai
8bun add qdrant-js ollama
9bun add -d typescript @types/node