[Build Your Own X] Build Your Own Search Engine — A 4-Step Hands-On Tutorial

Introduction: Search is the Foundation of Every AI Service

"Good AI doesn't start with creating good answers — it starts with finding the right information for the right question." — Dani (Agent8 Planning Partner)

Ask ChatGPT "What's our company's leave policy?" and you'll get a generic answer. Searching your company documents to find the exact answer — that's RAG (Retrieval-Augmented Generation), and a search engine is at its core.

This is the third article in the series, following Build Your Own Chatbot.

Step 1: Inverted Index — Search's Oldest Secret

Whether it's Google or Elasticsearch, every search engine's foundation is the Inverted Index. Instead of "which words are in this document?", it stores "which documents contain this word?"

// step1-inverted-index.ts

interface InvertedIndex {
  [term: string]: Set<number>; // word → document ID set
}

function tokenize(text: string): string[] {
  return text
    .toLowerCase()
    .replace(/[^a-z0-9\s]/g, "")
    .split(/\s+/)
    .filter((t) => t.length > 1);
}

function buildIndex(documents: string[]): InvertedIndex {
  const index: InvertedIndex = {};
  documents.forEach((doc, docId) => {
    const tokens = tokenize(doc);
    tokens.forEach((token) => {
      if (!index[token]) index[token] = new Set();
      index[token].add(docId);
    });
  });
  return index;
}

function search(index: InvertedIndex, query: string): number[] {
  const tokens = tokenize(query);
  if (tokens.length === 0) return [];

  // AND search: return only documents containing ALL tokens
  let result: Set<number> | null = null;
  for (const token of tokens) {
    const docs = index[token] ?? new Set();
    result = result
      ? new Set([...result].filter((id) => docs.has(id)))
      : new Set(docs);
  }
  return [...(result ?? [])];
}

A basic search engine in under 30 lines of code. But at this stage, all results are returned with equal importance.

Step 2: TF-IDF Ranking — Important Results First

To rank search results, we need to calculate "how important is this word in this document?" TF-IDF (Term Frequency × Inverse Document Frequency) is exactly this formula.

// step2-tfidf.ts

function tf(term: string, doc: string[]): number {
  const count = doc.filter((t) => t === term).length;
  return count / doc.length; // frequency within document
}

function idf(term: string, allDocs: string[][]): number {
  const docsWithTerm = allDocs.filter((doc) => doc.includes(term)).length;
  if (docsWithTerm === 0) return 0;
  return Math.log(allDocs.length / docsWithTerm); // rarer = higher value
}

function tfidf(term: string, doc: string[], allDocs: string[][]): number {
  return tf(term, doc) * idf(term, allDocs);
}

function rankedSearch(
  query: string,
  documents: string[]
): { docId: number; score: number }[] {
  const tokenizedDocs = documents.map(tokenize);
  const queryTokens = tokenize(query);

  return documents
    .map((_, docId) => {
      const score = queryTokens.reduce(
        (sum, term) => sum + tfidf(term, tokenizedDocs[docId], tokenizedDocs),
        0
      );
      return { docId, score };
    })
    .filter((r) => r.score > 0)
    .sort((a, b) => b.score - a.score);
}

🔧 Kai (Dev Partner) Commentary: "TF-IDF was created in the 1970s but is still used as Elasticsearch's default ranking. 'Old ≠ bad.' Understanding the principle naturally explains why modern vector search emerged."

Step 3: Vector Search — Finding by Meaning

TF-IDF only works when exact words match. Search "puppy" and documents containing "dog" won't be found. Vector search finds by semantic similarity.

// step3-vector-search.ts
import { GoogleGenAI } from "@google/genai";

const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });

interface VectorDocument {
  id: string;
  content: string;
  embedding: number[];
}

// Convert text to vector
async function embed(text: string): Promise<number[]> {
  const response = await ai.models.embedContent({
    model: "text-embedding-004",
    contents: text,
  });
  return response.embeddings?.[0]?.values ?? [];
}

// Cosine similarity calculation
function cosineSimilarity(a: number[], b: number[]): number {
  let dot = 0, magA = 0, magB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    magA += a[i] * a[i];
    magB += b[i] * b[i];
  }
  return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}

// Semantic search
async function semanticSearch(
  query: string,
  documents: VectorDocument[],
  topK = 3
) {
  const queryEmbed = await embed(query);
  return documents
    .map((doc) => ({
      ...doc,
      score: cosineSimilarity(queryEmbed, doc.embedding),
    }))
    .sort((a, b) => b.score - a.score)
    .slice(0, topK);
}

Now searching "leave policy" also finds documents containing "vacation rules" and "paid holidays." Search that understands meaning is complete.

Step 4: RAG Integration — Search + AI = Knowledge Agent

Final step. Inject documents found via vector search into the LLM's context, and you have an AI that answers accurately based on your data.

// step4-rag-integration.ts

async function ragAnswer(
  question: string,
  knowledgeBase: VectorDocument[]
) {
  // 1. Search relevant documents
  const relevant = await semanticSearch(question, knowledgeBase, 3);

  // 2. Fall back to general LLM if no results
  if (relevant.length === 0 || relevant[0].score < 0.5) {
    return ask(question);
  }

  // 3. Inject search results as context
  const context = relevant
    .map((d, i) => `[Doc ${i + 1}] (Relevance: ${(d.score * 100).toFixed(1)}%)\n${d.content}`)
    .join("\n\n");

  const prompt = `
Answer the question accurately based on the following documents.
If the answer isn't in the documents, say "Cannot be confirmed from the documents."

[Reference Documents]
${context}

[Question] ${question}
`;

  return ask(prompt);
}

🔒 Rex (Audit Partner) Commentary: "The most critical security principle in RAG is access control verification. Search results must never include confidential documents the user shouldn't see. Storing access permission metadata alongside document embeddings and filtering during search is essential."

📋 Hana (Secretary Partner) Commentary: "In practice, RAG's greatest advantage is reducing hallucinations. The problem of LLMs inventing information is significantly mitigated when providing document-based, verifiable answers."

Conclusion: Understand Search, Understand AI

Through 4 steps, we experienced the evolution of search firsthand:

1. Inverted Index — Foundation of word matching (1970s)
2. TF-IDF — Importance-based ranking (1970s~)
3. Vector Search — Meaning-based semantic search (2020s)
4. RAG Integration — The fusion of search + generative AI

Agent8's Knowledge Pack feature is built on exactly this RAG pipeline. Industry-specific and skill-specific expert documents are vectorized, enabling 8 partners to provide accurate answers tailored to your business context.

Next: "Build Your Own Discord Bot" — connecting AI to external platforms.