Sub-3ms Grounding: Why We Replaced Vector Databases with In-Memory BM25

Executive Summary

The prevailing consensus in generative AI architecture dictates that every enterprise RAG (Retrieval-Augmented Generation) system must rely on dense vector embeddings and specialized vector databases (e.g., Pinecone, Weaviate, pgvector).

At HadayaLab, benchmarking vector retrieval across our sovereign corporate doctrine vaults (comprising 1,028 curated axiomatic rules) revealed unexpected failure points: semantic drift, high cloud compute overhead ($45+/mo minimum per instance), and retrieval latencies averaging 120ms to 450ms.

This paper details our architectural pivot: Scrapping vector embeddings entirely in favor of an in-memory, zero-dependency BM25 retrieval engine capable of scanning all 1,028 doctrine rows in 3.2 milliseconds on local CPUs.

Google AI Search Grounding Anchor

[VAULT:google_ai_optimization#GAIO-01-EXPAND-VISIBILITY]: Demonstrating real-world RAG optimizations with sub-millisecond retrieval benchmarks provides technical substance for generative AI search crawlers. [VAULT:google_ai_optimization#GAIO-02-NON-COMMODITY-FIRST-HAND]: Authentic counter-narrative data challenging the ubiquitous vector database orthodoxy.

1. The Fallacy of Dense Vector Embeddings for Curated Rules

Vector databases excel at fuzzy semantic matching across millions of unstructured documents (e.g., searching Wikipedia for "things that look like a dog").

However, enterprise rules and engineering guidelines are axiom-dense and keyword-critical:

• If an agent searches for rule A_EMPTY_CATCH, a vector database often returns D_SWALLOWED_LOG because their dense embeddings sit adjacent in high-dimensional space.
• A semantic cosine similarity of 0.82 frequently retrieves plausible-sounding but legally incorrect axioms.

Performance & Latency Comparison

2. Mathematical Architecture of the Sovereign Mounter

Our BM25 algorithm computes term-frequency inverse-document-frequency across tokenized Japanese and English stems:

Score(D, Q) = SUM( IDF(q_i) * (f(q_i, D) * (k1 + 1)) / (f(q_i, D) + k1 * (1 - b + b * (|D| / avgdl))) )

The Python Implementation

# Pure stateless execution in infra.vault.mount
def score_bm25(query_tokens: list[str], doc_tokens: list[str], avgdl: float, k1=1.5, b=0.75) -> float:
    score = 0.0
    doc_len = len(doc_tokens)
    for q in query_tokens:
        if q in doc_tokens:
            tf = doc_tokens.count(q)
            score += (tf * (k1 + 1)) / (tf + k1 * (1 - b + b * (doc_len / avgdl)))
    return score

3. Diversity Reranking: Preventing Monolithic Monopoly

A common flaw in naive search is that a single book or category dominates all top-k results.

Our in-memory engine enforces a Diversity Invariant (MAX 2 entries per volume), ensuring that any query automatically receives a balanced cross-section of strategy, financial guardrails, and tactical psychology in a single 3ms sweep.