DeutschAI-Designers
Most RAG projects don’t fail because of the database, and they don’t fail because of the language model. They fail at a component almost no one talks about: the embedding model. It decides whether your search finds the right paragraph or lands just beside it. And many companies hand this very model to an outside API provider – along with every line they want to search through.
It doesn’t have to be that way. You can run an embedding model yourself with modest effort. Often it even fits on hardware you already own. This article explains why the move pays off, which model suits which purpose, and what to watch for in production.
What an embedding model actually does
An embedding model turns text into numbers. A sentence, a paragraph, or a whole invoice line becomes a vector – a list of several hundred decimal numbers. Texts with similar meaning end up close together in this numeric space, even when they share no words at all. „Notice period“ and „ending a contract early“ move toward each other, where a keyword search would come up empty.
This single step sets the quality of the entire pipeline. Reranking, hybrid search, your choice of vector database – all of it works on the vectors the embedding model delivers. If the vectors are weak, no language model behind them can save the result. It simply receives the wrong paragraphs and spins them into a confident-sounding but wrong answer.
So the close look is worth it. An embedding model is not an interchangeable cog. It is the foundation your search stands on.
Why you should bring the model in-house
The first reason is data protection. Anyone generating embeddings through a third-party API ships their documents to an outsider – contracts, personnel files, engineering drawings. For many regulated industries, that alone is a dealbreaker. A self-hosted model processes everything inside your own network. Not a single byte leaves the building.
The second reason is cost. Embeddings don’t happen just once. Every new document has to be embedded, every search query too, and each model switch forces you to re-index the entire corpus. Across millions of text chunks, that adds up fast. Your own model on existing hardware turns these ongoing costs into something you can plan around.
The third reason is control. You decide when a model changes. No provider deprecates a version overnight and forces you into a re-indexing scramble. And you can adapt the model to your own technical language – a lever a closed API never offers in the first place.
Which model fits your use case
The good news: you don’t need to train anything yourself. Several freely available model families now deliver quality that was reserved for the big providers just two years ago. What matters is picking by your actual need, not by the top spot in some leaderboard.
For German-language or mixed collections you need a multilingual model. BGE-M3 and multilingual-e5-large cover more than a hundred languages and return solid hits on German technical text. Jina Embeddings v3 and Nomic Embed are further serious candidates, with Nomic standing out for its open training dataset. For English-only content, a smaller model like gte-base or mxbai-embed-large often does the job.
Watch three numbers:
- Vector size (dimensions): More dimensions mean more memory and slower search. 768 or 1024 are a good middle ground for most cases. Some models can run truncated without losing much quality.
- Maximum input length: Does a typical paragraph fit? Models capped at 512 tokens silently cut off longer sections – a common, invisible mistake.
- Language coverage: An English-only model returns weak vectors on German contracts, even when it shines in the benchmark.
Hardware and operations: less than you’d expect
An embedding model is many times smaller than a language model. The common ones run to a few hundred million parameters and fit comfortably on a single mid-range graphics card. For smaller collections, even the CPU is often enough, especially if you serve the model as ONNX or in quantized form.
For production, skip the home-grown script. Ready-made servers like Hugging Face’s Text Embeddings Inference, or wiring the model into your existing inference server, batch requests, keep the GPU evenly loaded, and expose a clean interface. That way you process thousands of chunks per second instead of pushing each one through the pipeline alone.
Separate two load profiles cleanly. The initial indexing of a large archive is a batch job you run overnight. The query at runtime needs low latency but only touches a short piece of text. Run both with the same configuration and you throw away either throughput or response time.
Tuning it to your own technical language
This is where self-hosting really pays off. A general model knows everyday language well, but your internal abbreviations, product names, and standard designations it treats like foreign words. A machine builder with thousands of part numbers, or a law firm with its own forms, feels this immediately in mediocre hits.
With a few thousand example pairs – a question and its matching text chunk – you can sharpen an open model. This fine-tuning often takes only hours on a single GPU and lifts the hit rate in your own domain noticeably. You harvest the example pairs from support tickets, from how your staff search, or from queries you’ve already answered. With an outside API, that path is simply closed.
The most common pitfalls
The costliest mistake is a late model switch. Vectors from different models are not compatible. Change models and you have to re-embed the entire corpus – a job of days on large archives. So choose the model carefully and early, rather than swapping it later under pressure.
Second pitfall: chunk length. Embed whole pages into a single vector and the meaning blurs, leaving search fuzzy. Cut too small and the context is gone. Aim for sensibly bounded sections that fit your model’s input limit.
Third point: check that your database expects the same normalization your model produces. Many models output normalized vectors, some don’t. If the two don’t match, the similarity search computes the wrong thing – and no one notices, because results still appear. So test with real questions from daily work from the start, not with constructed examples.
Conclusion: small in effort, large in effect
The embedding model is the most underrated lever in any AI-powered search. Hosting it yourself costs less hardware and effort than a language model, protects your data completely, and opens the door to tuning for your technical language. Choose carefully here and you lay a foundation every other RAG component can build on reliably.
Want to build a RAG search on your own documents – privacy-compliant and tailored to your industry? The experts at AI-Designers guide you from model selection all the way to production. Get in touch.
Images: AI-Designed