How Modern Search Engines Work Using TF-IDF and BM25 and Embeddings

This text presents a practical, end-to-end approach to building a modern hybrid search engine that combines TF-IDF, BM25, and embeddings to deliver more robust search results than any single method alone. TF-IDF provides fast, literal keyword matching, while BM25 improves lexical search through better term weighting and document length normalization. Embeddings add a semantic layer, allowing the system to capture conceptual similarity even when exact words do not overlap. Each method can be viewed as an independent “judge” scoring document relevance from a different perspective. The system normalizes and combines these scores using weighted fusion to produce a final ranking. An optional cross-encoder re-ranking step further refines the top candidates using deeper query-document interaction. The example demonstrates how this hybrid approach handles real search failures such as synonyms, short queries, and overly broad semantic matches. The text explains why purely lexical or purely semantic systems are insufficient in isolation. It highlights that hybrid retrieval is now the standard design pattern in real-world RAG and search systems. Finally, it outlines realistic paths for scaling this approach, either by leveraging existing web search APIs for discovery or by building a focused crawler and index, while clarifying why indexing the entire internet is far beyond a small-scale setup.