Find the BEST RAG Strategy with Domain Specific Evals

Creating custom RAG chunking and embedding strategies with domain specific evaluation experiments

Resources:
Notebook Repo: https://github.com/ALucek/custom-rag-...
Prior Chunking Breakdown:    • The BEST Way to Chunk Text for RAG  
ChromaDB Repo: https://github.com/brandonstarxel/chu...
ChromaDB Colab Notebook: https://colab.research.google.com/dri...
ChromaDB Research Report: https://research.trychroma.com/evalua...

Chapters:
00:00 - Why Measure Chunking & Embedding
03:44 - Creating a Custom Chunking Strategy
06:12 - Breaking Down Eval Metrics
08:00 - Metrics: Eval Dataset
09:17 - Metrics: Recall, Precision, IoU
13:27 - General Evals: Describing Test Set
14:32 - General Eval: Process & Running Test
17:58 - General Eval: Embedding Test
19:00 - Running Multiple Evals Across Strategies
20:36 - Multiple Evals: Interpreting Results
23:18 - Domain Specific Dataset Generation & Filtering
27:06 - Running Domain Specific Evals
29:32 - Final Thoughts

#ai #machinelearning #datascience