How do vector (search) databases work? ft: turbopuffer

For memberships: join this channel as a member here:
   / @thegeeknarrator  

Summary:
In this conversation, Kaivalya Apte and Simon Eskildsen talk about vector databases, particularly focusing on TurboPuffer. They discuss the importance of vector search, embeddings, and the challenges associated with building efficient search engines. The conversation covers various aspects such as cost considerations, chunking strategies, multi-tenancy, and performance optimization. Simon shares insights on the future of vector search and the significance of observability and metrics in database performance. The discussion emphasizes the need for practical application and experimentation in understanding these technologies.

Chapters:
00:00 Introduction to Vector Databases
10:34 Understanding Vectors and Embeddings
15:03 Example: Designing a Search Engine for Podcasts
27:53 Scaling Challenges in Vector Search
36:46 Indexing and Querying in TurboPuffer
38:12 Understanding Indexing and Query Planning
45:45 Exploring Index Types and Their Performance
50:27 Data Ingestion and Embedding Retrieval
54:19 Use Cases and Challenges in Vector Search
01:01:22 Metrics and Observability in Vector Databases
01:03:52 Future Trends in Vector Search and Databases

References:
How do build a database on Object Storage?    • How would you design a database on Object ...  
Turbopuffer https://turbopuffer.com/
Continous Recall measurement: https://turbopuffer.com/blog/continuo...
Turbopuffer architecture: https://turbopuffer.com/architecture

Don't forget to like, share, and subscribe for more insights!

=============================================================================
Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription.

https://app.codecrafters.io/join?via=...

=============================================================================
Database internals series:    • Write-ahead-logging  

Popular playlists:
Realtime streaming systems:    • Realtime Streaming Systems  

Software Engineering:    • Software Engineering  

Distributed systems and databases:    • Distributed Systems and Databases  

Modern databases:    • Modern Databases  

Stay Curios! Keep Learning!

#vectors #vectorsearch #embeddings #TurboPuffer #searchengines #AI #machinelearning #data #storage #objectstorage #distributedsystems #multi-tenancy