DuckDB: How to Build 100x Faster Analytics Databases (with Co-Creator Hannes Mühleisen)

SQLite could do with a little competition, so when I invited the co-creator of DuckDB in to talk, I thought we'd be discussing the perils of trying to build a new in-process database engine. I quickly realised things went much deeper than just a tech refresh.

Hannes Mühleisen joins me this week to blend his academic credentials as a database researcher with his vehement need to make that research practical. And so we dive into what modern database literature has to say on making queries faster, more parallelizable, and closer to the metal, and how it all comes together in a user-friendly package that’s found its way into my day-to-day workload, and might well help out yours.

If you’re curious about the gory details of database queries, how they can take advantage of modern hardware, or how all that research actually turns into a useful tool, Hannes has some great answers.

--

DuckDB: https://duckdb.org/
Database Systems Book: http://infolab.stanford.edu/~ullman/d...
Kris’ first computer: https://en.wikipedia.org/wiki/File:ZX...
Volcano Query Evaluation System [pdf]: https://paperhub.s3.amazonaws.com/dac...
Morsel Query Engine [pdf]: https://cs.brown.edu/~kayhan/papers/m...
Unnesting Arbitrary Queries [pdf]: https://cs.emis.de/LNI/Proceedings/Pr...
Papers Hannes' team have published: https://duckdb.org/why_duckdb#peer-re...

DuckDB on Mastodon: https://mastodon.social/@duckdb
Kris on Twitter:   / krisajenkins  
Kris on LinkedIn:   / krisjenkins  
Kris on Mastodon: https://mastodon.social/@krisajenkins

--

#softwaredevelopment #podcast #programming #database #duckdb #sql #sqlite

0:00 Intro
2:15 Podcast
8:15 From Professor to Implementor
15:28 Deciding What To Build
22:43 How does DuckDB work?
42:50 Parallelization
1:10:18 A real world use-case
1:18:59 Outro