InfluxDB Storage Engine Internals | Metamarkets

Recorded at DataEngConf SF '17

InfluxDB is an open source time series database developed over the last 3 years. In that time we've tried different storage engines starting with LevelDB and testing out HyperLevelDB, RocksDB and BoltDB. Over a year ago we made the decision to write our own storage engine from scratch. Inspired by the LSM Tree underlying LevelDB and its variants, we created a new storage engine we're calling the TSM Tree (Time Structured Merge Tree). Over the last eight months we've added to this storage engine to provide index capabilities for mapping metadata to underlying time series.

This talk will briefly cover our journey with other storage engines and why we ultimately decided to write our own from scratch. The underlying InfluxDB storage engine is more like two storage engines in one: a time series storage engine and an inverted index for metadata. This talk will dive into the details about how each of these systems work, their design considerations and lessons learned along the way. We'll cover compression techniques for columnar time series storage, Robin Hood Hashing for quickly index lookups, and sketches for estimation of series cardinality at scale.

Speaker: Paul Dix, Metamarkets

ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.

FOLLOW DATA COUNCIL:
Twitter:   / datacouncilai  
LinkedIn:   / datacouncil-ai  
Facebook:   / datacouncilai  
Eventbrite: https://www.eventbrite.com/o/data-cou...