Cheap Log Analytics

Tired of observability vendors charging you per gigabyte? Watch me build a production-grade AWS CloudFront analytics system that handles 15 TiB of data for just $279/month ($3,348/year).

In this video, I'll show you how to set up CloudFront log analytics with ClickHouse Cloud and visualize it with Grafana - all while avoiding the per-gigabyte pricing trap.

WHAT YOU'LL LEARN
✅ Set up AWS CloudFront logging to S3
✅ Configure ClickHouse Cloud with S3Queue table engine
✅ Avoid AWS Data Transfer Out (DTO) charges completely
✅ Query latency percentiles with SQL
✅ Visualize with CLI, ClickHouse Console, GNUPlot, and Grafana
✅ Build production dashboards on a fixed budget

COST BREAKDOWN
ClickHouse Cloud 8GiB Instance: $220/month
S3 Storage (15 days staging): $8/month
ClickHouse Storage (90 days, 2 TiB): $51/month
S3 API Requests: $0.43/month
TOTAL: $279/month for 15 TiB of analytics capability

Compare that to vendors charging per gigabyte!

TIMESTAMPS
00:00 Introduction - Breaking the per-gigabyte pricing trap
01:00 The "Walk" stage of observability
02:00 Architecture overview
03:00 Budget breakdown and cost optimization
05:00 Implementation Step 1: Enable CloudFront logging
06:00 Implementation Step 2: AWS IAM roles (the hard part)
07:00 Implementation Step 3: ClickHouse S3Queue schema
09:00 Implementation Step 4: Debugging S3 access
10:00 Query latency percentiles with SQL
11:00 ClickHouse Cloud Console dashboards
12:00 Grafana Cloud integration
13:00 Tradeoffs: Latency and storage
15:00 Conclusion and call to action

RESOURCES
📚 Full Blog Post
https://cardinality.cloud/blog/aws-we...

📖 ClickHouse S3Queue Documentation
https://clickhouse.com/docs/engines/t...

🔐 Accessing S3 Data Securely
https://clickhouse.com/docs/cloud/dat...

☁️ CloudFront Logging Guide
https://docs.aws.amazon.com/AmazonClo...

📊 Grafana Cloud Free Tier
https://grafana.com/products/cloud/

🗄️ ClickHouse Cloud
https://clickhouse.com/cloud

ABOUT CARDINALITY CLOUD
I'm Jack Neely, Independent Observability Architect and owner of Cardinality Cloud, LLC - where observability is leverage, not a tax. With 25 years of SRE and DevOps experience and over a decade as an observability subject matter expert, I help teams avoid six-figure observability bills.

Need help building something like this for your team? Let's talk before your CFO asks why observability is a six-figure line item.

🌐 Website: https://cardinality.cloud/
💼 LinkedIn:   / jack-neely-47316575  
📧 Contact: jjneely@cardinality.cloud

#observability #aws #cloudfront #clickhouse #grafana #devops #sre #costoptimization #loganalytics