How to Self-Host Your Own Private Local AI Stack - Ollama, Open WebUI, Whisper, searXNG, and more

In this tutorial we'll walk through my local, private, self-hosted AI stack so that you can run it too.

If you're looking for the overview of this stack, you can out the video here:    • Self-Hosted AI That's Actually Useful  

Video Notes: https://technotim.live/posts/ai-stack...

Products:
GPUs
12GB is probably "good enough" for small models.

3000 Series
👇 My value pick. 3060, 12GB RAM, affordable, can play games too.
MSI Gaming GeForce RTX 3060 12GB: https://amzn.to/3WerOUI
GIGABYTE GeForce RTX 3060 Gaming OC 12G: https://amzn.to/3WdqHV9
NVIDIA GeForce RTX 3090 Founders Edition Graphics Card Renewed 24GB: https://amzn.to/4cTIS7Q
Zotac Gaming GeForce RTX 3090 Trinity OC, 24GB: https://amzn.to/3WdhJqV

4000 Series
ASUS TUF Gaming GeForce RTX™ 4090 OG OC Edition Gaming Graphics Card 24GB: https://amzn.to/463LHRF
MSI Gaming GeForce RTX 4060 8GB: https://amzn.to/4eVxrON
ASUS Dual GeForce RTX™ 4070 White OC Edition 12GB: https://amzn.to/460rqN0

CPUs
You'll want a modern CPU, if you are going desktop class here are a few I would choose
Intel Core i7-12700K Gaming Desktop Processor: https://amzn.to/3XTj1Zu
Intel Core i7-13700K Gaming Desktop Processor: https://amzn.to/3Lg7M5V
Intel Core i7-14700K Gaming Desktop Processor: https://amzn.to/3WftehU

Storage
For flash storage, I always go with these SSDs
Samsung 870 EVO SATA: https://amzn.to/3XTW6gA
SAMSUNG 990 PRO SSD NVMe: https://amzn.to/4cW1xA9

(Affiliate links may be included in this description. I may receive a small commission at no cost to you.)

Support me on Patreon:   / technotim  
Sponsor me on GitHub: https://github.com/sponsors/timothyst...
Subscribe on Twitch:   / technotim  
Merch Shop 🛍️: https://l.technotim.live/shop
Gear Recommendations: https://l.technotim.live/gear
Get Help in Our Discord Community: https://l.technotim.live/discord
Main channel:    / @technotim  
Talks Channel:    / @technotimtalks  

00:00 - Intro
01:14 - Hardware Specs
02:23 - GPU Considerations
03:10 - GPU Perspective
04:54 - Proxmox
07:05 - Server OS
07:25 - Drivers
08:31 - NVIDIA Container Toolkit
09:44 - Docker
10:25 - Folder Layout
11:20 - Stack Overview
12:15 - Traefik & Docker Networking Considerations
12:55 - Ollama
17:29 - Open WebUI
20:55 - Starting the Stack
23:08 - Ollama Models
25:50 - Chatting with Ollama & Performance
32:05 - searXNG
33:33 - Stable Diffusion & ComfyUI
39:54 - Stable Diffusion Models
44:42 - ComfyUI Workflows
47:00 - Verifying Model Checksum
48:48 - Integrating ComfyUI into Open WebUI
51:59 - Whisper
56:22 - Home Assistant Assist - Chat, Voice, and Text to Voice
01:04:15 - Code Suggestions and Chat Assistant in VSCode (Free Co-pilot)

Thank you for watching!