Billion-parameter brains in pocket-sized chips: The local AI revolution [VIDEO VERSION]

#AI #LLM #embeddedsystems #IoT #privacy #wetalkiot

In this episode, we explore how engineers are embedding powerful AI directly into hardware – no cloud connection required.
Michaël Uyttersprot from Avnet Silica and Cedric Vincent from Tria Technologies reveal how they run ChatGPT-quality language models on resource-constrained embedded devices. What once demanded data centre infrastructure now fits onto chips with just 2GB of RAM.
The conversation covers the technical challenges of cramming billion-parameter models into embedded systems, real-world applications from conference rooms to industrial robotics, and the three compelling reasons driving this shift: data privacy, power efficiency, and cost control.
Michaël and Cedric discuss hardware platforms from AMD, NXP, and Qualcomm, explain techniques like quantisation and mixture of experts, and demonstrate applications including a vintage telephone box that lets you call avatars from different time periods.
Tune in to learn why the future of AI might not be in the cloud at all – and what that means for industries from manufacturing to healthcare.

Summary of this week's episode:
(02:48) What makes large language models special
(05:27) Why run LLMs locally on embedded devices
(07:42) Real-world applications: Vision LLMs and OCR
(11:12) Technical deep dive: How to fit billions of parameters into tiny devices
(18:52) Understanding temperature: Making AI creative or accurate
(22:41) Industries moving fastest: OCR, security, and robotics
(24:52) Future applications: Robotic arms and time series analysis
(28:00) The biggest technical hurdle: Power consumption
(30:55) Advice for engineers: Start with llama.cpp

Show notes:
Michaël Uyttersprot:   / micha%c3%abl-uyttersprot-aaa971211  

Cedric Vincent:   / cedric-vincent-19222910  
Tria Technologies: https://www.tria-technologies.com/

Generative AI at the Edge: https://my.avnet.com/silica/solutions...

The podcast episode where the generative AI examples where discussed: https://www.podbean.eu/ep/pb-9juiy-d4...
How to enhance embedded systems with Generative AI and Local LLMs |

Michael Uyttersprot at HWPMAX25:    • How to enhance embedded systems with Gener...  

Listen to the "We Talk IoT" Soundtrack on:

Spotify: https://open.spotify.com/playlist/05M...

YouTube:    • Jamiroquai - Cosmic Girl (Official Video)  

The Llama song:    • The Llama(.cpp) Song - AI-Generated track ...  

About Avnet Silica:
This podcast is brought to you by Avnet Silica—the Engineers of Evolution.
Subscribe to our newsletters here:

https://my.avnet.com/silica/resources...

You can connect with us on LinkedIn:   / silica-an-avnet-company  . Or find us at www.avnet-silica.com.