Building a Multi-Cluster Privately Hosted LLM Serving Platform on Ku... Julian Bright & Noah Yoshida
Автор: CNCF [Cloud Native Computing Foundation]
Загружено: 2023-11-13
Просмотров: 2634
Building a Multi-Cluster Privately Hosted LLM Serving Platform on Kubernetes - Julian Bright & Noah Yoshida, Predibase
Large language models (LLMs) have taken the tech industry by storm, due to their powerful capabilities, and accessibility through APIs like ChatGTP. However, hosting your own LLM can be very challenging due to their large size and GPU resource requirements. In this session, we will take you through our journey at Predibase in building a cloud agnostic privately hosted LLM serving platform on Kubernetes.. We will cover in detail the architecture of our control plane, and dataplane secured with an Istio service mesh, as well as our use of KEDA for event driven auto scaling to support serverless inference of open-source models. By the end of the talk, attendees will have a better understanding of some of the challenges in deploying LLMs, and how to apply some of the tools and techniques we adopted in their own organization.
Доступные форматы для скачивания:
Скачать видео mp4
-
Информация по загрузке:
