OM1: A multimodal stack for thinking machines by Jan Liphardt, Founder of OpenMind

Founder of OpenMind, Jan Liphardt, at the Democratize Intelligence Summit on January 24th, 2025 in San Francisco, CA.

Jan Liphardt, a bioengineering professor at Stanford (with a physics background from Berkeley), introduces a new approach to robotics that leverages large language models (LLMs) as if they were “blind and deaf,” receiving sensor inputs in natural language and producing actions through an open-source operating system called OM1. He emphasizes that LLMs from providers like OpenAI, Claude, and others can “speak robot” without heavy modifications, making it straightforward to connect text outputs to hardware commands. His team’s software pipeline turns multisensory inputs into English sentences—akin to narrating the world for a visually and hearing-impaired relative—and fuses these sentences into a paragraph sent to an LLM, which in turn directs a robot’s movements, expressions, and more. Liphardt envisions a future where thinking machines, possibly smarter than humans within a few years, will need new frameworks for identity, governance, and collaboration, prompting his interest in decentralization tools (e.g., Ethereum) and “one-time models” for secure sharing of AI capabilities. He cites examples of robots “hallucinating” as dogs or cats simply by tweaking text prompts, illustrating how easy it is to reprogram behaviors. While China currently leads in affordable robot hardware, Liphardt sees major players like Tesla, Amazon, and OpenAI quickly entering robotics, drawn by the same potential for flexible, prompt-based control. He concludes by stressing that natural language provides transparent debug-ability and rapid retraining, although more specialized solutions may emerge for edge cases or narrowly focused tasks like Roombas.