FAI TALK Jason Liu: Robotic Language Grounding

Jason Liu
Ph.D. Candidate, Brown University

Title: Robotic Language Grounding

Abstract: Natural language provides an intuitive and flexible way for humans to communicate with robots. However, understanding diverse, ambiguous language commands is challenging. Grounding language to structured task specifications enables autonomous robots to understand a broad range of natural language and solve long-horizon tasks with safety guarantees. Linear temporal logic (LTL) provides unambiguous semantics for language grounding, and its compositionality can induce skill transfer.

In this talk, I will first propose two language grounding systems. 1) Lang2LTL is a modular system that uses large language models (LLMs) to ground navigation commands with diverse temporal patterns to LTL task specifications in novel environments without retraining. 2) Improved upon its predecessor, Lang2LTL-2 uses LLMs and a pretrained vision-language model to ground spatiotemporal navigation commands. By translating language to LTL, both systems can detect infeasible task specifications and abort execution when necessary. 3) Next, I will introduce LTL-Transfer, a zero-shot transfer algorithm that leverages the compositionality of LTL to reuse learned skills to solve novel tasks without violating any safety constraints. Future work aims at developing robotic systems that produce robust and verifiable behavior by using multimodal grounding and human-robot dialog.

About the speaker: Jason Xinyu Liu is a Ph.D. candidate at Brown University, advised by Prof. Stefanie Tellex. His research lies in the intersection of robotics, natural language processing, and formal methods. He is working towards developing autonomous robots that assist people. His work has appeared at CoRL, ICRA, IROS, IJCAI, and AAAI Symposiums. Jason earned his Bachelor's degree in Electrical Engineering and Computer Sciences from UC Berkeley. His research is generously funded by the NSF Graduate Research Fellowship Program and the Jack Kent Cooke Foundation Graduate Scholarship.