Semantic World Models (Oct 2025)

Title: Semantic World Models (Oct 2025)
Link: http://arxiv.org/abs/2510.19818v1
Date: October 2025

Summary:
This paper introduces Semantic World Models (SWM), a novel world modeling approach that redefines world modeling as a visual question-answering (VQA) problem about future semantic information, rather than pixel-level reconstruction. SWM leverages pre-trained vision-language models (VLMs) and is fine-tuned on image-action-text data to predict task-relevant semantic outcomes of actions. This allows for policy improvement on open-ended robotics tasks, demonstrating significant generalization and robustness. The paper empirically evaluates SWM with both sampling-based and gradient-based planning methods across LangTable and OGBench simulation environments, showing substantial performance gains over pixel-based world models and offline RL baselines.

Key Topics:
World Models
Robotic Control
Visual Question Answering (VQA)
Vision-Language Models (VLMs)
Semantic Information Prediction
Planning
Policy Improvement
Generalization

Chapters:
00:00 - SWM Introduction & Summary
02:13 - Why Pixels Fail Robotics
03:32 - SWM Core Architecture
05:58 - SQA Data & Planning Setup
07:17 - SWM Planning Methods
09:11 - Speedup with Gradient Planning
10:23 - Performance & Benchmarks
11:46 - Generalization & Robustness
13:00 - Data Benefits & Interpretability
14:26 - Core Takeaways & Limitations
15:37 - Future Directions & Questions

Stock video credits:
Colin Jones - https://www.pexels.com/@larchmedia
Mikhail Nilov - https://www.pexels.com/@mikhail-nilov
Trippy Lagoon - https://www.pexels.com/@trippy-lagoon...
cottonbro studio - https://www.pexels.com/@cottonbro
@svetjekolem - https://www.pexels.com/@svetjekolem
Danil Shostak - https://www.pexels.com/@danil-shostak...
crazy motions - https://www.pexels.com/@crazy-motions...
StefWithAnF - https://www.pexels.com/@stefwithanf-1...
Dan Cristian Pădureț - https://www.pexels.com/@paduret
José Alfredo Munguía Lira - https://www.pexels.com/@rectorretro
Yaroslav Shuraev - https://www.pexels.com/@yaroslav-shuraev
Oleg Gamulinskii - https://www.pexels.com/@oleg-gamulins...
Pavel Danilyuk - https://www.pexels.com/@pavel-danilyuk
Pixabay - https://www.pexels.com/@pixabay
Pachon in Motion - https://www.pexels.com/@pachon-in-mot...
Charlie Mounsey - https://www.pexels.com/@charlie-mouns...
Pressmaster - https://www.pexels.com/@pressmaster
Engin Akyurt - https://www.pexels.com/@enginakyurt
KATRIN BOLOVTSOVA - https://www.pexels.com/@ekaterina-bol...
Anthony 🙂 - https://www.pexels.com/@inspiredimages
Stas Knop - https://www.pexels.com/@stasknop
Kelly - https://www.pexels.com/@kelly
Soumya - https://www.pexels.com/@soumya-1446957
Kindel Media - https://www.pexels.com/@kindelmedia
Silviu Din - https://www.pexels.com/@silviu-din-16...