Visual Question Answering on Remote Sensing Images, Sylvain Lobry, Universit´e de Paris, France

Abstract: Remote sensing images contain a wealth of information which can be useful for a wide range of tasks including land cover classification, object counting or detection. However, most of the available methodologies are task-specific, thus inhibiting generic and easy access to the information contained in remote sensing data. As a consequence, accurate remote sensing product generation still requires expert knowledge. With Visual Question Answering for Remote Sensing (RSVQA), we propose a system to extract information from remote sensing data that is accessible to every user: we use questions formulated in natural language and use them to interact with the images. With the system, images can be queried to obtain high level information specific to the image content or relational dependencies between objects visible in the images. Using an automatic method, we built two datasets (using low and high resolution data) of image/question/answer triplets. The information required to build the questions and answers is queried from OpenStreetMap (OSM).

Bio: Sylvain Lobry is an assistant professor in Computer Science since 2020. He is a researcher at the SIP team of the LIPADE laboratory and teaches at UFR de Mathématiques et Informatique in Université Paris Cité. He obtained his PhD in image processing from Télécom Paris in 2017. His research interests are in the areas of methodological developments in image processing with applications in particular to remote sensing imagery. This includes high-resolution optical images processing using deep learning techniques and change detection, classification and regularization on multi-temporal series of SAR images using Markov Random Fields models. During his PhD, he was working on the SWOT mission, dedicated to the study of the world’s oceans and its terrestrial surface waters. Since 2019, he also works on the interactions between remote sensing data and natural language. In particular, he introduced the task of Visual Question Answering for Remote Sensing (RSVQA).