Abstract: Navigational signs are common aids for human wayfinding and scene understanding, but are underutilized by robots. We argue that they benefit robot navigation and scene understanding, by directly encoding privileged information on actions, spatial regions, and relations. Interpreting signs in open-world settings remains a challenge owing to the complexity of scenes and signs, but recent advances in vision-language models (VLMs) make this feasible. To advance progress in this area, we introduce the task of navigational sign understanding which parses locations and associated directions from signs. Additionally, we present a baseline approach using VLMs, and demonstrate their promise on navigational sign understanding. We also outline different applications, such as localization and navigation, which benefit from the spatial-symbolic information encoded by navigational signs.

Host: Prof. Kai Hormann