RxR Competition

This competition reports results on the RxR Test-Standard split, a set of 12,249 navigation instructions in English, Hindi and Telugu associated with paths in the Matterport3D dataset. Only the instructions and the starting position of each path in the test set are made public. For scoring, participants must upload the paths generated by their Vision-and-Language Navigation (VLN) agents. Paths are defined by sequences of Matterport3D viewpoints, using the navigation graphs from the Matterport3D Simulator. This is an ongoing competition with no submission deadline.

Participate

Leaderboard

This leaderboard shows the performance of official submissions ranked by NDTW (see below for details).

For each official submission we report Path Length (PL), Navigation Error (NE ↓), Success Rate at reaching the end of the path (SR ↑), Success weighted by inverse Path Length (SPL ↑), Normalized Dynamic Time Warping (NDTW ↑), and Success weighted by normalized Dynamic Time Warping (SDTW ↑). See Anderson et al. (2018) and Ilharco et al. (2019) for a discussion of VLN path fidelity metrics. Trained in Test (TT) and Explored in Test (ET) indicate easier problem settings which are described in detail on the Participate tab.