RxR-Habitat Competition

NEW! The RxR-Habitat Competition is on again at CVPR 2023!

The RxR-Habitat competition is a challenge that requires Vision-and-Language Navigation (VLN) agents to follow RxR navigation instructions in Matterport3D environments using the Habitat simulator. Compared to the standard RxR competition, in which paths are defined in a navigation graph, the continuous environments in Habitat emphasize the need for robust control using low-level actions, as illustrated in the demo video. Results are reported on the RxR Test-Challenge split. For scoring, participants must upload the paths generated by their agents in the Habitat simulator. As with the standard RxR competition, navigation instructions are in English, Hindi and Telugu.

Participation Guidelines

For fair comparison, all entries to the CVPR 2023 Challenge must use a standardized configuration for the VLN agent's observation and action space. This configuration, which is the same as the last two years, is based on a 30° turn angle, 30° look up / look down action, 0.25m step size and 480x640 RGBD vision. Precise details of the configuration are provided with the starter code (see below). CVPR Challenge entrants will need to certify (via a checkbox) that their agent uses this configuration when submitting to the leaderboard. Any submission received between 20 March and 15 May 2023 using the standardized configuration will be considered an entry to the CVPR 2023 Challenge. Submissions that do not use the standardized configuration (e.g., using panoramic vision, or larger step sizes) are welcome at any time but will not be part of the CVPR 2023 Challenge. These submissions should answer 'No' to the question asking if the agent was trained using the standardized challenge config.

Starter Code

We provide PyTorch starter code for an RxR-Habitat agent on github. In the starter code, all Habitat simulator settings (i.e., agent height and width, sensor configuration and action space configuration) are set to the standardized configuration. Paths in the RxR dataset were ported to the Habitat simulator using the VLN-CE codebase. Note that some paths could not be ported to Habitat so the RxR-Habitat dataset is slightly smaller than the original RxR dataset.

Important Dates for the CVPR 2023 Challenge
  • 17 March RxR-Habitat competition launch
  • 20 March Leaderboard submissions open
  • 15 May Leaderboard submission deadline at 23:59:59 UTC
  • 19 June Winners announced at the CVPR 2023 Embodied AI Workshop
Organizers

The organizers of the RxR-Habitat competition are:

RxR-Habitat first person agent demonstration
CVPR

The RxR-Habitat competition was first proposed as part of the CVPR 2021 Embodied AI Workshop, and repeated at CVPR 2022 and CVPR 2023. The video below provides further details of the task, starter code, and results from the first workshop:


How to Participate

Submit the paths generated by your VLN agent in response to the RxR Test-Challenge instructions using the JSON Lines format detailed below. Each submission is scored by comparing the generated path to the true path for each navigation instruction, using a variety of standard path fidelity metrics.

The results obtained by each submission are made available on the leaderboard as part of this competition. You will have access both to your personal leaderboard, as well as to the global leaderboard that will rank all submissions we receive.

UPDATE: There is currently a problem with the automatic submission process. If it doesn't work please email your submission file and team details us at rxrvln@google.com.

Prerequisites

You must have a Google Account to create a profile to participate in the competition.

Submission Format

Submissions must be in JSON Lines format, also called newline-delimited JSON. Each line should be a JSON object containing the instruction id and the agent's predicted path for one instruction. The path must be specified by an array of Matterport3D xyz coordinates, with the first coordinate matching the agent's starting location. We provide starter code that generates predicted paths in the correct format. 

{
  "instruction_id": int,
  "path": Sequence[Tuple[float, float, float]]
}

For example:

{
  "instruction_id": 0,
  "path": [
    [18.866100311279297, 0.9523793458938599, 1.3258600234985352],
    [18.789167404174805, 0.7878450155258179, 1.08799147605896],
    ...
  ]
{

You can use test attempts to validate your submission format without having it scored or counted as an official attempt.

Valid Training & Inference Settings

The VLN task is most difficult when the agent has no prior experience with the test environment. For fair comparisons in the RxR-Habitat competition, agents are not permitted to use the test environments during training in any way. Each submitted path should be the outcome of a single attempt using only local, sequential observations (like a person would). If there is any doubt about these requirements we encourage participants to contact us well in advance of submission.

You must be logged in and completed profile page to submit an attempt.
PrivacyTermsAbout GoogleGoogle Products