RxR

User FAQ

How do I make sure that I am submitting a valid file?

The submission format is defined on the Competition pages. You can use test attempts to validate your submission format without having it count as an official attempt.

What is the difference between a test attempt and an official attempt?

Official attempts have restrictions on the frequency of submissions. This is to promote healthy competition, prevent overfitting and maintain the integrity of the test sets. Before submitting an official attempt, you should submit a test attempt to ensure that your file format is correct.
Test attempts only validate your submission file and are not scored. Test attempts are shown with the icon on the “Mydashboard” page and only the last 10 are shown.

I have submitted my attempt, how frequently can I submit further attempts?

The minimum time intervals between official attempts are listed below. Note that in the RxR competition, time intervals are only applied to official attempts with the same responses for the Trained in Test (TT) and Explored in Test (ET) questions. For example, you can make two official attempts on the same day if one uses beam search decoding (ET=Yes) and the other does not (ET=No). Refer to the RxR Competition page for more information about the meaning of TT and ET.

Attempt	Minimum Interval
Between 1st and 2nd	7 days
Beyond 2nd	30 days

Will I get notified via email or otherwise if my submission produces errors?

No. Please check back for status updates on the submission page.

Can I submit an English-only VLN agent? Why don't you show leaderboard results by language?

Test submissions need to cover instructions in all three languages (English, Hindi and Telugu). We made that decision so that the competition wouldn't gradually become English-centric over time. For reporting results and differences across languages, we encourage researchers to use the Val-Seen and Val-Unseen dataset splits.

What's the difference between the two competitions, 'RxR' and 'RxR-Habitat'?

Both competitions use the same training and validation data from the RxR dataset but with different simulators. In the RxR competition, VLN agents navigate through navigation graphs from the Matterport3D simulator. In the RxR-Habitat competition, VLN agents navigate continuous environments using the Habitat simulator. So, while the instructions and the environments are the same, the RxR competition emphasizes visual realism (the agent always sees real images). The RxR-Habitat competition emphasizes more realistic control, since the agent has to not only understand the instructions but also avoid bumping into things and getting stuck behind furniture!

I would like to not show my submission on the leaderboard, is that possible?

For a healthy competition, we do not allow on-demand removal of submissions from the leaderboard.

I submitted an official attempt, but now I would like to withdraw it. How do I do that?

This is not supported. Once you make an official submission, there is no way to cancel or prevent it from showing up on the leaderboard. To test for errors, please use test attempts. Test attempts are not shown in the leaderboard.

How long will my data be kept in MLJam? Is my data shared with any other Google products?

Your submission files are deleted after 30 days. Your profile data is kept as long as you have an active profile. Your profile data and submissions are not shared with any other Google product. We may contact you for similar competitions if you opt in for it on your profile page.

Is it ok to publish my results on social sites and in blogs?

Yes! We encourage you to share your position on the leaderboard on social media, blogs, etc. We also recommend that the leaderboard results (which are assessed on the RxR Test-Standard split) be reported in academic publications.

How do I delete all my data?

Data deletion can happen in two ways:

If you explicitly request to delete your data from the Profile page, all your previous attempts (test and official) will be deleted. Any official attempts will be made anonymous on the leaderboard.
If you delete your Google account, your MLJam data will also be deleted and your position on the leaderboard will be anonymized.

In both cases, you will lose access to your data and your nickname will be available for use again by someone else.

Where should I report issues and requests?

If you have a technical question regarding the dataset or publication, please create an issue in our github repository. For other issues, feedback or concerns, please email us at rxrvln@google.com.