How do I make sure that I am submitting a valid container model? If you follow the instructions provided on the Natural Questions GitHub page, there are high chances that you will end up with the correct container. You can also
Submit test attempts or
Test your container locally on your machine using the instructions provided.
What is the difference between test attempt and official attempt? Official attempts have restrictions on frequency of submissions. This is to promote healthy competition, prevent “Hill Climb” and maintain the test set integrity. So, when you are submitting an official attempt, you want to make sure that docker image and other items are valid. Test attempt or submission is only for validation of your docker image. We run the submissions against a very small test set and they are not scored. Test attempts are shown with the in “Mydashboard” page and limited to last 10. Will my Google Cloud account be billed for the compute cost of running my submissions? No. We will run all the submissions under Google internal quotas, and no billing will be incurred to your Google Cloud account for compute cost. However, you are responsible for the storing costs of your files in your Google Cloud account (if any, as there are free options for new accounts, etc.). I have submitted my attempt, how frequently I can submit further attempts? Each team is allowed one attempt every 7 days. We strongly discourage teams from trying to submit more frequently by creating multiple accounts. We reserve the right to remove the results of teams that abuse the dataset through the use of multiple accounts. For how long will my docker image be allowed to run to produce the test outputs? Any official submission will be allowed to run for a maximum of 24h. If, at the end of that time, there are not enough outputs created, the run is terminated and the submission is considered unsuccessful. Will I get notified via email or otherwise if my submission produces errors? No. Please check back for status updates on the submission page. I have submitted my attempt, it’s been a while and still showing status as “Pending” Depending on the complexity of the model you submitted, it can take a few hours (we allow up to 24h) to complete the evaluation process. If you know your submission should not take this long but it seems that it does, you can email us at natural-questions@google.com. I think the scores for my model are not accurate. How is my model scored? Official submission output answers are scored against the reference answers in the hidden test set. We have released the official evaluation script and we also provide the option to verify your submission on the 200 item sample from the dev set. Please check your submission's results on this 200 item sample, against the results you achieve locally to ensure that your submission is correct. If you have any further questions, please contact us at natural-questions@google.com. Will the dataset ever change? In the future, we may increase the size of the training set and refresh the test set. However, to provide a stable basis for the comparison of methods, we will not do this frequently. I would like to not show my submission on the leaderboard, is that possible? For a healthy competition, we do not allow on-demand removal of submissions from the leaderboard. I submitted a model, however I would like to withdraw that attempt. How do I do that? This is not supported. Once you make a submission, there is no way to cancel or prevent it from showing up on the leaderboard. To test for errors, please use test attempts. Test attempts are not shown in the leaderboard. How long will my data be kept? Is my data shared with any other Google products? Your submissions and profile data is kept as long as you have an active profile. Your profile data, as well as your submissions are not shared with any other Google product. We may contact you for similar competitions if you opt in for it on your profile page. Is it ok to publish my results on social sites and in blogs? Yes! We encourage you to brag about your position on the leaderboard on social media, blogs, etc. We also encourage you to use the leaderbord results and metrics for your academic publications. How do I delete all my data? Data deletion can happen in a couple different ways
If you delete your Google account - all of your submitted data will also be deleted and your position on the leaderboard will be anonymized.
When you explicitly request to delete your data from “Profile page", All your previous attempts (test and official) will be deleted. If your attempt has made it to the leaderboard, it will be anonymized.
In both cases, you will lose access to your data your nickname will be available for use again by someone else.
PrivacyTermsAbout GoogleGoogle Products