The EfficientQA leaderboard displays performance on an unreleased test set. To evaluate your system on this test set, you will need to submit an attempt on this page. We have given an overview of the submission process below, and there are complete walk-through tutorials on the EfficientQA competition site. If you have any problems making a submission, please reach out to us at nqopen@google.com.

Your user profile, and all of your submissions, will be associated with the Google account that you used to log in. You can delete your data at any time by emailing us at nqopen@google.com from the Google account that you used to sign up.

To create a submission you will need to upload a Docker image to the Google Container Registry, and then choose it as your submission using the form above. You can make one official submission every three days, and this will be shown on the leaderboard. You can make as many test attempts as you like, to validate your submission. Test attempts are run on 100 examples sampled from the EfficientQA development set, and are only visible to you.

Your submission will be run on a GCloud instance of type n1-highmem-16 that has access to two Nvidia Tesla P100 GPUs. Test attempts are given 20 minutes to run, and official attempts are given 6 hours.

Below, we've given a high level overview of the submission process. However, if you do not have experience with Docker and GCP, the easiest way to get started is by following one of the complete examples that we have provided on the EfficientQA competition site.

Prerequisites

You will need to Google Cloud Platform account at cloud.google.com. to store your submissions. While storage costs should be negligible, we encourage you to use Google Cloud's free credits. You will not be charged for the cost of running your submission through this site.

Once you have an account, you need to create a project for your submissions in your console, and you need to give us permission to access this project as follows:

1. Go to the storage tab in the your console.
2. Locate the artifacts bucket. It should have a name like artifacts.<project-name>.appspot.com.
3. From this bucket's drop-down menu, select "Edit bucket permissions".
4. Grant "Storage Object Viewer" permissions to mljam-compute@mljam-205019.iam.gserviceaccount.com. You can find this option under "Storage" when selecting roles for new members.

Creating a submission

You should create a Docker image that contains an executable script submission.sh in its root directory, that will be run using the following command:

/submission.sh <path_to_input_file> <output_file>

where <input_file> is a file that contains one JSON input example per line:

{"question": "who won the women's australian open 2018"}
{"question": "kuchipudi is a dance form of which state"}
{"question": "who did stephen amell play in private practice"}
{"question": "who created the chamber of secrets in harry potter"}

and you should write your predictions to <output_file> one per line, as follows:

{"question": "who won the women's australian open 2018", "prediction": "Caroline Wozniacki"}
{"question": "kuchipudi is a dance form of which state", "prediction": "Tamil Nadu"}
{"question": "who did stephen amell play in private practice", "prediction": "a pedestrian"}
{"question": "who created the chamber of secrets in harry potter", "prediction": "the Heir of Salazar Slytherin"}.