The Natural Questions leaderboard displays performance of submissions on a blind test set. To evaluate your system on this test set, you will need to upload it to this site. Instructions for doing so are given below. We also provide a script that lets you test your upload locally on NQ's GitHub site, and we have provided a complete walkthrough of creating an upload for the BERT-joint baseline below.

If you have any issues making a submission, or have any questions, please reach out to us at natural-questions@google.com.

How to Participate

To create a submission you will need to upload a Docker image to the Google Container Registry, and then choose it as your submission using the form above. You are limited to one official attempt per week, but you can make as many test attempts as you like. Test attempts are run on the 200 example development set sample, and they let you ensure that your submission is running properly.

We have removed all answer annotations from the test set input examples, so check that your code does not expect them. Your submission will be run on a GCloud instance of type n1-highmem-16 that has access to two Nvidia Tesla V100 GPUs. Test attempts are given 20 minutes to run, and official attempts are given 24 hours.

Prerequisites

First, you will need to Google Cloud Platform account at cloud.google.com. This account will be used to store your submissions and, while storage costs should be negligible, we encourage all participants to make use of Google Cloud's free credits. We will not charge you for the cost of running your model through this site.

Once you have a Google Cloud account, you will need to create a project for your Natural Questions submissions in your console, and you will need to give us permission to access this project as follows:

1. Go to the storage tab in the your console.
2. Locate the artifacts bucket. It should have a name like artifacts.<project-name>.appspot.com.
3. From this bucket's drop-down menu, select "Edit bucket permissions".
4. Grant "Storage Object Viewer" permissions to mljam-compute@mljam-205019.iam.gserviceaccount.com. You can find this option under "Storage" when selecting roles for new members.

You should also install the Cloud SDK, which will let you interact with Google Cloud Project using command-line tools.

If you want to build and test your submissions locally, you should also install Docker, and optionally nvidida-docker if you want to use a GPU. All submissions will be built and run with Nvidia support, by default.

Creating a Submission

Your model will be run in a Docker container. The only assumption we make is that this Docker container contains a script /nq_model/submission.sh that will be run as follows:

/nq_model/submission.sh <path_to_input_file> <path_to_output_file>

where <path_to_input_file> points to the test set in gzipped jsonlines, and <path_to_output_file> is where /nq_model/submission.sh will write predictions in the json format used by NQ's eval script.

We provide a test script that will build and run your Docker, and then run NQ's eval script on the output predictions. This should let you troubleshoot most issues locally, before uploading your model to the cloud. We also provide detailed instructions for creating a submission below.

If you are sure that your Docker image is running correctly, then you will need to build it in the cloud as follows:

gcloud auth login
  gcloud config set project <your_project_id>
  gcloud builds submit --tag gcr.io/<your_project_id>/<submission_name> .

and once this is successful, you can submit this image by entering gcr.io/<your_project_id>/<submission_name> in the form above. Remember that you can find your project ID on your Cloud console. You can retrieve the names of all of your images by running the command below.

gcloud container images list --repository="gcr.io/<your_project_id>"

Example Submission

If you are not familiar with Docker, and the Google Cloud Platform, the easiest way to learn how to make a submission is by following an example. Here, we give a walk-through of building and uploading a submission for the BERT-joint baseline which you can download as follows:

git clone https://github.com/google-research/language.git

We will create the Docker image in the bert_joint directory. First, we will copy the model's parameters to a model subdirectory:

cd language/language/question_answering/bert_joint
  mkdir -p "model"
  gsutil cp "gs://bert-nq/bert-joint-baseline/bert_config.json" "model/"
  gsutil cp "gs://bert-nq/bert-joint-baseline/vocab-nq.txt" "model/"
  gsutil cp "gs://bert-nq/bert-joint-baseline/bert_joint.ckpt.data-00000-of-00001" "model/"
  gsutil cp "gs://bert-nq/bert-joint-baseline/bert_joint.ckpt.index" "model/"

and then we create a minimal Dockerfile that uses a tensorflow Docker container as the base image. This Dockerfile installs two python dependencies, and then simply adds everything under the bert_joint directory to the container's /nq_model directory.

cat <<'HERE' | tee Dockerfile
# Docker file for submission of the BERT-joint baseline to the Natural Questions
# competition site: https://ai.google.com/research/NaturalQuestions/competition.

ARG TF_VERSION=latest-gpu

FROM tensorflow/tensorflow:$TF_VERSION

# Install the BERT and Natural Questions libraries.
RUN pip install --trusted-host pypi.python.org bert-tensorflow
RUN pip install --trusted-host pypi.python.org natural-questions

# Add everything in the current directory to a /nq_model directory in the
# Docker container.
ADD . /nq_model
HERE

Now we know that the Docker container will contain the model in its /nq_model/model/ directory. Recall from above that the Docker container must also contain a script at /nq_model/submission.sh. Here, we create a submission.sh script for BERT-joint. All parameters apart from the input and output paths must be hard coded within this script.

cat <<'HERE' | tee submission.sh
#!/bin/bash
#
# submission.sh: The script to be launched in the Docker image.
#
# Usage: submission.sh <input_data_pattern> <output_file>
#   input_data_pattern: jsonl.gz NQ evaluation files,
#   output_file: json file containing answer key produced by the model.
#
# Sample usage:
#   submission.sh nq-dev-0?.jsonl.gz predictions.json

set -e
set -x

INPUT_PATH=$1
OUTPUT_PATH=$2

cd /nq_model
python -m run_nq \
  --logtostderr \
  --bert_config_file="/nq_model/model/bert_config.json" \
  --vocab_file="/nq_model/model/vocab-nq.txt" \
  --init_checkpoint="/nq_model/model/bert_joint.ckpt" \
  --output_dir="/tmp" \
  --do_predict \
  --predict_file="$INPUT_PATH" \
  --output_prediction_file="$OUTPUT_PATH"
HERE

Now that we have a Dockerfile and our submission script, we can build the Docker image

IMAGE_NAME="nq-submission-test"
docker build --tag=$IMAGE_NAME .

and we can test it as follows (remove runtime=nvidia) if you are not using a GPU.

DATA_DIR="/tmp/nq-submission-test-data"
mkdir -p "$DATA_DIR"
gsutil cp -R "gs://bert-nq/tiny-dev" "$DATA_DIR"

docker run --runtime=nvidia -a stdin -a stdout -a stderr -v "$DATA_DIR":/data \
  "$IMAGE_NAME" bash "/nq_model/submission.sh" \
  "/data/tiny-dev/nq-dev-sample.no-annot.jsonl.gz" \
  "/data/predictions.json"

This creates a local temporary data directory, downloads the 200 example dev set sample that we provide for testing, and runs the docker container. The argument -v $DATA_DIR:/data mounts your temporary data directory to the /data directory within the container. Once you have run the container, you should be able to see your model's predictions at "$DATA_DIR/predictions.json". Check that they are as expected by running the evaluation script.

pip install --user natural-questions
python -m natural_questions.nq_eval \
  --gold_path="$DATA_DIR/tiny-dev/nq-dev-sample.jsonl.gz" \
  --predictions_path="$DATA_DIR/predictions.json"

Provided everything is running as expected, you can now create your submission as follows:

gcloud auth login
gcloud builds submit --tag "gcr.io/<your_project_id>/<submission_name> ."

and once this has completed, you will be able to submit your attempt by entering gcr.io/<your_project_id>/<submission_name> in the form above.