Competition

Prerequisites

1. Sign up for a Google Cloud Platform account at cloud.google.com. Few important points below
   1. If this is a new account, you will be following the sign up process which may require billing information.
   2. For the purpose of this competition, your GCP account will be used only for the storage of the container image(s), which allows you to maintain all ownership of your container image(s). The compute cost of model evaluation will not be charged to your GCP account. You can start with free credit, if not already exhausted.
   3. Post-model evaluation, canceling your Google cloud billing will not eliminate your leaderboard entries or scores.
2. Install Cloud SDK. This is needed to interact with Google Cloud Project using command-line tools such as gcloud.
3. Install Docker. This is needed to build a Docker image for your model.

Building your Docker image

You must submit your model as a Docker image. You are allowed to use whatever software dependencies you want, but those dependencies must be included in your Docker image. Let's say your model is a Tensorflow model. For this, you can use the official tensorflow Docker container as the base container image:

FROM tensorflow/tensorflow:latest
ADD conceptual-captions /conceptual-captions/

The first line of this Dockerfile says to use the official Tensorflow Docker image as the starting point of the image. The second line says to add the contents of a directory called "conceptual-captions" to a folder called "/conceptual-captions" inside the image. Read the Docker manual for more details on how to use Dockerfiles.

The folder "/conceptual-captions" is expected to contain a script called "submission.sh". The submission.sh script should call your model to get captions for each input image, and write them to file. Here is an example submission.sh.

#!/bin/bash
#
# submission.sh: The script to be launched in the Docker image.
#
# Usage: submission.sh <image_basename_file> <image_dir> <output_file>
#   image_basename_file: Path to file containing image file basenames (e.g. "123.jpg"), one per line
#   image_dir: Path to directory containing image files (e.g. "/tmp/image_dir")
#   output_file: Path to file where image captions should be written, one per line
#
# Sample usage:
#   submission.sh images.txt image_dir captions.txt

IMAGE_BASENAME_FILE=$1
IMAGE_DIR=$2
OUTPUT_FILE=$3

for image in $(cat $IMAGE_BASENAME_FILE); do
  # YOUR CODE HERE!
  #
  # For example:
  #   python generate_caption.py --image_path=$IMAGE_DIR/$image >> $OUTPUT_FILE
  echo "dummy caption" >> $OUTPUT_FILE
done

If you have configured your Docker image correctly, you should be able to run docker build to create your Docker image. Example:

$ docker build .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM tensorflow/tensorflow:latest
latest: Pulling from tensorflow/tensorflow
297061f60c36: Pull complete
e9ccef17b516: Pull complete
dbc33716854d: Pull complete
8fe36b178d25: Pull complete
686596545a94: Pull complete
ed66f2c5f3d9: Pull complete
8405b6c3f141: Pull complete
070615ca3a03: Pull complete
306ac2321f8e: Pull complete
c30111bc1e74: Pull complete
7aa552c3f7f7: Pull complete
4db41af3662a: Pull complete
bf5fbadacf01: Pull complete
Digest: sha256:1cc84937252fcc6e8521901cbbe180c7a93300792aceb0da8a53a5728360320a
Status: Downloaded newer image for tensorflow/tensorflow:latest
 ---> e7a53807ee54
Step 2/2 : ADD src /conceptual-captions/
 ---> 60f7545b8562
Successfully built 60f7545b8562

To help you get started, we have created a skeleton you can use as a starting point. See the official GitHub page for more details.

Testing your Docker image

Once you have built your Docker image, you can test it against a small chunk of test data using the following command:

docker run -a stdin -a stdout -a stderr <DOCKER_IMAGE_ID> \
    bash /conceptual-captions/submission.sh \
      /conceptual-captions/test_images.txt \
      /conceptual-captions/test_images \
      /dev/stdout

Where <DOCKER_IMAGE_ID> is the hex identifier of your image, as printed by docker build (60f7545b8562 in the example above).

You should see exactly one caption printed to /dev/stdout per image in test.images.tsv.

Pushing your Docker image

Once you are able to build your Docker image, you must push it to Google Container Registry.

1. Create your Google Cloud Platform project if you haven't already by visiting the console.
2. Obtain the numeric PROJECT_ID for your Google Cloud Platform project. This should be visible from your project's homepage.
3. Build and push to Google Container Registry:

gcloud auth login
gcloud config set project [PROJECT_ID]
gcloud builds submit --tag gcr.io/[PROJECT_ID]/my-submission .

Take note of the value of the --tag argument. You will use that when you create your submission.

Note: Read the quickstart guide for more details on how to use Google Container Registry.

Grant Access to our Service Account

One last step is required. You must grant our service account read access to your project's Docker images. Otherwise your Docker images will be private and we won't be able to run them.

1. Go to the storage tab in the gcloud console.
2. Locate the artifacts bucket. It should have a name like artifacts.<project-name >.appspot.com
3. Click the dropdown for the bucket and select "Edit Bucket Permissions".
4. Grant Storage Object Viewer permissions to the following user: mljam-compute@mljam-205019.iam.gserviceaccount.com

Submitting a Model

Competitors will be allowed to have first two submissions one week apart (but not sooner). After that, they will be allowed one submission per month.

See FAQ for more details.

Each submission must process all test images in under 24 hours or they will be considered failed. The execution environment has 52GB available memory and limited disk. Exceeding the amount of storage on the system will be considered a failed submission.