User FAQ


How do I make sure that I am submitting a valid container model? If you follow the instructions in How to Participate section, there are high chances that you will end up with the correct container. You can also
  • Submit test attempts or
  • Test your container locally on your machine using the instructions provided.

What is the difference between test attempt and official attempt? Official attempts have restrictions on frequency of submissions. This is to promote healthy competition, prevent “Hill Climb” and maintain the test set integrity. So, when you are submitting an official attempt, you want to make sure that docker image and other items are valid.
Test attempt or submission is only for validation of your docker image. We run the submissions against a very small test set and they are not scored. Test attempts are shown with the in “Mydashboard” page and limited to last 10.
Will my Google Cloud account be billed for the compute cost of running my submissions? No. We will run all the submissions under Google internal quotas, and no billing will be incurred to your Google Cloud account for compute cost. However, you are responsible for the storing costs of your files in your Google Cloud account (if any, as there are free options for new accounts, etc.).
I have submitted my attempt, how frequently I can submit further attempts?

Following table will help understand the frequency of official attempts that you can submit.

AttemptMinimum Interval
Between 1st and 2nd7 days
Beyond 2nd30 days

For how long will my docker image be allowed to run to produce the test outputs? Any official submission will be allowed to run for a maximum of 24h. If, at the end of that time, there are not enough outputs created, the run is terminated and the submission is considered unsuccessful.
Will I get notified via email or otherwise if my submission produces errors? No. Please check back for status updates on the submission page.
I have submitted my attempt, it’s been a while and still showing status as “Pending” Depending on the complexity of the model you submitted, it can take a few hours (we allow up to 24h) to complete the evaluation process. If you know your submission should not take this long but it seems that it does, you can email us at conceptual-captions@google.com.
I think the scores for my model are not accurate. How is my model scored? Official submission output captions are scored against the reference captions from the hidden test set using this implementation of the CIDEr (primary), ROUGE-L and SPICE metrics. If you have further questions, please contact us at conceptual-captions@google.com.
Are there different versions of dataset? Yes. The last version mentioned below is the current one.
Version 1.0: The numbers published in the original paper are based on this version.
Version 1.1: The Validation and Test sets have changed (they are a subset of the v1.0 version, as we eliminated some examples that were too close to training examples). The Training set is the same as in v1.0.
I would like to not show my submission on the leaderboard, is that possible? For a healthy competition, we do not allow on-demand removal of submissions from the leaderboard.
I submitted a model, however I would like to withdraw that attempt. How do I do that? This is not supported. Once you make a submission, there is no way to cancel or prevent it from showing up on the leaderboard. To test for errors, please use test attempts. Test attempts are not shown in the leaderboard.
How long will my data be kept in MLJam? Is my data shared with any other Google products? Your submissions and profile data is kept as long as you have an active profile. Your profile data, as well as your submissions are not shared with any other Google product. We may contact you for similar competitions if you opt in for it on your profile page.
Is it ok to publish my results on social sites and in blogs? Yes! We encourage you to brag about your position on the leaderboard on social media, blogs, etc. We also encourage you to use the leaderbord results and metrics for your academic publications.
How do I delete all my data? Data deletion can happen in a couple different ways
  • When you If you delete your Google account - your MLJam data will also be deleted and your position on the leaderboard will be anonymized.
  • When you explicitly request to delete your data from “Profile page", All your previous attempts (test and official) will be deleted. If your attempt has made it to the leaderboard, it will be anonymized.
In both cases, you will lose access to your data your nickname will be available for use again by someone else.
PrivacyTermsAbout GoogleGoogle Products