Q. Do I have to register to participate?
A. Yes. To participate in the
Challenge, participants need to register your team in the CMT system (to be announced). The CMT system will be also the venue for you to submit your prediciton results and papers.
Q. Can I compete even if I am not affiliated
with an academic or industrial lab?
Q. How do you evaluate the system?
A. The measurements are main page of this website. This year, we will do offline evaluation.
Q. Is this competition open to entrants
who live outside the United States?
A. Individuals who live in
Cuba, Iran, North Korea, Sudan, or Syria are not eligible to participate in the
competition. Individuals living in other countries or regions may participate.
Q. Who is not eligible to compete in the
A.The challenge is not open
- Entrants who are younger than 18 years of age;
- Residents of any of the following countries: Cuba, Iran, North Korea, Sudan, and
- Employees of Microsoft Corporation or an employee of a Microsoft subsidiary;
- Persons involved in any part of the administration and execution of the competition;
- Immediate family (parent, sibling, spouse, child) or household members of a Microsoft
employee, an employee of a Microsoft subsidiary, or a person involved in any part
of the administration and execution of this Contest.
Q. Why does the challenge include separate
training, development and the test datasets?
A. The test dataset is used to determine the winners
of the challenge. There, we use expert judgments to determine the retrieval quality. In contrast, the training
dataset is sampled from Bing's usage logs that contain only query, image and click information. Since clicked images
are not necessarily the most relevant to the queries, and not-clicked images are not necessarily irrelevant, the
challenge fpr the participants is to distill the large amount of click information to arrive at good relevant judgments.
To help training such a system, a development set created in the same manner as the test dataset is provided so that
participants can utilize the development set to refine their systems. Please see the "Datasets" section of the website
for more details.
Q. Why does the challenge restrict the
number of submissions on the final day of the competition?
A. The challenge will only accept
three final test entries from each team. One of these entries must be listed
as a primary, which is the only entry that is eligible to win a prize. The other two entries will be run against the final dataset to provide teams with feedback about
the quality, but they will not be eligible for prizes (even if
they outperform primary entries). We have restricted the number of submissions to avoid
giving an unfair advantage to teams with the ability to develop a large number of
Q. Why do I need to choose one of my submissions
as my "primary entry?" Can’t you just do that for me?
A. You are required to determine
which system is your primary entry to be considered
for the cash prizes. This limitation is designed to prevent participants with fewer
resources from being disadvantaged, since participants with more resources could
just provide three entries and get three chances at winning a prize. This also requires
teams to judge for themselves which system they think is best – and then see if
their intuition was correct.
We permit secondary submissions in order to enable teams to get feedback on more
than one system, but the secondary submissions are not eligible for any prizes
(regardless of whether they outperform other primary ones).
Q. Will you release the ground truth of the final Test
A. No. We will not. We will build a service to automatically evaluate prediction results submitted after the challenge.
Q. Who can I contact for other questions
on the Challenge?
A. You can contact us at firstname.lastname@example.org
or participate in the community discussion (see below).
Q. Is there a place where I can interact
with the participants of the Challenge?
A. Yes. We encourage participants
to use the discussion forum of the
Facebook and Twitter.