Q. Do I have to register to participate?
A. Yes. To participate in the
Challenge, participants need to register using their Microsoft Account.
Q. Can I compete even if I am not affiliated
with an academic or industrial lab?
Q. How do you evaluate the system?
A. The measurements are explained in the "Rules" section
of this website.
Q. Is this competition open to entrants
who live outside the United States?
A. Individuals who live in
Cuba, Iran, North Korea, Sudan, or Syria are not eligible to participate in the
competition. Individuals living in other countries or regions may participate.
Q. Who is not eligible to compete in the
A.The challenge is not open
- Entrants who are younger than 18 years of age;
- Residents of any of the following countries: Cuba, Iran, North Korea, Sudan, and
- Employees of Microsoft Corporation or an employee of a Microsoft subsidiary;
- Persons involved in any part of the administration and execution of the competition;
- Immediate family (parent, sibling, spouse, child) or household members of a Microsoft
employee, an employee of a Microsoft subsidiary, or a person involved in any part
of the administration and execution of this Contest.
Q. Why does the challenge include separate
training, development and the test datasets?
A. The test dataset is used to determine the winners
of the challenge. There, we use expert judgments to determine the retrieval quality. In contrast, the training
dataset is sampled from Bing's usage logs that contain only query, image and click information. Since clicked images
are not necessarily the most relevant to the queries, and not-clicked images are not necessarily irrelevant, the
challenge fpr the participants is to distill the large amount of click information to arrive at good relevant judgments.
To help training such a system, a development set created in the same manner as the test dataset is provided so that
participants can utilize the development set to refine their systems. Please see the "Datasets" section of the website
for more details.
Q. Why does the challenge restrict the
number of submissions on the final day of the competition (June 17, 2013)?
A. Microsoft will only accept
three final test entries from each team, as determined by the entries listed on your
team site as of 07:59 AM PDT on June 17, 2013. One of these entries must be listed
as a primary, which is the only entry that is eligible to win a prize. The secondary
entries will be run against the final dataset to provide teams with feedback about
the quality, but they will not be eligible for prizes (even if
they outperform primary entries). The restriction and timing limits the load on
our servers. In addition, we have restricted the number of submissions to avoid
giving an unfair advantage to teams with the ability to develop a large number of
Prior to June 17th, you may “submit” an unlimited number of test runs for your speller(s)
to see results against the sample dataset and improve your speller.
Q. Why do I need to choose one of my submissions
as my "primary entry?" Can’t Microsoft just do that for me?
A. You are required to determine
which system is your primary entry to be considered
for the cash prizes. This limitation is designed to prevent participants with fewer
resources from being disadvantaged, since participants with more resources could
just provide three entries and get three chances at winning a prize. This also requires
teams to judge for themselves which system they think is best – and then see if
their intuition was correct.
We permit secondary submissions in order to enable teams to get feedback on more
than one system, but the secondary submissions are not eligible for any prizes
(regardless of whether they outperform other primary ones).
Q. Will you release the final Bing Test
A. No. We will not release
the Bing Test Dataset. However, as soon as the winners are determined, the website will open to
accept more submissions indefinitely for future researchers who would like to benchmark against this
Q. What if the website doesn’t work, or
seems to be responding slowly?
A. Send email to
MSRBingChallenge@microsoft.com or check community announcements in the social
media (see below).
Q. Where can I find the requirements for
building an entry?
A. To view the Challenge
Rules, please click on the "Rules" tab at the top of this page.
Q. Where can I find the Official Contest
A. To view the full Official
Rules of the Contest go to Rules section of this website,
go to the Registration page at My team tab.
Q. Who can I contact for other questions
on the Challenge?
A. You can contact us at MSRBingChallenge@microsoft.com
or participate in the community discussion (see below).
Q. Is there a place where I can interact
with the participants of the Challenge?
A. Yes. We encourage participants
to use the discussion forum of the
Facebook and Twitter.
Q. When will the winners be announced?
A. The winners will be announced
after Microsoft receives and validates the winning entries. The official results
are expected to be announced in June 2013.