MSR-Bing Image Retrieval Challenge

Microsoft Research in partnership with Bing is happy to launch the second MSR-Bing Challenge on Image Retrieval

Do you have what it takes to build the best image retrieval system? Enter the MSR-Bing Image Retrieval Challenge in ACM Multimedia and/or ICME to develop an image scoring system for a search query.

In doing so, you can:

  • Try out your image retrieval system using real world data;
  • See how it compares to the rest of the community‚Äôs entries;
  • Get to be a contender for MSR-Bing Image Retrieval Challenge;
  • Be eligible to win a cash prize.

***IMPORTANT NOTICE (Last update: April 7, 2014)***

Update History

April 7: Password-protected test set is avaliable for downloading. Please check your email (sent from CMT)(New!).

April 1: Trial set is available for download (see details below) (New!)

Mar 27: CMT System is up

Mar 24: Announce evaluation rules

To enable more team to participant the grand challenge, we will change the evaluation method from online to offline:

You will be asked to download one compressed file (evaluation set) which contains two files in text formats. One is a list of key-query pairs, and the other is a list of key-image pairs. You will be running your algorithm to give a relevance score for each pairs in the first file, and the image content, which will be Base64 encoded JPEG files, can be found in the second file through the key.

The evaluation set, which is encrypted, will be available for downloading 3 days before the challenge starts. A password will be delivered to all participants to decrypt the evaluation set when the challenge starts.

One full day (24 hours) will be given to the participants to do predicitons on all the query-image pairs in the evaluation set.Before the end of the challenge, participants need to submit the evaluation results (which is a TSV file containing a list of triads: key, query, score) to a CMT system (see below). The order of the triads in the text file is not important. Running prediction in parallel is encouraged.

The evaluation set will be different from what we used last year. The number of query-image pairs will be increased significantly this time.A trial set will be available around one week before the start of the challenge.

One team can submit up to 3 runs in 3 zipped text files, and each file corresponds to the results of one run.

Trial Dataset Download (New!)

1. The trial dataset can be downloaded here (13.6M).

2. It is a .tar file. If you can untar it successully, you should get the file correctly.

3. After untar, you will get a zip file that is protected by a password.

4. The password will be sent to registered users via emails.

5. We will follow the sampe process to deliver final test set.

6. You can try uploading your prediction results and then email to confirm the submission format.

Important Dates (Updated)

April 1: Trial set available for download.

April 2: Deadline for the participants to register on the CMT system for submitting evaluation results.

April 7: Encrypted evaluation set is available for downloading.

April 10: Challenge start 0:00AM PDT, password will be delivered through emails registered in the CMT system.

April 11: Challenge ends 0:00AM PDT. Submission system closes.

April 16: Challenge paper submission deadline (4-6 pages following the same guideline of the main conference).

April 23: Notification.

April 30: Camera ready.

Prediction Result Submission Entrance:

Participants are required to register their teams by April 2.

Please note the prediction submission site share the same main entrance with ICME 2014 workshop papers. Please choose "MSR-Bing Image Retrevial Challenge" to submit you results (by the end of April 10) and papers (by the end of April 16.

Useful Links

Past: 2013 Grand Challenge Result (with 1M datasets)

Current: 2014 Grand Challenge @ ICME (with 1M dataset)

Next: 2014 Grand Challenge @ ACM Multimedia (with 1M dataset and optionally 40M dataset)

Dataset Download