百度Deep Image论文被质疑过度使用ImageNet评价服务器【英】

Large Scale Visual Recognition Challenge (ILSVRC)

Date: June 2, 2015

Dear ILSVRC community,

This is a follow up to the announcement on May 19, 2015 with some more details and the status of the test server.

During the period of November 28th, 2014 to May 13th, 2015, there were at least 30 accounts used by a team from Baidu to submit to the test server at least 200 times, far exceeding the specified limit of two submissions per week. This includes short periods of very high usage, for example with more than 40 submissions over 5 days from March 15th, 2015 to March 19th, 2015. Figure A below shows submissions from ImageNet accounts known to be associated with the team in question. Figure B shows a comparison to the activity from all other accounts.

The results obtained during this period are reported in a recent arXiv paper. Because of the violation of the regulations of the test server, these results may not be directly comparable to results obtained and reported by other teams. To make this clear, by exploiting the ability to test many slightly different solutions on the test server it is possible to 1) select the best out of a set of very similar solutions based on test performance and achieve a small but potentially significant advantage and 2) choose methods for further research and development based directly on the test data instead of using only the training and validation data for such choices.

We noticed these irregularities on May 14th, 2015. The authors of the paper were notified on May 17th, 2015. On May 22nd, 2015, upon further discussion and consultation with senior advisors in our research community, we informed the authors that

1. Their results obtained from the ImageNet test server are not directly comparable to results obtained by others, and,
2. There is also concern about any new submission from Baidu that builds on top of these results. This includes submissions to all of the ILSVRC challenge tasks (classification, localization, and detection) as parts of the datasets are shared. We therefore requested that they refrain from submitting to the evaluation server or the challenge for the next 12 months.

We are in communication with the team involved in this incident. They have asked us to forward the message included further below to the community. Note that in order to stay neutral and independent, we have not and cannot work with any team to interpret or review results.

The test server is now back online. We look forward to continued progress in visual recognition!

The ILSVRC organizers

Message from the team in question:

Dear ILSVRC community,

Recently the ILSVRC organizers contacted the Heterogeneous Computing team to inform us that we exceeded the allowable number of weekly submissions to the ImageNet servers (~ 200 submissions during the lifespan of our project).

We apologize for this mistake and are continuing to review the results. We have added a note to our research paper, Deep Image: Scaling up Image Recognition, and will continue to provide relevant updates as we learn more.

We are staunch supporters of fairness and transparency in the ImageNet Challenge and are committed to the integrity of the scientific process.

Ren Wu – Baidu Heterogeneous Computing Team

微博相关评论：

@龙星镖局：首先，我相信度厂的技术是没问题的，然后提交多次对最终结果影响有多大，这个其实很看参与人的功底的，功力越深厚影响越大，对于一般人，你提交多次，可能你也调不出多好的效果。大家同意不？

@梁斌penny：百度这次的情况，我虽然还没有完全掌握，但觉得去参加比赛这件事很奇怪。公司不是应该服务社会，服务群众嘛？参加比赛是证明自己牛逼，这是厂职工和就业预备队的需求啊。。百度还需要证明自己牛逼嘛？手上聚集了国内最优秀的人才。。