百度Deep Image论文被质疑过度使用ImageNet评价服务器【英】

Large Scale Visual Recognition Challenge (ILSVRC)

Date: June 2, 2015

Dear ILSVRC community,

This is a follow up to the announcement on May 19, 2015 with some more details and the status of the test server.

During the period of November 28th, 2014 to May 13th, 2015, there were at least 30 accounts used by a team from Baidu to submit to the test server at least 200 times, far exceeding the specified limit of two submissions per week. This includes short periods of very high usage, for example with more than 40 submissions over 5 days from March 15th, 2015 to March 19th, 2015. Figure A below shows submissions from ImageNet accounts known to be associated with the team in question. Figure B shows a comparison to the activity from all other accounts.

Figure A

Figure B

The results obtained during this period are reported in a recent arXiv paper. Because of the violation of the regulations of the test server, these results may not be directly comparable to results obtained and reported by other teams. To make this clear, by exploiting the ability to test many slightly different solutions on the test server it is possible to 1) select the best out of a set of very similar solutions based on test performance and achieve a small but potentially significant advantage and 2) choose methods for further research and development based directly on the test data instead of using only the training and validation data for such choices.

We noticed these irregularities on May 14th, 2015. The authors of the paper were notified on May 17th, 2015. On May 22nd, 2015, upon further discussion and consultation with senior advisors in our research community, we informed the authors that

  1. Their results obtained from the ImageNet test server are not directly comparable to results obtained by others, and,
  2. There is also concern about any new submission from Baidu that builds on top of these results. This includes submissions to all of the ILSVRC challenge tasks (classification, localization, and detection) as parts of the datasets are shared. We therefore requested that they refrain from submitting to the evaluation server or the challenge for the next 12 months.

We are in communication with the team involved in this incident. They have asked us to forward the message included further below to the community. Note that in order to stay neutral and independent, we have not and cannot work with any team to interpret or review results.

The test server is now back online. We look forward to continued progress in visual recognition!

The ILSVRC organizers


Message from the team in question:

Dear ILSVRC community,

Recently the ILSVRC organizers contacted the Heterogeneous Computing team to inform us that we exceeded the allowable number of weekly submissions to the ImageNet servers (~ 200 submissions during the lifespan of our project).

We apologize for this mistake and are continuing to review the results. We have added a note to our research paper, Deep Image: Scaling up Image Recognition, and will continue to provide relevant updates as we learn more.

We are staunch supporters of fairness and transparency in the ImageNet Challenge and are committed to the integrity of the scientific process.

Ren Wu – Baidu Heterogeneous Computing Team


微博相关评论:

@龙星镖局:首先,我相信度厂的技术是没问题的,然后提交多次对最终结果影响有多大,这个其实很看参与人的功底的,功力越深厚影响越大,对于一般人,你提交多次,可能你也调不出多好的效果。大家同意不?

@梁斌penny:百度这次的情况,我虽然还没有完全掌握,但觉得去参加比赛这件事很奇怪。公司不是应该服务社会,服务群众嘛?参加比赛是证明自己牛逼,这是厂职工和就业预备队的需求啊。。百度还需要证明自己牛逼嘛?手上聚集了国内最优秀的人才。。


原文发布于微信公众号 - 人工智能头条(AI_Thinker)

原文发表时间:2015-06-02

本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏大数据

有向无环图检测

01 — Spark背景介绍 Apache Spark 是专为大规模数据处理而设计的快速通用的计算引擎。Spark 是一种与 Hadoop 相似的开源集群计算环...

4107
来自专栏华章科技

你真的懂数据分析吗?一文读懂数据分析的流程、基本方法和实践

导读:无论你的工作内容是什么,掌握一定的数据分析能力,都可以帮你更好的认识世界,更好的提升工作效率。数据分析除了包含传统意义上的统计分析之外,也包含寻找有效特征...

1162
来自专栏携程技术中心

干货 | ElasticSearch相关性打分机制

作者简介 孙咸伟,后端开发一枚,在携程技术中心市场营销研发部负责“携程运动”项目的开发和维护。 携程运动是携程旗下新业务,主要给用户提供羽毛球、游泳等运动项目的...

1.4K8
来自专栏鹅厂优文

游戏人工智能 读书笔记 (四) AI算法简介——Ad-Hoc 行为编程

本书英文版: Artificial Intelligence and Games - A Springer Textbook

24110
来自专栏数说工作室

海量文本用 Simhash, 2小时变4秒! | 文本分析:大规模文本处理(2)

这是一个相似匹配的问题(文本相似匹配基础→ 词频与余弦相似度)。但是,亿级数据库,用传统的相似度计算方法太慢了,我们需要一个文本查询方法,可以快速的把一段文本的...

2783
来自专栏鸿的学习笔记

写给开发者的机器学习指南(九)

正如你所看到的,最高的权重给予了几乎立即得到电子邮件回复的电子邮件,而最低权重给予具有非常长的时间范围的电子邮件。这允许具有非常低频率的电子邮件仍然基于它们被发...

1011
来自专栏iOSDevLog

Scikit-Learn教程:棒球分析 (一)

一个scikit-learn教程,通过将数据建模到KMeans聚类模型和线性回归模型来预测MLB每赛季的胜利。

1772
来自专栏生信小驿站

主成分分析 factoextra

1023
来自专栏机器学习人工学weekly

机器学习人工学weekly-2018/9/23

Rosetta: Understanding text in images and videos with machine learning

985
来自专栏数据魔术师

运筹学教学 | 十分钟快速掌握最大流算法(附C++代码及算例)

—“运筹教科书到底能给你啥?” —“算法和实现离教科书有多远?” —“问题解决能力到底从哪来?” 今天刚起床就接到了BOSS的 提·问·三·连 小编表示 收到直...

4935

扫码关注云+社区