人工智能学术速递[6.21]

访问www.arxivdaily.com获取含摘要速递，涵盖CS|物理|数学|经济|统计|金融|生物|电气领域，更有搜索、收藏、发帖等功能！点击阅读原文即可访问

cs.AI人工智能，共计49篇

【1】 How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers 标题：如何训练你的VIT？视觉变换器中的数据、增强和正则化

作者：Andreas Steiner,Alexander Kolesnikov,Xiaohua Zhai,Ross Wightman,Jakob Uszkoreit,Lucas Beyer 机构：Google Research, Brain Team; †independent researcher 备注：Andreas, Alex, Xiaohua and Lucas contributed equally. We release more than 50'000 ViT models trained under diverse settings on various datasets. We believe this to be a treasure trove for model analysis. Available at this https URL and this https URL 链接：https://arxiv.org/abs/2106.10270 摘要：视觉变换器（ViT）在图像分类、目标检测和语义图像分割等领域具有很强的竞争力。与卷积神经网络相比，在较小的训练数据集上训练时，视觉变换器较弱的感应偏差通常会导致对模型正则化或数据增强（简称“AugReg”）的依赖性增加。为了更好地理解训练数据量、AugReg、模型大小和计算预算之间的相互作用，我们进行了系统的实证研究。作为这项研究的一个结果，我们发现，增加计算和AugReg的组合可以产生与在一个数量级以上的训练数据上训练的模型具有相同性能的模型：我们在公共ImageNet-21k数据集上训练各种大小的ViT模型，这些模型与在更大的数据集上训练的对应模型相匹配或优于它们，但JFT-300M数据集尚未公开。 摘要：Vision Transformers (ViT) have been shown to attain highly competitive performance for a wide range of vision applications, such as image classification, object detection and semantic image segmentation. In comparison to convolutional neural networks, the Vision Transformer's weaker inductive bias is generally found to cause an increased reliance on model regularization or data augmentation (``AugReg'' for short) when training on smaller training datasets. We conduct a systematic empirical study in order to better understand the interplay between the amount of training data, AugReg, model size and compute budget. As one result of this study we find that the combination of increased compute and AugReg can yield models with the same performance as models trained on an order of magnitude more training data: we train ViT models of various sizes on the public ImageNet-21k dataset which either match or outperform their counterparts trained on the larger, but not publicly available JFT-300M dataset.

【2】 MADE: Exploration via Maximizing Deviation from Explored Regions 标题：进行：通过最大限度地偏离勘探区域进行勘探

作者：Tianjun Zhang,Paria Rashidinejad,Jiantao Jiao,Yuandong Tian,Joseph Gonzalez,Stuart Russell 机构：† Department of Electrical Engineering and Computer Sciences, UC Berkeley, ‡ Department of Statistics, UC Berkeley, § Facebook AI Research 备注：28 pages, 10 figures 链接：https://arxiv.org/abs/2106.10268 摘要：在在线强化学习（RL）中，在报酬稀少的高维环境中，有效的探索仍然是一个特别具有挑战性的问题。在低维环境中，表格参数化是可能的，基于计数的置信上限（UCB）勘探方法可以获得接近最优速率的极大极小值。然而，如何在包含非线性函数逼近的实际RL任务中有效地实现UCB仍然是个未知数。为了解决这个问题，我们提出了一种新的探索方法，通过最大化下一个策略的占用率与探索区域的偏差。我们将此项作为自适应正则化器添加到标准RL目标中，以平衡勘探与开发。我们将新的目标与一个可证明收敛的算法配对，从而产生一个新的内在奖励来调整现有的奖金。所提出的内禀报酬算法易于实现，并与现有的RL算法相结合进行探索。作为概念证明，我们通过各种基于模型和无模型的算法对表格示例的新内在回报进行了评估，显示了对仅计数探索策略的改进。当在MiniGrid和DeepMind Control Suite的导航和移动任务上进行测试时，我们的方法比最新的方法显著提高了样本效率。我们的代码在https://github.com/tianjunz/MADE. 摘要：In online reinforcement learning (RL), efficient exploration remains particularly challenging in high-dimensional environments with sparse rewards. In low-dimensional environments, where tabular parameterization is possible, count-based upper confidence bound (UCB) exploration methods achieve minimax near-optimal rates. However, it remains unclear how to efficiently implement UCB in realistic RL tasks that involve non-linear function approximation. To address this, we propose a new exploration approach via \textit{maximizing} the deviation of the occupancy of the next policy from the explored regions. We add this term as an adaptive regularizer to the standard RL objective to balance exploration vs. exploitation. We pair the new objective with a provably convergent algorithm, giving rise to a new intrinsic reward that adjusts existing bonuses. The proposed intrinsic reward is easy to implement and combine with other existing RL algorithms to conduct exploration. As a proof of concept, we evaluate the new intrinsic reward on tabular examples across a variety of model-based and model-free algorithms, showing improvements over count-only exploration strategies. When tested on navigation and locomotion tasks from MiniGrid and DeepMind Control Suite benchmarks, our approach significantly improves sample efficiency over state-of-the-art methods. Our code is available at https://github.com/tianjunz/MADE.

【3】 Bridging the Gap Between Object Detection and User Intent via Query-Modulation 标题：通过查询调制弥合对象检测和用户意图之间的差距

作者：Marco Fornoni,Chaochao Yan,Liangchen Luo,Kimberly Wilber,Alex Stark,Yin Cui,Boqing Gong,Andrew Howard 机构：Google Research, University of Texas at Arlington 链接：https://arxiv.org/abs/2106.10258 摘要：当用户通过相机或图片与对象交互时，往往有特定的意图。例如，他们可能希望执行视觉搜索。然而，大多数目标检测模型忽略了用户的意图，依赖于图像像素作为其唯一的输入。这通常会导致不正确的结果，例如对感兴趣的对象缺乏高置信度检测，或者使用错误的类标签进行检测。在本文中，我们研究的技术，以调整标准对象检测器显式地解释用户的意图，表示为一个简单的查询嵌入。与标准对象检测器相比，查询调制检测器在检测给定感兴趣标签的对象时表现出更高的性能。由于从标准对象检测注释合成的大规模训练数据，查询调制检测器也可以优于专门的引用表达式识别系统。此外，它们可以同时训练来求解查询调制检测和标准目标检测。 摘要：When interacting with objects through cameras, or pictures, users often have a specific intent. For example, they may want to perform a visual search. However, most object detection models ignore the user intent, relying on image pixels as their only input. This often leads to incorrect results, such as lack of a high-confidence detection on the object of interest, or detection with a wrong class label. In this paper we investigate techniques to modulate standard object detectors to explicitly account for the user intent, expressed as an embedding of a simple query. Compared to standard object detectors, query-modulated detectors show superior performance at detecting objects for a given label of interest. Thanks to large-scale training data synthesized from standard object detection annotations, query-modulated detectors can also outperform specialized referring expression recognition systems. Furthermore, they can be simultaneously trained to solve for both query-modulated detection and standard object detection.

【4】 An Empirical Investigation into Deep and Shallow Rule Learning 标题：深层规则学习与浅层规则学习的实证研究

作者：Florian Beck,Johannes Fürnkranz 机构：Application-oriented Knowledge Processing (FAW), Department of Computer Science, Johannes Kepler University Linz, Austria 链接：https://arxiv.org/abs/2106.10254 摘要：归纳规则学习是机器学习中最传统的模式之一。尽管多年来我们在学习基于规则的理论方面取得了相当大的进步，但是所有最先进的学习者仍然学习直接将输入特征与目标概念联系起来的描述。在最简单的情况下，概念学习，这是一个析取范式（DNF）描述的积极类。很明显，从逻辑的角度来看，这是足够的，因为每个逻辑表达式都可以简化为一个等价的DNF表达式，然而，更结构化的表示形式，通过形成中间概念形成深层理论，可能更容易学习，与深度神经网络相比，深度神经网络的性能要优于浅层网络，尽管后者也是通用函数逼近器。在本文中，我们将深度和浅层规则学习与基于贪心小批量优化的统一通用算法进行了实证比较。我们在人工和真实基准数据上的实验表明，深规则网络的性能优于浅规则网络。 摘要：Inductive rule learning is arguably among the most traditional paradigms in machine learning. Although we have seen considerable progress over the years in learning rule-based theories, all state-of-the-art learners still learn descriptions that directly relate the input features to the target concept. In the simplest case, concept learning, this is a disjunctive normal form (DNF) description of the positive class. While it is clear that this is sufficient from a logical point of view because every logical expression can be reduced to an equivalent DNF expression, it could nevertheless be the case that more structured representations, which form deep theories by forming intermediate concepts, could be easier to learn, in very much the same way as deep neural networks are able to outperform shallow networks, even though the latter are also universal function approximators. In this paper, we empirically compare deep and shallow rule learning with a uniform general algorithm, which relies on greedy mini-batch based optimization. Our experiments on both artificial and real-world benchmark data indicate that deep rule networks outperform shallow networks.

【5】 Less is More: Feature Selection for Adversarial Robustness with Compressive Counter-Adversarial Attacks 标题：少即是多：利用压缩反对抗攻击实现对抗健壮性的特征选择

作者：Emre Ozfatura,Muhammad Zaid Hameed,Kerem Ozfatura,Deniz Gunduz 链接：https://arxiv.org/abs/2106.10252 摘要：关于对抗性攻击的一个常见观察是，它们主要导致倒数第二层的错误激活来欺骗分类器。假设这些激活值对应于输入的某些特征，目标就变成选择对分类最有用的特征。因此，我们提出了一种新的方法来识别重要的特征，采用对抗性攻击，这突出了倒数第二层对输入样本扰动的一致性。首先，我们从经验上证明了存在一个基于分类的特征子集，它弥补了干净和稳健精度之间的差距。其次，我们提出了一种简单而有效的机制，通过搜索输入样本的邻域来识别这些特征。然后通过观察倒数第二层激活值的一致性来选择特征。 摘要：A common observation regarding adversarial attacks is that they mostly give rise to false activation at the penultimate layer to fool the classifier. Assuming that these activation values correspond to certain features of the input, the objective becomes choosing the features that are most useful for classification. Hence, we propose a novel approach to identify the important features by employing counter-adversarial attacks, which highlights the consistency at the penultimate layer with respect to perturbations on input samples. First, we empirically show that there exist a subset of features, classification based in which bridge the gap between the clean and robust accuracy. Second, we propose a simple yet efficient mechanism to identify those features by searching the neighborhood of input sample. We then select features by observing the consistency of the activation values at the penultimate layer.

【6】 Active Offline Policy Selection 标题：活动脱机策略选择

作者：Ksenia Konyushkova,Yutian Chen,Thomas Paine,Caglar Gulcehre,Cosmin Paduraru,Daniel J Mankowitz,Misha Denil,Nando de Freitas 机构：DeepMind 链接：https://arxiv.org/abs/2106.10251 摘要：本文研究了具有大量日志数据，但交互开销非常有限的域中的策略选择问题。解决这个问题将使离线强化学习策略在工业、机器人和医疗保健等领域的安全评估和部署成为可能。已经提出了几种非策略评估（OPE）技术，仅使用记录的数据来评估策略的价值。然而，OPE的评价与真实环境下的完全在线评价相比还有很大差距。为了缩小这个差距，我们引入了一个新的\emph{active offline policy selection}问题公式，它结合了记录的数据和有限的在线交互来确定最佳策略。我们依靠OPE的进步来开始评估。我们建立在贝叶斯优化的基础上，迭代地决定要评估哪些策略，以便明智地利用有限的环境交互。许多候选策略可以被提出，因此，我们专注于使我们的方法具有可伸缩性，并引入一个核函数来模拟策略之间的相似性。我们使用了几个基准环境来表明，所提出的方法改进了最新的OPE估计和完全在线的政策评估，并且预算有限。此外，我们还证明了该方法的每个组成部分都是重要的，它适用于不同数量和质量的OPE估计，甚至适用于大量的候选策略。 摘要：This paper addresses the problem of policy selection in domains with abundant logged data, but with a very restricted interaction budget. Solving this problem would enable safe evaluation and deployment of offline reinforcement learning policies in industry, robotics, and healthcare domain among others. Several off-policy evaluation (OPE) techniques have been proposed to assess the value of policies using only logged data. However, there is still a big gap between the evaluation by OPE and the full online evaluation in the real environment. To reduce this gap, we introduce a novel \emph{active offline policy selection} problem formulation, which combined logged data and limited online interactions to identify the best policy. We rely on the advances in OPE to warm start the evaluation. We build upon Bayesian optimization to iteratively decide which policies to evaluate in order to utilize the limited environment interactions wisely. Many candidate policies could be proposed, thus, we focus on making our approach scalable and introduce a kernel function to model similarity between policies. We use several benchmark environments to show that the proposed approach improves upon state-of-the-art OPE estimates and fully online policy evaluation with limited budget. Additionally, we show that each component of the proposed method is important, it works well with various number and quality of OPE estimates and even with a large number of candidate policies.

【7】 A Coarse-to-Fine Instance Segmentation Network with Learning Boundary Representation 标题：一种具有学习边界表示的由粗到精的实例分割网络

作者：Feng Luo,Bin-Bin Gao,Jiangpeng Yan,Xiu Li 机构：Shenzhen International Graduate School, Tsinghua University, Shenzhen, China, Youtu Lab, Tencent, Shenzhen, China, Department of Automation, Tsinghua University, Beijing, China 备注：8 pages, Accepted by IJCNN 2021 链接：https://arxiv.org/abs/2106.10213 摘要：基于边界的实例分割以其高效的特点受到了广泛的关注。然而，现有的方法存在着长距离回归的困难。在本文中，我们提出了一个由粗到精的模块来解决这个问题。在粗化阶段生成近似的边界点，然后对这些点的特征进行采样，并将其输入到精化回归器中进行精细预测。由于差分采样操作在模块中得到很好的支持，因此它是端到端可训练的。此外，我们设计了一个整体的边界感知分支，并引入实例不可知监督来辅助回归。利用ResNet-101，通过单尺度训练和测试，我们的方法在COCO数据集上获得了31.7%的mask-AP，在附加参数和GFLOPs小于1%的情况下优于基线1.3%的mask-AP。实验还表明，与现有的基于边界的方法相比，该方法具有轻量级的设计和简单的流水线结构。 摘要：Boundary-based instance segmentation has drawn much attention since of its attractive efficiency. However, existing methods suffer from the difficulty in long-distance regression. In this paper, we propose a coarse-to-fine module to address the problem. Approximate boundary points are generated at the coarse stage and then features of these points are sampled and fed to a refined regressor for fine prediction. It is end-to-end trainable since differential sampling operation is well supported in the module. Furthermore, we design a holistic boundary-aware branch and introduce instance-agnostic supervision to assist regression. Equipped with ResNet-101, our approach achieves 31.7\% mask AP on COCO dataset with single-scale training and testing, outperforming the baseline 1.3\% mask AP with less than 1\% additional parameters and GFLOPs. Experiments also show that our proposed method achieves competitive performance compared to existing boundary-based methods with a lightweight design and a simple pipeline.

【8】 An Investigation into Mini-Batch Rule Learning 标题：关于小批量规则学习的研究

作者：Florian Beck,Johannes Fürnkranz 机构：Institute for Application-oriented Knowledge Processing (FAW), JKU Linz, Austria 备注：None 链接：https://arxiv.org/abs/2106.10202 摘要：我们研究是否有可能学习规则集有效地在一个网络结构与一个单一的隐藏层使用迭代细化在小批量的例子。第一个基本版本在除一个数据集之外的所有数据集上都显示了可接受的性能，尽管它还没有达到Ripper的性能级别。 摘要：We investigate whether it is possible to learn rule sets efficiently in a network structure with a single hidden layer using iterative refinements over mini-batches of examples. A first rudimentary version shows an acceptable performance on all but one dataset, even though it does not yet reach the performance levels of Ripper.

【9】 A Dynamic Spatial-temporal Attention Network for Early Anticipation of Traffic Accidents 标题：一种交通事故早期预测的动态时空注意网络

作者：Muhammad Monjurul Karim,Yu Li,Ruwen Qin,Zhaozheng Yin 机构： Stony Brook University, Zhaozheng Yin is with the Department of Biomedical Informatics, and AI Institute 备注：10 pages, 4 figures, submitted to a journal 链接：https://arxiv.org/abs/2106.10197 摘要：最近，自动驾驶车辆和配备高级驾驶员辅助系统（ADAS）的车辆正在出现。它们与完全由人类驾驶的常规车辆共用一条路。为了确保乘客和其他道路使用者的安全，自动驾驶车辆和自动驾驶辅助系统必须从自然驾驶场景中预测交通事故。交通代理的动态时空交互是复杂的，用于预测未来事故的视觉线索深深嵌入到仪表盘视频数据中。因此，交通事故的早期预测仍然是一个挑战。为此，本文提出了一种基于动态时空注意力（DSTA）的交通事故预警网络。提出的DSTA网络通过一个名为动态时间注意（DTA）的模块学习选择视频序列中有区别的时间段。它还通过另一个名为动态空间注意（DSA）的模块学习如何关注帧的信息空间区域。利用选通递归单元（GRU）网络联合学习事故的时空关系特征和场景外观特征。在两个基准数据集上对DSTA网络的实验评估证实，它已经超过了最先进的性能。一项彻底的消融研究评估了DSTA网络的各个组成部分的贡献，揭示了网络是如何实现这种性能的。此外，本文还提出了一种融合两个互补模型预测得分的新策略，并验证了该策略在进一步提高早期事故预测性能方面的有效性。 摘要：Recently, autonomous vehicles and those equipped with an Advanced Driver Assistance System (ADAS) are emerging. They share the road with regular ones operated by human drivers entirely. To ensure guaranteed safety for passengers and other road users, it becomes essential for autonomous vehicles and ADAS to anticipate traffic accidents from natural driving scenes. The dynamic spatial-temporal interaction of the traffic agents is complex, and visual cues for predicting a future accident are embedded deeply in dashcam video data. Therefore, early anticipation of traffic accidents remains a challenge. To this end, the paper presents a dynamic spatial-temporal attention (DSTA) network for early anticipation of traffic accidents from dashcam videos. The proposed DSTA-network learns to select discriminative temporal segments of a video sequence with a module named Dynamic Temporal Attention (DTA). It also learns to focus on the informative spatial regions of frames with another module named Dynamic Spatial Attention (DSA). The spatial-temporal relational features of accidents, along with scene appearance features, are learned jointly with a Gated Recurrent Unit (GRU) network. The experimental evaluation of the DSTA-network on two benchmark datasets confirms that it has exceeded the state-of-the-art performance. A thorough ablation study evaluates the contributions of individual components of the DSTA-network, revealing how the network achieves such performance. Furthermore, this paper proposes a new strategy that fuses the prediction scores from two complementary models and verifies its effectiveness in further boosting the performance of early accident anticipation.

【10】 Equilibrium Design for Concurrent Games 标题：并发博弈的均衡设计

作者：Julian Gutierrez,Muhammad Najib,Giuseppe Perelli,Michael Wooldridge 机构：Department of Computer Science, University of Oxford, Department of Computer Science, University of Göteborg 备注：None 链接：https://arxiv.org/abs/2106.10192 摘要：在博弈论中，机制设计关注的是激励机制的设计，以达到期望的博弈结果。在本文中，我们研究了激励的设计，从而得到一个理想的均衡，例如，一个满足给定时序逻辑性质的均衡，我们称之为均衡设计。我们的研究建立在这样一个框架之上：系统规范被表示为时序逻辑公式，博弈被表示为定量的并发博弈结构，参与者的目标被表示为平均收益目标。特别地，我们考虑了由LTL和GR（1）公式给出的系统规范，并证明了实现一种机制，以确保在博弈的某些/每个纳什均衡上满足给定的时序逻辑性质，无论这种机制何时存在，可以在PSPACE中完成LTL属性，在NP/$\Sigma^{P}{2}$中完成GR（1）规范。我们还研究了各种相关的决策和优化问题的复杂性，如解的最优性和唯一性，并证明了所有这些问题的复杂性都在多项式层次内。作为一个应用，均衡设计可以作为具有平均收益目标的并发博弈的理性综合和验证问题的替代解决方案，或者作为一种技术，在可能的情况下，以最优的方式修复具有不良理性结果的并发博弈（纳什均衡）。 摘要：In game theory, mechanism design is concerned with the design of incentives so that a desired outcome of the game can be achieved. In this paper, we study the design of incentives so that a desirable equilibrium is obtained, for instance, an equilibrium satisfying a given temporal logic property -- a problem that we call equilibrium design. We base our study on a framework where system specifications are represented as temporal logic formulae, games as quantitative concurrent game structures, and players' goals as mean-payoff objectives. In particular, we consider system specifications given by LTL and GR(1) formulae, and show that implementing a mechanism to ensure that a given temporal logic property is satisfied on some/every Nash equilibrium of the game, whenever such a mechanism exists, can be done in PSPACE for LTL properties and in NP/$\Sigma^{P}_{2}$ for GR(1) specifications. We also study the complexity of various related decision and optimisation problems, such as optimality and uniqueness of solutions, and show that the complexities of all such problems lie within the polynomial hierarchy. As an application, equilibrium design can be used as an alternative solution to the rational synthesis and verification problems for concurrent games with mean-payoff objectives whenever no solution exists, or as a technique to repair, whenever possible, concurrent games with undesirable rational outcomes (Nash equilibria) in an optimal way.

【11】 Rational Shapley Values 标题：有理Shapley值

作者：David S. Watson 机构： WATSON 1 1Department of Statistical Science, University College London 备注：20 pages, 3 figures, 7 tables 链接：https://arxiv.org/abs/2106.10191 摘要：解释不透明机器学习算法的预测是一项重要且具有挑战性的任务，尤其是在复杂模型越来越多地用于辅助高风险决策（如医疗和金融领域的决策）的情况下。大多数流行的事后解释人工智能（XAI）工具要么对上下文不敏感（如特征属性），要么难以总结（如反事实）。在本文中，我将介绍\emph{rational Shapley values}，这是一种新的XAI方法，它以严格、灵活的方式综合和扩展了这些看似不兼容的方法。我利用决策理论和因果建模的工具来形式化和实现一种实用的方法，解决XAI中的许多已知挑战。通过将随机变量的分布与给定解释任务的适当参考类配对，我通过理论和实验说明了用户目标和知识如何以迭代方式通知和约束解集。在一系列定量和定性的比较中，该方法优于最先进的XAI工具。 摘要：Explaining the predictions of opaque machine learning algorithms is an important and challenging task, especially as complex models are increasingly used to assist in high-stakes decisions such as those arising in healthcare and finance. Most popular tools for post-hoc explainable artificial intelligence (XAI) are either insensitive to context (e.g., feature attributions) or difficult to summarize (e.g., counterfactuals). In this paper, I introduce \emph{rational Shapley values}, a novel XAI method that synthesizes and extends these seemingly incompatible approaches in a rigorous, flexible manner. I leverage tools from decision theory and causal modeling to formalize and implement a pragmatic approach that resolves a number of known challenges in XAI. By pairing the distribution of random variables with the appropriate reference class for a given explanation task, I illustrate through theory and experiments how user goals and knowledge can inform and constrain the solution set in an iterative fashion. The method compares favorably to state of the art XAI tools in a range of quantitative and qualitative comparisons.

【12】 Self-supervised Incremental Deep Graph Learning for Ethereum Phishing Scam Detection 标题：自监督增量式深度图学习在以太网络钓鱼检测中的应用

作者：Shucheng Li,Fengyuan Xu,Runchuan Wang,Sheng Zhong 机构：National Key Lab for Novel Software Technology, Nanjing University, China 链接：https://arxiv.org/abs/2106.10176 摘要：近年来，网络钓鱼诈骗已成为第二大区块链平台以太坊涉案金额最大的犯罪类型。同时，图形神经网络（GNN）在各种节点分类任务中表现出了良好的性能。然而，对于以太坊事务数据而言，它可以自然地抽象为现实世界中的复杂图形，标签的稀缺性和事务数据的巨大容量使得利用GNN方法变得困难。本文针对这两个问题，提出了一种自监督的增量深度图学习模型（seave），用于以太坊网络钓鱼欺诈的检测。在我们的模型中，从空间和时间角度设计的两个借口任务帮助我们有效地从大量未标记的事务数据中学习有用的节点嵌入。增量范式允许我们有效地处理大规模事务数据，并帮助模型在数据分布急剧变化时保持良好的性能。我们从以太坊收集了大约半年的交易记录，我们的大量实验表明，我们的模型在传导和感应两种情况下都始终优于强基线。 摘要：In recent years, phishing scams have become the crime type with the largest money involved on Ethereum, the second-largest blockchain platform. Meanwhile, graph neural network (GNN) has shown promising performance in various node classification tasks. However, for Ethereum transaction data, which could be naturally abstracted to a real-world complex graph, the scarcity of labels and the huge volume of transaction data make it difficult to take advantage of GNN methods. Here in this paper, to address the two challenges, we propose a Self-supervised Incremental deep Graph learning model (SIEGE), for the phishing scam detection problem on Ethereum. In our model, two pretext tasks designed from spatial and temporal perspectives help us effectively learn useful node embedding from the huge amount of unlabelled transaction data. And the incremental paradigm allows us to efficiently handle large-scale transaction data and help the model maintain good performance when the data distribution is drastically changing. We collect transaction records about half a year from Ethereum and our extensive experiments show that our model consistently outperforms strong baselines in both transductive and inductive settings.

【13】 The Principles of Deep Learning Theory 标题：深度学习理论的基本原理

作者：Daniel A. Roberts,Sho Yaida,Boris Hanin 机构：based on research in collaboration with, arXiv:,.,v, [cs.LG] , Jun 备注：451 pages, to be published by Cambridge University Press 链接：https://arxiv.org/abs/2106.10165 摘要：这本书开发了一个有效的理论方法来理解实际意义的深层神经网络。从网络的第一性原理分量图出发，说明了如何通过求解层间迭代方程和非线性学习动力学来确定训练网络输出的精确描述。主要结果是网络的预测是近似高斯分布的，网络的深宽比控制着与无限宽高斯描述的偏差。我们解释了这些深度网络如何有效地从训练中学习非平凡的表示，并更广泛地分析了非线性模型的表示学习机制。从近似核方法的角度，我们发现这种模型的预测对底层学习算法的依赖性可以用一种简单而通用的方式来表示。为了得到这些结果，我们提出了表示群流（RG-flow）的概念来描述信号在网络中的传播。通过将网络调整到临界状态，我们给出了爆炸和消失梯度问题的一个实用解。我们进一步解释了RG流如何导致接近普遍性的行为，并让我们将从不同激活函数构建的网络分类为普遍性类。总之，我们证明了深度与宽度之比决定了训练网络集合的有效模型复杂性。通过使用信息论技术，我们估计了最佳的纵横比，在这个比例下，我们期望网络实际上是最有用的，并展示了如何使用剩余连接将这个比例推到任意深度。通过这些工具，我们可以详细了解体系结构、超参数和优化器的归纳偏差。 摘要：This book develops an effective theory approach to understanding deep neural networks of practical relevance. Beginning from a first-principles component-level picture of networks, we explain how to determine an accurate description of the output of trained networks by solving layer-to-layer iteration equations and nonlinear learning dynamics. A main result is that the predictions of networks are described by nearly-Gaussian distributions, with the depth-to-width aspect ratio of the network controlling the deviations from the infinite-width Gaussian description. We explain how these effectively-deep networks learn nontrivial representations from training and more broadly analyze the mechanism of representation learning for nonlinear models. From a nearly-kernel-methods perspective, we find that the dependence of such models' predictions on the underlying learning algorithm can be expressed in a simple and universal way. To obtain these results, we develop the notion of representation group flow (RG flow) to characterize the propagation of signals through the network. By tuning networks to criticality, we give a practical solution to the exploding and vanishing gradient problem. We further explain how RG flow leads to near-universal behavior and lets us categorize networks built from different activation functions into universality classes. Altogether, we show that the depth-to-width ratio governs the effective model complexity of the ensemble of trained networks. By using information-theoretic techniques, we estimate the optimal aspect ratio at which we expect the network to be practically most useful and show how residual connections can be used to push this scale to arbitrary depths. With these tools, we can learn in detail about the inductive bias of architectures, hyperparameters, and optimizers.

【14】 Toward Fault Detection in Industrial Welding Processes with Deep Learning and Data Augmentation 标题：基于深度学习和数据增强的工业焊接过程故障检测

作者：Jibinraj Antony,Dr. Florian Schlather,Georgij Safronov,Markus Schmitz,Prof. Dr. Kristof Van Laerhoven 机构： Laerhoven [a][a] University of Siegen 链接：https://arxiv.org/abs/2106.10160 摘要：随着计算机视觉领域中深度学习模型的兴起，它们在工业过程中应用的新可能性被证明带来了巨大的收益。然而，机器学习对于高度标准化的工业过程的实际适用性仍在争论之中。以激光束焊接质量控制为例，阐述了人工智能工具在工业实现中面临的挑战。我们使用来自TensorFlow对象检测API的对象检测算法，并使用转移学习使它们适应我们的用例。我们开发的基线模型用作基准，并与经过数据集缩放和超参数调整的模型进行评估和比较。我们发现，通过图像增强对数据集进行适度的缩放可以提高联合交集（IoU）和召回率，而高水平的增强和缩放可能会导致结果的恶化。最后，我们将结果放在底层用例的视角中，并评估它们的适合性。 摘要：With the rise of deep learning models in the field of computer vision, new possibilities for their application in industrial processes proves to return great benefits. Nevertheless, the actual fit of machine learning for highly standardised industrial processes is still under debate. This paper addresses the challenges on the industrial realization of the AI tools, considering the use case of Laser Beam Welding quality control as an example. We use object detection algorithms from the TensorFlow object detection API and adapt them to our use case using transfer learning. The baseline models we develop are used as benchmarks and evaluated and compared to models that undergo dataset scaling and hyperparameter tuning. We find that moderate scaling of the dataset via image augmentation leads to improvements in intersection over union (IoU) and recall, whereas high levels of augmentation and scaling may lead to deterioration of results. Finally, we put our results into perspective of the underlying use case and evaluate their fit.

【15】 Predicting gender of Brazilian names using deep learning 标题：基于深度学习的巴西人名性别预测

作者：Rosana C. B. Rego,Verônica M. L. Silva 机构：Program in Electrical and Computer Engineering, Federal University of Rio Grande do Norte, Brazil, Department of Engineering and Technology, Federal Rural University of Semi-Arid, Brazil 备注：9 pages, 8 figures 链接：https://arxiv.org/abs/2106.10156 摘要：通过名字来预测性别并不是一项简单的任务。在许多应用程序中，特别是在自然语言处理（NLP）领域，这项任务可能是必要的，主要是在考虑外来名称时。一些机器学习算法可以很好地进行预测。在本文中，我们研究并实现了前馈和递归的深层神经网络模型，如MLP、RNN、GRU、CNN和BiLSTM，通过名字来分类性别。巴西人名数据集用于训练和评估模型。我们分析了模型的准确度、召回率、精确度和混淆矩阵来衡量模型的性能。结果表明，通过将人名看作一组字符串的特征提取策略，可以进行性别预测。一些模型准确预测了90%以上病例的性别。递归模型克服了二值分类问题中的前馈模型。 摘要：Predicting gender by the name is not a simple task. In many applications, especially in the natural language processing (NLP) field, this task may be necessary, mainly when considering foreign names. Some machine learning algorithms can satisfactorily perform the prediction. In this paper, we examined and implemented feedforward and recurrent deep neural network models, such as MLP, RNN, GRU, CNN, and BiLSTM, to classify gender through the first name. A dataset of Brazilian names is used to train and evaluate the models. We analyzed the accuracy, recall, precision, and confusion matrix to measure the models' performances. The results indicate that the gender prediction can be performed from the feature extraction strategy looking at the names as a set of strings. Some models accurately predict the gender in more than 90% of the cases. The recurrent models overcome the feedforward models in this binary classification problem.

【16】 Classical Planning as QBF without Grounding (extended version) 标题：经典规划作为无接地的QBF(扩展版本)

作者：Irfansha Shaik,Jaco van de Pol 机构：Aarhus University, Department of Computer Science, Aarhus, Denmark 链接：https://arxiv.org/abs/2106.10138 摘要：大多数经典的规划者使用接地作为预处理步骤，将规划简化为命题逻辑。然而，接地带来了严重的内存开销，这导致了基于SAT/QBF的计划者需要大量的编码。尽管SAT/QBF编码进行了优化，如动作分割、紧凑编码和使用并行计划，但当动作有许多参数时，由于接地而导致的内存使用仍然是一个瓶颈，例如IPC 2018计划竞赛的有机合成问题（以其原始的非分割形式）。在本文中，我们提供了一个紧凑的QBF编码是对数的对象数，并避免了接地完全使用通用量化的对象组合。我们将不接地的QBF编码与简单的SAT编码进行了比较，同时也表明我们可以解决一些由于接地而无法被任何基于SAT/QBF的规划者处理的有机合成问题。 摘要：Most classical planners use grounding as a preprocessing step, reducing planning to propositional logic. However, grounding comes with a severe cost in memory, resulting in large encodings for SAT/QBF based planners. Despite the optimisations in SAT/QBF encodings such as action splitting, compact encodings and using parallel plans, the memory usage due to grounding remains a bottleneck when actions have many parameters, such as in the Organic Synthesis problems from the IPC 2018 planning competition (in its original non-split form). In this paper, we provide a compact QBF encoding that is logarithmic in the number of objects and avoids grounding completely by using universal quantification for object combinations. We compare the ungrounded QBF encoding with the simple SAT encoding and also show that we can solve some of the Organic Synthesis problems, which could not be handled before by any SAT/QBF based planners due to grounding.

【17】 Towards Distraction-Robust Active Visual Tracking 标题：朝向分散注意力的鲁棒主动视觉跟踪

作者：Fangwei Zhong,Peng Sun,Wenhan Luo,Tingyun Yan,Yizhou Wang 备注：To appear in ICML2021 链接：https://arxiv.org/abs/2106.10110 摘要：在主动视觉跟踪中，分心物体的出现是众所周知的困难，因为分心物常常通过遮挡目标或带来混乱的外观来误导跟踪器。为了解决这个问题，我们提出了一个混合的合作竞争多智能体博弈，其中一个目标和多个干扰者组成一个协作团队，与一个跟踪器对抗，使其无法跟踪。通过在游戏中的学习，分心者的各种分心行为自然出现，从而暴露了跟踪器的弱点，增强了跟踪器的分心鲁棒性。为了有效的学习，我们提出了一系列实用的方法，包括分心者的奖励函数、跨模式的师生学习策略和跟踪器的重复注意机制。实验结果表明，该跟踪器具有良好的分心鲁棒主动视觉跟踪性能，并能很好地推广到不可见环境中。我们还证明了多智能体博弈可以用来对抗地测试跟踪器的鲁棒性。 摘要：In active visual tracking, it is notoriously difficult when distracting objects appear, as distractors often mislead the tracker by occluding the target or bringing a confusing appearance. To address this issue, we propose a mixed cooperative-competitive multi-agent game, where a target and multiple distractors form a collaborative team to play against a tracker and make it fail to follow. Through learning in our game, diverse distracting behaviors of the distractors naturally emerge, thereby exposing the tracker's weakness, which helps enhance the distraction-robustness of the tracker. For effective learning, we then present a bunch of practical methods, including a reward function for distractors, a cross-modal teacher-student learning strategy, and a recurrent attention mechanism for the tracker. The experimental results show that our tracker performs desired distraction-robust active visual tracking and can be well generalized to unseen environments. We also show that the multi-agent game can be used to adversarially test the robustness of trackers.

【18】 Discerning Generic Event Boundaries in Long-Form Wild Videos 标题：在冗长的Wild视频中识别通用事件边界

作者：Ayush K Rai,Tarun Krishna,Julia Dietlmeier,Kevin McGuinness,Alan F Smeaton,Noel E O'Connor 机构：Alan F. Smeaton, Noel E. O’Connor, Insight Centre for Data Analytics, Dublin City University, Ireland 备注：Technical Report for Generic Event Boundary Challenge - LOVEU Challenge (CVPR 2021) 链接：https://arxiv.org/abs/2106.10090 摘要：检测泛型的、无分类的事件边界代表了对视频理解的一大进步。本文提出了一种基于平面三维卷积结构的双流伪造事件边界检测技术，该技术可以从视频中学习时空特征。我们的工作受到了一般事件边界检测挑战（CVPR2021长格式视频理解-LOVEU研讨会的一部分）的启发。在本文中，我们对所进行的实验进行了深入的分析，并对获得的结果进行了解释。 摘要：Detecting generic, taxonomy-free event boundaries invideos represents a major stride forward towards holisticvideo understanding. In this paper we present a technique forgeneric event boundary detection based on a two stream in-flated 3D convolutions architecture, which can learn spatio-temporal features from videos. Our work is inspired from theGeneric Event Boundary Detection Challenge (part of CVPR2021 Long Form Video Understanding- LOVEU Workshop).Throughout the paper we provide an in-depth analysis ofthe experiments performed along with an interpretation ofthe results obtained.

【19】 Label Mask for Multi-Label Text Classification 标题：多标签文本分类中的标签掩码

作者：Rui Song,Xingbing Chen,Zelong Liu,Haining An,Zhiqi Zhang,Xiaoguang Wang,Hao Xu 机构：School of Artificial Intelligence, Jilin University, Public Computer Education and Research Center, Jilin University, College of Computer Science and Technology, Key Laboratory of Symbolic Computing and 链接：https://arxiv.org/abs/2106.10076 摘要：多标签文本分类的关键问题之一是如何利用标签之间的相关性。然而，在一个复杂且未知的标签空间中，直接对标签之间的相关性进行建模是一个非常具有挑战性的问题。本文借鉴语言模型中完形填空的思想，提出了一种标签掩码多标签文本分类模型（LM-MTC）。通过训练前语言模型的强大功能，LM-MTC能够捕捉到标签之间的隐含关系。在此基础上，我们为每个潜在的标签分配一个不同的标记，并以一定的概率随机屏蔽该标记，建立一个基于标签的屏蔽语言模型（MLM）。同时对MTC和MLM进行训练，进一步提高了模型的泛化能力。在多个数据集上的大量实验证明了该方法的有效性。 摘要：One of the key problems in multi-label text classification is how to take advantage of the correlation among labels. However, it is very challenging to directly model the correlations among labels in a complex and unknown label space. In this paper, we propose a Label Mask multi-label text classification model (LM-MTC), which is inspired by the idea of cloze questions of language model. LM-MTC is able to capture implicit relationships among labels through the powerful ability of pre-train language models. On the basis, we assign a different token to each potential label, and randomly mask the token with a certain probability to build a label based Masked Language Model (MLM). We train the MTC and MLM together, further improving the generalization ability of the model. A large number of experiments on multiple datasets demonstrate the effectiveness of our method.

【20】 Learning to Plan via a Multi-Step Policy Regression Method 标题：通过多步策略回归方法学习计划

作者：Stefan Wagner,Michael Janschek,Tobias Uelwer,Stefan Harmeling 机构：Department of Computer Science, Heinrich Heine University D¨usseldorf, Germany 备注：Accepted at the 30th International Conference on Artificial Neural Networks (ICANN 2021) 链接：https://arxiv.org/abs/2106.10075 摘要：我们提出了一种新的方法来提高推理性能的环境中，需要一个特定的序列的行动，以解决。例如，在迷宫环境中，理想情况下确定了最佳路径。我们希望学习一个可以提前预测n个动作的策略，而不是一步一步地学习一个策略。我们提出的策略水平回归（PHR）方法利用A2C采样的环境知识，在一个策略蒸馏设置中学习一个n维的策略向量，每个观测值产生n个连续动作。我们在微网格和Pong环境下对我们的方法进行了测试，通过成功地预测单个观测的动作序列，在推理过程中显示出极大的加速。 摘要：We propose a new approach to increase inference performance in environments that require a specific sequence of actions in order to be solved. This is for example the case for maze environments where ideally an optimal path is determined. Instead of learning a policy for a single step, we want to learn a policy that can predict n actions in advance. Our proposed method called policy horizon regression (PHR) uses knowledge of the environment sampled by A2C to learn an n dimensional policy vector in a policy distillation setup which yields n sequential actions per observation. We test our method on the MiniGrid and Pong environments and show drastic speedup during inference time by successfully predicting sequences of actions on a single observation.

【21】 Contrastive Learning of Generalized Game Representations 标题：广义游戏表征的对比学习

作者：Chintan Trivedi,Antonios Liapis,Georgios N. Yannakakis 机构：Institute of Digital Games, University of Malta, Msida, Malta 备注：8 pages, 7 figures, CoG 链接：https://arxiv.org/abs/2106.10060 摘要：通过像素表示游戏为构建通用和多功能的游戏模型提供了一种很有前景的方法。虽然游戏不仅仅是图像，但基于游戏像素训练的神经网络模型往往捕捉图像视觉风格的差异，而不是游戏内容的差异。因此，即使在同类游戏中，这样的模型也不能很好地推广。本文以对比学习的最新进展为基础，展示了它对游戏表征学习的益处。学会对比游戏图像不仅能更有效地对游戏进行分类；它还产生了一些模型，这些模型通过忽略视觉风格，而将注意力集中在游戏内容上，从而以一种更有意义的方式将游戏分开。我们在一个大型体育视频游戏数据集上的研究结果表明，对比学习比传统的监督学习更适合于广义游戏表征的学习。这项研究的结果使我们更接近通用的游戏视觉编码器，它可以在以前看不到的游戏中重用，而不需要重新训练或微调。 摘要：Representing games through their pixels offers a promising approach for building general-purpose and versatile game models. While games are not merely images, neural network models trained on game pixels often capture differences of the visual style of the image rather than the content of the game. As a result, such models cannot generalize well even within similar games of the same genre. In this paper we build on recent advances in contrastive learning and showcase its benefits for representation learning in games. Learning to contrast images of games not only classifies games in a more efficient manner; it also yields models that separate games in a more meaningful fashion by ignoring the visual style and focusing, instead, on their content. Our results in a large dataset of sports video games containing 100k images across 175 games and 10 game genres suggest that contrastive learning is better suited for learning generalized game representations compared to conventional supervised learning. The findings of this study bring us closer to universal visual encoders for games that can be reused across previously unseen games without requiring retraining or fine-tuning.

【22】 Meta-control of social learning strategies 标题：社会学习策略的元控制

作者：Anil Yaman,Nicolas Bredeche,Onur Çaylak,Joel Z. Leibo,Sang Wan Lee 机构：Korea Advanced Institute of Science and Technology, Daejeon, Republic of Korea, Sorbonne Universit´e, Paris, France, Eindhoven University of Technology, Eindhoven, the Netherlands, DeepMind, London, UK 链接：https://arxiv.org/abs/2106.10015 摘要：社会学习是在没有实际经验的情况下模仿他人的行为，提供了一种成本效益高的知识获取方式。然而，它提出了一个基本问题，即哪些人拥有可靠的信息：成功的个人与多数人。前者和后者分别被称为基于成功的和循规蹈矩的社会学习策略。我们在这里表明，基于成功的策略充分利用了低不确定性的良性环境，但在不确定环境中失败了。另一方面，因循守旧策略可以有效地缓解这种不利影响。基于这些发现，我们假设个体和社会学习策略的元控制在不稳定和不确定的环境中提供有效的和样本有效的学习。在一组具有不同程度的波动性和不确定性的环境中进行的模拟验证了我们的假设。研究结果表明，社会学习的元控制通过利用他人的学习作为外部知识库，为个体提供了以最小的探索成本解决环境不确定性的杠杆。 摘要：Social learning, copying other's behavior without actual experience, offers a cost-effective means of knowledge acquisition. However, it raises the fundamental question of which individuals have reliable information: successful individuals versus the majority. The former and the latter are known respectively as success-based and conformist social learning strategies. We show here that while the success-based strategy fully exploits the benign environment of low uncertainly, it fails in uncertain environments. On the other hand, the conformist strategy can effectively mitigate this adverse effect. Based on these findings, we hypothesized that meta-control of individual and social learning strategies provides effective and sample-efficient learning in volatile and uncertain environments. Simulations on a set of environments with various levels of volatility and uncertainty confirmed our hypothesis. The results imply that meta-control of social learning affords agents the leverage to resolve environmental uncertainty with minimal exploration cost, by exploiting others' learning as an external knowledge base.

【23】 On the Connections between Counterfactual Explanations and Adversarial Examples 标题：论反事实解释与对抗性例证的联系

作者：Martin Pawelczyk,Shalmali Joshi,Chirag Agarwal,Sohini Upadhyay,Himabindu Lakkaraju 机构：University of Tübingen, Harvard University 链接：https://arxiv.org/abs/2106.09992 摘要：反事实解释和对抗性例子已经成为解决机器学习（ML）的可解释性和鲁棒性目标的关键研究领域。反事实解释的目的是向受到算法决策不利影响的个人提供求助，而对抗性例子则是为了暴露ML模型的弱点。虽然先前的研究已经暗示了这些框架之间的共同点，但是对于系统地探讨反事实解释和对抗性例子之间的联系的工作却很少甚至没有。在这项工作中，我们首次尝试将反事实解释和对抗性例子之间的联系形式化。更具体地说，我们从理论上分析了显著的反事实解释和对抗性例子生成方法，并强调了它们行为相似的条件。分析表明，Wachter等人、Carlini和Wagner等人（均方误差损失）提出的反事实解释和对抗性例子生成方法，以及Zhao等人提出的C-CHVAE和自然对抗性例子生成方法是等价的。我们还限定了Wachter等人产生的反事实解释和对抗性例子之间的距离，以及线性模型的DeepFool方法。最后，我们用大量的合成和真实数据集的实验来验证我们的理论发现。 摘要：Counterfactual explanations and adversarial examples have emerged as critical research areas for addressing the explainability and robustness goals of machine learning (ML). While counterfactual explanations were developed with the goal of providing recourse to individuals adversely impacted by algorithmic decisions, adversarial examples were designed to expose the vulnerabilities of ML models. While prior research has hinted at the commonalities between these frameworks, there has been little to no work on systematically exploring the connections between the literature on counterfactual explanations and adversarial examples. In this work, we make one of the first attempts at formalizing the connections between counterfactual explanations and adversarial examples. More specifically, we theoretically analyze salient counterfactual explanation and adversarial example generation methods, and highlight the conditions under which they behave similarly. Our analysis demonstrates that several popular counterfactual explanation and adversarial example generation methods such as the ones proposed by Wachter et. al. and Carlini and Wagner (with mean squared error loss), and C-CHVAE and natural adversarial examples by Zhao et. al. are equivalent. We also bound the distance between counterfactual explanations and adversarial examples generated by Wachter et. al. and DeepFool methods for linear models. Finally, we empirically validate our theoretical findings using extensive experimentation with synthetic and real world datasets.

【24】 On the Sample Complexity of Batch Reinforcement Learning with Policy-Induced Data 标题：策略诱导数据批量强化学习的样本复杂度研究

作者：Chenjun Xiao,Ilbin Lee,Bo Dai,Dale Schuurmans,Csaba Szepesvari 机构：University of Alberta, Google Research, Brain Team, DeepMind 备注：26 pages, 2 figures 链接：https://arxiv.org/abs/2106.09973 摘要：我们研究了有限马尔可夫决策过程（MDP）中学习一个好策略的样本复杂度这一基本问题，当可用于学习的数据是通过遵循一个必须在不知道底层MDP的情况下选择的日志策略获得的。我们的主要结果表明，当规划周期$H$有限时，样本复杂度，即获得一个好策略所需的和足够的最小转移次数，是相关量的指数函数。特别地，我们证明了获得$\epsilon$-最优策略的样本复杂度对于$\gamma$-折扣问题至少是$\Omega（\mathrm{A}^{\min（\mathrm{S}-1，H+1）}）$，其中$\mathrm{S}$是状态数，$\mathrm{A}$是动作数，$H$是有效视界，定义为$H=\lfloor\tfrac{\ln（1/\epsilon）}{\ln（1/\gamma）}\rfloor$；它至少是$\Omega（\mathrm{A}^{\min（\mathrm{S}-1，H）}/\varepsilon^2）$，其中，$H$是问题的规划范围。这个下限基本上与一个上限匹配。对于平均报酬设置，我们证明了在有限的数据量下没有算法可以找到$\epsilon$-最优策略。 摘要：We study the fundamental question of the sample complexity of learning a good policy in finite Markov decision processes (MDPs) when the data available for learning is obtained by following a logging policy that must be chosen without knowledge of the underlying MDP. Our main results show that the sample complexity, the minimum number of transitions necessary and sufficient to obtain a good policy, is an exponential function of the relevant quantities when the planning horizon $H$ is finite. In particular, we prove that the sample complexity of obtaining $\epsilon$-optimal policies is at least $\Omega(\mathrm{A}^{\min(\mathrm{S}-1, H+1)})$ for $\gamma$-discounted problems, where $\mathrm{S}$ is the number of states, $\mathrm{A}$ is the number of actions, and $H$ is the effective horizon defined as $H=\lfloor \tfrac{\ln(1/\epsilon)}{\ln(1/\gamma)} \rfloor$; and it is at least $\Omega(\mathrm{A}^{\min(\mathrm{S}-1, H)}/\varepsilon^2)$ for finite horizon problems, where $H$ is the planning horizon of the problem. This lower bound is essentially matched by an upper bound. For the average-reward setting we show that there is no algorithm finding $\epsilon$-optimal policies with a finite amount of data.

【25】 Novelty Detection via Contrastive Learning with Negative Data Augmentation 标题：基于负数据增强对比学习的新颖性检测

作者：Chengwei Chen,Yuan Xie,Shaohui Lin,Ruizhi Qiao,Jian Zhou,Xin Tan,Yi Zhang,Lizhuang Ma 机构：East China Normal University, Tencent Youtu Lab, Shanghai Jiao Tong University, Zhejiang Lab 备注：None 链接：https://arxiv.org/abs/2106.09958 摘要：新颖性检测是确定查询示例是否与所学习的训练分布不同的过程。以往的方法试图通过生成性对抗网络（generative敌对网络，GANs）来学习正态样本的表示。然而，他们会受到不稳定训练、模式下降和低辨别能力的影响。最近，各种各样的借口任务（如旋转预测和聚类）被提出用于新颖性检测中的自监督学习。然而，学习到的潜在特征仍然是低分辨的。我们通过引入一个新的解码器-编码器框架来克服这些问题。首先，生成网络（又称解码器）通过将初始化的潜在向量映射到图像来学习表示。特别地，该向量通过考虑训练数据的整体分布来初始化，避免了模式丢失的问题。其次，对比网络（又称编码器）的目标是通过互信息估计来“学习比较”，这直接帮助生成网络通过使用负数据扩充策略获得更具区分性的表示。大量实验表明，该模型比现有的新颖性检测方法具有明显的优越性，并在CIFAR10和DCASE等新颖性检测基准上取得了最新的结果。此外，与其他基于对抗的新颖性检测方法相比，我们的模型在非对抗性训练中更稳定。 摘要：Novelty detection is the process of determining whether a query example differs from the learned training distribution. Previous methods attempt to learn the representation of the normal samples via generative adversarial networks (GANs). However, they will suffer from instability training, mode dropping, and low discriminative ability. Recently, various pretext tasks (e.g. rotation prediction and clustering) have been proposed for self-supervised learning in novelty detection. However, the learned latent features are still low discriminative. We overcome such problems by introducing a novel decoder-encoder framework. Firstly, a generative network (a.k.a. decoder) learns the representation by mapping the initialized latent vector to an image. In particular, this vector is initialized by considering the entire distribution of training data to avoid the problem of mode-dropping. Secondly, a contrastive network (a.k.a. encoder) aims to ``learn to compare'' through mutual information estimation, which directly helps the generative network to obtain a more discriminative representation by using a negative data augmentation strategy. Extensive experiments show that our model has significant superiority over cutting-edge novelty detectors and achieves new state-of-the-art results on some novelty detection benchmarks, e.g. CIFAR10 and DCASE. Moreover, our model is more stable for training in a non-adversarial manner, compared to other adversarial based novelty detection methods.

【26】 Labelling Drifts in a Fault Detection System for Wind Turbine Maintenance 标题：风电机组检修故障检测系统中漂移的标注

作者：Iñigo Martinez,Elisabeth Viles,Iñaki Cabrejas 机构：esElisabeth VilesUniversity of Navarra - Tecnun 备注：None 链接：https://arxiv.org/abs/2106.09951 摘要：故障检测系统是实现预测性维修策略的第一步。一种流行的数据驱动的方法来检测早期故障和异常是通过应用机器学习技术（如前馈神经网络（FFNN）或极限学习机（ELM））来训练正常行为模型。然而，在工业资产运行的动态环境中，非平稳性的意外上升可能会恶化这些建模技术的性能。测量变量中这种不可预测的统计变化称为概念漂移。本文介绍了一个风电机组维修案例，其中各种非平稳性可能会意外发生。这种概念漂移事件需要通过统计检测器和基于窗口的方法来检测。然而，在实际的复杂系统中，概念漂移并不像人工生成的数据集那样清晰和明显。为了评估电流漂移探测器的有效性，并为这种特殊的工业应用设计一种合适的新技术，必须事先处理现有漂移的特征。在缺乏这方面信息的情况下，提出了一种在风力涡轮机寿命期内标记概念漂移事件的方法。这种方法将有助于建立一个漂移数据库，该数据库将既是概念漂移探测器的训练基地，也是增强复杂系统维护知识的宝贵信息。 摘要：A failure detection system is the first step towards predictive maintenance strategies. A popular data-driven method to detect incipient failures and anomalies is the training of normal behaviour models by applying a machine learning technique like feed-forward neural networks (FFNN) or extreme learning machines (ELM). However, the performance of any of these modelling techniques can be deteriorated by the unexpected rise of non-stationarities in the dynamic environment in which industrial assets operate. This unpredictable statistical change in the measured variable is known as concept drift. In this article a wind turbine maintenance case is presented, where non-stationarities of various kinds can happen unexpectedly. Such concept drift events are desired to be detected by means of statistical detectors and window-based approaches. However, in real complex systems, concept drifts are not as clear and evident as in artificially generated datasets. In order to evaluate the effectiveness of current drift detectors and also to design an appropriate novel technique for this specific industrial application, it is essential to dispose beforehand of a characterization of the existent drifts. Under the lack of information in this regard, a methodology for labelling concept drift events in the lifetime of wind turbines is proposed. This methodology will facilitate the creation of a drift database that will serve both as a training ground for concept drift detectors and as a valuable information to enhance the knowledge about maintenance of complex systems.

【27】 Evolving GANs: When Contradictions Turn into Compliance 标题：演变中的甘斯：当矛盾转变为顺从

作者：Sauptik Dhar,Javad Heydari,Samarth Tripathi,Unmesh Kurup,Mohak Shah 机构：America Research Lab, LG Electronics, Great America Pkwy, Santa Clara, CA, USA 备注：Generative Adversarial Networks, Universum Learning, Semi-Supervised Learning 链接：https://arxiv.org/abs/2106.09946 摘要：标记数据的有限可用性使得任何监督学习问题都具有挑战性。半监督学习和universum学习等替代学习设置减轻了对标记数据的依赖，但仍然需要大量的未标记数据，这些数据可能不可用或获取成本高昂。基于GAN的合成数据生成方法最近通过生成合成样本来改进手头的任务，显示出了良好的前景。但是，这些样品不能用于其他目的。在本文中，我们提出了一个GAN游戏，在有限的数据设置下提供了改进的鉴别器精度，同时生成真实的合成数据。这提供了一个额外的优势，即现在生成的数据可以用于其他类似的任务。我们提供了理论保证和实证结果来支持我们的方法。 摘要：Limited availability of labeled-data makes any supervised learning problem challenging. Alternative learning settings like semi-supervised and universum learning alleviate the dependency on labeled data, but still require a large amount of unlabeled data, which may be unavailable or expensive to acquire. GAN-based synthetic data generation methods have recently shown promise by generating synthetic samples to improve task at hand. However, these samples cannot be used for other purposes. In this paper, we propose a GAN game which provides improved discriminator accuracy under limited data settings, while generating realistic synthetic data. This provides the added advantage that now the generated data can be used for other similar tasks. We provide the theoretical guarantees and empirical results in support of our approach.

【28】 Goal-Directed Planning by Reinforcement Learning and Active Inference 标题：基于强化学习和主动推理的目标导向规划

作者：Dongqi Han,Kenji Doya,Jun Tani 机构：Cognitive Neurorobotics Research Unit, Okinawa Institute of Science and Technology, Okinawa, Japan, Neural Computation Unit 备注：Work in progress 链接：https://arxiv.org/abs/2106.09938 摘要：目标导向行为和习惯性行为有什么区别？我们提出了一种新的贝叶斯推理决策计算框架，其中所有的东西都集成为一个完整的神经网络模型。该模型通过自我探索学习预测环境状态转换，并通过随机内部状态采样$z$生成运动行为。习惯性行为是通过强化学习获得的，它是从$z$的先验分布中获得的。目标定向行为由$z$的后验分布决定，通过计划，使用主动推理，使目标观察的自由能最小化。我们通过在一个有摄像头观察和连续运动动作的感觉运动导航任务中的实验，证明了该框架的有效性。 摘要：What is the difference between goal-directed and habitual behavior? We propose a novel computational framework of decision making with Bayesian inference, in which everything is integrated as an entire neural network model. The model learns to predict environmental state transitions by self-exploration and generating motor actions by sampling stochastic internal states $z$. Habitual behavior, which is obtained from the prior distribution of $z$, is acquired by reinforcement learning. Goal-directed behavior is determined from the posterior distribution of $z$ by planning, using active inference, to minimize the free energy for goal observation. We demonstrate the effectiveness of the proposed framework by experiments in a sensorimotor navigation task with camera observations and continuous motor actions.

【29】 Embedding Heterogeneous Networks into Hyperbolic Space Without Meta-path 标题：在无元路径的双曲空间中嵌入异构网络

作者：Lili Wang,Chongyang Gao,Chenghan Huang,Ruibo Liu,Weicheng Ma,Soroush Vosoughi 机构： Department of Computer Science, Dartmouth College, Millennium Management LLC 备注：In proceedings of the 35th AAAI Conference on Artificial Intelligence 链接：https://arxiv.org/abs/2106.09923 摘要：现实世界中的网络是多种多样的。一种常见的网络类型是异构网络，其中节点（和边）可以是不同的类型。因此，人们一直致力于在低维空间中学习这些异构网络的表示。然而，现有的异构网络嵌入方法大多存在以下两个缺点：（1）目标空间通常是欧氏空间。相反，许多最近的研究表明，复杂网络可能具有非欧几里德双曲型的潜在解剖结构(2） 这些方法通常依赖于元路径，元路径选择需要特定领域的先验知识。此外，同一网络上的不同下行任务可能需要不同的元路径来生成特定于任务的嵌入。本文提出了一种新的无需元路径的自引导随机游走方法，将异构网络嵌入双曲空间。我们在两个公共数据集上对网络重建和链路预测任务进行了深入的实验，结果表明，我们的模型在所有任务中都优于各种已知的基线。 摘要：Networks found in the real-world are numerous and varied. A common type of network is the heterogeneous network, where the nodes (and edges) can be of different types. Accordingly, there have been efforts at learning representations of these heterogeneous networks in low-dimensional space. However, most of the existing heterogeneous network embedding methods suffer from the following two drawbacks: (1) The target space is usually Euclidean. Conversely, many recent works have shown that complex networks may have hyperbolic latent anatomy, which is non-Euclidean. (2) These methods usually rely on meta-paths, which require domain-specific prior knowledge for meta-path selection. Additionally, different down-streaming tasks on the same network might require different meta-paths in order to generate task-specific embeddings. In this paper, we propose a novel self-guided random walk method that does not require meta-path for embedding heterogeneous networks into hyperbolic space. We conduct thorough experiments for the tasks of network reconstruction and link prediction on two public datasets, showing that our model outperforms a variety of well-known baselines across all tasks.

【30】 Being Properly Improper 标题：适当地失当

作者：Richard Nock,Tyler Sypherd,Lalitha Sankar 机构：†Google Research, ‡School of Electrical, Computer, and Energy Engineering, Arizona State University, Tempe, AZ 链接：https://arxiv.org/abs/2106.09920 摘要：在今天的ML中，数据可以以各种方式扭曲（更改），无论是出于坏的目的还是出于好的目的。这种扭曲的数据挑战了监督损失适当性的基础理论，监督损失是许多流行的损失类别概率估计的基础。不幸的是，在它的核心，适当性确保了最优模型也学会了扭转。在本文中，我们分析了这类基于概率的损失，当它们被剥离强制性属性时；我们将扭度损失定义为能够从扭度中获得最佳（非扭度）估计值的损失，并表明S。阿里莫托是正确的。然后，我们转向一个理论，它提供了一些最佳的现成算法，以适当的损失，提高。提升可能需要获得损失凸共轭的导数来计算权重。由于计算或数学上的原因，这样的函数可能很难得到；原来阿里莫托的损失就是这样。我们绕过这个困难，将问题转化为：假设一个蓝图推进算法是用一个通用的权值更新函数实现的。最大限度减少的损失是什么？我们的答案是一个通用的boosting算法，它满足对弱学习者调用次数的最优boosting依赖；当应用于Arimoto的损失，它导致了一个简单的优化算法，其性能显示在几个领域和扭曲。 摘要：In today's ML, data can be twisted (changed) in various ways, either for bad or good intent. Such twisted data challenges the founding theory of properness for supervised losses which form the basis for many popular losses for class probability estimation. Unfortunately, at its core, properness ensures that the optimal models also learn the twist. In this paper, we analyse such class probability-based losses when they are stripped off the mandatory properness; we define twist-proper losses as losses formally able to retrieve the optimum (untwisted) estimate off the twists, and show that a natural extension of a half-century old loss introduced by S. Arimoto is twist proper. We then turn to a theory that has provided some of the best off-the-shelf algorithms for proper losses, boosting. Boosting can require access to the derivative of the convex conjugate of a loss to compute examples weights. Such a function can be hard to get, for computational or mathematical reasons; this turns out to be the case for Arimoto's loss. We bypass this difficulty by inverting the problem as follows: suppose a blueprint boosting algorithm is implemented with a general weight update function. What are the losses for which boosting-compliant minimisation happens? Our answer comes as a general boosting algorithm which meets the optimal boosting dependence on the number of calls to the weak learner; when applied to Arimoto's loss, it leads to a simple optimisation algorithm whose performances are showcased on several domains and twists.

【31】 Message Passing in Graph Convolution Networks via Adaptive Filter Banks 标题：基于自适应过滤银行的图卷积网络消息传递

作者：Xing Gao,Wenrui Dai,Chenglin Li,Junni Zou,Hongkai Xiong,Pascal Frossard 机构：‡Department of Electronic Engineering, Shanghai Jiao Tong University, ⋄Department of Computer Science, Shanghai Jiao Tong University, †Signal Processing Laboratory (LTS,), EPFL 链接：https://arxiv.org/abs/2106.09910 摘要：图卷积网络与消息传递图卷积网络（MPGCNs）一样，是网络数据表示学习的有力工具。然而，当数据是异构的时，大多数体系结构都受到限制，因为它们采用单一的策略来处理多通道图形信号，并且通常集中于低频信息。在本文中，我们提出了一种新的图卷积算子，称为BankGCN，它保留了消息传递模型的优点，但扩展了它们的能力，使之超越了“低通”特性。它将图上的多通道信号分解为子空间，并用自适应滤波器处理每个子空间中的特定信息。所有子空间的滤波器具有不同的频率响应，共同构成一个滤波器组。此外，谱域中的每个滤波器对应于一个消息传递方案，并且通过滤波器组实现不同的方案。重要的是，滤波器组和信号分解被联合学习以适应数据的频谱特性和目标应用。此外，与大多数现有mpgcn相比，这几乎是在没有额外参数的情况下实现的。实验结果表明，所提出的卷积算子可以在一组基准图数据集上实现良好的分类性能。 摘要：Graph convolution networks, like message passing graph convolution networks (MPGCNs), have been a powerful tool in representation learning of networked data. However, when data is heterogeneous, most architectures are limited as they employ a single strategy to handle multi-channel graph signals and they typically focus on low-frequency information. In this paper, we present a novel graph convolution operator, termed BankGCN, which keeps benefits of message passing models, but extends their capabilities beyond `low-pass' features. It decomposes multi-channel signals on graphs into subspaces and handles particular information in each subspace with an adapted filter. The filters of all subspaces have different frequency responses and together form a filter bank. Furthermore, each filter in the spectral domain corresponds to a message passing scheme, and diverse schemes are implemented via the filter bank. Importantly, the filter bank and the signal decomposition are jointly learned to adapt to the spectral characteristics of data and to target applications. Furthermore, this is implemented almost without extra parameters in comparison with most existing MPGCNs. Experimental results show that the proposed convolution operator permits to achieve excellent performance in graph classification on a collection of benchmark graph datasets.

【32】 Smoothed Multi-View Subspace Clustering 标题：平滑多视图子空间聚类

作者：Peng Chen,Liang Liu,Zhengrui Ma,Zhao Kang 机构： Jangsu Automation Research Institute, Lianyungang, Jiangsu, China, University of Electronic Science and Technology of China, Chengdu, Sichuan, China, Trusted Cloud Computing and Big Data Key Laboratory of Sichuan Province 备注：Accepted by International Conference on Neural Computing for Advanced Applications 2021 链接：https://arxiv.org/abs/2106.09875 摘要：近年来，多视点子空间聚类由于利用了多视点间的互补信息，取得了令人瞩目的效果。然而，多视图数据可能非常复杂，并且在实际应用中不容易聚类。现有的大多数方法都是对原始数据进行处理，可能得不到最优解。在这项工作中，我们提出了一种新的多视图聚类方法，称为平滑多视图子空间聚类（SMVSC），通过使用一种新的技术，即图过滤，来获得每个视图的平滑表示，其中相似的数据点具有相似的特征值。具体来说，它通过应用低通滤波器来保留图形的几何特征。因此，它产生了一个“聚类友好”的表示，极大地促进了下游的聚类任务。在基准数据集上的大量实验验证了该方法的优越性。分析表明，图过滤提高了类的可分性。 摘要：In recent years, multi-view subspace clustering has achieved impressive performance due to the exploitation of complementary imformation across multiple views. However, multi-view data can be very complicated and are not easy to cluster in real-world applications. Most existing methods operate on raw data and may not obtain the optimal solution. In this work, we propose a novel multi-view clustering method named smoothed multi-view subspace clustering (SMVSC) by employing a novel technique, i.e., graph filtering, to obtain a smooth representation for each view, in which similar data points have similar feature values. Specifically, it retains the graph geometric features through applying a low-pass filter. Consequently, it produces a ``clustering-friendly" representation and greatly facilitates the downstream clustering task. Extensive experiments on benchmark datasets validate the superiority of our approach. Analysis shows that graph filtering increases the separability of classes.

【33】 Towards Clustering-friendly Representations: Subspace Clustering via Graph Filtering 标题：面向聚类友好表示：基于图滤波子空间聚类

作者：Zhengrui Ma,Zhao Kang,Guangchun Luo,Ling Tian 机构：School of Computer Science and, Engineering, University of Electronic, Science and Technology of China, School of Information and Software 备注：Published in ACM Multimedia 2020 链接：https://arxiv.org/abs/2106.09874 摘要：在许多应用中，为特定的任务找到一个合适的数据表示是至关重要的。子空间聚类的成功与否取决于能否将数据划分成不同的子空间。然而，这个简单的假设并不总是成立的，因为原始数据可能不可分为子空间。为了恢复“聚类友好”的表示并便于后续的聚类，我们提出了一种图过滤方法，通过这种方法可以获得平滑的表示。具体地说，它通过应用一个低通滤波器将图的相似性注入到数据特征中，以提取有用的数据表示用于聚类。对图像和文档聚类数据集的大量实验表明，该方法改进了现有的子空间聚类技术。特别是，它与深度学习方法的可比性强调了简单图滤波方案在许多实际应用中的有效性。研究表明，图滤波可以去除噪声，保持图像的结构，提高分类的可分性。 摘要：Finding a suitable data representation for a specific task has been shown to be crucial in many applications. The success of subspace clustering depends on the assumption that the data can be separated into different subspaces. However, this simple assumption does not always hold since the raw data might not be separable into subspaces. To recover the ``clustering-friendly'' representation and facilitate the subsequent clustering, we propose a graph filtering approach by which a smooth representation is achieved. Specifically, it injects graph similarity into data features by applying a low-pass filter to extract useful data representations for clustering. Extensive experiments on image and document clustering datasets demonstrate that our method improves upon state-of-the-art subspace clustering techniques. Especially, its comparable performance with deep learning methods emphasizes the effectiveness of the simple graph filtering scheme for many real-world applications. An ablation study shows that graph filtering can remove noise, preserve structure in the image, and increase the separability of classes.

【34】 Effective Model Sparsification by Scheduled Grow-and-Prune Methods 标题：基于调度生长修剪方法的有效模型稀疏

作者：Xiaolong Ma,Minghai Qin,Fei Sun,Zejiang Hou,Kun Yuan,Yi Xu,Yanzhi Wang,Yen-Kuang Chen,Rong Jin,Yuan Xie 机构： DAMO Academy, Alibaba Group, Northeastern University, Princeton University 链接：https://arxiv.org/abs/2106.09857 摘要：深度神经网络（DNNs）是解决现实问题的有效方法。较大的DNN模型通常表现出更好的质量（如精度），但其过多的计算会导致较长的训练和推理时间。模型稀疏化可以在保持模型质量的同时减少计算量和内存开销。现有的稀疏化算法大多是单向地去除权值，而其他算法则是随机或贪婪地在每一层中寻找权值的一小部分。算法的低效性降低了可实现的稀疏性水平。此外，许多算法仍然需要预先训练密集模型，因此内存占用大，训练时间长。本文提出了一种新的无需对稠密模型进行预训练的计划增长与剪枝（GaP）方法。它解决了以往工作的不足之处，通过反复增长一个子集的层密集，然后修剪回稀疏后，一些训练。实验表明，在图像分类、目标检测、三维物体分割和平移等多种任务中，该模型在80%的稀疏度下都能达到或超过高度优化的稠密模型的质量。它们也优于其他最先进的剪枝方法，包括从预先训练的密集模型中剪枝。例如，通过GaP获得的90%稀疏ResNet-50在ImageNet上达到77.9%的top-1精度，使SOTA结果提高了1.5%。 摘要：Deep neural networks (DNNs) are effective in solving many real-world problems. Larger DNN models usually exhibit better quality (e.g., accuracy) but their excessive computation results in long training and inference time. Model sparsification can reduce the computation and memory cost while maintaining model quality. Most existing sparsification algorithms unidirectionally remove weights, while others randomly or greedily explore a small subset of weights in each layer. The inefficiency of the algorithms reduces the achievable sparsity level. In addition, many algorithms still require pre-trained dense models and thus suffer from large memory footprint and long training time. In this paper, we propose a novel scheduled grow-and-prune (GaP) methodology without pre-training the dense models. It addresses the shortcomings of the previous works by repeatedly growing a subset of layers to dense and then pruning back to sparse after some training. Experiments have shown that such models can match or beat the quality of highly optimized dense models at 80% sparsity on a variety of tasks, such as image classification, objective detection, 3D object part segmentation, and translation. They also outperform other state-of-the-art (SOTA) pruning methods, including pruning from pre-trained dense models. As an example, a 90% sparse ResNet-50 obtained via GaP achieves 77.9% top-1 accuracy on ImageNet, improving the SOTA results by 1.5%.

【35】 Disinformation, Stochastic Harm, and Costly Filtering: A Principal-Agent Analysis of Regulating Social Media Platforms 标题：虚假信息、随机伤害和代价高昂的过滤：监管社交媒体平台的委托代理分析

作者：Shehroze Khan,James R. Wright 机构：Department of Computing Science, Alberta Machine Intelligence Institute (Amii), University of Alberta 链接：https://arxiv.org/abs/2106.09847 摘要：在Facebook等社交媒体平台上传播辟谣，对社会有害。这种危害可能表现为公共话语的逐渐退化；但它也可以采取突发戏剧性事件的形式，比如最近在国会山发生的暴动。这些平台本身处于防止造谣传播的最佳位置，因为它们最容易获得相关数据并拥有使用这些数据的专业知识。然而，过滤造谣是昂贵的，不仅是因为实施过滤算法或采用人工过滤的努力，而且因为删除这种高度病毒性的内容影响用户增长，从而潜在的广告收入。由于有害内容的成本由其他实体承担，因此该平台将没有动力以社会最佳水平进行过滤。这一问题类似于环境监管问题，即不良事件的成本并非由企业直接承担，企业的缓解努力不可观察，有害后果与具体失败之间的因果关系难以证明。在环境监管领域，解决这一问题的一个办法是进行成本高昂的监测，以确保企业按照规定的规则采取充分的预防措施。然而，对造谣信息进行分类是有效果的，因此一个固定的规则随着时间的推移会变得不那么有效。将我们的域编码为马尔可夫决策过程，我们证明了基于静态规则的惩罚，无论多大，都不能激励平台进行充分的过滤。基于自适应规则的惩罚可以激励最佳努力，但与直觉相反，只有当监管者对有害事件作出足够的过度反应，要求更高水平的过滤。 摘要：The spread of disinformation on social media platforms such as Facebook is harmful to society. This harm can take the form of a gradual degradation of public discourse; but it can also take the form of sudden dramatic events such as the recent insurrection on Capitol Hill. The platforms themselves are in the best position to prevent the spread of disinformation, as they have the best access to relevant data and the expertise to use it. However, filtering disinformation is costly, not only for implementing filtering algorithms or employing manual filtering effort, but also because removing such highly viral content impacts user growth and thus potential advertising revenue. Since the costs of harmful content are borne by other entities, the platform will therefore have no incentive to filter at a socially-optimal level. This problem is similar to the problem of environmental regulation, in which the costs of adverse events are not directly borne by a firm, the mitigation effort of a firm is not observable, and the causal link between a harmful consequence and a specific failure is difficult to prove. In the environmental regulation domain, one solution to this issue is to perform costly monitoring to ensure that the firm takes adequate precautions according a specified rule. However, classifying disinformation is performative, and thus a fixed rule becomes less effective over time. Encoding our domain as a Markov decision process, we demonstrate that no penalty based on a static rule, no matter how large, can incentivize adequate filtering by the platform. Penalties based on an adaptive rule can incentivize optimal effort, but counterintuitively, only if the regulator sufficiently overreacts to harmful events by requiring a greater-than-optimal level of filtering.

【36】 Many Agent Reinforcement Learning Under Partial Observability 标题：部分可观测性下的多智能体强化学习

作者：Keyang He,Prashant Doshi,Bikramjit Banerjee 机构：Department of Computer Science, University of Georgia, University of Southern Mississippi 链接：https://arxiv.org/abs/2106.09825 摘要：最近对多智能体强化学习（MARL）的新兴趣产生了一系列令人印象深刻的技术，这些技术利用了深度强化学习，主要是演员-评论家体系结构，并且可以应用于有限范围的可观察性和通信设置。然而，这项工作的一个持续的局限性是，当涉及到基于联合行动的表示时，维数的诅咒，它随着代理的数量呈指数增长。在本文中，我们直接关注可伸缩性这一挑战。我们将行动匿名性的关键观点（导致联合行动的排列不变性）应用于最近提出的两种深层泥灰岩算法MADDPG和IA2C，并将这些实例与另一种利用行动匿名性的最新技术，即平均场泥灰岩进行了比较。我们使用最近引入的实用主义领域，证明了我们的实例可以在比平均场方法更广泛的一类代理网络中学习最优行为。 摘要：Recent renewed interest in multi-agent reinforcement learning (MARL) has generated an impressive array of techniques that leverage deep reinforcement learning, primarily actor-critic architectures, and can be applied to a limited range of settings in terms of observability and communication. However, a continuing limitation of much of this work is the curse of dimensionality when it comes to representations based on joint actions, which grow exponentially with the number of agents. In this paper, we squarely focus on this challenge of scalability. We apply the key insight of action anonymity, which leads to permutation invariance of joint actions, to two recently presented deep MARL algorithms, MADDPG and IA2C, and compare these instantiations to another recent technique that leverages action anonymity, viz., mean-field MARL. We show that our instantiations can learn the optimal behavior in a broader class of agent networks than the mean-field method, using a recently introduced pragmatic domain.

【37】 Deep reinforcement learning with automated label extraction from clinical reports accurately classifies 3D MRI brain volumes 标题：从临床报告中自动提取标签的深度强化学习精确地对3D MRI脑体积进行分类

作者：Joseph Stember,Hrithwik Shalu 机构：Memorial Sloan Kettering Cancer Center, New York, NY, US, Indian Institute of Technology, Madras, Chennai, India 链接：https://arxiv.org/abs/2106.09812 摘要：目的：图像分类可能是成像人工智能中最基本的任务。但是，为图像添加标签既费时又繁琐。我们最近证明了强化学习（RL）可以对MRI脑图像的2D切片进行高精度的分类。在这里，我们做了两个重要的步骤来加速图像分类：首先，我们自动从临床报告中提取类别标签。其次，我们将先前的二维分类工作扩展到我们机构的全三维图像体。因此，我们按照以下步骤进行：在第1部分中，我们使用SBERT自然语言处理方法自动从报表中提取标签。然后，在第2部分中，我们使用这些标签和RL来训练三维图像体的Deep-Q分类网络（DQN）。方法：第一部分，我们用90份放射报告来训练SBERT。然后，我们使用训练好的SBERT来预测第二部分中使用的类别标签。在第二部分中，我们应用了多步图像分类，以允许使用3D卷积和TD（0）Q学习的联合深度Q学习。我们训练了一组90张图片。我们在一组单独的61幅图像上进行了测试，同样使用了第1部分中训练过的SBERT从患者报告中预测的类别。为了进行比较，我们还在使用相同标签的同一组训练和测试图像上训练和测试了一个有监督的深度学习分类网络。结果：第1部分：通过对放射报告语料库的训练，SBERT模型对正常扫描和含转移扫描的准确率均为100%。第2部分：然后，使用这些标签，尽管监督方法很快过拟合训练数据，并且正如预期的那样在测试集上表现不佳（66%的准确率，只是超过随机猜测），但是强化学习方法达到了92%的准确率。结果具有统计学意义，p值为3.1x10^-5。 摘要：Purpose: Image classification is perhaps the most fundamental task in imaging AI. However, labeling images is time-consuming and tedious. We have recently demonstrated that reinforcement learning (RL) can classify 2D slices of MRI brain images with high accuracy. Here we make two important steps toward speeding image classification: Firstly, we automatically extract class labels from the clinical reports. Secondly, we extend our prior 2D classification work to fully 3D image volumes from our institution. Hence, we proceed as follows: in Part 1, we extract labels from reports automatically using the SBERT natural language processing approach. Then, in Part 2, we use these labels with RL to train a classification Deep-Q Network (DQN) for 3D image volumes. Methods: For Part 1, we trained SBERT with 90 radiology report impressions. We then used the trained SBERT to predict class labels for use in Part 2. In Part 2, we applied multi-step image classification to allow for combined Deep-Q learning using 3D convolutions and TD(0) Q learning. We trained on a set of 90 images. We tested on a separate set of 61 images, again using the classes predicted from patient reports by the trained SBERT in Part 1. For comparison, we also trained and tested a supervised deep learning classification network on the same set of training and testing images using the same labels. Results: Part 1: Upon training with the corpus of radiology reports, the SBERT model had 100% accuracy for both normal and metastasis-containing scans. Part 2: Then, using these labels, whereas the supervised approach quickly overfit the training data and as expected performed poorly on the testing set (66% accuracy, just over random guessing), the reinforcement learning approach achieved an accuracy of 92%. The results were found to be statistically significant, with a p-value of 3.1 x 10^-5.

【38】 LNN-EL: A Neuro-Symbolic Approach to Short-text Entity Linking 标题：LNN-EL：一种短文本实体链接的神经符号方法

作者：Hang Jiang,Sairam Gurajada,Qiuhao Lu,Sumit Neelam,Lucian Popa,Prithviraj Sen,Yunyao Li,Alexander Gray 机构：MIT, IBM Research, University of Oregon 备注：Accepted to ACL 2021 链接：https://arxiv.org/abs/2106.09795 摘要：实体链接（elentitylinking，EL）是通过将文本中提到的内容链接到知识图中的实体来消除歧义的任务，对于文本理解、问答或会话系统至关重要。由于上下文的限制，在短文本（例如，单个句子或问题）上的实体链接带来了特殊的挑战。虽然先前的方法使用启发式或黑盒神经方法，这里我们提出LNN-EL，一种神经符号方法，它结合了使用基于一阶逻辑的可解释规则的优点和神经学习的性能。即使受限于使用规则，LNN-EL也能与SotA黑盒神经方法进行竞争，具有可扩展性和可转移性的额外优势。特别地，我们证明了我们可以很容易地将现有的由专家给出的规则模板与多种类型的特征（先验、BERT编码、盒嵌入等）以及以前EL方法得到的分数进行融合，从而改进了这些方法。例如，在LC-QuAD-1.0数据集上，我们显示F1分数比之前的SotA增加了4$\%。最后，我们证明了使用逻辑所提供的归纳偏差会导致学习的规则在数据集之间传输良好，即使没有微调，同时保持高精度。 摘要：Entity linking (EL), the task of disambiguating mentions in text by linking them to entities in a knowledge graph, is crucial for text understanding, question answering or conversational systems. Entity linking on short text (e.g., single sentence or question) poses particular challenges due to limited context. While prior approaches use either heuristics or black-box neural methods, here we propose LNN-EL, a neuro-symbolic approach that combines the advantages of using interpretable rules based on first-order logic with the performance of neural learning. Even though constrained to using rules, LNN-EL performs competitively against SotA black-box neural approaches, with the added benefits of extensibility and transferability. In particular, we show that we can easily blend existing rule templates given by a human expert, with multiple types of features (priors, BERT encodings, box embeddings, etc), and even scores resulting from previous EL methods, thus improving on such methods. For instance, on the LC-QuAD-1.0 dataset, we show more than $4$\% increase in F1 score over previous SotA. Finally, we show that the inductive bias offered by using logic results in learned rules that transfer well across datasets, even without fine tuning, while maintaining high accuracy.

【39】 Multi-Task Learning and Adapted Knowledge Models for Emotion-Cause Extraction 标题：用于情感原因提取的多任务学习和自适应知识模型

作者：Elsbeth Turcan,Shuai Wang,Rishita Anubhai,Kasturi Bhattacharjee,Yaser Al-Onaizan,Smaranda Muresan 机构：Department of Computer Science, Columbia University, Data Science Institute, Columbia University, Amazon AI 备注：15 pages, 6 figures. Findings of ACL 2021 链接：https://arxiv.org/abs/2106.09790 摘要：在自然语言处理中，检测文本中表达的情感是一个被广泛研究的问题。然而，对更细粒度的情感分析的研究，如情感的成因，仍处于起步阶段。我们提出的解决方案，处理两个情感识别和情感原因检测在一个联合的方式。考虑到常识知识对理解隐含表达的情感及其原因起着重要的作用，我们提出了一种新的方法，通过自适应知识模型将常识知识与多任务学习相结合，进行情感分类和情感原因标注。当包含常识推理和多任务框架时，这两个任务的性能都得到了提高。我们提供了一个彻底的分析，以获得对模型性能的见解。 摘要：Detecting what emotions are expressed in text is a well-studied problem in natural language processing. However, research on finer grained emotion analysis such as what causes an emotion is still in its infancy. We present solutions that tackle both emotion recognition and emotion cause detection in a joint fashion. Considering that common-sense knowledge plays an important role in understanding implicitly expressed emotions and the reasons for those emotions, we propose novel methods that combine common-sense knowledge via adapted knowledge models with multi-task learning to perform joint emotion classification and emotion cause tagging. We show performance improvement on both tasks when including common-sense reasoning and a multitask framework. We provide a thorough analysis to gain insights into model performance.

【40】 Efficient Self-supervised Vision Transformers for Representation Learning 标题：用于表征学习的高效自监督视觉转换器

作者：Chunyuan Li,Jianwei Yang,Pengchuan Zhang,Mei Gao,Bin Xiao,Xiyang Dai,Lu Yuan,Jianfeng Gao 机构：Microsoft Research at Redmond, Microsoft Cloud + AI 备注：24 pages, 12 figures, file size 13.6MB 链接：https://arxiv.org/abs/2106.09785 摘要：研究了两种用于视觉表征学习的高效自监督视觉变换器（EsViT）。首先，我们通过一个全面的实证研究表明，具有稀疏自关注的多阶段体系结构可以显著降低建模复杂度，但代价是失去捕获图像区域间细粒度对应关系的能力。其次，我们提出了一个新的区域匹配预训练任务，使得模型能够捕捉到细粒度的区域依赖关系，从而显著提高了学习视觉表示的质量。我们的结果表明，结合这两种技术，EsViT在ImageNet线性探针评估中达到81.3%的top-1，在大约一个数量级的更高吞吐量下优于现有技术。当转移到下游线性分类任务时，EsViT在18个数据集中的17个数据集上优于其监督的同类。代码和模型将公开。 摘要：This paper investigates two techniques for developing efficient self-supervised vision transformers (EsViT) for visual representation learning. First, we show through a comprehensive empirical study that multi-stage architectures with sparse self-attentions can significantly reduce modeling complexity but with a cost of losing the ability to capture fine-grained correspondences between image regions. Second, we propose a new pre-training task of region matching which allows the model to capture fine-grained region dependencies and as a result significantly improves the quality of the learned vision representations. Our results show that combining the two techniques, EsViT achieves 81.3% top-1 on the ImageNet linear probe evaluation, outperforming prior arts with around an order magnitude of higher throughput. When transferring to downstream linear classification tasks, EsViT outperforms its supervised counterpart on 17 out of 18 datasets. The code and models will be publicly available.

【41】 Adapting the Function Approximation Architecture in Online Reinforcement Learning 标题：函数逼近结构在在线强化学习中的应用

作者：John D. Martin,Joseph Modayil 机构： large systems typically impose sparse con-nections from knowledge of the observation structure; forEqual contribution 1Department of Computing Science 链接：https://arxiv.org/abs/2106.09776 摘要：强化学习（RL）系统的性能取决于用于逼近值函数的计算结构。深度学习方法为从高维噪声观测值中逼近非线性函数提供了优化技术和结构。然而，流行的优化技术并不是为严格的增量在线更新而设计的。标准体系结构也不是为具有先验未知结构的观测而设计的：例如，光传感器随机分布在空间中。提出了一种在线RL预测算法，该算法采用自适应结构，能有效地发现有用的非线性特征。该算法在高维随机观测的空间域中进行评估。该算法的性能优于非自适应基线结构，接近给定侧信道信息的结构性能。在观测结构不可用的情况下，这些结果朝着可扩展的RL算法迈出了一步。 摘要：The performance of a reinforcement learning (RL) system depends on the computational architecture used to approximate a value function. Deep learning methods provide both optimization techniques and architectures for approximating nonlinear functions from noisy, high-dimensional observations. However, prevailing optimization techniques are not designed for strictly-incremental online updates. Nor are standard architectures designed for observations with an a priori unknown structure: for example, light sensors randomly dispersed in space. This paper proposes an online RL prediction algorithm with an adaptive architecture that efficiently finds useful nonlinear features. The algorithm is evaluated in a spatial domain with high-dimensional, stochastic observations. The algorithm outperforms non-adaptive baseline architectures and approaches the performance of an architecture given side-channel information. These results are a step towards scalable RL algorithms for more general problems, where the observation structure is not available.

【42】 Autoencoder-based cleaning in probabilistic databases 标题：概率数据库中基于自动编码器的清理

作者：R. R. Mauritz,F. P. J. Nijweide,J. Goseling,M. van Keulen 机构：University of Twente, Enschede, NL 备注：Submitted to ACM Journal of Data and Information Quality, Special Issue on Deep Learning for Data Quality 链接：https://arxiv.org/abs/2106.09764 摘要：在数据集成领域，数据的提取、合并和合并往往会遇到数据质量问题。概率数据集成方法表示概率数据库中有关不确定性等问题的信息。在本文中，我们提出了一个数据清洗自动编码器能够近自动的数据质量改善。它学习数据中的结构和依赖关系，以识别和更正可疑值。给出了一个理论框架，实验表明该方法能有效地去除分类概率数据和数值概率数据中的噪声。我们的方法不需要干净的数据。然而，我们确实证明了手动清理一小部分数据可以显著提高性能。 摘要：In the field of data integration, data quality problems are often encountered when extracting, combining, and merging data. The probabilistic data integration approach represents information about such problems as uncertainties in a probabilistic database. In this paper, we propose a data-cleaning autoencoder capable of near-automatic data quality improvement. It learns the structure and dependencies in the data to identify and correct doubtful values. A theoretical framework is provided, and experiments show that it can remove significant amounts of noise from categorical and numeric probabilistic data. Our method does not require clean data. We do, however, show that manually cleaning a small fraction of the data significantly improves performance.

【43】 Unsupervised Resource Allocation with Graph Neural Networks 标题：基于图神经网络的无监督资源分配

作者：Miles Cranmer,Peter Melchior,Brian Nord 机构：Princeton University, Princeton, NJ , USA, Fermilab, Batavia, IL , USA 备注：Accepted to PMLR/contributed oral at NeurIPS 2020 Pre-registration Workshop. Code at this https URL 链接：https://arxiv.org/abs/2106.09761 摘要：我们提出了一种通过学习如何在无监督的方式分配资源来最大化全局效用函数的方法。我们期望分配目标之间的相互作用是重要的，因此建议学习具有GNN的近似最优分配策略的报酬结构。通过放松资源约束，我们可以采用基于梯度的优化，而不是更标准的进化算法。我们的算法是由现代天文学中的一个问题驱动的，在这个问题中，我们需要根据10^9$个星系中有限的初始信息来选择那些详细测量将导致对宇宙组成的最佳推断的星系。我们的技术提供了一种灵活地学习分配策略的方法，只需要针对感兴趣的物理和测量过程的前向模拟器。我们期望我们的技术也能在一系列资源分配问题中得到应用。 摘要：We present an approach for maximizing a global utility function by learning how to allocate resources in an unsupervised way. We expect interactions between allocation targets to be important and therefore propose to learn the reward structure for near-optimal allocation policies with a GNN. By relaxing the resource constraint, we can employ gradient-based optimization in contrast to more standard evolutionary algorithms. Our algorithm is motivated by a problem in modern astronomy, where one needs to select-based on limited initial information-among $10^9$ galaxies those whose detailed measurement will lead to optimal inference of the composition of the universe. Our technique presents a way of flexibly learning an allocation strategy by only requiring forward simulators for the physics of interest and the measurement process. We anticipate that our technique will also find applications in a range of resource allocation problems.

【44】 CIRA Guide to Custom Loss Functions for Neural Networks in Environmental Sciences -- Version 1 标题：环境科学中神经网络的自定义损失函数的CIRA指南-第1版

作者：Imme Ebert-Uphoff,Ryan Lagerquist,Kyle Hilburn,Yoonjin Lee,Katherine Haynes,Jason Stock,Christina Kumler,Jebb Q. Stewart 机构：CIRA, ECE, CIRA, NOAA-GSL, CS, CIRES, NOAA-GSL 备注：37 pages 链接：https://arxiv.org/abs/2106.09757 摘要：神经网络在环境科学中的应用越来越广泛。此外，神经网络模型是通过最小化损失函数来训练的，对于环境科学应用来说，非常谨慎地选择损失函数是至关重要的，因为它决定了要优化什么。标准损失函数并不涵盖环境科学的所有需求，这使得科学家能够开发自己的自定义损失函数，以便能够实现环境科学中已经开发的许多经典性能度量，包括为空间模型验证而开发的度量，这一点很重要。然而，只有很少的资源能够全面地涵盖自定义损失函数开发的基础知识，据我们所知，没有一个资源关注于环境科学家的需求。本文试图通过提供如何编写面向环境科学应用的自定义损失函数的指南来填补这一空白。主题包括编写自定义损失函数的基础知识、常见陷阱、损失函数中使用的函数、示例（例如分数技能分数作为损失函数）、如何合并物理约束、离散化和软离散化，以及概念（例如焦点损失、稳健损失和自适应损失）。虽然本指南中目前提供了使用Keras的Python和TensorFlow后端的示例，但基本概念也适用于其他环境，例如使用PyTorch的Python。类似地，虽然这里提供的示例损失函数来自气象学，但这些只是如何创建自定义损失函数的示例。环境科学中的其他领域对自定义损失函数有着非常相似的需求，例如，有效地评估空间预测，这里讨论的概念也可以在那里应用。所有代码示例都在GitHub存储库中提供。 摘要：Neural networks are increasingly used in environmental science applications. Furthermore, neural network models are trained by minimizing a loss function, and it is crucial to choose the loss function very carefully for environmental science applications, as it determines what exactly is being optimized. Standard loss functions do not cover all the needs of the environmental sciences, which makes it important for scientists to be able to develop their own custom loss functions so that they can implement many of the classic performance measures already developed in environmental science, including measures developed for spatial model verification. However, there are very few resources available that cover the basics of custom loss function development comprehensively, and to the best of our knowledge none that focus on the needs of environmental scientists. This document seeks to fill this gap by providing a guide on how to write custom loss functions targeted toward environmental science applications. Topics include the basics of writing custom loss functions, common pitfalls, functions to use in loss functions, examples such as fractions skill score as loss function, how to incorporate physical constraints, discrete and soft discretization, and concepts such as focal, robust, and adaptive loss. While examples are currently provided in this guide for Python with Keras and the TensorFlow backend, the basic concepts also apply to other environments, such as Python with PyTorch. Similarly, while the sample loss functions provided here are from meteorology, these are just examples of how to create custom loss functions. Other fields in the environmental sciences have very similar needs for custom loss functions, e.g., for evaluating spatial forecasts effectively, and the concepts discussed here can be applied there as well. All code samples are provided in a GitHub repository.

【45】 PyKale: Knowledge-Aware Machine Learning from Multiple Sources in Python 标题：PyKale：基于Python的多源知识机器学习

作者：Haiping Lu,Xianyuan Liu,Robert Turner,Peizhen Bai,Raivo E Koot,Shuo Zhou,Mustafa Chasmai,Lawrence Schobs 机构：The University of Sheffield, Sheffield, United Kingdom, Indian Institute of Technology, Delhi, New Delhi, India 备注：This library is available at this https URL 链接：https://arxiv.org/abs/2106.09756 摘要：机器学习是一种多学科交叉研究的通用技术。然而，当大多数机器学习工具在不同领域分别开发时，在跨越学科界限方面存在着明显的障碍。我们介绍Pykale-一个Python库，用于图形、图像、文本和视频的知识感知机器学习，以支持和加速跨学科研究。我们在标准软件工程实践的基础上制定了新的绿色机器学习准则，并提出了一种新的基于流水线的应用程序编程接口（API）。PyKale专注于利用来自多个来源的知识进行准确和可解释的预测，从而通过最新的深度学习和降维模型支持多模式学习和迁移学习（特别是领域适应）。我们在Pytork上建立PyKale，并利用丰富的Pytork生态系统。我们基于管道的API设计加强了标准化和极简主义，通过减少重复和冗余、重用现有资源和跨领域回收学习模型，拥抱绿色机器学习概念。我们通过生物信息学、知识图、图像/视频识别和医学成像的例子来展示它的跨学科性质。 摘要：Machine learning is a general-purpose technology holding promises for many interdisciplinary research problems. However, significant barriers exist in crossing disciplinary boundaries when most machine learning tools are developed in different areas separately. We present Pykale - a Python library for knowledge-aware machine learning on graphs, images, texts, and videos to enable and accelerate interdisciplinary research. We formulate new green machine learning guidelines based on standard software engineering practices and propose a novel pipeline-based application programming interface (API). PyKale focuses on leveraging knowledge from multiple sources for accurate and interpretable prediction, thus supporting multimodal learning and transfer learning (particularly domain adaptation) with latest deep learning and dimensionality reduction models. We build PyKale on PyTorch and leverage the rich PyTorch ecosystem. Our pipeline-based API design enforces standardization and minimalism, embracing green machine learning concepts via reducing repetitions and redundancy, reusing existing resources, and recycling learning models across areas. We demonstrate its interdisciplinary nature via examples in bioinformatics, knowledge graph, image/video recognition, and medical imaging.

【46】 Optimizing robotic swarm based construction tasks 标题：基于机器人群的施工任务优化

作者：Teshan Liyanage,Subha Fernando 机构：Universiy of Moratuwa, Colombo, Sri Lanka, University of Moratuwa 备注：4 pages, 3 figures, submitted to 2021 7th International Conference on Control, Automation and Robotics (ICCAR) Singapore 链接：https://arxiv.org/abs/2106.09749 摘要：自然界中的群居昆虫，如蚂蚁、白蚁和蜜蜂，在一个非常有效的过程中协同构建它们的群体。在这些昆虫群落中，每一种昆虫都参与了各自的构建任务，表现出个体实体的冗余和平行行为。但是，由于现有的群机器人构建方法的局限性，这些群机器人行为的机器人适应性还没有在足够大的范围内适应现实世界的普遍应用。本文提出了一种结合现有群构造方法的群机器人系统，该系统能够以优化的方式构造给定的二维形状。 摘要：Social insects in nature such as ants, termites and bees construct their colonies collaboratively in a very efficient process. In these swarms, each insect contributes to the construction task individually showing redundant and parallel behavior of individual entities. But the robotics adaptations of these swarm's behaviors haven't yet made it to the real world at a large enough scale of commonly being used due to the limitations in the existing approaches to the swarm robotics construction. This paper presents an approach that combines the existing swarm construction approaches which results in a swarm robotic system, capable of constructing a given 2 dimensional shape in an optimized manner.

【47】 An Improved Single Step Non-autoregressive Transformer for Automatic Speech Recognition 标题：一种用于自动语音识别的改进单步非自回归变换器

作者：Ruchao Fan,Wei Chu,Peng Chang,Jing Xiao,Abeer Alwan 机构：Dept. of Electrical and Computer Engineering, University of California, Los Angeles, USA, PAII Inc., USA 备注：To appear in Interspeech2021 链接：https://arxiv.org/abs/2106.09885 摘要：非自回归机制可以显著减少语音转换器的推理时间，特别是当使用单步变量时。基于CTC校准的单步非自回归Transformer（CASS-NAT）的前期工作表明，与自回归Transformer（AT）相比，实时系数（RTF）有很大的提高。在这项工作中，我们提出了几种提高端到端CASS-NAT准确性的方法，并进行了性能分析。首先，卷积增强的自我注意块应用于编码器和解码器模块。其次，我们建议扩展每个令牌的触发掩码（声学边界），以增加CTC对齐的鲁棒性。此外，采用迭代损失函数来增强低层参数的梯度更新。在不使用外部语言模型的情况下，改进的CASS-NAT在Librispeech test clean/other测试集上的WER为3.1%/7.2%，在Aisell1测试集上的CER为5.4%，相对WER/CER提高了7%～21%。为了进行分析，我们绘制了解码器中的注意权重分布图，以可视化符号级声学嵌入之间的关系。当声学嵌入被可视化时，我们发现它们具有与单词嵌入相似的行为，这就解释了为什么改进的CASS-NAT的性能与AT相似。 摘要：Non-autoregressive mechanisms can significantly decrease inference time for speech transformers, especially when the single step variant is applied. Previous work on CTC alignment-based single step non-autoregressive transformer (CASS-NAT) has shown a large real time factor (RTF) improvement over autoregressive transformers (AT). In this work, we propose several methods to improve the accuracy of the end-to-end CASS-NAT, followed by performance analyses. First, convolution augmented self-attention blocks are applied to both the encoder and decoder modules. Second, we propose to expand the trigger mask (acoustic boundary) for each token to increase the robustness of CTC alignments. In addition, iterated loss functions are used to enhance the gradient update of low-layer parameters. Without using an external language model, the WERs of the improved CASS-NAT, when using the three methods, are 3.1%/7.2% on Librispeech test clean/other sets and the CER is 5.4% on the Aishell1 test set, achieving a 7%~21% relative WER/CER improvement. For the analyses, we plot attention weight distributions in the decoders to visualize the relationships between token-level acoustic embeddings. When the acoustic embeddings are visualized, we find that they have a similar behavior to word embeddings, which explains why the improved CASS-NAT performs similarly to AT.

【48】 Causal Bias Quantification for Continuous Treatment 标题：连续治疗的因果偏差量化方法

作者：Gianluca Detommaso,Michael Brückner,Philip Schulz,Victor Chernozhukov 机构：Massachusetts Institute of Technology & Amazon 链接：https://arxiv.org/abs/2106.09762 摘要：在这项工作中，我们开发了一个新的特征边际因果效应和因果偏见的连续治疗设置。我们证明了它们可以表示为关于条件概率分布的期望，这可以通过标准的统计和概率方法来估计。期望中的所有项都可以通过自动微分来计算，对于高度非线性的模型也是如此。我们进一步发展了一个新的通过协变量调整的因果效应可识别性的完整标准，如果满足该标准，则偏差等于零。我们研究了我们的框架在三种不同情景下的有效性：混杂、过度控制和内生选择偏差下的线性模型；一种非线性模型，由于数据丢失而无法完全辨识；他汀类药物与动脉粥样硬化性心血管疾病的模拟医学研究。 摘要：In this work we develop a novel characterization of marginal causal effect and causal bias in the continuous treatment setting. We show they can be expressed as an expectation with respect to a conditional probability distribution, which can be estimated via standard statistical and probabilistic methods. All terms in the expectations can be computed via automatic differentiation, also for highly non-linear models. We further develop a new complete criterion for identifiability of causal effects via covariate adjustment, showing the bias equals zero if the criterion is met. We study the effectiveness of our framework in three different scenarios: linear models under confounding, overcontrol and endogenous selection bias; a non-linear model where full identifiability cannot be achieved because of missing data; a simulated medical study of statins and atherosclerotic cardiovascular disease.

【49】 Optimising simulations for diphoton production at hadron colliders using amplitude neural networks 标题：用振幅神经网络优化强子对撞机双光子产生模拟

作者：Joseph Aylett-Bullock,Simon Badger,Ryan Moodie 机构：Institute for Particle Physics Phenomenology, Department of Physics, Durham University, Durham, Institute for Data Science, Durham University, Durham, DH,LE, United Kingdom 备注：31 pages, 12 figures, 2 tables 链接：https://arxiv.org/abs/2106.09474 摘要：机器学习技术有可能极大地优化事件生成和模拟。我们继续研究使用神经网络来近似矩阵元素的高多重散射过程。我们专注于通过胶子聚变产生回路诱导双质子的情况，并发展了一种可应用于强子对撞机观测的真实模拟方法。神经网络训练使用在NJET C++库中实现的单回路振幅，并与夏尔巴蒙特卡罗事件发生器接口，在这里我们对2美元-3美元和2美元-4美元散射问题进行详细研究。我们还考虑了当改变影响相空间和神经网络模拟的可靠性的运动切割时，训练网络的性能。 摘要：Machine learning technology has the potential to dramatically optimise event generation and simulations. We continue to investigate the use of neural networks to approximate matrix elements for high-multiplicity scattering processes. We focus on the case of loop-induced diphoton production through gluon fusion and develop a realistic simulation method that can be applied to hadron collider observables. Neural networks are trained using the one-loop amplitudes implemented in the NJet C++ library and interfaced to the Sherpa Monte Carlo event generator where we perform a detailed study for $2\to3$ and $2\to4$ scattering problems. We also consider how the trained networks perform when varying the kinematic cuts effecting the phase space and the reliability of the neural network simulations.