机器人相关学术速递[7.20]

访问www.arxivdaily.com获取含摘要速递，涵盖CS|物理|数学|经济|统计|金融|生物|电气领域，更有搜索、收藏、发帖等功能！点击阅读原文即可访问

cs.RO机器人相关，共计33篇

【1】 Know Thyself: Transferable Visuomotor Control Through Robot-Awareness 标题：认识自我：通过机器人感知实现可转移的视觉运动控制

作者：Edward S. Hu,Kun Huang,Oleh Rybkin,Dinesh Jayaraman 机构：GRASP Lab, Department of CIS, University of Pennsylvania 备注：Website: this https URL 链接：https://arxiv.org/abs/2107.09047 摘要：在新机器人上从头开始训练视觉运动机器人控制器通常需要生成大量特定于机器人的数据。我们能否利用以前在另一个机器人上收集的数据来减少甚至完全消除对机器人特定数据的需求？我们提出了一个“机器人感知”的解决方案范例，利用现成的机器人“自我知识”，如本体感觉，运动学和相机校准来实现这一点。首先，我们学习模块化动力学模型，将可转移的、机器人不可知的世界动力学模块与机器人特定的、分析的机器人动力学模块配对。接下来，我们设置视觉规划成本，区分机器人自身和世界。我们在模拟桌面操作任务和真实机器人上的实验表明，这些插件的改进极大地提高了视觉运动控制器的可转移性，甚至首次允许Zero-Shot转移到新机器人上。项目网站：https://hueds.github.io/rac/ 摘要：Training visuomotor robot controllers from scratch on a new robot typically requires generating large amounts of robot-specific data. Could we leverage data previously collected on another robot to reduce or even completely remove this need for robot-specific data? We propose a "robot-aware" solution paradigm that exploits readily available robot "self-knowledge" such as proprioception, kinematics, and camera calibration to achieve this. First, we learn modular dynamics models that pair a transferable, robot-agnostic world dynamics module with a robot-specific, analytical robot dynamics module. Next, we set up visual planning costs that draw a distinction between the robot self and the world. Our experiments on tabletop manipulation tasks in simulation and on real robots demonstrate that these plug-in improvements dramatically boost the transferability of visuomotor controllers, even permitting zero-shot transfer onto new robots for the very first time. Project website: https://hueds.github.io/rac/

【2】 Playful Interactions for Representation Learning 标题：用于表征学习的趣味互动

作者：Sarah Young,Jyothish Pari,Pieter Abbeel,Lerrel Pinto 机构：UC Berkeley, NYU 链接：https://arxiv.org/abs/2107.09046 摘要：视觉模仿学习的一个关键挑战是为给定的任务收集大量的专家演示。虽然通过遥操作方法和使用低成本辅助工具，收集人类演示的方法变得越来越容易，但我们通常仍然需要为每项任务提供100-1000个演示，以学习视觉表示和策略。为了解决这个问题，我们转向另一种不需要特定任务演示的数据形式——播放。游戏是儿童在早期学习中学习一系列技能、行为和视觉表现的基本方法。重要的是，游戏数据是多样的，任务不可知，而且获取成本相对较低。在这项工作中，我们建议以自我监督的方式使用有趣的互动来学习下游任务的视觉表现。我们在19个不同的环境中收集了2小时有趣的数据，并使用自我预测学习来提取视觉表征。基于这些表示，我们使用模仿学习来训练策略，用于两个下游任务：推送和堆叠。我们证明了我们的可视化表示比标准的行为克隆具有更好的通用性，并且只需要一半的所需演示次数就可以获得类似的性能。我们的表示，这是从零开始训练，比较有利与ImageNet预训练表示。最后，我们对不同的预训练模式对下游任务学习的影响进行了实验分析。 摘要：One of the key challenges in visual imitation learning is collecting large amounts of expert demonstrations for a given task. While methods for collecting human demonstrations are becoming easier with teleoperation methods and the use of low-cost assistive tools, we often still require 100-1000 demonstrations for every task to learn a visual representation and policy. To address this, we turn to an alternate form of data that does not require task-specific demonstrations -- play. Playing is a fundamental method children use to learn a set of skills and behaviors and visual representations in early learning. Importantly, play data is diverse, task-agnostic, and relatively cheap to obtain. In this work, we propose to use playful interactions in a self-supervised manner to learn visual representations for downstream tasks. We collect 2 hours of playful data in 19 diverse environments and use self-predictive learning to extract visual representations. Given these representations, we train policies using imitation learning for two downstream tasks: Pushing and Stacking. We demonstrate that our visual representations generalize better than standard behavior cloning and can achieve similar performance with only half the number of required demonstrations. Our representations, which are trained from scratch, compare favorably against ImageNet pretrained representations. Finally, we provide an experimental analysis on the effects of different pretraining modes on downstream task learning.

【3】 Learning compliant grasping and manipulation by teleoperation with adaptive force control 标题：通过自适应力控制的遥操作学习柔顺的抓取和操作

作者：Chao Zeng,Shuang Li,Yiming Jiang,Qiang Li,Zhaopeng Chen,Chenguang Yang,Jianwei Zhang 机构： Universit¨at Hamburg, Hunan University, Bielefeld University, University of the West of England 链接：https://arxiv.org/abs/2107.08996 摘要：在这项工作中，我们致力于改善机器人的灵巧能力，利用视觉传感和自适应力控制。TeachNet是一个基于视觉的遥操作学习框架，用于将人类手的姿势映射到多指机器人手。我们增加TeachNet，它最初是基于一个不精确的运动学映射和位置伺服，与仿生学习为基础的柔顺控制算法灵巧的操作任务。该柔顺控制器以TeachNet映射的机器人关节角度作为期望目标，计算期望的关节力矩。它是由人体运动学习中仿生控制策略的计算模型导出的，它允许在参考关节角度轨迹的执行过程中在线调整控制变量（阻抗和前馈力）。同时自适应的阻抗和前馈配置文件使机器人能够与环境进行互动，在一个顺从的方式。我们的方法已经在物理模拟的多个任务中得到验证，即抓取、开门、翻盖和触摸鼠标，并且显示出比现有的位置控制和基于固定增益的力控制方法更可靠的性能。 摘要：In this work, we focus on improving the robot's dexterous capability by exploiting visual sensing and adaptive force control. TeachNet, a vision-based teleoperation learning framework, is exploited to map human hand postures to a multi-fingered robot hand. We augment TeachNet, which is originally based on an imprecise kinematic mapping and position-only servoing, with a biomimetic learning-based compliance control algorithm for dexterous manipulation tasks. This compliance controller takes the mapped robotic joint angles from TeachNet as the desired goal, computes the desired joint torques. It is derived from a computational model of the biomimetic control strategy in human motor learning, which allows adapting the control variables (impedance and feedforward force) online during the execution of the reference joint angle trajectories. The simultaneous adaptation of the impedance and feedforward profiles enables the robot to interact with the environment in a compliant manner. Our approach has been verified in multiple tasks in physics simulation, i.e., grasping, opening-a-door, turning-a-cap, and touching-a-mouse, and has shown more reliable performances than the existing position control and the fixed-gain-based force control approaches.

【4】 CodeMapping: Real-Time Dense Mapping for Sparse SLAM using Compact Scene Representations 标题：CodeMapping：使用紧凑场景表示的稀疏SLAM实时密集映射

作者：Hidenobu Matsuki,Raluca Scona,Jan Czarnowski,Andrew J. Davison 备注：Accepted to IEEE Robotics and Automation Letters (RA-L) 2021 链接：https://arxiv.org/abs/2107.08994 摘要：我们提出了一个新的密集映射框架稀疏视觉SLAM系统，利用紧凑的场景表示。最先进的稀疏视觉SLAM系统提供了对摄像机轨迹和地标位置的准确可靠的估计。虽然这些稀疏地图对于定位很有用，但它们不能用于其他任务，如避障或场景理解。本文提出了一个稠密映射框架来补充稀疏视觉SLAM系统，该框架以SLAM系统产生的摄像机姿态、关键帧和稀疏点为输入，预测每个关键帧的稠密深度图像。我们建立在CodeSLAM的基础上，利用变分自动编码器（VAE）对稀疏SLAM的强度、稀疏深度和重投影误差图像进行预测，得到具有不确定性的稠密深度图。利用VAE，我们可以通过多视图优化来细化密集的深度图像，从而提高重叠帧的一致性。我们的映射器以松散耦合的方式在独立线程中与SLAM系统并行运行。这种灵活的设计允许在不延迟主SLAM过程的情况下与任意度量稀疏SLAM系统集成。我们的稠密映射器不仅可以用于局部映射，还可以通过TSDF融合进行全局一致的稠密三维重建。我们演示了使用ORB-SLAM3运行的系统，并显示了精确的密集深度估计，这可以实现机器人和增强现实等应用。 摘要：We propose a novel dense mapping framework for sparse visual SLAM systems which leverages a compact scene representation. State-of-the-art sparse visual SLAM systems provide accurate and reliable estimates of the camera trajectory and locations of landmarks. While these sparse maps are useful for localization, they cannot be used for other tasks such as obstacle avoidance or scene understanding. In this paper we propose a dense mapping framework to complement sparse visual SLAM systems which takes as input the camera poses, keyframes and sparse points produced by the SLAM system and predicts a dense depth image for every keyframe. We build on CodeSLAM and use a variational autoencoder (VAE) which is conditioned on intensity, sparse depth and reprojection error images from sparse SLAM to predict an uncertainty-aware dense depth map. The use of a VAE then enables us to refine the dense depth images through multi-view optimization which improves the consistency of overlapping frames. Our mapper runs in a separate thread in parallel to the SLAM system in a loosely coupled manner. This flexible design allows for integration with arbitrary metric sparse SLAM systems without delaying the main SLAM process. Our dense mapper can be used not only for local mapping but also globally consistent dense 3D reconstruction through TSDF fusion. We demonstrate our system running with ORB-SLAM3 and show accurate dense depth estimation which could enable applications such as robotics and augmented reality.

【5】 Hierarchical Few-Shot Imitation with Skill Transition Models 标题：基于技能转移模型的层次化Few-Shot模仿

作者：Kourosh Hakhamaneshi,Ruihan Zhao,Albert Zhan,Pieter Abbeel,Michael Laskin 机构：University of California, Berkeley, equal contribution 链接：https://arxiv.org/abs/2107.08981 摘要：自治代理的一个理想特性是既能解决长时间范围的问题，又能推广到不可见的任务。数据驱动技能学习的最新进展表明，从离线数据中提取行为先验知识可以使agent通过强化学习解决具有挑战性的长时间任务。然而，一般化的任务中看不到的行为训练仍然是一个突出的挑战。为此，我们提出了一种从离线数据中提取技能，并利用这些技能来推广到给定一些下游演示的不可见任务的算法，即技能转移模型（FIST）。FIST学习逆技能动力学模型，距离函数，并利用半参数方法进行模拟。我们证明了FIST能够推广到新的任务中，并且在需要穿越大型迷宫中看不见的部分的导航实验和需要在厨房操纵以前看不见的物体的7自由度机械臂实验中大大优于先前的基线。 摘要：A desirable property of autonomous agents is the ability to both solve long-horizon problems and generalize to unseen tasks. Recent advances in data-driven skill learning have shown that extracting behavioral priors from offline data can enable agents to solve challenging long-horizon tasks with reinforcement learning. However, generalization to tasks unseen during behavioral prior training remains an outstanding challenge. To this end, we present Few-shot Imitation with Skill Transition Models (FIST), an algorithm that extracts skills from offline data and utilizes them to generalize to unseen tasks given a few downstream demonstrations. FIST learns an inverse skill dynamics model, a distance function, and utilizes a semi-parametric approach for imitation. We show that FIST is capable of generalizing to new tasks and substantially outperforms prior baselines in navigation experiments requiring traversing unseen parts of a large maze and 7-DoF robotic arm experiments requiring manipulating previously unseen objects in a kitchen.

【6】 Untangling Dense Non-Planar Knots by Learning Manipulation Features and Recovery Policies 标题：通过学习操作特征和恢复策略解开密集的非平面结点

作者：Priya Sundaresan,Jennifer Grannen,Brijen Thananjeyan,Ashwin Balakrishna,Jeffrey Ichnowski,Ellen Novoseller,Minho Hwang,Michael Laskey,Joseph E. Gonzalez,Ken Goldberg 机构： 1AUTOLAB at the University of California, 2Toyota Research Institute 链接：https://arxiv.org/abs/2107.08942 摘要：由于一维可变形结构（如绳索、电缆和电线）具有无限维的构形空间、复杂的动力学特性和自闭塞的趋势，因此机器人操作具有挑战性。由于相邻电缆段之间难以抓取，分析控制器通常在密集配置的情况下失败。我们提出了两个算法，提高鲁棒性的电缆解开，洛基和蜘蛛侠，这两个操作旁边的绿巨人，一个高级规划师从以前的工作。LOKI使用操作特征的学习模型将粗略的抓取关键点预测细化为精确、优化的位置和方向，而SPiDERMan使用学习模型感知任务进度并应用恢复操作。我们使用达芬奇手术机器人在336节和超过1500个动作的实际电缆解开实验中评估了这些算法。我们发现，绿巨人，洛基和蜘蛛侠的组合能够解开密集的上手，图八，双上手，正方形，保龄球，奶奶，装卸工，和三上手结。在60个物理实验中，68.3%的实验成功地将电缆从密集的初始结构中分离出来，成功率比先前工作的基线高出50%。补充材料、代码和视频可在https://tinyurl.com/rssuntangling. 摘要：Robot manipulation for untangling 1D deformable structures such as ropes, cables, and wires is challenging due to their infinite dimensional configuration space, complex dynamics, and tendency to self-occlude. Analytical controllers often fail in the presence of dense configurations, due to the difficulty of grasping between adjacent cable segments. We present two algorithms that enhance robust cable untangling, LOKI and SPiDERMan, which operate alongside HULK, a high-level planner from prior work. LOKI uses a learned model of manipulation features to refine a coarse grasp keypoint prediction to a precise, optimized location and orientation, while SPiDERMan uses a learned model to sense task progress and apply recovery actions. We evaluate these algorithms in physical cable untangling experiments with 336 knots and over 1500 actions on real cables using the da Vinci surgical robot. We find that the combination of HULK, LOKI, and SPiDERMan is able to untangle dense overhand, figure-eight, double-overhand, square, bowline, granny, stevedore, and triple-overhand knots. The composition of these methods successfully untangles a cable from a dense initial configuration in 68.3% of 60 physical experiments and achieves 50% higher success rates than baselines from prior work. Supplementary material, code, and videos can be found at https://tinyurl.com/rssuntangling.

【7】 Towards synthesizing grasps for 3D deformable objects with physics-based simulation 标题：基于物理仿真的三维变形体综合抓取研究

作者：Tran Nguyen Le,Jens Lundell,Fares J. Abu-Dakka,Ville Kyrki 机构：Intelligent Robotics Group, Department of Electrical Engineering and Automation, School of Electrical Engineering, Aalto University, Finland 备注：4 pages, 5 figures. Published at Robotics: Science and Systems (RSS) 2021 Workshop on Deformable Object Simulation (DO-Sim) 链接：https://arxiv.org/abs/2107.08898 摘要：由于对可变形物体动力学行为建模和仿真的复杂性，对可变形物体的抓取问题的研究还不够深入。然而，随着支持柔体的物理仿真器的迅速发展，刚性物体和可变形物体之间的研究差距越来越小。为了充分利用这类仿真器的能力，并挑战迄今为止指导机器人抓取研究的假设，即物体刚性，我们提出了一种基于深度学习的方法，生成与刚性相关的抓取。我们的网络是根据一个基于物理的模拟器生成的纯合成数据进行训练的。同样的模拟器也被用来评估训练过的网络。结果显示，在抓取排名和抓取成功率方面都有所提高。此外，我们的网络可以根据刚度自适应抓取。我们目前正在一个更大的模拟测试数据集和一个物理机器人上验证所提出的方法。 摘要：Grasping deformable objects is not well researched due to the complexity in modelling and simulating the dynamic behavior of such objects. However, with the rapid development of physics-based simulators that support soft bodies, the research gap between rigid and deformable objects is getting smaller. To leverage the capability of such simulators and to challenge the assumption that has guided robotic grasping research so far, i.e., object rigidity, we proposed a deep-learning based approach that generates stiffness-dependent grasps. Our network is trained on purely synthetic data generated from a physics-based simulator. The same simulator is also used to evaluate the trained network. The results show improvement in terms of grasp ranking and grasp success rate. Furthermore, our network can adapt the grasps based on the stiffness. We are currently validating the proposed approach on a larger test dataset in simulation and on a physical robot.

【8】 Ab Initio Particle-based Object Manipulation 标题：从头开始基于粒子的对象操纵

作者：Siwei Chen,Xiao Ma,Yunfan Lu,David Hsu 机构：National University of Singapore, Reconstructed particle representation of some household objects, Grasping, Pushing, Placing 备注：Robotics: Science and Systems (RSS) 2021 链接：https://arxiv.org/abs/2107.08865 摘要：提出了一种新的基于粒子的对象操作（Prompt）方法，该方法无需事先建立对象模型，也无需对大对象数据集进行预训练。提示的关键元素是基于粒子的对象表示，其中每个粒子表示对象中的一个点，该点的局部几何、物理和其他特征，以及它与其他粒子的关系。与基于模型的操作分析方法一样，粒子表示使机器人能够对物体的几何和动力学进行推理，从而选择合适的操作动作。像数据驱动的方法一样，粒子表示是从视觉传感器输入实时在线学习的，特别是多视图RGB图像。因此，粒子表示将视觉感知与机器人控制联系起来。Prompt结合了基于模型的推理和数据驱动学习的优点。我们的经验表明，提示成功地处理各种日常对象，其中一些是透明的。它处理各种操作任务，包括抓、推等，。我们的实验还表明，即使不使用任何离线训练数据，Prompt在日常物体上的抓取效果也优于最新的数据驱动抓取方法。 摘要：This paper presents Particle-based Object Manipulation (Prompt), a new approach to robot manipulation of novel objects ab initio, without prior object models or pre-training on a large object data set. The key element of Prompt is a particle-based object representation, in which each particle represents a point in the object, the local geometric, physical, and other features of the point, and also its relation with other particles. Like the model-based analytic approaches to manipulation, the particle representation enables the robot to reason about the object's geometry and dynamics in order to choose suitable manipulation actions. Like the data-driven approaches, the particle representation is learned online in real-time from visual sensor input, specifically, multi-view RGB images. The particle representation thus connects visual perception with robot control. Prompt combines the benefits of both model-based reasoning and data-driven learning. We show empirically that Prompt successfully handles a variety of everyday objects, some of which are transparent. It handles various manipulation tasks, including grasping, pushing, etc,. Our experiments also show that Prompt outperforms a state-of-the-art data-driven grasping method on the daily objects, even though it does not use any offline training data.

【9】 Relative Localization of Mobile Robots with Multiple Ultra-WideBand Ranging Measurements 标题：基于多个超宽带测距的移动机器人相对定位

作者：Zhiqiang Cao,Ran Liu,Chau Yuen,Achala Athukorala,Benny Kai Kiat Ng,Muraleetharan Mathanraj,U-Xuan Tan 机构： Tan are with the Singapore University of Technology and Design, Mathanraj is with the Universityof Jaffna, Liu are also with theSouthwest University of Science and Technology 备注：Accepted by the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2021), Prague, Czech Republic 链接：https://arxiv.org/abs/2107.08842 摘要：无基础设施的自主机器人之间的相对定位是实现其导航、路径规划和编队的关键，在许多应用中，如应急响应，其中获取环境的先验知识是不可能的。传统的基于超宽带（UWB）的方法可以很好地估计机器人之间的距离，但是获取相对姿态（包括位移和方向）仍然是一个挑战。提出了一种通过为机器人配置多个超宽带测距节点来估计机器人相对姿态的方法。我们通过最小化所有UWB节点的测距残差来确定两个机器人之间的姿态。为了提高定位精度，我们提出通过基于滑动窗口的优化来利用里程计约束。在粒子滤波中，将优化后的姿态与里程计融合，实现了一组移动机器人的姿态跟踪。我们进行了大量的实验来验证该方法的有效性。 摘要：Relative localization between autonomous robots without infrastructure is crucial to achieve their navigation, path planning, and formation in many applications, such as emergency response, where acquiring a prior knowledge of the environment is not possible. The traditional Ultra-WideBand (UWB)-based approach provides a good estimation of the distance between the robots, but obtaining the relative pose (including the displacement and orientation) remains challenging. We propose an approach to estimate the relative pose between a group of robots by equipping each robot with multiple UWB ranging nodes. We determine the pose between two robots by minimizing the residual error of the ranging measurements from all UWB nodes. To improve the localization accuracy, we propose to utilize the odometry constraints through a sliding window-based optimization. The optimized pose is then fused with the odometry in a particle filtering for pose tracking among a group of mobile robots. We have conducted extensive experiments to validate the effectiveness of the proposed approach.

【10】 A Multi-UAV System for Exploration and Target Finding in Cluttered and GPS-Denied Environments 标题：一种用于杂波和GPS干扰环境下的多无人机探测和目标发现系统

作者：Xiaolong Zhu,Fernando Vanegas,Felipe Gonzalez,Conrad Sanderson 机构： Queensland University of Technology, Australia, Data, CSIRO, Australia, Griffith University, Australia 链接：https://arxiv.org/abs/2107.08834 摘要：多旋翼无人机（UAV）在搜索和救援以及遥感方面的应用正在迅速增加。然而，多旋翼无人机的续航能力有限。如果使用多个无人机团队，无人机的应用范围可以扩大。我们提出了一个无人机团队在复杂的障碍物环境中进行合作探测和发现目标的框架。无人机小组在一个杂乱无章的环境中用一张已知的地图自主地导航、探索、探测和发现目标。这种环境的例子包括室内场景、城市或自然峡谷、洞穴和隧道，其中GPS信号受到限制或阻塞。该框架基于一个概率分散的部分可观测马尔可夫决策过程，该过程考虑了传感和环境中的不确定性。该小组可以有效地进行合作，每个无人机在执行任务期间只共享有限的已处理观察结果及其位置。利用机器人操作系统和Gazebo对系统进行了仿真。测试了该系统在几种有障碍物的室内场景下，随着无人机数量的增加，系统的性能。结果表明，所提出的多无人机系统在时间成本、搜索区域比例和搜救任务成功率等方面都有改进。 摘要：The use of multi-rotor Unmanned Aerial Vehicles (UAVs) for search and rescue as well as remote sensing is rapidly increasing. Multi-rotor UAVs, however, have limited endurance. The range of UAV applications can be widened if teams of multiple UAVs are used. We propose a framework for a team of UAVs to cooperatively explore and find a target in complex GPS-denied environments with obstacles. The team of UAVs autonomously navigates, explores, detects, and finds the target in a cluttered environment with a known map. Examples of such environments include indoor scenarios, urban or natural canyons, caves, and tunnels, where the GPS signal is limited or blocked. The framework is based on a probabilistic decentralised Partially Observable Markov Decision Process which accounts for the uncertainties in sensing and the environment. The team can cooperate efficiently, with each UAV sharing only limited processed observations and their locations during the mission. The system is simulated using the Robotic Operating System and Gazebo. Performance of the system with an increasing number of UAVs in several indoor scenarios with obstacles is tested. Results indicate that the proposed multi-UAV system has improvements in terms of time-cost, the proportion of search area surveyed, as well as successful rates for search and rescue missions.

【11】 Visual Adversarial Imitation Learning using Variational Models 标题：基于变分模型的视觉对抗性模仿学习

作者：Rafael Rafailov,Tianhe Yu,Aravind Rajeswaran,Chelsea Finn 机构： Stanford University, University of Washington, Facebook AI Research 链接：https://arxiv.org/abs/2107.08829 摘要：奖励函数规范需要大量的人力和迭代，仍然是通过深度强化学习学习行为的主要障碍。相反，提供所需行为的视觉演示通常是一种更简单、更自然的方法来教授代理。我们考虑一个设置，其中一个代理提供了一个固定的可视化演示数据集，说明如何执行任务，并且必须学习使用所提供的演示和无监督的环境交互来解决任务。这种设置带来了许多挑战，包括视觉观察的表征学习、高维空间导致的样本复杂性以及缺乏固定奖励或学习信号导致的学习不稳定性。为了应对这些挑战，我们开发了一种基于变分模型的对抗性模仿学习（V-MAIL）算法。基于模型的方法为表征学习提供了一个强大的信号，提高了样本效率，并通过策略学习提高了对抗训练的稳定性。通过对多个基于视觉的运动和操作任务的实验，我们发现V-MAIL以一种样本高效的方式学习成功的视觉运动策略，与以前的工作相比具有更好的稳定性，并且获得了更高的渐近性能。我们进一步发现，通过传递学习到的模型，V-MAIL可以从视觉演示中学习新的任务，而无需任何额外的环境交互。所有结果（包括视频）都可以在\url上在线找到{https://sites.google.com/view/variational-mail}. 摘要：Reward function specification, which requires considerable human effort and iteration, remains a major impediment for learning behaviors through deep reinforcement learning. In contrast, providing visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We consider a setting where an agent is provided a fixed dataset of visual demonstrations illustrating how to perform a task, and must learn to solve the task using the provided demonstrations and unsupervised environment interactions. This setting presents a number of challenges including representation learning for visual observations, sample complexity due to high dimensional spaces, and learning instability due to the lack of a fixed reward or learning signal. Towards addressing these challenges, we develop a variational model-based adversarial imitation learning (V-MAIL) algorithm. The model-based approach provides a strong signal for representation learning, enables sample efficiency, and improves the stability of adversarial training by enabling on-policy learning. Through experiments involving several vision-based locomotion and manipulation tasks, we find that V-MAIL learns successful visuomotor policies in a sample-efficient manner, has better stability compared to prior work, and also achieves higher asymptotic performance. We further find that by transferring the learned models, V-MAIL can learn new tasks from visual demonstrations without any additional environment interactions. All results including videos can be found online at \url{https://sites.google.com/view/variational-mail}.

【12】 How does a robot's social credibility relate to its perceived trustworthiness? 标题：机器人的社会可信度与其感知的可信度有何关系？

作者：Patrick Holthaus 机构：School of Physics, Engineering, and Computer Science, University of Hertfordshire, Hatfield, United Kingdom, Index Terms—Human-robot interaction; social credibility;, trustworthiness 备注：Position paper submitted to SCRITA, a workshop at IEEE RO-MAN 2021: this https URL 链接：https://arxiv.org/abs/2107.08805 摘要：本文旨在强调和讨论机器人的社会公信力在与人类互动中的作用。特别是，我想探讨社会公信力和机器人的可接受性以及最终的可信度之间的潜在关系。因此，我也回顾并扩展了社会公信力的概念，作为衡量机器人在与有意识承认的概念互动过程中遵守社会规范的程度。 摘要：This position paper aims to highlight and discuss the role of a robot's social credibility in interaction with humans. In particular, I want to explore a potential relation between social credibility and a robot's acceptability and ultimately its trustworthiness. I thereby also review and expand the notion of social credibility as a measure of how well the robot obeys social norms during interaction with the concept of conscious acknowledgement.

【13】 Optimizing Gait Libraries via a Coverage Metric 标题：通过覆盖率度量优化步态库

作者：Brian Bittner,Shai Revzen 链接：https://arxiv.org/abs/2107.08775 摘要：许多机器人通过合成步进和转弯等基本动作在世界各地移动。要做到这一点，机器人不需要有对人类有直觉意义的原语。当机器人损坏，不再按设计移动时，这一点就变得至关重要。在这里，我们提出了一个目标函数，我们称之为“覆盖率”，它以一种与原语本身的细节无关的方式表示运动原语库的有用性。我们证明了在模拟和物理机器人上优化覆盖的能力，并且证明了在受伤后覆盖可以迅速恢复。这表明，通过优化覆盖范围，机器人即使在面临重大机械故障的情况下，也能保持在世界各地导航的能力。这种方法的优点是通过有效的样本、数据驱动的系统识别方法来增强的，这种方法可以快速通知原语的优化。我们发现，自由度的增加提高了模拟机器人的恢复速度，这在步态优化和强化学习领域是一个罕见的结果。结果表明，用树枝做肢体的机器人（没有CAD模型或第一性原理模型）能够快速找到有效的高覆盖率运动原语库。优化的原语对于人类观察者来说是完全不明显的，因此不太可能通过手动调整来实现。 摘要：Many robots move through the world by composing locomotion primitives like steps and turns. To do so well, robots need not have primitives that make intuitive sense to humans. This becomes of paramount importance when robots are damaged and no longer move as designed. Here we propose a goal function we call "coverage", that represents the usefulness of a library of locomotion primitives in a manner agnostic to the particulars of the primitives themselves. We demonstrate the ability to optimize coverage on both simulated and physical robots, and show that coverage can be rapidly recovered after injury. This suggests that by optimizing for coverage, robots can sustain their ability to navigate through the world even in the face of significant mechanical failures. The benefits of this approach are enhanced by sample-efficient, data-driven approaches to system identification that can rapidly inform the optimization of primitives. We found that the number of degrees of freedom improved the rate of recovery of our simulated robots, a rare result in the fields of gait optimization and reinforcement learning. We showed that a robot with limbs made of tree branches (for which no CAD model or first principles model was available) is able to quickly find an effective high-coverage library of motion primitives. The optimized primitives are entirely non-obvious to a human observer, and thus are unlikely to be attainable through manual tuning.

【14】 ObserveNet Control: A Vision-Dynamics Learning Approach to Predictive Control in Autonomous Vehicles 标题：观测网控制：自主车辆预测控制的视觉动力学学习方法

作者：Cosmin Ginerica,Mihai Zaha,Florin Gogianu,Lucian Busoniu,Bogdan Trasnea,Sorin Grigorescu 机构： Romania 3 Lucian Busoniu is with Technical University of Cluj-Napoca 备注：None 链接：https://arxiv.org/abs/2107.08690 摘要：自动驾驶的一个关键组成部分是自动驾驶汽车理解、跟踪和预测周围环境动态的能力。虽然在目标检测、跟踪和观测值预测方面有着重要的研究，但是还没有研究表明原始观测值预测可以用于运动规划和控制。本文提出了一种视觉动力学方法来解决自主车辆的预测控制问题。我们的方法由以下两部分组成：i）深度神经网络能够自信地预测未来10秒的时间范围内的感测数据；ii）时间规划者设计用于根据预测的感测数据计算安全车辆状态轨迹。考虑到车辆的历史状态和Lidar点云形式的传感数据，该方法旨在以一种自我监督的方式学习观察到的驾驶环境的动态，而不需要手动指定训练标签。以CARLA和RovisLab的AMTU移动平台为1:4比例的汽车模型，进行了模拟和真实的实验。我们评估了在攻击性驾驶环境下，如超车或侧线切断情况下的观察控制能力，同时将结果与基线动态窗口方法（DWA）和两种最先进的模拟学习系统，即欺骗学习（LBC）和轨道世界（WOR）进行了比较。 摘要：A key component in autonomous driving is the ability of the self-driving car to understand, track and predict the dynamics of the surrounding environment. Although there is significant work in the area of object detection, tracking and observations prediction, there is no prior work demonstrating that raw observations prediction can be used for motion planning and control. In this paper, we propose ObserveNet Control, which is a vision-dynamics approach to the predictive control problem of autonomous vehicles. Our method is composed of a: i) deep neural network able to confidently predict future sensory data on a time horizon of up to 10s and ii) a temporal planner designed to compute a safe vehicle state trajectory based on the predicted sensory data. Given the vehicle's historical state and sensing data in the form of Lidar point clouds, the method aims to learn the dynamics of the observed driving environment in a self-supervised manner, without the need to manually specify training labels. The experiments are performed both in simulation and real-life, using CARLA and RovisLab's AMTU mobile platform as a 1:4 scaled model of a car. We evaluate the capabilities of ObserveNet Control in aggressive driving contexts, such as overtaking maneuvers or side cut-off situations, while comparing the results with a baseline Dynamic Window Approach (DWA) and two state-of-the-art imitation learning systems, that is, Learning by Cheating (LBC) and World on Rails (WOR).

【15】 Topology-Guided Path Planning for Reliable Visual Navigation of MAVs 标题：拓扑引导的MAV可靠视觉导航路径规划

作者：Dabin Kim,Gyeong Chan Kim,Youngseok Jang,H. Jin Kim 机构：©, IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media 备注：8 pages, 9 figures, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) accepted 链接：https://arxiv.org/abs/2107.08616 摘要：视觉导航已广泛应用于微型飞行器的状态估计。为了实现稳定的视觉导航，微型飞行器应该生成感知路径，以保证足够的可见地标。以往许多感知路径规划的研究主要集中在基于抽样的规划器上。然而，它们可能会受到样本效率低下的影响，从而导致寻找全局最优路径的计算负担。为了解决这个问题，我们提出了一种感知路径规划器，它利用了环境的拓扑信息。由于路径的拓扑类与路径中可见的路标密切相关，该算法通过检查不同的拓扑类来选择具有丰富视觉信息的类。从环境的广义Voronoi图中提取拓扑图，得到具有不同拓扑类的初始路径。为了评估类的感知质量，我们将初始路径划分为离散的部分，每个部分中的点共享相似的视觉信息。选择具有高感知质量的最优类，利用基于图的规划器生成类内路径。通过仿真和真实世界的实验，我们证实了与感知不可知方法相比，该方法能够保证精确的视觉导航，同时显示出比基于采样的感知感知规划者更高的计算效率。 摘要：Visual navigation has been widely used for state estimation of micro aerial vehicles (MAVs). For stable visual navigation, MAVs should generate perception-aware paths which guarantee enough visible landmarks. Many previous works on perception-aware path planning focused on sampling-based planners. However, they may suffer from sample inefficiency, which leads to computational burden for finding a global optimal path. To address this issue, we suggest a perception-aware path planner which utilizes topological information of environments. Since the topological class of a path and visible landmarks during traveling the path are closely related, the proposed algorithm checks distinctive topological classes to choose the class with abundant visual information. Topological graph is extracted from the generalized Voronoi diagram of the environment and initial paths with different topological classes are found. To evaluate the perception quality of the classes, we divide the initial path into discrete segments where the points in each segment share similar visual information. The optimal class with high perception quality is selected, and a graph-based planner is utilized to generate path within the class. With simulations and real-world experiments, we confirmed that the proposed method could guarantee accurate visual navigation compared with the perception-agnostic method while showing improved computational efficiency than the sampling-based perception-aware planner.

【16】 Scalable Distributed Planning for Multi-Robot, Multi-Target Tracking 标题：多机器人、多目标跟踪的可扩展分布式规划

作者：Micah Corah,Nathan Michael 机构： Nathan Michael is affiliated with the RoboticsInstitute, Carnegie Mellon University (CMU) 备注：11 pages, 4 figures, Extended version of paper accepted to IROS'21. arXiv admin note: text overlap with arXiv:2102.04054 链接：https://arxiv.org/abs/2107.08550 摘要：在多机器人多目标跟踪中，机器人通过协调来监视多组目标在环境中的运动。我们通过制定一个具有互信息目标的后退视界多机器人感知问题来解决此类场景的规划问题。这类问题一般都是NP难问题。然而，我们的目标是子模，这使得某些贪婪的规划者能够保证常数因子次优。然而，这些贪婪的规划者要求机器人按顺序规划自己的行动，一次规划一个机器人，因此规划时间至少与机器人的数量成正比。对于大型团队来说，解决这些问题变得很难，甚至对于分布式实现也是如此。我们之前的工作提出了一种分布式规划器（RSP），它允许机器人并行规划，而忽略彼此的一些决策，从而将连续步骤的数量减少到一个常数，即使对于大量的机器人也是如此。虽然该分析不适用于目标跟踪，但我们证明了一个类似的保证，即RSP规划接近于全序贯规划器的性能保证，通过采用一个新的边界，利用目标运动的独立性来量化机器人观察和动作之间的有效冗余。此外，我们提出的分析，明确说明了实际执行的特点，包括接近目标和随时规划。通过开源版本获得的利用测距传感器进行目标跟踪的仿真结果表明，我们的规划人员在仅给出2-8个规划步骤的情况下，一致地接近顺序规划的性能（在位置不确定性方面），并且对于多达96个机器人，规划中的顺序步骤数量减少了24倍计划。因此，本文的工作使得多机器人目标跟踪规划在更大的尺度上易于处理，适用于实际规划和一般的跟踪问题。 摘要：In multi-robot multi-target tracking, robots coordinate to monitor groups of targets moving about an environment. We approach planning for such scenarios by formulating a receding-horizon, multi-robot sensing problem with a mutual information objective. Such problems are NP-Hard in general. Yet, our objective is submodular which enables certain greedy planners to guarantee constant-factor suboptimality. However, these greedy planners require robots to plan their actions in sequence, one robot at a time, so planning time is at least proportional to the number of robots. Solving these problems becomes intractable for large teams, even for distributed implementations. Our prior work proposed a distributed planner (RSP) which reduces this number of sequential steps to a constant, even for large numbers of robots, by allowing robots to plan in parallel while ignoring some of each others' decisions. Although that analysis is not applicable to target tracking, we prove a similar guarantee, that RSP planning approaches performance guarantees for fully sequential planners, by employing a novel bound which takes advantage of the independence of target motions to quantify effective redundancy between robots' observations and actions. Further, we present analysis that explicitly accounts for features of practical implementations including approximations to the objective and anytime planning. Simulation results -- available via open source release -- for target tracking with ranging sensors demonstrate that our planners consistently approach the performance of sequential planning (in terms of position uncertainty) given only 2--8 planning steps and for as many as 96 robots with a 24x reduction in the number of sequential steps in planning. Thus, this work makes planning for multi-robot target tracking tractable at much larger scales than before, for practical planners and general tracking problems.

【17】 Distributed Planning for Serving Cooperative Tasks with Time Windows: A Game Theoretic Approach 标题：带时间窗的协同任务分布式规划：博弈论方法

作者：Yasin Yazicioglu,Raghavendra Bhat,Derya Aksaray 机构：Received: date Accepted: date 链接：https://arxiv.org/abs/2107.08540 摘要：研究了多机器人系统的分布式规划问题，为分布在空间和时间上的协同任务提供最优服务。每个任务都需要足够多的机器人在指定的时间窗口内在指定的位置提供服务。任务在每一集中到达，机器人通过根据传入任务的规格规划自己的轨迹，试图在每一集中最大化服务的总价值。机器人被要求在环境中的指定位置开始和结束每一集。我们提出了一个博弈论的解决方案，将其映射到一个博弈中，其中每个机器人的动作是其在一个事件中的轨迹，并使用合适的学习算法以分布式方式获得最优的关节规划。我们提出了一个系统的方法来设计机器人的最小行动集（可行轨迹子集）的基础上规范的传入任务，以促进快速学习。然后，我们提供了在所有机器人都遵循最佳响应或噪声最佳响应算法迭代规划其轨迹的情况下的性能保证。当最佳响应算法导致纳什均衡时，噪声最佳响应算法导致高概率的全局最优联合计划。我们证明了所提出的博弈一般可以具有任意的差纳什均衡，这使得在已知任务规格具有某种特殊结构的情况下，噪声最佳响应算法更为可取。我们还描述了一类特殊情况，其中所有的平衡点都保证有界次优。仿真和实验结果证明了该方法的有效性。 摘要：We study distributed planning for multi-robot systems to provide optimal service to cooperative tasks that are distributed over space and time. Each task requires service by sufficiently many robots at the specified location within the specified time window. Tasks arrive over episodes and the robots try to maximize the total value of service in each episode by planning their own trajectories based on the specifications of incoming tasks. Robots are required to start and end each episode at their assigned stations in the environment. We present a game theoretic solution to this problem by mapping it to a game, where the action of each robot is its trajectory in an episode, and using a suitable learning algorithm to obtain optimal joint plans in a distributed manner. We present a systematic way to design minimal action sets (subsets of feasible trajectories) for robots based on the specifications of incoming tasks to facilitate fast learning. We then provide the performance guarantees for the cases where all the robots follow a best response or noisy best response algorithm to iteratively plan their trajectories. While the best response algorithm leads to a Nash equilibrium, the noisy best response algorithm leads to globally optimal joint plans with high probability. We show that the proposed game can in general have arbitrarily poor Nash equilibria, which makes the noisy best response algorithm preferable unless the task specifications are known to have some special structure. We also describe a family of special cases where all the equilibria are guaranteed to have bounded suboptimality. Simulations and experimental results are provided to demonstrate the proposed approach.

【18】 SENSORIMOTOR GRAPH: Action-Conditioned Graph Neural Network for Learning Robotic Soft Hand Dynamics 标题：感觉运动图：学习机器人软手动力学的动作条件图神经网络

作者：João Damião Almeida,Paul Schydlo,Atabak Dehban,José Santos-Victor 机构：Equal contribution 1Institute for Systems and Robotics, University of Lisbon 2Machine Learning Department, Carnegie Mellon University 备注：Accepted at 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2021) 链接：https://arxiv.org/abs/2107.08492 摘要：软机器人学是机器人学的一个蓬勃发展的分支，它从自然界中汲取灵感，使用廉价的柔性材料来设计适应性强的非刚性机器人。然而，这些机器人的柔性行为使其难以建模，这对于精确驱动和最优控制至关重要。对于系统建模，基于学习的方法已经显示出良好的结果，但他们没有考虑作为感应先验的系统的物理结构。在这项工作中，我们从感觉运动学习中得到启发，并将一个图神经网络应用于非刚性运动链（即机器人的软手）的建模问题，利用了两个关键特性：1）系统是组合的，也就是说，它由简单的通过边连接的交互部分组成，2）它是顺序不变的，也就是说，只有系统的结构与预测未来的轨迹有关。我们将我们的模型表示为“感觉运动图”，因为它从观测中学习系统连通性，并将其用于动力学预测。我们在不同的场景下验证了我们的模型，并表明它在动态预测方面优于非结构化基线，同时对结构变化、跟踪错误或节点故障具有更强的鲁棒性。 摘要：Soft robotics is a thriving branch of robotics which takes inspiration from nature and uses affordable flexible materials to design adaptable non-rigid robots. However, their flexible behavior makes these robots hard to model, which is essential for a precise actuation and for optimal control. For system modelling, learning-based approaches have demonstrated good results, yet they fail to consider the physical structure underlying the system as an inductive prior. In this work, we take inspiration from sensorimotor learning, and apply a Graph Neural Network to the problem of modelling a non-rigid kinematic chain (i.e. a robotic soft hand) taking advantage of two key properties: 1) the system is compositional, that is, it is composed of simple interacting parts connected by edges, 2) it is order invariant, i.e. only the structure of the system is relevant for predicting future trajectories. We denote our model as the 'Sensorimotor Graph' since it learns the system connectivity from observation and uses it for dynamics prediction. We validate our model in different scenarios and show that it outperforms the non-structured baselines in dynamics prediction while being more robust to configurational variations, tracking errors or node failures.

【19】 Robust Composition of Drone Delivery Services under Uncertainty 标题：不确定条件下无人机送货服务的稳健组合

作者：Babar Shahzaad,Athman Bouguettaya,Sajib Mistry 机构：∗School of Computer Science, The University of Sydney, Australia, †School of Electrical Engineering, Computing and Mathematical Sciences, Curtin University, Australia 备注：6 pages, 3 figures. This is an accepted paper and it is going to appear in the Proceedings of the 2021 IEEE International Conference on Web Services (IEEE ICWS 2021) 链接：https://arxiv.org/abs/2107.08450 摘要：我们提出了一个新的稳健的无人机交付服务的组成框架，考虑到在城市地区的风模式的变化。拟议的框架结合了无人机服务在充电站的动态到达。提出了一种概率前向搜索（PFS）算法，用于在不确定性条件下选择和组合最优无人机交付服务。通过一个真实无人机数据集的实验，验证了该方法的有效性和有效性。 摘要：We propose a novel robust composition framework for drone delivery services considering changes in the wind patterns in urban areas. The proposed framework incorporates the dynamic arrival of drone services at the recharging stations. We propose a Probabilistic Forward Search (PFS) algorithm to select and compose the best drone delivery services under uncertainty. A set of experiments with a real drone dataset is conducted to illustrate the effectiveness and efficiency of the proposed approach.

【20】 Deformation-Aware Robotic 3D Ultrasound 标题：变形感知机器人三维超声

作者：Zhongliang Jiang,Yue Zhou,Yuan Bi,Mingchuan Zhou,Thomas Wendler,Nassir Navab 机构： Technical University of Munich, Zhejiang University, Johns Hopkins University 备注：Accepted for publication in IEEE Robotics and Automation Letters; Video: this https URL 链接：https://arxiv.org/abs/2107.08411 摘要：超声成像中的组织变形在测量组织时由于探头施加的压力而导致几何误差。这种变形对3D-US体积有更大的影响，因为不一致的位置和几何形状限制了正确的合成。这项工作提出了一种基于患者特定刚度的方法来校正机器人三维超声采集中的组织变形。为了获得患者指定的模型，在组织上的取样位置执行机器人触诊。记录触诊过程中的接触力、超声图像和探头姿势。利用接触力和探针姿态估计组织的非线性刚度。图像被送入光流算法来计算像素位移。然后用二次回归方法描述了不同力作用下的像素级组织变形。为了校正在轨迹上看不见的位置处的变形以构建三维体积，基于在采样位置处计算的刚度值执行插值。根据刚度和记录的力，可以校正组织位移。在两个不同刚度的血管模型上进行了验证。结果表明，该方法能有效地修正力引起的变形，最终生成三维组织几何图形 摘要：Tissue deformation in ultrasound (US) imaging leads to geometrical errors when measuring tissues due to the pressure exerted by probes. Such deformation has an even larger effect on 3D US volumes as the correct compounding is limited by the inconsistent location and geometry. This work proposes a patient-specified stiffness-based method to correct the tissue deformations in robotic 3D US acquisitions. To obtain the patient-specified model, robotic palpation is performed at sampling positions on the tissue. The contact force, US images and the probe poses of the palpation procedure are recorded. The contact force and the probe poses are used to estimate the nonlinear tissue stiffness. The images are fed to an optical flow algorithm to compute the pixel displacement. Then the pixel-wise tissue deformation under different forces is characterized by a coupled quadratic regression. To correct the deformation at unseen positions on the trajectory for building 3D volumes, an interpolation is performed based on the stiffness values computed at the sampling positions. With the stiffness and recorded force, the tissue displacement could be corrected. The method was validated on two blood vessel phantoms with different stiffness. The results demonstrate that the method can effectively correct the force-induced deformation and finally generate 3D tissue geometries

【21】 Pre-trained Language Models as Prior Knowledge for Playing Text-based Games 标题：预先训练的语言模型作为玩文本游戏的先验知识

作者：Ishika Singh,Gargi Singh,Ashutosh Modi 机构：Department of Computer Science and Engineering, Indian Institute of Technology Kanpur (IITK), India 备注：55 Pages (8 Pages main content + 2 Pages references + 45 Pages Appendix) 链接：https://arxiv.org/abs/2107.08408 摘要：最近，文本世界游戏被提出，以使人工代理了解和推理现实世界的情况。这些基于文本的游戏对人工智能体来说是一个挑战，因为它需要在部分可观察的环境中使用自然语言进行理解和交互。在本文中，我们提出了一个简单的RL-with-LM框架，其中使用了基于转换器的语言模型和深层RL模型，从而提高了对agent的语义理解。我们对我们的框架进行了详细的研究，以展示我们的模型如何在流行游戏Zork1上胜过所有现有的代理，从而获得44.7分，比最先进的模型高出1.6分。我们提出的方法在其他文本游戏上的表现也与最先进的模型相当。 摘要：Recently, text world games have been proposed to enable artificial agents to understand and reason about real-world scenarios. These text-based games are challenging for artificial agents, as it requires understanding and interaction using natural language in a partially observable environment. In this paper, we improve the semantic understanding of the agent by proposing a simple RL with LM framework where we use transformer-based language models with Deep RL models. We perform a detailed study of our framework to demonstrate how our model outperforms all existing agents on the popular game, Zork1, to achieve a score of 44.7, which is 1.6 higher than the state-of-the-art model. Our proposed approach also performs comparably to the state-of-the-art models on the other set of text games.

【22】 A Duality-based Approach for Real-time Obstacle Avoidance between Polytopes with Control Barrier Functions 标题：一种基于对偶的屏障控制函数多面体实时避障方法

作者：Akshay Thirugnanam,Jun Zeng,Koushil Sreenath 备注：submitted to 60th Conference on Decision and Control (CDC 2021) 链接：https://arxiv.org/abs/2107.08360 摘要：多面体间避障控制器的研制是空间狭小导航中一个具有挑战性和必要性的问题。传统方法只能将避障问题描述为离线优化问题。为了解决这些问题，我们提出了一种基于对偶的安全临界最优控制方法，该方法利用控制势垒函数实现多面体之间的避障，并通过基于QP的优化问题实时求解。引入对偶优化问题来表示多面体之间的最小距离，并利用拉格朗日函数的对偶形式构造控制势垒函数。我们证明了所提出的控制器在一个移动沙发的问题，其中非保守机动可以实现在一个狭小的空间。 摘要：Developing controllers for obstacle avoidance between polytopes is a challenging and necessary problem for navigation in a tight space. Traditional approaches can only formulate the obstacle avoidance problem as an offline optimization problem. To address these challenges, we propose a duality-based safety-critical optimal control using control barrier functions for obstacle avoidance between polytopes, which can be solved in real-time with a QP-based optimization problem. A dual optimization problem is introduced to represent the minimum distance between polytopes and the Lagrangian function for the dual form is applied to construct a control barrier function. We demonstrate the proposed controller on a moving sofa problem where non-conservative maneuvers can be achieved in a tight space.

【23】 Vision-Based Autonomous Car Racing Using Deep Imitative Reinforcement Learning 标题：基于深度模拟强化学习的视觉自主赛车

作者：Peide Cai,Hengli Wang,Huaiyang Huang,Yuxuan Liu,Ming Liu 机构： andDepartment of Science and Technology of Guangdong Province Fund, )The authors are with The Hong Kong University of Science andTechnology 备注：8 pages, 8 figures. IEEE Robotics and Automation Letters (RA-L) & IROS 2021 链接：https://arxiv.org/abs/2107.08325 摘要：在机器人控制领域，自动赛车是一项具有挑战性的任务。传统的模块化方法需要精确的绘图、定位和规划，这使得它们计算效率低，并且对环境变化敏感。最近，基于深度学习的端到端系统在自动驾驶/赛车方面显示出了良好的效果。然而，它们通常是通过有监督的模仿学习（IL）来实现的，IL存在分布不匹配的问题，而强化学习（RL）则需要大量的有风险的交互数据。在这项工作中，我们提出了一个通用的深度模拟强化学习方法（DIRL），它成功地实现了敏捷自主赛车使用视觉输入。驾驶知识是从IL和基于模型的RL中获取的，在RL中，agent可以向人类教师学习，也可以通过与离线世界模型的安全交互来进行自我改进。我们在一个高保真驾驶模拟和一个实际的1/20比例RC车上验证了我们的算法。评价结果表明，该方法在样本效率和任务执行效率方面优于已有的IL和RL方法。演示视频可在https://caipeide.github.io/autorace-dirl/ 摘要：Autonomous car racing is a challenging task in the robotic control area. Traditional modular methods require accurate mapping, localization and planning, which makes them computationally inefficient and sensitive to environmental changes. Recently, deep-learning-based end-to-end systems have shown promising results for autonomous driving/racing. However, they are commonly implemented by supervised imitation learning (IL), which suffers from the distribution mismatch problem, or by reinforcement learning (RL), which requires a huge amount of risky interaction data. In this work, we present a general deep imitative reinforcement learning approach (DIRL), which successfully achieves agile autonomous racing using visual inputs. The driving knowledge is acquired from both IL and model-based RL, where the agent can learn from human teachers as well as perform self-improvement by safely interacting with an offline world model. We validate our algorithm both in a high-fidelity driving simulation and on a real-world 1/20-scale RC-car with limited onboard computation. The evaluation results demonstrate that our method outperforms previous IL and RL methods in terms of sample efficiency and task performance. Demonstration videos are available at https://caipeide.github.io/autorace-dirl/

【24】 The Effects of Learning in Morphologically Evolving Robot Systems 标题：学习在形态进化机器人系统中的作用

作者：Jie Luo,Jakub M. Tomczak,Agoston E. Eiben 机构：Artificial Intelligence dept., Vrije Universiteit Amsterdam, Amsterdam, The Netherlands 备注：9 pages, 11 figures, IEEE SSCI conference 链接：https://arxiv.org/abs/2107.08249 摘要：当机器人的控制器（大脑）和形态（身体）同时进化时，这会导致一个问题，即大脑和身体不匹配的问题。在本研究中，我们提出了一个终身学习的解决方案。我们建立了一个系统，模块化机器人可以通过重组和变异产生继承父母身体的后代。至于后代的大脑，我们用两种方法来创造它们。第一种是进化论，也就是说机器人孩子的大脑是从父母那里遗传下来的。第二种方法是进化加学习，这意味着孩子的大脑也是遗传的，但另外是由一种学习算法-RevDEknn开发的。我们在一个名为旋转的模拟器上进行实验，比较这两种方法的使用效率、效能和机器人的形态智能。实验表明，进化加学习方法不仅能提高机器人的适应度，而且能使机器人在形态上得到更大的进化。这构成了一个定量的证明，即大脑的变化可以诱发身体的变化，从而产生了形态智能的概念，这个概念由学习三角洲来量化，也就是说形态促进学习的能力。 摘要：When controllers (brains) and morphologies (bodies) of robots simultaneously evolve, this can lead to a problem, namely the brain & body mismatch problem. In this research, we propose a solution of lifetime learning. We set up a system where modular robots can create offspring that inherit the bodies of parents by recombination and mutation. With regards to the brains of the offspring, we use two methods to create them. The first one entails solely evolution which means the brain of a robot child is inherited from its parents. The second approach is evolution plus learning which means the brain of a child is inherited as well, but additionally is developed by a learning algorithm - RevDEknn. We compare these two methods by running experiments in a simulator called Revolve and use efficiency, efficacy, and the morphology intelligence of the robots for the comparison. The experiments show that the evolution plus learning method does not only lead to a higher fitness level, but also to more morphologically evolving robots. This constitutes a quantitative demonstration that changes in the brain can induce changes in the body, leading to the concept of morphological intelligence, which is quantified by the learning delta, meaning the ability of a morphology to facilitate learning.

【25】 Woodscape Fisheye Semantic Segmentation for Autonomous Driving -- CVPR 2021 OmniCV Workshop Challenge 标题：面向自动驾驶的Woodscape Fisheye语义分割--CVPR 2021 OmniCV研讨会挑战赛

作者：Saravanabalagi Ramachandran,Ganesh Sistu,John McDonald,Senthil Yogamani 机构：MaynoothUniversity 备注：Workshop on Omnidirectional Computer Vision (OmniCV) at Conference on Computer Vision and Pattern Recognition (CVPR) 2021. Presentation video is available at this https URL 链接：https://arxiv.org/abs/2107.08246 摘要：我们介绍了自主驾驶的WoodScape鱼眼语义分割挑战赛，该挑战赛是CVPR 2021全向计算机视觉（OmniCV）研讨会的一部分。这一挑战是研究界评估针对鱼眼相机感知的语义分割技术的第一个机会。由于强径向畸变，标准模型不能很好地推广到鱼眼图像，因此物体和实体视觉外观的变形需要隐式编码或显式编码。这个挑战作为一个媒介来研究挑战和新的方法来处理鱼眼图像感知的复杂性。这次挑战是在CodaLab上进行的，并使用了最近发布的WoodScape数据集，该数据集包含10k个样本。在这篇论文中，我们总结了吸引了71个全球团队参与的比赛，共有395份参赛作品。顶级团队的平均IoU和准确度得分显著高于基线PSPNet和ResNet-50主干。总结了获胜算法的方法，并对失败案例进行了分析。最后，我们提出了未来的研究方向。 摘要：We present the WoodScape fisheye semantic segmentation challenge for autonomous driving which was held as part of the CVPR 2021 Workshop on Omnidirectional Computer Vision (OmniCV). This challenge is one of the first opportunities for the research community to evaluate the semantic segmentation techniques targeted for fisheye camera perception. Due to strong radial distortion standard models don't generalize well to fisheye images and hence the deformations in the visual appearance of objects and entities needs to be encoded implicitly or as explicit knowledge. This challenge served as a medium to investigate the challenges and new methodologies to handle the complexities with perception on fisheye images. The challenge was hosted on CodaLab and used the recently released WoodScape dataset comprising of 10k samples. In this paper, we provide a summary of the competition which attracted the participation of 71 global teams and a total of 395 submissions. The top teams recorded significantly improved mean IoU and accuracy scores over the baseline PSPNet with ResNet-50 backbone. We summarize the methods of winning algorithms and analyze the failure cases. We conclude by providing future directions for the research.

【26】 SCV-Stereo: Learning Stereo Matching from a Sparse Cost Volume 标题：SCV-Stereo：从稀疏成本量中学习立体匹配

作者：Hengli Wang,Rui Fan,Ming Liu 机构：⋆ Hong Kong Unviersity of Science and Technology, Hong Kong SAR, China, † Tongji University, Shanghai , China 备注：5 pages, 3 figures and 2 tables. This paper is accepted by ICIP 2021 链接：https://arxiv.org/abs/2107.08187 摘要：基于卷积神经网络（CNN）的立体匹配方法通常需要一个密集的代价体（DCV）来进行视差估计。然而，产生这样的成本量是计算密集型和内存消耗，阻碍CNN训练和推理效率。为了解决这个问题，我们提出了一种新的CNN结构SCV-Stereo，它能够从稀疏代价体积（SCV）表示中学习密集立体匹配。我们的灵感来源于这样一个事实：DCV表示有点多余，可以用SCV表示代替。得益于这些SCV表示，我们的SCV立体可以以迭代的方式更新视差估计，从而实现精确有效的立体匹配。在KITTI立体基准上进行的大量实验表明，我们的SCV立体可以显著降低立体匹配的精度和效率之间的折衷。我们的项目页面是https://sites.google.com/view/scv-stereo. 摘要：Convolutional neural network (CNN)-based stereo matching approaches generally require a dense cost volume (DCV) for disparity estimation. However, generating such cost volumes is computationally-intensive and memory-consuming, hindering CNN training and inference efficiency. To address this problem, we propose SCV-Stereo, a novel CNN architecture, capable of learning dense stereo matching from sparse cost volume (SCV) representations. Our inspiration is derived from the fact that DCV representations are somewhat redundant and can be replaced with SCV representations. Benefiting from these SCV representations, our SCV-Stereo can update disparity estimations in an iterative fashion for accurate and efficient stereo matching. Extensive experiments carried out on the KITTI Stereo benchmarks demonstrate that our SCV-Stereo can significantly minimize the trade-off between accuracy and efficiency for stereo matching. Our project page is https://sites.google.com/view/scv-stereo.

【27】 Co-Teaching: An Ark to Unsupervised Stereo Matching 标题：协同教学：通向无监督立体匹配的方舟

作者：Hengli Wang,Rui Fan,Ming Liu 机构：⋆ Hong Kong Unviersity of Science and Technology, Hong Kong SAR, China, † Tongji University, Shanghai , China 备注：5 pages, 3 figures and 2 tables. This paper is accepted by ICIP 2021 链接：https://arxiv.org/abs/2107.08186 摘要：立体匹配是自主驾驶感知的重要组成部分。最近的无监督立体匹配方法由于其不需要视差地面真值的优点而受到了足够的重视。然而，这些入路在近闭塞处的表现较差。为了克服这一缺点，本文提出了一种新的无监督立体匹配方法CoT-Stereo。具体地说，我们采用了一种合作教学的框架，两个网络以无监督的方式交互地教授对方关于遮挡的知识，这大大提高了无监督立体匹配的鲁棒性。在KITTI立体声基准上进行的大量实验表明，CoT立体声在精度和速度上都优于所有其他最先进的无监督立体声匹配方法。我们的项目网页是https://sites.google.com/view/cot-stereo. 摘要：Stereo matching is a key component of autonomous driving perception. Recent unsupervised stereo matching approaches have received adequate attention due to their advantage of not requiring disparity ground truth. These approaches, however, perform poorly near occlusions. To overcome this drawback, in this paper, we propose CoT-Stereo, a novel unsupervised stereo matching approach. Specifically, we adopt a co-teaching framework where two networks interactively teach each other about the occlusions in an unsupervised fashion, which greatly improves the robustness of unsupervised stereo matching. Extensive experiments on the KITTI Stereo benchmarks demonstrate the superior performance of CoT-Stereo over all other state-of-the-art unsupervised stereo matching approaches in terms of both accuracy and speed. Our project webpage is https://sites.google.com/view/cot-stereo.

【28】 Dual Quaternion-Based Visual Servoing for Grasping Moving Objects 标题：基于对偶四元数的运动物体抓取视觉伺服

作者：Cristiana de Farias,Maxime Adjigble,Brahim Tamadazte,Rustam Stolkin,Naresh Marturi 机构： School of Metallurgy and Materials, University of Birmingham 备注：Accepted for 2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)- August 23-27, 2021, Lyon, France 链接：https://arxiv.org/abs/2107.08149 摘要：提出了一种新的基于双四元数的姿态视觉伺服方法。扩展了我们以前基于局部接触矩（LoCoMo）的抓取规划的工作，我们演示了三维空间中任意运动物体的抓取。与传统的轴角参数化方法不同，双四元数可以更紧凑地设计视觉伺服任务，并对机械手奇异点具有鲁棒性。给定一个目标点云，LoCoMo生成一个抓取姿势和预抓取姿势的排序列表，这些姿势作为视觉伺服所需的姿势。每当对象移动时（通过视觉标记跟踪跟踪），所需的姿势会自动更新。为此，利用双四元数空间距离误差，提出了一种动态抓取重排序度量，以选择运动目标的最佳抓取。这使得机器人能够随时跟踪和抓取任意移动的物体。此外，我们也利用我们的控制器来探索机器人的零空间，以避免关节的限制，从而在跟随移动物体时达到平滑的轨迹。我们评估了所提出的视觉伺服性能进行了仿真实验抓取各种物体的7轴机器人配备了2个手指抓取。得到的结果证明了我们提出的视觉伺服的有效性。 摘要：This paper presents a new dual quaternion-based formulation for pose-based visual servoing. Extending our previous work on local contact moment (LoCoMo) based grasp planning, we demonstrate grasping of arbitrarily moving objects in 3D space. Instead of using the conventional axis-angle parameterization, dual quaternions allow designing the visual servoing task in a more compact manner and provide robustness to manipulator singularities. Given an object point cloud, LoCoMo generates a ranked list of grasp and pre-grasp poses, which are used as desired poses for visual servoing. Whenever the object moves (tracked by visual marker tracking), the desired pose updates automatically. For this, capitalising on the dual quaternion spatial distance error, we propose a dynamic grasp re-ranking metric to select the best feasible grasp for the moving object. This allows the robot to readily track and grasp arbitrarily moving objects. In addition, we also explore the robot null-space with our controller to avoid joint limits so as to achieve smooth trajectories while following moving objects. We evaluate the performance of the proposed visual servoing by conducting simulation experiments of grasping various objects using a 7-axis robot fitted with a 2-finger gripper. Obtained results demonstrate the efficiency of our proposed visual servoing.

【29】 CoCo: Online Mixed-Integer Control via Supervised Learning 标题：COCO：基于监督学习的在线混合整数控制

作者：A. Cauligi,P. Culbertson,E. Schmerling,M. Schwager,B. Stellato,M. Pavone 机构：edu 3Department of Operations Research and Financial Engineering, PrincetonUniversity 链接：https://arxiv.org/abs/2107.08143 摘要：许多机器人问题，从机器人运动规划到目标操作，都可以建模为混合整数凸规划。然而，最先进的算法仍然无法足够快地解决用于在线使用的控制问题的MICP，并且现有的启发式算法通常只能找到可能降低机器人性能的次优解决方案。在这项工作中，我们转向数据驱动的方法，并提出了组合离线，凸在线（CoCo）算法，快速找到高质量的解决方案为MICPs。CoCo由两个阶段组成。在离线阶段，我们训练一个神经网络分类器，将问题参数映射到一个逻辑策略，我们定义为离散参数和与问题最优解相关的放松的big-M约束。在线上，在给定新问题参数的情况下，应用分类器选择候选逻辑策略；应用这种逻辑策略，我们可以将原MICP问题作为一个凸优化问题来求解。我们通过数值实验展示了CoCo如何找到机器人规划和控制中出现的MICPs的近似最优解，与其他数据驱动方法和求解器相比，该方法的求解速度提高了1到2个数量级。 摘要：Many robotics problems, from robot motion planning to object manipulation, can be modeled as mixed-integer convex programs (MICPs). However, state-of-the-art algorithms are still unable to solve MICPs for control problems quickly enough for online use and existing heuristics can typically only find suboptimal solutions that might degrade robot performance. In this work, we turn to data-driven methods and present the Combinatorial Offline, Convex Online (CoCo) algorithm for quickly finding high quality solutions for MICPs. CoCo consists of a two-stage approach. In the offline phase, we train a neural network classifier that maps the problem parameters to a (logical strategy), which we define as the discrete arguments and relaxed big-M constraints associated with the optimal solution for that problem. Online, the classifier is applied to select a candidate logical strategy given new problem parameters; applying this logical strategy allows us to solve the original MICP as a convex optimization problem. We show through numerical experiments how CoCo finds near optimal solutions to MICPs arising in robot planning and control with 1 to 2 orders of magnitude solution speedup compared to other data-driven approaches and solvers.

【30】 Autonomy 2.0: Why is self-driving always 5 years away? 标题：自主2.0：为什么自动驾驶总是5年后的事？

作者：Ashesh Jain,Luca Del Pero,Hugo Grimmett,Peter Ondruska 机构：Lyft Level , self-driving 链接：https://arxiv.org/abs/2107.08142 摘要：尽管机器学习在过去十年中取得了许多成功（图像识别、决策、NLP、图像合成），但自动驾驶技术还没有遵循同样的趋势。本文研究了现代自驱动堆栈的历史、组成和发展瓶颈。我们认为，进展缓慢是由于方法需要太多的手工工程、过度依赖道路测试和高昂的车队部署成本造成的。我们观察到，经典堆栈有几个瓶颈，这些瓶颈排除了捕获罕见事件长尾所需的必要规模。为了解决这些问题，我们概述了Autonomy 2.0的原则，这是一种ML优先的自动驾驶方法，是目前采用的最先进技术的可行替代方法。这种方法基于（i）可从人类演示中训练的完全可区分的AV堆栈，（ii）闭环数据驱动的反应模拟，以及（iii）大规模、低成本的数据收集，作为解决可伸缩性问题的关键解决方案。我们概述了总体架构，调查了在这方面有前途的工作，并提出了社区在未来要解决的关键挑战。 摘要：Despite the numerous successes of machine learning over the past decade (image recognition, decision-making, NLP, image synthesis), self-driving technology has not yet followed the same trend. In this paper, we study the history, composition, and development bottlenecks of the modern self-driving stack. We argue that the slow progress is caused by approaches that require too much hand-engineering, an over-reliance on road testing, and high fleet deployment costs. We observe that the classical stack has several bottlenecks that preclude the necessary scale needed to capture the long tail of rare events. To resolve these problems, we outline the principles of Autonomy 2.0, an ML-first approach to self-driving, as a viable alternative to the currently adopted state-of-the-art. This approach is based on (i) a fully differentiable AV stack trainable from human demonstrations, (ii) closed-loop data-driven reactive simulation, and (iii) large-scale, low-cost data collections as critical solutions towards scalability issues. We outline the general architecture, survey promising works in this direction and propose key challenges to be addressed by the community in the future.

【31】 Future Intelligent Autonomous Robots, Ethical by Design. Learning from Autonomous Cars Ethics 标题：未来的智能自主机器人，设计伦理。向自动驾驶汽车伦理学习

作者：Gordana Dodig-Crnkovic,Tobias Holstein,Patrizio Pelliccione 机构： Interaction Design Unit, Department of Computer Science and Engineering, Chalmers | University, of Technology Gothenburg, Sweden, Division of Computer Science and Software Engineering, School of Innovation, Design and 备注：11 pages, 1 figure, 1 table 链接：https://arxiv.org/abs/2107.08122 摘要：智能自主机器人技术的发展预示着它将对个人和社会产生预期的有益影响。在这种具有破坏性的新兴技术中，不仅有如何构建的问题，还有为什么构建以及产生什么样的后果的问题是重要的。智能自主机器人汽车的伦理领域是一个具有可操作实用价值的研究的好例子，在这个领域，各种利益相关者，包括法律制度和其他社会和政府行为体，以及公司和企业，进行合作，带来对技术的伦理和社会方面的共同看法。考虑到技术生命周期的不同阶段（开发、实施、测试、使用和处置）的人机界面，它可以作为智能自主机器人总体开发方法的启动平台。根据我们在自主智能机器人伦理方面的工作，以及现有的机器人伦理文献，我们的贡献包括一套价值观和伦理原则，以及确定的挑战和应对挑战的建议方法。这可能有助于智能自主机器人领域的利益相关者将伦理原则与其应用联系起来。我们对自主汽车道德要求的建议可用于其他类型的智能自主机器人，但对于需要更多研究与用户交互的社交机器人，则需注意。我们强调，现有的伦理框架需要以一种上下文敏感的方式应用，通过多标准分析，在跨学科、多能力团队中进行评估。此外，我们主张需要不断发展伦理原则、指导方针和法规，并根据技术进步和相关利益相关者的参与。 摘要：Development of the intelligent autonomous robot technology presupposes its anticipated beneficial effect on the individuals and societies. In the case of such disruptive emergent technology, not only questions of how to build, but also why to build and with what consequences are important. The field of ethics of intelligent autonomous robotic cars is a good example of research with actionable practical value, where a variety of stakeholders, including the legal system and other societal and governmental actors, as well as companies and businesses, collaborate bringing about shared view of ethics and societal aspects of technology. It could be used as a starting platform for the approaches to the development of intelligent autonomous robots in general, considering human-machine interfaces in different phases of the life cycle of technology - the development, implementation, testing, use and disposal. Drawing from our work on ethics of autonomous intelligent robocars, and the existing literature on ethics of robotics, our contribution consists of a set of values and ethical principles with identified challenges and proposed approaches for meeting them. This may help stakeholders in the field of intelligent autonomous robotics to connect ethical principles with their applications. Our recommendations of ethical requirements for autonomous cars can be used for other types of intelligent autonomous robots, with the caveat for social robots that require more research regarding interactions with the users. We emphasize that existing ethical frameworks need to be applied in a context-sensitive way, by assessments in interdisciplinary, multi-competent teams through multi-criteria analysis. Furthermore, we argue for the need of a continuous development of ethical principles, guidelines, and regulations, informed by the progress of technologies and involving relevant stakeholders.

【32】 Partially-Observed Decoupled Data-based Control (POD2C) for Complex Robotic Systems 标题：复杂机器人系统的部分观测解耦数据控制(POD2C)

作者：Raman Goyal,Ran Wang,Suman Chakravorty,Robert E. Skelton 机构： results in a composite perturbation feedbackdesign in the information state that can then be used to controlThe authors are with the Department of Aerospace Engineering, TexasA&M University 链接：https://arxiv.org/abs/2107.08086 摘要：提出了一种基于系统数据的高维机器人系统闭环反馈控制方法。我们首先利用自回归滑动平均（ARMA）模型将迭代线性二次型调节器（iLQR）推广到部分观测系统，该模型仅由输入输出数据生成。ARMA模型产生的信息状态维数小于或等于底层实际状态维数。然后利用该开环轨迹优化解设计局部反馈控制律，然后利用合成律解决部分观测反馈设计问题。通过控制复杂的高维非线性机器人系统，证明了该方法的有效性，该系统存在模型和传感不确定性，且分析模型不可用或不准确。 摘要：This paper develops a systematic data-based approach to the closed-loop feedback control of high-dimensional robotic systems using only partial state observation. We first develop a model-free generalization of the iterative Linear Quadratic Regulator (iLQR) to partially-observed systems using an Autoregressive Moving Average (ARMA) model, that is generated using only the input-output data. The ARMA model results in an information state, which has dimension less than or equal to the underlying actual state dimension. This open-loop trajectory optimization solution is then used to design a local feedback control law, and the composite law then provides a solution to the partially observed feedback design problem. The efficacy of the developed method is shown by controlling complex high dimensional nonlinear robotic systems in the presence of model and sensing uncertainty and for which analytical models are either unavailable or inaccurate.

【33】 DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation 标题：DeformerNet：一种三维可变形物体操作的深度学习方法

作者：Bao Thach,Alan Kuntz,Tucker Hermans 机构：University of Utah Robotics Center; †NVIDIA 备注：Published at RSS 2021 Workshop on Deformable Object Simulation in Robotics; received Honorable Mention for Best Paper Award 链接：https://arxiv.org/abs/2107.08067 摘要：本文提出了一种利用DeformerNet神经网络实现三维可变形物体操作的新方法。控制三维对象的形状需要一个有效的状态表示，它可以捕获对象的完整三维几何体。目前的方法都是通过在物体上定义一组特征点或只在二维图像空间中对物体进行变形来解决这个问题，这并不能真正解决三维形状控制问题。相反，我们显式地使用三维点云作为状态表示，并对点云应用卷积神经网络来学习三维特征。然后使用完全连接的神经网络将这些特征映射到机器人末端执行器的位置。一旦以端到端的方式进行了训练，DeformerNet就会直接将可变形对象的当前点云以及目标点云形状映射到机器人夹持器位置的所需位移。此外，我们还研究了给定目标形状和初始形状的操纵点位置的预测问题。 摘要：In this paper, we propose a novel approach to 3D deformable object manipulation leveraging a deep neural network called DeformerNet. Controlling the shape of a 3D object requires an effective state representation that can capture the full 3D geometry of the object. Current methods work around this problem by defining a set of feature points on the object or only deforming the object in 2D image space, which does not truly address the 3D shape control problem. Instead, we explicitly use 3D point clouds as the state representation and apply Convolutional Neural Network on point clouds to learn the 3D features. These features are then mapped to the robot end-effector's position using a fully-connected neural network. Once trained in an end-to-end fashion, DeformerNet directly maps the current point cloud of a deformable object, as well as a target point cloud shape, to the desired displacement in robot gripper position. In addition, we investigate the problem of predicting the manipulation point location given the initial and goal shape of the object.