OpenStack SR-IOV研究

关于 SR-IOV 本文就不再介绍了,具体可以查看 Intel® 82599 SR-IOV Driver Companion Guide

一、系统环境

操作系统: RHEL 7.2 OpenStack版本: OpenStack Mitaka Allinone 网卡型号: Intel Corporation 82599ES SR-IVO网卡名: ens1f0, ens1f0

二、服务器配置

在服务器 BIOS 中开启 VT-d 和 SR-IOV

三、操作系统配置

1. 编辑 /etc/default/grub 文件,加入以下内容

# vim /etc/default/grubGRUB_TIMEOUT=5GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"GRUB_DEFAULT=savedGRUB_DISABLE_SUBMENU=trueGRUB_TERMINAL_OUTPUT="console"-GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet"+GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet intel_iommu=on"GRUB_DISABLE_RECOVERY="true"

需要说明的是: ixgbe.max_vfs 参数已经废弃,故没有加入到内核参数中。

2. 重新生成 grub 文件

# grub2-mkconfig -o /boot/grub2/grub.cfg

3. 配置 SR-IOV 的网卡开机自启

编辑网卡配置文件,修改以下内容

BOOTPROTO=noneONBOOT=yes

4. 设置开机自动创建 VF(计算节点)

# vim /etc/rc.d/rc.localecho '0' > /sys/class/net/ens1f0/device/sriov_numvfsecho '7' > /sys/class/net/ens1f0/device/sriov_numvfsecho '0' > /sys/class/net/ens1f1/device/sriov_numvfsecho '7' > /sys/class/net/ens1f1/device/sriov_numvfschmod +x /etc/rc.d/rc.local

5. 重启服务器

6. 验证 VF 已经创建,并且是 up 状态

# lspci | grep Ethernet05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)05:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)# ip link show ens1f06: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 14:02:ec:82:96:c0 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto# ip link show ens1f17: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 14:02:ec:82:96:c1 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

四、OpenStack 配置

安装 sr-iov agent(计算节点)

# yum -y install openstack-neutron-sriov-nic-agent.noarch

控制节点配置

1. 修改 nova 调度,启用 PciPassthrough Filter

# vim /etc/nova/nova.conf[DEFAULT]scheduler_available_filters=nova.scheduler.filters.all_filtersscheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,PciPassthroughFilter

2. 在 ML2 中加载 sriovnicswitch mechanism driver,并设置网络绑定

# vim /etc/neutron/plugins/ml2/ml2_conf.ini[ml2]type_drivers = flat,vlantenant_network_types =mechanism_drivers = openvswitch,sriovnicswitchextension_drivers = port_security[ml2_type_vlan]network_vlan_ranges = provider,sriov1,sriov2[securitygroup]firewall_driver = neutron.agent.firewall.NoopFirewallDriver

3. 增加支持的 PCI 厂家的 VF 设备

  • 查看id # lspci -nn | grep -i ethernet 02:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01) 02:00.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01) 02:00.2 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01) 02:00.3 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01) 05:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 05:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 05:10.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.2 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.4 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.5 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.6 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.7 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.2 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.4 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.5 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.6 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.7 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
  • 配置设备ID # vim /etc/neutron/plugins/ml2/ml2_conf_sriov.ini [ml2_sriov] supported_pci_vendor_devs = 8086:10ed

4. 修改 neutron-server 启动文件,加载 ml2_conf_sriov.ini 文件

# vim /usr/lib/systemd/system/neutron-server.service[Service]Type=notifyUser=neutronExecStart=/usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log

5. 重启服务

# systemctl daemon-reload# systemctl restart neutron-server.service# systemctl restart openstack-nova-scheduler.service

计算节点配置

1. 配置 PCI 设备白名单

# vim /etc/nova/nova.conf[DEFAULT]pci_passthrough_whitelist = [{"devname":"ens1f0","physical_network":"sriov1"},{"devname":"ens1f1","physical_network":"sriov2"}]

2. 配置 SR-IOV neutron agent

# vim /etc/neutron/plugins/ml2/ml2_conf_sriov.ini[securitygroup]firewall_driver = neutron.agent.firewall.NoopFirewallDriver[sriov_nic]physical_device_mappings = sriov1:ens1f0,sriov2:ens1f1exclude_devices =

3. 启动/重启服务

# systemctl enable neutron-sriov-nic-agent.service# systemctl start neutron-sriov-nic-agent.service# systemctl restart openstack-nova-compute.service

五、测试

1. 创建网络

创建网络# neutron net-create --provider:network_type vlan --provider:physical_network sriov1 --provider:segmentation_id 10 --router:external net1# neutron net-create --provider:network_type vlan --provider:physical_network sriov2 --provider:segmentation_id 20 --router:external net2创建子网(禁用DHCP)# neutron subnet-create --name sriov1-net --disable-dhcp --ip-version 4 net1 10.0.1.0/24# neutron subnet-create --name sriov2-net --disable-dhcp --ip-version 4 net2 10.0.2.0/24

2. 创建 port

# neutron port-create net1 --binding:vnic-type direct# neutron port-create net2 --binding:vnic-type direct

3. 创建虚拟机

# neutron port-list+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+| id | name | mac_address | fixed_ips |+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+| ba446152-bd45-4a38-9947-1d539e538a68 | | fa:16:3e:42:6b:fd | {"subnet_id": "04ea17b5-08da-41cc-8114-e0781a1f8041", "ip_address": "10.0.1.3"} || bf8a7655-ac16-4dce-bb12-54efd2dd0967 | | fa:16:3e:77:9f:8a | {"subnet_id": "6612bf80-d682-474f-886e-93029a4a0964", "ip_address": "10.0.2.3"} |+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+# openstack server create --image rhel-server-7.5-x86_64-kvm --flavor 6 --nic port-id=ba446152-bd45-4a38-9947-1d539e538a68 --nic port-id=bf8a7655-ac16-4dce-bb12-54efd2dd0967 --config-drive True test

4. 查看虚拟机网卡

  • 速率
  • 型号

5. 查看 VF

# ip link show ens1f06: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 14:02:ec:82:96:c0 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 7 MAC fa:16:3e:42:6b:fd, vlan 10, spoof checking on, link-state auto# ip link show ens1f17: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 14:02:ec:82:96:c1 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 7 MAC fa:16:3e:77:9f:8a, vlan 20, spoof checking on, link-state auto


本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏JAVA同学会

Spring Data(一)概念和仓库的定义

Spring Data的主要任务是为数据访问提供一个相似的、一致的、基于Spring的编程模型,同时又保留着下面各个数据存储的特征。它使得使用数据访问技术非常的...

741
来自专栏木制robot技术杂谈

Python中os.path.dirname(__file__)的用法

os.path.dirname()的用途 os.path.dirname()用来获取文件的路径。 如: 1 2>>> os.path.dirname('/ho...

3857
来自专栏我的博客

Zend_Config使用笔记

1.zend_Config被设计在应用程序中简化访问和使用配置数据。它为在应用程序代码中访问这样的配置数据提供了一个基于用户接口的嵌入式对象属性。配置数据可能来...

2635
来自专栏何俊林

插件开发之360 DroidPlugin源码分析(三)Binder代理

Hook机制中Binder代理类关系图 Hook机制中Binder代理时序图 MyServiceManager ServiceManagerCacheBinde...

1899
来自专栏从零开始的linux

swap

Swap分区,即交换分区,系统在物理内存不够时,与Swap进行交换。 # free -m total used ...

4306
来自专栏互联网大杂烩

Spring MVC框架

前端控制器是DispatcherServlet;应用控制器其实拆为处理器映射器(Handler Mapping)进行处理器管理和视图解析器(View Resol...

552
来自专栏阿杜的世界

Spring实战5-基于Spring构建Web应用主要内容

写在前面:关于Java Web,首先推荐一篇文章——写给java web一年左右工作经验的人,这篇文章的作者用精练的话语勾勒除了各种Java框架的缘由和最基本的...

792
来自专栏LuckQI

SpringBoot~SpringBatch 使用

Spring Batch 提供了大量可重用的组件,包括了日志、追踪、事务、任务作业统计、任务重启、跳过、重复、资源管理。对于大数据量和高性能的批处理任务,Spr...

973
来自专栏学海无涯

Java Web之Spring Boot

我一直在尝试一个人写demo(Android和iOS)时,如何模拟服务器端返回的 JSON 数据,总的来说,我试过以下几种: 纯Servlet开发,这种方式配合...

2634
来自专栏xingoo, 一个梦想做发明家的程序员

Log4j官方文档翻译(四、如何在java中输出日志消息)

我们已经创建来配置文件,本章详细的介绍下如何生成调试信息,并把他们转化成文本文件。 基本的例子 下面就是创建的一个基本的例子: log4j.properties...

18310

扫码关注云+社区