OpenStack SR-IOV研究

关于 SR-IOV 本文就不再介绍了,具体可以查看 Intel® 82599 SR-IOV Driver Companion Guide

一、系统环境

操作系统: RHEL 7.2 OpenStack版本: OpenStack Mitaka Allinone 网卡型号: Intel Corporation 82599ES SR-IVO网卡名: ens1f0, ens1f0

二、服务器配置

在服务器 BIOS 中开启 VT-d 和 SR-IOV

三、操作系统配置

1. 编辑 /etc/default/grub 文件,加入以下内容

# vim /etc/default/grubGRUB_TIMEOUT=5GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"GRUB_DEFAULT=savedGRUB_DISABLE_SUBMENU=trueGRUB_TERMINAL_OUTPUT="console"-GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet"+GRUB_CMDLINE_LINUX="rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet intel_iommu=on"GRUB_DISABLE_RECOVERY="true"

需要说明的是: ixgbe.max_vfs 参数已经废弃,故没有加入到内核参数中。

2. 重新生成 grub 文件

# grub2-mkconfig -o /boot/grub2/grub.cfg

3. 配置 SR-IOV 的网卡开机自启

编辑网卡配置文件,修改以下内容

BOOTPROTO=noneONBOOT=yes

4. 设置开机自动创建 VF(计算节点)

# vim /etc/rc.d/rc.localecho '0' > /sys/class/net/ens1f0/device/sriov_numvfsecho '7' > /sys/class/net/ens1f0/device/sriov_numvfsecho '0' > /sys/class/net/ens1f1/device/sriov_numvfsecho '7' > /sys/class/net/ens1f1/device/sriov_numvfschmod +x /etc/rc.d/rc.local

5. 重启服务器

6. 验证 VF 已经创建,并且是 up 状态

# lspci | grep Ethernet05:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)05:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)05:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)05:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)# ip link show ens1f06: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 14:02:ec:82:96:c0 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto# ip link show ens1f17: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 14:02:ec:82:96:c1 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto

四、OpenStack 配置

安装 sr-iov agent(计算节点)

# yum -y install openstack-neutron-sriov-nic-agent.noarch

控制节点配置

1. 修改 nova 调度,启用 PciPassthrough Filter

# vim /etc/nova/nova.conf[DEFAULT]scheduler_available_filters=nova.scheduler.filters.all_filtersscheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,DiskFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,PciPassthroughFilter

2. 在 ML2 中加载 sriovnicswitch mechanism driver,并设置网络绑定

# vim /etc/neutron/plugins/ml2/ml2_conf.ini[ml2]type_drivers = flat,vlantenant_network_types =mechanism_drivers = openvswitch,sriovnicswitchextension_drivers = port_security[ml2_type_vlan]network_vlan_ranges = provider,sriov1,sriov2[securitygroup]firewall_driver = neutron.agent.firewall.NoopFirewallDriver

3. 增加支持的 PCI 厂家的 VF 设备

  • 查看id # lspci -nn | grep -i ethernet 02:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01) 02:00.1 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01) 02:00.2 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01) 02:00.3 Ethernet controller [0200]: Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet PCIe [14e4:1657] (rev 01) 05:00.0 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 05:00.1 Ethernet controller [0200]: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection [8086:10fb] (rev 01) 05:10.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.2 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.4 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.5 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.6 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:10.7 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.0 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.1 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.2 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.3 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.4 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.5 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.6 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01) 05:11.7 Ethernet controller [0200]: Intel Corporation 82599 Ethernet Controller Virtual Function [8086:10ed] (rev 01)
  • 配置设备ID # vim /etc/neutron/plugins/ml2/ml2_conf_sriov.ini [ml2_sriov] supported_pci_vendor_devs = 8086:10ed

4. 修改 neutron-server 启动文件,加载 ml2_conf_sriov.ini 文件

# vim /usr/lib/systemd/system/neutron-server.service[Service]Type=notifyUser=neutronExecStart=/usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log

5. 重启服务

# systemctl daemon-reload# systemctl restart neutron-server.service# systemctl restart openstack-nova-scheduler.service

计算节点配置

1. 配置 PCI 设备白名单

# vim /etc/nova/nova.conf[DEFAULT]pci_passthrough_whitelist = [{"devname":"ens1f0","physical_network":"sriov1"},{"devname":"ens1f1","physical_network":"sriov2"}]

2. 配置 SR-IOV neutron agent

# vim /etc/neutron/plugins/ml2/ml2_conf_sriov.ini[securitygroup]firewall_driver = neutron.agent.firewall.NoopFirewallDriver[sriov_nic]physical_device_mappings = sriov1:ens1f0,sriov2:ens1f1exclude_devices =

3. 启动/重启服务

# systemctl enable neutron-sriov-nic-agent.service# systemctl start neutron-sriov-nic-agent.service# systemctl restart openstack-nova-compute.service

五、测试

1. 创建网络

创建网络# neutron net-create --provider:network_type vlan --provider:physical_network sriov1 --provider:segmentation_id 10 --router:external net1# neutron net-create --provider:network_type vlan --provider:physical_network sriov2 --provider:segmentation_id 20 --router:external net2创建子网(禁用DHCP)# neutron subnet-create --name sriov1-net --disable-dhcp --ip-version 4 net1 10.0.1.0/24# neutron subnet-create --name sriov2-net --disable-dhcp --ip-version 4 net2 10.0.2.0/24

2. 创建 port

# neutron port-create net1 --binding:vnic-type direct# neutron port-create net2 --binding:vnic-type direct

3. 创建虚拟机

# neutron port-list+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+| id | name | mac_address | fixed_ips |+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+| ba446152-bd45-4a38-9947-1d539e538a68 | | fa:16:3e:42:6b:fd | {"subnet_id": "04ea17b5-08da-41cc-8114-e0781a1f8041", "ip_address": "10.0.1.3"} || bf8a7655-ac16-4dce-bb12-54efd2dd0967 | | fa:16:3e:77:9f:8a | {"subnet_id": "6612bf80-d682-474f-886e-93029a4a0964", "ip_address": "10.0.2.3"} |+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+# openstack server create --image rhel-server-7.5-x86_64-kvm --flavor 6 --nic port-id=ba446152-bd45-4a38-9947-1d539e538a68 --nic port-id=bf8a7655-ac16-4dce-bb12-54efd2dd0967 --config-drive True test

4. 查看虚拟机网卡

  • 速率
  • 型号

5. 查看 VF

# ip link show ens1f06: ens1f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 14:02:ec:82:96:c0 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 7 MAC fa:16:3e:42:6b:fd, vlan 10, spoof checking on, link-state auto# ip link show ens1f17: ens1f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000 link/ether 14:02:ec:82:96:c1 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 3 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 6 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 7 MAC fa:16:3e:77:9f:8a, vlan 20, spoof checking on, link-state auto


本文参与腾讯云自媒体分享计划,欢迎正在阅读的你也加入,一起分享。

发表于

我来说两句

0 条评论
登录 后参与评论

相关文章

来自专栏乐沙弥的世界

记一次离奇的TNS-12545 TNS-12560 TNS-00515

      最近reportDB监听无法随系统自启动,现象比较怪异。因为该服务器上的另一个实例的监听可以正常启动,这个不能自启动实例的监听手动启动又是正常的。因...

882
来自专栏运维

linux文件树

以前有意找这方面的资料,今天突然发现在系统中就有 linux系统用man hier solaris用man  filesystem 其结果如下     ...

532
来自专栏闵开慧

django中mysql配置及使用

# Django settings for mysite2 project. DEBUG = True TEMPLATE_DEBUG = DEBUG ADMIN...

34210
来自专栏Hongten

python开发_os.path

=========================================

805
来自专栏后端云

openstack两个region分别用各自glance服务-&gt;共用glance服务

登陆dashboard,切换RegionOne和RegionTwo的镜像一览,由原来各自的镜像列表变成了一样的镜像列表

583
来自专栏后台及大数据开发

CentOS下redis集群安装

环境: 一台CentOS虚拟机上部署六个节点,创建3个master,3个slave节点

662
来自专栏Netkiller

Hyperledger Fabric Chaincode 开发

中国广东省深圳市龙华新区民治街道溪山美地 518131 +86 13113668890 <netkiller@msn.com>

57411
来自专栏有困难要上,没有困难创造困难也要上!

Linux下启动Oracle服务

1072
来自专栏乐享123

How to Penetrate GFW With ShadowSocks Docker Container on Centos7

1594
来自专栏聊聊技术

原 初学算法-基于最小堆的优先级队列C++

3059

扫码关注云+社区