网卡中断绑定
网卡中断
- 获取网卡设备所有中断
[root@PowerCycle_1_DL380G7 ~]# cat /proc/interrupts|grep -iw eth0
52: 12403 2 13291 26 14094 20 13577 32 14576 21 13772 29 PCI-MSI-edge eth0-0
53: 2524 0 2647 2 2570 5 2653 9 2407 0 2419 20 PCI-MSI-edge eth0-1
54: 2770 0 2046 2 2311 6 2387 6 2243 8 2241 6 PCI-MSI-edge eth0-2
55: 31 0 631 1 729 3 684 0 496 1 647 0 PCI-MSI-edge eth0-3
56: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth0-4
57: 345 0 406 0 389 0 518 0 425 1 464 0 PCI-MSI-edge eth0-5
58: 11484 4 12300 15 13035 26 12789 26 13259 27 12407 23 PCI-MSI-edge eth0-6
59: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth0-7
- 查看机台node信息 （附录：What is NUMA）
[root@PowerCycle_1_DL380G7 ~]# cat /sys/devices/system/node/node0/cpulist
0,2,4,6,8,10
[root@PowerCycle_1_DL380G7 ~]# cat /sys/devices/system/node/node1/cpulist
1,3,5,7,9,11
[root@PowerCycle_1_DL380G7 ~]# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 2 4 6 8 10
node 0 size: 8189 MB
node 0 free: 111 MB
node 1 cpus: 1 3 5 7 9 11
node 1 size: 8191 MB
node 1 free: 63 MB
node distances:
node 0 1
0: 10 20
1: 20 10
- 获取连接网卡设备的
numa node id（附录：numa_node for pci device）
[root@PowerCycle_1_DL380G7 ~]# cat /sys/class/net/eth0/device/numa_node
0
- 依次绑定网卡中断到同node 的cpu core，若网卡中断数大于node core数，则重新从第一core开始，循环绑定所有中断 （附录：IRQ-affinity）
echo 0 > /proc/irq/52/smp_affinity_list # 将中断52绑定到core 0
注意事项
- 默认情况下，有一个irqbalance服务对IRQ进行负载均衡，在设置中断绑定时，可以根据需要停止这个服务。
- kernel启动参数:
isolcpus=CPU编号列表。将列表中的CPU从内核SMP平衡和调度算法中剔除。[注意]提出后并不是绝对不能再使用该CPU的，操作系统仍然可以强制指定特定的进程使用哪个CPU(可以通过taskset来做到)。该选项的目的主要是用于实现特定cpu只运行特定进程的目的。CPU编号从"0"开始计数，列表的表示方法有三种：numA,numB,...,numNnumA-numN以及上述两种表示方法的组合：numA,...,numM-numN例如：0,3,4-7,9
附录
What is NUMA
==============================================================================
1 - Introduction
==============================================================================
Systems employing a Non Uniform Memory Access (NUMA) architecture contain
collections of hardware resources including processors, memory, and I/O buses,
that comprise what is commonly known as a NUMA node.
Processor accesses to memory within the local NUMA node is generally faster
than processor accesses to memory outside of the local NUMA node.
DT defines interfaces that allow the platform to convey NUMA node
topology information to OS.
==============================================================================
2 - numa-node-id
==============================================================================
For the purpose of identification, each NUMA node is associated with a unique
token known as a node id. For the purpose of this binding
a node id is a 32-bit integer.
A device node is associated with a NUMA node by the presence of a
numa-node-id property which contains the node id of the device.
numa_node for pci device
What: /sys/bus/pci/devices/.../numa_node
Description:
This file contains the NUMA node to which the PCI device is
attached, or -1 if the node is unknown. The initial value
comes from an ACPI _PXM method or a similar firmware
source. If that is missing or incorrect, this file can be
written to override the node. In that case, please report
a firmware bug to the system vendor. Writing to this file
taints the kernel with TAINT_FIRMWARE_WORKAROUND, which
reduces the supportability of your system.
IRQ-affinity
================
SMP IRQ affinity
================
/proc/irq/IRQ#/smp_affinity and /proc/irq/IRQ#/smp_affinity_list specify
which target CPUs are permitted for a given IRQ source. It's a bitmask
(smp_affinity) or cpu list (smp_affinity_list) of allowed CPUs. It's not
allowed to turn off all CPUs, and if an IRQ controller does not support
IRQ affinity then the value will not change from the default of all cpus.
/proc/irq/default_smp_affinity specifies default affinity mask that applies
to all non-active IRQs. Once IRQ is allocated/activated its affinity bitmask
will be set to the default mask. It can then be changed as described above.
Default mask is 0xffffffff.
Here is an example of restricting IRQ44 (eth1) to CPU0-3 then restricting
it to CPU4-7 (this is an 8-CPU SMP box)::
[root@moon 44]# cd /proc/irq/44
[root@moon 44]# cat smp_affinity
ffffffff
[root@moon 44]# echo 0f > smp_affinity
[root@moon 44]# cat smp_affinity
0000000f
[root@moon 44]# ping -f h
PING hell (195.4.7.3): 56 data bytes
...
--- hell ping statistics ---
6029 packets transmitted, 6027 packets received, 0% packet loss
round-trip min/avg/max = 0.1/0.1/0.4 ms
[root@moon 44]# cat /proc/interrupts | grep 'CPU\|44:'
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
44: 1068 1785 1785 1783 0 0 0 0 IO-APIC-level eth1
As can be seen from the line above IRQ44 was delivered only to the first four
processors (0-3).
Now lets restrict that IRQ to CPU(4-7).
::
[root@moon 44]# echo f0 > smp_affinity
[root@moon 44]# cat smp_affinity
000000f0
[root@moon 44]# ping -f h
PING hell (195.4.7.3): 56 data bytes
..
--- hell ping statistics ---
2779 packets transmitted, 2777 packets received, 0% packet loss
round-trip min/avg/max = 0.1/0.5/585.4 ms
[root@moon 44]# cat /proc/interrupts | 'CPU\|44:'
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7
44: 1068 1785 1785 1783 1784 1069 1070 1069 IO-APIC-level eth1
This time around IRQ44 was delivered only to the last four processors.
i.e counters for the CPU0-3 did not change.
Here is an example of limiting that same irq (44) to cpus 1024 to 1031::
[root@moon 44]# echo 1024-1031 > smp_affinity_list
[root@moon 44]# cat smp_affinity_list
1024-1031
Note that to do this with a bitmask would require 32 bitmasks of zero
to follow the pertinent one.
/proc/interrupts 文件示例
[root@PowerCycle_1_DL380G7 ~]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11
0: 7677 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-edge timer
1: 2 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-edge i8042
3: 3 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-edge
4: 2 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-edge
5: 481 0 0 0 500520 0 0 0 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1, ohci_hcd:usb2, ohci_hcd:usb3
8: 1 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-edge rtc0
9: 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-fasteoi acpi
12: 4 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-edge i8042
14: 0 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-fasteoi sata_svw
36: 63 0 0 0 391 0 0 0 0 0 0 0 IO-APIC-fasteoi ioc0
44: 303 0 0 0 0 0 0 0 301 0 0 0 IO-APIC-fasteoi radeon
45: 36 0 0 0 0 1 0 0 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4, hpilo
46: 175 0 0 0 0 0 0 0 0 0 0 0 IO-APIC-fasteoi ipmi_si
48: 1032219 178024 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge hpsa0
52: 12403 2 13280 26 14087 20 13568 32 14570 21 13771 29 PCI-MSI-edge eth0-0
53: 2524 0 2647 2 2570 5 2653 9 2407 0 2419 20 PCI-MSI-edge eth0-1
54: 2770 0 2046 2 2309 6 2386 6 2242 8 2241 6 PCI-MSI-edge eth0-2
55: 31 0 631 1 729 3 684 0 494 1 647 0 PCI-MSI-edge eth0-3
56: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth0-4
57: 345 0 406 0 389 0 518 0 425 1 464 0 PCI-MSI-edge eth0-5
58: 11484 4 12292 15 13032 26 12782 26 13249 27 12405 23 PCI-MSI-edge eth0-6
59: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth0-7
60: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge cnic
61: 405593 2251345 943965 871 768474 804 578198 1670 484746 1838 1123713 1273 PCI-MSI-edge eth2-0
62: 465934 6153157 1980804 1812 545042 1584 340214 339763 337969 1223 404626 21366 PCI-MSI-edge eth2-1
63: 257035 10370169 647710 1672 292647 7128 158399 2205 445043 924542 678751 1016 PCI-MSI-edge eth2-2
64: 184523 4048634 3021307 56075 279545 2023263 373303 947 284113 61405 329394 381417 PCI-MSI-edge eth2-3
65: 210612 2736712 313831 1164 328921 5050 294789 4633 358082 2089 266162 2241 PCI-MSI-edge eth2-4
66: 140689 950354 4326650 372421 195675 1804 292528 7952 232210 562 1612581 51576 PCI-MSI-edge eth2-5
67: 232630 11041990 621602 213384 330390 4303 1243816 3393 300024 1636 194228 1588 PCI-MSI-edge eth2-6
68: 231324 2579573 3257214 5004 221930 2400 198914 4274 245901 832 198333 739 PCI-MSI-edge eth2-7
69: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge cnic
70: 1265332 247 808639 969285 1011166 172305 2868575 9972 2150200 4946 1838264 2397 PCI-MSI-edge eth3-0
71: 112867 1508675 514259 94410 246320 1428 148543 84828 179966 1299 160826 691 PCI-MSI-edge eth3-1
72: 248297 1052967 399836 171589 356165 5245 280966 3043 468521 2306 358288 13413 PCI-MSI-edge eth3-2
73: 165245 41545 331112 4011 303745 617 282643 2946 141572 3838 290459 476364 PCI-MSI-edge eth3-3
74: 283641 185921 352780 870608 379490 2143 390607 8430 274116 4940 346358 1555 PCI-MSI-edge eth3-4
75: 191985 51025 346074 311959 317937 1017 201560 1901 215988 4426 279054 1396 PCI-MSI-edge eth3-5
76: 799826 948064 694805 86918 663542 118124 504472 2727 584446 4405 482636 2864 PCI-MSI-edge eth3-6
77: 177551 586292 464620 53725 223075 1289 237082 124488 251205 575656 202223 1727 PCI-MSI-edge eth3-7
78: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge cnic
79: 1 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth1-0
80: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth1-1
81: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth1-2
82: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth1-3
83: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth1-4
84: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth1-5
85: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth1-6
86: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge eth1-7
87: 0 0 0 0 0 0 0 0 0 0 0 0 PCI-MSI-edge cnic
NMI: 183 193 311 300 82 73 46 36 44 45 45 34 Non-maskable interrupts
LOC: 125614374 129491009 125967521 122174709 44175771 40677695 7410066 5246562 6503443 4989630 6236192 3668821 Local timer interrupts
SPU: 0 0 0 0 0 0 0 0 0 0 0 0 Spurious interrupts
PMI: 183 193 311 300 82 73 46 36 44 45 45 34 Performance monitoring interrupts
PND: 0 0 0 0 0 0 0 0 0 0 0 0 Performance pending work
RES: 14550969 10563446 39027775 39346875 3473546 1777228 1337871 487759 1124219 419067 1008728 394662 Rescheduling interrupts
CAL: 182225 436776 30628 23849 349233 39870 562660 85246 275418 8636969 1778242 158955 Function call interrupts
TLB: 1199549 1119584 8878827 8912733 91958 77262 19831 17436 1843 1023 1034 172 TLB shootdowns
TRM: 0 0 0 0 0 0 0 0 0 0 0 0 Thermal event interrupts
THR: 0 0 0 0 0 0 0 0 0 0 0 0 Threshold APIC interrupts
MCE: 0 0 0 0 0 0 0 0 0 0 0 0 Machine check exceptions
MCP: 7545 7545 7545 7545 7545 7545 7545 7545 7545 7545 7545 7545 Machine check polls
ERR: 0
MIS: 0
本文参与 腾讯云自媒体同步曝光计划，分享自微信公众号。
原始发表：2020-03-13，如有侵权请联系 cloudcommunity@tencent.com 删除
本文分享自 WriteSimpleDemo 微信公众号，前往查看
如有侵权，请联系 cloudcommunity@tencent.com 删除。
本文参与 腾讯云自媒体同步曝光计划 ，欢迎热爱写作的你一起参与！
评论
登录后参与评论
推荐阅读
腾讯云开发者
