RabbitMQ——内存调优（一）
【前言】
在《RabbitMQ——调优参数》一文中提到了rabbitmq相关的参数，对相关参数进行调整后测试发现在队列无堆积的情况下， 生产消费速度有明显提升；而在队列消息堆积的情况下，生产消费速度还是没有明显变化。在此场景中，磁盘IO、CPU、内存均未达到瓶颈，疑惑了一段时间后，对erlang的内存管理、GC、调度器等知识做了一定的学习研究，并通过erlang的lcnt分析erts（erlang运行时系统erlang runntime system）内部的线程锁，发现其内存分配的效率比较低，又进一步挖掘了下erlang的内存分配管理相关知识，并通过相关参数调优后，其性能有了显著的提升，因此对相关知识进行总结以便后续回顾。内存的调优分两部分：本文主要总结erlang内存相关的概念知识，下一篇文章总结rabbitmq内存相关知识与调优。
【erlang内存分配相关概念】
1、block&carriers
block是一块连续的内存，在erlang虚拟机中用于存储诸如erlang进程的堆、栈、二进制数据（binary）、ets等等数据。
carrier也是一块连续的内存，但在其内部会再划分成一个或多个block，即可以理解为carrier是block的容器。
仅包含一个block的carrier称为SBC（single block carrier），包含多个block的carrier称为MBC（multi block carrier）。
2、allocator
allocator负责carrier与block的管理，allocator会从系统申请内存，组织成不同的carrier，再从carrier中按需划分成block供使用。一个allocator中同时管理了多个MBC与SBC，下图是一个简单的示例。
在erlang虚拟机中有许多不同类型的allocator，针对不同类型的数据，由不同类型的allocator负责其内存的分配。例如：
eheap_alloc用于堆的内存分配；binary_alloc用于binary数据的内存分配；ets_alloc用于ets的内存分配等。
而所有这些allocator的内存最终都来自于sys_alloc、mseg_alloc；也就是说sys_alloc/mseg_alloc这两个allocator真正从系统申请内存，然后分配给不同的allocator，不同类型的allocator再用于不同类型数据的内存分配。
通常sys_alloc使用malloc的方式从系统申请内存，mseg_alloc采用mmap的方式从系统申请内存。
3、内存分配算法
前面讲到carrier可分为mbc与sbc。
sbc只包含一个block，通常是存放一个大块的数据，所以sbc的分配是很简单的。一般来说，sbc的内存最终都是通过sys_alloc申请分配的。
mbc中则包含多个block，每个block的大小不一定相同，同时一个allocator中管理了多个mbc，这些mbc会组成一个池子，每个mbc中会同时存在已分配的block和空闲的block，申请分配block时，先从mbc池子中找到合适的mbc，在从mbc中找到合适的block。
在erlang内部，对于carrier和block的分配，提供了多种不同的分配策略。例如：
Best fit策略——找最小满足需求的Block；
Address order best fit策略——找最小满足需求的block，如果同时存在多个这样的block，则选地址最小的那个block；
Address order first fit策略——选取地址最小的并且能满足需求的block；
等等，详细可以参考官方介绍文档。
这些不同的策略，有的是为了减少内存碎片，提高内存分配的效率；有的是为了加快分配速度，但可能会存在一定的内存浪费。
4、内存整体框架
erlang虚拟机内部，有多个调度器线程，负责调度运行erlang进程
默认情况下，调度器线程的个数与CPU核数相同
而每个调度器线程各自都有一个独立的allocator实例，
即eheap_alloc、binary_alloc、ets_alloc等在每个调度器线程中都有一个实例
这样这些allocator在内存分配时，尤其是MBC中block的分配，完全是无锁的，因此性能会比较高。
然而，每个调度器都从各自的allocator实例中进行内存的分配，会带来内存浪费的问题。
首先调度器默认使用策略是"full load or not", 也就是说低ID的调度器如果没饱和的话，不会用下一个调度器。
在高负载的情况下，更多的调度器被启用，该调度器上的内存被缓冲，留在池子里。
当工作负载下去的的话，因为压力没到，高ID的调度器没机会被使用，
也就是说这个时候，这个调度器上的内存就浪费掉了
从整个erlang虚拟机的角度来看，内存的碎片率就很高
为了解决这个问题，erlang引入的策略是当每个调度器池子里面的内存使用率低于一定程度的时候，就把该块内存让出来，让有需要的调度器能够利用起来。这样虽然解决了内存碎片的问题，但由于多个调度器可能会操作同一个mbc，那么对mbc的处理也就需要有锁的保护，因为是多线程来访问的。使用的时候需要综合来考虑。
整体构架如下图所示
【erlang内存分配相关参数】
erlang内存分配提供了一些设置参数，而这也是调优最多的地方，一些常用的参数有：
- 分配器的启用禁用
+M<S>e true|false
其中S是一个变量，不同的分配器有不同的标识符（下面配置项中的<S>均表示该意思）
B：binary_alloc
D：std_alloc
E：ets_alloc
F：fix_alloc
H：eheap_alloc
I：literal_alloc
L：ll_alloc
M：mseg_alloc
R：driver_alloc
S：sl_alloc
T：temp_alloc
X：exec_alloc
Y：sys_alloc
其中sys_alloc、literal_alloc和temp_alloc始终是启用的，即不可禁用；
mseg_alloc也始终是启用的（前提是支持mmap的情况下）；其他的分配器则可以通过参数来控制是否启用，被禁用的分配器，默认采用sys_alloc来替代。
- sbc分配的阀值
+M<S>sbct <size>
单位为KB，即申请的block大小超过该大小时采用sbc的方式来分配
- allocator中MBC大小的最大值、最小值，增长的幅度
+M<S>lmbcs <size>
设置allocator中MBC大小的最大值(largest multi block size),单位是KB
+M<S>smbcs <size>
设置allocator中MBC大小的最小值(smallest multi block size),单位是KB
+M<S>mbcgs <size>
设置allocator中MBC的递增幅度
即MBC的分配是按大小递增分配的，合理的设置MBC的最大值，最小值，即递增幅度有助于提高分配效率。
- 设置内存分配策略
+M<S>as
可选的值包括
bf（best fit）
aobf（address order best fit）
aoff（address order first fit）
aoffcbf（address order first fit carrier best fit）
aoffcaobf（address order first fit carrier address order best fit）
ageffcaoff（age order first fit carrier address order first fit）
ageffcbf（age order first fit carrier address order best fit）
gf（good fit）
af（a fit）
- 最大缓存个数
+MMmcs
其他更多的设置可参考官方文档说明
【erlang相关命令查看内存分配信息】
1、erlang:system_info(allocator).
通过这个命令可以看到erlang运行时系统中，当前所有类型调度器的概况。
2> erlang:system_info(allocator).
{glibc,[2,12],
[sys_alloc,temp_alloc,sl_alloc,std_alloc,ll_alloc,
eheap_alloc,ets_alloc,fix_alloc,literal_alloc,exec_alloc,
binary_alloc,driver_alloc,mseg_alloc,sys_aligned_alloc,
literal_mmap],
[{sys_alloc,[{e,true},{m,libc},{tt,131072},{tp,0}]},
{temp_alloc,[{e,true},
{t,true},
{atags,false},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,90},
{rsbcmt,80},
{rmbcmt,100},
{mmbcs,131072},
{mmmbc,18446744073709551615},
{mmsbc,256},
{lmbcs,10485760},
{smbcs,1048576},
{mbcgs,10},
{acul,0},
{acnl,0},
{acfml,0},
{mbsd,3},
{as,...}]},
{sl_alloc,[{e,true},
{t,true},
{atags,false},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,80},
{rsbcmt,80},
{rmbcmt,50},
{mmbcs,32768},
{mmmbc,18446744073709551615},
{mmsbc,256},
{lmbcs,5242880},
{smbcs,262144},
{mbcgs,10},
{acul,0},
{acnl,0},
{acfml,0},
{as,...}]},
{std_alloc,[{e,true},
{t,true},
{atags,false},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,20},
{rsbcmt,80},
{rmbcmt,50},
{mmbcs,32768},
{mmmbc,18446744073709551615},
{mmsbc,256},
{lmbcs,5242880},
{smbcs,262144},
{mbcgs,10},
{acul,0},
{acnl,0},
{acfml,...},
{...}]},
{ll_alloc,[{e,true},
{t,true},
{atags,false},
{ramv,false},
{sbct,18446744073709551615},
{asbcst,0},
{rsbcst,0},
{rsbcmt,0},
{rmbcmt,0},
{mmbcs,524288},
{mmmbc,18446744073709551615},
{mmsbc,0},
{lmbcs,5242880},
{smbcs,262144},
{mbcgs,10},
{acul,0},
{acnl,...},
{...}|...]},
{eheap_alloc,[{e,true},
{t,true},
{atags,false},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,50},
{rsbcmt,80},
{rmbcmt,50},
{mmbcs,131072},
{mmmbc,18446744073709551615},
{mmsbc,256},
{lmbcs,5242880},
{smbcs,262144},
{mbcgs,10},
{acul,...},
{...}|...]},
{ets_alloc,[{e,true},
{t,true},
{atags,false},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,20},
{rsbcmt,80},
{rmbcmt,50},
{mmbcs,32768},
{mmmbc,18446744073709551615},
{mmsbc,256},
{lmbcs,5242880},
{smbcs,262144},
{mbcgs,...},
{...}|...]},
{fix_alloc,[{e,true},
{t,true},
{atags,false},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,20},
{rsbcmt,80},
{rmbcmt,50},
{mmbcs,32768},
{mmmbc,18446744073709551615},
{mmsbc,256},
{lmbcs,5242880},
{smbcs,...},
{...}|...]},
{literal_alloc,[{e,true},
{t,false},
{atags,false},
{ramv,false},
{sbct,18446744073709551615},
{asbcst,0},
{rsbcst,0},
{rsbcmt,0},
{rmbcmt,0},
{mmbcs,1048576},
{mmmbc,18446744073709551615},
{mmsbc,0},
{lmbcs,...},
{...}|...]},
{exec_alloc,[{e,true},
{t,false},
{atags,false},
{ramv,false},
{sbct,18446744073709551615},
{asbcst,0},
{rsbcst,0},
{rsbcmt,0},
{rmbcmt,0},
{mmbcs,0},
{mmmbc,18446744073709551615},
{mmsbc,...},
{...}|...]},
{binary_alloc,[{e,true},
{t,true},
{atags,true},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,20},
{rsbcmt,80},
{rmbcmt,50},
{mmbcs,32768},
{mmmbc,...},
{...}|...]},
{driver_alloc,[{e,true},
{t,true},
{atags,true},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,20},
{rsbcmt,80},
{rmbcmt,50},
{mmbcs,...},
{...}|...]},
{test_alloc,[{e,false}]},
{mseg_alloc,[{amcbf,4194304},{rmcbf,20},{mcs,10}]},
{alloc_util,[{mmc,18446744073709551615},
{ycs,1048576},
{sac,true}]},
{erts_mmap,[{scs,0}]},
{instr,[{t,false}]},
{lock_physical_memory,no}]}2、erlang:system_info({allocator,AllocType}).
通过这个命令可以看到erlang运行时系统中，不同类型分配器实例的状况。
这里AllocType是一个变量，有效值为不同分配器（binary_alloc、eheap_alloc等）
例如：
4> erlang:system_info({allocator,binary_alloc}).
[{instance,0,
[{versions,"0.9","3.0"},
{options,[{e,true},
{t,true},
{atags,true},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,20},
{rsbcmt,80},
{rmbcmt,50},
{mmbcs,32768},
{mmmbc,18446744073709551615},
{mmsbc,256},
{lmbcs,5242880},
{smbcs,262144},
{mbcgs,10},
{acul,0},
{acnl,0},
{acfml,0},
{as,aoffcbf}]},
{mbcs,[{blocks,12,31,31},
{blocks_size,256288,1341744,1341744},
{carriers,2,4,4},
{mseg_alloc_carriers,1},
{sys_alloc_carriers,1},
{carriers_size,1081344,3440640,3440640},
{mseg_alloc_carriers_size,1048576},
{sys_alloc_carriers_size,32768}]},
{sbcs,[{blocks,0,1,1},
{blocks_size,0,547896,547896},
{carriers,0,1,1},
{mseg_alloc_carriers,0},
{sys_alloc_carriers,0},
{carriers_size,0,548864,548864},
{mseg_alloc_carriers_size,0},
{sys_alloc_carriers_size,0}]},
{calls,[{binary_alloc,0,277},
{binary_free,0,265},
{binary_realloc,0,4},
{mseg_alloc,0,10},
{mseg_dealloc,0,9},
{mseg_realloc,0,0},
{sys_alloc,0,1},
{sys_free,0,0},
{sys_realloc,0,0}]}]},
{instance,1,
[{versions,"0.9","3.0"},
{options,[{e,true},
{t,true},
{atags,true},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,20},
{rsbcmt,80},
{rmbcmt,50},
{mmbcs,32768},
{mmmbc,18446744073709551615},
{mmsbc,256},
{lmbcs,5242880},
{smbcs,262144},
{mbcgs,10},
{acul,60},
{acnl,1000},
{acfml,0},
{as,...}]},
{mbcs,[{blocks,15,46,46},
{blocks_size,22480,80608,80608},
{carriers,2,2,2},
{mseg_alloc_carriers,1},
{sys_alloc_carriers,1},
{carriers_size,294912,294912,294912},
{mseg_alloc_carriers_size,262144},
{sys_alloc_carriers_size,32768}]},
{mbcs_pool,[{blocks,0},
{blocks_size,0},
{carriers,0},
{carriers_size,0},
{entrance_removed,0,0},
{skip_race,0,0},
{skip_homecoming,0,0},
{skip_not_pooled,0,0},
{skip_busy,0,0},
{skip_size,0,0},
{fetch,0,0},
{fail,0,0},
{fail_pend_dealloc,0,0},
{fail_shared,0,0},
{fail_pooled,0,0}]},
{sbcs,[{blocks,0,0,0},
{blocks_size,0,0,0},
{carriers,0,0,0},
{mseg_alloc_carriers,0},
{sys_alloc_carriers,0},
{carriers_size,0,0,0},
{mseg_alloc_carriers_size,0},
{sys_alloc_carriers_size,0}]},
{calls,[{binary_alloc,0,592},
{binary_free,0,577},
{binary_realloc,0,0},
{mseg_alloc,0,3},
{mseg_dealloc,0,2},
{mseg_realloc,0,0},
{sys_alloc,0,1},
{sys_free,0,0},
{sys_realloc,0,0}]}]},
{instance,2,
[{versions,"0.9","3.0"},
{options,[{e,true},
{t,true},
{atags,true},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,20},
{rsbcmt,80},
{rmbcmt,50},
{mmbcs,32768},
{mmmbc,18446744073709551615},
{mmsbc,256},
{lmbcs,5242880},
{smbcs,262144},
{mbcgs,10},
{acul,60},
{acnl,1000},
{acfml,...},
{...}]},
{mbcs,[{blocks,0,0,0},
{blocks_size,0,0,0},
{carriers,1,1,1},
{mseg_alloc_carriers,0},
{sys_alloc_carriers,1},
{carriers_size,32768,32768,32768},
{mseg_alloc_carriers_size,0},
{sys_alloc_carriers_size,32768}]},
{mbcs_pool,[{blocks,0},
{blocks_size,0},
{carriers,0},
{carriers_size,0},
{entrance_removed,0,0},
{skip_race,0,0},
{skip_homecoming,0,0},
{skip_not_pooled,0,0},
{skip_busy,0,0},
{skip_size,0,0},
{fetch,0,0},
{fail,0,0},
{fail_pend_dealloc,0,0},
{fail_shared,0,0},
{fail_pooled,0,...}]},
{sbcs,[{blocks,0,0,0},
{blocks_size,0,0,0},
{carriers,0,0,0},
{mseg_alloc_carriers,0},
{sys_alloc_carriers,0},
{carriers_size,0,0,0},
{mseg_alloc_carriers_size,0},
{sys_alloc_carriers_size,0}]},
{calls,[{binary_alloc,0,0},
{binary_free,0,0},
{binary_realloc,0,0},
{mseg_alloc,0,0},
{mseg_dealloc,0,0},
{mseg_realloc,0,0},
{sys_alloc,0,1},
{sys_free,0,0},
{sys_realloc,0,0}]}]},
{instance,3,
[{versions,"0.9","3.0"},
{options,[{e,true},
{t,true},
{atags,true},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,20},
{rsbcmt,80},
{rmbcmt,50},
{mmbcs,32768},
{mmmbc,18446744073709551615},
{mmsbc,256},
{lmbcs,5242880},
{smbcs,262144},
{mbcgs,10},
{acul,60},
{acnl,...},
{...}|...]},
{mbcs,[{blocks,0,0,0},
{blocks_size,0,0,0},
{carriers,1,1,1},
{mseg_alloc_carriers,0},
{sys_alloc_carriers,1},
{carriers_size,32768,32768,32768},
{mseg_alloc_carriers_size,0},
{sys_alloc_carriers_size,32768}]},
{mbcs_pool,[{blocks,0},
{blocks_size,0},
{carriers,0},
{carriers_size,0},
{entrance_removed,0,0},
{skip_race,0,0},
{skip_homecoming,0,0},
{skip_not_pooled,0,0},
{skip_busy,0,0},
{skip_size,0,0},
{fetch,0,0},
{fail,0,0},
{fail_pend_dealloc,0,0},
{fail_shared,0,...},
{fail_pooled,...}]},
{sbcs,[{blocks,0,0,0},
{blocks_size,0,0,0},
{carriers,0,0,0},
{mseg_alloc_carriers,0},
{sys_alloc_carriers,0},
{carriers_size,0,0,0},
{mseg_alloc_carriers_size,0},
{sys_alloc_carriers_size,0}]},
{calls,[{binary_alloc,0,0},
{binary_free,0,0},
{binary_realloc,0,0},
{mseg_alloc,0,0},
{mseg_dealloc,0,0},
{mseg_realloc,0,0},
{sys_alloc,0,1},
{sys_free,0,0},
{sys_realloc,0,0}]}]},
{instance,4,
[{versions,"0.9","3.0"},
{options,[{e,true},
{t,true},
{atags,true},
{ramv,false},
{sbct,524288},
{asbcst,4145152},
{rsbcst,20},
{rsbcmt,80},
{rmbcmt,50},
{mmbcs,32768},
{mmmbc,18446744073709551615},
{mmsbc,256},
{lmbcs,5242880},
{smbcs,262144},
{mbcgs,10},
{acul,...},
{...}|...]},
{mbcs,[{blocks,0,9,9},
{blocks_size,0,9936,9936},
{carriers,1,1,1},
{mseg_alloc_carriers,0},
{sys_alloc_carriers,1},
{carriers_size,32768,32768,32768},
{mseg_alloc_carriers_size,0},
{sys_alloc_carriers_size,32768}]},
{mbcs_pool,[{blocks,0},
{blocks_size,0},
{carriers,0},
{carriers_size,0},
{entrance_removed,0,0},
{skip_race,0,0},
{skip_homecoming,0,0},
{skip_not_pooled,0,0},
{skip_busy,0,0},
{skip_size,0,0},
{fetch,0,0},
{fail,0,0},
{fail_pend_dealloc,0,...},
{fail_shared,...},
{...}]},
{sbcs,[{blocks,0,0,0},
{blocks_size,0,0,0},
{carriers,0,0,0},
{mseg_alloc_carriers,0},
{sys_alloc_carriers,0},
{carriers_size,0,0,0},
{mseg_alloc_carriers_size,0},
{sys_alloc_carriers_size,0}]},
{calls,[{binary_alloc,0,9},
{binary_free,0,9},
{binary_realloc,0,0},
{mseg_alloc,0,0},
{mseg_dealloc,0,0},
{mseg_realloc,0,0},
{sys_alloc,0,1},
{sys_free,0,0},
{sys_realloc,0,0}]}]}]当然还可以通过recon库提供的相关函数来方便的查看这些信息。
【总结】
本文粗略的介绍了erlang内存分配相关的概念，以及对应的参数设置项，了解这些原理会有助于帮助我们进一步分析上层业务的瓶颈点。