一次永久代泄漏（perm泄漏）排查

春哥大魔王

发布于 2019-07-31 10:57:09

1.3K0

发布于 2019-07-31 10:57:09

文章被收录于专栏：服务端技术杂谈服务端技术杂谈

写在前面

网上关于perm区泄露的文章比较少，特别是对于动态类加载方面问题的分析比较少，在此记录下。

perm区问题一般两种解决方案：

启动时perm区问题，一般修改PermSize,MaxPermSize参数即可
运行时动态生成类加载问题，这种问题比较难搞，需要关心动态加载了哪些类

周末早晨收到几台机器告警(fullgc告警)(perm大于90%告警)，为快速解决问题，先把几台机器重启解决，留下了一台禁用端口保留现场进行问题分析。平时上线发版比较频繁，发版后jvm回收，如果较长时间没有发版可能会造成泄露，收到告警。

吃完早饭后登上机器进行排查。

排查过程

登入机器，查看内存使用高的进程：

top

pid:15298

既然是perm区问题，查看永久代情况：

jmap -permstat pid > 15298dump.permstat

class_loader	classes	bytes	parent_loader	alive?	type

<bootstrap>	3630	21866152	  null  	live	<internal>
0x0000000705e4df00	1	2008	0x00000006c002e450	dead	sun/reflect/DelegatingClassLoader@0x00000007c0050c30
0x00000007282e1040	4	23480	0x00000006c002e450	dead	com/facebook/swift/codec/internal/compiler/DynamicClassLoader@0x00000007c377f040

通过awk统计type类型，查看加载了哪种类型的类：

awk '{ arr[$6]+=$3 } END { for (key in arr) printf("%s\t%s\n", key, arr[key]) }' 15298dump.permstat | sort -k2,2

ava/net/URLClassLoader@0x00000007c01c1598	0
java/util/ResourceBundle$RBClassLoader@0x00000007c0337380	0
type	0
sun/reflect/DelegatingClassLoader@0x00000007c0050c30	11361872
sun/reflect/misc/MethodUtil@0x00000007c777bf60	134528
<internal>	21866152
com/alibaba/fastjson/util/ASMClassLoader@0x00000007c44de0e0	4785232
N/A	5146
sun/misc/Launcher$AppClassLoader@0x00000007c021c2d8	5423216
com/facebook/swift/codec/internal/compiler/DynamicClassLoader@0x00000007c377f040	5488288
sun/misc/Launcher$ExtClassLoader@0x00000007c01c1978	615216
org/eclipse/jetty/webapp/WebAppClassLoader@0x00000007c06e0da0	738107040

WebAppClassLoader加载最多，达到了738107040，近738m。

知道了加载的类最多，怎么分析具体加载了哪种类呢？

之前一般通过Java -verbose查看启动类加载过程。

对于运行时貌似有两种方式：

基于ClassFilter写一个拦截器，dump特定的类，使用SA的jar($JAVA_HOME/lib/sa-jdi.jar)编译好类，在编译好的类目录下调用下面的命令进行dump
问了(政威老师)，准备试试arthas，https://alibaba.github.io/arthas/classloader.html

机器上安装好arthas-boot.jar，并启动：

java -jar arthas-boot.jar

选择进程，执行类后台导出，便后续分析：

classloader -a >> &

打开导出文件：

hash:null, BootstrapClassLoader
 [B
 [C
 [D
 [F
 [I
 [J
 [Lcom.sun.activation.registries.MimeTypeFile;
 [Lcom.sun.imageio.plugins.jpeg.DHTMarkerSegment$Htable;
 [Lcom.sun.imageio.plugins.jpeg.ImageTypeProducer;
 [Lcom.sun.imageio.plugins.jpeg.JPEGImageReader$CallBackLock$State;
 [Lcom.sun.imageio.plugins.jpeg.JPEGImageWriter$CallBackLock$State;
 [Lcom.sun.imageio.plugins.jpeg.SOFMarkerSegment$ComponentSpec;
 [Lcom.sun.imageio.plugins.jpeg.SOSMarkerSegment$ScanComponentSpec;
 [Lcom.sun.jmx.mbeanserver.ClassLoaderRepositorySupport$LoaderEntry;
 [Lcom.sun.jmx.mbeanserver.MXBeanMapping;
 [Lcom.sun.org.apache.xalan.internal.utils.FeatureManager$Feature;
 [Lcom.sun.org.apache.xalan.internal.utils.FeaturePropertyBase$State;
 [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$Limit;
 [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$NameMap;
 [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityManager$State;
 [Lcom.sun.org.apache.xalan.internal.utils.XMLSecurityPropertyManager$Property;
 [Lcom.sun.org.apache.xerces.internal.impl.XMLEntityManager$CharacterBuffer;
 [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMLeaf;
 [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMNode;
 [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.CMStateSet;
 [Lcom.sun.org.apache.xerces.internal.impl.dtd.models.ContentModelValidator;
 [Lcom.sun.org.apache.xerces.internal.impl.dv.DatatypeValidator;
 [Lcom.sun.org.apache.xerces.internal.impl.dv.XSSimpleType;
 [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.AbstractDateTimeDV$DateTimeData;
 [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.TypeValidator;
 [Lcom.sun.org.apache.xerces.internal.impl.dv.xs.XSSimpleTypeDecl;
 [Lcom.sun.org.apache.xerces.internal.impl.xpath.regex.RegularExpression;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.SchemaGrammar;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.SubstitutionGroupHandler$OneSubGroup;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.XSAnnotationImpl;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.XSAttributeUseImpl;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.XSComplexTypeDecl;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.XSElementDecl;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.XSGroupDecl;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.XSNotationDecl;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.XSParticleDecl;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.identity.IdentityConstraint;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.identity.XPathMatcher;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.models.XSCMLeaf;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.models.XSCMValidator;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.opti.DefaultNode;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.opti.NodeImpl;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.traversers.OneAttr;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.traversers.XSDocumentInfo;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.util.SimpleLocator;
 [Lcom.sun.org.apache.xerces.internal.impl.xs.util.XInt;
 [Lcom.sun.org.apache.xerces.internal.util.Status;
 [Lcom.sun.org.apache.xerces.internal.util.SymbolHash$Entry;
 [Lcom.sun.org.apache.xerces.internal.util.SymbolTable$Entry;
 [Lcom.sun.org.apache.xerces.internal.util.XMLAttributesImpl$Attribute;
 [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$Limit;
 [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$NameMap;
 [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityManager$State;
 [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$Property;
 [Lcom.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$State;
 [Lcom.sun.org.apache.xerces.internal.xni.QName;
 [Lcom.sun.org.apache.xerces.internal.xni.XMLLocator;
 [Lcom.sun.org.apache.xerces.internal.xni.grammars.Grammar;
 [Lcom.sun.org.apache.xerces.internal.xni.grammars.XSGrammar;
 [Lcom.sun.org.apache.xerces.internal.xs.ShortList;
 [Lcom.sun.org.apache.xerces.internal.xs.XSAnnotation;
 [Lcom.sun.org.apache.xerces.internal.xs.XSAttributeUse;
 [Lcom.sun.org.apache.xerces.internal.xs.XSComplexTypeDefinition;
 [Lcom.sun.org.apache.xerces.internal.xs.XSElementDeclaration;
 [Lcom.sun.org.apache.xerces.internal.xs.XSIDCDefinition;
 [Lcom.sun.org.apache.xerces.internal.xs.XSModelGroupDefinition;
 [Lcom.sun.org.apache.xerces.internal.xs.XSNamespaceItem;
 [Lcom.sun.org.apache.xerces.internal.xs.XSNotationDeclaration;
 [Lcom.sun.org.apache.xerces.internal.xs.XSObject;
 [Lcom.sun.org.apache.xerces.internal.xs.XSParticle;
 [Lcom.sun.org.apache.xerces.internal.xs.XSSimpleTypeDefinition;
 [Lcom.sun.org.apache.xerces.internal.xs.XSTerm;
 [Lcom.sun.org.apache.xerces.internal.xs.XSTypeDefinition;
 [Lcom.sun.org.apache.xerces.internal.xs.datatypes.XSDateTime;
 [Lcom.sun.org.apache.xml.internal.dtm.DTM;
 [Lcom.sun.org.apache.xml.internal.dtm.DTMAxisTraverser;
 [Lcom.sun.org.apache.xml.internal.dtm.DTMIterator;
 [Lcom.sun.org.apache.xml.internal.dtm.ref.ExpandedNameTable$HashEntry;
 [Lcom.sun.org.apache.xml.internal.dtm.ref.ExtendedType;
 [Lcom.sun.org.apache.xpath.internal.Expression;
 [Lcom.sun.org.apache.xpath.internal.ExpressionNode;
 [Lcom.sun.org.apache.xpath.internal.XPathVisitable;
 [Lcom.sun.org.apache.xpath.internal.axes.LocPathIterator;
 [Lcom.sun.org.apache.xpath.internal.axes.PathComponent;
 [Lcom.sun.org.apache.xpath.internal.axes.PredicatedNodeTest;
 [Lcom.sun.org.apache.xpath.internal.axes.SubContextList;
 [Lcom.sun.org.apache.xpath.internal.objects.XObject;
 [Lcom.sun.org.apache.xpath.internal.patterns.NodeTest;
 [Lcom.sun.xml.internal.ws.org.objectweb.asm.Item;
 [Lcom.sun.xml.internal.ws.org.objectweb.asm.Type;
 [Ljava.awt.AWTKeyStroke;
 [Ljava.awt.Dimension;
 [Ljava.awt.Queue;
 [Ljava.awt.event.ActionListener;
 [Ljava.awt.event.ComponentListener;
 [Ljava.awt.event.FocusListener;
 [Ljava.awt.event.HierarchyBoundsListener;
 ......

统计最多的类：

#!/usr/bin/python

from collections import Counter

package_name_count_dic = {}
for s in open('14.txt'):
    full_qualified_name = s.strip()
    if not full_qualified_name:
        continue
    class_name_index = full_qualified_name.rfind('.')
    if class_name_index >= 0:
        package_name, class_name = full_qualified_name[:class_name_index], full_qualified_name[class_name_index + 1:]
    else:
        package_name = class_name = full_qualified_name
    if package_name in package_name_count_dic:
        package_name_count_dic[package_name] += 1
    else:
        package_name_count_dic[package_name] = 1
k = Counter(package_name_count_dic)
high = k.most_common(5)
for p, c in high:
    print p, c

最多的几个类：

真凶：ma.glasnost.orika.generated。

查看代码中谁使用了orika类库。

发现是闪购同学，由于我们的系统目前对接多方，闪购同学还在我们系统做代码开发，在codereview上存在一些问题。

@Component
public class SGMapperFactory implements FactoryBean<MapperFactory> {
    @Override
    public MapperFactory getObject()  {
        return new DefaultMapperFactory.Builder().build();
    }

    @Override
    public Class<?> getObjectType() {
        return MapperFactory.class;
    }

    @Override
    public boolean isSingleton() {
        return true;
    }
}

每次调用getObject都会新创建DefaultMapperFactory对象。MapperGenerator 每次会动态产生类。

至此问题排查，推动闪购同学改动。