Recently, a machine on the line gave an alarm after running for more than 10 days, and fgc frequently appeared. After cutting off the traffic, it took the application heapdump file from the operation and maintenance side. At the beginning of fgc, I went to the container platform to check the gc logs, which are as follows:

It can be seen from the log that it is obviously better than fgc caused by insufficient space of metaspace, and fgc is carried out continuously, and the space of metaspace cannot be recycled. Then check the jvm startup parameters, which are as follows:

Here, both Metaspace and MaxMetaspace are set to 256M. It's strange that fgc appears when the Metaspace in gc log only uses 165M. Isn't it the newly loaded 90M like space? I'm sure it's not. If it's not caused by the newly applied 90M space, it's only caused by the memory fragment of Metaspace. So through mat analysis of heapdump, we found that there are more than 1100 delegating classloaders, so let's first look at what delegating classloader is? It belongs to the sun.reflect package. The code is as follows:

classDelegatingClassLoader extendsClassLoader { DelegatingClassLoader(ClassLoader var1) { super(var1); }

Prove that it is a ClassLoader.

What objects are referencing these classloaders? Through mat, it is found that the GeneratedMethodAccessor is referencing these classloaders. After further tracking, it is found that the reflectors of mybatis have applied these objects. OK, so I continue to check the code of the Reflector. The code snippet is as follows:

privateMap<String, Invoker> setMethods= newHashMap<String, Invoker>();privateMap<String, Invoker> getMethods= newHashMap<String, Invoker>();privateMap<String, Class<?>> setTypes= newHashMap<String, Class<?>>();privateMap<String, Class<?>> getTypes= newHashMap<String, Class<?>>();

This Reflector object will cache the getter setter method of entity class in orm, and mybatis needs to convert the records in the table into java entity class. In order to improve the efficiency of reflection, mybatis will cache the methods and constructors of entity class. During the running process, mybatis will create a Reflector for each entity class through ReflectorFactory for subsequent reflection calls.

The question is, why are there so many delegatingclassloaders? It can be analyzed through mat that these classloaders are ultimately referenced by java Method objects.

After analyzing the method creation process and method call process, it is found that method will create a methodaccessor in the call process and use MehtodAccessor as a field called methodaccessor. In order to improve the performance of reflection call, java uses an expansion method (from jni call to classbytes call) through the parameter - dsun. Reflect. I Nflationthreshold is controlled by 15 by default. When the number is less than this, native method will be used to call methods. If the number of method calls exceeds the specified number, bytecode method will be used to generate method calls. If bytecode method is used, DelegatingClassLoader will be generated for each method. The specific source code is as follows: Method.invoke method:

Method.acquireMethodAccessor method:

ReflectionFactory.newMethodAccessor method:

Nativemethodeaccessorimpl.invoke method:

publicObject invoke(Object var1, Object[] var2) throwsIllegalArgumentException, InvocationTargetException { if(++this.numInvocations > ReflectionFactory.inflationThreshold() && !ReflectUtil.isVMAnonymousClass(this.method.getDeclaringClass())) { MethodAccessorImpl var3 = (MethodAccessorImpl)(newMethodAccessorGenerator()).generateMethod(this.method.getDeclaringClass(), this.method.getName(), this.method.getParameterTypes(), this.method.getReturnType(), this.method.getExceptionTypes(), this.method.getModifiers()); this.parent.setDelegate(var3); }
 returninvoke0(this.method, var1, var2);}

MethodAccessorGenerator.generateMethod method method fragment:

ClassDefiner.defineClass method:

In addition, the checkinitiated method of RefectionFactory will take sun.reflect.inflationThresholdproperty through the System.getProperty method. The default value is 15. The code flow is not very long, so it is easy to understand. The next step is to verify whether the java reflection is caused by the way of Inflat. Then write the following example for verification:

/-XX:MetaspaceSize=64M -XX:MaxMetaspaceSize=64M -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCTimeStamps-XX:+PrintGCDetails -Dsun.reflect.inflationThreshold=0/public static voidmain(String[] args) throwsIOException, InvocationTargetException, IllegalAccessException { ReflectorFactory reflectorFactory = newDefaultReflectorFactory(); System.out.println("load class start"); //model has 1000 methods: Reflector 1 = reflectorFactory.findForClass(TestModel.class); Reflector 2 = reflectorFactory.findForClass(TestModel2.class); Reflector 3 = reflectorFactory.findForClass(TestModel3.class);
 System.out.println("load class finished");
 //model has 1000 methods testmodel testmodel = - newTestModel();
 Object[] empty = {}; Object[] one1 = {"a"};
 TestModel2 testModel2 = newTestModel2();
 TestModel3 testModel3 = newTestModel3();
 System.out.println("method invoke start"); for(inti = 0; i < 1; i++) { for(intj = 0; j < 1000; j++) { reflector1.getSetInvoker("field"+ j).invoke(testModel, one1); reflector1.getGetInvoker("field"+ j).invoke(testModel, empty);
 reflector2.getSetInvoker("field"+ j).invoke(testModel2, one1); reflector2.getGetInvoker("field"+ j).invoke(testModel2, empty);
 reflector3.getSetInvoker("field"+ j).invoke(testModel3, one1); reflector3.getGetInvoker("field"+ j).invoke(testModel3, empty); } } System.out.println("method invoke finished"); System.in.read();}

By not setting the parameter sun.reflect.inflationThreshold and setting the parameter to 0, the running result is as follows: in case of not setting:

Set to 0:

It can be seen that the memory occupation of Metaspace under the two settings is quite different, and the results of basic verification analysis are correct. Finally, the solutions for frequent fgc repair caused by Metaspace are as follows:

• Increase Metaspace space

• At the expense of some performance, add the parameter - Dsun.reflect.inflationThreshold to the application startup parameter, and set its value to be large enough.