1: Background
1. Tell a story
In the middle of last month, a friend from the planet came to me on wechat and said that his program was running, and the memory would continue to grow slowly and could not be released. How to solve it?
Well, it seems that the planet has to be done well!!! 😂😂😂 Anyway, talk to windbg first.
2: Windbg analysis
1. Empirical reasoning
From my friend's screenshot, there are a large number of 8216 bytes []. What does this mean? Friends who follow this series should know that there is also a byte [] array with size= (8216-24=8192) in the dump of a class III hospital. His problem is that there is a problem with OraBuf in the sdk when reading a large field in Oracle. In other words, it must be caused by the pool object in the underlying or third-party library, Next, from the managed heap.
2. View managed heap
0:000> !dumpheap -stat Statistics: 00007ffe107248f0 483707 15478624 System.Threading.PreAllocatedOverlapped 00007ffe1079c160 483744 15479808 System.Threading.ThreadPoolBoundHandle 00007ffe1079cff8 483701 23217648 System.Threading._IOCompletionCallback 00007ffe106e7a90 483704 23217792 Microsoft.Win32.SafeHandles.SafeFileHandle 00007ffe1079b088 483703 30956992 System.IO.FileSystemWatcher+AsyncReadState 00007ffe1079ceb0 483707 34826904 System.Threading.OverlappedData 00007ffe1079ccb0 483707 34826904 System.Threading.ThreadPoolBoundHandleOverlapped 0000016c64651080 245652 1473128080 Free 00007ffe105abf30 488172 3977571092 System.Byte[]
After sweeping the managed heap and lying in the slot, byte [] didn't attract me. Instead, it was attracted by System.IO.FileSystemWatcher+AsyncReadState. After all, it has been tossed by System.IO.FileSystemWatcher for many times, and it has deeply penetrated into my mind... After all, it's the program that gets stuck and the handle explodes high... This time, it probably caused trouble again. It seems that many programmers are still planted here.
In order to be rigorous, I still start with the largest System.Byte [], group it according to size, and then in descending order according to totalsize. I won't send ugly scripts, but directly upload the script output results.
!dumpheap -mt 00007ffe105abf30 size=8216,count=483703,totalsize=3790M size=8232,count=302,totalsize=2M size=65560,count=6,totalsize=0M size=131096,count=2,totalsize=0M size=4120,count=11,totalsize=0M size=56,count=301,totalsize=0M size=88,count=186,totalsize=0M size=848,count=16,totalsize=0M size=152,count=85,totalsize=0M size=46,count=242,totalsize=0M size=279,count=38,totalsize=0M !dumpheap -mt 00007ffe105abf30 -min 0n8216 -max 0n8216 -short 0000016c664277f0 0000016c66432a48 0000016c6648ef88 0000016c6649daa8 0000016c6649fb00 0000016c664a8b90 ...
From the output results, there are 48w byte s [] with size=8216, and the script also lists some address addresses with size of 8216. Next, use! gcroot, look at the references to these addresses.
0:000> !gcroot 0000016c664277f0
HandleTable:
0000016C65FC28C0 (async pinned handle)
-> 0000016C6628DEB0 System.Threading.OverlappedData
-> 0000016C664277F0 System.Byte[]
Found 1 unique roots (run '!gcroot -all' to see all roots).
0:000> !gcroot 0000016c667c80d0
HandleTable:
0000016C65FB7920 (async pinned handle)
-> 0000016C663260F8 System.Threading.OverlappedData
-> 0000016C667C80D0 System.Byte[]
From the output, we can see that these bytes [] are async pinned, that is, the storage space that needs to be filled for byte [] when asynchronous IO comes back. Next, let's see how to find the byte [] defined as 8192 in the source code through OverlappedData.
If you know the FileSystemWatcher, the reverse lookup chain is probably overlapped data - > threadpoolboundhandleoverlapped - > system.io.filesystemwatcher + asyncreadstate - > buffer [], which involves the binding of ThreadPool and SafeHandle.
0:000> !do 0000016C663260F8
Name: System.Threading.OverlappedData
MethodTable: 00007ffe1079ceb0
EEClass: 00007ffe107ac8d0
Size: 72(0x48) bytes
File: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Private.CoreLib.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffe106e3c08 40009ce 8 System.IAsyncResult 0 instance 0000000000000000 _asyncResult
00007ffe104a0c68 40009cf 10 System.Object 0 instance 0000016c66326140 _callback
00007ffe1079cb60 40009d0 18 ...eading.Overlapped 0 instance 0000016c663260b0 _overlapped
00007ffe104a0c68 40009d1 20 System.Object 0 instance 0000016c667c80d0 _userObject
00007ffe104af508 40009d2 28 PTR 0 instance 00000171728f66e0 _pNativeOverlapped
00007ffe104aee60 40009d3 30 System.IntPtr 1 instance 0000000000000000 _eventHandle
00007ffe104ab258 40009d4 38 System.Int32 1 instance 0 _offsetLow
00007ffe104ab258 40009d5 3c System.Int32 1 instance 0 _offsetHigh
0:000> !do 0000016c663260b0
Name: System.Threading.ThreadPoolBoundHandleOverlapped
MethodTable: 00007ffe1079ccb0
EEClass: 00007ffe107ac858
Size: 72(0x48) bytes
File: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.Private.CoreLib.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffe1079ceb0 40009d6 8 ...ng.OverlappedData 0 instance 0000016c663260f8 _overlappedData
00007ffe1079b818 40009c0 10 ...ompletionCallback 0 instance 0000016f661ab8a0 _userCallback
00007ffe104a0c68 40009c1 18 System.Object 0 instance 0000016c667ca0e8 _userState
00007ffe107248f0 40009c2 20 ...locatedOverlapped 0 instance 0000016c66326090 _preAllocated
00007ffe104af508 40009c3 30 PTR 0 instance 00000171728f66e0 _nativeOverlapped
00007ffe1079c160 40009c4 28 ...adPoolBoundHandle 0 instance 0000000000000000 _boundHandle
00007ffe104a7238 40009c5 38 System.Boolean 1 instance 0 _completed
00007ffe1079b818 40009bf 738 ...ompletionCallback 0 static 0000016f661ab990 s_completionCallback
0:000> !do 0000016c667ca0e8
Name: System.IO.FileSystemWatcher+AsyncReadState
MethodTable: 00007ffe1079b088
EEClass: 00007ffe107a9dc0
Size: 64(0x40) bytes
File: C:\Program Files\dotnet\shared\Microsoft.NETCore.App\5.0.10\System.IO.FileSystem.Watcher.dll
Fields:
MT Field Offset Type VT Attr Value Name
00007ffe104ab258 400002b 30 System.Int32 1 instance 1 <Session>k__BackingField
00007ffe105abf30 400002c 8 System.Byte[] 0 instance 0000016c667c80d0 <Buffer>k__BackingField
00007ffe106e7a90 400002d 10 ...es.SafeFileHandle 0 instance 0000016c66326028 <DirectoryHandle>k__BackingField
00007ffe1079c160 400002e 18 ...adPoolBoundHandle 0 instance 0000016c66326058 <ThreadPoolBinding>k__BackingField
00007ffe107248f0 400002f 20 ...locatedOverlapped 0 instance 0000016c66326090 <PreAllocatedOverlapped>k__BackingField
00007ffe1079b8c8 4000030 28 ...eSystem.Watcher]] 0 instance 0000016c66326078 <WeakWatcher>k__BackingField
< buffer > k above__ The backingfield was originally thrown to OverlappedData as a buffer for asynchronous IO reading and writing, and then look at the source code of System.IO.FileSystemWatcher+AsyncReadState.
With these principles, you can then ask your friend if you have set reloadonchange=true for appsettings. The friend found the following code, which is roughly written as follows:
public object GetxxxFlag()
{
string value = AppConfig.GetConfig("appsettings.json").GetValue("xxxx", "0");
return new
{
state = 200,
data = value
};
}
public class AppConfig
{
public static AppConfig GetConfig(string settingfile = "appsettings.json")
{
return new AppConfig(settingfile);
}
}
public class AppConfig
{
private AppConfig(string settingfile)
{
_config = new ConfigurationBuilder().AddJsonFile(settingfile, optional: true, reloadOnChange: true).Build();
_settingfile = settingfile;
}
}
From the perspective of source code logic, I guess my friend thinks that the GetConfig method is singleton after it is marked as static. Calling new AppConfig(settingfile) again will not repeat, so the problem is here.
But interestingly, the FileSystemWatcher in the previous two articles will cause the program to get stuck. Why not in this one? It happens that he didn't put the log file in the program root directory, otherwise... 😄😄😄， But I didn't expect to escape the card death, but I didn't escape the soul torture of a watcher's default 8byte space... 😂😂😂
3: Summary
In general, be careful when setting reloadOnChange: true. It may cause your program to get stuck, handle leak, memory leak, etc!!! I won't talk about the improvement plan. Refer to my previous series of articles.