Reference document

https://blog.51cto.com/seekerwolf/2357220
https://www.centos.bz/2017/08/zabbix-zbx_mem_malloc-out-of-memory/

Environmental Science

CentOS 7.5.1804
Zabbix 3.4.15

phenomenon
zabbix web graphical interface shows that zabbix server is not running. Enter zabbix server systemctl status zabbix server display service to down

Troubleshooting
1. Try to restart ZABBIX server, failed to start
2. View the log / var/log/zabbix/zabbix_server.log

The following errors are found first

zabbix_server [7988]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_server [8008]: cannot open log: cannot create semaphore set: [28] No space left on device
zabbix_server [8022]: cannot open log: cannot create semaphore set: [28] No space left on device

Modify the kernel.sem parameter according to the online data. The default is as follows

[root@zabbix zabbix]# cat /proc/sys/kernel/sem
250     32000   32      128

The meanings are as follows

250 semmsl Max semaphores per array signal set holds the maximum number of signals   
32000 semmns Max semaphores system wide maximum number of all signals 
32 semopm Max OPS per semop call calls the maximum number of signals in a single signal set 
128 maximum number of arrays

Double directly and make it effective

[root@zabbix zabbix]# echo 'kernel.sem = 500 64000 64 256' >> /etc/sysctl.conf
[root@zabbix zabbix]# sysctl -p
kernel.sem = 500 64000 64 256

Finally, kill the shared channel occupied by zabbix

[root@zabbix zabbix]# ipcs -s|wc -l
143
[root@zabbix zabbix]# ipcs -s | grep zabbix | awk '{print $2}' | xargs -n 1 ipcrm -s
[root@zabbix zabbix]# ipcs -s|wc -l
10

3. the following errors were found during the second startup of zabbix-server

8599:20200401:140301.791 using configuration file: /etc/zabbix/zabbix_server.conf
8599:20200401:140301.798 current database version (mandatory/optional): 03040000/03040007
8599:20200401:140301.799 required mandatory version: 03040000
8599:20200401:140302.063 __mem_malloc: skipped 0 asked 120 skip_min 4294967295 skip_max 0
8599:20200401:140302.063 [file:dbconfig.c,line:90] zbx_mem_malloc(): out of memory (requested 120 bytes)
8599:20200401:140302.063 [file:dbconfig.c,line:90] zbx_mem_malloc(): please increase CacheSize configuration parameter

The error message indicates that oom appears when zabbix allocates memory. By default, the memory of zabbix server is sufficient. Fortunately, you are prompted to add CacheSize. After checking the data, you can find that this value can be increased in the configuration file. By default, it is 8M, increased to 1024M. Finally, the third time you start zabbix server, it can start normally.

[root@zabbix zabbix]# grep CacheSize /etc/zabbix/zabbix_server.conf 
### Option: VMwareCacheSize
# VMwareCacheSize=8M
### Option: CacheSize
CacheSize=1024M
### Option: HistoryCacheSize
# HistoryCacheSize=16M
### Option: HistoryIndexCacheSize
# HistoryIndexCacheSize=4M
### Option: TrendCacheSize
# TrendCacheSize=4M
### Option: ValueCacheSize
# ValueCacheSize=8M