Recently, one of the servers on the zabbix monitoring platform alarmed: High swap space usage (less than 50% free)
Log on to the alarm server and check the virtual memory usage
From the above figure, it is found that the physical memory of this server is very sufficient, there are 11 G, and the virtual memory has used 273M, operating 50% of the total virtual memory.
So how to disarm the alarm?
solution:
Manually release virtual memory
##Before performing the following operations, check whether your swap partition is full. You first check how much space is left in your actual memory, and then check how much your swap space is used.First, make sure that the actual remaining memory is larger than your swap memory space in advance, and then perform the operation, otherwise it will crash!
First we stop the swap partition, check the swap partition before stopping
swapon -s will see where your swap partition is mounted:
For example, the swap partition of this server is mounted to /swapfile
[[email protected] ~]# swapon -s
Filename Type Size Used Priority
/swapfile file 524284 279808 -2
Then stop the swap partition
[[email protected] ~]# swapoff /swapfile
[[email protected] ~]# free -h
total used free shared buff/cache available
Mem: 31G 19G 365M 34M 11G 10G
Swap: 0B 0B 0B
It takes a while to stop, because it will release the virtual memory to the real memory.
Then start our swap partition
[[email protected] ~]# swapon -a
[[email protected] ~]# free -h
total used free shared buff/cache available
Mem: 31G 19G 366M 34M 11G 10G
Swap: 511M 0B 511MReference link:
How to release the memory and the swap partition is full under linux
Linux memory mechanism and manual release of swap and memory
Zabbix Server suddenly reports «high swap space usage (less than 50% free)». Briefly describe the current Zabbix Server environment. The operating system is CentOS 8. MySQL, Zabbix Server, PHP and other components are deployed on it.
[root@Zabbix ~]# more /etc/redhat-releaseCentOS Linux release 8.2.2004 (Core)[root@Zabbix ~]# uname -mx86_64
Log in to the Zabbix server and check the memory as follows:
[root@Zabbix ~]# free -mtotal used free shared buff/cache availableMem: 7809 4911 153 71 2744 2529Swap: 8099 4082 4017[root@Zabbix ~]# cat /proc/zoneinfo |grep -E "Node|pages free|nr_inactive_anon|nr_inactive_file|min|low|high"|grep -v "high:"Node 0, zone DMAnr_inactive_anon 318071nr_inactive_file 531494pages free 3836min 32low 40high 48Node 0, zone DMA32pages free 21648min 6091low 7613high 9135Node 0, zone Normalpages free 14002min 10771low 13463high 16155Node 0, zone Movablepages free 0min 0low 0high 0Node 0, zone Devicepages free 0min 0low 0high 0
The Linux server still has a lot of free memory, but it uses a lot of Swap space (nearly half of the Swap partition is used). Using vmstat observation (si/so indicator), you can also see that memory pages are frequently replaced to Swap partitions or written to memory from Swap partitions)
si: the size of writes from the swap area to memory per second;
so: the memory size written to the swap area per second;
[root@Zabbix ~]vmstat -an 1
I once suspected that the server did not disable NUMA, which caused Mysql to consume a large amount of swap partition space. A Bug in MySQL would cause a serious «swap insensity» problem in MySQL under the x86 system. In short, when you allocate most of the host memory to InnoDB, you will find that there is still a lot of memory in the operating system, but a lot of memory is swapped to the swap partition. There are also a large number of such cases on the Internet. However, during the analysis process, I analyzed and counted which processes occupy the swap size. I was surprised to find that PHP FPM processes consume a lot of swap space. As follows
for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done | sort -k 2 -n -r | less
Restart the PHP FPM service. It is found that the swap partition utilization of the system has dropped sharply. It seems that the PHP FPM process has consumed a lot of swap space
#systemctl restart php-fpm
At /etc/php-fpm In the d/www.conf configuration file, find the error log file of PHP: php_admin_value[error_log] = /var/log/php-fpm/www-error.log, but I only see «error log file re opened» in the error log. I am not familiar with PHP and do not continue to explore it in depth.
[root@Zabbix ~]# cd /var/log/php-fpm[root@Zabbix php-fpm]# lserror.log error.log-20201025 error.log-20201101 error.log-20201108 error.log-20201115[root@Zabbix php-fpm]#more error.log[15-Nov-2020 03:28:01] NOTICE: error log file re-opened[18-Nov-2020 17:06:21] NOTICE: Terminating ...[18-Nov-2020 17:06:21] NOTICE: exiting, bye-bye![18-Nov-2020 17:06:22] NOTICE: fpm is running, pid 205515[18-Nov-2020 17:06:22] NOTICE: ready to handle connections[18-Nov-2020 17:06:22] NOTICE: systemd monitor interval set to 10000ms
Conclusion:
In the process of diagnosing the problem, I was once fixed by the preconceived idea that «MySQL has a serious» Swap insensity «problem in x86 system». The analysis has been deviated from the correct direction until I counted and checked which processes consumed Swap space. Then I suddenly realized that the php FPM process was abnormal. Then I was anxious to verify the conjecture and restarted the php FPM service, resulting in some problems that could not be verified, For example, the number of php FMP processes. In addition, I am not familiar with php and cannot further analyze it. However, a simple conclusion can be drawn: php FPM is abnormal, causing it to consume a lot of Swap space. It is impossible to explore whether it is a memory leak or other reasons!
touch /home/swap Execute command: DD if=/dev/zero of=/home/swap bs=1024 count=5120000
Execute command: /sbin /mkswap /home /swap
Execute command: /sbin /swapon /home /swap
vi /etc/FSTAB adds the following sentences /home/swap swap swap defaults 0 0
Intelligent Recommendation
Godaddy free space WordPress usage record
Free 10G space for domain registration at Godaddy, 300G monthly traffic, it’s really affordable. I have always wanted to set up a WordPress blog, so I experimented in the free space, and reco…
More Recommendation
Solution notes — CPU usage is fixed to 50%
problem: Control CPU usage is fixed to 50%? Solution Idea: To control the usage rate curve of the CPU, you can make the CPU running BUSY and IDLE two different loops. The BUSY loop can perform an empt…
URL parameter is SQL has a space-free solution
Here you will display the page 404 In developing Web services, it is found that the server may not be identified when the URL is 400 Error: Bad Request Sytanx. When the analysis, the server contains s…
In this article you will learn how to Fix Zabbix Warning: Lack of Free Swap Space, and manually clear swap memory in Linux.
The problem has 3 solutions, create or increase swap space, increase server RAM, and finally we can clear the swap manually.
What is Swap Space?
Swap space in Linux is used when the amount of physical memory (RAM) is full. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. While swap space can help machines with a small amount of RAM, it should not be considered a replacement for more RAM. Swap space is located on hard drives, which have a slower access time than physical memory.
Manually Clear Swap Memory
Check swap space in Linux using free
Before manually clear swap memory lets check free swap space using Linux command free:
# free -m
Example output:
In this example our system used only 12Mb of swap space.
Disable swap
Now time to disable swap:
# swapoff -a
It can take some time ~60sec, please wait until finished.
Enable swap
Then re-enable swap:
# swapon -a
Now swap space should be cleared.
Lets check again:
As you see there is 0Mb used space.
That’s It!
In short:
How to Clear Swap Memory Linux
- Disable swap
# swapoff -a
- Enable swap
# swapon -a
Discuss article in ArsTech Forum
If you like what you are reading, please:
Buy me a coffee
Zabbix agent item ‘system.swap.size[,pfree]‘ mays show wrong value on virtualized Windows hosts (VMWare, Hyper-V etc). Behavior is documented:**
https://www.zabbix.com/documentation/4.4/manual/config/items/itemtypes/zabbix_agent
Note that this key might report incorrect swap space size/percentage on virtualized (VMware ESXi, VirtualBox) Windows platforms. In this case you may use the perf_counter[700(_Total)702] key to obtain correct swap space percentage.
So, maybe we should use performance counter in the template instead of system.swap.size item?
In my case I’ve just replace it to ‘perf_counter_en[«Paging file(_Total) % Usage»]‘ and change trigger expression as showing below:
E.g. I have VM shv-vexch03 where performance counter returns ~8% page file usage and system.swap.size[,*pfree*] which returns 66% usage (100 — system.swap.size[,pfree]):
