Tips for quick optimization of Linux servers for high loads
1. Netfilter connection tracking
Simply put, this is a huge hash table that stores information about the state of all connections in the context of the device (server). This is necessary for all sorts of filters, firewalls, etc.
In certain cases, when the server is protected by another device in front of it, and requires additional resources this tracking can be disabled.
sudo modprobe -r nf_conntrack
echo “blacklist nf_conntrack” | sudo tee /etc/modprobe.d/nf_conntrack.conf
Why increase net.netfilter.nf_conntrack_max?
If, nevertheless, disabling tracking is not an option, but at the same time the server processes a large number of simultaneous connections, then a small value of this setting may make it impossible to create new connections, which, as you understand, is not ok.
Do not forget that in addition to increasing this parameter, we also need to increase the hash table itself where connection data is stored:
echo num > /sys/module/nf_conntrack/parameters/hashsize
2. File descriptors
Linux allocates a certain number of file handlers and the system will not let you open more files than the number of these handlers.
This setting fs.file-max sets the maximum number of them. But don’t forget about per-process settings with ulimit .
Here it is important not to forget that everything in Linux is a file and sockets as well, and therefore this setting also affects them.
3. Reusing a connection in the TIME_WAIT state
When a connection is closing, there is a certain amount of time before it closes to wait for lagging packages. At this time, the connection is in the TIME_WAIT state. And if the net.ipv4.tcp_tw_reuse option is not enabled, then with a large number of connections, due to these timeouts, there is a big chance to exhaust all available connections.
But this should not be used for NATs, proxies, or load balancers, as routing can be broken.
4. Limit sockets in the TIME_WAIT state
The net.ipv4.tcp_max_tw_buckets parameter specifies how many sockets can be in this state. If the limit is exceeded, the system starts actively closing sockets in this state and blocking the possibility of opening new ones.
You understand that with a large number of connections, this parameter should not be small.
5. Ports for outgoing connections.
As you may know, Linux limits the number of ports to 65,535. The net.ipv4.ip_local_port_range setting sets the range of ports that can be used for outgoing connections. Often 32768–60999 which gives us 28232 ports. If we need more, then this scale can be expanded.
But remember that 0–1024 is reserved for system services.
6. Setting up the system swap
Swap is used to flush part of the cache from memory to disk, which is useful when there is little memory, but extremely not useful when there is a lot of it, since the disk is much slower.
Setting vm.swappiness = 0 will force the cache to be swapped only when there is not enough memory.
7. Memory for system kernel
The system reserves a certain amount of memory that is used only by the kernel. A low indicator may cause the kernel to compete with user processes, as a result, the system will become very slow.
Configured with the vm.min_free_kbytes parameter
Remember that the kernel is actively involved in processing connections and passing traffic. With high network loads, it is better to increase it.
A good rule of thumb is to set it to the value resulting from the equation vm.min_free_kbytes = TotalMem * reserved_ratio. The reserved_ratio is typically around 0.01–0.03.
8. Package backlog
When a packages enters the system, it is queued for further processing. But if the packages arrive faster than they are being processed then the queue grows to the set limit, after which new packages are dropped.
The net.core.netdev_max_backlog parameter sets the limit for this queue. Increasing the parameter is necessary so that in the case of, for example, spike loads, the server can, albeit with a delay, process all packages without drops.
9. Backlog of SYN packages
Similar to the previous one, but it controls connection requests(SYN Packages).
And set by the net.core.somaxconn parameter
10. TCP stack memory limit
Global TCP stack memory limit
It is set by the net.ipv4.tcp_mem parameter and has 3 values: low, pressure and high. This parameter is set in cache pages. To see how much memory this is, use getconf PAGE_SIZE (often around 4kb).
To a low value, memory is allocated to everything on demand. With a value under load, the system tries to regulate memory and try to reduce consumption. More than a high value, the system can not allocate memory, and here comes our friend OOM Killer.
Buffer limit per socket
Set by the net.ipv4.tcp_wmem and net.ipv4.tcp_rmem options to write and read, respectively. It also has three values: minimum, standard and maximum.
Standard — the one with which the socket is created.
Let’s don’t forget that there is also a global maximum limit for the same thing. Parameters net.core.rmem_max and net.core.wmem_max
11. Monitoring connection metrics
By default, the system keeps connection metrics for the same host connection.
Provided that you have a lot of short connections, it makes sense to disable this behavior using the net.ipv4.tcp_no_metrics_save parameter
12. TCP congestion control algorithm.
This net.ipv4.tcp_congestion_control setting sets the algorithm that will be used to prevent the sender from sending more data than the network can handle.
Different algorithms may be better suited for specific workloads.
I did not test this topic myself, but the guys recommended it.
13. Configuring Selective Package Acknowledgment (SACK)
By default, the delivery confirmation comes from the last package. But if the network is not stable, this option can lead to extreme instability.
Often this happens with Wi-Fi and similar connections that can be subject to strong physical interference.
Controlled by the parameters net.ipv4.tcp_sack, net.ipv4.tcp_dsack, net.ipv4.tcp_fack
I myself have never used this topic, since there was no particular need.
