In this post we will review the most important Linux settings to adjust for performance tuning and optimization of a MySQL database server. We’ll note how some of the Linux parameter settings used OS tuning may vary according to different system types: physical, virtual or cloud. Other posts have addressed MySQL parameters, like Alexander’s blog MySQL 5.7 Performance Tuning Immediately After Installation. That post remains highly relevant for the latest versions of MySQL, 5.7 and 8.0. Here we will focus more on the Linux operating system parameters that can affect database performance.
Server and Operating System
Here are some Linux parameters that you should check and consider modifying if you need to improve database performance.
The value represents the tendency of the kernel to swap out memory pages. On a database server with ample amounts of RAM, we should keep this value as low as possible. The extra I/O can slow down or even render the service unresponsive. A value of 0
disables swapping completely while 1
causes the kernel to perform the minimum amount of swapping. In most cases the latter setting should be OK:
|
# Set the swappiness value as root
echo 1 > /proc/sys/vm/swappiness
# Alternatively, using sysctl
sysctl –w vm.swappiness=1
# Verify the change
cat /proc/sys/vm/swappiness
1
# Alternatively, using sysctl
sysctl vm.swappiness
vm.swappiness = 1
|
The change should be also persisted in /etc/sysctl.conf
:
XFS
XFS is a high-performance, journaling file system designed for high scalability. It provides near native I/O performance even when the file system spans multiple storage devices. XFS has features that make it suitable for very large file systems, supporting files up to 8EiB in size. Fast recovery, fast transactions, delayed allocation for reduced fragmentation and near raw I/O performance with DIRECT I/O.
The default options for mkfs.xfs
are good for optimal speed, so the simple command:
|
# Use default mkfs options
mkfs.xfs /dev/target_volume
|
will provide best performance while ensuring data safety. Regarding mount options, the defaults should fit most cases. On some filesystems you can see a performance increase by adding the noatime
mount option to the /etc/fstab
. For XFS filesystems the default atime
behaviour is relatime
, which has almost no overhead compared to noatime
and still maintains sane atime
values. If you create an XFS file system on a LUN that has a battery backed, non-volatile cache, you can further increase the performance of the filesystem by disabling the write barrier with the mount option nobarrier
. This helps you to avoid flushing data more often than necessary. If a BBU (backup battery unit) is not present, however, or you are unsure about it, leave barriers on, otherwise you may jeopardize data consistency. With this options on, an /etc/fstab
file should look like the one below:
|
/dev/sda2 /datastore xfs defaults,nobarrier
/dev/sdb2 /binlog xfs defaults,nobarrier
|
ext4
ext4 has been developed as the successor to ext3 with added performance improvements. It is a solid option that will fit most workloads. We should note here that it supports files up to 16TB in size, a smaller limit than xfs. This is something you should consider if extreme table space size/growth is a requirement. Regarding mount options, the same considerations apply. We recommend the defaults for a robust filesystem without risks to data consistency. However, if an enterprise storage controller with a BBU cache is present, the following mount options will provide the best performance:
|
/dev/sda2 /datastore ext4 noatime,data=writeback,barrier=0,nobh,errors=remount–ro
/dev/sdb2 /binlog ext4 noatime,data=writeback,barrier=0,nobh,errors=remount–ro
|
Note: The data=writeback
option results in only metadata being journaled, not actual file data. This has the risk of corrupting recently modified files in the event of a sudden power loss, a risk which is minimised with a presence of a BBU enabled controller. nobh
only works with the data=writeback
option enabled.
ZFS
ZFS is a filesystem and LVM combined enterprise storage solution with extended protection vs data corruption. There are certainly cases where the rich feature set of ZFS makes it an essential option to consider, most notably when advance volume management is a requirement. ZFS tuning for MySQL can be a complex topic and falls outside the scope of this blog. For further reference, there is a dedicated blog post on the subject by Yves Trudeau:
Most modern Linux distributions come with noop
or deadline
I/O schedulers by default, both providing better performance than the cfq
and anticipatory
ones. However it is always a good practice to check the scheduler for each device and if the value shown is different than noop
or deadline
the policy can change without rebooting the server:
|
# View the I/O scheduler setting. The value in square brackets shows the running scheduler
cat /sys/block/sdb/queue/scheduler
noop deadline [cfq]
# Change the setting
sudo echo noop > /sys/block/sdb/queue/scheduler
|
To make the change persistent, you must modify the GRUB configuration file:
|
# Change the line:
GRUB_CMDLINE_LINUX_DEFAULT=“quiet splash”
# to:
GRUB_CMDLINE_LINUX_DEFAULT=“quiet splash elevator=noop”
|
AWS Note: There are cases where the I/O scheduler has a value of none
, most notably in AWS VM instance types where EBS volumes are exposed as NVMe block devices. This is because the setting has no use in modern PCIe/NVMe devices. The reason is that they have a very large internal queue and they bypass the IO scheduler altogether. The setting in this case is none
and it is the optimal in such disks.
Ideally different disk volumes should be used for the OS installation, binlog, data and the redo log, if this is possible. The separation of OS and data partitions, not just logically but physically, will improve database performance. The RAID
level can also have an impact: RAID-5
should be avoided as the checksum needed to ensure integrity is costly. The best performance without making compromises to redundancy is achieved by the use of an advanced controller with a battery-backed cache unit and preferably RAID-10
volumes spanned across multiple disks.
AWS Note: For further information about EBS volumes and AWS storage optimisation, Amazon has documentation at the following links:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/storage-optimized-instances.html
Database settings
Non-uniform memory access (NUMA) is a memory design where an SMP’s system processor can access its own local memory faster than non-local memory (the one assigned local to other CPUs). This may result in suboptimal database performance and potentially swapping. When the buffer pool memory allocation is larger than size of the RAM available local to the node, and the default memory allocation policy is selected, swapping occurs. A NUMA enabled server will report different node distances between CPU nodes. A uniformed one will report a single distance:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
|
# NUMA system
numactl —hardware
available: 4 nodes (0–3)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 65525 MB
node 0 free: 296 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 65536 MB
node 1 free: 9538 MB
node 2 cpus: 16 17 18 19 20 21 22 23
node 2 size: 65536 MB
node 2 free: 12701 MB
node 3 cpus: 24 25 26 27 28 29 30 31
node 3 size: 65535 MB
node 3 free: 7166 MB
node distances:
node 0 1 2 3
0: 10 20 20 20
1: 20 10 20 20
2: 20 20 10 20
3: 20 20 20 10
# Uniformed system
numactl —hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 64509 MB
node 0 free: 4870 MB
node distances:
node 0
0: 10
|
In the case of a NUMA system, where numactl
shows different distances across nodes, the MySQL variable innodb_numa_interleave should be enabled to ensure memory interleaving. Percona Server provides improved NUMA support by introducing the flush_caches variable. When enabled, it will help with allocation fairness across nodes. To determine whether or not allocation is equal across nodes, you can examine numa_maps
for the mysqld process with this script:
|
# The perl script numa_maps.pl will report memory allocation per CPU node:
# 3595 is the pid of the mysqld process
perl numa_maps.pl < /proc/3595/numa_maps
N0 : 16010293 ( 61.07 GB)
N1 : 10465257 ( 39.92 GB)
N2 : 13036896 ( 49.73 GB)
N3 : 14508505 ( 55.35 GB)
active : 438 ( 0.00 GB)
anon : 54018275 (206.06 GB)
dirty : 54018275 (206.06 GB)
kernelpagesize_kB: 4680 ( 0.02 GB)
mapmax : 787 ( 0.00 GB)
mapped : 2731 ( 0.01 GB)
|
Conclusion
In this blog post we examined a few important OS related settings and explained how they can be tuned for better database performance.
While you are here …
You might also find value in this recorded webinar Troubleshooting Best Practices: Monitoring the Production Database Without Killing Performance
Related
Spyros Voultepsis
Spyros earned his stripes after more than 20 years’ experience in the IT and Telco sectors. He has direct in-depth experience of distributed systems, databases, virtualization, and application delivery. He joined Percona in 2017 as a Senior MySQL DBA in the Managed Services team.