One of my GCP VM is running at f1-Micro type which only has limited memory, 614MB. From syslog, I found it has a PHP invoked OOM killer messages. This usually means this server essentially ran out of memory and extra memory should add into this system, Based on some posts online, especially this 2010’s post :
“The Linux kernel likes to always allocate memory if applications asking for it. Per default it doesn’t really check if there is enough memory available. Given that behavior applications can allocate more memory as really is available. At some point it can definitely cause an out of memory situation. As result the OOM killer will be invoked and will kill that process:”
Determining Free Memory and Swap Activity
Before start optimizing your server, let’s review memory using on it. You can use the following command to display memory:
$ free -m
To see a list of your running processes sorted by memory use:
$ ps -eo pmem,pcpu,rss,vsize,args | sort -k 1 -r | less
Checking Logs Error
more /var/log/syslog
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315747] php-fpm7.2 invoked oom-killer: gfp_mask=0x15080c0(GFP_KERNEL_ACCOUNT|__GFP_ZERO), nodemask=(null), order=0, oom_score_adj=0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315750] php-fpm7.2 cpuset=/ mems_allowed=0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315758] CPU: 0 PID: 961 Comm: php-fpm7.2 Not tainted 4.15.0-1040-gcp #42-Ubuntu
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315759] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315760] Call Trace:
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315774] dump_stack+0x8e/0xcb
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315780] dump_header+0x71/0x285
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315782] oom_kill_process+0x220/0x440
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315784] out_of_memory+0x2d1/0x4f0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315786] __alloc_pages_slowpath+0xa53/0xe00
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315793] ? memcg_kmem_charge_memcg+0x7d/0xb0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315795] __alloc_pages_nodemask+0x29a/0x2c0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315800] alloc_pages_current+0x6a/0xe0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315805] pte_alloc_one+0x17/0x40
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315810] __pte_alloc+0x1e/0x120
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315811] copy_pte_range+0x535/0x7a0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315814] copy_page_range+0x3ed/0x6a0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315819] copy_process.part.35+0xe1e/0x1ba0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315822] _do_fork+0xdf/0x400
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315826] ? __do_page_fault+0x2aa/0x4c0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315828] SyS_clone+0x19/0x20
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315834] do_syscall_64+0x7b/0x150
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315840] entry_SYSCALL_64_after_hwframe+0x42/0xb7
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315844] RIP: 0033:0x7efcde571b1c
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315845] RSP: 002b:00007ffc4e3848c0 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315847] RAX: ffffffffffffffda RBX: 00007ffc4e3848c0 RCX: 00007efcde571b1c
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315847] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315848] RBP: 00007ffc4e384970 R08: 00007efce0ceac00 R09: 0000000000000000
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315849] R10: 00007efce0ceaed0 R11: 0000000000000246 R12: 0000000000000000
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315850] R13: 0000000000000020 R14: 0000000000000000 R15: ffffffffffffffff
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315852] Mem-Info:
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315856] active_anon:105955 inactive_anon:8694 isolated_anon:0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315856] active_file:69 inactive_file:144 isolated_file:0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315856] unevictable:0 dirty:0 writeback:0 unstable:0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315856] slab_reclaimable:4470 slab_unreclaimable:7737
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315856] mapped:8482 shmem:8871 pagetables:2067 bounce:0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315856] free:7589 free_pcp:94 free_cma:0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315860] Node 0 active_anon:423820kB inactive_anon:34776kB active_file:276kB inactive_file:576kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:33928kB dirty:0kB writeback:0kB shmem:35484kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315860] Node 0 DMA free:2948kB min:788kB low:984kB high:1180kB active_anon:8952kB inactive_anon:1892kB active_file:4kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:32kB pagetables:156kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315865] lowmem_reserve[]: 0 540 540 540 540
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315867] Node 0 DMA32 free:27408kB min:27460kB low:34324kB high:41188kB active_anon:414868kB inactive_anon:32884kB active_file:276kB inactive_file:576kB unevictable:0kB writepending:0kB present:612340kB managed:579656kB mlocked:0kB kernel_stack:2492kB pagetables:8112kB bounce:0kB free_pcp:376kB local_pcp:376kB free_cma:0kB
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315870] lowmem_reserve[]: 0 0 0 0 0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315873] Node 0 DMA: 27*4kB (UME) 21*8kB (UME) 39*16kB (UME) 20*32kB (UME) 12*64kB (UME) 5*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2948kB
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315882] Node 0 DMA32: 278*4kB (UMEH) 421*8kB (UMEH) 307*16kB (UMEH) 147*32kB (UMEH) 40*64kB (UME) 16*128kB (UME) 2*256kB (U) 2*512kB (UE) 1*1024kB (M) 1*2048kB (M) 1*4096kB (U) = 27408kB
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315894] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315896] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315896] 9092 total pagecache pages
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315898] 0 pages in swap cache
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315899] Swap cache stats: add 0, delete 0, find 0/0
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315900] Free swap = 0kB
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315900] Total swap = 0kB
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315901] 157083 pages RAM
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315901] 0 pages HighMem/MovableOnly
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315902] 8192 pages reserved
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315902] 0 pages cma reserved
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315903] 0 pages hwpoisoned
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315903] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315908] [ 396] 0 396 27480 230 200704 0 0 systemd-journal
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315909] [ 400] 0 400 24427 44 90112 0 0 lvmetad
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315911] [ 412] 0 412 10926 397 118784 0 -1000 systemd-udevd
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315913] [ 726] 100 726 20011 182 176128 0 0 systemd-network
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315914] [ 746] 101 746 17688 175 167936 0 0 systemd-resolve
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315916] [ 870] 103 870 12525 182 135168 0 -900 dbus-daemon
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315918] [ 885] 0 885 157192 186 147456 0 0 lxcfs
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315919] [ 918] 0 918 17663 200 176128 0 0 systemd-logind
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315921] [ 919] 0 919 72021 234 192512 0 0 accounts-daemon
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315922] [ 920] 0 920 163116 3151 233472 0 -900 snapd
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315924] [ 924] 0 924 7083 51 106496 0 0 atd
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315926] [ 925] 102 925 66817 322 167936 0 0 rsyslogd
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315927] [ 949] 111 949 25333 85 98304 0 0 chronyd
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315929] [ 961] 0 961 115363 1861 602112 0 0 php-fpm7.2
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315930] [ 972] 0 972 42721 1955 237568 0 0 networkd-dispat
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315932] [ 977] 0 977 7937 75 98304 0 0 cron
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315933] [ 987] 0 987 1157 16 57344 0 0 sshguard-journa
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315935] [ 990] 0 990 28251 140 233472 0 0 journalctl
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315936] [ 991] 0 991 4091 99 65536 0 0 sshguard
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315937] [ 1003] 0 1003 1157 16 57344 0 0 sshg-fw
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315939] [ 1006] 0 1006 35435 386 233472 0 0 nginx
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315940] [ 1007] 33 1007 36181 680 241664 0 0 nginx
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315942] [ 1137] 33 1137 153381 27842 835584 0 0 php-fpm7.2
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315943] [ 1169] 0 1169 72220 219 212992 0 0 polkitd
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315945] [ 1299] 0 1299 46931 1985 270336 0 0 unattended-upgr
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315946] [ 1362] 0 1362 4103 37 73728 0 0 agetty
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315947] [ 1405] 112 1405 372876 45749 753664 0 0 mysqld
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315949] [ 1455] 0 1455 3722 34 73728 0 0 agetty
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315950] [ 1456] 0 1456 18852 3644 196608 0 0 google_network_
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315952] [ 1469] 0 1469 18916 3687 192512 0 -999 google_accounts
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315953] [ 1472] 0 1472 18866 3606 192512 0 0 google_clock_sk
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315955] [ 1482] 0 1482 18074 189 180224 0 -1000 sshd
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315956] [ 2375] 33 2375 146970 21314 782336 0 0 php-fpm7.2
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315957] [ 9304] 0 9304 10048 157 110592 0 0 monit
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315959] [20018] 33 20018 138997 13458 720896 0 0 php-fpm7.2
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315960] [22124] 0 22124 115363 1432 557056 0 0 php-fpm7.2
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.315961] Out of memory: Kill process 1405 (mysqld) score 308 or sacrifice child
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.324223] Killed process 1405 (mysqld) total-vm:1491504kB, anon-rss:182996kB, file-rss:0kB, shmem-rss:0kB
Sep 1 10:46:22 ubuntu18niginxmariadb2 kernel: [74395.389736] oom_reaper: reaped process 1405 (mysqld), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Sep 1 10:46:25 ubuntu18niginxmariadb2 systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
Sep 1 10:46:25 ubuntu18niginxmariadb2 systemd[1]: mariadb.service: Failed with result 'signal'.
Sep 1 10:46:28 ubuntu18niginxmariadb2 systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
Sep 1 10:46:28 ubuntu18niginxmariadb2 systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
Sep 1 10:46:28 ubuntu18niginxmariadb2 systemd[1]: Stopped MariaDB 10.4.7 database server.
Sep 1 10:46:28 ubuntu18niginxmariadb2 systemd[1]: Starting MariaDB 10.4.7 database server...
Sep 1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] /usr/sbin/mysqld (mysqld 10.4.7-MariaDB-1:10.4.7+maria~bionic-log) starting as process 22301 ...
Sep 1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Using Linux native AIO
Sep 1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Sep 1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Uses event mutexes
Sep 1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
Sep 1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Number of pools: 1
Sep 1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Using SSE2 crc32 instructions
Sep 1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
Sep 1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
Sep 1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: Completed initialization of buffer pool
Sep 1 10:46:29 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:29 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
Sep 1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=589758381
Sep 1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Last binlog file '/var/log/mysql/mariadb-bin.000014', position 22151790
Sep 1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
Sep 1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
Sep 1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Creating shared tablespace for temporary tables
Sep 1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
Sep 1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
Sep 1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Waiting for purge to start
Sep 1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: 10.4.7 started; log sequence number 589758390; transaction id 501113
Sep 1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] Plugin 'FEEDBACK' is disabled.
Sep 1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Sep 1 10:46:30 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:30 0 [Note] Recovering after a crash using /var/log/mysql/mariadb-bin
Sep 1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:31 0 [Note] Starting crash recovery...
Sep 1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:31 0 [Note] Crash recovery finished.
Sep 1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:31 0 [Note] Server socket created on IP: '127.0.0.1'.
Sep 1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:31 0 [Note] Reading of all Master_info entries succeeded
Sep 1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:31 0 [Note] Added new Master_info '' to hash table
Sep 1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:31 0 [Note] /usr/sbin/mysqld: ready for connections.
Sep 1 10:46:31 ubuntu18niginxmariadb2 mysqld[22301]: Version: '10.4.7-MariaDB-1:10.4.7+maria~bionic-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 mariadb.org binary distribution
Sep 1 10:46:31 ubuntu18niginxmariadb2 /etc/mysql/debian-start[22342]: Upgrading MySQL tables if necessary.
Sep 1 10:46:31 ubuntu18niginxmariadb2 systemd[1]: Started MariaDB 10.4.7 database server.
Sep 1 10:46:32 ubuntu18niginxmariadb2 /etc/mysql/debian-start[22345]: Looking for 'mysql' as: /usr/bin/mysql
Sep 1 10:46:32 ubuntu18niginxmariadb2 /etc/mysql/debian-start[22345]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Sep 1 10:46:32 ubuntu18niginxmariadb2 /etc/mysql/debian-start[22345]: This installation of MariaDB is already upgraded to 10.4.7-MariaDB, use --force if you still need to run mysql_upgrade
Sep 1 10:46:32 ubuntu18niginxmariadb2 /etc/mysql/debian-start[22354]: Checking for insecure root accounts.
Sep 1 10:46:32 ubuntu18niginxmariadb2 /etc/mysql/debian-start[22358]: Triggering myisam-recover for all MyISAM tables and aria-recover for all Aria tables
Sep 1 10:46:32 ubuntu18niginxmariadb2 mysqld[22301]: 2019-09-01 10:46:32 0 [Note] InnoDB: Buffer pool(s) load completed at 190901 10:46:32
Bascically, php-fpm7.2 invoked OOM killer to kill mysql process on this VM. MariaDB was able to be re-launched successfully this time. But many times, whole system become unaccessed even ssh session.
GCP Monitoring shows CPU, Network Traffic and Disk IO increased during OOM killer killing mysql process.
There are a couple of ways to do it. One dirty way I tried is to use monit monitoring system resources or those critical processes and reboot machine. Another good way is to follow the post’s recommendation, to change the kernel behavior which stop overcommitting the memory for application requests such as php-fpm7.2. There are two values need to be included into the /etc/sysctl.conf , which will be automatically applied on start-up:
vm.overcommit_memory = 2 vm.overcommit_ratio = 90
This overcommit_memory switch knows 3 different settings:
- 0: The Linux kernel is free to overcommit memory (this is the default), a heuristic algorithm is applied to figure out if enough memory is available.
- 1: The Linux kernel will always overcommit memory, and never check if enough memory is available. This increases the risk of out-of-memory situations, but also improves memory-intensive workloads.
- 2: The Linux kernel will not overcommit memory, and only allocate as much memory as defined in overcommit_ratio.
Overcommit_ratio, this setting is only used when overcommit_memory = 2, and defines how many percent of the physical RAM are used. Swap space goes on top of that. The default is “50”, or 50%. I have tried different values from 75, 80, 90 for my vm. Eventually I found 90 is best value for my VM to get MariaDB started.
sysctl vm.overcommit_memory=2 : This will make the value active in this session without requiring a restart.
You can also configure your server to reboot when an OOM-Killer event occurs with following values in /etc/sysctl.conf based on this post:
vm.panic_on_oom=1
kernel.panic=5
or from command line, type:
sysctl vm.panic_on_oom=1
sysctl kernel.panic=5
References: