1. What is Monit?
Monit is a utility for managing and monitoring processes, programs, files, directories and file systems on a Unix like system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. E.g. Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources. You can use Monit to monitor files, directories and file systems for changes, such as timestamps changes, checksum changes or size changes.
Monit is controlled via an easy to configure control file (/etc/monit/monitrc) based on a free-format, token-oriented syntax. Monit logs to syslog or to its own log file (/var/log/monit.log) and notifies you about error conditions via customisable alert messages.
2. Install
netsec@ubuntu:~$ sudo apt install monit Reading package lists... Done Building dependency tree Reading state information... Done The following package was automatically installed and is no longer required: grub-pc-bin Use 'sudo apt autoremove' to remove it. Suggested packages: exim4 | postfix | mail-transport-agent sysvinit-core The following NEW packages will be installed: monit 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 318 kB of archives. After this operation, 946 kB of additional disk space will be used. Get:1 http://us-central1.gce.archive.ubuntu.com/ubuntu bionic/universe amd64 monit amd64 1:5.25.1-1build1 [318 kB] Fetched 318 kB in 0s (9343 kB/s) debconf: delaying package configuration, since apt-utils is not installed Selecting previously unselected package monit. (Reading database ... 45688 files and directories currently installed.) Preparing to unpack .../monit_1%3a5.25.1-1build1_amd64.deb ... Unpacking monit (1:5.25.1-1build1) ... Setting up monit (1:5.25.1-1build1) ... Processing triggers for systemd (237-3ubuntu10.28) ... jon_netsec@ubuntu-min-nginx:~$
3. Service Control and Show Service Status
jon_netsec@ubuntu-min-nginx:~$ sudo monit -t Control file syntax OK jon_netsec@ubuntu-min-nginx:~$ sudo service monit restart
jon_netsec@ubuntu-min-nginx:~$ sudo service monit status
● monit.service - LSB: service and resource monitoring daemon
Loaded: loaded (/etc/init.d/monit; generated)
Active: active (running) since Thu 2019-09-05 18:20:09 UTC; 26s ago
Docs: man:systemd-sysv-generator(8)
Process: 27892 ExecStop=/etc/init.d/monit stop (code=exited, status=0/SUCCESS)
Process: 27896 ExecStart=/etc/init.d/monit start (code=exited, status=0/SUCCESS)
Tasks: 1 (limit: 690)
CGroup: /system.slice/monit.service
└─27911 /usr/bin/monit -c /etc/monit/monitrc
Sep 05 18:20:09 ubuntu-min-nginx systemd[1]: Stopped LSB: service and resource monitoring daemon.
Sep 05 18:20:09 ubuntu-min-nginx systemd[1]: Starting LSB: service and resource monitoring daemon...
Sep 05 18:20:09 ubuntu-min-nginx monit[27896]: * Starting daemon monitor monit
Sep 05 18:20:09 ubuntu-min-nginx monit[27896]: ...done.
Sep 05 18:20:09 ubuntu-min-nginx systemd[1]: Started LSB: service and resource monitoring daemon.
jon_netsec@ubuntu-min-nginx:~$
4. Monitor System CPU, Memory, Load
Configure monit to monitor system load, cpu usage and memory usage. Once gauge exceeded, monit will trigger a reboot using command ‘/sbin/shutdown -r now’.
netsec@ubuntu:~$ sudo vi /etc/monit/monitrc
check system $HOST
if loadavg (1min) > 1 then alert
if loadavg (5min) > 1.05 then exec "/sbin/shutdown -r now"
if cpu usage > 50% for 5 cycles then alert
if cpu usage > 75% for 3 cycles then exec "/sbin/shutdown -r now"
if memory usage > 85% then alert
# if memory usage > 95% then reboot
if swap usage > 25% then alert
Here are some logs when system resource threshold reached , and monit service was triggered:
netsec@ubuntu:~$ cat /var/log/monit.log [EDT Sep 4 15:46:17] info : Starting Monit 5.25.1 daemon [EDT Sep 4 15:46:17] info : 'ubuntu18niginxmariadb2' Monit 5.25.1 started [EDT Sep 4 20:02:19] warning : 'ubuntu18niginxmariadb2' cpu usage of 98.7% matches resource limit [cpu usage > 75.0%] [EDT Sep 4 20:02:19] warning : 'ubuntu18niginxmariadb2' cpu usage of 98.7% matches resource limit [cpu usage > 50.0%] [EDT Sep 4 20:02:19] error : 'ubuntu18niginxmariadb2' loadavg(5min) of 1.6 matches resource limit [loadavg(5min) > 1.0] [EDT Sep 4 20:02:19] info : 'ubuntu18niginxmariadb2' exec: '/sbin/shutdown -r now' [EDT Sep 4 20:02:19] error : 'ubuntu18niginxmariadb2' loadavg(1min) of 4.1 matches resource limit [loadavg(1min) > 1.0] [EDT Sep 4 20:02:24] info : Monit daemon with pid [1121] stopped [EDT Sep 4 20:02:24] info : 'ubuntu18niginxmariadb2' Monit 5.25.1 stopped [EDT Sep 4 20:02:48] info : Starting Monit 5.25.1 daemon [EDT Sep 4 20:02:48] info : 'ubuntu18niginxmariadb2' Monit 5.25.1 started [EDT Sep 5 01:32:50] error : 'ubuntu18niginxmariadb2' loadavg(1min) of 1.0 matches resource limit [loadavg(1min) > 1.0] [EDT Sep 5 01:34:51] warning : 'ubuntu18niginxmariadb2' cpu usage of 87.7% matches resource limit [cpu usage > 75.0%] [EDT Sep 5 01:34:51] warning : 'ubuntu18niginxmariadb2' cpu usage of 87.7% matches resource limit [cpu usage > 50.0%] [EDT Sep 5 01:34:51] error : 'ubuntu18niginxmariadb2' loadavg(5min) of 1.8 matches resource limit [loadavg(5min) > 1.0] [EDT Sep 5 01:34:51] info : 'ubuntu18niginxmariadb2' exec: '/sbin/shutdown -r now' [EDT Sep 5 01:34:51] error : 'ubuntu18niginxmariadb2' loadavg(1min) of 4.0 matches resource limit [loadavg(1min) > 1.0] [EDT Sep 5 01:34:55] info : Monit daemon with pid [977] stopped [EDT Sep 5 01:34:55] info : 'ubuntu18niginxmariadb2' Monit 5.25.1 stopped [EDT Sep 5 01:35:18] info : Starting Monit 5.25.1 daemon [EDT Sep 5 01:35:18] info : 'ubuntu18niginxmariadb2' Monit 5.25.1 started [EDT Sep 5 02:23:19] error : 'ubuntu18niginxmariadb2' loadavg(1min) of 1.3 matches resource limit [loadavg(1min) > 1.0] [EDT Sep 5 02:25:19] warning : 'ubuntu18niginxmariadb2' cpu usage of 50.7% matches resource limit [cpu usage > 50.0%] [EDT Sep 5 02:25:19] error : 'ubuntu18niginxmariadb2' loadavg(1min) of 2.0 matches resource limit [loadavg(1min) > 1.0] [EDT Sep 5 02:27:19] warning : 'ubuntu18niginxmariadb2' cpu usage of 99.5% matches resource limit [cpu usage > 75.0%] [EDT Sep 5 02:27:19] warning : 'ubuntu18niginxmariadb2' cpu usage of 99.5% matches resource limit [cpu usage > 50.0%] [EDT Sep 5 02:27:19] error : 'ubuntu18niginxmariadb2' loadavg(5min) of 2.1 matches resource limit [loadavg(5min) > 1.0] [EDT Sep 5 02:27:19] info : 'ubuntu18niginxmariadb2' exec: '/sbin/shutdown -r now' [EDT Sep 5 02:27:19] error : 'ubuntu18niginxmariadb2' loadavg(1min) of 4.3 matches resource limit [loadavg(1min) > 1.0] [EDT Sep 5 02:27:22] info : Monit daemon with pid [1003] stopped [EDT Sep 5 02:27:22] info : 'ubuntu18niginxmariadb2' Monit 5.25.1 stopped [EDT Sep 5 02:27:47] info : Starting Monit 5.25.1 daemon [EDT Sep 5 02:27:47] info : 'ubuntu18niginxmariadb2' Monit 5.25.1 started [EDT Sep 5 02:41:47] error : 'ubuntu18niginxmariadb2' loadavg(1min) of 2.1 matches resource limit [loadavg(1min) > 1.0] [EDT Sep 5 02:43:48] warning : 'ubuntu18niginxmariadb2' cpu usage of 99.5% matches resource limit [cpu usage > 75.0%] [EDT Sep 5 02:43:48] warning : 'ubuntu18niginxmariadb2' cpu usage of 99.5% matches resource limit [cpu usage > 50.0%] [EDT Sep 5 02:43:48] error : 'ubuntu18niginxmariadb2' loadavg(5min) of 1.9 matches resource limit [loadavg(5min) > 1.0] [EDT Sep 5 02:43:48] info : 'ubuntu18niginxmariadb2' exec: '/sbin/shutdown -r now' [EDT Sep 5 02:43:48] error : 'ubuntu18niginxmariadb2' loadavg(1min) of 4.1 matches resource limit [loadavg(1min) > 1.0] [EDT Sep 5 02:43:52] info : Monit daemon with pid [1041] stopped [EDT Sep 5 02:43:52] info : 'ubuntu18niginxmariadb2' Monit 5.25.1 stopped [EDT Sep 5 02:44:16] info : Starting Monit 5.25.1 daemon [EDT Sep 5 02:44:16] info : 'ubuntu18niginxmariadb2' Monit 5.25.1 started
5. Enable Monit Web Service
netsec@ubuntu:~$ sudo nano /etc/monit/monitrc
jon_netsec@ubuntu-min-nginx:~$ sudo service monit restartjon_netsec@ubuntu-min-nginx:~$ netstat -na Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:2812 0.0.0.0:* LISTEN tcp 0 0 10.128.0.3:59048 169.254.169.254:80 ESTABLISHED tcp 0 0 10.128.0.3:80 52.53.175.20:61496 TIME_WAIT tcp 0 0 10.128.0.3:59058 169.254.169.254:80 ESTABLISHED tcp 0 0 10.128.0.3:80 3.83.192.114:46676 TIME_WAIT tcp 0 0 10.128.0.3:80 54.193.33.178:52530 TIME_WAIT tcp 0 0 10.128.0.3:80 3.90.252.110:24496 TIME_WAIT tcp 0 0 10.128.0.3:59044 169.254.169.254:80 CLOSE_WAIT tcp 0 384 10.128.0.3:22 173.194.90.33:40188 ESTABLISHED tcp 0 0 10.128.0.3:59052 169.254.169.254:80 ESTABLISHED tcp6 0 0 :::22 :::* LISTEN tcp6 0 0 :::2812 :::* LISTEN udp 0 0 127.0.0.1:323 0.0.0.0:* udp 0 0 127.0.0.53:53 0.0.0.0:* udp 0 0 10.128.0.3:68 0.0.0.0:* udp6 0 0 ::1:323 :::* raw6 0 0 :::58 :::*Browse to your Monit server’s http://yourserverip:2821 page
