Modifying Icinga2 to Use Master/Client Model on CentOS 6 or CentOS 7

Icinga2 is a powerful monitoring system, and when used in a master-client model, it can replace the need for NRPE-based monitoring checks. The master-client model is more secure for various reasons. It also allows for more distributed monitoring by allowing checks to be modified on one server and propagate to others. It also gives one centralized location for changes.

For example: Let’s say you have 4 different load-balanced web servers. If you know that you are going to have a load spike coming up (maybe a sale or something for which you anticipate a lot of activity), you may want to increase the levels at which high CPU usage alerts the system admins. If you’re using NRPE checks, you will need to go onto each server and modify the /etc/nrpe.d/common_commands.cfg document on each of them. In the master-client model, you can simply modify their host definitions in the /etc/icinga2/repository.d/hosts/ folder, then reload Icinga. The checks will propagate to each server and will be picked up automatically. You can also add checks very easily by modifying the /etc/icinga2/repository.d/services/ files, and they will automatically propagate and become recognized, as long as the plug-ins exist on the client servers.

Prerequisites

• A server instance with Icinga2 and Icingaweb2 already installed and configured (instructions) named icinga2master.example.com.
• A fresh Vultr CentOS 6 or 7 server instance you wish to monitor named icinga2client.example.com.
• Sudo access.

Some of this guide will mirror the install instructions listed above, but this must be done on the client machine not the server machine.

Step 1: Update the system

Add the “ICINGA” repo to the client system to install the Icinga packages.

sudo cat > /etc/yum.repos.d/ICINGA-release.repo << 'EOF'
[icinga-stable-release]
name=ICINGA (stable release for epel)
baseurl=http://packages.icinga.org/epel/$releasever/release/
enabled=1
gpgcheck=1
gpgkey=http://packages.icinga.org/icinga.key
EOF

Step 2: Install required packages

Install the required packages.

# CentOS 6
sudo yum -y install icinga2 mysql-server mysql icinga2-ido-mysql nagios-plugins-disk nagios-plugins-load nagios-plugins-procs nagios-plugins-ssh nagios-plugins-users nagios-plugins-swap nagios-plugins-ping icingacli

# CentOS 7
sudo yum -y install icinga2 mariadb-server mysql icinga2-ido-mysql nagios-plugins-disk nagios-plugins-load nagios-plugins-procs nagios-plugins-ssh nagios-plugins-users nagios-plugins-swap nagios-plugins-ping icingacli

CentOS 7 uses MariaDB instead of MySQL, so the packages necessary are slightly different. Start the services, and set them to start on boot.

sudo chkconfig icinga2 on
sudo service icinga2 start

Set the MySQL service to start on boot and start it now so that we can secure it.

# CentOS 6
sudo chkconfig mysqld on
sudo service mysqld start

#CentOS 7
sudo systemctl enable mariadb.service
sudo systemctl start mariadb.service

Secure the MySQL/MariaDB installation. Both CentOS 6 and 7 use the same command.

sudo mysql_secure_installation

During the process, answer the questions as shown:

Enter current password for root (enter for none): Enter
Set root password? [Y/n]: Y
New password: $newRootPassword
Re-enter new password: $newRootPassword
Remove anonymous users? [Y/n]: Y
Disallow root login remotely? [Y/n]: Y
Remove test database and access to it? [Y/n]: Y
Reload privilege tables now? [Y/n]: Y

Make note of $newRootPassword as you will need it several times, and it’s a pain to reset.

Step 3: Set up the database

You need to create the icinga username, and import the schema into the database. This is done the same way on both CentOS 6 and 7.

sudo mysql -u root -p
>CREATE DATABASE icinga;
>GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW, INDEX, EXECUTE ON icinga.* TO 'icinga'@'localhost' IDENTIFIED BY '$someIcingaPassword'; 
>exit

Next, import the schema into the database you have just created.

sudo mysql -u root -p icinga < /usr/share/icinga2-ido-mysql/schema/mysql.sql

You will need to enter the $newRootPassword MySQL/MariaDB password you used during the mysql_secure_installation command above, not the $someIcingaPassword you have just created – that one is used later.

Enable the ido (database) module.

sudo icinga2 feature enable ido-mysql

You’ll need to edit the /etc/icinga2/features-enabled/ido-mysql.conf file to add the database information. You can do so with the following command.

sudo vi /etc/icinga2/features-enabled/ido-mysql.conf

In that file, find the lines that are commented out (start with //).

//user = "icinga"
//password = "icinga"
//host = "localhost"
//database = "icinga"

…and edit them using the user/password you have just created.

user = "icinga"
password = "$someIcingaPassword"
host = "localhost"
database = "icinga"

Make sure that you uncomment the lines by removing the initial two slashes.

Next, restart the service to pick up the changes.

sudo service icinga2 restart

You will need to add a connection between the client server and the Icinga2 master that has already been configured. You’ll need to do that using either Firewalld on CentOS 7, or IPTables in either CentOS 6 or 7.

# CentOS 6
sudo vi /etc/sysconfig/iptables
...
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5665 -j ACCEPT
...
#Icinga2 Master
-A OUTPUT -p tcp -m state --state NEW -m tcp --dport 5665 -j ACCEPT

sudo iptables restart

# CentOS 7    
sudo firewall-cmd --add-port=5665/tcp --permanent
sudo firewall-cmd --reload

Now, edit the hosts file to add the master as a local domain name.

sudo vi /etc/hosts
...
$masterIP  icinga2master icinga2master.example.com

Step 4: Initialize the master

This step needs to be done on the Icinga2 master. You’ll need to initialize it as a master, then generate a certificate signing request that your client will use to communicate.

sudo icinga2 node wizard

This will run the node wizard, which will ask several questions. The screen will look like the following, and the answers you will give are listed.

Welcome to the Icinga 2 Setup Wizard!

We'll guide you through all required configuration details.

Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: n
Starting the Master setup routine...
Please specify the common name (CN) [icinga2-master1.localdomain]: icinga2master.example.com
Checking for existing certificates for common name 'icinga2master.example.com'...
Certificates not yet generated. Running 'api setup' now.
information/cli: Generating new CA.
information/base: Writing private key to '/var/lib/icinga2/ca/ca.key'.
information/base: Writing X509 certificate to '/var/lib/icinga2/ca/ca.crt'.
information/cli: Generating new CSR in '/etc/icinga2/pki/icinga2master.example.com.csr'.
information/base: Writing private key to '/etc/icinga2/pki/icinga2master.example.com.key'.
information/base: Writing certificate signing request to '/etc/icinga2/pki/icinga2master.example.com.csr'.
information/cli: Signing CSR with CA and writing certificate to '/etc/icinga2/pki/icinga2master.example.com.crt'.
information/cli: Copying CA certificate to '/etc/icinga2/pki/ca.crt'.
Generating master configuration for Icinga 2.
information/cli: Adding new ApiUser 'root' in '/etc/icinga2/conf.d/api-users.conf'.
information/cli: Enabling the 'api' feature.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
information/cli: Created backup file '/etc/icinga2/zones.conf.orig'.
Please specify the API bind host/port (optional):
Bind Host []: <hit enter>
Bind Port []: <hit enter>
information/cli: Created backup file '/etc/icinga2/features-available/api.conf.orig'.
information/cli: Updating constants.conf.
information/cli: Created backup file '/etc/icinga2/constants.conf.orig'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
Done.

Now restart your Icinga 2 daemon to finish the installation!

Then, restart Icinga2 to pick up the changes.

sudo service icinga2 restart

Now that it’s a master, you can use it to generate the certificate signing request string.

sudo icinga2 pki ticket --cn icinga2client.example.com

You will get some string $pkiString. Copy that string, as you’ll need it on the client.

Step 5: Initialize the client

Run the node wizard to generate the certificates and connect to the master.

sudo icinga2 node wizard

It will come up with a similar page to the above. This page is shown below with answers included.

Welcome to the Icinga 2 Setup Wizard!

We'll guide you through all required configuration details.



Please specify if this is a satellite setup ('n' installs a master setup) [Y/n]: Y
Starting the Node setup routine...
Please specify the common name (CN) [icinga2client.example.com]:
Please specify the master endpoint(s) this node should connect to:
Master Common Name (CN from your master setup): icinga2master.example.com
Do you want to establish a connection to the master from this node? [Y/n]: Y
Please fill out the master connection information:
Master endpoint host (Your master's IP address or FQDN): icinga2master.example.com
Master endpoint port [5665]: 5665
Add more master endpoints? [y/N]: N
Please specify the master connection for CSR auto-signing (defaults to master endpoint host):
Host [icinga2master.example.com]: icinga2master.example.com
Port [5665]: 5665
information/cli: Created backup file '/etc/icinga2/pki/icinga2client.example.com.key.orig'.
information/cli: Created backup file '/etc/icinga2/pki/icinga2client.example.com.crt.orig'.
information/base: Writing private key to '/etc/icinga2/pki/icinga2client.example.com.key'.
information/base: Writing X509 certificate to '/etc/icinga2/pki/icinga2client.example.com.crt'.
information/cli: Fetching public certificate from master (icinga2master.example.com, 5665):

Certificate information:

 Subject:     CN = icinga2master.example.com
 Issuer:      CN = Icinga CA
 Valid From:  Jan 10 21:08:37 2017 GMT
 Valid Until: Jan  7 21:08:37 2032 GMT
 Fingerprint: FE 72 AB F3 18 A5 12 E0 0C 5D 94 8B 96 C4 57 3B 00 5C E0 04

Is this information correct? [y/N]: Y
information/cli: Received trusted master certificate.

Please specify the request ticket generated on your Icinga 2 master.
 (Hint: # icinga2 pki ticket --cn 'icinga2client.example.com'): $pkiString

information/cli: Requesting certificate with ticket '$pkiString'.

warning/cli: Backup file '/etc/icinga2/pki/icinga2client.example.com.crt.orig' already exists. Skipping backup.
information/cli: Writing signed certificate to file '/etc/icinga2/pki/icinga2client.example.com.crt'.
information/cli: Writing CA certificate to file '/etc/icinga2/pki/ca.crt'.
Please specify the API bind host/port (optional):
Bind Host []: <enter key>
Bind Port []: <enter key>
Accept config from master? [y/N]: Y
Accept commands from master? [y/N]: Y
information/cli: Disabling the Notification feature.
Disabling feature notification. Make sure to restart Icinga 2 for these changes to take effect.
information/cli: Enabling the Apilistener feature.
warning/cli: Feature 'api' already enabled.
information/cli: Created backup file '/etc/icinga2/features-available/api.conf.orig'.
information/cli: Generating local zones.conf.
information/cli: Dumping config items to file '/etc/icinga2/zones.conf'.
information/cli: Created backup file '/etc/icinga2/zones.conf.orig'.
information/cli: Updating constants.conf.
information/cli: Created backup file '/etc/icinga2/constants.conf.orig'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
information/cli: Updating constants file '/etc/icinga2/constants.conf'.
Done.

Now restart your Icinga 2 daemon to finish the installation!

Go ahead and restart Icinga2 to pick up the changes.

sudo icinga2 restart

This may already be done, but you need to ensure that the command and API features are enabled.

sudo icinga2 feature enable command
sudo icinga2 feature enable api

If they weren’t enabled, it will say “feature enabled”, so you’ll need to restart the Icinga2 daemon. But first, you need to change ownership of the files in the /etc/icinga2 directory, as some of them have the wrong permissions after running the node wizard. They should all have their owner and group set to icinga. Change the permissions by using the following command:

sudo chown -R icinga:icinga /etc/icinga2/

Now, you can restart the Icinga2 daemon.

sudo service icinga2 restart

Step 6: Configure the client

The client configuration requires editing a few different files.

• /etc/icinga2/zones.conf: Contains the endpoints (servers) and zones that contain them. This is how the servers are partitioned. Make sure the client names match the server names.
• /etc/icinga2/icinga2.conf: You are accepting commands and configs from the master, so you need to do a minor edit on this file to avoid duplicate command definitions.

First, we’re going to modify the zones.conf file to define the server and the zone it’s in, as well as creating a “global” zone for command definitions that are universal to ALL servers, instead of unique to each (such as disk checks).

sudo vi /etc/icinga2/zones.conf

object Zone clientZone { 
        endpoints = [ "icinga2master.example.com" ]
}

object Zone "global-templates" { # Add global templates zone
        global = true
}

object Endpoint NodeName {
        host = "$icinga2clientIP" 
}

object Zone ZoneName {
        endpoints = [ NodeName ]
        parent = clientZone
} 

It is important to know, there can only be 2 nodes per zone at the moment. You can make as many zones as you wish, but if there are more than 2 nodes per zone there is a chance that the Icinga2 service will not work properly. This is not a problem on the client, unless you already have 2 nodes in the zone clientZone, in which case you will need to make another zone name.

Now, make the modification in the /etc/icinga2/icinga2.conf file. You need to comment out the line include_recursive "conf.d".

sudo vi /etc/icinga2/icinga2.conf
...
//include_recursive "conf.d"
...

Verify config to ensure that everything is formatted correctly.

sudo icinga2 daemon -C

If you get permissions problems, it’s likely because the item is owned by root:root. Chown to icinga:icinga to resolve.

sudo chown -R icinga:icinga /etc/icinga2/

There are other issues with the configs, generally the information given from the output of the sudo icinga2 daemon -C command will be verbose enough to track them down. Addressing every potential issue that arises is outside the scope of this guide.

If the verification succeeds, go ahead and reload the service:

sudo service icinga2 restart

You’ll need to give the icinga user sudo permissions so that the commands can be run. These sudo permissions will be restricted to the specific plugin directory. This can just be put at the end of the file.

sudo vi /etc/sudoers
...
Defaults:icinga !requiretty
icinga          ALL=(ALL) NOPASSWD: /usr/lib64/nagios/plugins/

Now the client configuration is officially finished. You shouldn’t need to make any more modifications, any service checks will configured on the master and will propagate down to the client. You will need to install any additional plug-ins you might need, however. Plug-ins can also be written using python, perl, BASH, PHP, and other scripted languages.

Step 7: Add client to master config

You will need to get back onto the master to configure the client so that it can be connected. There are a few steps that will need to be taken:

1. Add host config to repository.d.
2. Add host to zone as endpoint.
3. Configure any checks for the client. This is slightly out of scope of this guide, I’ll provide only a basic explanation of the steps involved.

The host config will be entered in the /etc/icinga2/repository.d/hosts/icinga2client.example.com.conf file.

sudo vi /etc/icinga2/repository.d/hosts/icinga2client.example.com.conf
object Host "icinga2client.example.com" {
  import "satellite-host"
  address = "$icinga2clientIP"
  vars.os = "Linux"
}
sudo chown icinga:icinga /etc/icinga2/repository.d/hosts/icinga2client.example.com.conf

Now the host exists in the client config, but it needs to be put into a zone. This zone will be in the /etc/icinga2/repository.d/zones folder, with the same name as the zone name to make it easier to keep track.

sudo vi /etc/icinga2/repository.d/zones/clientZone.conf
object Zone "clientZone" {
        endpoints = [ "icinga2client.example.com" ]
        parent = "icinga2master.example.com"
}

Next, you need to add the global-templates zone.

sudo vi /etc/icinga2/repository.d/zones/global-templates.conf
object Zone "global-templates" {
        global = true
}

Finally, add the client endpoint so that the “clientZone” endpoints array is populated when it runs.

sudo vi /etc/icinga2/repository.d/endpoints/icinga2client.example.com.conf

#Change values to match the host you're adding
object Endpoint "icinga2client.example.com" {
        host = "$icinga2clientIP"
}

In order to make sure there are no permissions issues, go ahead and change the owner/group again.

sudo chown -R icinga:icinga /etc/icinga2

Verify configuration changes to make sure there are no more issues.

sudo icinga2 daemon -C

If verification succeeds, go ahead with the reload.

sudo service icinga2 restart

If you visit your Icinga2 website, you will see the server become accessible.

Step 8: Custom commands and checks

You may encounter errors about “command not found” or “check not defined”. This is where the global-templates zone comes into play. You will need to copy the configuration in /etc/icinga2/conf.d/ files and put them into /etc/icinga2/zones.d/global-templates/ files. For example, to create a “check_software” command, you would need to do do the following.

sudo mkdir /etc/icinga2/zones.d/global-templates
sudo vi /etc/icinga2/zones.d/global-templates/commands.conf
object CheckCommand "check_software" {
  import "plugin-check-command"
  command = [ "/usr/bin/sudo",  PluginDir + "/software_service_check.sh" ]
}

The software_service_check.sh file will need to exist in the /usr/lib64/nagios/plugins folder on the client.

Next, add the service check into the /etc/icinga2/repository.d/hosts/services.conf file.

sudo vi /etc/icinga2/repository.d/hosts/services/services.conf
apply Service "Check Software Service" {
  import "mail-service"
  check_command = "check_software" # This check command is the same name as the one you defined in global-templates/commands.conf

  assign where host.vars.client_endpoint # This will apply to every client server. If you need it to be more restrictive, you should look into custom variables
}

You’ll need to change the permissions again.

sudo chown -R icinga:icinga /etc/icinga2

Then, re-verify configuration.

sudo icinga2 daemon -C

Finally, if it succeeds, reload:

sudo service icinga2 restart

Congratulations! You are now using Icinga2 in a master-client model and can now avoid having to use the insecure NRPE checks!