Operating System CentOS

Icinga 2 Distributed Monitoring: Master and Agents on Debian 13 and Ubuntu 26.04

Convert Icinga 2 to the master/agent model with the node wizard, sign agent certificates and define checks on the master that run on the agent.

This guide turns a single Icinga 2 server into a distributed setup: a master that holds all configuration and one or more agents that run checks locally and report back over a TLS-secured connection. You will run the node wizard on both sides, sign the agent's certificate, and write host and service definitions on the master that execute on the agent. The commands are for Debian 13 and Ubuntu 26.04 LTS; they are the same on Debian 12 and Ubuntu 24.04 LTS.

What older tutorials call the "master/client model" is now called distributed monitoring, and "clients" are agents. The bottom-up mode that stored discovered objects in /etc/icinga2/repository.d/ and the icinga2 node update-config command were removed from Icinga 2 years ago. The supported approach is top-down: everything is configured on the master. CentOS 6 and CentOS 7 are both end of life (November 2020 and June 2024); see the next section for what that means for RHEL-family hosts.

Where current packages come from

Icinga publishes its own package repository at packages.icinga.com. Access depends on the distribution:

  • Debian, Ubuntu, Raspberry Pi OS, Fedora and openSUSE: the repositories are free to use. They carry builds for Debian 13 ("trixie"), Debian 12, Ubuntu 26.04 and Ubuntu 24.04.
  • RHEL, SLES and Amazon Linux: the installation documentation states that "a paid repository subscription is required for RHEL repositories". The repository file lives under packages.icinga.com/subscription/rhel/ and needs the username and password from your subscription. Icinga says the RHEL packages may also be used on binary-compatible distributions such as AlmaLinux, Rocky Linux and Oracle Linux.

If you do not have a subscription, run the master on Debian or Ubuntu. For RHEL-family machines you only want to monitor, either buy the subscription for the agent packages or check them from the master with SSH-based or SNMP checks.

Requirements

  • A master server, here icinga-master.example.com (203.0.113.10), and a server to monitor, here agent1.example.com (203.0.113.20). Both must resolve each other's names, through DNS or /etc/hosts.
  • Root access on both. The commands below are run as root; open a root shell with sudo -i.
  • TCP port 5665 on the master reachable from the agent. Open it in your firewall for the agent's address only.
  • Clocks in sync on both machines (certificate validation depends on it).

Keep versions aligned: the master should run the same or a newer Icinga 2 version than its satellites and agents. Upgrade the master first, then satellites, then agents.

Install Icinga 2 on both servers

On Debian, add the repository and install the daemon and the check plugins:

apt update
apt -y install apt-transport-https wget
wget -O icinga-archive-keyring.deb "https://packages.icinga.com/icinga-archive-keyring_latest+debian$( . /etc/os-release; echo "$VERSION_ID" ).deb"
apt install ./icinga-archive-keyring.deb
DIST=$(awk -F"[)(]+" '/VERSION=/ {print $2}' /etc/os-release)
echo "deb [signed-by=/usr/share/keyrings/icinga-archive-keyring.gpg] https://packages.icinga.com/debian icinga-${DIST} main" > /etc/apt/sources.list.d/${DIST}-icinga.list
apt update
apt install icinga2 monitoring-plugins

On Ubuntu, only the keyring file name and the repository path differ:

apt update
apt -y install apt-transport-https wget
wget -O icinga-archive-keyring.deb "https://packages.icinga.com/icinga-archive-keyring_latest+ubuntu$( . /etc/os-release; echo "$VERSION_ID" ).deb"
apt install ./icinga-archive-keyring.deb
. /etc/os-release
echo "deb [signed-by=/usr/share/keyrings/icinga-archive-keyring.gpg] https://packages.icinga.com/ubuntu icinga-${UBUNTU_CODENAME} main" > /etc/apt/sources.list.d/${UBUNTU_CODENAME}-icinga.list
apt update
apt install icinga2 monitoring-plugins

If you already have a working master with Icinga Web 2, skip the installation there and continue with the wizard.

Set up the master

Run the wizard on icinga-master.example.com:

icinga2 node wizard

Answer the prompts as follows:

  • Please specify if this is an agent/satellite setup: n (this makes it a master).
  • Common name: press Enter to accept the FQDN, icinga-master.example.com.
  • Master zone name: press Enter for master.
  • Additional global zones: press Enter for none.
  • API bind host / port: press Enter for the defaults (all addresses, port 5665).
  • Disable the inclusion of the conf.d directory: the default is yes. In a distributed setup, objects belong in /etc/icinga2/zones.d/. If your existing master still has hosts and services in /etc/icinga2/conf.d/, answer n for now and move them later, otherwise they stop being loaded.

The wizard enables the api feature, creates a certificate authority in /var/lib/icinga2/ca if there is none, issues the master's certificate and updates zones.conf. Restart the service:

systemctl restart icinga2

Back up /var/lib/icinga2/ca. If the CA key is lost, every node certificate has to be reissued.

Still on the master, create a ticket for the agent and print the master certificate's fingerprint. You need both in the next step:

icinga2 pki ticket --cn agent1.example.com
openssl x509 -noout -fingerprint -sha256 -in "/var/lib/icinga2/certs/$(hostname --fqdn).crt"

Set up the agent

Run the same wizard on agent1.example.com:

icinga2 node wizard
  • Agent/satellite setup: press Enter (yes).
  • Common name: press Enter for agent1.example.com. It must match the name used for the ticket.
  • Parent endpoint common name: icinga-master.example.com.
  • Establish a connection to the parent node from this node: press Enter (yes), then enter the master's address 203.0.113.10 and accept port 5665.
  • Add more master/satellite endpoints: n.
  • The wizard shows the parent's certificate. Compare the fingerprint with the one you printed on the master and confirm with y only if they match.
  • Request ticket: paste the ticket from the master.
  • API bind host / port: press Enter.
  • Accept config from parent node: y.
  • Accept commands from parent node: y.
  • Local zone name and parent zone name: press Enter (the agent's FQDN and master).
  • Global zones: press Enter.
  • Disable the inclusion of the conf.d directory: press Enter (yes). The agent gets its work from the master and needs no local objects.
systemctl restart icinga2

If you leave the ticket empty, the agent sends a signing request instead. Approve it on the master with icinga2 ca list followed by icinga2 ca sign and the request's fingerprint.

Register the agent on the master

The master must know the agent's endpoint and zone. Add them to /etc/icinga2/zones.conf on the master:

object Endpoint "agent1.example.com" {
  // no host attribute: the agent opens the connection
  log_duration = 0
}

object Zone "agent1.example.com" {
  endpoints = [ "agent1.example.com" ]
  parent = "master"
}

An endpoint only needs a host attribute on the side that actively connects. If the agent sits behind a firewall you can reverse the direction instead: set host = "203.0.113.20" here and answer "no" to the connection question in the agent wizard.

Now define what to monitor. Create the directory for the master zone and a host object in /etc/icinga2/zones.d/master/hosts.conf:

mkdir -p /etc/icinga2/zones.d/master
object Host "agent1.example.com" {
  check_command = "hostalive"
  address = "203.0.113.20"
  vars.agent_endpoint = name
}

Then add services in /etc/icinga2/zones.d/master/services.conf. The command_endpoint attribute tells the master to have the check executed by the agent instead of running it locally:

apply Service "disk" {
  check_command = "disk"
  command_endpoint = host.vars.agent_endpoint
  assign where host.vars.agent_endpoint
}

apply Service "load" {
  check_command = "load"
  command_endpoint = host.vars.agent_endpoint
  assign where host.vars.agent_endpoint
}

apply Service "agent-health" {
  check_command = "cluster-zone"
  display_name = "cluster-health-" + host.name
  vars.cluster_zone = host.name
  assign where host.vars.agent_endpoint
}

The agent-health service runs on the master and reports whether the agent's zone is connected, so a dead agent shows up as one clear problem. Each additional agent needs only the wizard run, an Endpoint/Zone pair in zones.conf and a Host object with vars.agent_endpoint; the apply rules pick it up automatically.

Validate and reload

Check the configuration on the master and reload:

icinga2 daemon -C
systemctl reload icinga2

Confirm that the agent is connected. On either node, the log shows the peer connecting, and port 5665 should be listening:

ss -tlnp | grep 5665
tail -n 50 /var/log/icinga2/icinga2.log

In Icinga Web 2 the new host appears with its three services after the first check interval. To change a threshold, set the corresponding custom variable on the master (for example vars.disk_wfree = "15%" on the host or service), validate and reload. Nothing has to be edited on the agent.

Adding satellites

A satellite is a node between the master and a group of agents, typically one per data centre or network segment. It is set up with the same wizard as an agent (parent: the master), gets its own zone with parent = "master", and agents behind it use the satellite as their parent and its zone as their parent zone. Configuration for a satellite zone goes into /etc/icinga2/zones.d/<zone name>/ on the master and is synced down, which is why the satellite must accept configuration. Templates and commands that every zone needs go into the global zone directory /etc/icinga2/zones.d/global-templates/. The distributed monitoring chapter of the Icinga 2 documentation describes the three-level scenario in full.

Troubleshooting

  • Agent never connects: test the port from the agent with openssl s_client -connect 203.0.113.10:5665. A timeout means a firewall; a certificate dump means the network path is fine.
  • Checks stay "pending" or return "not connected": the endpoint name in the master's zones.conf must be exactly the agent's common name, and accept_commands = true must be set in /etc/icinga2/features-enabled/api.conf on the agent.
  • "Check command does not exist" on the agent: the plugin is missing there. Install monitoring-plugins on the agent; custom CheckCommand definitions belong in the global-templates zone so they reach every node.
  • Certificate request not signed: run icinga2 ca list on the master. Unsigned requests are removed automatically after a week.

Frequently asked questions

Do I still need NRPE?

No. The agent runs the same plugins that NRPE would, but over Icinga's own mutually authenticated TLS connection on port 5665, and the check parameters are defined on the master. NRPE is only worth keeping for devices where you cannot install the Icinga 2 package.

Which side should open the connection?

Either works, and once connected the link is used in both directions. Letting agents connect to the master means only one inbound port on one server has to be opened, which suits agents behind NAT. Letting the master connect outwards suits a master in a protected network that agents cannot reach.

What happens to checks when the master is down?

With command_endpoint checks, the master schedules every check, so an agent does nothing while the master is unreachable. A satellite is different: it has its own synced configuration, keeps checking and replays the results when the connection returns.

Can the agent run a different Icinga 2 version than the master?

The documented rule is master version greater than or equal to satellite, greater than or equal to agent. Older agents may keep working but this is not guaranteed, so plan agent upgrades soon after upgrading the master.

Share story:

Continue reading other guides on How VPS.