Our infrastructure team member, Jamila, recently completed an upgrade of our entire monitoring infrastructure for Windows computers, which is a key part of how we keep our client systems running, up to date, and virus free. This upgrade was almost 7 years in the works! We’re so excited to finally be up to date, using a set up that involves Icinga2, an open source monitoring software, with specific Powershell scripts to make it work nicely with Windows. We wanted to share the details of our set up so that others can replicate or iterate on it for their own use.
Our new set up
Each Windows desktop runs Puppet agent, which we use to keep software packages up to date. In our new set up, we also run our own Icinga2 server installed with Icinga2 Director, a newly released configuration tool specifically for Icinga2. Icinga Director allows for more powerful and easeful configuration of Icinga2, with configuration templates for specific computers or groups of computers attached to a central host template.
Our old set up
We used to use NSClient installed by Puppet, which would run commands locally and then report them to Icinga1.
Why did we upgrade from Icinga1 to Icinga2?
There were a few reasons for this. The first is that Icinga1 is past its end of life, so it’s not getting the vibrant support and updates that Icinga2 is now. NSClient also isn’t really a secure option. It uses a passphrase to encrypt its communication but there is no verification that the computer sending information via NSClient is the one that it is supposed to be. Icinga2 has a much better user interface, which makes it easier for our helpdesk team to keep tabs on which systems are and aren’t working. Importantly, Icinga2 also has the ability to work well with scripts that we use to automatically fix problems such as telling the antivirus system to run a scan if it hasn't recently. Icinga2 can check the results of those scripts, whether they completed successfully, what their error codes were, etc. so that we know which problems are still open and which are resolved.
Prerequisites for this set up
We recommend the following prerequisites for this monitoring system:
- Puppet 6 set up and running with a central server and puppet agents installed on individual desktops
- Computers running Windows 10 with Powershell 5 installed
- C:\Temp folder exists on each computer and is writable by the Puppet agent
- Icinga2 with Icinga Director
- This should be set up with a host template for Windows Desktop, with the appropriate variables and services attached to it
Powershell script
With those prerequisites all set up, Jamila then wrote a Powershell script that, when run from the C:\Temp folder, linked each Windows desktop to Icinga2. We’ve pasted it in full below, with comments for what each line of code does.
If (Test-Path -Path C:\Temp\icingadirector_ranonce.txt) { Exit };
The script creates a txt file if it runs successfully through to the end. This line checks for the presence of that file and exits if it finds it. Effectively, if this script has already ran successfully once, it won't try to run again.
Install-PackageProvider -Name NuGet -Force;
Powershell uses package providers to install modules, but doesn't come with any by default, this adds the provider we will use to install the module.
Set-PSRepository -Name "PSGallery" -InstallationPolicy Trusted;
Adding the repository the module will be downloaded from.
Install-Module -Name icinga-powershell-framework;
Installing the icinga powershell module.
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned;
This is a slight security risk, it is set to enable running the following commands without any interaction.
Import-Module icinga-powershell-framework;
Opening up the icinga powershell framework just installed.
Use-Icinga;
Initiating the icinga powershell framework
Running the main command with all the options:
Start-IcingaAgentInstallWizard
start the installer
-SelfServiceAPIKey '%APIKEY%'
Icinga2 director for windows uses a self service api key that is unique for the host template you create
This key is in the Agent tab for the windows desktop template
It is the same for all the hosts we are using
-UseDirectorSelfService 1
Use that key in the previous option
-DirectorUrl '%ICINGA2DIRECTORURL%'
The full URL to the director instance
-OverrideDirectorVars 0
No, do not override variables that Icinga2 director sets
-AutoUseFQDN 1
Yes, use the FQDN as the name of this host when contacting director
-AutoUseHostname 0
No, do not use the hostname
-LowerCase 1
Yes, make the FQDN all lowercase
-UpperCase 0
No, do not make it uppercase
-AllowVersionChanges 1
Yes, allow this script to change the version of Icinga2 that is installed
-UpdateAgent 1
Yes, update the agent
-AgentVersion 'release'
Install the latest release version of the icinga2 agent
-PackageSource 'https://packages.icinga.com/windows/'
Install it from this url
-Endpoints <%= $winmonitoring::endpoints %>
Puppet can fill out the appropriate endpoints
The default for this is just the fqdn of the main Icinga2 node
If some windows desktops are behind a firewall and can't see the main Icinga2 node, but they can see a server running Icinga2 that can see the main Icinga2 node, this can be a comma delimited list of the nodes that the desktop should connect to, starting with the main Icinga2 node
-CAPort 5665
Port for the Certificate authority
-EmptyCA 1
Yes, we are starting from no certificate
-AcceptConnections 1
Yes, accept connections from Icinga2 main node
-AddFirewallRule 1
Yes, add firewall rules for the Icinga2 agent
-ConvertEndpointIPConfig 0
No, do not use the IP configuration of the windows desktop
-EndpointConnections <%= $winmonitoring::endpointconnections %> Puppet can fill out the appropriate endpoint connections
The default of this is the IP address or hostname of the main Icinga2 node
This list should be the IP addresses or hostnames of the Endpoints listed above
-ParentZone master
Icinga2 zones have to be set up correctly, the parent zone is master
-AddDirectorGlobal 1
Yes, add the directorglobal zone
-AddGlobalTemplates 1
Yes, add templates from that zone
-GlobalZones '%MAINICINGA2NODEFQDN%'
Add the zone that matches the main node, if it is not 'master'
-CAEndpoint '%ICINGA2CAHOSTNAME%'
Add the hostname (or IP address) for the Certificate authority
-Ticket ''
Our ticket is empty
-EmptyTicket 1
Yes, our ticket is empty
-ServiceUser 'NT Authority\System'
Choose which user to run the Icinga2 agent as
NT Authority\System gives the agent more permissions locally, which is useful, but can be a security risk
-InstallFrameworkPlugins 0
No, don't install the powershell framework plugins
-InstallFrameworkService 0
No, don't install the powershell framework service
-Reconfigure
If Icinga2 is already installed, reconfigure it with this information
-RunInstaller;
Install the agent
echo $null >> "C:\Temp\icingadirector_ranonce.txt";
Create a txt file now that it has run successfully through to the end
After running this script
Once that script is run, go to the command line on your Icinga2 server and run the following to see the certs that need to be signed:
icinga2 ca list
To sign them:
icinga2 ca sign CERT
If necessary, edit the host address in Icinga2 Director hosts to match the external pingable IP address for the location of the host, not the LAN IP address.
Then apply the configuration in Icinga2 Director. Check if Icinga2 service is running on the host machine. If it’s not, try giving it a restart.