NRPE: Unable to read output
I've been working on setting up nrpe to allow a nagios server to execute check commands on another server.
From looking at NRPE, it seems to be a simple tcp server, a bit like a telnet server that receives a command, executes it on the host machine, and the sends the result back. NRPE also uses ssl encryption between the client and nagios server, so the communication seems quite secure.
============= =============
| Nagios Server | -> | Remote Server |
============= =============
check_nrpe -> nrpe server -> check_ping
So I've compiled nrpe from source to get the nrpe server daemon and the check_nrpe plugin.
As a note, I used the nrpe server daemon I compiled on the remote box, and I compiled nrpe on my nagios server. Then I copied the check_nrpe binary to the nagios plugin folder.
The point I'm making is, I didn't compile everything on one server, I compiled nrpe on both servers and used the server daemon on the remote box, and the check plugin on the nagios server.
Stops you getting compile / execute errors, due to package differences between the two servers.
Once I got nrpe up and running, I tried to use the check_nrpe script from the nagios server to connect to the remote server and run a check_users command. It would fail with NRPE: Unable to read output.
But if i sent a bogus command that wasn't defined in the nrpe.cfg file on the remote server, nrpe would come back saying the command didn't exist.
STRANGE indeed. That means NRPE is fine, the SSL is working, but something else is causing a problem.
In my /var/log/messages on the remote server I would find entries like this:
sudo(pam_unix)[15694]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=nagios
These entries would appear after I ran the check_nrpe plugin from the nagios server.
It turns out, nrpe on the remote host, needs to execute the check commands as root. This makes sense as the data the check_users plugin needs, would only be available as root. Here sudo now gets involved.
So nrpe fires up an sudo command to run the nagios plugin to get say our check_users data.
The command would be something like sudo -u nagios /etc/nagios/plugins/./check_users -w 4 -c 8
If you were to goto the remote server, and do the following:
# su nagios (become the nagios user)
# sudo -u nagios /etc/nagios/plugins/./check_users -w 4 -c 8
After the sudo, you would see it request a password. As nrpe can't enter passwords, this is what is causing the error. It invokes the command, but can't complete it, as sudo wants a password.
To solve this, you need to edit the /etc/sudoers file.
Here is a good doc explaining it in detail:
http://www.chinalinuxpub.com/doc/www.siliconvalleyccie.com/linux-hn/sudo.htm#_Toc32905575
The short of it is, I uses visudo and added this to the sudoers file.
nagios ALL=(ALL) NOPASSWD: /usr/local/nagios/libexec/check_users
For some reason you have to add and entry for each file. With this done, nrpe will work correctly.
Cheers for now.
From looking at NRPE, it seems to be a simple tcp server, a bit like a telnet server that receives a command, executes it on the host machine, and the sends the result back. NRPE also uses ssl encryption between the client and nagios server, so the communication seems quite secure.
============= =============
| Nagios Server | -> | Remote Server |
============= =============
check_nrpe -> nrpe server -> check_ping
So I've compiled nrpe from source to get the nrpe server daemon and the check_nrpe plugin.
As a note, I used the nrpe server daemon I compiled on the remote box, and I compiled nrpe on my nagios server. Then I copied the check_nrpe binary to the nagios plugin folder.
The point I'm making is, I didn't compile everything on one server, I compiled nrpe on both servers and used the server daemon on the remote box, and the check plugin on the nagios server.
Stops you getting compile / execute errors, due to package differences between the two servers.
Once I got nrpe up and running, I tried to use the check_nrpe script from the nagios server to connect to the remote server and run a check_users command. It would fail with NRPE: Unable to read output.
But if i sent a bogus command that wasn't defined in the nrpe.cfg file on the remote server, nrpe would come back saying the command didn't exist.
STRANGE indeed. That means NRPE is fine, the SSL is working, but something else is causing a problem.
In my /var/log/messages on the remote server I would find entries like this:
sudo(pam_unix)[15694]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=nagios
These entries would appear after I ran the check_nrpe plugin from the nagios server.
It turns out, nrpe on the remote host, needs to execute the check commands as root. This makes sense as the data the check_users plugin needs, would only be available as root. Here sudo now gets involved.
So nrpe fires up an sudo command to run the nagios plugin to get say our check_users data.
The command would be something like sudo -u nagios /etc/nagios/plugins/./check_users -w 4 -c 8
If you were to goto the remote server, and do the following:
# su nagios (become the nagios user)
# sudo -u nagios /etc/nagios/plugins/./check_users -w 4 -c 8
After the sudo, you would see it request a password. As nrpe can't enter passwords, this is what is causing the error. It invokes the command, but can't complete it, as sudo wants a password.
To solve this, you need to edit the /etc/sudoers file.
Here is a good doc explaining it in detail:
http://www.chinalinuxpub.com/doc/www.siliconvalleyccie.com/linux-hn/sudo.htm#_Toc32905575
The short of it is, I uses visudo and added this to the sudoers file.
nagios ALL=(ALL) NOPASSWD: /usr/local/nagios/libexec/check_users
For some reason you have to add and entry for each file. With this done, nrpe will work correctly.
Cheers for now.
posted by crash at
4:53 AM
2 Comments:
Doesn't adding something to the sudoers file open you up for a potential security risk. i.e. if via a buffer overflow someone is able to drop a file with the same name in the same location, they could execute it without knowing the root password and gain control of the box :S Is there no way to somehow put a signature (SHA1 or MD5) of the application in the sudoers file to prevent that type of attack. i.e. the injected file won't have the same signature and thus doesn't execute.
It definitely does create a security hole using sudo.
Even though the servers sit in a DMZ or on a LAN, it shouldn't be the only line of defense.
The best way to protect yourself from this attack is to use xinetd or a very tightly configured nrpe.cfg file.
There are options to set which IP address (or nagios server) the nrpe requests are accepted from.
This is not 100% fool proof as a hacker could spoof the IP address.
But if you accept requests from all IP's you are looking for trouble.
Next is to move the nrpe server off the default port, and use a random port. Not fool proof by any means.
Don't use the default nagios user, bad idea. Change it something else.
And well, nrpe should not be used over the internet. You could build alot of security to protect the nrpe server, but a good hacker will get in.
Even though the connections are SSL encrypted, I could easily get my nrpe_plugin to connect to a complete "strangers" nrpe server. The SSL is just to stop the whole "OMG!? it's like telnet across the internet".
Again, the fun of open source.
I like the hash check. Brilliant idea.
To be honest nrpe seems like something that got slapped together in a weekend.
Post a Comment
Subscribe to Post Comments [Atom]
<< Home