Tag Archives: nagios

alpine, nagios and display filters

I’ve been aware of alpine’s “display filter” feature for some time, used as it is for on-the-fly GPG interpretation amongst other things. But I’d never really examined the feature before now. The manual says:

The [filter] command is executed and the message is piped into its standard input. The standard output of the command is read back by Alpine.

This says it all: display filters turn out to be an extremely powerful generic mechanism for reformatting and enhancing text; it works particularly well when applied to machine generated messages. Maybe its power is best explained by the example which caused me to investigate in in the first place:

An example (the nagios bit):

A longstanding irritant to me has been a my difficulty in shutting nagios up. For a long time I’ve been relying on a filter to parse nagios’ incoming emails and generate a URL. The display filter closes the loop, automatically injecting that magic URL at the end of the message.

Here’s a simplified version of the filter, reminiscent of the one in the previous post:


#!/usr/bin/gawk -f
# Crude detection of problem type for acknowledgement link
# Don't forget to validate these inputs...
/Notification Type: / { TYPE=$3; }
/Service:/ { SERVICE=substr($0,length($1)+1,length($0)); }
/Host:/ { HOST=$2; }
# Important: this is a filter, so don't forget to print input lines back out!
// {print;}
# Now add the acknowledgement link below:
END {
    if (HOST && TYPE == "PROBLEM") {
        # this is the script which generates the URL.
        # ideally this should be replaced with some awk to do the same thing
        cmd="~/bin/nagack "HOST" "SERVICE
        cmd | getline url
        close(cmd)
        # now add the link to the email.
        print "[Acknowledgement link: "url" ]"
}

Now, to alpine’s Display Filters setting, add:


Display Filters    = _LEADING("***** Nagios")_ /path/to/nagios-filter-script

that’s it! My emails from nagios now look like:


***** Nagios *****
Notification Type: PROBLEM
Service: ssh
Host: myhost
Address: 192.168.12.34
State: CRITICAL
...
[Acknowledgement link: https://nagiosserver/nagios/cgi-bin/cmd.cgi?cmd_typ=3... ]

Important caveats:

  • If you’re not careful, by adding these filters you will have introduced a trivial local shell injection attack to your mail client. Validate your inputs — just like I didn’t above!
  • The developers have this to note about running filters on every message:

    Testing for the trigger and invoking the filter doesn’t come for free. There is overhead associated with searching for the trigger string, testing for the filter’s existence and actually piping the text through the filter. The impact can be reduced if the Trigger Modifying Tokens […] are employed.

    I’ve certainly noticed a small (immeasurable, but noticeable) delay in opening messages with triggers. Large enough to be annoying if I’d planned to filter every message, even using a trivial bash filter which itself is quick to complete.

  • One additional caveat on DICE: if your alpine session outlives your AFS credentials, and you’ve stored your display filters in your home directory, you will find that the display filters simply disappear. As good a reminder as any to renew, and thankfully a “renc” is all that’s required to restore your filters to former glory.

That’s it! Surprisingly trivial, and with a handful of these triggers, the benefits are huge. I’m using five so far, mostly generating clickable links to some of our automated systems, but I’d be pleased to hear what other people are doing with these filters.

Nag nag nag nag nagios

Nagios is an extremely useful tool, until it isn’t.  Which is to say, it’s nothing but a hindrance to have nagios continue to bombard you with IMs and emails when you’re already working on the problem.

Surely you can just acknowledge the fault and shut it up…?

Well, sometimes, but it is hardly convenient to break out a Firefox session when you’re attached to a serial console with your lovely secure-shell-enabled phone.  And even if you are on a DICE machine it’s a bit of a pain to have to navigate the slightly clunky Nagios UI to find the host and service you wish to silence.

I started with a dumb bash script. Continue reading

monitoring postgresql

This should serve as yet another account of “naïve” addition of monitoring to an LCFG component.  It proved to be surprisingly straightforward: PostgreSQL on my test server is now being monitored; built-in support is now available on any host:

#include <dice/options/postgresql-monitoring.h>

though take care to follow the instructions within.

Continue reading

Passive Monitoring

Familiar?

Profile Translation WARNING 1/4 WARNING: Service checked passively

If you’re getting this warning for a service which is checked passively by default, the solution is simple: just disable active checking!

It seems as if a more appropriate message would be something along the lines of “WARNING: cannot perform active checking” or similar. Still, now I know, if it saves someone else attempting to re-enable every option three times over… or reading documentation…