Category Archives: promotion

alpine descent

Fans (and regular users) of the text-based email client “alpine” might be interested to know that as of 3rd December we will be reverting to the previous upstream release, 2.11 on DICE SL7. The noticeable effect of this change should be minimal, except if you are in the set of users affected by bugs in the 2.20 release, in which case it should be wholly beneficial. DICE SL6 won’t be affected by this, as it has remained on the comparatively stable 2.10 release for some time now.

For those who have not been following the development of pine/alpine in recent years, this history gives a good account.

The version we’ve been using for a little while (2.20-1) was taken from an alternative fork to previous versions and contained several flaws – some of which we’ve noticed in practice – and unfortunately the updated package (2.20-2) referenced above adds a further bug affecting normal workflow (specifically: entering the “Compose” menu causes a repeatable fault if keylabels are disabled; in due course this should be reported upstream).

Given the comparatively unstable state of upstream alpine packaging, we’ve taken the decision to revert to 2.11 for now, and in future we will almost certainly build and package our own release of alpine as we had done for most of the course of DICE SL5 and SL6, but based on Eduardo Chappa’s releases. This invites some additional maintenance load to using our upstream distribution, but should hopefully result in a better, more stable alpine on SL7.

alpine, nagios and display filters

I’ve been aware of alpine’s “display filter” feature for some time, used as it is for on-the-fly GPG interpretation amongst other things. But I’d never really examined the feature before now. The manual says:

The [filter] command is executed and the message is piped into its standard input. The standard output of the command is read back by Alpine.

This says it all: display filters turn out to be an extremely powerful generic mechanism for reformatting and enhancing text; it works particularly well when applied to machine generated messages. Maybe its power is best explained by the example which caused me to investigate in in the first place:

An example (the nagios bit):

A longstanding irritant to me has been a my difficulty in shutting nagios up. For a long time I’ve been relying on a filter to parse nagios’ incoming emails and generate a URL. The display filter closes the loop, automatically injecting that magic URL at the end of the message.

Here’s a simplified version of the filter, reminiscent of the one in the previous post:


#!/usr/bin/gawk -f
# Crude detection of problem type for acknowledgement link
# Don't forget to validate these inputs...
/Notification Type: / { TYPE=$3; }
/Service:/ { SERVICE=substr($0,length($1)+1,length($0)); }
/Host:/ { HOST=$2; }
# Important: this is a filter, so don't forget to print input lines back out!
// {print;}
# Now add the acknowledgement link below:
END {
    if (HOST && TYPE == "PROBLEM") {
        # this is the script which generates the URL.
        # ideally this should be replaced with some awk to do the same thing
        cmd="~/bin/nagack "HOST" "SERVICE
        cmd | getline url
        close(cmd)
        # now add the link to the email.
        print "[Acknowledgement link: "url" ]"
}

Now, to alpine’s Display Filters setting, add:


Display Filters    = _LEADING("***** Nagios")_ /path/to/nagios-filter-script

that’s it! My emails from nagios now look like:


***** Nagios *****
Notification Type: PROBLEM
Service: ssh
Host: myhost
Address: 192.168.12.34
State: CRITICAL
...
[Acknowledgement link: https://nagiosserver/nagios/cgi-bin/cmd.cgi?cmd_typ=3... ]

Important caveats:

  • If you’re not careful, by adding these filters you will have introduced a trivial local shell injection attack to your mail client. Validate your inputs — just like I didn’t above!
  • The developers have this to note about running filters on every message:

    Testing for the trigger and invoking the filter doesn’t come for free. There is overhead associated with searching for the trigger string, testing for the filter’s existence and actually piping the text through the filter. The impact can be reduced if the Trigger Modifying Tokens […] are employed.

    I’ve certainly noticed a small (immeasurable, but noticeable) delay in opening messages with triggers. Large enough to be annoying if I’d planned to filter every message, even using a trivial bash filter which itself is quick to complete.

  • One additional caveat on DICE: if your alpine session outlives your AFS credentials, and you’ve stored your display filters in your home directory, you will find that the display filters simply disappear. As good a reminder as any to renew, and thankfully a “renc” is all that’s required to restore your filters to former glory.

That’s it! Surprisingly trivial, and with a handful of these triggers, the benefits are huge. I’m using five so far, mostly generating clickable links to some of our automated systems, but I’d be pleased to hear what other people are doing with these filters.

tar-a-dice?

I recently encountered a .tar file which DICE tar wouldn’t extract, giving multiple warnings:

tar:  Ignoring unknown extended header keyword `SCHILY.[...]'

before bailing out due to previous “errors”. The file in question appeared to have been created on a Mac, whose archive utility adds extended metadata to its files.


This is confirmed and fixed in May 2007, but some more venerable Linux distributions continue to ship older versions.  Tar’s behaviour is a bug, since these warnings should be non-fatal.  However, the warnings can be avoided altogether by addition of an argument:

 --pax-option="delete=KEYWORD.*"

for each problematic keyword.

[Source: Tar mailing list]

Python Profiling

Profiling isn’t often important for my python programs, but when a performance black hole appears out of nowhere it’s very useful for narrowing down the problem, even on smaller scripts.

(Recently I had cause to profile a script manipulating large-ish (<50Mb) files, whose performance had taken a turn for the worse. I had a suspicion that poor list manipulation was to blame, but the results spoke for themselves: 99% of CPU time was spent in the function list.pop(0)! Based on this I discovered PEP-290 which describes the collections.deque structure, significantly more efficient than a basic list — the replacement function, deque.popleft(), barely registers a percentage of overall execution time(!)

I could have discovered this using basic profiling, but whilst playing with the profiler I discovered a very straightforward, easy-to-read python profile visualiser, and thought it worth sharing. It doesn’t do anything very beautiful or complicated, but it’s very quick, provides a neat “squaremap” view of your code performance, and importantly Just Works. RunSnakeRun is its name.

Continue reading

Real Mac UK Keyboard Layout

Edit: Oct 2013

Rejoice! OS X 10.9 “Mavericks” now provides a keyboard layout called “British – PC” which faithfully maps a real UK keyboard (so far as five minutes’ examination has shown) onto the Mac (so long as caps lock is not engaged).

Original article (for OS X upto 10.8 “Mountain Lion”):

I hate the Apple UK keyboard layout. I detest it. Defensible as it may be to enhance cross-Atlantic keyboard familiarity, and rational as it might be to place double-quote above single, I still cannot stand it. If I were a ‘switcher’, and had decreed that, from last March, only Apple computers were sufficiently worthy to be graced by my fingertips, I might have come to live with it but I object to having to remap my brain-finger pathways every time I move from one platform to another (remembering which clipboard / paste buffer to use is struggle enough).

All this is an elaborate way of saying that I’ve made a set of truly UK-compatible keyboard layouts for my MacBook Pro and the standard UK USB keyboard I sometimes plug into it. These layouts work for 10.5 “Leopard” and possibly others. They can be found by in my RealUKKeyboardLayouts.zip file.

In this zip file you’ll find the two layouts and two similarly-named .icns files which allow you to identify the layouts at a glance. They’re not very pretty but they do the job.

Unzip the archive and, for each user who wants to “type proper” again, place the files in ~/Library/Keyboard Layouts/ (creating the directory if required). Log out and back in to allow Mac OS to discover the new layouts (supposedly /Library/Keyboard Layouts/ can be used for system-wide layouts, but this was not the case on my first attempt and, as I’m the only user of my laptop who cares about such things, I didn’t delve further).

Now open the “International” Preference Pane. On the “input menu” tab, check “Show input menu in menu bar”. You’ll need this because, just occasionally, Leopard will switch you back to a system ‘blessed’ layout and you’ll spend hours cursing your mistaken “@”s until you figure out what’s happened. Now scroll through the keyboard layouts and check “Real UK” and “Real UK – IBM/PC” to allow these layouts to be selected from the input menu.

Close the preference pane, select the appropriate layout from the menu bar, and that’s it.

Oh yes: in case you were wondering, these layouts were created with the eccentric but indispensible Ukelele which guides you through the entire process. This will be particularly useful if you want a layout for a keyboard other than my provided MacBook Pro or IBM/PC layouts.

offlineimap and alpine

Edit: The future is here! I’ve shortened my wishlist since OfflineIMAP now supports the IDLE command.

Further Edit: Kerberos instructions for Mac OS now available

For some time I’ve been meaning to make use of some sort of mail caching, in order to use my favourite email client whilst offline.  The end result of this process is that my incoming mail now takes a somewhat circuitous route of:

imap server
  |
offlineimap - local uw-imapd - alpine
                    |
               local cache

on my laptop.

Continue reading