Imagine power cycling a DICE server with redundant PSUs, using our lovely power bar control files. You don’t know where it’s installed, so you have to search for it. Thank goodness for the -xf script:

$ rfe -xf fpdu/myserver
s90.outlets:outlet 3 myserver on
s91.outlets:outlet 3 myserver on


Great. But now we have to open and edit each of these by hand:

$ rfe -xf fpdu/s90.outlets
outlet 1 thingy on
outlet 2 another on
outlet 3 myserver off⌶
outlet 4 [...]
…then…
$ rfe -xf fpdu/s91.outlets
outlet 1 thingy on
outlet 2 another on
outlet 3 myserver off⌶
outlet 4 [...]

and check the console, and again…

$ rfe -xf fpdu/s90.outlets
outlet 3 myserver off⌶
…and…
$ rfe -xf fpdu/s91.outlets
outlet 3 myserver off⌶
[...]


much easier to use my new rfeall script:

#!/bin/bash
# rfe edit all matching maps
search=$*; map=${search%/*}
maps=$(rfe -xf "$search" \
    |& awk -F ':' '/^rfe:/{exit;}{print $1;}' \
    | sort | uniq | xargs)
if [[ -n $maps ]]; then
    args=""
    for m in $maps; do args="${args}${map}/`filename ${m}` "; done
    rfe -S $args
else
    echo "No matches found for $search." >&2
    exit 1;
fi

which allows a far simpler edit:

$ rfeall fpdu/myserver

Oh. But we still have to edit two files, twice…


outlet 3 myserver off⌶
[...]
outlet 3 myserver off⌶
and relaunch, and…
outlet 3 myserver on⌶
[...]
outlet 3 myserver on⌶


So how much further can we take this?

Well, for a start I don’t want to have to navigate to line three every time. So let’s save the tedious effort of searching four times for the same thing. Turns out to be as easy as writing your own editor. Wait, what?

/usr/bin/vim -c 'call search("myserver")' $*

Maybe a slight exaggeration. But it needs to be a real, callable thing if you’re going to make it your $EDITOR, thus:

# rfe edit all matching maps; seek to match:
[...]
for m in $maps; do args="${args}${map}/${m} "; done
editr=/tmp/rfeall.$$
echo "/usr/bin/vim -c 'call search(\"$file\")' \$*" > $editr && chmod 700 $editr
EDITOR=$editr rfe -S $args
[...]

vim now opens at precisely the right location, give or take a word. But we still have to change those words by hand. Time for some vim scripting…

function! SwitchOn()
    s/o\(ff\|n\) *$/on/
endfunction
function! SwitchOff()
    s/o\(ff\|n\) *$/off/
endfunction
nmap   \O  :call SwitchOn()
nmap   \o  :call SwitchOff()

My vim’s not so sharp, so I have a nagging feeling there’s a nicer way to do this, but putting this into my .vimrc is enough to allow me to use two keys to turn the machine on the current line \On or \off.

So… you can now do your dual-PSU power cycle with approximately twenty keystrokes from the first time you hit <rtn>:


$ rfeall fpdu/myserver
\o :wq
\o :wq
…check your console, hit up, return…
\O :wq
\O :wq


all done. Since rfeall uses the same maps as rfe, it’s moderately useful to allow the rfe tab completion to work for the new script, too. In your ~/.bash_completion file (you have one, don’t you?):

complete -o nospace -F _rfe rfeall

Yes, I suppose could automate even the last step by setting ‘sed’ as my editor, but there’s a limit, I think – I trust myself little enough to turn off the right machine without delegating it to my dubious pattern-matching skills…

The Research and Teaching unit recently noted that users’ crontabs (sometimes required, for example, on research servers to start or check on custom services) are not routinely backed up. They are of course necessarily minimal and easy to recreate, but their fate following a disaster (or even a reinstallation) is not obvious. Even if users are aware of this limitation, it prevents us from performing a completely automated recovery where users’ crontabs are involved.

It is possible to back up crontabs on demand or routinely, but we’ve no procedure to do this on machines that typically have no other data to back up. So the purpose of this post is simply to draw attention to the above, and ask a few general questions:

• Is it well-known that crontabs are “at risk”? Does anyone care?
• Should crontabs be backed up routinely? on all servers? desktops? at all?
• Is there anything else we should be doing about this?

Comments are encouraged, below or by email.

1. crontab is the setuid executable which manages /var/spool/cron/ for this purpose. The location is notable: on DICE, the /var directory (or filesystem) consists of data which is either transient or generated and managed automatically. With the exception of some database files, this data is not typically backed up since its value is only to the running system. Indeed most cron entries on a machine are not created by users; they are configured by LCFG and (re)created automatically.
2. Note that using cron on DICE requires a few steps to work with (or around) AFS.

https://svn.theon.inf.ed.ac.uk/trac/wiki/PgluserAndDice

Have I missed anything?

The [filter] command is executed and the message is piped into its standard input. The standard output of the command is read back by Alpine.

This says it all: display filters turn out to be an extremely powerful generic mechanism for reformatting and enhancing text; it works particularly well when applied to machine generated messages. Maybe its power is best explained by the example which caused me to investigate in in the first place:

An example (the nagios bit):

A longstanding irritant to me has been a my difficulty in shutting nagios up. For a long time I’ve been relying on a filter to parse nagios’ incoming emails and generate a URL. The display filter closes the loop, automatically injecting that magic URL at the end of the message.

Here’s a simplified version of the filter, reminiscent of the one in the previous post:

#!/usr/bin/gawk -f
# Crude detection of problem type for acknowledgement link
# Don't forget to validate these inputs...
/Notification Type: / { TYPE=$3; }
/Service:/ { SERVICE=substr($0,length($1)+1,length($0)); }
/Host:/ { HOST=$2; }
# Important: this is a filter, so don't forget to print input lines back out!
// {print;}
# Now add the acknowledgement link below:
END {
    if (HOST && TYPE == "PROBLEM") {
        # this is the script which generates the URL.
        # ideally this should be replaced with some awk to do the same thing
        cmd="~/bin/nagack "HOST" "SERVICE
        cmd | getline url
        close(cmd)
        # now add the link to the email.
        print "[Acknowledgement link: "url" ]"
}

Now, to alpine’s Display Filters setting, add:

Display Filters    = _LEADING("***** Nagios")_ /path/to/nagios-filter-script

that’s it! My emails from nagios now look like:

***** Nagios *****
Notification Type: PROBLEM
Service: ssh
Host: myhost
Address: 192.168.12.34
State: CRITICAL
...
[Acknowledgement link: https://nagiosserver/nagios/cgi-bin/cmd.cgi?cmd_typ=3... ]

Important caveats:

• If you’re not careful, by adding these filters you will have introduced a trivial local shell injection attack to your mail client. Validate your inputs — just like I didn’t above!
• The developers have this to note about running filters on every message:

  Testing for the trigger and invoking the filter doesn’t come for free. There is overhead associated with searching for the trigger string, testing for the filter’s existence and actually piping the text through the filter. The impact can be reduced if the Trigger Modifying Tokens [...] are employed.

  I’ve certainly noticed a small (immeasurable, but noticeable) delay in opening messages with triggers. Large enough to be annoying if I’d planned to filter every message, even using a trivial bash filter which itself is quick to complete.

• One additional caveat on DICE: if your alpine session outlives your AFS credentials, and you’ve stored your display filters in your home directory, you will find that the display filters simply disappear. As good a reminder as any to renew, and thankfully a “renc” is all that’s required to restore your filters to former glory.

That’s it! Surprisingly trivial, and with a handful of these triggers, the benefits are huge. I’m using five so far, mostly generating clickable links to some of our automated systems, but I’d be pleased to hear what other people are doing with these filters.

This is easy to fix using vim’s ftdetect system. Some examples for simple types:

" These files are always POD in disguise
au BufRead,BufNewFile *.pod.cin : set filetype=pod
" Slightly contentious: a new filetype is needed, really, but this is a decent match.
au BufRead,BufNewFile *.def.cin : set filetype=cpp
" For other, unknown types, detect from the as-yet undefined shebang:
au BufRead,BufNewFile *.cin : if getline(1) =~ '^#!@SHELL@' | set filetype=sh | endif
au BufRead,BufNewFile *.cin : if getline(1) =~ '^#!@PERL@' | set filetype=perl | endif

(note the latter two lines are specified separately, rather than elseifed, purely for readability). It’s fairly obvious that this can be extended to any file type, and there’s also scope for adding an automatic mapping to allow all files of form file.typ.cin to be mapped automatically to their default .typ. “sub-extension” file type.

Anyway, the above has already improved my productivity no end so I’ll leave the latter exercise to the reader. Comments and contributions are welcome, as always — so long as they’re not suggestions to use Emacs(!)

Our PXE server is configured automagically by spanning map and, effectively, keyed on MAC, so it’s unlikely to present the correct configuration (reliably) when the new host differs from the old one in some way.

The workaround is to override the PXE configuration on the *existing* server (on the basis that you weren’t planning on reinstalling it, anyway, were you?):

!pxeclient.platforms mADD(new_plat_name) /* e.g. sl6_64 */

/* And, if you need to add or remove serial console support: */
!pxeclient.serial_port mSET(ttyS0) /* or () */

Post-PXE, the dhclient component is aware of subnet differences and will ensure your machine receives the correct profile for installation (though, to prevent future confusion, remove this as soon as the installer has done its work!).

All it does is chain up “rpmbuild -bs [spec]; pkgforge submit [srpm]” but it’s a nice timesaver nonetheless. Side-benefits include the automatic generation of a readable ID and provision of a tracking link for pkgforge so that you can anxiously refresh the page to watch the build progress (or you could just wait for the report email…).

So, here is is; my very simple and stupid RPM automation. Suggested name: ‘rpmbus’.

#!/bin/bash
if [[ -z $2 ]]; then
    echo "RPMbus: build -> submit assist"
    echo "Usage: `basename $0`   [pkgforge args]"
    exit 1
fi
bucket=$1; shift
spec=$1; shift
args=$*

output=`rpmbuild -bs ${spec} | tail -n 1`
pkg=`echo ${output} | sed -e 's_^Wrote: __'`

if [[ ! -e ${pkg} ]]; then
    echo "Package wasn't built: ${output}"
    exit 1
fi

id=`basename ${spec} | sed -e 's_\.spec__' -e 's_\.__g'`-`date +"%s"`
echo -e "Found source package:\n  ${pkg}"
echo "  Extra args: ${args:-none}"

read -p "Submit to '${bucket}'?" foo
if [[ ${foo} != 'y' ]]; then
    echo "Cancelled"
    exit 1
fi

echo "Submitting to ${bucket}..."
pkgforge submit --id ${id} -B ${bucket} ${args} ${pkg} && \
echo "  https://pkgforge.inf.ed.ac.uk/job/view?id=${id}"

Caveats: well, they’re numerous and they’re pretty apparent. But it took five minutes to write and it WFM :)