offlineimap and alpine

Edit: The future is here! I’ve shortened my wishlist since OfflineIMAP now supports the IDLE command.

Further Edit: Kerberos instructions for Mac OS now available

For some time I’ve been meaning to make use of some sort of mail caching, in order to use my favourite email client whilst offline.  The end result of this process is that my incoming mail now takes a somewhat circuitous route of:

imap server
  |
offlineimap - local uw-imapd - alpine
                    |
               local cache

on my laptop.

The following is my experience of configuring this software on Mac OS X 10.5; I’ve chosen this perspective as it’s pretty much a superset of the configuration required for Linux. The only major difference worth noting is that you should expect to use a different authentication method for your remote IMAP server; if you’re lucky you can use Kerberos, otherwise you might wish to consider either interactive password entry, or perhaps the Python gnome-keyring API.

Alpine is a fast, responsive email client which is outstandingly powerful, but as it sits very close to the IMAP protocol (indeed, in complete compliance) it can be very sensitive to server issues.  More upholstered clients such as Apple’s Mail.app cache mail locally for searching, but for best results require you to … use them.

The need to separate myself from my mailserver becomes increasingly relevant with *pine if one finds oneself switching connection frequently; disconnected operation is not one of its strengths.  Also, on some servers, as server or bandwidth utilisation increases, user experience can reduce to unacceptable levels.

the IMAP bit

offlineimap provides the answer: it will synchronise between two IMAP servers, or from one server to a local mail store.  Sadly, its local mailstore is in the Maildir format, supported only partially using a third-party (and moreover, almost officially disapproved of) patch to *pine, so for full Alpine access, a local IMAP server is the only option.

At present I would recommend pulling the latest git revision, rather than the latest release (v6.0.3); see the end of this post for details.

The natural choice of IMAP server for Alpine is the UW-IMAPD, built using the same core as Alpine.  Its installation is trivial on any supported platform, and its configuration is famously nonexistent.  I chose to download and compile directly into /usr/local/.  Without any further configuration, you now have a working IMAP daemon, suitable for secure, internal connections:

$ /usr/local/sbin/imapd
* PREAUTH [CAPABILITY IMAP4REV1 I18NLEVEL=1 LITERAL+ IDLE UIDPLUS NAMESPACE CHILDREN
MAILBOX-REFERRALS BINARY UNSELECT ESEARCH WITHIN SCAN SORT THREAD=REFERENCES
THREAD=ORDEREDSUBJECT MULTIAPPEND] Pre-authenticated user [...] IMAP4rev1 [...]

alpine / imapd / preauth

IMAP’s preauth mode means that there’s no need to mess about with authentication (unless you are going to be allowing remote TCP connections). But there’s no reason to restrict yourself to purely local access, if your local client accepts inbound SSH. Alpine and latter versions of pine will connect to a remote host using SSH automatically:

  • set the alpine configuration variables:
    • SSH Open Timeout” >= 5.
    • SSH Command“: %s %s -l %s exec /etc/r%sd.
    • SSH Path“: path to your ssh binary
  • create a link at /etc/rimapd to /usr/local/sbin/imapd (yes, bad FHS practice, but some versions of *pine/imapd do not appear to work with non-default locations)
  • Make sure to alter any other mailboxes or collections you have with the /norsh flag to prevent failed SSH attempts to those hosts.

(Remember, for locking purposes it is important that you only access these files from imapd, when a copy of imapd is running.)

further IMAP

This gives you instant access, effectively, to the whole contents of your home directory, at least from your local machine if not further. It would not be unreasonable to assume that your mail will be stashed in a subdirectory therein, rather than ~.  This is configurable by setting a folder prefix in alpine, offlineimap and any other mail client you use; or if, like me, you will be using your new IMAPd for nothing else, you can take the “super-dangerous” option of adjusting the IMAP root in your ~/.imaprc file, thus:

I accept the risk
set mail-subdirectory .offlinemail

(Don’t forget to create your mail directory).

Depending on your preference you might also wish to append

set new-folder-format mix

which uses a newer, faster method of storing mail.  You gain the advantage of faster mail access and search from IMAP clients, but lose the ability to usefully search mail by simply grepping the mailboxes (users of Spotlight, Beagle, etc. take note).

the offline bit

OfflineIMAP configuration is relatively straightforward, but somewhat long-winded.  Simplest to reproduce this verbatim:

[general]
accounts = Work
pythonfile = ~/.offlineimap.py
ui = Noninteractive.Quiet

[Account Work]
localrepository = LocalImap
remoterepository = Remote
autorefresh = 10 # full sync every ten minutes
quick = 5 # perform a quick sync (don't change flags on existing messages) every (10/5) minutes.

[Repository LocalImap]
type = IMAP
remotehost = localhost
preauthtunnel = /usr/local/sbin/imapd
maxconections = 2

[Repository Remote]
type = IMAP
ssl = yes
remotehost = [IMAP server here]
remoteuser = [IMAP username here]
remotepasseval = o.getPassword('[IMAP server here]')
holdconnectionopen = yes
maxconections = 2

# pick one of the below:
# (for a single server)
nametrans = lambda foldername: re.sub('^INBOX', 'INBOX_work', foldername)
# (for multiple-servers)
# nametrans = lambda foldername: re.sub('^(.*)$', '\\1_work', foldername)

Note the nametrans line above, which modifies the name of your remote inbox.
It’s important to define at least this, translation, since the special folder INBOX will not be allowed on a machine without a full local IMAP server configuration. However, if you want to define more than one remote mail server it’s probably worth translating all folder names to prevent conflicts. One would then obviously modify the ‘work’ suffix for each account.

the key part

So far so good.  But this configuration isn’t going to work.

A pre-authenticated local connection is all very well, but connecting to your remote server will require retrieval of some credentials.  If you’re fully Kerberised, OfflineIMAP claims to be, also.  I couldn’t get this working on my laptop, though didn’t expend much effort; I already have my mail password stored in my OS X Keychain thanks to Apple Mail, so thought I’d make use of it.

Note the lines:

pythonfile = ~/.offlineimap.py
remotepasseval = o.getPassword('[IMAP server here]')

The former defines a file to source before running offlineimap for the first time.  The latter defines a python expression which evaluates to a password for that server.

The python source in question references a small script, the essence of which is:

#!/usr/bin/python
import keychain

class OIHelper:
  def __init__(self):
    self.k = keychain.Keychain('/Users/grape/Library/Keychains/login.keychain')

  def getPassword(self, server):
    result = self.k.getinternetpassword(server)
    return result['password']

The ‘keychain’ module in question is an exceptionally crude interface to the Mac OS ‘security’ command found elsewhere; I made some improvements and, for now, it works.  With some embarrassment, linked is the python keychain interface. If you’re not comfortable with this scripted stored-password retrieval, you can authenticate in any other way you like, the simplest of which is probably to set remotepasseval = pwd and in your python script:

from getpass import getpass
p = getpass()

first run

Debug by launching offlineimap -u Curses.Blinkenlights, and launch your configured mailer. After an extended first synchronisation, watch the mail come rolling in!

Note that offlineimap has a fail-fast philosophy. Pretty much anything which it doesn’t like the look of, will cause it to bail out before any data is written (offlineimap performs a sort of dry-run merge before any copying takes place). It generally bails out with a stack trace, so don’t be intimidated if it appears to have fallen over completely simply because you’ve not made the mail subdirectory, for example.

the future

For me, a few tasks remain:

  • Add IMAP IDLE support to offlineimap.  This would improve mail response times and decrease bandwidth / server load. In the meantime, the latest repository version contains a patch allowing offlineimap to respond to signals; I have scripted a forced-check each time I launch my mailer using this. latest repository version contains IDLE support: simply set the configuration variable idlefolders = ['INBOX', ...] within each [repository].
  • Investigate why new mail is not flagged ‘recent’ when delivered to the local inbox. ‘Quick’ sync does not appear to be to blame for this.  This is a feature of UW-IMAP: offlineimap’s modifications to a mailbox when it inserts messages automatically precludes any messages from appearing ‘recent’ to a subsequent, human access of the mailbox.  Some manual, non-compliant tweak would probably be required to reset ‘recent’…
  • Investigate Kerberos authentication to the external mail server. Simply requires python Kerberos support: see below.
  • Launch offlineimap more reliably in the background.  For the moment a simple bash script loop suffices, but there should be no reason why launchd or similar cannot perform this task; indeed it’s designed to do just that.  A further problem lies in retrieving your keychain details (and indeed even prompting for your keychain password) in the context of launchd, which requires more knowledge of its internals than I presently have.
  • If you’re having trouble getting the pre/postsynchook configuration values to work, it’s because they don’t, in v6.0.3. My patch in git HEAD fixes this.

Bonus: Kerberos

Dead easy, but requires python-kerberos support. This is available in the usual ways on most Linux distributions, but is squirreled away from easy sight of Mac users in the CalendarServer project repository.

On the Mac:

$ svn export http://svn.calendarserver.org/repository/calendarserver/PyKerberos/tags/release/PyKerberos-1.1/
$ cd PyKerberos-1.1
$ python setup.py build && sudo python setup.py install

Now make sure to unset any password-related resources (remotepasseval if you followed the instructions above) for the Kerberised server. Also make sure you have the latest repository version of offlineimap (April 2009), which includes a fix to allow autorefresh+kerberos.

3 thoughts on “offlineimap and alpine

  1. gdutton Post author

    Actually, adding remote IMAP support in launchd is quite straightforward. I’ve posted the plist files I’ve used (to be placed in /Library/LaunchDaemons/) for IMAP and IMAPS.
    (I couldn’t get launchd to execute imapd except when it was placed in /usr/local/libexec/ – *sigh*).

  2. Pingback: Email clients « CPBL’s trivia

Leave a Reply

Your email address will not be published. Required fields are marked *