"IMAP"

I’ve spent the past couple of weeks starting the coding to PeerGroup, an IMAP based Personal Information Manager.  Seeing as IMAP is so central to the whole operation, I’ve spent that time reading up on IMAP, designing and writing the object model and refining it into a lean, mean, message-receiving machine.  (Actually at the moment, the library doesn’t support receiving messages, but you can do whatever you like to mailboxes!)

In looking around the internet, I’ve found that no one seems to have done a Critical Appraisal of IMAP (although there is this report comparing libraries).  No one seems to have looked back and thought ‘why on Earth did we do it that way?’ or ‘Wow, that Mark Crispin sure had some foresight!’.  So I thought I might document my experience with IMAP so far and share my thoughts on what’s right and what’s wrong.

I’ve happened to implement quite a few network protocols recently.  One of the things I was involved with at work before I left last week was investigating ISA100, a wireless mesh network stack standard (like TCP\IP and Ethernet but for wireless meshes).  At the boundary of the system is the gateway, which allows communications to pass from the wireless mesh to a standard TCP\IP network and back.  In order to get data out of the network a program (such as one that sits around waiting for thickness readings from oil and gas pipe walls to come in and put them in a database) can communicate with the gateway using a protocol called GSAP.  The specifics of GSAP don’t seem to be set down in the ISA100.11a standard, but the implementation we were using, from a company called Nivis, was very nice to work with.

In Nivis GSAP, each packet has a fixed length header.  The header tells you a version, a service ID (which tells you what type of request\response\indication the packet is), a transaction ID (which requests and responses for the same command share), the length of the payload and a CRC.  In the library I made to communicate with the demo kit we had, my program read off the header into an object (a GsapPacketHeader), inspected that for the service ID and used that to create an object for the actual packet (such as a GsapTopologyResponsePacket).  All requests and responses were nicely and simply defined and every request had a response, unless you really messed things up.

If only the same could be said about IMAP.  The phrase ‘simple protocol design’ didn’t seem to be words that could be found in Mark Crispin’s head as he developed IMAP.  But to be fair we need some context.  The first public version of IMAP, IMAP2 was developed in the late 80s, and even IMAP4’s most recent revision in RFC3501 was done in 2003, updating RFC2060, which was published in 1996.  Therefore technologies and techniques that we take for granted today like JSON, AJAX, XML, JavaScript, HTTP, HTML, 32-bit processors, integrated IDE controllers on motherboards, 486s and the like are all new fangled things according to IMAP and its backward compatibility requirements.  Nowadays, we’d probably make up some HTTP-based RESTful XML or JSON based protocol so we could all drink web flavour kool-aid while we sent our emails, but back then absolutely none of that existed and context-sensitive protocol grammars were the way forward.

So how does IMAP work?  An IMAP connection can be in one of four states: Disconnected, Unauthenticated, Authenticated and Selected.  You move between them by connecting, logging in, and choosing a mailbox to read messages from.  The user does this by issuing commands through their IMAP client, to which the server responds.  Each command and response ends in a new line.  The server can (and usually does) respond in two ways.  Each IMAP command has a tag to identify when it, so a response can be sent when its done.  Such notifications are called Tagged Responses and are sent once for each command with a status of OK, BAD or NO and an optional code and server message.  The other type of response is an untagged response.  These usually contain the data we’re interested in and are often encountered in response to a variety of commands.  In fact, the IMAP standard says that some untagged responses can be sent in response to any command, and that you need to deal with them otherwise everything breaks.  So server responses are the first pain, but not too insurmountable.  You just need to check each untagged response to see whether it needs some more general processing first before passing it back to whatever method sent the original command.

So think you’ve got responses done?  Think again.  IMAP is a text-based protocol, so everything is words and servers are expecting nice and lovely text to come flying through in their requests.  Unfortunately the world is not perfect and in a text based protocol you’ll eventually work out that you need to work out which text is protocol text (not-text text) and which text is protocol text (text text) and how are you going to encode the text text to keep it from the not text text?  In IMAP this is solved by one of three means: atoms are single words with no fancy characters in them at all, quoteds are “strings of text within quotes, but only allow the ASCII printable characters”, so ©, ç and ½ are all off limits and literals are just strings of data prepended with their length in {}s, followed by a new line.  You keeping up?  In case you missed it, this now means you’ve got to check whether the new line you thought delimited an untagged response is actually in the middle of a literal value you’re probably interested in.

So that’s the big gotchas with responses.  Now mailboxes!  IMAP says that just because you have a mailbox named INBOX\Computers\IMAP, it doesn’t automatically make Computers a valid up and proper mailbox.  No, in fact if Computers was a valid up and proper mailbox, it would be entirely valid to delete it, but for INBOX\Computers\IMAP to continue to exist at the exact same path.  That struck me as a bit weird.  What else is weird is that mailbox names also need to be encoded into a modified form of UTF7 encoding.  For those of you not up on the latest trends in textual encoding, as mailboxes are likely to contain interesting characters, and IMAP servers are unlikely to work with these interesting characters, they need to be converted to UTF7, which is a way of representing Unicode characters using just the printable ASCII characters.  Beautiful eh?

So having finally gotten your head around some of the interesting details of the IMAP protocol, you now need to find a library.  Clearly the language of choice for PeerGroup is going to be C# – it’s going to be a Windows-based desktop app that’s needs to look pretty and C# ticks all those boxes whilst also pretending to be portable with Mono.  But you then realise that there isn’t a single good quality IMAP Library around.  Sure, a Google search will pull up a few, but in no way are any of these serious enough for PeerGroup, which will need to support caching of mailboxes, make use of as much data from the server as possible and have a good, solid object model.  The good object model in an IMAP Library is key to hiding some of the hideous detail of the IMAP protocol.  I shouldn’t need to know that mailbox names should be encoded first, I should be getting some kind of Mailbox class from a List method and I shouldn’t have to be forming my own commands.  On the one hand, this suprised me, as there’s good C# libraries for doing other things, but I guess people aren’t so interested in writing implementations of protocols, for whom things like Linux, Active Directory and PowerBuilder are new, sexy and exciting technologies.

I would conclude by saying that, although IMAP is old, and has its fair share of gotchas, it is well documented and implemented in a wide variety of languages and frameworks.  In fact, its the wide support that got me thinking about PeerGroup.  Nowadays of cause we would do IMAP differently, but that’s hindsight for you, some of which was probably gleaned from the gotchas in IMAP and other protocols of the late 80s\early 90s, although it is nice that IMAP isn’t just another HTTP-based RESTful protocol serving up its own horrors of XML and JSON.  But that doesn’t stop it from being really annoying, vague and open at times.