Jon Udell: Email information management

Tangled in the Threads
Jon Udell, May 8, 2002
Email information management

A reluctant trial of Microsoft Outlook proves more congenial than expected

We are creatures of habit. That's true for most things in life, but it's particularly true for our most basic software tools -- namely, the ones we write with. I wouldn't be surprised to learn that there are grizzled journalists still pecking out columns in XyWrite. I am, in fact, pecking out this column in Lugaru Software's Epsilon, an emacs-like program I've used under DOS, OS/2, Irix, and every version of Windows. Nowadays, the software ships as a bundle of executables for Windows, Linux, FreeBSD, OS/2 & DOS.

My long and happy relationship with Epsilon shows no signs of strain. But the same can't be said for another long relationship I've had with Netscape's mail/news client. Because it offers a bunch of features that I care deeply about, I stuck with it (and the entire Communicator 4.x suite) long past the point of diminishing returns. For me, though, the Mozilla mail/news client has failed to equal, never mind surpass, the Communicator version it aims to replace. So recently, I decided to find out what the program used by so many people I know -- Microsoft Outlook -- can do.

First things first

The version I have is Outlook 2000. In order to use it seriously, I had to acquire a digital ID that would enable me to sign (always) and encrypt (rarely) my messages. My failure to accomplish this task in Mozilla's mailer is, in fact, what has prevented me from switching to it from the Netscape 4.x mailer. Activating the S/MIME features in an email client isn't the first thing most people do. In fact, it's something most people never do. But I decided a long time ago it was a good practice to strongly assert my identity in email. I use Thawte's free personal certificates, and my account there has enabled me to generate certificates for many versions of the Netscape mailer, for Outlook Express, and now for Outlook.

Like Netscape but unlike Outlook Express, Outlook can be used according to a policy that requires typing the digital ID passphrase for each outgoing message. I may be the only person on the planet who thinks that's a good idea. I do it for three reasons. First, it makes signing the message seem realer. When I sign a paper document, I always stop and think. Even though I send a lot of emails every day, I like to impose this extra bit of protocol on myself, to underscore that every email is a significant act. Second, it affords a measure of safety. By not letting Outlook remember and reuse the passphrase, I'm making life harder for worms that might want to take over my email client and wreak havoc with it. Third, routine signing disseminates my public key so that correspondents can encrypt mail to me -- though, of course, this rarely happens.

Mail filters in Outlook

In an all-day meeting a month ago I looked over a colleague's shoulder and noticed that more than 200 email messages had spooled up in his inbox that day. Was Outlook's filtering more trouble than it was worth, I wondered? Now, with several weeks of experience under my belt, I can report that while there are many things I'd like to see improved, Outlook's filtering is reasonably competent. I can't imagine letting hundreds of undifferentiated emails land in my inbox.

If you use Outlook or Eudora or any other filter-capable mailer and, like many people, have never tried filtering, there's one pattern that you might find really helpful. I learned this years ago from my former BYTE colleague John Montgomery, who's now a .NET product manager at Microsoft. The idea is simply to redirect anything that is not To:, Cc:, or Bcc: your primary address. Sending this stuff to another folder, which I typically entitle ProbablySpam, diverts the overwhelming majority of spam messages. I still have to look through the ProbablySpam folder because the filter catches some things that really are for me. But the strategy enables me to focus on a much smaller set of important messages in my inbox, and deal with the rest separately.

The steps to create such a filter in Outlook are as follows:

Tools->Rules Wizard->New

Select "Check Messages when they Arrive", click Next twice

Check "Move to the specified folder" (create the folder if needed), click Next

Check "Except if my name is in the To or Cc box", click Finish

Outlook's Rules Wizard is, unfortunately, a fixed-function tool that writes rules to a binary file. Netscape, by contrast, uses a text file with entries like this:
name="ProbablySpam"
enabled="yes"
type="1"
action="Move to folder"
actionValue="Inbox.sbd/ProbablySpam"
condition=" AND (to,doesn't contain,judell@mv.com) AND ..."
This looks like programming but, of course, that's just what rule-writing is. Hiding the text of the "program" behind a fixed-function wizard, and squirreling it away in a binary file, doesn't make the programming any easier. What would, maybe, is a more open format that would make it easier for people to share rules, and scripts for generating rules.

Creating filters in Outlook is almost wonderful. The wonderful part is the Organize wizard which streamlines the process of creating a filter to move messages from some sender into some folder. It spares you the step-by-step interaction with the Rules Wizard, and will act on messages in the current folder as well as messages yet to be received.

Unfortunately the Organize wizard doesn't quite get me all the way. My working life revolves around projects with sets of associated email addresses, and independent colleagues who have individual addresses. Routing messages into folders organized along these lines is really important to me. Outlook's Organize wizard comes tantalizingly close to making that a one-click operation. But there are a few programming details it doesn't seem to handle. One is the loop exit called "Stop processing more rules." Unless I dive into the Rules Wizard and check that box, the same message can end up in two places. The other detail is the order of the rules. The project- and colleague-specific ones need to have a crack at the incoming message first. Only if they fail should the ProbablySpam rule fire. I'm still a relative Outlook newbie. Readers of this column will very possibly show me ways to automate these chores. Meanwhile, despite these irritations, I'm finding Outlook's filtering usable, and quite useful.

There's one very nice feature worth noting. In Netscape's mailer, I never found a convenient way to route replies to folders, when a filter had moved the replied-to message to a non-inbox folder. Outlook can do this handily, though you have to drill down deep for the checkbox. In Tools -> Options -> Preferences -> E-mail Options -> Advanced E-mail Options, you can check the option labeled "In folders other than the Inbox, save replies with original message." This ties conversational threads together, and is an extra incentive to classify inbound messages and move them from the inbox to other folders.

Flexible views

Outlook's offers impressive customization of views of folders. You can include or exclude a vast array of common or obscure fields, and there's an SQL-like ability to group and sort the data. For most folders, I've settled on a simple view that groups messages by thread title, and sorts by date within those groups. In a perfect world, I wouldn't do it this way because each message title (i.e., Subject: header) would be unique and relevant, and threading based on Message-ID: would form a tree of these useful and descriptive titles. But I've given up hoping this will ever happen. In the real world, we're stuck with message grouping based on Subject: rather than Message-ID:, it seems, and in that case I choose to collapse the endlessly repeating Subject: headers into a single group title.

I'm also enjoying the auto-preview folder, which partly expands unread messages so you can peek at the first few lines. This, in combination with foldering, yields an information display that's rich with entry points. The tyranny of the inbox is that it inclines you to process mail in a linear fashion. I'm sure I'm not the only one who's been waylaid by message #2 when I really should have plunged in first to message #19. Diverting messages to folders increases the surface area through which you interact with your mail. Auto-previewing messages does the same thing within folders.

One of the drawbacks of extensive foldering is that you can lose a sense of what messages are current. I was pleased to discover that Outlook's Advanced Find is a very effective way to restore that linear context. I saved an instance of it, configured to search for today's messages and to order them by date and time. Typically I launch that instance from a shortcut on the desktop. Since the view updates itself automatically, as messages flow in and out and are routed to folders, it behaves more like a meta-folder than like a one-time search.

Outlook annoyances

Although my impression of Outlook is more positive than I'd have expected, it has its fair share of annoyances. There's no type-ahead address completion, and no next-message function to advance through unread messages in all folders. To paste an URL inline I have to strip off its protocol prefix, paste, then reattach the prefix. Otherwise, for some brain-dead reason, an URL pasted into a message becomes an attachment. The "friendly name" feature, which hides email addresses, seems like a bug to me. If my address book contains two instances of a name mapped to the same address, there's no easy way to tell them apart.

Administrative chores are needlessly painful. Importing my Netscape addresses was a two-step shuffle that involved exporting from Netscape to an LDIF file, using Outlook Express to convert from LDIF to an Outlook format, then finally importing into Outlook. Of course it didn't work very well, so I had to insert some Perl scripting into the loop. Backup is a nightmare too, because the outloook.pst file lives in a hidden directory. I have a lifetime of software experience, and it wasn't easy for me to figure this out. What do regular folks do?

Perhaps Outlook 2002 solves some of these problems. There is also, of course, a scripting layer in Outlook which I have yet to explore. But annoyances notwithstanding, the truth is that an application lives or dies by what it does right out of the box. And I have to say that if you're somebody who cares about how information is organized, Outlook comes out of the box with a lot of powerful (and fairly accessible) tools.

Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He is the author of Practical Internet Groupware, from O'Reilly and Associates. Jon now works as an independent Web/Internet consultant. His recent BYTE.com columns are archived at http://www.byte.com/tangled/.

This work is licensed under a Creative Commons License.