Jon Udell: NNTP, IMAP, and the Semantic Web

Tangled in the Threads
Jon Udell, April 25, 2001
NNTP, IMAP, and the Semantic Web

More thoughts on our messaging dilemma

Email is the root cause of infoglut. It's also the only conceivable solution.

An acquaintance who runs a software company wrote recently -- in email -- to complain about information overload. The problems are familiar to us all:

"the information deluge is preventing us from working effectively"

"because customers invariably use email, we can't use software logic to help solve their problems"

"the current mode of internal and external mail management is a patchwork of half-done solutions"

"only a few of the documents written in the company are stored in a way visible to others"

Another acquaintance, in another company, says these problems so exhaust him that he sometimes wants to throw in the towel.

I feel their pain! I've just learned that my book, Practical Internet Groupware, is out of print. It will remain available on Safari, and I know it has affected some people deeply, but it didn't have the mainstream appeal I'd hoped for. One of the reasons for that, I think, is that I focused heavily on NNTP discussions. In the mid-1990s, this technology reinvented itself in remarkable ways. But ultimately, it could not separate itself from its historical roots.

There was a method to my madness. I've always thought -- and I still believe -- that an installed base is the most precious commodity in the software universe. By 1995, the installed base included not only the ubiquitous browser and email client, but also their lesser-known cousin, the newsreader. And this, I was delighted to discover, was much more than a conventional USENET client. It could connect promiscuously to an array of private and public discussion servers. It could connect securely using password authentication, optionally strengthened with SSL. The Netscape version could perform fulltext searches against the companion Netscape news server. This new breed of newsreader offered, in short, the vision of an open-standards "Lotus Notes for the rest of us."

What made this so compelling, to me, was the deep integration of the obscure newsreader with the all-important email client. Although these two applications use different protocols, they share a common environment for reading and writing messages. This was crucial, and remains so for any solution to our messaging dilemma. People live, and work, in the email environment. A groupware solution that asks people to leave that environment and go elsewhere -- to an issue tracker, a contact-management database, a calendar, a shared file repository -- is asking for trouble.

Here's an example of the kind of problem that email/newsgroup synergy can help solve. You receive an email from a new acquaintance, and you want to publish that person's contact information to your team. Even if your team has a shared contact manager available on the LAN or the Internet, the odds are slim to none that you will switch from your email application to that other application in order to enter that data. Why? Because this is only one of the hundred messages that you have to plow through this morning before you focus on whatever it was you really set out to accomplish. Disruption of context is unacceptable. Any action within the email context that will take more than two seconds is unacceptable. So, if this contact information is going to be shared at all, it will be shared by forwarding the message to the team. We've all received too many such messages. They're painful because, unless you have immediate need for the information, it's just distraction and clutter. Should you file such a message? Transfer the information elsewhere?

Advantages of a centralized group message store

The solution that I hit upon, when I ran a project team and controlled its LAN/extranet environment, was to create a searchable newsgroup for these contact records. It was then possible to use email-like protocols to share the information in a more orderly way. The most succinct protocol is to cc: the newsgroup when replying to the sender. This method has the following advantages:

Minimal time/effort. It's the same as just replying to the email message, but for the mention of a newsgroup on the cc: line.

Appropriate prioritization. Odds are this isn't something your team needs to know right way. If you cc: the message to team members as email, you're placing a greater demand on their scarce attention than the situation warrants. A newsgroup that's checked once or twice a day is an appropriate lower-priority channel.

Notification. If you were to simply enter the contact record into a database, you'd be missing the opportunity to alert the team that this record has been inserted into the database. This notification is crucial. You want your team members to know that you've been contacted, by whom, and for what purpose. The trick is to navigate between Scylla (emailing everything to everybody all the time) and Charybdis (not communicating things that other people need to know). A team newsgroup, with its lower priority, can help you strike this delicate balance.

Centralized storage. Storage of email messages can be centralized, but typically is not. A given message might be in any of these places: a disk on my notebook or desktop PC, a server, a disk on your notebook or desktop PC. It is rarely guaranteed to be in a place that's accessible to current and future group members. In the case of a private, non-expiring newsgroup, a given message is guaranteed to be in at least that one canonical place, although (as with email) it may also be on one or more local disks. This has all sorts of powerful implications. The canonical message store is, for example, URL-addressable. Information in it can be shared "by reference" (i.e., sending an URL), rather than "by value" (i.e., sending a copy of the message). That means that follow-on collaboration can then be captured in the central store. I hate to use the phrase "knowledge management" because it's so overhyped and misunderstood, but if there going to be any such thing, it has to begin with this kind of information capture.

Drawbacks to quick-and-easy email/newsgroup integration

This quick-and-easy "cc the message to the newsgroup" technique has, admittedly, some disadvantages.

Poor privacy/security. The Netscape newsreader adds a Newsgroups: header to the message, with the name of the group to which you are "cc'ing" the message. Unfortunately, this header appears in the email reply, and your email correspondent is then alerted to the existence of what you meant to be a private newsgroup. This could easily have been solved with a newsgroup variant of the Bcc: protocol, but evolution along this path stalled and that never happened.

Can't label or contextualize. It's quick to combine the email reply and the newsgroup posting into a single action. But if you do, they'll share a Subject: header. This is not ideal. The appropriate Subject: header for the email reply might be: "re: our meeting on the 24th." The appropriate Subject: header for the newsgroup posting would instead be something like "John Doe, XYZ Corp., marketing manager." My solution to this problem was, in fact, to split the single action into two parts. First, I'd reply to the email message in the normal way. Then, I'd forward the same email message to the newsgroup. That gave me the chance to properly form the all-important message title, and also to frame the message -- that is, to say things about this contact that I wouldn't tell the person, but would want to tell the group (and my future self). This made a 2-second operation into a 5-second operation, which is (I'm not kidding) a serious impediment. But at least it didn't require a catastrophic switch to another application. Here too, had evolution not stalled, there might have been an easy fix. Along with a Bcc-to-newsgroup: option, why not also Subject-for-newsgroup: and Summary-for-newsgroup: options?

No structured querying. This technique doesn't enable a formal search for all contacts located in, say, a particular zip code. But in practice, that isn't the main limitation. Once you've got the stuff, you can combine fulltext search of unstructured data with navigational views of semi-structured data (by date, by sender, etc.) to pretty effectively find whatever is there. The real problem is that, typically, there isn't any "there" there. Nothing is captured in the first place, so there's nothing to find.

NNTP vs IMAP

The NNTP techniques that I have used in the past, and that I advocated in my book, have clearly not taken the world by storm. The subject came up this week in the BYTE.com discussion groups:

Mark Wilcox:

Nobody, not even geeks, really wants NNTP any more. Sites like SourceForge could have added NNTP. But nope, they give you email lists and web discussion forums; most traffic is on email.

One reason for this is the relative difficulty of managing multiple group memberships in NNTP space, versus in email space:

Andrew Ducker:

With news, you need a different newsgroup for each group of people. This is fine in-house, but of very little use when they are spread all over. It's fundamentally open, so if you want to reach people spread all over the place, you either need to run your own private newsgroups on a server (in which case your clients need to be set up for each news server they have groups on), or you'll be allowing anyone at all to read your private messages.

Also, I can pick up my email from 24 groups (I run 7 and belong to 17) anywhere by simply setting up a client to pick up my mail. To do the same for 24 newsgroups is trickier by far.

These problems could have been solved in NNTP space, but probably that won't happen now. Of course the email equivalent to NNTP -- IMAP with public folders -- isn't a slam dunk either. In the last five years, IMAP-based collaboration has made made no more progress than NNTP-based collaboration has.

Mark Wilcox:

At UNT [University of North Texas] we tried the public folders strategy. The biggest issue isn't IMAP, it's the fact that most people don't even know about folders, neither their own nor public ones. So getting people to use those folders is a daunting task.

I think clients are going to have to recognize this fact and make it easier to do. For example, by having a button marked "Lists" or "Discussions."

Writing the Semantic Web

Mark's proposal implies enriching the context in which email communication occurs. Separating interpersonal from group messages only scratches the surface. Much more is possible, and is desperately needed. But richer modes of collaboration, though supported in some ways by current technologies such as NNTP and IMAP, are only slowly sinking into mainstream consciousness. To speed up the process, we need to give people a tool that makes collaboration an automatic side-effect of doing things that they do anyway.

Consider Napster. Would it have taken off in the same way, had the default been not to share files? In fact, you could not use Napster without adding value to a shared repository.

The next generation of email needs to have this quality too. We need an app that makes central storage, reliable threading, fulltext indexing, foldering, metadata tagging, and other collaboration-enabling stuff just happen as a side-effect. To the user, it should only look like a cool new email/IM program. So cool that they install it in droves, whether or not IT wants them to.

Tim Berners-Lee and some of his colleagues have a wonderful vision called the Semantic Web. You can read about it in their recent article in Scientific American. I'm all for it. But it struck me as odd that there was only one passing reference to how the Sematic Web will be written. Describing a document which "knows" about people, organizations, dates, and relationships among these entities, the article notes in passing:
"...these semantics were encoded into the Web page when the clinic's office manager (who never took Comp Sci 101) massaged it into shape using off-the-shelf software for writing Semantic Web pages."

Whoa! This isn't just a detail to be dismissed with casual handwaving. Writing software is the key to this kingdom. And it will have to be something that people perceive to be email, or very like it. Because that's what we use.

Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He is the author of Practical Internet Groupware, from O'Reilly and Associates. Jon now works as an independent Web/Internet consultant. His recent BYTE.com columns are archived at http://www.byte.com/index/threads

This work is licensed under a Creative Commons License.