Tangled in the Threads

Jon Udell, April 25, 2001

NNTP, IMAP, and the Semantic Web

More thoughts on our messaging dilemma

Email is the root cause of infoglut. It's also the only conceivable solution.

An acquaintance who runs a software company wrote recently -- in email -- to complain about information overload. The problems are familiar to us all:

Another acquaintance, in another company, says these problems so exhaust him that he sometimes wants to throw in the towel.

I feel their pain! I've just learned that my book, Practical Internet Groupware, is out of print. It will remain available on Safari, and I know it has affected some people deeply, but it didn't have the mainstream appeal I'd hoped for. One of the reasons for that, I think, is that I focused heavily on NNTP discussions. In the mid-1990s, this technology reinvented itself in remarkable ways. But ultimately, it could not separate itself from its historical roots.

There was a method to my madness. I've always thought -- and I still believe -- that an installed base is the most precious commodity in the software universe. By 1995, the installed base included not only the ubiquitous browser and email client, but also their lesser-known cousin, the newsreader. And this, I was delighted to discover, was much more than a conventional USENET client. It could connect promiscuously to an array of private and public discussion servers. It could connect securely using password authentication, optionally strengthened with SSL. The Netscape version could perform fulltext searches against the companion Netscape news server. This new breed of newsreader offered, in short, the vision of an open-standards "Lotus Notes for the rest of us."

What made this so compelling, to me, was the deep integration of the obscure newsreader with the all-important email client. Although these two applications use different protocols, they share a common environment for reading and writing messages. This was crucial, and remains so for any solution to our messaging dilemma. People live, and work, in the email environment. A groupware solution that asks people to leave that environment and go elsewhere -- to an issue tracker, a contact-management database, a calendar, a shared file repository -- is asking for trouble.

Here's an example of the kind of problem that email/newsgroup synergy can help solve. You receive an email from a new acquaintance, and you want to publish that person's contact information to your team. Even if your team has a shared contact manager available on the LAN or the Internet, the odds are slim to none that you will switch from your email application to that other application in order to enter that data. Why? Because this is only one of the hundred messages that you have to plow through this morning before you focus on whatever it was you really set out to accomplish. Disruption of context is unacceptable. Any action within the email context that will take more than two seconds is unacceptable. So, if this contact information is going to be shared at all, it will be shared by forwarding the message to the team. We've all received too many such messages. They're painful because, unless you have immediate need for the information, it's just distraction and clutter. Should you file such a message? Transfer the information elsewhere?

Advantages of a centralized group message store

The solution that I hit upon, when I ran a project team and controlled its LAN/extranet environment, was to create a searchable newsgroup for these contact records. It was then possible to use email-like protocols to share the information in a more orderly way. The most succinct protocol is to cc: the newsgroup when replying to the sender. This method has the following advantages:

Drawbacks to quick-and-easy email/newsgroup integration

This quick-and-easy "cc the message to the newsgroup" technique has, admittedly, some disadvantages.

NNTP vs IMAP

The NNTP techniques that I have used in the past, and that I advocated in my book, have clearly not taken the world by storm. The subject came up this week in the BYTE.com discussion groups:

Mark Wilcox:

Nobody, not even geeks, really wants NNTP any more. Sites like SourceForge could have added NNTP. But nope, they give you email lists and web discussion forums; most traffic is on email.

One reason for this is the relative difficulty of managing multiple group memberships in NNTP space, versus in email space:

Andrew Ducker:

With news, you need a different newsgroup for each group of people. This is fine in-house, but of very little use when they are spread all over. It's fundamentally open, so if you want to reach people spread all over the place, you either need to run your own private newsgroups on a server (in which case your clients need to be set up for each news server they have groups on), or you'll be allowing anyone at all to read your private messages.

Also, I can pick up my email from 24 groups (I run 7 and belong to 17) anywhere by simply setting up a client to pick up my mail. To do the same for 24 newsgroups is trickier by far.

These problems could have been solved in NNTP space, but probably that won't happen now. Of course the email equivalent to NNTP -- IMAP with public folders -- isn't a slam dunk either. In the last five years, IMAP-based collaboration has made made no more progress than NNTP-based collaboration has.

Mark Wilcox:

At UNT [University of North Texas] we tried the public folders strategy. The biggest issue isn't IMAP, it's the fact that most people don't even know about folders, neither their own nor public ones. So getting people to use those folders is a daunting task.

I think clients are going to have to recognize this fact and make it easier to do. For example, by having a button marked "Lists" or "Discussions."

Writing the Semantic Web

Mark's proposal implies enriching the context in which email communication occurs. Separating interpersonal from group messages only scratches the surface. Much more is possible, and is desperately needed. But richer modes of collaboration, though supported in some ways by current technologies such as NNTP and IMAP, are only slowly sinking into mainstream consciousness. To speed up the process, we need to give people a tool that makes collaboration an automatic side-effect of doing things that they do anyway.

Consider Napster. Would it have taken off in the same way, had the default been not to share files? In fact, you could not use Napster without adding value to a shared repository.

The next generation of email needs to have this quality too. We need an app that makes central storage, reliable threading, fulltext indexing, foldering, metadata tagging, and other collaboration-enabling stuff just happen as a side-effect. To the user, it should only look like a cool new email/IM program. So cool that they install it in droves, whether or not IT wants them to.

Tim Berners-Lee and some of his colleagues have a wonderful vision called the Semantic Web. You can read about it in their recent article in Scientific American. I'm all for it. But it struck me as odd that there was only one passing reference to how the Sematic Web will be written. Describing a document which "knows" about people, organizations, dates, and relationships among these entities, the article notes in passing:

"...these semantics were encoded into the Web page when the clinic's office manager (who never took Comp Sci 101) massaged it into shape using off-the-shelf software for writing Semantic Web pages."

Whoa! This isn't just a detail to be dismissed with casual handwaving. Writing software is the key to this kingdom. And it will have to be something that people perceive to be email, or very like it. Because that's what we use.


Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He is the author of Practical Internet Groupware, from O'Reilly and Associates. Jon now works as an independent Web/Internet consultant. His recent BYTE.com columns are archived at http://www.byte.com/index/threads

Creative Commons License
This work is licensed under a Creative Commons License.