Tangled in the Threads
Jon Udell, April 10, 2002Zope Lessons Learned
For over a year now, I've been tinkering on and off with a small groupware application based on Zope. It's used by the editorial team of a monthly magazine to centralize the many versions of files that precede the final versions sent to production, and to manage a shared calendar. I sometimes think of turning this app into a Zope plug-in (aka Product) that could be used by similar teams. Laziness aside, there are two things standing in my way. First, it's not a 100% pure Zope (or Python) app. When a Perl script spawned by the cron daemon was the easiest way for me to do something, I took the path of least resistance. So while Zope itself is wonderfully portable, I'd have to rework some pieces -- for example, I'd probably want to replace my use of cron with Zope's Xron scheduler. I'd also need to learn (and perhaps require administrators to learn) Xron's un-cron-like syntax. This reminds me of the language versus environment discussion from an earlier column. It seems silly to recreate, in a scripting platform, services that already exist in the environment. On the other hand, since those services aren't guaranteed to exist in the same way in every environment, you can argue that the scripting platform should be self-sufficient even if that means re-inventing wheels. Stuck on the horns of this dilemma, I've not fixed what wasn't (for me) broken.
A second factor that keeps my from trying to package and distribute this code is my sense that groupware built for one team won't satisfy another. Working on this app has been an iterative process. I had a rough idea of what would work; users had their own different ideas; the team itself has morphed since we started. The result still doesn't satisfy everyone all the time, but it strikes a pretty good balance between what I saw was possible and practical, and what the team collectively discovered that it wanted. My role is that of toolsmith. I listen to feedback, and spend a spare hour now and then working on improvements. I've long believed that the ideal ratio of knowledge workers to toolsmiths is about one to seven. In other words, in every small workgroup, there ought to be one person who pays special attention to infrastructure, tools, and data. Ideally that person is a team member who is mainly engaged with team responsibilities. That's what makes a part-time toolsmith really tune into the needs of the group.
Desktop software has, for years, aimed to empower toolsmiths working as members of teams. There's often one person who learns to customize spreadsheets, or write word-processor macros, to encapsulate common idioms. That role, in a team of networked knowledge workers, becomes more challenging. Scripting a database-backed web application server is not (yet) something many people are able to do.
Zope appealed to me for lots of geeky reasons: its object database, its unique model of inheritance (by environmental acquisition), its deep scripting orientation. But it also appealed to me for a distinctly non-geeky reason: its web-based management UI. Right out of the box, Zope empowers a motivated non-programmer to build simple content-sharing applications, and to manage content repositories through the web with a great deal of power and sophistication.
Looking back, I'd have to say the outcome has been mixed. The app works well, and is used every day. The division of labor between the parts I've scripted and the native management features of Zope has been a huge win. Using role-based permissions, I selectively allow a subset of users to delete or cut-and-paste files and folders, and maintain data structures (lists of keywords, user attributes) that drive the app. This eliminated the need to write a bunch of application code.
So what's the problem? It's taken me a long time to get comfortable with some of Zope's basic idioms. Maybe passing a few of them along will help you ramp up more quickly. Here, then, are a few of the lessons I've learned along the way.
How to use the Zope Catalog
Every groupware application needs to be able to focus attention on recent activity. Mine does this with a Wiki-style RecentChanges page. It's based on a Zope Catalog that stores metadata about the files uploaded to the repository. I created it by adding a ZCatalog object to the root folder of the repository. I populated it, initially, by using the Find Objects tab on the management page of my catalog instance. I specified objects of type File and Image, the two kinds of things that users upload and download. By default, the catalog will index the id, metatype, and modification time of found objects -- which is all that's needed for a RecentChanges view. It took me a while to get the hang of using the Catalog; maybe this explanation will help you cut to the chase.
Here's the method that searches the catalog, and delivers the most recent 30 changes:
results = context.myCatalog (sort_order='reverse',sort_on='bobobase_modification_time')
list = []
for i in range(30):
result = results[i]
obj = result.getObject()
if ( obj != None ):
creator = obj.getProperty('creator')
url = result.getURL()
path = result.getPath()
time = result.bobobase_modification_time
list.append( [ '<a href="' + url + '">' + path + '</a>', creator, time ] )
return list
Because this method is a Python script, it automatically receives a context object, and it's through that object that the script accesses the catalog object. A call to the catalog object produces a list of results, reverse-ordered by modification time. Each item in the list is what's called (rather confusingly) a Catalog Brain -- it's an object you can use to get at the item's found object, its URL, and its path. Why check for an empty Brain -- i.e., obj != None? It's possible that a catalogued object will have been deleted.
The catalog searcher returns a list-of-lists. Another method processes this structure into an HTML table. Interestingly, the searcher may be called directly from the URL-line, and will return a text representation of the list-of-lists. This is handy for debugging and development. But it also makes the method into a kind of poor man's web service, a pattern that holds generally true in Zope. Although in this case I feed the searcher's output directly into a formatter, any application authorized to talk to my server could retrieve that output directly.
Since files are constantly being added to the repository, it's necessary to repeat the Find Objects procedure in order to get these new files into the catalog. An hourly refresh is sufficient for my purposes. Here's a case where I've used a Perl script, running from cron, to automate a scheduled operation. That's possible because, like any web application, Zope's HTML forms can be reduced to URL-line-callable functions. Here's part of the Perl script that updates the catalog:
my $req = new HTTP::Request 'GET';
$req->url("http://host:port/repository/myCatalog/manage_catalogFoundItems? \
obj_metatypes:list=Image&obj_metatypes:list=File&obj_mspec=%3c \
&obj_permission=Access+contents+information&search_sub:int=1 \
&btn_submit=Find+and+Catalog");
I admit to feeling vaguely guilty for doing things in this non-Zopish way. Perhaps if Zope were to export its management UI by way of WSDL, discovery and use of the API would feel less sneaky. Certainly it would be easier. I'm fairly certain that were Zope designed from scratch today, its API would be SOAP/WSDL accessible, and the extensive HTML management UI would be more cleanly separated from the API. As it stands, the API is rather deeply entangled with its HTML manifestation. That said, it's really not hard, as you can see, to automate use of the HTML-based API.
Using Python, instead of DTML, for a tree control
In an earlier column I showed how to customize the display of Zope's <dtml-tree> tag, in order to sort differently-typed levels in appropriate ways. It worked fine, but the outline view at the heart of my application -- which had already been a bit sluggish -- became unacceptably slow. After exhausting the options available in Zope's DTML (Document Template Markup Language), I decide to scrap <dtml-tree> and recreate the behavior I needed in a Python script.
It worked out nicely, and sped things up a whole lot. The script, myTree, is called initially as the URL http://host:port/repository/myTree. It composes descendant URLs on the same pattern, to navigate deeper into the tree, and decorates rows of output with links to level-appropriate functions. I'll walk you through some of the crucial idioms.
We start by querying for Folder, File, and Image objects, initializing some variables, and saving shortcuts to some methods.
l = context.objectValues(['Folder','File','Image']) # list the objects at this level storyFolder = 0 quote = context.pyUrlQuote pathPrefix = context.getTreePath parentFolder = context.parentFolder user = context.REQUEST['AUTHENTICATED_USER'] role = user.getRoles()[0]
Now, differentiate between storyFolders (containing files and images) and higher-level folders (containing folders). Sort storyFolders by modification time, other folders by name.
nonfolders = filter ( lambda x: x.meta_type != 'Folder', l) if ( len(nonfolders) == 0 ): l.sort(lambda x, y: cmp(y.getId(), x.getId())) else: storyFolder = 1 l.sort(lambda x, y: cmp(y.bobobase_modification_time(), x.bobobase_modification_time()))
Initialize a list of strings that will be joined to create the output page.
ret = []
ret.append('<html><body>')
Provide a breadcrumb trail and uplink.
path = pathPrefix() + '/' + context.getId() + '/../myTree'
ret.append ('<p>')
parent = parentFolder().getId()
if (parent != ''):
ret.append ( '<a href="' + path + '">' + parentFolder().getId() + '</a> / <b>' + context.getId() + '</b>')
ret.append('</p>')
Now iterate through the list of objects at this level. Indicate whether subfolders are expandable or empty. For each subfolder, form an URL (wrapped around a folder icon) that calls myTree to traverse it. For each file or icon, form an URL (wrapped around the appropriate icon) that invokes the object directly, sending it straight to the browser. For managers only, folders are also decorated with an URL suffixed with '/manage', invoking the Zope management API for renaming, deleting, or copy/paste operations. Finally, emit the page.
for i in (range(len(l))):
o = l[i]
ret.append('<div>')
if (o.meta_type == 'Folder'):
ret.append ('<tt>')
items = len(o.objectIds())
if ( items > 0 ):
ret.append ( ' + ' ) # indicate content inside folder
else:
ret.append ( ' ' ) # indicate folder is empty
ret.append ('</tt>')
ret.append('<a href="' + quote(o.getId()) + '/myTree">') # recurse
else:
ret.append('<tt> </tt><a href="' + quote(o.getId()) + '">')
ret.append( '<img src="/misc_/OFSP/' + o.meta_type + '_icon.gif" border="0" />' )
ret.append(' ' + o.getId() + '</a>')
if ( o.meta_type == 'Folder' and role == 'Manager' ):
ret.append(' <a href="' + o.getId() + '/manage">[manage]</a>')
if ( o.meta_type == 'File' or o.meta_type == 'Image' ):
ret.append ( ', ' + o.bobobase_modification_time().pCommon() + ', ' + o.creator)
if ( storyFolder ):
ret.append(' <a href="' + quote(o.getId()) + '/addFile">[add File]</a>')
ret.append(' <a href="' + quote(o.getId()) + '/addImage">[add Image]</a>')
ret.append('</div>')
ret.append('</body></html>')
return context.REQUEST.RESPONSE.write(string.join(ret,''))
This version is really snappy, a major improvement over its <dtml-tree> predecessor. It isn't much more complex either, and since Python is so much cleaner than DTML I find it easier to work with. In general, I do a lot less in DTML these days, and a lot more directly in Python. There is, of course, an heir to DTML. It's not one language, but three: TAL, METAL, and TALES. Together they attack the problem of blending what designers do, in WYSIWYG tools like DreamWeaver, with what programmers must do to implement those designs. Sounds great, but seems like overkill for the modest design needs of this project, so I haven't tried it. Meanwhile, ditching DTML in favor of Python, for anything non-trivial, has turned out to be the right strategy for me.
The road to enlightenment
I'm pleased to see my little application serving its purpose, day in and day out. Zope has proven itself a robust and reliable engine. The ZODB is storing thousands of files and images, and receiving new content daily, with never a hiccup. Delegating management tasks to authorized users, without having to write code to support those tasks, has been a blessing -- and indeed, this delegation model was central to Zope's design.
Nevertheless, my journey along the road to Zope enlightenment hasn't been smooth or straight. It's taken me a long time to work out the idioms that enable me, finally, to do some simple things in simple ways. There's still much in Zope and in Python that I don't know well and can't do easily. Can this potent combination can be made accessible to a wider audience -- in particular, to the kinds of folks who in an earlier era were the toolsmiths who created and shared wordprocessor or spreadsheet macros? That's an open question to which, someday, I hope the answer will be "Yes."
Jon Udell (http://udell.roninhouse.com/) was BYTE Magazine's executive editor for new media, the architect of the original www.byte.com, and author of BYTE's Web Project column. He is the author of Practical Internet Groupware, from O'Reilly and Associates. Jon now works as an independent Web/Internet consultant. His recent BYTE.com columns are archived at http://www.byte.com/tangled/.
This work is licensed under a
Creative Commons License.