TidBITS#675/07-Apr-03 ===================== Remember Glenn Fleishman's bandwidth nightmare? It's over now - read on for our look at the final bill and what Glenn learned in the process. Adam offers a look into our deliberations about our next generation content management system, and Matt Neuburg reviews his current favorite digital shoebox - Casady & Greene's iData Pro X. In the news, Apple announces a slew of high-end digital video editing tools and SETI@home 3.08 closes a security hole. Topics: MailBITS/07-Apr-03 Final Cut Pro 4, DVD Studio Pro 2, Shake 3 Announced Help Us Choose Among Content Management Systems The Boy Who Cried Bandwidth The Digital Shoebox: iData Pro X 1.0.5 Copyright 2003 TidBITS Electronic Publishing. All rights reserved. Information: Comments: --------------------------------------------------------------- This issue of TidBITS sponsored in part by: * READERS LIKE YOU! Help keep TidBITS going via our voluntary <------ NEW! contribution program. Special thanks this week to Amy Teel, Michael Edmondson, and Michael Luskin for their kind support! * SMALL DOG ELECTRONICS: eMacs Available Now!<----------------------- NEW! eMac G4/700 128/40/CD Only $699! eMac G4/700 CD-RW Only $899 eMac G4/700 128/40/Combo Only $959! Visit: - 802-496-7171 * SIX DEGREES puts your email back to work for you. It eliminates <-- NEW! the need to refile email or search for that elusive message. Your email isn't going away, so you may as well put it to work. ========> * Memory To Go -- We have RAM for all your Macintosh memory needs! iBook/Titanium -------------> 256 MB: $40 or 512 MB: $112 PowerBook G4 (PC2100 DDR) --> 256 MB: $37 or 512 MB: $109 Mention TidBITS/$2 off shipping! * Bare Bones Software TextWrangler 1.0 -- New general-purpose and <-- NEW! richly featured tool for composing, modifying, and transforming text stored in plain-text files. For more info, to download a demo, or to purchase: --------------------------------------------------------------- MailBITS/07-Apr-03 ------------------ **SETI@home 3.08 Security Fix** -- The distributed computing project SETI@home has released version 3.08 of their client software to eliminate the possibility of a buffer overflow error in the networking code that could result in a security hole. Personally, I've given up using SETI@home as a screen saver; instead I run it as a hidden application all the time, quitting it whenever I'm doing something that requires the full CPU power of my dual 1 GHz Power Mac G4 (Apple's CPU Monitor utility shows that my standard activities use only a fraction of the available processing power). If you run SETI@home, consider joining the TidBITS team, which is currently in 143rd place overall. [ACE] Final Cut Pro 4, DVD Studio Pro 2, Shake 3 Announced ---------------------------------------------------- by Jeff Carlson Apple grabbed the director's chair at the National Association of Broadcasters (NAB) event in Las Vegas last weekend, announcing updates to its professional line of video editing applications. Headlining the news is Final Cut Pro 4, featuring RT Extreme (real-time compositing and effects for Macs that support it), high-quality 8- and 10-bit uncompressed formats, and 32-bit floating-point-per-channel video processing for higher-quality rendering. Apple plans to ship three stand-alone programs with Final Cut Pro 4: LiveType, for creating animated titles; Soundtrack, for creating music; and Compressor, for batching and exporting to multiple file formats (such as MPEG-2, used in DVD creation, with support for one- or two-pass variable bit-rate encoding). Apple has also thrown in Cinema Tools, a high-end database for managing cut lists, which previously sold for $1,000. Final Cut Pro 4 will also support more advanced audio mixing, offer a customizable interface, and include a host of other improvements. Look for it to be available in June of 2003 for $1,000, or as a $400 upgrade for registered owners of previous versions of Final Cut Pro. Apple also announced DVD Studio Pro 2, a new version of the company's high-end DVD authoring program. It features an improved interface that Apple claims makes it easier to build DVD content, including a new menu editor and timeline-based track editing, a library of professionally designed templates, and the Compressor application that also comes with Final Cut Pro 4. DVD Studio Pro 2 should be available in August of 2003 for $500; as of 06-Apr-2003, you can purchase DVD Studio Pro 1.5 for $500 (down from $1,000), and if you do buy version 1.5, you will be able to upgrade to version 2 for a shipping and handling fee of $30 through Apple's Up-to-Date program. Finally, Apple announced Shake 3, advanced compositing software used for professional special effects and production. Apple expects to make version 3 available in June of 2003. Shake 3 adds new Qmaster network rendering management software, which uses Rendezvous to handle rendering operations across multiple machines, plus features that promise to speed the compositing process. Shake 3 for Mac OS X costs $4,950, includes unlimited render licenses, and must be purchased through an Apple Authorized Pro Film Reseller. Shake 3 is also available for Linux and Irix platforms (where the original program ran before Apple purchased it), but at a cost of $9,900 plus an annual maintenance fee of $1,485. Apple has been aggressively muscling into the high-end video and film market for the past several years. Even if you're not involved with these fields, it will be interesting to watch how Apple's current lineup of beefy professional applications affects the computer, film, and video industries at large. Help Us Choose Among Content Management Systems ----------------------------------------------- by Adam C. Engst I've mentioned in passing that we've been looking into new systems to replace the aging hardware that runs TidBITS (all our servers use pre-G3 PowerPCs) and cobbled-together software (FileMaker, Lasso, HyperCard, AppleScript, and Retrospect). The hardware decision is easy - an Xserve ought to do everything we want and more, and it will slot nicely into a rack at digital.forest, our server hosting service. But the content management software is another story. A content management system is a collection of programs for storing, managing, displaying, and archiving various types of content. A full content management system generally includes a database for storing the content, middleware software for extracting and presenting the content, and a Web server for the actual serving. We have articles, polls, and TidBITS Talk messages, along with collections of those items in issues, article series, and message threads. Plus, we publish our content in different formats (setext, HTML, RSS, and more) and in different venues (email, Web, FTP, Usenet news, and so on). The need for a content management system has become increasingly common as the Web has matured. Content producers realized they needed tools that help put content on a single Web page (Claris HomePage or Symantec's Visual Page, for instance). That expanded to needing tools that help manage an entire site (Adobe SiteMill or the modern day Adobe GoLive or Macromedia Dreamweaver). Now, however, many needs have grown once again to include software that helps to manage content on a regular basis, simplifying the process of adding new content, archiving old content, and providing access to both old and new content in appropriate ways. Everyone's needs are different, of course, but the need for some sort of content management system turns out to be remarkably common. Whether you're an individual trying to maintain a personal weblog or a business trying to maintain a catalog of your products, you need a content management system. That's why so many software packages call themselves content management systems. Some are designed for local newspapers, while others aim at providing information systems for entire college campuses. Some are so proprietary that you can use them only if you contract with the developers to build and host your system, whereas others adhere to open source precepts (unfortunately often including the one about documentation being for losers and fools). Add in a myriad of blogging programs and you can see how difficult it is to find the perfect content management system for a particular purpose. After long discussions about how to plan and create our new content management system, Tonya and I finally decided to work with our friend Keith Kubarek, who had spent 16 years working at Cornell University but recently left to concentrate on his Web design and development company, One Bad Ant. The process has been going well, and it's been tremendously helpful and educational to watch someone else try to figure out precisely what TidBITS does - we're too close to our systems to view them objectively. Keith's business analysis of what we do was particularly interesting for the way it helped us clarify not just what we do now, but those areas in which we hope to do different things in the future. **Initial Technical Analysis** -- Now that we've completed the business analysis, Keith has moved on to the technical analysis, in which he's evaluating existing content management systems to see which deserve further investigation and possible adoption for some or all of our eventual solution. What are our primary needs? Along with the basics of storing, managing, displaying, and archiving content, we need features like rock-solid integration with email for issue distribution, user account management so users can manage their own subscriptions, links between articles and between TidBITS Talk messages, management of sponsorship appearances, unified circulation statistics, support for different issue formats and venues, integration of PayBITS, and more. Trying to figure out which content management packages provide these features (or can be extended to provide these features) is a Herculean task. Luckily, we were able to eliminate many of the seeming contenders based on two simple criteria. * Platform. We will be using a Macintosh and Mac OS X. Others need not apply, and if Unix developers couldn't be bothered to state clearly whether or not Mac OS X was supported, they were dinged instantly too. * Price. We're not opposed to the concept of spending some money, but only within reason, so we ditched packages priced above $2,000 right away (some ran as high as $50,000!). Other pricing problems included annual fees and traffic-based pricing schemes, since our income isn't directly related to our level of traffic. **Narrowing the Field** -- After that initial level of winnowing, we applied additional criteria to narrow the list further. These criteria weren't as all-or-nothing as the previous ones, but if a program failed on several counts, it fell off our list. * Basic coherence. If we couldn't figure out the basic capabilities of the software from its Web site, it was hard to muster enthusiasm for using it. * Documentation. Some programs came closer to being dropped from our list thanks to poor, incomplete, or non-existent documentation. * Email. Many content management systems look no farther than the Web, which may be fine for others. For us, though, email is essential. We don't necessarily expect full mailing list capabilities, but we need some way to send content to subscribers via email, manage subscriptions, and handle bounces. * Customization. This one's a balancing act. We don't want to shoehorn TidBITS into some other publication's mold, but at the same time, we don't want to spend huge amounts of time or money on a custom site written for us from scratch. * Available knowledge. All other things being equal, we'd prefer to use a system that others who we know are also using so we can learn from them or get help as necessary. It's even better if those experts are people who read TidBITS regularly and appreciate what we're trying to do. Open systems generally win out over proprietary systems in this regard. * Maintenance. We want to spend our time researching and writing, not baby-sitting our server. Simply running on an Xserve in Mac OS X should do away with some of the problems we face now, but we don't want to sign up for more if at all possible. * Stability, reliability, and performance. Determining how any given application performs in these respects ahead of time is tricky, but we'll look more at these criteria as we get closer to final candidates. **Current Contenders** -- So here's where you come in - after all, TidBITS readers are the people who will use whatever we end up creating. We've come up with a short list of packages that deserve additional investigation, but we're under no illusion that we've even identified every possibility. If you know of something else that might fit our needs, let us know and we'll check it out. Similarly, if you have educated opinions or deep knowledge about any of these packages, we'd love to know that as well. I plan to be holding these discussions primarily via TidBITS Talk (please send comments to ), so if you want me to keep your message private, just say so and send it to me personally. Here's the list. * Aegir is an end-user content management system based on the Midgard development framework, which in turn relies on Apache, MySQL, and PHP. Although Aegir doesn't talk about Mac OS X specifically; Midgard does, so we assume both should work. * Bricolage is currently used by Macworld magazine as their content management system. It uses the PostgreSQL database to store content. Bricolage is actually built on Mason, which is a Perl-based Web site development and delivery engine. * Cofax is an open source package developed originally for Knight Ridder's newspaper Web sites. Although Cofax's Web site doesn't talk about Mac OS X specifically, Mac OS X-compatible software like MySQL and Mac OS X's Java implementation should be able to meet its requirements, and it was sufficiently interesting to warrant a closer look. * Blue World's Lasso now works with SQL databases, including MySQL. We currently use Lasso with FileMaker Pro, so we might be able to reuse some of that code. Lasso also has tight integration with Dreamweaver and GoLive, which could prove useful. * PHP-Nuke and its offshoot PostNuke are weblog/portal/content management systems that run on PHP-enabled Web servers. They can work with a variety of databases, including MySQL. Both offer numerous pre-built modules for specific functions like polls, searching, site statistics, and so on. * Xoops is yet another PHP-based content management system that can sit on top of a variety of different databases. * Zope is a Web application server written primarily in Python. It includes its own Web server, but can also run on Apache. A content management framework is available for Zope, and a full-fledged open source content management system called Plone runs on top of Zope and its content management framework. * 4D is best known as a database, but the current version is actually an application development and Web development environment so it might be able to meet our needs. **The Next Step** -- We realize that the current set of contenders represents a jumble of options for scripting languages, underlying databases, and supported technologies, and our heads are spinning as we try to analyze the possibilities. So tell us what you think, and we'll be sure to write more about our progress. The Boy Who Cried Bandwidth --------------------------- by Jeff Carlson Two weeks ago, Adam told the story of our colleague Glenn Fleishman's potential bandwidth disaster, in which Glenn gave away a free PDF version of our book Real World Adobe GoLive 6 only to be faced with the possibility of excess bandwidth charges running as high as $15,000. As Glenn's office-mate and co-author of the book, I've had a front-row seat to watch the process of determining the final damage. After a lot of uncertainly and a great outpouring of support from TidBITS readers and GoLive users, the punchline has arrived: the charges are _zero_. **Running the Numbers** -- Level 3, Glenn's hosting company, charges based on the busiest hour of the month, excluding a certain percentage (about 5 percent) to avoid charging for fluke peak hours. According to Glenn's logs and preliminary information from the provider, it looked as though the bill could run as high as $15,000 or possibly higher, due to the sustained download rate of 10,000 retrievals over 36 hours. (If the company charged on pure bytes, not bandwidth, the charges would not have been out of the stratosphere.) Level 3 doesn't have reporting software that makes it easy to understand the usage numerically, and they were unwilling to give final numbers until, ironically, April 1. So how did Glenn squeak by with charges of zero? The provider said that the cutoff point where excessive bandwidth charges would have started to hit was just above the point at which Glenn pulled the file. If the download had been accessible for just a few more hours, Glenn estimates his bill could have run from $2,000 to $5,000, based on the information Level 3 provided. **Passing the Hat** -- Thanks to the generosity of nearly 200 separate donors, Glenn raised about $1,800 between 20-Mar-03 and 01-Apr-03. As noted previously in TidBITS and on the book's Web site, he promised to donate any excess to Project Gutenberg, the grandfather of all electronic book projects on the Internet. Because the bandwidth bill turned out to be zero, Glenn sent email to donors offering refunds via PayPal and informing them that they could request cancellations via the Amazon.com Honor System. So far, only one person has asked for a refund, ironically due to his own unexpected excess bandwidth charges that cropped up last month. A hefty check, including an additional donation from Glenn, will help Project Gutenberg carry out its good work. Does Glenn regret the fund-raising? "No. If I hadn't struck while the iron was hot and the worst-case situation had turned out to be true, I doubt many donations would have come in," he said. "Because I had the backup plan of a charity, suggested by Adam, instead of Level 3 receiving the cash, a good cause will get an unexpected donation." **Lessons Learned** -- It's tempting to apply Glenn's experience to a broader context: Is electronic publishing still not a viable mechanism? On the contrary, Glenn's experience suggests that it can be more effective than one might expect, provided you have realistic expectations. Offering something for free was a more powerful attraction than Glenn anticipated. Further, he found that a complete book is worth more than a full chapter. In previous experiments, making limited content available always resulted in a modest number of downloads, such as when we provided a 128-page excerpt of the book for free. Many of those downloads were hosted on a bandwidth-restricted site, too, such as in our office (with a 768 Kbps SDSL line). Distributing the entire book seemed too good to be true, and people jumped on the download before it went away - which it did, though only briefly. Distributing bandwidth turned out to be an important strategy. Using many sites that aren't charged for bandwidth, such as the Info-Mac Archive and Bare Bones Software, who also hosted the PDF, has so far resulted in at least another 6,000 downloads (that Glenn can track directly) of the book. But bandwidth aside, Glenn's experience reinforces the importance of never underestimating the kindness of others. "It's hard to be a stranger when you're on the Internet all the time," he said, "and hundreds of individuals viewed me and my problem most kindly. Thank you all." The Digital Shoebox: iData Pro X 1.0.5 -------------------------------------- by Matt Neuburg What's in your digital shoebox? You know, the place where you stash those pesky snippets of pure text, be they a few words or many paragraphs, snippets that you know you'll need later but you just can't categorize. A database would be overkill; an outliner's hierarchy would be useless. So you just toss them into a virtual pile, a deliberate mass of miscellaneous clutter - a digital shoebox. Here's some of what's in mine: Tips on what certain Mac OS X keyboard commands do; some Unix phrases I need to utter in the Terminal now and then; someone's opinion of what router to buy, copied from a Usenet newsgroup; directions to my house, fit for emailing to visitors; the URL of something I'm selling on eBay; and my vacuum cleaner model number. How miscellaneous can you get? Yet I can lay my hands on any of them instantly. Now, I am, as you know, hopelessly obsessed with storage and retrieval of information. I like hierarchies, hyperlinks, keywords, and categories. A digital shoebox is the opposite of all those things! Nevertheless, there are times when simpler is better; and the excellence of a true digital shoebox is to be really, really simple. In fact, there seem to be just two main requirements for good shoebox-hood: a very plain interface and a very fast Find. My digital shoebox is currently iData Pro, from Casady & Greene. This program has a venerable lineage, having appeared under such previous names as InfoGenie and QuickDex. How venerable? Well, QuickDex was a desk accessory, if you remember what those were, and my research trail, which shows that it had a vociferous cult following, turned up the fact that it was first released in 1987. Let's just say that iData Pro is a Mac OS X incarnation of a very old favorite. The main reason iData Pro meets the requirements for shoebox excellence is that its files are just text, nothing more. One font, size, and style apply to each file when it displays. The only unusual feature of the file is the ASCII 06 control character used to separate entries. If iData Pro went on the fritz tomorrow, all my data would still be sitting there, plain as day, perfectly legible in BBEdit or Microsoft Word. And when a file opens, the whole thing loads into memory, so searching for specific text is extremely fast. **Free-form Files** -- iData Pro can make two kinds of files; you specify which you want when you create a file. The first, and most shoebox-like, is the free-form file. What you see when you look at a free-form file is the text of one entry, and that's all - the entry has no title, no keywords, no identifying marks of any kind other than its content. There is also a Find field at the top of the window. The entry content area is a decent text editor. Keyboard navigation and selection, and drag & drop, all work as expected. Option-Tab jumps between the Find field and the entry content area. In this window, you can add a new entry, or delete the current entry. You can navigate to the next or previous entry, or the first or last entry; but mostly you won't do that except to get an overall sense of what's in the file. Instead you'll navigate by finding: In the Find field, you'll type a word and hit Return to jump to its next occurrence in the file, in the same or another entry. One is reminded (and I'm not the first to make this comparison) of navigation and finding in HyperCard. That's basically all there is to it. You add an entry, enter some text, and return to it later by remembering some word within that text. In practice you'll probably use your knowledge of your own mental processes to make sure the text contains words you're likely to think of when seeking it later. For example, when I pasted in the opinion about a good router, I deliberately typed "opinion about good router" at the start of the entry, because the word "router," which I would expect to search on later, didn't occur in the pasted material. The fact that you see just one entry at a time, by the way, is not generally problematic. You can open a second (read-only) copy of a file, letting you see two entries at once. Also, instead of finding successively, you can winnow the set of currently available entries (called "selecting"); subsequent selecting can be from the currently selected set of entries or from its inverse. Thus, one way or another, you'll have a pretty good sense of what's in a file, and you can narrow in on the entry or entries you're looking for quickly and easily. **Field-Based File** -- A field-based file looks a little different from a free-form file: every entry consists of several named fields. When you create a field-based file, you declare these field names. (Don't worry; the names and order of the fields can be changed later, fields can be added and removed, and so forth.) There is thus some superficial similarity to a flat file database, but this similarity really is superficial, since ultimately there's no difference between iData Pro's two file types. The difference lies in how you're shown the text of the file; in a field-based file, each paragraph is portrayed as a separate field. Thus, no field can consist of multiple paragraphs except the last one. How does a field-based file's window, and what you can do there, differ from a free-form file? First of all, the field names appear down the left side of the window. You can tab from field to field, entering or modifying text. Also, you can switch the window to a "list view," a grid of cells with each row representing one entry and each column representing one field; you can specify that certain fields shouldn't appear in list view. You can do a Find or Select that looks in just one particular field. And you can sort on one or two fields; you can sort a free-form file too, but less powerfully, and you're less likely to want to. (Actually, even in a free-form file, if entries have a consistent structure, you can use that structure to some extent when sorting and selecting.) What are some candidates for a field-based file? An address book (last name, first name, address, address line 2, phone number, and so on) is an obvious example. The first field-based file I actually made was a holiday gift list; the fields were the person, the gift, whether I'd bought it yet, whether I'd sent it yet, and a notes field (a common use for the last field, because it can have multiple paragraphs). It would have been possible to use a more powerful program for this purpose - a database, or a spreadsheet - but for something so basic, iData Pro's simplicity was perfect. **Other Features** -- If all your files live in the same place, they all appear in a special menu (the Datafiles menu), from which you can open any of them. Files thus become a level of categorization within the total mass of your data. Also, particular files can be set to open automatically whenever you start up iData Pro, so that your most commonly needed data is always accessible. You can export and import data. Mostly you'll use tab-delimited format, but iData Pro also knows about the format of some common email programs and can import entire mailboxes. Also, iData Pro has a built-in notion of extracting an address, so that if you have, say, a field-based address book file, you can dictate how to assemble the various fields to make an address. Even more important, iData Pro is scriptable, so this behavior, as well as other tasks, can be even more precisely customized. iData Pro can dial a selected phone number, through your modem, in various ways that you can configure; indeed, the program turns out to be extraordinarily good at this. You can print labels and envelopes, using templates that you can configure. Email addresses and Web URLs can be live links, so that clicking them creates an email message or goes to a Web page in your preferred helper application. There's also hot-syncing to your PDA, but I don't have a Palm to test this. **Conclusions** -- iData Pro does have some irksome shortcomings, having mostly to do with how it has been ported to Mac OS X. For example, its windows don't respond to my mouse's scroll wheel, making it just about my only remaining program with this problem. Also, iData Pro's notion of text is unnecessarily primitive; it knows nothing whatever of the rich Unicode world that surrounds it, and can't access more than the first 230-odd characters of whatever font is used to display a file. To rewrite so simple a program in Cocoa, as a true Mac OS X native application, wouldn't be difficult, and I hope that Casady & Greene will eventually do so. At least the program is actively supported; there's a good bulletin board hosted at the developer's site, and bug fix releases appear quite regularly. iData Pro costs $40, with competitive upgrades from various other organizer and database programs for $30. A free demo version is available for download. PayBITS: Did Matt's review of iData Pro help you organize your important information? Consider thanking him via PayBITS! Read more about PayBITS: $$ Non-profit, non-commercial publications may reprint articles if full credit is given. Others please contact us. We don't guarantee accuracy of articles. Caveat lector. Publication, product, and company names may be registered trademarks of their companies. This file is formatted as setext. For more information send email to . A file will be returned shortly. For information: how to subscribe, where to find back issues, and more, email . TidBITS ISSN 1090-7017. Send comments and editorial submissions to: Back issues available at: And: Full text searching available at: -------------------------------------------------------------------