While making the changes to Mori’s code for 1.7, I encountered some oddities in test results, and it turned out there was a bug which I had introduced in an earlier release. While it doesn’t appear to endanger data in Mori notebooks, it might not return all the results you expect in a search, or in entry summaries.

In addition, it has what I hope are a couple of performance improvements, continued improvements to Italian localization, and a work-around for Leopard’s insistence to treat non-Apple Spotlight metadata files as third-class citizens.

Normally when Spotlight discovers a file has been created or changed, it will ask the responsible program to figure out what’s inside, and feed it back to Spotlight. But one of the drawbacks to Spotlight’s design is it lacks the ability to define containers, or documents which contain logically distinct elements such as the chapters of a book, pictures in a photo album, or entries from a Mori notebook; and which can nest other containers as well. Treating a document as a single entity, Spotlight will open a document at the beginning (or maybe the place where the cursor was the last time it was open), even if what you’re looking for is somewhere near the end.

Because it doesn’t understand that a file can have distinct elements, the development teams for other Apple software (e.g., iPhoto, Safari, Stickies, etc.) came up with a scheme to trick Spotlight by creating new files with the data for those elements. So that’s how Jesse coded Mori’s behavior: duplicate the data for that logically distinct element in its own file. A separate copy of each element’s data in its own file. One extra file per element. That means the space taken up by your data is easily half again more than if Apple just added a container definition for Spotlight metadata (once for the notebook, another for the entry metadata file, and the third copy in Spotlight’s database).

But that isn’t all. While we’d like to keep all those extra files inside a notebook bundle (a directory which Finder treats as a file), because Spotlight treats a document as a single element it won’t look for those files inside the bundle. So Mori creates those files in the metadata cache folder (in your Library/Caches/Metadata folder), along with the metadata files from some of Apple’s programs. If you open the metadata folder and look at these files, you’ll see they have numbers to help Mori figure out which entry contains that data. But when you do a search using the Spotlight menu, and when you select menu item ‘Show All’ and the results are displayed in the Finder, you won’t see the numbers; you’ll see the titles for the entries they represent.

Leopard however, isn’t so democratic; which is why users where complaining about the entries when Leopard was released. First off, it ignores any non-Apple metadata files in the cache folder unless you set your Spotlight preferences to use those files. Secondly, it will ignore the title info embedded in the entry metadata file and just display the file’s actual title, meaning the number. How’s that for Apple undermining the work of third-party developers?

So the workaround I came up with is to add the entry’s title (or Untitled, if it has none) at the beginning of the filename, so you at least have an idea which entry matches your search terms.

(Thanks for wasting about a whole month total of my development time on that alone, Apple. I feel the love.)

I am, of course, more than happy to eat crow should I be proven to be completely mistaken or speaking from out-dated information. It’s easily worth it in order to improve the user experience.

Regardless of the rationale for the design decisions, enjoy, and thank you for being part of the community and continuing to support Mori!

When I purchased Mori, one of the first things I did was generate some documents about its codebase. For this, the main tool I used was Xcode’s Class Model tool to generate charts of the various classes involved. I spent several days laying out the classes on the charts, then printing and then folding and taping the pages together. (This is a process which I am replacing with specific related classes that occupy only one or two pages, so I can keep them in a notebook; or sending a PDF to Kinko’s the next time I need to print the whole chart.) They helped me get an understanding of how code was laid out, and their relationships.

After a while, the tape that held the charts up on the wall would lose their adhesiveness and down they’d come…again and again, eventually falling into disuse. Having gotten a digital camera, I spent a day reassembling the crumpled remains of the charts about a week or so ago, and snapped a few shots which I present here.

There are 33 classes, and 25 protocols defined for the document back-end plugin, and quite a few supplementary category methods extending Cocoa’s foundation classes.

The user interface plugin has 85 classes, 18 protocols, and its own quantity of supplementary category methods.

At least, that’s what Mori’s classes looked like when the shots were taken. Today it looks a tad different.

Oneill, the branch for Mori 1.7 is under active development again and will bring substantial changes to the UI and back-end architecture. You shouldn’t notice any hiccups in the file format, but you should see better functionality and performance.

At least that’s what the testing support should help me do. Thankfully, I make heavy use of the Subversion SCM system. Just in case.

This has taken much longer than I hoped for, and it lacks the features I wanted to add in there, but due to another empty-window bug I’m releasing Mori 1.6.9 today as a stopgap measure.

I had hoped for a quick release to get the fix out to those who are upgrading now (particularly from older, Hog Bay Software versions), but I had upgraded my Xcode installation mid-stream, which is generally a bad idea if you can’t test the effects a tool migration will have on your projects. And Mori is quite unique in the way it’s constructed, being almost entirely plugins, and plugins which run plugins.

It took a week to resolve unit testing problems, some of which were due to Mori’s unique project files, some of which were due to changes in the layout of Xcode’s tools directories and one due to a bug in the unit tests for the ‘Check and Repair Notebooks’ functionality (which I wasn’t going to risk going out with a bug whose cause I hadn’t determined).

Anyway, testing problems aside, I’m packaging the latest for release now. You’ll be able to read the release info shortly.

In case you’re running one of the older releases of Mori, one with a bug in the update mechanism, this is just a brief note to let you know Mori v1.6.8 has now been released!

The fixes include:

• Rewrote user settings handling subsystem (to prevent wayward settings from affecting other operations).
• Improved notebook check and repair functionality to fix orphaned entries and entry data records.
• Fixed stall when dropping items into Mori’s source or entries lists.
• Fixed (or at least improved) Italian localization.
• Fixed faulty version update notification.

The code I had hoped to incorporate into the update, but which didn’t make it (because it was already three weeks or so overdue) included fixes for Smart Folder Rules, non-wildcard searches, and word-count bugs, which are among those to be included in v1.6.9. Also to be included are user-settable font settings in the source and entries views, and the fix for the font prefs for notes.

Work on v1.6.9 has already begun, and my work in v1.6.8 improved the ability to perform testing in new releases. The Beta Test Gang will find the first test version on the site Jan 2nd.

it’s been excessively frustrating and tiring. To cap off the season I was hoping to release the overdue 1.6.8 today, but there are still difficulties in getting the new defaults system working correctly.

The debugger is failing in the middle of an initializer method (self becomes 0×1, and the argument becomes invalid), which typically indicates an invalid pointer, but I’m just using -[NSCharacterScanner scanInt:], which shouldn’t be so problematic. (But it is, after all, Cocoa.) And it turns out that isn’t the correct tactic to use anyway as I want to extract info from the internal version number (MMmmb.bs.rr), not the public (MM.mm.bb[s[rr]]), and Cocoa doesn’t have an API for handling fixed-length records. (Possibly the only functionality from COBOL I actually miss! I mean, there are BCD libraries for C, right?)

So I’ve spent much of the day tussling through docs and links for various RegEx frameworks for Cocoa, and trying to add RegexKit to the Blocks plugin framework; first 0.5beta and then 0.4beta via compilation. Now I’m attempting to use the prebuilt binary, which is problematic because frameworks are normally used by applications, not plugins, so it doesn’t exactly fit right (or run). But as the clock’s been ticking, it looks like it’s time to punt and attempt another tactic by hard-coding a parser for the internal version scheme or using defaults for the optional fields in the public version scheme.

This is just to fix the bugs in the version class I added in an earlier version, which I have to get out of the way to continue fleshing out the revamped defaults system I created with unit tests and component code in order to release! Argh!

It hasn’t been all sweat and tears, though. I went through a period of idea generation (both software and marketing/entrepreneurism) early during the process and also, last night as I was organizing several years of accumulated papers, bills and what-not, found several new UI ideas I had come up with (and forgotten) and the missing FreeHand install CDs.

Now to get these final PITA problems licked so I can continue to move Mori forward!

As I was reviewing the code changes I had made, I slowly realized that the Mac’s user preferences system had a minor philosophical flaw which makes for dangerous situations in the program.

One of the goals of this version is improved recovery from faulty stored values relating to the preferences and UI, so you can continue with only minor inconvenience and perhaps a need to again set preferences to your liking. Not adequately performing this resulted in some of the missing toolbar and empty window problems that were experienced before. (For a more exciting example of cascading faults, read about the 2003 Northeast blackout.)

So after some delays in getting the unit tests for this new component to execute properly, I’m now adding that component into the program. Once I have the Cocoa preferences system replaced, I’ll be releasing 1.6.8. Then I’ll work on the possible fix for the SpotLight ‘odd name’ bug for a quick-turnaround (later today) 1.6.9.

If that particular bug can’t be quickly fixed, I’ll work on some other long-standing issues in Mori for an update that will be released after a few days.

Incidentally, Tobias had a valid concern regarding the update process: “Where does testing feature in this? In olden times an update was a build with only known problems since a few people played with it for a week before it was awarded a version number. WebKit offers a choice between bleeding edge nightly r1234 and release 3.0.”

I plan to continue to rely on the Beta Test Group for testing updates, with a release candidate at least one version ahead of what is shipping. However, at the time of this writing there are 320 open bug reports for Mori, even after having pre-release testing candidates as part of the process. That just isn’t right.

I’m primarily depending on the unit tests to prevent shipping defects to you. They serve to expose bugs in existing and new code; and by adding tests for the types of bugs identified in the bug reports, the tests will continue becoming more valuable to the release process.

The purpose is to continuously speed up the process by improving its results. It shouldn’t be a shock when a program works right off the bat. The delays are (partly) associated with having so many steps of repeated inspections on the same system. It multiplies the time it takes to get a new release out the door. The less I tolerate it, the quicker it gets done correctly.

Because of the difficulties fixing the toolbar bugs and getting Leopard compatibility complete (or reasonably so), Mori has quickly approached version 1.6.10 (not yet, only 1.6.7 has been released so far, but bear with me). This has some odd psychological barrier attached to it, as we seem to recognize it as a significant occasion, a hurdle we do not wish to cross.

I’ve been collecting fixes into a single release, attempting to conserve version numbers. There’s currently improved “Check and Repair Notebook”, more cautious handling of user preferences, improved Italian localization, a fix on the Drag and Drop stall, clearing of compiler warnings, some refactorings, and more unit tests. A couple more fixes I’d like to incorporate into this one: fixed word count (whether it’s in English or Greek), correct Smart Folder behavior and making wildcards optional in search terms.

But anything.10 is an artificial milestone, rather than a significant one. And MOX has already passed it and even gone on to 10.4.11, so what’s troublesome about it now? And with the new versioning class I added to Mori back in 1.6.4 or so, it should be able to handle even version 1.6.99 if necessary!

So I’ll be trying to post new updates more frequently. I’m not certain how frequently it’ll be, but I’d like to get to the point where there was a nightly build, like the Safari team provides. That’s too frequent for most users, of course, but then you’d be able to skip a few interim releases until something you need is included. The other benefit it would provide is allow me to move all the apps forward a little bit at a time, rather than doing continuous development on one app for weeks at a stretch before rotating development to the others.

The past week was one of the worst in recent computer experiences I’ve had, surpassing the Leopard install. In fact, the last time my experience has been this bad was when I last fiddled with website software. I was going to abandon the whole server transfer/upgrade plan due to the issues the upgrades were causing, but I remembered the transfer itself should be fairly trouble free. I’ll only upgrade a minor version; not a whole version, which would require adding a lot of code to custom modules Jesse wrote. Should be.

I can appreciate the time and effort invested by those who work on open source projects, but developers who complain that users aren’t migrating to their latest efforts quickly enough are both arrogant and naive. Customers prefer to be able to continue using the data they’ve stored in the current versions to losing them when they migrate to the newer ones. I’m pretty sure it’s the same for OSS users. Much as I love writing code, I have enough to write already thank you very much. And I don’t appreciate having to spelunk through your code to figure out what those data structures you’ve added are supposed to do (”ooh, shiny, new!”) just because you’re too lazy to write the docs (”the code are the docs!”) and impatient to play with your new ideas.

Although the site I inherited from Jesse used a CMS system distinct and incompatible from the one I had selected when I started my own, I didn’t want to switch it around or try doing new things with it because I felt the continuity was better than taking the time to rebuild a site from scratch. But now I can appreciate Jesse’s decision to do such a thing. If I have to spend the downtime upgrading parts that already work well just to get the features needed for other modules, I might as well fix it to my own needs and ignore the chaos that OSS developers are creating in the system. Now the whole NIH question is thrown out the window. It’s no longer a matter of, “Oh, I can do a cooler system.” Now it’s a question of the developers themselves causing users to abandon their system. A question of distrust and self-preservation.

Hopefully, any remaining misconfigurations in the website will be rectified before a second person notices it.

If you sent an email in the past 24 hours (of 2007-12-05) and you haven’t gotten a response already I apologize, but you’ll need to resend it. My site host doesn’t transfer files, emails, settings, or the like between accounts; and well, it has probably been crushed by the lumbering floes. In fact, if you’re waiting for a response from me on any matter, please jog my memory with another email.

Anyway, at least the site now has some breathing room, and I can continue improving the products.

Well, it looks like the latest toolbar fix has finally stuck. There were actually a few, very subtle interrelated items, and some procedural issues that cascaded into others. Code was shifted around hither and yon, resulting in elimination of two of a main class’ instance variables and simplification of logic in several methods.

Speaking of which, I’d really love to try out Xcode3’s new refactoring tools, but that would mean

1. spending more time in Leopard, which freezes on me,
2. and spending more time in Xcode3, whose text editing I already loathe.

I will need to investigate how well refactoring works in TextMate (which I already own and use from time to time when bumping against some other Xcode2 limitation.)

The pressure cooker development level of the past couple of weeks really put the development process here through the wringer. Some things held up. Some things fell apart. So more ‘behind the scenes’ development is going to take place here to minimize the interference caused by the mismatch between what needs to be done and what the tools require me to do next.

Some observations of the release’s development:
1. The primary, agile-based, processes were abandoned in the panic over the continued toolbar problems and Leopard incompatibilities.
   There was no established process for handling emergencies, so the chaotic edit-compile-test behavior resurfaced. This is not to be repeated.

   Corrective Measures: A plan of action for dealing with emergencies must be established. Exercises and drills prepared and performed so the proper outcome occurs next time.

2. Users seemed to be unaware, or unconvinced, of my plans for Leopard.
   Although I had made statements regarding the planned support for Leopard, it was only in response to questioning by users. I had failed to provide information ahead of time and prominently.

   Corrective Measures: While users refer to the forum when there are questions or they have some issue to resolve, they refer to the blog constantly. It’s best to have already disclosed plans so users have time to assimilate needed info and prepare accordingly. Blog. Blog. Blog. Forum posting: link to blog. Then blog.

3. Unit testing got skipped.
   It was unavoidable given the current process because tests didn’t cover the critical issue being developed: the user interface (i.e., toolbar items and empty windows).

   Corrective Measures: Mock objects are inadequate to sufficiently test the UI. For all my blustering that the UI is testable, it’s clearly time I seek or develop the necessary tools and put them to use.

4. Subversion is a win, but it’s an ugly win.
   Being able to restore files, or the project, of a particular beta or release build, or of a particular date and time, is great. Being able to make wholesale changes to the project, then abandoning them, or keeping them along a separate branch to continue experiments at a later date without juggling project directories here and there, is great. Being able to merge or contrast multiple working project directories from separate environments (Tiger, Leopard, laptop, and the v1.6.3 release) quickly and easily, is great.

   Nibs saved as a smattering of files in the repository: not so great. Confusing subversion with its own metadata when copying/adding directories: not so great. Performing an add/delete when moving or copying directories and files instead of adding “move” or “copy” semantics to the system: not so great. Poor integration mechanism for multiple offline revisions: not so great.

   SCM is great. I’ve used CVS for years. I’ve even set up another project or two using Subversion before I bought Mori and Clockwork from Jesse. But I don’t see where it’s an obviously better package than CVS, but perhaps that’s because I’ve learned to work around CVS’ shortcomings. (Reviewing the feature list at http://subversion.tigris.org, the only practical feature I see that means anything to someone actually using Subversion is command-level manipulation of directories, although the handling of renames is poor. Hrmm. And svn+ssh access to the repository, with Xcode integration even!)

   Corrective Measures: For the time being, the messiness of the metadata in copies and moves will have to remain unsettled. As long as the structure ends up correct, and it doesn’t take long to get it to that state, it remains tolerable in the face of bigger issues. A procedure must be established to handle the circumstance where developers are offline and must “commit” in some manner which is transferrable to the repository when they are back online.

5. Debugging is severely inadequate for professional-grade development.
   Can you tell Apple’s team has a severe case of he-man, mouse-haters club insecurities? (Either that, or they’re operating in crunch mode so much, they don’t have time to develop the appropriate tools. Nah!) The debugging facilities they provide for developers operate primarily through the use of environment variables, console output and functions and output which require gdb to access.

   ZeroLink didn’t work well and was abandoned; Fix and Continue is still problematic; and the vaunted debug libraries (to help you catch errors in the parameters passed to the AppKit and Foundation libraries) hasn’t worked right since 10.4.3.

   My own debugging facilities are rudimentary and lacking in depth. More of the failures in the field should be communicated back to me (remember Safari’s ‘bug’ button?). The crash reporting mechanism should work when the crash occurs, not the next time Mori runs. Also, exceptions are logged to the console, but otherwise go unnoticed by Mori and the user.

   Corrective Measures: Extend the runtime monitoring and browsing tools. Rewrite the crash reporting mechanism to activate when the application terminates. Add an exception handler which sends reports back. Develop proper high-level debugging tools for Objective-C, the UI, CoreData and Bindings. Coding takes place at a higher level. Debugging should be at least at that same level. -fobjc-gc exists for a reason. Take advantage of it!

6. The build process is inconsistent, confused and unstable.
   The various plugins’ build settings aren’t consistent. Extraneous resources (images, sounds, etc.) are stored in the nibs. At times the build directory must be purged for a functioning executable to be built.

   Corrective Measures: A Build Settings Table has already been prepared and Mori and its plugins have had their settings documented. The settings for the Blocks plugins has yet to be documented. The effect of some settings has to be determined, which can then be propagated to the targets as appropriate. The project file should be purged of duplicated actions, unnecessary references, etc.

7. Debugging Mori when it’s used for normal work results in too much human activity thrashing.
   Because the debug and test versions use the same preferences/file settings, the release version used for normal work had to be exited to avert data corruption in the notebooks.

   Corrective Measures: This issue was resolved with the Oneill branch. However, that uses a separate target to achieve its distinction. Some means of specifying a special bundle id for the debug and test builds must be developed to accomplish a similar effect, perhaps through preprocessing the Info.plist.

8. Checklists are great
   Being able to state that processes are being followed, builds are complete, and updates were released correctly, is great.

   Corrective Measures: Checklists are great, but scripted procedures are better. Automate as much of every process as possible.

9. To-Do lists, Getting Things Done (GTD), or whatever Time Management activity that is put into practice definitely helps.
   I’ve got the audio book. Mori’s got the plugin. I’ve got Taskpaper, too. I’ve got a Handspring Visor Edge and its Palm Desktop software. Something, anything, that helps track tasks that need to be done so nothing falls between the cracks is a plus. They are effective at keeping things moving forward, but I’m not efficient at it and a lot of discipline and effort must be used to keep moving things forward.
   Corrective Measure: I used TaskPaper to handle the tracking during this hectic period. I’ve got a blog entry in preparation, but basically, I found it a great way to get into the GTD system due to its simple interface. I must continue developing my understanding of this system to manage my activities, and see about getting the various software/devices integrated better.

Next Batch of Changes

I’ve already begun work on v1.6.8: Improved checking and repair functionality for notebooks. Correct Italian localization, thanks to Mario Pettenghi. The code for the Blocks framework will be tweaked so it compiles without triggering warnings (e.g., unused parameters, missing prototype, etc.). Additional unit testing for the UI, and refactoring of UI code.

These improvements to the code will set up the continuation of progress for v1.7, which should then be ready during the holidays. At least that’s the current goal.

Since it appears that the users still experiencing the “empty window” or missing notes problem are running Leopard and have some strange mixture of preferences (particularly “Enable check spelling as you type in new windows”), I should be able to get some update out today to Tiger users at least.

I received word a short while ago from users experiencing this specific symptom, and byakkie, who thankfully took the time to experiment with all the preferences to help pin down the most likely causes, regarding the results of running an experimental build to deal with the big remaining issues. With those results, and knowing it’s Leopard-specific (those more stringent parameter requirements!) I’ll reboot into Leopard in a moment and get down to exorcising that gremlin. If it’s like the other Leopard problems encountered it should be relatively easy to find the problem, and hopefully not-too-difficult to engineer a solution.

While I had awaited word, I worked on streamlining and pruning certain aspects of the build process and decided to tackle the recurring error message on my console:

*** -[NSKeyedUnarchiver decodeObjectForKey:]: cannot decode object of class (WebView)

which is always cause for concern for developers when things go wrong that you didn’t even realize were going on! “WebView?! Since when is there a WebView in an outliner?”

So I tackled it and discovered it is part of Blocks’ software update service, which checks for new versions of Mori and displays the release notes when found. Once I linked the WebKit framework into the Blocks software update plug-in, the release notes window popped up to announce a new version! Then it had a redirect issue I had to fix (because www.apokalypsesoftware.com redirects to apokalypsesoftware.com). Then I discovered the links in the release notes weren’t pointing to the correct location. And so on.

This all leads me to one conclusion: you folks are hard-core Mori users! Do you Fred Flintstone your way to work? Do you eat stale bread? Do you really love Mori’s features so much that you’ve been willing to put up with the broken stuff for so long? (And just what Mori features are you particularly fond of? I’d like to intensify that experience for you!) It’s just a more pleasant experience when the software does things for you like it should. Wow!

Well, while I am half-joking, I am truly impressed and grateful that you’ve been willing to endure some inconvenient obstacles just for the productivity that you gain with Mori. Thank you so much! I plan to continue fixing these annoyances as features are added, so that you find there’s more that can be done by Mori and less roadblocks it puts in your way.

Anyway, now on to Leopard and that fix!