The Mac developer scene is loaded with a lot of helpful and supportive developers out there, sharing ideas, code and encouragement with one another. I’ve benefitted from the development community, as have the Mori and Clockwork code. I say this not only as one who’s currently putting new code into them, but seeing the comments Jesse left in the code before me!

So I’d like to drop an idea into your lap you might find useful for the right type of application. As there are different types of software one can develop, e.g., operating systems, utilities, paint programs and the like, they have particular usage characteristics that they encourage. Some, like digital notebooks and desktop timers for example *cough*, tend to remain in operation while the user is logged into his account. Usually, this isn’t a problem for the user. Unless, of course, the user is also someone further developing said software. Then several questions have to be answered, such as, “How do I run unit tests on builds while still running a stable version of my app?” and “How do I try out the latest build without affecting the files which are currently open in my app?”

On operating systems based on more primitive process management, it isn’t troublesome because each built application is fairly independent of others, and often multiple invocations of a single app can be run concurrently. It wasn’t problematic on the Mac either back on MacOS 9 and earlier, and even in the early MOX releases. But now with Launch Services, running test versions of apps you run on an all-day basis can be problematic. It’s best to give that test version some different signature so its defaults and file changes don’t corrupt your normal environment.

Note that this tip doesn’t prevent two versions from munging the same files; even two unrelated apps can mess up a data file if they’re updating it at the same time. What this tip does is provide an alternate identity, if you will, for test builds so MOX doesn’t try to hand it the same data set it gives the release builds of your app. You mustn’t try to open files in the test build that are currently opened by a release build.

Here’s how I do it:

1. In the Info.plist file, use the C pre-processor’s conditional compilation directive to give your app its normal metadata when a release build is made, but a special set of metadata for all other build configurations:

#if defined(Release)
<key>CFBundleIdentifier</key>
<string>com.apokalypsesoftware.Mori</string>
#else
<key>CFBundleIdentifier</key>
<string>com.apokalypsesoftware.Mori-debug</string>
#endif

Update: I forgot to mention that Xcode 3 and later versions now use specialized editors for the Info.plist files. To override this editor selection, bring up the file’s Info window (by selecting the file and pressing cmd-I) or the Inspector window and change the File Type setting from “text.plist.whatever” to “text”. Once you add the conditional to Info.plist you can change File Type back and access Info.plist in its raw form from its target’s Properties tab. That way, you can view it in either mode within Xcode. However, making any changes in Xcode’s plist editor will wipe out any conditional directive in the file, so use it only for reviewing current settings, not for actual editing!

2. In the Target Info window (brought up via the ‘Project > Edit Active Target “Mori”‘ menu item), select the Build tab, specify “All Configurations” for the Configuration pop-up menu, and enter “plist” in the search toolbar widget to bring up the relevant setting items.

3. For the “Info.plist Other Preprocessor Flags” setting, enter -traditional as the value. While not needed to support the test version twin, it will help to prevent any URLs included in the Info.plist from being swallowed up by the pre-processor and possibly causing you sleepless nights because neither MOX nor the Xcode build system have the slightest idea of what the proper format for URLs are, but will behave erratically when it isn’t just so. So it’s best to make it a non-issue from the start. (No charge for that and other tips related to usage of the pre-processor you can find in Technical Note TN2175. Incidentally, TN2175 says to use “-traditional”, but I use “-CC”. Use whatever works for the version you’re on.)

4. For the Info.plist Preprocessor Definitions setting, enter $(CONFIGURATION). Xcode will change the display to read <multiple values>, but that’s okay because the value will vary according to the current build configuration. It will define a value that matches the build configuration that the pre-processor will test against in the Info.plist. (Try switching the setting in the Configuration pop-up menu of the Build tab to watch the value change to match the current setting.) This is what will direct the pre-processor to output the correct value for CFBundleIdentifier.

5. Check the Preprocess Info.plist File checkbox so Xcode invokes the pre-processor on the Info.plist before copying it to the application.

When built, you’ll be able to run the app concurrently with the release version, and without causing conflicts with data files in use by it. Of course, you won’t have your preferences set up, but that’s only a problem until you save preferences for the test version. (Remember, if you try to get around that problem by copying the release version’s defaults file that it also holds the recently opened files. Purge, close or what-have-you as appropriate!)

One of the extra things I attempted to do in this technique was to provide an alternate name for the private builds or test versions so they would be distinctly named for testers. We used to use naming conventions such as “Mori 1.7β3” when distributing such versions back in the pre-MOX days, but including the metadata keys CFBundleExecutable, CFBundleName, and CFBundleDisplayName don’t have any noticeable effect. Any tips in this regard would be greatly welcomed.

I had also attempted to get the pre-processor to generate the bundle identifier with “Mori-$(CONFIGURATION)” that fit whatever build configuration was in effect automatically, which would’ve avoided the use of the #if defined directive. Unfortunately, because the macro expansion would generate Mori-Release that method proved ineffective.

A corresponding idea is to use an alternate icon (via the CFBundleIconFile specifier) to represent the test version of your app. It helps you and your testers realize when odd behavior is due to running the wrong build just by looking at the dock or application-switcher. It’s also fun to double-click a document only to realize the test app is opening vital data meant only for the stable version. Hilarity is sure to ensue. But I’m a professional, don’t you kids try this at home!

If you don’t have the construction guy handy, you can resort to the technique I used before I discovered Info.plist processing: MOX’s own stamping of missing dock files makes for a handy visual guide to your test app. To try it yourself, add the debug version from within your build/Debug folder to your dock items. That’s right, the debug version (or whatever you use for test builds). Now delete it and empty the trash. (You will be building new ones, won’t you?) Now click on the app icon you just added to the dock. It adds a big fat ‘?’ to your lovely icon, doesn’t it? (At least it should if you deleted the correct icon.) Now rebuild your app and run it from within Xcode.

Now bring up the application switcher (cmd-tab). What do you see? A big fat ol’ ‘?’ emblazoned on the debug version of your app! Oh, you don’t? You see the normal icon? Yeah, you’re on Leopard, aren’t you? This trick doesn’t work as well on Leopard because MOX validates the file state more often than Tiger did. Such as, starting the test build from the Dock instead of only within Xcode. On Leopard, clicking on the dock icon when there’s a built debug version of your app in the debug folder will restore its unstamped icon. But anytime the question mark goes away, just delete the debug version, empty the trash, and click on the dock icon again.

Have fun, and keep ‘em flying!

As I was reviewing the code changes I had made, I slowly realized that the Mac’s user preferences system had a minor philosophical flaw which makes for dangerous situations in the program.

One of the goals of this version is improved recovery from faulty stored values relating to the preferences and UI, so you can continue with only minor inconvenience and perhaps a need to again set preferences to your liking. Not adequately performing this resulted in some of the missing toolbar and empty window problems that were experienced before. (For a more exciting example of cascading faults, read about the 2003 Northeast blackout.)

So after some delays in getting the unit tests for this new component to execute properly, I’m now adding that component into the program. Once I have the Cocoa preferences system replaced, I’ll be releasing 1.6.8. Then I’ll work on the possible fix for the SpotLight ‘odd name’ bug for a quick-turnaround (later today) 1.6.9.

If that particular bug can’t be quickly fixed, I’ll work on some other long-standing issues in Mori for an update that will be released after a few days.

Incidentally, Tobias had a valid concern regarding the update process: “Where does testing feature in this? In olden times an update was a build with only known problems since a few people played with it for a week before it was awarded a version number. WebKit offers a choice between bleeding edge nightly r1234 and release 3.0.”

I plan to continue to rely on the Beta Test Group for testing updates, with a release candidate at least one version ahead of what is shipping. However, at the time of this writing there are 320 open bug reports for Mori, even after having pre-release testing candidates as part of the process. That just isn’t right.

I’m primarily depending on the unit tests to prevent shipping defects to you. They serve to expose bugs in existing and new code; and by adding tests for the types of bugs identified in the bug reports, the tests will continue becoming more valuable to the release process.

The purpose is to continuously speed up the process by improving its results. It shouldn’t be a shock when a program works right off the bat. The delays are (partly) associated with having so many steps of repeated inspections on the same system. It multiplies the time it takes to get a new release out the door. The less I tolerate it, the quicker it gets done correctly.

Because of the difficulties fixing the toolbar bugs and getting Leopard compatibility complete (or reasonably so), Mori has quickly approached version 1.6.10 (not yet, only 1.6.7 has been released so far, but bear with me). This has some odd psychological barrier attached to it, as we seem to recognize it as a significant occasion, a hurdle we do not wish to cross.

I’ve been collecting fixes into a single release, attempting to conserve version numbers. There’s currently improved “Check and Repair Notebook”, more cautious handling of user preferences, improved Italian localization, a fix on the Drag and Drop stall, clearing of compiler warnings, some refactorings, and more unit tests. A couple more fixes I’d like to incorporate into this one: fixed word count (whether it’s in English or Greek), correct Smart Folder behavior and making wildcards optional in search terms.

But anything.10 is an artificial milestone, rather than a significant one. And MOX has already passed it and even gone on to 10.4.11, so what’s troublesome about it now? And with the new versioning class I added to Mori back in 1.6.4 or so, it should be able to handle even version 1.6.99 if necessary!

So I’ll be trying to post new updates more frequently. I’m not certain how frequently it’ll be, but I’d like to get to the point where there was a nightly build, like the Safari team provides. That’s too frequent for most users, of course, but then you’d be able to skip a few interim releases until something you need is included. The other benefit it would provide is allow me to move all the apps forward a little bit at a time, rather than doing continuous development on one app for weeks at a stretch before rotating development to the others.

Well, it looks like the latest toolbar fix has finally stuck. There were actually a few, very subtle interrelated items, and some procedural issues that cascaded into others. Code was shifted around hither and yon, resulting in elimination of two of a main class’ instance variables and simplification of logic in several methods.

Speaking of which, I’d really love to try out Xcode3’s new refactoring tools, but that would mean

1. spending more time in Leopard, which freezes on me,
2. and spending more time in Xcode3, whose text editing I already loathe.

I will need to investigate how well refactoring works in TextMate (which I already own and use from time to time when bumping against some other Xcode2 limitation.)

The pressure cooker development level of the past couple of weeks really put the development process here through the wringer. Some things held up. Some things fell apart. So more ‘behind the scenes’ development is going to take place here to minimize the interference caused by the mismatch between what needs to be done and what the tools require me to do next.

Some observations of the release’s development:
1. The primary, agile-based, processes were abandoned in the panic over the continued toolbar problems and Leopard incompatibilities.
   There was no established process for handling emergencies, so the chaotic edit-compile-test behavior resurfaced. This is not to be repeated.

   Corrective Measures: A plan of action for dealing with emergencies must be established. Exercises and drills prepared and performed so the proper outcome occurs next time.

2. Users seemed to be unaware, or unconvinced, of my plans for Leopard.
   Although I had made statements regarding the planned support for Leopard, it was only in response to questioning by users. I had failed to provide information ahead of time and prominently.

   Corrective Measures: While users refer to the forum when there are questions or they have some issue to resolve, they refer to the blog constantly. It’s best to have already disclosed plans so users have time to assimilate needed info and prepare accordingly. Blog. Blog. Blog. Forum posting: link to blog. Then blog.

3. Unit testing got skipped.
   It was unavoidable given the current process because tests didn’t cover the critical issue being developed: the user interface (i.e., toolbar items and empty windows).

   Corrective Measures: Mock objects are inadequate to sufficiently test the UI. For all my blustering that the UI is testable, it’s clearly time I seek or develop the necessary tools and put them to use.

4. Subversion is a win, but it’s an ugly win.
   Being able to restore files, or the project, of a particular beta or release build, or of a particular date and time, is great. Being able to make wholesale changes to the project, then abandoning them, or keeping them along a separate branch to continue experiments at a later date without juggling project directories here and there, is great. Being able to merge or contrast multiple working project directories from separate environments (Tiger, Leopard, laptop, and the v1.6.3 release) quickly and easily, is great.

   Nibs saved as a smattering of files in the repository: not so great. Confusing subversion with its own metadata when copying/adding directories: not so great. Performing an add/delete when moving or copying directories and files instead of adding “move” or “copy” semantics to the system: not so great. Poor integration mechanism for multiple offline revisions: not so great.

   SCM is great. I’ve used CVS for years. I’ve even set up another project or two using Subversion before I bought Mori and Clockwork from Jesse. But I don’t see where it’s an obviously better package than CVS, but perhaps that’s because I’ve learned to work around CVS’ shortcomings. (Reviewing the feature list at http://subversion.tigris.org, the only practical feature I see that means anything to someone actually using Subversion is command-level manipulation of directories, although the handling of renames is poor. Hrmm. And svn+ssh access to the repository, with Xcode integration even!)

   Corrective Measures: For the time being, the messiness of the metadata in copies and moves will have to remain unsettled. As long as the structure ends up correct, and it doesn’t take long to get it to that state, it remains tolerable in the face of bigger issues. A procedure must be established to handle the circumstance where developers are offline and must “commit” in some manner which is transferrable to the repository when they are back online.

5. Debugging is severely inadequate for professional-grade development.
   Can you tell Apple’s team has a severe case of he-man, mouse-haters club insecurities? (Either that, or they’re operating in crunch mode so much, they don’t have time to develop the appropriate tools. Nah!) The debugging facilities they provide for developers operate primarily through the use of environment variables, console output and functions and output which require gdb to access.

   ZeroLink didn’t work well and was abandoned; Fix and Continue is still problematic; and the vaunted debug libraries (to help you catch errors in the parameters passed to the AppKit and Foundation libraries) hasn’t worked right since 10.4.3.

   My own debugging facilities are rudimentary and lacking in depth. More of the failures in the field should be communicated back to me (remember Safari’s ‘bug’ button?). The crash reporting mechanism should work when the crash occurs, not the next time Mori runs. Also, exceptions are logged to the console, but otherwise go unnoticed by Mori and the user.

   Corrective Measures: Extend the runtime monitoring and browsing tools. Rewrite the crash reporting mechanism to activate when the application terminates. Add an exception handler which sends reports back. Develop proper high-level debugging tools for Objective-C, the UI, CoreData and Bindings. Coding takes place at a higher level. Debugging should be at least at that same level. -fobjc-gc exists for a reason. Take advantage of it!

6. The build process is inconsistent, confused and unstable.
   The various plugins’ build settings aren’t consistent. Extraneous resources (images, sounds, etc.) are stored in the nibs. At times the build directory must be purged for a functioning executable to be built.

   Corrective Measures: A Build Settings Table has already been prepared and Mori and its plugins have had their settings documented. The settings for the Blocks plugins has yet to be documented. The effect of some settings has to be determined, which can then be propagated to the targets as appropriate. The project file should be purged of duplicated actions, unnecessary references, etc.

7. Debugging Mori when it’s used for normal work results in too much human activity thrashing.
   Because the debug and test versions use the same preferences/file settings, the release version used for normal work had to be exited to avert data corruption in the notebooks.

   Corrective Measures: This issue was resolved with the Oneill branch. However, that uses a separate target to achieve its distinction. Some means of specifying a special bundle id for the debug and test builds must be developed to accomplish a similar effect, perhaps through preprocessing the Info.plist.

8. Checklists are great
   Being able to state that processes are being followed, builds are complete, and updates were released correctly, is great.

   Corrective Measures: Checklists are great, but scripted procedures are better. Automate as much of every process as possible.

9. To-Do lists, Getting Things Done (GTD), or whatever Time Management activity that is put into practice definitely helps.
   I’ve got the audio book. Mori’s got the plugin. I’ve got Taskpaper, too. I’ve got a Handspring Visor Edge and its Palm Desktop software. Something, anything, that helps track tasks that need to be done so nothing falls between the cracks is a plus. They are effective at keeping things moving forward, but I’m not efficient at it and a lot of discipline and effort must be used to keep moving things forward.
   Corrective Measure: I used TaskPaper to handle the tracking during this hectic period. I’ve got a blog entry in preparation, but basically, I found it a great way to get into the GTD system due to its simple interface. I must continue developing my understanding of this system to manage my activities, and see about getting the various software/devices integrated better.

Next Batch of Changes

I’ve already begun work on v1.6.8: Improved checking and repair functionality for notebooks. Correct Italian localization, thanks to Mario Pettenghi. The code for the Blocks framework will be tweaked so it compiles without triggering warnings (e.g., unused parameters, missing prototype, etc.). Additional unit testing for the UI, and refactoring of UI code.

These improvements to the code will set up the continuation of progress for v1.7, which should then be ready during the holidays. At least that’s the current goal.

It looks like the showstoppers which were part of 1.6.4, 1.6.5 (and even 1.6.3 counting the toolbar) have been dealt with. The latest build even seems to function normally in Leopard.

Thus, now that Leopard 10.5.1 has been released, and my Leopard install was for testing purposes anyway and not as my primary development environment, I’m going to install it and test the 1.6.6 build against it as well to make sure it still works. That is, after a stopover in Tigerland just to make sure that nothing in Leopard loused up my main volume.

The problems in Leopard appear to result, not from changes in the architecture of the Cocoa classes, but in how stringent the standards for values passed to them are, and how they deal with values which are unacceptable (invalid parameters and exception handling). Sure, code should only pass correct data all the time, but sometimes our expectations are off. Sometimes we get incorrect data ourselves (Garbage In Garbage Out), and sometimes, bad things happen in the real world in which computers operate.

Tiger used to be somewhat more non-chalant about this issues. It would ignore all but the most egregious problems, unless the programmer or the user asked otherwise. Leopard seems to be more restrictive and demanding of Mac developers and the code they write. Again, not bad by any means, except for the unexpected and the change in behavior for things that used to work before.

So, with more testing and better debugging (so why can’t I get “debug” suffix to work?!), we’ll hopefully see this occur less often in Mori’s code.

Now…back to that checklist.

Apple, fresh from delivering AppleTV and in the middle of pushing iPhone out the door, has announced a delay in the planned shipping date of October 2007 for Mac OS X (MOX).

While some bloggers denounce Apple’s claim as deceitful (there isn’t even an entry in its press releases, only a statement for April 12, 2007 on the Hot News page), I’d say it’s more likely a half- (or less) truth. There are plenty of causes for MOX’s delay.

Apple knows it has a process problem. Steve discussed only the AppleTV and iPhone during his MacWorld 2007 keynote.

Apple has a huge backlog of duplicate and unresolved bugs in its radar database. My own entry is #3665065, entered on 23-May-2004 12:26 AM and still open.

Apple doesn’t eat its own dogfood.

Apple doesn’t have enough bodies to handle its current plans. Here’s a listing of the groups with current software engineering needs from a recent job posting:

The following teams at Apple are now hiring:

- X Grid
- Core OS
- File Systems
- Build and Integration
- Kernel Development
- Vector Numerics
- Java
- Networking
- QuickTime
- Compiler
- User Interface
- Graphics and Imaging
- Frameworks
- Security
- Localization and Release Engineering
- Core Audio
- Video Codec
- Desktop Management Solutions

On top of that, the Professional Audio, Professional Video, FileMaker, Desktop Apps (incl. iTunes), Interactive Media, iTunes Store, iPod, AppleTV and other teams I wouldn’t even guess about need developers to develop products currently on the market as well as future goodies and you can see why Apple’s in the pickle it’s in.

After working to release AppleTV and iPhone with their own variants of MOX, the Leopard team most likely has to reintegrate those branches back into the main branch. As a bonus, they will most likely develop some inability to let you run an AppleTV as a standalone MOX system in the future and cripple the iPhone’s OS to prevent that sort of thing as well.

There are serious scope and process issues within Apple’s development groups. Do not expect a speedy resolution.

[Update - 2007-04-20] Here’s another loss to one of Apple’s product teams. His story is far more indicative of Apple’s problems.

Jeff Langr wrote about the importance of pair programming, which I’ve already discussed before.

However, one important issue he raises is what to do on non-paired time. The tasks he outlines are very important for any agile project as they definitely will increase your overall velocity.

His list:

• work on any non-production code: the acceptance test framework, other tools, build scripts, etc. In most shops, these are in dire need of attention.
• work on documentation
• work on spikes for future stories
• learn how to use a new third-party product; learn a new coding technique; and so on
• identify problem spots in the production code that need refactoring
• refactor tests
• improve the existing test coverage if necessary

I would augment “learning to use a new third-party product” to be “learning to use the software development tools at your disposal”. How much extra effort are you expending because you don’t use a document versioning tool in your workflow, or you aren’t using a source-level debugger, or don’t fully know the features and commandset of the debugger or text editor you use? How many possible bugs would you catch if you enabled certain warnings on compiles, or used a test coverage analyzer or performance profiler (e.g., gcov and gprof respectively)?

Here’s a great quote from Mike Ash:

“not knowing how to use the right tool is not a good reason to use the wrong tool”
— mikeash

Tools are probably already available for the things you need to do, so stop reinventing the wheel. After all, others have been writing software for over sixty years. What workflow-related (or even non-workflow) speedbumps could you eliminate if you just took the time to internalize a better mastery of your workflow?

How much more agile can you be if you spend a little effort improving your process?

Technical Debt. Although it sounds more like an issue to be heard in bankruptcy court than software development. I would prefer a term like “workload debt”, but oh well.

In any case, this is precisely the term I was looking for to describe the build up of costly delays in rework due to putting off the necessary tasks of software development.

Huzzah!

Yes, writing tests up front is time-consuming and not directly contributory to the final code. But writing the tests is like filling your auto’s tank with gas before starting a long trip. It will save you from the frequent stops you’ll have to take without it.

If you’re using a compiler, you’re likely already using automated testing as they perform static type checking. You have to declare the variable before using it, along with its type specifier. You have to use casts for parameters and assignments when the types don’t conform to the prototypes.

You’re content, for the most part, to use static types in your code because they eliminate an entire class of defects and reduce time you’d have to spend debugging for them.

Unit Tests and Assertions
Don’t use just unit tests to verify the production code works as expected, but also assertions within the production code to verify parameters and environment meet assumptions.

What to test
Interface - can be part of TDD
Parameters
Errors
Results

Measurements are an integral part of the testing process. In manufacturing processes, they provide an indication of how the final product will turn out. If the temperature is too high, resins will become too fragile. If raw materials are left in the open air too long, the loaves of bread won’t have the right texture.

Note the variation that was acceptable at the most commercially successful, and highest quality, studio at the time. And yet they still measured raw output. Other studios enforced a more inflexible production quota (bean counters!), and their work suffered as a result.

Measurement is an indirect means of testing the velocity and quality of a process, and is as valid for software development as it is for animation.