Author: sqc1w

Truth be told, I’m not big on software development. I became a programmer by hanging out at Radio Shack and trying to figure out what made the beasties run, but it wasn’t the career I had intended to enter. I grew up doing cartoon animation, and it was an expensive hobby in those days.

But this posting isn’t about my career choices. This is about software development and the myth, often voiced by programmers, that what they do is an art.

Now, there are exceptional programmers. And there are exceptional programmers who are artists. There are men who, with an understanding of the programming language and the architecture of the hardware on which it runs, can craft simple and elegant solutions in less than one tenth of the execution space and time that ordinary programmers do. But the programming profession is far from an art form.

Programming isn’t an art form not because it’s derived from mathematics or is executed on emotionless, sterile computing machinery; but because not enough is understood about what makes an effective program and how to build one. The vagaries of the practice don’t prove it’s art. It proves it isn’t.

To demonstrate this difference in more concrete terms I’ll illustrate with an overview of the development of an animated film, which I hope you would consider to be an art.

Initially, a story treatment is presented giving an overview of the film’s premise, the characters involved and the plot. If interested, the studio will pick up a treatment and secure a script. If the script is approved then various studies are made dealing with specific aspects of the story, characters, lighting, colors, sets, props, pacing, etc. Voice talent is selected for the characters. As the story is developed, storyboards are made illustrating the progression of a scene. An animatic, or Leica reel, is made combining dialogue, storyboard sketches and sound effects to provide the team with a sense of the film so they can detect weak or unworkable issues. As action is animated, the drawings are filmed as pencil tests to offer a quality review. As pencil tests are shot and approved and final dialogue, sound effects and scores are recorded they’re incorporated into the animatic so the team can gauge the film’s progress. The animators’ drawings are inked and painted, formerly by hand. Backgrounds and special effects are drawn and painted and merged with the colored animation and filmed. Eventually the animatic is transformed into the final picture and can be packaged and distributed. Of course, artwork and performances for marketing materials, trailers, and publicity are also necessary.

Production proceeds in a manner that is logical from a production standpoint, not one parallel to the film’s time. Sequences are animated out of linear order. Voice actors in a scene are usually not recorded in the same session. One animator draws the lead character in one sequence of the film while another draws the same character in a later sequence, and additional characters in those sequences are drawn by yet other animators. And normally, different sequences are in different phases of production at any given point. Scenes may have to return to an earlier phase several times before approval, but the film will still be completed properly.

This development cycle typically lasts three to four years for a feature-length animated film. How is it that a schedule for something as vague and variable as a filmed story can be well-defined enough to meet a release date? Because the problem is well-understood. Because the metrics (e.g., duration of a scene, characters per scene, feet of film an animator can output per week, etc.) are known. Because decisions and directions are constantly tested and screened every step of the way. Of course, though steps have been taken to minimize the risk of script changes, re-animating or re-filming sequences, etc., sometimes substantial and costly changes must be made as the film nears completion. In animation, as in software development, it’s very expensive to fix poor design decisions when they’ve been executed.

Every artist participating in the development process knows that though his contributions are a tiny portion of the final product, the rest of the team are dependent on his output for its success. Consider the outcome of the artist’s mindset, and how lacking it is in software development today: a development team of three to four hundred individuals capable of completing a three-year project on schedule and within budget repeatedly!

No, software development isn’t even close to being an art.

The animation industry has found that not everyone is alike. Someone who can give the appearance of weight and movement to fluffy bunnies is generally not as convincing when animating elephants or waves of water. Some can’t even animate at all, but are superb at choosing the colors or sounds or music to evoke a feeling of fright or happiness in a scene. So if an artist wants to do the animation for a particular on-screen personality or object they must be cast for the role, just as live-action actors are.

Let’s examine just how many artists it takes to make an animated film; specifically, Disney’s Hercules. According to the Yahoo! Movies entry for the film there were: 52 vocal artists; 2 directors; 5 screenwriters; 1 story artistic supervisor; 13 story team members; 3 producers; 1 associate producer; 1 color model artist; 1 scene planning/camera senior manager; 1 video reference production coordinator; 6 color stylists; 4 color stylist trainees; 7 color model mark-up artists; 1 digital film printing and opticals camera operations coordinator; 4 video reference cast artists; 1 video reference cameraman; 1 video reference crew camera assistant; 1 color timer; 7 digital film printing and opticals camera/film recorder operators; 2 associate editors; 1 additional editing member; 3 assistant editors; 1 dialogue editor; 1 supervising assistant editor; 3 negative cutters; 2 editors; 1 artistic coordinator; 1 character sculptures artist; 1 assistant artistic coordinator; 3 assistants supervising painting; 1 art director; 1 production designer; 1 costume designer; 1 recording assistant; 4 re-recording mixer; 1 original dialogue mixer; 1 ADR supervisor; 1 sound effects editor; 2 foley editors; 1 ADR assistant editor; 2 foley artists; 5 additional dialogue recorders; 1 supervising sound editor; 1 sound designer; 1 score recorder and mixer; 1 sound effects assistant editor; 1 composer; 1 lyricist; 1 executive music producer; 1 orchestrator; 1 song arranger; 1 song & score conductor; 1 vocals arranger; 1 song producer and arranger; 1 vocalist; 1 music production supervisor; 2 songs recorder and mixers; 1 additional score orchestrations; 1 assistant music editor; 2 orchestra contractors; 2 vocal contractors; 1 supervising music copyist; 2 additional song recorders; 2 music coordinators; 2 music editors; 1 dance choreographer; 12 supervising character animators; 69 character animators; 4 assistant character animators; 12 lead key clean-up animators; 38 in-between clean-up animators; 6 in-breakdown clean-up animators; 2 supervising key assistant clean-up animators; 59 key assistant clean-up animators; 43 assistant clean-up animators; 41 breakdown clean-up animators; 4 supervising visual effects animators; 27 visual effects animators; 1 effects animating assistant; 4 effects technical directors; 16 effects key assistants; 15 effects assistants; 28 effects in-betweeners; 6 effects breakdown animators; 4 effects animator trainees; 1 visual effects assistant scene set-up artist; 1 digital mark-up artist; 19 2-D animation processors; 2 senior animation checkers; 11 animation checkers; 8 scene planners; 3 scene planning and effects data entry; 35 background artists; 4 digital painters; 1 animation checker; 1 digital film printer; 12 key visual development artists; 1 paint/final checker; 7 CGI technical directors; 2 CGI look development & lighting; 1 CGI modeler; 19 rough in-betweeners; 12 journeyman; 1 color model assistant supervisor; 5 final checkers; 5 compositors; 1 animator editor; 1 assistant animator editor; 1 title designer; 1 key assistant; 1 layout key assistant; 1 layout stylist; 5 blue sketch artists; 5 dancers; 2 painting registration leads; 7 paint mark-up artists; 39 painters; 2 layout artistic supervisors; 2 visual effects artistic supervisors; 1 CGI artistic supervisor; 1 production stylist artistic supervisor; 2 backrounds artistic supervisor and 3 clean-up artistic supervisors.

This doesn’t include the producers and management, members of the orchestra, the staffs of the several outside production companies contracted in, and quite a few individuals left out of that Y! Movies crew listing. It also doesn’t count the administrative departments like accounting, legal, etc. which are just as necessary in producing the film you see because they help manage the production, and license or contract the artists and performers which are integral to the production.

To narrow this example down, and to make the point clearer, I’ll now drill down into the positions of the visual artists of the animation process to the animators. There are artists who draw, artists who paint, and also artists who make 3D models in wood or clay, called maquettes, of the characters. Of the artists who draw, there are storyboard artists, character designers, costume designers, set designers, prop designers and animators. Of the animators, there are character animators and effects animators. Of the character animators (and I’ll enumerate the animators for the grown up Hercules here), there is a supervising animator, one or more lead (or key) animators (10), assistant animators (unknown number), and rough-inbetweeners (unknown number), clean-up animators (1 lead key animator, 1 assistant animator, 8 in-betweeners, 8 key assistant animators, 6 assistant clean-up animators, and 2 breakdown animators).

The lead animator draws the key frames, or poses, of the character in a scene. His focus is on the movement of the character and his actions. The style isn’t polished and his drawings are called roughs for this reason. When his roughs have been filmed and the resulting pencil test approved by his higher-ups, they are handed off to the assistant animator. The assistant animator will add buttons, hair, and other character details pertinent to the scene. This work is also reviewed and pencil tested and when approved is given to the clean-up animator (Disney now uses rough in-betweeners to render in-betweens before the drawings are cleaned up to further specialize the animator’s role). The clean-up animator will trace the key drawings onto a clean sheet of paper, being careful to maintain the shape and flow of movement of the drawing while conforming to the proportions, appearance and details of the character. Finally, the inbetweener will draw the poses which are still missing, or in between, the key frames. There are moredetailed, and Disney Studios specific, descriptions of the roles on the net.

Think about all the steps required to draw a single scene in an animated cartoon and the specialization of the artists: three different animators are directly responsible for each key pose in the film! And another, or two, for the non-key pieces! All for a single character that appears onscreen. This collection of animators must create drawings that not only give the illusion of movement, weight and personality, but as if they were all created by an individual. The same goes for each character in the film. And all the combined elements of the film, the characters, sets, voices, music, etc. must present unified design principles that would appear to come from one mind.

We use a similar principle in the software we write: Single Responsibility. It is becoming a standard practice for the software we develop. For our teams to be more agile, it will have to be a requirement of our developers as well.

This strategy also applies to solopreneurs and microISVs. When you’re writing a marketing piece about your products, you don’t want to be in the middle of a phone conversation or chatting on IRC. Neither do you want to add error checking code when you’re writing out the logic of your functions. Focus on the one aspect of your code you’re dealing with at the time, then tackle the other issues separately and in their own phase.

Software Development Isn’t an Art III

One pitfall that programmers typically fall into is the need to just dive right into the code. (And production code at that.) They’re in such a hurry to do the job they enjoy the most that they fail to consider how to do it best. That means design. That means analyze the problem before you try to code the solution. That means test what you’ve come up with to validate it before you attempt to codify it.

Pulling examples from the animation profession again, here is a list of the test processes and artifacts used to validate the direction the production’s elements are going (or get it off the ground). There are story treatments, scripts (AKA screenplay), storyboards, and animatics (leica reels) which go through constant revision. There are concept drawings and paintings, character sketches, model sheets, maquettes, set designs, and color teststo develop the look and feel of the film. There are musical scores, bar sheets (AKA dope sheets), and exposure sheets (X-sheets) to work out the timing of the vocals and music with the action. There are layouts to determine camera angles and how motion will be staged.

Even after all this design and timing work has been done, an animator will thumbnail how his drawings will progress before actually beginning (or if he needs to, rethink) his drawing. As far as the animated drawings are concerned, there are roughs which are pencil test and studied before the inbetweens which are then cleaned up before the final ink & paint, and none of these drawings are even seen in the final production. In CGI they use wireframes to review the animation and camera movements before a scene’s rendered frames are generated.

As software developers, we have quite a few tools available to us to speed up the development process. There is the ol’ flowchart and pseudocode. Or we can go with a more in-depth, and sophisticated, Nassi-Schneiderman, Warnier-Orr, Jackson Structured Programming, or UML diagramming notation.

There are Story Cards, CRC Cards and my personal, more permanent, whiteboard substitute.

We can write Use Cases, Unit Tests and Functional Tests (also known as acceptance tests).

There are even Spike Solutions and Prototypes to test ideas, not just UIs, before committing too much effort and cost to the wrong direction.

How many of these or other non-production materials do you use to design your software and provide documentation for new members to your team (think agile communication that scales)? The importance of these tools in improving the quality of your code cannot be understated. Better designs make for better code.

If you read the comments for the link to bar sheets above, you’ll see that this issue of pre-planning strikes at the heart of the quality of the final product. Particularly this comment (which I’m reproducing here because Safari doesn’t scroll directly to it):

Stephen Worth said…

The most sloppily drawn, poorly timed television programs have the highest budgets. It doesn’t cost any more to draw well and plan out your timing. In fact, careful timing can save money…

It eliminates the need to “trim” shows to “make them play”, cutting down on the amount of wasted animation.

If action is timed to a beat, even if it’s a bit slow or a bit fast, it still feels “right”. When you time with no rhythm, you have to do multiple pencil tests and revise your work to parallel park it into looking “natural”.[Emphasis mine. — AG]

Animation is the one medium where every single aspect of what is on the screen can be carefully planned. [O RLY? — AG] If a director chooses to ignore some aspect and let it go to chance, that just means he’s a lazy director. It has nothing to do with money.

We have quite a few tools available to us as software developers. We just need to exercise the discipline to use them if we’re going to raise the quality of our work and be recognized as a true engineering profession.

Software Development Isn’t An Art IV

Let’s face it. The software industry just doesn’t eat its own dog food.

Debuggers? What are they for? Why doesn’t the system point out when memory is clobberred or a pointer is NULL?

Hardware is cheap. Software development costs are much higher. Why is that? Hardware developers have no trouble using simulators in the development process. The cost for building first-runs is quantifiable, as is the errors and rework associated with them. Hardware engineers and their managers can see the cost of defects in manufactured products is too high not to use additional tools, such as mock-ups and prototypes, to aid designers. So why don’t software projects also use simulations and other tools to improve the quality of the process? Poor management. Proud and arrogant developers.

Programming today is done much the way animating was during the twenties, before Walt Disney pushed for process improvements in his studio. After a director selected a storyline, the animators would have to produce a certain amount of feet of film, and come up with the gags and bits of story to fill it. Then the work produced by the various animators would be spliced together and, hopefully, they had a reel where each animator’s contributions formed a somewhat coherent plot.

Visual gags were a requirement because the story and character development were not considered important. So the use of gags and other gimmicks became necessary to sell a cartoon to film distributors.

Code shouldn’t be designed as it’s written. If you believe this contradicts the agile philosophy, in particular XP, you’re probably opposed to TDD as well which is a heavier process method than just straight writing code. Writing the actual code to be used in a production environment should be one of the last steps in the development process. A lot of effort must go into reviewing and validating the requirements, the implementation languages and the development toolchain, the architecture and interfaces, the components’ roles and interactions, the execution platform, etc. before the production code itself can be written.

If you do not have the behavior of the code firmly in mind before you begin writing it, you cannot reasonably expect it to behave well with others, and you will be doing a lot of rewrites. And debugging. A lot of it.

Software Development Isn’t an Art IV

For precept must be upon precept, precept upon precept; line upon line; here a little, and there a little. — Isaiah 28:10

The identification and use of design patterns has moved our understanding and the current state of our profession forward. But without an understanding of the critical factors for developing and successfully completing quality software projects, stagnation at the current state is all that can occur. And stagnation is all that’s occurring due to our unwillingness to exert more rigor in the design of software.

When Walt made storytelling the linchpin of his animated films, the other studios were unable to compete on a long-term basis regardless of the gimmicks used. They hired away his animators. They hired away his composer. They hired some of his storymen. But they didn’t use his process of refinement and experimentation to develop and improve the stories they told, and it reflected in the quality of their films.

Developing software without a written, documented and tested plan is like making a movie without developing the characters and the plot. Or, to use a metaphor you might find easier to relate to, like building a house without a blueprint. You’re just eyeballing it and you’ll have to continue making adjustments to your code as you integrate it into the rest of the system. And hope it works out for the best.

But you know it won’t. You know you”ll have to spend more time trying to get components to work together properly. You know you’ll have to redo production code and unit tests repeatedly. You know customers will change requirements that will adversely affect the system’s integrity downstream. Does it seem like a good idea to add extra work to your plate because you didn’t take the time to execute a proper design in the first place?

The design isn’t in your code. “If” statements aren’t design constructs. Neither are functions. They’re execution constructs meant for computers to execute.

Design languages help you understand and analyze designs. They help you switch contexts much quicker than code ever could. So when you design, use the language of design; don’t use code. Code is for computers to follow, not people.

Measurements are an integral part of the testing process. In manufacturing processes, they provide an indication of how the final product will turn out. If the temperature is too high, resins will become too fragile. If raw materials are left in the open air too long, the loaves of bread won’t have the right texture.

Note the variation that was acceptable at the most commercially successful, and highest quality, studio at the time. And yet they still measured raw output. Other studios enforced a more inflexible production quota (bean counters!), and their work suffered as a result.

Measurement is an indirect means of testing the velocity and quality of a process, and is as valid for software development as it is for animation.