Avoiding Development Mercantilism

(Note: This has been cross-posted to the Guidewire Development Blog.)

Prior to the rise of the capitalist view of economics, the prevailing economic world view was known as mercantilism. Mercantlism was essentially based on the theory that trade was a zero-sum game; there was a limited amount of wealth (in the form of gold or silver or other bullion) in the world, and one nation having more of it inevitably meant that other nations had less of it. One of the revolutions of the capitalist view was in recognizing that wealth is not, in fact, a zero-sum game, and that the amount of wealth in the world can increase as a result of increases in productivity due to things like economies of scale and technological improvements. The way to become richer as a nation is not, in other words, merely to try to make sure that more wealth enters the nation than leaves it via trade imbalances; instead, you can and should look for ways to increase the total amount of wealth being produced.

That's perhaps an overly-simplistic summary of things, but the point here is not to debate economics. Rather, it's to point out a common mistake that many development organizations make: they view development output as a fixed commodity just as the mercantilists viewed the amount of wealth in the world as fixed. But as every developer knows, development output is in no way fixed, and is highly dependent on factors such as the toolset being used, the fit of the developer's skillset and experience and temperment to a particular task, the developer's enthusiasm for the task, the current state of the code base (size, cleanliness, documentation), and the amount of organizational drag (in the form of meetings, reports, e-mails, etc.).

So what does it mean to be a development capitalist? The easier part, in my experience, is in matching people to the right tasks and in trying to reduce the amount of organizational overhead. Those, at least, are easy decisions to make, as they tend not to come with too much potential risk or up-front cost. The hard decisions, then, are around technical considerations: how much time do you spend building infrastructure and tools, and how much time do you spend trying to keep the code base clean and small? Both those efforts, and especially the effort to build infrastructure or tools, come with a potentially huge up-front cost and, at best, a speculative payout down the road. It's tempting, then, to simply see them as costs to be minimized. Doing so, however, can miss the huge potential productivity gains you can get down the line that will more than make up for any up-front investment.

As an example, consider what I spent my half of my day doing on Friday: performance tuning a couple of critical tests that I (and presumably everyone else on the team) run every time before checking in. The tests verify that all of our UI configuration files and gscript classes are error-free, and as such they catch a ton of errors and it's critical to keep them clean. I probably run the tests an average of about 10 times a day, and before I started I had to run two tests that, combined, took about 200 seconds to execute. The tests had some overlap, however, so I could tell that there might be some low-hanging fruit for further optimization, and about four hours of work later I had managed to combine the tests into a single test that took 140 seconds to execute. Four hours of work to save 60 seconds of test execute time might seem like a bad return on investment, but it won't take long to pay off. At my rate of test execution, it'll take about 24 days of development time for me to make back that four hours: but there are also seven other people on my team that will probably save about 5 minutes a day each, and another 40 or so across development that will eventually benefit from the optimizations.

So was the time worth it? It's always a hard call, and it can be hard to know when to stop; we certainly can't spend all our time building infrastructure or we'll never get our products out the door. We've got release deadlines just like everyone else, so it's always easy to say "we'll do that later, we can't afford the cost right now." But the payoff for taking a few days (or weeks or months) up-front to do things with seemingly small payoffs can, over the long-haul with months (or years) of development time ahead and dozens of programmers working with that code base, lead to huge time-savings and huge productivity increases for the team. Eventually it can make the difference between development grinding to a halt under the weight of an ever-growing code base and maintaining the ability to make steady forward progress.

That will really only happen if you have an organizational commitment to taking the long view and recognizing the long-term benefits of making a constant investment in your infrastructure, tooling, and code base quality. It also requires giving developers the freedom to make those improvements when they see an opportunity as well as the freedom to take some risks; not all such bets are going to pay off, and I could very well have spent four hours on Friday without being able to improve the performance of those tests in the least. Being willing to take those risks and let people scratch their particular technological itches every now and then will almost always pay off in the long run, and in my experience such investments usually pay off much faster (within weeks or months) than you initially think they will. And the next time you find yourself dividing a project into developer-days or man-months (which we should all know are Mythical anyway), make sure to ask yourself if you're falling into a mercantilist mindset where all costs are fixed instead of looking for ways to make the whole organization more effective.

Favoring Composition Over Inheritance

(Note: I'm now posting on the Guidewire Development Blog, so I'll be cross-posting to this blog with everything I deem interesting enough.)

It's something of a widely-stated belief in Object Oriented Programming circles that composition is often a better idea than inheritance, but my personal experience has been that, like so many other software best practices, it's more often said than done, and that in practice most developers (myself included) often revert to simple inheritance when they want to share behavior between two classes. There are reasons for that, of course, especially in a language like Java that offers no built-in support for composition: inheritance is generally easy to code, maintain, and understand and generally leads to less code than inheritance requires.

The path of least resistance, of course, is not always the best long-term strategy, and there are compelling reasons to prefer a compositional model instead of an inheritance model, especially with larger and more complex code bases. As a result, much of my time over the last several months has been devoted to untangling some of the more overgrown inheritance hierarchies that we've developed over the years and replacing them with cleaner compositional models.

Before going any further, it might help to clarify some terminology. Inheritance is the standard OO practice of subclassing something, inheriting all of its fields and methods and then overriding and/or adding to them. Composition can mean one of a few different things, but generally it refers to the idea of having behaviors defined in a separate class that your class then calls through to, commonly known as delegation (and the helper class is commonly known as a delegate). In practice, I almost always combine a delegation approach with an interface that my class implements by delegating each method to the delegate class. For example, if your ORM framework maps database rows to objects and you want the generic ability to ask if an object can be viewed by a given user, you could implement it with inheritance by creating a superclass for all your objects and adding a canView(User user) method to it (or simply adding to the existing superclass) or you could create a Viewable interface with a canView method on it, a ViewableDelegate class that provides a standard implementation of the canView method, and then each object would independently implement the Viewable interface by calling through to the ViewableDelegate object (generally held as a private instance variable on the class).

Clearly the delegation approach is, at least in Java, much more work to code and maintain than the inheritance approach. If you have 20 Viewable objects in your system and you decide to add a canCurrentUserView() method, if you're using inheritance you only have to add the method in one place, whereas with composition you'll need to modify the interface, the delegate, and then all 20 classes that make use of the delegate.

In spite of those obvious drawbacks, though, I've come around to the conclusion that I should be using composition much more than I have in the past, for the following reasons:

• Cleaner abstractions and better encapsulation. In my opinion this one is the most important reason to avoid deep inheritance hierarchies. With inheritance, there's a temptation to use it even when there isn't really an is-a relationship between two classes simply because it allows you to easily reuse code. Unfortunately, that tends to lead to fuzzy abstractions at the top of the hierarchy: your base classes begin to acquire a lot of methods that only apply to some subtypes simply because it's a convenient place to put things, and pretty soon your base class starts looking like it should be named "Thing" because the methods on it don't support any one unifying concept. Splitting the base class out into several smaller interfaces, then implementing the interfaces on an as-needed basis on the subtypes, makes the abstractions much clearer and better encapsulated, which makes the whole system easier to understand.
• More testable code. Along with cleaning up the abstractions and better encapsulating code comes improved testability. Classes with deep inheritance hierarchies are generally very difficult to test; it generally becomes impossible to test the subtype without also testing its supertype, and it's difficult to test the supertype in isolation from its subtypes. A delegation model can clean that up, as you can often more easily test the delegate in isolation and then have the choice of testing the classes that use the delegate either via an interaction model (to just test that they do indeed call through to the delegate) or via a state model (i.e. actually testing that they implement the interface and treating the delegate as an implementation detail). Either way it becomes more obvious what exactly to test and easier to do it.
• Decoupling. Another important factor on large systems (like, say, a Policy Administration or Claims system) is to keep coupling to a minimum. The only realistic way to deal with the complexity of such a large application is to keep it as compartmentalized as possible so that you only have to worry about one part at a time, and a high degree of coupling means that you can't make changes to one part of the application without worrying about how they might affect other seemingly-unrelated parts, which naturally leads to a higher probability of introducing bugs with any given change. By letting you define tighter abstractions and discouraging unnecessary code sharing, compositional models tend to reduce coupling to the absolute minimum (i.e. the interface boundaries). In addition, if you find that you need to further decouple things by splitting out sub-interfaces or having different delegates that implement the interface differently it's much easier to do than it is if everything inherits from the same superclass. And as painful as it can be to modify lots of code when you introduce a new method to an interface, you're at least forced to think whether it really makes sense in all those places: with an inheritance model it's far too easy to just add something to the superclass and never think about whether or not it makes sense for each of its subclasses to have that method on them, which quickly makes it difficult to figure out which of those methods is actually safe to call from a particular subclass.

While Java makes composition fairly difficult, languages like C++ that feature multiple inheritance give you a way out of this by simply inheriting from multiple parents, though that can lead to dangerous ambiguities and even more confusion. Dynamically-typed languages make composition much easier, as the desired methods can simply be added onto the class at runtime; Ruby does this via its mixin facilities while you could do it in JavaScript simply by programmatically attaching the methods you want directly to the prototype objects (or directly onto the instances, for that matter).

Making it work well in Java is, unfortunately, more difficult. If you don't like doing the delegation by hand, Java does give you a few other options. The simplest one is brute-force code generation; for example, we already generate code for our domain objects, so we've simply added the ability to specify interfaces and delegate classes for the code generator to add in when it generates the domain objects. If you're feeling more ambitious, you can also do the composition dynamically at runtime, using either the Java Proxy class or (taking it one step further) dynamically generating a java class using a library like javassist. It's not always easy, but the conceptual clarity that comes with using composition instead of inheritance is often worth the tradeoff.