Components should automate their own usage

(Achieving high productivity in complex software environments) Peter Ruttan Netron Inc. pruttan@netron.com, (416)636-8333 November 10, 1997

Introduction

Everyone has an appreciation for the complexities of dealing with multiple hardware platforms, operating systems, communications protocols, languages, etc. Standards will reduce the complexity in these areas, as will industry consolidation. Rather than focusing on those phenomena, this paper explores how to enable a development staff to be productive in spite of the environmental complexities that burden the system.

Let us accept the following facts:

there will be an explosion in the number of problems that are solved with new technologies throughout our lifetime;
it will be impossible for any efforts at global scale standardization to keep up with technological advancements;
it is going to be necessary to make selective use of bleeding edge technology to maintain a competitive position in the market place.

All of this means that the complexities of today's software development are inescapable, presenting an everlasting challenge; it does keep life interesting.

Being a bold optimist, the author is confident that every complex problem facing developers today can be solved. The challenge is to ensure that each problem is assigned to someone who can solve it. It is unlikely that any one person can know enough to solve all problems. Even if that were possible, we can't afford to be dependent on any one individual to solve every problem.

Achieving high productivity requires establishment of an environment that allows technological problems to be solved without bringing development to a grinding halt. To accomplish this we need several things:

the ability to isolate the technological problems from the business problems;
the ability to evolve business solutions knowing -- not hoping but knowing -- that when the technological problem is solved, the solution can be slotted into its place in the application without reworking the application;
the ability to decide if the technological problem is not ready to be solved. When making this decision, it must be possible to slot a low-tech solution in place of the high-tech one, and have a system that provides the necessary business functionality; and,
most important, the assurance that staff will never have to solve the same problem twice.

Characterizing the Problem

Let's focus on source code, remembering that this is only part of the development process, and that all of the parts are interrelated (that's why we refer to it as a life cycle). In the end, source code is the one thing we can't skip if we're going to get a system into production. All solutions to all of our business problems and the implementation of all of our standards must get rendered in source code. And, no, we won't quibble over whether we wrote, rented, or bought the solution.

There is an aspect of writing software that is not getting enough attention: Too many people are writing too much code over and over again. We spend all of our time solving difficult problems that are very interesting and in doing so ignore the obvious. Here's an example:

For years, many people have written about the different layers of abstraction appropriate for communications protocols. So many people have thought about this problem that we have quite a few standards to choose from. Vendors have figured out that implementing these standards is tricky enough that people will pay to have it done for them. This gives us a number of products that we can purchase to handle the hard part of getting application components to talk to each other. Without belaboring the fact that most of these vendors add a little something extra to improve upon the standards, or address some dimension of the problem space that the standard hasn't gotten to yet, it bears mentioning that we are still left needing to write fussy code to make use of these middleware packages. IDL, of course, comes to mind, but IDL really just displaces the problem to being one of needing to write around the fussy middleware code.

There are two principal concerns about this approach to handling technology problems:

We tend to stop before we've asked ourselves if we have made life as simple as possible for an application programmer. In most cases, taking that last step towards hiding unnecessary complexity would be a tiny amount of incremental effort.
We tend to solve the tough problem before we give application developers the information they need to do their job (i.e. writing an application that will make use of our "solution").

We don't have to know how we are going to solve a problem to know what information we need from the application or what information we are going to give back to the application when we're done. In fact, virtually all development methodologies say that you should define your inputs and outputs before you address the internals. But still, we tend to defer clearly defining to application developers what they need to do until after we've got the hard stuff ready for production.

The result of these two behaviors is:

We are demanding specialized knowledge of an entire community when localized specialization is all that is needed.
We are solving problems linearly when they could be solved in parallel.
A less obvious consequence of this approach to problem solving is that by requiring the application developer to be expert in both application issues and technological issues, we are increasing the likelihood that the application code and the technological code will be intertwined.

A quick survey of the source code for most applications will show that in spite of our best intentions, code dealing with technology is tightly coupled to code dealing directly with the application at hand. In the interest of meeting deadlines, it is our nature to cheat and defer structuring the application for maintainability. We all know that once we defer the clarity and structure required to have a maintainable application, it is deferred forever. Once an application is functionally ready for production, the risk of breaking the functionality makes it virtually impossible to adjust the code for the sake of making it better.

Solving the problem: make components usable and stable

So what can we do? There is no question that component-based development will improve our lot. It will force us to be more conscious of problem decomposition. We still have a huge amount to learn about the appropriate level of granularity of components. This will come over time. The rapid pace of improvements in hardware performance makes it much easier for us to optimize for clear boundaries between components and accept the performance penalty that highly layered call structures introduce.

While we are going through the natural learning process, we need to fundamentally change our measure of success in producing components for reuse. In addition to the fairly routine evaluation based on the architecture of the component, we need to consider a component's usability and stability:

How much does a developer have to know to use the component?
Does the component include guidance for how it is to be invoked? Or better, does the component automate the creation of the code required to use itself?
Is the component able to identify itself and its domain to a developer who may need to use it?
Can the component be specialized without a compromise in integrity?
Can the component be replaced? Can it be replaced without any side effects? Can it be replaced without the user needing to know or care?
Can the component's domain be extended without any impact on current users?

These questions need to be explicitly asked from the point of view of an application developer. Far too often, it is the developer's perception that it is easier to just write the code than it is to find and figure out how to use a component.

Why is it so hard to inventory components? The problem domain is much smaller than that of managing the parts used to, say, build a 747. The parts are all in a readable form so relevant information about them can be extracted by a relatively simple utility. The only apparent constraint is the discipline to make the cataloging of the component an inherent part of the creation of the component.

The author's organization took this approach, leading to the conclusion that the incremental effort is tiny and the payoff enormous. Components can actually be found in not much more time than it would take to decide what to name a new component.

Once a developer finds a component, the next challenge is to use the component correctly. This means figuring out what source code has to be written to (a) establish a connection to the component and (b) invoke the component with the right parameters and calling conventions.

Having committed to inventorying components, the next logical step is to automate the usage of each component. In essence this means associating with the component some wizard directives that can drive a dialog with a prospective user and translate the developer's responses into the source code required to achieve the desired objective. The degrees of freedom and the calling conventions of a component can, and should, be available from the component itself.

If we are going to the trouble of designing and building well-defined components, the incremental effort to capture the usage rules is relatively small. Substantial gains are realized because:

the developers' work is simplified because they can think in terms of functionality rather than ordering and values of parameters;
there are fewer errors because the syntax is always correct, the parameters are always in the right order and the parameters always have legal values; and,
the developers do not work with out-of-date documentation because the documentation is derived from the component itself.

Anyone who has been watching closely will have noted that in making it easy for developers to use a component, an abstraction layer is created around the use of the component. The existence of this abstraction layer affords the ability to evolve the component at no risk to a user of the component. Many people who have tried to upgrade a component have noticed that the biggest constraint is the retrofit of users of the component.

Making the component easy for a developer to use also solved, in advance, the problem of retrofitting the use of changing components.

In the end, by being careful up front about the domain of a component, clear about the definition of input and output data fields and not considering the component development to be complete until its usage by a developer has been automated, productivity was more than doubled throughout the development life cycle.