Jimako’s Blog

 

 

There’s something fundamentally broken with the whole software development industry. And no, I am not talking about the process of creating software itself. I am talking about the way we interface with the rest of the business world, the way we are perceived and the crazy expectations that are placed upon us.

If I see one more "design" that has been done by a design house, or a usability consultant, or an interaction designer, I swear I will scream.

Look, people. You can’t design something if you don’t know anything about how it might be implemented. I’ve sat in on meeting after meeting where a design consultant or an interaction designer, with a perfectly straight face, told a client that the designs they produce are "not constrained by the technology" so that they are able to create "the best possible experience" or "have the most impact" or whatever buzz-words are current.

Really?

How great is your design if it can’t actually be implemented? The design is not the end product. It is simply a tool that has just one objective: to move us towards the actual product. I don’t really like analogies (even though I seem to be using a lot of them on this blog), but here’s another one for you. 

If you commissioned a design for a building, would you accept a design that had not done the basic calculations about whether it could actually be built? A disclaimer along the lines of "we don’t consider actual material load capacities as we don’t want our design constrained by the technology" would immediately be recognised as absurd. While that massive open auditorium looks great in the design, the builders will actually need to put lots of pillars at various places to support the roof. And the builder (even in consultation with the building owner) will probably not really be able to optimise the placement of those pillars to optimise the viewing angles, accoustics and seating capacity of the auditorium.

In reality, the design firm did no design at all. They abdicated the hard design decisions and left them for the builders and the owner. What they did do is create an unrealistic expectation about what is possible, and used a whole lot of the time and dollar budget of the project. And because it’s an "artistic" exercise, I will bet you dollars to donuts that it took much longer and cost much more than was originally estimated. After all, you can’t do "design" to a timetable — creativity marches to its own beat.

Recently I’ve been involved in a project where a major usability firm produced what they called a functional specification. What it actually turned out to be was a thin document about how various workshops had been run, a few JPGs showing mocked-up screens obviously created in Powerpoint (their IDE of choice) and a spreadsheet that was to be considered a data dictionary. The client had requested that the design firm provide them with the HTML for the interface so that it could be used as a template for the working system. What the provided was a set of HTML pages that contained exactly one image each — yes, you guessed it, the JPGs already mentioned.

The system was to provide a browser-based UI to a legacy back-end system. The UI was designed to streamline the workflow so that users would be able to use the system without extensive training, even if they only accessed it intermittently and at wide intervals. It was a thing of beauty.

It’s a real pity that the new UI needed data elements and processing logic that the back-end system just doesn’t have — after all, that’s just a technology detail. Or the screen that requires 4 different calls to very slow web services per displayed row and so takes several minutes before it can be rendered — surely "the technology people" can just make it faster.

And the icing on the cake? Even though the "construction" of the software didn’t start until several months after it was scheduled (because, as is so often the case, the design took longer than expected), the delivery date couldn’t slip, because commitments had been made to the powers that be. Couldn’t we just compress the timeframes for this last little bit?

AAARRRGGHHH!!!

Let me be very clear.

Software development is NOT a construction activity. We are REALLY, REALLY efficient in constructing software. It’s called a build, and even with full automated regression testing and QA, it takes minutes to hours to do. That’s it — period. Construction is so cheap and easy that it is essentially free.

Software development is actually a design activity. 

It starts out by doing a high-level design, mapping out the major components and identifying the technology stack and supporting ecosystem of tools and libraries. It then works at different levels of abstraction and granularity, generally from more general and broad to more specific and focused, but often diving deep into the details of particular areas that are considered tricky or high risk. 

The most detailed level of design is done in a programming language, whether a standard language like C#, Java or Ruby, or a Domain Specific Language (or DSL) specifically crafted to support the particular problem domain — I consider Ruby on Rails an example of the latter even though it is technically a Ruby framework and not a separate language.

As all this design is going on, situations are uncovered that likely nobody had considered before. These edge-cases need business input, and sometimes they force quite significant redesign of the entire system. They can be encountered at any time, but in most cases they are only uncovered by the act of writing the code. 

A developer gets to a point where she types "else" and then looks blankly at the screen, totally at a loss as to how to handle that case. It had literally not occurred to anyone that this was a possible combination of the various system states and inputs. It is not that someone had "failed" in the design phase, it’s just that we only now got to a sufficiently detailed level of design that the situation became discernable.

And the answer is not more high-level design up front. At the end of the day, you need to get to the absolutely most fine-grained level of detail to fully flesh out a design, and that’s what coding is all about. When a developer is working at a high level, she is coding using a handful of basic constructs (like sequence, selection and iteration) to control a bunch of what are often as-yet unwritten code blobs (whether these blobs are objects or procedures is irrelevant). When she is working at the detail level, she is focusing on writing specific data to the database, handling specific input values from the screen, performing an actual calculation or transformation on some data items. And throughout a typical work day, she will switch between these two extremes and countless intermediate levels many times.

This is the type of design that is required to deliver a functioning system. Design is code and code is design. "Design" using a design tool is simply using a different notation to do the same thing, except that notation can’t express the most detailed levels of design in a clear and unambiguous manner. I say that confidently because if it COULD do those things, then by definition it would be a programming language, the ultimate design tool.

So, how does a smart company manage a software development project? Here’s how I would do it.

First, don’t commit to a delivery date yet. How can any date you put forward make any sense at all? Wait until you have more data. Get approval for a high-level design project only.

Now, don’t spend your budget on a consultant who is not going to be part of the final delivery. If I don’t have to worry about delivery, I can put anything into a Word document. Instead, find a technology partner that you can trust, and get that partner to gather all the required skills for you.

You see, I am not discounting the value of interaction or interface design — in fact, I think that good design of these aspects is often the difference between a barely usable application and an outstanding success. It’s just that I think that they need to report to both the business and the team charged with implementation, so that the designs meet the business requirements and can in fact be built.

So, how do you find a technology partner you can trust? It’s actually a lot simpler than you think, but it is not easy to do in most large enterprises. The way you do it is the way you would select any craftsman. By all means look at what they have done (in the guise of references and recommendations); after all, you need to ensure that they have some record of success. But the number one factor you need to consider is this: do they understand what you are trying to do and are you confident that they will work with you to make it happen? Talk to the people you will be working with. Insist on this — talking to the sales guy is worse than useless: it is dangerous as you’ll be told what you would like to hear rather than the truth.

Also, I would not lock in a long-term contract. Instead, I would engage them on a month-to-month basis so that I could terminate our arrangement if it wasn’t working out, if they tried to slip in a bunch of juniors, or simply stopped paying me attention. The flip side is that they could also fire me if I was becoming unreasonable. It’s amazing how focused both sides become when they need to work at preserving the relationship.

Then, as the initial design starts to firm up (including the interaction and interface design), I would be seeking from the technology partner and indication of the expected time to implement. Don’t impose a level of precision on them at this stage — it doesn’t work like that. If you’ve just managed to pin down the scope of the project and have a bunch of wireframes to look at, asking for a +/- 10% estimate is totally unrealistic. 

Yes, you would like that level of precision but no, you can’t actually have it. In reality, at that stage, the highest possible level of precision is one close order of magnitude (that is, one power of 2, or +100%/-50%). And that’s the best possible case. It’s not until much later in the project, typically when the majority of the coding is done and the users are getting a chance to play with it that we are able to have a precision as high as 10%.

For budget purposes, get approval for some of the work, and use an iterative development process (typically, some form of agile methodology) to deliver some of the highest-value functionality.

Lather – Rinse – Repeat.

As time goes on, both the business and the technology partner get a feel for the complexities or otherwise of the various parts of the solution, and the quantum of functionality that can be delivered for a given time and dollar investment.

And if you find that your technology partner is dropping the ball, then change them! Changing is not free — there will be some down time, and the replacement will need some time to come up to speed, but at least you have the option to do so. Just make sure that all the build artifacts are yours — the source code repository, the build system, the testing infrastructure, the "stories" or equivalent documentation.

Of course, this actually is almost impossible to do in a large enterprise. Management wants firm and precise estimates for budgeting purposes, and can’t understand why they can’t get them. The perception is that software development folks are just refusing to play by the rules, that they just don’t want to make a commitment. 

I just don’t know how to get the message across. Asking a software development team (or for that matter an individual) to provide estimates based on the broad and often internally contradictory wish-list of functionality that is titled a Request For Tender is a bit like going to a builder and asking for a quote for "a building that will have between 20 and 200 stories and will be built on a location as yet to be determined". It just doesn’t work that way in the real world, and software development is orders of magnitude more complex than constructing a building, even if it is less physically demanding.

I would like to think that we are getting better at developing software. I actually think that we are. What we are failing at abysmally is getting our customers to understand the reality of what we do and work with us in their best interests. They’ve been burnt too many times by the smooth-talking consulting organisations who promise the world and will agree to the most rediculous engagement terms just to get in with a client, relying on lock-in to keep them there.

And you know what? At the end of the day, these consulting firms get away with this behaviour because the clients let them. I’ve lost lots of work when I gave a client an honest estimate of the work, saying that I estimated it at between 9 and 15 team months when a competitor agreed to the client’s "estimate" of 3 months. Two full years later, the project is still not completed and the client has spent far more than was originally planned. The winner here? The competitor has been paid for a team of developers for the whole two years. I won’t name the competitor, but it is a big name, multi-national consulting organisation that you have definitely heard of. Whose fault is it? I would suggest that the client got what they deserved.

If someone is telling you what you exactly want to hear, maybe you should tread with caution.

 

No comments