In the waterfall development model, what is the most expensive part of software development?

Of all the ways of approaching the software development lifecycle, the waterfall is probably the worst. So why does it continue to dominate our thinking? Simple: We have failed to let everyone know how software development really works.

Although few projects would admit to using anything as flawed as the waterfall software development lifecycle, it continues to impose linear, mechanical thinking on our projects. We need to change the conversation in order to prevent the waterfall lifecycle from damaging any more projects.

Some Flaws

The waterfall approach assumes that it's possible to capture all requirements and complete analysis before design starts. For most projects this is ludicrous; by the time you've written down all of the requirements, they will have changed. Unfortunately, it's also very expensive to write down all of the requirements, because people are not used to specifying things in minute detail.

The waterfall approach is expensive and takes too long because it was designed to minimize the use of very expensive computing resources. Guess what? Things have changed. Computer time might have been very expensive back in 1970, but now development time is the most expensive part of a project budget.

The worst part, however, is the assumption that it's possible to separate software development activities into nice, neat, non-overlapping phases. Many problems are wicked problems, in which the design overlaps with and influences the requirements. With wicked problems, new requirements emerge out of the design decisions, making it impossible to create a clean separation between requirements and design.

But Nobody Really Uses the Waterfall Method, Right?

You might think that with all of the material that has been written over the years that some form of incremental development would be the default strategy. Guess again. I've lost count of the number of project plans I've seen that show systems analysts finishing their analysis work, then handing their work off to designers, who then hand specifications off to programmers, who then pass their work off to testers. It sure looks like the waterfall is alive and well!

Even the labels we assign to the roles in projects match those of the waterfall. It should also come as no surprise that the jobs at the top of the waterfall have higher status and rewards than the jobs at the bottom. No wonder systems analysts look down on lowly maintenance programmers.

In an era where companies are starting to talk about knowledge management, we still organize many software projects in a way that's guaranteed to minimize knowledge transfer. A systems analyst takes the time to learn the problem domain and then hands the work to a designer. The designer has to learn secondhand, and then subdivide and parcel up that knowledge into detailed specifications that get handed off to the programmers. Is this inefficient, or what?

But Are the Handoffs a Real Problem?

No, not from a linear, mechanical way of thinking. It's the most effective way to do it. Systems analysts can specialize and become good at requirements elicitation. Designers can specialize in creating detailed designs, and programmers can specialize in their favorite programming language. Sounds great, doesn't it?

Well it would be, if it were not for those wicked little problems that creep into the picture. If we've learned anything about software development, we know that each new system enables our users to think up completely new ideas for must-have features. But guess what? The user doesn't know who to tell these ideas to. Should she tell the maintenance programmer about the enhancement request? Or maybe the tester who is doing the usability testing with her? Or maybe the programmer who was asking her to decide between two different screen layouts? Or maybe the designer who was asking about the subtleties of one of the business rules? Or maybe the analyst who interviewed everyone 18 months ago?

Whoever she tells, it will be the wrong person, because nobody on the development side really understand the big picture.

The Waterfall Creates Schedule Slips

The most fundamental problem with the waterfall approach is that it's impossible to measure progress. Sure, it looks great that we can say we're 90% done on an analysis task, but what does that 90% mean? Not very much.

The problem is that you can always do more analysis to try to uncover more subtleties about the problem domain. At some point you have to say that you've done enough, but you can never be sure. The same goes for the design phase. How do you know whether your elegant design ideas are really going to work when the programmers implement them? How do you know the level of detail to put into the design documentation so that the programmer can understand the design—can you assume that the programmer is familiar with the problem domain or not?

Small wonder that most PERT charts show downstream tasks slipping and overrunning their due dates. The up-front tasks came in on schedule, so it must be the designers, programmers, and testers who are at fault. Yeah, I really believe that.

These slips are a natural consequence of the waterfall approach. And guess what? They put severe schedule pressure on the programming and testing activities. No wonder the quality of the resulting application is low. No wonder there's the temptation to ship "good enough" software.

## Software development life-cycle (SDLC)

The software waterfall model often uses some variation of following phases:

• Requirements specification (Requirements Analysis) resulting in Requirements Document
• Software design resulting in Software Design (SD) document
• Implementation resulting in the actual software
• Integration
• Testing (or Validation)
• Deployment (or Installation)
• Maintenance
  

This model expects that each phase is finished before the next one starts. Between each of those phases there are often “gates” that consist of reviews of produced material. In order to pass those “gates”, review of produced material needs to confirm that the previous phase is indeed complete. It usually ends with some form of a sign off. For example, requirements specification can be reviewed by customers. They are expected to sign off the document thus confirming that the result of the requirements phase (for example requirements specification document) is satisfactory and the team can proceed to the design.

Waterfall model discourages revisiting and revising any completed phase. For that reason there are “gates” that are acting as barriers towards the next phase. Each needs to be “perfect” since there is no going back.

McConnell shows that a bug found in the early stages (such as requirements specification or design) is cheaper in money, effort, and time to fix than the same bug found later on in the process. Waterfall responds to this claiming that time spent early on making sure requirements and design are correct saves much time and effort later. Thus, the thinking of those who follow the waterfall process goes, make sure each phase is 100% complete and absolutely correct before proceeding to the next phase. Program requirements should be set in stone before design begins (otherwise work put into a design based on incorrect requirements is wasted). Same can be said for the rest of phases. Implementation should not start until the design is 100% complete. Integration needs implementation, testing needs integration… Each phase should be “perfect” before people begin to implement the next one.

Criticism

Waterfall model was a good start in the software industry at the beginning. There was no other model to compare with. In the mean time, problems started to become evident.

In almost all but non-trivial projects it is impossible to finish a phase perfectly. Customers often cannot know all the details of the end product up front, design is prone to fail to be 100% accurate without doing part of the implementation, testing will find defects that might require the team to change the design.

“Many of the details only become known to us as we progress in the implementation. Some of the things that we learn invalidate our design and we must backtrack.” – David Parnas

One of the side effects of the waterfall model is creation of “silos”. Bigger companies, while adopting the model and growing bigger tend to create departments for each of the phases. As a result, requirements specification phase is done by business analysts, design by architects, implementation by developers, etc. Each of those departments tends to work in partial (if not full) isolation from others. End result of one phase is often “thrown over the wall” to the team in charge of the next phase. That same department often moves on to the next project and has limited availability to provide support to the team working on the next phase.

The “silos” effect often prevents the team to establish necessary cohesion required for employment of techniques like Continuous Integration (CI) and Behaviour Driven Development (BDD).

Common result of having phases and “silos” is increase in management and documentation effort. It is not uncommon that medium to big projects spend one third or even half of the budget in documentation and management. Since team communication is reduced as the result of phases and silos, it needs to be compensated with the increase in documentation.

Each phase with its own team suffers from different problems that are getting accumulated with time. Those listed below are not applicable to all companies practising waterfall model. However, due to the nature of the model, its fostering of silos and types of organizations that still practice this type of delivery, they are more common than not.

Estimations

One of arguments for the usage of the waterfall model is that it has more accurate estimations and budget and that pleases the customers. However, since it is based on the idea that it will start with the “perfect” plan this often tends not to be true due to increases in costs produced by inevitable changes that are requested later. Even when estimations are accurate that is done at the expense of big increase in costs that extensive documentation and planning requires. Due to pressures to do the project within the budget, estimations tend to be inflated so that everyone is safe from possible repercussions.

Requirements Specification (Requirements Analysis)

Requirements specification is often done by business analysts. Their primary role is to establish communication with the client and produce a document that is signed off and will be used as a “bible” of what should be designed by architects and coded by developers. BAs often lack skills to make actual decision about the final solution and act as proxies between client on one side and the rest of the team on the other. Due to the nature of phases, that communication is limited and tends to be based mostly on revision of already produced documents.

Software Design

Another common side effect of a specialization to work in one specific phase is conversion of best developers and testers into “MS Office users”. Senior developer would be promoted to, for example, architect. Since implementation is the job of developers, architects dedicate a lot (if not all) of their time writing the design document that is passed to developers. Ex “star developer” gets converted into a person that is more and more disconnected from the code resulting in someone who knows how things were done before but is not up to date with current tendencies nor actual implementation of previous projects.

Implementation

Division into silos can be even more fractured with developers belonging to product or component teams. They can be working for departments made around different products. It is hard to get a full picture of the solution that will be built when knowledge and skills are concentrated on one specific part of the system. Division is inevitable. However, while some other methods foster the idea of one team and shared knowledge while acknowledging that each member of the team has some specialization in which he is more productive than others (Java, C, front-end, back-end, DB, etc), result of the waterfall model and silos tends to go to extremes where one developer belongs to a group that is highly specialized (for example, only JavaScript front-end) and has limited knowledge regarding the rest of components.

Developers are supposed to “blindly” implement the Requirements Document as it is, often without having the direct contact with the customers. They often don’t know their motivations and reasons that resulted in a request for that product. Common phrase when it is discovered that the solution is not what the customer wants is “I did it as it is written in the Requirements Document”. This problem is increased even more with further splitting development team into the above mentioned product or component teams. Not only that in that case developers might not have enough information, but they sometimes receive or are interested only in requirements for their product or component.

Developers tend to rely on the next phases to validate their work and do not have incentives to switch towards Test Driven Development (TDD).

Integration

If development team is fractured into product and component teams, integration of these products and components is often left for the phase after the implementation, resulting in the “integration hell”. This is the phase when everyone crosses fingers that components and products done individually by separate teams are working well together. Continuous Integration is made to solve this problem and make sure that everything is integrated most of the time. However, teams working with waterfall model tend to not use the CI due to separations of teams and tendency to do their part. In an extreme (but not uncommon) case integration is done by a separate team that puts it all together, does smoke testing and passes problems back to developers.

Testing (or Validation)

Since testing is a separate phase that is performed after the integration, work is mostly based on quality checking (QC). It is too late to assure that the implementation is done with quality built-in. The effect is often that testers are acting as “police” trying to prevent that the implementation passes their “gate” and is deployed unless all checking passed successfully.

This phase suffers from the similar problem as development. Since there is often no direct contact with the customer and all the work in this phase is based on the Requirements document, validation can check whether the software conforms to the document, not whether it fulfills customer expectations.

Since testing is left for late stages of the project, actual discovery of bugs is postponed. McConnell’s claim that a bug found in the early stages is cheaper in money, effort, and time to fix than the same bug found later on in the process tends to work against the waterfall model even though that is the main argument used by those practicing it.

Test automation is often limited or non-existent since testers often don’t have enough coding skills to build the automated solution. On the other hand, developers are working on implementation and are concerned mostly to do their phase and fulfill their schedules. Due to the lack of cooperation between teams, testers are left on their own and often forced to do most of the work manually. As a result, testing phase can last as long as the implementation if not longer.

Maintenance

In extreme but very common cases, maintenance is handled by a dedicated team. Those that did the implementation and testing phase often move to the next project leaving the maintenance team to fix bugs that will occur in production. In such a situation knowledge of what was done during the project is based on documents instead actual experience and the team needs more time to debug and solve problems.

Change requests

Change requests are probably the most painful part of the process. Since the model assumes that each phase is complete before the next one starts and that the project has the “perfect” plan, any change to the scope tends to break the whole process, moves the team to the beginning and costs a lot of effort. For all but the most trivial projects, there is no perfect plan nor there is the perfect scope and design defined in advance. Change will happen and when it does it must be rejected or it will produce a major setback.

Conclusion

Waterfall model can be useful for static projects where there will not be many changes. For most others it is outdated, too inflexible and too expensive.

Often overlooked aspect of the waterfall model is its indirect effect on the company structure and the organization of teams that tends to create silos of highly specialized people and the expense of decreased collaboration. There is increase in management costs and creation of rigid environment with little space left for improvement.

photo credit: Mustang Joe via photopin cc