|Chapter 11 - Iterative and Incremental|
There are many approaches to developing software within a defined process. Some are skeletal methods around which you build your bright ideas. Others are quite prescriptive. You can build your software development methods around these approaches, or gradually fit their approach to methods already in place. Either way, they will help you define a repeatable process. Repeatability means consistency in approach and delivery. Delivering on time and to expectations will gain you respect, and respect will give you the latitude to create architecture.
I mentioned earlier in this book that the software development process presented so far is based upon the Unified Process, also known as the Rational Unified Process, or RUP. Many software houses use this or similar methods, and tailor them to their own organisational structure and beliefs.
The history and detail of the Unified Process is explained in The Unified Software Development Process, and also The Rational Unified Process. It is a collation of best practice software development methodologies, fully integrated with the modelling notation of the Unified Modelling Language.
To some extent, the modelling language is separate from the process of software development, but in other ways they are intrinsically tied. For example, the models created with UML are supported by the process, and the process itself is modelled using UML. This is not to say that other process methodologies do not support UML, for they do.
UML is a modelling language software architects can use to model the architecture and design of software systems. The Unified Process is an architecturally led method to ensure your wonderful architectures get delivered. It is quite possible for them to be used separately, but the likelihood of success with either is diminished without the other.
We should revisit the waterfall in all its glory. It is a superb metaphor for software development, and we must understand it better.
The traditional software development waterfall is an anachronism. It arrived before its time in a simplified and inaccurate rendering. It is this trifling, downward stepping waterfall we must eradicate from our memories, and instead embrace the waterfall of the future.
A real waterfall is an exciting, churning mass of water. It generates spray and static electricity. It mixes air into the water so that it might better support the life that exists downstream. The true waterfall: the churning, variable, flow and contraflow is the perfect model for software development.
The Unified Process claims to be iterative and incremental. Other words in other processes imply similar methods. In the spiral model, we home in on the software output through repeating requirements, analysis, development and testing phases.
Figure 11.1 The spiral model
The basis of the spiral and the iterative, incremental approach, mean that a bit of requirements gathering is followed by a bit of analysis, design, coding, test and release. This small iteration is used as the foundation for the second iteration of requirements, analysis, design etc.
The intent of iterative development is to finish a project quicker.
In a non-iterative, or linear approach, the requirements are gathered, the analysis and design done and the coders get to work. When they finish, the testers give it a good thrash and hurl it out into the dark world of live software. Usually when the testers finish their first test, a huge amount of rework is discovered. Some requirements are missing entirely, some were wrongly interpreted, and the whole thing just doesn't work; it crashes, hangs, behaves peculiarly during certain keystroke sequences, and generally behaves like an uncontrollable schoolboy.
What leaves development is often thought to be a finished product. What leaves test is sometimes a disaster of rework. Suddenly a project goes from an estimated 100% complete to only 50% complete; timeframes stretch. The developers get back to work, and release their trials to test, who return it again, this time 75% complete. Gradually through development/test iterations, the product homes in on completeness and is released.
Figure 11.2 Development/test iterations and project completeness
To improve this situation, we design the iterations and the increments into the process. An iteration is a cycle through the steps of requirements, analysis, design, code and test, and an increment is the piece of the product being focused on. That is, a part of the project is selected, and worked through the development cycle. Following test, another bit of the project is added in, while bugs are fixed and deviations resolved in the part already developed. On completion of the first iteration, it is easier to correct small mistakes which if left longer would grow into huge mistakes. It is no different than weeding the garden. If you do it often, you have a beautiful garden. The plants are healthy and the flower and vegetable beds free of weeds. However, leave those weeds for a few weeks, and they are as big as your plants. Some weeds even strangle the plants and take over their area. Leave it too long, and weeds are all you have.
Correcting problems before they grow, means there are no huge tumbles in perceived completeness following testing, and less time is spent developing the wrong thing.
Figure 11.3 Iterative incremental development finishes the job sooner
It's as simple as that. Running projects under an architectural umbrella can make move that finishing line left again by reusing components and delivering functionality within the architecture. Of course, there is still the upfront work to create the architecture.
Figure 11.4 Architecturally led iterative development finishes even sooner
The drawback of a small amount of requirements is that when you get your second set of requirements, you find that you did something wrong, or asked the wrong question when getting the first set of requirements. By the final incremental iteration, your product can end up such a mess that no-one understands how it came to be so.
Figure 11.5 The iterative, incremental waterfall
The best way to approach software creation is to gather requirements as fully as possible, and then select a subset for the first iteration of analysis, design, development and test. A birds-eye view of the entire product must be kept in mind during the design sessions to maintain a product view rather than an increment view. Without the birds-eye view, the final product may look disconnected, or functionally divided. While it is true that iterations of development are most easily defined upon a completed design, a carefully considered set of iterations is a better approach for two reasons. First, the feedback loop from the output of one iteration to the beginning of the next will provide valuable insights, and second, the completed design may not be as complete as believed, and later reflection may uncover time wasted due to an earlier misunderstanding.
The new improved waterfall model goes like this:
Figure 11.6 The born again Waterfall
A project begins as a twinkle in the eye of someone who can afford to sponsor it. It is then laid out in conversation as a set of documented requirements. During this requirements gathering process, analysis of the requirements begins. In analysis, the requirements are tested for logic, realism and completeness. The final stage of requirements is led by analysis as the requirements become ever more likely to form the foundation of a software project. Some time during analysis, the design phase begins. Along with ever reducing analysis of the requirements, the design now takes the lead in realising and improving the requirements.
By the time analysis of the requirements is all but over and design of the software is in its later stages, development has begun, either prototyping, or laying down the foundations of the software system.
Design continues on through the early iterations of coding as prototyping brings life to the project, and the sponsor (or the sponsorís rapid response team) can see, clarify, and improve the design.
Code a bit, test a bit, code a bit, test a bit then becomes the norm, perhaps for three or four iterations before release.
Of course, all through this slide down the waterfall, we have turbulence and frictions to be resolved. We have spray leaping back, carried by the wind into the face of requirements, even after release.
This then becomes the larger iteration on top of the code/test iterations. We do try to keep spray to a minimum, yet accept it exists.
We release. We take the feelings of those the product is released to and feed their emotion, knowledge and logic into the next phase of release, so requirements to release is the larger iteration moulded by spray from the last large iteration.
Now we see the Waterfall in its true glory. From a distance, its journey is a simple top to bottom flow, but closer in we see many other flows, and points of great turbulence within the main flow. Occasionally, there are flows against the flow, pools of eddies, waterspouts and huge churning masses which indicate energy being expended yet not contributing to the flow. We shall come in a moment to the waterfall's width, and what that might represent, but for a moment, we shall take a look at the terms used within RUP.
Figure 11.7 Phases and activities in the Unified Process lifecycle
The unified process uses the terms inception, elaboration, construction and transition as the phases of the lifecycle. These phases contain the same activities in varying degrees.
The first phase is inception, which covers the early work of domain modelling or domain analysis, gathering requirements, analysing them and beginning the design. Elaboration is the improvement of requirements through analysis, and into design. Construction is finalising the design, writing code and setting up and running tests. Transition is the movement of the software product from development, through test and on to the live system.
Also included in this representation of the Unified Process are the supporting workflows of configuration management, project management and setting up the development, test and release environments.
These are the words, diagrams and ideas used when discussing an architecturally led, iterative and incremental software process. Along the bottom of figure 11.7, the iterations lead step by step across the phases.
As shown in figure 11.7, there are other streams of work being carried out in parallel to the main effort of creating a product. There are possibly also other products being developed in other projects by other teams, all impinging upon the work within the project we are focussed upon.
These other projects, and other workflows, are represented by the width of the waterfall. The width is the entire set of concurrent projects being run by an enterprise. Work occurring in one project will have implications on another, indicated by water heaving sideways around a boulder. Occasionally the water is split into two channels, rejoining on the far side of another boulder. There at the joining point is more turbulence and frothing as the parted waters, or parted projects, rejoin and energy is spent as they realign themselves.
Our waterfall can only support a certain flow of water. Our software development departments can only support a certain number of projects.
Figure 11.8 The waterfall from above
What could the boulders in the water represent in your work? The rightmost boulder looks like a difficult technological problem and the development team are held up, using a lot of energy to break past it. Projects A and B share a piece of work, i.e. the oval boulder at the bottom. There will be turbulence at the downstream end where projects A and B merge again. And what of the leftmost boulder: People off sick? Old team members leaving, or new ones joining, or developers being reassigned from one project to the other? Both projects are affected bringing replacements up to speed.
Each individual project has a beginning and an end, but the torrent continues. Another related project or later iteration of an earlier release may follow on, and we begin to talk about streams of work. Projects are constrained in a stream, and together the streams of sequential projects make up a programme of work.
Figure 11.9 Streams of work
The Waterfall is quite a superb model of software development. It indicates the overall process starts and ends, it represents more than one project and the associated workflows. It shows energy being lost, diverted, there are pools of swirling peace, areas of torment, but mostly, the water gets to the bottom one way or another unless splashed onto the rocks of reality, or evaporated into clouds of dreams. Everything that happens in software creation is reflected in the waterfall.
Figure 11.10 An idealised iterative project Gantt chart
Figure 11.10 shows an iterative project plan. A team is assigned, requirements gathering begins, shortly followed by analysis. All of the activities overlap, as indicated in figure 11.7, so that implementation is not complete until testing is also complete. Implementation 1.2 also overlaps 1.1, requiring careful configuration management or source code control.
However, this level of detail is not necessary, nor even helpful. In many projects there are so many tasks that defining them down to this level of detail may cloud the picture rather than clarify it. It depends upon the project team, how comfortable they are working iteratively within the linear norm of a project plan, and just how much detailed control and understanding the project manager wishes to have over the team's activities. A non-iterative uncluttered project plan is probably the best structure to create within. This leaves the project manager clear of detailed development, and the development team free from measuring and marking every increment. Progress in a project plan is based on milestones; in iterative development, milestones are gradual changes rather than a concrete post to tick off when passing. These two views can only be reconciled by abstracting the detail for the plan, and using a percentage complete marker rather than a binary done or not done.
Here is a rough picture of how long might be spent cumulatively on each activity.
Figure 11.11 Dividing up project time
The easiest answer is it all depends. If you know the business, you will spend little or no time analysing it. If you are working on a real time, multi-threaded, military operation support system with built in redundancy across three geographical locations in case of attack, you will spend relatively more time on design and testing. The key is not how much time you spend, but how you avoid spending more time later in support and fixes, and also later within the cycles of development. There are no hard and fast rules, but more complex systems need relatively more analysis and design.
Projects have large phases. Phases are broken down into iterations. Iterations have development, test and review activities. Somehow all of these linear steps and iterative, incremental ideas must end up in a project plan.
What is important, is to declare key points in the process as if it was a smooth linear process with concrete deliverables, while knowing that really it is not. To manage the project, you might come up with a set of milestones for the project plan like this:
Figure 11.12 The key steps of a typical software project
As a software architect, you may form your own plan for the architecture you are creating or supporting and extending. Your own activities on a single project may be seen only during design and review. Nevertheless, you will have your own vision to head for and you must steer projects in long meetings and painful conversations, driving, explaining and reviewing, seeking to understand what is required, and to be understood when you have requirements of your own. You will have your own increments and deliverables, based on programme planning rather than project planning. Only your level in the business hierarchy defines how you are involved within the programme and its projects.
Rational is a software house providing software to help develop your software. They have a product called RUP (The Rational Unified Process) which is a management tool for the software delivery process. They also offer requirements gathering, automated testing, and one of the best known UML modelling tools in Rational Rose.
We are not discussing the software tool here, but the actual process as applied to architecturally led software development. For the rest of this discussion, RUP and the Unified Process refer to the software process rather than the software tool.
RUP is claimed to:
It is based on the following premise:
Planning precedes doing
But isnít that obvious?
Figure 11.13 RUP's 4 influences
RUP has 4 influences, which are: requirements, tools, technology and skills. Each of these influences addresses a different area of need or ability, and attempts to separate out the areas of interest in any development process.
RUP identifies the roles in the software development process as breadth and depth roles. They are:
|Discipline||Breadth role||Depth role|
|Business Modeling||Business Process Analyst
Discovers all business use cases.
Details a single set of business use cases.
Discovers all requirement use cases.
Details a single set of requirement use cases.
|Analysis and Design||Software Architect
Decides on technologies for the whole solution.
Details the analysis and design for a single set of use cases.
Owns the build plan that shows what classes will integrate with one another.
Codes a single set of classes or a single set of class operations.
Ensures that testing is complete and conducted for the right motivators.
Selects what to test based on the motivators.
Decides what tests should be automated vs. manual and creates automations.
Implements automated portions of the test design for the iteration.
Runs a specific test.
Oversees deployment for all deployment units.
|Tech Writer, Course Developer,
Create detailed materials to ensure a successful deployment.
|Project Management||Project Manager
Creates the business case and a coarse-grained plan; makes go / no go decisions.
Plans, tracks, and manages risk for a single iteration. (Note that this discipline has only one role. Assigning the depth view to a project coordinator can provide relief for overburdened project managers.)
Owns the process for the project.
Creates guidelines for using a specific tool.
|Configuration and Change Management||Configuration Manager
Sets up the CM environment, policies, and plan.
Change Control Manager
Establishes a change control process.
Creates a deployment unit, reports on configuration status, performs audits, and so forth.
Change Control Manager
Reviews and manages change requests.
(Again, note that breadth and depth roles are assigned to the same people in this discipline; assistant or associate managers in the depth roles would be helpful.)
RUP has its evangelists and critics. Most of the criticism is aimed squarely at its project focus and its view of development as the whole world.
Scott Ambler puts some of this criticism into a white paper. He suggests RUP has another phase following inception, elaboration, construction and transition, which is production. The activities of production are in operating and supporting the software after it has been released.
The process workflows are joined by operations and support. The supporting workflows are joined by Infrastructure management.
There are many other software processes, such as the Open Process, the Object Oriented Software Process
There are many software processes. They are all presented differently, but essentially similar to RUP in the way they are used. Here are a few of the alternative processes in Scott Ambler's paper, selected for the ease with which they are pictorially displayed and how they have contributed to the software lexicon.
This process has been developed by the Open Consortium, as a comprehensive software process which enhances the use of object oriented methods.
Although the OPEN process is displayed as interconnected boxes, the coding part of the process is still incremental, as shown by the multiple boxes and the n iterations between development and user review.
Unbounded activities are displayed as rounded rectangles, and tightly bound activities as normal rectangles.
Figure 11.14 The OPEN Process
The left hand side represents the activities for a single project and the right hand side represents cross project activities.
Beyond the single project, which is the sole focus of the Unified Process, are cross project, or programme issues. I have used the term programme management previously. Programme management is a definitive term in OPEN.
Other terms used are OOA, OOD and OOP, which are object oriented analysis, design and programming. V&V means validation and verification, or testing.
The OPEN process from the Open Consortium should not be confused with The Open Group's Architectural Framework.
OOSP is a similar, yet a semantically different process yet again. What OOSP has in its favour is that it recognises the project management, quality and personal improvement that are lacking in the other processes.
Figure 11.15 The Object Oriented Software Process
None of these processes define how a software architect fits into them other than rather loosely. So which one should you use? They all have their favoured terms, their not quite linear approach to process, and their multitude of boxes, lines and arrows.
The problem is that software architecture is only part process. The rest is made up of things that do not fit comfortably in a process model. The big arrow along the bottom of the OOSP diagram looks like a rather faint attempt at spreading lots of tasks across the whole project, but as you can see, the others miss them out entirely, or bundle them into a single task, project manage.
As a software process, OOSP is more complete, probably because it is based on the Capability Maturity Model (more later) of the Software Engineering Institute.
All are rather lacking in direct help for a software development house who wants an off the shelf process covering everything from hiring good developers to getting the most out of them. Most software organisations hire on personality and superficial testing. We have no software qualifications which are respected as much as other areas such as accountancy. People who work in accounting study for one exam after another. Studying lays a foundation upon which excellence is built. In software we are lucky if we can keep up with the changing technologies.
And how do you get the best out of people? You want the best interaction, communication, standard of code, use of algorithms, speed of development, effort against unconquerable odds etc. Such things can only be measured within the review points of a software process. Or does slamming a software process down on developer's heads destroy their creativity, their skills and their will to develop outstanding software?
As software architect, you need to exist somewhere in a process. You need to be around during the initial ideas stage. It may be you who is generating the ideas. You also need to be around at the macro design stage, when architecture is defined. Somehow the information you create needs transferred into developers' minds, and you need to ensure it is happening as you planned by having a look along the way.
Figure 11.16 A linear process for a software architect
Thatís it. This figure is your path to becoming a software architect. Your reality may be slightly different, but the essence will remain. Some points may come easy, others more difficult. I should imagine the step Identify yourself as a software architect might not be as easy as it looks. Patience alone will help you conquer that one, providing that you have laid the foundations with previous steps.
So where does that leave you with a software process. You have your architectural one, and we have three other examples in the Unified Process, OOSP and OPEN.
That answer is easy. You need to create a process which will fit your organisation, and you need to make it happen. You need to identify your role as software architect and make that happen too. Furthermore, you must tailor your process to each programme or project you undertake, for they will not all be the same.
Tailoring your process for each piece of work will allow you to improve your process incrementally. Stating the process at the beginning of each piece of work will give you both the freedom and control you need to be a software architect.
CMM measures the quality and depth of your software process. Within it, are the initiatives you can take to improve your delivery of software.
Most people in business are familiar with the quality initiative ISO 9001. Some software houses have used ISO 9001 to structure their software delivery. It can help you structure the way you approach development, and ensure it is a repeatable process, but ISO 9001 does not have any advice to help with the specifics of software development. For that, you need to look to CMM from the Software Engineering Institute (SEI).
The SEI has developed a quality based system for software development, named the Capability Maturity Model (CMM). They have also developed a Team Software Process (TSP) and Personal Software Process (PSP) to augment CMM.
The Capability Maturity Model has been taken up by some industry heavyweights such as NASA and Lockheed Martin. It is very popular in the huge software industry of India, where companies like Infosys have reached the highest maturity, level 5, as have NASA and Lockheed Martin. This is a considerable achievement as the SEI recommends the improvement to a continuously optimising process will take ten years.
CMM is a five step process, in which organisations can climb through an improving structure for managing their software development process.
Figure 11.17 The five levels of the Capability Maturity Model
At level 1, the software process is characterised as ad hoc, or even chaotic. Few processes are defined, and success depends on individual effort and heroics.
Basic project management processes are established to track cost, schedule, and functionality. The necessary process discipline is in place to repeat earlier successes on projects with similar applications.
The software process for both management and engineering activities is documented, standardised, and integrated into a standard software process for the organisation. All projects use an approved, tailored version of the organisation's standard software process for developing and maintaining software.
Detailed measures of the software process and product quality are collected. Both the software process and products are quantitatively understood and controlled.
Continuous process improvement is enabled by quantitative feedback from the process and from piloting innovative ideas and technologies.
Each of these levels is further defined by a set of measurable organisational processes:
No defined process
|Requirements Management||Software Project Planning||Software Project Tracking and Oversight||Software Subcontract Management||Software Quality Assurance||Software Configuration Management|
|Organisation Process Focus||Organisation Process Definition||Training Program||Integrated Software Management||Software Product Engineering||Intergroup Coordination||Peer Reviews|
|Quantitative Process Management||Software Quality Management|
|Defect Prevention||Technology Change Management||Process Change Management|
Your organisation is already at a minimum of level 1. If you have no process at all, or a few defined processes such as coding standards and best practice guides, you are likely to be level 1.
To acheive level 2, which is likely to be your organisationís first target level, you will have to have the following processes in place.
Establish a common understanding between the customer and the software project team of the customer's requirements that will be met by the software project
Software Project Planning
Establish reasonable plans for carrying out the project.
Software Project Tracking and Oversight
Establish adequate visibility into actual progress so that management can take effective actions when the project's performance deviates significantly from the project plans
Software Subcontract Management
Select qualified software subcontractors and manage them effectively
Provide management with appropriate visibility into the process being used by the project team and of the product(s) being built
Software Configuration Management
Establish and maintain the integrity of the products of the project throughout its lifecycle.
Level 3 is built upon the sound implementation of level 2.
Organisation Process Focus
Establish the oranisational responsibility for software process activities that improve the organisation's overall software capability
Organisation Process Definition
Develop and maintain a usable set of software process assets that improve process performance across projects and provide a basis for cumulative, long term benefits for the organisation
Develop the skills and knowledge of individuals so they can perform their roles effectively and efficiently
Integrated Software Management
Integrate the software engineering and management activities into a coherent, defined software process that is tailored from the organisation's standard software process and related process assets
Software Product Engineering
Consistently perform a well defined engineering process that integrates all the software engineering activities to produce correct, consistent software products efficiently and effectively
Establish a means for the software engineering group to participate actively with the other engineering groups so the project is better able to satisfy the customer's needs.
Remove defects from the software work products early and efficiently. An important effect is to develop a better understanding of the software work products and of the defects that can be prevented.
Level 4 is built upon the sound implementation of level 3.
Quantative Process Management
Control the process performance of the software project quantatively
Software Quality Managment
Develop a quantitative understanding of the quality of the project's software products and achieve specific quality goals
Again, Level 5 is built upon a sound implementation of level 4.
Identify the causes of defects and prevent them from reoccurring
Technology Change Management
Identify beneficial new technologies (tools, methods and processes) and transfer them into the organisation in an orderly manner
Process Change Management
Continually improve the software processes used in the organisation with the intent of improving software quality, increasing productivity and decreasing the cycle time for product development
Read more about CMM in The Capability Maturity Model or visit the Software Engineering Institute's website
Applying the CMM to your software development process can be very beneficial. It can take a long time to rise through the ranks, but even taking the spirit of it, or picking and choosing from its guidance can be beneficial.
Remember, it is your software development process. It does not come out of a box. You must decide what is good for you and your software delivery efforts. If you even get one idea from reading through these methods, then your time reading it will be repaid many times over in time that would otherwise have been wasted in an unstructured development environment.
As you assess these methods, it would be good to keep the following five questions in mind:
The greatest problem with CMM is that consultancies will come in and help you get from level 1 to level 5, and charge you a lot of money in the process. This was the failure of ISO 9001, where companies paid vast sums to the big audit houses for little more than a collection of nice forms and a pretty certificate to stick on the door. Yes, things were probably better, but could equally have been without the cost.
Quality in software is driven by the practitioners and the process they agree to follow. It is not driven by form ticking activities and climbing the CMM ladder.
We may even find that the reason for the differences in the software processes is a symptom of the fact that we have not conquered the software disease. It may be yet more complicated than we had imagined, and the various processes have cured a single symptom of the disease.
On the rise now are dynamic development processes. They use dynamic simulation models to compare predictions to history, then use 'peel the onion' analysis to find the causes of change in time and quality. Peeling the onion is a metaphor for removing one piece of the puzzle at a time and viewing its contribution. It's very interesting, but another story entirely.