The agile manifesto: where do things go wrong?

This is published under the terms of the licence summarized in the footnote.

All free-to-read materials on the Avancier web site are paid for out of income from Avancierís training courses and methods licences.

If you find them helpful, please spread the word and link to the site in whichever social media you use.

 

Abstract

This paper follows the Agile Manifesto paper that listed ďa top ten agile imperativesĒ.

If you want to promote agile principles, you must understand why some people find those ten imperatives challenging.

This paper discusses why and where you may find our top ten imperatives for agility are challenging Ė reasons why many resist agile principles Ė reasons why people stray from the agile path.

Contents

The manifesto for agile software development 1

Project preconditions. 1

Why the ten agile imperatives are difficult and threatening2

Ten lessons learned. 3

Other contexts where you may find agility is difficult 5

So whatís wrong with the Agile Manifesto?6

 

The manifesto for agile software development

This manifesto <http://www.agilealliance.org> presents the consensus view of several agilists about the values and principles they look for in a successful project.

 

We are uncovering better ways of developing software by doing it and helping others do it.

Through this work we have come to value:

That is, while there is value in the items on the right, we value the items on the left more.

 

This ten agile imperatives below elaborate on these four values.

Project preconditions

Many agile methods place considerable emphasis on preconditions for a successful agile project. DSDM proposes four keys to agility

         Flexible and prioritised requirements (80/20 rule, Value stack, MoSCoW)

         Time/cash-boxed (Rather than requirements-boxed, We deliver what we deliver

         Incremental development (Vertical slices added, evolutionary prototype)

         Close user involvement (Joint application development (JAD) workshops)

 

I have found many projects on which agile methods simply canít be applied, at least, not as far as agilists would like them to be.

Analysis of scores of projects led me to produce this scoring system.

 

What kind of project

Time/cost-driven

0, 1, 2, 3

Mandatory requirements-driven

Users available for JAD

0, 1, 2, 3

Users not available

Developers empowered

0, 1, 2, 3

Developers not empowered

What kind of system and work

Incremental releases possible

0, 1, 2, 3

No partial solution can be released

Client/user interface dominated

0, 1, 2, 3

Server/database dominated

Output/enquiry dominated

0, 1, 2, 3

Input/update dominated

Simple business rules (CRUD)

0, 1, 2, 3

Complex business rules (non-CRUD)

On-line

0, 1, 2, 3

Batch

Stand-alone

0, 1, 2, 3

Highly integrated w other systems

 

If your project score more than 10, then you will find it difficult meet our top ten imperatives, listed below.

That is not to say you shouldnít try to be agile, rather to explain why youíll find it difficult and may have to accept some compromises.

For example, you may find your customer forces you to capture detailed requirements in some kind of requirements catalogue and develop a pyramid of documentation below that.

 

The trick is to be as agile as you can be.

But Iíve studied why people resist agile methods

And come to conclude there is more to be understood about the challenges to agility than the conditions above.

Why the ten agile imperatives are difficult and threatening

People can find agile to be a harsh discipline. And if the whole team is not committed, it can decay.

 

1) If we are going to fail, we want to fail faster

People used to wallowing in a waterfall process find this a harsh discipline.

The supplier working on a time and materials contract needs nerves of steel to hold to the view that early failure on this project will be perceived as a success.

It also risks the premature cancellation of projects that simply need more time for analysis, for architecture definition, for technology familiarisation etc.

 

2) We develop iteratively, driven by priorities

Incremental development suits web sites and some other software products.

It is much harder to apply the principle a batch program suite.

 

3) We use prototypes to tackle risks early

Prototypes consume time and resources. Sometimes they reveal little.

Often, the final product is so different that the lessons drawn from the prototype are of limited value.

Sometimes it is well-nigh impossible to engage the users with what a prototype does.

Perhaps you will be able to allocate one of two of the team members to the role of users.

Sometimes the system is principally batch processing, where even the keenest of users find it difficult to engage with and validate a prototype.

 

4) We empower a small team (including users) to get on with it

Many project managers find this threatening.

 

5) We maintain continuous and close stakeholder involvement

Persuading customers that ongoing customer acceptance is a key to development, and all stakeholders must be involved, can be very difficult.

Customers may simply be unable to do acceptance testing on a half-finished batch job.

Worse, acceptance tests may prove useless because they fail to guarantee the operational system will meet non-functional requirements to do with performance, availability, recovery and security.

Michael Z: I haven't met many folks who wouldn't love to maintain continuous and close stakeholder involvement.

The problem of course is trying to get business management to let you have a good person as part of the team on a near full-time basis for an extended duration.

Chris B: I have heard some nominally agile projects fail for the interesting reason that one key stakeholder was not allowed to be involved. In one instance the key stakeholder was a user of the system.

In another it was an IT guy in the central office who was tasked to enforce a company-wide policy of some kind.

 

6) We focus on program code rather than on specifications

What if the customer insists on a detailed functional specification?

What if your analysts know no other way? And that way has worked for them in the past?

 

7) We focus on tests rather than on specifications

What if your analysts and developers perceive test data preparation as a low status activity, or boring compared with specification?

 

8) We iteratively refactor code written so far

Refactoring does not seem to apply to data structures as well asto process structures.

Specific advice on how to speed up database structure changes would be interesting and valuable (more so than classifications of changes that can be made).

 

9) The customer must be willing to change and sacrifice requirements

This is a huge challenge.

Flexing requirements is not tenable in many commercial customer-supplier relationships.

Welcoming late changes is impossible given an immovable deadline and in some commercial customer-supplier relationships.

Some requirements are fixed by legislation, or contract, or legacy interfaces, or legacy databases, or the requirement to mimic a legacy system.

 

10) We get customers on side before we start

Often, the customer and or users wonít play ball with agile principles.

E.g. The customer insists every specification is approved by a committee that meets once a month.

The customer will never delete a requirement.

The project manager is quite likely to think of several of these points, and conclude than most of our day to day problems lie in projects where agile principles cannot be readily applied.

Ten lessons learned

This section lists some lessons that projects have reported in connection with the use of the ten imperatives above.

It is intended to help you steer others away from making the same mistakes. It is not an explanation of any agile method or a definitive list of agile principles.

 

An agile method is more than pick 'n mix principles:

It is a good idea to apply agile principles where you can, but that is not the same as applying an agile method.

An agile method implies specific constraints and consequences.

The best-known agile method today is probably Scrum

There are others: DSDM, XP, Feature-Driven Development and Adaptive Development).

Each is a method with specific disciplines, suited to specific circumstances.

Lesson: Do not pretend you can use these methods if your team is not trained and the circumstances are not right.

 

An agile method implies a customer with contractual obligations:

Lesson: Never adopt an agile approach that you have not first presented to and agreed with your customer.

You will hear: "the customer has a short timescale, can we use RAD?" "can we do extreme programming to speed things up?Ē. "can we do it faster and cheaper using an agile method?"

Perhaps yes, but if your customer is not signed up, ideally in the contract, to play their part and meet their obligations then say no!

An agile method means "what gives is what you get".

If time and resources are fixed, then the requirements must be flexible.††††††

Is your customer signed up to revising requirements, and postponing some to a later iteration?

If not, then donít pretend you can follow an agile method, though some agile principles might be useful.

 

Prioritisation means willingness to sacrifice:

Lesson: listing 90% of the requirements as "must have" is not prioritisation.

Will your customers/users accept a system that meets significantly less than the originally requested functional or non-functional requirements?

If not, then donít pretend you can follow an agile method, though some agile principles might be useful.

 

Empowerment means decisions are immediate and final:

Lesson: a customer/user committee is not an empowered team.

Agile methods require close knit communication and co-operation, and usually a degree of physical proximity that is incompatible with offshore development.

Can you work daily with a small number of customer/user representatives who have the authority to revise and reprioritise requirements?

If not, then donít pretend you can follow an agile method, though some agile principles might be useful.

 

Flexible requirements imply light-weight or slick change management:

An empowered team is daily revising and reprioritising requirements.

This is incompatible with heavy-weight change management.

 

Incremental development yields software releases:

Incremental development (which covers both refinements and increments) is about delivering working code, ideally to live use, or to acceptance testing.

But if you canít get customers involved in the testing, then you arenít really doing incremental development.

 

Iterative development really does yield software releases:

Redrafting specifications is not iterative development.

Prototyping is very often not iterative development.

Most projects find uses for prototypes of various kinds.

The kind of prototype important to an agile method involves demonstration of the evolving system itself, and requires empowered users to make accept/reject decisions during the demonstration.

 

Agility comes from training, experience and skill:

It takes years of training for a boxer to be light on his feet when it matters.

Agile methods assume staff know what they are doing.

That includes knowledge not only of the chosen technologies and agile method, but also of structured methods and techniques, which will still be useful here and there.

And managing an agile project is more taxing than managing a structured project!

 

Schedule in technical architecture and testing:

In commercial SI projects, elements of the regular project life cycle are usually needed.

You still need to attend to your solution development process.

Software specialists still need sufficient specification of requirements.

Customers are not satisfied with a prototype rather than a tested, scalable system.

We have yet to find a customer willing to accept a system that does not scale up to support the real number of users and volume of transactions.

We have yet to find a customer prepared to forgo acceptance testing (which implies some degree of system testing).

Other contexts where you may find agility is difficult

Customer reluctance

Above all, some customers and end users are simply unable or unwilling to engage in agile and iterative styles of development, especially, unwilling to accept flexing requirements

Other non-agile contexts

Some projects are not well suited to an agile approach.

Characteristics to watch out for include:

 

The typical project lacks many of the agilist's preconditions for success. You will often find:

Example

Consider a troublesome project I reviewed.

There existed a legacy system to apportion the money from many ticket sales between a few business owners.

The system had to replace the legacy system, run daily instead of weekly, in a 4 hour batch window each night.

The business rules, previously hard-coded, must be converted into data attributes modifiable by business users

Our new system could easily have printed the blank cheques for the business owners, or cheques with wrong numbers on.

The trouble is, it took us several years

         to work out how to process the input data in the narrow time window available for batch processing,

         to print the right numbers on the cheques, and

         to prove to the customerís satisfaction that the numbers were right.

 

How would you approach incremental development in a case like that?

What partial system could the customer accept as an increment - before the system is fully finished?

When pressed, one agilist mumbled unconvincingly about doing ďearned value analysisĒ along the way.

(Writing a paper dissing that technique would be to shoot fish in a barrel.)

What we need are good indicators for when agility works well and when it doesn't.

People need to understand the trade-offs and compromises they have to make.

Several challenges are discussed below, starting with the challenge of a large-scale systems integration programme.

Enterprise project lifecycle standards

Some enterprises, looking to control the costs and ensure the value of projects, impose a simple sequential process on all projects carried out by their IT department or IT services provider.

Here is an example, drawn from a major international company: Appraise - Select - Define - Build Ė Operate.

The enterprise management team insists that all requirements are defined in detail during the Define stage.

The idea is to control the costs of the Build phase and ensure the value of the Operate phase.

 

So even though the IT services provider may prefer agile development, but they have to work to the terms the enterprise sets.

Enterprise managers believe that imposing stop/go reviews at the phase boundaries is the best way to spot and stop a runaway train.

Supplier standards

Some IT services providers (OK, most of them), insist on requirements capture and tight change control.

Because thatís the only way they have of stopping the customer from endlessly changing their mind about what they want.

And then refusing to pay the bill for the work the provider has done on earlier (now redundant) requirements.

Bid-time work

An IT service provider may have to spend months working on a bid for a large project.

They simply cannot start coding the solution.

They have to work with more abstract specifications.

So whatís wrong with the Agile Manifesto?

The agile manifesto is perfectly reasonable, but I think it is also fair to make these observations.

 

It is hard to direct or assess a project using the values.

Our top ten imperatives offer a stronger challenge to common practice, and give us some directions to follow.

 

It promotes conditions that occur naturally in easy projects.

We need help with the projects where we find the opposite conditions.

         There are many mandatory requirements (and perhaps heavy-weight change management processes

         Users are not available

         Developers are not empowered

         No partial solution can be released

         The solution is server/database dominated

         The solution is input/update dominated

         There are complex business rules (non-CRUD)

         There is batch processing, and highly performance-constrained

         There are many or complex interfaces w other systems

         There are data management or migration challenges.

 

It can encourage untutored, undisciplined, trial and error.

Surely no manifesto author intended this.

However, left to their own devices, many software developers will disregard any contract, maintain only slight documentation of requirements, and develop iteratively, learning as they go along.

And the agile manifesto enables them (albeit improperly) to aggrandize this by calling it agile development.

 

 

Footnote: Creative Commons Attribution-No Derivative Works Licence 2.0†††† 12/03/2015 20:22

Attribution: You may copy, distribute and display this copyrighted work only if you clearly credit ďAvancier Limited: http://avancier.websiteĒ before the start and include this footnote at the end.

No Derivative Works: You may copy, distribute, display only complete and verbatim copies of this work, not derivative works based upon it.

For more information about the licence, see http://creativecommons.org