Agile 2 – On Agile Software Development

Copyright Graham Berrisford. One of several hundred papers at http://avancier.website. Last updated 12/06/2019 12:16

Avancier’s agile papers

This is the second in a series of papers.

1.     On the beginnings of agile

2.     On agile software development

3.     On agile businesses and systems

4.     On systems thinking ideas used in agile

5.     What is agile architecture?

6.     EA in the world of agile architecture

 

Agility in software development

In 1970 Royce promoted what some called a waterfall approach to software engineering..

He wrote: “Hopefully, the iterative interaction between the various phases is confined to successive steps”

 

Years later, Jacobsen wrote: “The waterfall approach to software development is only predictable in one sense: it almost always fails to deliver the desired outcomes.

Detailed project plans and comprehensive documentation, coupled with exhaustive reviews and sign-offs, fail to deliver better results.”

From c1990 people promoted rapid application development based on iterative/incremental methods.

 

E.g. Feature-Driven Development (FDD) - 1997

A light-weight agile method that expresss requirements as features.

“Features are a primary source of requirements and the primary input into your planning efforts."  

"A feature is a small, client-valued function expressed in the form <action><result><object>.

E.g. Calculate the total of a sale, Validate the password of a user, Authorize the sales transaction of a customer".  Scott Ambler

 

E.g. The Unified Process – 1999

Requirements are expressed as uses cases, typically named after the goal “Do something to achieve aim”

E.g. Log in, Book train ticket, Apply for Loan.

 

By 2000, people were promoting agile methods.

And the principles common to several methods were published in the Agile Manifesto.

 

Skunk works – 1943 Kelly’s rules

Agile manifesto – 2001

Use a small number of good people.

We value individuals and interactions - over processes and tools

The contractor must test his final product in flight [and] in the initial stages.

A minimum number of reports required, but important work must be recorded thoroughly.

We value working software - over comprehensive documentation

There must be mutual trust between the [customer] and the contractor

[and] very close cooperation and liaison on a day-to-day basis

We value customer collaboration - over contract negotiation

A very simple... release system with great flexibility for making changes must be provided.

We value responding to change - over following a plan

 

"Two of the greatest rallying cries … are the slogans:

·       ‘Do the Simplest Thing that Could Possibly Work’ and

·       ‘You Aren't Going to Need It’ (known as YAGNI).

Both manifestations of the XP practice of Simple Design.“ Martin Fowler

 

Other agile principles include:

·       Fail faster is good!

·       Delivery early, commit late.

·       High-level documentation

·       Test-driven rather than model-driven.

 

An agile software development method is not only iterative, but also flexible about the requirements, the solution and the process being followed.

·       It favours negotiation over planning, and flexibility about requirements.

·       It presumes user involvement and feedback is essential to success.

·       It encourages early testing for usability and performance.

·       It looks to deliver a “minimal viable product” and extend that incrementally.

·       It encourages a team to change, rebuild and re-test a software system on a daily basis.

 

Agile methods were designed for a single application development team.

If agile software development has an underlying theory, it is at least partly a psycho-bio-sociological one.

Agile software development methods draw from socio-cultural systems thinking.

 

Scrum and Kanban are now the most popular agile methods.

Kanban or Scrum?

This section is distilled from this source by Brendan Wovchko

 

“Scrum and Kanban aren’t competitors, they are experiments every team should try.

I’m willing to bend my rule about prescription and offer some advice on five conditions under which I’ve found Kanban to be a better fit than Scrum.

 

1. Low Tolerance for Change

Many organizations [resist] change… even if they aren’t getting results.

Their solution isn’t to rethink how they work, it’s usually to make their teams work longer and harder—which isn’t a real solution.

Scrum is a threat to change-adverse organizations primarily because of how quickly it transforms roles and meetings.

 

Kanban doesn’t use transformative change, it embraces evolutionary change.

Kanban employs a start with what you do now mindset that introduces teams to the shallow end of the pool before taking them toward the deep end of maturity.

Kanban doesn’t require any changes to roles or meetings when first getting started.

 

2. Obvious at All Levels

Kanban… is immediately intuitive to anyone.

A Kanban board is an instant sense-making device; it requires zero explanation to understand.

However…. Kanban is far more sophisticated than a simple tool for visualization.

Kanban is built for speed but most teams will never master the behaviors that produce those results because their commitment to learning Kanban stops at visualization.

 

3. Fluid Priorities

Scrum produces the best results when a team commits to a batch of work - and is empowered to remain focused for the duration of their iteration.

Scrum requires informed and empathetic stakeholders who are bought-in to being agile.

 

Kanban is able to survive conditions where agile culture doesn’t yet exist because it encourages optionality.

A Kanban team prefers to not commit to work in batches and they don’t commit to work until they start it.

This means a Kanban team can… respond to emergencies or changing priorities without needing to renegotiate commitments.

 

4. Small Teams

The ideal size of a Scrum team is a two pizza team.

This ensures that a team is small enough to be efficient and large enough where the time investment in meetings makes sense.

 

If your team is smaller, Kanban is the best option.

 

5. Complex Collaboration

My favorite attribute of Scrum is the cross-functional team… comprised of people from different departments and disciplines.

Its not just engineers and testers anymore; today, we have user experience, visual design, writing, editing and many other activities.

If your team has a large number of activities, the strategies of sprinting ahead or using a scaling framework may introduce unnecessary complexity and delay.

 

Kanban doesn’t utilize time boxing to create predictability, it uses lead time—so it is capable of sustainably supporting an unlimited number of activities and collaborations.

 

I’d continue to encourage you to resist the idea that Scrum and Kanban are competitors or enemies.

Both are a means to help teams and their stakeholders achieve sustainable success.

Don’t assume that whichever you best understand or most frequently use is superior.

Adopt a true agile mindset and experiment with both!”

Obstacles to agility

Has agile proved more successful than waterfall?

In this Standish Group analysis of Agile vs. Waterfall, the numbers favour agile development

But like many surveys; it may be misleading, because it compares projects that are not like for like.

People naturally use agile methods on "easy" projects.

Especially in a product development or maintenance and extension context.

Indeed, some agile methods were reverse engineered from successes in those contexts.

 

People naturally use agile methods on "easy" projects, also on software maintenance and extension.

They revert to waterfall where there are obstacles to agile development such as:

·       Most/all requirements are mandatory.

·       Users/domain experts are not readily available.

·       Developers are not empowered to make decisions.

·       The system’s first release must be large safety-critical, money-handling, legacy replacement, or regulatory compliance).

·       The complexity is server-side rather than UX side.

·       The complexity is in input and update rather output and report.

·       The business rules are complex and/or money-handling.

·       The processing is batch rather than transactional.

·       The database schema is immature and evolving.

·       There are many integration points with other systems.

·       There are exceptional speed or volume requirements

 

In practice: “I estimate that 75% of those organizations using Scrum will not succeed in getting the benefits that they hope for from it.” Ken Schwaber in an interview posted on Agile Collab

 

All is well given flexible requirements for a system that can start out small and simple.

People are forced to use a more "waterfall" approach in more “difficult” projects.

When given rigid requirements for a large and complex system that must be complete and right first time.

When the application requires a large and complex database structure and/or data migration.

As sometimes is required in a legacy replacement situation.

 

Every developer wants to do agile. But is it always possible?

How to assess a project for suitability?

Architects need a score chart to assess projects up front.

Because a project that scores highly on the measures below is an inherently difficult project.

And you probably need to substantially increase the time and budget for the work.

(Our architect training courses include a score chart.)

Footnotes

 

Does agile benefit from some up front design?

Kelly’s rule 10: “the specifications of hardware must be agreed to well in advance of contracting”.

Similarly, agile development methods recommend stabilising the “infrastructure” in advance of software development.

Where the infrastructure can include platform technologies and any persistent data structure that must be populated and maintained.

 

Building on a sound data structure

Many business applications are built on top of a large persistent data structure that records entities and events of interest to business people.

Agile system development proceeds best when the structure of this state data (this memory) is stable.

So, it helps to get the data structure as complete and right as possible before coding – to minimise refactoring later.

It also helps to implement the logical data structure as directly as possible, to minimise the complexity and processing overhead of any data abstraction layer.

 

Refactoring

Clearly, a system that emerges from incremental development is not optimised for requirements unknown at the start.

To keep the system manageable may require substantial “refactoring” of the system from time to time.

This requires investment over and above normal change requests.

 

How much model-driven engineering?

Why didn’t multi-level model-based systems engineering take off?

There are many reasons, but here, the challenge is the amount of documentation it requires

Every documentation level is an overhead; every transformation between levels of documentation is an overhead.

The closer the levels to each other, the more levels are needed, the bigger the documentation overhead, and the less value added by a transformation.

The further apart the levels, the less the documentation overhead, the more valuable but also difficult and costly a transformation.

 

Model-driven engineering is intended to improve quality and verifiability or auditability.

But user acceptance testing is impossible until the bottom level executable description is complete.

At which point, the users may clarify or change their minds about what they really wanted the system to do.

So, the agile development principle is to maintain only high-level or “lean” architectural documentation.

It is impossible to get all the advantages of both approaches; you have to find the right balance for each situation.