Agile 2 – On Agile Software Development

Copyright Graham Berrisford. One of several hundred papers at Last updated 29/06/2019 19:17

Avancier’s agile papers

This is the second in a series of papers.

1.     On the beginnings of agile

2.     On agile software development

3.     On agile businesses and systems

4.     On systems thinking ideas used in agile

5.     What is agile architecture?

6.     EA in the world of agile architecture

Waterfall development

In 1970, most presumed a waterfall approach. Royce wrote: “Hopefully, the iterative interaction between the various phases is confined to successive steps”.


Waterfall  development









Roll out



Years later, Jacobsen wrote: “The waterfall approach to software development is only predictable in one sense: it almost always fails to deliver the desired outcomes. Detailed project plans and comprehensive documentation, coupled with exhaustive reviews and sign-offs, fail to deliver better results.”


Even in 1970, IBM promoted Joint Application Development workshops. And by 1990, James Martin’s “Information Engineering” (an early EA framework) presumed Rapid Application Development at the project level. RAD methods were refined during the 1990s.

Incremental development

Deliver a partial system early, learn from feedback, and polish the process.


Iterative / incremental development


A, D,

B, T, R

Product v2

A, D,

B, T, R

Product v3

A, D,

B, T, R



1997 Feature-Driven Development (fine-grained processes)

“Features are a primary source of requirements and the primary input into your planning efforts." 

Feature - a small, client-valued function in the form <action><result><object>.

E.g. Calculate sale value. Validate the password of a user. Authorize the sales transaction of a customer".


1999 The Unified Software Development Process (coarser—grained processes)

Use cases are the primary source of requirements and input into planning increments.

Named <action to achieve goal>.

E.g. Log in. Book train ticket. Apply for loan.


The Open-Closed Principle

Open to extension of existing viable product, closed to amendment of what exists.

Beware that adding new features can break old features.

The structure grows to be un-maintainable – needs refactoring

Agile development


1994 Dynamic System Development Method

Turned the requirements, time and cost triangle upside down.

You address the requirements that time and money for the next increment allow.


2001 The Agile Manifesto

Not the first such manifesto.


Skunk works – 1943 Kelly’s rules

Agile manifesto – 2001

Use a small number of good people.

We value individuals and interactions - over processes and tools

The contractor must test his final product in flight [and] in the initial stages. A minimum number of reports required, but important work must be recorded thoroughly.

We value working software - over comprehensive documentation

There must be mutual trust between the [customer] and the contractor [and] very close cooperation and liaison on a day-to-day basis

We value customer collaboration - over contract negotiation

A very simple... release system with great flexibility for making changes must be provided.

We value responding to change - over following a plan


Agile principles according to Fowler

"Two of the greatest rallying cries:

 ‘Do the Simplest Thing that Could Possibly Work’ [KISS]

 ‘You Aren't Going to Need It’ [YAGNI]

Both manifestations of the XP practice of Simple Design.” Martin Fowler.





EA traditions

Fail faster is good!

Failures should be minimized

Delivery early, commit late

Deliver at a distant horizon

High-level documentation

“Excruciating detail” Zachman

Test-driven “emergent architecture”

Model-driven “intentional architecture”


Agile principles for the social network – as in SCRUM

Three pillars of SCRUM: Transparency, Inspection, Adaptation

Team members share five values

·       Commitment: to achieving their team goals in every sprint.

·       Courage: to work through conflict and challenges together

·       Focus: exclusively on team goals and the sprint backlog (no work outside the backlog)

·       Openness: with stakeholders about the work and any challenges

·       Respect: others to be capable and work with good intent.


Just-in-time requirement specification - as in Kanban

Kanban is a scheduling system for lean and just-in-time manufacturing, developed by Taiichi Ohno at Toyota.

It “originates from the simplest visual stock replenishment signalling system, an empty box.”

“first developed in the UK Spitfire factories during the Second World War the late 1940s.”

In software, the “stock items” are typically processes (epics, use cases) to be developed


Just in time process decomposition - as in Kanban

Epics (cf. use cases) are identified and prioritised for incremental development

Epics are decomposed into user stories (cf. features) during development.


An agile software development method is not only iterative, but also flexible about the requirements, the solution and the process being followed.

·       It favours negotiation over planning, and flexibility about requirements.

·       It presumes user involvement and feedback is essential to success.

·       It encourages early testing for usability and performance.

·       It looks to deliver a “minimal viable product” and extend that incrementally.

·       It encourages a team to change, rebuild and re-test a software system on a daily basis.


Scrum and Kanban are now the most popular agile methods.

Kanban or Scrum?

This section is distilled from this source by Brendan Wovchko


“Scrum and Kanban aren’t competitors, they are experiments every team should try.

I’m willing to bend my rule about prescription and offer some advice on five conditions under which I’ve found Kanban to be a better fit than Scrum.


1. Low Tolerance for Change

Many organizations [resist] change… even if they aren’t getting results.

Their solution isn’t to rethink how they work, it’s usually to make their teams work longer and harder—which isn’t a real solution.

Scrum is a threat to change-adverse organizations primarily because of how quickly it transforms roles and meetings.


Kanban doesn’t use transformative change, it embraces evolutionary change.

Kanban employs a start with what you do now mindset that introduces teams to the shallow end of the pool before taking them toward the deep end of maturity.

Kanban doesn’t require any changes to roles or meetings when first getting started.


2. Obvious at All Levels

Kanban… is immediately intuitive to anyone.

A Kanban board is an instant sense-making device; it requires zero explanation to understand.

However…. Kanban is far more sophisticated than a simple tool for visualization.

Kanban is built for speed but most teams will never master the behaviors that produce those results because their commitment to learning Kanban stops at visualization.


3. Fluid Priorities

Scrum produces the best results when a team commits to a batch of work - and is empowered to remain focused for the duration of their iteration.

Scrum requires informed and empathetic stakeholders who are bought-in to being agile.


Kanban is able to survive conditions where agile culture doesn’t yet exist because it encourages optionality.

A Kanban team prefers to not commit to work in batches and they don’t commit to work until they start it.

This means a Kanban team can… respond to emergencies or changing priorities without needing to renegotiate commitments.


4. Small Teams

The ideal size of a Scrum team is a two pizza team.

This ensures that a team is small enough to be efficient and large enough where the time investment in meetings makes sense.


If your team is smaller, Kanban is the best option.


5. Complex Collaboration

My favorite attribute of Scrum is the cross-functional team… comprised of people from different departments and disciplines.

Its not just engineers and testers anymore; today, we have user experience, visual design, writing, editing and many other activities.

If your team has a large number of activities, the strategies of sprinting ahead or using a scaling framework may introduce unnecessary complexity and delay.


Kanban doesn’t utilize time boxing to create predictability, it uses lead time—so it is capable of sustainably supporting an unlimited number of activities and collaborations.


I’d continue to encourage you to resist the idea that Scrum and Kanban are competitors or enemies.

Both are a means to help teams and their stakeholders achieve sustainable success.

Don’t assume that whichever you best understand or most frequently use is superior.

Adopt a true agile mindset and experiment with both!”

Obstacles to agility

Has agile proved more successful than waterfall?


Agile vs. Waterfall

In this Standish Group analysis of Agile vs. Waterfall, the numbers favour agile development.

Like many surveys, it is misleading,

It is natural to use agile methods on "easy" projects, maintenance and extension

People revert to waterfall where projects are “difficult” because there are obstacles to agile development


“I estimate that 75% of those organizations using Scrum will not succeed in getting the benefits that they hope for from it.” Ken Schwaber in an interview posted on Agile Collab


11 obstacles to agile development

1.     Most/all requirements are mandatory.

2.     Users/domain experts are not readily available.

3.     Developers are not empowered to make decisions.

4.     The system’s first release must be large (safety-critical, money-handling, legacy replacement, or regulatory compliance).

5.     The complexity is server-side rather than UX side.

6.     The complexity is in input and update rather output and report.

7.     The business rules are complex and/or money-handling.

8.     The processing is batch rather than transactional.

9.     The database schema is immature and evolving.

10.  There are many integration points with other systems.

11.  There are exceptional speed or volume requirements


In practice: “I estimate that 75% of those organizations using Scrum will not succeed in getting the benefits that they hope for from it.” Ken Schwaber in an interview posted on Agile Collab


All is well given flexible requirements for a system that can start out small and simple.

People are forced to use a more "waterfall" approach in more “difficult” projects.

When given rigid requirements for a large and complex system that must be complete and right first time.

When the application requires a large and complex database structure and/or data migration.

As sometimes is required in a legacy replacement situation.


Every developer wants to do agile. But is it always possible?

How to assess a project for suitability?

Architects need a score chart to assess projects up front.

Because a project that scores highly on the measures below is an inherently difficult project.

And you probably need to substantially increase the time and budget for the work.

(Our architect training courses include a score chart.)



Does agile benefit from some up front design?

Kelly’s rule 10: “the specifications of hardware must be agreed to well in advance of contracting”.

Similarly, agile development methods recommend stabilising the “infrastructure” in advance of software development.

Where the infrastructure can include platform technologies and any persistent data structure that must be populated and maintained.


Building on a sound data structure

Many business applications are built on top of a large persistent data structure that records entities and events of interest to business people.

Agile system development proceeds best when the structure of this state data (this memory) is stable.

So, it helps to get the data structure as complete and right as possible before coding – to minimise refactoring later.

It also helps to implement the logical data structure as directly as possible, to minimise the complexity and processing overhead of any data abstraction layer.



Clearly, a system that emerges from incremental development is not optimised for requirements unknown at the start.

To keep the system manageable may require substantial “refactoring” of the system from time to time.

This requires investment over and above normal change requests.


How much model-driven engineering?

Why didn’t multi-level model-based systems engineering take off?

There are many reasons, but here, the challenge is the amount of documentation it requires

Every documentation level is an overhead; every transformation between levels of documentation is an overhead.

The closer the levels to each other, the more levels are needed, the bigger the documentation overhead, and the less value added by a transformation.

The further apart the levels, the less the documentation overhead, the more valuable but also difficult and costly a transformation.


Model-driven engineering is intended to improve quality and verifiability or auditability.

But user acceptance testing is impossible until the bottom level executable description is complete.

At which point, the users may clarify or change their minds about what they really wanted the system to do.

So, the agile development principle is to maintain only high-level or “lean” architectural documentation.

It is impossible to get all the advantages of both approaches; you have to find the right balance for each situation.