Agile Architecture: an oxymoron?
Copyright Graham Berrisford. One of several hundred papers at http://avancier.website. Last updated 09/03/2019 19:59
This paper is a supplement to Enterprise architecture challenges.
Advocates of agile architecture (including SAFe) tend to confuse concepts that should be distinguished.
“Thanks! You article contains some really good reflections on the difference between developing with agile methods and designing solutions to be agile.”
What is an Agile Architecture?
Does it mean agile system development, agile documentation or agile systems?
Agile system development to produce “the simplest system that can possibly work” does not produce an agile system.
Because there is a basic trade off between design for simplicity and design for flexibility.
And what is the goal of Agile Architecture?
To develop systems fast? develop them using agile methods? or develop agile systems?
One management consultant writes of a business making “the agile transformation journey” thus:
"the goal is to enable the organization to generate instant, frictionless, intimate, incremental, risk-free value at scale...
as exemplified by the five largest and fastest-growing firms on the planet: Amazon, Apple, Facebook, Google and Microsoft."
This brief statement prompts many questions.
What kind of business?
The famous five (above) are peculiarly software-centred businesses.
Different businesses change at different rates.
Changing web sites and algorithms is one thing.
Changing physical assets and workers with special abilities, knowledge or skills is another.
What kind of scalability?
Various scales affect information system development. E.g.
· The number of entity types that are monitored and directed by the system.
· The number of event types the system must recognise.
· The number of entity and event instances the system must process.
· The number of people in the system development organisation.
These and other variables contribute to the size and/or complexity of the system to be built
And combine in different ways to have impacts on system development.
How fast is change needed?
Different business system changes have different time and cost profiles.
A business strategy can trigger a long-term transformation project that last years and cost millions.
An architect in a global business told me of a project completed in 2016 that cost $200 million.
That same business has to respond to quickly to immediate problems and pressures.
Sometimes, managers want a new business system at three months notice; or even faster.
What if managers insist they need a point solution tomorrow?
Enterprise architects can give waiver for an innovative and/or agile project to proceed outside of normal governance.
But without architectural oversight, both buying packages and agile software development can create disintegrated and/or un-maintainable systems.
The discussion here is about agile software development in an enterprise architecture context.
Agile means willing and able to speedily respond to change.
What architecture means is less clear.
For some software architects, the architecture of a software system is its basic structure - in the code.
It is the stable abstractions people writing the code depend on.
The abstractions include an artificial domain-specific vocabulary, starting with the names of atomic data types.
(This vocabulary is necessary whether domain-driven design is used or not.)
Along with some stable compositions, interfaces and generalisations.
For enterprise and solution architects, architecture is higher level design-time description - rather than code.
It usually means an output of up-front planning, but can instead be reverse-engineered from a run-time system.
It is written in natural language – or near to it - and communicates the intent of the code.
Either way, if the architecture changes much or often, you’ve got a problem.
To use a metaphor: the architecture of a concrete building should allow for the fittings to be changed.
But to change the architecture might well mean discarding the fittings and re-building from scratch.
The term “Agile Architecture” is used with several meanings that might be more clearly called something else.
This term embraces a variety of practices that enable the rapid incremental development of a system.
The first release of a system (a minimum viable product) is followed by short-cycle releases that extend the system’s capability.
An agile development method is not only iterative, but also flexible about the requirements, the solution and the process being followed.
· It favours negotiation over planning, and flexibility about requirements.
· It presumes user involvement and feedback is essential to success.
· It encourages early testing for usability and performance.
· It favours short-cycle iterative development.
· It looks to deliver a “minimal viable product” and extend that incrementally.
· It encourages a team to change, rebuild and re-testing a software system on a daily basis.
Today, commonly used agile development methods or practices include SCRUM and Kanban, not discussed here.
In an extreme version of agile development, there is no up-front architecture or architect.
Rather, architectural documentation is reverse-engineered from an emergent design, and maintained in step with software development.
This is contrary to the classical notion that an architect draws diagrams that builders follow.
“One of the more persistent myths of agile development is that up-front architecture and design are bad …” Bob Martin
“While we must acknowledge emergence in design and system development, a little planning can avoid much waste.” James O. Coplien.
“Lean architecture comes from applying the principles of the Toyota Production System to software architecture.
Lean architecture is both about an architecture with no fat, and about the consistency and reduction of waste in the process surrounding its creation and use.”
Read this article for more http://leanmagazine.net/lean/lean-architecture/.
Architecting a system to be agile
This appears to be a goal in SAFe, and is explored below.
Given that most development teams use SCRUM, and many use Kanban, as their delivery method, what does SAFe add?
The quotes in this section are from the authors of SAFe.
“Agile Architecture is a set of values and practices that support the active evolution of the design and architecture of a system while implementing new system capabilities.”
How far is SAFe scalable above software architecture?
Is it about the abstract architecture of a system, or the system itself?
And is it a coherent framework or a collection of sometimes conflicting ideas?
The core ideas are listed in this sentence:“The Agile Architecture principles support Lean-Agile development through collaboration, emergent design, intentional architecture, design simplicity, as well as designing for testability, deployability and releasability.”
Let us take collaboration between architects, developers and others for granted.
Intentional design would mean designing a system ahead of development.
Emergent design means that the design of a system emerges from incremental development.
Intentional architecture implies some up-front, high-level, design documentation.
Explicit articulation of higher-level architecture(s) is the basis of all enterprise and solution architecture frameworks (including TOGAF of course).
For sure, at least some architectural oversight of agile system development is desirable.
And people ought to optimise the balance between that and emergent design, depending on various factors.
Where does agility enter the picture?
Remember agile mean “willing and able to speedily respond to change”.
Intentional agile architecture implies:
· Documenting an architecture that can be readily changed and/or
· Designing a system that can be readily changed – an agile system.
The first might be achieved by minimising the architectural documentation, keeping it lean.
The second might be achieved by generalising the architecture, keeping it abstract and/or designing for flexibility.
These are two different ideas.
You may know that agile development favours the Keep it Simple (KISS) principle.
You may not know there is a fundamental design trade off.
Designing a system for flexibility usually makes it more complex, and run slower.
Does this mean minimising the design documentation, or minimising the code written?
Or minimising the number of locations, components, messages, network hops, processes, databases?
Or minimising the size or length of components, messages, network hops, processes, databases?
There are many trade offs between these qualities.
And none of those design strategies necessarily results in an agile system.
In an article promoting Agile Architecture, a CTO is quoted as saying the following.
“The only way to manage a large distributed system is to keep things as simple as possible.”
OK, but isn’t that precisely because the "architecture" of large distributed systems is inherently complex?
If you want a simpler system, don’t distribute it!
As you may know, Fowler’s first law of distributed objects is: “Don’t distribute your objects”.
“Keep things simple by making sure there are no hidden requirements in the design.”
What are hidden requirements as opposed to visible ones?
You may know that agile development favours the You Ain’t Gonna Need It (YAGNI) principle.
The idea is not to anticipate future requirements, and to refactor when it becomes necessary.
But architecting an agile system does mean including features in the design that anticipate future requirements.
And a system cannot be agile in every possible way, it has to be designed with specific kinds of change in mind.
[And no] hidden dependencies in the design.”
What are hidden dependencies as opposed to visible ones?
We can make all dependencies visible in architectural documentation
OK, but that implies a heavy non-agile documentation overhead.
We can all remove synchronous interactions between distributed components,
OK, but that increases the complexity of maintaining data integrity dependencies and compensating transactions.
Read our Microservices paper for some decoupling trade offs.
“Cut technology to the minimum you need to solve the problem you have.”
It certainly seems a good idea not to use technologies you don’t need.
But what does this actually mean in an enterprise application design context?
Use thin clients with no application code on client devices?
Don’t use middleware when you can use REST?
Many enterprise architects promote middleware use.
Read Smart endpoints and dumb pipes for discussion of the synchronous v asynchronous trade off.
“It doesn’t help the company to create artificial and unneeded layers of complexity.”
OK, but nobody does this deliberately unless they are trying to design for maintainability.
And if asked, many sponsors will say they do want an agile (flexible, configurable) system.
To achieve this, you’ll have to make the system more complex than is needed in the short term.
For sure, test-driven design is an agile development principle.
Tests are built as “executable specification”.
But there is a trade off, since the more test cases you build, the harder it is to change the test cases.
And therefore, the harder it is to release a new system.
SAFe includes six or seven principles; some are variations on agile development principles.
Three of the more architectural principles are discussed below.
This means having enough (say 3 to 9 month) infrastructure to complete near-term product backlog without refactoring.
What is infrastructure? The term covers many things.
It can include client devices, server devices, operating systems, database management systems, middleware, data storage etc.
To a software engineer, it includes what they use by way of a database schema and module libraries (class libraries, DLLs...).
The trouble is, 6 months is a rolling future, and 1 month from now, the 6 month future will be different.
So, surely maintaining the runway to anticipate the 6 month ahead implies continual refactoring?
What does it mean to build an architecture rather than a system?
Building the simplest system that can possibly work is an agile development principle for sure.
But what does building the simplest architecture mean?
If it means the same, then it can inhibit the design of a system that is flexible and configurable.
If it means something different, then does it conflict with agile development principle?
How to “ simplify both the design and the communication between the teams.”
The proposal in SAFe is: “Domain-driven design, design patterns, and applying a system metaphor”
Again, the proposal questionable.
Wrt design patterns in general...
First, remember that “If you can’t draw it, it isn’t a pattern” Christopher Alexander.
Some articles on SAFe equate principles or standards with patterns.
But design patterns are not standards, and they often come in pairs.
What architects need is guidance on the trade-offs between patterns.
Then, note that following design patterns tends to complexify rather than simplify.
Many common design patterns are designed to make a system more flexible by decoupling communicating parties
But, by adding a level of indirection, and perhaps allowing temporary disintegrity, they make a system more complex.
Generally, making a system more agile tends not only to make the architecture more complex, but to make the system slower as well.
Wrt domain-driven design in particular...
In 2002, Martin Fowler observed that Domain-Driven Design is difficult to learn and best reserved for complex systems with “rich domain models”.
And in January 2017, Wikipedia said: “Microsoft recommends that [domain-driven design] be applied only to complex domains.”
(Where rich probably implies substantial use of inheritance.)
In the 1980s. the UK government’s structured design method (SSADM) had transformed an object-based domain model into event-oriented procedures.
Fowler wrote that event-oriented “Transaction Scripts” are simpler for many enterprise applications.
Both sources presumed reuse would be achieved by delegation to subroutines from event procedures or transaction scripts.
Read Domain-Driven Design, Transaction Script and Smart UI for discussion of the simple v complex trade off.
It seems likely that many if not most enterprise applications are more easily designed and maintained using Transaction Script or Smart UI patterns.
You simply factor out data object-based modules where data-centric rules are invoked from several transactions or screens.
Transaction scripts can equally well publish events (for others to consume) and log events (for subsequent query and replay).
Wrt a system metaphor...
A good system metaphor helps to facilitate communication.
Use this link to find a good discussion of the idea
Note that the metaphor for an agile system has to be generalised, which can weaken the clout of the metaphor.
And beware, of course, that metaphors can be misleading.
Analogies with the architecture of concrete buildings, airplanes and city plans have misled many about the nature of enterprise, solution and software architecture.
You may look at the architectures of buildings (in run-time reality) and at architectural drawings of them (in design-time descriptions).
You can see immediately (visually) whether the run-time reality resembles design-time description – or not.
By contrast you cannot look at the run-time reality of a software system.
You can see only a user interface that reveals nothing of the software architecture.
Enterprise and solution architects rarely look at the executable design-time description.
And looking at the code of a software system can be of little or no help anyway.
Instead, architects document a much more abstract design-time description using natural language and graphical models.
They then govern software development for compliance of the code to the intent of that documentation – rather than to its visual form.
OK, but how does this help us produce an agile architecture?
Surely here, to be agile means being able in mid-course software development to adopt an innovation with relative ease.
However useful an innovation might prove to be, adopting it is a departure from incremental development.
Until the innovation is understood, we don’t know how to anticipate it.
Until the innovation is on the road map, we don’t know we need to anticipate it.
And at that point, adopting the innovation is likely to incur the time and cost of substantially re-architecting or re-enginering the system.
So, in what sense does researching innovations enable “Agile Architecture”?
To the mirage of agile architecture we should add the mirage that after a series of YAGNI and KISS increments, substantial refactoring to resolve a major problem or introduce a major innovation can readily be accommodated without major time and cost implications.
Enterprise architecture methods are traditionally for planning major changes business systems, to a target that may be a year, two years or more away.
Often, initially-stated requirements unclear, inaccurate, incomplete – and change over time.
Agile development methods are about what a team does to continually improve and extend a software system, with release cycle of days or weeks.
The outcome should be faster delivery of something useful, though it may not do everything customers want.
Agile development does not necessarily produce agile systems.
The plain fact is, digitisation of systems is and always has been difficult.
To make a useful software system is a painstaking task.
Reengineering an old system using new (innovative) technologies may make little difference.
Replacing like for like may do little or nothing to make a system more agile.
To do that usually requires a redesign that makes the system more complex and slower.
(In one story, after being rewritten in Java, a single-server legacy system required a dozen servers.)
Moreover, the notion of an agile system is not a simple or single idea
Since a system can be flexible in several ways, as described in the table below.
yet might also be
its architecture may be
A versatile system
do many different things
rigid and hard to change or extend.
large or complex
A malleable system
be readily changed to a new version
limited and not versatile
slight or simple
An extendable system
be extended with extra features
not malleable (the Open-Closed principle)
fixed (must be right first time)
A self-organising system
change its own roles and rules
reliant on human ability, barely a system at all
slight and simple
Designing a complex business system so that it can be quickly changed is a challenge; and there are always trade offs.
Generally, building flexibility into a system makes it more complex – which adds time and cost to development.
It may mean shifting business rules from procedural instructions into data variables that end users can change – which tends to make the system slower.
And maintaining flexibility may require substantial “refactoring” of the system from time to time – which requires investment over and above normal change requests.
The discussion of SAFe above is limited to core ideas and principles.
Other management-oriented ideas include what some call "lean portfolio management" and "flexible budgets”.
A theme of this paper is that all is trade offs.
· Read SOA as per the Bezos mandate for some SOA history.
· Read our Microservices paper for some decoupling trade offs.
· Read Domain-Driven Design, Transaction Script and Smart UI for discussion of the simple v complex trade off.
· Read Smart endpoints and dumb pipes for discussion of the synchronous v asynchronous trade off.
Some see agile development as the anti-thesis of enterprise architecture.
However, both approaches are used in a “bimodal” IT department.
Naturally, agile development of small and/or silo systems can create the issues that led people to promote EA in the first place.
The same can be said of the fashion to develop “microservices”.
Mitigating that risk, inevitably, involves tightening up of project milestones
Typically, solution architects must complete “solution outline” documents and get them approved.
Architects are expected to document some up front high-level design and principles.
They may recommend that an agile development method is used.
They may then govern the acceptability of changes requested during an agile development project.
And reflect accepted changes in architecture-level documentation.
Agile development methods do not always give the hoped-for benefits.
They do not always mean quicker and cheaper delivery of a system.
It often takes a long time to develop applications that require a large/complex database structure and/or data migration.
Some systems cannot be partially implemented – must be right first time.
Architects need a score chart to define the suitability of agile development methods.
Obstacles to agility include:
1. Most/all requirements are mandatory.
2. Users/domain experts are not readily available.
3. Developers are not empowered to make decisions.
4. The system’s first release must be large (safety-critical, money-handling, legacy replacement, or regulatory compliance).
5. The complexity is server-side rather than UEX side.
6. The complexity is in input and update rather output and report.
7. The business rules are complex and/or money-handling.
8. The processing is batch rather than transactional.
9. The database schema is immature and evolving.
10. There are many integration points with other systems.
11. There are exceptional speed or volume requirements
A project that scores highly on these measures is an inherently difficult project.
So, consider substantially increasing the time and budget for the work.
An architecture method is designed to ensure there are agreed objectives, models of baseline and target systems, and governance of system delivery.
There is a down side - the necessary bureaucracy – to create documentation and analyse the impacts of change requests against that documentation.
Of course, untrained/unthinking completion of document templates and untrained/unthinking approval of the resulting documents is ineffective.
The problem is not the document templates; it is ill-educated, ill-informed document completion; and naivety in document reviewers.
There is no substitute for employing well-trained experienced and knowledgeable people at all levels.
All is trade offs in architecture and design (as our architect training courses emphasise).
And there are trade offs in methodology too.
Taken to the extremes, both EA bureaucracy and agile development lead to difficulties.
Architects have to understand the extremes and strike a balance between them.
Being appropriately Agile and having a well understood Architecture are both good things.
The challenge is create the right balance between the two based on an understanding of specific requirements (functional and non-functional), foreseeable changes and deadlines.
Why didn’t multi-level model-based systems engineering take off?
There are many reasons, but here, the challenge is the amount of documentation it requires
Every documentation level is an overhead; every transformation between levels is an overhead.
The closer the levels to each other, the more levels are needed, the bigger the documentation overhead, and the less value added by a transformation.
The further apart the levels, the less the documentation overhead, the more valuable but also difficult and costly a transformation.
Model-driven engineering is intended to improve quality and verifiability or auditability.
But user acceptance testing is impossible until the bottom level executable description is complete.
At which point, the users may clarify or change their minds about what they really wanted the system to do.
So, the agile development principle is to maintain only high-level or “lean” architectural documentation.
It is impossible to get all the advantages of both approaches; you have to find the right balance for each situation.
All free-to-read materials at http://avancier.website are paid for out of income from Avancier’s training courses and methods licences.
If you find the web site helpful, please spread the word and link to avancier.website in whichever social media you use.