Agile Architecture: an oxymoron?
Copyright Graham Berrisford. One of several hundred papers at http://avancier.website. Last updated 14/12/2018 23:04
What is an Agile Architecture?
Is the aim to produce an Agile Architecture or an agile system?
This paper is a supplement to the earlier paper on Enterprise architecture challenges.
“Thanks! You article contains some really good reflections on the difference between developing with agile methods and designing solutions to be agile.”
Different businesses change at different rates.
Changing abstract descriptions is one thing; changing concrete realities is another.
A business that is little more than a web site can change rapidly.
A business that owns/employs physical assets and workers with special physical abilities, knowledge or skills cannot.
Different business system changes have different time and cost profiles.
A business strategy can trigger a long-term transformation project that last years and cost millions.
An architect in a global business told me of a project completed in 2016 that cost $200 million.
That same business has to respond to quickly to immediate problems and pressures.
Sometimes, managers want a new business system at three months notice; or even faster.
What if managers insist they need a point solution tomorrow?
Enterprise architects can give waiver for an innovative and/or agile project to proceed outside of normal governance.
But without architectural oversight, both buying packages and agile software development can create disintegrated and/or un-maintainable systems.
The discussion here is about agile software development in an enterprise architecture context.
Agile means “willing and able to speedily respond to change”.
What architecture means is less clear.
For some, the architecture of a software system is its basic structure - in the code.
It is the stable abstractions (compositions, interfaces and generalisations) around which the system is built.
For others, architecture is higher level design-time description rather than code.
It may be an output of up-front planning, or reverse-engineered from a run-time system.
Either way, if the architecture changes much or often, you’ve got a problem.
The architecture of a concrete building should allow for the fittings to be changed.
By contrast, to change the architecture might well mean discarding the fittings and re-building from scratch.
Analogies with the architecture of concrete buildings, airplanes and city plans have misled many about the nature of enterprise, solution and software architecture.
You may look at the architecture of a building (in run-time reality) and the architectural drawings of it (a design-time description).
It is immediately clear how well the run-time reality visually resembles design-time description – or not.
You cannot look at the run-time reality of a software system.
You see only a user interface that reveals nothing of the software architecture
Enterprise and solution architects rarely look at the executable design-time description - the code of the software system
Instead, they document a much more abstract design-time description using natural language and graphical models.
They then govern software development for compliance of the code to the intent of that documentation - not to its visual form.
Beware the several meanings of agile architecture
The term “Agile Architecture” is used with several meanings that might be more clearly called something else.
This term embraces a variety of practices that enable the rapid incremental development of a system.
The first release of a system (a minimum viable product) is followed by short-cycle releases that extend the system’s capability.
An agile development method is not only iterative, but also flexible about the requirements, the solution and the process being followed.
· It favours negotiation over planning, and flexibility about requirements.
· It presumes user involvement and feedback is essential to success.
· It encourages early testing for usability and performance.
· It favours short-cycle iterative development.
· It looks to deliver a “minimal viable product” and extend that incrementally.
· It encourages a team to change, rebuild and re-testing a software system on a daily basis.
Today, commonly used agile development methods or practices include SCRUM and Kanban, not discussed here.
Obviously, architectural documentation can be reverse-engineered from an emergent design, and maintained in step with software development.
In an extreme version of agile development, there is no up-front architecture or architect.
‘One of the more persistent myths of agile development is that up-front architecture and design are bad …” Bob Martin
“While we must acknowledge emergence in design and system development, a little planning can avoid much waste.” James O. Coplien.
“Lean architecture comes from applying the principles of the Toyota Production System to software architecture.
Lean architecture is both about an architecture with no fat, and about the consistency and reduction of waste in the process surrounding its creation and use.”
Read this article for more http://leanmagazine.net/lean/lean-architecture/.
Architecting a system to be agile
This appears to be a goal in SAFe, explored below.
Remember that agile development and agile systems are two different things.
How far is SAFe scalable above software architecture?
Is it about the architecture or the system itself?
And is it a coherent framework or a collection of sometimes conflicting ideas?
“Agile Architecture is a set of values and practices that support the active evolution of the design and architecture of a system while implementing new system capabilities.”
The Agile Architecture principles support Lean-Agile development through
· emergent design,
· intentional architecture,
· design simplicity, as well as
· designing for testability, deployability and releasability.”
Let us take collaboration for granted.
Emergent design means that the design of a system emerges from incremental development.
By contrast, intentional design means designing a system ahead of development.
(It is also, by the way, an intensional definition of a set member.)
Intentional architecture implies some up front documentation, and some intent to design an agile system.
For sure, some architectural oversight of agile system development is desirable.
Agile Architecture favours lean up-front architecture, and encourages people to optimise the balance between that and emergent design.
You may know that agile development favours the Keep it Simple (KISS) principle.
You may not know there is a fundamental design trade off.
Designing a system for flexibility usually makes a system more complex, and run slower.
In an article promoting Agile Architecture, a CTO is quoted as saying the following.
“The only way to manage a large distributed system is to keep things as simple as possible.”
OK, but isn’t that precisely because the "architecture" of large distributed systems is inherently complex?
“Keep things simple by making sure there are no hidden requirements in the design.”
What are hidden requirements?
You may know that agile development favours the You Ain’t Gonna Need It (YAGNI) principle.
The idea is not to anticipate future requirements, and to refactor when it becomes necessary.
But architecting an agile system does mean including features in the design that anticipate future requirements.
And a system cannot be agile in every possible way, it has to be designed with specific kinds of change in mind.
[And no] hidden dependencies in the design.”
What are hidden dependencies?
We can make all dependencies visible in architectural documentation
OK, but that implies a heavy non-agile documentation overhead.
We can all remove synchronous interactions between distributed components,
OK, but that increases the complexity of maintaining data integrity dependencies and compensating transactions.
Read this Microservices paper for some decoupling trade offs.
“Cut technology to the minimum you need to solve the problem you have.”
It certainly seems a good idea not to use technologies you don’t need.
But what does this actually mean in an enterprise application design context?
Use thin clients with no application code on client devices?
Don’t use middleware when you can use REST? Some enterprise architects promote the reverse.
“It doesn’t help the company to create artificial and unneeded layers of complexity.”
OK, but nobody does this deliberately unless they are trying to design for maintainability.
And if asked, many sponsors will say they do want an agile (flexible, configurable) system.
To achieve this, you’ll have to make the system more complex than is needed in the short term.
Designing for testability, deployability and releasability
For sure, test-driven design is an agile development principle.
Tests are built as “executable specification”.
But there is a trade off, since the more test cases you build, the harder it is to change the test cases.
And therefore, the harder it is to release a new system.
SAFe includes half a dozen or so principles; some are variations on agile development principles.
Three of the more architectural principles are mentioned below.
#2 The Bigger the System, the Longer the Runway.
This means having enough (say 3 to 9 month) infrastructure to complete near-term product backlog without refactoring.
What is infrastructure? The term covers many things.
It can include client devices, server devices, operating systems, database management systems, middleware, data storage etc.
To a software engineer, it includes what they use by way of a database schema and module libraries (class libraries, DLLs...).
The trouble is, 6 months is a rolling future, and 1 month from now, the 6 month future will be different.
So, surely maintaining the runway to anticipate the 6 month ahead implies continual refactoring?
#3 Build the Simplest Architecture That Can Possibly Work.
What does it mean to build an architecture rather than a system?
Building the simplest system that can possibly work is an agile development principle for sure.
But what does building the simplest architecture mean?
If it means the same, then it can inhibit the design of a system that is flexible and configurable.
If it means something different, then does it conflict with agile development principle?
“Domain-driven design, design patterns, and applying a system metaphor, simplify both the design and the communication between the teams.”
Aside from applying SOA as per the Bezos mandate - hiding each teams system(s) behind APIs - this advice seems contrary to design for simplicity.
First, in some articles on SAFe, principles or standards are presented as patterns.
Remember, “If you can’t draw it, it isn’t a pattern” Christopher Alexander.
Second, design patterns are not prescriptions, and often come in pairs (e.g. centralise versus distribute).
What architects need is guidance on the trade-offs between them.
And to build an agile system requires us to successfully predict how future changes will shift those trade offs.
Third, many design patterns, by adding a level of indirection, make a system more complex.
(Generally, making a system more agile tends to make the architecture more complex, and make the system slower.)
Finally, on a specific point, Martin Fowler says domain-driven design is best reserved for complex systems.
He says transaction scripts are simpler for most systems.
#6 There Is No Monopoly on Innovation.
Surely, here, to be agile means being able in mid-course software development to adopt an innovation with relative ease.
But however useful an innovation might be, adopting it is a departure from incremental development.
Adopting a significant innovations is likely to incur the time and cost of substantially re-architecting or re-enginering the system.
And until the innovation is understood, we don’t know how to anticipate it.
And until the innovation is on the road map, we don’t know we need to anticipate it.
So, in what sense does researching innovations enable “Agile Architecture”?
Enterprise architecture methods are traditionally for planning major changes business systems, to a target that may be a year, two years or more away.
Often, initially-stated requirements unclear, inaccurate, incomplete – and change over time.
Agile development methods are about what a team does to continually improve and extend a software system, with release cycle of days or weeks.
The outcome should be faster delivery of something useful, though it may not do everything customers want.
Agile development does not necessarily produce agile systems.
The plain fact is, digitisation of systems is and always has been difficult.
To make a useful software system is a painstaking task.
Reengineering an old system using new (innovative) technologies may make little difference.
Replacing like for like may do little or nothing to make a system more agile.
To do that usually requires a redesign that makes the system more complex and slower.
(In one story, after being rewritten in Java, a single-server legacy system required a dozen servers.)
Moreover, the notion of an agile system is not a simple or single idea
Since a system can be flexible in several ways, as described in the table below.
yet might also be
its architecture may be
A versatile system
do many different things
rigid and hard to change or extend.
large or complex
A malleable system
be readily changed to a new version
limited and not versatile
slight or simple
An extendable system
be extended with extra features
not malleable (the Open-Closed principle)
fixed (must be right first time)
A self-organising system
change its own roles and rules
reliant on human ability, barely a system at all
slight and simple
Designing a complex business system so that it can be quickly changed is a challenge; and there are always trade offs.
Generally, building flexibility into a system makes it more complex – which adds time and cost to development.
It may mean shifting business rules from procedural instructions into data variables that end users can change – which tends to make the system slower.
And maintaining flexibility may require substantial “refactoring” of the system from time to time – which requires investment over and above normal change requests.
All is trade offs.
Some see agile development as the anti-thesis of enterprise architecture.
However, both approaches are used in a “bimodal” IT department.
We know that the agile development of small and/or silo systems can create the issues that led people to promote EA in the first place.
The same can be said of the fashion to develop “microservices” (as this Microservices paper explains).
Mitigating that risk, inevitably, involves tightening up of project milestones
Typically, solution architects must complete “solution outline” documents and get them approved.
Architects are expected to document some up front high-level design and principles.
They may recommend that an agile development method is used.
They may then govern the acceptability of changes requested during an agile development project.
And reflect accepted changes in architecture-level documentation.
Agile development methods do not always give the hoped-for benefits.
They do not always mean quicker and cheaper delivery of a system.
It often takes a long time to develop applications that require a large/complex database structure and/or data migration.
Some systems cannot be partially implemented – must be right first time.
Architects need a score chart to define the suitability of agile development methods.
Obstacles to agility include:
1. Most/all requirements are mandatory.
2. Users/domain experts are not readily available.
3. Developers are not empowered to make decisions.
4. The system’s first release must be large (safety-critical, money-handling, legacy replacement, or regulatory compliance).
5. The complexity is server-side rather than UEX side.
6. The complexity is in input and update rather output and report.
7. The business rules are complex and/or money-handling.
8. The processing is batch rather than transactional.
9. The database schema is immature and evolving.
10. There are many integration points with other systems.
A project that scores highly on these measures is an inherently difficult project.
So, consider substantially increasing the time and budget for the work.
An architecture method is designed to ensure there are agreed objectives, models of baseline and target systems, and governance of system delivery.
There is a down side - the necessary bureaucracy – to create documentation and analyse the impacts of change requests against that documentation.
Of course, untrained/unthinking completion of document templates and untrained/unthinking approval of the resulting documents is ineffective.
The problem is not the document templates; it is ill-educated, ill-informed document completion; and naivety in document reviewers.
There is no substitute for employing well-trained experienced and knowledgeable people at all levels.
All is trade offs in architecture and design (as our architect training courses emphasise).
And there are trade offs in methodology too.
Taken to the extremes, both EA bureaucracy and agile development lead to difficulties.
Architects have to understand the extremes and strike a balance between them.
Being appropriately Agile and having a well understood Architecture are both good things.
The challenge is create the right balance between the two based on an understanding of specific requirements (functional and non-functional), foreseeable changes and deadlines.
Why didn’t multi-level model-based systems engineering take off?
Every documentation level is an overhead; every transformation between levels is an overhead.
The closer the levels to each other, the more levels are needed, the bigger the documentation overhead, and the less value added by a transformation.
The further apart the levels, the less the documentation overhead, the more valuable but also difficult and costly a transformation.
Model-driven engineering is intended to improve quality and verifiability or auditability.
But user acceptance testing is impossible until the bottom level executable description is complete.
At which point, the users may clarify or change their minds about what they really wanted the system to do.
So, the agile development principle is to maintain only high-level or “lean” architectural documentation.
It is impossible to get all the advantages of both approaches; you have to find the right balance for each situation.
All free-to-read materials at http://avancier.website are paid for out of income from Avancier’s training courses and methods licences.
If you find the web site helpful, please spread the word and link to avancier.website in whichever social media you use.