Agile 6 - EA in the world of agile architecture
Copyright Graham Berrisford. One of several hundred papers at http://avancier.website. Last updated 16/08/2019 20:32
What does agile
mean?
What does
architecture mean?
What does agile
architecture mean?
And what is good EA
in the age of agile architecture?
This is the sixth in a series of papers.
2. On agile software development
3. On agile businesses and systems
4. On systems thinking ideas used in agile
5. What is agile architecture?
6.
EA in the world of agile
architecture
Enterprise Architecture (EA) is a
product of the Information Age.
Enterprise architects
don’t design buildings or other concrete things.
Their interest is in
activity systems in which business roles and processes create and use data.
(Where
the data represents entities and events of importance to the business.)
And
in how far those activity systems can and should be digitised using information
technology.
Software
development teams contain bright people who can do more than programming and
testing.
In the agile
development world, they do analysis and project/product management.
In the
dev/ops world, they deploy applications into the production environment, and
manage that.
In the agile
architecture world, architecture is said to “emerge” from modularisation of a
software system.
EA emerged in the 1980s as a reaction to the results that emerged from silo business system development.
Scores or hundreds of local system development projects had produced a sprawling wasteful mess.
There were duplications and disintegrities between systems, and difficulties with data quality and data analysis.
IT departments concluded they needed more enterprise-wide design thinking about business data and processes.
This article
is about EA in the world described in recent papers on "agile
architecture”.
Agile means willing and able to speedily
respond to change; what architecture means is less clear.
Agile system design has different principles
from agile development.
Agile development is based on the idea that
system mutations should be small and frequent.
Principles include YAGNI (You Ain’t Gonna Need
it) and KISS (Keep it Simple).
Agile system design means designing a system so
it does not need to change when the environment changes.
This implies designing the system - in
anticipation of those changes – so it can respond to change by homeostatic
change or by reconfiguration.
Principles include WAGNI (We Are Gonna Need
it) and CFC (Complexify for Configurability).
Agility |
When the environment changes
the system |
The system change may
be called |
Agile
development |
Must
be changed to a new version or generation |
Evolution,
incremental development |
Agile
system design |
Need
not be changed, is flexible, configurable |
Homeostasis,
reconfiguration, growth |
For some software developers, the
architecture of a system is found in the code itself.
Or
found in diagrams showing how software components/objects are connected to
complete required behaviors.
For enterprise and solution architects,
architecture is a higher-level design-time description.
It is an architecture definition or model
that communicates the intent of the architect.
It is written in natural language or near to
it.
What is the purpose of an architecture definition or model?
It is to help people understand and change a system that the business wants to be stable and under change control.
Architects model business systems that exhibit
"regular, repeated or determinate" behaviors (Ashby, 1956).
Business structures and behaviors that are continually changing cannot be modeled.
Business managers must direct and manage them on a daily basis.
And if an architecture
changes much or often, you’ve got a problem.
To use a metaphor: the
architecture of a concrete building should allow for the fittings to be
changed.
But to change the
architecture might well mean discarding the fittings and re-building from
scratch.
So, what is an agile architecture?
Is it an architecture that can easily or speedily be changed - which implies the less there is, the more agile it is?
Or an architecture that need not be changed because it anticipates future requirements - contrary to the KISS and YAGNI of agile development?
This section summarises the discussion in Agile paper 5.
1. Intentional architecture
This is an up-front architecture drives the
development of a system.
It may be revised iteratively and/or expand
incrementally.
“One of
the more persistent myths of agile development is that up-front architecture
and design are bad …” Bob Martin
2 Emergent architecture
This architecture is documented after a
system has been built or changed.
This turns the architect from a designer to a
recorder.
Of course, this is contrary to the classical notion that an
architect draws diagrams that builders follow.
3 Lean
(high-level, light) architecture
This
need not be changed because it is so abstract.
The
trouble is, the more abstract it is, the less helpful it is (which is why EA needs
solution architects.)
4 The
architecture of an agile system
This
need not be changed because it defines a flexible or configurable system.
The
difficulty: the complexity/flexibility trade off.
E.g
design patterns for extreme scalability tend to increase complexity - contrary
to KISS & YAGNI
Agile
system principle |
Agile
development principle |
|
Decouple
subsystems |
contrary to |
Keep the
system simple (KISS) |
Design for
future flexibility |
contrary to |
You ain’t
gonna need it (YAGNI) |
5 Architecture that facilitates agile
development
Scaling up from one team to a wider
organisation.
In recent years, a primary concern of “agile
architecture”.
Principles to scale up agile development
·
modularise
the architecture into subsystems, assignable to (relatively) autonomous
development teams.
·
decouple
subsystems, so teams can work (relatively) autonomously
·
release
a “minimum viable product”, followed by short-cycle releases that extend the
system’s capability
·
use
agile development methods and practices, such as SCRUM and Kanban.
All of this must be subordinate to the need
to support and enable whatever roles and processes feature in the request for
architecture work.
Agilists tend to deprecate what they see as
current EA practices.
There is indeed some EA bad practice out
there; so what is good EA?
To scale up agile development, one has to consider the
following.
What kind of enterprise?
Different kinds of businesses change at different rates.
The likes of Facebook, Amazon, Netflix, Google and Spotify (FANGS) are peculiarly software-centred businesses.
Changing web sites and algorithms is
one thing.
Changing physical assets and workers
with special abilities, knowledge or skills is another.
See Agile paper 3 for discussion of FANGS.
What kind of scalability?
One might count the
number of:
· entity and event types that are monitored and directed by a system (variety or
complexity).
· entity and event instances a system must process (throughput)
· people in the system development organisation
(human power).
“Scaling up agile” usually relates primarily to the last.
It means scaling up to the level of an enterprise with
many (even hundreds) teams working in parallel.
Longer term planning
EA typically works
to a strategic horizon.
EA methods are traditionally for
planning major changes business systems, to a target that is one, two or more years ahead.
Due the time scale, initially-stated
requirements are often unclear, inaccurate, incomplete – and change later.
Agile development methods are about
what a team does to continually improve and extend a software system, with
release cycle of days or weeks.
The outcome should be faster delivery of something useful, though it may not do everything customers want.
Development teams
typically work iteratively in sprints of 10 to 30 days.
The backlog (e.g.an
epic list) may be up to two to three iterations ahead, say 3 months
A challenge for EA
is collaborate with each team - at the level of their backlog.
Higher level agility
Development teams are agile within a narrow scope, at low level of
design
A challenge for EA is to be agile at a wider scope, and a higher level.
In Scrum, the product owner is
expected to assume responsibility for the full product life-cycle, including the business case.
A challenge for EA is to ensure this
is coordinated with other business cases.
Synchronizing deliverables
In agile architecture frameworks, development teams are supposed to be
autonomous and self-organising.
A challenge for EA is coordinate tens to hundreds of development teams,
where necessary.
For example, it may be necessary to synchronise subsystems in one
release of a wider system.
Is the vision of EA achievable?
EA was a reaction
against difficulties caused by silo systems not interacting as they should do.
· Duplications: same data captured or cached in
different data stores
· Delays: processes are slowed by
inter-application messaging.
· Disintegrities: and the complexity of
compensating transactions to restore integrity.
· Data analysis difficulties: where a
management information requires data from several data stores.
However, EA is difficult politically and
technically.
It can inhibit the agile system development
an enterprise also needs to respond to changing conditions.
Generally, agile means willing and able to
respond to change. It can apply to:
·
Business
actors – working business operations
·
Software
developers – delivering working software
·
Architects
– delivering architecture definitions
·
Anybody
working in a “self-organising system”
Inevitably, architecture does constrain
agility in some ways
·
In
business operations… business actors can change how they perform business
operations - within the constraints of
the software systems they use.
·
In the
design and build of systems… designers and builders can change systems they
release to business actors - within the constraints of the architecture given
to them.
·
In
architecture… architects can change the architecture they release to system
developers - within the constraints of the business context.
1 Start by
understanding the specific business context
Good EA always starts from the particular business
context.
What is your business?
·
Data processing?
Bank, insurance, tax, Ecommerce, Application-centric internet giant (FANGS, as
discussed in Agile paper 3).
·
Material
processing? Manufacturing, retail, logistics etc.
·
Delivery
of human skills? Entertainment, security, prison, education etc.
Clearly, data
processing businesses are candidates for digitisation, even digital
transformation.
The white paper discussed in Agile paper 7 is based on
lessons learned in such “digital enterprises”.
The exemplars are application-centric internet giants
like Facebook, Amazon, Netflix, Google and Spotify (FANGS).
Many programmers think it cool to copy what FANGS do -
regardless of relevance, cost and complexity.
But most businesses are not like FANGS.
And design patterns that FANGS have developed for extreme
scale web applications (see below) come with costs and downsides.
2 Look to
strike the optimal balance
EA and agile software development are opposites in some ways.
Taken to the extremes, both lead to
difficulties.
Both are used in a “bimodal” IT department;
they can, should and must collaborate.
Architects have to understand the extremes
and strike a balance between
·
up-front
intentional architecture
·
incremental
development and emergent architecture
3
At every level, system design is a creative and challenging process
(In systems theory
terms, this is the territory of classical
cybernetics.)
By contrast, the teams themselves must be
relatively “self-organising” social networks.
They must be flexible, responsive to
interaction with stakeholders, responsive to requirements change.
Team members must
think in creative ways, consider tradeoffs, and make decisions without all the
information they want.
4
Delegate to competent architects/designers
Good EA
looks to make coarse-grained architecture decisions, leaving room for others to
make fine-grained decisions.
It operates
at the level of an enterprise’s long running business processes and
coarse-grained application portfolio.
It likely
delegates project-level architecture definition to people playing solution
architect roles (read EA v SA)
It
encourages agile software development within an application project/product.
It
recognises that where agile practices are not a good fit, software development
usually takes longer and costs more.
5
Recognise tradeoffs between design patterns
Good EA
knows there is no silver bullet or
one-size-fits-all method or solution.
Different
design patterns fit different problems and requirements.
In architecture and
design (as our architect training courses emphasise) all is trade offs
There are
tradeoffs, for example between design for simplicity and for flexibility.
EA strives
to minimise wasteful duplications between systems, and to
integrate systems that work on the same data.
At the same time, it recognises that a large
and complex system should be modularised for manageability.
6 Decouple subsystems
to meet a defined need, rather than by default
Good EA recognises Constantine’s principles of modularity
(1968).
· high cohesion within a subsystem
·
loose
coupling between subsystems.
OK, but these principles apply differently at coarser and finer-grained levels of granularity.
Physical decoupling does not remove business
requirements for logical coupling.
It can lead to inter-system delays, disintegrities and
difficulties analysing business-wide data.
And as Craig Larman points out, decoupling subsystems
where there is no clear need is not time will spend.
Inevitably, division into smaller simpler subsystems
increases the complexity of inter-subsystem messaging.
This is great for the middleware vendor, but while
pub-sub middleware and event-driven messaging are useful, they are not best
used everywhere.
7 Recognise
Conway’s law
Good EA recognises Conway’s law (also 1968), which says
there is a natural correspondence between:
·
the
modularisation of a software system into subsystems
·
the
modularisation of a human organisation into development teams.
This is crystalized
in the Bezos mandate.
8 Design
subsystems with higher level processes in mind
Good EA recognises that large and complex system should be modularised into subsystems that are relatively autonomous.
OK, but scatter-brained silo system (or
microservice) development leads to chaos.
Remember to optimise the end-to-end business
processes.
To do this by coordinating scores or hundreds of
subsystems/teams does require some oversight.
Good EA is concerned to ensure that subsystems support higher
level processes and maintain consistency of data.
Fragmenting applications and delegating decisions may
lead to diversification and duplication, the very issues EA was created to
reduce.
It can fragment data stores, making it harder to maintain
data integrity and do cross-system data analysis.
Mitigating these risk, inevitably, involves review project
milestones
Typically, solution architects must complete “solution
outline” documents and get them approved.
9 Accommodate short term needs
Different business
system changes have different time and cost profiles.
A business strategy
can trigger a long-term transformation project that last years and cost
millions.
An architect in a
global business told me of a project completed in 2016 that cost $200 million.
That same business
has to respond to quickly to immediate problems and pressures.
Sometimes, managers
want a new business system at three months notice; or even faster.
What if managers insist
they need a point solution tomorrow?
Good EA can give
waiver for an innovative and/or agile project to proceed outside of normal
governance.
(Beware without
architectural oversight, both buying packages and agile software development
can create disintegrated and/or un-maintainable systems.)
10 Employ good architects
An architecture method is designed to ensure
there are agreed objectives, models of baseline and target systems and
governance of system delivery.
There is a down side - the necessary bureaucracy.
Pro-forma documentation is problematic
An EA framework can waste your time and money
if you wrongly read it as a prescription for what to do.
It can help if you know what you are doing
and use it as a menu of products and processes.
(Wrt a coherent and consistent
enterprise-wide repository, it seems the jury is out on what of that kind is
optimally practical and useful.)
Employ well-trained experienced and
knowledgeable people at all levels.
And expect them to share their knowledge.
11 Guerilla and Collective EA
If you cannot afford an EA function, consider
Guerilla EA
For more on that read on.
EA provides business change initiatives with a measure of centralised
oversight and leadership
It aims to support, enable and coordinate business actors in completing
end-to-end business processes that yield results of the value to the business.
And to integrate systems so as to ensure data quality and enable
cross-system data analysis.
The scope of an EA team varies from enterprise-wide to one business
segment, capability or function.
Its members may include enterprise
and solution architects and specialist domain experts.
Some or most of its members may be
assigned to change projects, full time or part time
EA teams often model a whole baseline business at a high level, but
rarely (if ever) look to change the whole.
EA is not top-down command and
control; it responds to diverse requests for work from sponsors.
It may initiate some changes; but more often steers changes
requested by others.
It looks at each change initiative as a distinct programme or project.
EA does not manage change projects; it shapes and steers them
It steers or governs them with
business goals in mind
It governs them optimise the wider
enterprise – minimising waste by standardisation and integration.
It governs then against whatever enterprise-wide or strategic road
maps, standards, principles are in place.
EA does not prevent prototyping or
innovation.
It encourages agile software
development, and presumes responsiveness to requirements change.
But EA does presume a judicious
amount of design up front!
None of this is easy; it can be more political than technological.
To use an analogy given to me by a
colleague: is EA like piloting an oil tanker or herding cats?
An oil tanker’s journey must be
carefully planned because it cannot easily change direction.
By contrast, herding cats requires
continual adaptation to an ever-changing situation.
Both approaches have some utility, but
most of what we need to do in practice lies between the extremes.
EA
is political, and there are natural cycles in the status of EA in
organisations.
Much hangs on the personality and persuasiveness of those at the centre, and what some call the "culture" of the enterprise.
What about small-to-medium enterprises - who can't afford a substantial
central team?
Most businesses still need solution architects to shape and steer
projects.
And they need some central thinking about SFIA calls "Strategy and
Architecture".
This central team, function, forum or person may be responsible for supplying solution architects to projects.
EA can be about making small steps towards the vision of digitised, standardised, integrated business processes and data.
“Guerrilla EA” does what it can, when and where it can, to
these ends.
It is the practice of advancing EA through the efforts of distributed
solution architects who are coordinated and self-organised to that end.
A
minimal central team is employed to
· provide
cross-organisational oversight
· encourage architect to cooperate with EA aims in mind.
· socialise architects into a group that cooperates by aligning and integrating solutions
· give architects some time and opportunity to do strategic EA level analysis and planning
· encourage them to do what they can, when and where they can, to move the enterprise towards an EA vision.
E.g. The organisation in the table below has no enterprise architect, and only four solution architects.
But they collaborate with an EA mind set when the opportunity arises.
A university |
total |
c75 people in business system
change |
total |
c165 people in IT services
management |
total |
Campus data centres |
2 |
Solution architects (TDAs with
EA mind set) |
4
|
Groupware / mailing list |
3 |
Schools |
26 |
Supplier relationship managers
(licences) |
2 |
Senior management |
5 |
Buildings (25m) |
300 |
Data integration specialists |
5 |
Data centre / hardware /
operations |
6 |
Staff |
9000 |
Application development team |
13 |
File store / directory /
identity management |
7 |
Students |
31000 |
User engagement and requirements specialists |
24 |
Service desk |
9 |
|
PMO (programme/project
managers, back office) |
27 |
Client Devices, Desktop apps,
OS and printers |
10 |
|
Clerical administration |
10 |
||||
IS/IT projects per year |
150 |
OS / databases / networks |
30 |
||
Business applications |
600 |
|
|
Hardware – desk tops,
projectors etc. |
85 |
There being no EA function, the principal solution architect sent two others, not called “architect”, to attend our architect training.
An education in enterprise and solution architecture can be relevant and useful to people with no EA function.
Setting up a central EA function without a solid foundation is likely to fail.
First you probably need:
· your change/project initiation and management processes to
include approval of a solution architecture definition (SAD) before the project
starts
· those processes to be adhered to
· a team or pool of solution architects who can (with SME
assistance) produce SADs and help managers to shape and steer projects
· an inspiring, or at least good, leader for that team/pool
· positive feedback from the team’s efforts.
Then you probably need:
· to identify some strategic and cross-organisational problems
that adding an EA function on top of that team will solve
· to convince some sponsors to back the EA function and create a
cross-organisational architecture board with clout.
· willing and active business stakeholders
· a business change function to address changes to people's
roles.
And some good fortune in your initial
endeavours.
Read the papers on the Agile Architecture page at http://avancier.website
2. On agile software development
3. On agile businesses and systems
4. On systems thinking ideas used in agile
5. What is agile architecture?
6.
EA
in the world of agile architecture
Also
1. Commentary on “Agile Architecture in the Digital Age”
You’ll find more
here – though it seems more for large-scale solution/system architecture than
EA.
DSDM for portfolio
management http://www.agilebusiness.org
Scaled Agile
Framework (SAFE) http://www.scaledagileframework.com
For large integrated solutions that require hundreds of people to
develop and maintain
Spotify-engineering-culture https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/
People organisation: Squads, Tribes, Chains, Guilds.
Product organisation: Clients, Features, Infrastructure.