Agile 6 - EA in the world of agile architecture

https://bit.ly/2UPrjOC

Copyright Graham Berrisford. One of several hundred papers at http://avancier.website. Last updated 16/08/2019 20:32

 

What does agile mean?

What does architecture mean?

What does agile architecture mean?

And what is good EA in the age of agile architecture?

 

This is the sixth in a series of papers.

1.     On the beginnings of agile

2.     On agile software development

3.     On agile businesses and systems

4.     On systems thinking ideas used in agile

5.     What is agile architecture?

6.     EA in the world of agile architecture

For a slide show that draws from the above email me here.

Preface

Enterprise Architecture (EA) is a product of the Information Age.

Enterprise architects don’t design buildings or other concrete things.

Their interest is in activity systems in which business roles and processes create and use data.

(Where the data represents entities and events of importance to the business.)

And in how far those activity systems can and should be digitised using information technology.

 

Software development teams contain bright people who can do more than programming and testing.

In the agile development world, they do analysis and project/product management.

In the dev/ops world, they deploy applications into the production environment, and manage that.

In the agile architecture world, architecture is said to “emerge” from modularisation of a software system.

 

EA emerged in the 1980s as a reaction to the results that emerged from silo business system development.

Scores or hundreds of local system development projects had produced a sprawling wasteful mess.

There were duplications and disintegrities between systems, and difficulties with data quality and data analysis.

IT departments concluded they needed more enterprise-wide design thinking about business data and processes.

 

This article is about EA in the world described in recent papers on "agile architecture”.

Agile means willing and able to speedily respond to change; what architecture means is less clear.

What does agile mean

Agile system design has different principles from agile development.

Agile development is based on the idea that system mutations should be small and frequent.

Principles include YAGNI (You Ain’t Gonna Need it) and KISS (Keep it Simple).

 

Agile system design means designing a system so it does not need to change when the environment changes.

This implies designing the system - in anticipation of those changes – so it can respond to change by homeostatic change or by reconfiguration.

Principles include WAGNI (We Are Gonna Need it) and CFC (Complexify for Configurability).

 

Agility

When the environment changes the system

The system change may be called

Agile development

Must be changed to a new version or generation

Evolution, incremental development

Agile system design

Need not be changed, is flexible, configurable

Homeostasis, reconfiguration, growth

What does architecture mean?

For some software developers, the architecture of a system is found in the code itself.

Or found in diagrams showing how software components/objects are connected to complete required behaviors.

 

For enterprise and solution architects, architecture is a higher-level design-time description.

It is an architecture definition or model that communicates the intent of the architect.

It is written in natural language or near to it.

 

What is the purpose of an architecture definition or model?

It is to help people understand and change a system that the business wants to be stable and under change control.

Architects model business systems that exhibit "regular, repeated or determinate" behaviors (Ashby, 1956).

What does agile architecture mean?

Business structures and behaviors that are continually changing cannot be modeled.

Business managers must direct and manage them on a daily basis. 

 

And if an architecture changes much or often, you’ve got a problem.

To use a metaphor: the architecture of a concrete building should allow for the fittings to be changed.

But to change the architecture might well mean discarding the fittings and re-building from scratch.

 

So, what is an agile architecture?

Is it an architecture that can easily or speedily be changed - which implies the less there is, the more agile it is?

Or an architecture that need not be changed because it anticipates future requirements - contrary to the KISS and YAGNI of agile development?

 

This section summarises the discussion in Agile paper 5.

 

1. Intentional architecture

This is an up-front architecture drives the development of a system.

It may be revised iteratively and/or expand incrementally.

“One of the more persistent myths of agile development is that up-front architecture and design are bad …” Bob Martin

 

2 Emergent architecture

This architecture is documented after a system has been built or changed.

This turns the architect from a designer to a recorder.

Of course, this is contrary to the classical notion that an architect draws diagrams that builders follow.

3 Lean (high-level, light) architecture

This need not be changed because it is so abstract.

The trouble is, the more abstract it is, the less helpful it is (which is why EA needs solution architects.)

 

4 The architecture of an agile system

This need not be changed because it defines a flexible or configurable system.

The difficulty: the complexity/flexibility trade off.

E.g design patterns for extreme scalability tend to increase complexity - contrary to KISS & YAGNI

 

Agile system principle

Agile development principle

Decouple subsystems

contrary to

Keep the system simple (KISS)

Design for future flexibility

contrary to

You ain’t gonna need it (YAGNI)

 

5 Architecture that facilitates agile development

Scaling up from one team to a wider organisation.

In recent years, a primary concern of “agile architecture”.

 

Principles to scale up agile development

·       modularise the architecture into subsystems, assignable to (relatively) autonomous development teams.

·       decouple subsystems, so teams can work (relatively) autonomously

·       release a “minimum viable product”, followed by short-cycle releases that extend the system’s capability

·       use agile development methods and practices, such as SCRUM and Kanban.

 

All of this must be subordinate to the need to support and enable whatever roles and processes feature in the request for architecture work.


Agilists tend to deprecate what they see as current EA practices.

There is indeed some EA bad practice out there; so what is good EA?

Scaling up to the wider enterprise

To scale up agile development, one has to consider the following.

 

What kind of enterprise?

Different kinds of businesses change at different rates.

The likes of Facebook, Amazon, Netflix, Google and Spotify (FANGS) are peculiarly software-centred businesses.

Changing web sites and algorithms is one thing.

Changing physical assets and workers with special abilities, knowledge or skills is another.

See Agile paper 3 for discussion of FANGS.

 

What kind of scalability?

One might count the number of:

·       entity and event types that are monitored and directed by a system (variety or complexity).

·       entity and event instances a system must process (throughput)

·       people in the system development organisation (human power).

 

“Scaling up agile” usually relates primarily to the last.

It means scaling up to the level of an enterprise with many (even hundreds) teams working in parallel.

 

Longer term planning

EA typically works to a strategic horizon.

EA methods are traditionally for planning major changes business systems, to a target that is one, two or more years ahead.

Due the time scale, initially-stated requirements are often unclear, inaccurate, incomplete – and change later.

 

Agile development methods are about what a team does to continually improve and extend a software system, with release cycle of days or weeks.

The outcome should be faster delivery of something useful, though it may not do everything customers want.

Development teams typically work iteratively in sprints of 10 to 30 days.

The backlog (e.g.an epic list) may be up to two to three iterations ahead, say 3 months

A challenge for EA is collaborate with each team - at the level of their backlog.

 

Higher level agility

Development teams are agile within a narrow scope, at low level of design

A challenge for EA is to be agile at a wider scope, and a higher level.

 

Wider responsibility

In Scrum, the product owner is expected to assume responsibility for the full product life-cycle, including the business case.

A challenge for EA is to ensure this is coordinated with other business cases.

 

Synchronizing deliverables

In agile architecture frameworks, development teams are supposed to be autonomous and self-organising.

A challenge for EA is coordinate tens to hundreds of development teams, where necessary.

For example, it may be necessary to synchronise subsystems in one release of a wider system.

Eleven principles of good EA – in the age of agile architecture

 

Is the vision of EA achievable?

EA was a reaction against difficulties caused by silo systems not interacting as they should do.

·       Duplications: same data captured or cached in different data stores

·       Delays: processes are slowed by inter-application messaging.

·       Disintegrities: and the complexity of compensating transactions to restore integrity.

·       Data analysis difficulties: where a management information requires data from several data stores.

 

However, EA is difficult politically and technically.

It can inhibit the agile system development an enterprise also needs to respond to changing conditions.

 

Generally, agile means willing and able to respond to change. It can apply to:

·       Business actors – working business operations

·       Software developers – delivering working software

·       Architects – delivering architecture definitions

·       Anybody working in a “self-organising system”

 

Inevitably, architecture does constrain agility in some ways

·       In business operations… business actors can change how they perform business operations -  within the constraints of the software systems they use.

·       In the design and build of systems… designers and builders can change systems they release to business actors - within the constraints of the architecture given to them.

·       In architecture… architects can change the architecture they release to system developers - within the constraints of the business context.

 

1 Start by understanding the specific business context

Good EA always starts from the particular business context.

What is your business?

·       Data processing? Bank, insurance, tax, Ecommerce, Application-centric internet giant (FANGS, as discussed in Agile paper 3).

·       Material processing? Manufacturing, retail, logistics etc.

·       Delivery of human skills? Entertainment, security, prison, education etc.

 

Clearly, data processing businesses are candidates for digitisation, even digital transformation.

The white paper discussed in Agile paper 7 is based on lessons learned in such “digital enterprises”.

The exemplars are application-centric internet giants like Facebook, Amazon, Netflix, Google and Spotify (FANGS).

Many programmers think it cool to copy what FANGS do - regardless of relevance, cost and complexity.

But most businesses are not like FANGS.

And design patterns that FANGS have developed for extreme scale web applications (see below) come with costs and downsides.

 

2 Look to strike the optimal balance

EA and agile software development are opposites in some ways.

Taken to the extremes, both lead to difficulties.

Both are used in a “bimodal” IT department; they can, should and must collaborate.

Architects have to understand the extremes and strike a balance between

·       up-front intentional architecture

·       incremental development and emergent architecture

 

3 At every level, system design is a creative and challenging process

EA and agile development teams define regular business systems in which regular roles and processes create and use business data.

(In systems theory terms, this is the territory of classical cybernetics.)

By contrast, the teams themselves must be relatively “self-organising” social networks.

They must be flexible, responsive to interaction with stakeholders, responsive to requirements change.

Team members must think in creative ways, consider tradeoffs, and make decisions without all the information they want.

 

4 Delegate to competent architects/designers

Good EA looks to make coarse-grained architecture decisions, leaving room for others to make fine-grained decisions.

It operates at the level of an enterprise’s long running business processes and coarse-grained application portfolio.

It likely delegates project-level architecture definition to people playing solution architect roles (read EA v SA)

It encourages agile software development within an application project/product.

It recognises that where agile practices are not a good fit, software development usually takes longer and costs more.

 

5 Recognise tradeoffs between design patterns

Good EA knows there is no silver bullet or one-size-fits-all method or solution.

Different design patterns fit different problems and requirements.

In architecture and design (as our architect training courses emphasise) all is trade offs

There are tradeoffs, for example between design for simplicity and for flexibility.

EA strives to minimise wasteful duplications between systems, and to integrate systems that work on the same data.

At the same time, it recognises that a large and complex system should be modularised for manageability.

 

6 Decouple subsystems to meet a defined need, rather than by default

Good EA recognises Constantine’s principles of modularity (1968).

·       high cohesion within a subsystem

·       loose coupling between subsystems.

 

OK, but these principles apply differently at coarser and finer-grained levels of granularity.

Physical decoupling does not remove business requirements for logical coupling.

It can lead to inter-system delays, disintegrities and difficulties analysing business-wide data.

And as Craig Larman points out, decoupling subsystems where there is no clear need is not time will spend.

Inevitably, division into smaller simpler subsystems increases the complexity of inter-subsystem messaging.

This is great for the middleware vendor, but while pub-sub middleware and event-driven messaging are useful, they are not best used everywhere.

 

7 Recognise Conway’s law

Good EA recognises Conway’s law (also 1968), which says there is a natural correspondence between:

·       the modularisation of a software system into subsystems

·       the modularisation of a human organisation into development teams.

 

This is crystalized in the Bezos mandate.

 

8 Design subsystems with higher level processes in mind

Good EA recognises that large and complex system should be modularised into subsystems that are relatively autonomous.

OK, but scatter-brained silo system (or microservice) development leads to chaos.

Remember to optimise the end-to-end business processes.

To do this by coordinating scores or hundreds of subsystems/teams does require some oversight.

 

Good EA is concerned to ensure that subsystems support higher level processes and maintain consistency of data.

Fragmenting applications and delegating decisions may lead to diversification and duplication, the very issues EA was created to reduce.

It can fragment data stores, making it harder to maintain data integrity and do cross-system data analysis.

Mitigating these risk, inevitably, involves review project milestones

Typically, solution architects must complete “solution outline” documents and get them approved.

 

9 Accommodate short term needs

Different business system changes have different time and cost profiles.

A business strategy can trigger a long-term transformation project that last years and cost millions.

An architect in a global business told me of a project completed in 2016 that cost $200 million.

That same business has to respond to quickly to immediate problems and pressures.

Sometimes, managers want a new business system at three months notice; or even faster.

What if managers insist they need a point solution tomorrow?

Good EA can give waiver for an innovative and/or agile project to proceed outside of normal governance.

(Beware without architectural oversight, both buying packages and agile software development can create disintegrated and/or un-maintainable systems.)

 

10 Employ good architects

An architecture method is designed to ensure there are agreed objectives, models of baseline and target systems and governance of system delivery.

There is a down side - the necessary bureaucracy.

Pro-forma documentation is problematic

An EA framework can waste your time and money if you wrongly read it as a prescription for what to do.

It can help if you know what you are doing and use it as a menu of products and processes.

(Wrt a coherent and consistent enterprise-wide repository, it seems the jury is out on what of that kind is optimally practical and useful.)

Employ well-trained experienced and knowledgeable people at all levels.

And expect them to share their knowledge.

 

11 Guerilla and Collective EA

If you cannot afford an EA function, consider Guerilla EA

For more on that read on.

What is the role of a central EA team?

EA provides business change initiatives with a measure of centralised oversight and leadership

It aims to support, enable and coordinate business actors in completing end-to-end business processes that yield results of the value to the business.

And to integrate systems so as to ensure data quality and enable cross-system data analysis.

 

The scope of an EA team varies from enterprise-wide to one business segment, capability or function. 

Its members may include enterprise and solution architects and specialist domain experts.

Some or most of its members may be assigned to change projects, full time or part time

 

EA teams often model a whole baseline business at a high level, but rarely (if ever) look to change the whole.

EA is not top-down command and control; it responds to diverse requests for work from sponsors.

It may initiate some changes; but more often steers changes requested by others.

It looks at each change initiative as a distinct programme or project.

 

EA does not manage change projects; it shapes and steers them

It steers or governs them with business goals in mind

It governs them optimise the wider enterprise – minimising waste by standardisation and integration.

It governs then against whatever enterprise-wide or strategic road maps, standards, principles are in place.

 

EA does not prevent prototyping or innovation.

It encourages agile software development, and presumes responsiveness to requirements change.

But EA does presume a judicious amount of design up front!

 

None of this is easy; it can be more political than technological.

To use an analogy given to me by a colleague: is EA like piloting an oil tanker or herding cats?

An oil tanker’s journey must be carefully planned because it cannot easily change direction.

By contrast, herding cats requires continual adaptation to an ever-changing situation.

Both approaches have some utility, but most of what we need to do in practice lies between the extremes.

Guerilla EA

EA is political, and there are natural cycles in the status of EA in organisations.

Much hangs on the personality and persuasiveness of those at the centre, and what some call the "culture" of the enterprise.

What about small-to-medium enterprises - who can't afford a substantial central team?

 

Most businesses still need solution architects to shape and steer projects.

And they need some central thinking about  SFIA calls "Strategy and Architecture".

This central team, function, forum or person may be responsible for supplying solution architects to projects.

 

EA can be about making small steps towards the vision of digitised, standardised, integrated business processes and data.

“Guerrilla EA” does what it can, when and where it can, to these ends.

It is the practice of advancing EA through the efforts of distributed solution architects who are coordinated and self-organised to that end.

A minimal central team is employed to

·       provide cross-organisational oversight

·       encourage architect to cooperate with EA aims in mind.

·       socialise architects into a group that cooperates by aligning and integrating solutions

·       give architects some time and opportunity to do strategic EA level analysis and planning

·       encourage them to do what they can, when and where they can, to move the enterprise towards an EA vision.

 

E.g. The organisation in the table below has no enterprise architect, and only four solution architects.

But they collaborate with an EA mind set when the opportunity arises.

 

A university

total 

c75 people in business system change

total 

c165 people in IT services management

 total

Campus data centres

2

Solution architects (TDAs with EA mind set)

4

Groupware / mailing list

3

Schools

26

Supplier relationship managers (licences)

2

Senior management

5

Buildings (25m)

300

Data integration specialists

5

Data centre / hardware / operations

6

Staff

9000

Application development team

13

File store / directory / identity management

7

Students

31000

User engagement and requirements specialists

24

Service desk

9

 

PMO (programme/project managers, back office)

27

Client Devices, Desktop apps, OS and printers

10

Clerical administration

10

IS/IT projects per year

150

OS / databases / networks

30

Business applications

600

 

 

Hardware – desk tops, projectors etc.

85

 

There being no EA function, the principal solution architect sent two others, not called “architect”, to attend our architect training.

An education in enterprise and solution architecture can be relevant and useful to people with no EA function.

Moving from guerrilla EA to centralised EA

Setting up a central EA function without a solid foundation is likely to fail.

 

First you probably need:

·       your change/project initiation and management processes to include approval of a solution architecture definition (SAD) before the project starts

·       those processes to be adhered to

·       a team or pool of solution architects who can (with SME assistance) produce SADs and help managers to shape and steer projects

·       an inspiring, or at least good, leader for that team/pool

·       positive feedback from the team’s efforts.

 

Then you probably need:

·       to identify some strategic and cross-organisational problems that adding an EA function on top of that team will solve

·       to convince some sponsors to back the EA function and create a cross-organisational architecture board with clout.

·       willing and active business stakeholders

·       a business change function to address changes to people's roles.

 

And some good fortune in your initial endeavours.

For more

Read the papers on the Agile Architecture page at http://avancier.website

1.     On the beginnings of agile

2.     On agile software development

3.     On agile businesses and systems

4.     On systems thinking ideas used in agile

5.     What is agile architecture?

6.     EA in the world of agile architecture

 

Also

1.     Commentary on “Agile Architecture in the Digital Age”

2.     Commentary on “SAFe”

 

You’ll find more here – though it seems more for large-scale solution/system architecture than EA.

DSDM for portfolio management http://www.agilebusiness.org

Scaled Agile Framework (SAFE) http://www.scaledagileframework.com

For large integrated solutions that require hundreds of people to develop and maintain

Spotify-engineering-culture  https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/

People organisation: Squads, Tribes, Chains, Guilds.

Product organisation: Clients, Features, Infrastructure.