Agile 6 - EA in the world of agile architecture

https://bit.ly/2UPrjOC

Copyright Graham Berrisford. One of several hundred papers at http://avancier.website. Last updated 03/07/2019 17:42

Avancier’s agile papers

This is the sixth in a series of mostly short papers.

1.     On the beginnings of agile

2.     On agile software development

3.     On agile businesses and systems

4.     On systems thinking ideas used in agile

5.     What is agile architecture?

6.     EA in the world of agile architecture

For a slide show that conveys some of the above email me here.

Preface

Enterprise Architecture (EA) is a product of the Information Age.

Enterprise architects don’t design buildings or other concrete things.

Their concern is activity systems in which processes create and use data.

The data represents entities and events of importance to the business.

Enterprise architects don’t recruit, motivate and manage individual human actors.

Their concern is the roles those actors play in creating and using data.

The focus is on roles and processes that can – to some extent - be digitised using information technology.

 

Software development teams contain bright people who can do more than programming and testing.

In the agile development world, they do analysis and project/product management.

In the dev/ops world, they deploy applications into the production environment, and manage that.

In the agile architecture world, architecture is said to “emerge” from modularisation of a software system.

 

EA emerged in the 1980s as a reaction to the results that emerged from silo business system development.

Scores or hundreds of local system development projects had produced a sprawling wasteful mess.

There were duplications and disintegrities between systems, and difficulties with data quality and data analysis.

IT departments concluded they needed more enterprise-wide design thinking about business data and processes.

 

This article is about EA in the world described in recent papers on "agile architecture”.

Agile means willing and able to speedily respond to change; what architecture means is less clear.

What does architecture mean?

For some software developers, the architecture of a system is found in the code itself.

For enterprise and solution architects, architecture is higher level design-time description - rather than code.

It is an architecture definition, written in natural language or near to it, that communicates the intent of the architect.

 

Either way, if the architecture changes much or often, you’ve got a problem.

To use a metaphor: the architecture of a concrete building should allow for the fittings to be changed.

But to change the architecture might well mean discarding the fittings and re-building from scratch.

What does agile architecture mean?

This section summarises the discussion in Agile paper 5.

 

1. Intentional architecture

This is an up-front architecture drives the development of a system.

It may be revised iteratively and/or expand incrementally.

“One of the more persistent myths of agile development is that up-front architecture and design are bad …” Bob Martin

 

2 Emergent architecture

This architecture is documented after a system has been built or changed.

This turns the architect from a designer to a recorder.

Of course, this is contrary to the classical notion that an architect draws diagrams that builders follow.

3 Lean (high-level, light) architecture

This need not be changed because it is so abstract.

The trouble is, the more abstract it is, the less helpful it is (which is why EA needs solution architects.)

 

4 The architecture of an agile system

This need not be changed because it defines a flexible or configurable system.

The difficulty: the complexity/flexibility trade off.

E.g design patterns for extreme scalability tend to increase complexity - contrary to KISS & YAGNI

 

Agile system principle

Agile development principle

Decouple subsystems

contrary to

Keep the system simple (KISS)

Design for future flexibility

contrary to

You ain’t gonna need it (YAGNI)

 

5 Architecture that facilitates agile development

Scaling up from one team to a wider organisation.

In recent years, a primary concern of “agile architecture”.

 

Principles to scale up agile development

·       modularise the architecture into subsystems, assignable to (relatively) autonomous development teams.

·       decouple subsystems, so teams can work (relatively) autonomously

·       release a “minimum viable product”, followed by short-cycle releases that extend the system’s capability

·       use agile development methods and practices, such as SCRUM and Kanban.

 

All of this must be subordinate to the need to support and enable whatever roles and processes feature in the request for architecture work.

 

Agilists tend to deprecate what they see as current EA practices.

There is indeed some EA bad practice out there; so what is good EA?

Scaling up to the wider enterprise

To scale up agile development, one has to consider the following.

 

What kind of enterprise?

Different kinds of businesses change at different rates.

The likes of Facebook, Amazon, Netflix, Google and Spotify (FANGS) are peculiarly software-centred businesses.

Changing web sites and algorithms is one thing.

Changing physical assets and workers with special abilities, knowledge or skills is another.

See Agile paper 3 for discussion of FANGS.

 

What kind of scalability?

One might count the number of:

·       entity and event types that are monitored and directed by a system (variety or complexity).

·       entity and event instances a system must process (throughput)

·       people in the system development organisation (human power).

 

“Scaling up agile” usually relates primarily to the last.

It means scaling up to the level of an enterprise with many (even hundreds) teams working in parallel.

 

Longer term planning

EA typically works to a strategic horizon.

EA methods are traditionally for planning major changes business systems, to a target that is one, two or more years ahead.

Due the time scale, initially-stated requirements are often unclear, inaccurate, incomplete – and change later.

 

Agile development methods are about what a team does to continually improve and extend a software system, with release cycle of days or weeks.

The outcome should be faster delivery of something useful, though it may not do everything customers want.

Development teams typically work iteratively in sprints of 10 to 30 days.

The backlog (e.g.an epic list) may be up to two to three iterations ahead, say 3 months

A challenge for EA is collaborate with each team - at the level of their backlog.

 

Higher level agility

Development teams are agile within a narrow scope, at low level of design

A challenge for EA is to be agile at a wider scope, and a higher level.

 

Wider responsibility

In Scrum, the product owner is expected to assume responsibility for the full product life-cycle, including the business case.

A challenge for EA is to ensure this is coordinated with other business cases.

 

Synchronizing deliverables

In agile architecture frameworks, development teams are supposed to be autonomous and self-organising.

A challenge for EA is coordinate tens to hundreds of development teams, where necessary.

For example, it may be necessary to synchronise subsystems in one release of a wider system.

Eleven principles of good EA – in the age of agile architecture

 

Is the vision of EA achievable?

EA was a reaction against difficulties caused by silo systems not interacting as they should do.

·       duplications, 

·       disintegrities,

·       delays and

·       difficulties with cross-organisational data analysis.

 

However, EA is difficult politically and technically.

It can inhibit the agile system development an enterprise also needs to respond to changing conditions.

 

Generally, agile means willing and able to respond to change. It can apply to:

·       Business actors – working business operations

·       Software developers – delivering working software

·       Architects – delivering architecture definitions

·       Anybody working in a “self-organising system”

 

Inevitably, architecture does constrain agility in some ways

·       In business operations… business actors can change how they perform business operations -  within the constraints of the software systems they use.

·       In the design and build of systems… designers and builders can change systems they release to business actors - within the constraints of the architecture given to them.

·       In architecture… architects can change the architecture they release to system developers - within the constraints of the business context.

 

1 Start by understanding the specific business context

Good EA always starts from the particular business context.

What is your business?

·       Data processing? Bank, insurance, tax, Ecommerce, Application-centric internet giant (FANGS, as discussed in Agile paper 3).

·       Material processing? Manufacturing, retail, logistics etc.

·       Delivery of human skills? Entertainment, security, prison, education etc.

 

Clearly, data processing businesses are candidates for digitisation, even digital transformation.

The white paper discussed in Agile paper 7 is based on lessons learned in such “digital enterprises”.

The exemplars are application-centric internet giants like Facebook, Amazon, Netflix, Google and Spotify (FANGS).

Many programmers think it cool to copy what FANGS do - regardless of relevance, cost and complexity.

But most businesses are not like FANGS.

And design patterns that FANGS have developed for extreme scale web applications (see below) come with costs and downsides.

 

2 Look to strike the optimal balance

EA and agile software development are opposites in some ways.

Taken to the extremes, both lead to difficulties.

Both are used in a “bimodal” IT department; they can, should and must collaborate.

Architects have to understand the extremes and strike a balance between

·       up-front intentional architecture

·       incremental development and emergent architecture

 

3 At every level, system design is a creative and challenging process

EA and agile development teams define regular business systems in which regular roles and processes create and use business data.

(In systems theory terms, this is the territory of classical cybernetics.)

By contrast, the teams themselves must be relatively “self-organising” social networks.

They must be flexible, responsive to interaction with stakeholders, responsive to requirements change.

Team members must think in creative ways, consider tradeoffs, and make decisions without all the information they want.

 

4 Delegate to competent architects/designers

Good EA looks to make coarse-grained architecture decisions, leaving room for others to make fine-grained decisions.

It operates at the level of an enterprise’s long running business processes and coarse-grained application portfolio.

It likely delegates project-level architecture definition to people playing solution architect roles (read EA v SA)

It encourages agile software development within an application project/product.

It recognises that where agile practices are not a good fit, software development usually takes longer and costs more.

 

5 Recognise tradeoffs between design patterns

Good EA knows there is no silver bullet or one-size-fits-all method or solution.

Different design patterns fit different problems and requirements.

In architecture and design (as our architect training courses emphasise) all is trade offs

There are tradeoffs, for example between design for simplicity and for flexibility.

EA strives to minimise wasteful duplications between systems, and to integrate systems that work on the same data.

At the same time, it recognises that a large and complex system should be modularised for manageability.

 

6 Decouple subsystems to meet a defined need, rather than by default

Good EA recognises Constantine’s principles of modularity (1968).

·       high cohesion within a subsystem

·       loose coupling between subsystems.

 

OK, but these principles apply differently at coarser and finer-grained levels of granularity.

Physical decoupling does not remove business requirements for logical coupling.

It can lead to inter-system delays, disintegrities and difficulties analysing business-wide data.

And as Craig Larman points out, decoupling subsystems where there is no clear need is not time will spend.

Inevitably, division into smaller simpler subsystems increases the complexity of inter-subsystem messaging.

This is great for the middleware vendor, but while pub-sub middleware and event-driven messaging are useful, they are not best used everywhere.

 

7 Recognise Conway’s law

Good EA recognises Conway’s law (also 1968), which says there is a natural correspondence between:

·       the modularisation of a software system into subsystems

·       the modularisation of a human organisation into development teams.

 

This is crystalized in the Bezos mandate.

 

8 Design subsystems with higher level processes in mind

Good EA recognises that large and complex system should be modularised into subsystems that are relatively autonomous.

OK, but scatter-brained silo system (or microservice) development leads to chaos.

Remember to optimise the end-to-end business processes.

To do this by coordinating scores or hundreds of subsystems/teams does require some oversight.

 

Good EA is concerned to ensure that subsystems support higher level processes and maintain consistency of data.

Fragmenting applications and delegating decisions may lead to diversification and duplication, the very issues EA was created to reduce.

It can fragment data stores, making it harder to maintain data integrity and do cross-system data analysis.

Mitigating these risk, inevitably, involves review project milestones

Typically, solution architects must complete “solution outline” documents and get them approved.

 

9 Accommodate short term needs

Different business system changes have different time and cost profiles.

A business strategy can trigger a long-term transformation project that last years and cost millions.

An architect in a global business told me of a project completed in 2016 that cost $200 million.

That same business has to respond to quickly to immediate problems and pressures.

Sometimes, managers want a new business system at three months notice; or even faster.

What if managers insist they need a point solution tomorrow?

Good EA can give waiver for an innovative and/or agile project to proceed outside of normal governance.

(Beware without architectural oversight, both buying packages and agile software development can create disintegrated and/or un-maintainable systems.)

 

10 Employ good architects

An architecture method is designed to ensure there are agreed objectives, models of baseline and target systems and governance of system delivery.

There is a down side - the necessary bureaucracy.

Pro-forma documentation is problematic

An EA framework can waste your time and money if you wrongly read it as a prescription for what to do.

It can help if you know what you are doing and use it as a menu of products and processes.

(Wrt a coherent and consistent enterprise-wide repository, it seems the jury is out on what of that kind is optimally practical and useful.)

Employ well-trained experienced and knowledgeable people at all levels.

And expect them to share their knowledge.

 

11 Guerilla and Collective EA

If you cannot afford an EA function, consider Guerilla EA

For more on that read on.

What is the role of a central EA team?

EA provides business change initiatives with a measure of centralised oversight and leadership

It aims to support, enable and coordinate business actors in completing end-to-end business processes that yield results of the value to the business.

And to integrate systems so as to ensure data quality and enable cross-system data analysis.

 

The scope of an EA team varies from enterprise-wide to one business segment, capability or function. 

Its members may include enterprise and solution architects and specialist domain experts.

Some or most of its members may be assigned to change projects, full time or part time

 

EA teams often model a whole baseline business at a high level, but rarely (if ever) look to change the whole.

EA is not top-down command and control; it responds to diverse requests for work from sponsors.

It may initiate some changes; but more often steers changes requested by others.

It looks at each change initiative as a distinct programme or project.

 

EA does not manage change projects; it shapes and steers them

It steers or governs them with business goals in mind

It governs them optimise the wider enterprise – minimising waste by standardisation and integration.

It governs then against whatever enterprise-wide or strategic road maps, standards, principles are in place.

 

EA does not prevent prototyping or innovation.

It encourages agile software development, and presumes responsiveness to requirements change.

But EA does presume a judicious amount of design up front!

 

None of this is easy; it can be more political than technological.

To use an analogy given to me by a colleague: is EA like piloting an oil tanker or herding cats?

An oil tanker’s journey must be carefully planned because it cannot easily change direction.

By contrast, herding cats requires continual adaptation to an ever-changing situation.

Both approaches have some utility, but most of what we need to do in practice lies between the extremes.

Guerilla EA

EA is political, and there are natural cycles in the status of EA in organisations.

Much hangs on the personality and persuasiveness of those at the centre, and what some call the "culture" of the enterprise.

What about small-to-medium enterprises - who can't afford a substantial central team?

 

Most businesses still need solution architects to shape and steer projects.

And they need some central thinking about  SFIA calls "Strategy and Architecture".

This central team, function, forum or person may be responsible for supplying solution architects to projects.

 

EA can be about making small steps towards the vision of digitised, standardised, integrated business processes and data.

“Guerrilla EA” does what it can, when and where it can, to these ends.

It is the practice of advancing EA through the efforts of distributed solution architects who are coordinated and self-organised to that end.

A minimal central team is employed to

·       provide cross-organisational oversight

·       encourage architect to cooperate with EA aims in mind.

·       socialise architects into a group that cooperates by aligning and integrating solutions

·       give architects some time and opportunity to do strategic EA level analysis and planning

·       encourage them to do what they can, when and where they can, to move the enterprise towards an EA vision.

 

E.g. The organisation in the table below has no enterprise architect, and only four solution architects.

But they collaborate with an EA mind set when the opportunity arises.

 

A university

total 

c75 people in business system change

total 

c165 people in IT services management

 total

Campus data centres

2

Solution architects (TDAs with EA mind set)

4

Groupware / mailing list

3

Schools

26

Supplier relationship managers (licences)

2

Senior management

5

Buildings (25m)

300

Data integration specialists

5

Data centre / hardware / operations

6

Staff

9000

Application development team

13

File store / directory / identity management

7

Students

31000

User engagement and requirements specialists

24

Service desk

9

 

PMO (programme/project managers, back office)

27

Client Devices, Desktop apps, OS and printers

10

Clerical administration

10

IS/IT projects per year

150

OS / databases / networks

30

Business applications

600

 

 

Hardware – desk tops, projectors etc.

85

 

There being no EA function, the principal solution architect sent two others, not called “architect”, to attend our architect training.

An education in enterprise and solution architecture can be relevant and useful to people with no EA function.

Moving from guerrilla EA to centralised EA

Setting up a central EA function without a solid foundation is likely to fail.

 

First you probably need:

·       your change/project initiation and management processes to include approval of a solution architecture definition (SAD) before the project starts

·       those processes to be adhered to

·       a team or pool of solution architects who can (with SME assistance) produce SADs and help managers to shape and steer projects

·       an inspiring, or at least good, leader for that team/pool

·       positive feedback from the team’s efforts.

 

Then you probably need:

·       to identify some strategic and cross-organisational problems that adding an EA function on top of that team will solve

·       to convince some sponsors to back the EA function and create a cross-organisational architecture board with clout.

·       willing and active business stakeholders

·       a business change function to address changes to people's roles.

 

And some good fortune in your initial endeavours.

Further agile papers

7.     Commentary on “Agile Architecture in the Digital Age”

8.     Commentary on “SAFe