Agile 6 - EA in the world of agile architecture

https://bit.ly/2UPrjOC

Copyright Graham Berrisford. One of several hundred papers at http://avancier.website. Last updated 04/05/2019 12:52

Avancier’s agile papers

This is the sixth in a series of mostly short papers.

1.      On the beginnings of agile

2.      On agile software development

3.      On agile businesses and systems

4.      On systems thinking ideas used in agile

5.      What is agile architecture?

6.      EA in the world of agile architecture

 

Software development teams contain bright people who can do more than programming and testing.

In the agile development world, they do analysis and project/product management.

In the dev/ops world, they deploy applications into the production environment, and manage that.

In the agile architecture world, architecture is said to “emerge” from modularisation of a software system.

 

Enterprise architecture (EA) emerged in the 1980s as a reaction to the results that emerged from silo business system development.

Scores or hundreds of local system development projects had produced a sprawling wasteful mess.

There were duplications and disintegrities between systems, and difficulties with data quality and data analysis.

IT departments concluded they needed more enterprise-wide design thinking about business data and processes.

 

This article is about EA in the world described in recent papers on "agile architecture”.

What does agile architecture mean?

In different contexts, the term can have any of seven meanings discussed in Agile paper 5.

1.      Intentional (up front) architecture that is revised in response to changing requirements

2.      Intentional (up front) architecture that expands incrementally to drive incremental software development

3.      Emergent (post hoc) architecture

4.      High-level (light or lean) architecture

5.      Architecture that enables scaling up systems to handle very high event throughput

6.      Architecture that facilitates scaling up agile development from a team to the wider system or enterprise.

7.      The architecture of an agile (flexible or configurable) system

 

The primary meaning here is the 6th in the list.

·         An architecture that facilitates scaling up agile development from a team to the wider system or enterprise.

·         A particular (ideally stable) high level design that facilitates agile software development.

 

Agile architecture frameworks typically assume

·         modularising a large system into subsystems each assignable to a (relatively) autonomous development team.

·         decoupling subsystems so teams can work (relatively) autonomously

·         using design patterns used also in extreme scale systems

·         each team can release a “minimum viable product” followed by short-cycle releases that extend their subsystem’s capability

·         each team uses agile development methods and practices such as SCRUM and Kanban.

 

Agilists tend to deprecate what they see as current EA practices.

There is indeed some EA bad practice out there; so what is good EA?

Scaling up to the wider enterprise

To scale up agile development, one has to consider the following.

 

What kind of enterprise?

Different kinds of businesses change at different rates.

The likes of Facebook, Amazon, Netflix, Google and Spotify (FANGS) are peculiarly software-centred businesses.

Changing web sites and algorithms is one thing.

Changing physical assets and workers with special abilities, knowledge or skills is another.

See Agile paper 3 for discussion of FANGS.

 

What kind of scalability?

One might count the number of:

·         entity and event types that are monitored and directed by a system (variety or complexity).

·         entity and event instances a system must process (throughput)

·         people in the system development organisation (human power).

 

“Scaling up agile” usually relates primarily to the last.

It means scaling up to the level of an enterprise with many (even hundreds) teams working in parallel.

 

Longer term planning

EA typically works to a strategic horizon.

EA methods are traditionally for planning major changes business systems, to a target that is one, two or more years ahead.

Due the time scale, initially-stated requirements are often unclear, inaccurate, incomplete – and change later.

 

Agile development methods are about what a team does to continually improve and extend a software system, with release cycle of days or weeks.

The outcome should be faster delivery of something useful, though it may not do everything customers want.

Development teams typically work iteratively in sprints of 10 to 30 days.

The backlog (e.g.an epic list) may be up to two to three iterations ahead, say 3 months

A challenge for EA is collaborate with each team - at the level of their backlog.

 

Higher level agility

Development teams are agile within a narrow scope, at low level of design

A challenge for EA is to be agile at a wider scope, and a higher level.

 

Wider responsibility

In Scrum, the product owner is expected to assume responsibility for the full product life-cycle, including the business case.

A challenge for EA is to ensure this is coordinated with other business cases.

 

Synchronizing deliverables

In agile architecture frameworks, development teams are supposed to be autonomous and self-organising.

A challenge for EA is coordinate tens to hundreds of development teams, where necessary.

For example, it may be necessary to synchronise subsystems in one release of a wider system.

Ten principles of good EA

 

Start by understanding the specific business context

Good EA always starts from the particular business context.

The white paper discussed in Agile paper 7 is based on lessons learned in very peculiar “digital enterprises”.

The exemplars are application-centric internet giants like Facebook, Amazon, Netflix, Google and Spotify (FANGS, as discussed in Agile paper 3).

Many programmers think it cool to copy what FANGS do - regardless of relevance, cost and complexity.

But most businesses are not like FANGS.

And design patterns that FANGS have developed for extreme scale web applications (see below) come with costs and downsides.

 

Look to strike the optimal balance

At their most extreme, EA and agile software development might be seen as opposites.

However, both approaches are used in a “bimodal” IT department.

They can, should and must collaborate.

 

Taken to the extremes, both EA bureaucracy and agile development lead to difficulties.

Architects have to understand the extremes and strike a balance between them.

They have to strike a balance between up-front design and incremental development.

 

At every level, system design is a creative and challenging process

Both EA and agile software development teams design business systems.

They design deterministic systems in which regular processes create and use business data.

(In systems theory terms, this is the territory of classical cybernetics.)

 

By contrast, both teams are self-organising in the sense they define their own roles and processes.

Their processes are flexible and iterative, driven by continual interaction with stakeholders and requirements change.

Team members must think in creative ways, consider tradeoffs, and make decisions without all the information they want.

 

Delegate what can be delegated to competent architects/designers

Good EA looks to make coarse-grained architecture decisions, leaving room for others to make fine-grained decisions.

It operates at the level of an enterprise’s long running business processes and coarse-grained application portfolio.

It likely delegates project-level architecture definition to people playing solution architect roles (read EA v SA)

It encourages agile software development within an application project/product.

It recognises that where agile practices are not a good fit, software development usually takes longer and costs more.

 

Recognise tradeoffs between design patterns

Good EA knows there is no silver bullet or one-size-fits-all method or solution.

Different design patterns fit different problems and requirements.

In architecture and design (as our architect training courses emphasise) all is trade offs

There are tradeoffs, for example between design for simplicity and for flexibility.

EA strives to minimise wasteful duplications between systems, and to integrate systems that work on the same data.

At the same time, it recognises that a large and complex system should be modularised for manageability.

 

Decouple subsystems to meet a defined need, rather than by default

Good EA recognises Constantine’s principles of modularity (1968).

That is, there should be high cohesion within a subsystem and loose coupling between subsystems.

But it also recognises these principles apply differently at coarser and finer-grained levels of granularity.

And that physical decoupling does not remove business requirements for logical coupling.

Physical decoupling can lead to inter-system delays, disintegrities and difficulties analysing business-wide data.

And as Craig Larman points out, decoupling subsystems where there is no clear need is not time will spend.

 

Recognise Conway’s law

Good EA recognises Conway’s law (also 1968), which says there is a natural correspondence between:

·         the modularisation of a software system into subsystems

·         the modularisation of a human organisation into development teams.

 

Design subsystems with higher level processes in mind

Good EA recognises that to coordinate scores or hundreds of subsystems/teams does require some oversight.

It encourages the modularisation of a large and complex system into subsystems that are relatively autonomous.

But it discourages scatter-brained silo system (or microservice) development.

 

Good EA is concerned to ensure that subsystems support higher level processes and maintain consistency of data.

Fragmenting applications and delegating decisions may lead to diversification and duplication, the very issues EA was created to reduce.

It can fragment data stores, making it harder to maintain data integrity and do cross-system data analysis.

Inevitably, division into smaller simpler subsystems increases the complexity of inter-subsystem messaging - great for the middleware vendor.

While pub-sub middleware and event-driven messaging are useful, they are not best used everywhere.

Mitigating these risk, inevitably, involves review project milestones

Typically, solution architects must complete “solution outline” documents and get them approved.

 

Accommodate short term needs

Different business system changes have different time and cost profiles.

A business strategy can trigger a long-term transformation project that last years and cost millions.

An architect in a global business told me of a project completed in 2016 that cost $200 million.

That same business has to respond to quickly to immediate problems and pressures.

Sometimes, managers want a new business system at three months notice; or even faster.

What if managers insist they need a point solution tomorrow?

Good EA can give waiver for an innovative and/or agile project to proceed outside of normal governance.

(Beware without architectural oversight, both buying packages and agile software development can create disintegrated and/or un-maintainable systems.)

 

Employ good architects

An architecture method is designed to ensure there are agreed objectives, models of baseline and target systems, and governance of system delivery.

There is a down side - the necessary bureaucracy – to create documentation and analyse the impacts of change requests against that documentation.

Of course, untrained/unthinking completion of document templates and untrained/unthinking approval of the resulting documents is ineffective.

The problem is not the document templates; it is ill-educated, ill-informed document completion; and naivety in document reviewers.

Good EA employs well-trained experienced and knowledgeable people at all levels.

And expects them to share their knowledge.

What is the role of a central EA team?

EA provides business change initiatives with a measure of centralised oversight and leadership

It aims to support, enable and coordinate business actors in completing end-to-end business processes that yield results of the value to the business.

And to integrate systems so as to ensure data quality and enable cross-system data analysis.

 

The scope of an EA team varies from enterprise-wide to one business segment, capability or function. 

Its members may include enterprise and solution architects and specialist domain experts.

Some or most of its members may be assigned to change projects, full time or part time

 

EA teams often model a whole baseline business at a high level, but rarely (if ever) look to change the whole.

EA is not top-down command and control; it responds to diverse requests for work from sponsors.

It may initiate some changes; but more often steers changes requested by others.

It looks at each change initiative as a distinct programme or project.

 

EA does not manage change projects; it shapes and steers them

It steers or governs them with business goals in mind

It governs them optimise the wider enterprise – minimising waste by standardisation and integration.

It governs then against whatever enterprise-wide or strategic road maps, standards, principles are in place.

 

EA does not prevent prototyping or innovation.

It encourages agile software development, and presumes responsiveness to requirements change.

But EA does presume a judicious amount of design up front!

 

None of this is easy; it can be more political than technological.

To use an analogy given to me by a colleague: is EA like piloting an oil tanker or herding cats?

An oil tanker’s journey must be carefully planned because it cannot easily change direction.

By contrast, herding cats requires continual adaptation to an ever-changing situation.

Both approaches have some utility, but most of what we need to do in practice lies between the extremes.

Guerilla EA

EA is political, and there are natural cycles in the status of EA in organisations.

Much hangs on the personality and persuasiveness of those at the centre, and what some call the "culture" of the enterprise.

What about small-to-medium enterprises - who can't afford a substantial central team?

 

Most businesses still need solution architects to shape and steer projects.

And they need some central thinking about  SFIA calls "Strategy and Architecture".

This central team, function, forum or person may be responsible for supplying solution architects to projects.

 

Guerilla EA is the practice of advancing EA through the efforts of distributed solution architects who are coordinated and self-organised to that end.

A minimal central team is employed to

·         provide cross-organisational oversight

·         encourage architect to cooperate with EA aims in mind.

·         socialise architects into a group that cooperates by aligning and integrating solutions

·         give architects some time and opportunity to do strategic EA level analysis and planning

·         encourage them to do what they can, when and where they can, to move the enterprise towards an EA vision.

 

E.g. The organisation in the table below has no enterprise architect, and only four solution architects.

But they collaborate with an EA mind set when the opportunity arises.

 

A university

total 

c75 people in business system change

total 

c165 people in IT services management

 total

Campus data centres

2

Solution architects (TDAs with EA mind set)

4

Groupware / mailing list

3

Schools

26

Supplier relationship managers (licences)

2

Senior management

5

Buildings (25m)

300

Data integration specialists

5

Data centre / hardware / operations

6

Staff

9000

Application development team

13

File store / directory / identity management

7

Students

31000

User engagement and requirements specialists

24

Service desk

9

 

PMO (programme/project managers, back office)

27

Client Devices, Desktop apps, OS and printers

10

Clerical administration

10

IS/IT projects per year

150

OS / databases / networks

30

Business applications

600

 

 

Hardware – desk tops, projectors etc.

85

 

There being no EA function, the principal solution architect sent two others, not called “architect”, to attend our architect training.

An education in enterprise and solution architecture can be relevant and useful to people with no EA function.

Moving from guerrilla EA to centralised EA

Setting up a central EA function without a solid foundation is likely to fail.

 

First you probably need:

·         your change/project initiation and management processes to include approval of a solution architecture definition (SAD) before the project starts

·         those processes to be adhered to

·         a team or pool of solution architects who can (with SME assistance) produce SADs and help managers to shape and steer projects

·         an inspiring, or at least good, leader for that team/pool

·         positive feedback from the team’s efforts.

 

Then you probably need:

·         to identify some strategic and cross-organisational problems that adding an EA function on top of that team will solve

·         to convince some sponsors to back the EA function and create a cross-organisational architecture board with clout.

·         willing and active business stakeholders

·         a business change function to address changes to people's roles.

 

And some good fortune in your initial endeavours.

Further agile papers

1.      Commentary on “Agile Architecture in the Digital Age”

2.      Commentary on “SAFe