Enterprise and Solution Architecture challenges

https://bit.ly/2OlDn6j

Copyright Graham Berrisford. One of several hundred papers at http://avancier.website. Last updated 05/11/2018 18:16

 

The paper is the second of two that positions mainstream enterprise architecture in relation to solution architecture.

It is written to reflect the history of architecture roles and the reality of modern architect job descriptions.

It is suggested as pre-course reading for Enterprise and Solution Architecture (ESA) certification courses in this Training Calendar.

 

·         RK “All in one article...” [now two articles]

·         VE “Thanks for sharing.”

·         SW “This is good reading. Thank you for sharing.”

·         TRB “Good read.”

·         SG “Very much insightful. Thanks a ton.”

Contents

Preface. 1

Governance of solution architects by enterprise architects. 2

Populating your Strategy and Architecture team.. 3

Guerilla EA.. 4

Architecture methodology. 5

Agility in its various guises. 7

References. 10

 

Preface

It is presumed here that you are familiar with context discussed in this previous paper:

·         Enterprise and solution architecture

·         Architect roles in SFIA

·         Architect roles in the TOGAF standard

·         Architect roles in advertised role definitions

·         Decades of solution architecture

·         Decades of point solution proliferation

·         Decades of EA as business system planning

 

EA has always been challenging, both technically and politically.

Decades ago, US OBM Memoranda 97-16 guidance on enterprise architecture said this:

the balance between centralization and decentralization and the pace of change within the agency, should be clearly understood.”

 

The balance between centralisation and decentralization?

What if nobody sees a need for current “silo systems” to interact?

Suppose the only business-wide view required is financial summary information collected from disparate business units?

A highly diversified business presents a challenge to the notion that the enterprise is a holistic system.

 

Still, a Strategy and Architecture team has a role to play in governance of solution architects working on discrete business systems.

Suppose your enterprises wishes to standardise, integrate and optimize systems to the benefit to the enterprise as a whole, and mitigate change risks.

Then it needs standards, principles, reference models, best practices and road maps for business system change.

And it ought to govern solution architects with reference to that collateral.

 

The pace of change?

What if managers insist a point solution is needed tomorrow?

Enterprise architects can give waiver for an innovative and/or agile project to proceed outside of normal governance.

Agile development and agile systems (two different things) are discussed below.

Read on for discussion of:

 

·         Governance of solution architects by enterprise architects  2

·         Populating your Strategy and Architecture team   3

·         Guerilla EA   4

·         Architecture methodology  5

·         Agility in its various guises  7

Governance of solution architects by enterprise architects

A solution architect’s vision usually has a relatively short time frame and narrow scope.

In the 1980s, the need to take more strategic and cross-organisational view of business processes and business data became clear.

The cost and quality issues caused by “silo systems” led many to call for higher-level “enterprise architecture”.

Enterprise architects take the broadest possible view, look to standardise, integrate and coordinate business systems across an enterprise.

They respond to the outputs of business planning, align system changes with them, and may influence them.

 

EA is challenging politically as well as technically.

The idea is to take a cross-organisational and strategic perspective of business systems and changes to them. 

Enterprise architects need not only appropriate knowledge, skills and resources, but also cross-organisational authority.

They act as a central design authority and risk management function.

They guide and govern lower level solution and technical architects working on specific programmes and projects.

 

All is trade offs

The aims of EA include the de-duplication and integration of many systems - the rationalisation and optimisation of business system estates.

 

Point solutions (innovations or not) are contrary to a holistic enterprise architecture.

But enterprise architects may sanction point solutions.

Architects should assess trade offs and make a decision, rather than allow point solutions to proliferate unconstrained.

 

Similarly, duplication of data storage by different applications is contrary to holistic enterprise architecture.

But architecture governance may allow some duplicate data to continue, or even to be created.

 

EAs may choose, or help to choose, generic and enterprise-wide platform technologies.

But the imposition by some EAs of middleware technologies on SAs might well have cost more than it saved.

Populating your Strategy and Architecture team

The number of roles called “architect” varies widely between organisations.

Some enterprises have never set up an EA practice; some have done so and struggled to make an impact.

However, many do have a team called “strategy and architecture” or the like.

Every enterprise has to work out for itself how its roles cover the cells of the table below.

 

Domain

Level

Business

Data/Information

Applications

Infrastructure technology

Enterprise

architecture

Business roles & processes,

standardisation, integration

and road maps

Business data

standardisation, integration

and road maps

Business application portfolio

standardisation, integration

and road maps

Platform technology portfolio

standardisation, integration

and road maps

Solution

architecture

Outline design of a solution’s

required processes

Outline design of a solution’s

data architecture

Outline design of a solution’s

application(s) architecture

Outline design of a solution’s

IT production environment

Detailed

design

Detailed design of

processes & use cases

Detailed design of

data stores and flows

Detailed design of

software architecture

Detailed design of

IT production environment

 

First, there must be a manager (CIO or other) who appreciates and wants to sponsor enterprise and/or solution architect roles.

That manager may create a Strategy and Architecture team – covering some or all of the table above.

How the activities in the table are mapped to roles and people depends on the manager, and the size and nature of the enterprise.

And by the way, it is normal to engage security specialists in each architecture domain.

 

Division of roles by scope

A large and diverse enterprise may be divided into “segments”, each with its own Strategy and Architecture team.

The segments likely correspond to distinct business functions/capabilities.

E.g. a bank might have retail banking, wealth management and investment segments.

 

Division of roles by level

A Strategy and Architecture team may divide its roles according the three rows of the table above.

So there is a hierarchy of enterprise, solution and software/technical specialist architects

Some insert a “segment” level between the enterprise and solution levels.

Where an enterprise is happy with silo solutions that duplicate and don’t integrate – the focus may be on solution architecture in the second row of the table.

 

Division of roles by domain

A Strategy and Architecture team may divide its roles according the columns of the table above.

“The enterprise architect" is rare, since it is difficult for one person to manage portfolios in all four domains.

A Strategy and Architecture team might employ an enterprise architect in charge of each architecture domain (the columns of the table).

By contrast, a solution architect is often expected span all domains, and join up the perspectives of specialists working in each domain on a project.

This makes broadly educated solution architects the lynch pin of an architecture function.

Guerilla EA

What if there is no “enterprise architect” in the organisation?

There is often some kind “Strategy and Architecture” team that supplies solution architects to projects.

A CIO, CTO or architecture team manager can

·         ask solution architects to work with an EA mindset.

·         socialise architects into a group that cooperates by aligning and integrating solutions

·         give architects some time and opportunity to do strategic EA level analysis and planning

·         encourage them to do what they can, when and where they can, to move the enterprise towards an EA vision.

 

Here, this is called “Guerilla EA”.

E.g. The organisation in the table below has no enterprise architect, and only four solution architects.

But they collaborate with an EA mind set when the opportunity arises.

 

A university

total 

c75 people in business system change

total 

c165 people in IT services management

 total

Campus data centres

2

Solution architects (TDAs with EA mind set)

4

Groupware / mailing list

3

Schools

26

Supplier relationship managers (licences)

2

Senior management

5

Buildings (25m)

300

Data integration specialists

5

Data centre / hardware / operations

6

Staff

9000

Application development team

13

File store / directory / identity management

7

Students

31000

User engagement and requirements specialists

24

Service desk

9

 

PMO (programme/project managers, back office)

27

Client Devices, Desktop apps, OS and printers

10

Clerical administration

10

IS/IT projects per year

150

OS / databases / networks

30

Business applications

600

 

 

Hardware – desk tops, projectors etc.

85

 

There being no EA function, the principal solution architect sent two others, not called “architect”, to attend our architect training.

An education in enterprise and solution architecture can be relevant and useful to people with no EA function.

Architecture methodology

Architecture is high level design and planning of systems that must be designed, tested, integrated and replaced.

How to build a coherent architecture method that makes enterprise transformation quicker, cheaper, better or less risky?

One has to draw many ideas from many sources.

And then direct each method user to decide which ideas best suit their particular transformation challenge.

Design thinking

Design thinking is not a particular approach, but a label for a collection of ideas such as.

·         Capture the inspiration, the vision.

·         Take a human-centric view of business roles and processes.

·         Manage stakeholders and value propositions.

·         Treat all design as re-design, as a baseline to target transformation.

·         Make ideas tangible to facilitate communication.

·         Use visual languages, sketch diagrams and technical drawings to show abstract requirements may be met by concrete systems.

·         Design process steps can occur in parallel and be repeated (aren't linear)

·         Double loop learning.

 

These ideas have long and widely been used in conventional system design methods.

Most feature in most enterprise and solution architecture methods. E.g.

 

A design thinking process (Wikipedia)

TOGAF’s ADM process

Avancier Methods phases

1 Define

phase A

Initiate

2,3,4 Research, Ideate, Prototype

phases B, C, D

Architect

5 Choose

phase E

Plan

6 Implement

phases F and G

Govern development

7 Learn

phases G and H.

Govern change

 

Architecture methods encourage parallel and iterative paths both through the phases of its process and within each phase.

E.g. TOGAF says the timing and order of steps should be adapted to the situation.

True, it encourages some big-up-front design, but how much you do is up to you.

True, it does not encourage prototyping in phases B to D, but that is a perfectly reasonable thing to do.

The enterprise as a system

Some use the terms system, component, process and service interchangeably.

Some use them with different (and sometimes contradictory) meanings.

 

In some standards from The Open Group

In Fowler’s writings for programmers

A structural building block that offers multiple services (a service portfolio) to its clients

Component

A unit of software that is independently replaceable and upgradeable.

A behavior composed of process steps in a sequence that leads to a result.

Process

An instance of a component executed on a computer.

A discretely requestable behaviour that encapsulates one or more processes.

Service

An out-of-process component, typically reached by a web service request or RPC.

 

It is commonly said that “enterprise architecture views the enterprise as a system, or system of systems”.

But there are profound misunderstandings of what this means.

To call every problem, situation or business “a system” is unhelpful.

It is important to distinguish:

·         a social network in which people communicate

·         a social system in which people realise role and rules.

 

An enterprise is one social network that realises many social and technological systems.

Architects may find some systems overlap, duplicate or even conflict with each other.

Enterprise architects apply some change control to large-scale, generational, system change.

For more on these underpinning concepts, read papers 5 and 6 on this system theory page.

Management of system design

EA frameworks like TOGAF are not primarily design methods,

They are abstract management frameworks for design thinking.

TOGAF is intended to help people manage the time, cost and risk of work to change business systems.

It does not teach architects how to do architecture.

Solution architects need more concrete training in architectural design principles and patterns.

System testing

Architecture methods don’t include testing itself, but must define systems in a way that is testable.

Whether you use diagrams or  not, testable requirements are specified in the domain-specific language of the system to be tested.

Templates for specifying required system behaviors include user stories, use case descriptions, and service contracts (signatures and semantics).

All are variations on the theme of specifying system behaviours (services and processes) declaratively by their entry and exit conditions.

·         Entry conditions may be defined as triggers, inputs and preconditions.

·         Exit conditions may be defined as outputs, post conditions and the values of those to system users.

 

(Read The science and philosophy of systems for the underpinning theory of system specification.)

System integration

Architecture methods should include not only system definition, but also patterns for system integration.

Systems can be integrated by users, messaging or database consolidation, online or off line, and in ACID or BASE styles.

System lifecycles and technical debt

Architecture methods have to address the life cycle of a system, and when to replace old systems.

Many applications are badly made, they have poorly structured databases and/or give poor user experiences.

Many are over-engineered: too many layers, vacuous abstractions, facades on facades, needlessly rich domain models and needless use of middleware.

Even well-engineered application – after they have been amended in various ways – can become badly engineered.

But if an application works, and doesn’t cost much, it may not be a concern to enterprise architects.

 

Technical debt isn’t a measure a poor software design, and refactoring all poor design would be impossible.

Technical debt is a measure of the technical insurance premium worth paying to increase system longevity.

First you have to assess the size and likelihood of the risk.

·         How important is the application? Does it work well enough now?

·         How long will the application be needed? Will it work for the foreseeable future?

·         How likely will it have to be changed much, or re-platformed, in future? What will be the cost of doing that?

·         What is the cost of refactoring now to make the application easier, quicker and cheaper to change in the future?

·         Would it be better simply to replace the application by another?

System road maps

Architects produce road maps that set out how one or more baseline systems are replaced by one or more target systems..

A solution architect drew the following simple road map for replacing an old accounting system by a new one.

System element

Baseline  state

Transition state 1

Transition state 2

Target state

Reporting

Old system

New system

New system

New system

Purchase to Pay (PtP)

Old system

Old system

New system

New system

Order to Cash (OtC)

Old system

Old system

Old system

New system

Temporary data flows

Old-to-new PtP & OtC

Old-to-new OtC

 

EA involves coordinating several road maps, strategic and tactical, that cut across each other.

The road map for replacement of the accounting system may have to be shaped in the light of other road maps for

·         replacing the CRM application used to capture orders

·         making all core business data accessible via interfaces conforming to the OData standard

·         reorganising the business from product-focused to customer-focused (or vice-versa).

 

There are always trade offs and compromises to be made; so EA is political.

Who has authority to make decisions depends on the enterprise’s management and governance structures and processes.

Agility in its various guises

Different businesses and business systems change at different rates.

A business strategy can trigger long-term transformation projects that last years and cost millions.

(In 2016, an architect in a global business told me of a recent project that cost $200 million.)

The same business must respond to quickly to immediate problems and pressures.

Sometimes a new system is needed at three months notice; or even faster.

Agile software development in an EA context

Architecture methods are traditionally for planning major changes business systems, to a target that may be a year, two years or more away.

Often, initially-stated requirements unclear, inaccurate, incomplete – and change over time.

Agile methods encourage speedy implementation of a partial system, followed by incremental delivery and fast and flexible response to changing requirements.

Agile development methods are about what a team does to continually improve and extend a software system, with release cycle of days or weeks.

 

An agile development method is not only iterative, but also flexible about the requirements, the solution and the process being followed.

·         It favours negotiation over planning, and flexibility about requirements.

·         It presumes user involvement and feedback is essential to success.

·         It encourages early testing for usability and performance.

·         It favours short-cycle iterative development.

·         It looks to deliver a “minimal viable product” and extend that incrementally.

·         It encourages a team to change, rebuild and re-testing a software system on a daily basis.

The outcome should be faster delivery of something useful, though it may not do everything customers want.

 

Some see agile development as the anti-thesis of enterprise architecture.

However, both approaches are used in a “bimodal” IT department.

Architects may recommend (in a solution outline document) that an agile development method is used.

Architects may then govern the acceptability of changes during an agile development project.

And reflect accepted changes in architecture-level documentation.

Obstacles to agile software development

Agile development methods do not always give the hoped-for benefits.

They do not always mean quicker and cheaper delivery of a system.

It often takes a long time to develop applications that require a large/complex database structure and/or data migration.

Some systems cannot be partially implemented – must be right first time.

Architects need a score chart to define the suitability of agile development methods.

 

Obstacles to agility include:

1.      Most/all requirements are mandatory.

2.      Users/domain experts are not readily available.

3.      Developers are not empowered to make decisions.

4.      The system’s first release must be large (safety-critical, money-handling, legacy replacement, or regulatory compliance).

5.      The complexity is server-side rather than UEX side.

6.      The complexity is in input and update rather output and report.

7.      The business rules are complex and/or money-handling.

8.      The processing is batch rather than transactional.

9.      The database schema is immature and evolving.

10.  There are many integration points with other systems.

 

A project that scores highly on these measures is an inherently difficult project.

So, consider substantially increasing the time and budget for the work.

Agile systems?

Agile software development does not necessarily produce agile systems.

And reengineering old systems may make little difference.

The fact is, digitisation is and always has been difficult.

To make a useful software system is a difficult and painstaking task.

To rebuild a system using new technologies, replacing like for like, may not make the system more agile.

To make a system more agile usually implies making it more complex, and slower.

 

Moreover, the notion of an agile system is not a simple or single idea

Since a system can be flexible in several ways, as described in the table below.

 

can

yet might also be

its architecture may be

A versatile system

do many different things

rigid and hard to change or extend.

large or complex

A malleable system

be readily changed to a new version

limited and not versatile

slight or simple

An extendable system

be extended with extra features

not malleable (the Open-Closed principle)

fixed (must be right first time)

A self-organising system

change its own roles and rules

reliant on human ability, barely a system at all

slight and simple

 

Designing a complex business system so that it can be quickly changed is a big challenge; and there are always trade offs.

Generally, building flexibility into a system makes it more complex – which adds time and cost to development.

It may mean shifting business rules from procedural instructions into data variables that end users can change – which tends to make the system slower.

And maintaining flexibility may require substantial “refactoring” of the system from time to time – which requires investment over and above normal change requests.

Agile architecture?

Of course it is possible to abstract architecture-level documentation from ongoing system development and deployment.

Years ago, Scott Ambler called this Agile Model-Driven Development; today it is called evolutionary design, but is it design at all?

Isn’t that agile documentation rather than agile architecture?

Surely, an architect must describe a system's structure and behaviour at a level of abstraction that is as stable as possible?

If that highest-level architecture is continually changing, that undermines the notion of architecture.

 

Architects cannot maintain detailed documentation of business systems that change frequently.

But they should maintain just enough documentation to govern their portfolio effectively.

Enough to be able to trade off the benefits of architecture frameworks and agile software development or deployment practices.

The impact of agile development on architecture methods

The agile development of small systems and silo systems can create the issues that led people to promote EA in the first place.

The same can be said of the fashion to develop “microservices” (read this Microservices paper for trade offs that must be considered).

Addressing these issues inevitably involves tightening up project milestones

Typically, solution architects must complete “solution outline” documents and get them approved.

 

An architecture method is designed to ensure there are agreed objectives, models of baseline and target systems, and governance of system delivery.

There is a down side - the necessary bureaucracy – including impact analysis of change requests against documented system descriptions.

Of course, mindless completion and approval of document templates is ineffective, but the problem is not the document templates.

The problem is ill-educated, ill-informed document completion; and naivety in document reviewers.

There is no substitute for employing well-trained experienced and knowledgeable people at all levels.

 

Taken to the extremes, both EA bureaucracy and agile development lead to difficulties.

Architects have to understand the extremes and strike a balance between them.

References

Read 50 years of digitisation and EA for a more extensive EA history, and ends with a link to a long list of references from 1970 onwards.

There is a common theme in the referenced sources – a focus on business roles and processes that create and use business data.

Read EA and TOGAF: how do they differ? for discussion of EA with reference to TOGAF.

 

All free-to-read materials at http://avancier.website are paid for out of income from Avancier’s training courses and methods licences.

If you find the web site helpful, please spread the word and link to avancier.website in whichever social media you use.