Who joins up
the dots?
Does agile system development create a sub-optimal enterprise?
Does optimising the enterprise-wide system hinder agility?
How does Conway’s Law apply?
This paper is published under the terms and conditions in the
footnote.
IT does for a modern business what even a huge labour force could never have done in years gone by.
By that measure, enterprises are under-spending on IT with respect to other mission-critical functions.
Yet it is often hard to make a convincing business case for any particular IT project.
Architects see their enterprise’s system estate as a mess; and believe there must be a better way.
EA
frameworks encourage people to standardise and integrate systems.
Yet deciding how best to do that, and where to spend money is unclear.
How to join up the dots, where the dots are separately developed systems?
EA
frameworks propose drawing up and maintaining comprehensive plans and
specifications.
Yet agile development methods discourage them, and encourage reliance on high-skilled
teams.
Extremists in each school appear to hold irreconcilable positions.
Who joins
up the dots, where the dots are enterprise architects and agile software
developers?
This paper concludes that without good solution architects, who work collaboratively, EA is somewhere between difficult and doomed.
Contents
How
does agile differ from iterative development?
The
primacy behaviour over structure
Abstraction
from software to enterprise architecture
The
principle of sub-optimisation
Beware
naïve claims about agility
What
about the new world of SaaS?
The
need for some top-down governance
Appendix:
separating system descriptions from operational systems
Does what works for software architecture work for enterprise architecture?
First, what does work for software system design, build and operation?
People have tried waterfall, iterative and agile approaches.
Waterfall?
To begin with, software engineering was seen as a variation of hardware engineering, which followed a waterfall method.
Given a big product is to be delivered, the waterfall approach is to schedule a long project that starts by specifying all the requirements.
By 1990, the waterfall approach was deprecated for the development of any but small software systems.
And people noticed that teams are better at maintaining applications than developing them.
Iterative?
Application maintenance teams neither produce nor follow substantial requirements specification documentation.
The team starts with an existing product that already works.
The team is given a pipe line of discrete requests (cf. user stories) in some kind of change request log.
The change requests are prioritised for attention, and each is typically completed between 1 and 10 days.
There is a set of "regression tests" that can be run automatically before each change is released into production.
Many people looked to make app/product development work in a similar way
The idea is to turn a project to build a big thing into the continuous development of a product.
Since 1990s, such an iterative or incremental approach has been the recommended approach for application development.
So, a system’s "functionality" is divided into “use cases” that each meet a goal and/or deliver value to users.
The use cases are prioritised so that the development team can deliver business value as early as possible.
Uses are grouped into batches that can be designed, tested and delivered incrementally
Successive cycles of development, test and release add more and more use cases to the system.
Agile?
Agile development methods
encourage very short delivery cycles, dividing use cases for delivery
into shorter user stories.
And performing
continuous (daily) integration, to prevent subsystem interfaces getting out of
step.
Great things can be achieved in
small increments.
But even very short step iteration is not enough to qualify a project as "agile" in the sense of the agile manifesto.
Here, “agile” means more than
that; it means the applying principles and practices found in agile methods
like SCRUM.
What makes agile methods different is the willingness and ability to respond to changes.
The agile team is willing and able
to accept new requirements, change the design structure, and change the team’s
roles and processes.
Capitalising on brain power
Digitising business roles and processes is inherently difficult, as is designing, building testing and change information systems.
There are complex class, deployment and network diagrams that only high-skilled professionals can understand.
The design, build and test process is intellectually challenging; no silver bullet technology can replace humans in it.
In practice, most system complexity is managed in the heads of the people who "know" the system/product.
And agile methods are built around that assumption.
Agile methods
promote employing a team of skilled people who know their product/system well
and who interact closely.
They capitalise on the power
of the human mind and the effectiveness of person-to-person communication
Given the principle of favouring
interaction over documentation, the architecture-level description of the
product may be light.
Yet it still not far too removed
from bottom level operations and data item types, since the system scope is
limited to what one team can manage.
And if needed, some documentation may be reverse-engineered from code.
Leader roles
Traditionally, a software project
is led by a triumvirate:
·
A lead business
analyst accountable for defining use cases that deliver value to system users.
·
A lead architect who
oversees all matter technical and the meeting of non-functional requirements.
·
A project manager
accountable for time and budget.
There are parallels (not matches)
between the roles above and the roles in SCRUM.
|
Responsibility |
Traditional roles |
Closest agile role |
|
Required behaviour/functions |
Business analyst |
Product owner |
|
Non-functional requirements Solution design structure |
Architect |
Dev lead |
|
Project roles and processes |
Project manager |
SCRUM master |
Where is the project manager role in a SCRUM world?
The traditional role of a manager is challenging in a technical project in which the work is highly intellectual.
The workers are capable of deciding what to be done, solving product, process, and people/role issues.
The SCRUM master is a facilitator, and a resolver of process problems.
The product owner minds the time and cash boxing of work.
Accountability for time and budget
Agile methods promote time and
cash boxing.
In SCRUM, the product owner may manage time and budget for the next time and cash box.
The “fail faster is good”
principle implies a project will be canned sooner rather than later if it is
not succeeding as hoped.
The approach fits most naturally to “soft target” systems and in-house developments
It is more difficult to apply
where there is a large, complex “hard target”, and the overall cost is
predicted to be millions.
And where the
project has been awarded to an external supplier, with time and budget agreed
in a contract.
Following system theory, a traditional principle of system design is to define required behaviour before designing the component structure.
All activity systems (business, software and infrastructure) have both structural and behavioural elements.
A behavioural view has start and end points (as in a use case definition, flow chart, state chart or interaction diagram in UML).
· Behavioural elements describe what a system does - are typically called processes or activities – and have a time dimension.
· Behavioural elements are usually repeated, and sometimes cyclical.
A structural view has no start and end points (as in a network diagram, a data model, or a class diagram in UML).
· Structural elements describe what a system is made of – are typically called components, actors or roles – and are addressable.
· Active components perform processes.
· Passive objects are structures on which active components perform processes.
Product/system definition by behaviour
In most system development
approaches, a product/system is initially defined in terms of required
behaviour, not designed structure.
This can be seen in SCRUM and in more
traditional requirements capture mechanisms.
|
Agile requirements |
Traditional requirements |
|
Epics User stories Test cases |
Use cases (aka “function definitions”) Use case steps and automated services Test cases |
An epic is large or
longer than a story, a use case is longer than use case step, but the
distinctions are subjective.
A popular format for user stories is: As a <type of
user> I <want/can/need to....> so that <some reason>.
· As customer, I want to order goods on-line via a credit card, so that I don’t have to visit the shop.
· As an employee, I want to cancel my parking pass for a week, so that don’t pay for a parking space while I am holiday.
By comparison, a system use case is named after its outcome, goal or desired effect.
· Orders goods on-line.
· Temporarily cancel parking pass.
A use case definition may elaborated using a detailed template, but the point here is only that it represents required behaviour.
Use cases are supposed to be written such that system testers can make test cases; the earlier the better.
Refactoring of designed structure
Some assume a large system
structure can and should be designed up front, before anything is delivered.
They further assume a high-level
structure can be defined and successively decomposed into components that are
readily coded and implemented.
This approach has such a poor
track record in large software projects that Agile
methods tend to discourage “big design up front”.
Instead, agilists
encourage building whatever structure is needed to perform the immediately
required behaviours.
Then iteratively refactoring the
system’s structure, to optimise the performance of new behaviours and the reuse
of old ones
This refactoring is recognised as
a good thing, rather than considered a consequence of poor design up front.
However, it is one thing to
refactor the active structural
elements (components, classes, modules etc.)
And another thing to refactor the passive structural elements that active
components work on.
Changes to data types and data
structures:
·
require the migration of test data and persistent data from old
structures to new ones.
·
can ripple through many software components.
In database-centric systems, at least,
there is a case for striving to stabilise data types and structures as early as
possible.
And this is an area where an
enterprise data architect may impose some top down constraints.
“The method proposed by systems
theory is to model complex entities created by the multiple interaction
of components by abstracting from
certain details of structure and component.”
Software architecture involves
designing and maintaining the internal structure of one application.
There is a definitive bottom level of system description - data item types and performable process steps.
A level or two higher there are complex class diagrams that only high-skilled professionals can understand.
One application may have 1,000
modules or classes (some applications have more).
At a higher-level, an enterprise
may have 1,000 applications (some businesses have more).
Usually, that portfolio is a mess
of applications, developed separately and in different ways.
There may be a mix of products: some
generic off-the-shelf, some maintained overseas, and some developed in-house.
Some applications overlap in the
scope of the roles and processes they support, or the data they maintain.
Enterprise architecture involves
the challenge of tidying up the messy application portfolio.
A goal is deduplication,
standardisation and integration of applications.
Is enterprise
architecture more complex that software architecture?
Do the numbers above mean the
enterprise architect manages a system one thousand times more complex than the
application software architect?
No!
“At
this point we must be clear about how a "system" is to be defined.
Our first impulse is to point at [some machine] and to say "the system is that thing there".
This method, however, has a fundamental disadvantage: every material object contains no less than an infinity of variables and therefore of possible systems.
Any suggestion that we should study "all" the facts is unrealistic, and actually the attempt is never made.
What is necessary is that we should pick out and study the facts that are relevant to some main interest that is already given.” (Ashby in “Introduction to Cybernetics”).
The complexity (aka complication) of a system is not and cannot be measured in the operational system itself.
Every real machine is infinitely complex, beyond comprehension, especially where the machine involves human activity.
Complexity can only be measured in a description a system, be it low level or abstract.
The complexity managed by an architect is only as complex as the system description that architect uses.
Compare the size and complexity of an enterprise application portfolio and a single application.
Size? The enterprise’s software estate may include one million
software modules or classes.
But both the single application
and the whole application portfolio have one thousand discrete components, and
by that simple measurement are the same size.
Complexity? A software system configuration is usually more complex
than an application portfolio configuration.
The software configuration, class
and deployment diagrams, of one application may be complex.
Yet be invisible to an enterprise
applications architect, whose higher level applications communication diagram
(aka data flow diagram) is probably simpler.
At the enterprise level, the complexity management task is probably simpler than that of the application software architect.
Any enterprise-wide system
description is necessarily far removed by abstraction from the bottom level
operations and data item types.
So far removed as to be of little use to an application
dev/maintenance team.
Scaling up from one team to many teams
What happens when (by increments) the scope of a system reaches a size and complexity that one team can no longer manage?
The scope has to be divided between teams; and now, hand-offs between teams produce information scatter and waste.
But at the size and complexity of "the enterprise as a system", hand offs are inevitable.
At some point in scaling up, the challenge less about how to make one team work better and more about how to manage interfaces between teams.
There are interfaces not only between teams at the same level, but between teams in different organisations and at different levels.
If the enterprise is to be regarded as a system, then the enterprise architecture must be divided into manageable components
Each component delivers services through defined interfaces and obtained services from shared components.
For 40 years, software architects have discussed how best to do this under the headings of modular design, component-based design and SOA.
The challenge is to do this at the enterprise level, not the team level.
Conway’s law
Agile development gurus like to quote Conway’s law (Melvyn Conway, 1967)
“Any organization that designs a system will produce a design whose structure is a copy of the organization's communication structure.”
The law sounds profound, but what does it mean, and isn’t it rather trite?
Does it mean:
· the division of software into modules will reflect the division of developers into teams? or
· the division of developers into teams will reflect the division of software into modules? or both?
And whichever it means - how could it ever be otherwise?
The principle of sub-optimization
“Optimizing the outcome for a subsystem will in general not optimize the outcome for the system as a whole.”
“When you try to optimize the global outcome for a system consisting of distinct subsystems, you might try to do this by optimizing the result for each of the subsystems separately.
This is called "sub-optimization".
The principle of sub-optimization states that sub-optimization in general does not lead to global optimization.”
Reference: Heylighen F. (1992) : "Evolution, Selfishness and Cooperation", Journal of Ideas, Vol 2, # 4, pp 70-76.
The
Bennett-Berrisford law
This combines a corollary to Conway’s law with the principle
of sub-optimization
"People naturally coalesce into work groups that optimise their system of interest, and in doing so, de-optimise any wider system of which their system is a part".
Agile
development methods and principles are designed for a team of skilled people
who interact closely.
They are willing and able to
respond to change.
And to change
not only the requirements, but the design, and the roles and processes of the
team.
Obviously,
one agile development team cannot design or maintain the entire software estate
of an enterprise.
Given that
the enterprise-wide system has been divided for maintenance between
loosely-coupled teams, the systems also become loosely coupled.
Different
teams duplicate what could be common components.
They duplicate
the capture and storage of data about the same things.
The define
business rules (data types, constraints, derivation rules) differently, and
apply them separately.
Enterprise architecture is often promoted as increasing business and IT agility.
Standardisation and integration can improve the integrity of information captured and provided, and reduce costs by removing duplication.
But as a general rule, integration between systems = more complexity = less agile systems.
A silo system can be changed more easily than one with many interfaces, and many shared components and resources.
Agile development methods are one thing; agile systems are another.
Agile methods might make a system that is easier to change.
However, agile principles like YAGNI (You Ain’t Gonna Need it) and a willingness to refactor don’t necessarily make agile systems.
They might even discourage forethought about how to avoid the need for refactoring.
“In my view, for agile teams to remain agile, a portion of the product backlog should explicitly address 'cost of change' issues.
By investing in 'features' that help reduce cost of change (e.g. devops, providing reusable integration points, wrapping proprietary solutions in generic interface definitions, building modular (substitutable) architectures, automated database migration scripts, making source code available for others to contribute to and use, keeping infrastructure components up-to-date), it should be possible to accommodate unplanned changes quicker, cheaper and easier.
I would argue that approach alone would make significant inroads in addressing EA concerns.
Having, say 20% , of agile product backlogs focused on this would be reasonable.” Darragh O’Grady
In the past, many packages were inflexible; and this remains true of some packages today.
Package purchases build an eco-system of add-ons and interfaces, in and around the package, to meet their needs.
They then find it hard to upgrade to the next package version.
They put off version upgrades for years, making the problem worse.
When they finally commit to an upgrade, it is very expensive and parts of the eco-system break down.
The latest generation of packages are more configurable, and (it is promised) bespoke additions can survive upgrades.
We may be moving to a word of SaaS solutions in which all data centre infrastructure concerns are hidden from the enterprise.
Performance, scalability, security and disaster recovery in the SaaS data centre will be better than most enterprises can manage.
But still, no package will do everything every purchaser wants.
Even if the purchaser can add components without disabling upgrades, interfaces to other systems will still be needed.
Your package should offer structured ways for exporting/importing data.
But the performance and other qualities of interfaces between systems can still be a concern.
Ad-hoc solutions in different scripting languages make dependency management and testing a challenge in practice.
Integration of discretely-managed information systems (packaged or in-house) can be challenging.
Each system may have power over data consumers, but perhaps no power over data sources.
Suppliers of data may have little or no responsibility for data feeds they provide.
Inter-system data exchanges create complex dependencies, may be unreliable or problematic in other ways.
Enterprise need a strategy for efficient and effect systems integration.
Ad hoc system integrations must be constrained by governance to strategic principles and standards.
This does not mean simply purchasing an expensive middleware suite and telling everybody to use it.
Architects still have to determine where business state data is stored, where it is duplicated, what master-copy relationships are .
Basically, architects are needed to join up the dots.
Agile principles, applied to each team separately, may lead the teams in different directions.
Division by behaviour/processes creates a challenge by way of duplication or maintenance of common structural components.
Division by structure may make it harder when integrating components to perform behaviour.
Finding the time and budget for cross-team refactoring, and achieving it, is challenging.
An enterprise could accept that agile development teams will produce systems that sub-optimise the enterprise.
And accept there is duplication and dis-integrity between systems.
The fact is, most large enterprises are in that situation now, with or without agile development.
If enterprise mangers recognise the messy estate as an issue worth addressing, then they need over-arching enterprise architecture of some kind.
SCRUM enthusiasts might propose a collective of product owners and dev team leaders can perform this function.
But this is difficult where some apps have no product owners or dev teams, some apps are maintained overseas, and some apps are off-the-shelf products.
Enterprise architecture is about tidying up the messy system estate by standardisation and integration.
Enterprise applications architects look to rationalise the application portfolio.
An aim is to simplify - by
standardisation and reuse - what software architects are asked to do.
Enterprise architecture is partly about imposing enterprise-wide principles and standards on all
teams. E.g.
·
A canonical data
format for name, address and telephone number data types,
·
XML document format
for transporting messages,
·
SOAP/TCP/IP protocol
stack
·
Oracle v12 for
database management
Enterprise architecture is partly about integrating systems and partly about constraining system integration.
Allowing agile programmers to create inter-system dependencies is risky, if not outright dangerous.
Allowing software (via automated service discovery) to do the same could prove worse.
If the enterprise is to know and govern what is going on, surely the enterprise needs
· data architects who know what business data sits where, and constrains who can use it?
· service catalogue architects who decide which remote services are useful and reliable, and constrains who uses them?
However, there is often a gulf between the enterprise architects who devise principles and standards, and the software developers who are supposed to implement them.
Who bridges the gap?
There is no sure-fire way to do enterprise architecture without imposing unworkable governance overheads.
But capitalising on the brain power of solution architects is a more agile approach.
Without good solution architects, who work collaboratively, EA is somewhere between difficult and doomed.
It does seem that enterprise and
software architecture are different problems, requiring different solutions.
The challenge
is to connect them in a coherent organisation, bearing in mind the
Bennett-Berrisford law.
"People naturally coalesce into work groups that optimise their system of interest, and in doing so, de-optimise any wider system of which their system is a part".
Agile methods are popular and successful in the context of a software engineering team.
They are geared to the world of the team who builds and maintains a system/product that is changed as and when needed.
How to scale up from the software configuration of one product to the whole enterprise-as-a-system?
Our challenge is to understand where and how best to spend the money on EA level thinking.
Do we spend an EA budget on agile methods, on turning projects into products?
Every product has a life cycle; and EA is about managing product lifecycles.
Agile methods strive to make app development work in a similar way to app/product maintenance.
Still, the initial development of an app/product requires a budget, and an enterprise should have procedures for approving budgets.
Key architectural decisions are made at the start of the product lifecycle, before software developers are even engaged.
Do we spend an EA budget on populating the Zachman framework with architectural documentation?
The further you abstract system descriptions from the bottom level, the more vacuous and less directly implementable they become.
In any case, the higher-level design decisions are of a different, more political, nature.
The decisions are not about implementing the next user story; they are about investing millions in replacing one CRM system by another.
And then, on when, where and how best to integrate that system with others.
If the goal is to ensure architectural concerns are addressed by agile teams.
Then the solution does not lie in populating the populating the Zachman framework or in modifying agile methods to become EA methods.
The solution lies in employing experienced and well-trained solution architects to shape and steer what is architecturally significant before and during system development.
Those solution architects may well be employed in an "EA team" reporting to an "EA manager", but their primary role is to shape and steer solutions.
Solution architects work at the level of solution vision, then solution outline, before any software developers are employed.
A role of the EA manager is to ensure solution architects have the enterprise-wide understanding they need to optimise the enterprise-as-a-system.
Without good solution architects to join up the dots, EA is somewhere between difficult and doomed.
For solution architect
training and certification, go to http://avancier.website.
Footnote: Creative Commons Attribution-No Derivative Works Licence
2.0 01/04/2015 19:35
Attribution: You may copy, distribute and display this copyrighted work
only if you clearly credit “Avancier Limited: http://avancier.website” before
the start and include this footnote at the end.
No Derivative Works: You may copy, distribute, display only complete and verbatim
copies of this page, not derivative works based upon it.
For more information about the
licence, see http://creativecommons.org
In general system theory, a theoretical system is abstracted from an empirical system.
“It is important to set apart a theoretical system from an empirical system.
· The former is a complex of concepts, suppositions, and propositions having both logical integration and empirical reference.
· The latter is a set of phenomena in the observable world that is amenable to description and analysis by means of a theoretical system.” (Laszlo and Krippner)
It is important to realise that
architects describe and manage a theoretical system that is much simpler than
the empirical one.