Agile 3 – On agile businesses and systems
Copyright Graham Berrisford. One of several hundred papers at http://avancier.website. Last updated 12/06/2019 12:44
Avancier’s agile papers
This is the third in a series of papers.
Enterprise architecture (EA) has always been challenging, both technically and politically.
In the 1990s, the US OBM Memoranda 97-16 guidance on EA declared that.
“the balance between centralization and decentralization and the pace of change within the agency, should be clearly understood.”
Today, the centralisation-decentralisation balance, and the pace of change, remain huge issues.
Gurus like to discuss what a business should do to survive in a changing world.
The vision is of an agile business, enabled and supported by agile systems.
In this context, what does agility mean? And what trade offs have to be made?
This questionably McKinsey article described two paradigms.
The article discusses how, in the early 20th century, Henry Ford and Frederick Taylor promoted specialisation of roles under a management hierarchy.
It characterises the organisation as a machine in the following way.
1. “In an environment of scarcity, we succeed by capturing value from competitors, customers, and suppliers for our shareholders.”
2. “People need to be directed and managed, otherwise they won’t know what to do—and they’ll just look out for themselves. There will be chaos.”
3. “To deliver the right outcome, the most senior and experienced individuals must define where we’re going, the detailed plans needed to get there, and how to minimize risk along the way.”
4. “To achieve desired outcomes, leaders need to control and direct work by constantly specifying tasks and steering the work of employees.”
5. “Technology is a supporting capability that delivers specific services, platforms, or tools to the rest of the organization as defined by priorities, resourcing, and budget.”
This seems a description of a straw man, since in my experience of business since the 1970s:
· Once trained, few employees are closely managed and controlled.
· Detailed planning of changes is often not possible, or done.
· Technologies are already embedded in business operations.
The article starts from some analysis that might be questioned.
1) “When machine organizations have tried to engage with the new environment, it has not worked out well for many.”
Is that a surprise? A business that purchases and makes and moves materials is completely different from one that captures data and manipulates it.
2) “A very small number of companies have thrived over time. Less than 10 percent of non-financial S&P 500 companies in 1983 remained in 2013.”
Is that a surprise? Doesn’t that show that the economy as whole is agile, and healthy in that new companies can overtake old ones?
3) “82% went through a redesign in the last three years. However, most efforts fail—only 23 percent were successful.”
Is that a surprise? Isn’t redesign what already-failing companies do?
(A Forbes article discussed below found half of senior leaders rate their business transformations a waste of time.)
Specialisation of roles under a management hierarchy goes much further back into history.
The ideas can be found in 19th century sociology, 18th century economics, and in the beginnings of civilisation.
Surely, the ideas will never go away.
· In a theatre: should a play have no director; should actors and stagehands be interchangeable?
· In a plane: should there be no pilot; should the cabin crew be able to steer the plane?
· In a surgical operation: should the surgeon not direct affairs; should your anaesthetist pick up the surgeon’s scalpel?
Having said that, it is clear that the pace of business change has increased over the last 100 years.
And a business needs its employees to be more flexible (or else, dare one say it, more replaceable).
The article’s analogy is ill-chosen, since organisms are machines.
And rather than redesign themselves, organisms die to make room for descendants.
Evolution favours those descendants better fitted to changes in the environment.
The article summarises the five trademarks as follows, to which some caveats have been added.
1) North Star embodied across the organization
“Recognizing the abundance of opportunities and resources available to us, we succeed by co-creating value with and for all of our stakeholders.
To give coherence and focus to their distributed value creation models, agile organizations set a shared purpose and vision… that helps people feel personally and emotionally invested.”
OK but beware others say: "Most corporate mission statements are worthless.” Russell Ackoff (1986). “Management in Small Doses”, Wiley
2) Network of empowered teams
“When given clear responsibility and authority, people will be highly engaged, will take care of each other, will figure out ingenious solutions, and will deliver exceptional results.”
OK if somebody has divided the enterprise into a network of responsibilities that connect in a coherent whole.
As above: “the balance between centralization and decentralization… should be clearly understood.” 1997
3) Rapid decision and learning cycles
“We live in a constantly evolving environment and cannot know exactly what the future holds.
The best way to minimize risk and succeed is to embrace uncertainty and be the quickest and most productive in trying new things.”
OK if you have no long term target, like sending a man to the moon, making a movie, or replacing a legacy system.
As above: “the pace of change within the agency, should be clearly understood.” 1997
4) Dynamic people model that ignites passion
“Effective leaders empower employees to take full ownership, confident they will drive the organization toward fulfilling its purpose and vision.”
OK if you can find them, and each team is already trained to the required level.
5) Next-generation enabling technology
“Technology is seamlessly integrated and core to every aspect of the organization as a means to unlock value and enable quick reactions to business and stakeholder needs.”
OK if your business is not primarily based on human knowledge and skills (teaching, catering, hospitality, entertainment, etc).
Of course, most businesses today sit somewhere between the two paradigms described in the article.
Let us focus on the last point: “Next-generation enabling technology”.
A major source of business change is the digitisation of business roles and processes.
So-called “digital transformations” are supposed to involve new technologies like Social Media Applications, Big Data, Machine Learning and other kinds of Artificial Intelligence.
An EA-like approach
This article on the Indian government’s approach bemoans digital transformation initiatives as technology-led rather than business service and data led.
It features eight reference models:
· Business, Application, Data and Technology (as in traditional EA), also
· Architecture, Performance, Integration and Security.
The steps for national, state and local governments to design architecture initiatives are outlined as below:
· Performance: Identify goals, targets and indicators. What/when/how to measure and monitor? When/How/Who to analyze and evaluate results? How/when to reports results?
· Business: Map current government services. Identify required services. Plan to transition.
· Data: What/where/when data is needed? Who needs and owns the data? How will it be used, consumed and protected from illegitimate use? Where is it stored?
· Integration: Identify integration requirements to realise whole-of-government perspective needed to achieve the goals.
“The results and outcomes require aggregation of data from various sources.
Realization of cross-domain, cross-sectoral services require data interchange, common data standards and communication protocols.”
This data-orientation sounds like classic, strategic, cross-organisational EA.
Where does agility some in to the picture?
This Forbes article quotes a survey finding half of senior leaders rate their business transformations a waste of time.
It suggests the issue is that most organizations are rushing to adopt the manifestations of digitisation.
For example, appointing chief digital officers and announcing programs to implement artificial intelligence.
Before getting to grips with what needs to change and therefore where their focus needs to lie.
“They don’t understand what is causing friction in the organization”
Five finding were:
· Worry about data before technology; reliable data is vital.
· Reengineer processes; don’t just digitise them.
· Smash the silos (in the company structure).
· It’s all about the business model.
· People first, technology second.
This people, process and data orientation sounds classic business process reengineering.
Again, where does agility some in to the picture?
A case study in business change https://lnkd.in/f5ehEF2
The link is to a case study in business management, planning and performance management.
Note that it is a bank, a data processing business, and therefore ripe for digitization.
Note also: “We were so bad, it was easy to improve” and the transformation was radical.
The bank was transformed over several years through a combination of:
· Better business performance metrics (e.g. related to customer experience).
· Better personal performance appraisals against KPIs
· Running meetings properly
· In-sourcing of what had been outsourced.
· Empowering employees to make incremental process improvement
· Migrating customers and bankers to digital operations.
Again, where does agility come into the picture?
Note first that several changes and their success can be attributed to effective top-down leadership.
It is a fine example of top-down leadership of strategic change.
Some agility emerged from encouraging people to improve their own processes.
Some agility emerged from being able to change software systems rather than change employees and material resources
All organisations must strike a balance between top-down and bottom-up change
Where the balance lies depends on the nature of the business.
The last thing you want in a nuclear power station is workers making up their own processes.
Long-term targets will evolve
Target setting (be it for a 15 day sprint or a 5 year transformation) is a basic tool of business management.
In the banking case study above, the directors sponsored a 5 year legacy replacement programme.
With the target building blocks in place, it took several more years to complete the digital transformation.
Contrary to what you might read, EAs rarely predict the long-term future in any detail.
And EAs has never presumed a target architecture cannot change.
To accommodate change, an EA framework must be iterative, with continual requirements change management.
Read the 1986 PRISM report that informed EA "Dispersion and Interconnection: Approaches to Distributed Systems Architecture".
The case study concludes with a merger that disrupted the plans drawn up by the architects.
“Roger many nights in bed wondering about whether he had done the right thing.
Could he have anticipated the merger, and adjusted the architecture accordingly?
Mark and Roger decided that there was nothing they could have done about it.
Had they tried to factor in a merger, they would have been unable to make any progress at all.”
Note also: the bank’s stated ambition was (and remains) to become an internet giant, but not every business can become an internet giant.
In the phrase “Agile Business”, what kind of business do we have in mind?
Discussions often focus on rather peculiar examples.
Notably, application-centric internet giants like Facebook, Amazon, Netflix, Google and Spotify (FANGS).
These businesses depend on others for their content - and the quality of that content.
· Amazon sell what others create; they can always replace a rejected element by another.
· Uber piggy back on drivers managing their own cars
· Air BnB piggy back on owners managing their own accommodation.
· Google extract information from users and sell it back (in patterns that are not always useful)
· Facebook, Twitter etc. share low quality data and low quality thinking (undermining the notion of authoritative sources).
This table contrasts FANGS with other kinds of business.
Characteristics of FANGS
Characteristics of many enterprises
The business is based on an application or software product.
Business operations are human (healthcare), material (retail) or mechanical (mining)
The business is a self-serve platform for its users.
Business operations are end-to-end pipelines or value streams
Business employees do not interact personally with customers
Business employees interact personally with customers
Operation/transaction throughput is extremely high
Operation/transaction throughput is relatively low
Low integrity, loose-coupled operations.
Tight integrity/quality requirements. e.g. money-handling, health, and safety-critical.
Mistakes and misjudgements are resolved after they happen
Mistakes and misjudgements in business operations cannot be redressed after-the-fact
Develop their own business-specific software.
Use more packaged COTS than bespoke software.
Many programmers think it cool to copy what FANGS do - regardless of relevance, cost and complexity.
But most businesses are not like FANGS.
And design patterns that FANGS have developed for extreme scale web applications come with costs and downsides.
Many systems thinkers promote the idea that every business should be a "learning organisation" (after Senge).
The learning organisation diverts resources from optimising what a business does now to considering what it might do differently in future.
It facilitates the learning of its members, and continuously transform itself.
An aim is to change, adapt, and transform to meet customer’s needs, with the current workforce.
The bank case study above illustrates the idea of the learning organisation can prove beneficial.
Successes were achieved in the process of transforming the nature of the bank’s business operations.
But let us not fool ourselves about how far and fast a business can change.
Some businesses are inherently more flexible than others
The likes of Facebook, Amazon, Netflix, Google and Spotify (FANGS) are peculiarly software-centred businesses.
Changing web sites and algorithms is one thing.
Changing physical assets and workers with special abilities, knowledge or skills is another.
No amount of organisational learning can save a bricks and mortar high street store and its sales assistants from the threat posed by the likes of Amazon.
Remember Blockbuster? It could not compete with Netflix, and went out of business.
Blockbuster's CEO had the chance to buy Netflix in the 1990s.
Supposing he had done so; the outcome would surely have been a reverse takeover.
It would have saved the Blockbuster name - but not its stores or its employees.
Evolution of the wider economy
Improvements do emerge from continual iterative sense-response adaptation.
But to preserve every business forever is impossible – and undesirable.
In regretting what happened to Blockbuster, people overlook it was good for Netflix.
At first sight, a high rate of business closures looks like a bad thing
It often mirrors a high rate of starts ups – which may well signify a healthy economy.
Sometimes, it is better that a business becomes extinct, and is replaced by a different kind of business.
The business world as a whole evolves that way, rather by maintaining every business entity forever.
Again, in the phrase “Agile Business”, what does agility mean?
Sometimes it is more about people than about systems
Teams and employees are empowered to make local changes.
Sometimes it is asserted that after empowering local teams “order will emerge from chaos”.
Sometimes it means employees are expected to be flexible.
It is expected that they will readily accept and handle changes that are made.
Sometimes it means replacing people (e.g. bank branch staff) by software.
What about the idea of designing a business system to be agile in the first place?
The notion of an agile system is not a simple or single idea
Since a system can be flexible in several ways, as described in the table below.
The system can
yet might also be
be extended with extra features
not malleable (the Open-Closed principle)
do many different things
complex and difficult to change
be readily changed
simple and not versatile
change its own roles and rules
unpredictable and disorderly
Agile development does not necessarily produce agile systems.
The plain fact is, digitisation of systems is and always has been a difficult and painstaking task.
Designing an agile system is a challenge; and there are always trade offs.
Gurus in agile development and architecture frameworks promote general principles.
It is important to recognise where principles can be in conflict.
A theme here is that balancing tradeoffs is more important than following uni-directional principles.
For many decades, general system design principles have included:
· Decomposition: modularise a large monolithic system into subsystems (Conway, 1968).
· Decoupling: strive for tight cohesion within a subsystem and loose coupling between subsystems (Constantine, 1968).
The aim of these principles is to facilitate the design, management and change of subsystems.
Taken too far however, both principles have negative effects, and designers must strike a balance between extremes.
Decomposition trade offs
This table contrasts the qualities of dividing a system into larger or smaller subsystems
Decomposing a system into small, simple subsystems creates complexity in the structure of the system.
It increases the volume and complexity of inter-subsystem messaging.
There are many different ways to implement messaging, ranging from tightly to loosely coupled.
Size matters in the sense that smaller, subsystems typically merit tighter coupling than larger subsystems.
Generally speaking, decoupling can lead to the “4 Ds”
· Duplications: caching of data from one data store in another.
· Delays: processes are slowed by the need for inter-application messaging.
· Disintegrities: and the extra complexity of compensating transactions to restore integrity.
· Data analysis difficulties: where a report requires data from several data stores.
Skillful decoupling of subsystems can help people manage and change each subsystem on its own.
But decoupling is not a single concept - our architect classes cover a dozen or more ways to decouple subsystems.
And increasing the agility of small subsystems can make it harder to meet higher/wider goals.
E.g. a principle of “microservices” is to “decentralise data management” by dividing one coherent database structure into smaller ones.
The aim is to divide the work of application development into smaller applications, to be developed by relatively autonomous development teams
In the right circumstances this is a good approach, but it can lead to downsides.
In “Applying UML and patterns”, OO thinker Craig Larman said to pick your battles.
“It is not high coupling per se that is the problem; it is high coupling to elements that are unstable in some dimension…
“If we put effort into “future proofing” or lowering the coupling when we have no realistic motivation, this is not time well spent.”
To make a system more agile usually requires a redesign that makes the system more complex.
Setting out to build an agile system adds time and cost to development.
Agile system principle
Agile development principle
Can be contrary to
Keep the system simple
Design for future flexibility
Can be contrary to
You ain’t gonna need it
Reengineering an old system to use new technologies (or for scalability) does not make a system more agile in other ways.
And it can complexify; in one case, after being rewritten in Java, a single-server legacy system required a dozen servers.
Agile system OR fast system?
An agile system is usually slower than a rigid one.
E.g. shifting business rules from procedural instructions into data variables (that end users can change) tends to make a system slower.