Agile 3 – On agile businesses and systems

Avancier’s agile papers

This is the third in a series of papers.

1. On the beginnings of agile

2. On agile software development

3. On agile businesses and systems

4. On systems thinking ideas used in agile

5. What is agile architecture?

6. EA in the world of agile architecture

Enterprise architecture (EA) has always been challenging, both technically and politically.

In the 1990s, the US OBM Memoranda 97-16 guidance on EA declared that.

“the balance between centralization and decentralization and the pace of change within the agency, should be clearly understood.”

Today, the centralisation-decentralisation balance, and the pace of change, remain huge issues.

Gurus like to discuss what a business should do to survive in a changing world.

The vision is of an agile business, enabled and supported by agile systems.

In this context, what does agility mean? And what trade offs have to be made?

Contents

“The five trademarks of agile organizations”. 1

The old paradigm “Organisation as Machine”. 1

The new paradigm “Organisation as Organism”. 3

On digital transformations 4

On the Internet giants (FANGS) 6

The limits to “the learning organisation”. 7

On agile systems. 7

System design tradeoffs. 8

“The five trademarks of agile organizations”

This questionable McKinsey article described two paradigms.

The old paradigm “Organisation as Machine”

The article discusses how, in the early 20^th century, Henry Ford and Frederick Taylor promoted specialisation of roles under a management hierarchy.

It characterises the organisation as a machine in the following way.

1. “In an environment of scarcity, we succeed by capturing value from competitors, customers, and suppliers for our shareholders.”

2. “People need to be directed and managed, otherwise they won’t know what to do—and they’ll just look out for themselves. There will be chaos.”

3. “To deliver the right outcome, the most senior and experienced individuals must define where we’re going, the detailed plans needed to get there, and how to minimize risk along the way.”

4. “To achieve desired outcomes, leaders need to control and direct work by constantly specifying tasks and steering the work of employees.”

5. “Technology is a supporting capability that delivers specific services, platforms, or tools to the rest of the organization as defined by priorities, resourcing, and budget.”

This seems a description of a straw man, since in my experience of business since the 1970s:

· Once trained, few employees are closely managed and controlled.

· Detailed planning of changes is often not possible, or done.

· Technologies are already embedded in business operations.

The article is questionable from start to end starting with this – an organism is a machine.

It suggests we should be surprised at what seems predictable.

“When machine organizations have tried to engage with the new environment, it has not worked out well for many.”

Is that a surprise? Presumably, the new environment is a digitised world.

A business that purchases and makes and moves materials is completely different from one that captures data and manipulates it.

“A very small number of companies have thrived over time. Less than 10 percent of non-financial S&P 500 companies in 1983 remained in 2013.”

Is that a surprise? Doesn’t that show that the economy as whole is agile? And healthy in that new companies can overtake old ones?

“82% went through a redesign in the last three years. However, most efforts fail—only 23 percent were successful.”

Is that a surprise? Isn’t redesign what already-failing companies do?

(A Forbes article found half senior leaders rate their business transformations a waste of time.)

The article suggests specialisation of roles under a management hierarchy is a bad thing.

But this goes way back, to 19th century sociology (Weber), to 18th century economics (Smith), and the beginnings of civilisation. Surely, the ideas will never go away.

· In a theatre: should a play have no director; should actors and stagehands be interchangeable?

· In a plane: should there be no pilot; should the cabin crew be able to steer the plane?

· In a surgical operation: should the surgeon not direct affairs; should your anaesthetist pick up the surgeon’s scalpel?

Having said that, it is clear that the pace of business change has increased over the last 100 years.

And a business needs its employees to be more flexible (or else, dare one say it, more replaceable).

The new paradigm “Organisation as Organism”

The article’s analogy is ill-chosen, since organisms are machines.

And rather than redesign themselves, organisms die to make room for descendants.

Evolution favours those descendants better fitted to changes in the environment.

The article summarises the five trademarks as follows, to which some caveats have been added.

1) North Star embodied across the organization

“Recognizing the abundance of opportunities and resources available to us, we succeed by co-creating value with and for all of our stakeholders.

To give coherence and focus to their distributed value creation models, agile organizations set a shared purpose and vision… that helps people feel personally and emotionally invested.”

OK but beware others say: "Most corporate mission statements are worthless.” Russell Ackoff (1986). “Management in Small Doses”, Wiley

2) Network of empowered teams

“When given clear responsibility and authority, people will be highly engaged, will take care of each other, will figure out ingenious solutions, and will deliver exceptional results.”

OK if somebody has divided the enterprise into a network of responsibilities that connect in a coherent whole.

As above: “the balance between centralization and decentralization… should be clearly understood.” 1997

3) Rapid decision and learning cycles

“We live in a constantly evolving environment and cannot know exactly what the future holds.

The best way to minimize risk and succeed is to embrace uncertainty and be the quickest and most productive in trying new things.”

OK if you have no long term target, like sending a man to the moon, making a movie, or replacing a legacy system.

As above: “the pace of change within the agency, should be clearly understood.” 1997

4) Dynamic people model that ignites passion

“Effective leaders empower employees to take full ownership, confident they will drive the organization toward fulfilling its purpose and vision.”

OK if you can find them, and each team is already trained to the required level.

5) Next-generation enabling technology

“Technology is seamlessly integrated and core to every aspect of the organization as a means to unlock value and enable quick reactions to business and stakeholder needs.”

OK if your business is not primarily based on human knowledge and skills (teaching, catering, hospitality, entertainment, etc).

Of course, most businesses today sit somewhere between the two paradigms described in the article.

Let us focus on the last point: “Next-generation enabling technology”.

On digital transformations

A major source of business change is the digitisation of business roles and processes.

So-called “digital transformations” are supposed to involve new technologies like Social Media Applications, Big Data, Machine Learning and other kinds of Artificial Intelligence.

An EA-like approach

This article on the Indian government’s approach bemoans digital transformation initiatives as technology-led rather than business service and data led.

It features eight reference models:

· Business, Application, Data and Technology (as in traditional EA), also

· Architecture, Performance, Integration and Security.

The steps for national, state and local governments to design architecture initiatives are outlined as below:

· Performance: Identify goals, targets and indicators. What/when/how to measure and monitor? When/How/Who to analyze and evaluate results? How/when to reports results?

· Business: Map current government services. Identify required services. Plan to transition.

· Data: What/where/when data is needed? Who needs and owns the data? How will it be used, consumed and protected from illegitimate use? Where is it stored?

· Integration: Identify integration requirements to realise whole-of-government perspective needed to achieve the goals.

“The results and outcomes require aggregation of data from various sources.

Realization of cross-domain, cross-sectoral services require data interchange, common data standards and communication protocols.”

This data-orientation sounds like classic, strategic, cross-organisational EA.

Where does agility some in to the picture?

Lessons learned

This Forbes article quotes a survey finding half of senior leaders rate their business transformations a waste of time.

It suggests the issue is that most organizations are rushing to adopt the manifestations of digitisation.

For example, appointing chief digital officers and announcing programs to implement artificial intelligence.

Before getting to grips with what needs to change and therefore where their focus needs to lie.

“They don’t understand what is causing friction in the organization”

Five finding were:

· Worry about data before technology; reliable data is vital.

· Reengineer processes; don’t just digitise them.

· Smash the silos (in the company structure).

· It’s all about the business model.

· People first, technology second.

This people, process and data orientation sounds classic business process reengineering.

Again, where does agility some in to the picture?

A case study in business change https://lnkd.in/f5ehEF2

The link is to a case study in business management, planning and performance management.

Note that it is a bank, a data processing business, and therefore ripe for digitization.

Note also: “We were so bad, it was easy to improve” and the transformation was radical.

The bank was transformed over several years through a combination of:

· Better business performance metrics (e.g. related to customer experience).

· Better personal performance appraisals against KPIs

· Running meetings properly

· In-sourcing of what had been outsourced.

· Empowering employees to make incremental process improvement

· Migrating customers and bankers to digital operations.

Again, where does agility come into the picture?

Note first that several changes and their success can be attributed to effective top-down leadership.

It is a fine example of top-down leadership of strategic change.

Some agility emerged from encouraging people to improve their own processes.

Some agility emerged from being able to change software systems rather than change employees and material resources.

All organisations must strike a balance between top-down and bottom-up change

Where the balance lies depends on the nature of the business.

The last thing you want in a nuclear power station is workers making up their own processes.

Long-term targets will evolve

Target setting (be it for a 15 day sprint or a 5 year transformation) is a basic tool of business management.

In the banking case study above, the directors sponsored a 5 year legacy replacement programme.

With the target building blocks in place, it took several more years to complete the digital transformation.

Contrary to what you might read, EAs rarely predict the long-term future in any detail.

And EAs has never presumed a target architecture cannot change.

To accommodate change, an EA framework must be iterative, with continual requirements change management.

Read the 1986 PRISM report that informed EA "Dispersion and Interconnection: Approaches to Distributed Systems Architecture".

The case study concludes with a merger that disrupted the plans drawn up by the architects.

“Roger many nights in bed wondering about whether he had done the right thing.

Could he have anticipated the merger, and adjusted the architecture accordingly?

Mark and Roger decided that there was nothing they could have done about it.

Had they tried to factor in a merger, they would have been unable to make any progress at all.”

Note also: the bank’s stated ambition was (and remains) to become an internet giant, but not every business can become an internet giant.

On the Internet giants (FANGS)

In the phrase “Agile Business”, what kind of business do we have in mind?

Discussions often focus on rather peculiar examples.

Notably, application-centric internet giants like Facebook, Amazon, Netflix, Google and Spotify (FANGS).

These businesses depend on others for their content - and the quality of that content.

· Amazon sell what others create; they can always replace a rejected element by another.

· Uber piggy back on drivers managing their own cars

· Air BnB piggy back on owners managing their own accommodation.

· Google extract information from users and sell it back (in patterns that are not always useful)

· Facebook, Twitter etc. share low quality data and low quality thinking (undermining the notion of authoritative sources).

This table contrasts FANGS with other kinds of business.

Characteristics of FANGS	Characteristics of many enterprises
The business is based on an application or software product.	Business operations are human (healthcare), material (retail) or mechanical (mining)
The business is a self-serve platform for its users.	Business operations are end-to-end pipelines or value streams
Business employees do not interact personally with customers	Business employees interact personally with customers
Operation/transaction throughput is extremely high	Operation/transaction throughput is relatively low
Low integrity, loose-coupled operations.	Tight integrity/quality requirements. e.g. money-handling, health, and safety-critical.
Mistakes and misjudgements are resolved after they happen	Mistakes and misjudgements in business operations cannot be redressed after-the-fact
Develop their own business-specific software.	Use more packaged COTS than bespoke software.

Many programmers think it cool to copy what FANGS do - regardless of relevance, cost and complexity.

But most businesses are not like FANGS.

And design patterns that FANGS have developed for extreme scale web applications come with costs and downsides.

The limits to “the learning organisation”

Many systems thinkers promote the idea that every business should be a "learning organisation" (after Senge).

The learning organisation diverts resources from optimising what a business does now to considering what it might do differently in future.

It facilitates the learning of its members, and continuously transform itself.

An aim is to change, adapt, and transform to meet customer’s needs, with the current workforce.

The bank case study above illustrates the idea of the learning organisation can prove beneficial.

Successes were achieved in the process of transforming the nature of the bank’s business operations.

But let us not fool ourselves about how far and fast a business can change.

Some businesses are inherently more flexible than others

The likes of Facebook, Amazon, Netflix, Google and Spotify (FANGS) are peculiarly software-centred businesses.

Changing web sites and algorithms is one thing.

Changing physical assets and workers with special abilities, knowledge or skills is another.

No amount of organisational learning can save a bricks and mortar high street store and its sales assistants from the threat posed by the likes of Amazon.

Remember Blockbuster? It could not compete with Netflix, and went out of business.

Blockbuster's CEO had the chance to buy Netflix in the 1990s.

Supposing he had done so; the outcome would surely have been a reverse takeover.

It would have saved the Blockbuster name - but not its stores or its employees.

Evolution of the wider economy

Improvements do emerge from continual iterative sense-response adaptation.

But to preserve every business forever is impossible – and undesirable.

In regretting what happened to Blockbuster, people overlook it was good for Netflix.

At first sight, a high rate of business closures looks like a bad thing

It often mirrors a high rate of starts ups – which may well signify a healthy economy.

Sometimes, it is better that a business becomes extinct, and is replaced by a different kind of business.

The business world as a whole evolves that way, rather by maintaining every business entity forever.

What is an agile system?

What does it mean for a system to be agile?

That is a difficult question - with many possible answers.

Sometimes it means employing people to play roles in a system.

And empowering them to determine their actions, or change their roles.

It can mean a system that does little – leaving people to choose how they use it.

Or a system that is flexible in that it is readily

· configurable in operation to meet a variety of requirements.

· modifiable at design time to meet new requirements.

Then, a system can be flexible in several different ways, as described in the table below.

Agile variation	The system can	yet might also be
Extendable	be extended with extra features	not malleable (the Open-Closed principle)
Versatile	do many different things	complex and difficult to change
Malleable	be readily changed	simple and not versatile
Self-organising	change its own roles and rules	unpredictable and disorderly

Agile development does not necessarily produce agile systems.

Digitisation of systems is and always has been a difficult and painstaking task.

Designing an agile system is a challenge; and there are always trade offs.

System design tradeoffs

To facilitate the design, management and change of systems

· Decompose (Conway, 1968) modularise a large monolithic system into subsystems

· Decouple (Constantine, 1968) strive for tight cohesion within a subsystem and loose coupling between subsystems.

Taken too far however, both have negative effects; designers must strike a balance.

Decomposition tradeoffs

This table contrasts the qualities of dividing a system into larger or smaller subsystems

	Within a subsystem		Between subsystems
	Size	Processes	Messaging	Coupling
Macro subsystems	Larger	Longer	Less	Looser
Micro subsystems	Smaller	Shorter	More	Tighter

Decomposing a system into small, simple subsystems creates complexity in the structure of the system.

It increases the volume and complexity of inter-subsystem messaging.

There are many different ways to implement messaging, ranging from tightly to loosely coupled.

Size matters in the sense that smaller, subsystems typically merit tighter coupling than larger subsystems.

Decoupling downsides

Decoupling can lead to

· Duplications: caching of data from one data store in another.

· Delays: processes are slowed by the need for inter-application messaging.

· Disintegrities: and the complexity of compensating transactions to restore integrity.

· Data analysis difficulties: where a report requires data from several data stores.

“It is not high coupling per se that is the problem; it is high coupling to elements that are unstable in some dimension… If we put effort into “future proofing” or lowering the coupling when we have no realistic motivation, this is not time well spent.” Craig Larman

Agile system or simple system?

To make a system more agile usually requires a redesign that makes the system more complex. Setting out to build an agile system adds time and cost to development.

Agile system principle		Agile development principle
Decouple subsystems	Can be contrary to	Keep the system simple
Design for future flexibility	Can be contrary to	You ain’t gonna need it

Reengineering an old system to use new technologies (or for scalability) does not make a system more agile in other ways.

And it can complexify; in one case, after being rewritten in Java, a single-server legacy system required a dozen servers.

Agile system or fast system?

An agile system is usually slower than a rigid one. E.g. shifting business rules from procedural instructions to data variables that end users can change.