Challenging architecture assumptions
This page is published under the terms of the licence summarized in the footnote.
All free-to-read materials on the Avancier web site are paid for out of income from Avancier’s training courses and methods licences.
If you find them helpful, please spread the word and link to the site in whichever social media you use.
In “The history of data processing” (http://www.infoq.com/presentations/db-history-data-processing)
Mark Madsen discusses the history data processing and looks to where the industry will go.
In the course of this, he lists eight choices facing people throughout history:
1. Top-down v Bottom up
2. Authority v Anarchy
3. Bureaucracy v Autonomy
4. Control v Creativity
5. Hierarchy v Network
6. Dynamic v Static
7. Powerful v Easy
8. Work up front v Postponed
Madsen says: “in every choice, something is lost and something is gained.”
In other words, choosing one at the expense of the other is never ideal.
Most of the eight choices surface in papers on the Software Architecture page at Avancier.website.
The first five characterise the tension between Enterprise Architecture (EA) and agile software development.
The sixth choice appears in the next point.
Analogies between system architecture and building architecture mislead people about what to focus on and how to go about it.
Our domain is the architecture of human and computer activity systems.
These systems are composed of components that cooperate in the performance of processes
The processes are what matters, since they deliver the desired results.
And this emphasis on the dynamic over the static makes activity system architecture different from building architecture.
Building architects (designing a house, bridge, ship or city)
Activity system architects (designing a human and computer activity system)
Focus primarily on defining system structure.
They do think about how people behave in or on their structures.
But don’t draw up detailed instructions for how people should behave.
Architectural drawings are of tangible components (walls, pillars, bulkheads, roads).
The buildings look like the drawings.
The drawings cannot be elaborated until they “work”
Builders step from description to concrete materials.
Focus primarily on defining system behaviour, and what is needed to perform it.
The structure is secondary; may be mostly hidden in a cloud.
They detail how humans and computers should behave in performable instructions.
Architectural drawings are of abstract roles and processes.
The buildings look nothing like the architectural drawings.
The “builders” painstakingly elaborate the specification until it can be performed
Builders do not step from description to concrete materials; all is abstraction.
True, an activity system employs concrete, tangible entities by way of human and computer actors.
In a human activity system, the component instances that perform procedures are people.
In a computer activity system, the component instances that perform programs are computers
Both human and computer components are general-purpose processors – we purchase or hire them as commodities - we largely take their abilities for granted.
We do not describe these components or make them; we specify only a few qualities they must possess and the roles they play in our system.
A silo system (or point solution) is an organisation unit, application or other system that:
· is not standardised - does not follow the same rules or processes as another doing the same thing
· is not integrated - does not share information with another doing something different
· does not share/reuse common services – at the business or technology level.
A silo system either does not interoperate with other systems, or resists attempts to make it interoperate.
The result is that silo systems are connected by only a few links, and loosely-coupled even there.
Inevitably, the difficulty of connecting silo systems leads designers to duplicate components in different systems.
And yet - silo systems can be a good thing. We can change one silo system without worrying too much about its effect on others.
Keeping silo systems apart stops the wider system from becoming too complex to understand and to manage.
In practice, people can and do manage scale and complexity by dividing a large and complex system into distinct silo systems.
In “On the Criteria to Be Used in Decomposing Systems into Modules” (1972) Parnas introduced the idea of information hiding.
The architectural fashion since the 1990s has been towards loosely-coupled architectures.
But loose-coupling has down sides. It can mean:
· longer end-to-end processes: due to slower inter-component communication
· higher process failure rates: due to inter-component communication failures
· more problems arising from inconsistencies: due to discrepancies between data maintained in different subsystems
· higher costs: due to the cost of network and mediator technologies to connect subsystems
· higher system complexity
· lower system change productivity.
As a crude generalisation, architects look for:
· Increasingly high cohesion within a system as it gets smaller.
· Increasingly loose coupling between systems as they get larger.
Within a system of a manageable size, you may do better to keep its subsystems closely coupled in some or all ways.
Only when the system reaches a size you can’t manage are you better advised to keep the systems loosely coupled in most of the ten ways.
For many years, technology vendors have used the following story to sell their middleware technology - it barely matters what kind of middleware it is.
“Your trouble (Mr Customer) is that you have hundreds of applications, which talk to each other in a point-to-point style.
Looked at as a whole, your application portfolio is unmanageable spaghetti.
I can radically improve your enterprise architecture.
Put my technology in the middle.
Connect all your applications to my technology, instead of to each other.
The spaghetti has disappeared.
Your new hub-and-spoke enterprise architecture will be much more efficient and easier to manage.”
This is a con trick – even if the salesman doesn’t realise it.
He started by talking about logical point-to-point data flows, then switched to talking about physical hub-and-spoke communication channels.
The enterprise can find itself painfully coding most if not all the same point-to-point data flows inside the vendor’s technology.
Not only is the spaghetti still there, each strand has been divided into two sections – to and from the hub.
And the enterprise now has yet another technology to operate and maintain.
There is a place for middleware, for many-to-many messages and where senders and receivers are volatile.
But it is not a universal panacea, and in some cases it complicates rather than simplifies.
Reuse does not always mean faster, better or cheaper.
Reuse is an ambiguous term; it can mean direct, copy or tailor reuse.
Direct reuse - one instance of a thing is used at run time by many.
This leads to the many service sharing challenges discussed in on the web site.
Copy reuse – many instances of a thing are used by many.
Copy reuse can be counter productive.
It means you started with something you didn’t quite want, and are constrained by that.
Copy and tailor reuse – many variations of a thing are made at design time.
To revise what you copy costs you time and money.
Then, you then have to test it before you can use it.
Your testing may reveal features of the copied thing that are contrary to your purpose, meaning you have to withdraw it and revise it a bit more.
You now have a bespoke thing, and you are obliged to maintain it separately from (in parallel with) the original thing.
Improving operational efficiency may trade off
Producing more – by automating production – perhaps by copy reuse, or copy and tailor reuse.
Producing less – by careful design – by not copying things - by consolidating several things to one – by direct reuse.
If architects on training courses are to be believed, many businesses have set out to consolidate and standardise their applications by buying a big COTS package, only to be disappointed.
They say their enterprise has spent millions of dollars/pounds/euros on an ERP or CRM package.
And then found it does little of what they want and/or the modules they want to use have to be expensively configured using specialist consultants.
The problem is rarely that the old system was no good.
Sometimes the old system is better than the new one.
The problem is sometimes that the intelligent design team who created the old system has moved on.
Legacy systems will die if there is no design team looking after the system with the same intelligence and care as the original design team.
In 1975: Michael Jackson wrote “The beginning of wisdom for a programmer is to recognise the difference between get a program to work, and getting it right.”
By right, he meant economical and maintainable.
Software systems have been built according to many different design methods and patterns.
When a system works, the team naturally assumes their methods and patterns are correct, and the system is well-designed.
Just because you have made a design pattern work, doesn’t mean that pattern was best or right for the system you had to build.
A personal anecdote
In his 1971 paper on “stepwise refinement”, Niklaus Wirth said “program construction consists of a sequence of refinement steps. In each step a given task is broken up into a number of subtasks.”
The paper vastly over complicated what should have been a simple program.
A year or two later, Djikstra later wrote up a different solution to the same program design problem.
A year or two later, I reworked it in an object-oriented style, designing it around an object that called another of the same type in a recursive fashion.
I was pleased to have worked through the intellectual challenge and impressed with my own work.
Many years later, I reworked it in event-oriented style, and found the program could be coded in about 20 lines of BASIC.
“All architecture is design but not all design is architecture.
Architecture represents the significant design decisions that shape a system, where significant is measured by cost of change.” Grady Booch
If a design is an abstraction from an executable specification, then an architecture description is an abstraction from a design.
But there is no fixed line between design and architecture.
There are scores of papers on the Avancier web site on the topics of enterprise, solution and software architecture.
You could replace all "architecture" with "design" in all the papers without any effect on their meaning.
A continuous abstraction hierarchy runs from the bottom-level executable specification to the highest-level system or enterprise description.
If you do draw a line between architecture and design, then it will shift up and down the abstraction hierarchy depending on
· how high you start your analysis
· how low you finish your design.
“I entirely concur. There is a design continuum that runs from more abstract to less abstract (also from more distributed to more local).
One end, by convention around the top of this continuum, can be referred to as 'architecture', with the bottom end 'implementation', but it is all design at some level…
The design continuum is often broken up by people into segments labeled 'architecture', 'design', 'detailed design', 'implementation'.
The boundaries of the segments are relative and depend on viewpoint. So any 'line' is a man-made, non-deterministic, judgment call, of convenience.” Ron Segal
Evolution is a bottom up process by which a system changes slightly, from one generation to the next.
Evolution by natural selection proceeds by random changes – not designed ones.
The opposite of evolution is intelligent design – by which wholly new things and large-scale changes are devised.
Architects are intelligent designers who describe new things and large-scale changes.
If you don’t have new things to make and large-scale changes to make, you don’t need architects.
Dion Hinchliffe says that web-oriented architecture (based on REST) enables systems to be built from the bottom-up.
Some beautiful systems may emerge from the bottom-up – by evolutionary development.
But you can’t call them architected if there is no architect, no architectural blueprint.
Architecting is a top-down process: architects are intelligent designers who produce blueprints for builders follow as best they can.
Evolution in biology means a continuing iteration of small changes that better adapt a species to its environment.
There is much to be said for evolutionary development, but the analogy with biological evolution is flawed.
The basis of biological evolution is that there is no architect, no intelligent design.
By contrast, every kind of IT development requires intelligent designers.
This table compares and contrasts evolution in IT with evolution in biology.
and growth through change
There is a continuing cycle of reproduction with tiny changes in each generation.
Changes that help individuals to thrive and reproduce (better than competitors) are more likely to be passed on.
Changes that hinder individuals from doing this are less likely to be passed on.
Viruses replicate themselves.
However, we don't willingly download virus updates.
Other kinds software are so widely useful they get not only copied but also updated.
For example, most of us have a copy of the Java Virtual Machine, and willingly download updated versions of it.
However, population growth is not what IT people think of when talking about evolution.
They focus on the improvement of a single system instance, through iterative change management. The key here is the term 'management'.
Source of change
Changes occur through chance mutations in genes.
Changes occur by design in response to requirements, test results or change requests.
This is a managed process that depends on intelligent design.
Speed of change
Genetic changes are very small and accumulate very slowly.
Hundreds or thousands of generations may pass without perceptible change.
Change can be very fast, a new system may be released every day.
Direction of change
Most genetic changes are deleterious.
Only a few are advantageous and survive.
Most software changes are improvements.
Intelligent designers don't look to make any deleterious change.
Consolidation of poor design
Biological evolution can consolidate what a designer considers a poor design.
For example, the photoreceptors in the eye point backwards.
(An example quoted by Richard Dawkins.)
Software enhancements can consolidate what a designer considers a poor design.
A widely reused system may be badly designed, may wastefully offer many services that nobody wants to use.
As long as the system fits the niche its users want, it survives and may be copied.
There are occasionally dramatic changes in the physical environment which cause mass extinctions.
There are frequently changes in the operating systems and other platform software upon which applications depend.
Size of change
There is absolutely no possibility of any substantial 'redesign' to optimise the structure of a system or make it work in a new environment.
In agile software development, intelligent designers are encouraged to continually ‘refactor’ the system to remove waste and optimise the system design.
Even a catastrophic change to the underlying platform software can be managed by intelligent designers.
However much users like an IT system, it will eventually die out.
And it will die sooner if there is no design team looking after it - with the same intelligence and care as the original design team.
Every kind of IT development requires intelligent designers.
There are several obstacles to completing and maintaining a single enterprise architecture model.
The obstacles include the excessive scale of the model and insufficient sponsorship.
Excessive scale of the model
Suppose your enterprise has 32 business functions, each with 32 software applications, each with 32 components, each component with 32 modules/classes.
A full top-to-bottom model of your enterprise architecture model would have one million software modules at the bottom level.
Perhaps one day we will have the resources, techniques and tools to complete a single coherent description with a million building blocks.
And connect them properly in single hierarchical structure; and keep it up to date with changes.
I haven’t seen one yet.
The code of a software application is a detailed description of a system’s structure and behaviour.
We have to complete it fully and correctly – we cannot get away with less - because a computer system cannot work from incomplete or incorrect instructions
One application may be composed from a thousand modules/classes.
To build and maintain an abstract description of a thousand modules requires a huge effort.
It requires much in the way of skilled human resources, techniques and tools.
A Gartner report claimed that less than 20% of software code is in fact documented.
Surely, to build and maintain a comparably detailed description of a human activity system is impossible?
We could never get investment, because we can get away with far, far less.
We trust that human beings can and will work from incomplete and incorrect instructions.
Higher level specifications acts to define the requirements and constraints for lower level specifications
Just as too much close coupling between systems is a bad thing so, too much traceability between levels of specification is a bad thing.
You cannot document every relationship between elements in lower-level and higher-level specifications
If you could, you would end up stitching them together into the humungous top-to-bottom architecture you were trying to avoid in the first place.
And you knew that would be too large and complex to maintain.
So here, in place of traceability records, we need a sense of duty and governance by human beings.
A lower-level architect has the duty to ensure their architecture conforms to the higher-level architecture as best they can.
The higher-level architect has the duty to govern the lower level, to assess the validity of the lower-level architecture during the course of architecture compliance reviews.
For many years, vendors having been selling new technologies as the answer to customers' system development and maintenance problems.
· Languages: 3GLs > 4GLs > OOPLs.
· Modelling: Modelling languages > Model-Driven Architecture tools
· Databases: CODASYL, Relational, NoSQL.
· Communication: Client-server, Middleware, RESTful Web Services.
New technologies can open up new opportunities
But it is far from obvious that productivity has improved as we move along any of these scales; some say it has got worse.
Have you read “The Mythical Man-Month: Essays on Software Engineering” by Fred Brooks?
"we cannot expect ever to see two-fold gains every two years" in software development, like there is in hardware development.
"there is no single development, in either technology or management technique, which by itself promises even one order of magnitude improvement within a decade in productivity, in reliability, in simplicity."
Creative Commons Attribution-No Derivative Works Licence 2.0 16/04/2015 22:37
Attribution: You may copy, distribute and display this copyrighted work only if you clearly credit “Avancier Limited: http://avancier.co.uk” before the start and include this footnote at the end.
No Derivative Works: You may copy, distribute, display only complete and verbatim copies of this page, not derivative works based upon it.
For more information about the licence, see http://creativecommons.org