This page is published under the terms of the licence summarized in the footnote.
It is safe to say that much said about SOA is either marketing for platform technologies or naive.
This paper discusses challenges to reuse of services, and reuse of other components, such as database tables.
A reader’s comment
SOA
business case and governance challenges
The challenge of the business case
The challenge of ownership and payment for shared service
The challenge of multiple versions
The challenge posed by changes to data structures
“SOA is an architectural approach that enables the creation of loosely-coupled interoperable business services that can easily be shared within and between enterprises.” BEA presentation. The key phrase here is “easily shared”. Service sharing is a lot tougher than it sounds. And you won’t succeed if you don’t understand the obstacles that get in the way.
Ref. 4 discusses SOA reference models. This paper uses this much simpler model, in which different use cases, even different applications in different enterprises, make use of shared business and/or data services.
|
Use Case 1 |
Use Case 2 |
|
User interface or process for use case 1 |
User interface or process for use case 2 |
|
Shared Business Services |
|
|
Shared Data Services |
|
Making a service easy to share – technology-wise - is one part of the challenge. Wrapping up components behind web services interfaces can help here. But making reuse happen requires much more than a technical point of view.
This paper discusses reuse, and the trade offs that must be made in design for reuse. It addresses several questions that are important in an SOA methodology that sets out to optimise sharing. (Notice “optimise” rather than maximise, since maximising sharing can cause more problems that it solves.)
An applications portfolio analyst catalogues the applications of an enterprise, usually with a view to rationalizing them, to de-duplicating the portfolio and saving money. Analysis addresses the ownership, funding and maintenance of applications.
The application portfolio of a typical large enterprise contains old “stovepipe” applications developed, paid for and owned by different business units. Each is a discrete body of software – a top to bottom structure of modules. Stovepipe applications may work alongside each other on desk top and data center computers, but don’t talk to each other. Many run on dedicated servers.
The SOA vision is that stove pipe applications will increasingly cooperate. Behind the scenes - behind the user interface of the applications that business users see on their computer screens- applications will share more and more modules.
The SOA vision is that few if any applications will remain as stovepipes. The end result might be viewed as one huge sprawling enterprise application, comprising thousands of thin “business process / workflow / UI” components which invoke thousands of “shared services”. Many shared services will be centered on the supply and/or maintenance of a parcel of business data that is of value to several business units.
We will still be able to describe a usable slice of the enterprise’s software as a top to bottom structure of modules. But the slices will not be discrete. Bottom level modules will be shared between business processes. Moreover, top-level modules will come and go as business processes change.
Implications are:
But then, perhaps the SOA vision will never be realized to the extent that these issues become significant?
This quote was copied on Jan 23 2007 from: http://www.infoq.com/news/2007/01/saugatuck-soa-reality-check
“In a freely available three-page document, the following key findings are highlighted:
Code reusability is intended to assist business agility.
According to Saugatuck, the three key inhibitors need to be overcome for successful SOA adoption are
This paper goes a long way to explain these inhibitors. But we start at the very beginning, by reviewing the basics of software reuse.
The principle that enables many different consumers to share a service is this:
The first thing in any discussion of reuse is to check what kind of reuse people are talking about.
Direct (run-time) reuse
In direct reuse, one copy of the service is used by many consumers.
Service M may be reused by some kind of “dynamic linking” from some kind of library (e.g. DLL).
Given a software application, you can usually draw (or at least imagine drawing) a structure diagram in which high-level processes and services invoke lower-level processes and services.
A general pattern for direct reuse is known as the Mosque Shape - discussed elsewhere in Avancier’s papers.
Copy (compile-time) reuse
In copy reuse, one Service is copied for deployment in several programs, applications or systems.
Service M appears by ‘static linking’ or ‘packaging’ in two or more compilation units
There is now some risk of copies getting out of step.
But copying is one way to allow different consumers to work with old and new versions of the service.
Tailoring (design-time) reuse
In tailoring reuse, consumers take a copy of Service M, then change it to fit their own purpose or context.
OK, but if we are always copying M, then perhaps should we redesign so it (or part of it) can be reused directly? Or abstract from it a more general template or pattern?
That’s the kind of thing Architects and Designers should be thinking about.
Discussions often confuse what are here called direct and copy reuse. Sometimes, what started out as direct reuse degenerates into copy reuse.
A software designer naturally works to save the time and effort of him/herself, or members of the same team.
The benefits the software designer is looking for are
All concerned are already funded to work on the solution and design it as best they can. They are both suppliers and customers of whatever shared services they can devise.
The people who own and pay for the work are the people who pay for the application. And if they realise the effort the software designer makes to optimise the design, then they should appreciate it, because it should mean they spend less. No business case for reuse need be made or assessed.
The hope in SOA is that we can scale up what the software designer does naturally within an application. We want to create common subroutines that designers of distinct applications want to reuse, and do reuse successfully.
The benefits the enterprise is looking for from shared services:
The truth is that the people who own and pay for one application aren’t interested in funding the optimisation of other future applications. They simply don’t want the cost of making services that might be reused in future, and might not.
So a business case for reuse needs be made and assessed. And any proper business case for reuse has to consider some of the risks and issues. This paper goes on to explore sharing and the trade offs in design for reuse.
We want faster, cheaper and better delivery and maintenance of business processes and consumer applications. Reuse of shared services is possibly unique amongst design strategies in promising all three of these benefits. Can reuse really help us break the iron triangle? The benefits are often exaggerated in a business case for shared services because the numbers presented hide some time, cost and quality issues.
There are three ways to quantify reuse in a business case.
e.g. Consumer application(s) invoke Service M 999 times per day.
A high invocation rate tells us very little. First of all, it depends on throughput. An old trick of systems integrators is to boast of building systems to handle N million transactions a day. While this may demonstrate the quality of the IT infrastructure, it tells us nothing about the ability of the system integrator to develop application software.
Within one application, some services are invoked more often that others. If you want to create a highly-invoked service, then you simply insert a trivial broker between many consumers and many services. Or you package many services up behind a thin façade component. The more highly used a common broker or thin facade becomes; the more it can become a bottleneck: both a design-time bottleneck and a run-time bottleneck. You may well be able to deliver your new application faster and cheaper without it.
e.g. 999 different consumers invoke Service M.
Some technical infrastructure services (say get the current date, or commit a transaction) are used by many consumers as a matter of course. Ditto some business infrastructure services (say to get the town and street address of a postcode, validate a credit card number, or display a drop-down list of reference data).
A high consumer-service ratio does suggest the service is designed well enough to be attractive to new consumers. But it does not measure the benefit to the designers of consumer applications. Many times a very small benefit is still a small benefit.
e.g. 999 different consumer applications have taken a copy of Service M at design and deployed it locally.
This too shows the service is designed well enough to be attractive to new consumers.
But why is it being copied rather than reused directly? And is it being tailored before being used?
None of these three measures tell you for a fact that all this reuse was actually a good idea, that it made for a faster, better or cheaper delivery and maintenance of consumer applications.
The highest reuse is found in highly generic infrastructure Services, and Services that act as a broker or façade in front of many other Services. Often these Services are small, so if you reuse them you do not actually save much construction or testing time. The benefit may be rather more to do with the quality of service received, and the reduction of effort at maintenance time.
How to measure what matters? That is, faster, cheaper and better delivery and maintenance of consumer applications or high-level processes that reuse services?
In practice, few make a serious effort to measure these things, to account for costs or indeed the constraints that sharing brings. People make extravagant claims based on naive assumptions.
Suppose a Service M built in one application or project, at a cost of $1,000, is reused in two subsequent applications or projects. The SOA design authority will likely claim that SOA has thus saved $2,000. But consider some of the costs that might be incurred in projects 2 and 3.
Hmm.. So how much did you save exactly? You certainly did not save the amount it took to specify, code and test Service M the first time around. You saved the time to code the service, but spent a significant amount of time on other activities. And there is the possibility a potential consumer’s research will lead them not to use Service M at all!
Estate agents sometimes describe a house with a shared entrance passage as having “unloved common parts”. Who wants to maintain them?
A service only exists to do what it is asked to do by a consumer. As long as there is only one consumer, the consumer can reasonably claim to own and control the required interface.
However, a shared service exists to serve many consumers. Whenever it is proposed that a service be reused, you ought to start out by asking some questions
When you know the answers to those questions, then you can define the service interface and service level agreement.
Above are the questions you have to ask of one service. You may be able to define the same regime for a group of services.
There is the broader more strategic question of how an enterprise manages its attempts to share lots of services, owned and managed by different departments. And how an enterprise designs or redesigns (componentizes) its software estate to optimises efforts in this direction.
But this paper is focused on the attention need to manage a single service (which might be a large component, or might not).
A service only exists to do what it is asked to do by a consumer. As a consumer, you don’t expect the owner of the service to
But this is exactly what may happen when a provider component is used by several consumer components. Beware different consumers want different things. Each consumer can find itself affected by the requirements of other consumers, as they are interpreted by the provider component. The interface may include information a consumer doesn’t want and shouldn’t touch.
Every time you change a Service, there is some tedious and unproductive administrative effort. You have to
So, as the maintainer of a Service, you will want to save effort overall by storing up changes and processing them in batches.
Sharing creates obligations and interdependencies. A shared service has obligations to all its consumers. Thus, sharing couples consumers who might otherwise know nothing about each other.
The more consumers depend on a service, the harder it gets to change the service. For any change, the service owner may have to negotiate with all the consumers, perhaps persuade them to work differently.
For impact analysis, the service owner needs to identify all the paths via which that service M is used, and all its consumers. The service owner needs to invert the service invocation hierarchy. Some CASE tools can show this for a given body of code, but not code distributed across a network.
So, the maintainer of a shared service can become a highly stressed bottleneck, and will naturally react to this stress by
In enterprise applications, you see this behaviour in the owners and maintainers of those shared services that are most stressed by many consumers - database tables, common code libraries and frameworks.
What can a poor overused and stressed service do? What if you cannot stabilise a shared service before application development? How to reduce maintenance and enhancement costs? The paper to meeting SOA management challenges discusses options.
There is no such thing as a fixed requirement. And no such thing as a business or data service that a consumer (current or new) will never want to extend or change. Changes raise two important questions.
The message to/from a web service is often a data flow that contains a serialised state. It is data structure or object graph that contains the properties of one or more objects. How to extend a web service without affecting existing consumers of that web service?
The paper to meeting SOA management challenges offers some advice on versioning strategy for services, and some notes on the same issue with product versions.
Without governance, an enterprise-wide SOA can be counter-productive. SOA has to come with a methodology for governing and interacting with an enterprise's software estate.
"The continued rise in the popularity of service-oriented architecture (SOA) has resulted in new technologies and directions but requires disciplined governance to achieve the promised reuse…. the evolving use of SOA presents challenging management issues, as does structured application integration".
Gartner 23rd Nov 2005
And later “An ungoverned SOA can become a liability for the enterprise, reversing the positive cycle and adding costs and disrupting processes.”
In fact, Gartner estimates that a lack of working governance mechanisms in mid-to-large-size (greater than 50 services) SOA projects is the most common reason for project failure. (Gartner, “Service-Oriented Architecture Craves Governance”, January 2006)
The following is edited from http://www.itbusinessedge.com/item/?ci=22039
“SOA Governance Too Focused on Services:
An enterprise’s web services register could contain thousands of functions that programmers happen to have implemented behind WS interfaces. If a SOA is to be manageable, it must centre on cataloguing and controlled use of relatively few business services.
We might create a manageable business service catalogue and still not obtain the desired benefits. The challenge was noted in the mid 1970s. “A module library is the only kind of library that everybody wants to put something into, and nobody wants to take something out of.” Michael Jackson. Decades later, one company found that the complexity of generic services created a long learning curve for developers, and slowed down delivery.
The general lesson is this: you need governance, more governance and more governance still. The surprise is that people on the SOA bandwagon have taken so long to admit that governance is the key issue in reuse of shared services. And governance requires considerable expertise.
Oracle (2006) have this to say about web services
Service-oriented architectures (SOA) offer significant advantages, however they place additional demands on visibility, control, and overall governance.
Although enterprise SOA initiatives are generally deployed incrementally, issues around SOA governance must be addressed early in the implementation process to gain long-term value and maintain quality and consistency.
Generally speaking, SOA governance has to ensure that:
But also, the services are used where they should be!
Libraries and frameworks may both contain modules that automate technical infrastructure functions (e.g. transaction management) and repetitious tasks (e.g. generation of a data abstraction layer). Both can help developers focus on business-specific code.
Libraries and frameworks are sometimes treated as the same. For us, a framework is more than a service catalogue or class library. It is code that
The general lesson of frameworks is this: unless you have time and money to spare, keep it simple.
This challenge applies to both data in motion and data in storage:
- changing a canonical data model has an impact on all service consumer components who use that data
- changing a data storage schema has an impact on all service provider components who use that data.
Most of this section is edited from my contributions to an internet data management discussion group (DM-discuss).
Why are data services the hardest to change? Partly because the database contains persistent data, so changing the database structure can mean changing a lot of stored test data, as well as changing the code. But also, because the database encapsulates business domain knowledge which is rarely understood well enough to be stable before development.
“Scott Ambler (Agile methods guru) frequently complains about stick-in-the-mud database administrators, who obstruct the efforts of the programmers to be agile. No doubt some DBAs are difficult to deal with, but I think there is more to it than that. And it is only to be expected that every complaint levelled against DBAs now will be levelled against the maintainers of data-centric services in a future SOA paradise.
The database is a highly reused part of a system. It sits at the centre of a system, with many processes dependent on it. Perhaps the most desirable state of affairs is when one database can support many and various applications.
As Scott suggested in one message, a DBA might measure dependencies at the level of one table - by counting how many programs/Services/classes/Services refer to that one table. Perhaps the higher the number the better?
It doesn't take a genius to work out that more things (A, B, C and D) depend on a thing (X), the harder it gets to change X, since every change to X requires attention to A, B, C and D.
Forget the database / program distinction for a moment. Suppose each database table is encapsulated behind a Java class, and we don't even know what kind of database there is. Exactly the same dependency issue arises with the Java class.
The OO designer who is responsible for that class will soon start to acquire the awkward reputation Scott gives to the DBA, as that OO designer starts to resist requests for change and look to optimise their workload.
Why is that exactly? See “The challenge of maintenance” earlier.
Perversely, the more rapidly changes are thrown at you, the more you will want to batch them up, because it is likely that some changes will make others pointless, and you can therefore optimise your effort even further.
In short, the more code that depends on a table, the more change requests that are made of a table, the more it is in the interests of the table maintainer to store up the change requests and process them in batches.
Like Terry I guess, I have worked on a project where the several teams of Java designers and developers each developed a class diagram to serve their area of UI or I/O functionality. The Java guys had the project manager's ear. They deprecated the database. They insisted the database was nothing but an unintelligent persistence layer, a means to persist the objects of their classes, and the database design would flow naturally from their OO class diagrams.
Since all the UI and I/O functions were related to the same business domain, of course the 'entities' in one class diagram overlapped and conflicted with entities in other class diagrams. The class diagrams could not be reconciled into one data structure.
Again, the database sits at the centre of a system, with many consumer processes dependent on it. So, the database structure is the point where conflicts between different use views become apparent. And conflict resolution takes time and resources. Of course all players must communicate to minimize and resolve conflicts, but that does not remove the central point here, which is that these conflicts are best discovered and resolved through the medium of data analysis and data structure design. Making sure your team are trained in those skills might be the single most effective thing you can do to give them the common framework they need to preempt conflicts, spot conflicts and resolve them speedily.
Something of the Java guys' attitude persists in the otherwise excellent Martin Fowler's discussions of software layering, where he has lumped the database into a layer called "infrastructure". Surely infrastructure is that which is general to a wide range of systems regardless of the business domain? The database is normally the reverse, since it encapsulates that which is business domain specific.
In my view a data model is two things:
The database that follows from the data model also captures business rules. At the very least it features business terms and facts. It normally features also business constraints and derivation rules. (David Hay likes to remove the constraints to minimize database refactoring, but moving the constraints into code isn't necessarily a better idea.)
In the same project above, the 2 database designers simply could not get the 25 Java designers to listen; they could not keep up with the flow of data structure requirements coming at them.
Databases hold persistent data; programs are transient. This creates a huge imbalance in terms of refactorability. You can change a program as you like. You cannot change a table without changing the data that it holds. Or in a developing system, changing the test data you have created so far. The general lesson is that changing database structures is hard. And there is only so much you can do to shield consumers from this by placing a layer of data services around the database. So try to get the database structure right, and allocate quality resources to the effort to get it right.
There are other challenges. Our training courses mention four issues that appear over and again in software design
Any SOA or software reuse initiative must centre on creation and management of an enterprise’s business service catalogue - not on a registry of all the components the enterprise happens to have wrapped up as web services.
A personal and perhaps controversial view: A plausible SOA strategy has to be based on PALM
The later papers say more about challenges and how to address them.
References
Ref. 1: “Modular design patterns
and principles” in the library at http://avancier.co.uk
Ref. 2: “SOA myths and half truths” in the library at http://avancier.co.uk
Ref. 3: “What SOA should mean” in
the library at http://avancier.co.uk
Ref. 4: “SOA reference models” in
the library at http://avancier.co.uk
Ref. 5: “SOA Business Case and
Governance” in the Library at http://avancier.co.uk
Ref. 6: “Object-Oriented Design”
in the library at http://avancier.co.uk
Ref. 7: “Architecture Myths” in the Library at http://avancier.co.uk
Footnote: Creative Commons
Attribution-No Derivative Works Licence 2.0
Attribution: You may copy,
distribute and display this copyrighted work only if you clearly credit “Avancier
Limited: http://avancier.co.uk”
before the start and include this footnote at the end.
No Derivative Works: You may
copy, distribute, display only complete and verbatim copies of this page, not
derivative works based upon it.
For more information about the
licence, see http://creativecommons.org