Introduction

The pseudo science of RoI quantification

Costs of running systems

The benefits of running systems

Costs of a transformation

Other things to think about

Appendix 1: example costings

Income generated and/or money saved

Legislation complied with

Pain relieved

Stories worth telling

Measuring what actually happens

On the inaccuracy of numbers and the need for trust

What about less formal “Value of Investment” or VOI?

Data disintegrity pains

Stakeholder feedback

Anecdote and scenario

Business cases: cost and benefit numbers

This page is published under the terms of the licence summarized in the footnote.

Measuring Return on Investment (RoI) is not the science some would have you believe.

In long term, the Total Cost of Ownership (TCO) is often considered a critical factor.

But what about the Total Benefit of Owernship (TBO)?

This paper discusses cost and benefit numbers, and factors to be considered in presenting a business case (for EA or anything).

Introduction. 1

The pseudo science of RoI quantification. 2

Costs of running systems. 3

The benefits of running systems. 4

Costs of a transformation. 6

Other things to think about 7

Appendix 1: example costings. 9

The business case for deploying or changing a system may be based on compliance with regulations or a political imperative.

It may be based reducing a particular business risk or pain that managers are keen to avoid.

And if business managers perceive a project as making money, or saving money, then they will likely commit funding.

However, in practice, attempts to quantify an economic case are often more works of art than science.

Consider how you would quantify the benefit to your business of

· the company's email system.

· upgrading from Microsoft Office version N to Office N+1 (or to 365).

· upgrading your standard operating system from Windows N to Windows N+1 (or N+2)

· commissioning or decommissioning a large packaged ERP application.

How to quantify the benefit?

You might be able to measure current operational costs.

And estimate future operational costs after a new or changed system has been deployed.

This may enable you to present the benefit of a change as a cost reduction.

But evaluating systems purely on the basis of how much they cost is naïve.

See “Can IT cost cutting go too far?” on the enterprise architecture page at avancier.website.

How quantify benefits arising from

· role and process enablement or improvement

· data qualities: confidentiality, integrity or availability

· compliance to regulations, or company or industry standards

· avoidance of unquantifiable risk?

Generally, the more strategic the decision, the less convincing a quantified business case is.

That doesn't mean the decision is wrong; it means that it is really political rather than economic.

Architects define and govern a transformation of business, data, applications and technology systems

· from the current, presumably chaotic or otherwise unsatisfactory, state

· to a future, rationalised, better, state.

So in theory, we can measure Return on Investment in terms of the differences between baseline and target systems:

1. Measure costs and benefits of running baseline systems: subtract costs from benefits

2. Estimate costs and benefits of running target systems: subtract costs from benefits

3. Subtract 1 from 2 to give the benefit of the transformation

4. Subtract the cost of the transformation from 3.

This table illustrates the process using example numbers.

	Baseline over 5 years	Target over 5 years	Transformation over 5 years
Benefits of running systems *	88m	99m
Costs of running systems	77m	66m
Benefit - Costs	11m	33m	22m	Gross benefit of transformation
			11m	Costs of the transformation
			11m	Return on Investment

* Note the potential for confusion between benefits and cost savings.

Make sure you know whether you are supposed to apply a Discounted Cash Flow formula or not.

The formulae in this process for calculating Return on Investment depend on three elements discussed below:

costs of running systems,
benefits of running systems, and
costs of a transformation.

How much should a business spend on IT? Industry variation is large.

Some businesses (e.g. entertainment businesses, a theatre, a football club) depend little on IT.

Some businesses are fundamentally about data processing, and their IT spend is high. Notably:

· Banks and other financial institutions. Some have staggering throughput, response time and availability targets.

· Telecommunications companies depend on IT devices and networks.

· Education now involves much IT use.

· Central government departments tend to have unique data processing systems

· Local authorities in the UK do much the same business, and systematically compare IT costs with each other (it is less clear how they compare IT benefits).

· Large retailers. Tesco’s Business Intelligence and Data Warehousing systems are reputedly key to the success of their logistics, and so are key intellectual property. (Though note that their profit comes more from treasury operations than sales.)

The costs of running systems can be divided in various ways.

The total cost of ownership (TCO) covers acquisition costs (capex) and operational costs (opex).

Within a few years, the latter usually outstrips the former.

Both of these costs can be divided between

Technology resources: client devices, server devices, network technologies, software licences etc.
Human resources: technical architects, operators, administrators, managers etc.
Environment: business/user environments, data centres, disaster recovery sites etc.

It is common to start with the costs of hardware, software and network technologies, then add the people and environment costs.

There is another consideration. Downtime costs money.

If critical applications can fail, the mean time to recovery becomes an issue, and you might consider including down time costs (the absence of benefits) in cost calculations.

A later section discusses complexity and downtime costs.

Appendix 1 shows some example costs, of the kind documented by an IT service provider when costing and pricing a service to an enterprise.

The IT department is often perceived as a cost by the rest of a business. In many cases, the annual bill for IT services is concrete and clear.

In most cases, the benefits provided by IT are less clear.

The result of this natural emphasis on cost is that the IT department is more often asked to reduce costs than to increase the benefits.

And where the benefits of IT are not presented separately, then people generally assume that benefits = costs.

However, it is possible to look at benefits separately.

This table classifies benefits, starting with the ones business managers are most likely to recognise and respond to.

How to measure the benefits provided by supporting applications, developing applications, upgrading technologies and pursuing improvement projects?

Some applications are vital for business success. They can be important to:

Business nature. E.g. the telecoms industry is driven by software.
Business competitiveness. E.g. in the finance industry, the movements of money, stocks and shares must be fast and continuous.
Business income. E.g. a billing application. It is conceivable that clerks could be employed to do the same job, but not realistic.
Business profit. E.g. a combination of billing and payment applications. The payment system can make money by paying suppliers the minimum amount at the latest date.

But how to measure their value?

Consider income generated and money saved, noting that it is not always easy to separate them.

Income

Some applications do create income (e.g. a stock trading system should do that).

They do things that a clerical system could not do with the required speed and reliability.

Some applications don’t create income, but they collect money for the delivery of some other product or service.

These save the cost of using people, pen and paper to do the same.

Savings

To measure savings implies being able to cost the alternative – of NOT having applications, of NOT pursuing projects.

When information systems were first computerised, this meant costing the clerical data processing staff time no longer needed.

The picture nowadays is a lot more complex.

Some applications save a very specific cost.

For example, one global logistics company saves millions of dollars in terms of transport time.

Hey do this by using special right-hand-turn truck routing software, which was commercial secret for some years.

Many EA projects are intended to save the costs of duplications:

Duplications of technologies, licences and operational costs.
Duplications of data entry, storage and update.
Duplications of processes, in different variations, and management of those processes

You can reasonably hope to measure the costs of redundant databases, applications and infrastructure technologies.

Remember the duplications involve not only technology resources, but also human resources and environment resources.

Some applications are required by legislation. This is especially true in government departments.

This may appear to be a non-quantifiable benefit.

However the savings made by IT systems can be still expressed in the traditional way.

That is, by comparison with the costs of creating a new clerical data processing organisation to implement the same legislation.

Sometimes, hiring clerks is cheaper!

Your business case can mention the costs of failing to comply with legislation, ranging from fines to the imprisonment of directors.

That is one kind of pain relief. Others are considered below.

Often, managers have to make decisions based on what they hear and see by way of problems and inefficiencies.

Look for the pains the business suffers from. Make sure managers recognise these pains.

Try to cost them. Set out to remove them. For example, consider legislative penalties and data disintegrity.

Many EA programmes are intended to save the costs of data integrity problems

Data not being correct: integrity errors and the resources used to correct those errors.
Data not being available to the right people, in the right places at the right times.
Data not being confidential, secure

The benefits of integrity reflect the costs of disintegrity.

Costs include:

the penalties for making mistakes on the basis of poor information,
the time and resources needed to

copy data between systems,
translate data between different formats,
find errors and correct them and
change several applications when a data item or processing rule changes.

One local government CIO measures the cost of data integrity errors, catalogues the risks, and uses this evidence to get buy-in from business managers to his EA work.

Managers and decision makers are reasonably influenced by opinions and anecdotes.

Locally, the decision to develop or deploy a new application is often a judgement call made by a manager.

Across the enterprise, the decision to employ enterprise architects is just as likely to be a judgement call made by senior managers.

So look to support these judgement calls.

Ask for and record the feedback and perceptions of your stakeholders.

Ask not only the top-level board member stakeholders, but also the lower-level stakeholders.

If somebody appreciates the contribution EA has made to a project, by way of direction or review, write that down.

Use surveys to quantify perceptions and feedback where you can.

It often helps to paint pictures and tell stories as well. So identify the business risks and pains caused by disintegrity and disinteroperability.

Paint pictures of these using examples and business scenarios. These stories and pictures often speak louder than numbers.

I have found the pain caused by a few data integrity errors can be enough to convince managers to allocate budget to correction work.

Estimating transformation costs is the province of conventional project management methods and not discussed in detail here.

Note that the costs of a transformation programme or project include the costs of:

Development and deployment: technology resources, human resources, project environment.
Transition: deployment of systems in the business environment, retraining of employees and other changes.

And that human resource costs should account for:

· recruitment costs

· employment costs (salary, benefits and equipment)

· availability (work days excluding holidays and sickness).

If you have a baseline measure (e.g. "last year we sold N million, with a 23% margin) then do periodic reviews (this year, we sold N + 27%, with a 34% margin).

Then you have numbers with which to measure the costs and benefits of work you do.

Demand 6 month and 1 year benefits reviews.

Look for cost savings in operations.

Look for benefits in sales.

Turn IT things into internal profit centres.

What about multiple causes for a benefit? During the year past you did

a business project (sales force training),
an IS project (a new CRM system) and
an IT project (server consolidation).

How much did each contribute to your success? Can you use 1950's time-and-motion Ford-era methods or newer Six Sigma VOC metrics?

The important thing for convincing people is to plan success metrics before the project, and stick to them.

Do present numerical business cases. And (Nic Harvard recommends) once you have presented numbers, stick to them.

But numbers are rarely reliable enough to be all and end all decisions.

Business managers want a business case with numbers in it. But they don’t want inaccurate numbers.

Unfortunately, EA is about the large, the wide, the long term.

Distance bring imprecision. The margins of estimating error are wide. There are many uncertainties - unquantifiable factors – unknowns.

Inaccuracies in system cost numbers

You can probably collect numbers for the current costs of client devices, data centers and application maintenance within the IT department.

Beware that sharing of resources complicates the picture.

Business units may share IT resources and costs.

The IT department may share an environment and its costs with other business departments.

System usage cost

Don’t forget that system usage has a cost, that poor systems waste time in unproductive activity.

Are you able to measure the business time wasted by log in and security procedures? lost passwords? Poor user interfaces? Poor quality data? Unavailable data? Application down time?

You might be able to.

Inaccuracies in system benefit numbers

The benefits of running baseline systems (as opposed to not running them) are rarely quantified.

People usually assume the benefits are equal to or greater than costs.

You might hope the benefits of target systems can be quantified.

The simplest case is where the target system removes elements of the current system – along with its costs.

But EA is not only about rationalisation by removal of redundancy.

And it is much harder to estimate other kinds of benefit.

It is generally hard to measure the benefit of computer applications to people executing business processes.

How do you value your use of email?

What is your comparator? The absence of your email application? The use of an alternative communication system?

Proper comparisons require painstaking time and motion studies of a kind that are rarely carried out.

Inaccuracies in transformation cost numbers

At the tactical level, much research has been done on small to medium software development project costs.

Grant Rule of Software Measurement Services reports that initial cost estimates have an error margin (compared with actual costs) of plus or minus a factor of four.

(Of course, the “cone of uncertainty” narrows as the project proceeds, and you can count actual project costs accurately.)

Good estimating practice involves dividing a big task into smaller shorter tasks.

The reasoning is that you can estimate the small more accurately than the large.

So at the start of a strategic EA programme, is it reasonable to expect that estimates will have a smaller error margin than for a single project?

The costs and benefits of running target systems remain inaccurate estimates for a long time.

Years later, when the target systems are running, and some benefits have been achieved, the numbers and the contribution made by EA to them will remain debatable.

Predicting the benefits of better systems integration – some years ahead - and for EA as a whole - is far from being a science. And the cone of uncertainty narrows but slowly.

It is possible to over-engineer the numbers in a business case, to present too many numbers, to present numbers that don’t stand up to inspection.

Beware presenting numbers that purport to be accurate to business manage who know there is uncertainty.

Of course quantified business cases should be offered for transformation efforts.

But numbers cannot be predicted accurately and completely enough for decision making to be a wholly scientific process.

There has to be some trust. An enterprise depends on the experience and wisdom of its managers to make good decisions, based on evidence of various kinds.

That’s why stakeholder management and gathering of anecdotal evidence are vital to enterprise architects and anybody who hopes to engineer transformational improvements over a long time.

If you can measure the VOI, then it turns into the ROI.

If you cannot measure the VOI, then you are left with trusting that the decision makers exercise good judgement about the future value to be gained from investment made today.

And you need evolutionary pressure from market forces (voters, customers) to weed out the poor systems and poor decision makers.

In theory, Enterprise Architects work with top-level decisions makers, and provide the scientific rationale for large-scale transformation proposals and migration plans.

In practice, top-level managers make speculative and big-risk decisions - often without reliable information.

Business cases are manufactured to suit the power and politics of the situation.

Sometimes a decision works out in the long run, sometimes it doesn't. Survival of fittest applies - more or less.

Eventually, the board, shareholders (and to a lesser extent, employees) provide the evolutionary pressure to remove a top-level decision maker who gets it wrong.

Even random decisions will generate a good VOI for the end consumer if there are competing systems and evolutionary pressures to kill of the bad systems.

The trouble arises when there are no competing systems and no way to kill of the bad ones, or the remove the bad decision makers.

Valuing any kind of infrastructure improvement is difficult. You may notice that roads in the USA are badly maintained.

Also road design and road signage are pretty dire.

Surely this is mostly for the lack of good enterprise wide (federal government) standards for junction approaches, sign sizes, colours and locations?

Now there is a role for the enterprise architect.

One time

Year 1

Year 2

Year 3

Year 4

Year 5

Hardware and software

Network

People

Environment

Down time (loss of benefit)

The examples below (2008) are intended only to illustrate the kind of information involved in costing and pricing an IT service.

People Costs	One Time	Year 1	Year 2	Year 3	Year 4	Year 5
C&SS LAN WAN Support	£900	£0	£0	£0	£0	£0
Data Centre Ops Support	£350	£0	£0	£0	£0	£0
Oracle DBA Support	£0	£0	£0	£0	£0	£0
Storage Support	£0	£0	£0	£0	£0	£0
UK Tools Support	£480	£598	£598	£598	£598	£598
UNIX (AIX) Support	£3,200	£11,968	£11,968	£11,968	£11,968	£11,968
WebSphere & MQ Support	£2,400	£7,181	£7,181	£7,181	£7,181	£7,181
WSSU Support	£0	£0	£0	£0	£0	£0
Oncall Rotas	£0	£0	£0	£0	£0	£0
Management
Service Manager	£0	£0	£0	£0	£0	£0
Technical Services Manager	£1,650	£0	£0	£0	£0	£0
Service Level Manager	£0	£0	£0	£0	£0	£0
Configuration Manager	£0	£0	£0	£0	£0	£0
MO Project Manager	£2,250	£0	£0	£0	£0	£0
Change Manager	£0	£0	£0	£0	£0	£0
Capacity Manager	£0	£0	£0	£0	£0	£0
Problem Manager	£0	£0	£0	£0	£0	£0
(24x7 desk) Incident Manager	£0	£0	£0	£0	£0	£0
TOTALS	£11,000	£20,000	£20,000	£20,000	£20,000	£20,000

People Costs	Day rate	Annual rate (190 days)
Service Manager	£550	£100,000
Technical Services Manager	£550	£100,000
Service Level Manager	£450	£80,000
Configuration Manager	£450	£80,000
MO Project Manager	£450	£80,000
Change Manager	£350	£60,000
Capacity Manager	£350	£60,000
Problem Manager	£350	£60,000
(24x7 desk) Incident Manager	£350	£60,000

Hardware & Software	One Time	Year 1	Year 2	Year 3	Year 4	Year 5
SFE2 DL 380 Server	£5,444	£307	£307	£307	£307	£307
Red Hat Linux Enterprise	£763	£763	£763	£763	£763	£763
CA NSM and Performance	£760	£127	£127	£127	£127	£127
Footprint for server	£0	£222	£222	£222	£222	£222
Power (KVA)	£0	£717	£717	£717	£717	£717
Training budget	£4,000	£0	£0	£0	£0	£0
WebSphere Application Server (version 6.1)	£5,552	£1,296	£1,296	£1,296	£1,296	£1,296
Wily Introscope	£3,000	£800	£800	£800	£800	£800
TLS KeyStore and Certificate	£500	£0	£0	£0	£0	£0
TOTALS	£20,019	£4,232	£4,232	£4,232	£4,232	£4,232

Hardware & Software	Information	Quantity	Capital Purchase	Annual Support
SFE2 DL 380 Server	Equanet Quote H015498, annual support cost is 24x7 4-hour fix	1	£5,444	£307
Red Hat Linux Enterprise	Linux operation system license	1	£763	£763
CA NSM and Performance	CA Unicentre agents (Tier 2 server) license	1	£760	£127
Footprint for server	Data Centre space for SFE1 server	1	£0	£222
Power (KVA)	Power requirements for SFE1 server	1	£0	£717
Training budget	Wily deployment training (knowledge spreading)	1	£4,000	£0
Wily Introscope	Wily Introscope license for SFE2 server	1	£3,000	£800
WebSphere Application Server	WebSphere licences (2 CPU) for SFE2 via Cerner	1	£5,552	£1,296
TLS Keystore and certificate	TLS endpoint for SFE2 server	1	£500	£0

DL 380 Server Spec - ACP-EBS-PR-SFE2	Qty	Unit	Total
HP ProLiant DL380 G5 High Performance – Rack - 2-way – 2 x Dual-Core Xeon 5160 / 3 GHz – RAM 4 GB – SAS - hot-swap 2.5" – HD: none – CD-RW / DVD-ROM	1	£3,000	£3,000
146GB 10K SAS 2.5 HP HDD	4	£200	£800
HP NC110T PCIe Gigabit Server Adapter	3	£500	£1,500
HP 8X Slim DVD+RW Drive	1	£100	£100
HP DL36X PCI-X Conversion Kit	1	£50	£50
			£5,450

Footnote: Creative Commons Attribution-No Derivative Works Licence 2.0 02/11/2014 19:0802/11/2014

Attribution: You may copy, distribute and display this copyrighted work only if you clearly credit “Avancier Limited: http://avancier.co.uk” before the start and include this footnote at the end.

No Derivative Works: You may copy, distribute, display only complete and verbatim copies of this page, not derivative works based upon it.

For more information about the licence, see http://creativecommons.org