Useful Measurements in an
Application Portfolio Management

Michael Oara

CEO, System Renewal, Inc.

Application Portfolio Management (APM) is recognized as one of the new disciplines that help aligning business and IT. APM gives the macro level view of all software applications used in a company, helping understand the costs vs. the benefits that each one provides. It is an indispensable tool in IT planning, helping to invest in those projects that would best benefits the company’s needs and long term objectives.

As any discipline based on the handling large amount of information, it benefits directly from a high degree of automation. In this case, the automation does not refer solely at the high level charts and reports, which support the IT decisions, but also at the more difficult process of gathering row data. This row data may be collected from a number of sources, such as software code, production incident files, performance logs or request logs. Software tools for APM are – or should be – able to use all these as sources of data. In fact, an APM tool should also be capable to incorporate – as automatically as possible – data coming directly from the business itself. As this data often sits in the same databases that the IT department manages anyway, so access to it should not be a problem.

APM should thus be an instrument to gather heterogeneous measurements from a variety of sources and organize and synthesize it such that the IT and business managers can have a clear view and make intelligent decisions. These measurements may be classified in three categories: technical metrics, stakeholder metrics and other business metrics, as they are defined bellow.

Technical metrics

In this category we place the metrics that describe intrinsic characteristics of and application. The static analysis of an application can reveal a number of measurements relevant for APM. These include

Size – that can be measured in various ways.

Perhaps the most familiar and easy to understand is Lines of Code (LOC), however other characteristics may help get a better understanding; one may look, for example, at the number of programs, screens, reports or files.
Complexity – could be measured as a composite of complexities of individual programs or of the application itself as a whole. In general complexity measurements are based on various characteristics of a graph, like, for example, the graph of decision points in a program connected through execution paths. Complexity measurements are not easy to derive and they require the usage of specialized static analysis tools (a number of such tools are currently commercially available)
Function Points – which describe the size of the application not from the technician’s point of view, but from that of a user. This measurement was introduced in the 60’s by Alan Albrecht of IBM. Function points do not care about the number of size of the technical artifacts, but the amount of functionality provided to the user of the application.
Technology mix – describes the relative usage of various technologies in an application. One may be interested in the usage of relational databases vs. flat files, or Cobol vs. Java. It makes sense to split this into at least three separate categories, for data (example: flat, relational, hierarchical, network), programming languages (Cobol, PLI, C, Java, HTML) and interfaces (3270 screens, HTML). The percentage of each one of these technologies could be an important factor in modernization decisions.
Connectivity – describes the degree to which one application is dependent of or uses assets from other applications.

Stakeholder metrics

As any application is built and run for the benefit of various business entities in an enterprise, it is important to take a point of view that is relevant to them. This could be accomplished by combining their own internal business metrics with application metrics.

A large part – but not all - of the business data relevant for APM is available from the various databases maintained by the IT applications. This is true in particular about financial data, which gives a good idea about the importance of various company operations. At a consolidated level, such data may reside not in the databases but in individual spreadsheets or other documents maintained by business people, which makes it harder to collect.

Some data or measurements, by its own nature, may not be at all recorded by any application managed by IT. This is the case of data external to the company (like, for instance, market prices) or company internal data reflecting planning and priorities.

As business data may describe so many aspects of the business, it is impossible to give a fix list of business indicators; they should rather be selected based on both availability and relevance for APM.

Other business metrics

We can look at this as sitting in-between the other two categories and describing the running of the application in the business context. We include here information collected directly from the running logs and from the business/IT communication on the subject of the application. Unlike the intrinsic technical metrics, which refer to the application artifacts, these “other business metrics” reflect the way in which the application performs and fulfils its mission.

Database size (number of records) – for each one of the important databases
Database usage (number of update, inserts, deletes, reads)
Transaction volume: transactions per day or per hour
Transaction performance: elapsed time for each type of transaction
Production incidents (per week, month or year)
User requests backlog

Although most of the data described above is recorded one way or another, it may reside in a number of heterogeneous files or databases and in most cases an explicit effort is required to collect it.

Correlation levels

As application (static or dynamic) data is correlated with business data, one needs to find the right level at which this correlation is made. There are two issues here: granularity and boundaries.

Granularity

The information gathered, processed and displayed by an APM system could be as granular as the effort to collect all significant detail of data. On the business side, it could go down from the level of the whole company to the smallest business units. On the IT side, it could start from the collective size of all applications and descend to the low level of technical artifacts. One may look at indicators as the ratio of all function points of all applications to the total size of the company, and compare it with the same ratio in other companies to draw o conclusion about the automation level offered by IT. On the other side of the spectrum, one may look at the impact of a particular transaction performance on the productivity of a small team that sells a particular product.

Ideally one should use some zooming paradigm, which would allow starting with the big picture and navigating from there to lower levels. In such a scenario, the user would start by analyzing the problems faced by a particular large business unit and zoom into the software applications supporting it, until an individual program is located which shows an unusual complexity and propensity to defects.

Boundaries

While on the business side the boundaries of individual units are usually quite clear, on the IT applications side this may not be the case. The reason is that in many cases there are no simple one to one correlations between business units and IT software applications. An application could be shared by many business units, while the same business unit may use a number of software applications. To complicate this even further, one business unit may rely on a precise component of an application, not caring about the other components.

Finding the exact boundaries of an application is less trivial than it seems at the first look. Software artifacts have intricate dependencies and experience shows that even for the people responsible to maintain them it is not easy to come with a precise inventory of all objects included or required. Often some static analysis tool is required to come to an exhaustive and precise classification of all technical artifacts into the proper applications to which they belong.

Even if the application boundaries are clearly designated, this may not be sufficient for a complete and useful APM solution, which would actually require classifications of IT software assets into multiple dimensions. If one considers a simple technical artifact, as for instance a program, this may be classified according to some of the following dimensions.

Application - usually a set of software artifacts that are employed together to perform a particular function, and for which a single IT team is responsible.
Data center – where the program is usually run.
Development and maintenance location – this is particularly important if the maintenance and development is performed at various locations, including offshore.
Business function to which the program contributes.

To accomplish such a multidimensional classification, a solution, conceptually simple but technically difficult, may be employed. The solution is to provide a way to freely tag the software artifacts, in any way that is found useful. The same program then can be tagged as belonging to the Inventory application, as being run from the Chicago data center, as being maintained in Bangalore and as providing a reporting function.

Tagging mechanisms are particularly important in order to impose the correct business perspective on the large amounts of data available in APM. Consider the following example. The Congress or the Administration may impose a certain rule about data security or privacy, for certain categories of data, as in the case of HIPA regulations in the Health Insurance field. Which data or function is affected by the regulation? It is perfectly possible that the technical artifacts have nothing in themselves to indicate if they are or are not subject to HIPA. IN this case, one may manually “color” some application assets with a “HIPA” tag. Now the APM diagrams or reports could indicate, for example, the percentage of the application affected by the regulation. APM could also compute the complexity of affected assets and even give an estimate of the work required in order to bring them in line with the regulation.

Putting it together

The power of APM tools comes not only from showing individual aspects of applications, but also from the capability to correlate these with business measurements. This can reveal interesting facts and help align the IT objectives with the business priorities. We list here a number of such possible correlations, fully aware that this list could be adapted or enlarged depending on what is considered to be relevant.

Starting with the three categories of metrics presented at the beginning of this article, we can also create various types of correlations. The series of tables bellow indicate the correlations as well as the meaning of high values and low values for the corresponding ratios. Many of these ratios make sense only in a relative sense, compared with the same ratios for other applications, development teams or business units.

Technical metrics

Ratio	Low value	High value
Function Points / Complexity	Poor architecture or code, which may require clean-up, refactoring or modernization	Excellent architecture and good coding practices
Team size / Function Points	Low team productivity	High team productivity
Percentage of technology X / percentage of technology X specialists	Low productivity or excessive number of programmers	High productivity

Technical metrics correlated to Other business metrics

Ratio	Low value	High value
Number of production defects or incidents / Complexity	Excellent architecture and good coding practices	Poor architecture or code, which may require clean-up, refactoring or modernization
Team size / Size of user requests backlog	Low team productivity	High team productivity
Percentage of technology X / percentage of technology X specialists	Low productivity or excessive number of programmers	High productivity

Technical metrics correlated with Stakeholders’ metrics

Ratio	Low value	High value
Number of production defects or incidents * Business cost per runtime defect	Acceptable level of quality	Unacceptable level of quality
Number of IT supported processes / Total number of business processes	Low degree of automation	High degree of automation
Importance of a business function / Planned expansion of applications supporting it	Poor allocation of IT resources	Good IT priorities

All formulas in the examples above may be calculated at various levels in the hierarchy of business units or IT applications. They become more relevant when the results are visually displayed or reported together for series of parallel business units or applications, such that comparisons could be easily made and Business/IT misalignments could be detected.

Beyond measurements

Although measurements and other values derived from them are very useful and easy to process and report, certain issues could only be discovered through a structural, rather then numerical analysis. Other disciplines, such as BPM, are capable to present an integrated IT/business view.

Here are two examples of misalignments that would require a structural rather then a numerical analysis.

Example 1: Duplication of functions

Two loan departments of a bank, A and B, use different processes but execute a similar task, the determination of credit worthiness of an applicant. This may happen for historical reasons, for example because the two departments originated from two separate banks that have merged. It makes sense to create a unique implementation of credit worthiness determination and present it in the form of a service provided by one of the applications and consumed by both.

Example 2: Lack of integration

During a particular business process, an operator executes the tasks A, B and C, in this particular order. However, A and C are supported by one application, while B by another. The operator needs log in the first application, execute action A, write down some results, log off the first application, log in the second, execute action B, then log off the second application and go back to the first application to complete the process.

We expect that such new disciplines and tools as APM and BPM will evolve to provide an increasing integrated view of business and IT. This will continue the trend of pushing the IT from a simple supporting role to that of an active participant in all business activities of a corporation.