IT Measurement is Still Immature

by Craig Symons

Executive Summary: Recently, Forrester asked IT decision-makers at North American enterprises about their IT measurement practices. Comparing the responses, we found that IT measurement is getting some attention; however, only 33% of respondents have a formal IT measurement framework in place. While progress has been made, there is still much work to be done.

IT Measurement Remains Immature

Based on recent inquiry trends, there is a growing interest on the part of Forrester clients around strategic IT measurement; however, a recently conducted survey of IT decision-makers to obtain a snapshot of IT measurement efforts revealed that only a third of IT organizations responding have implemented formal IT measurement frameworks. Respondents included CIOs, CTOs, CFOs, VPs, and IT directors and managers. A wide range of industries were represented, and 63% of the 134 respondents worked for companies with revenues of $1 billion or greater.

• IT measurement frameworks are not widely deployed. IT is under increasing pressure to demonstrate value from IT investments and to improve their alignment with business strategies. Implementing a strategic measurement framework is an effective tool for improving overall IT performance, improving IT/business strategy alignment, and communicating IT value to senior management and business unit clients. Despite this, only 33% of respondents indicated that they have adopted a formal IT measurement framework; however, 39% stated that they are currently working on developing a framework, and another 15% were thinking about it. Only 13% said that they had no plans at this time (see Figure 1-1). If all of the respondents can be believed, 72% will have measurement frameworks in place at this time next year.
• Improving IT performance and demonstrating value are the primary motivation. Improving the performance of the IT organization was the No. 1 motivator for implementing a measurement framework (53%). This was followed very closely by the effort to demonstrate value (52%), a CIO initiative (42%), and the need to reduce costs (40%). There are clearly no surprises in these responses. Of some interest was that 29% of respondents stated that creating more transparency into IT was a motivator (see Figure 1-2). This is in keeping with the general trend toward increased IT governance.
• CIOs take the lead. As you would expect, leadership of the IT measurement effort comes from within the IT organization. Fifty-six percent of the firms we spoke with have the CIO or CIO designate leading the effort, and another 24% are led by an internal IT team or task force. Only 1% of respondents said that the effort was led by the CFO (see Figure 1-3).

• IT measurement takes time. Implementing an IT measurement framework does not happen overnight. The largest number of respondents, 29%, said that their IT measurement framework took more than a year to develop, while 13% said it took from nine to 12 months, and 18% said it took six to nine months (see Figure 2-1).
• A variety of frameworks are in use. The largest number of respondents, 49%, developed an in-house custom measurement framework. Of established frameworks, 25% reported that they implemented an IT Balanced Scorecard; 13% implemented Six Sigma; 13% implemented ISO 9000; and 8% implemented EVA. An additional 12% used an outside consultant to design their measurement framework (see Figure 2-2). We were encouraged by the number of IT organizations that have implemented an IT Balanced Scorecard, as we believe it is the most effective strategic measurement system available.
• Multiple dimensions are measured. Most measurement frameworks are tracking multiple dimensions. IT operational metrics, such as availability and response time, are the No. 1 metric mentioned at 82%, followed closely by IT budgets at 80%, and customer satisfaction at 71%. It is important to note that these three categories are similar to three of the four IT Balanced Scorecard Perspectives. Other measures include business value at 50%; project ROI at 49%; human capital at 41%; and alignment at 32% (see Figure 2-3).

• Measurements are cascaded. Organizations that successfully execute strategy do so by creating a line of sight between the overall strategic objectives of the organization and the day-to-day activities of the individual employees. The most effective way to accomplish this is to cascade the goals and measurements down through the organization. The IT organizations represented by the survey have followed this principal to some degree. One-third of respondents cascade the measurements down to the IT department or team level, while 22% cascade them down to the functional level. When it comes to cascading measurement down to the individual level, 11% of do so. Only 14% stated that they did not cascade measurements (see Figure 3-1).
• Results are communicated regularly. IT organizations communicate their results frequently, with 52% of respondents communicating on a monthly basis and 41% on a quarterly basis. An impressive 8% reported that they communicate results continuously, implying that they have implemented a Web-based dashboard to report results (see Figure 3-2). With an increasing number of vendors improving IT management and reporting solutions, we would expect this number to increase rapidly going forward.
• The number of measures is limited. Forrester recommends that for effective performance management, the number of metrics must be kept small. Too many measures, and people become confused about what is really important. Respondents to the survey reflect this best practice with the majority (56%) using less than 20 metrics. The next largest category was 22% of respondents, who use between 21 and 30 metrics (see Figure 3-3).

More Work Remains To Be Done

IT measurement is important both to senior management and to IT itself. A good strategic measurement system can become a tool to transform the IT organization. Developing metrics that tie to strategic objectives, setting aggressive stretch targets, and implementing focused initiatives around each target is a proven methodology. For the 54% of IT organizations that are either working on a measurement framework or thinking about one, the payoff awaits.