The Service Management Soapbox

One aspect of IT management automation that has frustrated me for years is the relative lackluster progress we’ve made in the area of performance. With all the other wonderful innovations we’ve made, I’m confounded that performance remains largely in the dark ages. This need not be. This must not be. We are finally beginning to see welcome changes in both available software solutions and more importantly, attitudes toward performance. If we hope to attain true service management, this must be a priority.

The world’s requirements have shifted to a point where performance, done right, has become a necessity. The operative phrase here is done right. Many IT operations are stuck in the obsolete mode of approaching performance from a myopic perspective of the infrastructure and often based on arbitrary static thresholds.

I wrote a META Group paper in June 2003 entitled Performance Monitoring versus Performance Analysis that echoes much of what I state here. Despite the nearly 5 years of elapsed time since, little has changed. Many sit here in the genesis of 2008 and unfortunately still view performance through a 1998 lens. Vendors have been part of the problem, shipping products that do little more than collect raw metrics and generate reports. It is time to finally make the change … and we are. With more attention on best practices such as ITIL, the philosophical shift is finally happening. It is also coincident with a similar evolutionary step in automated management technologies.

Let us first examine the available reporting products and how they are changing. Performance tools fall into one or both of two categories: reporting and analysis. Reporting is the fairly simplistic category of tools to which I just alluded. Collection and reporting proved valuable in the early days (1990s) when we had no visibility at all. Our capacity challenges were centered around infrastructure (e.g., network links, servers) and our best alternatives were streams of raw textual data. The pretty graphs generated by performance reporters were a big step forward. Today, however, this type of reporting is so commoditized that some of the most popular tools are from the open source community (e.g., MRTG, Nagios).

Simple reporting is also very misleading. When thousands (maybe millions) of metrics are imposing varying degrees of influence on service quality, how can any level of manual review be possible? Indeed it is not possible, so the only alternative is to collect all the metrics and analyze only a few. This analysis is often only performed when trouble arises. Although such situations occur more often than they should, the likelihood that collected data is viewed, used, or otherwise accessed is very low. It turns out that over 95% of collected data is never touched. The remaining <5% gives a limited perspective of service health, but this subset drives a lot of decisions. When decisions are based on such a narrow view of the world, trouble is inevitable.

Automated performance analysis is where the real action lies. The best tools should encompass a wide swath of data, determine the relationships between the data elements, and process the data based on some admittedly esoteric mathematics. This is not easy, but it is necessary and a few vendors are stepping up to the challenge.

Two of the most promising vendors in the performance analysis market are Netuitive and NetQoS. Visit their web sites for much more than I can describe here, but the excitement I feel from both is because they are tackling the thorny issues of analysis with some genuinely cool technology based on some deep cerebral concepts that will make most of our heads spin. Yes, this is complex, but it is the responsibility of vendors to insulate the complexity and make it all seem easy. This is what Netuitive, NetQoS, and a small number of other innovators are doing. So far, the established vendors are mostly languishing in the old world view. It’s time for them to wake up and either innovate themselves, or go hunting for one of these smaller mavericks. BMC recently took the bold step to acquire Proactivenet. This was a good move and indicative of the future.

One of the simplest mechanisms, single-variable statistical baselining is finally becoming commonplace, even among the stodgier vendors. This form of baselining establishes a statistical pattern of normal behavior and determines when performance is straying outside the bounds of this normality. Being single-variable, it only focuses on one metric at a time, so it can monitor CPU utilization for a pattern, but it ignores the impact of memory swapping or network I/O.

Still, this is a huge improvement over static thresholds. Static thresholds are useful for a very limited set of metrics (e.g., application response time as defined in an SLA). For most other metrics, however, static thresholds are not just inadequate, but misleading. I expand on this more thoroughly in a presentation I delivered at the 2005 Computer Measurement Group conference. The bottom line is that static thresholds can generate lots of false alarms and ignore valid performance anomalies that lie below the threshold (or above, depending on the nature of the threshold). A good example is CPU utilization. If I set a threshold to alarm above 75%, it will alarm at 85% during a time when normal behavior is in the range of 80 to 95%. Conversely, if the CPU is at 20% during this time, it is an anomaly, yet a static threshold will ignore this situation and fail to notify.

Where analysis gets truly exciting is when we can extend beyond the single variable to determine the interrelationships between multiple variables. This is hideously difficult, but the innovators I mentioned earlier are now delivering such capabilities. These technologies will continue to evolve, as we are now only experiencing the infancy of automated analysis.

It doesn’t make sense to struggle with manual analysis based upon simple reporting. The world is getting far too complex for this mode. Let automation do the hard work for you. The technology is finally maturing to a point where it can offer great relief to the increasingly painful performance problems that plague most IT organizations.

Analysis has proven effective for fault management (evaluation of up/down conditions), but performance is a different animal. Whereas fault management deals with binary conditions of black and white, performance involves the full pallet of colors and shades of gray. Of course, dealing in colors is much more difficult than black and white, but help is now here.

In this age of HDTV, it’s a bit absurd that we still view performance in black and white!

This entry was posted on Wednesday, January 23rd, 2008 at 20:35 and is filed under SLM, ITIL, General ITSM, Management Tools, ITIL v3. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.