The HR Idea Lab: Analyze the Analysts

A few weeks ago I declared that I would be giving away my unused ideas in a series I refer to as The HR Idea Lab. The simple concept is to document my thoughts surrounding potentially viable opportunities and let you all run with them as you see fit. This will hopefully stimulate some interesting dialogue and perhaps provide some of you with a spark to creating a new venture. Or perhaps this will be a horrible and ill-conceived notion that flames out in a dramatic and humiliating fashion. No matter – I’m good either way.

Okay, so on to the inaugural HR Idea Lab — Analyze the Analysts.

The Problem

Regardless of the market segment, industry analysts serve as a critical force in rationalizing the construct and constitution of a given space as well as measuring the current and projected growth, the service provider landscape and the likely sustainability of its value proposition to the end users it intends to serve. We purchase their reports, attend their briefings, seek their guidance and generally assume that they are credible sources of unbiased advice and accurate metrics.

The problem sits in our assumption that all analyst firms – and as a subset, all analysts themselves – are created equal. Rarely do we take the time to double-click into their underlying assumptions, deconstruct their market data and CAGR projections, seek transparency into the financial relationships with the very same firms they rank and rate, or comprehend the confluence of factors that underpin their preferences and bias.

The service providers in a given segment face the quandary of attempting to cajole and influence these influencers to attain pole position for their firm while simultaneously using these same analyst market sizings and future projections to build budgets, seek funding and assess their respective market share. The buyers/end users use these same metrics to assess which service providers are most appropriately matched to their needs while determining whether a particular concept is just a fad or one that’s attained mainstream adoption (and therefore worth investing precious dollars and resources to pursue).

It’s a complex, messy and highly interconnected web that lacks independent oversight and an objective rating system.

The Opportunity

As the problem setup implies, the opportunity is for an unbiased third-party to establish a transparent mechanism to rate and rank industry analysts across a wide variety of metrics. Rating systems are not a new concept yet would be new if applied to the industry analyst community.

One existing entity who applies this construct to investment funds is Morningstar. Their rating system is one of the most commonly leveraged mechanisms for fund assessment in the world (and you can read up on how their establish their ratings by clicking here). My suggestion is to leverage this example, and others, to establish a well-documented and common set of rules in your rating of the analyst firms. Let me give you an example.

Given how important an analyst’s market sizing is to a given space, one could envision a mechanism whereby you compare an analyst firm’s compound annual growth rate (CAGR) projections against actuals for the segment. There are two ways to accomplish this; either, 1) you would calculate your own market sizing at the end of a given period and compare the analysts’ projections to your own; or 2) you would simply compare analyst projections against their own firms actuals for the following year. I think Option 2 is an easier place to start (although less accurate) because you’d basically be comparing two easily accessible numbers from published reports. Here’s how it could work:

[EXAMPLE ONLY] Analyst Firm X projects that the market for talent management will be $10B in 2011 and $13B in 2012, with 2010 estimates at $7.5B. They publish these numbers in their Fall of 2010 talent management study. In Spring of 2011, they release new numbers which show that the 2010 market size was actually $6.8B and the projections for 2011 are now $9.5B (and so on). You would give them an accuracy rating of 91% for 2010 and flag them for an adjustment of -5% for 2011 projections. You would do this every time they publish on the segment.

The Challenge

Comparative analysis for their own CAGR projections are relatively easy, but what about uncovering their inherent bias? And how do you force them to disclose the financial relationships with the service providers they study and rate? These are much more complex issues and the only means of attaining that level of detail is if the buyer/end user market looks to you as a trusted independent entity and effectivity forces the analyst firms to play ball. Alternatively, you may find that certain analyst firms love this idea because they have confidence in their ability to be beyond approach and they will therefore “score well”. They would welcome a third-party firm to expose their competitors for the frauds they are, right??

Don’t underestimate the complexity of this endeavor and the lives and reputations that would be at stake. The right management team with a properly constructed value proposition and very clear rules of engagement could overcome this resistance and serve a valuable role in the community, but don’t kid yourself that this would be easy.

How You Make Money

This is the trickiest part of the equation and one that the analyst firms themselves constantly wrestle with, namely how you take money from the same organizations you are analyzing. This would require some additional thought as you could see how well-performing analysts would want to leverage their “5 Star Rating” in their literature, on their websites, during briefings, etc., and how you could monetize their ability to do so through a subscription or per-instance licensing agreement.

Another alternative is to create an annualized membership fee that various stakeholder groups (venture capitalists, hedge funds, service providers, clients/end users, etc.) would pay in exchange for access to your rankings and ratings. Once again, the monetization model would need to be crisp and well documented so that no one could accuse you of colluding through back-channel financial arrangements.

Thoughts?

So that’s it for the first HR Idea Lab. What do you think?… is this a good idea and one the market would value?… should someone pursue it? There are many holes to be plugged and much more than one thousand words of thought to put into this but I’d greatly appreciate your feedback. Moreover, please share your thoughts on whole HR Idea Lab concept below, and as always, let’s keep the conversation going.