As many of you are aware, Phil Fersht of AMR Research had posted a simple survey asking buyers, providers and influencers to “rate the raters”.  This morning, Phil announced that the findings of the survey were tainted by false respondents claiming to be Fortune 500 organizations.  Per Phil’s Horses for Sources blog:

“Unfortunately, I received a very large number of suspicious survey responses from a host of “FORTUNE 500 buyers”, whose IP addresses – for some reason – all seemed to emanate from the couple of locations. I received a very large number of these survey submissions clustered within a short time-frame, and they had no names or email addresses attached. They also all had selected one particular list-maker as “highly credible”, while simultaneously describing the same 2 others as having “poor credibility”.”

As someone who has been in this industry for some time now, I am terribly saddened and disappointed to see such a simple attempt at an open information capture to be fraudulently tainted.  Unfortunately, this type of behavior simply perpetuates an area of the industry wrought with increasing distrust.  It seems to me that once again things will have to get worse before they get better.  

Let’s keep the conversation going and our integrity intact.  

Although concern surrounding certain list-makers’ methodologies has been brewing for some time, a frenzy erupted when BusinessWeek’s Steve Hamm wrote a very compelling article questioning the practices of Black Book of Outsourcing creators Brown & Wilson.  Per Hamm’s article:

“Claes G. Fornell, a professor at the University of Michigan Ross School of Business who specializes in customer satisfaction surveying, says Brown & Wilson’s methods aren’t sound. First, he says, the firm can’t be sure all the people who respond are qualified. Second, the results could be tilted in favor of companies that urge their customers to participate. ‘You’d be better off not doing anything than doing a survey like this,’ says Fornell.”

Picking up on these concerns, Deborah Kops (CMO for WNS Global Services) repurposed her May 2006 article (“The Book of Lists”) for Fersht’s audience.  Among many insightful statements, Kops questions the value of lists and what they don’t tell us:

“A key component of editorial calendars and other sponsorships in the outsourcing industry, rankings and lists can provide a service to communities by identifying players and trends. But let’s not delude ourselves; they are also a business imperative for publishers, associations and pundits to build membership and/or circulation and sell adverts, publications and reprints, playing on the sell side’s need for recognition. They generally make someone money! This is not necessarily a negative, but is rather the way the world goes around. What’s critical is that that which is editorial and that which is financial should be kept completely independent at all costs. Pay for play in any form must not be the modus operandi.”

I believe Deborah was very gracious by not explicitly referring to those entities whose lists, rankings and awards have tainted the perceived or real value of the industry’s providers.  Those of us who have been in the market for sometime now can easily identify the organizations who have handsomely profited from such point in time, pay-to-play assignments of value.

The Catch-22 of our current industry state is this: In lieu of an attractive alternative, provider marketing organizations will continue to pour precious funding into the pre-existing ranking and rating processes. Thus, the cycle perpetuates itself.  We don’t like the lists yet many are afraid to not be on them.  However, until there is a collective halting of such spend, we can’t expect the system to self-correct. In the interim, we can hold out hope that a better mechanism will emerge that is truly transparent, unbiased and effective. Let’s keep the conversation going.

_________________________________________________

It you want to voice your thoughts on these issues, place your votes by judging the judges.