This post is about indices (plural of index). It might be dry. However, bad indices are a blight on humanity which I must combat. Onward!
An index can be extremely useful. A good index will compile a combination of numbers (data points) that when combined tell a story.
An example of good index is the consumer price index. The CPI collects data about a basket of goods which reflect the overall prices of consumer products. As a representative index it has data that’s inherently connected. Even better, it outlines which questions it has been designed to answer:
The CPI affects nearly all Americans because of the many ways it is used. Following are major uses:
As an economic indicator. More.
As a deflator of other economic series. More.
As a means of adjusting dollar values. More.
The reason I wrote this post is because I came across an example of a bad index: the mediaite power ranking index. Mediaite (mediaite) is a news centric website created by media consultant Dan Abrams for media aficionados. A key “attraction” of the website is their power rankings. In these power rankings, they ranking people in a variety of different media fields from journalists to anchors to owners. Unfortunately these rankings make absolutely no sense.
In order to try and understand these rankings, I asked two questions: 1) What is it trying to measure? 2) What does it measure?
The answer to the first question is “answered” in the FAQ:
What is the Power Grid?
It is an objective ranking of roughly 1,500 known and important players in the media today, divided into categories including Media Moguls, TV Anchors and Hosts, Magazine Editors, Print and Online Editors, and Top TV Executives.
Mmhm…an objective ranking of players in the media today. Notice what’s missing, objective ranking based on what? Given the name, I guess we’re to assume that it’s a ranking of media power? Except that elsewhere on the same site the rankings are referenced an “influence” index. Influence over whom or what? Influence over other world affairs? Influence over public opinion? Influence over other media? Influence over Dan Abrams and his staff? I’m not sure we’re going to get an answer to that question. Let’s move on to analyzing what the “influence rankings” actually do measure.
What does this index measure? Well, for each different category it uses different metrics which are then put into a secret algorithm. The definitions of each metric are available on the FAQ page. Let’s look at some specific examples, first the anchors/hosts categories.
First of all this category makes no sense. Why would you compare Oprah/Katie Couric/ Bill O’Reilly? They are competing for very different audiences and have very different “power”. This is close to the definition of apples and oranges. Ranking them hierarchically makes no sense.
Having established the very absurdity of the category we’ll analyze the validity of the metrics and the importance of the assigned weights for each category.
First, are the metrics valid. In this case there are three primary metrics (and apparently a fourth less prominent category). The three prominent metrics are total viewers, google buzz, and blog buzz and if you dig deeper you’ll also find that the number of twitter followers is also included in the calculation. How are these measured?
Well, there is no “metric” in mediaite’s FAQ called “total viewers” the closest relevant metric is time-slot ratings which is described as follows:
Time slot Ratings – This metric is the total viewership of the program, as extrapolated from Nielsen-reported television ratings.Note: in some cases television ratings have been adjusted for individuals who appear on programs that only air once per week, or are part of a larger ensemble cast.
This is a long standing professional measure of media figures because we suppose that the more successful one is the more viewers they will attain (more viewers also is connected with more revenue for the media company). The next two categories, google buzz and blog buzz are more problematic:
Google Buzz of Name – This metric is the number of relevant hits yielded by a Google search of an individual’s name. Irrelevant hits, such as those for similarly-named individuals, are filtered out. For instance, the Google Buzz metric for James B. Stewart, the writer and reporter, filters out hits for Jimmy Stewart, the actor, and for James Stewart, Jr., the motocross racer.
Google Blog Buzz of Name – This metric is the number of relevant hits yielded by a Google blog search of an individual’s name. Irrelevant hits, such as those for similarly-named individuals, are filtered out. For instance, the Google Blog Buzz metric for James B. Stewart, the writer and editor, filters out hits for Jimmy Stewart, the actor, and for James Stewart, Jr., the motocross racer.
These metrics assume that any mention of a person on the internet adds to their power/influence. It’s because of these metrics that Katie Couric is ranked above her competitors Charlie Gibson and Brian Williams (they both have much higher viewership totals). Yet, if you do your own google search of the name Katie Couric you’ll learn that the high number of google hits has to do more from Sarah Palin and less with Katie Couric. Katie Couric happened to have an interview with Sarah Palin, which became famous because Palin appeared buffoonish. This interview is credited with doing significant damage to the Palin brand, and to the McCain campaign as a whole. Additionally, if you do a google blog search of Katie Couric you’ll find posts titled “TV Anchor Babes”, “Should Katie Couric Quit”, and “Katie Couric – The Face of Media Bias.” Mediaite claims that each of these posts increases Katie Couric’s power/influence. In my experience, blogs are more likely to cite a journalist or post their work if they disagree with the journalist. A search of prominent posts from the last two weeks at the website Daily Kos found 31 articles regarding Fox News, and fewer than 10 articles each regarding CBS, NBC, or ABC. Mediaite would claim that each of those pieces about Fox News shows it has more power/influence than any of the broadcast networks. Mediaite would also claim that this very post is increasing Couric’s power/influence.
These rankings do not measure anything. Each metric on its own tells a story which can then be deciphered in different ways, but when combined they make nonsensical noise—especially given that we don’t know what proportions of each metric were used for the final product. If these rankings were to remain Dan Abrams little parlor game, then there wouldn’t be much problem with his silly rankings. Unfortunately, Abrams is pushing these rankings as legitimate “objective” measures of professional success. The last thing the media needs right now is to enter a competition to get cited by the blogosphere.
In short, Mediaite fail.
If after reading this you care for more critiques of the rankings or a response from the people at mediatite check out this article (while it was written a while ago, I only found it after writing the above).