Recently it was made aware that Obsidian Entertainment, the company responsible for the excellent game Fallout: New Vegas, was not given any royalties or bonuses for the game because the game failed to garner a high enough score on aggregate review site Metacritic. Aside from this being a terrible story for the talented developers over at Obsidian, it is also a ludicrous practice for a publisher to utilize. There has been a bevy of articles and opinion pieces on the situation, with a lot of the argument aimed at Metacritic. But is Metacritic really to blame, or should the pointed finger of the outraged public be directed towards someone else?

Let’s start things off by clarifying the problem: Obsidian Entertainment’s game Fallout: New Vegas was only to be granted royalties and extra bonuses to its developers if the game achieved a score of 85 or higher on Metacritic. This decision came from New Vegas’ publisher, Bethesda Softworks.Metacritic gathers online and print reviews from credited publications and gives an overall “Metascore” based on the averages collected from said gaming publications. Since Metacritic only has so many gaming publications it can draw from at any given time (take a look and read over what it takes to qualify to be a Metacritic-approved publication), the given Metascore is based on only a handful of people. Assuming a game reviewer is doing their job properly, then they are reviewing the game accurately and giving their honest thoughts and opinions about the title. So what the hell does this all mean? It means that 81 critics determined the fate of Obsidian’s developer’s bonuses.

Being a game reviewer myself as well as a Fallout fan, I find this story to be completely and utterly ridiculous. I must clarify that I DO NOT put any blame for this on the individuals who reviewed Fallout: New Vegas; they did nothing wrong and in fact can be completely omitted from further discussion. I do find faults, however, with the publisher, Bethesda, and Metacritic itself.

The Problem

A Metascore alone should in no way sway a person’s choice one way or another.
I’ll avoid reiterating that numbered game reviews should be done away with, and say that consumers need to read before they decide. A review of a game is going to highlight the ups and downs of the title and should give you a multitude of insight on the game (How’s the story? Does it control well? What about the graphics? Are there any bugs to deal with?). 84 does not tell you about the game. Metacritic tries its best to create a Metascore that is indicative of a games quality, but a number can’t tell you how cool it was the first time you shot a ghoul in the head and saw it’s brains blast out the back of it’s cranium. This isn’t really the fault of Metacritic, but more an issue with us as consumers, wanting to get our information here and now. Taking the time to read why someone gave the game a 70 (Joystiq) instead of a 75 (1up) can’t be done with a Metascore.

A game’s performance should be the factor used to determine remuneration.
A quick search for New Vegas’ total income finds that the game has earned over $300 million. I’m not an industry insider or business analyst, but three hundred MILLION dollars sounds like a hell of a lot of money for one game to make. Clearly the game has sold well, so why hasn’t Bethesda given the game’s developers their due reward? Never mind the fact that New Vegas missed it’s Meta-goal by one point, but consider that the game sold millions of units, raked in lots of cash, and was generally positively reviewed. If 81 reviewers say the game is great with a Metascore of 84, then I guess the millions of gamers that bought the game don’t matter.

Metacritic’s system is flawed.
Not just anyone can write a review and be added to the aggregate for a game. You need to first meet Metacritic’s guidelines to qualify to be considered. But even then, certain sites are weighed more than others when determining the Metascore. This isn’t really a bad thing, since I would venture to guess that most gamers would want to hear what IGN says about a game over D+PAD Magazine. But even with this weighted system in place, things still aren’t entirely right with each Metascore. Not every site/magazine uses the 100 point scale that Metacritic does. Sure, Metacritic tries to convert scores as accurately as possible to fit their scale, but plenty is lost in translation. If you gave a game, say, a B, then what number value would you give it? 82? What about 85? 86? Metacritic has a system for getting this number from a letter grade, but where is the reviewer in this scenario? As a game critic myself, I’d be incredibly aggravated if someone translated my score to anything I didn’t fell was appropriate because that is my review they are talking about.No one should have a say in my review but myself. Metacritic is trying to do something great here, by offering the consumer an easy way to view an overall average rating for a game, but the system has flaws that rear their ugly heads the deeper you go. And when dealing with something as tremendous as a studio’s bonus paychecks, Metacritic just isn’t a reliable source for quality.

So where do we go from here? Unfortunately for Obsidian I think they are out of luck on getting their bonuses. But we as the gaming community shouldn’t settle for this practice. There are better ways to compensate developers than using the entirely subjective Meatcritic. Let’s take a look, shall we?

The Solution

Metacritic needs to change its ways.
If this situation has shown us anything it is that Metacritic’s system for averaging games isn’t the greatest. Sure, they can be faulted to a degree as it is their system that I am in disagreement with, but I can’t give them flak for trying. A better formula for gathering a game’s Metascore would be to only use the same 20 or so publications for each game. Make this group comprised of bigger publications that cover all kinds of games, likeGame Informer or GameSpot, as well as more specific sites like RPG Fan. This way you have a nice pool of minds that know what they’re doing, have a background in the genre, and can be a credible source. Of course, reviewers need to always remain truthful and opinionated in order for this to work. If this course of action were taken then one could safely assume that each publications score would be weighted equally, therefore giving the most accurate and true Metascore.

Review publications can either bend to Metacritic or downright refuse them.
Much of the trouble with converting a site or magazine’s score to Metacritic is that it does not follow the same scale. Metacritic, again, uses a 1-100 scale to rate a game. If every game reviewer used a 1-100 scale to rate a game, then there would be nothing lost in the translation of scores (especially if the above solution is also practiced). But the 1-100 scale system has flaws as well. Can you tell me the difference between an 84 and an 85? So perhaps a better solution would be to tell Metacritic sayonara andnotbe one of their affiliated sites. If your scores are being interpreted incorrectly then why allow yourself to be misquoted? Though this solution is a bit farfetched it would eliminate the overarching problem of using a Metascore to judge a game’s quality.

Publishers need to stop looking at Metacritic. Period.
Because the Metascore of a game isn’t a true representation of a game’s quality, publishers need to stop using it. But this is easier said than done. A Metascore is an easy number to slap on a game and feed to the masses. “Did you hear, Halo 4 got a Metascore of 90. It must be amazing!” is a whole lot nicer to say than, “I read over at Kotaku that the new Halo game had great graphics but felt stale with the outdated gameplay and uninspired weapons.” I can’t say that I don’t understand where Bethesda was coming from by doing this, but this practice does a massive injustice to everyone: developer, gamer, and the general population. If you want to give a bonus to a development studio and base it off the game’s performance, use more concrete facts, not the opinions of a handful of people.

Which brings us to the question posed at the beginning of this article: who is to blame? I can give you a beat-around-the-bush answer and say that it’s both the fault of Metacritic and the publisher (which it can be heavily argued is true), but I think the guild should be solely with the publisher. Shame on you for putting the fate of a person’s bonus in the hands of a group of subjective reviewers. Yes, Metacritic has its fair share of problems, and yes, Metacritic is a convenient way to sum up a game (albeit inaccurately), but it is in no way a means to base a game’s success. Bethesda should have more faith in it’s development staff at Obsidian. Any publisher for that matter should get into a method of paying out bonuses on a game’s financial and commercial success instead of critical reaction (even if it’s good!). Game reviewers do a fine job reporting on the quality and content of a game, but it is never a reviewer’s duty to be the deciding factor on a person’s compensation.

Laters,
Jsick

**Though I find Metacritic to be ultimately flawed, they have done something great that shouldn’t be skipped over. Take a look at a game/movie/TV show on the site and actually go to the site where the original review had been posted. Get the author’s full insight on the piece and try to understand why reviewer A gave one score and reviewer B gave a drastically different one. When I read up on a game I usually go to Metacritic because it has already done a lot of the legwork for me by collecting links to publications that I then go to and read myself. Seriously, this is arguably the best part of Metacritic and still allows for the author’s creative thoughts to best be understood.**

Sources:
Kotaku
Game Informer
Joystiq
Shiny Buttons
ggs Gamer