Analysis: What VGChartz Does (And Doesn't) Do For The Game Biz
June 23, 2008 8:00 AM | Simon Carless
Recently, there's been a significantly greater profile for the video game chart compilation site, VGChartz.
As well as beginning to contact major news sites on a regular basis to disseminate its news, the site was also the subject of a positive article on O'Reilly Radar from Robert Passarella, comparing its open data dissemination method favorably with The NPD Group, the generally agreed 'canonical' source for North American game charts.
Indeeed, as the Wikipedia page for VGChartz notes, Forbes, Fortune, The New York Post, and The New York Times have all referenced the site. And since it's been more aggressively marketed by the site's creators - especially regarding the 'holy grail' of global sales figure comparisons, its references in the news are rapidly increasing in frequency.
But how is the site actually compiled, and is it a good source for reasonably reputable news websites such as Gamasutra to be citing? Thus far, we have referenced VGChartz data twice - once with regard to Xbox Live Arcade game sales, and more recently because Michael Pachter has started to cite the data in his NPD game sales previews.
The second citation provoked a number of reader queries about the veracity of the data, so we embarked on some detailed analysis of VGChartz, and followed it up with a long series of emails with the site's creator, Brett Walton.
How Charts Are Compiled
So, let's start with the basics. Most worldwide game charts (of which NPD in North America, Media Create and Enterbrain in Japan, and Chart-Track in England are the most prominent) are compiled by extrapolation from sales figures provided by retailers. Thus, there's no third-party that uses regular access to publisher data on sales to make the charts - all of them take in data from major retailers, and then calculate sales from there.
This service is sold to major publishers and financial firms for a monthly or yearly subscription, allowing publishers to see how well their competitors are doing. Limited amounts of the data is made available to the public, generally in terms of a top 10, 20, or 50 either weekly or monthly, depending on the territory. Obviously, if the pay service was sufficiently 'off' from the internal numbers the publisher was getting, they would not want to pay for it.
But what VGChartz claims to do is a weekly estimate of every game published in every territory - including Japan, North America, and the particularly difficult to estimate Europe (there's no subscription-based pan-European chart right now, due to the fragmented nature of the market) - and offer all of that data for free. Which is extremely impressive, let's face it, because it enables them to provide real-time updated information for the entire market.
But how accurate is it? If I was, say, writing a story for the New York Times, what proof do I have that the 'correct' numbers are displayed on the site? Obviously, as mentioned above, all sales figures are by necessity estimates, and that's the crux of the issue - we'll get back to that later. But I asked Brett Walton his methodology, and he gave me the following, quite impressive answer:
"The methodology we use for all of our charts in all regions is the same and our data is arrived at by a combination of the following:
- Sampled direct sell-through data
- Industry knowledge and experience - applying past trends in terms of marketshares, regional breakdowns, casual vs hardcore and so on
- plenty of statistical analysis, regression calculations, market projections
- Contact with industry figures - buy-side analysts (such as Pachter / Divnich), sell-side analysts who work with us on specific products / projects, manufacturers who work with us to project sales of their key titles
- Retail checks - we have a team who talk to stores and estimate shipment figures for low-stock and hard to find items which we struggle to track with our normal data samples.
Exactly how we get from these various sources of data to final figures differs from game to game and console to console and our exact methodologies are confidential for obvious reasons."
Essentially, Walton is saying that he uses a number of high quality factors to produce his estimates, but can't mention any of the retail sources, or companies that VGChartz works with. Well, fair enough. But did you realize that VGChartz estimates can retroactively change by 100% or more based on 'official' chart results?
Iron Man & Retrofitting
One of the most unexpected results in the recent NPD charts for May was the appearance of the poorly reviewed Iron Man game for PlayStation 2 in the Top 10 of the charts, with 130,000 copies sold.
Thinking about it carefully, with the movie rocketing to unexpected success during the month, it would make sense that the game would sell well. But it's not the kind of game that you're likely to estimate in the Top 10 - and indeed VGChartz did not, estimating 53,000 units in sales, according to VGChartz staffers.
But what's surprising is that Iron Man for PlayStation 2 has been adjusted in its official VGChartz page so that its first four weeks of sales (encompassing May) add up to 111,000 units.
Clearly, these numbers have been changed after NPD debuted, showing a couple of things. Firstly, if you were a journalist, you could have cited VGChartz as saying Iron Man was a flop on PS2, selling half as many units - when NPD vibrantly disagrees. In addition, and more interestingly, it shows that VGChartz trusts NPD over their own prediction data by retroactively changing things to better match.
Apparently, this has happened before, because in a FAQ about North American VGChartz numbers, Brett Walton addresses this precise subject:
"Do we adjust our data? Not as such. Do we adjust our methods then? Yes - which will of course alter some data. On what basis? If we believe that a particular data set differs significantly from other sources of data (data released into the public domain by tracking firms, manufacturers, analysts) then we do re-check our data and make adjustments to the methods / scaling factors used.
This happens on a fairly infrequent basis - less often than we adjust due to internal data changes - and is something that every tracking firm and analyst does. I personally have no issues with "benchmarking" our data from time to time against other sources of data - as long as it has been made public."
In other words, if they are sufficiently out, then VGChartz will retrofit their results - either weekly or monthly - to conform to the more 'official' data. But they won't credit those firms as the source of the retrofitting - they'll just bump their numbers around without saying why on the site.
As a result, we get to what VGChartz actually is - a strange mixture of a prediction market (as consensus prediction site TheSimExchange is) and a retroactive, but non-credited reflection of charts that have historically been known for having more concrete data.
Where's The Beef?
OK, so you might say - and a lot of VGChartz' forumgoers do - what's the problem with that? If VGChartz gets close enough, and can adjust if it's too far off when top-end data comes out, then why would there be a problem?
Well, because you then have a moving target for checking/reporting purposes, and particularly because there's a high probability that VGChartz figures will be significantly wrong for those titles on the lower end of sales - those that lurk outside the top of the charts.
In other words, for those high-selling titles, VGChartz is checking against public data, and they will change their estimates if they are majorly off. Most of the time, they are quite close compared to the worldwide charts. That's because VGChartz is - like services such as The SimExchange - using common sense, Internet buzz, real-time data such as Amazon.com and analyst commentary to synthesize a sensible estimate.
But in covering all games, they are doing readers a disservice, because it's clear from the Iron Man example that they simply do not have the direct sale retail contacts to extrapolate unexpected but nonetheless true results. And if a title spikes but is outside public data, VGChartz will never catch it.
And the amount of concrete data available to VGChartz is low - as is freely admitted in a recent interview, VGChartz had 2-3% of the North American market as a sample at the time, whereas by estimate, NPD might have 60-65%. If this 2-3% was clean and canonical, this might not matter - but how do you explain the big Iron Man discrepancy, if so? Wouldn't VGChartz' retail sources have picked it up too?
So, let's take a step back and concentrate on some games that have sold in significant numbers, but have never made it into the Top 20 in North America for a significant time.
One good example is the Ben 10 series of games from D3 Publisher. VGChartz has the series listed at 590,000 sold worldwide to date. But when Gamasutra interviewed D3's Yoji Takenaka last week, he specifically said: "Ben 10 is selling well over a million units right now, since last Christmas."
So sure, Takenaka could be conflating shipped with sold - making the number closer to the estimate. But that's an awfully large discrepancy - one that most people won't care about because it's not a prominent or critically acclaimed game, and there's no way to refute VGChartz on it, but a discrepancy nonetheless.
Unfortunately, we don't have lifetime NPD data for this set of titles - but in researching this story, we spoke to a third party who had access to NPD lifetime to date sales that are not normally disclosed to the public.
We picked two titles released for one of the next-gen consoles over the previous year, neither of which had been in the public NPD charts for more than a month, leaving VGChartz to make estimates based on their own sources on their selling curve over time.
Well, somewhat spectacularly, in both cases, NPD and VGChartz disagreed by about 100%. In one case, VGChartz was citing 300,000 sales, whereas NPD had the game at 150,000 units. And in the other case, it was inversed - NPD had the game at around 200,000, but VGChartz had it at 100,000.
If VGChartz knew of this discrepancy, would they have retroactively changed their data? Probably so, given the Iron Man example. And this is essentially the problem - that with very limited access to retail numbers, especially over time, the downward curve of a game's sales becomes essentially a guessing game for VGChartz, whereas services like Media Create and NPD merge in greater real sales data to calculate their curve at much higher levels.
[Here's one more public datapoint, this time referencing NPD, but uncorrected by VGChartz, since I presume they didn't notice it or consider it important enough. Variety recently revealed that Brash's Alvin & The Chipmunks game had sold 286,000 copies since launch, according to NPD. VGChartz has the combined SKUs listed at just 110,000 units.]
Conclusion
Let's be clear. I think the concept behind VGChartz is a wonderful one - freely available data to let everyone see how well games are selling. And it's absolutely true that all data is an estimate - not even major services such as Media Create and NPD get it exactly right.
But VGChartz is staffed by amateurs working in their spare time to estimate sales, and while they are perfectly smart, they are much closer to the SimExchange model of estimation than the Media Create method.
What I'd like to see is some clear labeling of what is estimated data, and what is extrapolated or changed from companies that have greater access to retail sales. And not only does VGChartz have no intention of doing this, it is starting to claim major scoops based on data which, in some cases, estimates entire territories without any real data.
In particular, the site widely and loudly disseminated to the media its worldwide Day 1 Metal Gear Solid 4 sales, explaining:
"VGChartz can exclusively reveal that first day sales of Metal Gear Solid 4, released on June 12th 2008 in most major markets worldwide, were an impressive 1.3 million units."
The headline actually originally read 1.5 million, but was changed by a not insignificant 200,000 units after publication. Even more surprisingly, the figure debuted just 48 hours after the launch of the game - not a lot of time to compile data from retail sources.
I asked Brett Walton about the change, and why this figure was not advertised a little more prominently as an estimate, given the short amount of time to get real data, and he explained:
"It was based on first day Japan sales, first day America sales, and from that projecting for Europe / others which we didn't get direct day 1 for. We projected Europe would be ~20% higher than America given the larger install base and based on previous game releases, but it turned out at 430k for the week vs 510k for America - whereas we estimated it at more like 600k given America and Japan figures."
Firstly, Walton freely admits the numbers were based on zero actual data for the entire European market, just pure extrapolation. It's also very unclear how far the estimates for launch were based on real retail data for Japan and North America.
It's a reasonable figure, of course, because the VGChartz folks are smart people. But it's not a real figure. It's a educated guesstimate, and it's much more of an estimate than the subsequent Chart Track data for the UK, for example. Walton clarified due to my complaints:
"So yes, maybe we should be clearer with the word estimate, especially in early PR and this has been reflected in comments back to the guy who wrote the story. From now on we will label day 1 sales as preliminary for that very reason."
But that doesn't really change the main problem with the site. There's a place for a resource like VGChartz, but it'd be a site that clearly labels the source of its estimates (whether it be Chart-Track, NPD, Media Create - even if some of those sources have poor data dissemination and a fractious relationship with the media) and then labels which are its own estimates based on its own industry knowledge and whatever channel checks it has.
But if I was a writer or analyst trying to extrapolate significant information from the resource, especially regarding those titles which don't chart regularly, given the major discrepancies with other figures shown here, I would not recommend it.
[UPDATE: Someone has just pointed out to me that Brett Walton has accused me of reprinting a 'confidential email conversation' for this article. In the course of my discussion with Brett, I specifically asked him if our emails were on the record. He replied: "I have no issues reproducing the discussion - needs a tidy up of course." As far as I'm concerned, issues like this are symptomatic of why VGChartz cannot be trusted.]
Categories:
48 Comments
The best analysis (or should I say estimate?) of VGChartz I've read yet.
I would say their software numbers have historically been very inaccurate. Their hardware estimates are a bit better, but still unreliable, in my opinion.
Tony | June 23, 2008 12:20 AM
You know I don't remember NPD ever acknowledging the few times where they have been wrong either...
Besides what do you suggest when some numbers are wrong ?
Keep the wrong numbers and keep going like it never happened like NPD or fix the wrong numbers once you know they are wrong like VGChartz ?
Otherwise I will agree that VGChartz software numbers are not always right but their hardware estimates have been pretty much spot on for the last 6 months...
Aildiin | June 23, 2008 12:33 AM
Well I think you're picky.. It's not easy for them to set up such a site and you don't talk about the numbers that are correct? Like their Hardware Numbers? Thousand of Software Numbers AND they are doing it WORLDWIDE!! It's much easier to track on a small scale like NPD.. Give them credit for what they are trying to do.. And whats wrong for correcting your mistakes?
Nik | June 23, 2008 2:02 AM
This is the type of indepth story on vghcartz that we have all been waiting for. Thank you for taking the time, digging deep, doing the research and staying on top of Mr. Walton. You have addressed what so many of us already know. Walton misrepresents himself and mocks the industry.
Aildiin, when NPD has been wrong, they send a statement to the industry and work with it to avoid future errors with their model but even then the industry has been quick to fry them. I remember the last time after New York Times wrote a scathing article about them and some publishers called them out so yes, when NPD is wrong they are quick to correct.
vgchartz is more than wrong on some figures. Its software numbers are the most unreliable of all. Who cares aout hardware nowadays? We all know how easy it is to get hardware numbers directly from manufacturers. I want proper software!!
nicom | June 23, 2008 2:06 AM
@ nicom. You do know that their hardware numbers is about sold to consumers instead of shipped by the manufacturers?
Nik | June 23, 2008 2:13 AM
Very good read.
The one thing I miss, which would have made the article a bit more balanced, is that you only pick the cases where VGChartz is off, granted by quite a large percentage.
But you could have pointed towards the cases where the numbers are actually very precise. For example the GTAIV case where Take-Two gave exactly the same figure for first day sales a day after VGChartz.
Also console sales are very accurate, very much in line with the annual figures of the manufacturers.
Tom | June 23, 2008 2:33 AM
Does that matter? As a journalist, all you have to do is contact the right marketing liasion within each manufacturer and they will happily give you sales by region. Whether it's sell through or shipment doesn't matter right now because it's so early in the sales cycles for all three manufacturers.
TSE and even Pachter don't go that far because they are actually estimating, not claiming to make estimates and then contacting the manufacturers or using their released shipment figures. You will look reasonably reliable using this method right now but you would eventually be caught with your pants down as the market matures.
I am anxiously awaiting the flamewars and can see they're already beginning but let's not forget what it is that Simon has exposed here: Brett Walton is getting caught up in his own manipulation of the truth. He retrofits using data and insight but doesn't properly identify his sources. He charts using publicly available information but will not and cannot identify these sources because it would deflate the value of his site.
Beyond that, in one breath Brett claims not to compete with the research firms like Chart-Track, NPD and Famitsu, then he and his moderators will talk trash about Chart-Track, NPD and Famitsu at every possible opportunity. It gets old and for those of us that happen to work with the research firms, it fogs the mirror.
I have emailed with Brett in the past and have asked similar questions. To claim that insight into his retailer relationships were "confidential for obvious reasons" (this must be a canned response because he wrote the same thing to me) atuomatically raised the red flag. That is when I knew all wasn't as it appeared to be in the great land of Oz.
And we wonder why, for example, it is so hard to get closer to Chart-Track. One reason why they don't like to work with media and give out anything is because of the Brett Waltons of the world. NPD will likely throw in the towel, as will Famitsu and Media Create if this kind of garbage continues. That would be the ultimate irony, to have vgchartz as the the only available media resource, then watching as publishers and manufacturers crush it after providing completely false sales figures and research on titles and hardware sales.
nicom | June 23, 2008 3:01 AM
While you are somewhat accurate in this, I feel compelled to mention that VGChartz has every month posted a comparison with their data and NPD, and shown where they have been wrong.
Brett has also always said that small selling games are a weakness, which makes sense. The smaller the number, the more difficult it is to estimate. Apart from Iron Man last week, the top 10 and the hardware were quite accurate.
While the numbers may not be the most accurate, I have hopes that they continue to improve the methodology such that no one is forced to pay thousands for simple sales data.
omgwtfbbq | June 23, 2008 5:52 AM
Thank god someone is talking about this, VGChartz is an hurting us more then helping.
Emenis | June 23, 2008 5:58 AM
omgwtfbbq,
"no one is forced to pay thousands for simple sales data"
Have you ever worked / interact in business world ? We are used to pay for everything. If the value is there, we are willing to pay for it. Otherwise, why are we paying analyst or fund managers for them to help us pick the right stock / market to invest in ?
You do realize that there are different markets to serve : low end market with free stuff (VGC for non-business) and high end market with paid stuff (NPD for business).
It is a given that paid services (like cable TV)should be more (widely varied, accurate, or any other trait) than free services.
apujanata | June 23, 2008 6:06 AM
Well if you dont like vgchartz then don't go to the site.. And of course it matters if it's sell trough or retail: If I had made 30 million products but 29 million are sitting on a shelve, I couldn't possibly say that I did a good job selling my product..
Nik | June 23, 2008 6:14 AM
Thank you for writing such a thorough article!
Saint10118 | June 23, 2008 6:54 AM
Simon, great post. I've been enjoying this debate for sometime and I really like your analysis here. I think you have nailed a lot of the issues in a non-biased way. When I use or show others how to use VGChartz as part of their analysis -- I always point out that it shouldn't be your only source. VGChartz is part of the mosiac you need to create in order to understand sales data in general. As always with modeling, you need to be aware of what question you want to answer with the available data. For me VGChartz fills that need, even with its short comings.
Robet Passarella | June 23, 2008 7:08 AM
Wow at Nicom. Clearly part of the NPD hit squad.
Joe Schmoe | June 23, 2008 8:05 AM
Looks like ioi (Walton) is pitching a fit on his site, saying email conversations between him and this article's author should have been confidential. Funny because Walton sure will post any email sent to him at his leisure, regardless of the content. Then he has to go and throw NPD into his tie raid. Always with the NPD bashing whenever he can, making statements about research firms that he has absolutely no background or insight on, etc., yet he claims never to compete with them. What a hypocrite. What a baby.
Nice article S.C. Thanks for the truth about Walton. That he lifts the data he can from third party sources w/o giving them the credit (and the fact that if he did he would be sued) and then making simple estimates with the rest of it. Isn't that what it's really all about?
GmS | June 23, 2008 8:15 AM
I'm going to disagree about it being so easy to get hardware numbers from manufacturers.
Remember 360 sales for Holydays 2006 ?
Based on shipped numbers ( the only numbers manufacturers will give you, especially in MSFT case) the 360 sold like hotcakes during those holydays. And then proceeded to ship less than 500k units in the following quarter.
Reason being that holydays quarter Microsoft stuffed its channels like mad and it took retailers 6 months to sell the units they had..
And like some posters have pointed out, if you don't like VGChartz numbers, don't use them, noone is forcing you. But at this point they are the only one giving you a view of what weekly worldwide sales look like and how some titles can influence hardware sales or not on a weekly basis...
Aildiin | June 23, 2008 8:39 AM
And if it was so easy to get hardware numbers, NPD woudn't have a business out of selling those numbers............
Aildiin | June 23, 2008 8:42 AM
It's amusing how the gaffers rejoice over Mr. Carless' article here, and the VGCers rejoice over Mr. Passarella's article, and then get angry over the other one, respectively, when this article doesn't dispute what Passarella laid out as VGC's big strengths (speed, WW data, upfront NPD comparisons, community), but at the same time Passarella comes over and compliments this article, basically saying its all true.
Only the forum fanboys paint this as some sort of us vs. them, black and white war between VGC and NPD or other benchmark services. Some over at VGC are saying Mr. Carless can "go to hell" while on Mr. Passarella's comment section, people claim he just called VGC a "be-all, end-all." The NPD-bashing is based on the lateness or cost of their data, while the VGC-bashing is based on accuracy and personally discrediting Mr. Walton. All these people are only seeing things in one context, and unable to recognize the differing, non-competing values of the other service.
The point is, both articles are basically correct, but for Mr. Passarella the 4 things he listed are valuable despite accuracy shortcomings, while for Mr. Carless they are less so.
Disclaimer: I post over at VGC.
Erik Aston | June 23, 2008 9:33 AM
VGC is approaching the mainstream, it's already been referenced by several newspapers. I imagine in the next year or two, NPD or some other tracking firm will take legal action. Either that or they will cease public releasing their monthly reports. Unfortunately, it'll probably be the latter.
Gabe | June 23, 2008 9:54 AM
VGC's problem is two-fold: accountability and credibility.
It's true that it has appeared in some mainstream articles but in case you haven't noticed, VGC has no staying power within these media outlets ... sort of like when you wake up in the morning after a long night out with the boys and find a sea hag in your bed. Sure, you may have made a mistake but you'll be damned if you're going to go for another round.
Doctor | June 23, 2008 11:52 AM
Simon Carless Is A Cherry-Picking Douschebag. I mean seriously, am I going to die if I don't know how much Ironman and Ben 10 sold? No. I don't care about those lesser crap games. I care about the big ones. OK, so the guy CHERRYPICKED 2 examples, out of the thousands of games that VGChartz has gotten right, they've missed 2. Hmm... let's say they totally bomb 3 games for every 100 they track (A gross overestimate). That makes 3/100 -> 3%. 100 - 3% = 97%. Oh man, that 97% accurate site is Sooooooooooo Inaccurate.
Bullshit. Along with this entire stupid-ass article.
Simon Carless Is A Cherry-Picking Douschebag | June 23, 2008 1:02 PM
there are a number of factual corrections you
need to make (but probably won't):
"Well, because you then have a moving target for checking/reporting
purposes, and particularly because there's a high probability that
VGChartz figures will be significantly wrong for those titles on the lower
end of sales - those that lurk outside the top of the charts. In other
words, for those high-selling titles, VGChartz is checking against public
data, and they will change their estimates if they are majorly off."
Based on what exactly? You go from writing a factual article to making a number of (false)
assertions and it becoming very opinionated.
"Most of the time, they are quite close compared to the worldwide charts.
That's because VGChartz is - like services such as The SimExchange - using
common sense, Internet buzz, real-time data such as Amazon.com and analyst
commentary to synthesize a sensible estimate."
Actually, most Simexchange users are regulars on vgchartz and use our data
to form the basis of their predictions which in turn forms the basis of SE
numbers. Our data comes out before all analysts release their reports and
yes if we do use factors like marketing budgets, buzz, sales of past
titles, shipment figures and many other factors to augment our direct
sell-through data then what is wrong with that?
"But in covering all games, they are doing readers a disservice, because
it's clear from the Iron Man example that they simply do not have the
direct sale retail contacts to extrapolate unexpected but nonetheless true
results. And if a title spikes but is outside public data, VGChartz will
never catch it."
No, it is simply a case that a title like Iron Man sells well at retailers
like Walmart which neither we, nor NPD track. They "guess" it sold more
than we "guessed" at Walmart, both are guesses so made some tweaks after
talking to the publisher who indicated that NPDs data is too high but
closer than ours. Would you rather we don't adjust or falsely adjust the
following month's data to make up for it?
"And the amount of concrete data available to VGChartz is low - as is
freely admitted in a recent interview, VGChartz had 2-3% of the North
American market as a sample at the time, whereas by estimate, NPD might
have 60-65%. If this 2-3% was clean and canonical, this might not matter -
but how do you explain the big Iron Man discrepancy, if so? Wouldn't
VGChartz' retail sources have picked it up too?"
More crazy assumptions from that head of yours
"So, let's take a step back and concentrate on some games that have sold
in significant numbers, but have never made it into the Top 20 in North
America for a significant time."
Yes, I can show you hundreds that have an excellent correlation
"One good example is the Ben 10 series of games from D3 Publisher.
VGChartz has the series listed at 590,000 sold worldwide to date. But when
Gamasutra interviewed D3's Yoji Takenaka last week, he specifically said:
"Ben 10 is selling well over a million units right now, since last
Christmas."
So sure, Takenaka could be conflating shipped with sold - making the
number closer to the estimate. But that's an awfully large discrepancy -
one that most people won't care about because it's not a prominent or
critically acclaimed game, and there's no way to refute VGChartz on it,
but a discrepancy nonetheless."
No, it is simply a case (if you actually check the figures) that the
figures for Europe haven't been added for that game (or thousands of
others) - as clearly stated on the European pages of the site and obvious
when you look at the game page. Compare our American data to NPDs US data
(as with all figures remember that Americas = USA + lots of other
countries) and you'll see it ties up well.
"Unfortunately, we don't have lifetime NPD data for this set of titles -
but in researching this story, we spoke to a third party who had access to
NPD lifetime to date sales that are not normally disclosed to the public."
Eh? And what were the figures?
"We picked two titles released for one of the next-gen consoles over the
previous year, neither of which had been in the public NPD charts for more
than a month, leaving VGChartz to make estimates based on their own
sources on their selling curve over time.
Well, somewhat spectacularly, in both cases, NPD and VGChartz disagreed by
about 100%. In one case, VGChartz was citing 300,000 sales, whereas NPD
had the game at 150,000 units. And in the other case, it was inversed -
NPD had the game at around 200,000, but VGChartz had it at 100,000."
Lol, and you are not even going to disclose the figures?? I assume these
are the two absolute extremes that you can find out of 1000s of games and
we are to take your word on it??
"If VGChartz knew of this discrepancy, would they have retroactively
changed their data? Probably so, given the Iron Man example. And this is
essentially the problem - that with very limited access to retail numbers,
especially over time, the downward curve of a game's sales becomes
essentially a guessing game for VGChartz, whereas services like Media
Create and NPD merge in greater real sales data to calculate their curve
at much higher levels.
[Here's one more public datapoint, this time referencing NPD, but
uncorrected by VGChartz, since I presume they didn't notice it or consider
it important enough. Variety recently revealed that Brash's Alvin & The
Chipmunks game had sold 286,000 copies since launch, according to NPD.
VGChartz has the combined SKUs listed at just 110,000 units.]"
No, maybe we are just happy with our figures on most of the games we track.
"Conclusion
Let's be clear. I think the concept behind VGChartz is a wonderful one -
freely available data to let everyone see how well games are selling. And
it's absolutely true that all data is an estimate - not even major
services such as Media Create and NPD get it exactly right.
But VGChartz is staffed by amateurs working in their spare time to
estimate sales, and while they are perfectly smart, they are much closer
to the SimExchange model of estimation than the Media Create method."
How, when SE data changes widly everytime we do a weekly release and when
we actually estimate sell-through figures based on samples rather than
just speculate on what they could be?
"What I'd like to see is some clear labeling of what is estimated data,
and what is extrapolated or changed from companies that have greater
access to retail sales. And not only does VGChartz have no intention of
doing this, it is starting to claim major scoops based on data which, in
some cases, estimates entire territories without any real data."
It is all estimated data.
"In particular, the site widely and loudly disseminated to the media its
worldwide Day 1 Metal Gear Solid 4 sales, explaining:
"VGChartz can exclusively reveal that first day sales of Metal Gear Solid
4, released on June 12th 2008 in most major markets worldwide, were an
impressive 1.3 million units."
The headline actually originally read 1.5 million, but was changed by a
not insignificant 200,000 units after publication. Even more surprisingly,
the figure debuted just 48 hours after the launch of the game - not a lot
of time to compile data from retail sources.
I asked Brett Walton about the change, and why this figure was not
advertised a little more prominently as an estimate, given the short
amount of time to get real data, and he explained:
"It was based on first day Japan sales, first day America sales, and from
that projecting for Europe / others which we didn't get direct day 1 for.
We projected Europe would be ~20% higher than America given the larger
install base and based on previous game releases, but it turned out at
430k for the week vs 510k for America - whereas we estimated it at more
like 600k given America and Japan figures."
Firstly, Walton freely admits the numbers were based on zero actual data
for the entire European market, just pure extrapolation. It's also very
unclear how far the estimates for launch were based on real retail data
for Japan and North America."
It is hardly based on zero actual data any more than NPDs estimates for
Walmart are based on zero actual data.
"It's a reasonable figure, of course, because the VGChartz folks are smart
people. But it's not a real figure. It's a educated guesstimate, and it's
much more of an estimate than the subsequent Chart Track data for the UK,
for example."
No such thing as a real figure, just different levels of accuracy of
estimate. You are being quite ridiculous with this angle after admitting
yourself that all data of this sort is an estimate. You seem to have now
forgotten the fact of vgchartz data = free, Chart Track data = £1000s.
"Walton clarified due to my complaints:
"So yes, maybe we should be clearer with the word estimate, especially in
early PR and this has been reflected in comments back to the guy who wrote
the story. From now on we will label day 1 sales as preliminary for that
very reason."
But that doesn't really change the main problem with the site. There's a
place for a resource like VGChartz, but it'd be a site that clearly labels
the source of its estimates (whether it be Chart-Track, NPD, Media Create
- even if some of those sources have poor data dissemination and a
fractious relationship with the media) and then labels which are its own
estimates based on its own industry knowledge and whatever channel checks
it has."
Again, all data is from our own estimates.
"But if I was a writer or analyst trying to extrapolate significant
information from the resource, especially regarding those titles which
don't chart regularly, given the major discrepancies with other figures
shown here, I would not recommend it."
Shame, for many the fact that MGS4 sold ~1.3m in one day is newsworthy and
a great achievement. Whether the actual figure turns out to be 1.23m or
1.36m or whatever, it is written as an estimate and given as an estimate -
and it is a very good estimate made before anyone else and accurate enough
to be newsworthy. Whether you use the figure as a preliminary figure or
label it as an estimate or whatever, it is of worth - enough to make an
investment on as many watchers do. For the site as a whole, it is meant to
be fun. The figures are all in the right ballpark and give an instant idea
of whether a title is a success or failure. You make a big issue of being
100% out - saying that you found a couple of games that NPD claim to have
sold 150k and we claim 300k is hardly a big deal. There will be many cases
like this - compare Media Create and Famitsu figures (both claim to have
50-60% direct coverage) and you can find similar discrepancies. It doesn't
suddenly make the game a hit or a flop - the fact we are in the right
ballpark for 1000s of games for free is all we claim and all we want to be
used for.
Nobody has ever said that if you want to find perfectly accurate sales for
a certain title that you should come to us - why you make an argument
based on that fact in light of an article saying MGS4 sold 1.3m on day 1
is beyond me.
Finally, comparing our data to NPDs or any other single source is
inherantly wrong. Sure, they are far more accurate than we are but they
are far from perfect themselves - it is impossible for anyone to be unless
you collect 100% direct sell-through data and you are better having 10%
but spread evenly over all retailers than having half but missing some key
retailers. It is the difference from retailer to retailer that is so hard
to track - one Best Buy will sell pretty similarly to another so it is
really a case of diminishing returns unless you expand marketshare by
adding more retailers.
Brett
Brett Walton | June 23, 2008 1:13 PM
From VGChartz:
I was going to do an editorial on all this but I think this works better:
We get sample data every week for the Americas, if it weren't classified I could post it from last November through last week which is the period for when I've seen it. Its just not enough store variation for some software so there has to be a concerted effort to look at shipment info, how similar games performed on hw at a given time, etc. The real advantage NPD has over us isn't the larger sample size, so much as it is the diversified sample. But still, it is just a larger sample size, and since it covers the majority of the market, but not all of it, and tends to be conservative, we aim to put sell-through figures closer to shipments. NPD isn't perfect, whenever I talk to some of my contacts I hear stories about how NPD changes figures after public release, and spreads out the changes over months, supposedly conflicts between NPD and Microsoft internal figures happened for months on end at Microsoft.
We're not taking other people's figures and predictions since those are flawed and subjective. We have a sample, we weight it for the data that is provided. Most games perform in a similar pattern in spiraling down from week one sales in the sample data, and that is reflected on the site. Adjusting to NPD figures would be silly given that they do not cover the entire market, but if the pattern is different from what we have there is no reason not to recognize it as reflective of 60% of the market in the cases where we have limited data. Its not like we do (Pachter x 2)+(NPD x 10) + (Sample x 3) and divide by 15 to get data or something. Frankly when we have data that is diametrically opposed to what NPD reports we ignore it, when the data is similar we have no reason to change it, it when a game falls in between those two ends because our sample wasn't diversified enough for propper scaling then we go back and weight the sample information differently. That isn't copying NPD, thats us being modest and recognizing that our scaling factors don't always hold steadfast. So if No More Heroes or something sold better nationally than in our sample, it just means we're scaling up the sample, we don't copy NPD, we just use it as a better idea for weighting tough to track games closer to shipments.
If NPD didn't report data we would still be able to provide weekly data from using our sample by constantly looking at the number of stores in a given market, comparing our coverage to that figure, and making the necessary adjustments to weights based on shipment data.
TheSource | June 23, 2008 1:21 PM
And my second email sent about 12 hours ago which also received no reply:
"I'd rather you make some factual corrections to your article rather than
making numerous false and unsupported assumptions. List out the Ben 10
figures and compare them to the NPD figures, provide links and references,
show all the games that are within 10-15% correlation to NPD as well as
the few that are not. Accept my point that NPD making educated guesses
about Walmart data is no different and no less professional than VGC
making educated guesses about European sales for the isolated case of day
1 MGS4 figures and that we did sacrifice some potential care and
dilligence over the accuracy of the numbers to get a scoop. In future we
will be more careful and ensure that the data has been finalised before
making similar releases.
Anyway, I feel like your whole intention with the article from the start
was to cause trouble and you obviously listened to nothing I said when we
spoke initially so I accept that this will likely fall on deaf ears.
Brett"
Brett Walton | June 23, 2008 1:26 PM
Posted by ioi:
Yeah that's fine but I am not going to let him now call me a liar and say I gave him permission. For the record I have no issue with anything I said to him or anything he reproduced - if you actually read what I wrote instead of reading the way he has put it together and twisted it then there is no issue.
For MGS4 we had hard data for the Americas - all the normal data we get on a weekly basis. For Japan we had day 1 hard data. For Europe we had a couple of pieces of data from some smaller regions but not the usual data that I use for the weekly sales and we weren't going to be getting that on day 1. I made the decision to take an educated estimate for Europe based on the American figures, past MGS debuts, recent PS3 performance in others and the few pieces of data we did have. So I extrapolated them up using the normal sorts of forumula and filling blanks with educated estimates and scaling. This brought me to ~550k, plus the 400k in Americas and 350k in Japan - which was reasonable. So we went with 1.3m.
Actual data has come in a little lower - UK mainly so in fact first day for Europe was more like 380k. So yes, we overestimated in a preliminary day one article. Which I am happy to admit since there is a certain amount of transparency in our data. I will not give exact methods or data but will give a general idea of figures. For Japan, for example, we were working with figures of 721 and 195 before extrapolation from two medium retailers which we are happy working with to give good figures. For the Americas, we had 412 sold at one group of retailers, 1,328 at another, 7,143 at another and 4,551 at the fourth. Small numbers but covering most major types of retailer - electronics store, gaming stores, toy stores etc and scaled to match total marketshares that those stores occupy. From these we apply the normal factors and get our data out. At the same time we examine marketing budgets, sales of previous games(i.e retailer breakdown for prequels or similar titles), retailer expectations, shipments (~750k in the Americas), retail checks on stores we don't get data for (Walmart apparently had ~4 PS3 bundles per store and and ~30 MGS4s sold day one for example) and we put it all into some formulae to give the final figures. We try to plug gaps in retailers we don't cover using good logic, past trends and by talking to some of our contacts. Gamestop and walmart each represent ~25% marketshare in the US, Best Buy 15%, Target 10%, Circuit City / Toys R Us / Gamecrazy down at 3-4%, then Amazon etc. We have been doing this for long enough and have enough of a grasp on the situation to know how to take the samples we get and arrive at accurate and reliable figures. We are right far more than we are wrong.
Look at last months data. We were the only and the first site anywhere to put Wii Fit week 1 at 750k rather than the 1-1.5 million that everyone was expecting - since we actually knew how many were shipped rather than just speculating on how we think something will sell. We were crucified and told we would be wrong when NPD arrives and we were spot on. Same with Mario Kart, Smash Bros, GTA IV and MGS will be no different.
We do this for hundreds of games week after week and post the data for you guys to enjoy. In our spare time. For free, with very little thanks and only to get articles like this printed and be told we are liars and are misleading people.
Thanks
BHodgkiss | June 23, 2008 2:01 PM
I have done an analysis of VGC -- last year. I tracked VGC estimates, NPD numbers, and VGC revised historical estimates. It's laughable how bad VGC is. I have historical data. VGC is generally off in the range of +/-100%. And I have the data to prove it.
Val | June 23, 2008 2:02 PM
Simon, welcome to the wonderful world of Brett Walton, a "man" (and I use that term loosely) that doesn't like to compare his data to NPD's or any other source in one breath (as he said in this thread, it "is
inherantly wrong) but spits up vile comments at every opportunity about how this firm doesn't track Wal-Mart or that firm's numbers aren't any better than his.
There's so much more to tell about VGChartz that I can hardly stand not to jump on the bandwagon but it's best to let the industry figure it out on its own.
It appears it's well on its way to doing so, thanks to articles like yours.
VGChartz is simply nothing more than a group of two-bit thieves and manipulators who got caught up in a game and are having one hell of a time figuring out how to get out of it while saving face.
Doctor | June 23, 2008 2:42 PM
Doctor (David), don't NPD pay you to actually do any work?
Brett Walton | June 23, 2008 2:45 PM
Hey, Source, you're lying. We've seen an email that's been going around where Brett Walton declares that he doesn't actually track retailers so there's no need to keep the front going. Classified my ass.
You say NPD isn't perfect and that whenever you talk to some of your contacts you hear stories about how NPD changes figures after public release, and spreads out the changes over months, etc., etc.
This is a lie and the fact that you post this and identify yourself as a part of vgchartz is alarming and should be brought to NPD's attention immediately. Don't you think the iundustry would know about this and reporters would report on it? Do you VGCrapperz think we're as stupid as you wish we were?
whaaaaat? | June 23, 2008 2:48 PM
It's hilarious how defensive Brett's getting here. I work for a publisher, I know what our games are selling, and I've seen both NPD's and VGC's numbers for our games. NPD's are always vastly closer to the mark, as opposed to VGC's which have been off by as much as 100%.
It's beyond me why anyone pays attention to them--sure they're free, and regularly updated, but why does any of that matter if they're flat-out wrong? The Onion's a free weekly paper (though not nearly as funny as VGC's numbers) and I'm not dumb enough to look to it for factual information.
S.T.E.P.H.E.N. | June 23, 2008 2:56 PM
Doctor David? I prefer to be called Doctor Loooooove
Doctor | June 23, 2008 3:13 PM
Val,
you said this :
"I have done an analysis of VGC -- last year. I tracked VGC estimates, NPD numbers, and VGC revised historical estimates. It's laughable how bad VGC is. I have historical data. VGC is generally off in the range of +/-100%. And I have the data to prove it."
If it does not bother you, will you post URL to your analysis ? If you don't have it on the internet, you can e-mail me at [email protected]
apujanata | June 23, 2008 6:25 PM
"Well, somewhat spectacularly, in both cases, NPD and VGChartz disagreed by
about 100%. In one case, VGChartz was citing 300,000 sales, whereas NPD
had the game at 150,000 units. And in the other case, it was inversed -
NPD had the game at around 200,000, but VGChartz had it at 100,000."
Lol, and you are not even going to disclose the figures?? I assume these
are the two absolute extremes that you can find out of 1000s of games and
we are to take your word on it??
----
No Brett, according to the article, they picked two random titles given a set of criteria. VGC and NPD disagreed on both.
They haven't disclosed the figures because that data is copyright NPD and costs a lot.
Even if they were the two extremes, how can anyone tell if you just haven't ripped off NPD data like in the other cases?
----
"Look at last months data. We were the only and the first site anywhere to put Wii Fit week 1 at 750k rather than the 1-1.5 million that everyone was expecting. Same with Mario Kart, Smash Bros, GTA IV and MGS will be no different." - BHogkiss
Yes, but the article doesn't raise serious questions over the accuracy for top sellers, but for the other hundreds of lesser games.
"We do this for hundreds of games week after week and post the data for you guys to enjoy. In our spare time. For free, with very little thanks"
Don't take the piss. Sure you might not be getting paid right now. But no one, bar a few high school imbeciles believes that you're running VGChartz for purely altruistic reasons.
Why even hide behind that mask?
kwyjibo | June 23, 2008 7:24 PM
The number of hours that are spent developing vgchartz (the site has all been coded from scratch by myself - as a web developer how long that takes) and the number of hours spent painstakingly putting all the data together each week FAR outweight the couple of thousand dollars that the site makes per month - especially once server costs have been taken out. VG Chartz is run purely for the gaming public.
I have been approached on many occasion by industry firms and asked to sell them data (with some fantastic offers) but have point blank refused them. I have no interest in VG Chartz taking on NPD, we exist for a totally different purpose and the sooner NPD and everyone else understand that the better.
An extract from a post I made on VG Chartz:
"Most people totally understand the point of vgchartz and what we offer as a service.
The simple concept to grasp here is if the data gets better it is no longer free and vgchartz just becomes another NPD and none of you get any data. That would be fine with me, I could retire in a couple of years and live the high life, but that isn't why I am doing this.
The entire point of vgchartz is to offer a simplified, cut-down version of a professional tracking service. We sacrifice accuracy for timeliness and cost. As Robert Passarella said, that is enough for him to use us as one of a number of sources - and that is great but not my focus and not what vgchartz is about.
As I've said before, we are a McDonalds to a gourmet burger. A bit rough and ready, readily available, cheap, frowned upon by chefs but still a decent meal that is good enough for most people who can't afford to spend $20 on a decent burger.
And that is totally fine with me and was the very reason i created the site. We will never start making gourmet burgers because that would be totally against what we are about."
Brett Walton | June 24, 2008 12:26 AM
Brett, by declaring that you actually make money (a couple thousand dollars each month) you are essentially telling the world that you are, in fact, making money from those entities in which you are lifting information. Amazon, NPD, Gamecrazy, Famitsu, etc. Whether it's simple public charts (Gamecrazy, Wal-Mart top sellers, etc.) or more in-depth, illegal use of public information (NPD, Famitsu,etc.), what you're doing is wrong.
This marks the first time that you have disclosed the fact that not only does your site make money but that you are in the black.
You're not bright but you're not completely stupid, either. Of course you cannot sell your reports to any outside firm. Either that, or you don't disclose your complete methodology to those firms so they turn tail once they realize you're hiding something. In selling that data, both VGChartz and the firms that you claim want your data would be in some pretty deep water.
You might want to rethink just how much information you are providing to the general public, especially on something so anti-VGChartz. This could be used against you. In fact, I'd put money on the Gamecrazys and Amazons of the world keeping your statements on file for future consideration. The fact that you're in the UK can only protect you for so long.
Bat Masterson | June 24, 2008 2:23 AM
"The entire point of vgchartz is to offer a simplified, cut-down version of a professional tracking service. We sacrifice accuracy for timeliness and cost." - Brett
That's pretty much what I thought of VGC before the article, and that's pretty much what I think of VGC after the article.
I'm a bit surprised at the vehemently negative reaction from the guys at VGC.
kwyjibo | June 24, 2008 5:21 AM
@Bat Masterson (Mr Riley again I presume)
Nowhere in any email correspondance or any other contact with Mr Careless did I say we lift information from or indeed make any money out of an outside entity. VG Chartz makes just enough to survive and pay for its own costs, the site is definitely not being run at a profit.
My issues with this article are lines such as "That's because VGChartz is - like services such as The SimExchange - using common sense, Internet buzz, real-time data such as Amazon.com and analyst commentary to synthesize a sensible estimate." which is the writer's opinion and based on nothing that I have said. We use real data from real stores and use a combination of factors to scale this data to represent the whole market. By stating thing like that, Mr Carless is basically asserting his own opinion as fact and claiming that it is what I have said and misleading anybody who reads this.
Brett Walton | June 24, 2008 5:30 AM
As I stated on Gamaustra:
A very nice write up. Personally I believe VGC provides a very nice service for those looking to hedge their bets. I do not necessarily buy some of the arguments here and I do not necessarily believe VGC is perfect in their regard to provide clear data. However, their data is fast and normally within a range of estimation.
I read a piece last week about NPD versus VGC from a Wall Street analyst as well and he comes to the conclusion that VGC is quality data. I tend to agree and also agree with his point about having data on the fly 24/7 for those in the market. This debate will probably continue to rage on as there are so many parties on each side of the equation, but it is an interesting one no doubt. The internet age lives on.
Jay Lee | June 24, 2008 5:31 AM
You go ahead and keep presuming things like that, Mr. Paranoia.
You being upset at Simon is ridiculous. There is no reason not to believe Simon but every reason to doubt you. No - and I repeat NO - email correspondence with a reporter is ever confidential unless an NDA is agreed upon up front. In fact, any email that you send to anyone is fair game unless both parties reach an agreement PRIOR to you sending them anything (remember this because it's gonna come back and bite you square in the ass). And you are hardly one to bitch and gripe about taking email conversations out of context and posting them for your audience.
Keep kicking and screaming. I could play tennis with you all day.
Doctor | June 24, 2008 6:34 AM
Posted May 6, 2008:
http://news.vgchartz.com/news.php?id=1126
Posted May 7, 2008:
http://news.cnet.com/8301-10784_3-9937851-7.html
If you want people to take you seriously when writing an article, you should make sure to cover all sides of the argument.
the informer | June 24, 2008 8:46 PM
also, why don't you start hating on Wikipedia while you're at it?
the informer | June 24, 2008 8:55 PM
@informer, actually, T2I was telling press about this the morning of the 6th. I received an email from a T2I contact telling me the exact same thing Once again you take information and falsely tag it as yours.
Why don't you think major media did not pick up your story? And didn't Matt Richtel at New York Times run the print exclusive on the 7th using figures from T2I, which means he would have had to have heard of this from T2I on the 6th in order to make it to press? Damn straight!
IMO, the author of this story did cover all sides of the argument and based on my knowledge and that of my peers he did a damn good job.
GmS | June 25, 2008 2:52 AM
Do it better then.
Anonymus | June 25, 2008 8:38 AM
Sorry to have been out of the loop this week, this story kept me going. Did anyone see the last post on the positive article sight? If anyone is still out there it would be nice to know if there is any truth to it.
and to all a good night [06.26.08 02:38 AM]
an email has been going around supposed from someone close to ioi claiming it was from ioi himself to npd. who knows why ioi sent it around afterwards and why ioi didnt keep it closer to him but i called npd to ask if it was legit and all they said to me was that they will not comment on personal emails. that was enough for me to believe it is legit. much of it doesnt make sense but part of that email reads as follows. you want proof here it is and straight from the horses mouth.
:
All of our figures are essentially "guesstimates", made by a team of people who have extensive experience at monitoring sales figures and using indicators such as positions on Amazon charts, public bestsellers lists online and at stores like Walmart, Target, Gamecrazy etc and so on. All of our data is above board and legal - we basically fit data to chart positions and using historical information for reference. The constant slander that we are doing something illegal by your senior marketing manager is totally unacceptable. At a push, I'd agree we are maybe a little complimentary to ourselves saying we collect data from retailers - in reality we use public chart positions / press releases / our own estimates to come up with figures as well as our own store on the site. So we are collecting retailer data in effect but not actual figures and nothing which would breach any of your contracts.
Dark gKnight | June 26, 2008 11:48 AM
Rumor has it that our buddy at vgcrapz had his ass handed to him today by some Web content providers not all too happy about how he decided to start using their videos on his site. Not only does he steal research and lie to his community, he has lowered himself even more if that was at all possible in the first place, by actually attempting to steal and illegally use branded video content from other sites. Ladies and gentlemen, it just keeps getting better.
rumor miller | June 27, 2008 3:37 PM
Thanks for this; I've always regarded VGChartz's numbers for fringe titles (like Max Payne and Burnout Revenge) to be rather inaccurate. This only serves to bolster my cautious approach.
Fire | July 1, 2008 1:28 PM
Good Article. I'm an investor so I use this data weekly for guidance. It doesn't bother me to see VGC can be low on their sales data week to week. It does help in a big way to have a general guide of overall company sales per week though....keep up the good work VGChartz!
Forager | June 23, 2009 5:45 AM
Glad to see someone took the time to verify the inauthenticity of VGFakeChartz' numbers. If I just wanted someone's uninformed opinion of Day 1 sales of a game, I'd just turn and ask some guy on the street, or roll 2d10 and multiply the results by an appropriate factor of 10.
TiredoftheFlimFlam | April 8, 2010 5:59 PM