The size of the sample is irrelevant for its accuracy. That seems odd and statistics can be counterintuitive, but consider how a random sample of a few hundred people is enough to estimate who will win the elections within a few %.
What matters - beyond a certain reasonable minimum size, which is a few hundred - is if the sample is random and unbiased. Both NetApps and StatCounter are far, far above the minimum size, so the only questions are the other factors.
But more important to realize here is that they measure different things. StatCounter measures pages viewed, NetApps unique users. There is no reason to expect those to match up.
I (W3Counter, 70k sites) measure unique users, and it's always tracked much closer to StatCounter's numbers than NetApps. Same for every other site that tracks this stuff. NetApps has always been an outlier, especially on their IE numbers.
The difference for NetApps is how they track unique users, particularly for countries such as China.
My understanding is that they basically look at the total number of unique visitors from each country that they get, then weight that number by the number of reported internet users in that country. So if NetApps sees 10 users from country A and 5 users from country B, but country B has twice as many 'connected' people as country A, then they both get equal share.
What this basically leads to is that China, which has a very large population of people who infrequently use the internet but are still counted as internet connected (and may be using shared computers), gets inflated numbers based on total population. This is also why IE8 numbers are so large- because it's still in wide use in China.
The NetApps number probably makes sense if you were to ask 'what is the browser use of the individual users across the entire planet that could possibly load my website'. But when looking at browser usage by # of page loads or predicted visitors, the numbers could very well be much closer to StatCounter.
In that example, it is ok for A and B to get equal share - if you are measuring unique users and not usage. There is no inflation here, at least not in the methodology, which looks like a valid statistical one to me.
Of course it could be wrong in practice, if the data used to re-weight is misleading (while the unweighted raw data was more accurate). That's possible in theory, but seems less likely - there is fairly good information about internet usage in general which is what is used to re-weight, certainly compared to browser share.
It is very important for accuracy. Otherwise there wouldn't be a consistent bias between the two. Uniques vs page views doesn't account the large differences. As you said, independent random sampling would certainly result no significant difference, but neither Net Applications' nor Stat Counter's samples are random. Also, their samples are mostly consistent. When there vast differences between different countries, site topic, and market, sample size is very important.
To your other point, election statistics is much more delicate than you make it out to be. They are also not truly random samples, but they are different for each poll--which helps. They also weight the results by location and expected turn out which is a whole different set of assumptions. If surveying the first 1000 people to answer their phone with consent and using the raw numbers was enough, then there wouldn't be any significant differences in the polls.
A population is a population and a 1% sampling has much less meaning than a 99% sampling, whether or not it is random. A larger sample always has more statistical relevance/confidence.
GeneralMayhem pointed out that this is flat out wrong. Here's an example to demonstrate it:
Consider a population where 99% of users use IE, and 1% of
users use Chrome.
From that population you can draw a biased sample consisting of the 99% IE users and conclude that there are no Chrome users at all.
Or you could draw a random sample of 1% of the population, and assuming that 1% adds up to enough people, your chances of drawing a random sample that did not include a reasonable number of Chrome users too would be extremely low.
So depending on methods used, even a 99% sample may be entirely meaningless when compared to a 1% sample.
If you extrapolate the raw numbers, then yes, you can conclude there are no Chrome users at all. But no one would accept this as fact knowing the sampling methodology.
You can, however, still build confidence intervals based on biased samples. In your example, with 100% certainty, between 0-1% of the population uses Chrome. Yet of course, even the 99.999% CI will be wrong due to the severe bias. Now, if in your example only a 1% biased sample were looked at, the 100% CI would be 0-99%. Much less information. Note that you may also still see trends in biased samples if the sample is consistent.
If biased samples were meaningless, then how are Stat Counter or Net Applications results valuable at all since they are not random samples?
That's true, but if the size of your sample is 99% of the population, that sample is always going to be close to random. For all practical purposes it's not actually a sample any longer.
> It is very important for accuracy. Otherwise there wouldn't be a consistent bias between the two.
As mentioned above, they measure different things - page views vs. unique users. That is, usage vs users. There is no reason to expect them to be identical.
That sample size difference could matter depending on their methodology. If they do countrywise segmentation the sample size for many countries may be very small and any resulting error would get amplified by the weighting.
40,000 sites / 200 countries = 200 sites per country on average. How many participating sites do they have in smaller countries?
It is true that if they have a tiny sample in a large country, their result could be very wrong for that country, and leaving it small and not size-corrected would at least leave that error small.
But, even 200 sites per country is enough for a proper random sample (that's about the same size as in election surveys). Even more so when you consider that a site is not a single observation, but reports data about all the users visiting it.
>But, even 200 sites per country is enough for a proper random sample
Yes, but my concern is that they may only have a handful of sites (or none at all) for a long tail of smaller countries.
>Even more so when you consider that a site is not a single observation, but reports data about all the users visiting it.
True, so it depends on whether or not users of popular international sites from a particular country are representative of all users from that country.
What matters - beyond a certain reasonable minimum size, which is a few hundred - is if the sample is random and unbiased. Both NetApps and StatCounter are far, far above the minimum size, so the only questions are the other factors.
But more important to realize here is that they measure different things. StatCounter measures pages viewed, NetApps unique users. There is no reason to expect those to match up.