Firefox never beat chrome, and I like both of them the same amount.
When I first did the data dive, I thought Safari was worse than IE, which I thought made a much better blog post. That would have been way more surprising. It was after due diligence that I was forced to discard that result.
Firefox 4 and 5 in the first graph are significantly lower than Chrome. Obviously it's a pretty dodgy comparison to make since Chrome isn't split by versions, but I agree with hammock that it seems statistically unsound to arbitrarily drop some versions of some browsers, especially when they can't do the same for the Chrome results.
"All jobs run recently have lower error rates because our service continues to become more reliable as we fix bugs our users discover. "
This says to me that none of the results can really be trusted because the error count is not just errors in the browser - it's apparently significantly affected by errors in their system, and we don't really know by how much.
Plus it seems kind of anachronistic to come to some conclusion about how the browsers compare having removed FF4 and FF5. It's only of historical interest how FF3.6 compares since anyone that downloads it now is going to get FF6 now anyway.
It may be true that SauceLabs's code for each browser is the same, but Selenium surely has significantly divergent code for each browser. I wonder how to rule out crashes caused by Selenium's browser-specific code...
At first we thought the high error rate could be the result of the fact that we always run Safari on Windows, while it’s made by Apple. That’s easy to dismiss, because earlier versions of Safari were fine.
The earlier data points on the graph show low error rates for Safari browsers on Windows.
The article may have benefited from including a larger caveat about only testing one OS, but this seems like an intellectually honest elimination of a confounding variable.
Perhaps intellectually honest, but a weak conclusion nonetheless: in my understanding, getting Safari 5 to run on Windows requires porting parts of Apple's huge, complicated Cocoa framework to Windows. That framework is always changing, and so is the browser-- and they both primarily target OS X. With OS X, Windows, Cocoa, and Safari all moving quite a bit between Safari versions, I would be surprised if it didn't become harder to support Windows as the product grows.
"all FF3.5 jobs are from a long time ago, before we’d had much time to streamline and errorproof our system."
"This is the same graph as above, with the unfairly advantaged or disadvantaged browsers removed"
What? Why not take out all results from the period when your system was unstable, rather than removing the results that looked 'wrong' to you? Good grief.
The system was never very unstable. The error rates are all low. The point is that it's now much more stable.
The reason I didn't sort error rates by month was to preserve statistical significance. If we try to only look at error rates from specific periods of time, we have to decide how short those periods of time are. Too long and we don't have data from some of the browsers; too short and we don't have enough data to form a significant opinion. It may have been possible to find a middle ground, but why bother when we can still get unbiased results for 80% of the browsers?
I get what you're saying, but you still removed data that you thought was wrong in order to give the results you were expecting. Who's to say that the results for Chrome in that period were not also wrong?
What is the point of this article? It appears to be that Safari proxy mode is unreliable, which I guess is surprising but not that interesting.
Given that the actual article does not have anything to do with the headline ("error rates" is never really satisfactorily defined, the real story is proxy mode Safari, and IE6 came out 9 years ago), what makes this headline so compelling?
Do headlines with the word "surprising" always work? Is it that everyone deals with browsers, so a browser related story is likely to have a wider mind share? Is it that web developers all just want a chance to make fun of IE6, due to the collective pain that it has inflicted on all of us?
Oh look, free, anonymous Windows VMs for 20 minutes (it's trivial to open CMD.EXE from any browser in the saucelabs setup). Any statistics on what people run in those VMs, other than browsers?
1. Go to File > Open, then click Browse...
2. In 'File name', type 'c:\windows\system32' and press enter
3. Change the 'Files of type' field to 'All Files'
4. Drag any of the files into the 'cmd' icon
Bonus: type `explorer C:\` if you want to snoop around. There's a copy of Safari at C:\Program Files\Safari\Safari.exe
Regardless of possible testing or analysis concerns, it's pretty clear that Safari 5 has a higher error rate than Safari 4 or 3. Safari is the only browser to go down in quality over time. This appears to be further evidence that Apple is putting less resources towards Mac software in favor of supporting ios devices. Fairly concerning for all of us Mac users.
So you claim to have "significant data" but the first thing you do is throw out huge swaths of it because it's not actually significant?
Also regarding graph 2's commentary, 0.51 is not "almost" double 0.34. It turns out to be only 1.5x, so calling it "almost double" is about as meaningful as calling it "almost the same".
I'm usually a safari fan, but since upgrading last week nothing but trouble. Opened Chrome and Firefox a few times instead but I'm to accustomed to Safari now... and admittedly the gestures are AWESOME.
I was hoping the 10.7.1 update would have fixed it but nope :/ Maybe 10.7.2, I feel my days numbered ATM though
"Look, now Chrome is beating Firefox. But IE still isn't losing yet, so let's make up some reasoning and fudge some more until we're happy!"
"Ah, that's better, now IE looks like it sucks and my worldview is no longer shattered. Hooray for fudging data to make it say what we want it to!"