I was enjoying the article, but feel like it just abruptly ended. Why was the clone the winner? I would have loved to hear more about your process for judging the successful mockup.
Bump.... if Ugly Banking Site had lowest bounce rate why wasn't it the "winner"? If you're just looking at a static page what else is judged other than bounce rate? If you're measuring conversions, isn't most of that influenced by the existing site? Or do you attribute it to some lasting impression based on how the site was entered? Glad Ugly Banking Site didn't win, but I am curious about the critical metrics.
Fewer bounces doesn't mean more conversions. The winning site might have generated the most tours of office space but the Ugly Banking Site may have generated more overall page views.
I understand & agree, but my point was that conversions would still be accomplished from the existing site, just entered via a static landing page. The conversions weren't coming from the static page, it was just an entryway into the existing site.
Enter = no bounce, Exit = bounce
Correlating stats collected once they leave (click anything) with these static pages is a bit less scientific, so I was interested in hearing the logic.
Good point. I just added this note to the end of the article:
Our criteria for picking the winner came down to one number: tour request rate. In other words, what percentage of users found a listing they liked and contacted it using a particular design. We use other data points to augment this core metric (bounce rate, time on site, number of listings viewed, search criteria revisions), but ultimately a variation only wins if it's better than the control at getting users to an office space that they like.
Nice experiment. Do you have any insight whether this is due to the design being inherently superior, or if it is just the fact that users are familiar with the design due to it being a 1:1 clone of Google?
At our company, we've found that a cohesive design throughout a funnel will always* drive better results. If you think about google as a step in your funnel, then you'll be improving conversion through that same principle.
Also, you've guaranteed familiarity by using Google to buy users. It might be interested to buy users via Bing or Yahoo or something, to see if that same design wins.
But, that's purely gravy. You've probably hit a pretty good balance of testing and results with your experiment already.
Next question: Would it improve your success rate to have multiple versions of your site that echo the designs of the sites that drive most of your traffic, and automatically direct users to the version that matches their referrer?
There's probably more overhead involved than could ever be worthwhile, but now I'm curious.
When we rolled out hover2 in production it looked significantly different from Google, so we were concerned that we might lose the lift. But we didn't. So we chalked the lift up to the hover UX paradigm rather than the visual similarity to Google.
Cool, thanks for the info and the reply. I think there may be something to that type of design for comparing locations vs listings, it is similar to what sites like airbnb use. I generally like to think that those types of sites have done the testing already and came to that design (or stuck with it) for a reason.
I was enjoying the article, but feel like it just abruptly ended. Why was the clone the winner? I would have loved to hear more about your process for judging the successful mockup.