Well, we have made the experience that people that are not seriously considering to take the job are filtered out by a homework. It helps both to save a lot of time and energy interviewing people.
I'm genuinely not trying to be snarky, but how can you say that with any certainty?
The only way to know this for sure would be to ask half your applicant pool to do the test, and the other half to go through the exact same battery of interviews sans take home test and then see how many of the offers you extend are accepted.
Obviously this requires that you're a big enough company to split your applicants into two pools and still have a big enough sample size.
Whenever I talk to people involved in hiring about this, they balk at the idea (for okay reasons -- in theory doing that would mean less people filtered out by homework and more applicants rejected during in person interviews that take up engineer time.)
I guess you are right, we haven’t done a proper A/B test for this. We had no homework before however, and the overall results were different. It is my strong feeling though that many cancel after seeing the homework that are not completely convinced, which is ok.
