The review article "The Validity and Utility of Selection Models in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings"
sums up, current to 1998, much of the HUGE peer-reviewed professional literature on the industrial and organizational psychology devoted to business hiring practices. There are many kinds of hiring screens, such as resume reviews for job experience, telephone interviews, in-person interviews, checks for academic credentials, and so on. There is much published study research on how job applicants perform after they are hired in a wide variety of occupations.
The overall summary of the industrial psychology research in reliable secondary sources is that two kinds of job screening procedures work reasonably well (but only about at the 0.5 level, standing alone). One is a general cognitive ability test (an IQ-like test, such as the Wonderlic personnel screening test). Another is a work-sample test, where the applicant does an actual task or group of tasks like what the applicant will do on the job if hired. Each of these kinds of tests has about the same validity in screening applicants for jobs, with the general cognitive ability test better predicting success for applicants who will be trained into a new job. Neither is perfect (both operate at about 0.5 level in validation studies), but both are better than anything else that has been tested in rigorous research, across a wide variety of occupations. So if you are hiring for your company, it's a good idea to think about how to build a work-sample test into all of your hiring processes.
For legal reasons in the United States (the same consideration does not apply in other countries), it is difficult to give job applicants a straight-up IQ test (as was commonplace in my parents' generation) as a routine part of a hiring process. The Griggs v. Duke Power, 401 U.S. 424 (1971) case in the United States Supreme Court
held that cognitive ability tests used in hiring that could have a "disparate impact" on applicants of some protected classes must "bear a demonstrable relationship to successful performance of the jobs for which it was used." In other words, a company that wants to use a test like the Wonderlic, or like the SAT, or like the current WAIS or Stanford-Binet IQ tests, in a hiring process had best conduct a specific validation study of the test related to performance on the job in question. Some companies do the validation study, and use IQ-like tests in hiring. Other companies use IQ-like tests in hiring and hope that no one sues (which is not what I would advise any company). Note that a brain-teaser-type test used in a hiring process might be illegal if it can be shown to have disparate impact on some job applicants and is not supported by a validation study demonstrating that the test is related to successful performance on the job. Companies outside the United States are regulated by different laws.
I'm not too familiar with the US justice system, but I read the Wikipedia page the last time you brought up that specific case. Reading (a little) between the lines, Duke Power seems to have set out to discriminate against black employees. That seems very different from using an IQ test to select the best employees, especially for a job that actually is highly IQ-dependent.
It takes one bad apple to ruin it for the bunch, as they say.
That said, I lived and worked for a few years in a country where IQ tests were the norm for hiring people. When hiring for my own team (C++ devs), I did not use them as I found they were a very poor filter with both high false positives and high false negatives in terms of indicating whether or not you should hire a particular person.
Intent is often irrelevant. If something is shown to have disparate impact on a protected class then it is likely to be ruled discriminatory, and therefore illegal, regardless of intent.
You could argue -- probably with varying degrees of success, but still -- that the puzzle question is an attempt at an IQ question. Certainly, there are vaguely similar questions of reasoning ability, numeracy, and pattern recognition on actual IQ tests.
Anyone who's taken one of the commonly accepted IQ tests will recall that it's full of sequential pattern matching, word-unscrambling, and so forth. And I realize that most here have probably not taken the LSAT for law school, but that test is essentially a string of interview-style brainteasers. (The LSAT was initially derived from IQ tests, though it should be said that this topic is not without considerable historical controversy).
The last time I hired I asked two puzzle-type questions during the interviews: a math (statistics) puzzle and a programming puzzle.
The math puzzle had a definite correct answer that can be arrived at by anyone remotely qualified for the position, but which causes even statistics undergrads to scratch their heads for a minute. The real purpose of this is to see if someone is able, willing, and perhaps even eager to actually think.
The programming test does not have a single correct answer. As far as I know there's always a trade-off depending on the importance of competing priorities. I tried to lead them into discussing the design differences to handle the different scenarios. It's very interesting to observe the thought process. Some people think of one fairly workable design and then they are done. Period. Other people realize that a good design in one scenario is suboptimal in another and try a new design. Some go farther and look for an 80% solution to both.
You can call this screening to some degree. It's also about discovery, to be part of the larger picture of how this prospect would work with our team.
You have 50 red marbles, 50 blue marbles, and 2 buckets. You must put all the marbles into the buckets but you may distribute them any way you like. I randomly pick a bucket, and then randomly select a marble from that bucket. How, if possible at all, can you maximize the probability of me picking a red marble?
My intuitive answer would be to put one red marble in one bucket, and the other marbles in the other bucket, making it 100% likely to get red if you picked bucket 1 and almost 50% likely if you picked bucket 2, for a combined probability of just under 75%.
The expression I would want to maximize would be (r/(r+b))+((50-r)/((50-r)+(50-b))), where r and b are integers between 0 and 50 inclusive (I've forgotten the calculus required for this).
Most people somehow get stuck assuming they must distribute the marbles evenly (always 50 marbles in each bucket), even though it's neither stated nor implied. It's interesting to see how long it takes people to challenge their own assumptions.
I expect I would get stuck wondering if I'd be out on my ear if I answered that you could put all the marbles in one bucket, take them back out, and then put one red marble in each bucket.
TLDR: Note that a brain-teaser-type test used in a hiring process might be illegal if it can be shown to have disparate impact on some job applicants and is not supported by a validation study demonstrated that the test is related to successful performance on the job
Thanks, tokenadult. This is an interesting result, and may give pause to some of the readers here.
I doubt it's actually true, in that brainteasers aren't scored on a pass/fail basis but are used as tools to see "how you think".
Still, don't give Al Sharpton any ideas, or brainteasers may be the new frontier for anti-discrimination lawsuits.
Perhaps the end-game is a system where job interviews are illegal; instead, you tell the Department of Labor your needs and the Department of Labor (in fourteen to eighteen weeks) sends you your new employee, chosen according to race, gender, sexual preference and [if at all possible] qualifications.
Suppose you flip a coin on 100 candidates, one of whom is qualified. You'll get about 50 "hire" answers, and the odds are 50% that one of those hires is actually qualified.
The article is referring only to employees that were hired and were also given the GMA, so qualification was not an issue.
However, using a coin flip rather than a GMA score on a prospective employee would imply a score of either 0 (missed 'em all) or 100 (perfect), and thus would likely have 0 correlation with the actual employee performance, rather than .51.
So your conclusion is right even if your analysis is not.
http://mavweb.mnsu.edu/howard/Schmidt%20and%20Hunter%201998%...
sums up, current to 1998, much of the HUGE peer-reviewed professional literature on the industrial and organizational psychology devoted to business hiring practices. There are many kinds of hiring screens, such as resume reviews for job experience, telephone interviews, in-person interviews, checks for academic credentials, and so on. There is much published study research on how job applicants perform after they are hired in a wide variety of occupations.
The overall summary of the industrial psychology research in reliable secondary sources is that two kinds of job screening procedures work reasonably well (but only about at the 0.5 level, standing alone). One is a general cognitive ability test (an IQ-like test, such as the Wonderlic personnel screening test). Another is a work-sample test, where the applicant does an actual task or group of tasks like what the applicant will do on the job if hired. Each of these kinds of tests has about the same validity in screening applicants for jobs, with the general cognitive ability test better predicting success for applicants who will be trained into a new job. Neither is perfect (both operate at about 0.5 level in validation studies), but both are better than anything else that has been tested in rigorous research, across a wide variety of occupations. So if you are hiring for your company, it's a good idea to think about how to build a work-sample test into all of your hiring processes.
For legal reasons in the United States (the same consideration does not apply in other countries), it is difficult to give job applicants a straight-up IQ test (as was commonplace in my parents' generation) as a routine part of a hiring process. The Griggs v. Duke Power, 401 U.S. 424 (1971) case in the United States Supreme Court
http://scholar.google.com/scholar_case?case=8655598674229196...
held that cognitive ability tests used in hiring that could have a "disparate impact" on applicants of some protected classes must "bear a demonstrable relationship to successful performance of the jobs for which it was used." In other words, a company that wants to use a test like the Wonderlic, or like the SAT, or like the current WAIS or Stanford-Binet IQ tests, in a hiring process had best conduct a specific validation study of the test related to performance on the job in question. Some companies do the validation study, and use IQ-like tests in hiring. Other companies use IQ-like tests in hiring and hope that no one sues (which is not what I would advise any company). Note that a brain-teaser-type test used in a hiring process might be illegal if it can be shown to have disparate impact on some job applicants and is not supported by a validation study demonstrating that the test is related to successful performance on the job. Companies outside the United States are regulated by different laws.