That's true, but trivially so. Things that aren't infallible can still be useful, but understanding their limitations is important.
If you read that blog, you can find plenty of discussion of those limitations, for example, how the absence of markers of digital manipulation does not prove that an image is genuine. After all, there are plenty of staged photos out there, images that were framed in a misleading way, etc.
I'm not making a trivial point though. I believe the tools, techniques and expertise this guy professes in image analysis are modern day snake oil. I just think that he has deluded himself instead of being deliberately dishonest.
How then do you explain the fact that his methods have worked? He has outed doctored photos that appeared in news stories and they have been retracted after investigation.
That's not quite what I'm saying, I don't think his tests are scientific not that they show absolutely nothing (although probably fairly close to nothing with ELA). I also haven't doubted his ability to detect fakes but I do doubt the particular efficacy and explanation of his techniques. There is no doubt using PCA can highlight unusual changes in an image - but there are also other explanations. His famous suggestion that terrorist videos had books inserted into them could simply have been a slightly different coloured spot light.
If his techniques are unreliable, he should have some notable failures by now after having analyzed so many images publicly. Where can these failures be found?
No not really, he techniques may be unhelpful and he still might be good at picking fakes. But analyzed so many images? There is a handful on the site and the outcome is almost never in doubt. You could test his ability to detect fakes but even if it were supreme you really couldn't separate that into the portion provided by his skills and those of his techniques.
Much more interesting would be to find out what the techniques really show, which faking techniques show which signatures and what other natural occurrences mimic that.
What about the times he found cheating by the winners of photographic contests? Those were hardly obvious choices. And falsely accusing someone would have really hurt his reputation. Of course, investigation proved that he was right. He analyzes something every few weeks it seems like (I've read the blog for years now), so yeah, there's a long history for you to look at. The outcome is only "never in doubt" if you're using hindsight bias. Besides, even when he already knows an image is fake, he figures out what is fake and how it was faked. One example would be showing which of those lottery numbers was real: the whole row was fake... except for the last number. Something which helped explain why the 2nd and 3rd photograph were the way that they were.
He's quite up front about the fact that some tests give inconclusive results. I believe that he has discussed the limitations of the tests. But even a test that gives an inconclusive result half the time can be useful if it shows you which areas of the image need attention the rest of the time. That data is indeed useful. He has talked about it. And you could just politely email him if you wanted to know more.
And yes, it probably is hard to separate the success due to technique and skills. But it's hard to believe that he would be good at picking out fake images without understanding why he was good at picking them out.
Well I enjoyed the discussion but I don't think I'm making the point especially well anymore, its not about picking fakes, its about reliably saying something about the numbers return by a mathematical function on an image. It needs more study and less experts making consulting money from their special magic.
