Cherry Picking

Cherry picking is defined by Wikipedia:

Cherry picking, suppressing evidence, or the fallacy of incomplete evidence is the act of pointing to individual cases or data that seem to confirm a particular position, while ignoring a significant portion of related cases or data that may contradict that position

RationalWiki refer to it as:

Cherry picking, when used figuratively, refers to selective extraction of points in an argument in order to refute or affirm them while ignoring others which will not support the point(s) being made. Often, these cherry-picked factoids or references will be over-extrapolated and oversold to give the impression that they are representative, when they are not.

A now classic example of cherry picking in the context of the topic of this blog is spin on the paper by Daoud et al, that was widely touted in the minimalist and barefoot communities as proof that what they are doing is better. Vivobarefoot even produced a poster claiming it was proof that barefoot was better (despite all the subjects in the study actually wearing shoes!…don’t figure)¹.

barefoot_is_best

The study by Daoud et al (2012) was a retrospective review of 52 almost elite level track runners (that are not close to being representative of typical runners) which found the injury rate in the heel strike group was almost double the forefoot striking group. Vivobarefoot in their claims cherry picked that study, ignoring the work of Kleindienst (2003) (471 runners; no difference between rearfoot and forefoot strikers concerning the frequency of injury) and Walther (2005) (1203 runners; no difference in incidence of injury between rearfoot and forefoot strikers). Subsequent to this Vivobarefoot debacle there is now a fourth study on injury rates in forefoot vs rearfoot strikers. I will do a blog post analyzing the four studies soon. For more discussion on this Vivobarefoot debacle and the study, see Podiatry Arena and RunBlogger. It certainly damaged their brand.

Another example of cherry picking which also has elements of confirmation bias in it, is when insignificant or minor aspects of a study’s methodology is blown way out of proportion in an attempt to invalidate the study. For example, see what I wrote about the responses to the bone marrow edema study. Of course, all research should be critically appraised, but some will go to extraordinary lengths to discredit a study, grasping at straws if they do not like the results of the study, but then fail to hold up research that they do like to that exact same standard. You can not have it both ways. That is cherry picking at it best!

Another example is the use of the study from the late Alex Stacoff (RIP) and colleagues: Effects of foot orthoses on skeletal motion during running. Stacoff A, Reinschmidt C, Nigg BM, van den Bogert AJ, Lundberg A, Denoth J, Stüssi E. Clin Biomech (Bristol, Avon). 2000 Jan;15(1):54-64. This was study on the mechanism of action of foot orthotics and nothing to do if they worked or not (the evidence is clear on them working). The study is often held up as an exemplary piece of work as it used surgically implanted bone pins to measure the kinematics (~motion) of the bones and in many regards it is of high standard. Yet despite this, you often see this study cherry picked out of the 100 or so other kinematic study’s as evidence that foot orthotics don’t work, as the authors found that the “foot orthotics” did not change skeletal motion. However, all the subjects in the study: “were all injury free, were no overpronators, and had no previous injury history which may have influenced their locomotion patterns“, so the participants used were of the type that did not even need foot orthotics! The “foot orthotics” used in this study were not close to being of the type that clinicians even use in clinical practice and really just consisted of 1cm thick cork arch cookie shaped pads. Even the harshest critic of foot orthotics would have to agree that they were not “foot orthotics”. So all this study really found was that foot orthotics of the type that no one uses in clinical practice did not alter the skeletal motion in people who do not need foot orthotics! Yet despite this, this study is cherry picked ahead of others by the critics! … not to mention the only 5 subjects, but I won’t go that way as I think you see the point I am making on cherry picking and how it gets used.

As always, I go where the evidence takes me until convinced otherwise and if I ever engage in cherry picking, please tell me as it would not have been done intentionally and I will fix it.

¹For more discussion on this Vivobarefoot debacle and the study, see Podiatry Arena and RunBlogger. It certainly damaged their brand.

Last updated by .

No comments yet.

Leave a Reply