Understanding Educational Research - Problems With Secondary Sources

(Continued from part 2 - Understanding Peer-Reviewed Educational Journals)

The Internet, and better access to information has made it easier to come across reports on educational research. Many of us, in fact perhaps the majority of us don't have the time to spend a year or two reading research journals, so we rely on secondary sources, notably research summaries, news articles about specific research, or books. The latter has become far more common, as more books are published than previously, a kind of POP-Psychology approach but for education. For example, there are many many books on learning styles and how they affect teaching and learning.

Pop Education Books And The Research They Report

Over the last while, we've seen a growth of books on education, written for popular consumption. For example you may be familiar with Alfie Kohn, who wrote "Punished By Rewards", and other books of similar themes. Kohn's book, and others from others like Daniel Pink, and Malcolm Gladwell, end up popular and often quoted. if you read them, you'll find dozens of research studies cited to support a particular contention the author is presenting. In Kohn's case, his book contains citations to studies that purport to support his conclusions on the potential damage of extrinsic rewards.

These books tend to end up being quite influential.

Some Things You Should Know About Pop Ed. Books

Books Are For Profit Enterprises

It should go without saying that pop ed books are published for profit. There's nothing wrong with that, at least on the surface, and neither does it invalidate the motives of the author or the accurateness of the book content.  However, as an author of a number of books published by major publishers, I know a little about what does and doesn't get published. The publisher is motivated by sales. It's that simple. Major publishers will publish books they think will sell. Period.

There's nothing inherently wrong with that, at least until you understand what makes a book publishable, and marketable. And then you'll see the problem with books by Pink, Kohn, and Gladwell.

Publishable Book MUST Have A Strong Point Of View

Books that sell well, and hit the best seller lists (the gold publishers look for), have a strong point of view. While journal article are deliberately written using an objective writing style, hence they are always boring to read), and need to sound objective, books that sell must have a point of view. They must contain a major theme, some clear conclusions, and also titles that are marketable.

Kohn's book "Punished By Rewards" has a particular point of view, whether the author started out with trying to prove a particular point (bias), or whether he didn't. It tries to point out the dangers of rewards, and the studies included in it support that point of view.

If, for example, the book was entitled "Rewards Can Be Good and Bad" (which would be far more accurate), and it contained articles chosen to present a balanced and accurate view, the chances are that book would never be published. Who wants to read a book that says: "Well, sometimes rewards are a good thing, but sometimes rewards are a bad thing".

That's not to cast aspersions on the morality of these writers. However, the book publishing process pushes prospective writers to take a position, THEN look to support that position, rather than to start with a question: "Are rewards good or bad, and when are they good or bad?, and present a balanced view.

So what happens is that often the best sellers, the pop ed. books, contain lots of references, but we don't know whether the author has "cherry picked" the articles to present a particular point of view decided in advance.

Punished By Rewards - Questionable Interpretation

A few years after Kohn's book came out, and created quite the storm, I decided to look at his primary sources that he used to make his points. As a former social science researcher, and educational journal board member, I became curious as to the accuracy and quality of the research Kohn cited in one particular article he published, based on the book.

I tracked down a number of his citations, and when possible read the original research. What I find was rather distressing.

  • In some cases the research cited was so badly done, or contained so few subjects (sometimes as little as eight or ten people), that the research was worthless, even if it was published in a journal.
  • In some cases, Kohn did not appear to properly reflect the conclusions of the researchers themselves, seeming, at least to me, to take some liberties in going beyond the conclusions of the original authors.
  • In a few cases, I found that the research could easily be interpreted to mean the exact opposite of what both the researchers and Kohn thought the research results meant.

If you read ONLY Kohn's book and didn't look at any of the cited research, OR you looked at the primary research and lacked the expertise in research design, Psychology, and statistical analysis, you would "believe" Kohn's interpretation.

This a problem with Kohn specifically. But it did point out that relying on pop ed books to guide educational practice is highly problematic.


The Problem: Lack Of Competence

Kohn, Gladwell, and Pink are very smart guys. They have managed to write books and articles that are interesting, marketable, and sometimes quite compelling and even fun (I quite like Gladwell). But here's the problem. As best as I can tell, Kohn, Gladwell and Pink are writers and journalists. They are, and seem to never have been researchers, statisticians, psychologists, or have had enough formal training in research to be competent in reading and interpreting research studies.

This isn't a knock on any of them. We all know some things and don't know other things. The problem here is that it's dangerous to rely on third parties to interpret research findings -- to rely on their abilities to accurately make sense out of them when they don't have that formal training. There's an art to understanding research, to be able to find the holes in the research, to examine the methods used and the logic and to determine if the statistical analyses were appropriate and done correctly.

Yet, we often look to book authors who have no experience doing this kind of work to accurately reflect the results, and to do so in an unbiased way.

Implications For Educational Decisions

The danger of pop ed. books is obvious.

They appear to be scholarly, objective and even detailed, but we don't know the skills of the author in making sense of the research, whether the author started out with a conclusion, THEN looked primarily for evidence to support that conclusion, or whether the author started from a neutral point of view.

The appearance of objectivity and scholarliness makes the books very compelling when coupled with their popularity. Books that hit the best seller list don't necessarily do so because they are "right", but because they are well marketed, or controversial or have something else going for them. Thus, there's a tendency to "believe" that popular books are true and accurate, when they may not be.

There are some exceptions to this. For example, there are books out there that have been written by scholars, and researchers who have worked in their fields for decades. In other words, the authors are researchers FIRST, and authors second. Usually, you will find these more reputable experts have published a number of articles in journals, BEFORE writing pop ed. books. That's different from the people who are writers first, and publish their books FIRST, and then write a few journal articles.

The upshot is that while books like Kohn's Punished By Rewards are compelling, and appear to make strong, unbiased cases for a particular point of view, they may very well be biased, set up to sell rather than to inform, and have the potential to drive educators and teacher to make decisions based on exceedingly faulty information.

Next up and coming soon. What about other third party research sources?