Understanding Educational Research - Looking At The Sources

(continued from Part 1)

Most of us, and that includes teachers, school administrators and policy makers don't read the hundreds of studies available on any one particular topic. Let's look at the various sources where most of us learn about research and research findings.

Peer-Reviewed Journals - The Gold Standard

We should put the most credence in research that's published in peer-reviewed journals, but we'll also look at some potential flaws in the publishing system that can cause you to get the conclusions wrong, even form research in those journals.

Here's How Peer-Reviewed Journals Work

How do research studies get published in these journals? A researcher or research team will do some sort of study, write it up, and then send it on to a journal or journals that tend to publish research on that specific topic. They receive no pay for submitting, or getting published.

An editorial board will review each submission to screen out research that is off-topic, obviously flawed due to poor methodology or logic errors, or is stylistically inconsistent with the journal's approach. Those submissions are rejected out of hand (usually), but occasionally may be returned to the author for revision along with some suggestions.

More often, those that survive the initial screening will be assigned to several independent experts in that particular field, where the reviewers have volunteered to be independent and anonymous reviewers. The names of the authors are removed prior to forwarding to the external reviewers. try to eliminate bias creeping in on the basis of WHO did the research.

The reviewers are expected to tear the research apart, and to submit fairly detailed critiques back to the editorial board, and to make a recommendation as to whether the article should be rejected outright, accepted for publication, or should be sent back to the author with a request for specific modifications.

Then, the editorial board makes the final decision as to what happens next, whether they will publish, and when they will do so. Because space is limited, and there are far more submissions than can be published, more articles are rejected than accepted, and even for accepted articles, there can be long times before publishing, as much as a year or more.

So, if it sounds like the process is made as objective as possible, it is, but only kind of. Editorial boards have biases. Reviewers can have biases. Hopefully they get cancelled out.

Two Major Problems Even With Peer-Reviewed Journals

Peer reviewed journals are the gold standard, but they are far from perfect. There are two things you need to know.

Bias Towards Publishing POSITIVE Results

Pretty much every journal has a bias towards publishing research that has a positive outcome. Let's look at a simple example. Let's say there's a research article that shows that a certain approach to teaching has been show to improve children's reading scores. That's a positive result. On the other hand we can also have a study that shows that the same method does not show any improvement in reading scores, all things being equal.

The study with positive results is MUCH more likely to be reported and published than would be the study that shows no difference or advantage for this new approach to reading.

You should be seeing the implication here. Even if you read all the journals you can on a topic, and as many studies as you can, you will be far more likely to read positive results compared to negative results. Negative (no difference) results aren't sexy, or all that notable, and often simply do not come to light so you can find them. This is a historical thing that has been going on for decades.

That's not to say negative results are NEVER published, but they are published far less often, and that skews what you see, and the conclusions you can draw.

Journals Biased Based On Subject Matter

This gets a little sticky. While it shouldn't be the case, journals have a "point of view", based on their subject matter. Each journal has a set of implicit or explicit assumptions about its specialization. For example, the fictitious Journal Of Freudian Psychoanalysis (JFP) has a set of beliefs about Freudian Psychoanalysis; that it's a worthwhile subject to investigate, that it has value, that looking at things like ID, Super-Ego, and Ego are important.

However, the equally fictitious Journal of Skinnerian Behavior Modification has a completely different set of assumptions that contradict those of the JFP. Each journal has a built in and very strong bias about what it publishes. That affects what you see.

Here's an educational example. There are several specialized journals on psychological type (MBTI) and on leaning styles. Those journals operate on a set of assumptions about the value of psychological type, or learning styles. As such they are far less likely to publish research articles that basically suggest that the MBTI is a waste of time, or that learning styles have no positive implications for teaching. It's not that they will never publish such articles, but the overwhelming number of articles published are consistent with the specific journals' point of view, and biases.

The editorial board, and the external reviewers for those journals are not likely to be critics of the MBTI or learning styles. In fact, they are likely people actively involved in similar research (that makes them experts), and they hold similar views.

Going back to our example of the Freudian vs. Skinnerian journals, a Freudian journal is NOT going to have a Skinnerian behaviorist as a reviewer, and vice versa.

The result is that what you see in these journals is subject to a very strong form of bias in favor of the specific niche it operates in.

Implications For Practice

While peer-reviewed journals are the "gold" standard, and it may be the best system we can hope for to eliminate bias, it's still not terribly good. Since our purpose with research is not to find "truth" but to stimulate more research, it's not a huge problem for researchers, but it IS a huge problem for decision-makers, teachers and so on that want to use research findings to guide practice.

That's because what you see in research journals is not a complete reflection of all the research. The bias in favor of positive research, and the bias of speciality journals (i.e. journals that look at psychological type like the MBTI) have a built in bias that will make the concept appear more useful than it is.

Which again, highlights that if you want to make decisions about educational practice, you need to look at studies from different researchers, and from as many different journals as possible.

But you may say: "Well, most of the time I don't read the studies in journals, but hear about the findings from other sources, let's say news reports, or books, or conferences, and other secondary and tertiary sources.