FORUM on Corrections Research

Buyer beware: A consumer's guide to reading and understanding correctional research (Part II)

In the September 1993 issue of FORUM, we presented an article looking at some important, but often neglected, questions that we should ask about correctional research. That article wasn't the end of the story. There are other issues we should be aware of when reading about correctional research. Some of these issues are presented here, in the second article in this series. What is the question being answered? In his book Technopoly, social critic Neil Postman relates a tale of two priests who write to the Vatican for advice on smoking and praying at the same time. One priest asks whether it is all right to smoke while praying. The answer he receives is "No, because one's whole attention should be upon prayer." The other priest then asks whether it is all right to pray while smoking. The answer he receives is "Yes, because it is always appropriate to pray"⁽²⁾

In research, the question asked greatly affects the answer given. If we ask, for example, whether a certain program reduces recidivism (defined here as a reconviction), we can look at two groups - one consisting of offenders who were involved in the program and the other consisting of comparable offenders who were not - and compare the incidence of recidivism among members of each group.

If, however, we ask whether this program reduces crime among its graduates, we cannot get the answer simply by counting reconvictions. The reconviction rate tells us only about offenders who got caught and reconvicted. It provides no information about offenders who may have successfully continued their criminal career or who "beat the system" at some point after arrest. Further, if the reconviction rate goes down or stays the same, it could be the result of either intended or unintended effects of the program on any number of offenders - or not.

Of course, the nature of the program has a lot to do with the plausibility of alternative interpretations, but this illustration points out the fine distinctions that must be made when interpreting a research finding. The problem of measuring what we can't see To take the above analysis one step further, there really is no perfectly reliable way to determine whether program X reduces crime among its graduates. The only crimes we know about are those for which there were convictions.

Even if we followed up released offenders for 20 years, we would not likely get them to admit to crimes they got away with. Indeed, it is hard enough getting offenders to admit to the crimes for which they have been convicted.

But the problem is not simply crimes remaining hidden to researchers.

Our theories of criminal behaviour require us to measure some things that, for various reasons, are not easily measured. If something is required for a theory but cannot be observed directly, we call it a "construct" and try to measure it indirectly.

Intelligence is such a construct. We often want to explain the differences we observe among various individuals' problem-solving abilities. But, unlike shoe size, age, or number of convictions, intelligence cannot be measured directly. In fact, there is great controversy among psychologists about whether intelligence can be measured at all.

Therefore, if we, as consumers of research, put any faith in a study that uses an intelligence quotient (commonly known as IQ) as a predictor of anything, we are implicitly buying into one side of a rather heated controversy.⁽³⁾ Interpreting measurements of constructs A popular saying among psychologists goes something like "Intelligence is what intelligence tests measure." We must, therefore, be very careful in interpreting the results of studies of this construct.

Certain skills are required to perform well on an IQ test, and others are not. For example, a test that measures the ability to remember and use complicated numbers may leave us uninformed about the subject's ability to remember a complicated musical score. Is musical ability not at least an indication of intelligence? If it is, shouldn't we test for it? If it is not, then why do we regard Beethoven and Mozart as geniuses?

We would not naturally assume that an outstanding physicist would also be an outstanding pianist. By the same token, we should not naturally assume that an offender who lacks the ability to read would also lack the ability to crack a safe. The skills required for one activity may or may not be useful for another. Testing for one type of skill is not the same as testing for all.

Researchers are human, however, and some have been seduced into judging a person's intelligence solely by his or her score on an IQ test. We must remember that what the data show will be limited by the question(s) asked, and our conclusions must be limited to the context of the question(s) - nothing broader. Unfortunately, this sometimes does not prevent researchers from trying to use the answer to one question as though other questions had been asked. Surveys: Where the question really counts Surveys and opinion polls take researchers' questions directly to the people being studied. Therefore, the precise phrasing of survey questions is critical, and the questions must be understandable to people answering them.

Consider the difference between "Who would be the best prime minister?" and "Which party would you vote for if an election were held today?" These two questions would elicit two different answers from most people. Yet, the newspaper headline would read "The XYZ Party leads in poll," regardless of which of these questions had been asked.

To borrow an example from Postman's Technopoly (modifying the content to suit the present topic), let's assume that controversy erupts over the sentencing of an offender convicted of manslaughter. The individual is sentenced to 10 years' incarceration, and a ban is placed on the publication of trial information. However, the homicide was particularly grisly and many people are calling for a life sentence.

Given the outcry, we would likely soon hear about a poll indicating that something like "80% of Ontarians favour a review of this case and the imposition of a life sentence."

But our imaginary poll results could also have indicated that, of the 80% favouring a life sentence,

• 60% knew what the offender had been charged with;
• 10% knew what the average sentence for manslaughter is;
• none knew what evidence had been presented; none were aware of the judge's reasons for the sentence; and
• 40% knew the difference between murder and manslaughter.

Unfortunately, this information would probably not even be obtained, much less reported, because the questions would never have been asked.

As Postman writes, "Were pollsters to provide such information, the prestige and power of polling would be considerably reduced."⁽⁴⁾ Pollsters do not, as a matter of course, provide such information, but when they do, it does not make the headlines. However, responsible readers should ask the questions that would help to clarify survey results. The only way to improve the reporting of surveys is to make reporters and pollsters aware of our expectations. So, what can we say? Given the cautions cited above (and the list is far from complete), you may wonder whether anything can definitively be said in a research context. But, while the outlook may seem bleak, it is not entirely so.

Where something has been "left hanging," other researchers can pick up the thread. That is how we progress - further research on a topic is almost always necessary. The illusion that scientists can conduct a definitive study and answer all relevant questions within a specific time period is usually just that, an illusion.

From a purely administrative perspective, a good study might be one that answers a limited number of questions once and for all, within a specific time frame and budget. Calls for "further study" might be viewed with suspicion, as a means of leaving the door open for another research proposal - and further funding.

However, from a researcher's perspective, a good study raises more questions than it answers.

What we can say for sure is that there is an art to choosing questions wisely and there are difficulties in attempting to measure the unobservable, not the least of which is proving that something unobservable is actually present and not just a figment of our imagination.

---

⁽¹⁾ Travis Gee, Research and Statistics Branch, Correctional Service of Canada, 4B-340 Laurier Avenue West, Ottawa, Ontario K1A 0P9.
⁽²⁾ N. Postman, Technopoly: The Surrender of Culture to Technology (New York: vintage Books, 1993): 125-126.
⁽³⁾ For an insightful discussion of this topic, interested readers should see 5.1 Gould, The Mismeasure of Man (New York: W.W. Norton & Co., 1981).
⁽⁴⁾ Postman, Technopoly: 135.