There’s an old joke about a company that wanted to hire a statistician. The first candidate entered the room and asked a number of questions from a panel of three interviewers. At the end of the interview one of the interviewers stood and walked to a black board and wrote 2 + 2 = and asked the candidate to answer the question. The first person went to the black board and wrote the number 4, was thanked by the panel and escorted to the door. This continued through the day. At the end of the day with the final candidate the interviewer wrote the same problem on the black board, except this time the candidate looked around, walked to the windows and pulled the shades, walked to the door, opened it a crack to see if anyone was close by then closing the door walked to the panel and leaning close, whispered, what would you like the answer to be?
Whether you found the joke humorous or not, it raises an important point with research and data. Often the data you hear about in advertising, politics, and even science is biased. Sometimes this is on purpose and is done with the intention to either deceive or convince you of one thing or another. Sometimes it is not on purpose; but is simply sloppy research that allows bias to slip in.
Even what appears to be the best research may have it's flaws. For example Piaget's research on the topic of "conservation." I theorized when my oldest daughter was about three years old that if it was important enough for her, she would understand the concept. (When a liquid in one glass is poured into another glass of a different shape and size, she would still understand which glass had more.) I tried it with chocolate milk and or Kool-Aid and she got it every time. A similar experiment is sometimes duplicated with pennies. What does a little kid care about pennies. Try it with Skittles or M & Ms and see if they don't know the difference. More younger children will retain the concept of which is more with a higher motivation.
So, how can you tell the difference?
The first thing is to not simply believe what someone reports as the results of research; but to look at the actual research. At the bottom of this blog you will find the Google Scholar Search Engine. This is a place where you can search for actual research articles. In many cases you will only be able to see the abstract, which is a type of summary. If it looks like you want to read the whole article you will need to either pay for the article or go to a local university library and find the actual journal (some larger city libraries also carry journals).
Once you find the article you want to look at the research design. This will almost always be a section within the article. There you will want to especially look for two things; randomized subjects (randomized controlled samples) and blind assessment. I remember one article I read a few years ago where they said they randomized; however all of the children from one group lived in the city and received the extra services being research and all of the “control” subjects (children not receiving the extra service) lived out in the country. The reason they gave for this was that the children outside of the city lived too far away to receive the extra services. Unfortunately there are many other differences in the environment for the children. So many differences or “variables” that it would be impossible to say that the difference in results was the result of the additional services which were being researched. Another big problem is the natural bias of the researchers themselves. If you have a lot of work and effort in a research project, you may have an interest in seeing it successful or in getting the desired results. For this reason you want to look for “double blind” studies. These are studies where the researcher doing the assessments on the children do not know which children were involved in the “treatment,” or extra service, or whatever is being researched.
Learn to be a good critical thinker when looking at research, advertising, text books, or listening to politicians. Additional resources can be found here.
No comments:
Post a Comment