research

• Regarding the Furor Over the Study That Says "Unschoolers Don't Test Well"

  29 December 2011 / cat test, News, research, Teaching / 10 Comments

  There was a bit of hoopla in the fall over a Canadian study that (some folks believed) implied that unstructured homeschoolers fare poorly academically.

  Please give me a few minutes to present a critical thinking lesson about research.

  When reading a research study, these things must be considered:

  1) Generalizability- Can the results of this study describe the universal population of homeschoolers or only the local Canadian population of homeschoolers?

  The only way a research study is generalizable from the exact persons studied to an entire population is if the sample (people chosen for the study) is LARGE in NUMBER, from SEVERAL LOCATIONS and RANDOMLY SELECTED. Otherwise, the study is useful to describe the small, local group who participated in the study.

  To help explain, imagine this fictional study interpretation: Americans are poor performers of spoken English because they say the word, "Ya'll" (based on a study of 75 people from First Groovy Church in Atlanta, Georgia). You simply cannot infer that ALL Americans have poor spoken English. The sample was tiny (75), all from one place (Atlanta, where "ya'll" is part of normal spoken English), and there is no evidence of random selection since they are all from the same church.

  This study, The impact of schooling on academic achievement: Evidence from homeschooled and traditionally schooled students (Canadian Journal of Behavioral Science), compared 74 children living in Nova Scotia and New Brunswick- 37 homeschoolers and 37 public schoolers. Of the homeschoolers, 12 were unstructured homeschoolers.

  In the study, the homeschooled children who were educated with a structured curriculum all performed at higher grade-level than their traditionally-schooled peers (from .5 grade level above to 2.2 grade levels above). However, the children who were homeschooled with a non-structured approach (unschooling, as it were) scored lower than their traditionally-schooled peers across the board.

  Now, this is fine descriptive research. The results describe the achievements of 74 kids in Nova Scotia and New Brunswick. The results do NOT describe all homeschoolers everywhere because the sample of people studied was SMALL, from a fairly small location, and there is not evidence of randomization of the population.

  2) Validity of Interpretation- Are you interpreting the results appropriately?

  Let's go back to my fictional study of the folks at First Groovy Church in Atlanta. The inflammatory reading of the results imply that the word, "Ya'll", is an indicator of a person with poor spoken English. HOWEVER, if you ask anyone from First Groovy, "Ya'll" is the best word to use for second person plural, based on tradition dating back to Old English. Thus, to First Groovy, "Ya'll" is an indicator of APPROPRIATE English.

  Those of us who know unschoolers know that they do not measure their academic success by achievement test results- far from it. They are more likely to measure academic success with evidence of delight in chosen subjects, development of talents and interests, and rich experiences. Achievement tests can in no way measure these things.

  So the Canadian study, while interesting description of one tiny population, cannot be used to imply that unschoolers are under-educating their children.

  Next time you read a study, please note the population size, location, randomization of sample, and validity of interpretation.

  (Hey, just for fun- do your critical thinking, then weird your friends out with this "powerful" research fact: 97% of all criminals drank milk as children.)