Are WIDE World surveys representative?
WIDE World’s evaluation efforts rely quite a bit on surveys. Since response rates have been about 55% for our end-of-course evaluations and about 20% for our one year follow-up surveys, it is natural to ask what gives us confidence that the fraction who respond constitute a representative sample of course participants. Our web page, Course Evaluations, briefly discusses the ways we try to address the issue (both at the top and bottom of the page). The current piece will fill in some gaps and add some detail.
We regularly check to see whether survey respondents hold, on average, similar characteristics to those of nonrespondents. For example, we check whether our findings are applicable regardless of region, teaching subject, level of experience, or educational degree. Generally we find that they are, and the glaring exception turns out to be less important than one might think. Respondents tend to have amassed more participation points than nonrespondents have. But level of participation shows such low correlations (r ~ 0.0 to 0.2) with important outcomes such as course satisfaction, or appraisal of the course’s effects, that for these outcomes nonrepresentativeness in participation is scarcely an issue. And to reiterate what was described in the page listed above, in 2005 our special effort to individually reach nonrespondents (especially those with low participation) revealed surprisingly few differences in their opinions of the courses compared to respondents.
As for our recent one year follow-up survey, it is true that respondents were more likely than others to have gone on to become coaches in our program. However, the numbers in this group were small, and adjusting for this imbalance would have exerted only a slight effect on the findings reported at One Year Follow-up. Our paper entitled Beyond Self-Report, p. 12, provides another examination of such factors as they pertain to a follow-up survey from the previous year.
[RBS]