Exclusion of small subgroups annual AYP reporting

Here is one of those cases in which the reliance on a single metric to both provide feedback to the instructional system and, at the same time, hold folks accountable bites one in the posterior. The use of confidence intervals around measures with low sample sizes is a critical part of interpreting statistics. Most statistics text books cite a sample size of 15 as the minimum number justify the assumption of normal distribution. Given the measurement error associated with traditional standardized tests, sample sizes of 2-4 times that number would not be an unreasonable standard for reporting.

An inaccurate measure is fine for providing instructional feedback. Teachers have a wealth of information about student performance. The high stakes test result is just one observation among many. Low reliability is not a problem in this case. However, for high stakes, the use of low sample sizes (and the resulting lack of confidence in the point estimate) creates serious concerns about the consequential validity of any sanctions or rewards based on that measurement.

Chris

PS This Week in Education also has some good links on this story.