Friday, January 27, 2006

Data Quality Campaign and its 10 Criteria

What a list. Some of this is going to be very hard to accomplish.
  1. A unique statewide student identifier
    • Mobility makes this tough. State to state transfers increase the difficulty enormously
  2. Student-level enrollment, demographic and program participation information
    • Program participation remains tough to follow. Much of the money flows down to schools. Without the data processing capacity to track the relationship between teachers, students, and courses - something most district lack - this is impossible. Many programs (state and federal) require only counts by school and grade. There has never been an incentive to track individual student participation - with a few exceptions such as special and vocational education.
  3. The ability to match individual students’ test records from year to year to measure academic growth
    • Many states are now doing this. NCLB was a wakeup call. Still, many states are brand new at this and will take several years to build up useful longitudinal data.
  4. Information on untested students
    • Ideally many distrticts and states are going a step beyond this with induction testing for children who move into the district outside of the high stakes testing window. It is vitally important to make sure that all student are tracked and that schools are held accountable for all kids.
  5. A teacher identifier system with the ability to match teachers to students
    • Without this link it is impossible to provide teachers with the characteristics of their incoming kids. This like is also necessary if one is to study the effectiveness of a particular curriculum given varying levels of teacher experience and/or training. The tie between teacher and student is at the core of the "production function" of education.
  6. Student-level transcript information, including information on courses completed and grades earned
    • This data documents mastery of required content and shows students' opportunity to learn at a system level.
  7. Student-level college readiness test scores
    • While this seems like a good idea, it is tough to administer tests with no stakes attached. Many districts have attempted to use exit exams. There are mixed results.
  8. Student-level graduation and dropout data
    • This remains a serious measurement problem. Dropouts are the measurement of a non-event - a student no longer attends school. We don't know what happened - only that we don't see them any longer. This makes it tough to fix the problem. We don't know if it was a fairlure of the educational system or some other influence.
  9. The ability to match student records between the PreK–12 and higher education systems
    • Student outcomes after high school may be the best way to measure PK-12 productivity. Entrance and placement exams may be the most useful data available.
  10. A state data audit system assessing data quality, validity and reliability
    • It is possible to do technical audits for compliance, but to get this story right one would have to combine this with direct observation to see if reported data aligns with what one observes in the field.

No comments: