An Adoption Strategy for Social Software in the Enterprise. Many-to-Many:

While this post explictly addresses "social" software, its lessons can also be applied to decision support systems (DSS). DSS do not always contain collaborative tools (apart from email or notification features) they are about sensemaking at a distance. The goal of DSS is to help users and decision makers at different levels of the enterprise to make sense of larger or smaller sub-systems. The lessons of rollout and the role of leadership are appropriate for decision makers in the education sector as well.

Chris

Growth Models: Results of the pilot request evaluations

Secretary Spellings and the US Dept. of Ed publish the winners and losers in the first round of pilot states.

Chris

Measuring education outcomes, it's not as easy as it sounds

Graduation rates are an important measure of success for the educational system. The difficulty of measuring drop outs and graduate is outlined in this piece from Kansas. It's hard at the state level and even harder at the federal level. The author gets the problem statements right. We need to have higher quality data and we need to focus on how we are failing those kids who don't get diplomas.

Chris

Teacher commitment matters more than certification or experience

Well....I guess that's not to hard to understand. Committed teachers in early grades who take personal responsibility for student learning tend to produce more learning growth.

LoGerfo's other findings reinforce other widely held precepts - supportive leadership makes a difference, Catholic schools tend to show higher commitment than public schools.

As we work on getting the models right, it will be important to make sure that teacher practice and perceptions make it into the mix.

Chris

Hitting the high standards ceiling?

As Schrag points out, it is not surprising that the warts on NCLB are becoming increasingly clear several years into the implementation of the legislation. States with high standards, such as California, are being penalized since it is becoming ever more difficult for schools to meet the expectations of 100% proficiency in the face of tough standards. Indeed, a judge in California just issued a preliminary injunction against state barring it from using the high school exit exam to bar students from graduating. The argument was that many students were not given an adequate opportunity to learn - given their exposure to poor schools and teaching. Without equity of access to a quality education, it is argued that students should not be held to these high standards.

Schrag also points to extreme examples of the narrowing of the curriculum in many schools. Some schools went as far as teaching only math, reading, and gym. Is this what Congress intended?

Chris

Milwaukee Public Schools Superintendent used growth metrics as part of reform package

Milwaukee Public Schools is now in its fifth year of reporting value-added metrics for individual schools in the system. Sup. Andrekopoulos is using one of the core reporting forms from the value-added report. The district level report includes a simple 2-by-2 table of schools at the high, middle, and elementary levels. These tables lay out high and low attainment on the vertical axis and high and low student value added on the horizontal axis. The district is giving more autonomy to schools that show high attainment and high growth. Schools with low growth and attainment are being singled out and are receiving more direction from the central office, including the placement of "instructional facilitators" in each school. These facilitators report to central curriculum and instructional staff rather than to the local leadership.

It is important to note that this is not classroom or grade level value added. This is not a system that sanctions individual teachers. Rather, it focuses on research in the district on effective teaching strategies that correlates with improved student outcomes and seeks to implement these strategies in failing schools.

Chris

And they're off - 2 states allowed to experiment with growth models

It seems that North Carolina and Tennessee will both be allowed to added growth models to their official NCLB compliance frameworks. It remains to be seen how this will actually play out. Tennessee will be using a proprietary model that has never had any rigorous external evaluation. North Carolina was just in the news this week for the major cost overruns and failures surrounding it's state-wide student data system.

Chris

Data Quality and the risks of "running with what we have"

There is a great temptation for education (and other) organizations to just "get something up" and call it a data warehouse as part of a strategy of retaining the support of senior leaders. One of the problems data warehouse designers have in organizations with little history of decision support is that the clients (program area staff) literally cannot identify needs that extend past their current experience with data. One common solution is to take the current operational data and its definitions (such as they are) and simply load them into a warehouse. One can then take existing reports as the design documents for data marts.

The good thing about this approach is that it provides a wonderful teaching environment for bringing program staff into the discussion using data and representations that they know and can make sense of. The risk is that they will see this and want to run with it. It is guaranteed that these data (and definitions) will contain serious quality problems there were not exposed or stressed under the older, more constrained reporting system. While this might seem like an early success, going forward with this system can be very risky. Program staff are experts in their programs. Data problems will emerge and they are likely to blame the system rather than the data or collection processes. They are also likely to see the problems as someone else's problem and not be receptive to requests that they "clean" the data. They may come back with requests for IT to fix the transactional system.

This is a cautionary tale for states working on getting their warehouses up as rapidly as possible.

Chris

The Ed Trust compares proposed growth models

The Education Trust has been very explicitly concerned about the risks of relying only on growth models for accounability. There is the concern that focusing only on growth rates will lower attainment expectation for the nation's neediest students. The Trust compares the stark differences between the Alaska and Arkansas plans in their article and provides a table that compares the strengths and weakness of all of the proposals. This is an easy and informative read for those not prepared to wade through all of the propoals themselves.

Chris

Ed Week delivers research report on data use

An excellent survey of the lay of the land at the school, district, and state level with respect to data use and decision support.

Chris

Exclusion of small subgroups annual AYP reporting

Here is one of those cases in which the reliance on a single metric to both provide feedback to the instructional system and, at the same time, hold folks accountable bites one in the posterior. The use of confidence intervals around measures with low sample sizes is a critical part of interpreting statistics. Most statistics text books cite a sample size of 15 as the minimum number justify the assumption of normal distribution. Given the measurement error associated with traditional standardized tests, sample sizes of 2-4 times that number would not be an unreasonable standard for reporting.

An inaccurate measure is fine for providing instructional feedback. Teachers have a wealth of information about student performance. The high stakes test result is just one observation among many. Low reliability is not a problem in this case. However, for high stakes, the use of low sample sizes (and the resulting lack of confidence in the point estimate) creates serious concerns about the consequential validity of any sanctions or rewards based on that measurement.

Chris

PS This Week in Education also has some good links on this story.

Back from my 20th year grad school reunion

No computer on the road! Two weeks with no email or phone calls! Working though a large backlog of stuff to post.

Chris