VARC Blog

Guest post by Heather Johnson

2008-08-03T13:35:00.005-05:00

Hi,

I was interested when Heather asked me if she could do a guest post. The title link (above) links back to the teachingtips.com blog where one can find some of Heather's other posts.

While I agree with all of Heather's concerns about testing - curricular narrowing, too many eggs in the single test basket, etc. - I am even more worried that status measures (proficiency levels) are terrible indicators of how well a school/grade/classroom is educating a particular set of students. For all of their flaws, standardized tests could be used to provide more useful information for improvement.

I also agree with Heather that high stakes tests are here to stay. I'm pretty sure that performance pay linked to value-added measures in just over the horizon for many states and districts.

Chris

The Pros and Cons of Standardized Testing

The debate rages in educational circles: is standardized testing a fair assessment for measuring students? Everyone has their ideas on the issue and many people are still on the fence about the merits of standardized testing. The “No Child Left Behind Act” has put an emphasis on this system of testing. Legislators believe that we need to have a raw score for the students and this is how we should rate all students in the United States. Good or bad; this is going to be around for the long haul. In order to help you find your stance on this contentious issue we’ve come up with a list of the pros and cons of standardized testing.

Pros:

Reliability – a standardized test is reliable if the questions and answers are dependable. If students answer the same questions in the same fashion then a test is deemed dependable and the research that can be gleaned from such results is valuable.

Validity – a test is considered to hold validity if educators can make proper assessments of their students based on the information learned from their test results. The notion behind a standardized test is to find students flaws and correct them in the future.

Trends – educators need to be able to see trends developing in students’ answers on a broad scale for the standardized test to be considered important. We are not simply looking to see who is smart and who is not; we are trying to pinpoint areas in the curriculum where students can improve and how we, as educators, can help them improve.

Cons:

Diminished Time in the Classroom – with the advent of the standardized test it’s meant that teachers need to teach their students how to take a test properly. This means that there will be hours of practice tests and teaching students how to follow instructions properly. This takes away from actual teaching of subjects.

Teaching to the Test – this is one of the most common complaints among educators, parents and students alike when the subject of standardized testing arises. Some teachers feel they have to teach specifically to the material they believe will be on the test and department heads will shift a given curriculum to follow suit. This diminishes the role educators have in forming lesson plans.

Unfair to students – many students become extremely nervous for any sort of test. When they are aware of the weight a test will have, such as standardized test, their stress levels soars through the roof. The results of these tests then become skewed.

Whether you like it or not, standardized tests are here to stay and all we can do is hope the students are always the center of our attention.

By-line:

This article is contributed by Heather Johnson, who regularly writes on Alabama teacher certification courses. She invites your questions and writing job opportunities at her personal email address: heatherjohnson2323 at gmail dot com.

Projected Growth and Value-Added in Ohio

2008-08-01T17:41:00.000-05:00

This is so cool. A policy discussion that differentiates between growth and value-added.

Chris

Ohio Governor wants to talk about wacky ideas

2008-07-23T17:43:00.004-05:00

Governor Strickland wants put everything on the table for education reform. One of his points was:

"What if we created a value-added system that measured results and compensated teachers for improving student achievement?"

Well...he's more than half way there. The state now provides value-added analysis and will have access to classroom-level data. Four of the state's largest districts are recipients of federal Teacher Incentive Fund grant funds.

Chris

Colorado's State Growth Model Goes Live

2008-07-22T16:45:00.002-05:00

Colorado's growth model is official now. A number of outlets have picked up the press released (linked to above in the title) and discussed how this new analytical information can inform policy making in the state.

Colorado Charter Schools Blog has a post and links to a page with all of the charter schools in the state. Papers in the state begin to discuss how to read the diagrams and how parents might use the results (The Examiner, The Daily Sentinel, The Reporter Herald).

This level of visibility and dialog is a welcome addition to the national discussion of the appropriate usage of growth model data.

Chris

Data Quality Campaign 2007 Survey on Longitudinal Systems

2008-07-14T08:00:00.001-05:00

One of the most important contributions of the Data Quality Campaign has been to gather solutions and examples of policy making from state agencies across the 50 states. In particular, the work done in Florida to update its already impressive PK-Workforce system to take advantage new technologies and more flexible thinking about appropriate data use provides a broad backdrop for possible innovations across the country.

Chris

NSBA's Blog summarizes how parents feel about testing

2008-07-11T08:00:00.000-05:00

BoardBuzz posting a link to an Associated Press Poll about issues in educational problems and possible solutions. One of the interesting findings is the gap between fairly solid support for tests as instruments of school accountability but a belief that classroom work better represented student learning.

Chris

Collaborative provide V-A Training Support in Ohio

2008-07-08T08:00:00.000-05:00

The Ohio Resource Center is an interesting collaborative that combines scholars from the different schools of education in in Ohio and is supported by the legislature and Ohio Board of Regents. The "about" page describes the Center as follows:

ORC's resources are available primarily via the web and are coordinated with other state and regional efforts to improve student achievement and teacher effectiveness in preK–12 mathematics, science, and reading. The website is organized around Ohio's academic content standards.

What I find interesting, as a V-A guy, is that the Center has also developed a three module set of lessons on understanding value-added outcome metrics. It builds up knowledge about assessments and their uses, in general, and then goes on to describe how one should use V-A metrics.

A big thumbs up for the ORC!

Chris

Science Cannot Be Secret

2008-07-05T08:00:00.000-05:00

This blog post does a great job laying out some of the core concerns about complexity and proprietary information in accountability systems. Our mantra has been "simple is better, unless it's wrong". In the VA work we do, we might prefer a simpler model that has "strong" assumptions. However, we think it is important to develop model enhancements that deal with cases in which the assumptions cannot be met. We look at the results of both models to see what difference violating the assumption makes. I very much agree with the author of this blog that this is a policy decision and that the educational agency (state, local, etc.) must be engaged with the model design and be able to understand and explain any and all modeling choices.

Chris

Value-Added doesn't get any more real than this

2008-07-02T08:00:00.000-05:00

This is the face of classroom value-added on the ground. The Ohio Department of Education and Battelle for Kids (who provides training for the interpretation of VA scores) have used classroom level scores to identify teachers who produce large gains in student test scores.

Chris

VA and mobile students

2008-06-30T08:00:00.001-05:00

I've been trying to pull together how the different districts and states using VA measures for accountability are dealing with Mobile kids. The differences in how the issue is discussed in the three states currently using SAS's EVAAS for state-wide accountability. Bill Sanders has addressed the issue very clearly in his writings around the EVAAS model. The model being used by EVAAS can handle missing observations that are generated by mobile students. In all of the jurisdictions for which I can find documentation, there is a rule for who falls into the accountability system and his therefore included in a particular year's model. Students who do not meet the definition are excluded from the growth model.

Ohio Rules - (page 1 of ODE VA FAQ and FAQ at Battelle for Kids)
"How does high attrition or mobility affect the value-added measure?
Schools and districts are accountable for students enrolled at that school for a full academic year. Only students who are continuously enrolled from October count week through March testing will be included in the analysis. The Ohio Department of Education will match students test scores across years and schools using the SSID.

The two FAQs seem to imply that mobile students are included, but only when they meet the above definition. It is possible that some non-zero percent of students fall out of the analysis every year.
Pennsylvania Rules - (there are more detail proposed rules here - but they do not differ on this issue)

It seems that in Pennsylvania, the district - not the school - is accountable for the performance of students who do not attend for a "full academic year". As in the case of of Ohio, the EVAAS system explicitly takes into account missing data for individual children.

(page 21 of the Pennsylvania Consolidated State Application Accountability Workbook)
"Schools, LEAs and educational entities are accountable for mobile students in the same manner as they are for other students. The “full academic year” criteria are applied to all students. In Pennsylvania, it is not uncommon for students to move from one school to another within the same district during an academic year. In these instances, the school in which the student is enrolled at the time of the assessment bears responsibility for test administration; however, the district, rather than the school, will be accountable for the student’s performance."
Tennessee rules (several districts (pdf page 2 and other external sites) refer to 150 day enrollment requirement before the test )

My concern is for the implications this has for high mobility districts - particularly the large urban settings with high mobility rates. While I can see how techniques for dealing with missing data can be used to make good classroom and grade estimates, there might be a related incentive to not focus as consistently on the learning needs of mobile children - given the pressing needs of those fully included in the accountability system. A model that includes "dosage" or proportional assignment of student growth is what we should be shooting for. This would be relatively easy in a district in which most of the mobility is school-to-school within the district. However, as state-wide data system improve, it should be relatively easy to track student mobile within a state and get access to their test data and current school. A dose-based model does he math right and provides consistent incentives to school staff.

Chris

Accountability in Higher Education

2008-06-27T08:00:00.000-05:00

There have been discussions of using standardized tests in undergraduate institutions as a part of an institutional accountability system. A working committee was established by the ational Association of State Universities and Land-Grant Colleges (NASULGC) and the American Association of State Colleges and Universities (AASCU). They developed a framework that is called the Voluntary System of Accountability (VSA). The VSA can be implemented with a series of different examinations (C-Base, CLA, CAAP, MAPP, GRE and ACT WorkKeys). There are currently not enough questions in common across these assessments to support a simple value-added model. There is the notion, however, that VA measures are the goal for the test-based elements in the accountability system.

The piece linked to by the title suggests that this approach follows the recommendations of the Spellings Commission too closely. Tests will not capture the range of what undergraduates are expected to learn across a wide range of subjects. The Association of American Colleges and Universities (AACU) has established the Valid Assessment of Learning in Undergraduate Education (VALUE) project to expand assessment efforts beyond tests to include an e-portfolio approach that would document both growth and breadth of student learning and development.

Chris

Obama and McCain on VA and and teacher pay

2008-06-24T08:00:00.000-05:00

There don't currently seem to be enormous differences between the two presumptive presidential nominees on student assessment, school reform, and the use of VA results. Both support public charter schools and programs to bring highly qualified teachers to low performing schools. Obama seems to be more open to a wider range of performance measures (including teacher knowledge, observed practices, etc.) than McCain when discussing teacher performance pay. Obama also puts more emphasis on early childhood development efforts.

Chris

Adaptive tests as the assessment fix for NCLB's narrow approach to testing

2008-06-21T08:00:00.000-05:00

In the Value-Added Research Center work with districts and states, we have run into a number of instances of the NWEA MAP being used untested grades and subjects to fill in gap years between NCLB-mandated tests. This approach is particularly appealing in districts engaged in teacher and or school incentive projects. The ability to include more teachers in grade-to-grade growth models is appealing across the board. Administrators like the equity and external validation of external measures. Many educators like the respect given to tested subject and prefer not to have their performance measured by walk throughs or other observational measures only.

On the other hand, there is still great deal of research being done on the validity and reliability of growth models based on computer adaptive tests (over 2300 hits). Student under an adaptive test regime do not take the same form of the test. Much of the science around understanding growth rely on students taking the same form of a test. However, when one looks at the working being done on VA use of adaptive tests on gets 4 hits.

We are hoping to be able to work with one or more districts using the MAP to see how well this works in practice. We are also looking a districts using quarterly diagnostic assessment to predict performance on the annual high stakes. Likewise, we are likely to work with one or more districts who want to use the PLAN-ACT series of tests for measuring high school productivity.

There is certainly plenty of work to do.

Chris

Colorado Growth Model Introduced

2008-06-19T08:00:00.000-05:00

Colorado has been working for some time on a state-wide growth model. In early March, the state issued a press release and made a number of documents available on their web site.

Technical Report on a Colorado’s Academic Growth Model (pdf)
Presentation to district assessment directors (pdf)
Changes to the accreditation process presentation (ppt)

The Technical Report includes both the authorizing legislation and a technical paper by external consultant Damian Betenbenner. Betebenner and his colleagues at the Center for Assessment in Dover, NH are generally on the simpler is better side of student performance modeling recommendations.

I am not an economist, although I do play on on TV. However, I know from personal experience that we've been repeatedly put in the position of evaluating simple systems and the unintended consequences that flow from such models. As hard as one imagines it might be to do growth modeling well, it's probably 2 orders of magnitude harder than one imagines. In particular, what looks good to an outsider looks very different to a teacher or principal whose job performance or bonus is going to be based on that analysis. Educators suddenly discover a preference for complex models when the simpler models turn out to be unfair to some large portion of the adults in a system.

Chris

Coloado District Explains new state growth model

2008-06-18T12:35:00.002-05:00

A district assistant superintendent does a nice job presenting the new Colorado growth model and how it differs from previous accountability measures.

Chris

UK Education officials stuggle to explain attainment versus growth

2008-06-16T08:00:00.000-05:00

Educators and policymakers in the US are not the only folks to struggle with explaining what can appear to be contradictory outcome measures. A recent report identifying failing schools described a wide range of growth performance. In particular, this story points to 30 schools who were in the top 5% on attainment measures (number of GCSEs) but failing on growth.

This is something many people have trouble discussing. The notion of "controlling" for prior ability and demographic characteristics gets confused with expectations. What controls do is level the playing field by making a fair comparison. From a social policy point of view, positive or negative coefficients on for gender, economic status, or race have nothing to do with expectations. They are are the growth equivalents of attainment gaps. If economically disadvantaged children in the 5th grade show on average that their growth in test scale score is 4 points lower than non-poor students, this is the performance gap. This tells us how well we are doing helping students overcome the educational impact of the non-school economic resources. Policy should be focused on improving the rate of learning growth to erase the growth gap.

The challenge presented by interpreting high attainment versus low growth is actually not that hard to overcome. We all know of schools that are good at recruiting families and students with high prior test scores. Many adds for new homes included references to the attainment levels of local schools. High prior school-level attainment (and school magnate programs) tends to attract families with high attainment students. Recruiting students with high prior attainment is the simplest way to be a high performance school under an accountability system that focuses on attainment. A growth model, instead, controls for prior attainment and teases out what learning was delivered in that year. A school can be very good at recruiting while being not very good at challenging good students. The two things are quite different.

Chris

Bill Sanders responds to public criticisim of his VA model

2008-06-13T08:00:00.000-05:00

Ed Week covered a dispute (May 6, 2008) between Bill Sanders (SAS Inc.) and Audrey Amrein-Beardsley (Arizona State University). Assistant Professor Amrein-Beardsly published (Ed Researcher) her own analysis of data collected in a study Sanders conducted on the effectiveness of board-certified teachers in North Carolina.

What this exchange confirms is that smart, well meaning people can come to entirely different conclusions. In particular, the argument that a simple model is required for VA to be accepted is completely at odds with our experience that VA models have to be quite complex to be fair. There is no simple answer to the transparency-equity argument. It is a normative paradox that leaves scholars red-faced and exasperated on both sides of the argument.

Chris

U. S. Secretary of Education Margaret Spellings Approves Additional Growth Model Pilots for 2007-2008 School Year

2008-06-11T14:14:00.003-05:00

The next round of approvals of state Growth Models.

Washington, D.C. — U.S. Secretary of Education Margaret Spellings today announced approval of two high-quality growth models, which follow the bright-line principles of No Child Left Behind. Michigan is immediately approved to use the growth model for the 2007-2008 school year. Missouri's growth model is approved on the condition that the state adopt a uniform minimum group size for all subgroups, including students with disabilities and limited English proficient students, in Adequate Yearly Progress determinations for the 2007-2008 school year. (7thSpace)

Chris

New York regional education board provide VA PD

2008-06-10T08:00:00.000-05:00

The Capital Region Board of Cooperative Educational Services held a day long session on value added measures on May 28, 2008. Sponsors of the session included New York State School Boards Association (NYSSBA), New York State Council of School Superintendents (NYSCOSS), School Administrators Association of New York State (SAANYS), and Battelle for Kids in Ohio.

I ran across the blog linked to the posting title as I was looking for people around the net explaining VA to others. I am keenly interested in how folk exposed to VA discussions understand them and explain them to others. This is one of the better explanations I've found. It uses a particular form of reporting for its explanation structure - one found in Battelle materials. I'm on the lookout for other approaches such as this year's VA versus this year's attainment compared to last year's attainment with this year's VA. The two graph's tell different stories. Neither is wrong, but they have different purposes. I am hoping that the sophistication of the analysis used in PD efforts broadens to include a wider range of program and school evaluation questions.

Chris

Houston's VA PD and the criticism of complexity

2008-06-08T08:00:00.001-05:00

Whatever I think about the difficulty of training teachers and administrators to understand and use value-added measures, I agree with my colleague Rob Meyer who consistently argues that simple is better, unless it's wrong. I really like the quote from Bill Sanders at the end of the RedOrbit post linked above. Bill use a great teaching aid as well:

"I'm not going to trade simplicity of calculations for the reliability of the information," he said. "Before groups of teachers, I often hold up a cell phone and I say, 'I don't have a clue what's inside this, but I have to have trust that when I punch the numbers, it's going to call the right number.' "

There are two challenges when working with educators on understanding and using VA measures. First, one has to expose how the simplicity of attainment masks its underlying inadequacy as a performance measure. Second, one has to show that the more complex analysis used in VA models is more fair and gives educators credit for improving student learning no matter where the student starts across the range of prior ability.

There is no way to dodge the complexity bullet if we want to be fair to students and educators.

Chris

Houston's Delivers new Value-Added numbers at part of performance pay system

2008-06-06T08:00:00.000-05:00

Houston abandoned it's local performance measurement system after data quality and communication problems last year. The Houston Chronicle described the new approach at beginning of this school year. The new program, called ASPIRE, was designed to answer the core problems of the local system - clear rules, transparency around data quality, and equitable access to financial incentives. ASPIRE is a supported by value-added analysis provided by SAS EVAAS and professional development, online support services, and leadership consulting provided Battelle for Kids. This reworking of the HISD approach to performance pay was supported by the Broad and Bill and Melinda Gates Foundations.

We'll need to watch over the next few days and weeks to see how these effort play out.

Chris

School board member digs in on tough issues

2008-06-04T08:00:00.001-05:00

Wow. A recently elected school board member is really thinking hard about using VA results. Our local school board has been scheduling monthly VA briefings for the assessment subcommittee as we work to develop a statewide VA model for Wisconsin.

We are working to integrate our work with the Milwaukee and Madison school districts to show how the results and analysis differ in large and mid-size districts. We are also working with a regional service agency to explore the best ways to report results to small districts. Small districts have unique challenges for statistical analysis of VA given their low student counts. We will be looking at grouping similar rural districts into "quasi" districts to extract more explanatory power from the models.

Chris

Education Week on Using Value Added Data

2008-06-02T08:00:00.004-05:00

Ed Week reported on a meeting at the Urban Institute that was a summary/policy implications round up of a much more technical meeting held here in Madison in late April, 2008.

One of the most important things delivered by the April meeting was a concerted attempt to translate between economists, statisticians, psychometricians, and sociologists. There are is an emerging consensus on the minimum requirements for a value added model used for high-stakes decisions. As the number of scholars engaged in real world analysis converge on a set of recommendations, we should be able to form a more consistent, non-technical explanation of VA model features and assumptions. One of the biggest stumbling blocks facing the wide spread adoption of VA models is the perception that ordinary people cannot understand them. We are getting close to the prerequisites for a coherent set of explanations.

Chris

So, why have I been so busy?

2008-05-31T19:59:00.005-05:00

I've been working with VARC Director Rob Meyer to grow our research center. We are engaged in a range of work in Milwaukee Public Schools, Chicago Public Schools, a series of Teacher Incentive Fund projects, as well as basic research on extensions of our value-added model.

The work includes a series of program evaluations, professional development around the use of VA measures, random assignment experiments, improvements in operational systems, and the use of VA models for quarterly diagnostic assessments. The current focus is on adding a number of tweaks to our model that uses two prior scores to compute VA measures. The complexity of including corrections for measurement error, mid-year testing, retention in grade, etc. has required lots of graph paper and white board makers.

As for the development of the team, at the last PhD count, we were up to 2 in sociology, 2 in industrial engineering, 2 in educational leadership, 1 in statistics, and 6 in economics. We are probably going to add a couple of more MAs in economics and statistics and a PhD in applied mathematics. I can also tell you that Fortran is alive and well here. We are hoping to start working with the Condor Project to address some of the computational problems of more complex VA models.

As the guy in charge of strategic planning and human capital development for this merry band, it's been an exciting 18 months.

Chris

Massachusetts Education Commissioner interested in VA

2008-05-31T19:47:00.003-05:00

The new Commissioner for Education in MA is interested in VA for school and program evaluation. This notion that attainment alone does not tell enough of the story is popping up all over.

Chris

Ok. I'm going to try it again

2008-05-31T17:25:00.003-05:00

Hi,

I actually liked doing these posts. The pressure of starting up a major research and technical assistance center made this effort look like the back breaking straw at the end of a long day. I am at the end of a growth phase and have added quite a few staff members. My goal is actually to bring in a few guest bloggers amongst the VARCians to contribute analysis and materials.

Chris

Spellings adds to list of Growth Model states

2007-05-29T07:33:00.000-05:00

Secretary Spellings announced that Iowa and Ohio have been added to the list of states allowed to include growth models in their high-stakes accountability systems. The other states already on that list are North Carolina, Tennessee, Delaware, Arkansas, and Florida. Tony Bryk of Stanford University led the review panel.

Chris

Center for Teaching Quality publishes report on pay for performance

2007-05-27T09:03:00.000-05:00

CTQ recently produced a report on recommendations for reforming teacher pay. The twist here is that the report was co-authored by 18 award-winning teachers. The thing that warms my heart is the acknowledgment that while the focus for all kids must be high achievement, the reality on the ground is that only high rates of growth will get them there. The group recommends fairly comprehensive systems of performance measurements in all grades and subjects. They also recommend incentives for accumulating relevant skills (pay for course work related to district goals) and for knowledge transfer (pay for high growth teacher to work at the school or district level to hand on that ability).

Here are the high points:

Get the base-pay system right.
Supplement the base-pay system with a performance-pay system that is open to all teachers.
Reward teachers who help their students make significant academic gains.
Provide additional pay for additional degrees and professional development, but only if the training is relevant.
Allow local flexibility.
Encourage collaboration.
Offer incentives to teachers who want to teach in high-needs, low-performing schools, but only if they’re qualified.
Reward leadership, not seniority.
Be brave, be bold.
Finally, make sure to include accomplished teachers in any efforts to overhaul your teacher compensation plans.

Chris

Trying to do the right thing costs too much!

2007-05-22T07:36:00.000-05:00

The Delaware Department of Education attempted to introduce both adaptive testing to high stakes as well as provide series of formative test to provide more focused and timely diagnostic feedback to students and teachers. Faced with a stiff price, the state decided to stay with its current assessments for three more years. This is a challenge faced by many SEAs attempting to do the right thing. It's not that they don't know what a better testing system would look like. It's expensive to do this sort of work and most vendors don't have item banks large enough or well enough aligned with coursework. Until there is a mature market in this area, early movers will essentially be subsidizing the development work being done by test vendors.

Chris

No lack of interest - just a lack of time and new material

2007-01-09T09:18:00.000-06:00

I find myself slowing down here. We are engaged in more and more work with districts and states. Most of our senior staff are engaged in scholarly writing to get out the details of our approach. More and more, we are thinking of how we support analysis, evaluation, and decision support as an intervention. As such, we must focus on getting written product out through reviewed publications. This is the primary method for achieving economies of scale in our work.

We are also slogging through the hard work of scaling up ourselves. Aside from our work with two large urban districts, we are also beginning to help a number of districts using pay for performance schemes to reward teachers and leaders. The difficulty of attracting and hiring qualified staff for this esoteric work cannot be overestimated.

Chris

Data Quality Campaign a good idea, but....

2006-12-10T08:34:00.000-06:00

The Data Quality Campaign has done a good job highlighting the work of groups who have been working on the issues of data use in school improvement for years. The DQC has been helpful in that it both focused these (and other affiliated and similarly-minded groups) on a series of core metrics for measuring state-level progress in system building. The dilemma here is that except for the those states that provide school information systems to all schools from a central source, most of the data is annual accountability information and is almost irrelevant for local decision making at the school or classroom level. The data needed by states for accountability and monitoring is good for studying the impact of programs and policies, but lacks the real time and frequent needs of building level staff for planning lessons and addressing gaps in students' knowledge.

The way that these efforts may benefit student and teachers is the university requirements for student and teacher ID numbers and the application of modern information management principles and technologies to the education sector. The demand for more granular, high-quality data by state education agencies from local districts has provided much better direction to district research and IT staff as well as to vendors of systems at these levels. The real changes in data-informed decision making will take place as districts and schools have access to better training and tools for data use. The DQC can take some credit for raising the visibility of these issues.

Chris

Value-Added Research and educational infrastructure

2006-12-06T22:36:00.000-06:00

One of the things our research team struggles with on a regular basis is the huge gaps between what educational information systems track and what we need to correctly attribute programs to teachers and students. It has been rare - in my experience - to find that a district student information system comes close to managing the complexity of teaching and learning in modern schools. If we want to know the effectiveness of a particular 4th grade reading program, for example, one would want to have strong attribution of the following links.

Which adults are engaged in the instruction? Given multi-grade classrooms, team teaching, trade offs between subject matter exports who are not the primary instructor, etc. it is often difficult to determine who (and there may be several adults) is doing the teaching.
What is being taught? While we may know what books were purchased, it is difficult to know if a teacher is actually delivering the purchased curriculum as intended. There may be custom additions or replacement of sections. The curriculum may be difficult and the teacher is struggling with the material as well.
Which kids are in the room? Pull out programs, ability grouping, and student mobility may all cloud the picture and make it difficult to determine who was in the room to do the learning.
What resources does the teacher bring to bear? There are important aspects of teacher training (original university work and ongoing professional development) that provide important insights about what works from the input side of the production of student knowledge.

This is not to say that no one tracks these things. However, I would suggest that it is far more difficult that even most district administrators recognize. It is beyond the capacity of some large fraction of the school management software being sold today.

Chris

Fixing NCLB Suggestions

2006-12-01T07:39:00.000-06:00

The recent election results have revived calls for fixes to NCLB. A recent Star-Tribune opinion piece sums up most the changes recommended by many constituencies. The National School Board Association actually keeps a running list of relevant activities and publications that focus on reforming NCLB.

At the same time, both of the major teacher organizations have reviewed and updated their positions on NCLB and the reforms the see as necessary - American Federation of Teachers and
National Education Association.

Chris

Ties between high stakes assessment and dropouts

2006-11-29T12:43:00.000-06:00

One of the concerns about high stakes for high schools - in particular - is that there is increasing pressure for schools to focus resources on the students close to proficiency to "shove them over the bar". The message that this sends to other students who are farther from proficiency is that they are on the periphery and do not matter. This is very likely to rise their likelihood of dropping out. At worst, these students are encourage to move to other schools or drop out.

Chris

Challenges in addressing attainment and growth goals

2006-11-15T10:00:00.000-06:00

This debate summarizes much of the concern about growth or value-added models and the risk that they will dilute the push for high expectations for under-performing kids and schools. There are at least two ways that these concerns can be addressed. One thing that value-added measures can be used for is to identify schools that are "beating the odds" and delivering higher than average growth in student learning. This both provides an "existence proof" to show what is possible as well as a target for evaluation to figure out the mechanisms of success.

The other important contribution of value-added analysis to an high-stakes, high attainment system is to provide insights into what kind of growth exists in current systems. There is some concern that there many be very few schools in the nation capable of delivering the rates of growth needed to achieve NCLB standards by 2014. This is vital policy information. If nothing we are currently doing - in terms of teacher education, professional development, curriculum, etc. - can deliver the growth needed, then more radical interventions in these areas are necessary.

Chris

Producing high quality teachers

2006-11-09T08:04:00.000-06:00

In an opinion piece, William Graves - Dean of Old Dominion University's Darden College of Education - supports changes in teacher education programs. Old Dominion offers a warranty for its teachers. Graves points out that Old Dominion already implements many of the suggestions recently put forward in a report by a former Columbia Dean Arthur Levine. One of the most interesting things about the Levine report is that vast differences in quality between education programs. One of the things that I like about this discussion is the focus on what constitutes a quality program. The recommendations address the "how would you know?" question.

Chris

Lovaglia's Law - Bad news for Evidence-Based Decisionmaking

2006-11-07T07:27:00.000-06:00

Bob Sutton quotes Michael Lovaglia as follows:

Lovaglia’s Law: The more important the outcome of a decision, the more people will resist using evidence to make it.

As someone who works to help bring better data to bear on low- and high-stakes decision making in educational systems, this is troubling (but not that surprising) as a working hypothesis.

Chris

TIF Awards and growth at the district level

2006-11-06T07:30:00.000-06:00

The first Teacher Incentive Fund grantee has recently been announced. This will start a round of federally-supported experiments in generating and using school-, grade-, and classroom-level growth or value-added performance models. There is no telling how well thought out these models will be, but their very existence will drive forward the discussion of equitable treatment of teachers, leaders, and students. It will require districts to focus on both growth and attainment. While some of the discussion will inevitably be contentious, the ability to tie growth outcomes to educational practices is welcome shift.

Chris

DOE preparing to announce additional states allowed to test growth-model

2006-11-04T09:13:00.000-06:00

While there is a short list of possible candidates who will be considered for growth model permission for next year, that is only a foretaste of what might be on the horizon. Eric Hanushek - the Hoover Institute scholar who chaired the first review panel - notes that with the creation of student id systems and grade level testing in almost all states, the number of possible candidate will quickly climb to all states. So, despite the fact that the DoEd is taking the slow road to growth models (laying aside the issue that none these models actually study growth), it will be possible for states and districts to experiment outside of official program participation. Many groups, both national and at the state level, are asking questions about school productivity and attainment. As it becomes possible for states to explore growth models, we may start to see states driving policy experiments and the Feds playing catchup.

Chris

Help with answers to the "Now what?" question.

2006-10-23T07:33:00.000-05:00

The Center on Innovation and Improvement has a nice collection of research and reports focused on school and district improvement. This site points to recent research on effective practices as well as links to federal programs and support resources around the country.

Chris

New agenda for Texas education

2006-10-10T07:11:00.000-05:00

Whether one agrees with all of the efforts taken by Texas, it is not hard to see that education leaders are out ahead in most policy areas. Standards-driven reform linked to testing (that improved in quality from the early years of poor quality tests) have sent strong signals to all levels of the educational system. Jim Windham of the Texas Institute for Education Reform draws from a recent policy presentation to lay out his group's agenda:

• Enhance educator effectiveness: No education delivery system can be better than the educators in the school building. We need much better and more competitive preparation, certification reform, research-based professional development, effective mentoring, performance-based compensation, value-added evaluation, mandatory remediation and dismissal of ineffective educators.
• Raise standards: After 10 years, it is clear that TEKS needs a complete overhaul. The expectations for our kids are too low, there is no grade-level specificity, no progression of rigor from grade to grade and in many instances, the standards are not measurable.
• Strengthen accountability: We should phase into a 90 percent proficiency standard for accreditation of a campus, strengthen the consequences for school failure, adopt statewide public school choice, and expand charter school authority with equalized funding and tougher standards.
• Refine academic performance assessments: We should adopt value-added evaluation for charters, educator preparation programs and educator compensation; add end of course exams in high school; and connect all assessments to college and workplace readiness expectations.
Finally, we should create a comprehensive agenda for systemic long-term reform for public education that will fulfill the objective that every child in Texas will graduate from high school fully prepared for higher education, the 21st century workplace and responsible citizenship.

This potpourri of efforts attempts to address almost all elements of the production chain for primary and secondary education. The one thing that is not mentioned explicitly is increased accountability of schools of education to train better prepared educators and leaders.

Chris

Crappy People versus Crappy Systems

2006-10-08T12:17:00.000-05:00

In another post by Sutton, there is an important lesson about organizational improvement. As much as school turn-around efforts rely on great leaders, there is a fair body of evidence that even star performers cannot overcome broken systems. One of the concerns I see for school improvement along NCLB lines is that that blinders that focus on reading and math scores may induce district staff for focus on getting leaders into buildings who can turn things around. It may be more important to improve the alignment of curriculum and professional development resources than fix leadership. If it is the crappy system that is getting in the way of success, it is vital that we step back to take a look at the incentives and sanctions inherent in the current system.

Chris

Bad assumptions and submarines

2006-10-02T08:02:00.000-05:00

Bob Sutton does a great job laying out the problem with pay for performance policies that are sometimes linked to value-added measures at the classroom level. Pay might provide an incentive to a teacher to work harder or smarter, but there are a large number of things that have an impact on classroom level growth that are outside the teacher's control. If we are going to provide incentives at that level, it is crucial that we also ensure that resources are equally allocated and that we don't implement bad tests that force our teachers to teach to low/poor standards.

Chris

Testing to go nation-wide?

2006-09-30T13:44:00.000-05:00

The debate on national standards is alive and well.

Chris

Research Report at Pearson (PDF) discusses growth models

2006-09-28T17:01:00.000-05:00

This paper - buried on the Pearson research pages - provides some useful definitions of terms and lays out some of the difficulties one encounters when developing growth models.

Chris

A look at California's API and its use as an accountability measure

2006-09-25T21:56:00.000-05:00

California's API (Academic Performance Index) is criticized as inadequate for evaluating school progress or for judging the ability of schools to meet the needs of different sub-groups. The use of a complex indicator also makes interpretation difficult.

Chris

Return on Investment and Reform

2006-09-23T12:15:00.000-05:00

The Doyle Report takes an interesting look at our amazing inability to actually consider data when we think about school reform. There is probably ample evidence to support a hypothesis that year round schooling would be far better for achieving desired educational outcomes. In general, it would be interesting to actually with some level of confidence that education reform effort A actually adds more value to student learning than effort B.

Chris

On the road to the UK and France

2006-09-11T15:50:00.000-05:00

Posting is likely to be a bit spotty. I am off to give a couple of talks at the Strategies in Qualitative Research conference at the Durham University, UK on the maturity of qualitative research tools. I'm going to repeat that talk a few days later in Lyon as well as give a presentation to several teams within ICAR - CLAPI and others - on the scholarly history and current capabilities of an open source, video transcription and analysis tools we developed here at WCER - Transana. I'll be back in the states in two weeks.

Chris

An interesting back to school series from the Pittsburgh Gazette

2006-09-10T14:21:00.000-05:00

The piece on the fifth focuses on the impact of NCLB from narrowing of curriculum to holding schools accountable for achievement gaps. The follow on pieces look at other major trends in urban education.

Chris

Another sotory about how AYP woes play out in South Bend Indiana

2006-09-08T07:04:00.000-05:00

Here's another good piece to look at how AYP failures and successes play out in an Indiana town. There are many schools that area failing to meet expectations. There is going to be increasing pressure for change. The question remains is the change going to happen in the educational system or will members of the U.S. Congress find that they cannot stand the heat and back off on accountability.

Chris

11 Districts begin using value-added results in Pennsylvania

2006-09-05T07:40:00.000-05:00

Pennsylvania is using SAS to do value-added analysis of its test scores. 11 districts are already using the data as early adopter. Sanders does a nice job providing a non-technical description of value-added metrics. The state also provides some info on cost - $2.00/year/student - when all 501 districts are on board. Unlike a couple of other states, value-added assessments will be used diagnostically to track student progress but not to evaluate teacher performance. This piece gets to a lot of important issues, including growth of students who are already at high attainment. Take a look.

Chris

Examining the incentives for and against reducing high school dropout rates

2006-09-02T22:32:00.000-05:00

The Brownsville herald does a good job examing the incentives confronting teachers and schools around dropouts. Districts want dropout rates to decline to avoid systemic sanctions aimed at districts for high ratios. Individual teachers, on the other hand, may actually have incentives to counsel students out of attending, since the pressure on teachers is to get classroom, grade level, and school scores on state tests to increase. Students who are very far behind would require much more help than students who are near proficiency. The incentive is to neglect the students at the lowest levels.

The article goes on to study the economic impact of dropouts on the economy of the region. The fact that dropouts earn far less than do graduates means that they pay less in payroll taxes, are likely to consume lots of social and/or welfare services, and increase the likelihood that the individual will engage in criminal activity.

Chris

Californina has spent $70 million to almost be able to track students over time

2006-08-31T13:20:00.000-05:00

So, $70 million doesn't buy as much as it used to. This article claims that state officials are knowingly underreporting dropout rates to avoid federal sanctions. California is no exception in its difficulties to build a longitudinal system for tracking students and their outcomes. As the author points out, only Florida and Texas have truly robust systems. Other states, such has North Carolina, have had hugely expensive failures. A new bill has been introduced for this year's legislative session that would fund the completion of the system. But, even some of that bill's backers - namely the teachers' union - does not support the next logical step of linking teacher IDs to students.

Chris

The Plain Dealer worries about increasing complex school report cards

2006-08-27T16:37:00.000-05:00

The increasingly complex school report cards and the looming introduction of the "mind boggling" value-added analysis to the Ohio accountability system has folks worried that the will not be able to use or trust the new information. This is a real problem for NCLB. The bar on accountability is going up, but Joe Public doesn't know how to read the instructions for operation.

Chris

Money makes the world go around, the world go around,....

2006-08-25T19:16:00.000-05:00

Money may be an important fix. Very high salary levels would ramp up competition for slots that would squeeze out mediocre candidates. I'd like to see the math on that on what sort of wage would encourage enough people to retool for teaching.

Chris

Higher education and value-added - check the discussions

2006-08-24T00:07:00.000-05:00

When I first found this, I thought it was an interesting piece on the logic and difficulties of measuring the value added by a multi-path undergraduate experience. However, the discussion in the comments is even more interesting. Supporters and detractors present interesting cases that push the limits of where value-added can and cannot be applied. Attainment is still the goal. That's no different in higher education. However, the difficulty of coming up with useful (comparable) metrics is a tough problem. Generating the motivation to take any post-test seriously is another mountain of an obstacle - even if one could conceive of an appropriate testing regime.

Chris

Knox County, Tennessee Schools (and the rest of the state) dodge some bullets with growth models

2006-08-20T11:21:00.000-05:00

As predicted, the introduction of growth metrics as an alternative method for achieving adequate yearly progress has reduced the number of schools identified as failing. 8 schools met AYP using the new growth standard. Overall the number of schools not making AYP dropped from 159 to 96. More details about the response to being on the failing lest are available here.

Chris

About that "weighing the pig" analogy....

2006-08-18T10:03:00.000-05:00

Many of you know the "you don't make the pig grow by weighing it" analogy and have heard it applied to NCLB testing requirements. That's bugged me for quite some time as a pretty weak statement. I'm not a fan of testing for no reason and I hate tests that are not aligned to the education system's learing goals. Poorly aligned tests provide the worst sort of incentives and almost no vaulable feedback.

However, I still have a problem with this analogy. Let's think this through. I wouldn't want to compare education to fattening a pig for market. What do we know about pigs?

The growth trajectory of a pig from birth to slaughter weight is a pretty well understood piece of meat science. For more info specialization in this industry see the USDA background brief on hog farming.
Pigs have range of growth rates that is well understood. The grow at well understood rates throughout different periods of their maturation. The farmer only needs to provide food and water.
While weighing the pig does not make it go fatter, you better believe the farmer weights the inputs to pig growth (feed) very carefully. Farmers know very specifically how much of which kinds of nutrient sources a pig needs throughout its life cycle. It is in the farmer's interest to do this well and consistently through all stages of growth.
The growth characteristics of a pig are inherent to the pig. The only time the pig needs to be weighed is at time of sale to get the overall purchase weight. If the farmer does the production right, the pig will weigh what it is supposed to weigh at the time of sale. The weighing is about the commercial exchange between farmer and buyer.

What about student learning?

The growth trajectory of child learning is known and taught in child development courses, but we also know that there is great variance. We don't just want to get a child to some arbitrary amount of learning based on the child's characteristics. We have societal norms about what is expected for participation in a democratic society.
The growth of children is affected by the peer group. Pig A does not grow faster or more slowly if Pig B is a slower or faster learner. Peer effects in classrooms and schools is profound. One could think of this as one of the inputs to learning that we don't know much about.
The pig will grow if the farmer is poor or rich, black or white, if she likes or does not like pigs, etc. Kids learn better if they have supportive parents, come from homes with adequate resources, speak English well, etc. The thing pigs bring to the pen don't affect the pig's growth (apart from genetics around size itself).
We know almost nothing about the inputs to education - at least compared to our farmer. When we look at how much a student has learned, an enormous number of factors enter in to explain the variation one sees across the population. We collect almost none the data one would need to understand that variation.
The one thing we seem to have in common is the notion of the buyer weighing the pig/student. We publish test scores for schools and districts as a way of telling the buyer (taxpayers and voters) what they got for their investment in public education. That's about the only similarity I can see. Particularly since, we as citizens have already paid for the education at that point. This is a retrospective look to see what we eventually received for what we paid up front.

All in all, I don't see that the analogy holds much water.

Chris

eScholar Data Definitions - Embrace and Extend?

2006-08-16T12:16:00.000-05:00

K-12 data warehouse vendor eScholar provided data definitions for 750 elements to the pK-12 Data Model initiative sponsored by the U.S. department of Education, National Center for Education Statistics and mapped to both the SIF and NCES data standards.

This is a bold move for eScholar, but might provide additional leverage to market to states and districts, since they could guarantee data element alignment out of the box. Microsoft has a followed the embrace and extend model several times. This happens when an influential firm adopts an open standard and then adds substantial features to that standard that can, however, only be used by folks using the "extender's" systems.

This will be an interesting development to follow.

Chris

Critique and recommendations from the NEA as NCLB is considered for renewal

2006-08-13T21:47:00.000-05:00

The NEA discusses some of the unintended consequences of NCLB such as:

Clustering of students with disabilities in separate pull-out schools to reduce the risk of failure for regular neighborhood schools
Special education teachers are often required to demonstrate highly-qualified status in multiple disciplines. Many states do not yet have workable regimes for making this possible
Alternate assessments were specified by NCLB, but only 10 states currently have them in place. It is difficult create quality tests of any sort. Regular education testing has exceeded the testing industry's capacity. Alternate assessments are even further behind

This is a pretty lucid piece of argument from Patti Ralabate presented by the NEA to the Aspen Institute's Commission on No Child Left Behind.

Chris

Welcome to the math of NCLB - many schools slide into "failing" status

2006-08-11T08:13:00.000-05:00

In Vermont, schools categorized as failing climbed from 10 to 61. One of the causes of this five-fold increase was the inclusion of more children who had not been included in the testing system. This is the bane and boon of NCLB. The law requires that the schools be held accountable for teaching and testing all students. At the same time, the rising bar means that many schools will have to achieve gains in learning that may not be possible under any circumstances - particularly with the more need children.

State leaders cite growth models as a way of better reflecting what students are accomplishing on the path to proficiency. Many of the states that applied to the U.S. Department of Education's growth model pilot program cited reducing the number of schools on the "failing" list as one of the key motivators.

We had a study group here at WCER go through all of the applications for the growth model exception. While the is a reduction in the number of schools identified as failing in the short term, the attainment requirements don't get lifted. Under a growth model, it would simply put off the dramatic increase of schools identified as failing for a couple of years. At that point, the numbers would climb even more spectacularly.

That may actually be part of the strategy. The mechanistic application of the "failing school" label to the many, many schools that are actually doing pretty well for most students would likely create a huge political backlash against NCLB. I don't actually own a tin-foil hat, but this sort of logic would not surprise me.

Chris

Virginia pushes ahead with state-wide SIF Project

2006-08-09T22:40:00.000-05:00

Edustructures announces the launch of the second phase of their state-wide SIF implementation with a Student Locater application. The list of states deploying (or considering) state-wide SIF include; Wyoming, Oklahoma, and Pennsylvania. Delaware, Nevada, and South Dakota support the submission of some data using SIF.

Vertical integration, the ability to include XML markup that is not part of the SIF standard, and a web-services reporting API seem to be important feature in version 2.0 of the SIF that will continue to make it a compelling element of system designs for the at least the near- to mid-term.

Chris

Center for Analysis of Longitudinal Data on Education Research Founded

2006-08-07T15:41:00.000-05:00

Faculty from the University of Texas at Dallas are collaborating with the Urban Institute's Education Policy Center's prinicipal investigator, Jane Hannaway, as well as with scholars from Duke University, Stanford University, the University of Florida, the University of Missouri, and the University of Washington.

The researchers will be using state-wide databases from Florida, Missouri, New York, North Carolina, Texas, and Washington as the basis for their research. The research will focus on issues surrounding teacher quality (see Ed Week - subscription).

As Hannaway notes, this data will only be avaialble for secondary analysis to members of CALDER. FERPA concerns prohibit wider access. That's a bummer.

Chris

Blogging got overtaken by proposal writing

2006-07-22T20:07:00.000-05:00

Our value-added research group has been working on a number of proposals linked to various U.S. Department of Education and a couple of private foundation deadlines. It just so happens that they all fall in the latter half of July. Several are out the door and several more are due this coming week. We are proposing more work that links inputs (financial and human resources, professional development, curricular materials, etc.) with outputs (classroom practices, student tests, grades, attendance, etc.) in one urban district. This work would tie together ongoing school- and grade-level value-added analysis of test with data to other information systems that have traditionally been kept separate.

On the national front, we are bidding with one of applicants for the technical assistance center that will support the U.S. Department of Education's Teacher Incentive Fund. It's a bit of a pig in a poke since we don't know who will be awarded the TIF grants. So, there's no telling how challenging the work will be. We do know from years of experience that getting the models right is brutally hard. For example, if testing is done in late fall or early spring, growth cannot be simply assigned to a single grade or school. The credit for growth has to be apportioned proportionally to the two grades in which it occurred. Other issues such as how to handle mobile students and teachers and changes in test forms also complicate models. Students retained in grade present a challenge for growth modellers. If a student is retained in fourth grade and takes the fourth grade test again, what is the growth one would expect from the 4th grade to 4th grade?

If we win even half of the work proposed, we will have new windows into aspects of schooling and management that will advance our understanding of how best to support decision making at all levels of the educational system.

Chris

Indiana wants test scores measured over time

2006-07-18T22:15:00.000-05:00

Like many other states, Indiana wants to move to some sort of growth or value added modeling to keep more states off the failing list. We've been looking at this at Wisconsin Center for Education Research. We recently evaluated several of the growth models proposed by states in a series of brown bag sessions this summer. We used longitudinal research data we have in house to impose various state models on the same data set to get an apples to apples comparison. One of the clear factors common to most of the models is that they decrease the percentage of schools identified as failing by 10-12%. This is only a year one effect, however. By 2-3 years out, if the high growth rates predicted are not met (and that's not to surprising), all states are back at high percentages of schools failing. So, the current flurry of activity may only buy 2 or 3 years of coverage.

Chris

Ed Trust takes a look at teacher quality

2006-07-15T16:46:00.000-05:00

Just in case you wondered how much teach quality matters - particularly vs exposure to advanced curriculum, Ed Trust has some sobering findings. In a report published on July 8th, the authors (Heather G. Peske and Kati Haycock) provide some scary numbers that reflect things we've heard before like - students in schools with high percentages of poor and minority students are twice as likely to have novice teachers. They are also more likely to have teachers teaching out of their primary subject area/area of certification. This was a particularly scary quote:

[T]here were stunning differences in levels of readiness according to the quality of teachers in a school. In schools with just average teacher quality, for example, students who completed Algebra II were more prepared for college than their peers in schools with the lowest teacher quality who had completed calculus.

These numbers are the sort of thing that would make me want to go hide under the bed.

Chris

Just in case you were wondering why realtors support school report card sites

2006-07-13T21:07:00.000-05:00

Here we have a study that shows just how much house values in crease with a 20% increase in students meeting proficiency - 7%. It seems though that the authors have a funny take on value-added reporting when they study its impact on house values. Just looking at the change in proficiency rates between 4th grade and 9th grade cohorts isn't value added, since you don't know if it is the same kids or any of the qualities of the instrument.

Time to go back to school.

Chris

Missouri struggles to show that new tests meet NCLS quality standards

2006-07-11T19:42:00.000-05:00

Missouri administered new tests in grades 3-8 this spring and must now provide technical data to show that the tests meet test quality requirements set forth in NCLB. Maine and Nebraska are the only states that have had tests disallowed. In Maine it was the use of the SAT as the high stakes test for hight school students. Illinois is likely to encounter similar feedback as it attempts to use the ACT for high school assessment. NCLB requires that assessments be aligned with state standards. It would be hard to argue that national college entrance exams reflect local standards.

Chris

Discrepancies in graduation rates in the news

2006-07-10T07:24:00.000-05:00

A recent Editorial Projects in Education Research Center report published by Ed Week on graduation rates points out discrepancies in state graduation rate calculations (by state) and shows the status of efforts to implement a nationally-accepted definition of graduation rates proposed by the National Governors' Association. What the report (and the article linked to this post) reveal is how hard it is to get agreement on, define, and implement what sounds - on the face of it - to be a fairly simple concept. Image the difficulties surrounding dropouts and who gets the "credit" in a high stakes system for that drop out. The last school? What if the student was only there a week? Who gets the credit for a graduation if the student spent most of his or her time at one school and then switches to another, less effective school in the last semester?

These difficulties are all over high stakes data analysis.

Chris

Not a simple story, but evaluators have to love this headline

2006-07-07T14:44:00.000-05:00

"State hurting education by not funding data collection"

Hewlitt Foundation Education Program Director Marshall (Mike) Smith and Hewlitt Program Manger Kristi Kimball have some fairly strong words for the legislature's failure to provide adequate funding for the collection of high quality district data. This is one of the problems with which every state in the nation needs to grapple. The siloed organizational structures of school districts and state educational agencies are a manifestation of the compartmentalization of funding and accountability from both state and federal agencies. Decades of developing stove pipe reporting capacities are fundamentally inadequate for the task of addressing questions about "what works" in education. The effectiveness of complex social phenomena, such as effective educational practices for particular communities, are difficult to measure under the best of circumstances. In education we have data systems designed for different purposes and staff who have always been rewarded for hoarding data and reporting up - not using the data.

The inability of state systems to address "bang for the buck" questions continues to stymie legislatures. After years of building state and district capacity, California seems to have snatched defeat from the jaws of victory.

Chris

Michigan jumps into the Pay for Performance fray

2006-07-05T15:23:00.000-05:00

Michigan seems poised to join Minnesota and Colorado in using pay increases tied to school level performance as an incentive for teachers. As the reporter notes this comes in the wake of an announcement by the Bush Administration to provide $500 million in supplemental funds to support pay for performance plans. It looks like value-added evaluation capacity might be a highly sought set of skills.

Chris

Florida sums up its options under NCLB and 2007 testing

2006-07-03T08:03:00.000-05:00

Florida, like a number of other states, is looking at a substantial number of schools (535) failing to make AYP targets for 5 years running next year. If it happens again, they go under automatic restructuring. However, like many other states, there are a number of schools in this list that are successful in most areas and are only missing AYP in one area. There are no exceptions however. Schools missing AYP at all are subject to restructuring.

This dynamic is one of the reasons that Florida (and most of the other applicants) applied for the U.S. Department of Education's growth model experiments. Under a growth model that predicts eventual proficiency, students who are currently under AYP, but can be predicted to achieve proficiency in the future, can be counted as proficient.

It is no wonder that states with the capacity will want to take this path. It does not obviate the need to achieve 100% proficiency, but it puts the extreme sanctions into the more distant future.

Chris

A Policymaker's Guide to the Value of Longitudinal Student Data

2006-07-01T08:34:00.000-05:00

This posting comes from 2002, but these recommendations remain solid policy recommendations. The disturbing thing is how far many states and districts still are from making this a reality.

Chris

Growth Modeling and Grad Student Training

2006-06-26T10:32:00.000-05:00

Not sure when we will get some publications out of this, but we are running a set of meetings between our NSF-sponsored Interdisciplinary Training Program and the Value-Added Research Center. Here is our internal announcement. I'll post content as it becomes available.

Chris

ITP/VARC Summer Workshop

The Interdisciplinary Training Program and the Value-Added Research Center at WCER will jointly sponsor a summer workshop on "Growth Models for Adequate Yearly Progress."

When it comes to labeling good and bad schools, NCLB is caught on the horns of a dilemma: Use absolute targets to set the same standards for all schools, or focus on growth to recognize progress relative to student starting points. Until now, NCLB has taken an extreme position by focusing exclusively on absolute targets, which rise at varying rates but must reach 100% proficient by 2014.

Secretary Spellings recently granted waivers to two states -- Tennessee and North Carolina -- to pilot a "growth model" approach to Adequate Yearly Progress. Have these states solved the dilemma of recognizing growth while maintaining absolute standards? That is the question we will consider in our summer workshop. We will also examine the 6 unsuccessful applications to see why they were rejected and whether we like their solutions any better. Finally, we will examine real data from Milwaukee to see how the growth models will play out.

The summer workshop will be held on the following schedule in the 13th floor board room of Education Sciences:

Wednesday, June 7, 12:00-1:00pm -- Introduction: Adequate Yearly Progress under NCLB
--In this session we will review AYP, examine the call for growth models and the instructions to peer reviewers, and select state proposals for study.

Wednesday, June 21, Review of State Proposals and Peer Reviews

Wednesday, June 28, Review of State Proposals and Peer Reviews
--In these two sessions, participants will (a) describe selected state proposals; (b) present the reviewers' critique; (c) present their own critique; and (d) reach a conclusion on how close the proposal comes to solving the fundamental dilemma. As a group, we will decide which approaches are worth trying out with real data.

Wednesday, July 5, Conclusion: Application of Growth Models to Milwaukee
--In this concluding session, we will use data from Milwaukee to carry out the procedures selected in our review of state proposals, and reach a conclusion about the prospects for growth modeling under NCLB.

AEI weighs in on the politics of NCLB

2006-06-23T19:55:00.000-05:00

Frederick Hess and Michael Petrilli lay out the history of NCLB. They explicitly focus on the frustration felt by members of Congress about the lack of results from decades of reforms intended to address the achievement gap. There was a consensus that high expectations were important, but the censuses did not extend to root causes. Some thought that low expectations themselves were key. Others identified lack of resources for low achievers as the most important problem to address. A third explanation was school culture and a lack of effective organizational structures that led to institutional paralysis. However, the censuses that all students should be able to learn and all schools should be able to produce significant achievement gains bridged these different understandings.

Politically, the attraction of the NCLB consensus was that it allowed public officials to embrace high standards and champion equal opportunity without having to prescribe uncomfortable solutions or explain exactly what strategies would enable schools to succeed....Ultimately, NCLB was intended to provide political cover to superintendents and school board members to encourage them to take controversial and difficult steps to root out mediocre teachers and administrators, shift resources to poorer schools, challenge collective bargaining provisions regulating teacher transfer and preventing efforts to link pay to teacher quality, and overhaul central office processes.

Hess and Petrilli do a good job laying out the history and the trends that are likely to find their way into the reauthorization. The consensus around high expectations seems to be holding. The major change is the inclusion of growth in student learning as an additional feature (not a replacement of attainment requirements).

Chris

More folks looking at NCLB as a deeply flawed policy instrument

2006-06-12T07:36:00.000-05:00

Utah is officially adopting state legislation that preferences state standards over NCLB. Many states are moving to lower proficiency standards because NCLB expectations for 100% proficiency for all groups is unachievable.

"In some ways it's creating a race to the bottom," [Michael Petrilli of the Fordham Foundation] said. States are obliging schools' and parents' requests to make the tests easy enough to achieve "socially acceptable" pass rates, he said.

How do we keep the best of NCLB - the notation that all kids deserve a good education and that we should be able to know that they are getting it - without the ugly incentive to dumb things down?

Chris

I'm off to chaperon a bunch of middle and high school choristers on a 10 day tour. Back on the 22nd of June.

More info on the Sanders study of learning benefits of board certification

2006-06-10T09:10:00.000-05:00

The Greensboro News-Record provides a little more detail on the study done by William L. Sanders "found that students of nationally certified teachers make no greater classroom gains than do other students".

Not exactly what folks arguing for greater professionalization and higher qualitative standards in the teaching workforce are the key to greater student learning. One worry is that tying incentives to board certification may overlook equaly effective teachers.

Chris

North Carolina - Sorry about that test, do your best.

2006-06-08T21:22:00.000-05:00

It turns out that North Carolina decided not to use this year's new test to evaluate student performance. Teachers are instructed to use grades, classroom tests, and other tradtional measures to judge student proficiency for promotion. Hmmmm. This is one of the two states that is going to be allowed to use growth modelling? It seems that the last time the state delivered new tests, it go the cut points for proficiency wrong - based on the fact that too many students scored poorly on the tests. The state plans to spend the summer getting those numbers correct.

That seems a little strange that the numbers can be so subjective. I guess the right percent proficient is the one that shows most schools making their AYP numbers. That must be right.

Riiiiiiight.

Chris

Even Florida doesn't get the Growth thing right.

2006-06-06T16:18:00.000-05:00

Florida is widely thought to have to most comprehensive student information system in the US. This piece lays out the impact of not being selected for as a growth model pilot in pretty stark terms.

If it had been accepted, an estimated 43 percent of Florida's schools would have made Adequate Yearly Progress (AYP) under NCLB. Without it, only 29 percent are expected to make AYP, down from 36 percent last year.

With numbers like that, it is no wonder that states are concerned about access to this alternate method to show student progress.

Chris

Schools Matter - A not very flattering analysis of the recent growth model choices (and NCLB in general)

2006-06-04T20:16:00.000-05:00

Schools Matter takes on both the recent decision to allow only North Carolina and Tennessee and tinkering with sanctions to allow more supplemental services before schools become accountable. Blogger Jim Horn notes (as do many others) that NCLB seems designed to crush public schooling by showing nearly all schools to be failing under the law's inflexible and (probably) unachievable metrics.

Chris

NSBA outlines recommendations for the next round of NCLB modifications

2006-06-02T16:47:00.000-05:00

While most observers do not expect there to be significant changes in NCLB until an eventual reauthorization in 2009, the National School Boards Assocation is pushing for a series of fixes to the existing legislation.

Chris

An Adoption Strategy for Social Software in the Enterprise. Many-to-Many:

2006-05-31T18:48:00.000-05:00

While this post explictly addresses "social" software, its lessons can also be applied to decision support systems (DSS). DSS do not always contain collaborative tools (apart from email or notification features) they are about sensemaking at a distance. The goal of DSS is to help users and decision makers at different levels of the enterprise to make sense of larger or smaller sub-systems. The lessons of rollout and the role of leadership are appropriate for decision makers in the education sector as well.

Chris

Growth Models: Results of the pilot request evaluations

2006-05-29T13:40:00.000-05:00

Secretary Spellings and the US Dept. of Ed publish the winners and losers in the first round of pilot states.

Chris

Measuring education outcomes, it's not as easy as it sounds

2006-05-27T14:05:00.000-05:00

Graduation rates are an important measure of success for the educational system. The difficulty of measuring drop outs and graduate is outlined in this piece from Kansas. It's hard at the state level and even harder at the federal level. The author gets the problem statements right. We need to have higher quality data and we need to focus on how we are failing those kids who don't get diplomas.

Chris

Teacher commitment matters more than certification or experience

2006-05-25T21:15:00.000-05:00

Well....I guess that's not to hard to understand. Committed teachers in early grades who take personal responsibility for student learning tend to produce more learning growth.

LoGerfo's other findings reinforce other widely held precepts - supportive leadership makes a difference, Catholic schools tend to show higher commitment than public schools.

As we work on getting the models right, it will be important to make sure that teacher practice and perceptions make it into the mix.

Chris

Hitting the high standards ceiling?

2006-05-23T22:03:00.000-05:00

As Schrag points out, it is not surprising that the warts on NCLB are becoming increasingly clear several years into the implementation of the legislation. States with high standards, such as California, are being penalized since it is becoming ever more difficult for schools to meet the expectations of 100% proficiency in the face of tough standards. Indeed, a judge in California just issued a preliminary injunction against state barring it from using the high school exit exam to bar students from graduating. The argument was that many students were not given an adequate opportunity to learn - given their exposure to poor schools and teaching. Without equity of access to a quality education, it is argued that students should not be held to these high standards.

Schrag also points to extreme examples of the narrowing of the curriculum in many schools. Some schools went as far as teaching only math, reading, and gym. Is this what Congress intended?

Chris

Milwaukee Public Schools Superintendent used growth metrics as part of reform package

2006-05-20T09:38:00.000-05:00

Milwaukee Public Schools is now in its fifth year of reporting value-added metrics for individual schools in the system. Sup. Andrekopoulos is using one of the core reporting forms from the value-added report. The district level report includes a simple 2-by-2 table of schools at the high, middle, and elementary levels. These tables lay out high and low attainment on the vertical axis and high and low student value added on the horizontal axis. The district is giving more autonomy to schools that show high attainment and high growth. Schools with low growth and attainment are being singled out and are receiving more direction from the central office, including the placement of "instructional facilitators" in each school. These facilitators report to central curriculum and instructional staff rather than to the local leadership.

It is important to note that this is not classroom or grade level value added. This is not a system that sanctions individual teachers. Rather, it focuses on research in the district on effective teaching strategies that correlates with improved student outcomes and seeks to implement these strategies in failing schools.

Chris

And they're off - 2 states allowed to experiment with growth models

2006-05-18T20:42:00.000-05:00

It seems that North Carolina and Tennessee will both be allowed to added growth models to their official NCLB compliance frameworks. It remains to be seen how this will actually play out. Tennessee will be using a proprietary model that has never had any rigorous external evaluation. North Carolina was just in the news this week for the major cost overruns and failures surrounding it's state-wide student data system.

Chris

Data Quality and the risks of "running with what we have"

2006-05-17T07:08:00.000-05:00

There is a great temptation for education (and other) organizations to just "get something up" and call it a data warehouse as part of a strategy of retaining the support of senior leaders. One of the problems data warehouse designers have in organizations with little history of decision support is that the clients (program area staff) literally cannot identify needs that extend past their current experience with data. One common solution is to take the current operational data and its definitions (such as they are) and simply load them into a warehouse. One can then take existing reports as the design documents for data marts.

The good thing about this approach is that it provides a wonderful teaching environment for bringing program staff into the discussion using data and representations that they know and can make sense of. The risk is that they will see this and want to run with it. It is guaranteed that these data (and definitions) will contain serious quality problems there were not exposed or stressed under the older, more constrained reporting system. While this might seem like an early success, going forward with this system can be very risky. Program staff are experts in their programs. Data problems will emerge and they are likely to blame the system rather than the data or collection processes. They are also likely to see the problems as someone else's problem and not be receptive to requests that they "clean" the data. They may come back with requests for IT to fix the transactional system.

This is a cautionary tale for states working on getting their warehouses up as rapidly as possible.

Chris

The Ed Trust compares proposed growth models

2006-05-15T19:47:00.000-05:00

The Education Trust has been very explicitly concerned about the risks of relying only on growth models for accounability. There is the concern that focusing only on growth rates will lower attainment expectation for the nation's neediest students. The Trust compares the stark differences between the Alaska and Arkansas plans in their article and provides a table that compares the strengths and weakness of all of the proposals. This is an easy and informative read for those not prepared to wade through all of the propoals themselves.

Chris

Ed Week delivers research report on data use

2006-05-12T17:46:00.000-05:00

An excellent survey of the lay of the land at the school, district, and state level with respect to data use and decision support.

Chris

Exclusion of small subgroups annual AYP reporting

2006-05-11T19:32:00.000-05:00

Here is one of those cases in which the reliance on a single metric to both provide feedback to the instructional system and, at the same time, hold folks accountable bites one in the posterior. The use of confidence intervals around measures with low sample sizes is a critical part of interpreting statistics. Most statistics text books cite a sample size of 15 as the minimum number justify the assumption of normal distribution. Given the measurement error associated with traditional standardized tests, sample sizes of 2-4 times that number would not be an unreasonable standard for reporting.

An inaccurate measure is fine for providing instructional feedback. Teachers have a wealth of information about student performance. The high stakes test result is just one observation among many. Low reliability is not a problem in this case. However, for high stakes, the use of low sample sizes (and the resulting lack of confidence in the point estimate) creates serious concerns about the consequential validity of any sanctions or rewards based on that measurement.

Chris

PS This Week in Education also has some good links on this story.

Back from my 20th year grad school reunion

2006-05-10T19:25:00.000-05:00

No computer on the road! Two weeks with no email or phone calls! Working though a large backlog of stuff to post.

Chris

Technical documentation of value added metrics in the UK

2006-04-22T14:15:00.000-05:00

Whoa baby! Are those some wacky value-added metrics. The pretests are based on averaging across math, science, and reading. What in the world does that measure? Post tests are based on the highest eight scores on a battery of tests - so, individual post test scores aren't even based on the same tests.

I guess some of the state growth models being suggested in the U.S. don't look that bad.

Chris

Long and Short Decisions - We need flexible tools

2006-04-19T07:02:00.000-05:00

Engineers without Fears point out an important consideration for those building decision support systems for educational leaders. The large temporal grain size of state data systems lead to the creation of systems to support "long" decisions - those problems that allow time for consultation and consideration. The problem with this notion is that leaders a the district or school level often have to make "short" shoot-from-the-hip decisions and have little structured information to consider when the time to decide arrives. Many leaders will have to rely on the resources provided by the state - even though they were not provided for that purpose and may be unreliable in a "short" context.

The creation of robust use cases that help design "long" decision resources need to be accompanied by cases that show the appropriate use of such data in "short" situations. These cases should also outline how data likely to be available to local decision makers can be combine with state data to make better decisions.

Chris

How do modern leaders view social capital?

2006-04-17T21:18:00.000-05:00

I like the stuff at Connectedness. Bruce does a good job summarizing Brass and Krackhardt and their description of leadership under strong and weak tie constellations. The simple examples do a good job laying out career-stage and other important considerations. This sort of analysis is vital part of work engagement we do in working with state agency and school district program staff and administrators. Without understanding leaders' network strategies (often unarticulated) it would be difficult to intervene successfully.

Chris

National Transcript Center aims to deliver commercial transcript transmission solution

2006-04-14T06:57:00.000-05:00

Ok. I guess preventing fraud in transcripts is important. That wouldn't quite be might my first priority. The nirvana I seek is the ability to use transcript data within a district to look at students and their opportunity to learn. It's tough to understand differential outcomes of kids if all you know is their characteristics and maybe the teachers they had. We all know that scope and sequence matters when designing curriculum. It follows that we need to know this stuff to figure out if curricula, PD efforts, new approaches to educational guidance, etc. are working or not. That should be what is driving automation of transcripts.

Chris

Teaching leaders to see social capital

2006-04-12T13:47:00.000-05:00

As we consider methods for intervening in complex education organizations to improve decision making, I have been looking for resources to provide a wide range of metrics. It's clear that we will be able to improve standard report production efficiency for test data and other high-demand, high-cost products. However, it will be vital to demonstrate the payoffs to units and leaders without direct connections to large scale data warehouse efforts. Social network analysis may provide insights into the structure and effectiveness of data sharing and expertise networks that could radically improve agency performance.

Chris

One of the states being considered for permission to use growth models discusses the prospect

2006-04-10T08:03:00.000-05:00

The state superintendent's chief of staff downplayed the state's earlier criticism of NCLB. This is the tack that several states have pursued recently. Many people believe the expectations embedded in the legislation is unrealistic and unachievable. The sense seems to be that it will crash on its own illogical requirements. Instead, it makes sense for states to work on more realistic systems to study growth.

Chris

The fundamental problem with "over the bar " NCLB-style reporting from a district teaching resource-poor children

2006-04-08T22:37:00.000-05:00

This quote nails one of the primary problems with NCLB reporting. For schools or teachers working with resource-port children, attainment models can be fundamentally disenfranchising.

'I don't fundamentally disagree that we should be held accountable, but you have to look at where the students are when we get them and measure the growth by the time they leave you,' Young said. 'None of the growth that occurred below the bar is counted.'

As much as anything else, it is an issue of respect. People want to have progress acknowledged. A feedback system that says only this single point will be a legitimate reference for success is severely limited.

Chris