Nobody does scientific caution like Matt Di Carlo, blogger for the union-affiliated Albert Shanker Institute. Though “not so fast” isn’t the sexiest message in education debates, we could surely use more fair-minded and careful voices like his. But this week, Di Carlo takes his caution to the extreme, writing:
“Charter schools and their students may have done very well over the past year, or they may not have. These data cannot tell us either way. Period.”
Not so slow, Mr. Di Carlo.
The post’s main point is a fair one: there are more students, and different students, in the NYC charter school sector from one year to the next. Aggregate proficiency rates across a growing sector are not a measure of individual students’ growth. (They were never designed to be.)
Di Carlo’s main concern is that new students have arrived in charter schools, who presumably may have different characteristics than last year’s batch. So what would happen if we ignored the newcomers, looking only at the grade-level cohorts that were in tested grades last year also?
Di Carlo didn’t check, but we did. Long story short, the ELA proficiency rate was about the same (52.6% vs. 51.5%), and the math rate was even higher (75.4% vs. 72.0%). In other words, excluding the new kids brings the charter average up—as one would hope, given that the first-year cohorts of students have spent the least time in the charter sector. (See technical notes below.*)
| Comparison, all test cohorts | Comparison, consistent test cohorts | |||||
| 2011 | 2012 | Change | 2011 | 2012 | Change | |
| ELA | 44.5% | 51.5% | +7.0% | 45.9% | 52.6% | +6.7% |
| Math | 68.3% | 72.0% | +3.6% | 68.9% | 75.4% | +6.6% |
| Number of Schools | 91 | 105 | +14 schools | 90 | 90 | no change |
| Number of Students | 17,633 | 21,728 | +23.2% | 15,029 | 14,443 | -3.9% |
That doesn’t make proficiency rates a shining measure of school effectiveness, of course, but the casual “nothing to see here” is just too easy. In this fallen world where all data are limited, these numbers are something we can work with, at least until the state’s growth-based system comes online. Proficiency rates also aren’t too crude for the UFT bloggers at EdWize, which this week called charter-district comparisons “real enough.” Even more confident is AFT head (and Albert Shanker protégé) Randi Weingarten, who has even been tweeting about what a Daily News comparison of proficiency rates “disproves” about new DOE schools in the Mayor Bloomberg era.**
DN stats disprove that.”@howiewolf: @TeacherArthurG New schools far superior to the schls they replaced. Our strategy has been a. success.”
— Randi Weingarten (@rweingarten) July 22, 2012
What does Matt Di Carlo think of that same article?
The gross misinterpretation of this year’s NYC test results continues, this time on the other “side” – nydn.us/OhBD9s
— Shanker Blog (@shankerblog) July 22, 2012
Give him points for consistency. But why not spread the word to Randi? Scientific purity begins at home.
NOTES
*For each school, for the “consistent test cohort” 2011 results, we excluded exit grades (for example, grade 5 at a k-5 school) and all grade 8 results. For the “consistent test cohort” 2012 results, we excluded entering grades (for example, grade 5 at a 5-8 school) and all grade 3 results. To ensure that the number of studied school-grades was consistent, schools closing in 2011 as well as schools tested for the first time in 2012 were excluded. Like Di Carlo, we are unable to describe individual student mobility based on public data. Mobility issues are discussed at length in our State of the Sector Report.
**Here’s a response from the Bloomberg camp.
