Tuesday, August 09, 2005

Even more fun with data: What does it all mean?

Now that we have the 2003 and 2004 data the next logical step would be to look at each school system's year-to-year performance. The following chart shows the difference between the 2003 data and the 2004 data:

ChangeElementarySecondary
 ReadingMathReadingMath
All Schools+2.11%+1.94%+2.09%+2.68%
DCPS+1.84%+4.63%+0.81%-0.38%
Public Charters+6.06%+6.82%+9.09%+19.72%
BoE Charters-1.70%-1.58%+2.77%+3.33%


Its tempting to look at the above data and start making value judgments about the various school systems, but that would be misleading.

When comparing a group to itself over time you want that group to be consistent throughout the study. For example, if you were going to test the effects of a cholesterol drug on a group of 10 people over the space of a year it doesn't make much sense to allow half of the study group to leave half way through the study and replace them with anyone off the street. You want the group you start with to be the one you end with, or at least as close as possible. The bigger the difference the less sure we can be at speculating at the causes of the results.

For example: Let's take a hypothetical 10th grade public charter student. He has just transfered to a public charter school after attending a DCPS school for grades 1-9. He just recently tested as proficient in both math and reading on the Stanford 9 tests. How much of that proficiency should be attributed to public charter school he attends and how much should be attributed to his old DCPS schools? Remember, the exam test for the proficiency in things that a 10th grader should know. One possibility is that the student came into his 10th grade year having a good, solid 9 years worth of educational background provided by the DCPS system and the charter school did a good job of building upon that. Both school systems were equally good. Another possibility is that the student entered the charter school already possessing the knowledge needed for the test; i.e. his DCPS education was exceptional and it didn't matter how good or bad the charter school was. Yet another possibility is that the student entered the charter school with an inadequate education and the charter school education was exceptional and made up the difference. Without knowing a lot more information -- for instance knowing how many students transfer and tracking their scores -- we just don't know which, if any, of the above examples explains the student's test scores. And this is just for students that transfer. Each year you also have students who leave the system (due to graduation, relocation, dropping out, etc.) and students who enter the system (due to relocation, transferring from private schools, etc). Without quantifying the extent of these changes it can be difficult to make generalizations between each year.

So, back to our data. The population of both charter school systems is relatively small, and, although I don't have any hard numbers, I would assume that the percentage of charter students who have transferred into and out of the system each year is pretty substantial. For the reasons I outlined above I don't think the year-to-year comparisons would be that meaningful. On the other hand the DCPS system has a much larger population and, again absent of any hard data, I would suspect that, as a percentage, the number of students transferring into and out of the system is pretty low; i.e. unlike the charter school population the vast majority of DCPS students being tested they have been in that system from day one. Therefore, the year-to-year differences are a little more indicative of how the school system does at creating students who are proficient at math and reading for their grade level. However, there is another large caveat. We only have two (soon to be three) years of data here and something as large and complex as a municipal education system changes take time to implement. Two years of data is really a snapshot at best. It certainly doesn't give us any real insights, statistically, into the direction that the school systems are heading. That will take a few more years worth of test data.

So, to wrap up: its good to see that DC's students' proficiencies are rising. Lets hope that its just the start of a longer, significant trend.

No comments: