Wednesday, August 03, 2005

Fun with data (get yr statistics on, part 1)

Some interesting conversation popped up from yesterday's posting about the newly released No Child Left Behind (NCLB) statistics, so I figured I'd start playing with some NCLB data to see what, if anything, could be gleaned from them. This first batch of data that I am going to present is at a high, system-wide level. We will look at the District's public school system first, as a whole, and then subdivided into the following three catagories: 1) DCPS schools, 2) public charter schools, and 3) BoE charter schools.

Before we begin, some definitions:
  • DCPS Schools are the traditional DC public schools.
  • Public charter schools are schools who were granted their charter by the DC Public Charter School Board.
  • BoE charter schools are schools who were granted their charter by the Board of Education.
  • Private schools are not included.
If you want to play along at home, the data can be found here.

The first set of statistics we will look at is the 2004 NCLB Adequate Yearly Progress (AYP) metrics. Note: this is the latest full set of data I have available. These are calculated by looking at the students' results of the Stanford 9 tests. The results are broken out by elementary and secondary schools, and into separate math and reading scores. Each score shows the percentage of students who tested as "proficient" for their grade level.

Click on me for a larger picture

All Schools46.22%55.52%31.87%39.32%
Public Charters46.36%56.14%37.39%54.12%
BoE Charters42.15%43.43%28.77%30.18%

CORRECTION: Corrected the All Schools: Secondary Math number. It 39.32%, not 39.52%.

As you can see, at the elementary level DCPS and public charter schools are roughly equivalent for both math and reading scores, with BoE charter schools lagging (although I'm unsure how statistically significant the differences are between the three populations; this will be part of further analysis). When we get to the secondary level however, the numbers begin to spread out. All three school populations see reductions in proficiencies from their elementary school metrics, with public charter schools seeing the least amount of erosion: 9% in reading, and 2% in math. DCPS loses 15% in reading, and 20% in math. Again, we'll have do more analysis to determine the statistical significance of these differences, but at first glance the losses look particularly painful for the DCPS schools. Furthermore, considering that the BoE schools' elementary numbers were not very robust to begin with, that 13% drop in both sets of metrics really hurts.

CORRECTION: BoE charter schools lost 13% in both math and reading from elementary to secondary school, not 15% as originally written.

So, what does this all mean? Well, at first glance, DCPS and public charter elementary school students perform at roughly the same level in both reading and math . However, while both see subsequent erosion at the secondary school level, on average students at public charter schools outperform DCPS students, especially in math. Finally, students at BoE charter schools lag students at both DCPS and public charter schools, and this is observable at both the elementary and secondary levels.

To caution, this is very preliminary data, and I'll continue to crunch the numbers on this end. If I (or you!) find any errors in the analysis, I'll update this posting, documenting what has changed and why. As I said above, if you have a hankering to play with numbers, go ahead and check out the data for yourself.

UPDATE: I have done more number crunching on this data set. First, the population samples for each data segment:

All Schools20,55024,57520,52824,165
Public Charters1,8503,9841,8493,991
BoE Charters5211,6285231,618

CORRECTION: The PCS secondary math number is 3,991, not 9,991 as previously listed. This was a transcribing error, the analysis below was done with the correct data.

I then tested that to see if the test scores for each population were significantly (statistically) different. (See note below for methodology and caveat)

The results:
  • There is no significant difference between the level of proficiency from the elementary reading tests of DCPS and public charter schools. There is most likely a difference between the levels of proficiency from the elementary reading tests of the BoE charter schools and the other schools; the statistical tests were persuasive (p<.10) but not where we would want them to be certain (p>.05)
  • There is no significant difference between the level of proficiency from the elementary math tests for DCPS and public charter schools. The level of proficiency from the elementary math tests of the BoE charter schools are, however, significantly lower than the proficiency of the other two groups of schools.
  • At the secondary school level the difference between both the reading and math proficiency for each of the three school systems is significant: i.e. we can say that in both the math and reading tests that the public charter school students were more proficient than DCPS students, who in turn were more proficient than BoE charter students.
This verifies the earlier conclusions I had about the data.

One thing I want to stress is that this is an analysis at the school system level; this does not imply that any public charter high school had higher proficiency than any DCPS high school. Within each school system are schools with high proficiency rates and schools with low proficiency rates. This analysis simply looked at the entire test taking population of each school system. I do intend to do some school-by-school analysis in the future, but it was beyond the scope of this particular exercise.

That's it for tonight and this post, baring any data or statistical screw-ups anyone finds. Next up: similar analysis of the 2003 data and 2005 data when available, and a year-to-year comparison for each school system.

UPDATE 3: Commenter Ed Researcher, who does this sort of thing for a living, makes some good points in the comment section that are worth reading.

UPDATE 2: Here's an interesting tidbit. According to the help page for the AYP reports the Number in Group metric is
Number of students enrolled in the group for a ‘Full Academic Year’
If you add up the elementary and secondary school Numbers in Group for DCPS and BoE charter schools you get a grand total of 41,823 students. In 2003 the official enrollment number published by DCPS was 65,099 students. Even if you don't count the alternative programs, special education, and tuition grant students (I have no idea what their testing responsibility is) that still leaves over 60,000 students. I assume that the 2004 enrollment numbers aren't that far off of 2003 (if anyone has them let me know and I'll update this), so the question is, where did the other ~20K students go? Why were they not tested? Does anyone have a better understanding regarding who does and does not get tested, i.e. am I missing something?

UPDATE 2a: I did miss something. DCPS only tests 3rd, 5th, 8th and 10th grade, which explains why the test taking population was less than the entire school population.

I've been plugging away with statistics for about 15 years, but its been a donkey's age since I did much work that went beyond your bog-standard gaussian distributed data-sets. I'm a little rusty with statistics around categorical data, so if anyone notices that I've done something horribly wrong please drop me a line in the comments.

The data presented for the Districts public and charter schools only shows how many people took the test and what percentage of them passed it. What we don't know is how well or badly each individual test taker did. For example, if 60% was the passing grade for a test and it shows that 100 students passed the test we don't know if those 100 students were brilliant and had perfect scores, they all just scraped by with 60s or something in-between. All we know is that they either passed or failed. Thus, for each population we can only back out the number of students who passed and the number of students who failed (in NCLB lingo, students who were proficient vs. students who were not). So I used Fisher's Exact Test to compare the pass/fail rates between the various school systems.

Greetings Carnivalites. Feel free to make yourself at home and read all about DC education. There is more coverage of NCLB metrics here and here.


Bob said...


The enrollment numbers for secondary math do not add up. First, it doesn't make sense that the number tested in math is so much higher than the number testing in reading (which, according to these figures, is the case for all charter schools at both elementary and secondary levels). But the secondary math total number of students is 30,165, not 24,165. And I'm suspicious about the big jump in the number for public charter secondary math, which happens to be the highest performing group. Something is not right.

Nathan said...

Bob, thanks for the catch. It was a transcribing error. The number should have been 3,991. Thankfully the resulting tests were calculated with the correct numbers (3,991 total; 2,164 proficient; 1,827 not proficient) and not the 9,991 number. I have noted the change in the post body.

Ed Reseacher said...

Besides living in DC, I do statistical analysis of education data for a living and, while I haven't read this whole thing closely, I don't think you're way off.

A weakness I do see is in how you interpret these differences. First of all, it's not quite right to compare elementary and secondary and say that there is a trend. They are different sets of kids taking different tests and most importantly, entirely different schools. Most of those kids in chartered secondary schools probably attended regular public schools when they were in elementary grades.

The second point is that AYP measures student performance, not school performance. Imagine one school has an unusually high percentage of students exceeding a threshold and another had a low percentage. If those two schools swapped students, which school would have higher scores? If you guessed the second one then you correctly recognize that these measures don't get at "value added" by the schools, which is what policy makers need to measure if they want school accountability to be fair.

What I take away from this is that the public school board gives charters to groups serving lower achieving students, perhaps kids with greater economic disadvantages or learning disabilities.

Let me also say, however, thank you for going through this exercise and posting your thoughts and analysis. More citizen-bloggers should do this, and maybe a few pros too.

KOB said...

Mind numbing work for the math challenged (moi) but great work.