I’m super glad Bill Gates knows, as well as other education experts such as economists and Malcolm Gladwell– but it turns out that I had no idea.

The story starts out, as all roads to hell do, with good intentions. Two weeks ago my district provided all ELA teachers with their own in-house jerry-rigged estimations of what our January state exam scores were. (We generally receive these from the state in May, about six weeks before we never instruct the involved students again. The state has also refused to share with districts its score calculation formulas, leaving us as deeply in the dark as, oh, say, students who aren’t provided with transparent assessment expectations. Not that there’s any connection there.)

I was planning to rummage through my own fun house of stats right in front of everyone on the blog. I imagined myself bowing my head over my keyboard with determination and honesty, culling the admiration of my peers and stepping up to self-betterment with statements like “These scores aren’t high enough, and I need to get with the program.” Or “I’m disturbed by what these data indicate about my instruction of English Language Learners.” And then the disillusionment began.

~~~~~~~~~~~~~~~~~~~~~~

OK, so I’ll just show that approximately 82% of my students passed the exam, and that means I’m a 82nd percentile teacher. Not great, but not horrible, right? And then I’ll–

Wrong, my child.

Who the hell is that?

I am the Ghost of Statistical Analysis Yet to Come.

Er…there’s fog rolling into my classroom.

I visit those who wander this bleak land looking for dummy variables and fixed teacher effects.

Why can’t I see you?

Few can. (pause) And I have this really neat cloaking device called the New York State Scale Score Conversion Chart.

Um. Okay. So, how can you help me here?

You must remember first, child, that PERCENT is not the same thing as PERCENTILE.

Oh, yeah…this is coming back to me now from…

(eagerly) From a graduate class?…

Uh, no. We’re not required to take statistics in pre-service preparation in New York.

(disapppointed) Oh.

But okay, okay, I remember that thing about percentiles. So THAT means that I actually need last year’s scores for my current students, right? Then, I can compare them to this year’s scores. I’ll see how many of these students scored at or below proficiency last year, how many scored at or below proficiency this year, and THAT will give me my percentile.

Er…

Ok, we’ve got a plan. I just have to…well, I’ll need to email the district data person, because I don’t have the scores from last year. Well, I do, but they’re in this Excel sheet we got in September. I can just cut and paste the jerry-rigged ones in….oh…except… the jerry-rigged ones from 09 are on paper. Hey– do you have any magical powers that can shunt paper scores into already-existing Excel spreadsheets? In alphabetical order? And then massage them so they show something meaningful?

Look, I don’t have any magical powers. I’m not Michelle Rhee.

Well, jeez, no need to get upset.

Your ignorance is starting to become irritating. There are factors you’re not considering.

Oh, yeah, of course… like the 7th grade exam and the 6th grade exam test completely unrelated skill sets, so comparing their scores is useless.

Well, that’s not quite what I–

Or that our state exam is given in January, so Mrs. Johnson’s ability to teach my kids in 6th grade is mixed up with mine in any analysis of the exam scores.

Mrs. Johnson is not really the point–

And Mr. Allbright. Some of his kids are on our team this y–

ANYWAY, you’re still wrong about how to calculate teacher percentile.

I am?

Sigh.

A little tutorial, then. New York City has prepared a document for its teachers which explains how percentiles are calculated for human resources decisions.

I thought that was just an experiment for general knowledge. Isn’t that what the NYC DOE told the participating teachers?

Never mind that. (Waves hand to displace some fog, and a pdf appears).

(reading the pdf) oh….oh….OH. I get it now. So a teacher’s percentile is based on whatever standardized exam the students happen to take in a district from year to year?

Correct.

Not a nuanced national assessment of some kind that all kids have to pass. Or that all teachers have to pass, for that matter.

A what?

Never mind. So the district uses a formula to PREDICT gains for kids from year to year…

Yes…

and then however many of my students meet the predicted gain…

(supportively) Yes…

…is my percentile ranking.

No.

What are all those little brown filaments falling on the floor?

It’s your hair. Don’t worry, it’s a common occurrence around here. Now, there’s one step more you need to take.

OK…(scrutinizes the pdf again)…oh, all right. I have to look at the amount of gains my kids make–

Yes…

on the non-national exam–

Yes, yes–

that may or may not have been controlled for poverty, teacher experience, and non-random teacher assignment–

(cheerfully) But back to percentiles!

(sighs) So when you compare TEACHERS– me– against OTHER teachers in the district, and what percent of us met or didn’t meet the predicted amount of our kids scoring as proficient on the exam–

Yes, that’s right…

then THAT’s my percentile ranking.

You’ve grasped it! Finally!

But–

The exit is right over here. Excuse our dust. We’re constantly renovating, you know.

But–

Now what is it?!

My district hasn’t calculated any of this.

…What?

I said, my district hasn’t calculated a prediction for gains, or done any comparisons of teachers against teachers. So I can’t publicize what my percentile ranking is. I don’t even have the data to calculate it myself.

(long silence)

And anyway, doesn’t this mean that we can’t make any really meaningful comparisons between districts, or states, that use different exams? Or different gain calculation formulas?

Look here. I don’t like your argumentative tone. I’m trying to HELP–

And THEN, the Education Public Interest Center out of University of Colorado recently published a 50 page brief about how there are at least six calculable socioeconomic factors that affect school achievement: and that schools– that is, teachers– cannot overcome them alone. What about that?

(frantically) A good teacher can overcome any obstacles in his way! Just look at KIPP!

Uh, yeah. I will. Right after their teachers unionize because they’re killing themselves trying to overcome alone the six calculable socioeconomic factors that affect school achievement.

(losing control) You know what? I don’t give a crap. Don’t get me wrong. Creativity is good and whatever. But if the children don’t know how to read, I don’t care how creative you are. YOU’RE NOT DOING YOUR JOB!!

(Pauses, and heaves a heavy sigh. Begins to weep.)

No one really understands me.

(puts a comforting arm over GOSAYTC’s shoulders)

It’s ok. It’s really ok. I believe in you. We’ll get through this together. Somehow.

(looks around apprehensively at the fog, which is rolling in, in thicker and thicker waves…)

Somehow.

~~~~~~~~~~~~~~~~~~~~~~~