IB 2020: model-predicted grades

01 · CONTEXT

Grades assigned without an exam

With the final exams canceled, the International Baccalaureate (IB) was going to use a statistical model to assign students' grades. The article underlying this case cites more than 160,000 affected students, and the IB had 166,000 candidates in 144 countries in the previous cohort.

The final grade matters a great deal: according to the article, it lets the student graduate, apply to universities, and accept admission offers.

The statements attributed to the IB in this case are quoted by the article's author and do not come from IB documents themselves. The text is a critique of the announced method, and this case uses it to study the design and risks of a high-impact automated decision like this one.

As quoted in the article, the IB stated that these assessment arrangements represented the fairest approach it could take for all of its students.

1. Two inputs described in the article: the grades on projects and assignments submitted before the disruption, and the predicted grade, which is what the teacher believes the student would have obtained had the exams taken place as planned.

2. The method also uses historical data and data from the current session, as well as other data whenever it exists.

3. The model predicts the final grade from coursework grades and predicted grades.

4. The predicted grade is then combined with the coursework grades to obtain the final grade: the coursework enters both stages, which the author calls a double penalty.

5. The final grade is what lets the student graduate, apply to universities, and accept admission offers.

02 · THE INPUTS

What goes into the model and where it repeats

According to the article, the IB's method uses historical data and data from the current session. Among the described inputs are the grades on coursework submitted before the disruption, the grade predicted by the teacher, and other data, whenever it exists.

The author points to a double penalty: the model predicts the final grade from the coursework and the predicted grades, and that prediction is then combined with the coursework grades. In his view, those who did poorly on coursework are penalized twice.

The author cites a study according to which high school teachers tend to give lower predictions for non-white and disadvantaged students, and says this is problematic because the predicted grade has a prominent role in the model.

Regarding the additional data, the author says it can induce biases and make the predictions for some schools more accurate than for others.

03 · UNEQUAL ERRORS

The same model, different errors by school

According to the author, small schools, 15% to 30% of IB schools, would have larger and more frequent errors than large ones. The principle is simple: the more data from a school, the more accurate the predictions.

To give an order of magnitude, the author assumes the largest cohort at an IB school has 300 students. In that calculation, a student in a cohort of 5 would have an average error about 25% larger than students at the school with 300.

04 · NO ANSWER KEY

Getting the distribution right is not getting the student right

The IB canceled the cohort's final exams. The author says it cannot guarantee that students would end up in the same position in the distribution as they would if the exams had taken place.

The IB, according to the author, assured students that it would match the previous year's grade distribution. In his view, this is a cosmetic improvement that still gives the IB some plausible deniability.

To show the limit of this adjustment, the author imagines a model that gives random grades to each student and then adjusts the grade bands to match the previous year's distribution. In his view, this does not make up for the fact that the model gives grades disconnected from reality.

The author proposes assuming a model with 90% accuracy, a target he calls almost unrealistically ambitious. In that case, the IB would assign the wrong grade to at least 1 in every 10 students.

The author also reminds us that a modelo being predictive does not mean it is correct: an accurate model can be bad, and spurious correlations are problematic if they are not detected.

05 · THE EXPERIMENT

Even without sensitive attributes, the model infers them

The author cites experts to say that ignoring sensitive attributes is, in general, a practice somewhere between ineffective and harmful. To demonstrate, he builds an experiment that he says is very similar to what the IB does, but with data from New York schools.

1. The author trains a model with each school's county and scores on 3 tests, with no data on race, socioeconomic status, or gender.

2. This model predicts with almost 80% accuracy whether the graduation rate is above or below the national average.

3. In the probe, the author asks the same model whether the majority of the school's students are Black or Hispanic, and the model gets this right with higher accuracy than the graduation rate; the author concludes it is race-aware.

4. For socioeconomic status, the model recognizes majority economically disadvantaged schools and detects them with about 75% accuracy.

5. For majority gender, accuracy is below 50%, and the author concludes the model does not use this attribute.

6. When the author gives the majority race to an alternative model, the accuracy of the graduation prediction rises by only about 1%, which he reads as evidence that the original model was already race-aware.

Finally, the author states that there are three fairness criteria for predictions and that no model satisfies all three at once. So, in his reading, any IB model would discriminate in two of the three ways described.

06 · DESIGN LESSONS

What to take into an automated decision system

The author describes the context as an outsourced, black-box model, with limited historical data, no oversight of the decision mechanism, and 3 months for research and production.

His conclusion is that the situation calls for a process solution, not a modeling solution, and that a delayed, suboptimal solution is better than a discriminatory, error-ridden one.

For the design student: compliance and governance go into the design from the start. Define the human contestation path, the record of inputs and model version per decision, and the monitoring of error by group.