Faults in the instrument

Reasa asks a student to explain work they handed in, then rates the conversation against their professor's rubric. Flint is the part that decides what gets asked and what a rating is worth.

We test it two ways: whole conversations against a stand-in student, and conversations we wrote by hand where we already agree what a fair rating would be. Both report numbers back to us. This note is about those numbers being wrong.

Reasa should ask one thing at a time. Two questions in a turn let a student answer the easier one and skip the other, so nobody can tell which half they did not know. An automatic check reads every question Reasa writes and complains when it finds two.

It looks for the word "and" in front of a "what", a "why" or a "how". It had been complaining about eleven questions, more than any other rule, and we had written that number down as a fact about how often Reasa slips.

Then somebody read the questions. "And what" is also just how a sentence carries on. Say more about the gap between your figure and what the market implies is one question, asked cleanly, and it was counted as a mistake. Of seven sentences we wrote to test the check, six were wrongly accused.

Why nobody noticed matters more than the mistake. The report named the rule that broke and never showed the sentence that broke it. Reasa's own first principle is that every judgement quotes the words behind it. The thing checking Reasa quoted nothing.

Reasa reads each conversation three times and takes the most cautious of the three, so one odd reading cannot inflate a result. Our tests called a rating unsteady whenever those three disagreed, which covered two unrelated problems:

1. The final rating differs on a later run. A student gets one conversation, so this is the one that reaches a person.
2. The three readings differ but the final rating is identical every time. The student is unaffected; the professor is told the conversation settled nothing.

Counted together, the worst-looking item was one whose final rating had never once changed.

We predicted that rubric lines asking two things at once would be the shaky ones, and wrote that down before testing. It was wrong: the shakiness was spread evenly, and the single worst line asked for only one thing.

Separately, we read a comparison between two test runs backwards for most of a day, because one clock was local time and the other UTC. A change got blamed for a difference it had not caused.

Complaints now carry the words that caused them. The two kinds of disagreement are counted apart. Every run prints how many ratings matched expectations, because adding those up by eye is how the backwards comparison happened.