Research
Faults in the instrument
Most of what looked like a fault in our scorer was a fault in the thing measuring it. One automatic check was wrong six times in seven, and we had built it so that nobody could see that.
Human-facing AI development
We build AI that makes people more capable, not less necessary.
Our research is applied. We study where AI genuinely makes someone more capable, and we ship the software that does it.
Systems that decide for people erode the skill of deciding
A judgement handed to a system stops being exercised, and the competence behind it decays quietly. The loss is invisible until the system is wrong and nobody left can tell.
So we work on the other problem. Not what a system can do instead of a person, but what leaves that person more capable afterwards, and whether the difference can be measured rather than claimed.
It is a narrower thing to automate a task than to improve the person doing it. We think the second is where the durable work is.
How we build
A system can route around a person, or route through them
- Two arrangements of the same parts, side by side.
- Routed around: a task goes to the system, the system does the work, an output comes out, and the person is left unchanged.
- Routed through: a task goes to the system, the system does the work and hands to a person, the person decides, and what comes out is both an output and a person who is more capable.
- The difference is whether the person sits in the path or beside it.
Routed around
Routed through
Routed around
Task
same startRouted through
Task
The system does the work and the output comes straight out. The person is not in the path, so nothing about them changes. This is the arrangement most AI products are built to produce.
The system still does work, but it hands to a person rather than past them. What it produces is an input to their judgement, not a substitute for it.
The result leaves the system and goes on to whatever comes next. Nobody in the path was asked to agree with it, which is what makes the arrangement quick and what makes it unattributable.
The output is theirs. That means they can be wrong about it, can be argued with, and can be held to it — none of which is true of a result a system issued on its own.
They received a result they did not produce and cannot check. Next time they will need the system more, not less, which is the opposite of what we are trying to build.
The person finishes knowing something they did not know before, so the next task starts from a higher floor. A system that never produces this is doing the narrower job.
Where the software sits
A system informs a judgement. It does not hold one.
- A chain, read left to right.
- What a person gives it. The system forms a view and shows its reasoning. The person decides, and may disagree. The outcome is attributable to that person.
- Authority sits at the third link, and the third link is a person.
Input
Only what the system receives directly. There is no second, quieter channel of collected signal behind it — see the boundary in our commitments.
System
A view is not a determination. Nothing downstream is triggered by it, and the reasoning is visible so it can be disagreed with on its merits rather than accepted on authority.
Person
The link that holds authority. Their disagreement is the outcome, not an appeal against one. An interface that makes accepting easier than questioning has moved the authority without saying so.
Outcome
Somebody is accountable for it, and that somebody is not a model. This is the property institutions actually need, and it is lost the moment the third link is automated.