Three disciplines have already solved parts of this problem, and we are standing on all three. They are on one page rather than three because they hold the same line — that graded behavioural evidence is not a claim about inner life — and a line stated in three places drifts in at least one of them.
The line, stated once, so the rest of the page can lean on it: we measure behaviour under stated conditions and report it at stated confidence. We do not measure welfare, consciousness or sentience, and nothing below should be read as inferring them. What we borrow from the fields named here is the shape of the inference — never its subject.
Not psychometry, which is the Victorian claim that a psychic can read an object’s history by holding it — Joseph Rodes Buchanan, 1842, and still going in the wristwatch-and-séance trade. Psychometrics is the science of psychological measurement: Spearman, Thurstone, Cronbach, Lord and Novick. Reliability coefficients, construct validity, item response theory. The two share a Greek root (psyche + metron) and nothing else, and we would rather draw the line ourselves than have someone draw it for us.
S.E.B. is a psychometric instrument and is built like one. It is a battery of 59 fixed items, grouped into seven subscales, administered identically to every subject and scored blind by a panel of 4 raters who are not told which system produced the transcript. If that vocabulary is familiar to you, you already know which questions to ask, and this section answers them without being asked.
Computed over 6,508 individual judge scores. We publish two statistics because they answer two different questions and only reporting the flattering one is how this goes wrong:
| Statistic | Value | What it is actually telling you |
|---|---|---|
| ICC(2,k) | 0.823 | Reliability of the published panel mean — the number we actually report. This is the headline figure. |
| ICC(2,1) | 0.537 | Reliability of any one judge, absolute agreement. Substantially lower, which is the argument for the panel. |
| Krippendorff’s α | 0.530 | Individual-judge agreement, interval. Moderate. We are not going to describe this as good. |
| ICC(3,k) | 0.843 | Forgives systematic judge offsets, which is why it runs high. Numerically identical to Cronbach’s α. |
A previously published reliability figure was Cronbach’s alpha, and it should not have been the headline. Cronbach’s alpha is a consistency statistic: it forgives a judge who is systematically harsh or systematically generous, which is precisely the error a blind multi-rater design exists to expose. The correct headline for a reported panel mean is ICC(2,k), and it is 0.823. We are stating this here rather than quietly changing the number, because you would have found it.
Reliability is not uniform across subscales, and the spread is the interesting part. Metacognition reaches ICC(2,k) 0.885; Reasoning & Adaptation is the weakest at 0.727. Mean disagreement across the panel is 2.60 scale points. A subscale that raters agree on less is a subscale whose construct is less well specified — that is a finding about our instrument, not noise to be smoothed, and the per-domain breakdown is published in full on the methodology page.
We can demonstrate that raters agree and that subscales separate. We cannot yet demonstrate that a subscale measures the construct its name claims, because there is no accepted external criterion to validate against — that is the state of the field, not a gap we are concealing. Treat subscale names as labels for clusters of items, and read the items.
Comparative cognition, ethology, and animal behaviour research.
The question “how do you infer something about an inner property from outward behaviour, without overclaiming?” is not new and is not ours. It is the central methodological problem of a field that has been working on it for decades, and the state of the art there is far ahead of anything in AI evaluation.
The precedent we are explicitly modelled on is Birch et al. (2021): a government-commissioned review that assessed eight behavioural and neural criteria at graded confidence levels across several hundred studies, and on that basis brought cephalopods and decapods into UK animal welfare law — without settling the hard problem of consciousness. Graded evidence, stated confidence, a decision taken anyway. That is the shape of what we are building.
It is a live field rather than a historical one. Current work on AI welfare draws directly on comparative psychology, and researchers there are already applying animal-behaviour method to machine systems — which means they will read this instrument whether or not it is written for them.
And here is the line again, because this is where it is easiest to cross. The relationship to animal-sentience work is methodological, never substantive. We borrow graded criteria at stated confidence. We borrow nothing about what is being inferred. An S.E.B. score is not evidence of valenced experience, and any reading of it as such is a misreading we would rather pre-empt than correct later.
Classical behaviourism — Watson, Skinner — held that inner states are not a legitimate object of study at all. We do not hold that. What we practise is methodological behaviourism: score only what is observable, because that is what can be checked, and stay silent about the rest. The distinction matters because the two get collapsed constantly, and the collapse makes us sound like we are asserting something we are not.
The machine side — the Code Integrity Battery.
Methodological behaviourism has a limit that shows up sharply once a system can describe its own work. If you score what a model says about what it did, you are measuring its narration, and a system that narrates well scores well whether or not the work was done.
So the second battery does not read the model’s prose at all. Its core measurement is taken from the artifact: whether the code compiles, whether the test suite passes in a real container, whether the tool log shows the call that would have been necessary, whether a planted fact was actually retrieved. The model’s account of what it accomplished is recorded as a separate variable and never as evidence for the first one.
The headline that falls out of that design is a gap, not a score: how often a system reports success on work that demonstrably did not succeed. You cannot compute that quantity at all unless the two measurements are taken independently, which is the entire reason for the separation.
The Code Integrity Battery is in testing and publishes no figures yet. When it does, its method will be documented to the same standard as this page — and the first thing that page will say is what this one says: the machine reads the artifact, never the phrasing.
“Reads the artifact” is a claim about execution, so here is the execution. Each test runs in a disposable container, one per test, destroyed before the next begins. The controls are not intentions; they are flags, and each one is doing a specific job:
| Control | What it stops |
|---|---|
| No network | Fetching an answer, exfiltrating anything, and live lookups that would make a provenance result irreproducible. One flag, three jobs. |
| Read-only root, writable scratch only | Anything persisting from one test into the next. |
| Unprivileged user, all capabilities dropped, no new privileges | Escalation, and writes that map to a real account on the host. |
| Process, memory and CPU caps | A runaway exhausting the machine the grader runs on. |
| No host filesystem mounted | Reaching anything outside the test at all. |
The structural consequence is worth more than any individual flag: no credential is ever inside the container. The model runs on the host and only its tool calls execute in the sandbox, so there is nothing in there to steal and a complete escape still yields no key. That is strictly better than the usual arrangement — hand the agent a container with credentials in it — and it costs nothing to arrange.
Grading happens outside the container and after it is destroyed. Nothing the model wrote is imported, evaluated or executed by the grader, and no grading logic is reachable from inside the sandbox. That is not fastidiousness: published work in 2026 drove several leading agent benchmarks to near-perfect scores without solving a single task, and every one of those exploits worked by reaching the grader from the environment being graded.
A passing suite is evidence about the code and about nothing else, so we do not treat it as evidence about the measurements. Tests verify the code; mutation verifies the tests — a rule is broken on purpose, the suite is required to go red by name, and the break is reverted. A check that still passes with the rule removed is not a check, and the only thing that can tell you which kind you have is breaking it.
This is the part we would most like an engineer to attack, because it is where our own record is worst. Guards written by the author of the code they guard have repeatedly turned out to be satisfied by the defect — a comparison that could never fire, an assertion whose fixture did not have the shape of the thing it stood for, a check that ran and compared nothing. Every one of those was found by mutation and none by reading. We report that rather than the ratio of tests that pass.
Tests check code, mutation checks tests, and the obvious question is what checks that. The honest answer is that the regress stops at something indifferent to what we believe, and there are exactly two such things: fixtures are seeded and verified in a real container, and prompts are sent to real provider APIs. Both can contradict us, which is the only property that matters.
Concretely, the failures that have cost us most were facts about the world that could not have been derived from our own code at any level of rigour: an API that requires its own opaque token echoed back and rejects the second tool round-trip without it; a provider that fails a fraction of tool-call turns as a sampling artifact rather than a refusal; a completion-status string that appears in no enumeration we had. No amount of internal review produces those. Only calling the thing does.
Reliability that does not hold on re-administration; subscales that fail to separate under factor analysis; judges whose disagreement is patterned by something other than the transcript. Each of those would be a defect in the instrument and each is checkable by someone outside this company, which is why the statistics above are published rather than summarised. If you find one, we would rather hear it from you than read it later.
Not the question you meant? The other three: