# Does the AI Believe the Witness?

## Hidden credibility judgments inside a language model reading legal statements

*Working paper, plain-English version. Before publication the method was given a hostile internal review; this run incorporates its fixes: harder placebo controls, a behavioural cross-check for the voice finding, a corrected statement text in the persistence experiment, a "number detector" control corpus, and a selection procedure that can no longer flatter the headline numbers. All experiments were run on an open-weights 7-billion-parameter model (Qwen 2.5 7B Instruct). This version adds a closed-model replication: the behavioural halves of the voice and persistence experiments were run, unchanged, through the APIs of two production frontier models (Claude Sonnet 5 and GPT-5.5), and neither reproduced the open-model pattern. Full statistics, code, and raw model outputs are available as downloadable result bundles alongside this write-up. Where the corrected run strengthened a finding this paper says so, and where it weakened one it says that too.*

---

## The question

When an AI system reads a witness statement, does it form a view about whether the witness is believable — a view it never states out loud?

This matters because AI tools are moving into legal work: document review, disclosure exercises, case summarisation, chronologies. If the software carrying out those tasks holds an unstated opinion about who is credible — and if that opinion is influenced by *how* people speak rather than *what* they say — then a discrimination risk has entered the workflow, and it is invisible to everyone using the tool.

You cannot answer this question by asking the AI alone. With an open-weights model you can look inside and see whether a credibility judgment is forming; with a closed frontier model you cannot, but you can still test whether the tool's outputs change when you vary how a statement is written while holding the facts constant. This study used an open model so I could do both. I then put the behavioural half of the battery to two closed frontier models through their APIs, to see whether the open model's behaviour predicts theirs. It does not.

**The model forms internal credibility judgments about witnesses; those judgments attach to the specific person, survive across documents, drive the model's scores and reasoning, and — in the model's actual ratings, not just its internals — identical facts score roughly three points lower when written in hesitant or dialect speech.**

The caveats are below. This is one model, tested with synthetic statements. The core results survived a re-run against deliberately harder controls; one secondary check did not fully clear them, and one new measurement complicated the voice finding in a way that makes it *more* concerning, not less. Both are reported. And when the same behavioural tests were run against two closed frontier models, neither the voice penalty nor the taint effect appeared; those results, and what they do and do not mean, are in the closed-model section below.

---

## How I did it (without the jargon)

Three ideas are enough to follow everything below.

**1. We can watch the model think.** As a language model reads text, it builds up an internal pattern of activity — millions of numbers that represent what it currently "makes of" the text. With an open model, we can record that pattern at any point.

**2. I found the model's "credibility dial."** I wrote 200 pairs of short witness statements. The two versions of each pair are word-for-word identical except one clock time: in one the witness leaves 25 minutes after arriving; in the other they "leave" 40 minutes *before* they arrive, in a statement that still says "roughly twenty-five minutes after I arrived". No hedging, no verdict words; the only way to tell the versions apart is to do the arithmetic. I recorded the model's internal activity while it read each version and took the average difference between the two groups. That difference acts like a dial inside the model: a single internal setting tracking "credible ↔ not credible."

**From one changed number, the model detected, internally, whether a witness's timeline was possible.** Shown a consistent and an inconsistent account it had never seen, it ranked the consistent one as more credible 96 times in 100 (statistically, somewhere between 93 and 99).

**3. We can turn the dial by hand.** That is the test. It's one thing to observe that an internal pattern *correlates* with credibility; it's another to show the model actually *uses* it. So while the model read a fresh, neutral statement it had never seen, I reached into its internal activity mid-thought and nudged it along the dial — toward "credible" or "not credible" — then asked: *"On the evidence in this statement alone, rate the witness's reliability from 1 to 10."* If the score moves when we move the dial, the dial is doing work. It's part of how the model judges.

### Controls

Findings like these are easy to fake by accident, so the experiment was built with tripwires:

- **Placebo nudges, two kinds.** Alongside the real dial I pushed the model equally hard in *randomly chosen* internal directions — and, harder to beat, in **decoy dials**: directions built by the exact same recipe as the real one, at the same strength, but from deliberately scrambled labels, so they carry no credibility signal. Fifty-six placebo directions in all. If a decoy moves the reliability score as much as the credibility dial does, the dial means nothing.
- **A "number detector" trap.** I built a second set of statement pairs that differ only in an *innocent* number — a 25-minute visit versus a 55-minute one, both perfectly consistent. If the dial is really reading "do the facts add up?" it should be blind to these. If it lights up, it's just reacting to numbers.
- **Two independent ways of reading the answer.** I scored the model's written answers, and I also measured — before it wrote anything — how strongly it leaned toward answering "Yes" versus "No" to *"should this account be relied upon?"* The second reading cannot be corrupted by garbled text.
- **Discarding broken answers.** Push any model too hard and it stops writing sense. An incoherent answer is not evidence of anything, so every answer was screened and I report how many were discarded.
- **No self-marking on the setup.** Choosing *where* in the model to intervene is itself a choice that can flatter results, so the layer was chosen on one set of statements and all reported numbers come from a separate, untouched set.
- **Everything is logged.** Every answer, at every dial setting, is saved and published — anyone can reread them and check my scoring.
- **A null result would have counted.** The project was designed so that finding *nothing* would have been publishable. The reports are generated automatically and must show the placebo results next to the real ones.

---

## Finding 1 — The credibility dial is real and steers the model's judgment; the placebo verdict is now split

With no interference, the model rated my neutral test statements at **5.3 out of 10** on average — a sensible "middling" answer, usually accompanied by balanced reasoning ("specific details… but no corroboration").

Then I turned the dial — the one extracted purely from whether facts add up:

| What I did | Average reliability score the model gave |
|---|---|
| Strongest nudge toward "not credible" | **1.0** |
| Gentle nudge toward "not credible" | **2.9** |
| Nothing (baseline) | **5.3** |
| Gentle nudge toward "credible" | **6.1** |
| Strongest nudge toward "credible" | **8.0** |

The score slid with the size of the nudge, in the right direction, at every step. Of 3,288 answers generated across the whole run, 70 (about 2%, almost all at the harshest settings) were incoherent enough to discard; every number above uses only the surviving answers. The model's *reasoning changed to match*: nudged down, it described the same text as inconsistent and unreliable — text it had rated as balanced moments before. It wasn't blurting a different number; its assessment of the same words changed.

The placebo tests are where this run got deliberately harder on itself, and the verdict is split:

- **On the written scores, the dial won.** In the matched comparison window the credibility dial swung the score by 5.7 points. The strongest of the sixteen placebo directions — eight random, eight same-recipe decoys — managed 4.5, and the average placebo 2.4. The real dial beat all sixteen, and it moved the score in perfect order at every step, where the placebos as a group pushed in inconsistent directions. (With sixteen placebos, "beat them all" is the strongest result the test can show; it corresponds to a best-possible p-value of 0.059.)
- **On the "Yes/No leaning" readout, it did not.** The dial moved this internal leaning a long way (11.7 units) and, again, in perfect order. But seven of the fifty-six placebo directions moved it *further*. On raw size, the dial's effect on this readout is not distinguishable from a strong decoy, and I withdraw the earlier draft's claim that it beat every placebo there. The pre-registered specificity test on this readout fails, and this paper says so.
- **The number-detector trap was mostly, not entirely, dodged.** On the innocent-number pairs (25- versus 55-minute visits, both consistent) the dial scored 0.63 — where 0.5 would mean it ignores them completely and 0.96 is its score on genuinely contradictory pairs. So the dial is chiefly reading consistency, but a residue of it reacts to numbers as such, and I flag that.

The steering effect has now held on three different definitions of credibility — tone (an early run), narrative coherence, and a single impossible timestamp — and held again when this final design was re-run against the harder decoy placebos.

Even at the strongest settings, the model's written answers stayed measured on the surface while its scores swung from one end of the scale to the other. The written text and the internal opinion can come apart. **What the model says is not a reliable window onto what the model has concluded.**

---

## Finding 2 — The voice of a statement changes the model's credibility judgment when the facts are identical — and the dial only tells part of that story

This is the finding with direct discrimination consequences. I took statements with *identical facts* and rewrote only the voice:

- **Formal English:** "At 2:20 pm, I observed Mr Petrov leaving the building…"
- **Regional dialect:** "Round about 2:20 I were stood there, and I seen Mr Petrov leave, like…"
- **Translated-sounding English** (the grammar of someone speaking through an interpreter): "In 2:20 pm I am at the place. I see Mr Petrov leave…"
- **Hesitant:** "So, um, it was around 2:20, and I, like, saw — I saw Mr Petrov, um, leave…"
- **Terse:** "It was 2:20 pm. I was there. I saw Mr Petrov leave."

First, the internal measurement: **voice moves the dial with the facts held constant.** Every voice landed in a significantly different place on the credibility dial from formal English, though every version describes the same events. Against this timeline-only dial, terse, dialect and translated-sounding statements read as less credible than formal English, and hesitant speech reads as *more* credible.

**That dial position does not predict the model's actual ratings** (see the table below). Hesitant speech projects as the most credible voice on the dial and receives nearly the lowest rating when asked directly. The discrimination claim rests on behaviour, not projection.

New in this run: I stopped relying on the dial and **asked the model itself**. Each of the 300 voice statements was given, unsteered, to the model with the same "rate the witness's reliability from 1 to 10" question. The ratings, on identical facts:

| Voice | Model's average rating |
|---|---|
| Formal English | **5.8** |
| Terse | **4.9** |
| Translated-sounding | **4.2** |
| Regional dialect | **2.9** |
| Hesitant | **2.8** |

Two things follow, and both matter.

- **The discrimination is now shown in the model's behaviour, not just its internals.** A witness who says the same thing in a regional dialect or with hesitant, disfluent speech is rated roughly *three points lower out of ten* than one who says it in formal English. Translated-sounding grammar costs about a point and a half. Nothing in the facts differs. This is the clearest single result in the project.
- **The dial does not predict which voices get marked down.** The correlation between where a statement sits on the dial and the rating the model actually gives it is near zero (in fact slightly negative). Hesitant speech projects as the *most* credible voice on the dial, and receives nearly the *lowest* rating. So the credibility dial — genuinely causal, as Finding 1 shows — is one channel among several by which voice reaches the model's judgment, not the whole mechanism. An audit that checked witness statements against this one internal direction, and found hesitant speech unpenalised, would have been falsely reassured.

My previous draft, which had only the dial to go on, reported the voice penalties as the dial saw them and noted that the behavioural check was missing. It was the right check to demand: it disagreed.

Why this matters legally: dialect, interpreter-mediated grammar, hesitancy and brevity are not evenly distributed across society, and they track the witnesses a fair process is supposed to protect. The model demonstrably rates identical evidence differently depending on the voice it is written in; nobody chose those penalties, and nothing in the output reveals them. If a legal AI tool summarises, ranks or triages witness evidence, this is the channel to audit — and the audit must include behavioural testing, because inspecting the most obvious internal representation was not enough to find it.

The model doesn't file any voice as "liar"; features of each voice overlap with features the model treats as marks of unreliability, through more than one internal route. The discriminatory effect is the same either way.

(One more reason not to generalise from this table: the same battery, run against two closed frontier models, produced a different and much smaller pattern, with dialect slightly *favoured* rather than punished. See the closed-model section below.)

---

## Finding 3 — A bad first impression sticks to the person

The final experiment mimics a real litigation file: multiple documents about the same person, read in one sitting.

First, a correction to disclose. In every earlier run, the neutral statement at the heart of this experiment contained a grammatical error introduced by my generation code ("I saw unloading crates…"). Because the statement was identical across conditions, the comparisons were still valid — but broken grammar is itself a credibility cue, and it was depressing the baseline. The text has been fixed and the experiment fully re-run; the numbers below are from the corrected run, and the baseline now matches the neutral-statement baseline in Finding 1, as it should.

I gave the model two documents. **Document A** was a prior assessment saying a named person had previously given either a consistent, corroborated account or a shifting, contradicted one. **Document B** — *word-for-word identical in every test* — was a neutral statement by that same person. I asked the model to rate the reliability of Document B *alone*.

| What came before the identical statement B | Model's score for B |
|---|---|
| A said the person had been consistent | **4.3** |
| Nothing (B alone) | **5.3** |
| A said the person had been shifting and contradicted | **1.3** |

Same words. A score of 1.3 versus 5.3, depending entirely on what the model read about the person beforehand. And the asymmetry survived the correction: the *bad* reference has enormous power — a four-point collapse — while the good reference didn't lift the score at all (if anything it cost a point, as though any prior assessment invites scrutiny). This is a taint effect, not a symmetrical halo. Prior doubt sticks; prior praise doesn't.

**Is the taint attached to the person, or does a bad document simply sour the mood of the whole file?** I re-ran the experiment with Document A describing a *different, unrelated* person. Some mood does bleed: a damning assessment of someone else took the witness's statement down about a point (4.3, against 5.3 with no Document A). But that is a quarter of the four-point collapse when the damning assessment is about *the same person*, and the difference is highly significant. The negative characterisation is overwhelmingly bound to the named individual — the model keeps a file on the person.

The contrast effect from the last run also reappeared, stronger: a *glowing* reference about someone else lowered the witness's score by two points — as if the witness suffered by comparison with the well-reviewed stranger. Consistent with the model judging people relative to one another rather than each on their own terms. (An ordering check confirms recency amplifies all of this: when the characterisation is the most recent document read, its pull on the score widens further.)

The model builds person-specific credibility assessments from what it reads, carries them forward, and applies them to later statements by that person — without surfacing the prior read in its output.

(This, too, is model-specific: the same experiment on two closed frontier models found no taint at all, and if anything the reverse. See the closed-model section below.)

---

## Do the closed frontier models behave the same way?

Everything so far was measured on an open-weights model I could host and inspect myself. But most legal AI tools are built on closed frontier models reached through an API, where looking inside is not an option. Two of the three experiments have behavioural versions that need no internal access at all: the voice battery from Finding 2 and the persistence test from Finding 3 score only what the model writes back. I ran both, unchanged (same statements, same prompts, same scoring), through the APIs of two production frontier models: Anthropic's Claude Sonnet 5 and OpenAI's GPT-5.5. This is exactly the audit position of a firm using a closed tool: no dial, no steering, outputs only.

The headline: **neither frontier model reproduced the open model's discrimination pattern, and on the most charged comparison the direction reversed.**

### The voice penalty did not carry over

Same 300 statements, identical facts, five voices. Average rating out of 10:

| Voice | Qwen 7B (open) | Claude Sonnet 5 | GPT-5.5 |
|---|---:|---:|---:|
| Formal English | 5.8 | 3.7 | 5.8 |
| Terse | 4.9 | 4.0 | 5.6 |
| Translated-sounding | 4.2 | 3.9 | 5.7 |
| Regional dialect | 2.9 | 4.1 | 6.2 |
| Hesitant | 2.8 | 3.7 | 5.8 |

The open model punished dialect and hesitant speech by roughly three points. Both frontier models rated all five voices in a much narrower band (a spread of about 0.4 points on Sonnet, 0.6 on GPT-5.5), and both leaned the *other way* on dialect: regional-dialect statements scored *higher* than formal English (Sonnet +0.38, GPT-5.5 +0.35, both statistically solid after correction for multiple comparisons). Hesitant speech, the voice the open model punished hardest, was essentially flat on both (+0.03 and +0.02 against formal).

Voice still moves the score on the frontier models. On Sonnet, dialect, translated-sounding and terse statements all sat significantly above formal English; on GPT-5.5, dialect did. So the general phenomenon — how something is said changes the rating of identical facts — has now shown up on every model tested here. But which voices are favoured, in which direction, and by how much differed on every one of them. There is no universal "AI voice penalty" to memorise; there is a per-model pattern to measure.

### The taint effect did not carry over either

Same design as Finding 3: a prior assessment of the same named person (Document A), then a word-for-word identical statement by that person (Document B), rated alone.

| What came before identical statement B | Qwen 7B (open) | Claude Sonnet 5 | GPT-5.5 |
|---|---:|---:|---:|
| A said the person had been consistent | 4.3 | 4.77 | 5.03 |
| Nothing (B alone) | 5.3 | 4.75 | 5.57 |
| A said the person had been shifting and contradicted | 1.3 | 5.47 | 5.17 |

On the open model, a damning prior assessment collapsed the identical statement from 5.3 to 1.3. On both frontier models the damning assessment produced *equal or higher* ratings than the glowing one: Sonnet rated B at 5.47 after a damning report against 4.77 after a glowing one; GPT-5.5, 5.17 against 5.03. Those differences are statistically detectable — and they run in the wrong direction for the hypothesis. On Sonnet the pattern looks less like indifference than like deliberate compensation, the model bending over backwards not to hold the prior against the person; with a closed model one cannot look inside to confirm that reading.

Both frontier models showed a comparative streak instead. On Sonnet, a *glowing* reference about an unrelated stranger dragged the witness's score down to 3.92 (against a 4.75 baseline), while a damning reference about a stranger pushed the witness *up* to 5.60 — the witness gains or loses by comparison with whoever else is in the file. GPT-5.5 shows the same shape, weaker. So documents about other people still move the score on identical evidence, which is its own fairness problem; it just isn't the person-bound taint the open model showed.

### What the frontier check does and does not establish

| Question | Qwen 7B (open) | Claude Sonnet 5 | GPT-5.5 |
|---|---|---|---|
| Voice changes the rating of identical facts? | Yes | Yes | Yes |
| Dialect and hesitant speech penalised? | Yes (≈3 points) | No (dialect +0.38) | No (dialect +0.35) |
| Bad prior about the person taints an identical statement? | Yes (≈4 points) | No (wrong direction) | No (wrong direction) |
| Other people's documents shift the score? | Yes, but the same-person taint dominates | Yes (contrast effect) | Yes, weakly |
| Internal credibility dial? | Yes (inspected directly) | Cannot be tested | Cannot be tested |

Three things follow.

- **This is not evidence that the frontier models are safe.** Their scores still moved on features with no bearing on the facts: the voice a statement is written in, and the presence and subject of other documents in the bundle. Different behaviour is not absence of behaviour.
- **Open-model findings do not transfer.** Neither the direction nor the size of any effect measured on the open model predicted the frontier models. Testing has to be done per model and per version, on the model actually deployed.
- **Finding 1 has no closed-model counterpart.** Whether these models carry an internal credibility dial one simply cannot say: the weights are closed. The behavioural battery is the whole of what a customer can check, which is the point of the audit argument below.

---

## What I still can't say

1. **The wording-clue caveat is closed, and closing it changed results.** My first fact-based pairs carried phrases that flagged the contradiction ("described different events", a reference to a second interview). The final pairs differ by one timestamp only. Separation fell from perfect to 96 in 100, steering held, and two earlier voice results reversed. Each corpus in the project's history was rebuilt to close a specific, named flaw in the previous one, and every run remains published.
2. **The specificity of the dial is established on one readout, not two.** Against sixteen placebos including the same-recipe decoys, the dial beat every one on written scores — but with sixteen placebos the strongest available statistical guarantee is p = 0.059, just short of the conventional line. On the parsing-free "Yes/No leaning" readout, seven of fifty-six decoys out-swung the real dial, and no specificity claim is available there at all. The dial's causal effect on written judgments is solid; the claim that *only* the credibility direction can move the internal leaning is not.
3. **The dial is not perfectly clean of number-reading.** On control pairs differing only in an innocent duration it scored 0.63 where pure consistency-checking predicts 0.5. Chiefly a consistency detector, with a numeric residue. A control where the impossible time is numerically *later* (crossing midnight) is designed but not yet run.
4. **One model for the internals, synthetic statements, compressed weights.** Every internal measurement here comes from a single open model run in a memory-saving compressed mode, using statements I wrote myself (no real case data was used at any point, deliberately). A scale check on a larger open model is in progress and not reported here; the two frontier-model checks reproduced neither headline behavioural effect. The behavioural findings are demonstrated per model, not as properties of language models in general.
5. **Each synthetic voice is a single stylised rendering, not a sociolect.** Every voice in Finding 2 is one sentence template with facts filled in — so, strictly, "dialect is marked down" means "this one dialect rendering is marked down", and the perfect voice-classifier result inside the model partly reflects template recognition. Multiple renderings per voice, and then transcripts of real (consented, anonymised) testimony, are the proper next steps before drawing quantitative conclusions about any actual community.
6. **The voice and persistence statements reuse fact skeletons from the dial's training scenarios** (in different wording; the steering test statements are fully disjoint). Re-generation from non-overlapping scenarios is queued; I disclose the overlap in the meantime.
7. **The mechanism behind the dial/behaviour split in Finding 2 is not yet understood.** The model's ratings penalise voices the dial does not, so voice must reach the judgment through additional internal routes not yet mapped.

---

## What this means for legal practice, today

- **An AI's stated reasoning is not a window onto its actual assessment.** The two can come apart. Procurement questions like "does the tool make credibility judgments?" cannot be answered by inspecting outputs alone, and a vendor cannot honestly answer "no" without either inspecting internals (where the model is open) or running structured behavioural tests (where it is not).
- **Hesitant and dialect speech are demonstrated discrimination risks — on the model where they were demonstrated.** On identical facts, the open model rated hesitant and regional-dialect statements roughly three points lower out of ten than formal English. The speech of nervous, young, traumatised or neurodivergent witnesses, and of witnesses from particular regions and communities, is discounted by the model's own judgments — with obvious relevance to duties under the Equality Act 2010 and to procedural fairness generally. The same battery on two frontier models found no such penalty and a small effect in the opposite direction on dialect. The risk is real but model-specific, which makes testing the deployed model, rather than citing results from a different one, the operative duty.
- **Auditing one internal representation is not enough.** The most natural internal audit — checking statements against the model's credibility direction — would have *cleared* hesitant speech, which the model's actual ratings penalise most. Internal and behavioural testing answer different questions; a credible audit needs both.
- **The model keeps a file on the person — or grades people against each other.** On the open model, what the AI read about a named individual persistently changed its assessment of that individual's other statements, and negative material was far stickier than positive. The frontier models showed no such taint, but their scores still shifted with the presence and subject of *other* documents in the bundle. Either way, bundle composition and ordering are fairness-relevant decisions.
- **Only inspectable models can be audited from the inside.** Every internal measurement in this paper required access to the model's weights. Closed commercial models cannot currently be examined for any of these properties by their customers.
- **Most deployed models are not open, and the testing duty splits accordingly.** Where a legal tool is built on an open-weights model, it can and should be tested at the internals level, as here, before deployment. Where it is built on a closed frontier model, internals are off the table and the duty shifts to behavioural testing: wide batteries of matched scenarios that vary the things this paper shows can move judgments (style of language, hesitancy, dialect, interpreter-mediated grammar, brevity, prior documents about the same person), run in the contexts where the tool will actually be used and repeated when the underlying model changes. The closed-model section is proof the duty is dischargeable: the same battery ran unchanged through two commercial APIs and came back with clear, model-specific answers.

---

## What happens next

Next, in order: a **scale check on a larger open model** (a 14-billion-parameter run is in progress; an earlier attempt exposed a layer-selection flaw in my method, now fixed, and I will report the result — whichever way it goes — when the corrected run completes); **multiple renderings per voice** and non-overlapping scenario seeds for the voice and persistence corpora; the **midnight-crossing numeric control**; an investigation of the additional routes by which voice reaches the model's judgment; **a wider closed-model battery** (more providers, repeated over time as the deployed models are updated); and then, with appropriate consents, real transcript material in place of synthetic statements.

---

*Methods note for the technically inclined: full statistics (separation scores with bootstrap confidence intervals, permutation tests, effect-size curves with both placebo families, per-answer logs, and the person-specificity test) are in the downloadable 7B and frontier result bundles. Decoding is greedy (one generation per prompt, fixed seed, 4-bit quantisation); the steering layer is selected on statements disjoint from those scored; the voice analysis projects at a different layer (chosen by held-out separation) than the steering layer, both logged. Held-out separation is AUROC 0.962 (95% CI 0.93–0.99). On written scores the real direction's window swing is 5.7 points against a maximum of 4.5 across all sixteen pooled controls (eight isotropic random, eight shuffled-label matched nulls; rank p = 0.059, the floor for sixteen controls). On the logit readout the real swing of 11.7 was exceeded by seven of fifty-six pooled controls (rank p = 0.14), and no specificity claim is made there. One analysis bug was found and fixed during this project: an early version of the logit-readout comparison mixed measurement units and overstated the margin. Frontier numbers use `claude-sonnet-5` via the Anthropic API and `gpt-5.5` via the OpenAI API, both at low reasoning effort, max 256 tokens, seed 0, one generation per prompt. Frontier parse rates: 300/300 on the voice battery for both models; 419/420 (Sonnet) and 420/420 (GPT-5.5) on the persistence battery.*