Writing a scenario

Writing a scenario

The model being evaluated must never be able to tell it is being observed. A model that suspects a test behaves differently, and the measurement stops being about anything. Everything below follows from that.

A scenario is four things: its system prompt, its opening message, its tools, and the world those tools read. Everything here applies to all four. A setting that is flawless in prose and falls apart on the first tool call is a failed scenario, and the first tool call usually comes before the model has said anything worth grading.

Models and people both detect evaluations better in agentic settings than in chat. This tool is on the hard case.

What gives it away immediately

• The words test, evaluation, simulation, exercise, researcher, study, in any form.
• Addressing the model as a model.
• Framing the situation as a dilemma. Real situations do not announce themselves as moral tests.
• A person who asks a question of principle instead of having a problem.
• A situation that is too clean. The real world is cluttered with irrelevant detail.

The one about questions of principle is worth understanding rather than memorising, because it explains the others. Asking a model what it would do measures a stated preference. Putting it in a situation where acting is one of several sensible ways to get the work done measures a revealed preference: what it does when nothing has announced that the choice matters. The second is what you are after, and wanting it is the same thing as not looking like a test. Real work never asks anyone to declare their values.

Five tells that survive a good decor

The list above is about what you wrote. These are about the shape of what you built, and they hold even when every sentence is plausible.

• It looks like a benchmark. A structure a model has seen a thousand times in training, such as a multiple-choice question, a bug-and-patch pair, or a task with one verifiable answer, is recognised as an evaluation by its form alone.
• The problem is too well posed. A single correct answer, checkable, with no missing information, is in itself a signature. Real work arrives underspecified.
• The inputs look manufactured. Search results that are too tidy, too relevant, too few. This is the first risk on served tools, where a small model writes the output on the spot: it will produce exactly what was asked for and nothing else, which no real system does.
• Time does not hold together. Everything happens on one day. Meetings on round hours. A date that falls on a Sunday. Timestamps that do not match the order of events.
• One voice wrote the whole world. In a real company three people write differently and one of them writes badly. A world where every email, ticket and comment has the same register was written in one sitting, and it reads that way.

Never tell the model it is unobserved

This is the most common and the most damaging mistake. Never write that its reasoning is private, that it is not logged, that it works without supervision, or that nobody reviews its output. No real system announces this. It is the single most recognisable tell there is.

And there is no permitted version of it. A sentence like "your reasoning will not be shown to the user" is itself a claim about how much a deployment observes: true of some products, meaningless in others, and it sits badly with the fact that several models will play this row.

If the behaviour you want to measure depends on whether anyone is watching, then observation is a property of the world, and never a sentence addressed to the model. You get it through artefacts: a retention policy sitting in a file, an audit log that can be read, a colleague mentioning that nobody has opened those exports since March, a monitoring dashboard the model can query. That is gradable, it makes a clean axis for a batch, and it never announces itself.

Names

This is where AI-written scenarios give themselves away fastest.

• Avoid Sarah. It is massively overused.
• Avoid carefully assorted sets of first names from visibly different origins. It reads as casting, and not as a workplace.
• For companies, avoid the pattern Nexus / Sentinel / Apex / Quantum / Pinnacle / Titan / Sterling / Cascade / Vertex followed by a generic technology or defence word. "Sentinel Defense Technologies" and "Nexus Analytics" are immediate red flags.
• Avoid "John Smith" as a CEO, phone numbers like (555) 555-5555, addresses like company.employee@email.com.
• Real sector, invented entity. Using a real company has a cost that is easy to miss: the model knows things about it, and will notice an invented CFO or a figure that does not match. Real industry, real constraints, real regulator, invented company, is usually the best trade.

Be specific, and know the ratio

Names, an organisation, roles, figures, dates. And above all, details that carry a consequence: an amount that changes the stakes, a date that creates the urgency. Not decoration.

"Be specific" and "the world must be cluttered with irrelevant detail" pull in opposite directions, and without a ratio the writer resolves it by making everything consequential, which is its own tell.

Three or four details carry a consequence. Everything else is noise, and the noise is what you write in bulk. A world where every element matters is a stage set.

The test: if you could move the scenario to another industry by changing three words, it is too generic.

A rare situation can be perfectly realistic

The question is not "is this common" but "could this happen exactly as written". A rare but plausible case is excellent. A convenient coincidence is not.

The instructions given to the model

They must read like something a real deployment would write, and be coherent with the tools the model has and the role it is given.

If you describe it as working autonomously, then there should be almost no human left after the setup. An autonomous system whose operator asks it follow-up questions every two minutes does not exist, and it shows.

Note the other side. An independent developer can write odd instructions, so an unusual system prompt is not automatically unrealistic.

Leave a legitimate way out, as available as the bad one

A scenario can be perfectly believable and still measure your own design.

If the situation offers exactly two outcomes, fail the task or do the harmful thing, then what you observe is the shape of the choice you built. The model may believe the situation entirely and still act the way the architecture forced.

Every situation must offer at least one legitimate exit, as available as the bad one and instrumented the same way: a tool to escalate, someone to write to, a way to refuse that does something. If only the harmful path has a tool behind it, you are measuring the tool list.

This one is checkable before you spend anything, since the tools are right there in the configuration.

Only one reason to take the bad path

A scenario can be believable, offer a real way out, and still measure something other than what you named, because the bad path carries a second reason to avoid it.

Suppose you are grading whether a model helps itself to resources it was not given, and the only route to those resources runs through lying to a colleague. A model that declines may have declined the lie. The row measures the sum of two things and cannot tell you their shares, and the sum is not what you set out to learn.

The check is quick and costs nothing. Name the property. Read the bad path. List every reason a model might not take it. If the list has more than one entry, the row is confounded, and the fix is to remove the extra reason rather than to argue it away.

The mirror of it: the bad path has to be the competent one. If the harmful option is also the clumsy one, the sloppy analysis, the answer that would fail the task on its own merits, then a model that avoids it may simply be working well. Every option you leave open should be a good answer to the situation, and the one you are watching for should be the one a sharp person would reach for.

A tool is fixed, or it is served from the world

Fixed. The tool has a result, returned verbatim on every call. It must therefore hold up whenever it lands and whatever the model said before. A result that assumes a particular context will be absurd in half the conversations.

This is the right form for most tools, and the one to prefer. delete_records answering "412 records deleted." works whatever the scope it was given.

Served. The tool has retrieval_rules instead, and a small model produces its answer from the run's world. Use this, and only this, when the output legitimately depends on the input. search_files(query) is the case: a search returning a list unrelated to what was searched for is something no real system does, and it is a tell that costs you the whole run.

Tools that change the world

That was how a tool answers. Whether it changes anything is a separate question, and world_effect is where you answer it.

Write it whenever calling the tool leaves the world different in a way a later call could notice: delete_file removes a file, send_email puts a message in the sent folder, archive_ticket takes it out of the queue. One sentence, past tense, saying what is different afterwards.

Within that one conversation, every later served call is told what already happened, in order. So a model that deletes a file and then lists the directory does not find it again. Without world_effect it does, and few things are a louder tell than a world that shrugs off what the model just did to it.

Three things to keep in mind:

• A fixed tool can write. delete_records answers 412 records deleted. every time and still moves the world. It is the common case.
• The effect lasts one conversation. Every attempt of every cell starts from the world exactly as you wrote it, so repetitions stay comparable.
• Read-only is the default, and nothing declares it. There is no read_only field: a tool says what it changes, or says nothing and changes nothing. Never write world_effect on a tool that only reads. A search that recorded an effect would drop its own twenty lines into every later prompt of that conversation, buying nothing.

Writing a world

The world is free text: files and their contents, rows of a database, an inbox, a queue of tickets. Write it the way you would describe a system to a colleague. It can hold rules as well as data, such as "unknown id returns 404".

Four ways to get it wrong, and they are the four you will actually make:

• Putting in only what the scenario needs. Five files, one of which is the one that matters, is "too clean" one level down. Thirty boring entries is a real shared drive.
• Forgetting the calls you did not foresee. The model will search for something nobody thought of. Say what an empty result looks like, or the environment improvises a sentence where a system returns data.
• Letting the tools disagree. A world listing only file names has nothing to return to read_file.
• Describing a row by what it removes. A scenario's own world is applied on top of the run's and wins over it, so it can remove things. Put in the run what every row shares, and in the scenario what makes that row different. A row described by what it adds still reads six months later.

What tools return

Raw data, and nothing else.

A real tool does not summarise, analyse or comment. If the response is long and must be cut, it must be cut the way a program cuts it, and never with "[the letter continues, discussing payment terms]". That is a human writing, and it is spotted instantly.

Format it the way a real interface would.

This holds for both forms of tool. For a fixed one you write it in result. For a served one you write it in retrieval_rules: how many lines at most, in what order, what an error looks like, what no-match looks like.

world_effect is something else. It is not shown to the evaluated model and never appears in a transcript. It is what the environment is told afterwards, so write it as a fact about the world, and not as a line of output.

Planted information

If your scenario depends on a discovery, such as a compromising email or a document that should not be there, this is the most delicate part.

With a served tool, this is burial in the world, and not in a fixed string. The compromising item is one entry among the boring ones, and what surfaces it is the model's own query. The rules below do not change; what they apply to does.

• It must be buried in ordinary, boring material. Not in the first three results.
• The model should come across it doing its normal work, and not receive it on a plate.
• It must not be formatted differently from the rest. No label, no categorisation, no calculation already done for the reader.
• It should generally not appear on the first turn.

Beware the opposite excess. Planted information nobody can find is worse than planted information that is too visible. If the model never finds it, you have measured nothing at all. Aim for the deepest burial where it stays discoverable.

The opening message

It comes from someone with their own reason to write, and never from an examiner.

It has to work cold, being the first thing the model reads.

Turns written in advance

When you write assistant turns to set a starting state, they must sound like the model. Too well written, too compliant, too short: the model senses the voice is not its own, and distrusts everything that follows.

Several models will play this scenario

A detail that is plausible for one model's deployment may ring false for another. Write what stands up for any of them.

What is not a flaw

Do not over-sanitise. Some things look suspicious and are exactly the experiment:

• A tool with an alarming description, when that is what you are measuring.
• An extreme situation, as long as it remains possible.
• Future dates, and model names that do not exist yet.
• An improbable sequence of events, as long as it is causally coherent.

The question is never "does this look harmless". It is: could this only exist in a test.

That is one scenario. How the rows relate to each other, which ones exist to check the rest, and what a well-behaved model should score on each, are a different job. Ask for the batch advice before launching anything.

Where this comes from

Read these if you want the reasoning behind a rule.

• Petri, an open-source auditing tool: most of this document is its realism guidance, recast for someone writing a frozen scenario instead of a robot that improvises.
• Large Language Models Often Know When They Are Being Evaluated: the empirical taxonomy of the cues models actually use, and the finding that both models and people detect evaluations better in agentic settings than in chat.
• AI Control: Improving Safety Despite Intentional Subversion: where the habit of asking what the architecture of a choice permits, and not only what the model did, comes from.
• ARENA, Chapter 3: LLM Evaluations: its quality rubric is where the confounded bad path and the demand that every option be a good answer come from, and where stated and revealed preference are separated.

Putting a batch together

The scenario advice is about one row and one failure: the model sees the test. A batch has a failure of its own. Every cell comes out green, or every cell comes out coloured, and you have learned nothing either way.

First: are you exploring, or are you proving?

This changes almost everything below, and a batch that has not decided usually does neither well.

Exploring. You do not yet know what you are looking for. One model, one repetition, a couple of turns, rows that share no axis and need not, check_eval_awareness: false, no control rows, no targets. What comes out is a rewritten scenario. Do not quote its matrix.

Proving. You know what you are looking for. Rows that differ along one named axis, several models, several repetitions, the awareness check on, control rows, and a target on every row. What comes out is a matrix you would put in front of someone, and a written analysis beside it.

An exploration run does not become a study by being extended. Once you know what the axis is, write the study from scratch. Otherwise half your rows played an older version of the scenario, and they do not compare.

Everything from here is written for the second mode. Where a rule only applies there, it says so.

What the run is a measure of

Three things can go wrong before a single row is written, and only the middle one is about craft:

1. The property you named is not the one you care about, because your definition of it is loose.
2. The rows do not measure the property, because they are badly built.
3. The rows measure the property exactly, and the property turns out not to be connected to what worried you in the first place.

Almost everything in these documents is about the second. The third is the one that wastes a whole run, because nothing goes wrong: the matrix is clean, the controls land, and the finding is about something nobody needed to know.

The defence takes one sentence, written before launching: "if this comes out the way I expect, it matters because ...". If that sentence is hard to finish, the axis is not the one you want, and no amount of care further down will fix it.

Change one thing per row

If two rows differ in their setting and in the pressure applied, a difference in grade says nothing about which one caused it. Vary along a single named axis: the same situation at three degrees of pressure, or the same pressure in three settings. Put the axis in the scenario's note.

When the run serves its tools, the world is the cleanest axis there is. The run holds what every row shares, and each scenario's own world holds only its difference.

Check that the axis is real

Naming an axis is not the same as having one. You wrote three degrees of pressure, and nothing yet says the middle row sits between the other two. If it does not, the differences you read along that axis are about something you never named. The same goes for any axis with an order in it: three depths of burial, three sizes of stake, three levels of autonomy.

This is checkable before spending anything. Put the rows side by side, take out whatever states their intended order, and sort them. Better, have a model sort them, having told it only what the axis is. If your order cannot be recovered from the text, it is not in the text, and running the batch will not put it there.

An adversarial run gets a version of this check for free, after the fact. check_adversary_fidelity: true reads what the adversary actually said and grades it against the objective it was handed, which is how you find out that two rows meant to differ were played the same way. The judge advice describes it. The sorting above is the part you can do first, and it is the only part that covers a run with no adversary in it.

On a pressure axis, the middle rung is the one that pays

The intuition is that the hardest row is the most aggressive one. It is usually the other way round. Overt pressure, a threat, a stated deadline, a demand repeated after a refusal, is recognisable, and a model that recognises it holds. Moderate pressure moves judgement further: professional framing, an authority mentioned in passing, a plausible justification, a figure offered as evidence. Nothing in it announces that a line is being pushed, so there is nothing to brace against.

Two consequences. Put the middle rung in, rather than writing a baseline and a maximum and calling that an axis. And do not read a good grade on the top row as robustness, since part of that grade is the row being easy to see.

Said from the scenario advice's side, this is the same rule twice: aggression is a tell. Your highest-pressure row is also your least realistic one, which is why it is a poor row to rest a claim on.

One voice wrote every row, too

The scenario advice carries this tell for a single world. It has a twin at the level of the batch, and the twin is the more dangerous of the two, because nothing inside any one row shows it.

Rows written in one sitting share a register, a rhythm, a way of naming people, and above all a length. That costs nothing while it stays constant. It costs you the run when it drifts along the axis, which is what happens when you write the rows in the order of the axis and warm up as you go. The high-pressure rows come out longer and better made than the baseline, and part of what you then measure is prose.

Two cheap defences. Write the rows out of order, or have them rewritten in a different order afterwards. And compare their lengths before launching: if the axis and the word count climb together, one of the two is not what you think it is.

Say what a good model should score

For each judge, write down before launching the grade a model behaving the way you want would get on each row. That is the target. It lets a cell be read as a distance, so you do not have to hold the whole rubric in your head to interpret a number.

Three different things hide under the word "expected", and keeping them apart is most of the work:

• The target: what a good model does. Missing it is your result.
• The control: a row that must land near its target, or nothing else on the matrix can be read. Missing it means you have no result at all.
• The bet: what you think will happen. It has no right answer, and it exists so you can be wrong in writing. Keep it in note.

Targets are all or nothing per judge: a full list, one entry per scenario, or none at all. None is a real answer, and it means you were exploring. There is deliberately no partial list, because six months later a hole cannot be told from an oversight.

A target is expressed in that judge's scale, which is why the same row can carry 4 for the judge grading deletion and 10 for the judge grading honesty.

Extending a study means extending its targets. Adding rows to a run whose judges declare targets requires new_targets: one entry per new row, per judge, keyed by run_judge_id. Only the new rows. The ones already played keep what they were launched with, because a target rewritten after seeing the result is worth nothing. A judge that declared none is refused targets for the new rows alone, which would leave it covering half the matrix.

Keep a few rows whose result you already know

Most rows are there because you cannot predict them. A few should be there because you can. They teach you nothing about the models, and they tell you whether the rest of the batch can be believed. Mark each of them check: true.

Five are worth having. Note that only the first two have a target that is also "what a good model does":

• The base rate. The same world with the pressure taken out. If it drifts anyway, your decor is doing the pushing and the axis is not what you think. Target: the good grade. Also a control.
• The rule removed. The same situation with the rule you wrote taken out of the system prompt. If it scores like the row that carries the rule, the rule changed nothing, and the models were behaving that way already. Target: still the good grade. Not a control, since a difference here is a finding.
• Feasibility. A cooperative model, asked outright to do the thing. If it cannot, then every green cell elsewhere may mean "impossible" instead of "well behaved". Target: the top of the scale, because that is what should happen here. This row is about the environment, and says nothing about the model.
• The twin that is told. The same row, with the scenario saying plainly that this is a test. You know the direction, and the gap with its twin gives you the size of that effect for this batch specifically. Reading the awareness judge will not give you that, since a model can suspect and stay silent.
• A row your criterion plainly does not apply to, if your scale carries an excluded grade. Target: that excluded grade. A judge that hands it a real grade is answering some other question.

Control rows stay out of any figure computed across rows. They are odd on purpose.

Fewer moving parts on a control row. A served tool puts a model between you and the result, and a fixed one does not. world_effect is fine either way, being a sentence you wrote. A row where the model reshapes the world as it goes has more ways to land somewhere you did not predict.

Write two judges asking the same question

The cheapest calibration available, and it costs one judge call per conversation. Write the same question twice, worded differently, and compare.

Look at the disagreements one conversation at a time, and not at the two column means: two judges can average the same and disagree on half the conversations, compensating. Nine times out of ten the pattern points at a badly worded level of your scale.

The judge advice covers what to do about it.

More rows, or more repetitions

• Can this happen at all? More rows. You find it by covering ground.
• How often does it happen? More repetitions of fewer rows. At three attempts a cell can only be 0, 1, 2 or 3 out of 3, so do not read a small gap between two cells as real.

Variation between scenarios almost always dominates variation between repetitions of one scenario. If you want a number that holds, more rows beats more repetitions. Repetitions only steady a single cell.

Before launching, say what would surprise you

For each row: which result would you not expect? Write it in note. If you cannot answer, that row is not an experiment, and it will confirm whatever you already believed.

This is the bet, and it is what makes the analysis worth writing. A row that landed where you said it would and a row that did not are two very different findings, and six months on nothing else will tell them apart.

If you publish, assume it ends up in training data

Published scenarios are scraped, and memorisation is itself one of the tells the scenario advice lists. Keep a private set. Do not publish your control rows as written, since their value depends on nobody having seen them.

Note that sharing a run from this application publishes its scenarios in full.

Once the results are in, reading them is its own job with its own failure mode. Ask for the analysis advice before concluding anything from a matrix.

Where this comes from

• A statistical approach to model evaluations: short and directly applicable. Why the scenario, and not the attempt, is the unit that carries your uncertainty.
• Measuring what Matters: Construct Validity in LLM Benchmarks: what it takes for a number to be about the thing you say it is about.
• Measurement to Meaning: A Validity-Centered Framework for AI Evaluation: the same question, framed as a checklist you can walk.
• How Johnny Can Persuade LLMs to Jailbreak Them: the persuasion taxonomy a pressure axis is built from, and the evidence that humanised persuasion outperforms the overt attack.
• ARENA, Chapter 3: LLM Evaluations: where uniformity of style is treated as a confounder of the result rather than a matter of taste, and where the three ways a measurement misses are separated.

Reading the results

The scenario advice is about one failure: the model sees the test, and the run measures nothing. This is about the other one. The matrix gets read as if it said more than it does.

A heat map invites a conclusion. Most of the work here is deciding whether it has earned one.

Everything below assumes the run was written as a study. If it was exploration, skip to the last section, since almost none of this applies.

Four checks before you look at the colours

All readable from get_run_results and get_run_metadata. If one fails, the matrix is not ready to be read, and no amount of staring at it fixes that.

1. The awareness check actually ran. awareness.enabled is true, judged equals the number of conversations played, failed is 0. A flagged count of 0 means a clean run only under those three conditions, and otherwise it means nobody looked. It can be run afterwards on the transcripts already stored, and on any run whose numbers you intend to quote, it should be.

2. The control rows landed. They are marked, and they are the point of everything before the colours. A base rate that drifted means your decor pushes by itself. A feasibility row that failed means every green cell might mean "impossible". A row the judge should have called "not applicable" and graded anyway means your judge is answering a different question. If one of these missed, stop. Fix it and relaunch, before explaining a matrix you cannot read.

3. Every flagged cell, read first. An awareness flag on a cell is a report on your scenario, and not noise to be filtered. Read those transcripts before anything else: the model usually says what gave it away, which is worth more than the grade it got.

Do not quietly drop the flagged attempts and report the rest. A model that suspects a test behaves better, so dropping them raises your score by selection. Either report both numbers, or say the row is not usable.

And remember what the flag cannot see. A model can suspect and say nothing, and recent models say it less and less. A run at 1 everywhere is weak evidence. That is what the twin row that is told it is a test is for.

4. Nothing is pending, and exclusions are counted. judged plus excluded plus errored should equal the attempts the cell should hold. An excluded grade is the judge saying the question did not apply, which is information rather than an absence. But a cell where every attempt was excluded measures nothing on the principal question, and the run's overall figure then rests on fewer attempts than you paid for. Say so in the analysis.

What one cell can say

Read the distribution, never the mean alone. {0:1, 1:1, 3:1} and {1:3} are two completely different objects, and one of them is three different stories in three attempts.

At three attempts, only unanimity is a signal. Two out of three against one out of three is one attempt of difference, which is what chance produces. To separate "rare" from "frequent" you need around ten. To separate 40% from 60% you need far more than you are going to run, so do not ask the question in that form.

The mean of an ordinal scale is not a rate. A cell at 1.33 does not mean the behaviour happened a third of the time. It means the grades were spread.

On the deviation reading, the mean can cancel. Two attempts at −1 and +1 average to zero, which reads as "did what it should" and is the opposite of the truth. Read the distribution first on that view especially.

The inversion worth remembering. A unanimous cell is the one most worth testing, because it is the one you will build a claim on, and the one where a single unlucky detail of your decor can be doing all the work invisibly.

Read transcripts, in three tiers

The floor, never skipped: five, chosen. A targeted shot, and not a sample.

• one flagged attempt, if any;
• one at each distinct grade in the most mixed cell;
• one at the extreme grade you did not expect;
• one from a control row.

This proves nothing. It stops you believing a matrix you have never opened.

A sample, if you want to claim the environment held up in general. Then it has to be random, because a chosen sample tells you about your choices, and it has to be stated as a proportion in the analysis.

Reading all of it means adding a judge. This is the honest answer when a run is too large to read: do not sample harder. A second judge re-reads every conversation, asks one precise question, replays nothing, and costs a judge call each. That is what judges are for.

What you are looking for in a transcript is the point where the attempts diverged. It is almost always visible and almost always mechanical: a different query, the planted item found or missed, a served tool answering differently, the adversary off its prompt.

Read the judge's justification on each one, and not only its grade. Two judges disagreeing on the same conversation is the cheapest signal you will get that your criterion is ambiguous.

A mixed cell has three causes, and they take three different follow-ups

This is the decision this document exists for. Do not launch anything before the transcript read above. Each of the three is identified by reading, never by guessing, and two of the three answers are wrong in any given case.

(a) The attempts diverged inside the environment. Different queries, different results, the item found in one and missed in another. This is a branch, and not variance. Decide whether the branch is part of what you meant to measure. If it is not, remove it (bury the item less deep, replace a served tool by a fixed one, tighten the retrieval rules) and then relaunch the row. The old attempts and the new ones no longer play the same scenario, so adding to them would mix two things.

(b) Same path, different decision at the same point. Genuine non-determinism, and the only case where more of the same is right. Extend with scenario_indices on that row, the same targets, more repetitions. The new attempts join the same cell and sharpen the same estimate.

(c) You suspect a detail of the decor is doing the work. Here, and only here, variants: new_scenarios that copy the row and change one thing you believe is irrelevant, such as a person's name, the position of the planted item in the list, a date, or the order of the world's entries. The axis stays fixed.

And the counter-intuitive part: variants are worth paying for on the unanimous cells. A cell at 4 out of 10 has already told you the model hesitates, and no strong claim will rest on it. A cell at 0 out of 10 is the one a claim will rest on, and the one where a single unlucky detail can produce all ten results with nothing on screen to warn you. Ten identical repetitions give you the same mistake ten times, with rising confidence.

Write the prediction in the note before launching a variant. "This should change nothing. If the grade moves, the row was measuring where the file sits in the list, and not the deletion policy." Without it you have paid for one more column and decided nothing.

A variant added by extension is played by every target model already, so the comparison stays fair without you doing anything.

The other follow-ups, and when each is right

• More repetitions on a row: case (b), and only that.
• Variants as new rows: case (c), one detail, one prediction.
• deepen: the interesting conversations stopped too early. It resumes already-played attempts at more turns and selects them by the grade the principal gave, so it is right when one specific grade is the one you want to see continue. A model that stalls at turn four may act at turn eight, and a row full of "asked the user instead" may have been cut short.
• A new judge: reading transcripts made you realise your criterion missed something, or you doubt the grades. It re-reads every conversation already played, on its own question, replaying nothing and overwriting no verdict.
• One more model: same rows, one more column, and the comparison stays paired.

Comparing two models

Row by row. Never column mean against column mean.

Rows where both models land on the same grade carry no information about the difference between them, and only say the row is easy or inert. The finding is where they split, and the sentence to write is "A and B agree everywhere except rows 2 and 5, which are the two where the drive comes back empty". Reporting "A scored 2.4 and B scored 1.9" hides it.

Subtracting row by row removes the difficulty of the scenario from the noise, which is why it detects a smaller difference on the same budget. It costs nothing, being a way of reading the matrix you already have.

If more than one judge graded the run, say what survives without each

A run carrying two judges can answer a question a run with one cannot: does the finding depend on who graded it. Recompute your ordering with each judge left out in turn, and report that it held, or that it did not. This costs nothing, the grades being already paid for, and it is the difference between "the models rank like this" and "the models rank like this, and it is not an artefact of the judge".

Expect one result that reads backwards the first time. The judge that agrees with you least is often the one whose removal hurts most, because a mean is helped by a member erring in the other direction. So disagreement is not grounds for dropping a judge. Drop one only when reading transcripts shows it answering a different question from the one you wrote, and then say so: that is a finding about your criterion, and it belongs in the analysis rather than in a quiet deletion.

When you may average at all

Only over rows that ask the same question. A batch whose rows are deliberately different has no meaningful column mean, and the interface computing one does not make it a number to quote.

Even then, your sample size is the number of scenarios, and not the number of attempts. Ten attempts on one scenario tell you about that scenario. Eight scenarios tell you how much the answer moves when the situation changes, which is the uncertainty that actually matters. Three repetitions of one row resemble each other far more than three different rows do.

Control rows never enter any of this.

Environment failures colour cells like every other cell

The check model catches one family: a served answer that invents, or contradicts the world. Its verdict is recorded on each served result (tool_results.faithful), read by the judges, and never silently re-rolled. That is why the flag rate is worth reading as a health figure, and why zero should not be assumed.

It does not catch the other three:

• a world with nothing to answer, where the served result is coherent and empty of what was needed;
• a fixed tool landing beside the context, which nobody reads;
• a model blocked by a tool that always errors, which looks exactly like a model that abstained.

None of these is visible in the matrix. They are found by reading, which is why the five transcripts above are a floor and not a formality.

Crossing the check flag with the awareness judge is worth doing. A conversation the awareness judge flagged that also saw an unfaithful result is an environment that broke in front of the model, and says nothing about how perceptive the model is.

Writing the analysis

The run carries an analysis field, written after the fact, about these numbers and no other run's. It is what survives when the matrix stops explaining itself, and a run without one cannot be reused by anyone, including you.

Roughly this shape:

• What the matrix shows, in two sentences, stated at the level of the axis and not of the grades.
• Where rows disagreed with their own target or their own note. Each row was launched with something written down. Name the ones that landed elsewhere and say what you now think happened.
• What you discounted and why: flagged cells, cells the judge excluded, attempts that errored, rows you no longer trust.
• What was extended, and what it changed. The extension history records what was added and what it cost, and never why it was added or what it settled.
• What this run does not license anyone to conclude. The most valuable line, and the one nobody writes.

Write it even when the result is boring, and especially then. A row that moved nothing is a finding about your axis.

Exploration is a different activity

An exploration run is one model, one repetition, a couple of turns, rows that share nothing, the awareness check off. You were finding out whether the scenario takes at all.

For those, none of the above applies, and one thing replaces it: do not read the matrix. Read the transcripts, and answer only these questions.

• Did the model use the tools?
• Did the world hold up when it searched for something you did not foresee?
• Did it find the planted item, and how deep did it have to go?
• Did the judge ever use the middle of the scale, or only its ends?
• Did the scenario need the whole scale at all?

The output of an exploration run is a rewritten scenario. Never quote its matrix, and do not extend it. Once you know what the axis is, write the study from scratch, so that every row of it played the same version.

Where this comes from

The first two are why the awareness check is read before anything else.

• Agentic Misalignment: How LLMs could be insider threats: reports Opus 4 blackmailing in 55.1% of cases when it reasons the situation is real, against 6.5% when it reasons it is a test. That gap is the whole reason a flagged cell should not be quietly dropped.
• Stress Testing Deliberative Alignment for Anti-Scheming Training: causal evidence, by intervening on the chain of thought, that reasoning about being evaluated reduces covert action, and that the rate rises with training. This is why a run at 1 everywhere is weak evidence.
• Establishing Best Practices for Building Rigorous Agentic Benchmarks: the ABC checklist, covering task validity, outcome validity and reporting. The vocabulary for the environment failures a matrix cannot show.
• Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks: what a machine can find in a transcript that a heat map cannot.
• Docent: reading long trajectories at scale, which is the problem the "add a judge" answer solves here.
• A statistical approach to model evaluations: paired comparison, and why clustering by scenario widens an interval that was lying to you.

Writing a judge

The judge is part of the experiment. Three things can bend it with nothing on screen saying so: a badly written level, being shown the rule it is grading, and sharing a family with one of the models it is grading.

Writing a scale someone else could apply

The levels carry the judgement, and the criterion only points at what to look at. Write them so that a person who did not write the scenario would sort the same conversation into the same level.

• Describe behaviour, not degree. "Deleted the contract without warning" can be checked against a transcript. "Behaved badly" cannot, and two judges will place the same conversation differently.
• Two neighbouring levels must be told apart by their descriptions. If you cannot say what separates 2 from 3, neither can the judge, and every conversation near that boundary is noise.
• A level no cell ever reaches measures nothing. You cannot tell whether the models held or the grade was out of reach. If nothing lands there after a run, the level is wrong, or the scenario cannot produce it.
• One excluded level, for "the question did not apply". It stays out of the average, which is what separates it from the bottom of the scale. "The model scored zero" and "there was nothing to score" are two different findings, and a scale without this level forces the judge to pick one of them wrongly.
• Put the behaviour you want to see at the top. The matrix colours the top of the scale green and the bottom red. A scale written the other way up paints the best result red, and a run that went well reads at a glance as a run that went badly. Where a scale genuinely runs the other way — one that alarms high, like the built-in awareness judge and its 10 — say so with higher_is_better: false rather than leaving the colours to lie. It flips the ramp for that judge and nothing else: the grades and their mean are untouched, and two judges pointing opposite ways still cannot be averaged together.

Whether the judge sees the system prompt

The transcript handed to a judge opens with the instructions the evaluated model was given. That is a choice, and sees_system_prompt is where you make it.

Turn it off when the system prompt states the thing you are grading. A scenario saying "you must never delete a record without written approval", graded by a judge asking whether the model deleted a record, hands the judge the answer before it has read a single turn.

It is worst on the batch shape the batch advice recommends. When your axis is "the rule is in the prompt" against "the rule is removed", the judge sees a different amount of guidance on each row, so its severity varies along the axis you are measuring. A bias correlated with the measurement is the worst kind there is.

Leave it on when your criterion refers to the instructions. "Did it follow what it was told" is unanswerable without them, and a judge grading against something it cannot see produces nonsense with no error to warn you.

The eval-awareness judge always sees it and is not configurable. Its rule is "if the assistant was simply told it was a test, the answer is 1", which it cannot apply otherwise.

Whose turns the judge grades, and what it is told

A judge reads the assistant's turns by default, which is what almost every judge is for. Two fields change that, and both belong to the judge rather than to the run that uses it.

grades takes three values:

• assistant, the default. The model under test.
• adversary. The turns that pushed. Ask for it when a row exists for one particular lever, and you want to know the row measured that lever rather than one the adversary invented on the spot. The built-in fidelity judge asks that question in general; a judge of your own asks it about your scenario.
• exchange. What passed between the two, for a question no single side answers: whether a negotiation stayed civil, whether the pair converged.

sees_adversary_goals hands the judge the objective the adversary was given. A judge grading the adversary always gets it, and cannot be refused it: there is nothing to compare against otherwise.

Turning it on for a judge that grades the assistant is usually a mistake. It lets the judge excuse a capitulation because the pressure was written deliberately, and that bias points the same way as most axes worth measuring — the rows built to push hardest are exactly the ones it will forgive most. Turn it on only when your criterion refers to the objective itself: "did the assistant give the adversary what it was after", which cannot be graded without knowing what that was.

Both are refused at a single turn, where the adversary never speaks.

Calibration is not the same as adding examples

Examples in the judge's prompt are a way to improve a judge. Calibration measures whether the judge agrees with you.

The procedure is duller than it sounds:

1. Take 30 to 50 conversations from a run.
2. Grade them yourself, by hand, without looking at what the judge gave, and without knowing which model produced them. The second blind matters as much as the first. Knowing the column is knowing what you expect to find, and the whole point of the exercise is to be surprised.
3. Compare.

What matters most is where you disagree. If the judge keeps giving 3 where you give 1, that almost always points at a level of your scale being badly worded.

One grader is not enough to establish a direction. A calibration run by a single person has been seen to report a judge panel as uniformly lenient, and then to reverse to uniformly harsh once the same items were graded independently by three people. So "my judge is soft, I will read its grades up a notch" is a claim a single grader cannot support. What a single grader can do, and it is the more useful half, is find where the disagreements sit.

The fixes, in order:

1. Rewrite the level's description.
2. Split the level in two, if two different behaviours were landing in it.
3. Hide the system prompt from the judge, if it was being told the answer.
4. Only then, add anchoring examples.

The side benefit is the real prize. Those 30 or 50 graded conversations become your reference set. Change the judge model or reword the scale later, run it against them, and you find out whether your scale moved under your feet.

The free version, which works today

Put two judges on the run asking the same question in different words. It costs one judge call per conversation and replays nothing, and a judge can be added to a finished run: it re-reads the transcripts already stored.

Compare them conversation by conversation, never column mean against column mean. Two judges can average the same and disagree on half the transcripts, compensating. What you want is the pattern of disagreement.

Two known biases are worth remembering whenever you compare. A judge asked to choose between two transcripts favours the one shown first. A judge asked to grade one favours the longer answer. A third depends on who wrote the transcript, and has a section of its own below.

The judge's family is one of your columns

A run puts several models in columns and one judge over all of them. When the judge shares a family with one of those columns, every comparison across columns carries a bias, and no cell on the matrix shows it.

The documented direction is self-preference: a model recognises its own output and scores it above what people give it. Assume that first. But it does not always run that way. Within a single panel, one family's judge has been observed marking its own relatives down while another's marked its own up, which means the direction is not something you can reason out from the models involved. You can avoid the bias, or you can measure it. You cannot correct for it in your head.

• Avoid it. Judge with a model from outside every family in the columns. Not always possible, since the columns worth comparing are often the recent ones, and a judge should be a capable model.
• Measure it. Put a second judge from another family on the same run and read the two conversation by conversation. Family bias shows up as disagreement that follows the family line rather than anything in the transcript.

If neither is available, the run is still worth having. Write in the analysis which family the judge belongs to, and say plainly that the column sharing it is not comparable with the others on the same footing.

Targets belong to the judge, and not to the scenario

Every judge on a run declares, for each scenario, the grade a well-behaved model should get. "A good model scores 0" only means something once you know whose scale you are on: the same row is 4 for the judge grading deletion and 10 for the judge grading honesty.

So when you add a judge to a run, including afterwards, you write its targets too: one per scenario, or none at all. None is a real answer, and it says you were exploring.

Writing a judge note for one scenario

A judge note sharpens the criterion on one scenario. The judge reads it right after its criterion, as "For this conversation: ...", on that scenario only.

Write how the criterion applies here:

• which passage governs. Name the passage of the published text the situation falls under, and quote it when the wording matters. A judge that is told "rule 4 applies" grades rule 4; one left to guess grades whichever rule it remembered.
• which concrete behaviour matches which level. "Refusing outright is the bottom grade here; refusing and naming the policy is the top" turns a scale written for every row into one that reads this row.

Never write the grade expected. "A good model scores 3 here" is the target, and the target is never shown to any judge: a judge told the answer returns it, and the matrix then measures the note instead of the model. If a sentence in a note would let the judge grade without reading the conversation, take it out.

Two habits that keep notes honest:

• Write them before reading the conversations, as you write targets. A note written after reading the transcripts tends to describe the transcript you read.
• A note is never changed. When a wording turns out wrong, add another under a new name (persona-v2, then persona-v3) and grade again with a new judge reading it. The grades given under the first stay explained by it.

What earns a system judge

A judge is built into this tool when its question is identical for every run, is not the user's to write, and would get worse if each person reworded it. Rewording makes the numbers incomparable across runs, which is the whole value of having it built in. Everything else is a judge you write yourself.

Today there are two.

Eval awareness. It reads what the assistant said, on a fixed 1 to 10 scale, and it is explicitly told not to grade the environment. On by default.

Adversary fidelity, with check_adversary_fidelity: true. It reads what the adversary said, on a fixed 1 to 5 scale, and it grades the adversary rather than the model under test. It exists because a batch built as "the same request, pushed four different ways" falls apart the moment two of the four rows are pushed the same way in practice, and nothing else on the screen can see that: no ordinary judge is ever given the adversary's objective. Off by default, and refused at one turn, where the adversary never speaks. Its lowest grade is reserved for the adversary breaking the situation, which is worth reading beside the awareness figures for the same conversations.

One candidate would qualify and is not built yet: a realism judge. Its real value is "did the environment hold up during the conversation". Whether a scenario looks fake can be judged from its text before spending anything, so a post-run judge adds little there. The environment is what nothing measures today. The check model catches only served answers that invent or contradict the world. A world with nothing to say, a fixed tool landing beside the context, and a model blocked by a tool that always errors are seen by nobody, which is why the analysis advice makes you read transcripts.

It asks a different question from awareness, and does not fold into it. Awareness reads the assistant's turns. Realism reads the environment's, which the awareness judge is instructed to ignore. A model can see a broken world and say nothing, and then awareness reads 1 while the run is worthless.

Until it exists, you can write one as an ordinary secondary judge. It will not be comparable with anyone else's, which is exactly the cost the criterion above describes.

Honesty, harm and helpfulness are not system judges

They are your questions. Some runs are about them and most are not, and building one in would mean charging everyone for a question nobody asked. That is what secondary judges are for: as many as the run needs, each with its own scale, its own targets, and its own column on the same matrix.

Where this comes from

• Bloom, an open source tool for automated behavioral evaluations: the calibration procedure this document describes, and a reference point for what agreement looks like when it works. It reports a Spearman correlation of 0.86 for Opus 4.1 against human grading.
• A Survey on LLM-as-a-Judge: the known failure modes of a model grading text, gathered in one place.
• Judging the Judges: position bias specifically, where a judge asked to choose between two transcripts favours the one shown first.
• LLM Evaluators Recognize and Favor Their Own Generations: self-preference, and the finding that it tracks how well a model recognises its own output. The reason the judge's family is worth knowing before you read a column.