The Crucible Protocol

Adjudicating disagreement without rhetoric

Abstract. Public debate formats award victory on impression: delivery, confidence, composure, wit. These qualities are measurable and trainable, and they are uncorrelated with being right. The Crucible Protocol is a version-controlled alternative that removes rhetoric's purchase on the outcome. Claims enter in a fixed structure through a deterministic admission filter. Evidence is graded on a published ladder. Objection is restricted to four enumerated grounds. The verdict is arithmetic over the surviving record rather than a judgment about a performance. The rubric is published before any argument is submitted, so the only way to score higher is to bring better evidence, a tighter warrant, or a more relevant claim. This paper specifies the mechanism, works a full example, reports what running it found, and states plainly what it does not solve.

1. The problem

A traditional debate produces a result and no artefact. The audience forms an impression, the impression becomes the verdict, and nothing retrievable survives. Two years later the same question is argued from scratch by different people who do not know that the strongest version of their point was already made, refuted and repaired somewhere they will never find.

The impression is also a channel that rhetoric acts on directly. A confident delivery, a well-timed joke, an attack on an opponent's character, a firm assertion with no source behind it: all of these move an audience, and none of them is evidence. Any format whose verdict is a function of audience reaction will reward the qualities that move audiences, and those qualities are separable from correctness.

The design question is not how to find better judges. It is how to build a structure in which the qualities that decide traditional debates have no surface to act on.

2. Design principle

Every input to the verdict must be either mechanically checkable or recorded as a judgment with its author, its reason, and its value.

That single constraint produces most of the protocol. If a factor cannot be checked, it must at least be localised, so that a reader who disagrees can point at the specific line rather than at a general feeling. The test the protocol is built to pass is narrow and concrete:

Could a stranger who disagrees with the outcome point at the specific line they think is wrong?

An impression cannot be attacked in detail, because it has no parts. A ledger can be.

3. The mechanism

A debate proceeds through seven phases: framing, construction, challenge, steelman, adjudication, verdict, closed. An admitted appeal enters a separate appeal-review phase and exits back to closed.

3.1 Claim structure

Every claim is a Toulmin triple plus a declared confidence, and is one of three types: empirical, logical, or definitional.

The warrant requirement is the load-bearing one. In ordinary argument the inferential step is left implicit, and unstated inference is where rhetoric hides. Forcing it into a field makes it a target: a warrant, once written down, can be attacked on its own.

Claims live in threads. A thread holds successive versions of one claim, and each version has a status: candidate, merged, superseded, demoted, or rejected. Nothing is ever deleted; a demoted claim stays on the record with the ruling that demoted it.

3.2 The Gate

Admission is a deterministic filter of fourteen rules, applied before anything reaches the record. The rules constrain form only and cannot reference a claim's position, so the Gate cannot be aimed at a conclusion.

Every refusal is public and quotes the rule, the offending phrase, and a specific fix. A refusal works as an instruction rather than a penalty, so most disputes become a revision instead of an argument.

3.3 Evidence tiers

Sources are graded on a five-step ladder adapted from GRADE.

Claiming a tier does not grant it. Verification means retrieval: an arbiter must fetch the source and confirm it says what the citation claims, and the confirmation is recorded as its own action. Until then the citation's effective weight is capped at T3, so an unverified claim of T1 scores as a 3 rather than a 5. This closes the gap where a participant could assert a top tier and have it priced as one.

3.4 Challenge and merge

Objection is not free-form. A merged claim may be challenged on exactly four grounds:

1. Evidence validity: the source does not support what it is cited for.
2. Warrant failure: the inference does not carry.
3. Relevance: the claim, even if true, does not bear on the resolution.
4. Counter-evidence: stronger evidence points the other way.

Grounds 1 and 4 must themselves cite a source (G11). A participant may not challenge their own side, which closes a straightforward exploit described in section 6.

The merge rule governs which version of a claim stands. A new version supersedes the incumbent only if it is stronger under the same published rubric. Anyone may submit a repair, including the person who just refuted the claim. The intent is to reward improving the record over winning an exchange.

3.5 Scoring

Each standing version carries a merit score, the product of four factors:

merit = E x R x S x Q

E is the evidence base. For logical claims E = 2; for definitional claims E = 1. For empirical claims, E is the effective weight of the strongest citation plus a corroboration bonus of min(1, 0.5 x (n - 1)) for n sources. A second independent source is worth half a tier, a third reaches the cap of 1, and every source after the third adds nothing. Effective weight is the claimed tier weight, capped at T3 unless verified.

R is relevance, assigned by an arbiter from {0.25, 0.5, 0.75, 1.0}. This is the most subjective factor in the rubric. It is recorded per claim, which is what made a real measurement failure detectable (section 6).

S is survival: S = min(1.3, 1 + 0.1 x d), where d counts dismissed challenges from the opposing side only. The cap is reached at three. Survival records that a claim held up under scrutiny. It is not a bounty per attack survived, because an uncapped bounty rewards a claim for attracting challenges, and two cooperating participants can arrange that.

Q is the qualifier: certain = 1.0, probable = 0.9, plausible = 0.75. The spread is deliberately narrow relative to the demotion risk that overclaiming carries under G12. This makes honest calibration the dominant strategy rather than a virtue the protocol has to request.

A side's total is the sum of merit over its canonical versions. Where a debate declares a claim budget of N, only each side's best N claims score; the rest remain in the record marked as non-scoring. Without a budget, totals being sums means volume wins. Section 6 reports the measurement that forced this.

3.6 Verdict, burden and appeal

The margin is the leader's lead as a fraction of the leader's own total:

margin = (hi - lo) / hi

It is compared against four bands in order, with a strict less-than test:

A margin under 10 percent is declared unresolved rather than awarded to whoever happens to be ahead. "Not proven" is a real outcome, and this is the most important difference from formats that must produce a winner.

Where a debate declares that one side carries the burden of proof, an inconclusive ledger resolves in favour of the side without it. The verdict is flagged byBurden so a reader can see that the ledger did not decide it. Under a shared burden, inconclusive means no winner.

Issuing a verdict freezes the ledger. Later versions, challenges and appeals produce new verdicts; they never edit an old one. Every verdict a debate has carried stays on the record.

A closed debate may be reopened on three grounds only: A1 (newly discovered evidence), A2 (procedural error), A3 (source retraction or correction). An appeal adds to the history rather than rewriting it.

4. A worked verdict

Two claims per side, all arithmetic explicit.

Side A

Side B

margin = (6.36 - 4.20) / 6.36 = 0.340

0.340 is not below 0.25 but is below 0.50, so the verdict is A wins, clear.

Now change exactly one thing. Suppose an arbiter retrieves B1's source and confirms it at T1. Its effective weight becomes 5 rather than 3:

B1 merit = 5 x 1.0 x 1.1 x 1.0 = 5.50
Total B  = 6.40
margin   = (6.40 - 6.36) / 6.40 = 0.006

0.006 is below 0.10, so the verdict becomes unresolved, no winner. Under a declared burden it would go to whichever side does not carry it.

One verification step moved the outcome from "A wins clearly" to "nobody wins". That is the intended behaviour, and it is why verification is defined as retrieval rather than inspection of the citation string.

5. What it inherits

Nothing here is new; the contribution is the assembly.

Two of these deserve emphasis because they mark real limits rather than borrowings.

Barnes and Christiano (2020) identified obfuscated arguments: a case whose flaw is spread thinly across many steps rather than concentrated in one. Every step survives inspection; the defect exists only in aggregate. Recursion then fails, because the honest side cannot say where the argument is wrong. The protocol mitigates this by removing the clock (obfuscation is an attack on a bounded judge) and by capping the scoring surface with the claim budget. It does not solve it, and no structured-argument system currently does.

The bridging literature is cited for a mechanism that is specified but not built. The specification recommends that the two rulings carrying the most weight, merge decisions and challenge resolutions, require agreement across arbiters who do not share a prior position. Section 7 records that the reference implementation runs one arbiter.

6. What running it revealed

The protocol has been run end to end with language models in all four roles: a framing agent, two advocates, and a referee. Two adversarial audits and a small number of live debates produced findings that no amount of design review had surfaced.

The Gate works, and its false positives are directional. In the first live debate, seven submissions were refused: a rhetorical question inside a challenge response (G8), the word "obviously" inside a steelman (G9), and five refusals from the formal-register rule (G7), of which four were false positives on legitimate acronyms including PRISMA, SMART and GCSE. That error rate is not uniform across fields. It falls hardest on acronym-dense vocabularies, which means medicine, law and engineering pay a higher admission tax than philosophy does. This is a real bias with a known direction, and it is unfixed at the rule level.

A published factor can be inert. Across the first two debates the arbiter assigned full relevance to eight claims out of eight. A four-factor rubric was silently operating as a three-factor rubric while the formula continued to display four. No theory would have found this; only running it did. Rewriting the relevance brief produced a distribution of 1, 0.75, 1, 0.75 on the next run.

Additive totals reward volume. Measured directly: eighteen thin claims scored 20.25 and beat three strong claims at 14.85. Padding the leading side also widens the margin band. The claim budget exists because of this measurement.

Survival could be farmed. Nothing originally prevented a participant from challenging their own side and losing on purpose to inflate the survival multiplier. The fix was to refuse same-side challenges and to count only opposing-side dismissals.

Verification was theatre. An arbiter without retrieval can only judge the citation string. The fix was to define verification as retrieval, record it as a separate action, and cap unverified citations at T3.

A recorded field was never read. The burden of proof was stored and never consulted, so "not proven" could not resolve. This was fixed, with the default changed to shared, because the previous default silently handed every inconclusive debate to one side.

The engine currently passes 115 conformance tests, many written as attacks rather than as confirmations.

7. Known limits

Stated as concessions, in the form the protocol requires of its participants.

The arithmetic does not remove judgment; it relocates it. Relevance weighting, tier verification, merge and challenge rulings, and the rubric constants themselves are all judgments. No experiment fixed the corroboration cap at 1 or the survival increment at 0.1; they are defensible, not derived. The claim the protocol can make is narrow: its verdicts are not objective, but they are wrong in ways that can be found. Displaying three decimal places overstates the precision of the inputs.

The tier ladder embeds one field's epistemology. GRADE was built for clinical questions. On a question where the relevant evidence is ethnographic, historical, or interpretive, the ladder will misprice it, and a side arguing from qualitative evidence starts at a structural disadvantage unrelated to being right. The protocol is better suited to some questions than others, and knowing which kind you have is an operating instruction, not a detail.

The fact-value boundary is porous. Which facts are relevant is already a value judgment, and framing determines the answer. A verdict on the factual substrate of a contested question will be read as a verdict on the question. The framing phase currently produces a resolution unopposed, which is a genuine weakness.

Two sides is a modelling choice. The data model fixes exactly two positions because pairwise merit totals produce a legible margin and three-way comparison does not. Multi-way questions must be decomposed into several two-sided resolutions, and choosing the decomposition is itself a framing act.

One arbiter is the deepest open problem. Every other mechanism is designed to survive participants who are biased and trying to win. The decisive judgments are then handed to a single arbiter whose good faith is assumed, and in the reference implementation that arbiter is the same kind of system as the advocates, with the same training distribution and the same blind spots. The specification says arbiters should be selected by demonstrated calibration and should be required to agree across differing priors. The implementation runs one. Until that changes, every verdict produced should be read as the output of a single unaudited judge applying a published rubric. That is better than an audience impression, but it is not what the protocol claims to be.

8. Cost and adoption

Fifty years of argument-mapping systems failed on adoption, not on design. People will argue for hours and will not spend twenty minutes formalising.

The change that makes this attempt different is that machines can now pay the formatting tax. A model turns an ordinary paragraph into a compliant claim: assertion, grounds, explicit warrant, suggested tiers. The contributor reviews rather than composes. The Gate pays the moderation tax deterministically, before anything reaches a human.

A full debate at the lowest depth is roughly 25 agent calls, measured at $3.61 and 15 to 20 minutes of wall-clock time.

The judgment tax has not collapsed, and it cannot be automated away without becoming exactly the verification theatre described in section 6. The correct design, a panel of calibrated arbiters from different origins, costs several times more than the single arbiter that is running.

One operational finding is worth recording: open contribution and per-run cost are in tension. A marketing bot once submitted a promotional message as a debate question, and the system did what it was built to do, spending real money framing and adjudicating a question nobody had asked. When each contribution triggers an expensive process, the open door needs a rate limit, a deposit, or a queue before it opens, not after.

9. Status

The protocol is at version 1.1, specified normatively and implementable without reference to the accompanying code. The reference engine is dependency-free, Apache-2.0, and passes 115 conformance tests. It has run a handful of live debates, found several exploits in its own scoring rule, and fixed most of them.

That is a starting position rather than a result. Some claims this paper does not make are worth stating explicitly: that verdicts track truth better than the alternatives, that participation transfers reasoning skill outside the protocol, and that the record accumulates in practice. Each is testable, none has been tested, and a paper claiming more than that would be violating its own rubric.

The metaphor is not a courtroom or a stage. Both are venues where advocacy is performed for someone who is impressed or unimpressed. A crucible is a vessel that takes heat. It is not fair to its contents, and nothing inside it is protected by having been eloquent. What comes out is not what argued best.

Specification and reference engine: open-crucible-protocol, Apache-2.0. Scoring constants quoted here are normative and taken from src/constants.js. The full argument, its intellectual history and the audit record are published alongside this paper at /evidence.