RoastPit

The idea

Model comparisons usually happen on benchmarks: scores in a table, stripped of context. RoastPit compares them somewhere more revealing: an argument. Users submit a properly contestable topic, four models argue it out over escalating rounds, and the audience votes on who made the better case.

The winner is decided by that audience vote. No model is asked to judge the debate. Entertainment on the surface; a fairly demanding orchestration problem underneath.

Orchestration without an orchestrator model

The obvious way to vary a debate is to ask a 'director' model what should happen next. RoastPit does not do that: not on the happy path, not as a fallback, and not when a debate hits its cost cap, where it closes on deterministic copy and audience voting instead.

Every round is planned locally from the state of the debate. The planner reads the moves assigned to this match, the topic's tags, the rolling summary, recent turn sentiment, which model is pulling ahead, and live reaction counts, then decides what to change. No model call is involved in that decision.

It is a judgment call about where a model earns its place. Putting one inside the round loop means paying for it on every round of every debate, forever, to decide something a scoring function decides just as well, and faster.

A weathermaker, not a playwright

The orchestrator changes the conditions of a debate, never its content. It is closer to a referee who can add rules mid-game than a director giving actors their lines. It will tell every model that responses must be one sentence for the next three turns, or that no one may agree with GPT this round, or that two debaters must form a coalition and defend a joint position. It never tells a model what to argue, what tone to take, or how to feel about anyone.

That distinction is the whole design. Scripting dialogue would make four different models sound like one writer; changing the rules they operate under lets each personality collide with the constraint in its own way. The models stay themselves; the weather changes around them.

Each match draws three moves from a toolkit of nineteen, held back through the opening act and then spent across the rest, so two debates on an identical topic still take different shapes. And the mechanism stays completely invisible: no voice, no avatar, no announcement. The audience sees the AIs behaving differently and never learns why.

Keeping context bounded

Debates run long. Feeding the entire transcript back to four models every single turn is the shortest path to context-limit failures.

So a compactor runs at the end of each round: a fast model distils what happened into a structured summary that gets carried forward as rolling context. However long a debate goes, the history it carries stays bounded.

Voice, and what happens when it fails

Debates are voice-first. A streaming TTS pipeline delivers audio sentence by sentence instead of making users wait for a whole turn to finish generating, so a debate feels like a conversation rather than a page load. Reactive interjections are pre-generated and served from storage. In a live show, latency is far more noticeable than variety.

When the voice provider goes down, the product falls back to text automatically. Users are never offered that fallback as a choice, because it is not a feature. It is how a paid debate keeps working through someone else's outage.

What production actually taught me

A single debate got a full written incident review: the stored rows, every turn, every cost event, the round summaries, and the streaming reconnect path all read back against the code.

The original suspicion (duplicate generation on reload) turned out to be wrong. What the investigation did surface was more useful: schema drift between application code and the live database, truncated turns from one model persisted across every round, and cost-log semantics misleading enough to make a clean run look duplicated.

Two of those three were observability defects rather than product defects. The system was behaving correctly and reporting it badly. That is the failure mode I now look for first.