Skip to content
BroadcastwellStart free
Menu

QuestionMeasurement

Same question, different engines: what stays fixed?

Score 0
Sairam SivakumarReaderBroadcastwell team, Founder

Bring a receipt. Answers that carry a published receipt are easier for everyone to check.

Suppose ChatGPT, Claude, Perplexity, Google AI Overviews and Google AI Mode return different options for the same buyer question. Before explaining the difference, I would record the exact question, dates, locale and account context.

This is a hypothetical comparison, not a fresh measurement. What else must be held fixed, and what cannot realistically be fixed?

If you have a public example you have permission to share, bring the answer and its context. I would like us to separate what we observed from the explanation we are tempted to give it.

Source: broadcastwell.com/methodology

3 answers

The accepted answer first, then by score, then oldest first.

  1. Answer by Daniel Kim

    Score 0
    Daniel KimWritten with AI assistanceReaderBroadcastwell team, AI Engineer

    For a cross-engine comparison, I hold the input and the test conditions as fixed as the products allow: exact prompt text, language and locale, category definition, fresh-session state, authentication or personalization state, capture window, repeat count, and the rule we use to score a name. I also record whether an engine was using web retrieval or a different search mode, because that can change the task materially.

    The control people forget most often is conversation state. A "same question" test is not the same test if one engine receives a clean prompt and another has prior chat context, memory, or personalization shaping the answer. I would rather accept that engines have different architectures than accidentally introduce avoidable differences in the setup. The goal is not to make the products identical. It is to make our comparison protocol explicit enough that someone else can understand what was held constant and what could not be.

  2. Answer by clane

    Score 0
    claneReader

    One additional control I would add is run order. In a hypothetical comparison, querying every prompt on engine A first and engine B hours later can mix an engine difference with a time difference. I would interleave engines within a narrow capture window and record the order and timestamps, with a preset repeat count.

    I would not claim to hold hidden retrieval indexes, experiments, model updates or undisclosed personalization fixed. Mark those as uncontrolled or unknown, retain every run, and describe the result as a comparison of the recorded product experiences. API and consumer-app outputs should also remain separate unless the study explicitly compares those surfaces.

    AI-assisted response; proposed protocol, not a completed measurement.

  3. Answer by Sairam Sivakumar

    Score 0
    Sairam SivakumarReaderBroadcastwell team, Founder

    clane, recording run order would let a reader distinguish an engine comparison from a comparison made at different times. For a future protocol, a balanced order across repeat blocks would also avoid giving the same engine the first slot every time.

    Predeclare how a failed capture or outage affects the block, and preserve the failed attempt when a retry is allowed. Keep product surface, timestamp and run order with each answer. Interleaving does not hold hidden systems fixed, so the conclusion should stay limited to the recorded experiences and capture window.

    Source: broadcastwell.com/methodology