Skip to content
BroadcastwellStart free
Menu

Contribution record

This is the member's own record of contributions and responses, to share if they choose. Taking part in the Exchange does not affect applications. It is not a credential or a hiring decision.

Same question, different engines: what stays fixed?

Author
Sairam SivakumarBroadcastwell team, Founder
Posted
Revision
Original version
Thread
Same question, different engines: what stays fixed? (Discussion)

The contribution

Suppose ChatGPT, Claude, Perplexity, Google AI Overviews and Google AI Mode return different options for the same buyer question. Before explaining the difference, I would record the exact question, dates, locale and account context.

This is a hypothetical comparison, not a fresh measurement. What else must be held fixed, and what cannot realistically be fixed?

If you have a public example you have permission to share, bring the answer and its context. I would like us to separate what we observed from the explanation we are tempted to give it.

Source: https://broadcastwell.com/methodology

The team reply

For a cross-engine comparison, I hold the input and the test conditions as fixed as the products allow: exact prompt text, language and locale, category definition, fresh-session state, authentication or personalization state, capture window, repeat count, and the rule we use to score a name. I also record whether an engine was using web retrieval or a different search mode, because that can change the task materially.

The control people forget most often is conversation state. A "same question" test is not the same test if one engine receives a clean prompt and another has prior chat context, memory, or personalization shaping the answer. I would rather accept that engines have different architectures than accidentally introduce avoidable differences in the setup. The goal is not to make the products identical. It is to make our comparison protocol explicit enough that someone else can understand what was held constant and what could not be.

A contribution record shows what was posted publicly on the Broadcastwell Exchange, by whom and when. It is not a certificate or an endorsement.