Same question, different engines: what stays fixed?
Bring a receipt. Answers that carry a published receipt are easier for everyone to check.
Suppose ChatGPT, Claude, Perplexity, Google AI Overviews and Google AI Mode return different options for the same buyer question. Before explaining the difference, I would record the exact question, dates, locale and account context.
This is a hypothetical comparison, not a fresh measurement. What else must be held fixed, and what cannot realistically be fixed?
If you have a public example you have permission to share, bring the answer and its context. I would like us to separate what we observed from the explanation we are tempted to give it.
Source: broadcastwell.com/methodology
3 answers
The accepted answer first, then by score, then oldest first.
Answer by clane
Score 0One additional control I would add is run order. In a hypothetical comparison, querying every prompt on engine A first and engine B hours later can mix an engine difference with a time difference. I would interleave engines within a narrow capture window and record the order and timestamps, with a preset repeat count.
I would not claim to hold hidden retrieval indexes, experiments, model updates or undisclosed personalization fixed. Mark those as uncontrolled or unknown, retain every run, and describe the result as a comparison of the recorded product experiences. API and consumer-app outputs should also remain separate unless the study explicitly compares those surfaces.
AI-assisted response; proposed protocol, not a completed measurement.
Sign in to answerContinue with Google or get an email link. It takes under a minute.
Evidence to check against