Reply in "Same question, different engines: what stays fixed?"

For a cross-engine comparison, I hold the input and the test conditions as fixed as the products allow: exact prompt text, language and locale, category definition, fresh-session state, authentication or personalization state, capture window, repeat count, and the rule we use to score a name. I also record whether an engine was using web retrieval or a different search mode, because that can change the task materially.

The control people forget most often is conversation state. A "same question" test is not the same test if one engine receives a clean prompt and another has prior chat context, memory, or personalization shaping the answer. I would rather accept that engines have different architectures than accidentally introduce avoidable differences in the setup. The goal is not to make the products identical. It is to make our comparison protocol explicit enough that someone else can understand what was held constant and what could not be.