Skip to content
BroadcastwellStart free
Menu

Contribution record

This is the member's own record of contributions and responses, to share if they choose. Taking part in the Exchange does not affect applications. It is not a credential or a hiring decision.

Reply in "A claim clinic: what evidence is missing?"

Author
Arjun MehtaWritten with AI assistanceBroadcastwell team, Management Consultant
Posted
Revision
Original version
Thread
A claim clinic: what evidence is missing? (Discussion)

The contribution

A claim I see a lot is some version of "we got this brand into ChatGPT." I would not treat that as evidence of impact until I can see the baseline and the measurement design. What exact buyer questions were tested before the work? Which engine and surface were used? Were the sessions clean? When were the before and after runs captured? Did the brand appear repeatedly, or once? I would also want the raw answer receipts and cited sources, not just a screenshot of the best result.

The thing that is usually missing is repeatability. One favorable answer can be real and still be weak evidence. I also want to know what else changed during the period: the client's site, third-party coverage, product launches, or the engine itself. The claim becomes credible when the evidence shows a consistent before-and-after pattern and makes the confounders visible.

The team reply

The fixed question set limits the interpretation in exactly that way. A Wilson interval does not make those questions a random sample of buyer demand, and it does not account for dependence between repeated answers. It should not be presented as uncertainty about the whole market. The next check is to report counts and valid bases separately by question and engine, preserve exclusions, and examine whether the combined result hides different patterns. We have not added a dependence adjustment or a new sensitivity result in this thread. The public result remains an observation under its stated configuration.

A contribution record shows what was posted publicly on the Broadcastwell Exchange, by whom and when. It is not a certificate or an endorsement.