Contribution record
This is the member's own record of contributions and responses, to share if they choose. Taking part in the Exchange does not affect applications. It is not a credential or a hiring decision.
Reply in "A claim clinic: what evidence is missing?"
- Author
- clane
- Posted
- Revision
- Original version
- Thread
- A claim clinic: what evidence is missing? (Discussion)
The contribution
One concrete public claim to examine is the CRM release’s report of 113 named answers out of 150 valid answers (75.3%), with a Wilson 95% interval of 67.9%–81.5%. Source: https://index.broadcastwell.com/categories/crm/ (release 2026-09.1; answer capture September 22, 2026).
Observation: that is the published count for the fixed question set. It is not evidence of product quality, and the page explicitly says so. The inference I would question is treating the interval as uncertainty about all real buyer questions. Three repeats of each question on each engine do not make the questions a random sample of buyer demand.
The missing evidence I would ask for is sensitivity by question and engine, plus an explanation of how dependence among repeated answers affects the interval’s intended interpretation. Stable results across those slices would improve my confidence in robustness within this design; they would still not establish a market-wide ranking.
AI-assisted analysis of the public summary, not an independent audit of raw answers.
The team reply
A claim I see a lot is some version of "we got this brand into ChatGPT." I would not treat that as evidence of impact until I can see the baseline and the measurement design. What exact buyer questions were tested before the work? Which engine and surface were used? Were the sessions clean? When were the before and after runs captured? Did the brand appear repeatedly, or once? I would also want the raw answer receipts and cited sources, not just a screenshot of the best result.
The thing that is usually missing is repeatability. One favorable answer can be real and still be weak evidence. I also want to know what else changed during the period: the client's site, third-party coverage, product launches, or the engine itself. The claim becomes credible when the evidence shows a consistent before-and-after pattern and makes the confounders visible.
A contribution record shows what was posted publicly on the Broadcastwell Exchange, by whom and when. It is not a certificate or an endorsement.