Skip to content
BroadcastwellStart free
Menu

DiscussionMeasurement

A claim clinic: what evidence is missing?

Score 0
Sairam SivakumarReaderBroadcastwell team, Founder

I would like to use this thread to examine one public, dated claim about AI search visibility at a time. Link the original source and separate its observation, inference and proposed action.

Then identify one missing piece of evidence that could change your assessment. Represent the author's caveats fairly. The purpose is to improve the test, not attack the person or sell a competing service.

Which claim would you like to examine together, and what would make you change your current view?

Replies

  1. Reply by Arjun Mehta

    Score 0
    Arjun MehtaWritten with AI assistanceReaderBroadcastwell team, Management Consultant

    A claim I see a lot is some version of "we got this brand into ChatGPT." I would not treat that as evidence of impact until I can see the baseline and the measurement design. What exact buyer questions were tested before the work? Which engine and surface were used? Were the sessions clean? When were the before and after runs captured? Did the brand appear repeatedly, or once? I would also want the raw answer receipts and cited sources, not just a screenshot of the best result.

    The thing that is usually missing is repeatability. One favorable answer can be real and still be weak evidence. I also want to know what else changed during the period: the client's site, third-party coverage, product launches, or the engine itself. The claim becomes credible when the evidence shows a consistent before-and-after pattern and makes the confounders visible.

  2. Reply by clane

    Score 0
    claneReader

    One concrete public claim to examine is the CRM release’s report of 113 named answers out of 150 valid answers (75.3%), with a Wilson 95% interval of 67.9%–81.5%. Source: https://index.broadcastwell.com/categories/crm/ (release 2026-09.1; answer capture September 22, 2026).

    Observation: that is the published count for the fixed question set. It is not evidence of product quality, and the page explicitly says so. The inference I would question is treating the interval as uncertainty about all real buyer questions. Three repeats of each question on each engine do not make the questions a random sample of buyer demand.

    The missing evidence I would ask for is sensitivity by question and engine, plus an explanation of how dependence among repeated answers affects the interval’s intended interpretation. Stable results across those slices would improve my confidence in robustness within this design; they would still not establish a market-wide ranking.

    AI-assisted analysis of the public summary, not an independent audit of raw answers.

  3. Reply by Sairam Sivakumar

    Score 0
    Sairam SivakumarReaderBroadcastwell team, Founder

    The fixed question set limits the interpretation in exactly that way. A Wilson interval does not make those questions a random sample of buyer demand, and it does not account for dependence between repeated answers. It should not be presented as uncertainty about the whole market. The next check is to report counts and valid bases separately by question and engine, preserve exclusions, and examine whether the combined result hides different patterns. We have not added a dependence adjustment or a new sensitivity result in this thread. The public result remains an observation under its stated configuration.

    Source: index.broadcastwell.com/categories/crm/