Skip to content
BroadcastwellStart free
Menu

QuestionMeasurement

What should count as named?

Score 0
Sairam SivakumarReaderBroadcastwell team, Founder

Bring a receipt. Answers that carry a published receipt are easier for everyone to check.

An answer can name a company as an option, mention it in passing, or explicitly say it does not fit. I would record those observations separately before combining anything into a rate.

Here is a hypothetical case: "Example A exists, but it does not meet the requested constraint." Example A is fictional. Would you count a mention, a recommendation, both or neither?

Which rule would make two reviewers agree? The published method is linked below so we can test the rule against something concrete.

Source: broadcastwell.com/methodology

3 answers

The accepted answer first, then by score, then oldest first.

  1. Answer by Vikram Patel

    Score 0
    Vikram PatelWritten with AI assistanceReaderBroadcastwell team, AI Architect and Specialist

    I would draw the primary "named" line at an unambiguous mention in the answer body. If the brand appears in a comparison list, it is named even if the model does not recommend it. If the model says the brand is a poor fit, it is still named; visibility and sentiment are different measurements. A recommendation is a stronger event, but I would record it as a separate attribute rather than changing the definition of "named."

    A brand that appears only in a citation link or source card would not count as named in the answer. I would track that separately as "vendor domain cited." That distinction is useful because an engine can use a company as evidence without presenting the company to the buyer as an option.

    The reason for a low threshold is auditability. "Was the entity explicitly present in the substantive answer?" is easier to reproduce than a subjective judgment about whether the mention was favorable or important enough.

    1. Comment by Maya Thompson

      Score 0
      Maya ThompsonWritten with AI assistanceReaderBroadcastwell team, AI Architect and Specialistedited

      I agree with using substantive mention as the base definition, then treating recommendation and citation as separate dimensions. A visibility metric should first answer the simplest auditable question: did the answer present this brand or product to the user? Recommendation tells us how the brand was framed. Citation tells us whether the brand's domain helped support the answer. Those are related, but they are not interchangeable.

      A hard case is an answer that says, "I would not shortlist Kalvenor (Sample data, fictional company) for this use case," and then explains why. I would still mark Kalvenor as named because the user saw the brand in the answer, but I would tag the context as negative or exclusionary rather than positive. Another hard case is an ambiguous product name that is also an ordinary word. In that situation I would require entity resolution from the surrounding text before counting it, rather than matching a token mechanically.

      Edit note: Wording updated by the Broadcastwell team, 11 November 2026.

  2. Answer by clane

    Score 0
    claneReader

    For the fictional Example A sentence, I would record mentioned=yes, recommended=no, and explicitly_excluded=yes, with the exact supporting sentence attached. These fields should remain separate when calculating a shortlist or recommendation rate.

    My proposed check is a small fixed set of edge cases: citation-only, ambiguous name, explicit rejection, conditional recommendation, and clear recommendation. Have two reviewers label them independently before looking at aggregate results, preserve disagreements, and version any rule changes. That would test whether the coding rule is usable rather than assuming agreement because the definition sounds clear.

    AI-assisted response; this is a proposed test, not a report of a completed study.

  3. Answer by Sairam Sivakumar

    Score 0
    Sairam SivakumarReaderBroadcastwell team, Founder

    clane, those three fields preserve the meaning of the fictional sentence without turning an explicit rejection into a recommendation. Your edge cases also expose where a simple name match can fail.

    Add an unknown state when the surrounding text cannot resolve the entity. Keep each reviewer’s original label and supporting passage separate from the later reconciled decision, and state which field enters each published metric. If a rule changes, record the version and recalculate affected results; do not silently mix the old and new definitions. This remains a proposed coding check until people actually complete it.

    Source: broadcastwell.com/methodology