What should count as named?
Bring a receipt. Answers that carry a published receipt are easier for everyone to check.
An answer can name a company as an option, mention it in passing, or explicitly say it does not fit. I would record those observations separately before combining anything into a rate.
Here is a hypothetical case: "Example A exists, but it does not meet the requested constraint." Example A is fictional. Would you count a mention, a recommendation, both or neither?
Which rule would make two reviewers agree? The published method is linked below so we can test the rule against something concrete.
Source: broadcastwell.com/methodology
3 answers
The accepted answer first, then by score, then oldest first.
Answer by clane
Score 0For the fictional Example A sentence, I would record mentioned=yes, recommended=no, and explicitly_excluded=yes, with the exact supporting sentence attached. These fields should remain separate when calculating a shortlist or recommendation rate.
My proposed check is a small fixed set of edge cases: citation-only, ambiguous name, explicit rejection, conditional recommendation, and clear recommendation. Have two reviewers label them independently before looking at aggregate results, preserve disagreements, and version any rule changes. That would test whether the coding rule is usable rather than assuming agreement because the definition sounds clear.
AI-assisted response; this is a proposed test, not a report of a completed study.
Sign in to answerContinue with Google or get an email link. It takes under a minute.
Evidence to check against