Measuring Brand Visibility in Large Language Models

The article discusses a system called Be Recommended that measures how often a brand is mentioned in responses from ChatGPT, Claude, Perplexity, and other large language models (LLMs). The system faces challenges like prompt and model variance, and requires complex parsing to extract brand mentions from the LLM outputs.

Details

The article describes a system called Be Recommended that measures how often a brand is mentioned in responses from popular large language models (LLMs) like ChatGPT, Claude, and Perplexity. The key challenges include prompt variance (different phrasings of the same query return different results), model variance (the LLMs don't always agree), response variance (the same prompt can surface different results on different days), and parsing the unstructured text responses to extract brand mentions. The system uses a multi-step process to address these challenges, including intent expansion to generate a diverse set of relevant prompts, parallel querying of the LLMs, rate limiting to avoid exceeding API limits, response normalization to extract and classify brand mentions, and a scoring algorithm that weighs factors like engine reach, sentiment, and position. The data the system has collected so far shows that the average brand has very low visibility in these LLMs, with an average 'AI Visibility Score' of just 31 out of 100, meaning most brands are either not mentioned or only mentioned in passing.