How to measure whether your site shows up in AI answers (and why one query isn't enough)

View contents

A question that companies increasingly care about is "Do I show up when someone asks ChatGPT or Perplexity about what I offer?" According to OpenAI's report on ChatGPT usage in the first quarter of 2026, several Latin American countries (among them the Dominican Republic, Mexico, Brazil and Costa Rica) are among the ten that climbed the most in messages per capita, with the Dominican Republic moving up nine spots in the global ranking. And a January 2026 report from McKinsey and the World Economic Forum estimates that AI could add between 1.9% and 2.3% in annual productivity to the region. These numbers don't say your brand needs to appear in AI answers (those are separate things), but they do describe the ground: more and more people ask an assistant before they ask a search engine, and that's where the question of whether you're in that conversation comes from.

The problem shows up when you try to measure it. Today there are at least three ways to approach this, and it helps not to confuse them, because they measure different things.

1. Your analytics (GA4, for instance) shows you the clicks that arrived from an assistant: which model they came from and which page they land on, as long as you have it set up.
2. Search Console, with its generative AI performance report announced on June 3, 2026, shows you how your content appears in Google's own generative AI features (AI Overviews and AI Mode) and in Discover; worth knowing that Google is rolling it out gradually, for now to a limited set of sites, so you may not have it available in your account yet.
3. And asking the model directly tells you whether your brand appears in the generated response, click or no click, in any assistant. They're like a usability test and an interview: they can serve the same goal, but they aren't the same, and each sees a different part. Analytics only records the clicks that happened; Google's report measures impressions in Google Search's AI features, not in the conversational responses of ChatGPT, Claude or Perplexity; and direct sampling is the only one that measures whether you appear in those assistants' responses, which most of the time happens without anyone clicking. No single one gives you the full picture.

And sometimes you don't even have those options: in the project I mention below, the team didn't have Search Console set up, so asking the models directly was the only way in. In my experience, that's one of the first frustrations anyone hits when they want to review their users' data and understand where they come from.

In fact, at UXR we've already added an AI visibility section to our client dashboard, to track this specifically, and it seemed useful to organize what the literature and other colleagues around the world have been saying about how to do it well. This is in no way meant to be exhaustive, not at all. It's a topic I find genuinely interesting and I hope to keep learning and writing much more about it, but I'm sharing one of my most recent takeaways.

---

Why a single response doesn't work

The temptation is to open ChatGPT, type the question, check whether your brand appears, and draw a conclusion. The problem is that this single response means almost nothing :(

Language models generate text by sampling from a probability distribution: at each step they don't pick "the" right word, they draw among the possible ones according to their probability. That's why the same question, asked twice, even by the same person, can return your brand once and omit it the next time. It's not that the model contradicts itself: it's how it works. One detail I found surprising: this variation doesn't fully disappear even when you set the model's "temperature" to zero. Research on LLM reproducibility (several papers available on arXiv, listed in the references) shows that floating-point arithmetic and the order of operations on the hardware can alter the result even in theoretically deterministic setups. Anthropic itself recommends sampling multiple times to check output consistency.

The consequence: drawing a conclusion from a single response is like estimating a survey by asking one person. Knowing one answer, out of a wide range of possibilities, doesn't get you very far. That's why we take samples.

---

What to do, based on what we have so far

I have good news: you don't need a huge sample or expensive tools. With a bit of method and statistics, a small number of runs is enough for a reasonable estimate. This is what I've been applying, leaning mainly on Graphite's work ("Demystifying Randomness in AI," 2026) and on classic proportion statistics.

Run each query at least 10 times. That's the threshold where the error becomes manageable. According to Graphite's data, with 10 responses per query the mean absolute error in the visibility estimate sits around 5.6%, and more than 98% of queries land with an error of 10 points or less. You can go higher, but there are diminishing returns: estimates from 10 responses aren't very different from those you get with 200. One operational detail that helps: use a fresh chat for each run and turn off memory, so one response doesn't carry over into the next.

Measure visibility first, position second. It helps to separate two things. The first is binary: does your brand appear or not? The second, finer one, is where it appears relative to others. Position only becomes meaningful once visibility is already high: if your brand appears in 2 out of 30 responses, its "average position" is computed over so few cases that it's mostly noise. If you're starting out and don't yet appear consistently, visibility is the metric that matters.

Use the Wilson interval for the margin of error. Since visibility is a proportion (appearances over total runs), it carries an associated uncertainty worth stating. The Wilson confidence interval (proposed by Edwin B. Wilson in 1927 and still recommended as a default for proportions) works well precisely when the sample is small or the proportion is close to 0% or 100%, which is where the classic normal approximation fails. To make it concrete: if your brand appears in 5 of 10 responses, the 95% Wilson interval runs roughly from 24% to 76%. In other words, with 10 runs you can claim "appears / barely appears," but you can't confidently tell a 50% from a 60%. That takes a larger sample.

Measure on a cadence, weekly or biweekly. Because there's baseline noise, looking at a single measurement or measuring too often leads you to mistake normal variation for a trend. A regular cadence, read alongside the confidence intervals, lets you separate signal from noise and see whether a content change actually moved the needle.

---

A case: visibility can be zero

A result that comes up more often than you'd expect is the absolute zero. In a recent diagnostic for a project in the United States, where the team didn't yet have analytics or Search Console set up, direct sampling was the only way to estimate visibility. We measured the brand's presence across three language models (LLMs), ten queries each, derived from an analysis of its competitors: zero mentions out of thirty. The rest of the direct competitors were equally absent, too: a few large platforms showed up, but no player comparable to the project.

A clean zero like that is more informative than it looks. There's a bound (the "rule of three") stating that, when you observe zero appearances in n attempts, the upper limit of the 95% confidence interval is around 3/n. With 30 runs, that puts real visibility below roughly 10%. Put another way: if the brand had meaningful presence, we'd very likely have seen at least one appearance. We saw none. And there's a hidden advantage: a zero is an excellent baseline. Any future appearance, after a content intervention, becomes a clear and easy-to-defend signal.

I plan to write separately about this case and the full diagnostic it came from, because AI visibility was just one of several parts of the analysis, and not even the main one.

---

There's much more to this, and on a good part of it we're still learning in practice and reading others. Open questions remain: how to choose the query set well so it reflects how people actually search, how to handle responses with web search enabled, how to compare across models. Each is worth its own post. If you measure AI visibility a different way, or if you see something I'm missing here, I'd love to hear from you.

---

References

• Druck, G. & Smith, E. (2026). Demystifying Randomness in AI. Graphite.io. [Agency report; primary methodological basis for sample size and visibility tracking]. https://graphite.io/five-percent/demystifying-randomness-in-ai
• Wilson, E. B. (1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209-212. [Primary source for the Wilson confidence interval].
• OpenAI (2026). Report on ChatGPT usage, first quarter of 2026. [Latin America adoption data; cited via El Cronista, July 1, 2026].
• McKinsey & World Economic Forum (2026). From Potential to Productivity: Latin America in the Intelligent Age. [Estimate of AI's impact on regional productivity].
• Google Search Central (2026). Optimizing your website for generative AI features on Google Search. Official Google documentation. [Google's position: optimizing for generative AI is still SEO; best practices and clarification of which tactics aren't needed]. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
• Google Search Central (June 3, 2026). Introducing generative AI performance reports in Search Console. By Hillel Maoz and Moshe Samet. [Official announcement of the report measuring impressions in Google Search's generative AI features and Discover; rolling out gradually]. https://developers.google.com/search/blog/2026/06/gen-ai-performance-reports
• On LLM non-determinism (reproducibility, sampling and temperature): a set of works available on arXiv, including "Non-Determinism of Deterministic LLM Settings" (arXiv:2408.04667) and "Defeating Nondeterminism in LLM Inference" (Thinking Machines Lab, 2025). [Technical basis for why responses vary].

A note on sources: Graphite's work is an agency report, not a peer-reviewed paper; I use it for its transparent methodology and authors with a research background. The Wilson statistics and the non-determinism works are primary and academic sources.

---

Share

Get the next articles

I write about UX Research, transitioning from social sciences, and reflections from the craft. No fixed schedule — when there is something worth sharing.

This newsletter is currently published in Spanish only. Subscribe now and you'll start receiving English issues once our English-speaking list grows.

Subscribe

No spam. Unsubscribe anytime.