"We Rank #1 in AI Search" -- But Who Is Asking That Question?
The AEO measurement blind spot: measuring yourself with your own queries proves nothing. Unless you measure with the questions users actually ask AI, you cannot know if you are being cited.
What "Ranking #1 in AI Search" Really Means
AEO tools and consulting reports sometimes contain a line like this:
"We searched for your company in ChatGPT and you appeared as the top result."
This looks like a win. But there is one thing worth checking.
Who wrote that query?
If your marketing team crafted the question, typed it into ChatGPT themselves, and saw your company name appear -- that is not "being cited by AI." That is confirming what you expected to find.
Users Ask Differently Than You Do
In SEO, Google Search Console provides the ground truth. You can see what keywords users actually searched for. AI search has no equivalent. ChatGPT, Perplexity, and Gemini do not publish user query data.
This means when measuring AEO, you have to create the queries yourself. And this is where the fundamental problem lies.
Queries you design:
- "Best AEO optimization tools"
- "AI search monitoring tool comparison"
- "AEO monitoring service"
What users actually ask AI:
- "My site doesn't show up in ChatGPT, why?"
- "How do I get AI to mention my company?"
- "I'm doing SEO but AI search ignores me. What should I do?"
Measure with the first set and you might see "cited." Measure with the second set and the results could be completely different. The problem is not which is correct -- it is that you do not know which one users are actually asking.
The Brand Query Trap
There is an even more insidious problem. Most AEO measurement tools produce high scores on brand queries -- questions that include your company name.
Ask "What is Sighted?" and AI will talk about Sighted. Of course it will. You said the name.
Our data from 3,419 observations on our own domain:
| Query Type | Mention Rate | Citation Rate |
|---|---|---|
| Brand queries ("What is Sighted?") | 100% | 38% |
| Industry queries ("Best AEO companies?") | 0% | 0% |
If you only measure brand queries, you get "100% mention rate, 38% citation rate." If you measure the queries that matter -- where users do not already know your name -- the citation rate is zero.
This is not unique to us. Another company with 25,204 observations showed the same pattern: 24% on brand queries, 0% on industry queries. Ten times the data, same structure.
Without a Question Space, You Are Measuring Nothing
SEO has "keywords" as a shared language. "We rank #3 for this keyword" is unambiguous. AEO has no equivalent. What it needs is a defined Question Space -- the set of questions that matter for your business, designed in advance.
Three layers of a Question Space:
- Brand layer: Queries containing your name. Measuring these is almost meaningless -- you will always appear when asked by name
- Industry layer: Queries without your name, about your industry. "Best tools for X?" -- this is where the real competition happens
- Problem layer: Queries without industry terms, just the user's problem. "X isn't working" -- being cited here is the goal
Most AEO measurement stops at layer 1 and reports "we are being cited." Layers 2 and 3 are either unmeasured, or measured with queries that carry the creator's bias.
A Single Observation Proves Nothing
Even with a well-defined question space, there is another problem. AI responses vary every time. Ask the same question 10 times and you will get 10 slightly different answers with different source citations.
Concluding "we were cited" from a single observation is like rolling a die once, getting a 6, and declaring that the die only produces 6s.
In our experiments, we observe each question 20-30 times minimum. "Cited 15 out of 30 times (50%)" versus "cited 1 out of 30 times (3%)" are fundamentally different outcomes. A single observation cannot distinguish between them.
Three Requirements for Real Measurement
1. User-Centric Question Space
Design queries from the user's perspective, not your marketing team's. Use Search Console data, social media questions, and sales inquiries as source material. Include industry and problem layers, not just brand queries.
2. Statistically Significant Sample Size
At least 20-30 observations per query to smooth out AI response variance. 10 queries times 30 observations = 300 minimum. "Asking once and taking a screenshot" is not measurement -- it is anecdote.
3. Before/After Experimental Design
Take baseline measurements before any intervention. Execute one change at a time. Remeasure under the same conditions 2-4 weeks later. Running multiple interventions simultaneously makes attribution impossible.
Without these three elements, AEO measurement is confirmation bias -- picking favorable queries, asking once, and calling it success.
We Made This Mistake Too
We hypothesized that rewriting our landing page copy would improve citation rates. We rewrote the entire LP based on GPR experiment data. The before/after result: citation rate 9.1% to 9.1%. No change at all.
We tested 11 hypotheses. 9 were rejected. What remained: mismatched title tags and insufficient trust signals for Google. The simplest, lowest-cost root causes.
We only learned this because we measured. Without measurement, we would have concluded "the LP rewrite must have helped" and moved on.
We Formalized This Problem Academically
We formalized this "measuring without defining what you're measuring" problem in an academic paper. Existing AEO measurement tools lack a defined estimand -- the quantity being estimated. They produce numbers without specifying what those numbers represent.
Our proposed framework, intent-conditional estimation, estimates AI citation probability per user intent cluster and builds visibility measurement across the full question space with statistical rigor.
The paper will be published on SSRN. A link will be added here upon publication.
What to Check
If your company is running AEO initiatives, verify the following:
- Who created the measurement queries? If it was your marketing team, replace them with user-perspective queries
- Are you only measuring brand queries (with your company name)? Check your citation rate on industry queries (without your name)
- Are you judging from a single observation? Run the same query multiple times and compute statistical citation rates
- Did you take a baseline before your last initiative? If not, take one now before your next one