AI Search Visibility
Does Personalization Change AI Answers? Track Brands
Two people can get different brand recommendations from one AI engine. Here is why, and how to track visibility with repeated sampling.
Sheetal Shakshi3 Nov 202614 min read
Yes. ChatGPT and Google's AI Mode can personalize answers using memory, history and context, and public research shows brand lists also change from run to run. Track how often your brand appears across many runs, per engine and prompt wording, rather than one rank. Noma by Nurdd tracks 13 AI engines, polled daily, without promising a fixed ranking.
Yes, personalization in LLM responses is real, but it is only one of several reasons two people see different brands from the same AI engine. ChatGPT has a memory feature, Google's AI Mode can draw on past activity, and public research shows brand lists also change between identical runs. Tracking therefore means sampling repeatedly, not trusting one screenshot.
This guide separates the causes of variation, shows what public evidence says about each, and gives a repeatable method for measuring brand visibility when no two answers are identical. It also shows how Noma by Nurdd approaches the problem without promising a single fixed ranking. If you are new to the field, start with what generative engine optimization is.
Key takeaways
- Treat an AI answer as a distribution, not a position. Measure how often your brand appears across many runs, not whether it appeared once.
- Personalization is one source of variation. Run-to-run differences exist even for identical prompts.
- Track several wordings of each buying question, because phrasing changes which brands are named.
- Report a range (for example, appeared in 55 to 75 percent of runs), never a single rank, and compare trends over weeks.
- Check your own account's memory and history settings before you test, so your personal history does not skew what you see.
- Use a tool that polls on a schedule across engines, then judge the direction of the line, not any one day.
Do ChatGPT answers differ from person to person?
They can. OpenAI's help center has pages on Memory in ChatGPT and on temporary chat, which show that ChatGPT keeps account-level context that users can manage. Two accounts can therefore start from different stored context. Check the current wording of those pages before you test, because the options change.
A person who has told an assistant they prefer budget products may be shown a different shortlist than a new user asking the identical question. The brand shown is then a function of the person as well as the question and the web. Brand teams cannot see that memory layer from outside.
Does Google's AI Mode personalize answers too?
Yes, for users who opt in. Google Search Help says Personal Intelligence in AI Mode references previous searches and activity saved in a user's Search Services History to tailor suggestions. It also says personalization applies when history and personalized recommendations are enabled, and that AI Mode might not always get personalization right.
So on at least two major engines, the vendors themselves document a personal layer on top of the question. Whether a given user has it switched on varies, which is one more reason two people can see different brands. For Google-specific monitoring, see how to track Google AI Overviews.
What does personalization in LLM responses actually mean?
Personalization in LLM responses is any way an AI assistant adjusts an answer using information about the person asking, rather than only the text of the question. The information can be stored by the product (memory or history), supplied by the user (instructions or preferences), or inferred at request time (location, language, current conversation).
If the terms are new, the AI search glossary defines them. This guide separates five sources of variation in an AI answer: account memory, location and language, run-to-run variation, prompt wording, and engine and retrieval differences. Separating them matters because each needs a different response: you can reduce some, average out others and only monitor the rest.
The five sources of variation
| Source | What changes | Can you control it when testing? | What to do |
|---|---|---|---|
| Account memory and history | Saved memories and inferred preferences shape the shortlist | Partly: use a clean or temporary session | Test from a neutral session and note settings |
| Location and language | Local brands, availability, currency | Partly | Track the markets you sell in, separately |
| Run-to-run variation | List, order and length differ between identical runs | No | Repeat each prompt many times |
| Prompt wording | Different phrasing triggers different brands | Yes | Track a set of paraphrases per question |
| Engine and retrieval | Each engine uses different sources and updates | Yes | Measure each engine on its own |
Is the variation just personalization, or are AI answers inconsistent anyway?
The strongest public evidence suggests variation is large even for identical prompts. Rand Fishkin of SparkToro and Patrick O'Donnell of Gumshoe.ai studied whether AI brand recommendations repeat. Search Engine Journal, early 2026, reports an under-1-in-100 chance that ChatGPT or Google's AI in Search (AI Overviews and AI Mode) would return the same list of brands across repeated runs of the same prompt.
The same article says the team ran 2,961 prompts across ChatGPT, Claude and Google Search AI Overviews, using hundreds of volunteers over November and December. It adds that nearly every response was unique in three ways: the list of brands, the order of recommendations, and the number of items returned.
In the SparkToro and Gumshoe.ai study, identical brand-recommendation prompts almost never returned the same list of brands twice, and the order and number of items also changed.
Two limits apply. Search Engine Journal notes that Gumshoe.ai sells AI tracking tools, so read the framing with that in mind. And the findings describe how much answers varied, not why, so treat them as evidence that variation is normal, not as a measure of how much comes from personalization. The safe reading is that a single response is a weak measurement.
Why this breaks single-screenshot checks
If one person checks a question like best CRM for small teams once and sees their brand in third place, they have one sample from a wide distribution. A colleague checking an hour later may see it absent. Neither observation is wrong, and neither is a ranking.
How do you track brand visibility if AI answers are personalized?
You track the rate at which your brand appears across many independent runs, per engine and per prompt, and watch that rate over time. This replaces a rank with a frequency, which is the quantity that still makes sense when answers differ by person and by run.
Noma is a web dashboard from Nurdd that tracks whether a brand appears in answers from 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, polled daily. It reports citation frequency by category with competitor comparison and recommendations, rather than a fixed position.
What is the step-by-step method for tracking brands in personalized AI answers?
Use these steps whether you build it by hand or alongside a tool. Each one states the action, the result and the reason.
- Write the questions buyers ask. List 10 to 20 category questions, such as which project tool suits a ten-person agency. Result: a prompt set tied to buying intent. Why: visibility on questions nobody asks does not matter. The guide to which prompts to track covers how to choose them.
- Write three to five paraphrases of each. Change length, specificity and persona (for a startup, for an enterprise). Result: a wording spread. Why: phrasing changes which brands are named, so one wording overstates or understates you.
- Set a neutral testing session. Use a logged-out session or a temporary chat where the engine offers one, and record your memory and history settings if you use an account. Result: a baseline with less personal skew. Why: removing stored context makes your own account less likely to flatter or hide your brand.
- Repeat every prompt many times per engine. Record whether your brand appears, and whether it is cited as a source. Result: appearance counts. Why: run-to-run variation only averages out with repetition.
- Segment by market and persona. Run the set again with a location or buyer context that reflects a real segment. Result: a table of visibility by segment. Why: this is a deliberate, controlled version of personalization.
- Report a rate with a range. State appeared in X of N runs with the uncertainty. Result: a number you can defend. Why: it prevents false confidence from small samples.
- Trend it weekly. Compare rates by week and engine, and flag sustained moves only. Result: direction of travel. Why: day-to-day noise is expected; a sustained change is a signal.
- Act on causes, not on ranks. Pair each weak prompt with a content or citation fix. Result: a work list. Why: visibility improves through sources and clarity, which the AI search visibility analysis guide explains.
How many runs do you need? A worked example
The figures below are an illustration of arithmetic, not measured data. Suppose a brand appears in 14 of 20 runs of one prompt, a 70 percent appearance rate. A standard approximation for a 95 percent range on a proportion is 1.96 times the square root of p times (1 minus p) divided by n, where p is the rate and n is the number of runs.
| Runs (n) | Appearances | Rate | Approx. 95% range |
|---|---|---|---|
| 10 | 7 | 70% | about plus or minus 28 points |
| 20 | 14 | 70% | about plus or minus 20 points |
| 60 | 42 | 70% | about plus or minus 12 points |
| 200 | 140 | 70% | about plus or minus 6 points |
With 20 runs, the 70 percent rate could plausibly sit anywhere from roughly 50 to 90 percent. That is why a drop from 70 to 60 percent on a small sample is not evidence of anything. More runs narrow the range, which is why measurement should be repeated and not taken once.
Compare competitors the same way. If your brand is at 55 percent and a rival is at 60 percent on a small sample, the honest statement is that they overlap. Claim a lead only when the ranges clearly separate.
What should you do about each source of variation?
Different causes call for different responses, and mixing them up wastes effort. This decision table maps each cause to an action and to the evidence you should keep.
| If you see this | Likely cause | Action | Evidence to keep |
|---|---|---|---|
| Brand appears for you but not a colleague | Account memory or history | Retest from a neutral session | Settings used and run count |
| Brand appears in one run, then vanishes | Run-to-run variation | Repeat the prompt until the range narrows | Appearances out of runs |
| Brand appears only for detailed questions | Prompt wording | Publish pages for the specific use cases | Wording variants tested |
| Brand appears on one engine only | Engine and retrieval differences | Check which sources that engine cites | Per-engine rates |
| Brand appears in one region only | Location signals | Add market-specific pages and details | Per-market rates |
The table is a working framework from this guide, not a finding from a study. It is meant to keep a team from reacting to noise as if it were a trend.
For ecommerce teams, location and feed details matter more than most, and the guide to ChatGPT shopping optimization in the US and Europe shows how setup differs by market. Brands that want to understand why a competitor keeps appearing can read why a competitor is cited and you are not.
Why does prompt wording change which brands appear?
Engines build an answer from the question as phrased, so extra details such as budget, team size, region or use case change which brands fit. A broad question tends to surface category leaders; a specific one can surface niche fits.
A practical test using only public tools: take one buying question and write it five ways, from a bare query to a detailed brief. Run each several times in the same engine and record which brands appear. If your brand shows up only in the detailed versions, your visibility depends on a specific use case, which is a content gap to fix, not a ranking problem.
A wording ladder you can copy
| Level | Pattern | Example shape |
|---|---|---|
| 1 | Bare category | best [category] tools |
| 2 | Add a buyer type | best [category] tools for [team type] |
| 3 | Add a constraint | best [category] tools for [team type] on a limited budget |
| 4 | Add a market | best [category] tools for [team type] in [country] |
| 5 | Full brief | a paragraph describing needs, size, region and integrations |
Record your appearance rate at each level. A brand that appears at level 1 but fades by level 3 has a positioning problem with specific buyers. The sentence structure that gets cited and the guide on writing website content AI engines quote show how to make a page match specific phrasings.
Which controls reduce personalization when you test?
You cannot remove personalization for real users, but you can limit it in your own tests:
- ChatGPT memory: the OpenAI Help Center page on Memory covers managing ChatGPT's memory, so review it before you test and record what you chose.
- Temporary chat: OpenAI has a temporary chat help page. Read its current wording on what a temporary chat does and does not carry over, and check that each time.
- Google AI Mode: Google Search Help ties personalization to history and personalized recommendations being enabled, so test with them off, or logged out, and note which you used.
- Logged-out sessions: they carry no account history, though location and run-to-run variation still apply.
Record the settings beside every result, so a later reader knows which conditions produced which rate. Also test the surface your customers use, since behaviour in a custom or embedded assistant may differ from the main chat.
How does Noma handle personalized and variable AI answers?
Noma by Nurdd tracks a brand's presence across 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, polled daily. It shows citation frequency by category, a competitor comparison and recommendations to improve AI visibility. Nothing in that approach depends on one fixed ranking.
Noma does not claim to reproduce what any one named person sees, since no outside tool can read a user's private memory. It gives a comparable, scheduled measurement of how often a brand is cited, which is the number a marketing team can act on.
Book a demo at nurdd.club/demo to see Noma on your own brand.
Noma has a 3-day free trial. A card is taken at signup and billing starts on day 3; cancel before then and you are not charged. See the product at nomahq.app.
What a demo shows for your own brand
- Citation frequency for your brand by category.
- A competitor comparison for the same category.
- Recommendations to improve your AI visibility.
How do manual checks, spreadsheets and monitoring tools compare?
| Method | Handles variation | Scheduling | Covers many engines | Effort |
|---|---|---|---|---|
| One-off manual check | No | None | Only what you open | Low, but unreliable |
| Spreadsheet with repeated runs | Yes, if disciplined | Manual | Slow to extend | High and ongoing |
| Noma by Nurdd | Scheduled, comparable measurement, no fixed rank | Polled daily | 13 AI engines | Dashboard review |
For a deeper tool-by-tool view, see the AI search visibility tools comparison and the tracking tool review checklist.
What are the common mistakes when tracking personalized AI answers?
- Treating one screenshot as a rank. The research above shows identical prompts rarely return the same list.
- Testing only from your own logged-in account. Your history reflects your company's interests, which can flatter your own brand.
- Tracking a single phrasing. You measure the wording, not the topic.
- Merging all engines into one number. Engines draw on different sources. The engine comparison explains why they diverge.
- Reacting to daily swings. Use weekly trends, as in tracking a visibility score week over week.
- Ignoring how brands are described. Being named is only part of the picture, so read the wording engines use about you.
Checklist: a weekly routine for variable AI answers
- Confirm your prompt set still reflects real buyer questions.
- Check that every prompt has at least three wordings.
- Review appearance rate by engine, with the run count beside it.
- Mark changes only if they persist for two or more weeks.
- Compare against two or three named competitors using the same prompts.
- Note any new sources being cited and whether your pages are among them.
- Assign one content or citation fix per weak prompt.
- Use the GEO readiness scorecard to check the page-level basics.
Does this change how you think about GEO overall?
It shifts the goal from rank first to being in the set of brands engines consider, often and for the right questions. Content that states clearly what a brand is, who it serves and why gives engines a stable description to reuse; see writing a brand description for AI search.
It also means share of voice is a rate, not a place. The guide on AI search share of voice explains how to calculate it. Teams who prefer help can look at the generative engine optimization services from Nurdd, measured in Noma.
How we researched this
We reviewed Google's Search Help page on AI Mode personalization, a Search Engine Journal report on the SparkToro and Gumshoe.ai consistency study, Noma's public home page, and OpenAI's help center page titles for memory and temporary chat. The OpenAI pages could not be opened during research, so we do not quote their details; check them directly. We did not run hands-on tests, interviews or customer studies for this article, and the sampling table is arithmetic, not measured data. Engine behaviour changes often, so recheck settings before you test.
Frequently asked questions
Do ChatGPT answers differ from person to person?
Yes, they can. OpenAI offers a memory feature with user-managed settings, so accounts can start from different stored context. Answers also vary between identical runs, so two people asking the same question will rarely see identical brand lists.
How do you track brand visibility if AI answers are personalized?
Track the rate your brand appears across many runs, per engine and per prompt wording, and watch the trend over weeks. Use a neutral test session, add segment-specific prompts for your real markets, and report a range instead of a rank. Noma by Nurdd tracks 13 AI engines, polled daily, and shows citation frequency by category with competitor comparison.
Can I see what personalized answer a specific customer received?
No. A customer's memory and history are private, and no outside tool can read them. You can approximate segments by testing prompts that include the context that segment would give, such as location or role, and compare the visibility rates.
Does turning off memory make results objective?
It makes them less personal, not fixed. Variation between runs remains: Search Engine Journal reports an under-1-in-100 chance of the same brand list across repeated runs of a prompt in the SparkToro and Gumshoe.ai study.
How many times should I run each prompt?
More is better. The illustration above shows a range of about plus or minus 20 points at 20 runs and about 12 points at 60. Pick a count that makes your decisions safe, and combine runs across days rather than relying on one session.
Is a brand's AI ranking meaningful at all?
Position on a single response is a weak signal. Search Engine Journal reports that the order of recommendations varied between runs, along with the list and the number of items. Appearance frequency, citation frequency and trend over many runs are better measures.
Does Noma promise a fixed ranking in AI answers?
No. Noma shows citation frequency by category across 13 AI engines, polled daily, with competitor comparison and recommendations. It does not promise a single position, because AI engines do not produce one.
Next step
AI answers vary by person, by wording and by run, so the useful question is how often your brand appears, not where it sat once. Book a demo at nurdd.club/demo to see citation frequency, competitor comparison and recommendations for your own brand in Noma.