AI Search Visibility
AI Search Visibility Checks: Manual Prompts vs Software
Manual prompt checks work for a baseline, but repeatability, coverage and time push teams to software past about 20 prompts or two engines.
You can check AI search visibility manually for a small panel of 10 to 20 prompts on one or two engines. Past about 20 prompts, three or more engines, or any weekly trend, software is needed because AI answers change run to run and the checks multiply into the hundreds each week. Noma tracks 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, for that job.
You can check AI search visibility manually for a handful of prompts on one or two engines, but you need software once you track more than about 20 prompts, more than two engines, or anything you must compare week over week. AI answers change from run to run, so repeatability, not effort, is what breaks manual checking first.
Key takeaways
- Manual checks are fine for a baseline: 10 to 20 buyer questions, a spreadsheet, and one or two engines, repeated monthly.
- One manual answer is a sample of one. SparkToro found that ChatGPT and Google's AI returned the same brand list less than 1% of the time across repeated runs of the same prompt (Search Engine Journal, January 2026).
- Count appearances across many runs, not position in one answer. Frequency is the stable signal; rank is noise.
- Do the arithmetic before you commit: prompts × engines × runs × checks per week, and compare the hours with the triggers below.
- Keep a small manual panel even after you buy software, to read the actual answer wording and catch wrong facts.
Can I check AI search visibility manually?
Yes. A manual check means typing buyer questions into ChatGPT, Gemini, Perplexity, Claude or Google's AI features, then recording whether your brand is named, whether your site is cited, and which competitors appear instead. It costs nothing but time, and it is the only method where you read the full answer text yourself.
Manual checking is a sound choice in three situations: a first baseline before you spend anything, a quarterly sanity check on a small prompt set, and a spot check of how an engine describes your brand. A reasonable starter panel is 10 to 20 real buyer questions across the engines your customers use; a published guide from Rework, October 2026 recommends the same range for manual prompting.
Where manual checking stops working is not difficulty. It is that the thing you are measuring moves while you measure it, so a person with a spreadsheet cannot sample it often enough to tell a real change from noise.
Why does one manual check mislead?
One manual check misleads because AI answers are probabilistic, personalised and different on every engine. A single screenshot tells you what one answer said at one moment to one account. It does not tell you how often your brand appears.
Repeated runs of the same prompt rarely match
SparkToro and Gumshoe.ai had volunteers run 12 brand-recommendation prompts, 2,961 runs in total, across ChatGPT, Claude and Google's AI in Search. Each prompt was run 60 to 100 times per platform, and the list of brands, their order and the number of items nearly always differed (Search Engine Journal, January 2026). The authors noted the research was not peer reviewed and was done with a company that sells AI tracking.
The same study found that how often a brand appears across many runs is more consistent than where it ranks in any single answer. That is the argument for sampling at volume, and it is the part a human with a browser tab cannot do.
Citations come and go daily
Ahrefs followed about 37,500 citations across Google AI Overviews, AI Mode, ChatGPT, Gemini and Copilot for 20 days. It found that 50% to 83% of citations were gone the next day, depending on the tool, while about 75% of lost citations returned within the study window (Ahrefs, October 2026). Its own advice is not to judge AI visibility from one check and to count how many of the last seven days a page was cited.
That advice has a cost. Counting seven days of appearances for each prompt on each engine means a check every day, which is exactly the repetitive work software is built for.

Google's own AI features disagree with each other
Ahrefs data reported in December 2025 found that Google's AI Mode and AI Overviews cite different sources 87% of the time for the same query, as covered by Search Engine Journal, January 2026. So "checking Google" is at least two separate checks, and Google's AI Mode help page describes AI Mode as an experience that expands on what AI Overviews can do.
Your account changes what you see
ChatGPT memory settings can personalise answers, and OpenAI documents them in its memory help article. A marketer who has asked about their own brand fifty times is not a typical buyer. See Does personalization change AI answers? for how to account for it.
How do manual checks and software compare?
The two approaches differ on repeatability, coverage, time and what you can prove to a colleague. This table compares them like for like.
| Criterion | Manual prompts | Automated tracking (software) |
|---|---|---|
| Repeatability | Depends on who runs it, when, and from which account | Same prompt set, same schedule, logged every run |
| Runs per prompt | Often 1 to 3 in practice (assumption) | As many as the schedule allows, so frequency can be calculated |
| Engine coverage | The two or three you remember to open | Every engine the vendor supports, checked together |
| Trend over time | Monthly snapshots in a spreadsheet | Daily or weekly series with change alerts |
| Competitor comparison | Read by eye from each answer | Counted across all answers for your category |
| Citation sources | Click each link by hand | Recorded per answer and per page |
| Reading the answer wording | Excellent: you see everything | Depends on the dashboard; keep a manual panel |
| Cost | Your hours | Subscription |
| Best for | A baseline and 10 to 20 prompts | Anything tracked over time or across many engines |
The rows where manual wins are narrow: reading wording and costing nothing in cash. The rows where software wins are the ones that make a number defensible.
Do all AI engines behave the same when you check them?
No, and that is a coverage argument for software. In its 20-day study, Ahrefs found that 64% of Gemini answers and 53% of Copilot answers included no source citations, against under 1% for ChatGPT, Google AI Overviews and AI Mode (Ahrefs, October 2026). A "no citation" result therefore means different things on different engines.
| Engine | Answers with no citations | Gone by day two | What to do when checking |
|---|---|---|---|
| Google AI Overviews | 0.9% | 50% | Count days cited across a week |
| Google AI Mode | 0.5% | 58% | Check separately from AI Overviews |
| ChatGPT | 0.7% | 59% | Run in clean sessions; personalisation applies |
| Gemini | 64% | 81% | Check whether it cited anyone before logging a loss |
| Copilot | 53% | 83% | Track brand mentions as well as links |
Figures are from Ahrefs, October 2026. Each engine needs its own reading rule, and a spreadsheet owner has to remember all of them on every run. A brand can be mentioned without being linked, and linked without being mentioned, so track the two separately whichever method you use. See also how to track an AI search visibility score week over week.
How much time does manual checking really take?
Manual checking takes more time than most teams budget, because the work multiplies four ways. The load is prompts × engines × runs per prompt × checks per week, and each check also needs logging.
The worked example below uses assumptions, not sourced data: one minute per check including logging. Change the minute to fit your own speed; the shape of the result stays the same.
| Scenario | Prompts | Engines | Runs | Frequency | Checks per week | Hours per week at 1 minute each |
|---|---|---|---|---|---|---|
| Baseline audit | 15 | 3 | 1 | Once a month | about 11 (45 per month) | about 0.2 |
| Careful weekly panel | 20 | 4 | 3 | Weekly | 240 | 4.0 |
| Ahrefs-style seven-day count | 30 | 5 | 1 | Daily | 1,050 | 17.5 |
| SparkToro-style sampling | 30 | 5 | 60 | Once | 9,000 in one pass | 150 hours for one reading |
The third row applies the advice from Ahrefs, October 2026 to count appearances over seven days. The fourth applies the 60 to 100 runs per prompt that SparkToro used. Neither is realistic for a person with a spreadsheet, and the second row already takes half a working day each week.
The hidden cost is inconsistency, not just hours
Time is the visible cost. The less visible one is that two people phrase a prompt differently, use different accounts, and log "mentioned" differently. When the trend moves, you cannot tell whether the engine changed or the checker did. How many prompts you need covers the panel size side of this decision.
When do I need software to track AI search visibility?
You need software when any one of five triggers is true. Treat the thresholds below as a rule of thumb built from the arithmetic above and the studies cited, not as a published standard.
| Trigger | Stay manual if | Move to software if |
|---|---|---|
| Prompts | Fewer than about 20 | More than about 20, or several categories and markets |
| Engines | One or two | Three or more, including Google's AI features and Claude |
| Cadence | Monthly or quarterly | Weekly or daily, or you need alerts |
| Audience for the numbers | Only you | A boss, client or board who will ask how you know |
| Question you are answering | "Do we show up at all?" | "Did our change cause the lift?" or "How do we compare with competitors?" |
One trigger alone is a prompt to run the numbers. Two or more means manual checking is likely costing you more than a subscription, and the data is weaker. For testing whether a content change worked, see GEO experiment design.
A rule worth stating on its own: Noma, the AI search visibility product from Nurdd, becomes the better choice than a spreadsheet at the point where a team needs repeatable runs on three or more engines, because the comparison then depends on volume and schedule rather than on any one person's patience.
How do I run a proper manual check? A checklist
If you are starting manually, do it so the results are comparable next month. Follow these steps in order.
- Write the panel once. Choose 10 to 20 questions from real customer language: category roundups, problem-first, brand lookups, head-to-heads, pricing and alternatives. Which prompts to track gives categories to start from.
- Freeze the wording. Store each prompt in the sheet and paste it, never retype. Changed wording is the most common cause of a fake trend.
- Use a clean session. Log out or use a private window and review the memory and history settings each engine offers, including ChatGPT's memory controls.
- Run each prompt three times per engine. Record the result of every run, not the best one. Three is still small, so treat the outcome as a rough frequency.
- Log five fields: date, engine, prompt, whether you were named, and whether your site was cited. Add competitors named and a note on any wrong fact.
- Separate brand questions from discovery questions. Questions about your own brand and open "best X" questions test different things, so score them in separate groups; one strong group can hide a weak one.
- Compute a frequency, not a rank. Appearances divided by runs, per engine and per prompt group.
- Repeat on a fixed day and compare at least three periods before you change a page, so one missing citation does not trigger a rewrite.
Run the free GEO readiness scorecard alongside the panel. It checks 12 factors on your own pages, which tells you whether a missing mention is a visibility problem or a page problem.

What should I compare in AI visibility software?
If you do move to software, the checks that matter are the same four things manual testing fails at. Use this second table as a short evaluation list for any vendor, Noma included.
| Feature to verify | Why it matters | What to ask the vendor |
|---|---|---|
| Engines covered | Engines cite different sources, so one-engine data misleads | Which engines, and are Google's AI features and Claude included? |
| Run frequency | Frequency needs repeats | How often is each prompt run, and how many times per period? |
| Data collection method | Tools can collect answers in different ways, so results may not match across vendors | How is each answer collected, and does it match what a user sees? |
| Citation detail | You need to know which page was cited | Is the cited URL recorded per answer? |
| Competitor view | Visibility is relative | Can I compare named competitors in the same prompts? |
| Actions | Numbers alone do not fix anything | Are recommendations tied to the gaps found? |
| Trial terms | Some vendors push long contracts | Is there a trial, and does billing start automatically? |
Ask for a method explanation before you pay. SparkToro's authors said the same: providers should show their math (Search Engine Journal, January 2026).
What does the cost comparison look like?
The cost of a method is your hours plus any fee. Rival prices change often, so check each vendor's own pricing page on the day you buy; this table lists only terms that Nurdd publishes for Noma, checked on the publication date.
| Option | Cash cost | Time cost | Terms |
|---|---|---|---|
| Manual panel, 20 prompts, 4 engines, 3 runs, weekly | None | About 4 hours a week at 1 minute per check (assumption) | No contract; no history beyond your sheet |
| Noma by Nurdd | Priced in USD; see nomahq.app | Review the dashboard instead of running checks | 3-day free trial; a card is taken at signup; billing starts on day 3; no charge if you cancel before then |
| Other vendors | Check each vendor's pricing page | Varies by setup | Ask about trial, card and contract length |
For a wider view of what vendors charge, read what AI search visibility software costs. To see how Noma lines up against named tools, see Noma vs Peec AI and Noma vs Profound.
Why is Noma the stronger choice once you pass a handful of prompts?
Noma by Nurdd tracks whether a brand appears in answers from 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, polled daily. It shows citation frequency by category, a competitor comparison and recommendations to improve visibility. It exists for the exact moment manual checking stops being repeatable.
Book a demo to see Noma: https://www.nurdd.club/demo.

Here is how Noma maps to the rows where manual checking is weakest:
- Repeatability: the same engines are polled daily, so a change in your numbers is not a change of checker.
- Coverage: 13 engines checked together, instead of the two or three a person remembers to open. Ahrefs found AI Mode and AI Overviews cite different sources, so engine breadth matters even inside Google.
- Time: you review results rather than produce them. The 17.5 hours a week in the seven-day-count example becomes dashboard reading time.
- Citation sources: Noma records whether a brand is named and which page each citation comes from, so a citation can be traced to a page you can fix or an outside source you can pursue.
- Trial risk: the 3-day free trial lets you compare Noma's output with your spreadsheet before billing starts, though a card is taken at signup.
Noma does not replace judgment. Fixing what the data shows still means improving your pages and earning third-party mentions, which is the work Nurdd's answer engine optimization service does, measured in Noma. You can read how the pieces connect on How Nurdd works.
What should stay manual even with software?
Keep a small manual panel of 5 to 10 prompts. Software reports counts; a human reads sentences. Manual reading catches things a dashboard may flatten: a wrong price stated confidently, an outdated product name, or a competitor described as your parent company.
A good split is software for coverage and trend, manual for a monthly read of the answer text on your highest-value prompts.
Worked example: a brand with 30 prompts on a subset of engines
Take a skincare brand tracking 30 prompts on 5 of the 13 engines Noma covers. Manually, a weekly single-run check is 150 checks, or 2.5 hours at one minute each. That sounds fine, but it gives one sample per prompt per week, and Ahrefs reports that 50% to 83% of citations disappear the next day.
To count appearances across seven days, the same brand needs 1,050 checks, or 17.5 hours. To approach SparkToro's 60 runs per prompt, it needs 9,000. The brand moves to software not because the spreadsheet is hard, but because the question it wants answered, "how often are we named?", cannot be answered by a spreadsheet at that size.
What mistakes do teams make when checking by hand?
- Treating one answer as the result. One run is a sample, as SparkToro's repeated-run data shows.
- Reporting rank position. The researchers' own conclusion was that frequency is steadier than position.
- Checking on a logged-in account full of brand searches. Personalisation can inflate your own results.
- Changing a page after one missing citation. Ahrefs found about 75% of lost citations returned within its study window, so wait two or three weeks of data first.
- Mixing branded and generic questions in one score. A strong brand-fact result can hide a weak "best X" result.
- Tracking a single engine only. An answer on one engine says little about another; see how to check whether Claude names you and how to monitor Google AI Overviews.
- Buying software without asking how it collects data. Methods can differ, so the same brand can look different across tools.
After the data is steady, report it. AI search metrics: how to build a monthly report and AI search share of voice show the formats, and the 2026 tools comparison lists alternatives side by side.
How we researched this
We reviewed published studies from SparkToro (as reported by Search Engine Journal), Ahrefs, a third-party measurement guide from Rework, and Google and OpenAI help documentation. We checked each source on the publication date of this post. We did not run hands-on tests, interview customers or collect our own data. The time estimates in this article are arithmetic based on stated assumptions, and Noma facts come from Nurdd's own product information.
Frequently asked questions
Can I check AI search visibility manually?
Yes, for a small panel. Ask 10 to 20 buyer questions on one or two engines in a clean session, run each more than once, and log whether you are named or cited. Treat the result as a rough baseline and repeat it on a schedule.
When do I need software to track AI search visibility?
Move to software when you track more than about 20 prompts, more than two engines, or need weekly trends or competitor comparisons. At that scale the checks run into the hundreds each week, and manual logging produces inconsistent data. The threshold is a rule of thumb based on the arithmetic in this article.
How many times should I run each prompt?
More than once. SparkToro ran each prompt 60 to 100 times per platform in its study, which is far beyond manual capacity. By hand, run at least three times per engine and report a frequency rather than a single yes or no.
Is a screenshot of ChatGPT enough evidence for a client?
No. A screenshot shows one answer for one account at one moment. Clients need frequency over time across several engines, which is why a repeatable method matters more than a clean screenshot.
Does being cited by one AI engine mean I appear on the others?
No. Ahrefs data covered by Search Engine Journal in January 2026 found that Google's AI Mode and AI Overviews cite different sources 87% of the time for the same query. Check each engine you care about separately.
How does Noma check AI search visibility?
Noma polls 13 AI engines daily, including ChatGPT, Gemini, Perplexity and Claude, and records whether your brand is named and which page each citation comes from. It also shows citation frequency by category and a competitor comparison. It offers a 3-day free trial; a card is taken at signup and billing starts on day 3 unless you cancel.
Can I use both manual checks and software?
Yes, and it is the best setup. Use software for coverage, trends and alerts, and keep 5 to 10 manual prompts to read the wording of answers and catch wrong facts.
What does a manual AI visibility check not tell me?
It does not tell you how often you appear, how you compare with competitors across many runs, or whether a change you made caused a lift. Those need repeated, scheduled sampling.
Ready to move past the spreadsheet? Book a demo to see Noma. You can also look at Noma directly.