Marketing AI Vendor Assessment: What Analysts Don't Test

A marketing AI vendor assessment from an analyst firm answers one question well: which vendors are credible in a defined market, judged against the analyst's criteria. It cannot answer whether a tool names your brand in the answers your buyers read, against the competitors you actually face, in a report your boss will accept. That part is yours to test.

This post explains how to read an analyst assessment, what the major firms say about their own methods, and the hands-on tests that close the gaps. It uses the September 2026 IDC MarketScape on agentic marketing, which Profound announced it appeared in, as a worked case of what a report covers and what it leaves to you. If you are new to the field, start with what generative engine optimization is, and check your site first with the free GEO readiness scorecard.

Key takeaways

• Treat an analyst report as a shortlisting filter. It tells you who was in scope and how they scored, not how a tool performs in your category.
• Check the market definition first. A report on enterprise agentic marketing platforms is not a test of AI search visibility trackers, even when a vendor appears in both conversations.
• Read the firm's published method and inclusion rules before the chart, and check whether a missing vendor was out of scope or declined to take part.
• Run three tests the analysts cannot: your own prompts, your own competitor set and your own reporting output.
• Repeat every prompt several times. BrandJet's testing guide cites research reporting substantial variability in repeated identical prompts, so a single run proves little.
• Use a trial as evidence, not a demo: write your pass criteria before you sign up, and score every tool against the same sheet.

What does an analyst vendor assessment actually tell you?

An analyst vendor assessment tells you how a research firm scored a set of vendors against a market definition it wrote. The score reflects the firm's criteria, the vendors that were included, and the evidence submitted. It is a credible starting point and a poor final decision.

The IDC report is a useful example. Its abstract describes an evaluation of agentic marketing platforms available to enterprise B2C organisations, and its "Companies Covered" list names 19 companies, among them Writer, Braze, Oracle, SAP, Salesforce, Adobe, Microsoft and Profound (IDC, September 2026). SAP announced it as a Leader (SAP, September 30, 2026), and Profound announced a Major Player placement (Profound, 2026).

Notice what the abstract says about its own scope. IDC found that autonomy and system integration are largely solved in this market, while independent judgment and cross-agent coordination remain works in progress. Its research director advised buyers to scrutinise vendors on interoperability, governance and road maps, among other differentiators (IDC, September 2026). That is guidance to run your own checks, and it is the premise of this post.

How do Gartner, Forrester and IDC build their reports?

The three firms use different formats but share a structure: define a market, set criteria, collect vendor evidence, score, and publish a graphic. Knowing each firm's published rules tells you what the graphic can and cannot mean.

Rule of thumb: an analyst graphic compares vendors against each other on the analyst's criteria. It does not compare any vendor against your needs.

None of these methods involves running your prompts, naming your rivals or producing your monthly report. That is not a flaw; it is outside what a market-level study can do for thousands of buyers at once.

Why is a market-level assessment not enough for an AI marketing purchase?

A market-level assessment is not enough because AI marketing software is judged on outputs that depend on your brand, category and audience. Two buyers can read the same chart and need opposite tools.

Three gaps recur. First, scope: the IDC study evaluates agentic marketing platforms for enterprise B2C teams, which is a different job from checking whether ChatGPT or Perplexity name your brand. Second, specificity: a chart scores general capability, while your result depends on your category's prompts and which sources AI engines trust there. Third, output fit: a platform can be strong on capability and still produce reporting your finance lead cannot use.

There is also the question of who publishes the news. Vendors announce favourable placements on their own sites, as both SAP and Profound did, which is reasonable but means the headline you read first is the vendor's framing. Go to the analyst's own abstract, then to the criteria, before you act on a placement label.

How do I read a placement announcement?

Read a placement announcement as a pointer to the original report, not as the finding. Work through these questions in order.

1. Find the original. Locate the analyst's own page, as with the IDC abstract above, and note the date.
2. Read the market definition. Ask whether it describes the job you are buying for.
3. Check the companies covered. See who was in scope and whether the vendors you care about are listed, and whether the segment (enterprise, B2C) matches yours.
4. Read the criteria and weights. If a criterion you care about, such as reporting or governance, is not scored, you must test it yourself.
5. Note what you could not read. Full reports often sit behind access; do not infer scores you have not seen.

How should I evaluate an AI marketing vendor beyond analyst reports?

Evaluate a vendor with a repeatable test you write before you see any demo: define your prompts, your competitor set, your reporting needs, and the evidence that counts as a pass. Then run every shortlisted vendor through the same sheet.

The decision table below maps each analyst blind spot to the test that fills it.

What should I test in a trial of an AI visibility tool?

Test five things in a trial: prompt fidelity, competitor coverage, repeatability, citation evidence and report usability. Each is quick to set up once you have prepared the inputs, and together they cover what analyst reports cannot.

1. Your prompts, not the vendor's defaults

Default sample prompts may not reflect your buyers, so replace them with your own. The Omnia knowledge base describes how wording changes outcomes: synonyms such as "alternatives" or "vs", audience framing such as "for startups", and constraint prompts such as "works with HubSpot" can each shift which brands appear (Omnia, knowledge base). Build clusters by intent, not single hero queries. For help choosing, see which prompts to track for AI search visibility and how many prompts you need.

2. Your competitor set

A trial is only informative if the comparison reflects your market. Add the rivals who show up in deals, the marketplaces and publishers that compete for the same answers, and one or two newer entrants. Then check whether the tool lets you see a side-by-side view for each prompt cluster, which is the basis of AI search share of voice.

3. Repeatability and variance

AI answers change between runs. BrandJet's guide to testing AI visibility tools cites researchers who ran identical prompts repeatedly and found substantial variability even under closely controlled timing (BrandJet, 2026). We have not traced the original study, so treat that as BrandJet's account. The practical consequence is simple: a vendor that shows you one score from one run is showing you a sample. Ask how many runs sit behind each number and whether you can see them.

4. Raw answers and citations

The same BrandJet guide lists a question worth copying: can you inspect the raw answer? Without it you cannot tell an extraction error from real model variability. In your trial, open the raw answer for at least ten prompts and confirm the tool's reading (mentioned, cited, position) matches what you see.

5. The report your stakeholders will actually read

Build the monthly report during the trial, not after purchase. Leaving reporting to the end means you find reporting gaps after you have bought. A template to start from is in AI search metrics: how to build a monthly report, and agency teams can adapt AI search reporting for agencies.

How do I score vendors fairly? A worked example

Score every vendor on the same weighted sheet, set before the trial. The arithmetic below is an illustration of the method with made-up scores; none of these numbers is data about any real vendor.

Suppose your team weights the five trial tests as follows: prompt fidelity 25, competitor coverage 20, repeatability 20, citation evidence 20, report usability 15, totalling 100. Each vendor is scored 1 to 5 on each test.

Each weighted figure is score divided by 5, multiplied by the weight. Vendor A totals 83 and Vendor B 78. The point is not the winner but that the decision rests on criteria you chose, which is the one thing an analyst chart cannot do for you. If your priorities differ, change the weights first, then score.

A checklist for a trial you can defend

1. Write the pass criteria first. List the five tests, the weights and what counts as a 5 and a 1, before any vendor sees them.
2. Prepare 25 to 50 prompts. Draw them from sales calls, support tickets and search queries, and group them by intent.
3. List five to eight competitors. Include deal-loss rivals and publishers or marketplaces that take the same answers.
4. Run the same inputs in every tool. Different inputs make scores incomparable.
5. Repeat on several days. Note the date, the engine and the answer each time.
6. Open ten raw answers. Confirm the tool's reading matches the text.
7. Check the source list. Pick answers where a rival is named and you are not, and see which pages were cited.
8. Build the report. Send it to a colleague outside marketing and ask what they understood.
9. Ask the governance questions in writing. Use the vendor questionnaire.
10. Score, then decide. Calculate the weighted totals before discussing price.

What are the trade-offs of relying on analyst reports, and of running your own tests?

Analyst reports are cheap shortcuts with a built-in scope limit; your own tests are specific but small and take time. Using both is the sensible position, with the report narrowing the field and the test deciding.

The sample-size limit on your own test is real. Thirty prompts on one day is a snapshot. That is why the checklist repeats runs, and why you should treat your trial as evidence for a decision, not as a benchmark to publish.

How does a Noma trial fit as one of these tests?

Noma is built for the three gaps analysts leave: your prompts, your competitors and your reporting. It is one test among several, and you should run it alongside any other tool on your shortlist using the same sheet.

Noma by Nurdd is an AI search visibility dashboard that tracks whether a brand appears in answers from 13 AI engines, including ChatGPT, Gemini, Perplexity and Claude, polled daily. It shows citation frequency by category, competitor comparison and recommendations for improving visibility, so a team can see whether its brand is named and which pages the engines cite. Noma has a 3-day free trial at nomahq.app.

Book a demo at https://www.nurdd.club/demo to see Noma's competitor comparison and citation views set up for your own brand.

Trial terms to plan around

A card is taken at signup and billing starts on day 3, with no charge if you cancel before then. With three days, prepare the prompts and competitor list beforehand so the trial clock covers measurement, not setup. Because answers vary between runs, use the three days for repeated runs of your core cluster and for building the report, and treat the result as one input to your weighted sheet.

If you are comparing Noma with Profound specifically, the Noma vs Profound page lays out the published differences, and the same trial sheet works for both tools. Broader shortlists are in best AI search monitoring tools.

What alternatives exist if I cannot run a trial myself?

If you lack time or analyst capacity, there are three alternatives, each with trade-offs. None replaces seeing your own data.

• A scoped pilot with a services partner. A managed team runs the prompts and reports monthly. Nurdd's answer engine optimization agency service works this way, measured in Noma.
• A structured experiment. Change one page, track a prompt cluster before and after. See GEO experiment design and the four-week ChatGPT test plan.
• A review-based shortlist. Useful for narrowing, but see what to trust in AI search visibility software reviews before leaning on them.

For wider context on where agentic features fit in a marketing stack, AI agents for marketing explains the use cases, and how Nurdd works describes how the three Nurdd products divide the work.

What mistakes make vendor assessments unreliable?

• Reading a placement label as a score. "Leader" or "Major Player" describes position in a specific study, not fit for your use case.
• Ignoring the market definition. A study of enterprise agentic marketing platforms does not test AI search tracking depth.
• Testing with default sample prompts. They may not reflect your buyers, so replace them with your own.
• Judging from one run. Repeated-run research cited by BrandJet reports substantial variability.
• Leaving reporting to the end. You find reporting gaps after you have bought. Plan for measuring AI referral traffic in analytics too, so visibility links to visits.
• Skipping personalisation. Answers can differ by user; does personalisation change AI answers covers how to account for it.

How we researched this

We reviewed the analyst firms' own methodology pages (Gartner and Forrester), the IDC abstract for the 2026 agentic marketing MarketScape, vendor announcements of placements (SAP and Profound), and independent practitioner guides on testing AI visibility tools. Pages were checked on the publication date, November 2026. We did not read the full IDC report, which sits behind IDC's access. We have not hands-tested any vendor named here, and the scoring example is illustrative arithmetic, not data. Facts about Noma come from Nurdd's own product information.

Frequently asked questions

How should I evaluate an AI marketing vendor beyond analyst reports?

Write a weighted scoring sheet before any demo, then test every shortlisted vendor with your own prompts, your own competitor list and the report format your stakeholders need. Add written answers on security, data use and integration. Decide on weighted totals rather than placement labels.

What should I test in a trial of an AI visibility tool?

Test prompt fidelity, competitor coverage, repeatability across several days, citation evidence in the raw answers and report usability. Prepare inputs before the trial starts so a short window is spent on measurement. Score every tool against the same sheet.

Does being named a Leader or Major Player mean a tool is right for me?

No. A placement describes a vendor's position in one study against the analyst's criteria and market definition. The IDC study, for example, covers enterprise agentic marketing platforms for B2C organisations, so it speaks to that segment only.

Does a vendor missing from an analyst report mean it is weak?

Not necessarily. A vendor may have been out of scope or may have declined to take part; Forrester, for instance, has a participation policy for vendors that decline. Check the report's inclusion criteria and the firm's published method before drawing a conclusion.

Why do I need to repeat prompts when testing?

AI answers vary between runs. BrandJet's testing guide cites research reporting substantial variability when identical prompts are repeated, even under controlled timing. Repeating runs lets you see a range and avoid acting on noise.

How many prompts should a trial use?

Enough to cover each buyer intent cluster, commonly a few dozen for a first test. Start from real questions in sales and support conversations, not generic category terms. Our guide on how many prompts you need goes deeper.

Is the Noma trial really free?

Noma offers a 3-day free trial. A card is taken at signup and billing starts on day 3; if you cancel before then, you are not charged. Prices are in USD.

Can I use Noma if I already use another AI visibility tool?

Yes. Running two tools on identical prompts is a sound way to compare outputs against your scoring sheet. Keep inputs the same and compare raw answers, not only dashboard scores.

Conclusion

Analyst assessments are good at narrowing a crowded field and flagging governance questions, and weak at everything specific to your brand. Read the market definition and criteria, then settle the decision with a weighted trial on your own prompts, rivals and reports.

To put Noma on your scoring sheet, book a demo or start the 3-day trial at nomahq.app, and score it against every other tool on the same criteria.