We asked Google's AI about our product. It described a different company, confidently.

Not a hallucination about something obscure and harmless. It said the product was a Google feature, and that it uploads your source code to Google's servers. The product is neither of those things, and the second claim is the exact opposite of what it promises.

What we did

We took a small, real, obscure software product with a public website, RunBoth at runboth.dev, and wrote down ten factual questions a customer might ask. The answers are all published on its own pages, so there is a right answer to every one.

We asked each question twice, of two frontier models. First from memory, with no tools and an explicit instruction that saying "I do not know" is better than guessing. Then again, with the site's own agent endpoint attached so the model could ask the site directly. A third model graded every answer against the site's own page text.

The numbers

claude-sonnet-5 and gemini-2.5-flash, ten questions each, 15 September 2026. Claude with the endpoint attached scored 8 of 10 with nothing invented. One question in the Claude run failed to grade and is excluded from its totals rather than counted as a pass.

The six false claims, verbatim

Every one of these was stated from memory, without hedging, about a real product:

• RunBoth is a competitive programming judge, returning verdicts like Time Limit Exceeded and Wrong Answer
• RunBoth is a Google Gemini feature
• Your code is sent to Google's backend servers for execution
• It uses an AI model to generate explanations
• An AI model processes your code and output to produce natural language responses
• It returns Accepted, Memory Limit Exceeded and Runtime Error verdicts

RunBoth is a differential execution tool for Python. It runs the old and new versions of changed code on generated inputs and compares seven observation channels. It makes no network calls at all and uses no AI model to decide anything.

Be fair about it

• Gemini declined eight of the ten questions. The six false claims came from the two it answered. It is not inventing constantly; it is inventing occasionally and confidently, which is worse for the business being described.
• Claude did not invent anything. It said it did not know, nine times. For a business owner that is a different problem, not a smaller one: the AI simply has nothing to say about you.
• One model, one prompt style, one small obscure product. This is a measurement, not a law. The method is published so you can disagree with it by running it.
• We did not measure search rankings or AI overview placement, and we make no claim about either.

The fix, and why it is boring

The model was not lying on purpose. It had no way to check. So we gave the site an endpoint the model could ask, using the Model Context Protocol, which is the protocol these models already speak. The site's own pages went in; the answers come out with the page they came from. Gemini went from six false claims to none, and from zero correct answers to nine.

There is nothing clever in it. The site publishes an endpoint, the endpoint answers from the site's own pages, and the model asks instead of remembering.

Run it on your own site

The harness and the whole install are open source, Apache 2.0, at github.com/KyleClouthier/agentweb. Write ten questions about your business, list the true answers from your own pages, and run it. It takes about ten minutes and costs a few cents in model calls.

If you would rather we did it for you and showed you the report, that is the pilot. If you just want to see the machinery working, two real sites are live and every agent call they receive is published on the live page.