A retrieval-on versus retrieval-off test sounds like one of the cleanest experiments available in AI visibility.
If a brand appears only when retrieval is on, live search probably helped. If it survives when retrieval is off, the brand may already be familiar to the model.
A tool-style search payload returned an explicit error on one surface. That was inconvenient, but safe. The system told me that the requested contract was unsupported.
Fields such as enablesearch or websearch were accepted. The APIs returned HTTP 200 and produced normal-looking answers.
Had I launched the full collection, I could have saved those rows as retrieval-on observations. The final comparison might have looked methodical, balanced and completely wrong.
The failure exposed a distinction that AI visibility pipelines need to make explicit.
Observed retrieval state records what the collector can verify actually happened.
That is more honest than forcing the row into either the retrieval-on or retrieval-off group.
Requesting a specific language does not prove that the answer stayed in that language.
Requesting citations does not prove that the displayed URLs support the claims beside them.
HTTP 200 answers a narrow transport question: did the server successfully process the request well enough to return a response?
Some APIs reject unsupported parameters. That is the safest behavior because the failure is visible.
Others may accept, ignore or partially interpret fields they do not implement. Compatibility layers are especially risky because they may reproduce the shape of another provider’s API without reproducing every behavior behind it.
There is no universal field because provider implementations differ. But evidence can still be graded.
A documented retrieval mode accompanied by response fields specific to that mode.
A model can emit a familiar domain without fetching it in the current session. It can also produce a malformed or invented URL. A citation-like string proves that text was rendered. It does not automatically prove that live retrieval occurred.
When a provider exposes no independent signal, retrieval should remain unproven.
That may make the dataset smaller. It also prevents a configuration assumption from becoming a published finding.