01 The research
A benchmark sent AI shoppers to 555 business sites
If an AI agent cannot open your home page, nothing else you do for AI visibility is reachable.
At 43% of travel and mobility sites neither agent could read the home page, the worst of any sector measured.
Adrian de Valois-Franklin, Alex Bogdan, Anas Ibrahim, Ibrahim Shaer and Tamaz Andguladze published a merchant-readiness benchmark on arXiv on 12 October. On 30 September they ran two self-identified AI agents, a plain web client and a full browser, at 555 business websites named publicly as clients by two payment processors. Neither agent could read the home page at 105 of them, 18.9%.
Most sites were fine. About three quarters were readable to each agent and 391 were readable to both. The failures clustered by sector: 8.0% of sports and venue sites, 29.2% of retail, and 42.9% of travel and mobility, which is 15 of 35 and a small enough group that the authors flag the range as wide.
Not all of that is deliberate. The 18.9% counts every site neither agent could read, whatever the cause, and the number that answered both agents with a block, a challenge or an error is 76 sites, 13.7%. At 91 of the 119 sites that challenged an agent the authors could name the bot-management service from the response, and one service accounted for 61 of them.
The second half asks whether an agent can work out what a purchase costs. Across 330 sites assessed in detail, the all-in total was decidable at 18 and failed outright at 178, 137 of those because tax was not stated, was stated without a rate, or was left to checkout.
Then the detail that should bother anyone who has paid for AI-visibility work. Twelve sites published an llms.txt file, the file that tells AI systems how to read a site, while neither agent could read the home page at all. About a quarter of the sites publish one, and the authors treat it as a diagnostic rather than a score, keeping it outside the grade.
What to do about it
Fetch your own home page the way a self-identified AI agent does, or ask whoever maintains your site to do it. If it comes back a block, a challenge page or nothing, that is the limit on your AI visibility and no amount of content work raises it. Then ask the same question of the page that states your price and your cancellation terms. This bites hardest where the purchase carries a number and a condition: a private aviation charter operator quoting a trip, a luxury home builder quoting an all-in build cost, a law firm publishing a flat fee. If an agent cannot determine what the thing costs from your pages, it cannot put you in front of someone who gave it a budget.
Source arXiv, Adrian de Valois-Franklin, Alex Bogdan, Anas Ibrahim, Ibrahim Shaer and Tamaz Andguladze, “Get In, Decide, Prove: A Merchant-Side Readiness Benchmark for Agentic Commerce,” 12 October 2026 · arxiv.org/abs/2610.17378 (a preprint, not peer reviewed; access census of 555 sites observed 30 September 2026, declared assessment of 330 merchants observed 3 and 7 October, rubric version 0.1; the authors state the pilot “does not validate them or rank merchants” and that the weights “await validation against outcomes”, and the sample is a list of named payment-processor clients rather than a random draw)
02 The evidence
Businesses absent from what the assistant fetched were named 2.8%
Whether an assistant pulls up your own website during the answer is almost the whole of whether it names you.
The paper’s author works at the company whose customer data it analyses, and that company sells AI-visibility monitoring.
Benjamin Tannenbaum published a stage model of AI-search brand visibility on arXiv on 19 September. It analyses 34,960 answers to questions that never mentioned the business by name, across 2,854 distinct questions and 75 anonymised projects, on repeated ChatGPT and Gemini runs between June and September 2026.
The result is a four-rung ladder and the rungs are far apart. When neither the business name nor its own website appeared in what the engine fetched, it was named 2.8% of the time on ChatGPT and 3.8% on Gemini. With its own site among the pages cited, 49.0% and 58.4%. When the engine also typed the business name into one of its own follow-up searches, 91.4% and 100%.
Which questions trigger those follow-up searches is not random. In a clean subset of 90 real user questions, 78.3% of commercial questions produced a follow-up search against 3.6% of informational ones. The questions with money behind them are the ones where the engine goes looking.
Now the part that decides how much of this to believe. The author is at Aiso Boost Ltd, the data are that company’s own customer projects, and the company sells AI-visibility monitoring. The paper is open about the rest: it states that the model is “predictive and observational, not a causal description of proprietary engine internals”, and it reports its accuracy against a held-out last 30% of the data rather than against the data it was fitted to.
Forever Cited sells AI-visibility work and has the same reason to like this number. What makes it worth repeating is not who published it, but that the method is on the page and the limits are stated by the author rather than pulled out of him.
What to do about it
Ask the three questions a customer would ask before they knew your name, in ChatGPT and in Gemini, and watch what each assistant fetches before it answers. If your own domain never appears among the pages it pulls, the writing on those pages is not in the running, whatever it says. That is a different problem from ranking and it has a different fix: the page has to exist, be reachable, and match the question closely enough to be pulled for it. A surgical practice and an M&A advisor will discover this on completely different questions, so use your own.
Source arXiv, Benjamin Tannenbaum (Aiso Boost Ltd), “From Prompt to Recommendation: A Fitted Stage Model of Brand Visibility in AI Search,” 19 September 2026 · arxiv.org/abs/2609.23162 (a preprint, not peer reviewed, and a month old rather than this week; 34,960 unbranded prompt-engine observations across 2,854 prompts and 75 anonymised projects of the author’s own employer, June to September 2026, validated on a chronological holdout of the latest 30%; the author states the model is predictive and observational rather than causal)
03 The delegation
Eight AI models produced 258 unusable replies of 480
More than half the answers an AI agent hands back cannot be used by the person who asked for them.
One extra sentence in the request cut the failure rate from 52% to 43%, and a second model rewriting the reply took it to 36%.
Yilin Jiang, Fei Tan, Xuanyu Yin, Guanchu Wang, Xia Hu and Yutao Yue published this on arXiv on 14 October. They gave eight AI models from seven families the same four multi-step projects and scored one thing only: whether the reply handed back could be understood and used by the person who asked, who had not watched the work.
Of 480 replies, 258 failed, a rate of 54%. Seven of the eight failed on at least 48% of their replies, and the range ran from 30% for the best, GPT-5.6 Sol, to 82% for the worst, Claude Opus 5, which is the family of model this briefing is written with. Open-weight and closed models alike.
The failures were not wrong answers. Replies mostly pointed at material the requester had never seen, or used labels the model had invented during its own work: a file, a step or a category name that existed only inside the model’s process.
Two fixes worked and both are cheap. Stating explicitly what the reply had to contain cut the failure rate from 52% to 43%. Handing the reply to a second model to rewrite took it to 36%, which human annotators confirmed in a blind comparison.
The limits are stated on the page. The projects are scripted and synthetic rather than logged sessions of real people, and the automatic scorer stands in for one kind of reader: someone who knows the subject and has not seen the work. The authors name real sessions as the next step.
What to do about it
If you hand multi-step work to an AI agent, a research summary, a competitive scan, a first draft of a policy, say in the request what the reply must contain and what the reader will not have seen. The paper’s own fix is that plain. Then read the reply as the person who has to act on it rather than as the person who set the task, and strike every label the model invented along the way. An IT and cybersecurity firm delegating a vendor review and a franchise advisor delegating a market scan will both get back something that reads complete and names things nobody else can find.
Source arXiv, Yilin Jiang, Fei Tan, Xuanyu Yin, Guanchu Wang, Xia Hu and Yutao Yue, “Measuring and Reducing Unusable Replies of LLM Agents,” 14 October 2026 · arxiv.org/abs/2610.19659 (a preprint, not peer reviewed; eight models from seven families across four projects, 32 collaborations and 480 replies, with an automatic reader scorer validated against five human annotators; the authors state the collaborations are scripted and synthetic and that applying the measure to logged sessions of real users is the next step)
04 Who checks
Anthropic gave six staff from an outside lab internal access
The company selling the model is still the main source of the numbers you read about it.
Six people from the nonprofit METR now have Anthropic hardware, desk space and access to internal tools and systems.
Anthropic published this on 15 October, with a clarification added the following day. Six employees of METR, a nonprofit that evaluates AI models, have been given company hardware, office space, and access to Anthropic’s internal tools and systems. Anthropic says METR publishes its own findings.
Anthropic is building three tiers of evaluator access and says it hopes over time to have several organisations at the third. It has worked with METR on reviews of its own risk reports over the past year and says it is in early discussions with other organisations.
There is no deadline here and nothing a business owner has to do about it. It is on this list because it answers the question sitting underneath every other number in this briefing: who is permitted to check what an AI company says about its own product, and on whose terms.
Note the shape of it. The access is granted by the company being evaluated, in tiers that company defines, and it is a real improvement on the alternative, which is nothing at all. Both of those are true at once, and an arrangement can be the best available and still not be independence.
What to do about it
When anyone shows you a number for your own AI visibility, a score, a share, a mention rate, ask three things: who measured it, on what sample, and whether anyone outside the seller has checked the method. Those are the same three questions this briefing has to answer about itself, and the second story today is a live example of all three being awkward at the same time.
Source Anthropic, “An update on independent evaluators,” 15 October 2026, updated 16 October · anthropic.com/news/an-update-on-independent-evaluators (a first-party announcement with no sample, no deadline and no figure for how much of Anthropic’s internal work the arrangement covers)
