How an AI agent reads a web page

A browser and an agent start the same way. Each sends a request for a URL, and your server answers with a document in HTML, the markup language web pages are written in. From there they part. The browser downloads the scripts, images and style sheets that the HTML points to, runs the scripts, and draws the result. A program can stop after the first answer and work with the HTML alone.

The llms.txt proposal describes what that HTML looks like from a language model's side: "An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise." The price a shopper sees in large type may sit deep inside the markup, or it may not be in the markup at all.

By raw HTML we mean the first response, exactly as the server sent it. By the rendered page we mean what exists after the browser has run the page's scripts: those scripts can add text, swap parts of the page and fetch data from elsewhere.

Google documents how its own crawler handles the gap. Its guide to JavaScript SEO basics, last updated 4 March 2026, says that "Google processes JavaScript web apps in three main phases: Crawling · Rendering · Indexing". Between crawling and rendering, the page waits in a queue: "The page may stay on this queue for a few seconds, but it can take longer than that. Once Google's resources allow, a headless Chromium renders the page and executes the JavaScript." Headless Chromium is a browser engine running without a screen.

The same guide describes pages where the first response holds none of the content. Some sites "use the app shell model where the initial HTML does not contain the actual content and Google needs to execute JavaScript before being able to see the actual page content that JavaScript generates." An app shell, in our words, is a near-empty page that a script fills in after loading.

On 22 October 2026 we fetched the raw HTML of contentsignals.org: 1,966 bytes, holding the title "Content Signals" and a single script tag, the HTML element that loads or holds code for the browser to run. All of the site's text is inside the script, so a client that does not run it gets the title and nothing else.

Google's guide also advises: "server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript." Server-side rendering means the server builds the full HTML before sending it, so the first response already holds the content.

To see whether AI agents run a page's scripts, we read the crawler documentation of three AI companies, OpenAI, Anthropic and Perplexity, on 22 October 2026. None of the three pages contains the word "JavaScript" or says whether its crawlers or agents render pages. The pages do not say the agents skip scripts either, so all a store can conclude is that it cannot count on them running.

The figure below shows one made-up product page both ways. As a person sees it, the page has a name, a price, a stock line, a size picker and a returns note printed on the photo. As a program that reads only the raw HTML gets it, the page is missing the price and the stock, because a script fills those in after loading, and nothing in the HTML carries the returns note, because it exists only as pixels in the image.

<title>Ridge trail shoe | Example store</title><h1>Ridge trail shoe</h1><img src="/img/ridge.jpg" alt="Ridge trail shoe, side view"><p class="price"><span id="price"></span></p><p class="stock"><span id="stock"></span></p><label>Size  <select name="size">    <option>40</option><option>41</option><option>42</option><option>43</option>    <option>44</option><option>45</option><option>46</option>  </select></label><script src="/app.js"></script>

<script type="application/ld+json">{  "@context": "https://schema.org",  "@type": "Product",  "name": "Ridge trail shoe",  "image": "https://example.com/img/ridge.jpg",  "offers": {    "@type": "Offer",    "price": "129.00",    "priceCurrency": "EUR",    "availability": "https://schema.org/InStock"  }}</script>

# Ridge trail shoe Price: €129.00In stockSizes: 40 to 46Free returns for 30 days.

The structured data and the Markdown version in that figure are both things a store adds on purpose; they are covered after the section on the accessibility tree.

There is a second model of the page, kept for assistive technology, the software some people use to have a page read aloud or to operate it without a mouse. WAI-ARIA 1.2, a Recommendation of 6 June 2023 from the W3C (the World Wide Web Consortium, which publishes web standards), defines this accessibility tree as a "Tree of accessible objects that represents the structure of the user interface (UI)", and says that "Each node in the accessibility tree represents an element in the UI as exposed through the accessibility API". A node carries a role, such as heading, and an accessible name: "the name of a user interface element".

How HTML elements turn into nodes is set out in HTML Accessibility API Mappings (HTML-AAM), a W3C Working Draft dated 21 October 2026. An h1 maps to the "heading role, with the aria-level property set to the number in the element's tag name." A select shown as a drop-down box maps to the combobox role and takes its name from its label. An image gets its name from its alt attribute.

<h1>Ridge trail shoe</h1><img src="/img/ridge.jpg" alt="Ridge trail shoe, side view"><label>Size  <select name="size">…</select></label>

In our reading, the tree is a fair test of whether the page says in text what it shows in pictures. An image takes its name from its alt text, so a product photo without one tells the tree little about what it shows. The returns note in the sample photo appears nowhere in the tree, for the same reason it is missing from the HTML.

We found no source that says a particular shopping agent reads the accessibility tree, so this article makes no such claim. What it is worth to agents, if anything, is unpublished.

Structured data is a block of facts in a fixed vocabulary that a program can read without guessing. For products the vocabulary is schema.org, at version 30.1 of 16 September 2026. It defines a Product as "Any offered product or service. For example: a pair of shoes; a concert ticket; the rental of a car; a haircut; or an episode of a TV show streamed online." A product's offers property points to an Offer, which carries the terms of sale; schema.org defines an Offer as "An offer to transfer some rights to an item or to provide a service". The property can also point to a Demand.

On a shop page, three of the Offer's properties carry what a shopper checks first, in our reading. price is "The offer price of a product". priceCurrency is the currency of that price, and schema.org asks for codes from ISO 4217, the international list of currency codes: "Use standard formats: ISO 4217 currency format, e.g. "USD"". availability says whether the item can be had, with values such as "In stock, Out of stock, Pre-order".

The usual way to put this on a page is JSON-LD. In our words, JSON-LD is JSON written so that it names its vocabulary, and the block goes inside a script tag, which here holds data. Google's introduction to structured data calls it "A JavaScript notation embedded in a <script> tag" and says "Google recommends using JSON-LD for structured data if your site's setup allows it". The tag sits in the HTML itself, so if the server puts it in the first response, a program that reads only raw HTML still gets it. That is why, on the sample page, the price is missing from the HTML's visible text but present in its structured data.

<script type="application/ld+json">{  "@context": "https://schema.org",  "@type": "Product",  "name": "Ridge trail shoe",  "image": "https://example.com/img/ridge.jpg",  "offers": {    "@type": "Offer",    "price": "129.00",    "priceCurrency": "EUR",    "availability": "https://schema.org/InStock"  }}</script>

The same Google page adds: "Google can read JSON-LD data when it is dynamically injected into the page's contents, such as by JavaScript code". That describes Google's own renderer. A block that a script adds after loading is in the same position as the price in the figure: there only for a reader that runs the script.

Google also documents two product markups, on its page on product structured data: product snippets "for product pages where people can't directly purchase the product", and merchant listings "for pages where customers can purchase products from you", which have "more options for specifying detailed product information, like apparel sizing, shipping details, and return policy information." Those are rules for Google's search features. Nothing on the page says how AI agents use either one.

Markdown is plain text with light marks for headings, lists and links. A page in Markdown has no navigation, no scripts and no layout, which is what the llms.txt proposal is after. There are two ways to offer one, and they come from different places.

The first is the llms.txt proposal itself, titled "The /llms.txt file, v2", by Jeremy Howard, first published on 3 September 2024 and last modified on 10 August 2026. It is a proposal by its author, not a document of a standards body. It suggests "adding a /llms.txt markdown file to websites to provide LLM-friendly content": a file at the root of the site that "offers brief background information, guidance, and links to detailed markdown files." It also suggests a clean Markdown version of each page at the same address with .md appended, or with the extension replaced by .md, and a link to it from the HTML with rel="alternate" type="text/markdown".

The second is to let the client ask for Markdown at the page's own address. A request can carry an Accept header, the line where a client lists the formats it can take. Cloudflare's Markdown for Agents, documented as last updated on 9 October 2026, works on that header: "the client needs to add the Accept negotiation header with text/markdown as one of the options. Cloudflare will detect this, fetch the original HTML version from the origin, and convert it to Markdown before serving it to the client." It applies to sites where the feature is turned on. In Cloudflare's sample response the answer comes back as content-type: text/markdown; charset=utf-8 with vary: accept (in our words, a note to caches that the answer depends on the Accept header), and with a content-signal header that Cloudflare says it keeps if your server already sets one. How agents discover what a store offers covers what content signals are.

A media type is the label, such as text/html, that tells a client what format a response is in. text/markdown is registered in RFC 7763, an Informational RFC of March 2016; RFCs are the numbered documents in which the IETF (the Internet Engineering Task Force, which develops internet standards) publishes standards and related notes. The RFC registers the name and does not say how an agent should ask for Markdown. The two routes above are a proposal and one vendor's feature.

Google, for its part, says on its page on AI features in Search: "You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add." That page does not mention llms.txt by name, and it speaks only for Google's search features.

A Markdown version helps only if it says the same thing as the page. In the figure of one product page, as a person sees it and as an agent reads it, the Markdown version is the one pane that holds every detail, the returns note included, because the store wrote it by hand. A version written by hand can also fall out of date when a price changes, which is a cost the store takes on.

A browser is enough for most of these checks; the last one also uses a terminal.

• Open a product page with "view source" and search for the price and the stock text. View source shows the raw HTML. If they are not there, a program that does not run your scripts does not see them.
• In the same source, look for <script type="application/ld+json"> and check that it carries a Product with an Offer, the price, a priceCurrency code such as EUR, and availability. Check that the values match what the page shows.
• Open your browser's developer tools and look for its accessibility view. Check that product photos have names and that buttons such as "Add to cart" have labels.
• If you publish /llms.txt or .md versions of pages, open them and compare them with the live page.
• If your site is on Cloudflare, ask whoever runs it whether Markdown for Agents is turned on. You can test it with curl -H "Accept: text/markdown" https://your-domain/some-page and look at the content-type of the answer.

• None of OpenAI's, Anthropic's or Perplexity's crawler pages says whether their agents run JavaScript (checked 22 October 2026).
• No source we found says that any shopping agent reads the accessibility tree.
• llms.txt is a proposal by one author, and Markdown for Agents is one company's feature. The text/markdown media type is registered; the two ways described here for asking for it are a proposal and one company's feature.
• Google's statements cover Google's crawler and Google's search features. They say nothing about other agents.

Reading the page is one way for an agent to learn what a store sells. Some stores offer a second way in, an interface built for software, which UCP explained and ACP explained describe. A store with one still has product pages, and agents that do not speak those protocols still have to read them.