# How an AI agent reads a web page

Raw HTML, the rendered page, the accessibility tree, structured data, Markdown versions and llms.txt, and what each one means for an agent trying to read your product page.

By Valerii Checha, 13 Oct 2026

An AI shopping agent that lands on your product page has to work out what you sell, what it costs and whether it is in stock, and it often has to do that from whatever your server hands it. A person looks at the finished page in a browser. A program may get something quite different: the bare HTML before any script has run, a block of JSON (a plain-text format for data) hidden in that HTML, or a plain-text version of the page if you publish one. This article goes through those forms one at a time and says, for each, what the published documents say about it and where they say nothing.

It belongs to a series. [The hub article](https://storewalk.ai/blog/why-agents-cant-buy) sets out five walls between an agent and a finished purchase; reading the page is the first. Who the agent is and whether your server lets it in is the next article, [Bots, crawlers and agents: who is knocking](https://storewalk.ai/blog/bots-crawlers-agents).

Where we explain a term in plain words, the explanation is ours, not any specification's. Everything else comes from the documents linked in the text and listed under sources.

## What an agent gets when it asks for a page

A browser and an agent start the same way. Each sends a request for a URL, and your server answers with a document in HTML, the markup language web pages are written in. From there they part. The browser downloads the scripts, images and style sheets that the HTML points to, runs the scripts, and draws the result. A program can stop after the first answer and work with the HTML alone.

The [llms.txt proposal](https://llmstxt.org/) describes what that HTML looks like from a language model's side: "An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise." The price a shopper sees in large type may sit deep inside the markup, or it may not be in the markup at all.

## Raw HTML and the rendered page

By raw HTML we mean the first response, exactly as the server sent it. By the rendered page we mean what exists after the browser has run the page's scripts: those scripts can add text, swap parts of the page and fetch data from elsewhere.

Google documents how its own crawler handles the gap. Its guide to [JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics), last updated 4 March 2026, says that "Google processes JavaScript web apps in three main phases: Crawling · Rendering · Indexing". Between crawling and rendering, the page waits in a queue: "The page may stay on this queue for a few seconds, but it can take longer than that. Once Google's resources allow, a headless Chromium renders the page and executes the JavaScript." Headless Chromium is a browser engine running without a screen.

The same guide describes pages where the first response holds none of the content. Some sites "use the app shell model where the initial HTML does not contain the actual content and Google needs to execute JavaScript before being able to see the actual page content that JavaScript generates." An app shell, in our words, is a near-empty page that a script fills in after loading.

What happens to the page in Google’s process.

Example product page, made up for this figure.

The steps in Google’s process:

1. Crawling. The crawler fetches the URL and gets the HTML the server sends, the first response. No script has run yet. Source: [Google: JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics).
2. Waiting in the render queue. Google puts the page in a queue for rendering: “The page may stay on this queue for a few seconds, but it can take longer than that.” Source: [Google: JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics).
3. Rendering. “Once Google’s resources allow, a headless Chromium renders the page and executes the JavaScript.” Only now does the app shell have its content. Source: [Google: JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics).

The two pages:

- Sent complete by the server. The server puts the product details into the HTML it sends. Google calls server-side or pre-rendering “still a great idea”. Source: [Google: JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics).
- App shell, filled in by scripts. The first HTML is a shell: in Google’s words, “the initial HTML does not contain the actual content”, and scripts have to run before the page has any. Source: [Google: JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics).

| Sent complete by the server: on the page | Crawling | Waiting in the render queue | Rendering |
| ---------------------------------------- | -------- | --------------------------- | --------- |
| Product name                             | Yes      | Yes                         | Yes       |
| Price                                    | Yes      | Yes                         | Yes       |
| In stock                                 | Yes      | Yes                         | Yes       |
| Sizes                                    | Yes      | Yes                         | Yes       |

| App shell, filled in by scripts: on the page | Crawling | Waiting in the render queue | Rendering |
| -------------------------------------------- | -------- | --------------------------- | --------- |
| Product name                                 | Not yet  | Not yet                     | Yes       |
| Price                                        | Not yet  | Not yet                     | Yes       |
| In stock                                     | Not yet  | Not yet                     | Yes       |
| Sizes                                        | Not yet  | Not yet                     | Yes       |

Google names three main phases: crawling, rendering and indexing. The queue sits between the first two. This figure stops before indexing. Source: [Google: JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics).

A reader that does not run JavaScript never gets past the first response. Google says “not all bots can run JavaScript”. Source: [Google: JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics).

A real app shell: on 2026-10-22 the raw HTML of contentsignals.org was 1,966 bytes, holding the title “Content Signals” and a script tag. All of its text is in the script. Source: [contentsignals.org](https://contentsignals.org/).

On 22 October 2026 we fetched the raw HTML of [contentsignals.org](https://contentsignals.org/): 1,966 bytes, holding the title "Content Signals" and a single script tag, the HTML element that loads or holds code for the browser to run. All of the site's text is inside the script, so a client that does not run it gets the title and nothing else.

## What it costs when scripts do not run

Google's guide also advises: "server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript." Server-side rendering means the server builds the full HTML before sending it, so the first response already holds the content.

To see whether AI agents run a page's scripts, we read the crawler documentation of three AI companies, [OpenAI](https://developers.openai.com/api/docs/bots), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) and [Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), on 22 October 2026. None of the three pages contains the word "JavaScript" or says whether its crawlers or agents render pages. The pages do not say the agents skip scripts either, so all a store can conclude is that it cannot count on them running.

The figure below shows one made-up product page both ways. As a person sees it, the page has a name, a price, a stock line, a size picker and a returns note printed on the photo. As a program that reads only the raw HTML gets it, the page is missing the price and the stock, because a script fills those in after loading, and nothing in the HTML carries the returns note, because it exists only as pixels in the image.

One product page, as a person sees it and as an agent reads it.

Example product, made up for this figure.

As a person sees it:

- Name: Ridge trail shoe
- Price: €129.00
- In stock
- Sizes: 40, 41, 42, 43, 44, 45, 46
- Written into the product photo: Free returns for 30 days

As an agent reads it:

Raw HTML. The first response, before any script runs. Google says “not all bots can run JavaScript”; here the price and stock are filled in by a script, so they are not in it. Source: [Google: JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics).

```html
<title>Ridge trail shoe | Example store</title>
<h1>Ridge trail shoe</h1>
<img src="/img/ridge.jpg" alt="Ridge trail shoe, side view">
<p class="price"><span id="price"></span></p>
<p class="stock"><span id="stock"></span></p>
<label>Size
  <select name="size">
    <option>40</option><option>41</option><option>42</option><option>43</option>
    <option>44</option><option>45</option><option>46</option>
  </select>
</label>
<script src="/app.js"></script>
```

Structured data. A schema.org Product with an Offer, as JSON-LD in a script tag of the same HTML. The Offer carries the price, its ISO 4217 currency and the availability. Source: [schema.org Offer](https://schema.org/Offer).

```html
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Ridge trail shoe",
  "image": "https://example.com/img/ridge.jpg",
  "offers": {
    "@type": "Offer",
    "price": "129.00",
    "priceCurrency": "EUR",
    "availability": "https://schema.org/InStock"
  }
}
</script>
```

Markdown version. A plain-text page the store writes itself. The llms.txt proposal suggests one at the page’s URL with .md appended; it is a proposal by its author, not a standard. Source: [llms.txt](https://llmstxt.org/).

```markdown
# Ridge trail shoe

Price: €129.00
In stock
Sizes: 40 to 46
Free returns for 30 days.
```

OpenAI’s, Anthropic’s and Perplexity’s crawler pages do not say whether their agents run JavaScript (checked 2026-10-22). Sources: [OpenAI crawlers](https://developers.openai.com/api/docs/bots), [Anthropic crawlers](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) and [Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers).

| On the page               | Raw HTML | Structured data | Markdown version |
| ------------------------- | -------- | --------------- | ---------------- |
| Product name              | Yes      | Yes             | Yes              |
| Price                     | Missing  | Yes             | Yes              |
| In stock                  | Missing  | Yes             | Yes              |
| Sizes                     | Yes      | Missing         | Yes              |
| Returns note in the photo | Missing  | Missing         | Yes              |

The structured data and the Markdown version in that figure are both things a store adds on purpose; they are covered after the section on the accessibility tree.

## The accessibility tree

There is a second model of the page, kept for assistive technology, the software some people use to have a page read aloud or to operate it without a mouse. [WAI-ARIA 1.2](https://www.w3.org/TR/wai-aria-1.2/), a Recommendation of 6 June 2023 from the W3C (the World Wide Web Consortium, which publishes web standards), defines this accessibility tree as a "Tree of accessible objects that represents the structure of the user interface (UI)", and says that "Each node in the accessibility tree represents an element in the UI as exposed through the accessibility API". A node carries a role, such as heading, and an accessible name: "the name of a user interface element".

How HTML elements turn into nodes is set out in [HTML Accessibility API Mappings](https://www.w3.org/TR/html-aam-1.0/) (HTML-AAM), a W3C Working Draft dated 21 October 2026. An `h1` maps to the "heading role, with the aria-level property set to the number in the element's tag name." A `select` shown as a drop-down box maps to the `combobox` role and takes its name from its `label`. An image gets its name from its `alt` attribute.

From HTML elements to accessibility tree nodes.

Example elements of the product page above, made up for this figure.

```html
<h1>Ridge trail shoe</h1>
<img src="/img/ridge.jpg" alt="Ridge trail shoe, side view">
<label>Size
  <select name="size">…</select>
</label>
```

The nodes they become in the accessibility tree:

```text
heading (level 1) "Ridge trail shoe"
img "Ridge trail shoe, side view"
combobox "Size"
```

- \<h1>Ridge trail shoe\</h1> becomes heading (level 1) "Ridge trail shoe". HTML-AAM maps h1 to the “heading role, with the aria-level property set to the number in the element’s tag name”. WAI-ARIA lets a heading take its name from its contents. Sources: [HTML Accessibility API Mappings](https://www.w3.org/TR/html-aam-1.0/) and [WAI-ARIA 1.2](https://www.w3.org/TR/wai-aria-1.2/).
- \<img src="/img/ridge.jpg" alt="Ridge trail shoe, side view"> becomes img "Ridge trail shoe, side view". The image’s name comes from its alt text: HTML-AAM’s rule for img says “use alt attribute”. Whatever is drawn inside the picture, such as the returns note, is not in it. Sources: [HTML Accessibility API Mappings](https://www.w3.org/TR/html-aam-1.0/) and [WAI-ARIA 1.2](https://www.w3.org/TR/wai-aria-1.2/).
- \<select name="size"> becomes combobox "Size". HTML-AAM maps a select shown as a drop-down box to the combobox role. Its name comes from the rule for other form elements, “Otherwise use label element”, and the sample wraps it in a label that reads “Size”. Source: [HTML Accessibility API Mappings](https://www.w3.org/TR/html-aam-1.0/).

WAI-ARIA defines the tree as a “Tree of accessible objects that represents the structure of the user interface”, in which “Each node … represents an element in the UI as exposed through the accessibility API”. We found no source that says a particular shopping agent reads it. Source: [WAI-ARIA 1.2](https://www.w3.org/TR/wai-aria-1.2/).

In our reading, the tree is a fair test of whether the page says in text what it shows in pictures. An image takes its name from its `alt` text, so a product photo without one tells the tree little about what it shows. The returns note in the sample photo appears nowhere in the tree, for the same reason it is missing from the HTML.

We found no source that says a particular shopping agent reads the accessibility tree, so this article makes no such claim. What it is worth to agents, if anything, is unpublished.

## Structured data

Structured data is a block of facts in a fixed vocabulary that a program can read without guessing. For products the vocabulary is [schema.org](https://schema.org/Product), at version 30.1 of 16 September 2026. It defines a Product as "Any offered product or service. For example: a pair of shoes; a concert ticket; the rental of a car; a haircut; or an episode of a TV show streamed online." A product's `offers` property points to an [Offer](https://schema.org/Offer), which carries the terms of sale; schema.org defines an Offer as "An offer to transfer some rights to an item or to provide a service". The property can also point to a Demand.

On a shop page, three of the Offer's properties carry what a shopper checks first, in our reading. `price` is "The offer price of a product". `priceCurrency` is the currency of that price, and schema.org asks for codes from ISO 4217, the international list of currency codes: "Use standard formats: ISO 4217 currency format, e.g. "USD"". `availability` says whether the item can be had, with values such as "In stock, Out of stock, Pre-order".

The usual way to put this on a page is JSON-LD. In our words, JSON-LD is JSON written so that it names its vocabulary, and the block goes inside a script tag, which here holds data. Google's [introduction to structured data](https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data) calls it "A JavaScript notation embedded in a `<script>` tag" and says "Google recommends using JSON-LD for structured data if your site's setup allows it". The tag sits in the HTML itself, so if the server puts it in the first response, a program that reads only raw HTML still gets it. That is why, on the sample page, the price is missing from the HTML's visible text but present in its structured data.

Product structured data, key by key.

Example markup for a product made up for this figure. It is the structured data of the page above.

```html
<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Ridge trail shoe",
  "image": "https://example.com/img/ridge.jpg",
  "offers": {
    "@type": "Offer",
    "price": "129.00",
    "priceCurrency": "EUR",
    "availability": "https://schema.org/InStock"
  }
}
</script>
```

- \<script type="application/ld+json">. In our words: The tag that holds the markup. It sits in the page’s HTML, so a reader that does not run scripts still gets it, as long as the server sends it. What the definition says: Google: JSON-LD is “A JavaScript notation embedded in a \<script> tag”, and “Google recommends using JSON-LD for structured data if your site’s setup allows it”. Source: [Google: Introduction to structured data markup](https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data).
- Product. In our words: Says what the markup describes: a thing for sale. What the definition says: schema.org: “Any offered product or service. For example: a pair of shoes; a concert ticket; the rental of a car; a haircut; or an episode of a TV show streamed online.” Source: [schema.org Product](https://schema.org/Product).
- offers. In our words: Links the product to the terms it is sold on. What the definition says: schema.org: “An offer to provide this item—for example, an offer to sell a product, rent the DVD of a movie, perform a service, or give away tickets to an event.” Source: [schema.org Product](https://schema.org/Product).
- Offer. In our words: The terms of sale. In the example, the price, the currency and the availability sit inside it. What the definition says: schema.org: “An offer to transfer some rights to an item or to provide a service”. Source: [schema.org Offer](https://schema.org/Offer).
- price. In our words: The amount on its own. The currency goes in priceCurrency, not in this value. What the definition says: schema.org: “The offer price of a product, or of a price component when attached to PriceSpecification and its subtypes.” Its usage guidelines ask for priceCurrency “instead of including ambiguous symbols such as ‘$’ in the value”. Source: [schema.org Offer](https://schema.org/Offer).
- priceCurrency. In our words: Which currency the amount is in, as an ISO 4217 code: EUR for the €129.00 on the page. What the definition says: schema.org: “The currency of the price … Use standard formats: ISO 4217 currency format, e.g. “USD””. Source: [schema.org Offer](https://schema.org/Offer).
- availability. In our words: Whether the item can be had, given as an ItemAvailability value: here InStock. What the definition says: schema.org: “The availability of this item—for example In stock, Out of stock, Pre-order, etc.” Its values are of the type ItemAvailability. Source: [schema.org Offer](https://schema.org/Offer).

Google can also read JSON-LD that JavaScript injects; that is a statement about Google’s own renderer. Google documents two product markups: product snippets “for product pages where people can’t directly purchase the product”, and merchant listings “for pages where customers can purchase products from you”, which have “more options for specifying detailed product information, like apparel sizing, shipping details, and return policy information.” Sources: [Google: Introduction to structured data markup](https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data) and [Google: Product structured data](https://developers.google.com/search/docs/appearance/structured-data/product).

The same Google page adds: "Google can read JSON-LD data when it is dynamically injected into the page's contents, such as by JavaScript code". That describes Google's own renderer. A block that a script adds after loading is in the same position as the price in the figure: there only for a reader that runs the script.

Google also documents two product markups, on its page on [product structured data](https://developers.google.com/search/docs/appearance/structured-data/product): product snippets "for product pages where people can't directly purchase the product", and merchant listings "for pages where customers can purchase products from you", which have "more options for specifying detailed product information, like apparel sizing, shipping details, and return policy information." Those are rules for Google's search features. Nothing on the page says how AI agents use either one.

## Markdown versions and llms.txt

Markdown is plain text with light marks for headings, lists and links. A page in Markdown has no navigation, no scripts and no layout, which is what the llms.txt proposal is after. There are two ways to offer one, and they come from different places.

The first is the [llms.txt proposal](https://llmstxt.org/) itself, titled "The /llms.txt file, v2", by Jeremy Howard, first published on 3 September 2024 and last modified on 10 August 2026. It is a proposal by its author, not a document of a standards body. It suggests "adding a `/llms.txt` markdown file to websites to provide LLM-friendly content": a file at the root of the site that "offers brief background information, guidance, and links to detailed markdown files." It also suggests a clean Markdown version of each page at the same address with `.md` appended, or with the extension replaced by `.md`, and a link to it from the HTML with `rel="alternate" type="text/markdown"`.

The second is to let the client ask for Markdown at the page's own address. A request can carry an `Accept` header, the line where a client lists the formats it can take. Cloudflare's [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/), documented as last updated on 9 October 2026, works on that header: "the client needs to add the `Accept` negotiation header with `text/markdown` as one of the options. Cloudflare will detect this, fetch the original HTML version from the origin, and convert it to Markdown before serving it to the client." It applies to sites where the feature is turned on. In Cloudflare's sample response the answer comes back as `content-type: text/markdown; charset=utf-8` with `vary: accept` (in our words, a note to caches that the answer depends on the `Accept` header), and with a `content-signal` header that Cloudflare says it keeps if your server already sets one. [How agents discover what a store offers](https://storewalk.ai/blog/how-agents-discover-a-store) covers what content signals are.

A media type is the label, such as `text/html`, that tells a client what format a response is in. `text/markdown` is registered in [RFC 7763](https://www.rfc-editor.org/rfc/rfc7763), an Informational RFC of March 2016; RFCs are the numbered documents in which the IETF (the Internet Engineering Task Force, which develops internet standards) publishes standards and related notes. The RFC registers the name and does not say how an agent should ask for Markdown. The two routes above are a proposal and one vendor's feature.

Google, for its part, says on its page on [AI features in Search](https://developers.google.com/search/docs/appearance/ai-features): "You don't need to create new machine readable files, AI text files, or markup to appear in these features. There's also no special schema.org structured data that you need to add." That page does not mention llms.txt by name, and it speaks only for Google's search features.

A Markdown version helps only if it says the same thing as the page. In the figure of one product page, as a person sees it and as an agent reads it, the Markdown version is the one pane that holds every detail, the returns note included, because the store wrote it by hand. A version written by hand can also fall out of date when a price changes, which is a cost the store takes on.

## What you can check today

A browser is enough for most of these checks; the last one also uses a terminal.

- Open a product page with "view source" and search for the price and the stock text. View source shows the raw HTML. If they are not there, a program that does not run your scripts does not see them.
- In the same source, look for `<script type="application/ld+json">` and check that it carries a `Product` with an `Offer`, the `price`, a `priceCurrency` code such as `EUR`, and `availability`. Check that the values match what the page shows.
- Open your browser's developer tools and look for its accessibility view. Check that product photos have names and that buttons such as "Add to cart" have labels.
- If you publish `/llms.txt` or `.md` versions of pages, open them and compare them with the live page.
- If your site is on Cloudflare, ask whoever runs it whether Markdown for Agents is turned on. You can test it with `curl -H "Accept: text/markdown" https://your-domain/some-page` and look at the `content-type` of the answer.

## What is not settled

- None of OpenAI's, Anthropic's or Perplexity's crawler pages says whether their agents run JavaScript (checked 22 October 2026).
- No source we found says that any shopping agent reads the accessibility tree.
- llms.txt is a proposal by one author, and Markdown for Agents is one company's feature. The `text/markdown` media type is registered; the two ways described here for asking for it are a proposal and one company's feature.
- Google's statements cover Google's crawler and Google's search features. They say nothing about other agents.

Reading the page is one way for an agent to learn what a store sells. Some stores offer a second way in, an interface built for software, which [UCP explained](https://storewalk.ai/blog/ucp-explained) and [ACP explained](https://storewalk.ai/blog/acp-explained) describe. A store with one still has product pages, and agents that do not speak those protocols still have to read them.

## Sources

- [Google: JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)
- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots)
- [Anthropic: Does Anthropic crawl data from the web?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [contentsignals.org (raw HTML, fetched 2026-10-22)](https://contentsignals.org/)
- [W3C: WAI-ARIA 1.2](https://www.w3.org/TR/wai-aria-1.2/)
- [W3C: HTML Accessibility API Mappings 1.0](https://www.w3.org/TR/html-aam-1.0/)
- [schema.org Product](https://schema.org/Product)
- [schema.org Offer](https://schema.org/Offer)
- [Google: Introduction to structured data markup](https://developers.google.com/search/docs/appearance/structured-data/intro-structured-data)
- [Google: Product structured data](https://developers.google.com/search/docs/appearance/structured-data/product)
- [The /llms.txt file, v2](https://llmstxt.org/)
- [Cloudflare docs: Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/)
- [RFC 7763: The text/markdown Media Type](https://www.rfc-editor.org/rfc/rfc7763)
- [Google: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)

Last checked 22 Oct 2026