Aiscovery

Custom-built guide

Can AI agents shop a custom-built store?

They can when your public pages put product facts in the first response and your firewall lets identified bots through, and on a custom stack both are your own code, which is where it commonly breaks.

No platform sets defaults for you. Your robots.txt, CDN and firewall rules, how prices and stock reach the page and where policies live are decisions your team made, often before AI agents existed.

Request a demoCheck AI crawler rules free

Reviewed 21 October 2026 against official documentation. 20 sources

Example: what an agent runs into on custom-built stores

Common problems

An illustrative store, not a real result. Your own result shows only what is really on your store.

custom-built stores at a glance

robots.txt
Whatever your server returns at /robots.txt. A server error there tells compliant crawlers to stay out.
Bot protection
Your CDN or WAF. Cloudflare now sets Search, Agent and Training policies per domain.
Structured data
Only what you add. Google reads store-wide shipping and returns from Organization markup.
Agent checkout
Open protocols you can implement: ACP from OpenAI and Stripe, UCP from Google.

What usually breaks on custom-built stores

The usual suspects on a custom stack, roughly in the order an agent meets them.

  1. 01

    Firewall rules written for scrapers stop agents too

    Rate limits, challenges and user agent blocks catch AI crawlers and shoppers' agents along with scrapers. On Cloudflare, Bot Fight Mode can't be skipped with WAF rules, and new domains that show ads, added since 15 September 2026, block user-directed agents on those pages by default. A 403 tells Google the page doesn't exist; a 429 says slow down.[3][4][5]

  2. 02

    A robots.txt that errors, or blocks more than you meant

    If /robots.txt returns a server error, compliant crawlers must assume the whole site is disallowed. And a crawler that finds a group with its own name ignores the * group, so a GPTBot group with only Allow: / also lifts your cart and account rules for GPTBot.[1]

  3. 03

    Prices and stock arrive after the page loads

    Product HTML that says 'loading' while a script calls a price or stock API gives an agent nothing until the call returns, and agents that read the first response never see it.

  4. 04

    Policies in PDFs and chat widgets

    Returns terms in a downloadable PDF, or delivery answers that only a chat widget gives, are facts an agent commonly can't reach from a product page.

  5. 05

    Markup that disagrees with the page

    Google's merchant listings want an Offer per buyable product, not a price range, and settings in Merchant Center override on-page markup. A hand-built JSON-LD block that drifts from the page leaves agents with two different answers.[11]

How to fix it on custom-built stores

Each fix is config or a small change in the templates that render your product pages.

Check which AI crawlers your robots.txt allows, free
  1. 1

    Let verified bots past your custom rules

    Cloudflare's own pattern is a first rule that skips the remaining custom rules for verified bots. Shoppers' agents, such as ChatGPT agent, increasingly sign requests with Web Bot Auth, which Cloudflare treats as signed agents. When you rate limit, answer 429 with Retry-After rather than 403.[2][6][7]

    Cloudflare WAF custom rule
    Rule name:  Allow verified bots
Expression: (cf.client.bot)
Action:     Skip > All remaining custom rules
Placement:  First
  2. 2

    Serve robots.txt as a static file with explicit groups

    Make /robots.txt a static file that can't fail with your app. Name the crawlers you care about in groups, repeat private paths in each group, and decide training separately from search. User-requested fetchers like ChatGPT-User and Perplexity-User say robots.txt may not apply to them; Anthropic says its bots honor robots.txt, and Claude-User can be blocked there.[1][8][9][10]

    robots.txt
    # AI search and user-requested fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

# Model training: your call
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/

Sitemap: https://example.com/sitemap.xml
  3. 3

    Put price, stock and delivery in the first HTML

    Render the current price, stock status and delivery summary on the server, and let scripts update them afterwards if they must. Cache the page briefly instead of moving the facts into a browser call.

  4. 4

    Mark up offers, shipping and returns

    Give each buyable product an Offer with price, currency and availability. Put shipping and returns either on the offer or once, store-wide, on your Organization, which Google has supported for returns since 2024 and for shipping since November 2025.[11][12][13][14]

    JSON-LD in the product page (example values, use your own)
    {
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Rain jacket",
  "sku": "RJ-BLUE-M",
  "image": "https://example.com/images/rain-jacket.jpg",
  "offers": {
    "@type": "Offer",
    "url": "https://example.com/products/rain-jacket?size=m",
    "price": 129.0,
    "priceCurrency": "USD",
    "availability": "https://schema.org/InStock",
    "shippingDetails": {
      "@type": "OfferShippingDetails",
      "shippingRate": { "@type": "MonetaryAmount", "value": 4.95, "currency": "USD" },
      "shippingDestination": { "@type": "DefinedRegion", "addressCountry": "US" },
      "deliveryTime": {
        "@type": "ShippingDeliveryTime",
        "handlingTime": { "@type": "QuantitativeValue", "minValue": 0, "maxValue": 1, "unitCode": "DAY" },
        "transitTime": { "@type": "QuantitativeValue", "minValue": 2, "maxValue": 5, "unitCode": "DAY" }
      }
    },
    "hasMerchantReturnPolicy": {
      "@type": "MerchantReturnPolicy",
      "applicableCountry": "US",
      "returnPolicyCategory": "https://schema.org/MerchantReturnFiniteReturnWindow",
      "merchantReturnDays": 30,
      "returnMethod": "https://schema.org/ReturnByMail",
      "returnFees": "https://schema.org/FreeReturn"
    }
  }
}
  5. 5

    Turn PDFs and chat answers into pages

    Publish delivery and returns as HTML pages, link them from every product page and the footer, and keep the wording in step with what your checkout and support team actually do.

Agent protocols a custom store can adopt

None of these is needed for agents to read your pages, but they decide how agents identify themselves and check out. Newest last.

  1. Aug 2025Signed agents on CloudflareCloudflare started recognising agents that sign their requests with Web Bot Auth, with ChatGPT agent among the first.[7]
  2. Sep 2025Agentic Commerce ProtocolOpenAI and Stripe released ACP, an open checkout standard, with Instant Checkout in ChatGPT. Product feeds in ChatGPT are open to approved partners.[15][16][17]
  3. Sep 2025Agent Payments ProtocolGoogle announced AP2 for agent-led payments, built on signed mandates that record what the shopper authorised.[18]
  4. Oct 2025Visa Trusted Agent ProtocolVisa, with Cloudflare, published a protocol that helps merchants tell trusted shopping agents from malicious bots, based on HTTP message signatures.[19]
  5. Jan 2026Universal Commerce ProtocolGoogle announced UCP, an open standard for the whole shopping journey, co-developed with Shopify, Etsy, Wayfair, Target and Walmart.[20]
  6. Jul 2026Separate Search, Agent and Training policies on CloudflareCloudflare replaced its single AI bot switch with three behaviours, each allowed, blocked, or blocked on pages with ads, with new defaults from 15 September 2026: new domains that show ads block Agent on pages with ads and disallow Training, while Search stays allowed; sites without ads allow all three.[4]

What Aiscovery checks on custom-built stores

From the outside, the way an agent sees your store. Nothing to install, and agents stop before login, cart, checkout and payment.

Request a demoFree AI crawler checker
  • Whether /robots.txt answers, and what it declares for each AI crawler, group by group.
  • Reachability and redirects, and pages that fail for our identified checker. We never pretend to be an AI crawler, so rules for their user agents aren't tested.
  • Page basics and JSON-LD syntax on the HTML your server sends, read without running JavaScript.
  • AI-facing files: sitemaps, llms.txt and llms-full.txt.
  • GPT-6 Luna and Gemini try to find a product, its price, stock, delivery and returns in a real browser, stopping before cart and checkout.

Questions about custom-built stores and AI agents

Should I allow GPTBot on my store?

It's your call: GPTBot collects data for training OpenAI's models. Whether you appear in ChatGPT search depends on OAI-SearchBot, and ChatGPT-User fetches pages when a user asks. You can block the first and allow the other two.

Why do AI agents get 403 errors on my site?

Commonly a WAF, CDN or rate limit rule built for scrapers. Let verified bots skip your custom rules, check your CDN's AI bot settings, and answer real overload with 429 and Retry-After, which crawlers read as 'slow down' rather than 'gone'.

Do AI agents obey robots.txt?

Their crawlers say they do: GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot among them. Fetchers that act on a user's request, such as ChatGPT-User, Perplexity-User and Google-Agent, say robots.txt may not apply. Anthropic says its bots honor robots.txt, and Claude-User can be blocked there.

Do I need ACP or UCP for agents to shop my store?

Not to be read. Agents can browse your pages today. The protocols matter for checkout inside an AI assistant: OpenAI takes product feeds from approved partners, and Google's UCP is an open standard you can implement.

What structured data does a store need for AI agents?

Product markup with an Offer per buyable item: price, currency and availability. Add shipping and returns on each offer, or once on your Organization, and keep both in step with what the page says.

Do you need access to our code or servers?

No. Aiscovery tests your public storefront from the outside with an identified checker, so nothing is installed and nothing changes on your side.

Sources

Official documentation and announcements, checked on 21 October 2026. Platforms change often, so check the source before you change settings.

  1. 1.RFC 9309: Robots Exclusion ProtocolIETF, 2022-09
  2. 2.RFC 6585: 429 Too Many RequestsIETF, 2012-04
  3. 3.How HTTP status codes affect Google's crawlersGoogle Search Central
  4. 4.New AI traffic options: Search, Agent and TrainingCloudflare changelog, 1 July 2026
  5. 5.Bot Fight ModeCloudflare docs
  6. 6.Stop malicious bots: skip verified botsCloudflare docs
  7. 7.Signed agentsCloudflare, 28 August 2025
  8. 8.Overview of OpenAI crawlersOpenAI
  9. 9.Does Anthropic crawl data from the web, and how can site owners block the crawler?Anthropic
  10. 10.Perplexity crawlersPerplexity
  11. 11.Merchant listing structured dataGoogle Search Central
  12. 12.Shipping policy structured dataGoogle Search Central
  13. 13.More ways to share your shipping and returns policies with GoogleGoogle Search Central blog, 12 November 2025
  14. 14.Merchant return policy structured dataGoogle Search Central
  15. 15.Stripe powers Instant Checkout in ChatGPTStripe, 29 September 2025
  16. 16.Agentic Commerce ProtocolOpenAI and Stripe
  17. 17.Commerce in ChatGPT: get startedOpenAI
  18. 18.Announcing the Agent Payments Protocol (AP2)Google Cloud, 16 September 2025
  19. 19.Visa introduces Trusted Agent ProtocolVisa, 14 October 2025
  20. 20.New agentic commerce tools and the Universal Commerce ProtocolGoogle, 11 January 2026

Related guides

Platform names and logos belong to their owners. Aiscovery is not affiliated with them; we test any store from the outside.

See what agents see on custom-built stores.

Tell us your store and we'll show you, on a short call, what AI agents could and couldn't find there, and the fix for each.

Have a real workflow to test now? Apply as a launch partner

We only use this to arrange your demo. Privacy notice