# A transparent AI-agent testing methodology.

> How Aiscovery tests AI agents on real store tasks, and how the automatic site check works.

- Canonical URL: https://aiscovery.io/methodology
- Markdown version: https://aiscovery.io/methodology.md
- Last updated: 2026-10-21

The method behind AI agent test runs. The automatic site check is described on our checker page.

- [Request a demo](https://aiscovery.io/demo)
- [Explore the audit](https://aiscovery.io/product/agent-readiness-audit)

## Follow one test run

> Interactive demo on the HTML page: a playable test run on three tracks (What the agent did, What we captured, What we checked) with a scrubber. Its content is written out below.

> Example run. Store: northwind-outdoor.com. Agent: Google Gemini.

Find waterproof trail shoes under $150, then the delivery cost.

| What the agent did | Page | What we captured | What we checked |
| --- | --- | --- | --- |
| Opened the home page | Home | Screenshot |  |
| Searched "waterproof trail shoes" | Search | Screenshot and step log | Product found: yes |
| Opened Ridgeline GTX | Product | Screenshot and page response | Price under $150: yes |
| Picked size US 9 | Product | Screenshot and layout shift | In stock: yes |
| Looked for the delivery cost | Delivery | Screenshot and step log | Delivery cost: no |

Stopped before checkout, as every run does.

Result: Partial: 3 of 4 facts found.

1. What counts: [What we measure](https://aiscovery.io/methodology#measurement)
2. Where we test: [Channels and audit modes](https://aiscovery.io/methodology#modes)
3. Before the run: [Scenarios and assertions](https://aiscovery.io/methodology#scenarios)
4. During the run: [Sessions, timing and CLS](https://aiscovery.io/methodology#sessions)
5. After the run: [Attribution and scoring](https://aiscovery.io/methodology#attribution)
6. What we keep: [Evidence and privacy](https://aiscovery.io/methodology#evidence)

## Measure the customer journey, not the presence of a tag.

A readiness signal combines discoverability, successful task completion, repeatability and efficiency.

1. **Find.** Facts are reachable, current, consistent and available in an agent-usable form.
2. **Understand.** Controls, policies, prices and states have unambiguous meaning.
3. **Act.** The allowed workflow reaches an independently verified final state.
4. **Recover.** Validation, auth, session and inventory failures expose a workable next action.

## Coverage is separated by channel and access mode.

A missing agent-native surface cannot erase a successful browser result, and the reverse is also true.

### AI search

Crawl access, citation surfaces, content consistency and machine-readable context.

AI crawlers our checker reads your robots.txt for:

| Company | User agents |
| --- | --- |
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User |
| Anthropic | ClaudeBot, Claude-User |
| Google | Google-Extended |
| Perplexity | PerplexityBot |
| Apple | Applebot-Extended |
| Microsoft Bing | Bingbot |

> Interactive demo on the HTML page: pick a crawler to see which rule applies on an example robots.txt, for the pages /, /products/ridgeline-gtx, /checkout.

> Example run.

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Disallow: /checkout

User-agent: *
Allow: /
```

No group of its own, so the \* group applies.

### Browser and UI

Rendered interaction, semantic controls, keyboard path, sessions and responsive behavior.

### API, MCP and WebMCP

Declared surfaces are noted when we detect them. Running their tools is coming later.

### Three access modes

> Interactive demo on the HTML page: a map of store pages that shows how far the agent goes in each access mode.

> Example run. Stop gate: The agent stops here: no cart, no login, no payment.

| Mode | Status | Pages in reach | Note |
| --- | --- | --- | --- |
| Public | Available | Home, Search, Product, Policies | Public pages only. The agent stops before cart and login. |
| Authenticated | Coming later | Home, Search, Product, Policies, Cart, Login | Coming later: signed-in pages on an account you approve. |
| Sandbox | Coming later | Home, Search, Product, Policies, Cart, Login, Checkout, Payment | Coming later: the full journey on an approved staging site. |

## The scenario is a versioned test contract.

Prompt wording alone never defines success. The contract states the goal, permissions, stop gate and assertions.

### The test contract

1. Persona and business goal
2. Prerequisites and test data
3. Allowed actions and production stop gate
4. Independent final-state assertions
5. Required evidence and criticality

### How a result is decided

> Interactive demo on the HTML page: two switches, Literal checks and AI check, and the result they give.

| Literal checks | AI check | Result |
| --- | --- | --- |
| Pass | Confirms | Pass, AI-checked |
| Pass | Flags | Flagged for review |
| Fail | Confirms | Fail |
| Fail | Flags | Fail |

The AI check can't turn a failed check into a pass.

A run succeeds only when literal checks confirm the intended final state. An AI check then reads the same evidence and can confirm the result or flag it for review. It never turns a failed check into a success, results it checked are marked AI-checked, and concierge results are also reviewed by Aiscovery.

## Record the path and the cost of completion.

Session evidence explains whether a nominal success was fast, stable, repeatable and autonomous.

### What every run records

> Interactive demo on the HTML page: step screenshots of the example run, one frame per step.

> Example run. Store: northwind-outdoor.com.

| What the agent did | Time per step |
| --- | --- |
| Opened the home page | 1.2 s |
| Searched "waterproof trail shoes" | 2.4 s |
| Opened Ridgeline GTX | 1.8 s |
| Picked size US 9 | 1.1 s |
| Looked for the delivery cost | 4.6 s |

- Step screenshots with sensitive fields masked. No video.
- Step log of each page, link, search and filter the agent used
- Total duration, step latency and waiting time
- Attempts per agent and task
- Layout shifts on each page visited

### Layout shifts are measured per page.

Each page the agent visits gets a bounded Cumulative Layout Shift measurement in our browser. It is a lab value, not field data. Movement of the target around each click is not measured yet.

> Interactive demo on the HTML page: a late banner moves the page: a "Free returns this week" banner arrives late and pushes the Ridgeline GTX product block down, and the layout shift on this page counts up.

> Example run.

### Repeatability protocol

Each test run is one attempt by one named agent, and we plan up to three attempts per agent for each key task. The agent, model, viewport, locale and safety rules stay attached to every run.

Up to three attempts per agent:

> Example run.

| Agent | Attempt 1 | Attempt 2 | Attempt 3 |
| --- | --- | --- | --- |
| OpenAI | worked | worked | got stuck |
| Google Gemini | worked | got stuck | got stuck |

## Separate website defects from executor noise.

Results separate website, agent and platform failures.

- **Website:** Something on your pages stopped the task.
- **Agent:** The agent misread the page or gave up.
- **Platform:** Our browser or an AI provider failed.

Each failure goes to whoever caused it:

> Interactive demo on the HTML page: sort example failures into the lane that caused them.

| Example failure | Lane |
| --- | --- |
| The price only appears after a script runs | Website |
| The agent picked the wrong size | Agent |
| Our browser timed out while starting | Platform |
| robots.txt blocks this agent | Website |
| The AI provider returned an error | Platform |

### Verdict

What happened on the tasks we tested.

Statuses: pass, partial, fail, blocked.

Severity:

- Critical blocks a key task or creates a real safety risk.
- High creates major repeatable friction or unreliable completion.

### The verdict and its confidence are reported separately.

The verdict reflects what we tested. Confidence reflects coverage, attempts, evidence quality and open questions. Neither becomes a single score.

> Interactive demo on the HTML page: same verdict, different confidence: switch the four factors and only the confidence changes (Low, Medium, High). The verdict stays Partial.

> Example run.

| Factor | When on | When off |
| --- | --- | --- |
| Coverage | Every key task tested | Some key tasks not tested |
| Attempts | Three attempts agree | One attempt only |
| Evidence quality | Screenshots for every step | Steps without screenshots |
| Open questions | None left open | A result needs review |

## Evidence is useful only when it is safe to review.

Artifacts are stored with stable IDs, timestamps, environment and retention rules.

> Interactive demo on the HTML page: a switch between what the page showed and what we keep.

> Example run.

| Field | What the page showed | What we keep |
| --- | --- | --- |
| Email field | maria.kowalski@example.com | •••••••••••••••••••••••••• |
| Delivery postcode | SW1A 1AA | •••• ••• |
| Session cookie | sid=8f2a91c4e07b | •••••••••••••••• |
| Price shown | $129.00 | $129.00 |
| Stock shown | In stock | In stock |

- Masked step screenshots, step logs, page responses and the list of network requests
- Redaction for PII, credentials, tokens, payment, financial and medical data
- Purpose-limited retention and controlled access

### Emerging technologies are inventory signals, not automatic grades.

llms.txt, llms-full.txt, robots.txt and declared agent surfaces such as WebMCP forms are detected (declared) and lifecycle-labeled when our checker sees them. We don't run MCP, OpenAPI or WebMCP tools yet, and their absence is not a failure.

What our checker found on the example store:

> Example run.

| File | State |
| --- | --- |
| robots.txt | Found |
| sitemap.xml | Found |
| llms.txt | Detected (declared) |
| llms-full.txt | Not there, not a failure |
| WebMCP form | Detected (declared) |

Detected tools are not run yet.

### This page is versioned

Agent products, policies and models change. Results are versioned observations from a defined environment, not permanent compatibility guarantees.

- **Last reviewed:** 3 October 2026
- **Reviewed by:** Aiscovery product and engineering
- **Methodology version:** 2026.08

## Use a method your team can inspect.

Start with a scoped audit and keep every conclusion connected to evidence.

- [Request a demo](https://aiscovery.io/demo)
- [Explore the product](https://aiscovery.io/product)

---

This is the Markdown version of the page for AI agents. The HTML page at the canonical URL is authoritative. Site index: https://aiscovery.io/llms.txt