Methodology

A transparent AI-agent testing methodology.

The method behind AI agent test runs. The automatic site check is described on our checker page.

Updated

northwind-outdoor.comGoogle Gemini

Find waterproof trail shoes under $150, then the delivery cost.

Example run
  1. Opened the home pageScreenshot
  2. Searched "waterproof trail shoes"
  3. Opened Ridgeline GTX
  4. Picked size US 9
  5. Looked for the delivery cost
Stopped before checkout, as every run does.

Measure the customer journey, not the presence of a tag.

A readiness signal combines discoverability, successful task completion, repeatability and efficiency.

  1. FindFacts are reachable, current, consistent and available in an agent-usable form.
  2. UnderstandControls, policies, prices and states have unambiguous meaning.
  3. ActThe allowed workflow reaches an independently verified final state.
  4. RecoverValidation, auth, session and inventory failures expose a workable next action.

Coverage is separated by channel and access mode.

A missing agent-native surface cannot erase a successful browser result, and the reverse is also true.

AI search

Example run

Crawl access, citation surfaces, content consistency and machine-readable context.

AI crawlers our checker reads your robots.txt for

Pick a crawler to see which rule applies

User-agent: GPTBotDisallow: / User-agent: ClaudeBotAllow: / User-agent: Google-ExtendedDisallow: /checkout User-agent: *Allow: /

Page

  • GPTBotBlocked
  • OAI-SearchBotNo group of its own, so the * group appliesAllowed
  • ChatGPT-UserNo group of its own, so the * group appliesAllowed

Browser and UI

Rendered interaction, semantic controls, keyboard path, sessions and responsive behavior.

API, MCP and WebMCP

Declared surfaces are noted when we detect them. Running their tools is coming later.

Three access modes

Example run
Home
Search
Product
Policies
Cart
Login
Checkout
Payment

Stop gate. The agent stops here: no cart, no login, no payment.

The scenario is a versioned test contract.

Prompt wording alone never defines success. The contract states the goal, permissions, stop gate and assertions.

The test contract

  1. 1Persona and business goal
  2. 2Prerequisites and test data
  3. 3Allowed actions and production stop gate
  4. 4Independent final-state assertions
  5. 5Required evidence and criticality

How a result is decided

Literal checks

AI check

ResultPass, AI-checked

The AI check can't turn a failed check into a pass.

A run succeeds only when literal checks confirm the intended final state. An AI check then reads the same evidence and can confirm the result or flag it for review. It never turns a failed check into a success, results it checked are marked AI-checked, and concierge results are also reviewed by Aiscovery.

Record the path and the cost of completion.

Session evidence explains whether a nominal success was fast, stable, repeatable and autonomous.

What every run records

Example run
Step screenshotsnorthwind-outdoor.com

1 / 5

Opened the home page

Screenshot

Time per step

  • 1.2s
  • 2.4s
  • 1.8s
  • 1.1s
  • 4.6s
  • Step screenshots with sensitive fields masked. No video.
  • Step log of each page, link, search and filter the agent used
  • Total duration, step latency and waiting time
  • Attempts per agent and task
  • Layout shifts on each page visited

Layout shifts are measured per page.

Example run

Each page the agent visits gets a bounded Cumulative Layout Shift measurement in our browser. It is a lab value, not field data. Movement of the target around each click is not measured yet.

Ridgeline GTXAdd to cart

Layout shift on this page

0.00

Repeatability protocol

Example run

Each test run is one attempt by one named agent, and we plan up to three attempts per agent for each key task. The agent, model, viewport, locale and safety rules stay attached to every run.

Up to three attempts per agent123
OpenAI2/3 worked
Google Gemini1/3 worked

Separate website defects from executor noise.

Results separate website, agent and platform failures.

The price only appears after a script runs
  • Website0Something on your pages stopped the task.
    • Agent0The agent misread the page or gave up.
      • Platform0Our browser or an AI provider failed.

        Verdict

        What happened on the tasks we tested.

        • pass
        • partial
        • fail
        • blocked

        Severity

        Critical blocks a key task or creates a real safety risk.

        High creates major repeatable friction or unreliable completion.

        The verdict and its confidence are reported separately.

        The verdict reflects what we tested. Confidence reflects coverage, attempts, evidence quality and open questions. Neither becomes a single score.

        Verdict

        Partial

        Confidence

        Medium

        Evidence is useful only when it is safe to review.

        Artifacts are stored with stable IDs, timestamps, environment and retention rules.

        Example run
        • Email fieldmaria.kowalski@example.com
        • Delivery postcodeSW1A 1AA
        • Session cookiesid=8f2a91c4e07b
        • Price shown$129.00
        • Stock shownIn stock
        • Masked step screenshots, step logs, page responses and the list of network requests
        • Redaction for PII, credentials, tokens, payment, financial and medical data
        • Purpose-limited retention and controlled access

        Emerging technologies are inventory signals, not automatic grades.

        llms.txt, llms-full.txt, robots.txt and declared agent surfaces such as WebMCP forms are detected (declared) and lifecycle-labeled when our checker sees them. We don't run MCP, OpenAPI or WebMCP tools yet, and their absence is not a failure.

        What our checker found on the example store

        robots.txtFound
        sitemap.xmlFound
        llms.txtDetected (declared)
        llms-full.txtNot there, not a failure
        WebMCP formDetected (declared)

        Detected tools are not run yet.

        This page is versioned

        Agent products, policies and models change. Results are versioned observations from a defined environment, not permanent compatibility guarantees.

        Last reviewed
        3 October 2026
        Reviewed by
        Aiscovery product and engineering
        Methodology version
        2026.08

        Use a method your team can inspect.

        Start with a scoped audit and keep every conclusion connected to evidence.