Build a versioned scenario pack
Reuse the contract shape, not assumptions about the client stack. Each scenario needs a stable ID, goal, prerequisites, test data, allowed actions, stop gate, expected result, criticality and evidence requirements. Store domain-specific fixtures and assertions beside the scenario instead of burying them in an executor prompt.
Version the website release or deployment, environment, source snapshots, agent product and model, tool access, policy restrictions, viewport, locale, consent state and authentication fixture. Plan repeated runs before execution so a retry cannot be silently promoted into an ordinary first-pass success.
Run an evidence-safe baseline
Start with a deterministic browser baseline, then run named agent profiles against the same task contract. Isolate cookies, local storage, credentials and mutable fixtures between runs. Preserve successful and failed attempts so the client can distinguish repeatability from a single favorable outcome.
Capture the session recording, ordered actions, screenshots, accessibility tree, DOM snapshots, console and network signals, source citations, final-state assertion, completion time, waits, retries, interventions, cost, Cumulative Layout Shift and target movement. Redact secrets and client data before an artifact enters a shareable report, then apply the agreed access and retention policy.
- Use synthetic identities and reversible fixtures for authenticated or sandbox runs.
- Record both observation and action timing around unstable controls.
- Prove no unintended payment, message, booking, deletion or external notification occurred.
Separate cause from confidence
Classify the primary cause as website, agent-specific, agent policy, infrastructure or inconclusive. A deterministic baseline that passes while one named agent fails is different from a rendered state that blocks every executor. Preserve the exact step, decision, response or policy boundary that supports the classification.
Report readiness and confidence separately. Repetitions, fixture quality, evidence completeness and environment stability increase confidence, while retries and human interventions remain visible friction. A flaky pass is not equivalent to a clean first-pass result, and an untested area is not evidence of success.
Create an owner-ready backlog
Connect every priority finding to the affected customer task, observed result, expected result, reproduction steps, evidence artifact, primary cause and acceptance criteria. Add impact, effort, likely owner and dependencies so account teams can explain the business consequence while implementation teams can verify the fix.
Separate website remediation from agent compatibility notes, policy limitations and infrastructure incidents. Describe optional technologies such as llms.txt, OpenAPI, MCP or WebMCP by observed state and task impact. Do not turn their presence into a maturity claim or their absence into an automatic failure.
- Prioritize unsafe side effects, blocked critical journeys and data exposure before minor friction.
- Keep confidence, limitations and not-run areas visible in the executive summary.
- Avoid guaranteed visibility, universal agent support and unsupported automation claims.
Run a comparable retest
Link each retest to the baseline manifest and approved findings. Reuse the same scenario ID, fixtures, stop gate and assertions where possible. Record every intentional change to the website, environment, browser, agent version, policy or test data so the comparison does not attribute a moving executor to the client fix.
Compare verified outcome, repetitions, completion time, waits, retries, interventions, CLS, target movement and side effects. A finding is resolved only when its acceptance criteria pass with reviewable evidence. A new regression becomes a separate finding rather than being hidden inside an improved aggregate score.
Govern portfolio evidence and claims
Keep client workspaces, credentials, recordings and exports isolated. Define who may view raw traces, receive redacted reports, approve excerpts and request deletion. Portfolio summaries should aggregate compatible statuses without exposing client URLs, prompts, screenshots or business data outside the agreed audience.
Present only capabilities the service actually supports. White-label reports, automated exports, scheduled monitoring and cross-client benchmarks remain explicit product states, not implied promises. Every client-facing claim should link back to a dated scope, tested surface, executor profile and confidence level.
- Use a standard report structure while preserving client-specific limitations.
- Never compare scores from incompatible scopes without a clear qualification.
- Retain the evidence needed for the agreed retest window, then follow deletion rules.
Agency audit delivery checklist
- Record client owner, authorized domains, environments, routes and excluded surfaces.
- Declare public, authenticated or sandbox mode, allowed actions and production stop gate.
- Define one stable task ID with persona, fixtures, expected state and assertions.
- Version website release, sources, browser, agent, tools, policy, viewport and locale.
- Run an isolated deterministic baseline before named agent profiles.
- Preserve every repetition, retry and intervention without rewriting the first-pass result.
- Capture session, accessibility, DOM, console, network, timing, CLS and final-state evidence.
- Redact secrets and client data, then enforce evidence access and retention rules.
- Classify website, agent-specific, policy, infrastructure and inconclusive causes separately.
- Report readiness, confidence, limitations and not-run areas as distinct fields.
- Give each finding reproduction steps, task impact, owner and acceptance criteria.
- Record emerging technologies as observed states instead of automatic scoring requirements.
- Link retest to the baseline manifest and disclose every changed variable.
- Prove resolved findings with matching assertions and preserve new regressions separately.
- Publish only supported agency capabilities and authorized portfolio evidence.
Primary delivery and testing sources
- Accessibility for agentsChrome for Developers
- Lighthouse agentic browsing scoringChrome for Developers
- IsolationPlaywright
- RetriesPlaywright
- Trace ViewerPlaywright
- Cumulative Layout Shiftweb.dev
- Template for Accessibility Evaluation ReportsW3C Web Accessibility Initiative
- Planning and Managing AccessibilityW3C Web Accessibility Initiative
