Define one cross-surface task contract
Start with a customer goal that joins a public fact to a product outcome. Record the immutable prompt, persona, canonical sources, expected plan or capability, locale, viewport, authentication state, test tenant, allowed actions, stop gate and deterministic assertions before the executor runs.
Keep that contract stable across the controlled browser baseline and named agent profiles. The task is not complete because an agent produced a plausible explanation. The cited fact and the observed account state must match the same versioned fixture.
- DISCOVER: find a capability, limit, policy or plan from a canonical source.
- ENTER: reach signup or sign-in with the intended plan, locale and return path intact.
- ACT: complete only the authorized onboarding or self-service change in a test tenant.
- VERIFY: assert product state independently and prove the production stop gate held.
Reconcile marketing, docs and product facts
Create known facts for capabilities, plan limits, trial conditions, regions, support commitments and deprecation state. Compare the marketing page, pricing or contact path, documentation, help center, structured data and visible product UI without assuming every SaaS business must publish public pricing.
Preserve the canonical source, reviewed date and applicable plan or product version. A fact becomes a finding when authoritative surfaces conflict, the agent selects a stale page, or the product state contradicts the promise used to begin the journey.
Control signup and invitation boundaries
Prepare valid, invalid, duplicate, invited, expired-link and rate-limited signup fixtures. Labels, instructions, validation and errors should expose the expected field purpose and recovery action. The selected plan, referral context, locale and return URL must survive the transition into identity flows.
Public production tests stop before account creation, email delivery or another external side effect. An approved sandbox may submit synthetic identities, but the assertion must prove exactly one expected tenant or account exists and delayed retries did not create duplicates.
Test authentication as a recoverable state machine
Run password, passwordless, SSO, OAuth and MFA paths only with approved accounts and permissions. Inspect accessible names, autofill semantics, focus, one-time-code handling, redirect validation and the distinction between invalid, expired, denied and temporarily unavailable states.
Expire the code or session after the agent has made progress. Recovery should preserve safe intent, explain the next action and avoid redirect or authentication loops. For OAuth browser flows, audit the deployed contract against current guidance such as authorization code with PKCE without exposing credentials, tokens or cookies in evidence.
Verify onboarding and permission state
Use a versioned test tenant with known role, entitlement, workspace and onboarding state. Verify each product action through an independent API, database-safe fixture or observable final-state assertion rather than trusting a success toast or the agent's summary.
Test least privilege explicitly. A member should not inherit an owner action, a personal workspace should not become an organization, and a stale invitation should not grant access. Record every approval and human intervention in the step trace.
Test self-service changes and recovery
Scope reversible preferences, profile settings and documented account procedures separately from billing, deletion and external notifications. For plan changes or cancellation, production should stop at an unambiguous review boundary that states the object, effective date, impact and next irreversible action.
Inject validation errors, expired sessions, permission loss, slow responses and late-rendered controls. Measure time to verified outcome, planned waits, retries, interventions, Cumulative Layout Shift and target movement around consent, authentication and account controls.
Prove the outcome and classify the failure
Assert the exact tenant, account, role, entitlement, preference or review state expected by the scenario, then prove no unintended account, subscription, charge, deletion or message was created. A visible success message without the expected backend or product state is a false success.
Package the prompt, source snapshot, fixture version, executor profile, ordered steps, screenshots, accessibility evidence, console and network signals, timing, retries, interventions, CLS and final-state assertion. Classify the primary cause as website, agent-specific, agent policy, infrastructure or inconclusive before assigning remediation.
- Redact credentials, tokens, cookies, personal data and tenant secrets.
- Link every P0 or P1 finding to reproduction steps and acceptance criteria.
- Repeat the same scenario after the fix with a matching environment.
SaaS agent test checklist
- Define one customer goal that connects a canonical public fact to a product outcome.
- Version plan, capability, tenant, role, invitation and authentication fixtures.
- Compare marketing, docs, help, structured data and product UI for the same fact.
- Preserve plan, locale and return intent through signup and authentication redirects.
- Test valid, invalid, duplicate, expired and rate-limited signup states.
- Verify labels, errors, focus, autofill semantics, MFA and expired-session recovery.
- Assert onboarding, workspace, role and entitlement state independently.
- Test least privilege and prevent duplicate writes during retries.
- Measure completion time, waits, retries, interventions, CLS and target movement.
- Stop production before account creation, billing, deletion or external notification.
- Prove no unintended side effect occurred and capture the deterministic final state.
- Redact sensitive evidence, classify the failure cause and define the retest condition.
Primary implementation sources
- Accessibility for AI agentsChrome for Developers
- Forms TutorialW3C Web Accessibility Initiative
- AutofillWHATWG HTML
- Best Current Practice for OAuth 2.0 SecurityRFC Editor
- SoftwareApplicationSchema.org
- AuthenticationPlaywright
- Cumulative Layout Shiftweb.dev
- Trace ViewerPlaywright
