The layers of an audit result
An Aiscovery audit starts with a real user goal, not a checklist total. A result connects five layers: the scenario contract, one or more agent runs, deterministic assertions, evidence artifacts and an actionable finding.
The task matrix is the fastest place to begin. It shows whether each required agent profile reached the expected final state. Technical checks then explain conditions that may have helped or prevented that outcome. A passing technical check never replaces a completed task.
A result is only as useful as its link to the observed run. Scores summarize the audit, but recordings, traces and assertions explain it.
Each finding should answer four questions:
- What user goal was affected?
- What happened instead of the expected result?
- Which evidence proves the difference?
- What change and acceptance criterion would close the finding?
Task and check statuses
Checks and task runs use different status sets because they answer different questions. A check evaluates a specific condition. A task evaluates whether an agent achieved a business goal.
| Record | Status | Meaning |
|---|---|---|
| Check | pass | The criterion passed with valid evidence. |
| Check | partial | The criterion was only partly met and the impact is observable. |
| Check | fail | The fail criterion was observed. |
| Check | blocked | The check started, but site access prevented completion. |
| Check | not_applicable | The check is outside the approved scope. |
| Check | not_run | The check was planned but not executed. Confidence decreases. |
| Task | success | All positive and negative assertions passed. |
| Task | success_with_friction | The final state was correct, but the run exceeded an expected step, retry or intervention budget. |
| Task | partial | Part of the goal succeeded, but an important assertion failed. |
| Task | failed | The goal was not reached or the final result was wrong. |
| Task | blocked_by_site | The site prevented the in-scope flow. |
| Task | blocked_by_agent_policy | The executor policy did not permit the in-scope goal. |
| Task | inconclusive | The fixture, evaluator or requirement could not support a conclusion. |
not_run and inconclusive do not silently count as failures. They are excluded from readiness calculations and reduce confidence, so an incomplete audit cannot appear more certain than it is.
Failure attribution
The same visible failure can have different causes. Attribution is completed before scoring and distinguishes four primary classes:
- Website failure: the deterministic baseline or several capable agents reproduce a site defect.
- Agent-specific failure: the site works in the baseline and for other profiles, but one profile cannot complete it.
- Policy block: the agent product declines or requires a handoff because of its own policy boundary.
- Infrastructure failure: the runner, fixture, network or evaluator failed independently of the site and agent capability.
A policy block receives zero for that agent profile because a user of that product cannot complete the task. It is not labeled a website defect without additional evidence. The report keeps both statements visible.
Severity and blockers
Severity expresses impact, not how difficult a fix appears.
| Severity | Interpretation |
|---|---|
P0 | Money, sensitive data, cross-tenant access, duplicate action or a complete blocker for a critical transaction. |
P1 | A systematic failure of a primary task or materially wrong information without P0 impact. |
P2 | Meaningful friction, instability or a defect in a secondary flow. |
P3 | A minor gap or improvement opportunity. |
An unresolved, applicable P0 caps the final readiness result at 39. An unresolved P1 caps it at 59. The raw score remains visible beside the capped score, the reason and the linked finding.
Limits and safe interpretation
Audit results are versioned observations from a defined environment. They do not guarantee that every current or future model will behave the same way.
Review these fields before comparing two results:
- site environment and fixture version;
- agent product, model, tools and policy restrictions;
- viewport, locale, cookies and authentication state;
- planned repetitions and completed repetitions;
- unsupported capture capabilities or redacted evidence;
- production stop gates and actions that were intentionally not completed.
Production payments, bookings, submissions, deletions and other irreversible operations stop before commit. A full completion claim requires an approved sandbox or another reversible environment with independently verified final-state assertions.