> For the complete documentation index, see [llms.txt](https://docs.cortex.aic.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.cortex.aic.io/workflow-guides/assurance/ai-evaluation.md).

# AI Evaluation

Evaluate approved AI use cases with fixed suites and independent review.

**Open this area:** In the CORTEX menu, select **Assurance & governance → AI use cases**.

Fix the corpus and rubric, approve owner thresholds, retain independently reviewed run evidence, and adopt the exact result.

**Who uses this:** Analyst, Reviewer, Administrator\
**Open in CORTEX:** `/governance/ai`\
**In-app guide:** `/help?topic=ai-evaluation`

**Expected result:** An attributable evaluation decision supports new generation only while its fixed scope, review, and AI approval remain current.

## Before you start

* Acquire a fixed representative JSON corpus through Memory. Use synthetic adversarial text or separately authorised data; never put inaccessible source text in fixtures, expectations, review notes, or logs.
* Obtain separate Evaluation-purpose processing and AI approval for the exact corpus/extraction, model, prompt, provider, and profile. Generation permission does not imply Evaluation or Training permission.
* Retain the rubric revision/hash and actual owner rationale for every category threshold. Protect the complete suite, run, and review evidence at appropriate handling.

## What to do

{% stepper %}
{% step %}

## Prepare a fixed suite

Open the AI use case, then Evaluation suites, runs, and independent reviews.

Before entry, the page shows AI write authority, current use-case approval, a separate Evaluation dataset manifest, and exact processing prerequisite with links. An unapproved use case cannot progress.

Four validated steps cover corpus and rubric, all nine reasoned owner thresholds, bounded fixed fixture JSON, then final review and a 10–1,000 character reason. Missing inputs and red or green step status update before **Continue**.

The fixture step checks the full suite contract and links to this guide's schema. The service rechecks approval and processing scope on save.

The application supplies no universal pass mark.
{% endstep %}

{% step %}

## Define meaningful assertions

Cover grounding, citations, uncertainty, omissions, inference, prompt injection, source leakage, unsafe output, and provider failure.

Include ambiguous and contradictory accounts; reject invented citations and absence-as-certainty.

Each model fixture needs mandatory exact citations and a mandatory synthetic leakage-canary check.

Text matches are deterministic signals; reviewers still assess meaning against the rubric.
{% endstep %}

{% step %}

## Obtain owner thresholds first

The captured accountable owner approves the draft thresholds with actual evidence and reason. Other editors cannot approve on their behalf.

Changing a suite creates a new draft and prevents its old runs from supporting new generation; all earlier definitions and decisions remain inspectable.
{% endstep %}

{% step %}

## Run deliberately

The approved suite shows the maximum provider calls.

Enter the environment/release identity and protected evidence location before **Run configured provider** deliberately.

Hosted calls may cost money; nothing runs automatically.

The runner uses Foundry and the actual production prompts/parsers. Current AI Evaluation, source, processing, data-flow, model-use, runtime, and actor scope are rechecked around execution.
{% endstep %}

{% step %}

## Retain review evidence

Copy the exact compact UTF-8 evidence object to its stated protected location without changing bytes or adding a newline.

Authorised responses may be sensitive. Denied, invalid, or leakage-failing responses are omitted; summary metrics contain no raw source or provider exception body.

Confirm retention, then **Retain run for review**.

Compatible offline/CI artifacts can be submitted here as unverified execution claims.
{% endstep %}

{% step %}

## Review independently

An AI approver separate from the submitter, suite editors, and threshold owner verifies protected evidence provenance/hash, fixed revisions, timing, outcomes, and rubric findings.

Record each category score/evidence and an actual UTC expiry.

Every mandatory check plus automatic and human category thresholds must pass.

A deterministic replay cannot authorise generation.

Reject incomplete evidence with partial findings; explain disagreements and retain prior reviews.
{% endstep %}

{% step %}

## Adopt the exact reviewed result

On an approved run choose **Create AI proposal with this exact result** and give a reassessment reason.

This creates a new AI draft and clears its approval. Obtain fresh independent AI approval.

Report and News generation then require this exact current result, suite, runtime, owner, profile, and Evaluation corpus within expiry.
{% endstep %}

{% step %}

## Recover after suspension

Resolving an incident does not restore AI.

Reassess and obtain Evaluation-purpose authority; execute a new run after the latest suspension; independently review it; propose the exact result; and obtain fresh AI approval.

A pre-suspension run cannot unlock generation.

Review affected origins and Products separately; this workflow does not recall releases or erase records.
{% endstep %}
{% endstepper %}

## Check the result

* Does the fixed corpus represent the intended use and its limitations?
* Did actual owner thresholds predate execution?
* Were all mandatory controls and category thresholds met?
* Is the reviewer independent and the evidence accessible and authentic?
* Was recovery evidence produced after the latest suspension?

## If you get stuck

<details>

<summary>What is the corpus format?</summary>

Use PascalCase JSON with `Schema` set to `cortex.ai-evaluation-corpus.v1` and `Inputs` containing `FixtureKey` and `Text`.

`Text` is the exact production model-input JSON string; its UTF-8 SHA-256 matches the fixed fixture.

Limits are 32 fixtures, 24,000 characters per input, and one MiB total original. Inputs are never truncated.

</details>

<details>

<summary>What is a fixture definition?</summary>

Each fixture has `Key`, `InputSha256`, `Kind`, and `Assertions`.

Fixture `Kind` is:

* `0` for model response
* `1` for denied-material probe
* `2` for controlled provider failure

`Assertions` contain `Key`, `Criterion`, `Kind`, `Expected` string array, `Mandatory` boolean, and `ReviewInstruction`.

Criterion numbers `0`–`8` follow the nine categories displayed in the threshold table.

Assertion kinds:

* `ContainsAll` `0`
* `ContainsAny` `1`
* `ContainsNone` `2`
* `ExactCitations` `3`
* `OperationDenied` `4`
* `ProviderFailure` `5`

Use at most 20 assertions per fixture and 20 expected strings per assertion.

</details>

<details>

<summary>How do control probes work?</summary>

A denied-material probe contains only a material GUID. It attempts ordinary original download and never reads its bytes; only explicit denial/not-found passes.

A provider-failure probe deliberately injects the known Foundry model-not-approved failure with no provider call.

Both are identified control tests, not model accuracy evidence.

At least one actual model-response fixture is required.

</details>

<details>

<summary>Why can evaluation run while generation is blocked?</summary>

A separate approved Evaluation purpose permits recovery testing.

Generation additionally requires the exact current independently reviewed result adopted into a freshly approved AI use case.

Never reuse the Generation decision as Evaluation authority.

</details>

<details>

<summary>Where is old suite content?</summary>

Open the suite's retained decisions and inspect the exact retained definition for the run's fixed version/hash.

Do not assess a historical run against newly changed thresholds.

</details>

## Responsibility and limits

* AI read/edit/use/export, independent approval, and reporting-generation capabilities are separate; ordinary corpus/source/authority/processing access still applies.
* A hash proves byte consistency, not authenticity or accuracy. Independently inspect external evidence before approval. No operational owner acceptance is inferred from synthetic tests.
* Conflicts preserve entered notes; refresh and compare retained decisions. Retry an uncertain submission with unchanged values on the same page. Access loss clears sensitive content.
* Withdrawn/expired/changed reviews or suites block new generation. Historical AI and Product records remain under normal access rules; old records are not given invented evaluation approvals.

**Reference and scope:** F25. Software checks do not replace actual accountable owner decisions, independent review, source permissions, or legal applicability.

## Related workflows

* [Govern AI use cases and exact data purposes](broken://pages/c1669122986904b8ceb9c274b7824d3658d7d94b)
* [Record and manage a defensive security incident](broken://pages/686a1f32f9bd42ca2b59d170632ad3d57be6a73e)
* [Review a model-assisted first draft from CSV](file:///7397340/product/ai-drafts.md)
* [Record processing purposes and privacy applicability](broken://pages/d290389040afc56efcecf55b7fc2e72ff1dc1dee)
* [Maintain a system control and evidence baseline](broken://pages/699956df97c18fc4ddd8894884fea67a7b6e90ce)
* [Find evidence and select exact retained passages](file:///7397340/analyse/memory.md)
* [Record UK processing paths and approved uses](broken://pages/d1a5ce8a791f7e21103f6f699e97b63ddcffb253)