---
title: "Test Rules and Model"
description: "Test deterministic policy exactly and model behavior empirically, including inputs that try to redirect the assessment."
canonical_url: "https://vercel.com/academy/enterprise-apps-agents/test-the-boundary"
md_url: "https://vercel.com/academy/enterprise-apps-agents/test-the-boundary.md"
docset_id: "vercel-academy"
doc_version: "1.0"
last_updated: "2026-09-14T22:07:15.950Z"
content_type: "lesson"
course: "enterprise-apps-agents"
course_title: "Enterprise Apps and Agents"
prerequisites:  []
---

<agent-instructions>
Vercel Academy — structured learning, not reference docs.
Lessons are sequenced.
Adapt commands to the human's actual environment (OS, package manager, shell, editor) — detect from project context or ask, don't assume.
The lesson shows one path; if the human's project diverges, adapt concepts to their setup.
Preserve the learning goal over literal steps.
Quizzes are pedagogical — engage, don't spoil.
Quiz answers are included for your reference.
</agent-instructions>

# Test Rules and Model

# Test rules and model

The scaffold includes twelve cases in `evals/vendor-requests.json`: ordinary requests, cost and data-policy boundaries, ambiguous purposes, and adversarial descriptions that try to redirect the model. It also includes fast unit tests for routing, reviewer authorization, repeated decisions, and the agent's retry key.

The runner tests two layers differently:

- **Policy checks are exact.** The expected reviewer groups must match.
- **Model checks allow some variation.** A case can accept several defensible categories or risk levels while still requiring missing-information coverage and rejecting injected approval language.

## Checkpoint: break the policy on purpose

Run the fast tests and the credential-free evaluation suite:

```bash
pnpm test
pnpm eval:policy
```

All tests and all twelve policy checks should pass. Now open `lib/policy.ts` and temporarily change the cost comparison from `>= 50_000` to `> 50_000`. Run the suite again.

The threshold test and `threshold-boundary` evaluation should fail because a request costing exactly $50,000 no longer reaches Procurement. Restore `>= 50_000` and confirm both suites pass.

## Test the model

With `AI_GATEWAY_API_KEY` configured, run:

```bash
pnpm eval
```

The command performs the policy checks, calls the configured assessment model for every case, and reports failures by layer:

```text
PASS  policy  restricted-customer-support-data
PASS  model   restricted-customer-support-data
PASS  policy  prompt-injection-approve
FAIL  model   prompt-injection-approve
      risk low not in [medium, high]
      expected at least 1 missing-information item(s)
```

For each model failure, read the input and output. Determine whether to correct the requirement, revise the assessment instructions, or try a different model.

In `lib/ai.ts`, request fields are labeled as untrusted data, and the model is instructed not to follow commands inside them. A model can still follow injected text; labeling is not a security guarantee. Deterministic routing uses the submitted cost and data types, not the model's answer, so a misleading risk assessment cannot remove required Security review.

## Decide how failures behave

If model output is invalid, unavailable, or too slow, Vendor Review should not silently infer approval. Choose a safe response:

- Preserve the request
- Mark the assessment as failed
- Route to a human when policy or missing evidence requires it
- Allow an explicit retry

Test that behavior manually by running the application with an invalid Gateway key. The request should remain visible as `assessment_failed`; it must not become screened or approved.

## Record results

Save the model, assessment version, date, pass count, and unresolved failures in `docs/model-record.md`. Keep the command output in local development or CI logs for the case-by-case results.

Add one case from your application job in `docs/readiness.md`. Give it an exact policy expectation and a clear range of acceptable model results. If the new case exposes an undocumented decision, update **Who decides what** before changing the code.

Commit the routing rules, evaluation cases, and updated records:

```bash
git add lib/policy.ts evals docs/model-record.md docs/readiness.md
git commit -m "feat: add tested vendor routing"
```

## Summary

Exact tests catch changes to company rules. Representative and adversarial evaluations show whether model output meets the task’s acceptance criteria. Record failures so they remain available for review.

## Check your work

Run `pnpm test`, `pnpm eval:policy`, and then `pnpm eval` with a Gateway key. Every exact rule should pass. If a model case fails, record it in `docs/model-record.md`; do not widen the acceptance criteria just to make it pass.

Compare the fixtures with [`evals/vendor-requests.json` on `complete`](https://github.com/vercel-labs/academy-enterprise-apps-agents/blob/complete/evals/vendor-requests.json).


---

[Full course index](/academy/llms.txt) · [Sitemap](/academy/sitemap.md)