CatalogueDelivery Documentation & QA
Run a Skill Against the Demo Corpus
Runs one skill against the claims portal and scores what it found against the problems planted there.
What it examines
- A seeded copy of the portal, so the skill never sees the answer key
- The skill you name, run against that copy
- What was caught, what was missed, and what was a false alarm
- A saved score you can compare with the next run
What you get back
A score for that skill against the claims portal.
Dependencies
- An AI coding agent that reads SKILL.md files, such as Claude Code, Codex or Gemini CLI
- A clone of the full skills repository. The skill uses its demos folder, schema folder and scoring script, so the download alone is not enough.
- Bash and Git
- Docker with Compose, for the skills that need the live app (H3 and H4) or a presenter demo
- Python in a virtual environment at .venv at the repository root, with the repository’s tooling installed
- The skill you want to score, already installed
How it works
It makes a throwaway copy of the ClaimsPortal demo with planted problems, and checks that the answer file is not in the copy, so the skill under test never sees it.
You name a skill. It runs that skill against the copy, following that skill’s own instructions, and saves its findings.json outside the copy.
It then scores the findings against the planted problems: what was caught, what was missed and what was a false alarm. Scores can be saved to compare with the next run. Planted problems are never fixed unless you ask.
Setup
- Clone the skills repository and open it in your agent.
- Create the Python environment at the repository root: python3 -m venv .venv, then pip install -e ".[dev]".
- Make sure the skill you want to score is installed. If it is H3 or H4, start Docker.
- Ask your agent to run the run-demo-skill-eval skill and name the skill to score.
- Check and score the result: python schema/validate.py on the findings, then python demos/score.py with the --save flag.
Run it with your agent
Copy the prompt, fill in the parts in square brackets, and paste it into your agent.
Run the run-demo-skill-eval skill. Skill to score: [for example A4]. Prepare a seeded copy of the demo so the answer file is not visible, run that skill against it, then score the findings. Do not fix any planted problems.