skywalkskills

CatalogueDelivery Documentation & QA

H5Ready

Run a Skill Against the Demo Corpus

Runs one skill against the claims portal and scores what it found against the problems planted there.

What it examines

  • A seeded copy of the portal, so the skill never sees the answer key
  • The skill you name, run against that copy
  • What was caught, what was missed, and what was a false alarm
  • A saved score you can compare with the next run

What you get back

A score for that skill against the claims portal.

Dependencies

  • An AI coding agent that reads SKILL.md files, such as Claude Code, Codex or Gemini CLI
  • A clone of the full skills repository. The skill uses its demos folder, schema folder and scoring script, so the download alone is not enough.
  • Bash and Git
  • Docker with Compose, for the skills that need the live app (H3 and H4) or a presenter demo
  • Python in a virtual environment at .venv at the repository root, with the repository’s tooling installed
  • The skill you want to score, already installed

How it works

It makes a throwaway copy of the ClaimsPortal demo with planted problems, and checks that the answer file is not in the copy, so the skill under test never sees it.

You name a skill. It runs that skill against the copy, following that skill’s own instructions, and saves its findings.json outside the copy.

It then scores the findings against the planted problems: what was caught, what was missed and what was a false alarm. Scores can be saved to compare with the next run. Planted problems are never fixed unless you ask.

Setup

  1. Clone the skills repository and open it in your agent.
  2. Create the Python environment at the repository root: python3 -m venv .venv, then pip install -e ".[dev]".
  3. Make sure the skill you want to score is installed. If it is H3 or H4, start Docker.
  4. Ask your agent to run the run-demo-skill-eval skill and name the skill to score.
  5. Check and score the result: python schema/validate.py on the findings, then python demos/score.py with the --save flag.

Run it with your agent

Copy the prompt, fill in the parts in square brackets, and paste it into your agent.

Prompt

Run the run-demo-skill-eval skill. Skill to score: [for example A4]. Prepare a seeded copy of the demo so the answer file is not visible, run that skill against it, then score the findings. Do not fix any planted problems.

What is inside

The files that make up this skill, as your agent reads them.

packages/h5-demo-skill-eval

skill › SKILL.md

Sign in to download

Use the email and password you signed up with.