Evaluate agentskills.io-style skills from a CLI, SDK, or CI workflow.

Quickstart

npm install @agilelab/agent-skills-eval

OPENAI_BASE_URL=https://api.openai.com/v1 \
OPENAI_API_KEY=... \
npx @agilelab/agent-skills-eval ./skills \
  --target gpt-4o-mini \
  --judge gpt-4o-mini \
  --baseline \
  --strict

Run Modes

Target/judge calls can go through a plain OpenAI-compatible API, or be routed through an agentic CLI you already run day to day — opencode or Claude Code. In those two modes, skills aren't injected into the prompt at all: the provider symlinks the skill onto disk and the agent discovers and invokes it itself via its own skill tool, exactly like it would for a real user. That makes the eval a true test of whether your skill gets picked up in the wild — plus you get the agent's own tool use, permissions, and subagent delegation for free. See the README for full flag references, config, and caveats for each.

npx @agilelab/agent-skills-eval ./skills \
  --run-mode opencode \
  --target anthropic/claude-sonnet-5 \
  --opencode-agent build

npx @agilelab/agent-skills-eval ./skills \
  --run-mode claude-code \
  --target claude-sonnet-5

YAML Config

Use config files for repeatable local and CI runs. CLI flags override config values.

root: ./skills
workspace: ./agent-skills-workspace
baseline: true
target: gpt-4o-mini
judge: gpt-4o-mini
baseUrl: https://api.openai.com/v1
apiKeyEnv: OPENAI_API_KEY
concurrency: 4
layout: iteration
strict: true
report:
  enabled: true
  title: Agent Skills Report
logging:
  format: pretty
  verbose: false
targetParams:
  temperature: 0
judgeParams:
  temperature: 0

SDK Usage

import {
  OpenAICompatibleProvider,
  consoleReporter,
  evaluateSkills,
} from "@agilelab/agent-skills-eval";

const provider = new OpenAICompatibleProvider({
  baseUrl: "https://api.openai.com/v1",
  apiKey: process.env.OPENAI_API_KEY!,
  model: "gpt-4o-mini",
  providerName: "openai",
});

const result = await evaluateSkills({
  root: "./skills",
  workspace: "./agent-skills-workspace",
  baseline: true,
  workspaceLayout: "iteration",
  strict: true,
  target: { model: provider.model, provider },
  judge: { model: provider.model, provider },
  onEvent: consoleReporter(),
});

Skill Format

---
name: csv-analyzer
description: Analyze CSV files.
license: MIT
---

Identify trends and cite the relevant rows.

{
  "skill_name": "csv-analyzer",
  "evals": [
    {
      "id": "basic",
      "prompt": "Find the highest revenue month.",
      "files": ["evals/files/revenue.csv"],
      "expected_output": "The response names the highest revenue month.",
      "assertions": [
        "The answer names February."
      ]
    }
  ]
}

Artifacts And Reports

Runs produce the official iteration-N workspace layout with prompts, outputs, timings, grading, optional tool calls, benchmarks, and a static HTML report.

agent-skills-workspace/
  iteration-1/
    meta.json
    benchmark.json
    eval-basic/
      with_skill/
        prompts.json
        timing.json
        grading.json
        outputs/response.txt
      without_skill/
        ...