Set Up CI with the CLI | SkillEvaluator Documentation

Bring skill evaluation to your GitHub repository in one guided setup. SkillEvaluator finds your skills, prepares the workflows and opens a setup PR. If your repository already has skill checks, it can add reporting to that setup. Your CI gets Tier 1 checks, including SkillSpector security scanning.

Run inside your checked-out GitHub repository

pip install --upgrade skillevaluator
skillevaluator ci init

Adding reports to an existing shared component and using private Pages require SkillEvaluator 0.7.2 or later.

1. Choose

Select skill folders, a score threshold and feedback.

2. Merge

Review the setup PR and merge it when ready.

3. See results

Open a skill PR and follow its check, comment and report.

No evaluation code to copy into your repository. No CI scanners to install locally. The shared container supplies the runtime; you control the configuration.

What do I need before I start?

Use Python 3.12–3.14, Git and the GitHub CLI. Start from a clean checkout with a GitHub origin remote. Sign in with gh auth login --scopes workflow if you have not already done so. The login needs permission to add workflow files as well as access to the repository.

Automatic GitHub setup requires repository admin access. GitHub Actions and the referenced reusable workflows must be allowed. Checks-only setup supports public and private repositories. Hosted feedback supports public Pages and verified private Pages sites in eligible organization repositories. See private report hosting for the one-time prerequisite. This wizard supports GitHub. For GitLab, use the GitLab template.

Run the wizard

For a new scan, the wizard discovers folders containing SKILL.md. Answer five short questions; press Enter to keep a default.

Question	Your choice
Skill folders	Use the displayed numbers: `1,2` or `1 2` selects two folders.
Minimum score	`70` by default; a lower quality score fails the check.
PR scope	`auto` checks changed skills; `all` checks the selected collection.
Hosted feedback	Add hosted HTML reports and one PR comment that updates with new results.
Apply setup	Choose `pr`, review the summary, then confirm.

╭─ 1/5  Which skill folders? ───────────╮
│ 1  skills          12                │
│ 2  .agents/skills   3                │
│ Use folder numbers: 1,2 or 1 2.      │
╰─────────────────────────────────────╯
Folders [1]: 1,2
✓ Selected: skills, .agents/skills (15 skills)

Shortened example interview. Folder names and counts come from your repository.

The score threshold does not override required security or schema checks. Default Tier 1 checks use no model key. Tier 2 and Tier 3 are not part of this setup.

Already have .github/workflows/skill-checks.yml? The CLI recognizes a Skill checks workflow with one reusable scan job. It keeps your existing inputs, triggers and workflow reference, including a shared component’s @v3 tag. Review those settings, then choose whether to add hosted feedback. Reporting adds id-token: write to the scan job when needed and prepares the publisher workflow.

Shared components

If your scan calls a shared component such as your-org/ci/.github/workflows/skill-checks.yml@v3, its maintainer first grants id-token: write in the component job that calls the evaluator. This is a one-time component change; the CLI checks it before applying reporting setup. Then run skillevaluator ci init in each consuming repository.

The setup keeps @v3 and approves that component to select the evaluator revision. When the maintainer updates v3, future scans use the updated component without a consumer edit. Each published report retains the resolved component and evaluator commits in its execution receipt.

Private report hosting

Private Pages requires a private or internal project repository owned by a GitHub Enterprise Cloud organization. An administrator must first open Settings → Pages, choose GitHub Actions as the source and set the site visibility to Private. GitHub’s REST API cannot change site visibility, so this step is done in GitHub’s settings.

Rerun skillevaluator ci init after that one-time setup. The CLI verifies the site’s private visibility and uses GitHub’s assigned private URL. It can reuse an existing SkillEvaluator report site or a newly configured empty private site; an unrelated website is preserved.

Personal repositories cannot host access-controlled Pages. If private Pages is unavailable, choose checks only to retain reports in Actions artifacts. Repository privacy alone does not restrict access to a Pages site. See GitHub’s private Pages requirements.

Review your setup

After confirmation, the CLI configures your selected GitHub settings and opens a PR with the workflow files:

What changes in your repository

.github/workflows/
├── skill-checks.yml             Checks + downloadable reports
└── publish-skill-reports.yml    Optional hosting + PR comments

╭─ ✓ Setup PR ready ───────────────────╮
│ Merge this PR to enable checks.      │
╰─────────────────────────────────────╯
https://github.com/your-org/skills/pull/12
After merging, optionally create a test PR:
  skillevaluator ci demo

Example completion screen. Your CLI prints the real setup PR link.

Review and merge the setup PR yourself. The CLI does not merge it. A setup-only run may say Skipped because no skill changed.

Try a test PR

After merging setup, run:

skillevaluator ci demo

The CLI asks before adding a small Markdown comment to one selected SKILL.md on a new branch. Type y to create the test PR, or press Enter to skip. You can also make a small skill edit and open a PR yourself.

╭─ ✓ Test PR created ──────────────────────────────╮
│ Checks → Skill checks                            │
│ Bot comment → Open full report                   │
│ Comment follows publication. Close PR when done. │
╰─────────────────────────────────────────────────╯
https://github.com/your-org/skills/pull/13

Example test PR output with hosted feedback enabled. Checks-only setups link to the downloadable report.

Close the test PR when you finish; you do not need to merge it.

See the result on your PR

github-actions[bot]

Tier 1 scan result: PASSED

1 skill · 1 passed · 0 failed · 0 incomplete

Skill	Status	Quality / 100
`skills/evidence-table`	PASS	98.2 / A

Open full report · View workflow run

Illustrative bot comment. Each run supplies its own verdict, score and report links.

Where to look	What you get
PR → Checks	The scan result for the current commit.
PR → Conversation, with publishing enabled	One bot comment, updated when a new scan is published.
Open full report in that comment	Detailed HTML findings and scores.

Checks only: open the scan in Actions, download its report artifact, extract it and open index.html. Failed scans retain their available reports too.

Can I inspect the setup before applying it?

Choose the read-only option in the wizard to see the plan without changing files or GitHub settings. Previewing an existing scan workflow makes no network requests; GitHub access and private Pages checks happen when you apply setup. The save option writes only the local workflow files; choose pr for automatic GitHub setup.

Will this change my other workflows or branch rules?

The CLI preserves the settings of a recognized existing scan and leaves other workflows separate. It does not automatically make the scan a required merge check. After the first run, select its emitted check in GitHub branch rules if a failed or incomplete result should block merging; the current template emits skills / Tier 1 scan. Organization action restrictions are preserved. Repositories using an Actions allowlist must use the wizard’s save option, even when the required actions are already allowed. Review the saved files and complete setup through your normal repository process.

A scan ran, but I cannot find the comment

Confirm that setup is merged, the PR is open and its branch is in the same repository. Wait for Publish skill reports to finish; open that run if publication failed. Fork-origin PRs retain their available results in Actions. Checks-only setups do not post comments.

The check failed. Did onboarding fail?

Read the reported issue. A failed scan can still publish feedback: the result may identify a skill issue or a scanner that did not finish. Merging setup is not a passing evaluation. Push a fix to the same PR to rerun its checks and update the published feedback.

Prefer to copy a template?

Add the GitHub workflows yourself using the release files.

Using GitLab?

Add the shared scanner to your GitLab pipeline.