skill-evolution

仓库: heyvhuang/ship-faster
安装量: 51
排名: #14402

安装

npx skills add https://github.com/yonatangross/orchestkit --skill skill-evolution

Enables skills to automatically improve based on usage patterns, user edits, and success rates. Provides version control with safe rollback capability.

Overview

  • Reviewing how skills are performing across sessions

  • Identifying patterns in user edits to skill outputs

  • Applying learned improvements to skill templates

  • Rolling back problematic skill changes

  • Tracking skill version history and success rates

Quick Reference

| /ork:skill-evolution | Show evolution report for all skills

| /ork:skill-evolution analyze <skill-id> | Analyze specific skill patterns

| /ork:skill-evolution evolve <skill-id> | Review and apply suggestions

| /ork:skill-evolution history <skill-id> | Show version history

| /ork:skill-evolution rollback <skill-id> <version> | Restore previous version

How It Works

The skill evolution system operates in three phases:

COLLECT                    ANALYZE                    ACT
───────                    ───────                    ───
┌─────────────┐           ┌─────────────┐           ┌─────────────┐
│ PostTool    │──────────▶│ Evolution   │──────────▶│ /ork:skill  │
│ Edit        │  patterns │ Analyzer    │ suggest   │ evolve      │
│ Tracker     │           │ Engine      │           │ command     │
└─────────────┘           └─────────────┘           └─────────────┘
     │                          │                          │
     ▼                          ▼                          ▼
┌─────────────┐           ┌─────────────┐           ┌─────────────┐
│ edit-       │           │ evolution-  │           │ versions/   │
│ patterns.   │           │ registry.   │           │ snapshots   │
│ jsonl       │           │ json        │           │             │
└─────────────┘           └─────────────┘           └─────────────┘

Edit Pattern Categories

The system tracks these common edit patterns:

| add_pagination | User adds pagination to API responses | limit.*offset, cursor.*pagination

| add_rate_limiting | User adds rate limiting | rate.?limit, throttl

| add_error_handling | User adds try/catch blocks | try.*catch, except

| add_types | User adds TypeScript/Python types | interface\s, Optional

| add_validation | User adds input validation | validate, Pydantic, Zod

| add_logging | User adds logging/observability | logger\., console.log

| remove_comments | User removes generated comments | Pattern removal detection

| add_auth_check | User adds authentication checks | @auth, @require_auth

Suggestion Thresholds

| Minimum Samples | 5 | Uses before generating suggestions

| Add Threshold | 70% | Frequency to suggest adding pattern

| Auto-Apply Confidence | 85% | Confidence for auto-application

| Rollback Trigger | -20% | Success rate drop to trigger rollback

Subcommand: Report (Default)

Usage: /ork:skill-evolution

Shows evolution report for all tracked skills.

Implementation

# Run the evolution engine report
"${CLAUDE_PROJECT_DIR}/.claude/scripts/evolution-engine.sh" report

Sample Output

Skill Evolution Report
══════════════════════════════════════════════════════════════

Skills Summary:
┌────────────────────────────┬─────────┬─────────┬───────────┬────────────┐
│ Skill                      │ Uses    │ Success │ Avg Edits │ Suggestions│
├────────────────────────────┼─────────┼─────────┼───────────┼────────────┤
│ api-design-framework       │     156 │     94% │       1.8 │          2 │
│ database-schema-designer   │      89 │     91% │       2.1 │          1 │
│ fastapi-patterns           │      67 │     88% │       2.4 │          3 │
└────────────────────────────┴─────────┴─────────┴───────────┴────────────┘

Summary:
  Skills tracked: 3
  Total uses: 312
  Overall success rate: 91%

Top Pending Suggestions:
1. 93% | api-design-framework | add add_pagination
2. 88% | api-design-framework | add add_rate_limiting
3. 85% | fastapi-patterns | add add_error_handling

Subcommand: Analyze

Usage: /ork:skill-evolution analyze <skill-id>

Analyzes edit patterns for a specific skill.

Implementation

# Run analysis for specific skill
"${CLAUDE_PROJECT_DIR}/.claude/scripts/evolution-engine.sh" analyze "$SKILL_ID"

Sample Output

Skill Analysis: api-design-framework
────────────────────────────────────
Uses: 156 | Success: 94% | Avg Edits: 1.8

Edit Patterns Detected:
┌──────────────────────────┬─────────┬──────────┬────────────┐
│ Pattern                  │ Freq    │ Samples  │ Confidence │
├──────────────────────────┼─────────┼──────────┼────────────┤
│ add_pagination           │    85%  │ 132/156  │       0.93 │
│ add_rate_limiting        │    72%  │ 112/156  │       0.88 │
│ add_error_handling       │    45%  │  70/156  │       0.56 │
└──────────────────────────┴─────────┴──────────┴────────────┘

Pending Suggestions:
1. 93% conf: ADD add_pagination to template
2. 88% conf: ADD add_rate_limiting to template

Run `/ork:skill-evolution evolve api-design-framework` to review

Subcommand: Evolve

Usage: /ork:skill-evolution evolve <skill-id>

Interactive review and application of improvement suggestions.

Implementation

When this subcommand is invoked:

  • Get Suggestions:
SUGGESTIONS=$("${CLAUDE_PROJECT_DIR}/.claude/scripts/evolution-engine.sh" suggest "$SKILL_ID")
  • For Each Suggestion, Present Interactive Options:

Use AskUserQuestion to let the user decide on each suggestion:

{
  "questions": [{
    "question": "Apply suggestion: ADD add_pagination to template? (93% confidence, 132/156 users add this)",
    "header": "Evolution",
    "options": [
      {"label": "Apply", "description": "Add this pattern to the skill template"},
      {"label": "Skip", "description": "Skip for now, ask again later"},
      {"label": "Reject", "description": "Never suggest this again"}
    ],
    "multiSelect": false
  }]
}
  • On Apply:

Create version snapshot first

  • Apply the suggestion to skill files

  • Update evolution registry

  • On Reject:

Mark suggestion as rejected in registry

  • Won't be suggested again

Applying Suggestions

When a user accepts a suggestion, the implementation depends on the suggestion type:

For add suggestions to templates:

  • Add the pattern to the skill's template files

  • Update SKILL.md with new guidance

For add suggestions to references:

  • Create new reference file in references/ directory

For remove suggestions:

  • Remove the identified content

  • Archive in version snapshot first

Subcommand: History

Usage: /ork:skill-evolution history <skill-id>

Shows version history with performance metrics.

Implementation

# Run version manager list
"${CLAUDE_PROJECT_DIR}/.claude/scripts/version-manager.sh" list "$SKILL_ID"

Sample Output

Version History: api-design-framework
══════════════════════════════════════════════════════════════

Current Version: 1.2.0

┌─────────┬────────────┬─────────┬───────┬───────────┬────────────────────────────┐
│ Version │ Date       │ Success │ Uses  │ Avg Edits │ Changelog                  │
├─────────┼────────────┼─────────┼───────┼───────────┼────────────────────────────┤
│ 1.2.0   │ 2026-01-14 │    94%  │   156 │       1.8 │ Added pagination pattern   │
│ 1.1.0   │ 2026-01-05 │    89%  │    80 │       2.3 │ Added error handling ref   │
│ 1.0.0   │ 2025-11-01 │    78%  │    45 │       3.2 │ Initial release            │
└─────────┴────────────┴─────────┴───────┴───────────┴────────────────────────────┘

Subcommand: Rollback

Usage: /ork:skill-evolution rollback <skill-id> <version>

Restores a skill to a previous version.

Implementation

  • Confirm with User:

Use AskUserQuestion for confirmation:

{
  "questions": [{
    "question": "Rollback api-design-framework from 1.2.0 to 1.0.0? Current version will be backed up.",
    "header": "Rollback",
    "options": [
      {"label": "Confirm Rollback", "description": "Restore version 1.0.0"},
      {"label": "Cancel", "description": "Keep current version"}
    ],
    "multiSelect": false
  }]
}
  • On Confirm:
"${CLAUDE_PROJECT_DIR}/.claude/scripts/version-manager.sh" restore "$SKILL_ID" "$VERSION"
  • Report Result:
Restored api-design-framework to version 1.0.0
Previous version backed up to: versions/.backup-1.2.0-1736867234

Data Files

| .claude/feedback/edit-patterns.jsonl | Raw edit pattern events | JSONL (append-only)

| .claude/feedback/evolution-registry.json | Aggregated suggestions | JSON

| .claude/feedback/metrics.json | Skill usage metrics | JSON

| skills/<cat>/<name>/versions/ | Version snapshots | Directory

| skills/<cat>/<name>/versions/manifest.json | Version metadata | JSON

Auto-Evolution Safety

The system includes safety mechanisms:

  • Version Snapshots: Always created before changes

  • Rollback Triggers: Auto-alert if success rate drops >20%

  • Human Review: High-confidence suggestions require approval

  • Rejection Memory: Rejected suggestions aren't re-suggested

Health Monitoring

The system monitors skill health and can trigger warnings:

WARNING: api-design-framework success rate dropped from 94% to 71%
Consider: /ork:skill-evolution rollback api-design-framework 1.1.0

References

  • configure - Configure OrchestKit settings

  • doctor - Diagnose OrchestKit issues

  • feedback-dashboard - View comprehensive feedback metrics

← 返回排行榜