Self-Improving Agent Builder Purpose Run a closed-loop improvement cycle on any goal-seeking agent implementation: EVAL -> ANALYZE -> RESEARCH -> IMPROVE -> RE-EVAL -> DECIDE -> (repeat) Each iteration measures L1-L12 progressive test scores, identifies failures with error_analyzer.py , runs a research step with hypothesis/evidence/ counter-arguments, applies targeted fixes, and gates promotion through regression checks. When I Activate "improve agent" or "self-improving loop" "agent eval loop" or "run improvement cycle" "benchmark agents" or "compare SDK implementations" "iterate on agent scores" or "fix agent regressions" Quick Start User: "Run the self-improving loop on the mini-framework agent for 3 iterations" Skill: Executes 3 iterations of EVAL->ANALYZE->RESEARCH->IMPROVE->RE-EVAL->DECIDE Reports per-iteration scores, net improvement, and commits/reverts. Runner Script The self-improvement loop is implemented as a Python CLI:

Basic usage

python -m amplihack.eval.self_improve.runner --sdk mini --iterations 3

Full options

python -m amplihack.eval.self_improve.runner \ --sdk mini \ --iterations 5 \ --improvement-threshold 2.0 \ --regression-tolerance 5.0 \ --levels L1 L2 L3 L4 L5 L6 \ --output-dir ./eval_results/self_improve \ --dry-run

evaluate only, don't apply changes

Source: src/amplihack/eval/self_improve/runner.py The Loop (6 Phases per Iteration) Phase 1: EVAL Run the L1-L12 progressive test suite on the current agent implementation. Execution: python -m amplihack.eval.progressive_test_suite \ --agent-name < agent_name

\ --output-dir < output_dir

/iteration_N/eval \ --levels L1 L2 L3 L4 L5 L6 Output: Per-level scores and overall baseline. Phase 2: ANALYZE Classify failures using error_analyzer.py . Maps each failed question to a failure taxonomy (retrieval_insufficient, temporal_ordering_wrong, etc.) and the specific code component responsible. from amplihack . eval . self_improve import analyze_eval_results analyses = analyze_eval_results ( level_results , score_threshold = 0.6 )

Each ErrorAnalysis maps to:

failure_mode -> affected_component -> prompt_template

Phase 3: RESEARCH (New)

The critical thinking step that prevents blind changes. For each proposed

improvement:

State hypothesis

What specific change will fix the failure?

Gather evidence

From eval results, failure patterns, baseline scores

Consider counter-arguments

What could go wrong? Risk of regression?

Make decision

Apply, skip, or defer with full reasoning

Decisions are logged in

research_decisions.json

for auditability.

Decision criteria:

Apply

Clear failure pattern + prompt template available + low score

Skip

Score above 50% (likely stochastic variation)

Defer

Ambiguous evidence, needs more data

Phase 4: IMPROVE

Apply the improvements approved by the research step. Priority order:

Prompt template improvements (safest, highest impact)

Retrieval strategy adjustments

Code logic fixes (most risky, needs careful review)

Phase 5: RE-EVAL

Re-run the same eval suite after applying fixes to measure impact.

Phase 6: DECIDE

Promotion gate:

Net improvement >= +2% overall score: COMMIT the changes

Any single level regression > 5%: REVERT all changes

Otherwise: COMMIT with marginal improvement note

Configuration

Parameter

Default

Description

sdk_type

mini

Which SDK: mini/claude/copilot/microsoft

max_iterations

5

Maximum improvement iterations

improvement_threshold

2.0

Minimum % improvement to commit

regression_tolerance

5.0

Maximum % regression on any level

levels

L1-L6

Which levels to evaluate

output_dir

./eval_results/self_improve

Results directory

dry_run

false

Evaluate only, don't apply changes

Programmatic Usage

from

amplihack

.

eval

.

self_improve

import

run_self_improvement

,

RunnerConfig

config

=

RunnerConfig

(

sdk_type

=

"mini"

,

max_iterations

=

3

,

improvement_threshold

=

2.0

,

regression_tolerance

=

5.0

,

levels

=

[

"L1"

,

"L2"

,

"L3"

,

"L4"

,

"L5"

,

"L6"

]

,

output_dir

=

"./eval_results/self_improve"

,

dry_run

=

False

,

)

result

=

run_self_improvement

(

config

)

print

(

f"Total improvement:

{

result

.

total_improvement

:

+.1f

}

%"

)

print

(

f"Final scores:

{

result

.

final_scores

}

"

)

4-Way Benchmark Mode

Compare all SDK implementations side by side:

User: "Run a 4-way benchmark comparing all SDK implementations"

Skill: Runs eval suite on mini, claude, copilot, microsoft

Generates comparison table with scores, LOC, and coverage.

Integration Points

src/amplihack/eval/self_improve/runner.py

Self-improvement loop runner

src/amplihack/eval/self_improve/error_analyzer.py

Failure classification

src/amplihack/eval/progressive_test_suite.py

L1-L12 eval runner

src/amplihack/agents/goal_seeking/sdk_adapters/

All 4 SDK implementations

src/amplihack/eval/metacognition_grader.py

Advanced eval dimensions

src/amplihack/eval/teaching_session.py

L7 teaching quality eval