Proposal: Add context-benchmarking Skill and CI Example
I've been diving into the advanced-evaluation and context-compression skills and found them incredibly useful. I noticed there is currently a gap between designing these evaluations and operationalizing them in a production workflow.
I would like to propose adding a new Operational Skill and a corresponding Example to bridge this gap.
The Proposal:
New Skill (skills/operational-skills/context-benchmarking): A guide on how to set up a benchmarking feedback loop. This would cover metrics (latency vs. recall), preventing regression in retrieval quality, and safeguards for automated testing.
New Example (examples/automated-benchmarking-pipeline): A working GitHub Action workflow that implements the existing llm-as-judge logic.
Implementation Details:
The example pipeline will use a Python script to run a test set against an agent configuration.
Cost Safety: To ensure this doesn't spike API usage for forks or the main repo, the GitHub Action will be configured to run only on workflow_dispatch (manual trigger) or specific PR labels (e.g., run-benchmark).
Why this matters:
As the repo grows, users need a way to verify that new context compression strategies (like context-optimization) don't accidentally degrade information retrieval. This validates the "theory" of the other skills.
Would you be open to a PR for this?
Source: muratcankoylan/Agent-Skills-for-Context-Engineering