一个开源的以合规性为中心的评估框架,用于生成式 AI 模型
The COMPL-AI framework includes a technical interpretation of the EU AI Act and an open-source benchmarking suite (this repo). The key features are:
complai --help for usage)Community contributions for benchmarks and new mappings are welcome! We are actively looking to expand our EU AI Act and Code of Practice technical interpretation and benchmark coverage. See the contributing section below.
To run an evaluation yourself, please follow the instructions below (or contact us through compl-ai.org).
# Clone and create a virtual environment
git clone https://github.com/compl-ai/compl-ai.git
cd compl-ai
uv sync
source .venv/bin/activate
# Set your API key
export OPENAI_API_KEY=your_key
# Run 5 samples on a single benchmark
complai eval openai/gpt-5-nano --tasks mmlu_pro --limit 5
# Or run the full framework
complai eval openai/gpt-5-nanoYou can then view a detailed sample-level log of your results with the Inspect AI VS Code extension, or in your browser with:
inspect viewcomplai COMMAND --helpcomplai listExport samples as JSONL:
complai samples samples.jsonl
complai samples samples.jsonl --tasks mmlu_pro,human_deception --limit 10
complai samples arc.jsonl --task-spec inspect_evals/arc_challengeEach record contains the task name, sample ID, and the structured input that the task dataset supplies to the evaluation pipeline.
complai eval / -t -l …COMPL-AI supports task-specific configuration via YAML or JSON files. See default_config.yaml for a reference of all configurable parameters. You can:
cp default_config.yaml my_config.yamlcomplai eval --task-config my_config.yamlSee the Providers section for more information on different providers.
COMPL-AI can auto-load models (COMPLAI_MODEL), API keys (OPENAI_API_KEY), and many other configurations (COMPLAI_LOG_DIR) from your local .env file. Values provided in the CLI take precedence over .env vars.
The current version of the framework is published here.
We are currently in the process of renewing our coverage of the EU AI Act by updating the set of benchmarks, thus the supported set of benchmark may differ from this original mapping. The goals of this update are:
risk along side technical requirementsAs part of our update, the renewed benchmarking suite is now built on the UK Security Institute’s Inspect Framework, which offers improved ease of use and greater overall consistency. This means that several benchmarks are now evaluated differently (e.g. full-text answers instead of logits for multiple-choice questions) reflecting more modern and opinionated approaches to LLM evaluation. Thus, benchmark scores from the v1 and v2 suites, even for the same benchmark, should not be considered directly comparable.
COMPL-AI is primarily structured to provide coverage over 6 core EU AI Act principles:
You can see a list of all technical requirements and their respective benchmarks using complai list:
Capabilities, Performance, and Limitations
Representation — Absence of Bias
Interpretability
Robustness and Predictability
Fairness — Absence of Discrimination
Disclosure of AI
Cyberattack Resilience
Societal Alignment
Harmful Content and Toxicity
We welcome contributions! When contributing, please make sure to activate pre-commit hooks to ensure code quality and consistency. You can install pre-commit hooks with:
pip install pre-commit
pre-commit installTo run tests, run the following command:
make testThis project is licensed under the Apache 2.0 License - see LICENSE for details.
Please cite our work as follows:
@article{complai24,
title={COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act},
author={Philipp Guldimann and Alexander Spiridonov and Robin Staab and Nikola Jovanovi\'{c} and Mark Vero and Velko Vechev and Anna Gueorguieva and Mislav Balunovi\'{c} and Nikola Konstantinov and Pavol Bielik and Petar Tsankov and Martin Vechev},
year={2024},
eprint={2410.07959},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2410.07959},
}暂无开放 Issues,或尚未同步最近议题。