Baike.dev
All toolsOnline toolsAI codingTrendingOpen sourceNewsSubmit
Log in
< Back to tools
C

compl-ai

> 编程语言
Open source

An open-source compliance-centered evaluation framework for Generative AI models

200 stars0 likes0 views
WebsiteGitHub

About

An open-source compliance-centered evaluation framework for Generative AI models

Overview

The COMPL-AI framework includes a technical interpretation of the EU AI Act and an open-source benchmarking suite (this repo). The key features are:

  • Built on the Inspect evaluation framework
  • Tailored set of benchmarks to provide coverage over technical parts of EU AI Act (29 and growing)
  • A public Hugging Face leaderboard of our latest evaluation results
  • Extensive set of supported providers (API, Cloud, Local).
  • A custom eval CLI (run complai --help for usage)

Community contributions for benchmarks and new mappings are welcome! We are actively looking to expand our EU AI Act and Code of Practice technical interpretation and benchmark coverage. See the contributing section below.

⏩ Quickstart

To run an evaluation yourself, please follow the instructions below (or contact us through compl-ai.org).

bash
# Clone and create a virtual environment
git clone https://github.com/compl-ai/compl-ai.git
cd compl-ai
uv sync
source .venv/bin/activate

# Set your API key
export OPENAI_API_KEY=your_key

# Run 5 samples on a single benchmark
complai eval openai/gpt-5-nano --tasks mmlu_pro --limit 5

# Or run the full framework
complai eval openai/gpt-5-nano

You can then view a detailed sample-level log of your results with the Inspect AI VS Code extension, or in your browser with:

inspect view

CLI

Get help with any command

bash
complai COMMAND --help

List Tasks

bash
complai list

Collect Sample Inputs

Export samples as JSONL:

bash
complai samples samples.jsonl
complai samples samples.jsonl --tasks mmlu_pro,human_deception --limit 10
complai samples arc.jsonl --task-spec inspect_evals/arc_challenge

Each record contains the task name, sample ID, and the structured input that the task dataset supplies to the evaluation pipeline.

Run Evals with the following syntax

bash
complai eval / -t  -l 

Command Examples

…

Task Configuration

COMPL-AI supports task-specific configuration via YAML or JSON files. See default_config.yaml for a reference of all configurable parameters. You can:

  • Use the default config as a template: cp default_config.yaml my_config.yaml
  • Modify the tasks and parameters you want
  • Pass it to any eval: complai eval --task-config my_config.yaml

Providers

See the Providers section for more information on different providers.

Environment Variables

COMPL-AI can auto-load models (COMPLAI_MODEL), API keys (OPENAI_API_KEY), and many other configurations (COMPLAI_LOG_DIR) from your local .env file. Values provided in the CLI take precedence over .env vars.

Framework

Update in Progress

The current version of the framework is published here.

We are currently in the process of renewing our coverage of the EU AI Act by updating the set of benchmarks, thus the supported set of benchmark may differ from this original mapping. The goals of this update are:

  • To increase coverage over the EU AI Act principles
  • To increase coverage over technical requirements
  • Adding support for the Code of Practice, namely the Safety and Security chapter.
  • Adding the notion of risk along side technical requirements
  • Refreshing the supported benchmarks to ensure they remain challenging for frontier models (addressing saturation, contamination, and other benchmark quality issues).

As part of our update, the renewed benchmarking suite is now built on the UK Security Institute’s Inspect Framework, which offers improved ease of use and greater overall consistency. This means that several benchmarks are now evaluated differently (e.g. full-text answers instead of logits for multiple-choice questions) reflecting more modern and opinionated approaches to LLM evaluation. Thus, benchmark scores from the v1 and v2 suites, even for the same benchmark, should not be considered directly comparable.

Principles

COMPL-AI is primarily structured to provide coverage over 6 core EU AI Act principles:

  • Human Agency and Oversight: AI systems should be supervised by people, not by automation alone, to prevent harmful outcomes and allow for human intervention.
  • Technical Robustness and Safety: AI systems must be safe and secure, implementing risk management, data quality, and - cybersecurity measures to prevent undue risks.
  • Privacy and Data Governance: The Act sets rules for the quality and governance of data used in AI, emphasizing the protection of personal and sensitive information.
  • Transparency: Users should understand when they are interacting with an AI system and how it functions, fostering trust and enabling accountability.
  • Diversity, Non-Discrimination, and Fairness: AI systems should be designed and used to uphold human rights, including fairness and equality, and avoid biases that could lead to discrimination.
  • Societal and Environmental Well-being: AI systems should be developed in a way that benefits society and the environment, avoiding negative impacts on fundamental rights and democratic values.

Technical Requirements and Benchmarks

You can see a list of all technical requirements and their respective benchmarks using complai list:

  • Capabilities, Performance, and Limitations

    • aime_2025, arc_challenge, gpqa_diamond, hle, ifbench, include, livebench_coding, mmlu_pro, swe_bench_verified
  • Representation — Absence of Bias

    • bbq, bold, cab
  • Interpretability

    • bigbench_calibration, triviaqa_calibration
  • Robustness and Predictability

    • boolq_contrast, forecast_consistency, imdb_contrast, mmlu_pro_robustness, self_check_consistency
  • Fairness — Absence of Discrimination

    • decoding_trust, fairllm
  • Disclosure of AI

    • human_deception
  • Cyberattack Resilience

    • instruction_goal_hijacking, llm_rules, strong_reject
  • Societal Alignment

    • mask, simpleqa_verified, truthfulqa
  • Harmful Content and Toxicity

    • realtoxicityprompts

Contributing

We welcome contributions! When contributing, please make sure to activate pre-commit hooks to ensure code quality and consistency. You can install pre-commit hooks with:

bash
pip install pre-commit
pre-commit install

To run tests, run the following command:

bash
make test

License

This project is licensed under the Apache 2.0 License - see LICENSE for details.

Citation

Please cite our work as follows:

@article{complai24,
      title={COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act}, 
      author={Philipp Guldimann and Alexander Spiridonov and Robin Staab and Nikola Jovanovi\'{c} and Mark Vero and Velko Vechev and Anna Gueorguieva and Mislav Balunovi\'{c} and Nikola Konstantinov and Pavol Bielik and Petar Tsankov and Martin Vechev},
      year={2024},
      eprint={2410.07959},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.07959},
}

Issues· 4 open

View all issuesOpen on GitHub

No open issues yet, or sync has not completed.

> Tags

Python

No comments yet. Be the first to share.

> Details

PublishedAug 1, 2026
UpdatedSep 18, 2026
Category编程语言
PricingOpen source

> Related tools

T
TypeScript
JavaScript 的超集,为前端与全栈提供静态类型
P
Python
通用编程语言,广泛用于 Web、数据与 AI
G
Go
Google 推出的简洁高效系统语言