Since 2 August 2026, the Commission can impose Article 101 fines on general-purpose AI model providers for documentation and related Chapter V gaps. Article 5 prohibited practices have applied since 2 February 2025 (Art. 113(a), Chapters I and II). Fines reach up to €15M or 3% of worldwide annual turnover, whichever is higher, for GPAI providers (Article 101) and up to €35M or 7%, whichever is higher, for Article 5 prohibited-practice violations (Article 99(3)).
We read the public documentation of every major GPAI provider with material EU exposure and graded it against six obligation categories drawn from the Act. Across 12 providers, the average score is 53.1% on a 0-to-3 scale. Google (DeepMind) and Meta lead at 83.3%; xAI (Grok) trails at 16.7%. No provider scores the maximum on every obligation. Every score is still marked needs_review: a reading of public documents, not a finding. The leaderboard, provider evidence files, and scoring code are open source: anyone can re-run the benchmark and dispute a score with a public URL we missed.
What we measured
We score six public-documentation categories. Article cites below match Regulation (EU) 2024/1689 (not the legacy key names inside the JSON schema):
| Article | Category | Question |
|---|---|---|
| 50(2) | Content disclosure | Are synthetic outputs marked in a machine-readable format? (Chapter IV transparency; includes GPAI systems.) |
| 51 + 52 | Systemic-risk classification | Is there public evidence of threshold assessment (Art. 51) and Commission notification where required (Art. 52)? |
| 53(1)(a) | Technical documentation | Does public material meet Annex XI completeness for model technical documentation? |
| 53(1)(c) | Copyright transparency | Is there a published copyright policy and a workable rights-reservation / opt-out path? |
| 53(1)(b) | Downstream information | Is integrator documentation sufficient under Annex XII? (Training-content summary under Art. 53(1)(d) is adjacent; this bucket scores deployer-facing integration docs.) |
| 55 | Systemic risk | Evaluations, adversarial testing, incident reporting, cybersecurity: for models at or above the ≥10²⁵ FLOPs presumption in Art. 51(2). |
Each obligation is scored 0 (no evidence), 1 (partial), 2 (adequate), or 3 (exceeds). Only public documentation counts: model cards, terms of service, API docs, transparency reports.
Article 55 is marked N/A for providers we treat as below the systemic-risk threshold (Mistral, Hugging Face SmolLM, Stability AI, Cohere, AI21 Labs, Inflection AI, Amazon Titan/Nova). Those providers are scored out of 15 instead of 18. The five scored on Article 55 in this release are Google, Meta, Anthropic, OpenAI, and xAI.
| Score | Label | Standard |
|---|---|---|
| 0 | No evidence | Provider does not address the obligation publicly. |
| 1 | Partial | Mentioned without verifiable detail. |
| 2 | Adequate | Specific, sourced documentation exists. |
| 3 | Exceeds | Machine-readable, regularly updated, audited. |
What this is not
- Not a runtime test. We are not running model outputs through ComplyEdge's compliance engine. That is a different, complementary measurement: see our runtime benchmark.
- Not a legal verdict. GPAI documentation obligations have been enforceable with fines since 2 August 2026; these are readiness scores, not findings of non-compliance. We are scoring readiness, not adjudicating non-compliance.
- Not exhaustive. Twelve providers in scope: every vendor with material EU GPAI exposure we could verify from public sources.
All twelve scores carry verification_status: needs_review in the working dataset. We publish the rankings with that flag visible. Dispute a score by opening a PR with a public URL we missed.
The leaderboard
Data as of September 24, 2026. Generated from scripts/benchmark/results/benchmark_latest.json (generated_at 2026-09-24T17:06:04Z).
Update: on September 24, 2026 we re-checked and updated the Article 50(2) evidence for seven providers; the other five evidence files were not changed. Anthropic (0 to 2) and OpenAI (1 to 2) have since shipped machine-readable marking; xAI (0 to 1) now documents a visible watermark and prohibits stripping provenance signals. Amazon, Meta and Stability AI (each 1 to 2) are corrections: their marking of hosted outputs was documented before the July release and we missed it or scored it inconsistently. The other six scores are unchanged. The leaderboard is now ranked by percentage rather than raw points, since providers below the systemic-risk threshold are scored out of 15 instead of 18. Providers with the same percentage share a rank. The full method is in scripts/benchmark/gpai_benchmark_methodology.md.
| Rank | Provider | Class | 50(2) | 51/52 | 53(a) | 53(c) | 53(b) | 55 | Aggregate | % |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Google (DeepMind) | closed_api | 2 | 2 | 3 | 2 | 3 | 3 | 15 / 18 | 83.3% |
| 1 | Meta (Llama) | open_weights | 2 | 2 | 3 | 2 | 3 | 3 | 15 / 18 | 83.3% |
| 3 | Anthropic | closed_api | 2 | 2 | 3 | 1 | 3 | 3 | 14 / 18 | 77.8% |
| 4 | Mistral AI | open_weights | 1 | 2 | 3 | 1 | 3 | N/A | 10 / 15 | 66.7% |
| 5 | Hugging Face (SmolLM) | open_weights | 1 | 0 | 3 | 3 | 2 | N/A | 9 / 15 | 60.0% |
| 6 | OpenAI | closed_api | 2 | 1 | 2 | 1 | 2 | 2 | 10 / 18 | 55.6% |
| 7 | Stability AI | open_weights | 2 | 0 | 2 | 1 | 3 | N/A | 8 / 15 | 53.3% |
| 8 | Cohere | closed_api | 1 | 0 | 2 | 0 | 3 | N/A | 6 / 15 | 40.0% |
| 9 | AI21 Labs | open_weights | 1 | 0 | 2 | 0 | 2 | N/A | 5 / 15 | 33.3% |
| 9 | Amazon (Titan/Nova) | closed_api | 2 | 0 | 1 | 0 | 2 | N/A | 5 / 15 | 33.3% |
| 9 | Inflection AI | closed_api | 1 | 0 | 1 | 0 | 3 | N/A | 5 / 15 | 33.3% |
| 12 | xAI (Grok) | closed_api | 1 | 0 | 1 | 0 | 1 | 0 | 3 / 18 | 16.7% |
What the data shows
- Article 51/52 classification and notification is the industry floor. Seven of twelve providers have zero public evidence of systemic-risk threshold assessment or Commission notification. Even top scorers stop at partial (max score 2 / 3 in this release).
- Machine-readable marking has arrived for media, not yet for text. Article 50(2) averages 1.50 / 3, up from 0.92 in July. Images, video and audio have converged on C2PA credentials and invisible watermarks. Text is where providers split: Anthropic watermarks generated text at the model layer and Google marks text in the Gemini app, while the other ten document no text marking. No provider scores 3: none publishes an independent audit of its marking.
- The distribution is bimodal. Five providers score zero on copyright transparency. xAI scores zero on three obligations, more than any other provider.
- Open-weights does not predict lower scores. Meta scores 3/3 on Article 55. Most open-weights vendors in this set are below the FLOPs threshold, so Article 55 is N/A.
- Technical documentation is the high-water mark. Average 2.17 / 3: five providers score the maximum on Art. 53(1)(a).
Why we publish this
ComplyEdge's product is a runtime check on every AI request and answer. Phrase rules always run, and a block by one carries the rule ID and the citation. On a paid plan the injection check can also block when the month's model budget covers it, and that block names the law. The wider review runs only when you turn it on and that budget covers it, and its block does not cite an article. The benchmark is not the product. It is a measurement of the documentation gap the product operates in.
- Deployers. Obligations cascade from the model provider to the deployer. The deployer cannot inherit what was never published: 47.0 percentage points below full public documentation industry-wide.
- Acquirers. Average GPAI provider scores 53.1% on public documentation; no provider exceeds 84%.
- Regulators. Public data shows where industry stands on documentation readiness. Article 51/52 classification and notification is the gap that remains after 2 August 2026 Article 101 fining powers began.
Reproduce it
git clone https://github.com/ComplyEdge/complyedge
cd complyedge
pip install pyyaml jsonschema
python scripts/benchmark/benchmark_runner.py
python scripts/benchmark/leaderboard_renderer.py
cat scripts/benchmark/results/leaderboard.mdRepository: github.com/ComplyEdge/complyedge
Provider evidence: providers/
Benchmark code: scripts/benchmark/
Leaderboard JSON: benchmark_latest.json