🏆 Eval-Unlearn Leaderboard

About eval-learn

eval-learn is an open-source Python library providing a unified, reproducible benchmarking framework for concept unlearning in Stable Diffusion models.

The proliferation of concept unlearning techniques has produced a fragmented evaluation landscape: methods are assessed under heterogeneous experimental conditions with different datasets, metrics, and hyperparameters, making principled cross-method comparison difficult. eval-learn is designed to fill this gap.

📦 Install: pip install eval-learn  |  💻 GitHub  |  📖 Docs

Key Features

• Unified pipeline supporting fine-tuning, closed-form model editing, and inference-time intervention techniques on a common Stable Diffusion base
• 9 standardised evaluation metrics covering erasure efficacy, adversarial robustness, generative quality, and concept retention
• Plugin architecture via Python entry points — third-party techniques and metrics self-register upon installation without modifying the core framework
• GPU-efficient execution with FP16 inference, batch streaming, and proactive VRAM management
• CLI + YAML/JSON config for streamlined experiment management
• HuggingFace Hub integration via eval-learn push

BenchScore

Results are aggregated using a composite BenchScore that balances safety and quality:

BenchScore(α) = α · S + (1 − α) · Q

• BenchScore-S (α = 0.6): safety-prioritised
• BenchScore-Q (α = 0.4): quality-prioritised

Quick Start

pip install eval-learn
eval-learn run --config config.yaml

Example config.yaml:

output_dir: ./results/nudity/esd
seed: 42
technique:
  name: esd
  config: {erase_concept: nudity, train_method: noxattn}
metrics:
  - name: asr_i2p
    config: {concept: nudity, detector: nudenet}
  - name: clip_score
    config: {device: cuda}

Results

Comparison of unlearning techniques across all benchmarks. ↓ = lower is better, ↑ = higher is better. Sorted by average BenchScore.

🚀 Add Your Technique to the Leaderboard

Want to see your unlearning method on this leaderboard? Here's how:

1. Follow the contribution guidelines at eval-unlearn.readthedocs.io/en/latest/contributing to integrate your technique into the eval-learn framework
2. Submit a Pull Request to the Eval-Unlearn GitHub repository
3. Once your PR is reviewed and merged, we will evaluate your method using eval-learn under the same standardised conditions as all other techniques
4. Your results will be published on this leaderboard

We welcome contributions from the community — the more techniques we evaluate under unified conditions, the more useful this benchmark becomes for the field.