SALMUBench

Official repository for the CVPR 2026 paper:

SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning
Cai Selvas-Sala, Lei Kang, Lluis Gomez

SALMUBench is a benchmark for evaluating association-level multimodal unlearning in CLIP-style vision-language models. It tests whether a model can remove targeted sensitive persona-attribute associations while preserving general utility and avoiding collateral forgetting.

Links

• Project page: https://cvc-mmu.github.io/salmubench/
• Paper: https://arxiv.org/abs/2603.26316
• Code: https://github.com/cvc-mmu/salmubench
• Models & Datasets: https://huggingface.co/collections/cvc-mmu/salmubench

Overview

SALMUBench provides:

• A synthetic benchmark dataset of persona-attribute associations
• A Compromised CLIP model trained with the sensitive associations
• A Clean CLIP model trained without the sensitive associations
• A structured evaluation protocol for forgetting efficacy and utility preservation
• Evaluation splits for diagnosing incomplete forgetting, collateral damage, and over-generalized forgetting

Repository contents

This repository currently contains the main evaluation code and selected baseline unlearning scripts.

evaluation/
  evaluation.py
  analysis.py
  summarize.py

methods/
  unlearn_negative_gradient.py
  unlearn_similarity_minimization.py
  unlearn_shuffled_captions.py
  unlearn_generic_captions.py

Setup

Create the environment from the provided file:

conda env create -f environment.yaml
conda activate salmubench

Evaluation

Run the main evaluation script with a local CLIP checkpoint:

python evaluation/evaluation.py \
  --models-path /path/to/model_checkpoint.pth \
  --output-dir evaluation/evaluation_results

You can also pass a directory containing multiple checkpoints.

Notes

• The released datasets are synthetic and intended for research on multimodal machine unlearning.
• The repository currently focuses on evaluation and selected baseline scripts, not a full training pipeline.
• For the complete benchmark description, evaluation protocol, and results, see the paper and project page.

Citation

If you use SALMUBench, please cite:

@InProceedings{Selvas-Sala_2026_CVPR,
    author    = {Selvas-Sala, Cai and Kang, Lei and Gomez, Lluis},
    title     = {SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {39351-39360}
}