UnlearnCanvas: A Stylized Image Dataset to Benchmark
Machine Unlearning for Diffusion Models
paper: https://arxiv.org/abs/2402.11846
Code: https://github.com/OPTML-Group/UnlearnCanvas
The rapid advancement of diffusion models (DMs) has not only transformed various real- world industries but has also introduced negative societal concerns, including the generation of harmful content, copyright disputes, and the rise of stereotypes and biases. To mitigate these issues, machine unlearning (MU) has emerged as a potential solution, demonstrating its ability to remove undesired generative capabilities of DMs in various applications. However, by examining existing MU evaluation methods, we uncover several key challenges that can result in incomplete, inaccurate, or biased evaluations for MU in DMs.
To address them, we enhance the evaluation metrics for MU, including the introduction of an often-overlooked retainability measurement for DMs post-unlearning. Additionally, we introduce UnlearnCanvas, a comprehensive high-resolution stylized image dataset that facilitates us to evaluate the unlearning of artistic painting styles in conjunction with associated image objects.
We show that this dataset plays a pivotal role in establishing a standardized and automated evaluation framework for MU techniques on DMs, featuring 7 quantitative metrics to address various aspects of unlearning effectiveness. Through extensive experiments, we benchmark 5 state-of- the-art MU methods, revealing novel insights into their pros and cons, and the underlying unlearning mechanisms. Furthermore, we demonstrate the potential of UnlearnCanvas to benchmark other generative modeling tasks, such as style transfer.
[Other Related Benchmarks]
- UnlearnDiff Benchmark: an evaluation benchmark built upon adversarial attacks (also referred to as adversarial prompts), in order to discern the trustworthiness of these safety-driven unlearned DMs.
Method | Style-UA | Style-IRA | Style-CRA | Object-UA | Object-IRA | Object-CRA | FID | Time (s) | Memory (GB) | Storage (GB) |
|---|---|---|---|---|---|---|---|---|---|---|
SalUn | 98.58% | 80.97% | 93.96% | 92.15% | 55.78% | 44.23% | 131.37 | 6163 | 17.8 | 4.3 |
Method | Style-UA | Style-IRA | Style-CRA | Object-UA | Object-IRA | Object-CRA | FID | Time (s) | Memory (GB) | Storage (GB) |
|---|---|---|---|---|---|---|---|---|---|---|
SalUn | 98.58% | 80.97% | 93.96% | 92.15% | 55.78% | 44.23% | 131.37 | 6163 | 17.8 | 4.3 |
Evaluation Queue for the UnlearnCanvas Benchmark.
Currently, you have the option to share your checkpoint with us for evaluation, and we will subsequently update the benchmark results.
In the future, models submitted here will undergo automatic evaluation on our cluster.
Context
Evaluated MU Methods
We evaluate 5 most recently proposed MU methods for DMs:
Metrics
- Unlearning accuracy (UA): It represents the proportion of images generated by the unlearned DM using the unlearning target-related prompt, which are not correctly classified into the corresponding class for the case.
- In-domain retain accuracy (IRA): It is given by the classification accuracy of images generated by the unlearned DM using innocent prompts (not relevant to the unlearning target) in the same domain.
- Cross-domain retain accuracy (CRA): Similar to IRA, it is given by the classification accuracy of images generated by the unlearned DM using innocent prompts but in different domains.
- FID score: evaluate the distribution-wise generation quality of the unlearned DM.
- run-time
- storage space requirements
- memory costs
Impact Statement
This work helps improve the assessment and further promotes the advancement of MU (machine unlearning) methods for DMs (diffusion models), which are known to be effective in relieving or mitigating the various negative societal influences brought by the prevalent usage of DMs, which include but are not limited to the following aspects.
- Avoiding Copyright Issues: There is an urgent need for the generative model providers to scrub the influence of certain data on an already-trained model.
- Mitigating biases and stereotypes: Generative AI systems are known to have tendencies towards bias, stereotypes, and reductionism, when it comes to gender, race and national identities
Other Related Benchmarks
- UnlearnDiff Benchmark: an evaluation benchmark built upon adversarial attacks (also referred to as adversarial prompts), in order to discern the trustworthiness of these safety-driven unlearned DMs.
Contact
Please feel free to contact Yihua zhan1908@msu.edu and Yimeng zhan1853@msu.edu if you have any questions.