auto-attack

AutoAttack

"Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks"
Francesco Croce, Matthias Hein
ICML 2020
https://arxiv.org/abs/2003.01690

We propose to use an ensemble of four diverse attacks to reliably evaluate robustness:

• APGD-CE, our new step size-free version of PGD on the cross-entropy,
• APGD-DLR, our new step size-free version of PGD on the new DLR loss,
• FAB, which minimizes the norm of the adversarial perturbations (Croce & Hein, 2019),
• Square Attack, a query-efficient black-box attack (Andriushchenko et al, 2019).

Note: we fix all the hyperparameters of the attacks, so no tuning is required to test every new classifier.

News

• [Sep 2021]
  • We add automatic checks for potential cases where the standard version of AA might be non suitable or sufficient for robustness evaluation.
  • The evaluations of models on CIFAR-10 and CIFAR-100 are no longer maintained. Up-to-date leaderboards are available in RobustBench.
• [Mar 2021] A version of AutoAttack wrt L1, which includes the extensions of APGD and Square Attack (Croce & Hein, 2021), is available!
• [Oct 2020] AutoAttack is used as standard evaluation in the new benchmark RobustBench, which includes a Model Zoo of the most robust classifiers! Note that this page and RobustBench's leaderboards are maintained simultaneously.
• [Aug 2020]
  • Updated version: in order to i) scale AutoAttack (AA) to datasets with many classes and ii) have a faster and more accurate evaluation, we use APGD-DLR and FAB with their targeted versions.
  • We add the evaluation of models on CIFAR-100 wrt Linf and CIFAR-10 wrt L2.
• [Jul 2020] A short version of the paper is accepted at ICML'20 UDL workshop for a spotlight presentation!
• [Jun 2020] The paper is accepted at ICML 2020!

Adversarial Defenses Evaluation

We here list adversarial defenses, for many threat models, recently proposed and evaluated with the standard version of AutoAttack (AA), including

• untargeted APGD-CE (no restarts),
• targeted APGD-DLR (9 target classes),
• targeted FAB (9 target classes),
• Square Attack (5000 queries).

See below for the more expensive AutoAttack+ (AA+) and more options.

We report the source of the model, i.e. if it is publicly available, if we received it from the authors or if we retrained it, the architecture, the clean accuracy and the reported robust accuracy (note that might be calculated on a subset of the test set or on different models trained with the same defense). The robust accuracy for AA is on the full test set.

We plan to add new models as they appear and are made available. Feel free to suggest new defenses to test!

To have a model added: please check here.

Checkpoints: many of the evaluated models are available and easily accessible at this Model Zoo.

CIFAR-10 - Linf

The robust accuracy is evaluated at eps = 8/255, except for those marked with * for which eps = 0.031, where eps is the maximal Linf-norm allowed for the adversarial perturbations. The eps used is the same set in the original papers.
Note: ‡ indicates models which exploit additional data for training (e.g. unlabeled data, pre-training).

Update: this is no longer maintained, but an up-to-date leaderboard is available in RobustBench.

CIFAR-100 - Linf

The robust accuracy is computed at eps = 8/255 in the Linf-norm, except for the models marked with * for which eps = 0.031 is used.
Note: ‡ indicates models which exploit additional data for training (e.g. unlabeled data, pre-training).

Update: this is no longer maintained, but an up-to-date leaderboard is available in RobustBench.

MNIST - Linf

The robust accuracy is computed at eps = 0.3 in the Linf-norm.