RobustART


RobustART is the first comprehensive Robustness investigation benchmark on large-scale dataset ImageNet regarding ARchitectural design (49 human-designed off-the-shelf architectures and 1200+ neural architecture searched networks) and Training techniques (10+ general ones e.g., extra training data, etc) towards diverse noises (adversarial, natural, and system noises). Our benchmark (including open-source toolkit, pre-trained model zoo, datasets, and analyses): (1) presents an open-source platform for conducting comprehensive evaluation on diverse robustness types; (2) provides a variety of pre-trained models with different training techniques to facilitate robustness evaluation; (3) proposes a new view to better understand the mechanism towards designing robust DNN architectures, backed up by the analysis. We will continuously contribute to building this ecosystem for the community.

Comprehensive ArT

Thousands of architectures (49 human-designed off-the-shelf architectures and 1200 neural architecture searched networks) and training techniques (10+ general ones)

Large-scale Dataset

Extensive experiments are conducted on the large-scale dataset ImageNet.




Diverse Noise

Evaluate robustness on multiple noises including adversarial, natural, and system noises.

Framework

Highlighted
Features

  • Open-source toolkit.
  • Pre-trained model zoo.
  • Multiple datasets.
  • Comprehensive analyses.
Learn More

Contribution

Welcome to make contribution to RobustART!
Contributions including but not limited to:

  • Upload any model’s evaluation results of any robust dataset which are not included in current benchmark;
  • Add model(s) for our RobustART.models API;
  • Add metrics of other robust dataset for RobustART.metrics API;
  • Add new noise type for RobustART.noise.AddNoise API


To upload your contributions, click here to contact us !

Citation

To reference this work (APIs, leaderboards, model zoo, conclusions, and paper), just copy the following citation:

Bibtex

@article{tang2021robustart,
title={RobustART: Benchmarking Robustness on Architecture Design and Training Techniques},
author={Shiyu Tang and Ruihao Gong and Yan Wang and Aishan Liu and Jiakai Wang and Xinyun Chen and Fengwei Yu and Xianglong Liu and Dawn Song and Alan Yuille and Philip H.S. Torr and Dacheng Tao},
journal={https://arxiv.org/pdf/2109.05211.pdf},
year={2021}}

Authors

Shiyu Tang

Beihang University

Ruihao Gong

Beihang University, SenseTime

Yan Wang

SenseTime

Aishan Liu

Beihang University

Jiakai Wang

Beihang University

Xinyun Chen

UC Berkeley

Fengwei Yu

SenseTime

Xianglong Liu

Beihang University

Dawn Song

UC Berkeley

Alan Yuille

Johns Hopkins University

Philip H.S Torr

Oxford University

Dacheng Tao

JD Explore Academy

Institutions