In the image domain, adversarial perturbations can be crafted to be virtually indistinguishable to human perception, causing humans and state-of-the-art models to disagree. However, in the natural language domain, small perturbations are clearly perceptible, and the replacement of a single word can drastically alter the semantics of the document. Given these challenges, we use a black-box population-based optimization algorithm to generate semantically and syntactically similar adversarial examples that fool well-trained sentiment analysis and textual entailment models. We additionally show that the successful adversarial examples are classified to the true label by 20 human annotators, and are perceptibly quite similar to the original. Finally, we attempt to use adversarial training as a defense, but fail to yield improvement, demonstrating the strength and diversity of the generated examples. We hope our findings encourage researchers to pursue improving robustness in the natural language domain.

Existing black-box approaches to generating adversarial examples typically require a significant number of queries, either for training a substitute network or estimating gradients from output scores. We introduce GenAttack, a gradient-free optimization technique which uses genetic algorithms for synthesizing adversarial examples in the black-box setting. Our experiments show that GenAttack can successfully generate visually imperceptible adversarial examples against image recognition models with orders of magnitude fewer queries than existing approaches. Furthermore, we show that GenAttack's query efficiency and gradient-free nature enable it to successfully attack both the state-of-the-art ImageNet defense, ensemble adversarial training, and non-differentiable, randomized input transformation defenses. Our results suggest that evolutionary algorithms open up a promising area of research into effective gradient-free black-box attacks.

Most recent work in adversarial attacks focuses on discriminative classifiers, which only model the conditional distribution of the labels given the inputs. We propose the deep Bayes classifier, which improves classical naive Bayes with conditional deep generative models. We further develop detection methods for adversarial examples, which reject inputs that have negative log-likelihood under the generative model exceeding a threshold pre-specified using training data. Our experimental results suggest that deep Bayes classifiers are more robust than deep discriminative classifiers, and the proposed detection methods achieve high detection rates against many recently proposed attacks.

Master's thesis comprising of my previous work in this area. I present ZOO and EAD, and validate their effectiveness through attacking state-of-the-art models trained on the MNIST, CIFAR-10, and ImageNet datasets. In addition, I demonstrate that the proposed attacks can successfully attack recently proposed defenses in their corresponding limited access settings. I show that ZOO in the black-box case can succeed against the state-of-the-art ImageNet defense, Ensemble Adversarial Training, while EAD relying on transferability can succeed against the state-of-the-art MNIST defense, the Madry Defense Model, and input transformation defenses, such as Feature Squeezing.

Feature Squeezing is a recently proposed defense method which reduces the search space available to an adversary by coalescing samples that correspond to many different feature vectors in the original space into a single sample. It has been shown that feature squeezing defenses can be combined in a joint detection framework to achieve high detection rates against state-of-the-art attacks. However, we demonstrate that by increasing the adversary strength of said state-of-the-art attacks, one can bypass the detection framework with adversarial examples of minimal visual distortion. These results suggest for proposed defenses to validate against stronger attack configurations.

The Madry Lab recently hosted a competition designed to test the robustness of their adversarially trained MNIST model. Attacks were constrained to perturb each pixel of the input image by a scaled maximal L_∞ distortion ϵ = 0.3. This discourages the use of attacks which are not optimized on the L_∞ distortion metric. We demonstrate that by relaxing the competition's L_∞ constraint, EAD can generate transferable adversarial examples which, despite their high average L_∞ distortion, have minimal visual distortion. These results call into question the use of L_∞ as a sole measure for visual distortion, and further demonstrate the power of EAD at generating robust adversarial examples.

Despite the fact that L₁ distortion accounts for the total variation and in optimization encourages sparsity in the perturbation, little has been developed for crafting L₁-based adversarial examples. In this paper, we formulate the process of attacking DNNs via adversarial examples as an elastic-net regularized optimization problem. Our elastic-net attacks to DNNs (EAD) feature L₁-oriented adversarial examples and include the state-of-the-art L₂ attack as a special case. Experimental results on MNIST, CIFAR-10 and ImageNet show that EAD can yield a distinct set of adversarial examples with small L₁ distortion which match state-of-the-art performance in different attack scenarios. More importantly, EAD leads to improved attack transferability and complements adversarial training for DNNs, suggesting novel insights on leveraging L₁ distortion in adversarial machine learning.

We propose an effective black-box attack that only has access to the input (images) and the output (confidence scores) of a targeted DNN. However, different from leveraging attack transferability from substitute models, we propose using zeroth order optimization (ZOO) to generate adversarial examples by directly estimating the gradients of the targeted DNN. We use zeroth order stochastic coordinate descent along with dimension reduction, hierarchical attack and importance sampling techniques to efficiently attack black-box models, sparing the need for training substitute models and avoiding the loss in attack transferability. Experimental results on MNIST, CIFAR-10 and ImageNet show that the proposed ZOO attack is as effective as the state-of-the-art white-box attack and significantly outperforms existing black-box attacks via substitute models.