Adversarial attacks are intentionally crafted input perturbations designed to mislead deep neural networks. Adversarial robustness, i.e., the ability of a trained model to maintain its predictive power under such attacks, is an important property for many safety-critical applications. The most common approach to construct adversarially robust models is via adversarial training i.e., training a model over adversarially-constructed examples. This, however, is computationally intensive, taking as much as 30x longer than standard training. We investigate the use of the Hilbert-Schmidt independence criterion (HSIC) as a regularizer for training an adversarially robust deep neural network. In addition to the usual cross-entropy loss, we add HSIC regularization terms in every intermediate layer. We formally prove that the resulting regularizer reduces the sensitivity of the classifier to adversarial perturbations. Our experiments on benchmark datasets demonstrate that adding this regularizer yields models with competitive natural accuracy and improved adversarial robustness, both with and without access to adversarial training examples. We also show that, when combined with knowledge distillation, our regularizer can be used to prune a previously trained robust neural network while maintaining adversarial robustness, again without any further generation of adversarial examples. This yields significant computational dividends, compressing large networks 4-7x times faster than state-of-the-art adversarial pruning methods.