Title: Regularizing Black-box Models for Improved Interpretability. Abstract: Most of the work on interpretable machine learning has focused on designingeither inherently interpretable models, which typically trade-off accuracyfor interpretability, or post-hoc explanation systems, which lack guarantees about their explanation quality.  We explore an alternative to theseapproaches by directly regularizing a black-box model for interpretabilityat training time.  Our approach explicitly connects three key aspects ofinterpretable machine learning:  (i) the model’s internal interpretability, (ii)the explanation system used at test time, and (iii) the metrics that measureexplanation quality.  Our regularization results in substantial improvementin terms of the explanation fidelity and stability metrics across a range ofdatasets and black-box explanation systems while slightly improving accuracy.  Finally, we justify theoretically that the benefits of our regularizationgeneralize to unseen points.