Traditional machine learning techniques may suffer from evasion attack in which an attacker intends to have malicious samples to be misclassified as legitimate at test time by manipulating the samples. It is crucial to evaluate the security of a classifier during the development of a robust system a