As our antifraud ML models perform better and better in production, we understand that we want much more than the lonesome prediction score. We want to be able to explain WHY the model has made its decision. In our case at PayPal, this explanation must be actionable: it should serve us, the data scientists, in debugging and improving the model and should also help us to investigate model misses in collaboration with risk experts.

I am going to share our journey to develop a method that extracts actionable explanations to single predictions. I will describe our definition of a useful explanation, the drawbacks we found in using SHAP values output as-is and the enlightening approach of counterfactual explanations. I will present how combining all of the above enabled us to push our ML solutions a few steps further and share tips from our explainability experience that should be relevant to every data scientist.