...
Q: Why is the spam filter not catching all my fraud? Why is the spam filter tagging some applications that are not fraud incorrectly?
A. The initial model was built on historical data in which fraud was tagged by a small group of colleges (~8 colleges). Based on 3 weeks of production data, we have 162,000 total applications and confirmed false positives by the colleges were 453 (0.2%). We will also have a small percentage of false negatives that are yet to be determined. This current model will be in use for a short duration of time until we enable continuous re-training and we expect the performance of model to continue improving as it now uses tagged data from all the colleges.
...
Q: Why does my model catch some applications as fraud, while others as not fraud when I see some similar signatures in both groups of applications?
A. The model is built to use over 200+ features. These features are a combination of application fields and engineered features. The model uses all of these features in determining an answer. As a result, it is not straightforward to explain why the model made a given decision. Where reasonable, we can dig into specific application or groups of applications to study decisions made by models so we can explain its decision better.
...
Q: What about new fraud signatures?
A. If the signature is entirely new, the model will rely on initial feedback from the colleges to start learning the signatures. As such applications accumulate the model will comprehend these signatures and predict them accurately.
...
Q: How long will the model take to learn a new signature?
A. This depends on how much tagged data is available for the model to capture the underlying patterns. With accurate tagged data, the model will pick-up on the underlying patterns and signatures very quickly.
...
Q: How can the colleges help?
A. The model is designed to be a human-in-the-loop system. Feedback from the colleges is critical for the model to continue to preform well. The colleges can aid the system in the following two ways:
...