Real-time fraud detection using big data and ml techniques

Abstract

Fraud detection in financial transactions finds great importance in the fight against financial crime. In this thesis, two models are developed: CatBoost and Light Gradient Boosting Machine for classifying transactions into "Fraud," "No Fraud," or "Suspicious." Later, the suspicious transactions will be relabeled after investigation and reintegrated into the training data, which will enhance the model performance. Apache Kafka allows doing real-time processing of data to efficiently handle live transactions. Challenges regarding dataset imbalance were addressed by employing class weights proportional to the inverse of class frequencies, further adjusted by a scaling factor to ensure optimal balance during training. In developing the adaptive accurate detection of frauds, this work designed a real-time pipeline, using a feedback loop iteratively in model refinements. Both the models are yielding good results; LGBM gave the best regarding precision and recall. The reintegration of relabeled data greatly increased accuracy, and the optimization performed with a focus on loss ensured that detection was better compared to traditional metrics. This thesis aims to contribute to the domain of fraud detection by presenting an adaptive and scalable framework in combination with real-time processing and continuous learning coupled with machine learning models. It addresses the challenges related to handling imbalanced datasets and evolving fraud patterns in real-world scenarios.