Artificial Intelligence in Enhancing Fraud Detection in Banking Transactions

Abstract

Fraudulent activities in banking transactions pose significant threats to financial institutions and customers, with traditional rule-based detection systems proving insufficient against increasingly sophisticated fraud schemes. This study investigates the role of artificial intelligence (AI) in enhancing fraud detection through a mixed-methods experimental design combining quantitative evaluation of supervised, unsupervised, and hybrid models with qualitative assessments of explainability and regulatory compliance. Using a dataset of over five million transactions, models including Random Forest, Gradient Boosting, Deep Neural Networks, Autoencoders, and Graph Neural Networks were applied. Results reveal that AI-based models achieved superior accuracy, recall, and AUC values compared to conventional systems, with ensemble and hybrid approaches effectively reducing false positives. Federated learning demonstrated promise in enabling secure cross-bank fraud detection, while explainable AI frameworks such as SHAP ensured transparency and regulatory alignment. The findings highlight not only the technological advantages of AI in fraud detection but also its strategic implications in reducing losses, improving customer experience, and enhancing institutional trust. This research provides both academic and practical contributions, underscoring AI as a pivotal tool in safeguarding the integrity of modern banking transactions.