Author(s)

Emily Yip, Thomas Lau, Patrick Ho^*

Affiliation(s)

Department of Computing, The Hong Kong Polytechnic University, Hong Kong Region, China.

Corresponding Author

Patrick Ho

ABSTRACT

Financial fraud detection systems face significant challenges in adapting to evolving fraudulent behaviors while maintaining optimal balance between detection accuracy and operational efficiency in dynamic financial environments. Traditional supervised learning approaches struggle with the sequential decision-making nature of fraud detection, limited labeled fraud data availability, and the need for real-time adaptation to emerging fraud patterns that continuously evolve to circumvent existing detection mechanisms. The challenge lies in developing intelligent systems that can learn optimal detection strategies through interaction with financial transaction environments while balancing exploration of new fraud patterns with exploitation of known fraud indicators.

This study proposes AutoDetect, a novel Actor-Critic Reinforcement Learning (RL) framework that formulates fraud detection as a sequential decision-making problem where an intelligent agent learns optimal detection policies through continuous interaction with transaction data streams. The framework employs actor-critic architecture where the actor network generates detection decisions and the critic network evaluates the quality of these decisions based on fraud detection rewards and penalty structures. The RL approach enables autonomous learning of detection strategies that maximize long-term fraud detection effectiveness while minimizing false positive rates through dynamic policy optimization based on environmental feedback.

Experimental evaluation using large-scale financial transaction datasets demonstrates that AutoDetect achieves 53% improvement in fraud detection accuracy compared to traditional supervised learning approaches. The framework results in 46% better adaptation to novel fraud patterns and 42% reduction in false positive rates while maintaining real-time processing capabilities suitable for high-throughput financial transaction environments. AutoDetect successfully combines reinforcement learning with fraud detection domain knowledge to provide 38% better interpretability of detection decisions while supporting autonomous improvement through continuous learning from transaction feedback.

KEYWORDS

Reinforcement learning; Actor-Critic; Fraud detection; Sequential decision making; Financial transactions; Autonomous learning; Policy optimization; Real-time adaptation

CITE THIS PAPER

Emily Yip, Thomas Lau, Patrick Ho. AutoDetect: an actor-critic reinforcement learning framework for financial fraud detection. Social Science and Management. 2025, 2(3): 55-63. DOI: https://doi.org/10.61784/ssm3059.

REFERENCES

[1] Alex-Omiogbemi A A, Sule A K, Omowole B M, et al. Advances in cybersecurity strategies for financial institutions: A focus on combating E-Channel fraud in the Digital era. Journal of Cybersecurity and Financial Innovation, 2024, 12(3): 35-48.

[2] Xing S, Wang Y. Proactive Data Placement in Heterogeneous Storage Systems via Predictive Multi-Objective Reinforcement Learning. IEEE Access, 2025, 13, 117986-117998. DOI: 10.1109/ACCESS.2025.3586378.

[3] Mandal A. Fathoming fraud: unveiling theories, investigating pathways and combating fraud. Journal of financial crime, 2024, 31(5): 1106-1125.

[4] Olushola A, Mart J. Fraud detection using machine learning. ScienceOpen Preprints, 2024, 12(6).

[5] Bello H O, Ige A B, Ameyaw M N. Adaptive machine learning models: Concepts for real-time financial fraud prevention in dynamic environments. World Journal of Advanced Engineering Technology and Sciences, 2024, 12(02): 021-034.

[6] Nesvijevskaia A, Ouillade S, Guilmin P, et al. The accuracy versus interpretability trade-off in fraud detection model. Data & Policy, 2021, 3, e12.

[7] Saxena C. Identifying transaction laundering red flags and strategies for risk mitigation. Journal of Money Laundering Control, 2024, 27(6): 1063-1077.

[8] Ashfaq T, Khalid R, Yahaya A S, et al. A machine learning and blockchain based efficient fraud detection mechanism. Sensors, 2022, 22(19): 7162.

[9] Ikemefuna C D, Okusi O, Iwuh A C, et al. Adaptive fraud detection systems: Using ML to identify and respond to evolving financial threats. International Research Journal of Modernization in Engineering, 2024, 6, 2077-2092.

[10] Cao J, Zheng W, Ge Y, et al. DriftShield: Autonomous Fraud Detection via Actor-Critic Reinforcement Learning with Dynamic Feature Reweighting. IEEE Open Journal of the Computer Society, 2025, 6, 1166-1177. DOI: 10.1109/OJCS.2025.3587001.

[11] Ijiga O M, Idoko I P, Ebiega G I, et al. Harnessing adversarial machine learning for advanced threat detection: AI-driven strategies in cybersecurity risk assessment and fraud prevention. J. Sci. Technol, 2024, 11, 001-024.

[12] Nopiel E, Okunola A, Phine E, et al. Reinforcement Learning for Adaptive Fraud Detection Systems in Dynamic Financial Environments. 2025. https://www.researchgate.net/publication/394397407_Reinforcement_Learning_for_Adaptive_Fraud_Detection_in_Dynamic_FinTech_Environments.

[13] Michailidis P, Michailidis I, Kosmatopoulos E. Reinforcement learning for optimizing renewable energy utilization in buildings: A review on applications and innovations. Energies, 2025, 18(7): 1724.

[14] Varga G. Data-Driven Methods for Machine Learning-Based Fraud Detection and Cyber Risk Mitigation in National Banking Infrastructure. Nuvern Machine Learning Reviews, 2024, 1(1): 33-40.

[15] Talukder M A, Khalid M, Uddin M A. An integrated multistage ensemble machine learning model for fraudulent transaction detection. Journal of Big Data, 2024, 11(1): 168.

[16] Al-dahasi E M, Alsheikh R K, Khan F A, et al. Optimizing fraud detection in financial transactions with machine learning and imbalance mitigation. Expert Systems, 2025, 42(2): e13682.

[17] Popoola N T. Big data-driven financial fraud detection and anomaly detection systems for regulatory compliance and market stability. Int. J. Comput. Appl. Technol. Res, 2023, 12(09): 32-46.

[18] Chalapathy R, Chawla S. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407. 2019. DOI: https://doi.org/10.48550/arXiv.1901.03407.

[19] Quintin-John S, Valverde R. A perceptron based neural network data analytics architecture for the detection of fraud in credit card transactions in financial legacy systems. WSEAS Transactions on Systems and Control, 2021, 16: 358-374. DOI: 10.37394/23203.2021.16.31.

[20] Wang M, Zhang X, Yang Y, et al. Explainable Machine Learning in Risk Management: Balancing Accuracy and Interpretability. Journal of Financial Risk Management, 2025, 14(3): 185-198.

[21] Kim J, Jung H, Kim W. Sequential pattern mining approach for personalized fraudulent transaction detection in online banking. Sustainability, 2022, 14(15): 9791.

[22] Rahman M S, Bhowmik P K, Hossain B, et al. Enhancing Fraud Detection Systems in the USA: A Machine Learning Approach to Identifying Anomalous Transactions. Journal of Economics, Finance and Accounting Studies, 2023, 5(5): 145-160.

[23] Xing S, Wang Y, Liu W. Self-Adapting CPU Scheduling for Mixed Database Workloads via Hierarchical Deep Reinforcement Learning. Symmetry, 2025, 17(7): 1109.

[24] Wang Zhikui. Research on the adaptability of subway ticket machines to the elderly based on FBM behavior model. Modern Engineering and Applications, 2025, 3(3): 1-13. DOI: https://doi.org/10.61784/mea2001.

[25] Wang Zhikui. Analysis of the shared energy storage business model of building clusters in commercial pedestrian streets. Economic Management and Practice, 2025, 3(3): 1-21. DOI: https://doi.org/10.61784/emp2001.

[26] Lu Yangfan, Chen Caishan, Mei Yuan. Vertical Synergy Evaluation of Science and Technology Finance Policy from the Perspective of Structure and Function: Based on the Experience of Guangdong Province and Municipality. Economic Management and Practice, 2025, 3(3): 22-34. DOI: https://doi.org/10.61784/emp2002.

[27] Zhou Ziyuan, Liu Guorun. Research on the innovative mechanism of grassroots governance from the perspective of interface governance - taking the "four platforms of grassroots governance" in L Town as an example. Economic Management and Practice, 2025, 3(3): 35-44. DOI: https://doi.org/10.61784/emp2003.

[28] Rane N, Choudhary S P, Rane J. Ensemble deep learning and machine learning: applications, opportunities, challenges, and future directions. Studies in Medical and Health Sciences, 2024, 1(2): 18-41.

[29] Sreevallabh Chivukula A, Yang X, Liu B, et al. Game theoretical adversarial deep learning. In Adversarial Machine Learning: Attack Surfaces, Defence Mechanisms, Learning Theories in Artificial Intelligence. Cham: Springer International Publishing, 2022, 73-149.

[30] Zheng W, Liu W. Symmetry-Aware Transformers for Asymmetric Causal Discovery in Financial Time Series. Symmetry, 2025, 16(5): 153.

[31] Ji E, Wang Y, Xing S, et al. Hierarchical Reinforcement Learning for Energy-Efficient API Traffic Optimization in Large-Scale Advertising Systems. IEEE Access, 2025. DOI: 10.1109/ACCESS.2025.3598712.

[32] Alonge E O, Eyo-Udo N L, Ubanadu B C, et al. Enhancing data security with machine learning: A study on fraud detection algorithms. Journal of Data Security and Fraud Prevention, 2021, 7(2): 105-118.

[33] Cao W, Mai N, Liu W. Adaptive Knowledge Assessment via Symmetric Hierarchical Bayesian Neural Networks with Graph Symmetry-Aware Concept Dependencies. Symmetry, 2025, 17(8): 1305.