Towards a New MLOps Architecture: A Methodological Approach Driven by Business and Scientific Requirements

Abstract

This article proposes an innovative conceptual model for Machine Learning Operations (MLOps) pipelines, aiming to overcome the current challenges concerning the entire lifecycle of machine learning models and to meet the growing demands of both Academia and Industry. Based on a hybrid research approach, combining scientific works and insights from professionals in the field, this proposed MLOps pipeline model integrates advanced automation, robust governance, intelligent data and model management, and explainable monitoring. We explore the convergence between theory and practice, identifying gaps and proposing an approach that promotes the scalability, reproducibility, and reliability of ML systems in complex and dynamic production environments. A state-of-the-art conceptual model for MLOps pipelines was proposed, based on a rigorous analysis of the literature and valuable insights from professional practice. The model addresses the critical challenges of automation, data and model management, monitoring, governance, and usability, aligning research ambitions with operational needs. The results from applying the MLOps architecture demonstrated measurable efficiency with a perceived improvement in the scalability, reproducibility, and reliability of ML systems. Positive outcomes were observed in relation to the deployment time of Machine Learning models, which was reduced from approximately 6 months to a range of 3 to 5 days, depending on the team’s maturity and the application’s purpose. An increase in productivity and operational standardization was also noted, accompanied by gains in scalability and efficiency, evidenced by the elimination of the model deployment queue, the migration of over 3,200 users to the new environment, and the publication of more than 100 Data Science models in the first few months of the new environment’s operation. Additionally, the transition to a cloud infrastructure provided cost and financial resource optimization compared to the previous on-premises solution, and an enhancement of governance and security through the execution of standardized pipelines.

Ähnliche Arbeiten