Designing a scalable and reproducible machine learning workflow

Abstract

The advancement of machine learning (ML) applications necessitates the construction of end-to-end experimentation pipelines characterized by scalability, robustness, and reproducibility. This thesis aims to empower Data Scientists and ML Εngineers, enabling seamless pipeline execution, free from unexpected obstacles such as downtime, hardware unavailability, OS conflicts, or dependency issues. To realize these objectives, an ideal ML workflow should exhibit robust execution in a highly available environment, revisitable reproducibility, minimal manual intervention through automation, easy extendability, and scalable capabilities to handle larger tasks concurrently. Moreover, facilitating these attributes, a comprehensive toolkit is essential, encompassing Containerization/Virtualization for consistent environment management, Monitoring experiments for provisioning necessary training information, Data/Model Tracking to trace model and data versions throughout the workflow, Scalable Object Storage for secure data and model storage, and a Workflow Engine for workflow automation, scheduling, and monitoring. This thesis explores the challenges inherent in designing scalable and reproducible ML workflows, detailing the utilization of the aforementioned toolkit to overcome these challenges. Furthermore, industrial case studies are presented to underscore the practical benefits of a scalable and reproducible ML workflow.