ML Pipeline Workflow

Complete end-to-end MLOps pipeline orchestration from data preparation through model deployment.

Overview

This skill provides comprehensive guidance for building production ML pipelines that handle the full lifecycle: data ingestion → preparation → training → validation → deployment → monitoring.

When to Use This Skill Building new ML pipelines from scratch Designing workflow orchestration for ML systems Implementing data → model → deployment automation Setting up reproducible training workflows Creating DAG-based ML orchestration Integrating ML components into production systems What This Skill Provides Core Capabilities

Pipeline Architecture

End-to-end workflow design DAG orchestration patterns (Airflow, Dagster, Kubeflow) Component dependencies and data flow Error handling and retry strategies

Data Preparation

Data validation and quality checks Feature engineering pipelines Data versioning and lineage Train/validation/test splitting strategies

Model Training

Training job orchestration Hyperparameter management Experiment tracking integration Distributed training patterns

Model Validation

Validation frameworks and metrics A/B testing infrastructure Performance regression detection Model comparison workflows

Deployment Automation

Model serving patterns Canary deployments Blue-green deployment strategies Rollback mechanisms Reference Documentation

See the references/ directory for detailed guides:

data-preparation.md - Data cleaning, validation, and feature engineering model-training.md - Training workflows and best practices model-validation.md - Validation strategies and metrics model-deployment.md - Deployment patterns and serving architectures Assets and Templates

The assets/ directory contains:

pipeline-dag.yaml.template - DAG template for workflow orchestration training-config.yaml - Training configuration template validation-checklist.md - Pre-deployment validation checklist Usage Patterns Basic Pipeline Setup

1. Define pipeline stages

stages = [ "data_ingestion", "data_validation", "feature_engineering", "model_training", "model_validation", "model_deployment" ]

2. Configure dependencies

See assets/pipeline-dag.yaml.template for full example

Production Workflow

Data Preparation Phase

Ingest raw data from sources Run data quality checks Apply feature transformations Version processed datasets

Training Phase

Load versioned training data Execute training jobs Track experiments and metrics Save trained models

Validation Phase

Run validation test suite Compare against baseline Generate performance reports Approve for deployment

Deployment Phase

Package model artifacts Deploy to serving infrastructure Configure monitoring Validate production traffic Best Practices Pipeline Design Modularity: Each stage should be independently testable Idempotency: Re-running stages should be safe Observability: Log metrics at every stage Versioning: Track data, code, and model versions Failure Handling: Implement retry logic and alerting Data Management Use data validation libraries (Great Expectations, TFX) Version datasets with DVC or similar tools Document feature engineering transformations Maintain data lineage tracking Model Operations Separate training and serving infrastructure Use model registries (MLflow, Weights & Biases) Implement gradual rollouts for new models Monitor model performance drift Maintain rollback capabilities Deployment Strategies Start with shadow deployments Use canary releases for validation Implement A/B testing infrastructure Set up automated rollback triggers Monitor latency and throughput Integration Points Orchestration Tools Apache Airflow: DAG-based workflow orchestration Dagster: Asset-based pipeline orchestration Kubeflow Pipelines: Kubernetes-native ML workflows Prefect: Modern dataflow automation Experiment Tracking MLflow for experiment tracking and model registry Weights & Biases for visualization and collaboration TensorBoard for training metrics Deployment Platforms AWS SageMaker for managed ML infrastructure Google Vertex AI for GCP deployments Azure ML for Azure cloud Kubernetes + KServe for cloud-agnostic serving Progressive Disclosure

Start with the basics and gradually add complexity:

Level 1: Simple linear pipeline (data → train → deploy) Level 2: Add validation and monitoring stages Level 3: Implement hyperparameter tuning Level 4: Add A/B testing and gradual rollouts Level 5: Multi-model pipelines with ensemble strategies Common Patterns Batch Training Pipeline

See assets/pipeline-dag.yaml.template

stages: - name: data_preparation dependencies: [] - name: model_training dependencies: [data_preparation] - name: model_evaluation dependencies: [model_training] - name: model_deployment dependencies: [model_evaluation]

Real-time Feature Pipeline

Stream processing for real-time features

Combined with batch training

See references/data-preparation.md

Continuous Training

Automated retraining on schedule

Triggered by data drift detection

See references/model-training.md

Troubleshooting Common Issues Pipeline failures: Check dependencies and data availability Training instability: Review hyperparameters and data quality Deployment issues: Validate model artifacts and serving config Performance degradation: Monitor data drift and model metrics Debugging Steps Check pipeline logs for each stage Validate input/output data at boundaries Test components in isolation Review experiment tracking metrics Inspect model artifacts and metadata Next Steps

After setting up your pipeline:

Explore hyperparameter-tuning skill for optimization Learn experiment-tracking-setup for MLflow/W&B Review model-deployment-patterns for serving strategies Implement monitoring with observability tools Related Skills experiment-tracking-setup: MLflow and Weights & Biases integration hyperparameter-tuning: Automated hyperparameter optimization model-deployment-patterns: Advanced deployment strategies