A Complete Guide to Building a Fully Automated MLOps Pipeline with Terraform and Kubeflow

A Complete Guide to Building a Fully Automated MLOps Pipeline with Terraform and Kubeflow

The value of a machine learning model is not limited to development. Real business value depends on how quickly and reliably the model is deployed to production and continuously operated. This process—MLOps (Machine Learning Operations)—is far more complex than traditional software delivery because it involves many more variables (data versions, model artifacts, environment dependencies, and more).

Managing that complexity by hand is equivalent to accumulating technical debt. This guide presents a practical architecture blueprint for managing the entire AI workflow under Infrastructure as Code (IaC) principles, maximizing model reproducibility and scalability.

Why Automate the AI Workflow: Why IaC Is Essential

Traditional MLOps pipelines often hit the following bottlenecks:

1. Environment drift: Cloud resource settings differ in subtle ways across development, staging, and production.
2. Manual intervention: Steps such as running data-preprocessing scripts, triggering training, and building container images still leave room for human involvement, which makes errors more likely.
3. Versioning challenges: Infrastructure configs, dataset versions, and model code are managed separately, making it hard to track the overall system state.

IaC addresses all of these. By defining cloud infrastructure as code with tools such as Terraform, and defining workflows as code with orchestration tools such as Kubeflow, you can treat the entire system as a version-controlled artifact.

Step 1: Building the Foundation — Terraform

You first need to define as code the cloud environment where the MLOps pipeline will run (EKS clusters, storage buckets, and so on). Terraform is one of the most powerful tools for this job.

Assuming an AWS environment, we define an S3 bucket for model artifacts and datasets, and a Kubernetes cluster (EKS) to run the pipeline.

# AWS S3 버킷 정의 (데이터 및 모델 아티팩트 저장소)
resource "aws_s3_bucket" "ml_artifacts" {
  bucket = "my-mlops-artifact-bucket-unique"
  acl    = "private"
}

# EKS 클러스터 정의 (파이프라인 실행 환경)
resource "aws_eks_cluster" "mlops_cluster" {
  name     = "mlops-pipeline-cluster"
  role_arn = aws_iam_role.eks_master.arn
  version  = "1.28"
  # ... 기타 설정 생략 ...
}

# EKS 노드 그룹 정의 (실제 워크로드가 구동될 컴퓨팅 자원)
resource "aws_eks_node_group" "workers" {
  cluster_name    = aws_eks_cluster.mlops_cluster.name
  node_group_name = "ml-workers"
  subnet_ids      = var.private_subnet_ids
  instance_type   = "t3.medium"
  scaling_config {
    desired_size = 3
    max_size     = 10
    min_size     = 2
  }
}

The moment you run terraform apply on this code, the cloud infrastructure is provisioned in the desired state. That process itself is version control for infrastructure.

Step 2: Defining the Core Workflow — Kubeflow

Once the infrastructure is in place, you need to define the core model-development logic. Kubeflow Pipelines lets you define the entire process—data collection $\rightarrow$ preprocessing $\rightarrow$ model training $\rightarrow$ model validation $\rightarrow$ deployment—as a DAG (Directed Acyclic Graph) in code.

The following is a conceptual Kubeflow Pipeline YAML example that defines hypothetical data-preprocessing and model-training steps.

# Kubeflow Pipeline YAML (개념적 예시)
apiVersion: kubeflow.org/v1
kind: PipelineRun
metadata:
  name: model-training-pipeline
spec:
  template:
    # 1. 데이터 전처리 단계 (Data Preprocessing)
    components:
    - name: data_prep
      containerImage: myregistry/data-processor:v1.2
      arguments:
      - --input-data-path: s3://my-mlops-artifact-bucket/raw/data.csv
      - --output-path: s3://my-mlops-artifact-bucket/processed/data.parquet
    
    # 2. 모델 학습 단계 (Model Training)
    - name: model_trainer
      dependencies: [data_prep] # data_prep이 완료된 후에 실행
      containerImage: myregistry/trainer:latest
      arguments:
      - --train-data-path: s3://my-mlops-artifact-bucket/processed/data.parquet
      - --hyperparameters: '{"lr": 0.001}'
      - --output-model-path: s3://my-mlops-artifact-bucket/models/run_$(date).pkl
      
    # 3. 모델 검증 및 배포 준비 단계 (Validation & Deployment Artifact)
    - name: model_validator
      dependencies: [model_trainer]
      script: |
        # 모델 로드 및 성능 지표 계산 로직 실행
        if [ $accuracy > 0.9 ]; then
          echo "Validation Success. Triggering deployment."
          # 이 단계에서 ArgoCD 또는 Flux를 통해 GitOps 배포를 트리거할 수 있음
        else
          echo "Validation Failed. Stopping pipeline."
          exit 1
        fi

This YAML is not merely a list of scripts. It is a contract that specifies execution order and dependencies.

Step 3: Integration and Validation — GitOps-Based CI/CD

True automation is complete when these two elements (infrastructure code + workflow code) are connected through Git as a single source of truth. That is the core principle of GitOps.

[System flow: Terraform $\rightarrow$ EKS $\rightarrow$ Kubeflow]

1. Git Commit: An engineer commits model-code or infrastructure changes to Git.
2. CI (Continuous Integration): A CI tool (such as GitHub Actions) is triggered to test the code and to build and push the required container images to a registry.
3. CD (Continuous Deployment):
   • Infrastructure deployment: If the Terraform code has changed, the CD tool runs terraform plan and apply to update the EKS cluster state.
   • Workflow deployment: If the Kubeflow pipeline YAML has changed, the CD tool applies it to the cluster to deploy the new pipeline definition.
4. Execution: When a user merges to a deploy branch in Git or applies a specific tag, Kubeflow autonomously allocates resources according to the defined DAG and starts training.

Manual Deployment vs. IaC-Based Deployment

Combining IaC (Infrastructure as Code) with Workflow as Code is the core of modern MLOps.

With this systematic approach, development teams can focus on model development itself, while operations teams can run predictable, reliable pipelines.

References: Official Documentation

The primary source for the behavior, configuration, and errors discussed in this article is the following official documentation. Check it for version-specific options and exact behavior.

• Terraform Official Documentation