Publishing MLFlow in GCP using Terraform: Step Step Directory

Setting * Docker * Auth once:

gcloud auth configure-docker europe-west2-docker.pkg.dev

Imagine construction:

docker build -t mlflow-gcp .

The mark:

docker tag mlflow-gcp europe-west2-docker.pkg.dev/test-ml-flow-terraform/test-ml-flow-terraform-repo/mlflow-imagine:latest

Pay:

docker push europe-west2-docker.pkg.dev/test-ml-flow-terraform/test-ml-flow-terraform-repo/mlflow-imagine:latest

Step 8: Create an example of a database

1. First, we need to create an assembly that will hold our database (you must create a new .TF file for the database).
   
   1. • The version used is PostGRESQL 14 (The latest currently available), the minimum configuration of this version is 1 CPU and 3.75 GB RAM, so a customized layer must be created (Read more about creating custom levels here).
      • The counterpart needs to get some IP, so either use IPv4_enabled or create a private network if this is a better option for you.

        data "google_project" "project" {
          project_id = var.project_name
        }
        
        resource "google_sql_database_instance" "main" {
          name             = "${var.project_name}-${var.env}-db"
          database_version = "POSTGRES_14"
          project          = var.project_name
        
          settings {
            tier            = "db-custom-1-3840" # vCPUs: 1 Memory: 3.75 GB
            disk_size       = "10"               # HDD storage: 10 GB
            disk_type       = "PD_HDD"
            disk_autoresize = false
            ip_configuration {
              ipv4_enabled  = true
              authorized_networks {
                name  = "Your VPN"
                value = var.vpn_to_access_db
              }
            }
          }
        }

        Use Terraform Plan. If there are no problems, then implement Terraform app. Be aware that creating database counterparts may take anywhere from 5 to 20 minutes.

        It should look like this:

2. Once the counterpart is finished and ready, we can create a database. You will comment on the counterpart we have previously created.

   resource "google_sql_database" "ml_flow_db" {
     name     = "mlflow-${var.project_name}-${var.env}"
     instance = google_sql_database_instance.main.name
     project  = var.project_name
   }

3. The next step is to create a user. The password is created using Random_Password Resource to create random symbols for you. The login address, password, and URL for the database are added to the secrets, where it will be needed later.

   resource "google_sql_user" "ml_flow_user" {
     name     = "mlflow-${var.project_name}-${var.env}"
     instance = google_sql_database_instance.main.name
     password = random_password.db_password_ml_flow.result
     project  = var.project_name
   }

   Create a secret log in:

   resource "google_secret_manager_secret" "db_login_ml_flow" {
     secret_id = "mlflow_tracking_username"
     project   = var.project_name
     replication {
       automatic = true
     }
   }
   
   resource "google_secret_manager_secret_version" "ml-flow-username-version-basic" {
     secret      = google_secret_manager_secret.db_login_ml_flow.id
     secret_data = google_sql_user.ml_flow_user.name
   }
   

   Create a password with password:

   resource "random_password" "db_password_ml_flow" {
     length           = 16
     special          = true
     override_special = "!#$%&*()-_=+[]{}<>:?"
   }
   
   resource "google_secret_manager_secret" "db_password_ml_flow" {
     secret_id = "mlflow_tracking_password"
     project   = var.project_name
     replication {
       automatic = true
     }
   }
   
   resource "google_secret_manager_secret_version" "ml-flow-pass-version-basic" {
     secret      = google_secret_manager_secret.db_password_ml_flow.id
     secret_data = random_password.db_password_ml_flow.result
   }

   Create a secret with the database connection link that we will use later:

   resource "google_secret_manager_secret" "mlflow_database_url" {
     secret_id = "mlflow_database_url"
     project   = var.project_name
     replication {
       automatic = true
     }
   }
   
   resource "google_secret_manager_secret_version" "mlflow_database_url-version-basic" {
     secret      = google_secret_manager_secret.mlflow_database_url.id
     secret_data = "postgresql+psycopg2://${google_sql_user.ml_flow_user.name}:${random_password.db_password_ml_flow.result}@/${google_sql_database.ml_flow_db.name}?host=/cloudsql/${google_sql_database_instance.main.connection_name}"
   }

   use Terraform Plan And then Terraform app.

Step 9: Create service accounts for Cloudrun and MLFlow

Next, we have two service accounts to create them, one with Cloudrun and the second to call MLFlow. We will set their rights in a separate file (a new .TF file for service accounts) must be created.

1. For Cloudrun – it will be used only by Cloudrun:

   resource "google_service_account" "cloudrun-mlflow" {
     account_id   = "cloudrun-mlflow-${var.env}"
     display_name = "Service Account for Cloud Run running ML Flow"
   }

2. To contact MLFlow and create objects in a bucket, you will need the key to connect to MNFlow using a text program.

   resource "google_service_account" "bucket_object_creator" {
     account_id   = "mlflow-connect-${var.env}"
     display_name = "Service Account for create objects in ML Flow bucket."
   }
   
   resource "google_service_account_key" "key_to_bucket_object_creator" {
     service_account_id = google_service_account.bucket_object_creator.name
   }
   
   resource "google_secret_manager_secret" "bucket_object_creator_account_key" {
     secret_id = "mlflow_service_account_key"
     project   = var.project_name
     replication {
       automatic = true
     }
   }
   
   resource "google_secret_manager_secret_version" "mlflow_key-version-basic" {
     secret      = google_secret_manager_secret.bucket_object_creator_account_key.id
     secret_data = base64decode(google_service_account_key.key_to_bucket_object_creator.private_key)
   }

after Terraform Plan and Terraform appYou should be able to see two accounts of the new service; One must have a key.

Step 10: Create a storage bucket for MNFlow artifacts

At this stage, we will create a storage bucket for our MNFlow artifacts, the bucket is created as regional, but if you need higher availability, you can use other types (you must create a new .f file for the bucket).

resource "google_storage_bucket" "mlflow_artifacts_bucket" {
  name                        = "${var.project_name}-mlflow-${var.env}-${var.region}"
  project                     = var.project_name
  location                    = "EUROPE-WEST2"
  storage_class               = "REGIONAL"
  uniform_bucket_level_access = true
  public_access_prevention    = "enforced"
  force_destroy               = false
}

URL must be inserted in secret as we will need it later.

resource "google_secret_manager_secret" "mlflow_artifact_url" {
  secret_id = "mlflow_artifact_url"
  project   = var.project_name
  replication {
    automatic = true
  }
}

resource "google_secret_manager_secret_version" "ml-flow-artifact-url-version-basic" {
  secret      = google_secret_manager_secret.mlflow_artifact_url.id
  secret_data = google_storage_bucket.mlflow_artifacts_bucket.url
}

Step 11: IAM formation

1. Add rights to the recognition (pre -created) account to be able to contact the database (the new IAM file must be created.

   resource "google_project_iam_member" "cloudsql_connect" {
     project = var.project_name
     role    = "roles/cloudsql.client"
     member  = "serviceAccount:${google_service_account.cloudrun-mlflow.email}"
   }

2. Add rights to the Cloudrun service account, so that he can read the secrets.

   resource "google_secret_manager_secret_iam_member" "db_login_ml_flow_accessor" {
     secret_id = google_secret_manager_secret.db_login_ml_flow.id
     role      = "roles/secretmanager.secretAccessor"
     member    = "serviceAccount:${google_service_account.cloudrun-mlflow.email}"
   }
   
   resource "google_secret_manager_secret_iam_member" "db_password_ml_flow_accessor" {
     secret_id = google_secret_manager_secret.db_password_ml_flow.id
     role      = "roles/secretmanager.secretAccessor"
     member    = "serviceAccount:${google_service_account.cloudrun-mlflow.email}"
   }
   
   resource "google_secret_manager_secret_iam_member" "mlflow_artifact_url_accessor" {
     secret_id = google_secret_manager_secret.mlflow_artifact_url.id
     role      = "roles/secretmanager.secretAccessor"
     member    = "serviceAccount:${google_service_account.cloudrun-mlflow.email}"
   }
   
   resource "google_secret_manager_secret_iam_member" "mlflow_database_url_accessor" {
     secret_id = google_secret_manager_secret.mlflow_database_url.id
     role      = "roles/secretmanager.secretAccessor"
     member    = "serviceAccount:${google_service_account.cloudrun-mlflow.email}"
   }

   All secrets should look like this:
   

3. Create a bucket for a bucket so that your team can take a look at the objects, and to make a service account to create objects.

data "google_iam_policy" "mlflow-bucket-policy" {
  binding {
    role    = "roles/storage.objectViewer"
    members = concat(var.bucket_users_list, ["serviceAccount:${google_service_account.cloudrun-mlflow.email}"])
  }
  binding {
    role = "roles/storage.objectCreator"
    members = [
      "serviceAccount:${google_service_account.bucket_object_creator.email}",
    ]
  }
}

resource "google_storage_bucket_iam_policy" "policy_bucket_object_create" {
  bucket      = google_storage_bucket.mlflow_artifacts_bucket.name
  policy_data = data.google_iam_policy.mlflow-bucket-policy.policy_data

  depends_on = [google_service_account.cloudrun-mlflow, google_storage_bucket.mlflow_artifacts_bucket]
}

You should be able to see these permissions in the bucket:

Step 12: Access to the Secret Director

If everything is done correctly, you should see this:

Step 13: Forming the operation of the cloud

This resource will be the most complex ever. Below is the full resource (you must create a new .TF file for Cloud Run):

• Service_account – The service account that Terraform will use
• Template/sizes – Cloudrun tells that he needs to contact the database
• Template/containers – carries information about what the container should be used; You can also advertise Env values ​​for the container here:

  data "google_project" "project" {
    project_id = var.project_name
  }
  
  resource "google_cloud_run_v2_service" "mlflow_on_cloudrun" {
    name     = "mlflow-${var.project_name}-${var.env}"
    location = var.region
    ingress  = "INGRESS_TRAFFIC_ALL"
    client   = "cloud-console"
  
    depends_on = [google_service_account.cloudrun-mlflow, google_artifact_registry_repository.ar_mlflow]
  
    template {
      service_account = google_service_account.cloudrun-mlflow.email
      volumes {
        name = "cloudsql"
  
        cloud_sql_instance {
          instances = [
            google_sql_database_instance.main.connection_name,
          ]
        }
      }
      scaling {
        max_instance_count = 10
      }
      containers {
        image = "${var.region}-docker.pkg.dev/${var.project_name}/${google_artifact_registry_repository.ar_mlflow.name}/${var.image_name}:latest"
        env {
          name  = "GCP_PROJECT"
          value = data.google_project.project.number
        }
        volume_mounts {
          mount_path = "/cloudsql"
          name       = "cloudsql"
        }
  
        resources {
          cpu_idle = true
          limits = {
            cpu    = "1000m" # 1 vCPU
            memory = "1024Mi"
          }
        }
      }
    }
  }

  This part of the symbol allows everyone access to the link to the MLFlow, which is then secured by logging in and password.

  resource "google_cloud_run_service_iam_binding" "access_to_mlflow" {
    location = google_cloud_run_v2_service.mlflow_on_cloudrun.location
    service  = google_cloud_run_v2_service.mlflow_on_cloudrun.name
    role     = "roles/run.invoker"
    members = [
      "allUsers"
    ]
  }

If you are planning to download sensitive data to MLFlow, I suggest adding another safety layer.

conclusion

However, we got to the end of this guide. I hope you have found this insightful and useful tutorial.

As you can see, the MNFlow setting on the GCP using Terraform leads to enhancing automation and simplifying ML. But Terraform provides more than just automation-as it provides major benefits such as the code reuse, simplified management, and reduce savings time, all of which are vibrant in a fast-paced ML environment.

With automatic learning progress, staying up and adapting to emerging trends and techniques is very important. If you find this guide useful, think about subscribing to our newsletter. We plan to publish more educational programs that can greatly enhance your business efficiency.

Fill in the bulldozer data (on the screen, you can see the data form). Think about choosing “Multiregion type” if your project needs higher availability. You can adjust other options as required.