Overview

Model monitoring is the close tracking of the performance of ML models in production so that production and AI teams can identify potential issues before they affect the business.

If production traffic differs from training data or varies substantially over time, the quality of the answers your model produces is probably affected. When that happens, you will want to be alerted automatically and responsively so that you can anticipate problems before they affect your customer experiences or your revenue streams.

Objectives

• Create a Workbench Instance Notebook.
• Deploy a pre-trained model.
• Configure model monitoring.
• Generate some artificial traffic.
• Interpret the data reported by the model monitoring feature.

Vertex AI offers two Notebook Solutions, Workbench and Colab Enterprise.

Workbench

Vertex AI Workbench is a good option for projects that prioritize control and customizability. It’s great for complex projects spanning multiple files, with complex dependencies. It’s also a good choice for a data scientist who is transitioning to the cloud from a workstation or laptop.

Vertex AI Workbench Instances comes with a preinstalled suite of deep learning packages, including support for the TensorFlow and PyTorch frameworks.

Setup and requirements

For each lab, you get a new Google Cloud project and set of resources for a fixed time at no cost.

1. Sign in to Qwiklabs using an incognito window.

2. Note the lab's access time (for example, 1:15:00), and make sure you can finish within that time.
   There is no pause feature. You can restart if needed, but you have to start at the beginning.

3. When ready, click Start lab.

4. Note your lab credentials (Username and Password). You will use them to sign in to the Google Cloud Console.

5. Click Open Google Console.

6. Click Use another account and copy/paste credentials for this lab into the prompts.
   If you use other credentials, you'll receive errors or incur charges.

7. Accept the terms and skip the recovery resource page.

Task 1. Launch Vertex AI Workbench instance

1. In the Google Cloud console, from the Navigation menu (), select Vertex AI.

2. Click Enable All Recommended APIs.

3. In the Navigation menu, click Workbench.

   At the top of the Workbench page, ensure you are in the Instances view.

4. Click Create New.

5. Configure the Instance:

   • Name: lab-workbench
   • Region: Set the region to
   • Zone: Set the zone to
   • Advanced Options (Optional): If needed, click "Advanced Options" for further customization (e.g., machine type, disk size).

6. Click Create.

This will take a few minutes to create the instance. A green checkmark will appear next to its name when it's ready.

7. Click OPEN JUPYTERLAB next to the instance name to launch the JupyterLab interface. This will open a new tab in your browser.

Click Check my progress to verify the objective. Launch Vertex AI Workbench instance

Task 2. Clone a course repo within your JupyterLab interface

To clone the training-data-analyst notebook in your JupyterLab instance:

Step 1

In JupyterLab, click the Terminal icon to open a new terminal.

Step 2

At the command-line prompt, type in the following command and press Enter.

git clone https://github.com/GoogleCloudPlatform/training-data-analyst

Step 3

Confirm that you have cloned the repository by double clicking on the training-data-analyst directory and ensuring that you can see its contents. The files for all the Jupyter notebook-based labs throughout this course are available in this directory.

Click Check my progress to verify the objective. Clone a course repo within your JupyterLab interface

Task 3. Monitor your Vertex AI model

1. In the notebook interface, navigate to training-data-analyst > courses > machine_learning > deepdive2 > machine_learning_in_the_enterprise > labs, and open model_monitoring_for_custom_model_batch_prediction_job.ipynb.

2. A pop-up will appear for you to select a kernel. Choose the TensorFlow 2-11 (Local) kernel from the options.

3. Click Edit > Clear All Outputs.

Click Check my progress to verify the objective. Create and run Model Monitoring Jobs

Task 4. Clean up

1. In the Google Cloud console, on the Navigation menu, click Vertex AI > Batch prediction.
2. Click on your endpoint (for example, bp_example), click the overflow menu () and then click Delete batch prediction job.
3. Click DELETE to confirm the undeployment.

End your lab

When you have completed your lab, click End Lab. Qwiklabs removes the resources you’ve used and cleans the account for you.

You will be given an opportunity to rate the lab experience. Select the applicable number of stars, type a comment, and then click Submit.

The number of stars indicates the following:

• 1 star = Very dissatisfied
• 2 stars = Dissatisfied
• 3 stars = Neutral
• 4 stars = Satisfied
• 5 stars = Very satisfied

You can close the dialog box if you don't want to provide feedback.

For feedback, suggestions, or corrections, please use the Support tab.

Copyright 2024 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.