arrow_back

Discover and Catalog Vertex AI Assets with Dataplex

Sign in Join
Get access to 700+ labs and courses

Discover and Catalog Vertex AI Assets with Dataplex

Lab 1 hour 30 minutes universal_currency_alt 5 Credits show_chart Intermediate
info This lab may incorporate AI tools to support your learning.
Get access to 700+ labs and courses

GSP1260

Overview

For many organizations, finding the right datasets and other assets for machine learning workflows with Vertex AI can be tricky. There may already be datasets ready to use for model training, but they may not be easy to find or it is not clear how the data can or should be used.

This is where Dataplex Universal Catalog can help! You can think of Dataplex as a centralized platform that unifies data assets distributed across an organization, and supports data governance without the need to move or transfer data from its original location. Dataplex provides easy-to-use tools for discovering, cataloging, and managing data to power your analytics and machine learning workflows, while also leveraging the security of IAM in Google Cloud.

In this lab, you walk through a typical use case for using Dataplex to catalog and discover data assets that can be used to train Vertex AI models. Full details below.

Note: For additional support throughout the lab, you can get help by expanding the hint boxes. Challenge yourself by expanding hints progressively (or not!), as you need more support.

Pre-created data assets in each project

To get you started, some data assets have been pre-created for you across two Google Cloud projects and include data assets in BigQuery and Cloud Storage, which you use to create a new Vertex AI dataset in the first task.

Project Project ID Assigned region Available Data
1 Cloud Storage bucket named -bucket containing the images of damaged car parts
2 Three BigQuery datasets: damaged_car_ownership (containing a table for owners of the damaged cars); damaged_car_image_info (containing a table with additional image information, such as owner ID and location/timestamp of the image); and damaged_car_image_metadata (an object dataset with metadata derived from the image files in Cloud Storage such as storage path and updated date)

Multiple projects and users

First, you take on the role of the Data Engineer who is helping their organization get data into usable and accessible formats for machine learning workflows. In Project 1, you create a Vertex AI dataset from images in Cloud Storage, and then you create a custom tag template to tag that Vertex AI dataset (and other potentially useful data in BigQuery), so that it is easy for others in your organization to find using Dataplex.

Then, you transition to the role of the Data Scientist or Machine Learning Engineer who is looking for existing data assets that they can use to train new models. In Project 2, you use the tags applied by the Data Engineer to search for relevant data assets in Dataplex. Then, you create a new custom tag template for personally identifiable information (PII) and tag additional assets that can be useful for future modeling efforts.

User Goal Username Primary project
1 - Data Engineer Create and apply tags to help others find available data for a Vertex AI model to be named damaged_car_parts across multiple projects.
2 - Data Scientist or ML Engineer Use tags to search for Vertex AI assets to train the model to be named damaged_car_parts, and create and apply a new tag for PII to flag protected data.

What you'll do

In this lab, you learn how to:

  • Create a Vertex AI managed dataset.
  • Tag Vertex AI assets using custom tag templates.
  • Search for Vertex AI assets using filters and tags.
  • View lineage of Vertex AI assets.

Prerequisites

While not required, it is helpful to have some previous knowledge about how Dataplex and Vertex AI are commonly used within Google Cloud workflows. For an introduction to these tools before you begin this lab, you can complete the following labs:

Setup and requirements

Note: To begin this lab, follow the instructions below to log into Project 1:

as Username 1: .

Before you click the Start Lab button

Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources are made available to you.

This hands-on lab lets you do the lab activities in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials you use to sign in and access Google Cloud for the duration of the lab.

To complete this lab, you need:

  • Access to a standard internet browser (Chrome browser recommended).
Note: Use an Incognito (recommended) or private browser window to run this lab. This prevents conflicts between your personal account and the student account, which may cause extra charges incurred to your personal account.
  • Time to complete the lab—remember, once you start, you cannot pause a lab.
Note: Use only the student account for this lab. If you use a different Google Cloud account, you may incur charges to that account.

How to start your lab and sign in to the Google Cloud console

  1. Click the Start Lab button. If you need to pay for the lab, a dialog opens for you to select your payment method. On the left is the Lab Details pane with the following:

    • The Open Google Cloud console button
    • Time remaining
    • The temporary credentials that you must use for this lab
    • Other information, if needed, to step through this lab
  2. Click Open Google Cloud console (or right-click and select Open Link in Incognito Window if you are running the Chrome browser).

    The lab spins up resources, and then opens another tab that shows the Sign in page.

    Tip: Arrange the tabs in separate windows, side-by-side.

    Note: If you see the Choose an account dialog, click Use Another Account.
  3. If necessary, copy the Username below and paste it into the Sign in dialog.

    {{{user_0.username | "Username"}}}

    You can also find the Username in the Lab Details pane.

  4. Click Next.

  5. Copy the Password below and paste it into the Welcome dialog.

    {{{user_0.password | "Password"}}}

    You can also find the Password in the Lab Details pane.

  6. Click Next.

    Important: You must use the credentials the lab provides you. Do not use your Google Cloud account credentials. Note: Using your own Google Cloud account for this lab may incur extra charges.
  7. Click through the subsequent pages:

    • Accept the terms and conditions.
    • Do not add recovery options or two-factor authentication (because this is a temporary account).
    • Do not sign up for free trials.

After a few moments, the Google Cloud console opens in this tab.

Note: To access Google Cloud products and services, click the Navigation menu or type the service or product name in the Search field.

Task 1. Create a Vertex AI managed dataset

For this task, be sure that you have logged into Project 1 () as Username 1 ().

In a typical workflow to create a new Vertex AI dataset for image classification, you create a CSV input file, in which each row contains a URL image path to a training image and the associated label for that image.

In this lab, this CSV file has been pre-created for you in Cloud Storage as:

gs://-bucket/damaged-car-images/data.csv

In this task, begin by updating the image paths in data.csv with your bucket name, and then uploading the revised CSV file back to your Cloud Storage bucket. Then, use this file when creating a new Vertex AI dataset to connect the dataset to the training images in that Cloud Storage bucket.

Remember that you get help by expanding the hint boxes as you need them!

Update data.csv with the path to your bucket

Start by downloading the data.csv file locally to Cloud Shell, so you can update the image paths and upload the revised file back to Cloud Storage.

  1. Click Activate Cloud Shell at the top of the Google Cloud console.

  2. Create a local copy of data.csv from your pre-created bucket, so that you can easily update the image paths with your bucket name:

gsutil cp gs://{{{project_0.project_id}}}-bucket/damaged-car-images/data.csv .

If prompted, click Authorize.

  1. Next, create a variable for the path to your bucket and subdirectory:
export DIRECTORY={{{project_0.project_id}}}-bucket/damaged-car-images
  1. Review data.csv to see that the original paths:
cat ./data.csv

The results display the current path (gs://damaged-car-parts-vertex-dataset/), which you update in the next step.

  1. Now update the image paths in data.csv with your bucket name and subdirectory:
sed -i -e "s|damaged-car-parts-vertex-dataset|${DIRECTORY}|g" ./data.csv
  1. Review the updated data.csv to see that the paths have been updated to reference your desired bucket and subdirectory:
cat ./data.csv

The results display the updated paths that begin with gs://{{{project_0.project_id}}}-bucket/damaged-car-images/.

  1. Last, upload the revised data.csv back to your Cloud Storage bucket:
gsutil cp ./data.csv gs://${DIRECTORY}

Check that data.csv has been updated in Cloud Storage

Though it is not a required step of this workflow, it is a good idea to check that the data.csv file has been uploaded to the desired location and successfully updated.

Check that data.csv was updated in your bucket using any method you already know, or expand the hint below for some helpful steps.

Full solution (Expand to see all of the steps!)

Create a managed dataset in Vertex AI

Next, create a new managed dataset in Vertex AI for single label image classification.

  1. Navigate to Vertex AI in the Google Cloud console by clicking on the Navigation menu () > Vertex AI > Dashboard.

  2. Click Enable All Recommended APIs.

  3. In the Vertex AI navigation menu, under Data, click Datasets.

These first three steps got you to the appropriate location in your Google Cloud project.

  1. Now, on your own, create a Vertex AI dataset named damaged_car_parts for single-label image classfication in the assigned region for Project 1 ().

If the option to select a region is null, you can assume that it will default to the assigned for this lab.

Expand the hints below for some helpful steps!

Option 1: Review the docs

Option 2: Adapt steps from a related lab

Full solution (Expand to see all of the steps!)

When you have successfully created the dataset, the page refreshes and provides new options for selecting an import method, which you use in the next section.

Connect your Vertex AI dataset to your training images using data.csv

Now, use your updated data.csv to import the image files of damaged car parts to the Vertex AI dataset.

  1. For Select an import method, click Select import files from Cloud Storage.

  2. In the Select import files from Cloud Storage section, click Browse, and navigate to data.csv in your Cloud Storage bucket (gs://-bucket).

  3. Click Select.

    Once you've selected an appropriate file, a green checkbox appears to the left of the file path.

  4. Click Continue.

Note: It can take around 10 to 15 minutes for your images to import and be aligned with their categories. You need to wait for this step to complete before checking your progress for this task.

During this time, you can leave this browser tab open, proceed to Task 2 in a new browser tab, and complete the first section titled Create a tag template in Dataplex.

Click Check my progress to verify the objective.

Create a Vertex AI dataset

Task 2. Tag a Vertex AI dataset using a custom tag template

For this task, stay logged into Project 1 () as Username 1 ().

Now that you have created the Vertex AI dataset, it is time to tag it, so that others can easily find it in Dataplex.

In this task, begin by creating a new tag template for Vertex AI model names. Then, you use that tag template to tag the Vertex AI dataset and associate it with the model name for damaged_car_parts.

Create a tag template in Dataplex

Start by creating a custom tag template in Dataplex to tag assets with associated model names.

  1. Navigate to Dataplex Universal Catalog in the Google Cloud console by clicking on the Navigation menu () > View all products > Analytics > Dataplex Universal Catalog.

  2. In the Dataplex menu, under Manage metadata, click Catalog.

  3. Click Create tag template (Deprecated), and then click Proceed.

Note: Tag templates are being replaced by aspect types but are still functional at this time. You can proceed with the next steps to create a tag template.
  1. Provide the required information to define the tag template:
Property Value
Template Display Name VertexAI_Data
Template ID Leave the default value.
Location
Visibility Public
  1. Click Add a field, and enter the required information to add a new field to the template:
Property Value
Field Display Name Related Models
Field ID Leave the default value.
Field description Add a text description of your choice for this field.
Type String
  1. Click Done.

  2. Click Create.

Click Check my progress to verify the objective.

Create a tag template in Dataplex

Search for Vertex AI assets using filters

Note: Before you continue, check that your Vertex AI dataset named damaged_car_parts has successfully been created.

Next, find the Vertex AI dataset that you created by using filters for systems and data types in Dataplex.

  1. In the Dataplex menu, under Discover, click Search.
Note: For this lab, use the default Data Catalog version of the search platform (already enabled at the top of the Search page), rather than updating to Dataplex Catalog [NEW].
  1. For Filters > Systems, enable the checkbox for Vertex AI.

  2. For Filters > Data Types, enable the checkbox for Dataset.

  3. Click on the damaged_car_parts dataset.

Add a tag to the Vertex AI dataset

Now, add a tag to the Vertex AI dataset to associate it with the model name damaged_car_parts.

  1. Click Attach tags. (You may have to scroll to the far right side of the page to see this option.)

If this option is not visible, ensure that you are using the Data Catalog version of the search platform (see note in previous section).

  1. For Choose the tag templates, select the template name VertexAI_Data, and click OK.

  2. Under VertexAI_Data, for Related Models, type damaged_car_parts, and click Save.

Remain on this details page for the damaged_car_parts dataset.

Click Check my progress to verify the objective.

Add a tag to the Vertex AI dataset

Add an overview description to the Vertex AI dataset

Adding an overview for a dataset asset in Dataplex can help others understand the intended purpose of the data and how it can or should be used.

Using the available options on the current page for the damaged_car_parts dataset, add an overview for the dataset that can be useful to others when they discover this dataset.

For example, the text can be something like: This Vertex AI dataset contains images of damaged car parts including bumpers, windshields, etc. Each image has been assigned a single label to categorize the images by these car part types.

Full solution (Expand to see all of the steps!)

Task 3. Tag other data assets using the same tag template

For this task, stay logged into Project 1 () as Username 1 ().

In the previous task, you created a reusable tag template that can be used to tag multiple data assets with associated model names.

  1. Now, use that same tag template to tag two BigQuery datasets that provide more information about the damaged car part images, so that they are also associated with the model name for damaged_car_parts:
  • damaged_car_ownership containing a table for owners of the damaged cars
  • damaged_car_image_info containing a table with additional image information (such as owner ID and location/timestamp of the image)
  1. Add some descriptive text to the overview for each dataset in Dataplex (just like you did previously for the Vertex AI dataset).

You can expand the hints below for some helpful steps!

Option 1: Review Task 2

Option 2: Adapt steps from a related lab

Full solution (Expand to see all of the steps!)

Click Check my progress to verify the objective.

Tag other data assets using tag templates

Task 4. Search for assets using tags and view lineage

For this task, begin by logging into Project 2 () as Username 2 ().

Expand the hint below for help with switching to a new project and user.

Full solution (Expand to see all of the steps!)

Now that the Data Engineer (you as Username 1!) has created a custom tag template and used it to tag data assets with associated model names, you can experience being the data scientist or machine learning engineer that benefits from these useful tags to search for relevant data assets and view their lineage.

Two options to search for assets using tags

In Dataplex, you can use tags to search for assets in multiple ways. Here are two useful ways. Both search options result in a list of data assets that have been tagged with VertexAI_Data.

Option 1: Search using text input

  1. In the Dataplex menu, under Discover, click Search.

  2. In Search input box (middle of page), type: tag:vertexai_data

  3. Click Search.

  4. Click on the asset name to see more details (such as the damaged_car_parts dataset).

or

Option 2: Search using checkboxes for tags

  1. In the Dataplex menu, under Discover, click Search.

  2. For Filters > Tags, enable the checkbox for VertexAI_Data.

  3. Click on the asset name to see more details (such as the damaged_car_parts dataset).

Search for Vertex AI Assets using tags

Now that you have the steps to search for assets using tags, search for the three data assets that you previously tagged.

  1. Use the tag to find the following three assets and review the stored information for Overview and Related Model:
  • damaged_car_parts (Vertex AI dataset for the images of damaged car parts)
  • damaged_car_ownership (BigQuery dataset containing a table for owners of the damaged cars)
  • damaged_car_image_info (BigQuery dataset containing a table with additional image information)
  1. Notice that the BigQuery object dataset named damaged_car_image_metadata is not listed in your search results. Why not?

Expand to see the answer

View lineage for assets

Last, review the lineage of some of these assets to learn more about when and how they were created.

  1. In the Dataplex search options, click on damaged_car_parts > Lineage.

  2. Then, click on the asset name for damaged_car_parts on the Graph tab.

In addition to the date and time that the dataset was created, you can also see that it was created in a different project () than the current project (), and the region in which it was created ().

  1. Now, return to the search page, and search for damaged_car_image_metadata using its full name.

  2. Click on damaged_car_image_metadata > Entry list.

  3. Then, click on the table name for bumper_images, and then click Lineage.

  4. Last, click on the asset name for bumper_images on the Graph tab.

Similar to damaged_car_parts, you can see the created date, time, project ID, and region in which the asset was created.

If provided more time than this lab duration, Dataplex lineage can also populate more information, in which case you would see the full lineage of the bumper_images table in the object dataset: .jpg files from a Cloud Storage directory named bumper are used in simple data pipeline to create the resulting bumper_images metadata table.

Challenge Task. Tag protected data with a custom PII template

For this task, stay logged into Project 2 () as Username 2 ().

While working with the data assets tagged by the Data Engineer, you notice that there are additional tags that should be applied. Specifically, the BigQuery dataset with ownership information for the damaged cars contains personally identifiable information (PII) that should be handled carefully.

Challenge yourself to create a simple custom tag template with a PII flag that you can apply to specific columns in the BigQuery table named owner_info in the damaged_car_ownership dataset.

You already created a tag template in Task 2. You can use those steps as a starting point and modify them to the specifications provided below.

Remember that you get help by expanding the hint boxes as you need them!

Create tag template for protected data

Begin by creating a new public tag template called Protected Data with one enumerated field called Protected Data Flag with YES and NO values. Be sure to create the template in the assigned region for Project 2 ().

Option 1: Review the docs

Option 2: Adapt steps from a related lab

Full solution (Expand to see all of the steps!)

Note: If the progress check is not successful after implementing the full solution, check that you are logged into Project 2

as Username 2: .

Click Check my progress to verify the objective.

Create tag template for Protected Data

Apply tag template for protected data

Next, apply the tag template for Protected Data to the BigQuery table named owner_info in the damaged_car_ownership dataset.

You decide not to tag the columns for owner_id, state, and age as PII (as they cannot be used to identify specific individuals), and instead focus on tagging the following columns:

  1. first_name
  2. last_name
  3. email
  4. city
  5. zip

Option 1: Review the docs

Option 2: Adapt steps from a related lab

Full solution (Expand to see all of the steps!)

Click Check my progress to verify the objective.

Apply tag template for Protected Data

Search for protected data using tags

Last, use the PII tag to search for data flagged as containing PII.

Remember that you searched for assets using tags in Task 4. Repeat those steps to find data assets flagged as containing PII, or expand a hint below for some helpful steps.

Option 1: Ask Gemini

Option 2: Adapt steps from a related lab

Full solution (Expand to see all of the steps!)

Congratulations!

In this lab, you learned how to tag Vertex AI assets using custom tag templates, search for Vertex AI assets using filters and tags, and view lineage of Vertex AI assets. Now you have what it takes to start cataloging and discovering models and datasets in your organization's projects using Dataplex!

Next steps / Learn more

Google Cloud training and certification

...helps you make the most of Google Cloud technologies. Our classes include technical skills and best practices to help you get up to speed quickly and continue your learning journey. We offer fundamental to advanced level training, with on-demand, live, and virtual options to suit your busy schedule. Certifications help you validate and prove your skill and expertise in Google Cloud technologies.

Manual Last Updated June 11, 2025

Lab Last Tested June 11, 2025

Copyright 2025 Google LLC. All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.

Before you begin

  1. Labs create a Google Cloud project and resources for a fixed time
  2. Labs have a time limit and no pause feature. If you end the lab, you'll have to restart from the beginning.
  3. On the top left of your screen, click Start lab to begin

This content is not currently available

We will notify you via email when it becomes available

Great!

We will contact you via email if it becomes available

One lab at a time

Confirm to end all existing labs and start this one

Use private browsing to run the lab

Use an Incognito or private browser window to run this lab. This prevents any conflicts between your personal account and the Student account, which may cause extra charges incurred to your personal account.