Build a Data Mesh with Dataplex: Challenge Lab Reviews

Somu B. · Reviewed 8 months ago

Had trouble troubleshooting my last step. The failure was due to not "seeing" my qwiklabs project/dataset.

Chrissy F. · Reviewed 8 months ago

Partho M. · Reviewed 8 months ago

Stephane V. · Reviewed 8 months ago

Stephane V. · Reviewed 8 months ago

Maja S. · Reviewed 8 months ago

Saurabh S. · Reviewed 8 months ago

Sahu S. · Reviewed 8 months ago

I have a rule named 'NOT NULL' , but task 5 does not recognize.

Allan S. · Reviewed 8 months ago

H C. · Reviewed 8 months ago

I have a rule named 'NOT NULL' , but task 5 does not recognize.

Allan S. · Reviewed 9 months ago

Mirosław G. · Reviewed 9 months ago

Mirosław G. · Reviewed 9 months ago

Vaishnavi V. · Reviewed 9 months ago

The last task is impossible to finish. metadata_registry_defaults: dataplex: projects: qwiklabs-gcp-01-0110787d189f locations: us-west1 lakes: sales-lake zones: curated-customer-zone row_filters: NONE: filter_sql_expr: |- True rule_dimensions: - consistency - correctness - duplication - completeness - conformance - integrity rules: NOT NULL: rule_type: NOT_NULL dimension: completeness rule_bindings: VALID_USER: entity_uri: bigquery://projects/qwiklabs-gcp-01-0110787d189f/datasets/customer_orders/tables/ordered_items column_id: user_id row_filter_id: NONE rule_ids: - NOT NULL VALID_ORDER: entity_uri: bigquery://projects/qwiklabs-gcp-01-0110787d189f/datasets/customer_orders/tables/ordered_items column_id: order_id row_filter_id: NONE rule_ids: - NOT NULL # Rule named 'NOT NULL' not found for the user_id column in the data quality task. # Please re-create the task including this rule as mentioned in the instructions. # Please create a data quality job named 'customer-orders-data-quality-job' as mentioned in the task. # If already done, wait until the 'results' table is created in 'orders_dq_dataset' dataset along with # data quality results. # Log Using the default container image Waiting for container log creation PYSPARK_PYTHON=/opt/dataproc/conda/bin/python JAVA_HOME=/usr/lib/jvm/temurin-11-jdk-amd64 SPARK_EXTRA_CLASSPATH= :: loading settings :: file = /etc/spark/conf/ivysettings.xml OS runtime details: PRETTY_NAME="Debian GNU/Linux 11 (bullseye)" NAME="Debian GNU/Linux" VERSION_ID="11" VERSION="11 (bullseye)" VERSION_CODENAME=bullseye ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" Python executable path: /opt/dataproc/conda/bin/python Python Version: sys.version_info(major=3, minor=9, micro=15, releaselevel='final', serial=0) PySpark working directory: PosixPath('/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491') PySpark directory content: ['.clouddq-executable.zip.hashsum.crc', 'clouddq-executable.zip', 'clouddq_pyspark_driver.py', '.dq-customer-orders.yaml.crc', '.clouddq_pyspark_driver.py.crc', 'clouddq-executable.zip.hashsum', 'dq-customer-orders.yaml', '.clouddq-executable.zip.crc'] Input PySpark arguments: ['/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/clouddq_pyspark_driver.py', 'clouddq-executable.zip', 'ALL', 'gs://qwiklabs-gcp-01-0110787d189f-dq-config/dq-customer-orders.yaml', '--gcp_project_id="qwiklabs-gcp-01-0110787d189f"', '--gcp_region_id="us-west1"', '--gcp_bq_dataset_id="orders_dq_dataset"', '--target_bigquery_summary_table="qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results"'] User-specified CloudDQ YAML configs: gs://qwiklabs-gcp-01-0110787d189f-dq-config/dq-customer-orders.yaml Creating configs directory at: `/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/configs` Copying YAML file dq-customer-orders.yaml to configs directory `configs`... Configs directory contents is: [PosixPath('configs/dq-customer-orders.yaml')] CloudDQ executable expected hexdigest: a14971737fa18e443390047a3d8b04a5378768bea0fed8a5bc34adabdba131c1 Executing commands: /opt/dataproc/conda/bin/python clouddq-executable.zip ALL /tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/configs --gcp_project_id="qwiklabs-gcp-01-0110787d189f" --gcp_region_id="us-west1" --gcp_bq_dataset_id="orders_dq_dataset" --target_bigquery_summary_table="qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results" 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.integration.gcp_credentials[77] INFO Successfully created GCP Client. 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Starting CloudDQ run with configs: {"clouddq_run_configs": {"rule_binding_ids": "ALL", "rule_binding_config_path": "/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/configs", "environment_target": "dev", "gcp_project_id": "qwiklabs-gcp-01-0110787d189f", "gcp_region_id": "us-west1", "gcp_bq_dataset_id": "orders_dq_dataset", "gcp_service_account_key_path": null, "gcp_impersonation_credentials": null, "metadata": "{}", "dry_run": false, "progress_watermark": true, "target_bigquery_summary_table": "qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results", "intermediate_table_expiration_hours": 24, "num_threads": 8, "debug": false, "print_sql_queries": false, "skip_sql_validation": false, "summary_to_stdout": false, "enable_experimental_bigquery_entity_uris": true, "enable_experimental_dataplex_gcs_validation": true, "bigquery_client": null, "gcp_credentials": {"credentials": "<google.auth.compute_engine.credentials.Credentials object at 0x7f370cf1e280>", "project_id": "qwiklabs-gcp-01-0110787d189f", "user_id": "984150749013-compute@developer.gserviceaccount.com"}}} 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.runners.dbt.dbt_connection_configs[77] INFO Using Application-Default Credentials (ADC) to authenticate to GCP... 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.runners.dbt.dbt_connection_configs[77] INFO Using Application-Default Credentials (ADC) to authenticate to GCP... 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Writing rule_binding views and intermediate summary results to BigQuery dq_summary_table_name: `qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.dq_summary`. 2024-01-17 19:56:22 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Using dq_summary_dataset: qwiklabs-gcp-01-0110787d189f.orders_dq_dataset 2024-01-17 19:56:23 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Using target_bigquery_summary_table: `qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results`. 2024-01-17 19:56:23 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Preparing SQL for rule bindings: ['VALID_USER', 'VALID_ORDER'] 2024-01-17 19:56:23 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.classes.dq_configs_cache[77] INFO Calling Dataplex Metadata API to retrieve schema for entity_uri: {'entity_uri': 'bigquery://projects/qwiklabs-gcp-01-0110787d189f/datasets/customer_orders/tables/ordered_items', 'rule_binding_ids_list': 'VALID_ORDER,VALID_USER'} 2024-01-17 19:56:24 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.classes.dq_configs_cache[77] INFO Retrieved Dataplex Entities: [DataplexEntity(name='projects/qwiklabs-gcp-01-0110787d189f/locations/us-west1/lakes/sales-lake/zones/curated-customer-zone/entities/ordered_items', createTime='2024-01-17T18:51:46.713702Z', updateTime='2024-01-17T18:51:46.713702Z', id='ordered_items', type='TABLE', asset='customer-orders', dataPath='projects/qwiklabs-gcp-01-0110787d189f/datasets/customer_orders/tables/ordered_items', system='BIGQUERY', format={'format': 'OTHER'}, schema=DataplexEntitySchema(fields=[{'name': 'delivered_at', 'type': 'STRING', 'mode': 'NULLABLE'}, {'name': 'shipped_at', 'type': 'STRING', 'mode': 'NULLABLE'}, {'name': 'created_at', 'type': 'TIMESTAMP', 'mode': 'NULLABLE'}, {'name': 'status', 'type': 'STRING', 'mode': 'NULLABLE'}, {'name': 'inventory_item_id', 'type': 'INT64', 'mode': 'NULLABLE'}, {'name': 'sale_price', 'type': 'DOUBLE', 'mode': 'NULLABLE'}, {'name': 'user_id', 'type': 'INT64', 'mode': 'NULLABLE'}, {'name': 'order_id', 'type': 'INT64', 'mode': 'NULLABLE'}, {'name': 'returned_at', 'type': 'TIMESTAMP', 'mode': 'NULLABLE'}, {'name': 'id', 'type': 'INT64', 'mode': 'NULLABLE'}], partitionFields=None, partitionStyle=None))] 2024-01-17 19:56:24 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO target_entity_summary_configs: {'qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__order_id_1': {'rule_binding_ids_list': ['VALID_ORDER']}, 'qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__user_id_1': {'rule_binding_ids_list': ['VALID_USER']}} 2024-01-17 19:56:24 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.runners.dbt.dbt_utils[77] INFO Executing dbt command: ['run', '--profiles-dir', '/tmp/srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491/dbt', '--vars', '{"entity_dq_statistics_models": ["qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__order_id_1", "qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__user_id_1"]}', '--target', 'dev'] 19:56:24 Running with dbt=1.1.5 19:56:24 Partial parse save file not found. Starting full parse. 19:56:25 Found 6 models, 0 tests, 0 snapshots, 0 analyses, 191 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics 19:56:25 19:56:25 Concurrency: 8 threads (target='dev') 19:56:25 19:56:25 1 of 5 START view model orders_dq_dataset.VALID_ORDER .......................... [RUN] 19:56:25 2 of 5 START view model orders_dq_dataset.VALID_USER ........................... [RUN] 19:56:26 2 of 5 OK created view model orders_dq_dataset.VALID_USER ...................... [OK in 0.75s] 19:56:26 1 of 5 OK created view model orders_dq_dataset.VALID_ORDER ..................... [OK in 0.76s] 19:56:26 3 of 5 START table model orders_dq_dataset.qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__user_id_1 [RUN] 19:56:26 4 of 5 START table model orders_dq_dataset.qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__order_id_1 [RUN] 19:56:29 3 of 5 OK created table model orders_dq_dataset.qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__user_id_1 [CREATE TABLE (1.0 rows, 7.8 KB processed) in 2.57s] 19:56:29 4 of 5 OK created table model orders_dq_dataset.qwiklabs_gcp_01_0110787d189f__customer_orders__ordered_items__order_id_1 [CREATE TABLE (1.0 rows, 7.8 KB processed) in 2.80s] 19:56:29 5 of 5 START incremental model orders_dq_dataset.dq_summary .................... [RUN] 19:56:32 5 of 5 OK created incremental model orders_dq_dataset.dq_summary ............... [MERGE (2.0 rows, 5.0 KB processed) in 2.60s] 19:56:32 19:56:32 Finished running 2 view models, 2 table models, 1 incremental model in 6.39s. 19:56:32 19:56:32 Completed successfully 19:56:32 19:56:32 Done. PASS=5 WARN=0 ERROR=0 SKIP=0 TOTAL=5 19:56:42 Error sending message, disabling tracking 2024-01-17 19:56:42 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO dbt invocation id for current execution is 2550959a-af66-4b73-974d-2018cd72494c 2024-01-17 19:56:44 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.integration.bigquery.dq_target_table_utils[77] INFO Table qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results already exists and query results are appended to the table. 2024-01-17 19:56:44 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq.integration.bigquery.dq_target_table_utils[77] INFO Loaded 2 rows to qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results. {"clouddq_job_completion_config": {"invocation_id": "2550959a-af66-4b73-974d-2018cd72494c", "target_bigquery_summary_table": "qwiklabs-gcp-01-0110787d189f.orders_dq_dataset.results", "summary_to_stdout": false, "target_rule_binding_ids": ["VALID_USER", "VALID_ORDER"], "partition_date": "2024-01-17", "num_rows_loaded_to_target_table": 2}} 2024-01-17 19:56:44 gdpic-srvls-batch-b4ef72c7-2a73-4119-b3eb-0f121450f491-m clouddq[77] INFO Job completed successfully. Waiting up to 5 seconds. Sent all pending logs.

Damir H. · Reviewed 9 months ago

Pawel J. · Reviewed 9 months ago

Mufti A. · Reviewed 9 months ago

Fernando F. · Reviewed 9 months ago

George C. · Reviewed 9 months ago

Jitendra S. · Reviewed 9 months ago

My job succeeded but still i am getting below error message as "Rule named 'NOT NULL' not found for the user_id column in the data quality task. Please re-create the task including this rule as mentioned in the instructions."

Sivakumar P. · Reviewed 9 months ago

shawn r. · Reviewed 9 months ago

good

Sivakumar P. · Reviewed 9 months ago

shawn r. · Reviewed 9 months ago

good

Sivakumar P. · Reviewed 9 months ago