[{"id":"76565","title":"Introduction","description":"

This module covers the course outline

","steps":[{"id":"528711","prompt":null,"isOptional":true,"activities":[{"id":"521558","href":"/course_templates/264/video/521558","isLocked":false,"duration":156000,"title":"Course Introduction","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false}],"expanded":false},{"id":"76566","title":"Monitoring","description":"

In this module, we learn how to use the Jobs List page to filter for jobs that we want to monitor or investigate. We look at how the Job Graph, Job Info, and Job Metrics tabs collectively provide a comprehensive summary of your Dataflow job. Lastly, we learn how we can use Dataflow’s integration with Metrics Explorer to create alerting policies for Dataflow metrics.

","steps":[{"id":"528712","prompt":null,"isOptional":true,"activities":[{"id":"521559","href":"/course_templates/264/video/521559","isLocked":false,"duration":190000,"title":"Job List","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528713","prompt":null,"isOptional":true,"activities":[{"id":"521560","href":"/course_templates/264/video/521560","isLocked":false,"duration":34000,"title":"Job Info","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528714","prompt":null,"isOptional":true,"activities":[{"id":"521561","href":"/course_templates/264/video/521561","isLocked":false,"duration":202000,"title":"Job Graph","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528715","prompt":null,"isOptional":true,"activities":[{"id":"521562","href":"/course_templates/264/video/521562","isLocked":false,"duration":454000,"title":"Job Metrics","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528716","prompt":null,"isOptional":true,"activities":[{"id":"521563","href":"/course_templates/264/video/521563","isLocked":false,"duration":129000,"title":"Metrics Explorer","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528717","prompt":null,"isOptional":false,"activities":[{"id":"521564","href":"/course_templates/264/quizzes/521564","isLocked":false,"duration":600000,"title":"Quiz","description":null,"type":"quiz","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528718","prompt":null,"isOptional":true,"activities":[{"id":"521565","href":"/course_templates/264/documents/521565","isLocked":false,"duration":1800000,"title":"Additional Resources","description":null,"type":"document","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false}],"expanded":false},{"id":"76567","title":"Logging and Error Reporting","description":"

In this module, we learn how to use the Log panel at the bottom of both the Job Graph and Job Metrics pages, and learn about the centralized Error Reporting page.

","steps":[{"id":"528719","prompt":null,"isOptional":true,"activities":[{"id":"521566","href":"/course_templates/264/video/521566","isLocked":false,"duration":376000,"title":"Logging","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528720","prompt":null,"isOptional":true,"activities":[{"id":"521567","href":"/course_templates/264/video/521567","isLocked":false,"duration":74000,"title":"Error Reporting","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528721","prompt":null,"isOptional":false,"activities":[{"id":"521568","href":"/course_templates/264/quizzes/521568","isLocked":false,"duration":600000,"title":"Quiz","description":null,"type":"quiz","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528722","prompt":null,"isOptional":true,"activities":[{"id":"521569","href":"/course_templates/264/documents/521569","isLocked":false,"duration":1800000,"title":"Additional Resources","description":null,"type":"document","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false}],"expanded":false},{"id":"76568","title":"Troubleshooting and Debug","description":"

In this module, we learn how to troubleshoot and debug Dataflow pipelines. We will also review the four common modes of failure for Dataflow: failure to build the pipeline, failure to start the pipeline on Dataflow, failure during pipeline execution, and performance issues.

","steps":[{"id":"528723","prompt":null,"isOptional":true,"activities":[{"id":"521570","href":"/course_templates/264/video/521570","isLocked":false,"duration":135000,"title":"Troubleshooting workflow","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528724","prompt":null,"isOptional":true,"activities":[{"id":"521571","href":"/course_templates/264/video/521571","isLocked":false,"duration":338000,"title":"Types of troubles","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528725","prompt":null,"isOptional":false,"activities":[{"id":"521572","href":"/course_templates/264/quizzes/521572","isLocked":false,"duration":600000,"title":"Quiz","description":null,"type":"quiz","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528726","prompt":null,"isOptional":false,"activities":[{"id":"521573","href":"/course_templates/264/labs/521573","isLocked":false,"duration":7200000,"title":"Serverless Data Processing with Dataflow - Monitoring, Logging and Error Reporting for Dataflow Jobs","description":"In this lab, you a)configure logging for your dataflow jobs, and b)explore the Error Reporting page ","type":"lab","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528727","prompt":null,"isOptional":true,"activities":[{"id":"521574","href":"/course_templates/264/documents/521574","isLocked":false,"duration":1800000,"title":"Additional Resources","description":null,"type":"document","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false}],"expanded":false},{"id":"76569","title":"Performance","description":"

In this module, we will discuss performance considerations we should be aware of while developing batch and streaming pipelines in Dataflow.

","steps":[{"id":"528728","prompt":null,"isOptional":true,"activities":[{"id":"521575","href":"/course_templates/264/video/521575","isLocked":false,"duration":269000,"title":"Pipeline Design","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528729","prompt":null,"isOptional":true,"activities":[{"id":"521576","href":"/course_templates/264/video/521576","isLocked":false,"duration":213000,"title":"Data Shape","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528730","prompt":null,"isOptional":true,"activities":[{"id":"521577","href":"/course_templates/264/video/521577","isLocked":false,"duration":147000,"title":"Source, Sinks & external systems","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528731","prompt":null,"isOptional":true,"activities":[{"id":"521578","href":"/course_templates/264/video/521578","isLocked":false,"duration":146000,"title":"Shuffle and streaming engine","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528732","prompt":null,"isOptional":false,"activities":[{"id":"521579","href":"/course_templates/264/quizzes/521579","isLocked":false,"duration":900000,"title":"Quiz","description":null,"type":"quiz","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528733","prompt":null,"isOptional":true,"activities":[{"id":"521580","href":"/course_templates/264/documents/521580","isLocked":false,"duration":1800000,"title":"Additional Resources","description":null,"type":"document","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false}],"expanded":false},{"id":"76570","title":"Testing and CI/CD","description":"

This module will discuss unit testing your Dataflow pipelines. We also introduce frameworks and features available to streamline your CI/CD workflow for Dataflow pipelines.

","steps":[{"id":"528734","prompt":null,"isOptional":true,"activities":[{"id":"521581","href":"/course_templates/264/video/521581","isLocked":false,"duration":299000,"title":"Testing and CI/CD Overview","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528735","prompt":null,"isOptional":true,"activities":[{"id":"521582","href":"/course_templates/264/video/521582","isLocked":false,"duration":349000,"title":"Unit Testing","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528736","prompt":null,"isOptional":true,"activities":[{"id":"521583","href":"/course_templates/264/video/521583","isLocked":false,"duration":124000,"title":"Integration Testing","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528737","prompt":null,"isOptional":true,"activities":[{"id":"521584","href":"/course_templates/264/video/521584","isLocked":false,"duration":73000,"title":"Artifact Building","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528738","prompt":null,"isOptional":true,"activities":[{"id":"521585","href":"/course_templates/264/video/521585","isLocked":false,"duration":756000,"title":"Deployment","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528739","prompt":null,"isOptional":false,"activities":[{"id":"521586","href":"/course_templates/264/quizzes/521586","isLocked":false,"duration":600000,"title":"Quiz","description":null,"type":"quiz","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528740","prompt":null,"isOptional":true,"activities":[{"id":"521587","href":"/course_templates/264/labs/521587","isLocked":false,"duration":5400000,"title":"Serverless Data Processing with Dataflow - Testing with Apache Beam (Java)","description":"In this lab you will learn and implement testing concepts for Cloud Dataflow.","type":"lab","isComplete":false,"inProgress":false,"score":null,"disabled":false},{"id":"521588","href":"/course_templates/264/labs/521588","isLocked":false,"duration":7200000,"title":"Serverless Data Processing with Dataflow - Testing with Apache Beam (Python)","description":"In this lab you will learn and implement testing concepts for Dataflow.","type":"lab","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528741","prompt":null,"isOptional":false,"activities":[{"id":"521589","href":"/course_templates/264/labs/521589","isLocked":false,"duration":7200000,"title":"Serverless Data Processing with Dataflow - CI/CD with Dataflow","description":"In this lab you will learn and implement testing and CI/CD concepts for Cloud Dataflow.","type":"lab","isComplete":false,"inProgress":false,"score":null,"disabled":true}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528742","prompt":null,"isOptional":true,"activities":[{"id":"521590","href":"/course_templates/264/documents/521590","isLocked":false,"duration":1800000,"title":"Additional Resources","description":null,"type":"document","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false}],"expanded":false},{"id":"76571","title":"Reliability","description":"

In this module we will discuss methods for building systems that are resilient to corrupted data and data center outages.

","steps":[{"id":"528743","prompt":null,"isOptional":true,"activities":[{"id":"521591","href":"/course_templates/264/video/521591","isLocked":false,"duration":151000,"title":"Introduction to Reliability","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528744","prompt":null,"isOptional":true,"activities":[{"id":"521592","href":"/course_templates/264/video/521592","isLocked":false,"duration":238000,"title":"Monitoring","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528745","prompt":null,"isOptional":true,"activities":[{"id":"521593","href":"/course_templates/264/video/521593","isLocked":false,"duration":136000,"title":"Geolocation","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528746","prompt":null,"isOptional":true,"activities":[{"id":"521594","href":"/course_templates/264/video/521594","isLocked":false,"duration":401000,"title":"Disaster Recovery","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528747","prompt":null,"isOptional":true,"activities":[{"id":"521595","href":"/course_templates/264/video/521595","isLocked":false,"duration":246000,"title":"High Availability","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528748","prompt":null,"isOptional":false,"activities":[{"id":"521596","href":"/course_templates/264/quizzes/521596","isLocked":false,"duration":600000,"title":"Quiz","description":null,"type":"quiz","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528749","prompt":null,"isOptional":true,"activities":[{"id":"521597","href":"/course_templates/264/documents/521597","isLocked":false,"duration":1800000,"title":"Additional Resources","description":null,"type":"document","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false}],"expanded":false},{"id":"76572","title":"Flex Templates","description":"

This module covers Flex Templates, a feature that helps data engineering teams standardize and reuse Dataflow pipeline code. Many operational challenges can be solved with Flex Templates.

","steps":[{"id":"528750","prompt":null,"isOptional":true,"activities":[{"id":"521598","href":"/course_templates/264/video/521598","isLocked":false,"duration":103000,"title":"Classic templates","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528751","prompt":null,"isOptional":true,"activities":[{"id":"521599","href":"/course_templates/264/video/521599","isLocked":false,"duration":149000,"title":"Flex templates","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528752","prompt":null,"isOptional":true,"activities":[{"id":"521600","href":"/course_templates/264/video/521600","isLocked":false,"duration":155000,"title":"Using flex templates","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528753","prompt":null,"isOptional":true,"activities":[{"id":"521601","href":"/course_templates/264/video/521601","isLocked":false,"duration":167000,"title":"Google provided templates","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528754","prompt":null,"isOptional":false,"activities":[{"id":"521602","href":"/course_templates/264/quizzes/521602","isLocked":false,"duration":900000,"title":"Quiz","description":null,"type":"quiz","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528755","prompt":null,"isOptional":true,"activities":[{"id":"521603","href":"/course_templates/264/labs/521603","isLocked":false,"duration":7200000,"title":"Serverless Data Processing with Dataflow - Custom Dataflow Flex Templates (Java)","description":"In this lab you (i)convert a custom pipeline into a Dataflow Flex Template, ii) and run a Dataflow Flex Template.","type":"lab","isComplete":false,"inProgress":false,"score":null,"disabled":false},{"id":"521604","href":"/course_templates/264/labs/521604","isLocked":false,"duration":7200000,"title":"Serverless Data Processing with Dataflow - Custom Dataflow Flex Templates (Python)","description":"In this lab, you (i)convert a custom pipeline into a Dataflow Flex Template, (ii)run a Dataflow Flex Template.","type":"lab","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false},{"id":"528756","prompt":null,"isOptional":true,"activities":[{"id":"521605","href":"/course_templates/264/documents/521605","isLocked":false,"duration":1800000,"title":"Additional Resources","description":null,"type":"document","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false}],"expanded":false},{"id":"76573","title":"Summary","description":"

This module reviews the topics covered in the course

","steps":[{"id":"528757","prompt":null,"isOptional":true,"activities":[{"id":"521606","href":"/course_templates/264/video/521606","isLocked":false,"duration":277000,"title":"Course Summary","description":null,"type":"video","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false}],"expanded":false},{"id":"next-steps","title":"Your Next Steps","description":null,"steps":[{"id":"badge-step","prompt":null,"isOptional":true,"activities":[{"id":"badge","href":null,"isLocked":true,"duration":null,"title":"Course Badge","description":null,"type":"badge","isComplete":false,"inProgress":false,"score":null,"disabled":false}],"isComplete":false,"isTestedOut":false,"allActivitiesRequired":false}],"expanded":true}]

Serverless Data Processing with Dataflow: Operations