create a Data Fusion instance and deploy a sample pipeline

Create a Data Fusion instance and deploy a sample pipeline that reads an input file from Cloud Storage, transforms and filters the data to output a subset of the data into BigQuery.

click Data Fusion Instance view

Task 4. Deploy a sample pipeline

Sample pipelines are available through the Cloud Data Fusion Hub, which allows you to share reusable Cloud Data Fusion pipelines, plugins, and solutions.

In the Cloud Data Fusion web UI, click HUB on the top right.

Highlighted HUB link

In the left panel, click Pipelines.

Click the Cloud Data Fusion Quickstart pipeline, and then click Create on the popup that appears.

Highlighted Cloud Data Fusion Quickstart tile on the Pipelines page

In the Cloud Data Fusion Quickstart configuration panel, click Finish.

Click Customize Pipeline. A visual representation of your pipeline appears in the Pipeline Studio, which is a graphical interface for developing data integration pipelines. Available pipeline plugins are listed on the left, and your pipeline is displayed on the main canvas area. You can explore your pipeline by holding the pointer over each pipeline node and clicking the Properties button that appears. The Properties menu for each node allows you to view the objects and operations associated with the node.

Note: A node in a pipeline is an object that is connected in a sequence to produce a Directed Acyclic Graph. E.g. Source, Sink, Transform, Action, etc.

Pipeline Studio displaying a visial representation of the pipeline

In the top right menu, click Deploy. This submits the pipeline to Cloud Data Fusion. You will execute the pipeline in the next section.

 

click HUB

click pipelines

Task 5. View your pipeline

The deployed pipeline appears in the pipeline details view, where you can do the following:

View the pipeline's structure and configuration.

Run the pipeline manually or set up a schedule or a trigger.

View a summary of the pipeline's historical runs, including execution times, logs, and metrics.

Task 7. View the results

The pipeline writes output into a BigQuery table. You can verify that using the following steps.

Click to open this link to the BigQuery UI in Cloud console or right-click on the console tab and select Duplicate, then use the Navigation menu (Navigation menu) to select BigQuery.

In the Classic Explorer pane, click your Project ID (it will start with qwiklabs).

Under the GCPQuickstart dataset in your project, click the top_rated_inexpensive table.

Click + SQL Query, paste the query below, and then click Run.

SELECT * FROM `qwiklabs-gcp-03-0710fb149621.GCPQuickStart.top_rated_inexpensive` LIMIT 10

Copied!

Wait for the query to finish. A similar Results will appear.