start portlet menu bar

HCLSoftware: Fueling the Digital+ Economy

Display portlet menu
end portlet menu bar
Select Page

Let us begin with understanding of Google Cloud dataflow what it is all about before moving to our GCP Cloud Dataflow plugin and how it benefits to our workload automation users.

Data is generated in real-time from websites, mobile apps, IoT devices and other workloads. Capturing, processing, and analyzing this data is a priority for all businesses. However, data from these systems is not often in the format that is conducive for analysis or for effective use by downstream systems. That’s where Dataflow comes in! dataflow is used for processing and enriching batch or stream data for use cases such as analysis, machine learning or data warehousing.

Google Cloud Dataflow

Dataflow templates offer a collection of pre-built templates with an option to create your own custom ones! Here in workload automation, you can implement Text files on Cloud Storage to BigQuery template.

The Text Files on Cloud Storage to BigQuery pipeline is a batch pipeline that allows you to stream text files stored in Cloud Storage, transform them using a JavaScript User Defined Function (UDF) that you provide, and append the results to BigQuery.

Let us now understand the plugin part with job definition parameters:

Log in to the Dynamic Workload Console and open the Workload Designer. Choose to create a new job and select “GCP Cloud Dataflow” job type in the Cloud section.

Job Definition PageFigure 1: Job Definition Page


Establishing connection to the Google Cloud server:

Use this section to connect to the Google Cloud in two ways.

  1. GCP Default Credentials — If Customer VM’s already resides inside GCP environment then no need to provide credential explicitly.
  2. GCP Server Credentials – Manually provide below details.

Service Account – The service account associated to your GCS account. Click the Select button to choose the service account in the cloud console.

Project ID – The project ID is a unique name associated with each project. It is mandatory and unique for each service account.

Test Connection – Click to verify if the connection to the Google Cloud works correctly.

Connection PageFigure 2: Connection Page


In Action tab specify the bucket name and operation which you want to perform.

  • Job Name – Specify the name of the job, must be unique among running jobs.
  • Region – Choose a dataflow regional endpoint to deploy worker instances and store job metadata.
  • Template Path – Text files on Cloud Storage to BigQuery.

Ex. gs://your-project-region/latest/GCS_text_to_BigQuery

Batch pipeline. Reads text files stored in Cloud Storage, transforms them using a JavaScript user-defined function (UDF), and outputs the result to BigQuery.

  • Javascript UDF path in Cloud Storage – The Cloud Storage path pattern for the JavaScript code containing your user-defined functions.

Ex: gs://your-bucket/your-transforms/*.js

  • JSON Path – The Cloud Storage path to the JSON file that defines your BigQuery schema. Ex: gs://your-bucket/your-schema.json
  • Javascript UDF Name – The name of the function to call from your JavaScript file. Use only letters, digits, and underscores. Ex: transform_udf1.
  • BigQuery Output Table – The location of the BigQuery table in which to store your processed data. If you reuse an existing table, it will be overwritten. Ex: your-project:your-dataset.your-table.
  • Cloud Storage Input Path – The path to the Cloud Storage text to read. Ex: gs://your-bucket/your-file.txt
  • Temporary BigQuery Directory – Temporary directory for the BigQuery loading process. Ex: gs://your-bucket/your-files/temp-dir.
  • Temporary Location – Path and filename prefix for writing temporary files. Ex: gs://your-bucket/temp.
  • Network – Network to which workers will be assigned. If empty or unspecified, the service will use the network “default”.
  • Subnetwork – Subnetwork to which workers will be assigned, if desired. Value can be either a complete URL or an abbreviated path. If the subnetwork is located in a Shared VPC network, you must use the complete URL.

Action Page

Action PageFigure 3: Action Page

Submitting your job

It is time to Submit your job into the current plan. You can add your job to the job stream that automates your business process flow. Select the action menu in the top-left corner of the job definition panel and click on Submit Job into Current Plan. A confirmation message is displayed, and you can switch to the Monitoring view to see what is going on.

Submit Job Into Current Plan PageFigure 4: Submit Job Into Current Plan Page

Monitor Page

Monitor Page

Monitor PageFigure 5: Monitor Page

Users can cancel the running the job by clicking kill option.

Job Log Details

Job Log PageFigure 6: Job Log Page

WorkFlow Page

Workflow Details PageFigure 7: Workflow Details Page

Are you curious to try out the GCP Cloud Dataflow plugin? Download the integrations from the Automation Hub and get started.

Authors Bio

Suhas H NSuhas H N, Senior Developer at HCL Technologies 

Works as a Plugin Developer in Workload Automation. Acquired skills on Java, Spring, Spring Boot, Microservices, AngularJS, JavaScript.



Rabic Meeran KRabic Meeran K, Senior Engineer at HCL Technologies 

Responsible for design and develop integration plug-ins for Workload Automation. Having two decades of experience in creating software products to enterprise customers.



Saket SauravSaket Saurav, Tester (Senior Engineer) at HCL Technologies

Responsible for performing Automation and Manual Testing for different plugins in Workload Automation using Java Unified Test Automation Framework. Hands-on experience on Java programming language, Web Services with databases like Oracle and SQL Server.


Comment wrap
Automation | July 24, 2023
Workload Automation: More than 100 Plugins Lets You Automate More (and Better)
The Automation Hub is an innovative work-in-progress as we regularly add many more items to it. It showcases HCL Workload Automation’s ability to orchestrate IT and business workloads.
Automation | February 10, 2023
Banking Case Study: New Client Adoption and Reverse Check: HWA+DRYiCE-iControl
We look at a banking case study wherein we are going through the “New Client Adoption and Reverse Check” business process.
Automation | September 28, 2022
See your scheduling metrics on Prometheus and Grafana
HCL Workload Automation has exposed its metrics for the main components, the back-end which reports metrics around job execution.