Connect AWS Batch with HCL Workload Automation

Let us first understand about the AWS Batch, before moving on to AWS Batch plug-in and how it is benefits to our workload automation users.

The AWS Batch is available on Automation Hub, download it to empower your Workload Automation environment.

AWS Batch enables developers, scientists, and engineers to run hundreds of thousands of batch computing jobs easily and efficiently on AWS. AWS Batch dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted. With AWS Batch, there is no need to install and manage batch computing software or server clusters that you use to run your jobs, allowing you to focus on analysing results and solving problems. AWS Batch plans, schedules, and executes your batch computing workloads across the full range of AWS compute services and features.

Now let us see how easy it is to start and monitor AWS Batch job.

Create a new job and select “AWS Batch” in the Cloud section.

Figure 1: Job Definition page

Connection

First, establish a connection to the AWS server by entering the required details.

Then, you can test the connection to AWS server by clicking on Test Connection.

Figure 2: Connection page

Figure 3: Connection page > Test Connection

Action

After having successfully tested the connection, you can go to the Action tab and specify the details about the job queue and define what actions you want to perform.

Job Name: Specify the name of the job, it must be unique among the running jobs.
Job Queue: Jobs are submitted to a job queue where they reside until they can be scheduled to run in a compute environment. If you do not have any job queue, create one. You can choose job queue with scheduling priority or without scheduling priority.
Job Definition: It specifies how the jobs should run. You can create/use (if exist) single node job definition or multiple job definition. Select any one of the job definition.
Share identifier: Use any text as share identifier if you use only job queue with scheduling priority. Do not enter any text on share identifier field if you use job queue without scheduling priority.

Figure 4: Action page

Job Configurations:

VCPUs: Choose the number of vCPUs that your compute environment should launch with. As your job queue demand increases, AWS Batch can increase the desired number of vCPUs in your compute environment. The minimum vcpu you can use is 1.

Memory: Specify the memory hard limit (in MiB) present to the container.

GPUs: There are different types of GPU available, specify the GPUs value. This is an optional field.

Figure 5: Action page > Job Configurations

Command:

User can use different commands for AWS batch to display in job logs for Ex:

User can write “echo” in first field

Click on + button

Then write any text like “Job Success”

This message should display in job logs after job success

Environment Variable:

AWS Batch automatically sets specific environment variables in container jobs. These environment variables provide introspection for the containers inside jobs, and you can use the values of these variables in the logic of your applications. All variables that are set by AWS Batch begin with the prefix, AWS_BATCH_.

For Ex:

Name: AWS_BATCH_JQ_NAME Value :Job queue name

Figure 6: Action page > Environment

Parameters:

When you submit a job, you can specify parameters that should replace the placeholders or override the default job definition parameters. Parameters in job submission requests take precedence over the defaults in a job definition.

For Ex:

Name: command Value: [“abc” , “data”]

Figure 7: Action page > Parameters

Submitting your job

Submit your job into the current plan. You can add your job to the job stream that automates your business process flow. Select the action menu in the top-left corner of the job definition panel and click on Submit Job into Current Plan. A confirmation message is displayed, and you can switch to the Monitoring view to see what is going on.