What Is Batch Automation—and Is It Right for You?

Batch processing of data has a long and fascinating history, dating back well over a century to the Industrial Revolution.

One of the earliest cited instances of batch processing dates to 1890, when workers at the U.S. Census Bureau gathered census data onto punch cards, which were then fed in batches through an electronic tabulator.

By the mid-twentieth century, with the advent of the mainframe computer, batch programs were created on magnetic tape to process data in sequential order throughout the workday. Today, all types of firms use batch processing for a wide range of uses, including generating automated reports, updating databases, and processing high volumes of transactional data within specified time windows.

What is batch automation?

Batch automation is the term used to describe the method of scheduling high-volume, repetitive computer jobs to be automatically processed at one time.

Typically, batch jobs are scheduled to be processed during specified batch windows, often after hours when system usage is lower than during business operational periods. This is because batch workloads are often high volume and data rich, thus requiring significant amounts of computational power and resources.

Certain workload scenarios make the most sense for using batch automation. These include:

• When it would be too inefficient, time consuming, and resource intensive to run individual data jobs.
• When running jobs during normal operating times would use too many CPU resources, negatively impacting business operations.
• When the information, data, or reporting outputs don’t need to be accessed immediately or on a real time or continuous basis.

Batch processing is used in a number of common scenarios across many industries.

For example, in financial services, batch processing has been commonly used for high-volume applications like after-hours check processing, overnight reporting, account statement generation, and ACH transaction processing.

For insurance companies, batch processing is often used for generating premiums, as well as in high-volume claims processing scenarios such as workers’ compensation filings.

What are the three types of batch automation tools?

Batch automation tools come in three distinct flavors: service orchestration and automation platforms (SOAPs), workload automation (WLA), and schedulers. Each type of automation tool has pros and cons when used for automating batch processes.

SOAPs are premium, high-end, modern automation tools. They offer users a “single-pane-of-glass” approach to managing back-office automation by providing a single, consolidated view and centralized management of automation across multiple distributed servers and applications. They are also capable of spanning cloud, on-premises, and hybrid servers and data centers.

SOAPs have six core competencies[i]:

1. Application workflow orchestration: This refers to the ability to create truly complex business flows that cross applications, operating systems, and anything in between to properly automate modern business processes.
2. Event-driven automation: This type of automation is triggered by specific events or actions that occur within a business, rather than being limited to time- and dependency-based flows.
3. Scheduling, monitoring, visibility, and alerting: These advanced features allow users to define when workflows run, easily monitor their status, and receive intelligent alerts of any potential processing issues.
4. Self-service automation: Business users with no IT skills can be given the ability to manage their own workflows through a secure, easy-to-use interface.
5. Resource provisioning: SOAPs typically offer end users the ability to dynamically build up and tear down infrastructure in response to workload and resource needs. Such flexibility is becoming critically important as environments increasingly move to the cloud.
6. Managing data pipelines: This functionality refers to advanced file transfer management and deep integrations into databases, data warehouses, and data lakes. It empowers users to move data around the enterprise as needed to enable the further automation of business processes.

WLA tools can be viewed as a less feature-rich version of a SOAP. Like a SOAP, WLA tools frequently have the ability to define complex chains of business processes that incorporate conditional logic. They also have broad integration lists that can automate many applications from a single tool. Typically, they offer event-driven automation, allowing changes in the business to trigger automation events. WLA solutions also offer a feature-rich scheduling system that allows for time- or calendar-based run plans.

Monitoring, visibility, and alerting capabilities exist, but they're typically less robust than those offered by a SOAP. Where SOAPs might provide customized views for external business stakeholders or senior management, WLA tools typically provide this functionality only for automation operators. SOAPs can often predict if a job won’t complete on time or is behind schedule, while WLA tools do not. Self-service automation might be missing entirely or not be delivered in an easy-to-use model, preventing the tool from transitioning beyond IT to a business-wide automation and orchestration platform.

WLA tools tend to be weakest in resource provisioning and managing data pipelines. Functionality delivered here is often not deeply integrated. Dynamic workloads and infrastructure management capabilities are generally cumbersome. WLA can handle simple file transfers but not more complex and sophisticated scenarios.

Schedulers are the simplest tools available for batch automation. They typically interface with fewer servers, applications, and data sources than SOAPs or WLA tools and offer only time-based scheduling with limited workflow composition or dependencies. Schedulers are meant to be used with a single server, so automating more than one server requires employing multiple unintegrated copies of the software. Most operating systems have simple schedulers that come bundled with their core features and functionality.

Which tool will work best for your batch processing scenario?

Deciding which automation tool is best for your unique batch processing scenario comes down to a few factors—mainly complexity and cost. SOAP has emerged as the best choice for automating today’s complex, multi-server, and hybrid cloud environments as it offers centralized management and maximum control. For many organizations with sophisticated needs, it’s often well worth the extra expense.

[i] As defined by Gartner