Configuring the System Batch Queue

Overview

Batch processes are an extremely efficient way of processing large amounts of data. The ability to schedule and prioritise work based on Service Level Agreements (SLAs) lets you allocate resources for best utilisation.

CCLAS 6 uses the Spring Batch open source framework for batch processing.

Spring Batch is the de facto standard for batch processing on the Java Virtual Machine (JVM). It lets you create high-performing, scalable batch applications that are resilient enough for your most mission-critical processes.

Spring Batch implements common batch patterns, such as chunk-based processing and partitioning, for the processing of large volumes of records, with logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management, and with more advanced features for optimising the through-put of high-volume and high performance batch jobs.

Spring Batch—Reference Documentation

Spring Batch—Tutorial

It is important to protect the system batch queue from overload so that more important batch queue tasks can be executed, for example, to place priorities on system batch jobs by job type or to limit the requests made by a user to generate job paperwork for the same job within a small time frame.

Process

Prioritising System Batch Jobs by Job Type

System batch jobs submitted to the batch queue can be submitted with a processing priority. The BATCH_QUEUE_TASKS_PRIORITY application preference defines the default priority at which system batch jobs are executed in the system batch queue. The Scope for this preference is usually set at laboratory level.

Set the BATCH_QUEUE_TASK_PRIORITY preference

Note: These timeout settings should be used with caution as some system batch jobs may require frequent running within a short period of time.

Placing Queue Limits on System Batch Jobs

When a system batch job is submitted to the system batch queue, then the system determines whether queue limits are in place for the job type, to potentially proceed with or deny the system batch job (with logging to indicate a denial). This ensures that particular types of batch queue jobs are not created excessively within a fixed time period, but are ignored.

The system does this by constructing a parameter checksum in the BATCH_JOB_QUEUE table from the system batch job's parameters, and assessing whether the number of system batch job entries based on that checksum is less than a) an allowable maximum number of duplicate entries, within b) a given time frame, where:

The allowed maximum number of duplicate entries is set by the batch.queue.control.<batch_job_name>.allow.max.duplicate.entry system user preference, for example, 5, and
The last defined time period is set by the batch.queue.control.<batch_job_name>.check.timeframe.in.minutes system user preference, for example, 60,

with each preference defined in the MSEPRF—User Preferences application, and where the particular batch_job_name is one of the job types specified in the BATCH_JOB_QUEUE table.

Note: These preferences are NOT lab or user specific, but are system-wide. These preferences are NOT installed by default, and therefore, no limits are applied for a given batch job name until BOTH preferences exist and are defined for that job type. Also, it takes up to ten minutes for the system to apply changes to system user preferences.

Where batch queue limits are exceeded for a particular job type, such that a requests for that job type is denied, then the denial details are recorded in the system server logs.

Note: Apart from the system server log and the lack of new system batch queue jobs being added to the system batch queue, there are NO other alerts created for the users to know that the management mechanism has kicked in and prevented the creation of excessive system batch jobs.

Example to specify a maximum number of ten system batch jobs to create/update report requests for the creation of paperwork and labels within any five minute period, using the CCBPGEN job type for the system batch job in the above table, set the following system user preferences:

batch.queue.control.ccbpgen.allow.max.duplicate.entry = 10

batch.queue.control.ccbpgen.check.timeframe.in.minutes = 5.

Example to specify a maximum number of five system batch jobs to generate reports from report requests within any five minute period, using the CCBRGEN job type for the system batch job in the above table, set the following system user preferences:

batch.queue.control.ccbrgen.allow.max.duplicate.entry = 5

batch.queue.control.ccbrgen.check.timeframe.in.minutes = 5.

Placing Execution Limits by Report Request Code on CCBPGEN and CCBRGEN System Batch Jobs

The detection and processing of system batch jobs ensures that for a CCBPGEN (Paperwork and Labels Report Generation) or CCBRGEN (Certificate and Analysis Report Generation) job type, the system batch processor will not process another system batch job for the same report request until the first system batch job has a status of COMPLETED or FAILED, that is, while the system batch job still has a STARTED state.

For other job types, this restriction is not in place. That is, where a job type other than CCBPGEN or CCBRGEN is in a STARTED state on the system batch processor, then another system batch job of the same job type can be triggered and executed.

Given this, it is advisable that job types like instrument result upload tasks are not scheduled more frequently that the typical time it would take to process the average sized instrument data file, otherwise, queued system batch jobs would eventually bank up and block the batch servers.

Note: If a system batch job's status is STARTED, and the batch server is subsequently restarted, then these in-progress tasks are NOT automatically picked up by other system batch servers.

These started but never completed system batch jobs linked to the system batch processor can be monitored, and if more than 1 day old, then they could be readily deleted.

Note: The setting of the concurrent processing is via System Configurations only and is not definable via the CCLAS UI.

In the system configuration files, the two Batch Tasks (CCBPGEN and CCBRGEN) are defined as being restricted. Future releases might extend this to other Batch Tasks — that is, cclas.batch.job.names.queue.check=CCBPGEN,CCBRGEN.

Important: If these started but not completed system batch jobs are of a CCBPGEN (Paperwork and Labels Report Generation) or CCBRGEN (Certificate and Analysis Report Generation) job type, then these should be deleted as it would prevent the specific report request from triggering further report generation jobs on the system batch queue.

Placing Timeouts on Processing System Batch Jobs

A number of batch jobs have a custom transaction timeout value set, the timeout defaults to 5 minutes if not set. This custom setting exists in a batch configuration file in the CCLAS code repository, and this file is included in the CCLAS application package (the EAR).

Note: The batch configuration file is not our own format. The batch job system/infrastructure in the Java Interface (JI) component of CCLAS 6 relies on a third-party library. The format and settings in the configuration file follow the standard specified by that third-party library.

The timeout configuration is by batch job step, that is, if a batch job has multiple steps, then each step can have a different timeout.

Refer to Configuring the System Batch Queue for prioritising jobs on the system batch queue by job type and protecting the system batch queue from excessive requests.

System batch jobs submitted to the batch queue can be submitted with a processing priority.

Example of CCLAS-DM settings - Ansible playbook: 7.2.2-boot file: 7.2.2-boot\vars\cclas.yml

-Dcclas.batch.libman.export.timeout={{ cclas_batch_libman_export_timeout | default(7200) }}
-Dcclas.batch.libman.upload.timeout={{ cclas_batch_libman_upload_timeout | default(7200) }}
-Dcclas.batch.libman.upload.notifier.timeout={{ cclas_batch_libman_upload_notifier_timeout | default(7200) }}
-Dcclas.batch.schedule.job.generator.timeout={{ cclas_batch_schedule_job_generator_timeout | default(3600) }}
-Dcclas.batch.schedule.job.sample.recovery.timeout={{ cclas_batch_schedule_job_sample_recovery_timeout | default(3600) }}
-Dcclas.batch.schedule.job.sample.recovery.by.job.timeout={{ cclas_batch_schedule_job_sample_recovery_by_job_timeout | default(3600) }}
-Dcclas.batch.generate.paperwork.timeout={{ cclas_batch_generate_paperwork_timeout | default(3600) }}
-Dcclas.batch.email.timeout={{ cclas_batch_email_timeout | default(3600) }}
-Dcclas.batch.print.timeout={{ cclas_batch_print_timeout | default(3600) }}
-Dcclas.instrument.parse.step.timeout={{ cclas_instrument_parse_step_timeout | default(7200) }}
-Dcclas.instrument.sync.timeout={{ cclas_instrument_sync_timeout | default(3600) }}
-Dcclas.batch.workbook.trigger.check.analysis.timeout={{ cclas_batch_workbook_trigger_check_analysis_timeout | default(300) }}
-Dcclas.batch.generate.report.timeout={{ cclas_batch_generate_report_timeout | default(3600) }}
-Dcclas.batch.generate.report.email.timeout={{ cclas_batch_generate_report_email_timeout | default(3600) }}
-Dcclas.batch.generate.report.print.timeout={{ cclas_batch_generate_report_print_timeout | default(3600) }}
-Dcclas.batch.invoice.gen.timeout={{ cclas_batch_invoice_gen_timeout | default(3600) }}
-Dcclas.batch.invoice.gen.email.timeout={{ cclas_batch_invoice_gen_email_timeout | default(3600) }}
-Dcclas.batch.invoice.gen.print.timeout={{ cclas_batch_invoice_gen_print_timeout | default(3600) }}
-Dcclas.batch.invoice.merge.timeout={{ cclas_batch_invoice_merge_timeout | default(86400) }}
-Dcclas.batch.invoice.combine.timeout={{ cclas_batch_invoice_combine_timeout | default(86400) }}
-Dcclas.batch.invoice.reprice.timeout={{ cclas_batch_invoice_reprice_timeout | default(86400) }}
-Dcclas.batch.invoice.change.pricing.mode.timeout={{ cclas_batch_invoice_change_pricing_mode_timeout | default(86400) }}
-Dcclas.batch.invoice.recalculate.wip.est.timeout={{ cclas_batch_invoice_recalculate_wip_est_timeout | default(86400) }}
-Dcclas.batchscript.executor.timeout={{ cclas_batchscript_executor_timeout | default(3600) }}
-Dcclas.batch.job.clean.timeout={{ cclas_batch_job_clean_timeout | default(3600) }}
-Dcclas.batch.ssa.file.remover.timeout={{ cclas_batch_ssa_file_remover_timeout | default(3600) }}
-Dcclas.batch.cclas.cleaner.timeout={{ cclas_batch_cclas_cleaner_timeout | default(7200) }}
-Dcclas.batch.msbcsb.timeout={{ cclas_batch_msbcsb_timeout | default(3600) }}