Archiving Jobs

• Archiving Jobs
  • Overview
    • Extent
  • Process
    • Archiving Jobs
    • Accessing Job Data

Overview

For those organisations that intend to utilise CCLAS in a larger domain, the amount of transactional data is expected to grow continually. As with all database-based systems, indefinite database growth eventually impacts the performance and efficiency of the CCLAS application. As with most transactional-based systems, eventually the relevance of the vast majority of the operational data diminishes to the point that it has no real value to the current organisation. However, there are often corporate governance and legal requirements to store and recover electronic data for a period of time (sometimes for seven years or more), so there is a corresponding need to make it possible to access this electronic data later, for review or audit. Therefore the need of keeping sufficient operational data live to enable the laboratory to perform their day to day tasks, and the need to routinely reduce the number of records in the database to keep performance consistent over time, requires balance. Archiving of operational transactional data, once it is not required to be kept live in the database, ensures optimal performance of the CCLAS product.

Archiving of operational transactional data supports:

• The ability of a laboratory to define the live retention time required on specific CCLAS record sets in the Live Production Environment, for example, keep operational data live for six months. This ensures live data is kept at optimal overall size, to ensure data volumes are managed, yet still provide current data for operational requirements. This optimisation strategy reduces operational data costs and optimises performance times for user interaction with the system.
• The archiving of transactional data from the live CCLAS databases into an Archive Repository.
• The ability to restore meta-data records from the archive database, so that they are fully available within the CCLAS application for normal processing requirements.
• The removal of transactional data from the archive database into an Offline Storage facility, at an independent retention cycle (for example, after 24 months). This includes the exporting of available files (reports etc) and essential CCLAS data into a computer-readable format (eg XML files).

The following functional requirements are supported by CCLAS archiving:

• There is no perceived and significant performance degradation of use of the application due to more laboratories and jobs coming online in the Live Production Environment.
• Operators performing operational tasks in the Live Production Environment are not overly compromised nor impacted due to aged jobs in the system.
• Laboratory supervisors working over typical operation periods of time can locate the records they require without having to consider whether records are archived or not.
• Laboratory managers have access to their typical managerial data without having to consider whether data is archived and missing from their management views or reports.
• Laboratory users who require access to aged jobs need to decide whether to request live or archived jobs when searching for jobs using the Registration or Status Report applications.
• Laboratory users require security permissions to retrieve a job from the Archive Repository back to the Live Production Environment.
• System administrators have configuration control as to when it is suitable to move the data that support jobs, invoices and reports from the Live Production Environment into the Archive Repository. This configuration is scoped by laboratory and by job type, so that the specific operational requirements of each laboratory, in terms of retaining their live data, is managed. This is also balanced across the whole CCLAS system to ensure that performance of the whole is not compromised by the data retention requirement of the few.
• System administrators have configuration control as to when it is suitable to move CCLAS record sets in the Archive Repository into the Offline Storage. This configuration is scoped by laboratory and by job type, so that the specific operational requirements of each laboratory, in terms of retaining their archived data, is managed.
• System users require security permissions to extract CCLAS record sets from the Offline Storage, to make documents and reports are available, and the CCLAS record sets available in some electronic format that is suitable for simple viewing/reading.
• The Archive Repository and Offline Storage should be backed up to ensure that data loss is minimised.
• When the CCLAS application is upgraded, it is not impacted by the archiving environment, and vice versa. For example, data model changes to CCLAS are propagated to the Archive Storage, and the Archived Storage data is still retrievable back into the Live Production Environment despite data model changes occurring across an upgrade.

Extent

• Jobs—Each job has a definitive starting and ending period where the job is actively worked on in the laboratory. Jobs may be completed in a number of days or weeks. The job's life-cycle typically ends when the final analysis certificate or results and invoices are presented to the customer. After this, since the customer might request additional details or work on the samples, then the jobs should be kept live in the system for some months after the job is initially completed.

  Note: All template, schedule and draft jobs are only kept in the online space, and therefore have an Archive Status of Online Only.

• Laboratory batch jobs—Each laboratory batch job has a definitive starting and ending period where the job is actively worked on in the laboratory. Laboratory batches may be completed in a number of days. The life-cycle typically ends when the final results are validated, thereby allowing the samples to become completed in the related production jobs. Some laboratory batch results might be included in the reports of the production jobs. As laboratory batch jobs typically include multiple production jobs, then the laboratory batch jobs should be kept live in the system independently of the production jobs.
• Certificate reports—Certificate reports can contain results from multiple production jobs, therefore, the certificate reports should be kept live in the system longer than the jobs they contain. Being able to locate and extract existing certificate reports is a common request, so unless a report has to be regenerated for some reason, then retaining the underlying meta-data may not be needed.
• Invoice documents—Commercial invoices documents can comprise pricing across multiple production jobs, therefore, the invoice documents should be kept live in the system longer than the jobs they contain. Being able to locate and extract existing invoice documents may be required, however, retaining the underlying meta-data may not be needed.
• QC history records—QC history records contain results of specific QC samples, held independently of the jobs in which they were analysed. Therefore, as QC history records provide valuable data and statistics to the laboratory, typically, this data has time-based relevance. The system administrator only removes QC history records from the CCLAS system once they no longer have relevance, independently from the archiving process.
• Management Operational Data—Summary records of an operational performance nature may be required with a retention period beyond the life-cycles of the individual jobs. This data provides summary statistics of the workloads and throughputs of a laboratory, and may be required to be kept over several operational reporting periods.
• Financial Operational Data—Summary records of an invoicing or financial performance nature may be required with a retention period beyond the life-cycles of the individual jobs. This data provides summary statistics of the workloads and throughputs of a laboratory, and may be required to be kept over several operational reporting periods.
• Sample Storage Data—Physical samples may be stored by the laboratory for a period of time after analysis. There is a chain-of-custody responsibility on the laboratory to keep records of the samples even after the physical samples are returned or destroyed. Typically, commercial laboratories may not take on very long-term storage of the sample materials, however, if they do, the electronic, as well as the physical records, must be retained. Once samples are returned to the customer or destroyed, these electronic records may be required retention for a period of time, should there be any questions raised by the laboratory or the customer.

Process

When a job record is created by any means, the job's Archive Status defaults to Live Production Environment.

When any job child record, sample record, sample child record or job audit record is created, the entity's Archive Status is inherited from the job, which ensures that the job is not located across multiple partitions, but in one partition for efficient retrieval.

Archiving Jobs

Jobs are archived periodically by system processes.

Accessing Job Data

When Maintaining Jobs, Locating Jobs for Analysis and Maintaining Jobs for a Report Request, then jobs are searchable by their archive status.

When Maintaining Jobs to review or update a job, then the archive status of the job is displayed on the Primary Details tab.

When Maintaining Samples, Maintaining Composite Samples and Maintaining Samples for a Report Request, then samples are searchable by their archive status.

When Maintaining Samples to review or update a sample, then the archive status of the sample is displayed on the Primary Details tab.