We are rolling out a 3-part blog series on how to test your clinical automation system effectively and accurately. Subscribe here so you don’t miss an article.

Part 1: A Practical Approach to Risk-Based Automation System Testing

Laboratories have always advocated quality, efficiency, and safety in the management of their diagnostic and companion data. Yet, when it comes to implementing and validating automation systems, there is little available support or guidelines. Technology brings both opportunities and presents challenges when faced with having to validate an automation system. Unfortunately, there is no universal guidebook or detailed instructions to follow to ensure us that our automation system’s functionality will operate in support of and in conjunction with the lab’s workflow and data requirements. At the end of the day, it’s up to us to ensure that our automation system meets our clinical and business requisites before it goes into service.

Don’t Panic! First, Understand the Scope

Many laboratories struggle with the reality of making sure their new automation system works in their lab. Labs are intimately familiar with validating new tests, instruments, and processes as a standard lab activity. But the introduction of a system with many moving parts and variable data sources, increases the need to systematically plan the validation effort based on an orderly verification of the data outcomes and workflow impacts. Many labs ask – where do we start and how can we accomplish what seems like a monumental effort to ensure that the system we bought fits our quality and requirement needs?

Any system automation verification plan should start with understanding the scope and purpose of testing. Your automation system vendor will certainly validate the functionality of the automation application – they will test that it performs according to the manufacturer’s specifications. They will verify that the automation system performs against a standard manufacturer’s feature-function set to ensure code continuity and operation within the system. They generally do not, however, verify how the automation system performs within the laboratory’s own environment, against the laboratory’s specific configuration settings, the lab’s workflow, and the lab’s decision rules. The lab, not the vendor is responsible for ensuring the system operates as expected within its own unique environment. We call this type of verification, user validation, user testing, or user verification. For this blog, we are going to use the term ‘User Validation’.

Typically, two distinct phases of User Validation need to be completed to ensure your automation system is ready for live operation: Dry Phase Testing and Wet Phase Testing. Together, these phases cover the clinical and operational aspects of your system's functionality.

In this blog we will discuss the Wet Phase Testing, sometimes also referred to as the ‘live system testing’ phase of your automation system implementation project. It is assumed that once you move to the Wet Phase Testing, that the Dry Phase Testing is completed and your rules and related workflows have been established and tested as thoroughly as possible in an isolated dry system environment. You will experience a smoother go-live of your automation system when you fully test your software decision rules and workflows prior to moving to the Wet Phase Testing.

The 3 Elements of Wet Phase Testing

The Wet Phase Testing section of your User Validation Plan should include at least three discrete testing elements, designed to confirm different aspects of your system performance in conjunction with your Host Systems (e.g., LIS, LIMS, or other connected systems). Ideally, each testing element should be executed in the following order to reduce repeat testing cycles because each testing element builds on the previous one.

Wet Phase Testing Element 1: Order and result verification

Confirmation that each individual order, profile, battery is received from the Host Systems to the automation system and transmitted to the instrumentation and devices, as designed by the workflow logic. Verification that results are transmitted back to the Host Systems, and in the correct format.

Wet Phase Testing Element 2: Workflow and rules logic confirmation

Validation of each workflow, combining single, multiple, and complex order combinations on the automation system, confirming specimen movement to each instrument and module. Subsequent confirmation of the rules logic for repeat, reflex, and dilution management including specimen storage according to the automation system configuration.

Wet Phase Testing Element 3: Peak volume performance certification

Confirmation of overall system performance by conducting at least two high-volume testing sessions of up to 300-500 orders that approximates the peak hourly volume of the planned system capacity. This is usually the final validation test exercise before the automation system is ready for live operation.

Design Your Wet Phase Testing Strategy

Before the Wet Phase Testing can begin, each of the three elements are scoped, defined, and aligned with your testing objectives:

Objective 1: Organize your testing based on risk

The first step in designing your wet testing strategy is to factor in risk. Ideally, focus on the high-impact and most critical areas of your automation system. This approach consists of identifying the risks associated with the different automation processing units and their associated workflows. Each workflow is assigned a technical and a clinical score to help you rank them and prioritize the testing effort. A risk-based approach ensures that your testing is allotted the right amount of effort to test the higher-risk workflows. Determining the focus of your testing helps you assign testers, the timing of materials and infrastructure needs, and the timeline of each Wet Phase Testing element. We will show you how to create a risk-based approach later in this blog.

Objective 2: Identify human proficiency required for testing

Once you have identified your testing scope and priorities based on your risk analysis, you are ready to assign personnel to the different elements of the Wet Phase Testing. It’s best to base your assignments on expertise and area of responsibility. Other considerations such as positive attitude, openness to change, natural curiosity, collaboration and good communication are essential in building a strong testing team that can work together to validate your system operation.

The different elements of Wet Phase Testing can be performed by multiple testers based on their skill sets. You might assign IT support to verify order and result interface result workflow and a clinical technologist to review the rule and workflow logic. The peak performance test could be a combined effort of IT, clinical and business stakeholders to ensure the system is meeting your clinical and operational needs.

Objective 3: Coordinate testing environments, instruments/devices, specimen, and supply chain variables for testing

One of the biggest areas of failure in automation testing is not planning the right infrastructure to ensure that the three elements of Wet Phase Testing receive adequate support. Each element will require different supplies, expertise, and test data. A support document should describe, for example, the staging system, instruments, and devices that will be used as well as the related materials and specimens needed. Your testing infrastructure will include IT and security support as well. Part 2 of this three-part blog series will give you a stepwise approach for planning your Wet Phase Testing infrastructure.

Objective 4: Time constraints

No laboratory has unlimited time to verify their automation system operation and performance. An estimated timeline should be assigned to the three elements of Wet Testing based on your risk assessment of the functionalities, workflows, tester availability, and infrastructure requirements. External organizational business factors or other constraints including those which are beyond your control will also be factored into your timeline. Your risk analysis will not only help you plan the testing schedule and your targeted go-live date, it will also give you an invaluable tool for managing the testing when the timeline is truncated or cut short due to unexpected circumstances.

How Much Testing is Enough?

There is no getting around it: allocating time and resources for testing will take away from other activities, so planning Wet Phase Testing using a risk-based approach just makes sense. Your risk-based testing will identify and eliminate waste in the testing process and focus instead on the automated workflows that represent the priorities of your laboratory business and best practices.

Risk-based testing (RBT) is a methodology that provides a framework for accessing the resources required to provide adequate testing. RBT testing involves in this context, prioritizing your testing based on the complexity of your workflow and rules logic. In general, the amount of testing is weighed against technical complexity, dependencies impacting the test, and the effort and technical skills required to perform the testing.

Here are the steps for getting started with your RBT testing. Diagram 1 gives an example of a workflow risk register that represents the risk-based approach for each of our three elements of Wet Testing.

Create a workflow risk register of your major automation workflows based on the disciplines, instruments, and modules supported by the automation system. Identify 10-15 key workflows with associated orders and document them in a spreadsheet. The intent is to cover your general rule types. Each individual rule and variable/value input will have been verified in the Dry Phase Testing.

Step 1: Define each major workflow by name, order type, and automation system module or components included in the workflow

Step 2: Identify any specific configuration, rules, or settings per the workflow

Step 3: Identify the type of test

• Order and result verification

• Workflow and rules logic confirmation

• Volume/peak performance

Step 4: Grade the workflow from low to high complexity using a numerical system. Consider for each workflow: technical criticality, testing dependencies, and complexity covering the major functions and features of the automation system.  Be careful not to get so granular in your initial assessments. Avoid trying to cover every possible workflow or rule combination. Suggested risk assessments for each type of testing:

• Technical complexity by the number of inputs (orders), rules, or routing logic

• Dependencies on other workflows, rules, or operational logic

• Estimated effort (time required to prepare and execute the testing)

• Technical skills required to complete the testing

Step 5: Total up the scores for each workflow from step 3.

Step 6: Review each workflow and eliminate any out-of-scope testing or redundant testing across workflows

Step 7: Re-sort and rank each workflow from high to low complexity for planning your testing time and effort. Your risk register will focus you on high-risk workflows that have a bigger impact on the success of your automation system performance. This risk methodology also identifies your most critical clinical and business practices and possible gaps in logic that support each workflow.

Build and organize your test plans for each of the three testing elements of the overall Wet Phase Testing, giving greater weight to the highest risk workflows.

Diagram 1: CBC Workflow Testing Risk Analysis for Order Confirmation with LIS

Test Smarter, Not Harder

When it comes to Wet Testing try the smarter risk-based strategy rather than the “just test everything” approach. You will gain a sensible framework for testing and an effective approach for allocating scarce project resources.

Continue on with our clinical automation testing blog series to learn how to apply the risk-based method to our three elements of Wet Testing. In part 2 we will discuss how to build maintainable test plans based on your risk assessments.