GSI Health - Litmus Paper: Quality Assurance

It is common parlance in software engineering that there is no such thing as bug-free software. Similarly, it is often said that quality is a continuum, and the amount of quality one receives depends on how much quality assurance one is willing to pay for. So, in this age of interoperability and interconnected healthcare systems sharing sensitive patient data, and opening organizational boundaries for the sake of information exchange, how much quality assurance is enough to deem this new generation of healthcare IT systems safe? Asked more simply, before you commit precious data from your organization, your region, or even your entire state, into the networks and databases of your chosen healthcare IT solution, how will you know it will work? Shockingly, the answer for many is, “We don’t know.” They don’t know if the architecture they built upon is likely to cause a major privacy breach. They don’t know if the performance of the system will grind to a halt in six months or six hours. They don’t know the expected error rate for correctly matching patient identities across care settings, or how often filters to prevent certain protected data from entering the network lets a wayward record slip through. They don’t know how protected they are from errors occurring across the network in systems they do not control. In short, an enormous healthcare IT infrastructure is being constructed piece by piece across the nation with woefully inadequate levels of testing planned; but you can still protect yourself through a proactive, defensive quality assurance (QA) program punctuated by a strong independent verification and validation effort (IV&V).

When evaluating a healthcare IT solution, you need to consider several litmus indicators before concluding that your plans for upgrading your healthcare IT architecture and/or systems will not leave you exposed to unforeseen-but-preventable errors that disrupt operations and increase liability risk. The following table lists some of these important considerations.

Litmus Indicator	Description
Functional-Test Fallacy	A lot of solution vendors will perform some kind of functional testing to demonstrate that the requirements given to them by the customer have been implemented. However, most functional testing neglects negative cases (testing for things that aren’t supposed to happen), and is typically light on data-driven testing (systematically using voluminous, well designed data for testing scenarios). Functional testing alone is inadequate to ensure safe operation.
Acceptance-Test Fallacy	Some vendors and contractors leave it to the customer to perform acceptance tests to verify that they have done all they were contracted to do. Most purchasers of healthcare IT, however, are not equipped to perform thorough and professional testing of a sophisticated IT system. Having acceptance tests is a good idea, but the onus is on the customer to make those tests robust and meaningful. Often acceptance tests boil down to demonstrations and self-attestations by the vendor or contractor. This is where IV&V is very valuable.
Vendor, Test-Thyself Fallacy	Ceding all testing to the hired vendor or contractor is abdicating responsibility for correct operation. This is especially dangerous when the vendor product is interfaced with other products, which should prompt a comprehensive end-to-end integration testing activity to the extent possible in addition to any testing the vendor or contractor does themselves.
Test Data Inadequacy	Most test data is very small in volume, has little resemblance to real-world data, and is often used as a support for other kinds of testing. Data-driven testing, which systematically uses voluminous, well designed data for testing scenarios, provides for much more solid quality assurance.
Production-Data-as-Test-Data Fallacy	Many vendors and contractors will request to use a slice of real production data from the customer environment as test data. The idea is that this will provide more realistic testing, which it does compared to most synthetic test data. The problem with this is that the extract contains random data which defies characterization in a useful way to see what amount of test coverage is accomplished by using it. It is better to have realistic data that can be used in a systematic way to march through explicit data-based test scenarios.

GSI Health Can Help You! GSI Health has extensive experience in testing healthcare IT systems, and focuses on data-driven testing for understandable and thorough test batteries. Our consultants can assist you in navigating the issues discussed in this paper, and provide the professional IV&V services you need to properly test your healthcare IT applications and architecture. Our testing methods account for adherence to national standards, and are extensible to accommodate local requirements. To learn more about how GSI Health can help make you successful in deploying a strong quality assurance program as a part of your healthcare IT project, please contact us by phone at 1-888-206-4237 or email HITvision@gsihealth.com.