In this post I will share some performance testing experiences from executing formal software performance verifications in healthcare industry. This is how performance testing of a medical product resulting in an official report is called there.
It is a bit different affair than testing in other domains.
What is characteristic about the healthcare domain
Starting a first project, I already knew that attention to detail and documentation is crucial in healthcare. That is understandable, as it’s checked by auditors or regulators like FDA etc. after all.
Now, having few projects in that industry under my belt I can say that there are also a few counter-intuitive aspects for me.
The biggest one was that the customer seems to care more about receiving a binary result for each test (passed/failed), than to get a clear set of conclusions and recommendations.
In order to tackle it you need to formally structure your performance tests in a similar fashion to functional tests with a very clear set of steps and pass criteria.
The issue is that the outcome of performance tests quite often is not binary. It is complex and has different facets. Some parts might be failing, but they could have no impact on the overall condition of the system.
Another surprising thing is that the quality of the non-functional requirements seems to be on similar level as elsewhere. And nope, that is not a compliment…
As a result you might need to deal with obscure requirements like: ‘Application should respond in reasonable time’.
Initial painful experience
Recalling my first project in the medical industry, planning and executing extensive set of tests actually backfired against me.
Some tests were planned to put the system on its knees and make it collapse, to give the client some data about system’s limits. Quite amazingly for me, after concluding the formal report, the customer was not asking technical questions concerning nature of the issues and the recommendations, but rather about other trivial reported stuff in my opinion.
I’ve learned my lesson and for the next assignments I chose to perform only the minimum set of tests covering the requirements during the formal verification. If I felt that other tests, like specific stress tests could deliver interesting information, I only did them as non-formal tests and haven’t mentioned them in the documents to avoid all the hassle.
Few advises on how to approach performance testing in healthcare projects
- First of all, beware that the healthcare industry is very strict and formal and the priority is to adhere to the process and have coherent documentation.
- If possible, try influencing specification of requirements. Make them concrete to allow you to make a clear decision whether the test passed.
- When the requirements are already signed and non-changeable, define explicit pass/fail criteria in your test plan. Formal test report is too late to come up with interpretation of the requirements.
- When performing formal tests, think twice whether it’s necessary to ‘save the world’ and execute multitude of scenarios.
- Make sure you have an unambiguous set of requirements and failing criteria and focus on tests that cover those requirements. Test other valuable scenarios as non-formal tests.
- Last, but not least, deliver state of the art test plan and report. Hopefully it can help you avoid a headache.