How to perform a clinical evaluation for Medical Device Software
- January 28, 2021
- Posted by: confinis
- Categories: confinis, Regulations
Last March, the Medical Device Coordination Group endorsed this guide for establishing the appropriate level of clinical evidence required for MEDICAL DEVICE SOFTWARE (MDSW) to fulfil the requirements set out in the MDR and in the IVDR.
Software that qualifies as a MD or an IVD (to know if a SW qualifies as MD under the MDR or IVDR, you can refer to this other guidance) is subject to the same general clinical evaluation (MDR) or performance evaluation (IVDR) principles as other MDs/ IVDs.
Both for clinical evaluation (MDR) and performance evaluation (IVDR), there is a shared expectation for providing sufficient clinical evidence to demonstrate conformity with relevant General Safety and Performance Requirements (GSPRs) under the normal conditions of the device’s intended use. Clinical evidence should be sufficient and appropriate in view of the characteristics of the device, clinical risks and its intended purpose. The level of clinical evidence necessary should be specified and justified by the manufacturer.
The following process flow contextualizes three key components that should be taken into account when compiling clinical evidence for every MDSW.
- Scientific validity or valid clinical association (Stage 1). During this stage, the MDSW’s output (e.g. concept, conclusion, calculations) based on the inputs and algorithms selected is associated with the targeted physiological state or clinical condition. For example, MDSW that detects heart arrhythmia by analysing auscultation sound obtained by a digital stethoscope requires demonstrating valid clinical association between abnormal cardiac sounds and heart arrhythmia. Scientific validity or valid clinical association can be demonstrated through the use of existing clinical performance data, while taking into account the generally acknowledged state-of-the-art.
- Validation of the technical performance / analytical performance (Stage 2). These performance are the demonstration of the MDSW’s ability to accurately, reliably and precisely generate the intended output, from the input data. Identification of gaps during this stage could require generation of new evidence, for example, to demonstrate generalizability with real-life datasets or to extend the usability evaluation to omitted user groups.
- Validation of the clinical performance (Stage 3) is the demonstration of a MDSW’s ability to yield clinically relevant output in accordance with the intended purpose. In this stage, the manufacturer should demonstrate that the MDSW has been tested for the intended use(s), target population(s), use condition(s), operating- and use environment(s) and with all intended user group(s). A MDSW may have multiple features with only some features claiming a specific clinical benefit, and the clinical performance is only applicable to those features. Since MDSW can be modular in nature, this validation is also permissible on module level when the functionality of the modules is independent of the other modules. For class III and implantable devices (MDR), this validation shall include data from a clinical investigation unless the conditions of Article 61(4), (5) or (6) of the MDR have been fulfilled. For MDSW falling under the IVDR, the evaluation of clinical performance requires the carrying out of clinical performance studies regardless of the classification of the device, unless due justification is provided for relying on other sources of clinical performance data. For MDSW not claiming clinical benefits that can be specified through measurable, patient-relevant clinical outcome(s), clinically relevant outputs are achieved through demonstrated predictable and reliable use and usability. To determine and justify the level of clinical evidence, both amount and quality of supporting data should be evaluated.
In the following table, we report some practical examples on the application of this flow.
| MDSW | Intended use and claims | Manufacturer’s claims | Scientific validity | Technical performance | Clinical performance |
| MDSW intended to analyse sleep quality data. | The independent MDSW takes into account accelerometer and microphone data to determine quality of sleep and to estimate the expected success rate of CPAP (continuous positive airway pressure) treatment for sleep apnoea.
| – Determines the quality of sleep that impacts the general well-being. – Monitors quality of sleep in patients with sleep disorders such as sleep apnoea (using phone sensors/wearable devices). – Estimates the expected success rate of CPAP therapy. | Review literature. – Objective quality of sleep is measured by sleep duration, efficiency and fragmentation. It is well-established that quality of sleep impacts general well-being such as concentration, risk-factors for cardiovascular disease, mood, cognitive abilities, etc. – It is not well-established that the success of CPAP therapy can be predicted by monitoring the quality of sleep. – Address the association of accelerometer and microphone data to established quality of sleep parameters (e.g. sleep duration, efficiency and fragmentation. | Confirm with verification and validation tests that the app can reliably and reproducibly calculate sleep quality scoring. Confirm compatibility between the MDSW and the device equipped with the sensors to ensure data can be utilized in the intended way. | In addition to the usability assessment, the manufacturer should perform a retrospective study on previously obtained data to confirm that success of CPAP therapy can be predicted based on the quality of sleep. |
| MDSW intended for image segmentation. | The independent MDSW allows automatic detection of organs and anatomical structures in CT scans with the accuracy of a radiologist.
| – Detects abdominal aortic aneurisms on abdominal CT scans. – Detects compression fractures on vertebrae. – Detects liver cysts. | Review literature. – The normal shape and size of anatomy is well established. -Segmentation techniques on cross-sectional images correlates well with the actual size and shape. The VALID CLINICAL ASSOCIATION has been established without gaps identified. | Confirm with verification and validation tests the basic technical performance such as display, modification, window levelling of images, measurements including confirmation of accuracy, sensitivity and reliability of the MDSW as per the expected performance. | Usability assessment including the intended user groups in conjunction with the scientific validity and validation of technical performance results has been determined as sufficient to demonstrate conformity with relevant GSPRs. – In cases where data is available, a retrospective analysis can be performed. In cases where data does not represent the variability of input parameters, the missing data could be generated in a prospective clinical investigation. |
| MDSW intended to detect inflammatory bowel diseases (IBD). | Self-testing independent MDSW intended for the semi-quantitative detection of calprotectin from a faecal sample. Reagents are added to the sample resulting in a colour change. The sample is then photographed on a smartphone, and the image is evaluated by an MDSW application (app) running on the phone. The MDSW app detects the colour change in the sample and interprets the concentration of calprotectin. The test is intended as an aid in monitoring and staging of patients with inflammatory bowel disease (IBD). | – Aids in monitoring and staging the disease level of patients with inflammatory bowel diseases (IBD). – Aids in differentiation between IBD and functional bowel disorders. – Helps patients avoid unnecessary clinical visits. | Review literature. The focus is on how the calprotectin level corresponds to the IBD level and stages. Furthermore, it should address, whether calprotectin levels are suitable to differentiate between IBD and functional bowel disorders. – It is well-established that calprotectin concentration in faecal matter can be reliably measured in test strips by change of colour. – The colour intensity is directly representative of the concentration of calprotectin. | Confirm the MDSW app can detect reliably and accurately the colour of the test strip compared to human observation, taking into account environmental factors. | The manufacturer should assess the initial performance and feasibility by creating clinical performance metrics, taking into account sensitivity, specificity and confidence intervals. – Any claims regarding clinical benefit should be supported by sufficient clinical performance data. – Usability should be confirmed by the manufacturer. |
If the MDSW is used for the determination of a patient’s future state (e.g. predisposition, prognosis, prediction) or if the output of the MDSW impacts clinical outcomes (e.g. treatment efficacy) or patient management decisions, then a prospective study may be required. In other situations, retrospective analysis may be more appropriate to generate the necessary data to support compliance with the GSPRs.
The manufacturer should compile evidence, perform the benefit-risk analysis and document the clinical or performance evaluation and its output in the clinical evaluation (MDR) / performance evaluation /IVDR) report.
Last but not least, safety, effectiveness and performance of the MDSW should be actively and continuously monitored by the manufacturer for example with post-market information such as complaints, PMCF/ PMPF data, real-world performance data, direct end-user feedback or newly published research / guidelines.
In summary, software that qualifies as a MD or an IVD is subject to the same general clinical evaluation (MDR) or performance evaluation (IVDR) principles as other MDs/ IVDs. Three key aspects are very important for providing sufficient clinical evidence to demonstrate conformity with relevant general safety and performance requirements: scientific validity or valid clinical association, validation of the technical performance / analytical performance, and validation of the clinical performance. We reported in this blog the process flow that contextualize these aspects and a table with some practical examples on how the clinical evidence can be proved on different kinds of MDSW.
We provided a quick overview on the topic, and we recommend you refer to the MDR, the IVDR, and to the MDCG 2020-1 if you need to establish clinical evidence for MDSW.
If you have questions or need support, please contact us at info.ch@confinis.com.
If you liked this blog and you are interested in how the notifies bodies assess the clinical evaluation, stay tuned for our next post in two weeks!
Authors: Dr. Martina Coscia, Dr. Andrea Biasiucci
Credits: the picture for this blog series has been inspired from a poster by R. Pradhan, S. Shrestha and U. Satyal entitled “Development of a digital tool for risk management, clinical evaluation and post-market surveillance of medical devices” for the 2020EuroConvergence conference.