The Zika ThreatASM Acts to Counter Zika Virus Outbreak.
The American Society for Microbiology (ASM) appreciates the opportunity to submit comments to the Food and Drug Administration on its Guidance Document regarding the criteria used to determine whether specific laboratory tests are waived from certain requirements of the Clinical Laboratory Improvements Amendments of 1988 (CLIA). The ASM is the largest single life science society in the world with more than 42,000 members representing a broad spectrum of subspecialties, including microbiologists who work in biomedical, clinical, public health, and industrial laboratories. The mission of ASM is to enhance the science of microbiology to better understand basic life processes and to promote the application of this knowledge for improved health and well-being.
The ASM's comments regarding the document, Guidance for Clinical Laboratory Improvement Amendments of 1988 (CLIA) Criteria for Waiver: Draft Guidance for Industry and FDA, are consistent with the spirit of the law (Food and Drug Administration Modernization Act of 1997) which requires simplicity, accuracy, and no unreasonable risk of harm to the patient. The ASM supports the use of waived tests, if these tests are truly "simple," contain controls adequate to provide quality tests, are medically useful, and are safe and accurate.
The Guidance Document lists several characteristics describing when a test is simple. However, nearly all automated instruments and most other instruments tend to be complex and require some electronic or mechanical maintenance. Because of the complexity of instruments, they should not be waived unless designed so that the user is locked out in the event of a malfunction, absence of specimen, improper insertion of test cartridge, etc. There should be no requirement for the preparation of reagents. The ASM recommends that the FDA place its emphasis on the safety of tests and the accuracy of test results, before placing emphasis on the simplicity of tests. This could mean the difference between the life and death of patients.
Demonstrating Insignificant Risk of Erroneous Result
Quality control is relevant for test accuracy but does not address the issue of "risk of erroneous results." The risk associated with an erroneous result should relate to the medical consequences of an erroneous result, e.g., lack of treatment, selection of inappropriate treatment, incorrect diagnosis, etc. ASM also strongly believes that a test that can cause harm should, under no circumstances, be waived. FDA's Medical Device Advisory Committee should be consulted to determine the criteria that the FDA could use in determining that a test will "pose no unreasonable risk of harm to the patient if performed incorrectly."
Quality control procedures should not be optional. There is no precedence for this with any other laboratory test. Additionally, the quality control procedures should be incorporated into the test design, as has been done with many of the immunoassays such as those for group A Streptococcus.
ASM supports the availability of well-characterized reference methods and/or materials as an essential part of the waived test assessment. To ensure "insignificant risk of an erroneous result" and "no unreasonable risk of harm," controls must be a mandatory component of these test systems to provide some measure of safety. The FDA should propose mandatory "failure alert mechanisms." These would include internal controls whenever possible and lockout mechanisms if external controls are required but not performed by the user. All external controls should be part of the test system and provided by the manufacturer. Test systems that require timed steps would have a built in timing control (e.g., color develops after X minutes). Also test systems that must have a specified sample volume (e.g., glucose meters), must not provide a result when the sample volume is inadequate. The statistical assessment should be based on the sensitivity, specificity, and predictive values of the test.
Finally, tests with low sensitivity should have labeling directing the user to consider alternate testing (e.g., negative Strep A tests require culturing). Other tests for the diagnosis of diseases of public health importance require labeling directing the user to inform the patient of the consequences of a positive result (e.g., HIV, gonorrhea).
The ASM supports the principle that a waived test should provide an accurate result with no significant clinical or statistical error when compared to a measure of truth. Because no oversight is required for waived tests, the need for accuracy is critical in order to protect the public's health from erroneous testing. ASM is concerned that the Guidance Document states that if the test is not accurate, "then it will not be waived unless the Secretary determines that it poses no unreasonable risk of harm to the patient if performed incorrectly, or if the test is otherwise determined to be simple with an insignificant risk of erroneous result (page 3, step 3)." ASM believes that accuracy must always be considered in the waiver criteria; an inaccurate test should never be approved for use.
ASM disagrees with the FDA's definition of accurate, which is defined as a test which renders the same results, when performed by the hands of untrained users and the hands of laboratory professionals, when using the device under realistic conditions. ASM supports the definition of analytical accuracy, which is defined by test sensitivity, test specificity, test reproducibility, and predictive value of positive and negative tests, and supports the use of such data in the 510k or pre-analytical market process, as well as in the waiver process. For example, if a test is always negative regardless of the patient population and the skill level of the testers, then the test would be highly reproducible but have a sensitivity of zero, which is an unacceptable performance. As part of the waiver process, ASM recommends that the FDA consult with or obtain these data relating to accuracy from the FDA Medical Device Advisory Committee. The acceptable accuracy threshold should be determined based on the clinical relevance and consequence of the test. If the analytical accuracy is poor (poor being defined by the Medical Device Advisory Committee), not only should the test not be waived, the test system should not be approved.
If the FDA were to apply a different model to determine the waived status of a test, it would also have to determine the accuracy threshold to use for life threatening and non-life threatening clinical situations. Determining these thresholds for infectious diseases is difficult because of the varied effect on individuals and populations. ASM strongly recommends that the FDA Medical Device Advisory Committee be consulted to determine the various accuracy thresholds for infectious disease tests.
ASM believes that parameters for test accuracy should be defined for all groups of tests (qualitative and quantitative). However, the error rate parameters defined for qualitative tests in the Guidance Document (as high as 20%) are an unacceptable level of performance and should be reconsidered. ASM recommends that waived tests be configured such that errors in test performance or interpretation of results by untrained users occur in no more than 2-5% of all tests. Demonstration that this standard has been met could be part of the clinical trials data submitted by the manufacturer to FDA during the approval/classification process. The ASM recognizes that some waived tests already approved by the FDA have a lower sensitivity, e.g., Group A Streptococci. Consideration should be given to the reexamination of the accuracy of the data used to approve such tests. Alternatively, post surveillance data on such tests should be required.
Accuracy should be determined using a well-characterized reference method and a clinical algorithm for diagnosis. ASM believes that clinical algorithms allow for clinical significance. The test must be evaluated on patient specimens in a "clinical trial." For example, a false negative test result for streptococcal A antigen could result in a life-threatening case of rheumatic fever or suppurative sequelae, whereas a false negative test result for vaginitis would not be life-threatening in most cases.
ASM encourages the FDA to consult with the FDA Medical Devices Committee to determine whether the device has been evaluated adequately.
Demonstrating Accuracy via Studies with Untrained Users
ASM supports the position that a test performed in a lay-setting should be compared to the same test performed in a CLIA-certified lab. Furthermore, the clinical trials conducted in a lay setting should include a diversely representative group of prospective users of the test. A waived test, when performed by untrained users, should provide a test result that shows no user error when compared to the same test performed by a trained user in a CLIA-certified lab. Furthermore, ASM believes that the untrained user should arrive at the same results as a trained user in a CLIA-certified lab if the test is "simple and accurate and the directions are written clearly." While such test comparisons will undoubtedly increase the cost of medical devices due to the manufacturer's need to conduct additional clinical trials, ASM asserts that comparisons are a necessary part of the waived test process. ASM firmly believes that any testing done that could cause harm, should not be waived.
The ASM recommends that FDA apply clinical performance criteria and a reference method as the accuracy threshold for a waived test because the majority of tests for infectious diseases provide only a positive or negative result. Waived tests should be held to the same performance criteria as all other FDA-approved tests, and the decision regarding waiver status be made independently. The latter decision should be based on an assessment of the risk of harm to the patients, should testing be performed or results interpreted incorrectly by untrained users.
The same environmental and inference studies required for approval of non-waived tests should be required for waived tests. The manufacturer should define the environmental conditions and interfering substances in the package insert. The test should be performed according to these instructions. The package insert should be written so that a lay individual who performs the test in any setting where a waived test is performed could be able to interpret and record the results accurately. Product inserts should also state the sensitivity and specificity of the test.
Finally, ASM recommends that the Guidance Document address initial mandatory training by the manufacturers andsubsequent training of new personnel. Training should focus on the following: proper collection of the sample for testing, step-by-step instructions for conducting the test, quality assurance practices including the performance and interpretation of quality control results, record keeping and documentation, and actions to be taken when systems become inoperable.
ASM appreciates the opportunity to comment as FDA decides the appropriate criteria for determining whether or not certain laboratory tests can be classified as "waived" in its Guidance Document. We are pleased to provide any additional information or assistance you may require as this process moves forward.