The American Society for Microbiology (ASM) appreciates the opportunity to submit comments to the Food and Drug Administration for its review of the criteria used to determine whether specific laboratory tests are waived from certain requirements of the Clinical Laboratory Improvements Amendments of 1988 (CLIA). The ASM is the largest single life science society in the world with more than 42,000 members representing a broad spectrum of subspecialties, including microbiologists who work in biomedical, clinical, public health, and industrial laboratories. The mission of ASM is to enhance the science of microbiology to better understand basic life processes and to promote the application of this knowledge for improved health and for economic and environmental well-being.
The ASM's comments regarding criteria for waived tests reflect the spirit of the law (Food and Drug Administration Modernization Act of 1997) which adheres to simplicity, accuracy, and no unreasonable risk of harm to the patient.
(1) Under FDAMA (1997), waived tests are currently defined as "laboratory examinations and procedures that have been approved by the Food and Drug Administration for home use or that, as determined by the Secretary, are simple laboratory examinations and procedures that have an insignificant risk of an erroneous result, including those that:
(A) employ methodologies that are so simple and accurate to render the likelihood of erroneous results by the user negligible, or
(B) the Secretary has determined pose no unreasonable risk of harm to the patient if performed incorrectly."
General Questions for Public Input
FDA Question 1:
What criteria should be used to demonstrate that a waived test is a simple laboratory examination and procedure with "an insignificant risk of an erroneous result?" For example:
- Should a waived test, when performed by untrained users, provide an accurate result with no significant clinical or statistical error when compared to a measure of truth? This requires availability of well-characterized reference methods and/or materials as part of the waived test assessment. The current threshold for waiver as established by CDC is no significant inaccuracy and no significant imprecision.
- Should a waived test, when performed by untrained users, provide a test result that shows no user error when compared to the same test performed in a CLIA-certified lab by a trained user? This requires comparison of the test in a lay-user setting with performance of the test in a CLIA-certified lab by a trained user. The threshold for waiver would be no difference in performance in the two settings.
- Should FDA apply a different model to determine the waived status of a test?
- The ASM supports the principle that a waived test, when performed by a lay person, should provide an accurate result with no significant clinical or statistical error when compared to a measure of truth. In addition, the test must be useful for the diagnosis or monitoring of a medical problem. Because no oversight is required for waived tests, the need for accuracy is critical in order to protect the public's health from erroneous testing. ASM also supports the availability of well-characterized reference methods and/or materials as an essential part of the waived test assessment. The statistical assessment should be based on the sensitivity, specificity, and predictive values of the test.
- ASM supports the position that a test performed in a lay-setting should be compared to the same test performed in a CLIA-certified lab. Furthermore, the clinical trials conducted in a lay setting should include a diversely representative group of prospective users of the test. A waived test, when performed by untrained users, should provide a test result that shows no user error when compared to the same test performed by a trained user in a CLIA-certified lab. Furthermore, ASM believes that the untrained user should arrive at the same results as a trained user in a CLIA-certified lab if the test is "simple and accurate and the directions are written clearly." While such test comparisons will undoubtedly increase the cost of medical devices due to the manufacturer's need to conduct additional clinical trials, ASM asserts that comparisons are a necessary part of the waived test process. ASM firmly believes that any testing done that could cause harm, should not be waived.
- A waived test should be so accurate and simple to perform that the likelihood of an erroneous result is negligible. If the FDA were to apply a different model to determine the waived status of a test, it would also have to determine the accuracy threshold to use to distinguish life threatening from non-life threatening clinical situations. Determining test thresholds for infectious diseases is difficult because of the varied effect on individuals and populations.
ASM strongly recommends that the FDA Medical Device Advisory Committee be consulted to determine the various accuracy thresholds for infectious disease tests.
FDA Question 2:
What criteria should FDA use to determine if a methodology is "so simple and accurate to render the likelihood of erroneous results by the user negligible?"
Should a waived test be so accurate when performed by untrained users that inaccurate results will not occur?
- Should a waived test have variable accuracy if used adjunctively; is it acceptable to waive tests that have inaccurate results but do not have any major negative clinical impact? How should FDA make this assessment?
- ASM recommends that waived tests should be configured such that errors in test performance or interpretation of results by untrained users occur in 1% or less of instances of use. Demonstration that this standard has been met could be part of the clinical trials data submitted by the manufacturer to FDA during the approval/classification process.
- A waived test should not have variable accuracy if used adjunctively; it is not acceptable to waive tests because they are inaccurate and do not have any major negative clinical impact. The acceptable accuracy threshold should be determined based on the clinical relevance and consequence of the test.
FDA Question # 3:
What criteria should FDA use in determining that a test will "pose no unreasonable risk of harm to the patient if performed incorrectly?"
ASM strongly believes that a test that can cause harm should, under no circumstances, be waived. FDA's Medical Device Advisory Committee should be consulted to determine the criteria that the FDA could use in determining that a test will "pose no unreasonable risk of harm to the patient if performed incorrectly."
FDA Questions #4:
Should the waiver process be different for screening tests that require a second test for confirmation? Since there are no CLIA standards for performance of waived testing, except instructions to follow the manufacturer's package insert, what is the assurance that confirmatory testing will be performed? Should the need for confirmatory testing raise, lower, or have no impact on the threshold for a waiver decision?
The waiver process should not be different for screening tests that require a second test for confirmation. Any test that requires confirmation is a 2-step process. Currently, waived tests only require a 1-step process. Because there is no assurance that a second test would be conducted to confirm the results of the first test, ASM believes that tests requiring confirmation should not be considered for a waiver decision.
Specific Questions for Public Input
FDA Question 5:
Should accuracy be determined using comparison of the waiver test to a well- characterized reference method and/or materials, to a designated comparative method and/or materials, to a working laboratory method and/or materials, to a clinical algorithm for diagnosis, and/or to other endpoints?
Accuracy should be determined using a well-characterized reference method and a clinical algorithm for diagnosis. ASM believes that clinical algorithms allow for clinical significance. The test must be evaluated on patient specimens in a "clinical trial." For example, a false negative test result for streptococcal A antigen could result in a life-threatening case of rheumatic fever or suppurative sequelae, whereas a false negative test result for vaginitis would not.
ASM encourages the FDA to consult with the FDA Medical Devices Committee to determine whether the device has been evaluated adequately.
FDA Question # 6:
How many samples, what types of samples (real or artificial) by how many users and how many sites are appropriate to evaluate accuracy? (Current guidelines being followed by FDA are for performance to be demonstrated by laboratory users at a minimum of one site.)
ASM recommends that 100 patient samples be tested at a minimum of three sites by 2 different individuals to appropriately evaluate accuracy. The test should be performed by a lay person without any prior training where it is likely to be performed, i.e. POL, and a trained laboratory person. The 100 samples should include approximately 40 positive samples of varying degrees of reactivity. More samples may be necessary to generate statistically meaningful assurance that a waived test is performing acceptably.
FDA Question # 7:
What should be the background of these users?
Waived tests should be conducted by both those who work in CLIA certified laboratories and by a cross section of individuals who are likely to perform the testing.
FDA Question # 8:
What performance criteria (statistical or clinical) should FDA apply to the accuracy threshold for a waived test (e.g., t-test or McNemar test at key decision points, description of performance with confidence intervals at key decision points, use of set performance standards using a receiver operator curve --80%, 90%, 95%, or other-- at key decision points, and/or others)?
The ASM recommends that FDA apply clinical performance criteria and a reference method as the accuracy threshold for a waived test because the majority of tests for infectious diseases provide only a positive or negative result. Waived tests should be held to the same performance criteria as all other FDA-approved tests, and the decision regarding waiver status be made independently. The latter decision should be based on an assessment of the risk of harm to the patients, should testing be performed or results interpreted incorrectly by untrained users.
FDA Question #9:
How should FDA define precision for purposes of waiver determination, what types of samples, how many and what types of operators/sites are appropriate? The current CDC recommendation is for 20 samples at three levels representing appropriate decision points to be tested at three sites by lay users using materials in either artificial and/or real matrices depending on availability and biohazard issues.
ASM recommends that FDA should define precision with a panel of 20 patient specimens at three levels representing appropriate decision points tested three times in a row at three sites by lay users.
FDA Question # 10:
What performance thresholds should FDA use to determine whether the precision studies are appropriate for waiver status (e.g., ANOVA analysis, use of a predefined performance goals such as Tonks' formula, or percent agreement out of total repeat runs)?
The appropriate performance threshold for a waived test should be determined by a statistician.
FDA Question # 11:
What interference studies are appropriate to establish performance of waived tests (e.g., effects of hemolysis, lipemia, etc.)?
The same interference studies required for approval of non-waived tests should be required for waived tests. The manufacturer should list all known interfering substances in the package insert. The package insert should be written so that a lay individual who performs the test in any setting where a waived test is performed could be able to interpret and record the results accurately.
FDA Question # 12
What environmental studies or flex (stress) studies are appropriate to establish performance of waived tests (e.g., temperature or humidity stresses, short fills)?
The same environmental studies required for approval of non-waived tests should be required for waived tests. The manufacturer should define the environmental conditions in the package insert. The test should be performed according to these instructions. The package insert should be written so that a lay individual who performs the test in any setting where a waived test is performed could be able to interpret and record the results accurately.
FDA Question # 13:
What additional studies (if any) should be submitted for evaluation of qualitative tests for waiver?
No additional studies are required.
FDA Question # 14:
What additional studies (if any) should be submitted for evaluation of quantitative tests for waiver?
ASM is against waiving quantitative tests which require two or more steps because of the test complexity. Otherwise the test must contain failsafe mechanisms which render no result when the test system malfunctions or is out of the reportable range.
In closing, the ASM recommends that FDA separate the difference between the simplicity of tests and the safety and accuracy of tests, in relation to the type of medical follow up that will be required as a result of the test. The accuracy of results can mean the difference between the life and death of patients.
ASM appreciates the opportunity to comment as FDA decides the appropriate criteria for determining whether or not certain laboratory tests can be classified as "waived." We are pleased to provide any additional information you may require as this process moves forward.