2. Please indicate the size of your group:
____ Small ____ Medium __X__ Large

3. What did you gain from the Temporary Certification Program that you did not expect?

4. What part(s) of the Temporary Certification Program worked well and that you would not want to see changed?
Proctor was extremely helpful
5. What 3 Temporary Certification Program testing and certification process points were the confusing / most misunderstood? For these points, what could be improved and how?
(1) Data necessary for generating the CCD, it is sometimes difficult to backdate data.
(2) Pubic Health Surveillance, requirements were very vague
(3)

6. What 3 meaningful use measures and their corresponding certification criteria and test procedures seemed not to be in alignment and caused confusion for you?
(1) Smoking Status and recodes
(2) Content of the Clinical Summaries CCD C32 conflict with CMS definition of their measures
(3) Public Health Surveillance

7. Are the certification criteria clear?
_X __ Yes __X__ No
If no, which criteria and how it can be improved?
Exchange of Health information, what method of exchange should be used.
8. Was the level of specificity appropriate?
__X__ Yes ____ No
If no, how it can be improved?

9. Should certain certification criteria be combined or decomposed differently?
_X_ Yes ____ No
If yes, which criteria and why?
We actually combined and reordered some of the test steps as part of theDuring our certification. process we combined

10. Should certain certification criteria be scoped differently?
____ Yes ____ No
If yes, which criteria and why?

11. Please comment on the balance of process-oriented vs. outcome-oriented certification criteria.

12. Do you have any suggestions for improving the ONC-approved test procedures for EHR certification?

13. Would it be beneficial if more choreographed/combined test procedures were developed that permitted an EHR developer to satisfy multiple certification criteria at once?
____ Yes __X__ No

14. In what ways can the test procedures be combined to facilitate the testing of certification criteria within the context of clinical workflow?

15. Please provide specific instances where the ONC-approved test method can be improved to reduce ambiguity, inconsistency with criteria, and addition of implicit requirements beyond the certification criteria.
Patient List by Conditions, in the original test script from NIST
16. How would you rate your certification experience on a scale from 1 to 6 (circle one)
1 2 3 4 5 6
Difficult Easy

17. Other Comments or Suggestions

Like or Dislike: 2  0 (+2)

Intuit Health is the largest independent patient portal provider in the country measured by either patients (millions) or providers (tens of thousands) using our system. Our solution enables patients to communicate and transact electronically online with their medical practice— and get lab results, pay bills, refill medications refills, securely message their provider or their staff, and a multitude of other “front and back office” services like appointment reminders and bill pay. We have over 40 partnerships with electronic health records (EHRs) and practice management systems and offer standards based integration.

Since your goal is to obtain feedback on the process for certifying EHR technology, we would like to offer two viewpoints – that of the vendor and that of the provider (our customers). Our comments are limited to the certification process and are not directed at any of the specific questions.

Vendor Viewpoint

Intuit Health is certified under the “timely access” component of meaningful use as an “EHR module”. We have a number of provider customers who are purchasing a new ONC-“certified Complete” EHR which uses a different Patient Portal (to meet the Timely Access criteria). These customers do not want to use the portal built into the new EHR and are asking us if they can use our certified product for Timely Access and still be compliant (using a certified technology). Our customers have chosen our product and want to continue with it as it is ‘best of breed’. The problem is that under the current certification methodology these providers must buy “two portals” – the one that comes with the complete certification EHR and the other that they want (the Intuit Health Portal).

When we and others from the industry approached the Office of the National Coordinator (ONC) about this issue, ONC responded with two FAQs, that essentially say providers can either buy the full “solution” – and then also buy our portal (FAQ 14) or they can “not pay for Portal A in their contract” to avoid “buying two portals” (see FAQ 21 – specifically Scenario 1).

Unfortunately these FAQs do not provide any practical relief. Providers are not going to buy two products, and negotiating to not pay for a solution “contractually” is awkward. More importantly, it creates a threshold of complexity that unintentionally promotes “completely certified solutions” over “modular solutions” – or said another way, favors large EHR vendors and prejudices single solution vendors.

Additionally, this aspect of certification seems to be at odds with the expressed goals of interoperability, supporting best of breed technologies, and to move away from “siloed” solutions.

Proposal: ONC should work to ensure that vendors certify their product modularly. In this way, purchasers could assemble the HIT solution that best meets their needs, without being coerced into purchasing a complete EHR. This would improve market adoption, provider understanding and stimulate innovation in product development and increased competition – producing “best product” for the provider.

Provider Viewpoint

Prior to the EHR incentive program, most physicians and hospitals gradually developed their HIT capacity, building their data systems one piece at a time, selecting the product that best met their needs regardless of vendor. So, for example, a physician could select a patient-provider portal from us, later the physician could add patient specific education materials from Vendor B and an electronic prescribing system from Vendor C. Quickly developing their HIT infrastructure is not something that providers are used to doing. In fact, a measured and gradual building block approach was the preferred method for acquiring HIT technology and integrating the technology into clinician and administrator workflow.

Due to the ease of selecting a “completely certified solution,” as outlined earlier, there is an unfair and we think, an unintentional, market bias against modular providers that has the effect of encouraging inferior product and reducing interoperability.

Proposal: ONC should address the imbalance by requiring modular certification by EHR vendors seeking complete certification and by helping providers understand and embrace modularity as a best of breed approach. Failure to correct this marketplace distortion could negatively impact the quality of care delivered as well as the quality and ease of patient/provider and provider/provider communication.

If you have questions, or need more information, please contact me at (919) 882-2863 or Warwick_Charlton@intuit.com or Melissa Netram, in our DC office at (202) 484-2147 or Melissa_Netram@intuit.com.

Sincerely,

Warwick
Warwick Charlton, MD
Chief Medical Officer

Like or Dislike: 1  0 (+1)

Key Comments
• Temporary Certification is an interval measure. There is some risk that purchasers who acquire systems under that Temporary status, will not get a fully-certified product ,since a vendor can elect to discontinue prior to permanent certification.
• The Office of the National Coordinator (ONC) should look to the AAP for the vision and the end-product that pediatricians want and need. It will be far too expensive for pediatricians to afford and maintain the short-term, one-step-at-a-time approach.
• The role of the ONC should be only to ensure that certifications and standards are coordinated and the role of the ONC-ATBs should be only to measure adherence to certification.

Like or Dislike: 2  0 (+2)

Question 5. What 3 Temporary Certification Program testing and certification process points were confusing / most misunderstood? For these points, what could be improved and how?

(1) Modular EHR Certification of parts of a product following Complete EHR Certification. Some level of retesting should always be required to ensure continued compliance; however, there appeared to be inconsistency in practice among ATCB’s. Early policy clarification is needed. All decisions should be documented and ATCB training sessions conducted. Leaving this decision up to individual ATCB’s may result in increased risk for the providers purchasing these products to support meaningful use.
(2) Security Criterion Exemptions. Lack of clarity and failure to address ATCB questions in a timely manner appeared to result in inconsistencies in the granting of Security Criterion Exemptions among ATCB’s for like products. Policies or FAQs for specific situations should be developed in a timely manner and all ATCB’s trained. There may be inconsistency of security exemptions among like products on the current ONC CHPL which may result in increased risk for the providers purchasing these products to support meaningful use.
(3) ONC CHPL reporting rules. Requiring multiple listings on the CHPL for the same EHR product as a vendor returns to have additional criteria incrementally tested and added (toward achievement of a Complete EHR) is counterintuitive for providers selecting these products. Providers are unlikely to be purchasing or installing the partial EHR in these situations. Maintaining all previous listings for a product is not necessary in all situations. This is a source of confusion and dissatisfaction for both providers and vendors. Updating product certification line items with added functionality as they are achieved when pursuing Complete EHR certification would simplify the CHPL. Version changes, Private Labels and Modular certifications following a Complete certification should be a new line item.
In addition, there doesn’t appear to be a documented approach to modifying a certification once listed (e.g., if a vendor’s subsequent modification of the product failed to support the criteria, the vendor violated ONC marketing practices rules, or an ONC-ATCB’s interpretation of the testing procedures was deemed to be inadequate.)This, too, could result in increased risk for providers purchasing products.

Question 6. What 3 meaningful use measures and their corresponding certification criteria and test procedures seemed not to be in alignment and caused confusion for you?

(1) 170.302(n) Automated measure calculation. The Final Rule requires that at least the denominator be auto-calculated by the EHR. The ONC approved NIST Test Procedure allows manual creation and entry of numerator and denominator, as clarified by NIST. Also, CMS has issued an FAQ (Published 02/17/2011 04:42 PM | Updated 02/18/2011 12:35 PM | Answer ID 10465) allowing providers to enter numerators and denominators directly into the CMS incentive application form bypassing the requirement for providers to use certified technology for the automated measure calculation.
(2) 170.306(i) and 170.304(j) Calculate and submit clinical quality measures. There is confusion by vendors and providers regarding how to meet the requirements. There is no clarification provided in the ONC approved Test Procedure. For Example: NQF 0421 and PQRI 128 are listed together as one report in the quality measures. Both of them have the same title. The NQF measure has three population criteria each with its own numerator and denominator. The PQRI measure has only one numerator and denominator. The CPT and ICD-9 codes listed in the NQF specification do not match with the PQRI specification. This is just one example. There are many other core, alternate core and additional measures listed in the Final Rule which have an NQF and PQRI measure listed together with the same issue.
(3) 170.302(l) Public health surveillance. There were errors in the Final Rule pointing to an incorrect implementation guide for one of the submission standards. ATCB’s were instructed to allow and certify products demonstrating compliance with the incorrect implementation guide until the correction to the Final Rule was published. There are certified products on the CHPL that demonstrated compliance with the incorrect implementation guide which could pose significant issues for providers using those systems. ATCB’s were instructed that no retesting was required.

Question 7. Are the certification criteria clear?
____ Yes __X_ No
If no, which criteria and how it can be improved?

§170.302(a) Drug-drug, drug-allergy interaction checks
(1) Notifications. Automatically and electronically generate and indicate in real-time, notifications at the point of care for drug-drug and drug-allergy contraindications based on medication list, medication allergy list, and computerized provider order entry (CPOE).
(2) Adjustments. Provide certain users with the ability to adjust notifications provided for drug-drug and drug-allergy interaction checks.

Suggested improvement=define “adjust”. Also, users should not have the option to turn off drug-allergy interaction checks for patient safety reasons.

§170.302(g) Smoking status
Enable a user to electronically record, modify, and retrieve the smoking status of a patient. Smoking status types must include: current every day smoker; current some day smoker; former smoker; never smoker; smoker, current status unknown; unknown if ever smoked.

Suggested improvement=if the CDC Recodes (Numbers) are required, include them in the criteria.

§170.302(h) Incorporate laboratory test results
(1) Receive results. Electronically receive clinical laboratory test results in a structured format and display such results in human readable format.
(2) Display test report information. Electronically display all the information for a test report specified at 42 CFR 493.1291(c), (1) through (7).
(3) Incorporate results. Electronically attribute, associate, or link a laboratory test result to a laboratory order or patient record.

Suggested improvement=clarify for provider sites with their own internal laboratory. Also, the structured format should be defined.

§170.302 (m) Patient-specific education resources.
Enable a user to electronically identify and provide patient-specific education resources according to, at a minimum, the data elements included in the patient’s: problem list; medication list; and laboratory test results as well as provide such resources to the patient.

Suggested improvement=clarify if resources must be “automatically” determined by the system based on clinical data in the record.

§170.302(n) Automate measure calculation
For each meaningful use objective with a percentage-based measure, electronically record the numerator and denominator and generate a report including the numerator, denominator, and resulting percentage associated with each applicable meaningful use measure.

Suggested improvement=require the system to automatically calculate at least the denominator. This requirement could currently be met by using an Excel spreadsheet.

§170.302 (p) Emergency Access
Permit authorized users (who are authorized for emergency situations) to access electronic health information during an emergency.

Suggested improvement=this criterion should reflect “break the glass” functionality. The current criterion is unclear.

§170.302 (q) Automatic Log-Off
Terminate an electronic session after a predetermined time of inactivity.

Suggested improvement=please define “terminate”.

§170.304 (b) Electronic prescribing
Enable a user to electronically generate and transmit prescriptions and prescription-related information in accordance with:
(1) The standard specified in §170.205(b)(1) or §170.205(b)(2); and
(2) The standard specified in §170.207(d)

Suggested improvement: Please define the requirements for the demonstration of electronically “transmit”.

§170.304 (g) Timely Access
Enable a user to provide patients with online access to their clinical information, including, at a minimum, lab test results, problem list, medication list, and medication allergy list.

Suggested improvement: Please define “online access”.

§170.304(h) Clinical summaries
Enable a user to provide clinical summaries to patients for each office visit that include, at a minimum, diagnostic test results, problem list, medication list, and medication allergy list. If the clinical summary is provided electronically it must be:
(1) Provided in human readable format; and
(2) Provided on electronic media or through some other electronic means in accordance with:
(i) The standard (and applicable implementation specifications) specified in §170.205(a)(1) or §170.205(a)(2); and
(ii) For the following data elements the applicable standard must be used:
(A) Problems. The standard specified in §170.207(a)(1) or, at a minimum, the version of the standard specified in §170.207(a)(2);
(B) Laboratory test results. At a minimum, the version of the standard specified in §170.207(c); and
(C) Medications. The standard specified in §170.207(d).

Suggested improvement: Clarify that clinical summaries for a specific office visit may include data carried over from previous visits, for example, allergies entered at a previous visit should still display on the summary.

§170.304 (i) Exchange Clinical Information and Patient Summary Record
(1) Electronically receive and display. Electronically receive and display a patient’s summary record, from other providers and organizations including, at a minimum, diagnostic tests results, problem list, medication list, and medication allergy list in accordance with the standard (and applicable implementation specifications) specified in §170.205(a)(1) or §170.205(a)(2). Upon receipt of a patient summary record formatted in the alternative standard, display it in human readable format.
(2) Electronically transmit. Enable a user to electronically transmit a patient summary record to other providers and organizations including, at a minimum, diagnostic test results, problem list, medication list, and medication allergy list in accordance with:
(i) The standard (and applicable implementation specifications) specified in §170.205(a)(1) or §170.205(a)(2); and
(ii) For the following data elements the applicable standard must be used:
(A) Problems. The standard specified in §170.207(a)(1) or, at a minimum, the version of the standard specified in §170.207(a)(2);
(B) Laboratory test results. At a minimum, the version of the standard specified in §170.207(c); and
(C) Medications. The standard specified in §170.207(d).

Suggested improvement: Clarify “Upon receipt of a patient summary record formatted in the alternative standard, display it in human readable format.” to indicate that BOTH the CCD and the CCR must be received and displayed in the EHR in human readable format.

§170.306 (f) Exchange clinical information and patient summary record

Suggested improvement: Same as 170.304(i) above.

Question 8. Was the level of specificity appropriate?
____ Yes __X_ No
If no, how it can be improved?

See suggestions in #7 above.

Question 9. Should certain certification criteria be combined or decomposed differently?
__X_ Yes ____ No
If yes, which criteria and why?

Duplication exists in many of the approved test methods. The following groups of criteria could be combined and tested together easily:

Group #1
§170.302 (o) Access Control
Assign a unique name and/or number for identifying and tracking user identity and establish controls that permit only authorized users to access electronic health information.
And
§170.302 (t) Authentication
Verify that a person or entity seeking access to electronic health information is the one claimed and is authorized to access such information.

Group #2
§170.302 (u) General encryption
Encrypt and decrypt electronic health information in accordance with the standard specified in §170.210(a)(1), unless the Secretary determines that the use of such algorithm would pose a significant security risk for Certified EHR Technology.
And
§170.302 (v) Encryption When Exchanging Electronic Health Information
Encrypt and decrypt electronic health information when exchanged in accordance with the standard specified in §170.210(a)(2).

Group #3
§170.302 (d) Maintain active medication list
Enable a user to electronically record, modify, and retrieve a patient’s active medication list as well as medication history for longitudinal care.
And
§170.302 (b) Drug-formulary checks
Enable a user to electronically check if drugs are in a formulary or preferred drug list.
And
§170.304 (b) Electronic prescribing
Enable a user to electronically generate and transmit prescriptions and prescription-related information in accordance with:
(1) The standard specified in §170.205(b)(1) or §170.205(b)(2); and
(2) The standard specified in §170.207(d)

Group #4
§170.302 (i) Generate patient lists.
Enable a user to electronically select, sort, retrieve, and generate lists of patients according to, at a minimum, the data elements included in:
(1) Problem list;
(2) Medication list;
(3) Demographics; and
(4) Laboratory test results
And
§170.304 (d) Patient Reminders
Enable a user to electronically generate a patient reminder list for preventive or follow-up care according to patient preferences based on, at a minimum, the data elements included in:
• Problem list;
• Medication list;
• Medication allergy list;
• Demographics; and
• Laboratory test results

Group #5
§170.304 (f) Electronic copy of health information
Enable a user to create an electronic copy of a patient’s clinical information, including, at a minimum, diagnostic test results, problem list, medication list, and medication allergy list in:
(1) Human readable format; and
(2) On electronic media or through some other electronic means in accordance with:
(i) The standard (and applicable implementation specifications) specified in §170.205(a)(1) or §170.205(a)(2); and
(ii) For the following data elements the applicable standard must be used:
(A) Problems. The standard specified in §170.207(a)(1) or, at a minimum, the version of the standard specified in §170.207(a)(2);
(B) Laboratory test results. At a minimum, the version of the standard specified in §170.207(c); and
(C) Medications. The standard specified in §170.207(d).
And
§170.304(h) Clinical summaries
Enable a user to provide clinical summaries to patients for each office visit that include, at a minimum, diagnostic test results, problem list, medication list, and medication allergy list. If the clinical summary is provided electronically it must be:
(1) Provided in human readable format; and
(2) Provided on electronic media or through some other electronic means in accordance with:
(i) The standard (and applicable implementation specifications) specified in §170.205(a)(1) or §170.205(a)(2); and
(ii) For the following data elements the applicable standard must be used:
(A) Problems. The standard specified in §170.207(a)(1) or, at a minimum, the version of the standard specified in §170.207(a)(2);
(B) Laboratory test results. At a minimum, the version of the standard specified in §170.207(c); and
(C) Medications. The standard specified in §170.207(d).
And
§170.304 (i) Exchange Clinical Information and Patient Summary Record
(1) Electronically receive and display. Electronically receive and display a patient’s summary record, from other providers and organizations including, at a minimum, diagnostic tests results, problem list, medication list, and medication allergy list in accordance with the standard (and applicable implementation specifications) specified in §170.205(a)(1) or §170.205(a)(2). Upon receipt of a patient summary record formatted in the alternative standard, display it in human readable format.
(2) Electronically transmit. Enable a user to electronically transmit a patient summary record to other providers and organizations including, at a minimum, diagnostic test results, problem list, medication list, and medication allergy list in accordance with:
(i) The standard (and applicable implementation specifications) specified in §170.205(a)(1) or §170.205(a)(2); and
(ii) For the following data elements the applicable standard must be used:
(A) Problems. The standard specified in §170.207(a)(1) or, at a minimum, the version of the standard specified in §170.207(a)(2);
(B) Laboratory test results. At a minimum, the version of the standard specified in §170.207(c); and
(C) Medications. The standard specified in §170.207(d).

Question 10. Should certain certification criteria be scoped differently?
_X__ Yes ____ No
If yes, which criteria and why?

(1) §170.302 (p) Emergency Access—match to Final Rule and test both “Break the Glass” for a patient medical emergency and also for situations like natural disaster emergencies.
(2) §170.302(a) Drug-drug, drug-allergy interaction checks—the appropriateness of disabling drug-allergy interactions should be reviewed and removed from the scope of this criterion for patient safety reasons.
(3) §170.302 (j) Medication Reconciliation—the current criterion and test method do not test true medication reconciliation.

Question 12. Do you have any suggestions for improving the ONC-approved test procedures for EHR certification?
Yes.

(1) A pilot testing process for any test procedures should be conducted with a testing lab, or labs, and volunteer EHR vendor(s) to identify issues and fully vet the documented process before acceptance by ONC.
(2) All interoperability testing should have ONC approved testing tools to validate the correct format and structure of files. In addition, ONC should develop a testing harness for Testing Labs to use (for consistency) to allow EHR vendors to demonstrate compliance with file transmission requirements.
(3) Ensure the accuracy of all clinical data in any required or example data sets prior to release. Additionally, ensure that the test scenarios reflect accurate and commonly recognized clinical workflows.
(4) An easy to understand, step by step Test Script (clinical workflow steps) following the requirements in the approved test procedures and published by ONC would improve consistency among Testing Bodies. See item a. above for pilot testing recommendation.
(5) Freeze testing requirements once test procedures and any test scripts are approved and published by ONC. This creates an even playing field for all EHR vendors for testing and prevents a “moving target” for development efforts. It also ensures all certified products have met exactly the same standard.
(6) Recognize that virtual software testing requires the applicant to perform data entry and carry out the functions in the test procedure during the inspection of the product for efficiency and accuracy. Requiring the Tester to perform these functions is not efficient or productive.
(7) Eliminate duplicative test procedure requirements and create combined testing for multiple criteria in a clinically accurate demonstration of the workflow in patient care.

Question 13. Would it be beneficial if more choreographed/combined test procedures were developed that permitted an EHR developer to satisfy multiple certification criteria at once?
__X_ Yes ____ No

Question 14. In what ways can the test procedures be combined to facilitate the testing of certification criteria within the context of clinical workflow?

See comments in item #9 above.

Question 15. Please provide specific instances where the ONC-approved test method can be improved to reduce ambiguity, inconsistency with criteria, and addition of implicit requirements beyond the certification criteria.

(1) 170.302(p) Emergency access. The criterion statement is confusing and the test method does not reflect an accurate test of “break the glass” functionality in medical emergencies. The intent of the test procedure seems to be to demonstrate the ability for users (not administrators) to override security controls and obtain access to clinical data which is not a secure practice. The test procedure does not address any other emergency situations (such as a natural disaster for example) where emergency access to data may be required during a system outage as mentioned in the Final Rule.
(2) 170.304(b) ePrescribing. The test method requires clarification of allowable vocabularies. There is contradictory information between the Final Rule and the approved test method. At one point ONC advised that a specific vocabulary was allowable, and, after clarification from NIST, it was determined that the vocabulary was NOT allowable. This created a situation wherein the CHPL could include certified products tested against an inappropriate vocabulary.
(3) 170.302(s) Integrity. The demonstration of the actual message digest (hash value) as required in the test method is highly unusual. Vendors who have compliant processes and use the required standards have been forced to purchase add on software or develop their own programs just to display these values to the Tester during the inspection. Other methods of demonstration or self attestation may be more appropriate for this criterion. See 170.302(u) below.
(4) 170.302(u) General encryption. The test method requires a Tester to view an encrypted message and determine it is NOT “human readable” as proof of compliance. Likewise, the test method then requires the Tester to view a decrypted message and determine it IS “human readable” as proof of compliance. This is equivalent to not trusting the HTTPS in the browser and requiring browsers to display the actual encryption taking place. The test methods could be revised to enable attestation of the use of FIPS compliant encryption for Stage 1. For Stage 2 there should be specificity in transport standards so that the test can be accomplished by submitting data to an ONC specified website that illustrates adherence to the transport standard (NHIN Direct, NHIN Connect etc.).
(5) 170.302(v) Encryption when exchanging electronic health information. The test method is not explicit in the requirement to send the encrypted message and to demonstrate in the receiving system that the message can only be successfully decrypted using the key created by the originating system.
(6) 170.306(g) Reportable lab results. One data set in the Reportable Lab test procedure requires that you send public health entities information about an infection the patient does NOT have (Stool Culture with a negative result for Shigella). This seems to contradict the intent of the criteria.
(7) For all Criteria that require entry of laboratory test data, clarity is required regarding the methods of data entry that are acceptable. Manual data entry should be one method, but to match clinical workflows, other methods of incorporation of laboratory results into the EHR should also be mentioned as allowable, for example incorporation of a file via interface.
(8) For all Criteria that require entry of properly coded problems and procedures a similar situation exists with regard to the method of recording the codes using the ONC-specified vocabulary standards. Specialized coding systems may be used to establish the correct coded values for these items, particularly for billing purposes. Additional clarification is required regarding whether a coding system-to-EHR interface is an acceptable method for positioning the test data in the EHR for these particular tests.
(9) 170.302(a) Drug/drug and drug/allergy interactions. Requiring functionality that allows certain authorized system users to be able to “adjust notifications provided for drug-drug and drug-allergy checks (e.g., set the level of severity for which notifications are presented).” is contrary to patient safety for drug/allergy interactions.
(10) 170.302(f.3) Plot and display growth charts. The test method is not clear whether to use one male and one female patient with 3 recorded height/weights at 3 different age intervals, OR, 3 male and 3 female patients each with one set of height weight measurements. At one point in the test procedure it speaks to “6 patients”, however, the intent seems to be to demonstrate graphing which would require more than one data point per patient. We requested clarification from NIST several months ago but have not received a response at this point.
(11) 170.302(g) Smoking status. The published test methods require the system to demonstrate the mapping of the textual smoking values in the criterion to the actual numerical CDC Smoking Recodes. For example: 1=Current every day smoker, 2=Current some day smoker, etc. The Final Rule requires only the textual values.
(12) 170.302(i) Generate patient lists. The published test methods require the ability to sort by elements of data used to generate the reports. This is not realistic when you generate a report to show all patients on Coumadin for example, to sort by medications (since all patients will be on the same medication). From the approved test procedure:
a.The Tester retrieves and displays two or more lists of patients using data elements included in the Problem list, Medication list, Demographics, and Laboratory test results
b.The Tester sequences or arranges the data on each patient list using these data elements
(13) 170.302(k) Submission to immunization registries. The CDC has published updated CVX codes for some of the immunizations used in the required test data in the test procedure. ONC needs to approve the use of the updated CVX codes for testing.
(14) 170.302(n) Automate measure calculation. As mentioned earlier, the approved test procedure (and subsequent clarification from NIST) allows for the manual entry of both the numerator and denominator. The Final Rule requires that at least the denominator be auto-calculated by the EHR.
(15) 170.302(l) Public health surveillance. There was a mistake in the original Standards and Certification Final Rule that specified an implementation guide for Disease Reporting, not Syndromic Surveillance. A revision to the Final Rule removed the original implementation guide but did not specify a new one; hence, there is no implementation guide to test against.
(16) 170.302(m) Patient specific education resources. It is unclear whether the education resources should be automatically produced by the EHR based on clinical documentation entered into the system or whether this allows a provider to manually select the appropriate education resources upon review of the clinical information in the record.
(17) 170.302(q) Automatic log off. Additional clarity is needed regarding the meaning of “terminate an electronic session” in the criterion and the test method. Some vendors have interpreted this to mean a system lock with the screen (with patient information) still visible, and then when further access is sought it requires re-authentication into the EHR or takes the user back to the login screen. Clarity regarding what is allowable and what is not would create better consistency between ATCB’s.
(18) 170.302(w) Accounting of Disclosures (optional). Optional items should not be included in the certification criterion or test procedures.
(19) 170.306(f) Exchange Clinical Information and Patient Summary Record. The HITSP C83 required problem lists to be coded in SNOMED-CT, but the Final Rule allows ICD-9-CM and SNOMED-CT. This is confusing since the Final Rule references C83 as the standard.
(20) 170.306(i) and 170.304(j) Calculate and submit clinical quality measures. Clarification is required for measures which have multiple specifications. For Example:
a. NQF 0421 and PQRI 128 are listed together as one report in the quality measures. Both of them have the same title. The NQF measure has three population criteria each with its own numerator and denominator. The PQRI measure has only one numerator and denominator. The CPT and ICD-9 codes listed in the NQF specification do not match with the PQRI specification. This is one example. There are other core, alternate core and additional measures listed in the Final Rule which have an NQF and PQRI measure listed together and many of them have this same issue. Finally, the PQRI XML specification is ambiguous. The ED measures only require 2 data elements, yet some validators require 6 data elements, with the last 4 set to zero.

Question 17. Other Comments or Suggestions

ONC-ATCBs experienced several challenges in the early days of the program due to an implementation timeline that did not accommodate piloting of certification criteria and testing procedures. EHR companies without experience in certifying their products were unclear about how to interpret the ONC standards and criteria and the test scripts created by NIST. This confusion was compounded by errors in the final rule with regard to the public health surveillance standard and the evolving nature of the NIST test procedures. The latter created concern for ATCBs and EHR companies who may have already begun a certification process using an earlier procedure. While ONC and NIST have understood and tried to mitigate the impact of these changes, ONC-ATCBs are still required to manage two test processes at once for EHR companies who may be in different stages of certification.
The focus on modules that can uniquely meet the requirements of individual criteria has been especially problematic when certifying hospital developed or modified EHR technology. Hospitals often choose a “best of breed” approach to creating an EHR system that doesn’t always align with the way the current ONC criteria are structured. That is, two or more products may straddle one ONC criterion. While this modular approach allows maximum flexibility, may stimulate vendor innovation, and may result in a higher number of vendor certified EHR technologies, it may not be efficient for hospital developers who need to certify previously installed EHR technology. More guidance should be given to ATCBs in this area.

Like or Dislike: 1  1 (0)

4. What part(s) of the Temporary Certification Program worked well and that you would not want to see changed?
It was good to have a choice of certification bodies, but also the assurance that we could use one consistent set of NIST test procedures regardless of which ATCB we certified with. That brought more assurance of a consistent baseline (even though there may have been variations in some scripts that the ATCBs offered), we knew we could use the NIST procedures if we chose.

5. What 3 Temporary Certification Program testing and certification process points were the confusing / most misunderstood? For these points, what could be improved and how?
The EHRA presentation to the ONC titled “Meaningful Use and Certification Challenges and Opportunities” dated March 28, 2011 outlines a number of issues and associated recommendations we suggest the committee should review as well. We particularly would highlight the following.
(1) The restrictive definition of what counts as a certified module (disallowance of a module that has been certified as part of a complete EHR to be counted). This forces an unwieldy and expensive need to certify many modular combinations, because real world implementations often do NOT use just one complete EHR. Yet it is also impractical to certify every individual module standalone, because of the burdensome and not always applicable security requirements that would have to be applied to each module, whereas they are normally applied at a system-wide level.
(2) Certification overlaps logical product boundaries. When a certification criteria spans a logical product boundary it creates a scenario where customers may need to self certify modules. Multiple products that either span certification criteria or that only covers part of a criterion must be certified together. When a third party product makes up only part of a module, it cannot be certified alone. This results in a provider having to self certify the unique combination.
(3) The requirement to “possess” certified EHR modules for all MU objectives, even for those that are being deferred. This adds unnecessary cost and burden to EH and EP. It either forces vendors to create many combinations or modules into bundles (and pay extra to certify these bundles), or forces EH and EP into “self certification” at significant cost (the labor cost far exceeding the certification fees).

To address these problems, we recommend:
• Consider ways to leverage the concept of “base” in the certification process. Customers must possess the base.
• There needs to be an easier, less costly, minimalist way to certify software that covers different combinations of the same certification criteria. For instance, EHRA could work with the ONC-ATCBs to seek to have such testing able to be done in a timely and cost-effective manner and not to be unduly complicated by ONC’s requirement per 45 CFR 170.450 for separate privacy and security testing for each module.
• Where there are combinations including other 3rd party software within a module, there needs to be an easier way for customer to self certify the 3rd party solution or to have vendors amend their certification. For instance, if it were easy for vendors to add in those third parties or have more than one alternative as a valid certification module. (ATCBs need to be consistent in this interpretation).

6. What 3 meaningful use measures and their corresponding certification criteria and test procedures seemed not to be in alignment and caused confusion for you?
(1) Having to demonstrate encryption, when standard built-in features (e.g., browser, OS) are used. This is equivalent to distrusting your browser’s HTTPS encryption, and the test procedure essentially requires inserting a test tool (not part of the product) to demonstrate something that is normally “invisible.” Attestation should be permitted in this case.
(2) Having to perform at least one test of immunization registry and reportable labs to public health using the standards. Since states and public health agencies vary quite a bit in the standards they require or support, it is very unclear what must be done for meaningful use when the states do not support the same standards as specified by ONC. Questions submitted to the CMS questions website on this issue have gone unanswered since August 2010 until now.
(3) Exchanging key clinical information – since the means of transport of the clinical information was not specified (no standards), the impression was left with many organizations that any electronic transport would be acceptable. However, CMS recently surprised many with the ruling that use of portable electronic media is not acceptable. This should have been much clearer in the regulations and the tests, if only certain ways of exchanging (e.g., via network but not via media) were accepted.

Overall, while we appreciated ONC, NIST, and CMS responding to many of our questions along the way, we found that the certification process was quite difficult (and we had previous experience). For providers who would undergo self-developed certification, we think it would be even more difficult.

Thank you for listening.

David

Like or Dislike: 1  0 (+1)

The Academy and its members are supportive of adopting robust and applicable EHRs into their practice to improve quality of care and enhance patient safety. EHRs show promise as tools to facilitate information exchange between health care settings and promote greater coordination between health care providers. However, true meaningful use of EHRs in support of these goals will require EHRs to be flexible to the needs of specialists and compatible with in-office technology.

Although EHR certification criteria are intended to facilitate meaningful use of EHRs with the end goal of a fully interoperable health information system, the criteria as published in the final rule do not adequately address interoperability and information exchange capabilities for image-based specialties such as ophthalmology, radiology, and cardiology. Ophthalmologists frequently perform office-based testing with imaging and measurement devices during patient encounters, yet few vendors currently comply with data representation and exchange standards. This creates a significant obstacle to widespread adoption by the specialty, including difficulties involving manual data re-entry into patient records, image data residing in multiple locations, the need to scan results into EHR systems, and the need to develop proprietary device interfaces. We are concerned that this increases the risk of errors when such electronic data are entered incorrectly or not available at the point of care.

**To better address device interoperability and image exchange, the Academy requests that the Implementation Workgroup recommend changes to the certification criteria to include diagnostic images in the types of information that a certified EHR is required to electronically receive, display and transmit. (45 CFR § 170.304(i)).**

The inclusion of diagnostic images as a component of this criterion will promote adoption of the prevailing standard for image transfer, Digital Imaging and Communication in Medicine (DICOM), among EHR vendors and support the broader goals of device interoperability and information exchange.

**Both the Veterans’ Administration (VA) and the Department of Defense (DoD) Military Health System use the DICOM standard to enable images and associated diagnostic information to be retrieved and transferred between various manufacturers’ devices as well as provider workstations (Joint VA/DoD DICOM Conformance Requirements for Digital Acquisition Modalities, 2005). Without a similar regulatory mechanism for enforcing these standards in the private sector, physicians and other care providers will not be able to use their EHRs to their full potential to aggregate, store and retrieve patient information to the detriment of quality improvement and care coordination efforts. **

We encourage the committee to recommend the above addition to the certification criteria supporting health information exchange as soon as possible in order to meet the needs of physicians and to allow EHR vendors time to implement any necessary changes to their systems.

Sincerely,

Michael X. Repka, M.D.
Medical Director, Government Affairs
American Academy of Ophthalmology

Like or Dislike: 1  0 (+1)

Judy Murphy & Liz Johnson
Co-Chairs, HIT Standards Committee’s Implementation Workgroup
HIT Standards Committee’s Implementation Workgroup
c/o Judy Sparrow
Office of the National Coordinator for Health Information Technology
Department of Health and Human Services

Dear Ms. Murphy and Ms. Johnson:

The College of American Pathologists (CAP) appreciates the opportunity to comment on the HIT Standards Committee Implementation Workgroup’s request for feedback on the Temporary Certification Program. The CAP is a national medical specialty society representing more than 17,000 physicians who practice anatomic and/or clinical pathology. CAP members practice medicine in clinical laboratories, academic medical centers, research laboratories, community hospitals and federal and state health facilities. The CAP has significant HIT expertise; CAP is the original creator of SNOMED Clinical Terms® (SNOMED CT®), and CAP STS continues to develop and maintain SNOMED CT.

FRAMEWORK FOR CAP COMMENTS

With regard to the first question on the Workgroup’s survey, under current definitions, many of CAP’s members are eligible providers (EPs) who would need to use certified technology to qualify for MU incentives and avoid future penalties. With regard to the second question, we have members in small, medium, and large groups. The remainder of the questions do not impact pathology at this point in time, but we wish to take this opportunity to point out challenges that the certification requirements place on pathology’s qualification for meaningful use. We hope that the Implementation Workgroup will give serious consideration to these important macro issues even as the work group addresses issues related to certification criteria, test procedures, etc.

Certification Requirements and Laboratory Information Systems

Fully functional EHR systems as defined in the ONC final rule on standards, implementation specifications, and certification criteria are not used in pathology. Instead, pathologists and their laboratories have long relied on sophisticated computerized laboratory information systems (LIS’s) in order to support the work of analyzing patient specimens and generating test results, and it is with these LISs that EHRs or enterprise-wide clinical information systems exchange laboratory and pathology data. Pathologists generally do not currently directly use EHRs to input data. Therefore, pathologists, regardless of practice setting, cannot meet the MU Stage 1 requirement to maintain 80% of patient records in certified EHRs. Most of the ONC certification criteria are focused on office-based specialties and EHR use in such settings, particularly primary care, and apply to functions that are simply not applicable to pathology.

Display of Laboratory Results

CAP notes that Section 170.302(g) of the “Final Health Information Technology: Initial Set of Standards, Implementation Specifications, and Certification Criteria for Electronic Health Record Technology” (45 CFR Part 170) sets the certification criteria for the MU Stage 1 objective “Incorporate clinical lab-test results into certified EHR technology as structured data.” The criteria listed basically only require that CLIA requirements for laboratory reports are met. Going forward, we recommend that EHR certification criteria include requirements that laboratory result display and handling in an EHR is appropriate and flexible enough to account for the complexity of laboratory result display to support clinical interpretation and patient care.

Laboratory results are more than just numbers, and laboratory data in EHR systems must be an accurate representation of what is reported from the laboratory or patient safety issues could arise. EHR systems vary widely in their capability for effective and appropriate presentation of laboratory data. CAP and its member pathologists can attest to many instances of inadequate or poorly designed display of laboratory results in EHR systems that would (at best) be inconveniences and (at worst) present significant risk for misinterpretation by the health care provider and cause harm to the patients. Examples of the types of laboratory testing that may require unique considerations in data display (including beyond simply presenting simple numerical data such as chemistry and hematology data) include:

• Microbiology
• Blood bank/transfusion medicine
• Molecular pathology and genetic testing
• Interpretive testing that combines numerical results with interpretive text, e.g., coagulation panel interpretation, and serum protein electrophoresis
• Anatomic pathology, including surgical pathology, autopsy pathology, and cytopathology

Elements of laboratory reports that may be prone to suboptimal handling in EHR systems include:
• Reference ranges (normal ranges)
• Reflex test orders and results
• Interim reporting and amended results
• Sequential results reported on different dates
• Results that fall into “indeterminate” states between normal and abnormal values
• Misleading report formats
• Corrected result reporting and documentation
• Explanatory footnotes or comments
• Issuance of recent practice guideline or best practice changes and additional observations, interpretations and assessment based on test results, specific patient history and clinical presentations
• Proper identification of name and address of the performing laboratory, particularly in cases where more than one laboratory provides results to the EHR

While standards exist for messaging (HL7) and nomenclature (SNOMED and LOINC), standards for the stored structure of lab data itself are more nebulous, especially outside the realm of routine general labs. Further, the possible enumerated range that is valid for a specific field may not be a part of either the HL7 message or the coded representation. For example, a molecular test may be reported as “Positive” or “Negative” 99% of the time, but what are the standards for defining what to report when the test is limited due to technical or sample issues? “Unsatisfactory”, “Limited”, “Could not complete” also become necessary choices for clear communication. Should these be in upper case or lower case? In black type or red type? Font size? For instance, results that may appear binary are not necessarily so. Alternatively, a molecular test for a specific mutation may be correctly resulted as “positive” which would properly be interpreted as “negative” for the reason to order the test (e.g., KRAS mutation and chemotherapy sensitivity). Even though the assay result might be binary, which would be the “abnormal” result? Further, many seeming binary test results are not truly binary (e.g. dip-stick urinalysis; microbiology tests and contaminating organisms can affect body fluid tests).

CAP is pleased to work with other stakeholders on numerous standards-related efforts both in the private sector and through the ONC S&I Framework laboratory results interface initiative to address some of these critical issues.

Similar considerations may apply for considerations in the standards for CPOE for laboratory orders. CPOE will meet goals for laboratory test ordering only if 1) capabilities that are necessary to meet requirements of all of the nuances of laboratory test ordering exist in the CPOE system/module and 2) organizations and providers using CPOE configure the CPOE system in a way that ensures proper ordering of laboratory tests. Meeting these requirements necessitates involvement of laboratory-specific domain expertise on the part of CPOE system developers/vendors and organizations that are implementing such systems, respectively. In many ways, properly configured CPOE will enhance meaningful use of the data generated by the ordering provider.

Improperly designed or implemented CPOE systems can increase the chances for error in the laboratory test ordering process and can lead to:
• Incorrect test orders
• Incomplete test orders
• Inappropriate test orders
• Inefficiencies in and among laboratories and providers for test order problem resolution

CONCLUSION

While recognizing that many of the issues raised herein are outside the narrow scope of the Workgroup’s request, we nonetheless hope that the Workgroup and ONC will give serious consideration of these comments which have significant public health and practice implications. Should you have any questions or concerns, please contact Julie Cantor-Weinberg, Director, Public Health & Scientific Affairs.

Sincerely,

College of American Pathologists

Submitted via posting to the ONC blog

Like or Dislike: 2  0 (+2)

Where there were incremental changes in the NIST Test Scripts between versions 1.0 and 1.1, the changes were well-documented and accessible; it would however have been preferable and ensured consistency if the rules were changed to ensure that RETESTING is a repeat of a previous test and not a new test to a new script. As an example of such a change, the initial version 1.0 of the NIST Test Script for §170.302 (s) Integrity included two test procedures while version 1.1 added a third procedure; this new procedure required us to make changes in a previously certified module in order to be able to meet new requirements during retest.

Where there were changes in guidelines for interpreting the test procedures, there was inadequate documentation of the changes and difficulty in accessing the guidelines. In the absence of published guidelines and documented tracking of changes between versions, we were at times “surprised” by new and different requirements that we were expected to comply with. Especially given that the testing guidelines were not necessarily always directly derived from the NIST Test Scripts, this created some difficulties.

As an example of such an issue, I refer to the Test Script for §170.302 (p) Emergency Access which does not specify a specific scenario and in fact cites the Final Rule on the fact that the emergency access capabilities are not limited to a particular scenario; notwithstanding this fact, during retest we were presented with a guideline that only a “break the glass” scenario would be accepted and that our previously certified test scenario was therefore no longer deemed adequate.

We would urge that development of the Permanent Certification program take these problem into account, particularly with respect to:

1. A better formulated description of when attestation would be enough for recertification and when retesting would be required. This should take into account the possibility of situations that there are minor product changes in some program areas (which could be attested to) even if there are more significant changes in other areas (which would require testing).

2. To the extent that there are unavoidable changes in test scripts during the course of a certification period, a policy that would allow previously certified software that was being retested, to test to the original test script and not to the updated version. This would allow developers to concentrate on significant new functionality rather than running on a treadmill of changing requirements for previously certified functionality.

3. To the extent that there are guidelines on the interpretation of test scripts, to ensure publication and versioning of these guidelines so that changes can be tracked and followed. Again this would ideally be coupled with a policy that would allow previously certified software that was being retested, to test to the original test script and guidelines and not to the updated version.

Like or Dislike: 2  0 (+2)

Until we are totally confident we know how to design and deploy EHRs in a manner that will dramatically improve healthcare, why would we want to proliferate these systems? The thinking is that EHR interoperability will solve healthcare’s crisis. But ask yourself: Whenever you’ve received inadequate care, what was the root cause? Was it (1) because your doctor couldn’t access a medical record that was in some other doctor’s office? Was it (2) because your doctor did not have access to the clinical knowledge that would have led to accurate diagnosis and/or effective treatment? Or was it (3) because medical science, itself, just does not know enough?

Of those 3 causes for suboptimal healthcare, I believe the first one (lack of EHR interoperability) is actually the least impacting. For most clinical episodes, the treating physician is not truly handicapped by not being able to see what’s in some other physician’s record of your prior care. The second one seems to be considerably more instrumental. No physician can learn all s/he needs to learn, remember all that was learned, and apply it effectively during a brief clinical encounter. So we should clearly enable access to whatever is currently known by medical science, by providing computer-retrievable knowledge at the point of care. Not to do so is just plain foolish…or professionally arrogant.

The third cause, in my opinion, is actually the most significant deficiency in healthcare. Medical science just does not know enough. The reason for this is that healthcare does not learn from its own experiences. No one is retrospectively analyzing all the clinical encounters every day, to determine the early signs of what eventually become definitive diagnoses. No one is evaluating what treatments actually work best for various conditions, and under what circumstances. Medical science only moves forward via controlled clinical studies, which are too targeted and expensive to be our only strategy for advancing the science. We need to mine the data on real-life clinical encounters – nationwide. If you doubt this assertion, think about hormone-replacement therapy. The message here is that data interoperability, attained through a standardized clinical vocabulary, is more critical than operational interoperability.

Once we have determined, through data analyses (while controlling for potentially confounding variables), how to diagnose and treat more effectively, we must convert that learning into a “clinical guidance system”, operational at the point of care. We would monitor outcomes, assuming we can figure out how to measure them, so that the system can be empirically enhanced – thereby establishing continuous quality improvement (CQI) for healthcare. That, along with systematization of healthcare delivery, via processes like triage and rational incentives, is the only way that we can prevent the current crisis from turning into an apocalypse.

We need to conduct pilots of alternative EHR approaches, rigorously analyzing both the financial and clinical outcomes – so that we can learn what truly works best. The point-and-click documentation requirement of most existing EHRs has ironically been demonstrated to decrease the productivity of physicians. That is the last thing we need…particularly if there are no offsetting benefits derived from improved quality and value. Let’s figure out how to do it right: How to make data entry physician-friendly and highly efficient. Let’s bring the best minds together to design and evaluate these systems, which will determine the future of our nation’s healthcare. Let’s not throw money at this devastating problem until we know for sure it will buy the cure.

Like or Dislike: 2  0 (+2)

2. Please indicate the size of your group:
____ Small ____ Medium _X__ Large

3. What did you gain from the Temporary Certification Program that you did not expect?
We learned that the value of the complete EHR certification was not quite what we may have expected. Initially, and through the early months of the program, it is probably a safe bet most EHR vendors would have thought that the certification to go after for competitive reasons and for purpose of covering their client base demands for having certified EHR technology capable of covering most objectives. However, as became very apparent after our certification efforts to be certified as a complete EHR for both EP and Hospital in October 2010, the better path would have been modular certification. The reality is that aside from our hosted small hospital and physician office clients, most of our locally installed clients and larger clients do not buy from one vendor exclusively even for what is within the scope of meaningful use. There is almost always at least some component (e.g. public health reporting, quality measures or an ED system) purchased from another vendor. Given ONC’s policy, the complete EHR concept has two significant issues –
• It compels redundant software licensing from multiple vendors if a provider is to use any part of any given vendor’s complete EHR certification for meaningful use
• It does not allow a complete EHR certification to be leveraged in part if not as a whole

The consequence of this has been we have gone back and sought modular certification for most all of the component parts of our complete EHR certification so that our clients can pair our certified products at a modular level with those of other vendors so they do not have to license things in redundancy.
To do it again, we would seriously consider only pursuing modular certification. We would still look to cover all meaningful use objectives through the sum total of the effort, but by the combination of the certified modules – not as a complete EHR.

4. What part(s) of the Temporary Certification Program worked well and that you would not want to see changed?
The support and advice of CCHIT as our ONC-ATCB has been tremendous. CCHIT has been wonderful about providing guidance, and very fair in the testing process. They have been very willing to help talk through how to approach testing for certain objectives to ensure we had full understanding, and have worked with us extensively to understand how best to label our certifications and has been very responsive. ONC should also be commended. Special mention should go to Steve Posnack and the ONC staff for the help they gave us to think through our options for certification. The help we have gotten especially on how to deal with the consequences of the complete EHR policy has been invaluable.

5. What 3 Temporary Certification Program testing and certification process points were the confusing / most misunderstood? For these points, what could be improved and how?
(1) The testing of encryption and hashing (for integrity) of data in transit. The expectation of the test procedures and of the certifying bodies is that encryption in particular might be enabled as if at an end user option. Better guidance on how to test for these objectives by example would have been useful. Much encryption and hashing is done within the context of a secure communication channel using TLS or SSL 2.0 – and these are enabled as properties of the secure channel. End users do not do anything to enable their use. File encryption (if used) is not typically done through end user action to encrypt data that may be transmitted for normal transactional types of communication such as electronic prescribing or clinical document exchange. The encryption is done at a machine level as an automated process. We wonder if the tests for encryption and integrity could have been as well supported by showing the configuration of the secure communication channel itself. As it was, vendors were left to figure out how to isolate the encryption and hashing mechanisms and show them in a demonstrative way that modeled how what is otherwise an automated communication occurs. A better framework of guidance and example would have helped for what may have constituted a valid test. As it was, there was a lot of learn by doing, questioning of the ONC-ATCB as to what a valid test may be and proceeding forward.
(2) The testing of auditing discredited the potentially very valuable role of external security audit logs that could stand agnostic to source systems. There are several products on the market that provide external security audit logs that strongly support separation of duty and physical (as well as technical) protection of the audit log for accesses to ePHI from typical end user access to EHRs. The way the test procedure for auditing is written by NIST, it either requires the physical system of the EHR to maintain its audit logs locally or to specifically bind an external audit log to a particular source EHR to test both the creation of and the logging of audit events related to creation, modification or viewing of ePHI. The external security audit logs on the market are fully capable of standing agnostic to any given source EHR yet for them to have any standing; they have to be certified in combination with specific EHRs to be able to be part of any certification label. Security audit log products should be able to be certified for that part of the audit requirement that has to do with the ability to receive inbound audit events from source systems, and that has to do with the ability to view and sort audit logs. Should the requirements emerge for certification to interleaf audit events and data from more than one source system and/or to support the IHE ATNA standard for logging of audit events between nodes (as was originally proposed by the HITSC for Stage 1), the independent ability of such external security audit logs should be recognized by the definition of the test procedure. If NIST were to define two test procedures for auditing – one for creation of audit events and data and one for viewing and sorting of the audit log itself – vendors could choose to certify to one or both criteria, and security audit log products could certify to the latter without creating an explicit dependency on each and every EHR vendor product they happen to share a common client with.
(3) The testing of hospital clinical quality measures (CQMs) for ED systems needs to absolutely be fixed. ED systems should be able to clearly certify on the clinical quality measures for ED throughput in isolation of the other hospital quality measures. This flexibility is afforded to systems certifying under the EP program, and the Stroke and VTE measures typically require data only available from an inpatient encounter to properly be calculated. It is not very credible that an ED system could certify to the Stroke and VTE measures, and even if they could (which some vendors somehow did), data does not typically flow from the IP side to the ED side, but the other way. ONC should not have defined this certification criterion to require an EHR or EHR module to certify to all of the hospital CQMs. It should have allowed for a system to certify to Stroke, VTE or ED Throughput at a condition or categorical level. This is an easy fix. Allow systems to so certify to a condition or a category – especially appropriate to its venue.
On a related point to quality measure testing, there could be more elaborative guidance in the test procedures given to vendors as to how to go about the whole process, which would be helpful to preparation and dry runs for going through certification inspections. This could focus on such matters as

• Informing the vendor that they would need to show at least 1 outcome for met, 1 for not met, and 1 for exclusion (if applicable) for each measure.
• Helping the vendor know to show the actual patient data in the chart that attributed to each outcome above
• Telling the vendor to what degree they would need to show the .xml file for each measure
• Letting the vendor know if they could generate the .xml file during the inspection or if they could have a file already prepared prior
• Setting the vendor expectation if they need to be prepared to show a modification to a patient record during inspection that impacts at least 1 measure show the before and after in the patient effects on the measure calculation and in the output of the.xml file

These are important points to a vendor’s preparation and dry run efforts to get ready to go through inspection, and should not be left to tester discretion if not accounted for by the test procedure.

6. What 3 meaningful use measures and their corresponding certification criteria and test procedures seemed not to be in alignment and caused confusion for you?
(1) The structured lab objective may have been the hardest for us to define. Despite the guidance that the lab result should be a numeric value or a positive or negative affirmation, there still was a lot of room for interpretation of what types of lab procedures and results should be considered for numerator credit. The challenge particularly came into play for result values that could be short textual strings.
(2) The CPOE objective was made difficult considering what healthcare roles could actually be considered. The objective made reference to it counting those who were licensed and who could exercise clinical judgment….but was that someone able to order under their own authority and scope of practice? Someone who was licensed transcribing the order given by another? Did it include verbal orders subject to later verification by a physician? Did it include pharmacists transcribing orders faxed to the pharmacy from the nursing unit? Especially at first, there was just too much vagueness in who could place an order and have it counted, the types of orders that could be placed and what types of workflows for entering orders could be accommodated. If the intent was to promote physician adoption (the “P” in CPOE), it seems at odds with that intent to allow for all manners of order transcription to be considered.
(3) The electronic copy of the record objective was clouded by considerations of how the record could be provided, the form it needed to be provided in, whether that needed to be singular or multiple electronic files/outputs and what content really needed to be included…..and the impacts of any conditions the patient might place on the request. For example, what if the patient asked for an electronic copy of their information that was not included in the specified minimum content as defined by the certification criteria but still was based on information otherwise stored in the certified EHR? What if multiple outputs were necessary? What if some of the requested information was not available in the form requested by the patient? Was there an obligation to inform the patient of their right to receive an electronic copy under ARRA HITECH? These are all questions we actually got from clients as we got into practical application of actual use.

7. Are the certification criteria clear?
__x__ Yes ____ No
If no, which criteria and how it can be improved?

For the most part, they are reasonably clear. It is more the test procedures, and the interpretive application of the measurement definitions to actual use that are not as clear. See responses to above questions.
8. Was the level of specificity appropriate?
____ Yes __x__ No
If no, how it can be improved?
Much more effort should be put to provide guidance of what constitutes valid forms or manners of testing for objectives such as we mention in our comments about encryption and integrity above. If examples of valid testing approaches can be developed within the context of the test procedures, that would serve to help vendors prepare.

9. Should certain certification criteria be combined or decomposed differently?
__x__ Yes ____ No
If yes, which criteria and why?

We highlight the audit test procedure above as one such objective that should be decomposed into two procedures to allow for security audit log products to be able to test independently.
We also highlight all of the public health reporting objectives as objectives that may need to be decomposed to allow for public health reporting applications to be able to clearly standalone and be certified as EHR modules. That is possible under the current test procedure definition, but the test procedures are a little awkwardly defined for that purpose. The certification criteria is defined to focus on the ability of the EHR to submit public health reporting data in a conformant manner to a defined specification, but the test procedure lays out a presumption that manual data entry in a source EHR need be the starting point for testing the certification criteria. The test procedure should allow for a starting point that the system is able to acquire the inbound data from a source system by showing how such inbound files are obtained. If the test is of submission, the test procedure should not require data acquisition only be possible by direct data entry.

10. Should certain certification criteria be scoped differently?
__x__ Yes ____ No
If yes, which criteria and why?
We have highlighted in response to questions 5 and 9 above particular certification criteria (or their test procedures) that should be scoped differently or defined to allow for alternative testing approaches for

• Auditing
• Encryption and integrity
• Public health reporting (lab, syndrome and immunization)
• Clinical Quality Measures

11. Please comment on the balance of process-oriented vs. outcome-oriented certification criteria.

12. Do you have any suggestions for improving the ONC-approved test procedures for EHR certification?
Please see comments in response to other questions above.
13. Would it be beneficial if more choreographed/combined test procedures were developed that permitted an EHR developer to satisfy multiple certification criteria at once?
_x___ Yes ____ No
An example of this is the test procedures for Access Control and Authentication. There are four test steps that are entirely repeated between them. Another example is for testing Encryption of Data in Transit and General Encryption. We are aware that ONC-ATCBs have allowed vendors to test both of those procedures with the same example of encryption. Either combine the two test procedures or require different encryption capabilities to be tested between them. Otherwise, it seems pointless to have two test procedures for encryption.

14. In what ways can the test procedures be combined to facilitate the testing of certification criteria within the context of clinical workflow?
It seems that the test procedures that align to a clinical workflow could be combined into a storyboard of sorts that provided for a linear flow to allow for multiple test procedures to be tested in one overall flow….so for EPs, CPOE, clinical decision support, drug based alerting and eRX could be tested together. For Hospitals, CPOE, clinical decision support, drug-formulary checking and drug based alerting could be tested as a continuous flow. Other combinations also seem possible centered around discharge or departure from a physician office including medication reconciliation, discharge instructions or patient education and providing an electronic copy of the record.
15. Please provide specific instances where the ONC-approved test method can be improved to reduce ambiguity, inconsistency with criteria, and addition of implicit requirements beyond the certification criteria.
No other comments come to mind aside from those offered in response to other questions.
16. How would you rate your certification experience on a scale from 1 to 6 (circle one)
1 2 3 4 5 6
Difficult Easy

17. Other Comments or Suggestions
We reinforce our great appreciation for the help and efforts of both CCHIT as our ONC-ATCB and of ONC to try to help in the breach of clarity being available from the final rules and from the NIST test procedures. We know this was a lot of learn by doing and policy interpretation by FAQ that is still going on, and the efforts of both of those entities to help provide clarification and guidance on a continuous basis are very much appreciated.

Like or Dislike: 2  0 (+2)