eHealth Observatory

Metrics

Introduction

The following are some of the design components and metrics to evaluate HIS adoption/use.

Exhibits 1- 7

Exhibit 1

Exhibit 1.

Four study cohorts can be defined according to the adopter/user and HIS type.

Legend: *means only EMR or eDrug is adopted/used but not both; +/- means either EMR or eDrug is present for basic-adopters/users

Cohort EMR eDrug EHR
A: Non-adopter/user
B: Basic-adopter/user* +/- +/-
C: Intermediate-adopter/user
D: Advanced-adopter/user

Exhibit 2

Exhibit 2.

Each cohort in 3a can be further divided into three care settings each with different HIS types

Setting EMR eDrug EHR
Physician Office
Community Pharmacy
Acute/Tertiary/continuing care

Exhibit 3

Exhibit 3.

Each cohort in 3a can be further divided into three provider types each involved with different HIS types

Provider EMR eDrug EHR
Physician
Pharmasist
Authorized health professional

Exhibit 4

Exhibit 4.

Different study design options available based on composition of the cohorts and time periods being examined. Legend: N is sample size; T is time period; subscripts 0 to 5 denote the sample sizes and time periods for each cohort; ∑ refers to the column, row and grand total sample sizes. For example, Cohort-A at Time-0 can be defined to include all three types of providers across all three settings who have not adopted/used EMR, eDrug and EHR at the baseline period.

Examples Time-0 Time-1 Time-2 Time-3 Time-4 Total
Cohort-A A(N1T0) A(N2T1) A(N3T2) A(N4T3) A(N5T4) A(N1-5)
Cohort-B B(N1T0) B(N2T1) B(N3T2) B(N4T3) B(N5T4) B(N1-5)
Cohort-C C(N1T0) C(N2T1) C(N3T2) C(N4T3) C(N5T4) C(N1-5)
Cohort-D D(N1T0) D(N2T1) D(N3T2) D(N4T3) D(N5T4) D(N1-5)
Total ∑(N1T0) ∑(N2T1) ∑(N3T2) ∑(N4T3) ∑(N3T4) ∑(N1-5)

Exhibit 5

Exhibit 5.

Five kinds of studies to examine the information/system/service quality and use/satisfaction dimensions in the BE Framework. Legend: Cohorts-A, B, C, D are non-adopters/users with no HIS, basic-adopters/users with eDrug or EMR, intermediate-adopters/users with eDrug and EMR, and advanced-adopters/users with eDrug, EMR and EHR, respectively. Time-0 is the baseline period, Time-X is the next period, Time-Y the period after Time-X, and Time-Z after Time-Y, etc. Cohort-A with no HIS is excluded since these studies all require specific types of HIS being adopted/used.

Studies on BE Quality/use Dimensions Cohort-A Cohort-B Cohort-C Cohort-D
(1) Comparison: documented features vs. best practices Time-0 to Time-X/Z Time-X to Time-Y/Z Time-X to Time-Y/Z
(2) Usability: ease of use, functionality and usefulness Time-0 to Time-X/Z Time-X to Time-Y/Z Time-X to Time-Y/Z
(3) Qualitative case: survey, interview and observation on on percieved/actual use Time-0 to Time-X/Z Time-X to Time-Y/Z Time-X to Time-Y/Z
(4) Quantitive case: system use log triangulated with qualitative case findings Time-0 to Time-X/Z Time-X to Time-Y/Z Time-X to Time-Y/Z
Data quality audit: patient electronic records vs. manual chart Time-0 to Time-X/Z Time-X to Time-Y/Z Time-X to Time-Y/Z

Exhibit 6

Exhibit 6.

Five kinds of studies to evaluate the care quality and productivity dimensions in the BE Framework. Legend: Cohorts-A, B, C, D refer to non-adopters/users with no HIS, basic-adopters/users with eDrug or EMR, intermediate-adopters/users with eDrug and EMR, and advanced-adopters/users with eDrug, EMR and EHR, respectively. Time-0 is the baseline period, Time-X is the next period, Time-Y the period after Time-X, and Time-Z after Time-Y, etc.

Studies on BE Benefits Dimensions Cohort-A Cohort-B Cohort-C Cohort-D
(1) Efficiency: Call-to-pharmacy; special authority request adjudication Time-0 to Time-X/Y Time-0 to Time-X/Y Time-X to Time-Y/Z Time-X to Time-Y/Z
(2) Coordination: review/reconciliation of medication management Time-0 to Time-X/Y Time-0 to Time-X/Y Time-X to Time-Y/Z Time-X to Time-Y/Z
(3) Patient safety: adverse drug event and medical error prevention and detection Time-0 to Time-X/Y Time-0 to Time-X/Y Time-X to Time-Y/Z Time-X to Time-Y/Z
(4) Effectiveness: special authority compliance; disease management Time-0 to Time-X/Y Time-0 to Time-X/Y Time-X to Time-Y/Z Time-X to Time-Y/Z

Exhibit 7

Exhibit 7.

Two examples of time series delayed controlled matching cohorts. First is the two matching Pharmacy cohorts where baseline measures are taken at Time-0 with the original PharmaNet. Then Pharmacy1 deploys the new eDrug in Time-1 while Pharmacy2 continues with PharmaNet. Then Pharmacy2 deploys new eDrug in Time-2 while Pharmacy1 continues to use eDrug. Then Pharmacy1 deploys a new EMR while Pharmacy2 continues to use eDrug. Another example is the matching Physician cohorts for deployment of EMR, eDrug and EHR.

Examples Time-0 Time-1 Time-2 Time-3 Time-4
Pharmacy 1 Original PharmaNet New eDrug eDrug eDrug + New EHR eDrug + EHR
Pharmacy 2 Original PharmaNet Original PharmaNet New eDrug eDrug eDrug + New EHR
Physician 1 Exsisting EMR EMR + New eDrug EMR + eDrug + New EHR EMR + eDrug + EHR EMR + eDrug + EHR
Physician 2 No HIS New EMR EMR + New eDrug EMR + eDrug + New EHR EMR + eDrug + EHR

Reactive Analysis

Reactive Analysis of an Information System, from the IT Interaction Model by Silver et al. [xx]

  • Was the information system's design objective to IMPROVE the organization incrementally or to TRANSFORM it?
  • What are the information system's FEATURES? What does it do?
  • How does the information system FIT the firm's EXTERNAL ENVIRONMENT?
  • How does the system FIT the firm's STRATEGY?
  • How does the system FIT the firm's BUSINESS PROCESSES?
  • How does the system FIT the organizational STRUCTURE and CULTURE?
  • Can the organization's existing IT INFRASTRUCTURE support the system? Does the information system leverage the infrastructure? Does it extend it?
  • How and how effectively was the system IMPLEMENTED?
  • Who USES the system and how do they USE it? As intended?
  • What are the CONSEQUENCES of the system for PERFORMANCE, PEOPLE, and FUTURE FLEXIBILITY? Did the system accomplish its objectives?

Guiding Questions

Guiding Questions from Kotter's Organizational Change Model [xx]

  • Has the sense of urgency for change been established within the organization?
  • Has a guiding coalition been created within the organization?
  • Has the vision and strategy been developed within the organization?
  • Has the vision and strategy been communicated throughout the organization?
  • What is the mechanism to empower broad-based action for change throughout the organization?
  • What are the short-term wins for the organization that can be generated from this change?
  • What are the efforts to consolidate gains and produce more change within the organization?
  • How are the new approaches being anchored within the organizational culture?