Plan

How should the user information be collected?

Before conducting any form of system or process evaluation or redesign, a list of all possible end-user or participant types must be generated via user profiling. The information needed to create user profiles, which are categorical groupings of unique user types (i.e. nurses with high technological proficiency, physicians with minimal computer experience, pharmacists with vast domain experience, etc.), can be obtained by conducting a survey. Survey’s can be generally broken down into two categories: questionnaires and interviews. In order to gather the information necessary to form the user profiles, these two survey methods can be applied as follows:

1. a questionnaire can be used to poll the system end users for relevant user background information.
2. an interview with experts who have detailed knowledge about the system end users can be conducted to gather user background information.
   • a.SAMPLE TEMPLATES NEEDED for questionnaires / interviews from (Mayhew, 1999)

Because questionnaires gather information directly from the source (the end-system-users and process participants), it is considered to be the best practice for gathering the most accurate user background data. The drawback of using questionnaires is that creating, testing, revising, distributing, gathering, then analyzing questionnaires can be quite time consuming. An interview with a person with expert knowledge about the users under study, however, may be able to provide such user categorization and qualification information in a much more effective manner, likely saving both time and money.

The choice between the two user profiling methods depends not only on the budget and schedule of the RR study, but also on the system and process under evaluation or redesign as well. If the system or process solution is designed for the long-term (meaning it will undergo multiple revisions and tests in the future), then spending the necessary time to create a detailed user profile set through the use of a questionnaire may be the best option (since user profile sets can be re-used in future system and process tests, as long as the system and process still involves the same target user groups). If, however, there are no future plans for tests or evaluations of the system or process, then the best option may be creating a faster, yet possibly less accurate user profile set, by interviewing experts with end-system-user and process participant knowledge.

What user information should be collected?

There are several attributes that can affect a person’s use of a system. When creating the user profile set (either through questionnaires or interviews), it is important that such attributes are determined, so that any user behaviour that may affect the process or the system’s use is understood. Although there are countless human characteristics that may affect process participation or system use, the following three user attribute categories have been found to be most useful (for usability studies in particular, which serve as the cornerstone evaluation methodology of the Rapid Response methodology) (Kushniruk & Patel, 2004):

• Technology Experience - The degree to which the user feels comfortable around the use of technology (i.e. a scale of 1 to 4 defining growing levels of technology expertise, where 1= no technology experience whatsoever and 4 = technology guru).
• Domain Experience - The user's experience with the work that the information system is targeted for (i.e. if the system under evaluation was used for medical imaging, a scale of 1 to 4 could indicate users who: 1 = have no previous experience with medical imaging, to 4 = users who have years of daily experience with medical imaging).
• Disciplinary Background - The roles of the different users who are responsible to use the system under evaluation (i.e. nurses, physicians, etc.).

It may also be necessary to factor in other relevant criteria, such as the user's demographic background (i.e. age, gender, etc.) physical attributes (i.e. color blindness, auditory deficiencies, etc.), attitude towards change or information technology (i.e. extremely resistant to computer use), when creating the user profiles.

Create the data collection tool

Once the type of data collection tool has been selected, and the categories of desired user information have been chosen, the study designers can then create the data collection tool that will be used to gather such user data. For the survey data collection tool, it is recommended to use a ready-made template (if one already exists), given that it is applicable to the area under study and previously proven and validated. In many cases, however, a new survey tool will have to be developed from scratch.

There are several issues to take into consideration when designing a survey, including: question format, structure, ordering, biases, scoring, etc. All of such issues must be well understood before designing a new survey for practical use. For a detailed description of survey design and use, please see the following Web link:

• Research Methods Knowledge Base – Survey Research: http://www.socialresearchmethods.net/kb/survey.php

The following is an example of a user profile data collection questionnaire:

Example user profile questionnaire needed

The following is an example of a user profile data collection interview:

Example user profile interview needed

Return to tabs

Do

Once user profile data collection tool has been selected and/or created, the tool can then be put to use by distributing it to the users (either in the form of remote questionnaires or in-person interviews).

Questionnaires

For questionnaires, a traditional, paper-based approach can be used as the distribution method (i.e. by mailing the questionnaires out to the system users or process participants, along with pre-paid envelopes so that the participants can mail the answered questionnaires back to the study designers), or modern technology could be utilized to distribute the questionnaires electronically. Distributing questionnaires electronically is not always possible (i.e. in the case where not all of the user group has computers or the internet) and it is also not always the most efficient method to distributing questionnaires (i.e. in the case of a small sample group that can be easily and quickly reached by the distributors); however, in many cases Web-based questionnaires can be used to greatly increase the efficiency of the survey data collection process. Not only could a basic electronic questionnaire be created through a word processor and then emailed the users, but there are also several freely available Web survey tools can automate much of the data collection process. Such features as instant and free questionnaire distribution and retrieval, structured question responses, restricted survey access, automated statistical analysis, pre-made question design templates, and automated survey response monitoring can make online survey tools very beneficial to use for data collection.

The following are two suggested tools for conducting online questionnaires:

• For advanced survey design:
  • Lime Survey – an open source survey tool - http://www.limesurvey.org/
• For quick and basic surveys:
  • Survey Monkey - http://www.surveymonkey.com/

When using the questionnaire method to surveying, it is extremely important that the responses received from the questionnaire are representative of the total population being surveyed. If there are too few responses or if certain demographics don’t respond to the survey, then the user profiles will not accurately portray the user population. To counteract such problems this, it is important to take the response rate of the questionnaire into strong consideration before analyzing the results. A low response rate (generally below 50%) indicates that the quality of the survey results is likely to be low due to response biases (i.e. those who responded to the questionnaire are different from those who didn’t respond to it, yet the survey results will only indicate the opinions of those who responded to the questionnaire); therefore, everything in the questionnaire designer’s power must be done to attain a high response rate (i.e. by: shortening the questions on the questionnaire, providing incentives to the sample for answering the questionnaire, etc.).

Interviews

Interviewing experts with knowledge about the user/participant group under study is a much simpler and quicker method to gathering user information. The downside to interviewing such experts is that the accuracy of the user information may be reduced. A user expert, for instance, may not actually know enough about the different user/participant types as they lead the interviewer to believe, or the user expert may have personal biases, which skew the information delivered about specific user types (i.e. a physician user expert may be able to provide quality information about the use of an Electronic Health Record (EHR) system in a local practice by other physicians; however their idea about users from the nursing or office assistant community may be inaccurate). An option for improving the accuracy of the user expert interview results (albeit lowering the speed at which the results can be generated) is to conduct multiple interviews with different user experts (perhaps, one from each domain).

There are three methods to conducting interviews: structured, semi-structured, and unstructured. Structured interviews require that the interviewer follows the pre-defined questions exactly as planned, without wavering off topic. This is the most appropriate interview format to use when it is known exactly what information needs to be retrieved during the interview. A semi-structured interview follows the pre-defined question sheet, but allows the interviewer to explore previously unidentified areas of interest with the interviewee (i.e. if the interviewee commented on something that was previously missed by the drafted question sheet, yet it is of interest to the interviewer, the interviewer could then further pursue questions about that topic). The semi-structured interview method is generally recommended for most interviews. Unstructured interviews do not require a pre-defined question sheet. In such unstructured interviews, the interviewer normally starts with a generic question to the interviewee (i.e. “What are the different types of EHR users at this practice?”), and the rest of the questions are asked on an ad-hoc basis, based on the interviewee’s responses. This method is generally used when the interviewer has little understanding of the topic of interest going into the interview, and so they rely on the interviewee to expose the areas of interest during the interview, in which the interviewer could then focus on.

Return to tabs

Study

Once adequate system end user or process participant data has been gathered, the study designers need to categorize their findings. For each user answered questionnaire or for the user information created through the expert interviews, the unique user attributes (i.e. technological experience, domain background, physical considerations, etc.) need to be abstracted and noted. After analyzing each of the users’ questionnaire responses, or all of the expert interview responses, a completed set of user attributes should be available to then categorize into general user types (stereotypes). This can be accomplished by grouping the users by attribute type (i.e. having separate user profiles for physicians, nurses, medical office assistants, etc.), or by a combination of attribute types (i.e. physicians with no computer experience, physicians with moderate computer experience, physicians with a high degree of computer experience, nurses with no computer experience, etc.). For example, after analyzing all physician responses to a questionnaire that was given to assess their background information before designing a new Electronic Medical Record (EMR), it was determined that the computer experience levels of the physicians varied greatly. This prompted the study designers to create three physician stereotypes (physician group 1 – physicians with high degrees of computer experience, physician group 2 – physicians with moderate computer experience, and physician group 3 – physicians with little or no computer experience).

Return to tabs

Act

After the user group information has been categorized, the user profiles can be created. User profiles should be written in the form of abstract biographies, which describe each stereotypical type of end-system user. Each user profile should contain a summary of the categorical grouping of user attributes, which forms the unique user stereotype. When the stereotypes are completed, they should be validated by the end-system users or process participants for accuracy. If the user profiles are inaccurate, either make the necessary changes to the profiles (in the case of minimal user profiling errors), or repeat the PDSA cycle to gather more accurate user data (in the case of major user profiling errors).

The following are examples of user profiles:

• to be inserted

Return to tabs

PDSA Summary

Plan

Determine which survey method to use (questionnaire or interview) to collect user profile data.

Decide which information needs to be collected in order to create the user profiles.

Either customize an existing user profile data collection tool or create one (draft, pilot)

Do

If using a questionnaire:

• Use either a paper-based questionnaire tool (for small samples where users can easily be reached by mail or in person, or for studies where not all of the user population has access to the Web) or an electronic tool (when the study sample has access to the Web and survey automation is desired).
• Attain a high response rate from a representative user population

If using interviews:

• Select one or multiple interviewees who have expert knowledge about the user groups under study;
• Conduct the interview in either a structured, semi-structured, or unstructured manner.

Study

Categorize the user data into a complete set of user stereotypes that capture every unique user attribute, so that all actual system users or process participants can qualify as belonging to one of the user stereotype groups

Act

From the user profiles, create textual descriptions of the stereotypical groupings

Validate the user profiles with an expert

Return to tabs

References

References

• Kushniruk, A. W., & Patel, V. L. (2004). Cognitive and usability engineering methods for the evaluation of clinical information systems. Journal of Biomedical Informatics (37), 56-76.
• Mayhew, D. J. (1999). THE USABILITY ENGINEERING LIFECYCLE - a practitioner's handbook for user interface design. (D. Cerra, Ed.) San Francisco, California, United States of America: MORGAN KAUFMANN PUBLISHERS, INC.

Return to tabs