CMS-10294.Appendix F (8-23-10)

APPENDIX F
CASE STUDY DISCUSSION GUIDES AND LETTERS

CMS LETTERHEAD
LETTER OF ENCOURAGEMENT TO PARTICIPATE - EVALUATION OF THE
9TH QIO SOW
PROVIDER ORGANIZATIONS SELECTED FOR SITE VISITS
[DATE]
[NAME AND ADDRESS]
Dear [Dr./Mr./Ms.] [FILL LAST NAME]:
The Centers for Medicare & Medicaid Services (CMS) is sponsoring a study about the
Quality Improvement Organization (QIO) Program, which is a key component in CMS’ efforts
to improve the quality and efficiency of care for Medicare beneficiaries. The purpose of the
study is to evaluate the program’s effectiveness in helping health care providers to improve the
quality of care for Medicare beneficiaries and to find ways to improve the program.
Mathematica Policy Research, Inc. (MPR), an independent research organization, is
conducting the study on behalf of CMS, with partnered organizations Social & Scientific
Systems and Abt Associates. As part of this study, MPR and its partners will visit providers in
twelve states who worked with their local QIO.
Your participation in the site visits is voluntary, but your participation is invaluable for the
success of this important study. Your input will help Medicare to improve the design and
effectiveness of the QIO program and to ultimately improve care for Medicare beneficiaries. I
urge you to participate as described in the invitation letter included with this one.
Please be assured that your participation in the study will remain confidential to the extent
permitted by law. All data collected for the purposes of this study will be combined and reported
in aggregate form only. Neither you nor your organization will be identified by name in any
reports or documents produced from the study findings. Only Mathematica staff that work
directly on the evaluation will have access to the name of your organization and your name.
We look forward to including your valuable input in this study.
Sincerely,

CMS Project Officer
Enclosure

F.3

QIO DIRECTOR INVITATION LETTER
600 Maryland Ave. S.W.
Suite 550
Washington D.C. 20024-2512
Telephone (202) 484-9220
Fax (202) 863-1763
www.mathematica-mpr.com

[Date], 2010/2011
QIO Director
QIO name
Street address
City, ST zip
Dear [QIO Director name]:
The Centers for Medicare & Medicaid Services (CMS) would very much appreciate your
thoughtful input on your QIO’s experience with the 9th SOW for [STATE]. This is an invitation to
participate in a site visit being conducted as part of the evaluation of the 9th SOW Quality
Improvement Organization Program by Mathematica Policy Research and its subcontractors Social
& Scientific Systems and Abt Associates, sponsored by CMS. Participation will involve
discussions with you and the key QIO staff who lead themes or patient safety sub-themes for a
total of [TOTAL LENGTH OF INTERVIEWS] hours during the week of [TARGET WEEK]; this
total includes one hour for discussion with you, approximately 30 minutes per theme or patient
safety sub-theme with the leader of each of those efforts, and an additional 15 minutes for the
overall patient safety theme leader.
Your feedback is confidential to the extent permitted by law. Your input will be valuable in
assisting CMS in continuing to improve the QIO program with each new scope of work. The site
visits are part of a larger study to evaluate the 9th Scope of Work Quality Improvement
Organization Program, described in the attachment.
Please note that Section C.4.B.13 of the 9th SOW contract requires each QIO to provide data
for evaluation, thus, your time to participate in the site visit is an expense covered under the
contract. If you have any questions, please feel free to call [CMS OFFICIAL] at [PHONE
NUMBER].
On-site, we would like to discuss the following key topics:
• The impact you have had on quality improvement and patient safety related to each
theme (note we do not expect you to provide data, the purpose is to obtain your
thinking in summary form about your accomplishments)
• Any difficulties you have faced with the program contract, infrastructure, and supports
• The value of QIOSCs in facilitating your work
• Which types of activities you have undertaken since August 2008 were more and less
valuable for quality improvement

F.5

An Affirmative Action/Equal Opportunity Employer

LETTER TO:
FROM:
DATE:
PAGE:

QIO Director
Myles Maxfeild
[Date], 2010/2011
2

• How you recruited providers to work with you during the 9th SOW
• Types of actions taken by providers to improve care on the program-targeted measures,
and what prompted the actions
• Lessons learned from the 9th SOW experience
• What types of key barriers remain to quality improvement in this state
• The state quality environment, such as the roles of other organizations influencing
quality, and the interest among providers in collaborating and working for quality
improvement
All data collected for the purposes of this study will be combined and reported in aggregate
form only. Neither you nor your organization will be identified by name in any reports or
documents produced from the study findings. Only Mathematica staff that work directly on the
evaluation will have access to the name of your organization and your name.
With your help, the evaluation will provide critical information to CMS to help refine the
Quality Improvement Program. An evaluation staff member will call you to follow up in the next
couple of days. In the meantime, please feel free to call or e-mail [CONTACT INFORMATION
OF EVALUATION STAFF MEMBER] with any questions or to initiate scheduling. Thank you
very much in advance for your assistance—your input into the evaluation is highly valued by
CMS.
Sincerely,
Myles Maxfield, Ph.D.
Project Director

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless
it displays a valid OMB control number. The valid OMB control number for this information collection is xxx-XXX. The
time required to complete this information collection is estimated to average 0.8 hours or 48 minutes per response, including
the time to review instructions, search existing data resources, and gather the data needed, and complete and review the
information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving
this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer, Mail Stop C4-26-05,
Baltimore, Maryland 21244-1850.

F.6

PROVIDER INVITATION LETTER
600 Maryland Ave. S.W.,
Suite 550
Washington, DC 20024-2512
Telephone (202) 484-9220
Fax (202) 863-1763
www.mathematica-mpr.com
[Date], 2010/2011
Contact
Organization name
Street address
City, ST zip
Dear [contact name]:
This is an invitation to provide important feedback to the Centers for Medicare & Medicaid Services (CMS) through
agreeing to an on-site interview, roughly 45 to 50 minutes in length, to assist CMS in evaluating and improving its Quality
Improvement Organization Program. The CMS Quality Improvement Organization Program in this state is operated
through [NAME QIO]. If you have worked with [NAME QIO] on one or more quality improvement efforts since August
2008, the evaluation research team would appreciate your feedback through an on-site interview during [TARGET
WEEK], to be scheduled at your convenience. You will not need to make any special preparations for the visit, and your
input will remain confidential to the extent permitted by law. In particular, we would like to discuss:
•

The overall value and effect of [NAME QIO] activities related to [NAME THEME(S) THE PROVIDER
WORKED WITH THEM ON]

•

Why you decided to work with [NAME QIO] (if applicable) and which types of interactions, tools, resources,
etc. that were provided by [NAME QIO] were more and less helpful to you

•

The story of this [TYPE OF PROVIDER]’s quality improvement related to [NAME THEME(S)] since Summer
2008, and the major reasons for any changes in measured performance.

•

Lessons you have learned about how to improve quality since Summer 2008, and remaining barriers you face to
further improvement

•

What you know and think about the quality-related environment in your area, for example, how interested [TYPE
OF PROVIDER]s are in general in quality improvement, and whether they are willing to share information

CMS has contracted with Mathematica Policy Research, Inc. and its partners Social & Scientific Systems and Abt
Associates, to conduct the site visits as part of a larger study to evaluate the 9th Scope of Work Quality Improvement
Organization Program. A description of the larger study is attached.
All data collected for the purposes of this study will be combined and reported in aggregate form only. Neither you
nor your organization will be identified by name in any reports or documents produced from the study findings. Only
Mathematica staff that work directly on the evaluation will have access to the name of your organization and your name.
With your help, the evaluation will provide critical information to CMS to help refine its work to improve the quality
of care. Please see the attached letter of encouragement to participate from CMS. An evaluation staff member will call you
to follow up in the next couple of days. In the meantime, please feel free to call or e-mail [CONTACT INFORMATION
OF EVALUATION STAFF MEMBER] with any questions or to initiate scheduling. Thank you very much in advance for
your assistance—your input into the evaluation is highly valued by CMS.
Sincerely,
Myles Maxfield, Ph.D., Project Director
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid
OMB control number. The valid OMB control number for this information collection is xxx-XXX. The time required to complete this information
collection is estimated to average 0.8 hours or 48 minutes per response, including the time to review instructions, search existing data resources,
and gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time
estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer, Mail
Stop C4-26-05, Baltimore, Maryland 21244-1850.

F.7

An Affirmative Action/Equal Opportunity Employer

COMMUNITY HEALTH LEADER
INVITATION LETTER

600 Maryland Ave. S.W., Suite 550
Washington, DC 20024-2512
Telephone (202) 484-9220
Fax (202) 863-1763
www.mathematica-mpr.com

[Date], 2010/2011
Contact
Organization name
Street address
City, ST zip
Dear [contact name]:
As a key health leader representing [HOSPITALS/NURSING HOMES/PHYSICIANS], you are no doubt
aware of the investment the Centers for Medicare & Medicaid Services (CMS) makes in improving quality of
care through its Quality Improvement Organization Program, which in this state is operated through [NAME
QIO]. This is an invitation to provide important feedback to CMS through agreeing to an in-person interview,
roughly 45 to 50 minutes in length, to assist CMS in evaluating and improving its Quality Improvement
Organization Program. Your input will remain confidential to the extent permitted by law.
CMS has contracted with Mathematica Policy Research, Inc. and its partners Social & Scientific Systems
and Abt Associates, to conduct the site visits as part of a larger study to evaluate the 9th Scope of Work Quality
Improvement Organization Program. A description of the larger study is attached. The evaluation research team
would appreciate your feedback through an in-person interview during [TARGET WEEK], to be scheduled at
your convenience. You will not need to make any special preparations for the meeting. In particular, we would
like to discuss the following key topics:
•
•
•
•

The overall value and effect of [NAME QIO] activities related to quality improvement and patient safety
Which types of activities by [NAME QIO] since August 2008 that were more and less valuable for
quality improvement
What types of key barriers remain to quality improvement in this state, and how the CMS Quality
Improvement Organization Program could best help in the future
What the quality-related environment is like in your area, for example, how interested [TYPE OF
PROVIDER]s are in general in quality improvement, and whether they are willing to share information

All data collected for the purposes of this study will be combined and reported in aggregate form only.
Neither you nor your organization will be identified by name in any reports or documents produced from the
study findings. Only Mathematica staff that work directly on the evaluation will have access to the name of your
organization and your name.
With your help, the evaluation will provide critical information to CMS to help refine its work to improve
the quality of care. Please see the attached letter of encouragement to participate from CMS. An evaluation staff
member will call you to follow up in the next couple of days. In the meantime, please feel free to call or e-mail
[CONTACT INFORMATION OF EVALUATION STAFF MEMBER] with any questions or to initiate
scheduling. Thank you very much in advance for your assistance—your input into the evaluation is highly valued
by CMS.
Sincerely,
Myles Maxfield, Ph.D., Project Director
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a
valid OMB control number. The valid OMB control number for this information collection is xxx-XXX. The time required to complete this
information collection is estimated to average 0.8 hours or 48 minutes per response, including the time to review instructions, search existing
data resources, and gather the data needed, and complete and review the information collection. If you have comments concerning the
accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports
Clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland 21244-1850.

F.9

An Affirmative Action/Equal Opportunity Employer

DRUG SAFETY ORGANIZATION
INVITATION LETTER

600 Maryland Ave. S.W.,
Suite 550
Washington, DC 20024-2512
Telephone (202) 484-9220
Fax (202) 863-1763
www.mathematica-mpr.com
[Date], 2010/2011

Contact
Organization name
Street address
City, ST zip
Dear [contact name]:
As a [DESCRIBE ORGANIZATION TYPE IN APPROPRIATE, FLATTERING TERMS—such as
major health plan in the state], you are probably aware of the investment that the Centers for Medicare &
Medicaid Services (CMS) makes in improving quality of care through its Quality Improvement Organization
Program, which in this state is operated through [NAME QIO]. This is an invitation to provide important
feedback to CMS through agreeing to an in-person interview, roughly 45 to 50 minutes in length, to assist
CMS in evaluating and improving its Quality Improvement Organization Program. Your input will remain
confidential.
CMS has contracted with Mathematica Policy Research, Inc. and its partners Social & Scientific Systems
and Abt Associates, to conduct site visits as part of a larger study to evaluate the 9th Scope of Work Quality
Improvement Organization Program. A description of the larger study is attached. The evaluation research
team would appreciate your feedback through an in-person interview during [TARGET WEEK], to be
scheduled at your convenience. You will not need to make any special preparations for the meeting. In
particular, we would like to discuss the following key topics:
•
•
•
•

The overall value and effect of [NAME QIO] activities related to improving drug safety by reducing
inappropriate medications and drug-on-drug interactions
Which types of activities by [NAME QIO] since August 2008 that were more and less valuable for
improving drug safety
What types of key barriers remain to further improving drug safety this state, and how the CMS
Quality Improvement Program could best help in the future
What the quality-related environment is like in your area, for example, how interested providers are
in general in improving drug safety, and whether they are willing to share information on how best
to do so

With your help, the evaluation will provide critical information to CMS to help refine its work to
improve patient safety. Please see the attached letter of encouragement to participate from CMS. An
evaluation staff member will call you to follow up in the next couple of days. In the meantime, please feel free
to call or e-mail [CONTACT INFORMATION OF EVALUATION STAFF MEMBER] with any questions or
to initiate scheduling. Thank you very much in advance for your assistance—your input into the evaluation is
highly valued by CMS.
Sincerely,
Myles Maxfield, Ph.D., Project Director
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless
it displays a valid OMB control number. The valid OMB control number for this information collection is xxx-XXX. The time
required to complete this information collection is estimated to average 0.8 hours or 48 minutes per response, including the time
to review instructions, search existing data resources, and gather the data needed, and complete and review the information
collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please
write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland
21244-1850.

F.11

An Affirmative Action/Equal Opportunity Employer

CASE STUDY DISCUSSION GUIDES

F6A. QIO DIRECTOR AND THEME LEADERS
In the matrix that follows, due to limited space we have used abbreviated indicators for the QIO
staff who will be asked each question during the case study discussions.
Prior to the visits, we will produce a guide tailored to the staffing situation of the QIO, based on
information about its staffing that we learn as we schedule the visit. For example, if a single
individual is responsible for leading the Pressure Ulcer and Physical Restraints sub-themes, we
will concatenate the questions for each of those topics into a single guide. The likely wide
variation in who is responsible for which themes and sub-themes has caused us to prefer to
provide the questions in this matrix format until just prior to the visit.
Key to Abbreviations for QIO Director and Theme Leaders Participating in Case Study
Discussions:
QIO Dir: QIO Director
Pt Sfty: Patient Safety Theme Leader
PrU: Patient Safety – Pressure Ulcers leader
PR: Patient Safety – Physical Restraints leader
SCIP: Patient Safety - Leader for improving surgical safety and heart failure
MRSA: Patient Safety - Leader for reducing Methicillin-resistant Staphylococcus aureus in the
acute care setting
Rx Safety: Patient Safety - Leader for improving drug safety
Pre: Prevention Theme Leader
Pre Disp: Prevention – Disparities Theme Leader
Care Tr: Care Transitions Theme Leader
CKD: Prevention – Chronic Kidney Disease Theme Leader

F.13

QIO DIRECTOR AND THEME LEADERS
DISCUSSION GUIDE TOPICS
Topic for Discussion

F.14

1. Program Impact in the State
Has [QIO contract theme] had an impact on
quality of care or patient safety? [For QIO Dir,
repeat for each theme]
[If yes to pt. safety] Which components of
the patient safety theme have had the
largest impact on quality or safety during
the 9th SOW? Why?
[If yes] What do you view as the most
important evidence of impact?
Did all the contract evaluation measures
work reasonably well to assess progress?
If not, which ones were problematic?
Why?
Are there certain types of providers that
seemed to benefit more than others from
working with the QIO? If yes, which
types?
Are there any specific “success stories”
that you are aware of that you could share
with us?
Are you aware of the trend in quality among
providers in the state that you did not
specifically work with during the 9th SOW? [If
yes:] Did they improve?

QIO Pt
PrU PR SCIP MRSA NHIN Rx Pre Pre Care CKD
Dir Sfty
Sfty
Disp Tr
X

X

X

X

X
X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Topic for Discussion

F.15

QIO Pt
PrU PR SCIP
Dir Sfty
[If providers not worked with also
X
X
X
improved, on average:] What do you think
caused the providers you didn’t work with
to improve? Would this also have affected
the providers you did work with?
In your survey response, you indicated [name X
area of quality improvement posing greatest
challenge] was the biggest/most important
challenge for you in the 9th SOW. Could you
elaborate a little on that and whether and how
it may have affected improvements in provider
quality measures?
2. Program Contract, Infrastructure, and
Supports
[Request QIO Dir & TLs to review their survey
responses and provide any updates prior to the
visit. If the respondent has changed positions,
the new QIO Dir or TL will be asked to
respond to the survey in advance of the visit.
The following questions are designed to follow
up on survey responses.]
[For any negative response on the contractX
X
X
related matters section of the survey—repeat
as necessary:] Could you tell me a little more
about the specifics of the problem you
indicated in your response to [name question]?
Do you believe this [name problem indicated]
X
X
X
significantly lessened the results you were able
to achieve? [If yes] Why?
Are there [QIO Dir: any; Theme Leads: any X
X
X
X
other] barriers to this QIO’s effectiveness that
stem from the contract or CMS procedures?

MRSA NHIN Rx Pre Pre Care CKD
Sfty
Disp Tr
X
X
X
X
X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Topic for Discussion

F.16

QIO Pt
PrU PR SCIP
Dir Sfty
[If responded insufficient data and
X
X
X
information on any item, ask:] You indicated
you had insufficient data and information to
[name task]: could you tell me more about
what data and information you felt you needed
but did not have?
Do you believe this lack of information
X
X
X
significantly lessened the results you were able
to achieve? [If yes] Why?
You rated the following information types as
X
X
X
having high value in helping you shape and
refine interventions over time: [name them].
What was it about these information sources
that made them high value?
You rated the following information types as
X
X
X
having low value: [name them]. What was it
about these information sources that made
them low value?
[Follow-up on any negative responses to
X
X
X
knowledge base/tools items:] Could you
elaborate more on your response that [tailor to
the item]?
[If not already covered:] Could you talk about
X
the role of the QIOSC with respect to your
work? Specifically what did they contribute to
your ability to work effectively on your theme?
[Follow up on survey response:] I see from
X
X
X
your survey response that you found the
centrally developed change package
[useful/not useful] to your work. Could
you tell us more about that?

MRSA NHIN Rx Pre Pre Care CKD
Sfty
Disp Tr
X
X
X
X
X
X
X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Topic for Discussion

F.17

QIO Pt
PrU PR SCIP MRSA NHIN Rx Pre
Dir Sfty
Sfty
[If not already covered:] Could you talk about X
X
X
X
X
X
X
X
your experience in using the PATRIOT
system—that is, was it reasonably easy to enter
the required information into the system, and
retrieve anything you needed to retrieve from
it? [If problematic:] To what extent did this
impact your ability to get your work done most
effectively and efficiently? Could you provide
examples?
Could you talk about the experience with
X
physician practices reporting data from their
EHRs to a new CMS management information
system?
I see from your survey response that you found
X
X
X
X
X
X
the annual in-person meetings sponsored by
CMS specific to the patient safety theme to be
[useful/not useful]. Could you tell me more
about how they [were/were not] useful?
[If survey suggested 1 or more improvements] X
In your survey, you suggested [non-zero
number up to 3] potential high-priority
improvements to CMS-funded tools or
resources. Could you elaborate a little on your
ideas on this and how they could help QIOs
achieve
better
quality
and
safety
improvements?
Do you have any additional suggested X
improvements at this time?

Pre Care CKD
Disp Tr
X
X
X

X

Topic for Discussion

F.18

QIO Pt
PrU PR SCIP
Dir Sfty
[For any negatives in the contract support
X
X
X
and communication section:] In the survey
you indicated that [certain CMS staff had fair
to poor knowledge relative to their
responsibilities/communications were not
usually consistent among different CMS
personnel]. Could you elaborate on the
problem?
[For any “excellent” responses:] You indicated
X
X
X
the knowledge base was strong among [name
type of CMS staff]. Please tell us more about
how this may have assisted you in your work.
3. Types of High- and Lower-Value
Activities
Collaborative Activities – Relative to other
X
X
X
types of activities, you rated collaborative
activities [summarize value rating relative to
other activities].
[For low-value items:] Could you tell us more
X
X
X
about why you rated this type of activity
relatively low in value?
[For high-value items:] Could you tell us more
X
X
X
about why you rated this type of activity highvalue?
[Repeat for: Interactions with Individual
Providers, One-to-Many Activities, Business
Case Focus, Theme-Specific Items]
Are there any other types of activities we
X
X
X
haven’t discussed that you have found to be
high-value? [If yes] What are they?

MRSA NHIN Rx Pre Pre Care CKD
Sfty
Disp Tr
X
X
X
X
X
X
X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Topic for Discussion

F.19

4. Common Types of Actions Taken by
Providers
What were common types of improvement
actions taken by providers [for Rx Sfty: or
other organizations] you worked with?
About what proportion of providers you
worked with took meaningful actions?
[If not 100%] What do you think might explain
why some took meaningful actions and some
did not?
5. Lessons Learned, and QI Within the QIO
What lessons have you learned during the 9th
SOW about what works to improve quality or
safety?
In hindsight, is there anything you wish you
had done differently in the area of recruiting
providers [Rx Sfty: or other organizations] to
work with?
Is there anything you wish you had done
differently in the area of working with
providers [Rx Sfty: or other organizations]?
During the 9th SOW, all QIOs were required to
track their own approaches to QI and assess
their effectiveness, then use action plans and
PDSA to improve outcomes. Based on the
quarterly reports on this topic that we
reviewed, it looks like [summarize findings
from quarterly reports review]. Could you
elaborate on [name anything puzzling or
particularly interesting from the review]?

QIO Pt
PrU PR SCIP MRSA NHIN Rx Pre Pre Care CKD
Dir Sfty
Sfty
Disp Tr

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Topic for Discussion

F.20

QIO Pt
PrU PR SCIP MRSA NHIN Rx Pre Pre Care CKD
Dir Sfty
Sfty
Disp Tr
Has any part of your work included thinking
X
X
X
about the sustainability of any gains that are
achieved? What are the key issues in
sustainability?
[If some changes were recommended in X
survey:] In your survey response, you
suggested that CMS should make the following
change [name change recommended]. Could
you elaborate a bit on what led you to that
recommendation and any more specific
thoughts about what shape the change should
take? [repeat as necessary to cover all
recommended changes]
6. State Environment
Health Care Organizations Involved in Quality
Please tell us about any quality-related
X
X
X
X
X
X
activities undertaken by the relevant
professional or provider [Rx Sfty: or health
plan] association(s)? What if any role did the
QIO play in any of these?
Please tell us about any relevant quality or
X
X
X
X
X
X
safety-related activities undertaken by large
provider organizations [Rx Sfty: or health
plans] that operate in this state? Please be clear
if the QIO had a role in prompting or
facilitating any of these.

Topic for Discussion

F.21

QIO Pt
PrU PR SCIP MRSA NHIN Rx
Dir Sfty
Sfty
In your survey response, you listed the
X
X
X
following other organizations whose efforts
were proving important to achieving
improvements: [list]. Could you tell us a little
more about what they are doing, and why you
think their activities have been effective?
Please be clear if the QIO had a role in any of
these.
Provider Environment
During the 9th SOW, to what extent have
X
X
providers in this state been interested in
exchanging information with one another to
improve quality?
Has this [willingness/unwillingness] to share
X
X
been a significant [help/hindrance] to
improving quality during this period?
What do you think underlies the general
X
X
[willingness/unwillingness] to share in this
state?
Your survey response indicated that providers
X
X
X
X
X
in this state often [have enough/lack]
motivation to
improve
their
quality
performance. Could you say a little more about
why they have this level/lack of motivation?
Has this level of motivation been a significant
X
X
X
X
X
[help/hindrance] to improving quality during
this period?
Are there any other characteristics of the
X
X
culture of the providers in this state that make
assisting them with quality improvement
particularly challenging?

Pre Pre Care CKD
Disp Tr
X
X
X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Topic for Discussion

F.22

QIO Pt
PrU PR SCIP
Dir Sfty
What are the key barriers to further
X
X
X
improvements among providers [Rx Sfty: and
health plans] in this state?
Focusing on the poorest performers, let’s say
X
X
X
the bottom 10 percent, could you describe why
you think they are performing so poorly? What
if anything do you believe would help them
improve?
Are there characteristics of the provider [Rx
Sfty: or health plan] environment in this state
that make providers particularly receptive to
attempts to assist them with quality
improvement?
Are there characteristics of the provider [Rx
Sfty: or health plan] environment in this state
that make it particularly challenging for an
organization like the QIO to assist providers
with quality improvement?
7. Participating Providers Selection Process
How much discretion did the QIO have in
X
X
X
selecting practices to be participating
providers?
Please describe the state’s strategy in recruiting
X
X
X
providers [CKD and CT: and organizational
partners] under this theme—types of providers
[CKD and CT: and organizations] targeted,
type of approach, and selling points.
Could you tell us how this state ended up
working with the number of providers [CKD
and CT: and other partners] that it did?

MRSA NHIN Rx Pre Pre Care CKD
Sfty
Disp Tr
X
X
X
X
X
X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Topic for Discussion

F.23

QIO Pt
PrU PR SCIP MRSA NHIN Rx Pre Pre Care CKD
Dir Sfty
Sfty
Disp Tr
Did the geography of the areas you targeted
X
X
X
present
any
special
challenges
to
accomplishing the goals of the theme?
How receptive were providers [CKD and CT:
X
X
X
X
X
X
X
X
X
and other key organizations] to working with
you? Why?
We understand from your survey response that
X
X
X
X
X
X
[summarize selection process]. Could you tell
us about the advantages and disadvantages of
the way providers were selected to work with,
in terms of how well it worked to give you a
group that you could assist to improve quality
to meet your goals?
If you had instead worked with every provider
X
X
X
X
X
X
X
in the state who could have well-used your
help to improve performance, how much
bigger would the group have been?
8. Recruitment of Beneficiaries
Based on your survey response, we understand
X
this state recruited beneficiaries for DSME
[through
providers/directly
from
the
community]. Did you consider the other type
of strategy, and if so, why did you choose the
one you did?
How well did your recruiting strategy
X
work?
What lessons have you learned about
X
recruiting beneficiaries for DSME?

F6B. HOSPITAL QUALITY IMPROVEMENT DIRECTORS
All hospitals selected for site visits will have worked with the QIO on one or more themes/subthemes. In the matrix that follows, some questions include theme/sub-theme names in brackets.
This means the question would be asked in terms of one theme that is applicable to the selected
hospital. If there is time, the interviewer would cycle back and discuss a second topic if the
hospital worked with the QIO on more than one theme/sub-theme.

F.24

1. Background Information
[collect description of hospital organization prior to visit, including whether part of larger
system, level of services (e.g., tertiary, secondary), size of hospital, teaching status (major
teaching, minor teaching, non-teaching), tax status (FP/NFP), payor mix (% Medicaid; %
Medicare)]
[We will know from administrative data which themes and sub-themes the visited hospital
worked on with the QIO. For questions that are specific to a theme or sub-theme—where a
theme or sub-theme is indicated in brackets—we will decide in advance which theme/sub-theme
to cover first. If time, we will cycle back and continue the discussion regarding a second
applicable theme/sub-theme.]
2. Overall Quality and Quality Improvement
Where does quality fit into the hospital’s overall business strategy?
When it comes to how well this hospital is performing overall in terms of quality and safety
overall, on a scale of 1-10, where 10 is the best care possible, where would you place it
currently?
Where would you have placed it three years ago?
[If improved:] What have been the main factors that led to the improvement?
[If lower than 7:] What are the main reasons you wouldn’t give the hospital a higher score?
Setting aside this 1-10 scale, could you give us an overview of how the hospital measures its
quality?
In general, what types of activities has the hospital found to be most effective at improving
quality?
3. QIO Interactions
How did this hospital get involved in quality improvement activities with the QIO that have been
undertaken since August 2008?
Please tell us about this hospital’s interactions with [name QIO] over the past three years.
[capture frequency and types of interactions, positions/backgrounds of those who attended on
both sides]
[Make sure they are thinking about emails they received as well as any in-person or phone
conferences they attended]
[Make sure they cover whether they took a HLQAT and/or AHRQ Pt. Safety survey provided by
the QIO and whether there was any follow up interaction.]
Let me summarize what I think I heard regarding all the hospital’s interactions with the QIO
around [PrU, SCIP/HF, MRSA as applicable]: [summarize]. Is that correct?
4. QIO Impact on Hospital Operations
How valuable to the hospital was the interaction with [name QIO] around [PrU, SCIP/HF,
MRSA as applicable]?
[If valuable:] In what way was it valuable?
[If not valuable:] Why was it not valuable?
Did any of the interactions with [name QIO] around [PrU, SCIP/HF, MRSA, as applicable] lead
to changes in the hospital that ultimately improved patient care?
[If hospital took the HLQAT or AHRQ Pt. Safety survey:] What did you learn from the results?
Have any changes been made as a result of the survey(s) or related follow-up that strengthen
quality or safety at the hospital?
Did any other interactions with the QIO lead to changes in the hospital that ultimately improved
patient care?

F.25

If you were advising [name QIO] about improving the impacts of its interactions with hospitals
like yours, what would you say?
5. Story of Hospital’s Performance Trend on [PrU, SCIP/HF, MRSA]
Could you take us through the story of this hospital’s performance trend on [PrU, SCIP/HF,
MRSA] from the time you first started tracking it? By “story,” we mean what the trends were
and what lay behind them.
If [name measure(s)] improved over time, what did you do?
What motivated you to take these actions that led to improvement?
What knowledge and staff resources made the changes possible?
Do you believe this hospital has now achieved the best possible performance?
[If no] What barriers remain to achieving optimal performance on the [PrU, SCIP/HF, MRSA as
appropriate] measures?
Are there any needs you have from an outside organization such as a Medicare Quality
Improvement Organization in order for this hospital to achieve optimal performance on these
measures? [If yes] What types of help do you need? [If SCIP/HF:] On which measures?
6. Hospital’s Focus on QI for Measures of Interest vs. Other Measures
How much of the hospital’s quality improvement efforts have been focused on the [PrU,
SCIP/HF, MRSA] measures vs. others? [We will provide each visited hospital in advance and
on-site with a list of the specific measures we are referencing.]
[SCIP/HF only:] Within the SCIP/HF measures, have some measures received more focus
on improvement than others? Why?
Are there other measures or quality improvement areas that received more focus than the [PrU,
SCIP/HF, MRSA] measures? Why?
What influences the priority that is given to improving on various measures?
Are there any needs you have from an outside organization such as a Medicare Quality
Improvement Organization in order for this hospital to achieve optimal performance on other
measures you perceive as high priority? [If yes] What types of help do you need? On which
types of measures?
7. Interactions with Outside Organizations/People Regarding QI
What other organizations or people has this hospital interacted with over the past three years
regarding quality or patient safety improvement?
Have any of these had an important influence on the hospitals quality improvement or safetyrelated efforts? [If yes] Which one(s)?
[If yes] Please tell us about what the most important interactions were and what changed as a
result within the hospital.
Do you talk much with other hospitals about their perspectives on quality improvement?
How do these conversations tend to occur, for example, by phoning a friend, chatting at inperson meetings, via one-to-one emails, via group emails/listservs?
[If yes] How motivated to improve quality and safety are hospitals in this state?
[If yes] Do hospitals in this state generally perceive there to be a business case for quality
and safety improvement? [Please elaborate.]
[If yes] What other factors motivate hospitals in this state? Public reporting? Pay-forperformance?
[If no] Are hospitals in this state generally reluctant to share much information about what
they are doing?

F.26

8. Closing
In conclusion, if you were going to advise CMS about how to make the QIO program more
effective in assisting hospitals to improve quality and safety, what would your advice be?
What are the most important targets for improvement?
What are the most useful methods of assistance to focus on?
Turning to efficiency of the program, are there some things that [name QIO] has been doing that
could be done just as well for a larger region than just the state? In other words, could you tell us
whether the state-specific knowledge of [name QIO] was key to its effectiveness in assisting
hospitals over the past few years?
Are there some things that have been done have little value that CMS should ensure QIOs do not
do in the future?
Do you have any other comments on the QIO program or your experiences with [name QIO] that
you would like to share with us?

F.27

F6C. NURSING HOME QUALITY DIRECTORS
The questions that follow apply the same to all nursing homes selected for site visits—those that
worked with the QIO on the Patient Safety Theme, sub-themes Pressure Ulcers, Physical
Restraints, and/or Nursing Homes in Need.

F.28

1.Background Information
Collect descriptive information about the facility prior to visit, including:
Ownership
Profit/non-profit
Size
5-star rating on staffing, deficiencies, and QMs
Pressure ulcer prevalence rates over last three years
Restraint prevalence rates over last three years
Payor mix (% Medicaid; % Medicare)]
2. Overall Quality and Quality Improvement
Where does quality fit into the nursing home’s overall business strategy?
When it comes to how well this nursing home is performing in terms of quality and safety
overall, on a scale of 1-10, where 10 is the best care possible, where would you place it
currently?
Where would you have placed it three years ago?
[If improved:] What have been the main factors that led to the improvement?
[If lower than 7:] What are the main reasons you wouldn’t give your facility a higher score?
3. QIO Interactions
How did this nursing home get involved in quality improvement activities with the QIO that
have been undertaken since August 2008?
Please tell us about your interactions with [name QIO] over the past three years. [capture
frequency and types of interactions, positions/backgrounds of those who attended on both sides]
[Make sure they are thinking about emails they received as well as any in-person or phone
conferences they attended]
Let me summarize what I think I heard regarding your nursing home’s interactions with the QIO
around [PrU and Restraints]: [summarize]. Is that correct?
4. QIO Impact on Nursing Home Operations
How valuable to the facility was the interaction with [name QIO] around [PrU or Restraints, as
applicable]?
[If valuable:] In what way was it valuable?
[If not valuable:] Why was it not valuable?
Did any of the interactions with [name QIO] around [PrU or Restraints, as applicable] lead to
changes in the facility that ultimately improved resident care? If yes, please describe.
If you were advising [name QIO] about improving the impacts of its interactions with facilities
like yours, what would you say?
5. Nursing Home’s Performance Trend on [PrU or Restraints]
Could you provide some history on your facility’s performance on the [PrU or Restraint]
measure(s)? Specifically, how have your rates changed (or not) from the time you first started
tracking it/them?
If [name measure(s)] improved over time, what did you do?
What motivated you to take these actions that led to improvement?
What knowledge and staff resources made the changes possible?
Do you believe this facility has now achieved the best possible performance?
[If no] What barriers remain to achieving optimal performance on the [PrU or Restraint as
appropriate] measures?

F.29

Are there any needs you have from an outside organization such as a Medicare Quality
Improvement Organization in order for your facility to achieve optimal performance on these
measures? [If yes] What types of help do you need?
6. Nursing Home’s Focus on QI for Measures of Interest vs. Other Measures
How much of your quality improvement efforts have been focused on the [PrU or Restraint]
measures vs. others?
Are there other quality measures or quality improvement areas that received more focus than the
[PrU and Restraint] measures? Why?
How do you prioritize your quality improvement activities? What measure or issue is most
important to your facility? Least important? Why?
Are there any needs you have from an outside organization such as a Medicare Quality
Improvement Organization in order for your facility to achieve optimal performance on other
measures you perceive as high priority? [If yes] What types of help do you need? On which
types of measures or quality improvement areas?
7. Interactions with Outside Organizations/People Regarding QI
What other organizations or people has this facility interacted with over the past three years
regarding quality improvement?
Have any of these had an important influence on the facility’s quality improvement efforts? [If
yes] Which one(s)?
[If yes] Please tell us about what the most important interactions were and what changed as a
result within the nursing home.
Do you talk much with other nursing homes about their perspectives on quality improvement?
How do these conversations tend to occur, for example, by phoning a friend, chatting at inperson meetings, via one-to-one emails, via group emails/listservs?
[If yes] How motivated to improve quality are nursing homes in this State?
[If yes] Do nursing homes in this State generally perceive there to be a business case for
quality? [Please elaborate.]
[If yes] What other factors motivate nursing homes in this State? Public reporting? Pay-forperformance? Medicaid payment rates? Survey and certification?
8. Closing
In conclusion, if you were going to advise CMS about how to make the QIO program more
effective in assisting nursing homes to improve quality, what would your advice be?
What are the most important targets for improvement?
What are the most useful methods of assistance to focus on?
Turning to efficiency of the program, are there some things that [name QIO] has been doing that
could be done just as well for a larger region than just the state? In other words, could you tell us
whether the state-specific knowledge of [name QIO] was key to its effectiveness in assisting
nursing homes over the past few years?
Are there some things that have been done that lack value and should be discontinued?
Do you have any other comments on the QIO program or your experiences with [name QIO] that
you would like to share with us?

F.30

F6D. PHYSICIAN PRACTICE – LEAD PHYSICIAN CONTACT
The questions that follow apply to lead physician contacts from practices selected for visit
because they worked with the QIO on the Prevention theme, and/or the Prevention – Disparities
theme. Where indicated in brackets within questions, question wording is slightly tailored to
apply to the Prevention – Disparities theme.

F.31

1. Background Information
[collect description of practice prior to visit, including whether an FQHC or Rural Health Clinic,
whether part of larger medical group or health system, size of practice (number of physicians and
mid-level practitioners), specialties of clinicians, number of Medicare beneficiaries and % of
practice this represents][for disparities, also get % of practice that is underserved, and % with
diabetes]
Does the practice receive any quality report cards from outside organizations?
Does the practice participate in any quality improvement programs or activities, in addition to
the [preventive/diabetes] work you do with [name QIO]?
2. Interactions with the QIO and other CMS Contractors
How did the practice get involved in working with [name QIO] on [preventive/diabetes] care
activities?
What did the practice hope to gain?
[Pre – Disp only:] Are you aware of the diabetes self-management education that the QIO has
been sponsoring?
[Pre – Disp only:] In your view, how well do such programs, and this one in particular, work?
Please tell us about the practice’s [Pre – Disp: other] interactions with [name QIO] over the past
three years. [capture frequency and types of interactions, positions/backgrounds of those who
attended on both sides]
[Make sure they are thinking about emails they received as well as any in-person or phone
conferences they attended]
Please describe your experience with reporting data from your EHR to the CMS contractor on
the [preventive/diabetes] care measures—did all go smoothly?
[Pre – Disp only:] Did you submit blood pressure control data to PQRI? Why or why not? [If
yes] Was it easy to do?
3. QIO Impact on Practice Operations and Patients’ Health
How valuable to the practice was the interaction with [name QIO] around [preventive/diabetes]
care?
[If valuable:] In what way was it valuable?
[If not valuable:] Why was it not valuable?
[Pre – Disp only:] To your knowledge, did any of your patients receive diabetes selfmanagement education because of the QIO’s initiative to increase this? [If yes] About how
many?
[Pre – Disp only:] [If yes] Did you see any improvement in the condition of patients who
attended the training that you believe was attributable to the class?
Did any of the interactions between the practice and [name QIO] around [preventive/diabetes]
care lead to changes in practice operations that improved patient care?
If you were advising [name QIO] about improving the impacts of its interactions with practices
like yours, what would you say?
4. Practice’s Knowledge of its Performance and Trend on [Preventive/Diabetes] Care
Measures
What information does this practice generate or receive about the percentage of practice patients
that have received [measures of interest]?
When and how did the practice first become aware of how its patients were doing on these
measures?

F.32

Since that time, have you taken any particular actions to try to increase these rates? [If yes,]
What were they?
[If yes] What motivated you to take these actions?
How consistently are these actions applied across the practice?
What knowledge and staff (or consultant) resources made the changes possible?
Have you been able to see any trend yet in these measures that would indicate if your actions
were working?
Do you believe this practice has now achieved the best possible performance?
What is the role of your EHR in supporting good performance on [preventive/diabetes] care
measures?
[If no] What barriers remain to achieving optimal performance on the [preventive/diabetes care]
measures?
Are there any needs you have from an outside organization such as a Medicare Quality
Improvement Organization in order for this practice to achieve optimal performance on these
measures? [If yes] What types of help do you need? On which measures?
5. Practice’s Focus on QI for Measures of Interest vs. Other Measures
How much of the practice’s quality improvement efforts have been focused on the
[preventive/diabetes care] measures vs. others? Why? [we will share a list with them of the
specific measures we are referencing]
Within the [preventive/diabetes care] measures, have some measures received more focus on
improvement than others? Why?
Are there other measures or clinical areas that received more focus on improving measured
performance than the [preventive/diabetes] care measures? [If so] Why?
What influences the priority that is given to improving on various measures?
Are there any needs you have from an outside organization such as a Medicare Quality
Improvement Organization in order for this hospital to achieve optimal performance on other
clinical performance measures you perceive as high priority? [If yes] What types of help do you
need? On which types of measures?
6. Interactions with Outside Organizations/People Regarding Performance Measurement
and Improvement
What if any other organizations or people has this practice interacted with over the past three
years regarding performance improvement?
Have any of these had an influence on the practice’s operations? [If yes] Which one(s)?
[If yes] Please tell us about what the most important interactions were and what changed as a
result within the practice.
Do you talk much with other practices about their perspectives on performance measurement and
improvement?
How do these conversations tend to occur, for example, by phoning a friend, chatting at inperson meetings, via one-to-one emails, via group emails/listservs?
Are most practices in this state aware of how they are doing on these [preventive/diabetes]
care measures?
[If yes] How motivated to improve performance on quality measures are practices in this
state at this time? Why?
Do practices in this state generally perceive there to be a business case for them to improve
on these quality measures? [Please elaborate.]

F.33

Aside from any inherent business case and professional integrity, do any other factors
motivate practices in this state? Future or current public reporting? Future or current payfor-performance?
What do you think it would take to get practices to change operations enough to improve the
rates of [preventive/diabetes] care measures in the state to their optimal level?
7. Closing
In conclusion, if you were going to advise CMS about how to make the QIO program more
effective in assisting practices to improve quality of care and safety for their patients, what
would your advice be?
What are the most important targets for improvement?
What are the most useful methods of assistance to focus on?
Turning to efficiency of the program, are there some things that [name QIO] has been doing that
could be done just as well for a larger region than just the state? In other words, could you tell us
whether the state-specific knowledge of [name QIO] or their ability to come on-site was
important to their ability to effectively assist physician practices over the past few years?
Are there some things that have been done have no apparent value that CMS should ensure QIOs
do not do in the future?
Do you have any other comments on the QIO program or your experiences with [name QIO] that
you would like to share with us?

F.34

F6E. COMMUNITY HEALTH LEADERS
The set of questions that follows applies to three “community health leaders” identified for each
case study state. Usually these would be provider or professional association representatives
knowledgeable about quality, with one representing the physician sector, one the hospital sector,
and one the nursing home sector. On occasion, another type of respondent rather than a provider
or professional association representative may be selected to provide an external vantage point,
such as the leader of a large quality alliance in the state.

F.35

1.Background Information
[collect description of the leader’s position and role in health care quality improvement]
2. QIO Program Impact in the State
Could you tell us how and to what extent you know about and/or involved in the work of [name
QIO] as a Medicare QIO over the past few years?
Has the work of [name QIO] over the past few years as a Medicare Quality Improvement
Organization influenced the quality of care in this state? How?
3. Most Effective QIO Activities
Among the different kinds of activities you may be aware of that [name QIO] undertakes in its
role as a Medicare QIO, are there any that you could identify as particularly high-value? Low
value?
Is there anything you think [name QIO] could have done differently to be more effective in its
work that you are aware of? What would it have taken for this to have happened?
4. Quality Improvement in the State
What are the key organizations that influence quality or patient safety improvement activities in
this state? What major contributions has each made in the past three years?
What are the key barriers to further improvements among quality and patient safety in this state?
Are there characteristics of the provider environment in this state that make providers
particularly receptive to attempts to assist them with quality improvement?
Are there characteristics of the provider environment in this state that make it particularly
challenging for an organization like the QIO to assist providers with quality improvement?
5. Closing
If you were going to advise CMS about how to make the QIO program more effective as a
catalyst or technical assistance resource to improve quality and patient safety, what would your
advice be?
What are the most important targets for improvement?
What are the most useful methods of assistance to focus on?
Turning to efficiency of the program, are there some things that [name QIO] has been doing that
could be done just as well for a larger region than just the state? In other words, could you tell us
whether the state-specific knowledge of [name QIO] was key to its effectiveness over the past
few years?
Are there some things that have been done have no apparent value that CMS should ensure QIOs
do not do in the future?
Do you have any other comments on the QIO program or your experiences with [name QIO] that
you would like to share with us?

F.36

F6F. DRUG SAFETY PARTNER ORGANIZATION
The drug safety partner organization may be a health plan, a provider organization, or another
type of organization who has partnered with the QIO to improve drug safety under the QIO’s
Patient Safety theme. The QIO will provide us with contact information to interview the lead
contact from this organization.

F.37

1.Background Information
[collect description of drug safety partner organization and the effort that they worked on with
the QIO in advance of the visit]
2. The Drug Safety Initiative
Where does drug safety as a topic area for improvement fit into this organization’s overall
business strategy?
What influences the priority that is given to improving on various measures of patient safety and
quality?
Please tell us about the drug safety initiative that the QIO has been involved in with you.
[motivators, goals and measures, other partnered organizations, scope of the effort, timeframe,
level of effort over time from the various partners]
3. QIO Interactions
How, why, and when did [name organization] get involved in [name drug safety-related
activities] with the QIO?
Please tell us about [name organization]’s interactions with [name QIO] around drug safety over
the past three years. [capture frequency and types of interactions, positions/backgrounds of those
who attended on both sides]
[Make sure they are thinking about emails and materials they may have received as well as any
in-person or phone conferences they attended]
4. QIO Impact on Drug Safety Initiative and Drug Safety
How valuable to the drug safety initiative was the interaction with [name QIO]?
[If valuable:] In what way was it valuable?
[If not valuable:] Why was it not valuable?
Did any of the interactions with [name QIO] around drug safety lead to changes that have
improved drug safety?
If you were advising [name QIO] about improving the impacts of its interactions with other
organizations in order to improve drug safety, what would you say?
5. Story of the Organization or Initiative’s Performance Trend on Drug Safety Measures
Could you take us through the story of this [organization’s or initiative’s] performance trend on
the drug safety measures you have been tracking, from the time you first started tracking it? By
“story,” we mean what the trends were and what lay behind them.
If [name measure(s)] improved over time, what did you do?
What motivated you to take these actions that led to improvement?
What knowledge and staff resources made the changes possible?
Do you believe this [organization/initiative] has now achieved the best possible drug safety
performance?
[If no] What barriers remain to achieving optimal performance on the drug safety measures?
Are there needs from an outside organization such as a Medicare Quality Improvement
Organization in order for providers in the state to achieve optimal performance on drug safety
measures? [If yes] What types of help are needed? To influence what measures?
6. Interactions with Outside Organizations/People Regarding Drug Safety
What if any other organizations or people has this organization interacted with over the past
three years regarding drug safety, that haven’t yet been mentioned?
Have any of these had an important influence on the organization’s drug safety-related efforts?
[If yes] Which one(s)?

F.38

If yes] Please tell us about what the most important interactions were and what changed as a
result.
7. Closing
In conclusion, if you were going to advise CMS about how to make the QIO program more
effective in assisting other organizations with improving drug safety, what would your advice
be?
What are the most important targets for improvement?
What are the most useful methods of assistance to focus on?
Turning to efficiency of the program, are there some things that [name QIO] has been doing that
could be done just as well for a larger region than just the state? In other words, could you tell us
whether the state-specific knowledge of [name QIO] was key to its effectiveness in assisting with
drug safety over the past few years?
Are there some things that have been done have no apparent value that CMS should ensure QIOs
do not do in the future?
Do you have any other comments on the QIO program or your experiences with [name QIO] that
you would like to share with us?

F.39