Do Subspecialists Working Outside of Their Specialty Provide Less Efficient
and Lower-Quality Care to Hospitalized Patients Than Do Primary Care
Physicians?

 Arch Intern Med. 2002;162:527-532

Author Information
<http://archinte.ama-assn.org/issues/v162n5/rfull/#aainfo>   Scott R.
Weingarten, MD, MPH; Lynne Lloyd, MBA; Chiun-Fang Chiou, PhD; Glenn D.
Braunstein, MD
Background  Studies show that subspecialists can provide better quality care
than primary care physicians when working within their subspecialty for
patients with some medical conditions. However, many subspecialists care for
patients outside of their chosen subspecialty. The present study compared
the quality of care provided by subspecialists practicing outside of their
specialty, general internists, and subspecialists practicing within their
specialty.
Methods  The severity-adjusted mortality rate and the severity-adjusted
length of stay were used as indexes of quality of care. Data from 5112
hospital admissions (301 different physicians) for community-acquired
pneumonia, acute myocardial infarction, congestive heart failure, or upper
gastrointestinal hemorrhage at 6 hospitals in the greater Cleveland, Ohio,
area were used in this study. The data were severity adjusted with the
CHOICE Severity of Illness System.
Results  Subspecialists working outside of their subspecialty cared for 25%
of hospitalized patients. When comparing patients cared for by
subspecialists practicing outside of their subspecialty, severity-adjusted
lengths of stay were longer for patients with congestive heart failure (23%
longer; 95% confidence interval [CI], 15%-32%), upper gastrointestinal
hemorrhage (22% longer; 95% CI, 7%-39%), and community-acquired pneumonia
(14% longer; 95% CI, 5%-24%) than for patients cared for by subspecialists
practicing within their subspecialty. Patients also had a slightly higher
hospital mortality rate when cared for by subspecialists practicing outside
of their specialty than by subspecialists practicing within their
subspecialty (mortality rate odds ratio, 1.46; P = .047). In addition,
patients cared for by subspecialists practicing outside of their
subspecialty had longer lengths of stay, and prolongations of stay were
observed for patients with congestive heart failure (16% longer; 95% CI,
8%-26%), upper gastrointestinal hemorrhage (15% longer; 95% CI, 2%-30%), and
community-acquired pneumonia (18% longer; 95% CI, 9%-28%) than patients
cared for by general internists.
Conclusions  Subspecialists commonly care for patients outside of their
subspecialty, despite the fact that their patients may have longer lengths
of stay than those cared for by subspecialists practicing within their
specialty or by general internists. In addition, such patients may have
slightly higher mortality rates than those cared for by subspecialists
practicing within their subspecialty.
Arch Intern Med. 2002;162:527-532
IOI10114
THERE HAS BEEN significant discussion and debate regarding the role of
subspecialists and primary care physicians in providing care to patients
with diverse medical conditions. 1-7
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r1>  Several studies have
reported that subspecialists have more up-to-date medical knowledge and
provide better quality of care than primary care physicians when caring for
patients with conditions within their chosen specialty (eg, cardiologists
providing care to patients with congestive heart failure). 1
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r1>  For example, when
patients with acute myocardial infarction, acute nonhemorrhagic stroke, and
asthma are cared for by subspecialists, they may have better outcomes than
when they are cared for by general internists. 1
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r1>  Moreover, a survey 8
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r8>  of primary care
physicians showed that primary care physicians believed that the scope of
conditions that they treat had increased significantly, and that 24%
believed that the scope of care that they were expected to provide was
greater than it should be.
Recent studies 9-11 <http://archinte.ama-assn.org/issues/v162n5/rfull/#r9>
have suggested that there may be a surplus of subspecialists, as determined
by projecting physician manpower needs from managed care subspecialty
requirements to a population of patients. A possible surplus of
subspecialists may result in some subspecialists expanding the scope of care
that they provide and treating conditions outside of their chosen specialty.
3 <http://archinte.ama-assn.org/issues/v162n5/rfull/#r3>
Although many studies have compared subspecialists practicing within their
chosen specialty with primary care physicians, few have examined the quality
and efficiency of care provided by subspecialists practicing outside of
their specialty. Using a valid severity of illness model, 12
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r12> , 13
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r13>  the present study
compared the quality of care provided by subspecialists caring for patients
outside of their specialty with that provided by general internists and by
subspecialists caring for patients within their specialty.



SUBJECTS AND METHODS



OUTCOME MEASURES

The primary outcome measures used to indicate the quality of care that
patients received were the severity-adjusted mortality rate and hospital
length of stay (LOS). The models were constructed based on patients'
demographic and clinical data.
DATA SOURCE

Six hospitals in the greater Cleveland area, in northeast Ohio, provided
information on their physician subspecialties and patients cared for by
these physicians to this study. All of these hospitals were members of the
Cleveland Health Quality Choice Coalition Consortium. 12-14
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r12>  Among them, 1 is a
rural hospital and 5 are community hospitals. Of the 6 hospitals, only 1 had
a hospitalist program, and none had an internal medicine training program,
family practice training program, or full-time faculty. Two of the hospitals
were part of a health care system and are coded as a single hospital
(hospital 4) ( Table 1
<http://archinte.ama-assn.org/issues/v162n5/fig_tab/ioi10114_t1.html> ). The
Cleveland Health Quality Choice program was a regional effort of health care
organizations to compare and improve hospital performance. 12-14
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r12>
Lengths of stay and mortality rates for patients with acute myocardial
infarction, congestive heart failure, upper gastrointestinal hemorrhage, or
community-acquired pneumonia were examined in this study.
Data, including sociodemographic variables, admission source, medications,
medical history, vital signs, selected variables from the physical
examination, results of laboratory tests, electrocardiographic findings,
echocardiographic findings, and do not resuscitate status, were abstracted
from the medical record of each patient by medical record technicians. There
were explicit protocols for data abstraction, double keystroke entry,
identification of out-of-range variables, and independent verification of
data quality at each hospital.
PHYSICIANS AND PHYSICIAN CLASSIFICATION

The primary physician for selected patients was obtained independently by
each hospital participating in this study. The selected physician was the
attending physician of record in each case. The subspecialty status of
physicians was verified by reviewing information supplied by each hospital
(medical staff office), information provided on the American Medical
Association and American Board of Internal Medicine Web sites, and other
available information on physician subspecialty. 15
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r15>
Physicians were classified as those practicing within their subspecialty,
those practicing outside of their subspecialty, general internists, or
family practitioners. For community-acquired pneumonia, physicians were
classified as practicing within the specialty if they were trained in
infectious diseases or pulmonary medicine. For upper gastrointestinal
hemorrhage, physicians were classified as practicing within the specialty if
they were gastroenterologists. For congestive heart failure or acute
myocardial infarction, physicians were classified as practicing within the
subspecialty if they were cardiologists.
PATIENTS AND PATIENT CLASSIFICATION

Patients were classified based on the International Classification of
Diseases, Ninth Revision, Clinical Modification, principal diagnosis code.
Only information on patients who were hospitalized in the 6 hospitals for
acute myocardial infarction, congestive heart failure, upper
gastrointestinal hemorrhage, or community-acquired pneumonia between January
1, 1997, and December 1, 1997, was used in this study. Data of patients who
were younger than 18 years or transferred from other acute-care hospitals
were excluded. Patients may have been hospitalized more than one time, and
each hospitalization was considered separately.
SEVERITY ADJUSTMENT AND THE CHOICE SEVERITY OF ILLNESS SYSTEM

The CHOICE Severity of Illness System was developed by the Cleveland Health
Quality Choice program. 12
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r12> , 13
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r13>  The system has
multivariate models that were developed separately for each diagnosis. These
models enable us to predict the mortality (0%-100%) and expected LOS (in
days) for patients with community-acquired pneumonia, congestive heart
failure, upper gastrointestinal hemorrhage, or acute myocardial infarction.
These models were built from the demographic and clinical variables
ascertained within the first 48 hours of hospitalization and were validated
by the CHOICE Severity of Illness System in several steps. The initial
models were derived from factors that independently contributed to the risk
of death or LOS (P<.01) in logistic or linear regression models. Length of
stay data were log transformed because the data were heavily skewed. The LOS
models excluded patients who died in the hospital or were transferred to
other hospitals.
The performance of the mortality models was examined by the receiver
operating characteristic curve and calibration was examined by the
Hosmer-Lemeshow goodness-of-fit test, 16
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r16>  while the
performance of the LOS models was assessed by the value of R2 and by
analysis of residuals. Models used in this study were those reestimated and
examined using the larger data set. The receiver operating characteristic
curve areas and R2 were similar among the diagnoses studied. The receiver
operating characteristic curve areas for mortality and the R2 for LOS for
the diagnoses are as follows: acute myocardial infarction, 0.89 and 0.19,
respectively; congestive heart failure, 0.85 and 0.14, respectively;
pneumonia, 0.88 and 0.25, respectively; and upper gastrointestinal
hemorrhage, 0.91 and 0.23, respectively. Performance of the model was also
similar in different types of hospitals. We used our data to calculate the
area under the receiver operating characteristic curves, and found similar
findings.
STATISTICAL ANALYSIS

Differences in patients' demographics and outcomes between physician groups
were analyzed: differences in patients' age, LOS, and risk of death
(severity-adjusted mortality rate) were examined using a Wilcoxon
nonparametric test, while those in sex, race, and mortality were examined
using a chi2 test. Multivariate analyses were also performed to analyze
differences in patients' outcomes of physician groups and to adjust patient
outcomes for patient severity of illness. Individual patient severity of
illness was first determined based on each patient's demographics (eg, age)
and clinical factors (eg, coexisting diseases, laboratory results, and vital
signs) using the CHOICE Severity of Illness System, assuming no unmeasured
selection effects associated with mortality rates and LOS due to the
limitation of available data.
To estimate the magnitude of the difference in severity-adjusted LOS between
the physician groups, a linear regression analysis was used. A dummy
variable for each physician group was used within the model to ascertain the
differences between each group relative to one another. Because LOS is log
transformed, the antilog of the coefficient in the linear regression model
represents the ratio of severity-adjusted LOS between each physician group
and the reference group. If the coefficient is 0.1, for example, the ratio
of severity-adjusted LOS between a specific physician group and the
reference group is 1.26 (e0.1). It can also be interpreted that the
severity-adjusted LOS of the specific physician group is 26% higher than the
one of the reference group.
To analyze differences in mortality rates, logistic regression was used with
a dummy variable to ascertain the differences in mortality rates of
physician groups. With a certain formula of antilog transformation, the
variable estimate for each dummy variable in the logistic regression model
measures the change in the mortality rate between each physician group and
the reference group.
Confidence intervals were then calculated for each estimate to reflect the
variation within the data and the statistical significance of the findings.
P<.05 was considered statistically significant. SAS statistical software was
used for all of the statistical analyses. 17
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r17>



RESULTS



PATIENT DEMOGRAPHICS AND CLASSIFICATION

There were a total of 6485 patient hospital admissions that were potentially
eligible for the study. Of these patient hospital admissions, 1373 included
patients who were not clearly identified as being primarily treated by an
internal medicine subspecialist, a general internist, or a family
practitioner. The remaining 5112 patient admissions were enrolled in the
study. Among them, 1143 patients (22%) had an acute myocardial infarction,
610 (12%) had an upper gastrointestinal hemorrhage, 1946 (38%) had
congestive heart failure, and 1413 (28%) had community-acquired pneumonia.
When patients were classified according to the type of physician who
provided their care, as seen in Table 2
<http://archinte.ama-assn.org/issues/v162n5/fig_tab/ioi10114_t2.html> , a
total of 1776 patients (35%) were treated by a physician practicing within
his or her specialty, 1083 (21%) were treated by an internist without an
identified specialty, 990 (19%) were treated by a family practitioner, and
1263 (25%) were treated by a subspecialist practicing outside of his or her
specialty. There were 301 different physicians.
The mean plusmnSD age of the patients was 72.2 plusmn13.9 years, 93% were
white, and 51% were men. About 72% of the patients had Medicare insurance,
and 23% had commercial insurance. The mean plusmnSD LOS was 5.6 plusmn3.9
days.
IN-HOSPITAL MORTALITY

The mean in-hospital mortality was 5.4%. The mean severity-adjusted
mortality was 5.5%. Mortality rates for each hospital are listed in Table 1
<http://archinte.ama-assn.org/issues/v162n5/fig_tab/ioi10114_t1.html> .
Patients cared for by subspecialists practicing outside of their specialty
had higher mortality rates than those cared for by subspecialists practicing
within their specialty (P = .047) (analysis 1 in Table 3
<http://archinte.ama-assn.org/issues/v162n5/fig_tab/ioi10114_t3.html> ).
There were no significant differences in the mortality rates when comparing
patients cared for by general internists with those cared for by
subspecialists practicing outside of their specialty (P = .17) or when
comparing patients cared for by general internists with those cared for by
subspecialists practicing within their subspecialty (P = .65) (analysis 2 in
Table 3
<http://archinte.ama-assn.org/issues/v162n5/fig_tab/ioi10114_t3.html> ).
Similar results were found for the severity-adjusted mortality rate. Too few
patients with an upper gastrointestinal hemorrhage died to compare mortality
rates by physician types.
LENGTH OF STAY

The mean patient hospital LOS was 5.7 days. The mean LOS was 5.5 days for
general internists' patients, 5.6 days for family practitioners' patients,
5.2 days for patients cared for by subspecialists practicing within their
specialty, and 6.6 days for those cared for by subspecialists practicing
outside of their specialty. The severity-adjusted LOS was longer for
patients treated by subspecialists practicing outside of their specialty
than for those cared for by subspecialists practicing within their specialty
( Table 4
<http://archinte.ama-assn.org/issues/v162n5/fig_tab/ioi10114_t4.html> ).
These differences were observed for patients with acute myocardial
infarction, congestive heart failure, gastrointestinal hemorrhage, and
pneumonia. In addition, patients cared for by subspecialists practicing
outside of their subspecialty had longer LOSs than those treated by general
internists ( Table 4
<http://archinte.ama-assn.org/issues/v162n5/fig_tab/ioi10114_t4.html> ).



COMMENT



This study demonstrated that subspecialists caring for patients outside of
their specialty may provide less efficient care, as evidenced by longer
LOSs, than either subspecialists practicing within their subspecialty or
general internists. In addition, patients cared for by physicians practicing
outside of their specialty may have slightly higher mortality rates than
those cared for by subspecialists practicing within their specialty. The
odds ratio for subspecialists caring for patients outside of their
subspecialty when compared with subspecialists caring for patients within
their subspecialty was 1.46 (P = .047). Patients cared for by physicians
outside of their specialty also had 19% longer LOSs for the total population
of patients, and significantly longer LOSs for patients with congestive
heart failure, upper gastrointestinal hemorrhage, and community-acquired
pneumonia. These differences suggest that subspecialists practicing outside
of their specialty may provide less efficient care and possibly
lower-quality care when compared with physicians providing care within their
subspecialty.
When comparing the mortality rates of patients treated by physicians
practicing outside of their specialty with those of patients cared for by
general internists, there were no statistically significant differences.
However, LOSs for patients cared for by subspecialists practicing outside of
their specialty were 17% longer than those of patients cared for by general
internists, and prolongations of stay were observed for patients with
congestive heart failure, upper gastrointestinal hemorrhage, and
community-acquired pneumonia. Therefore, LOSs were shorter when patients
were treated by general internists rather than subspecialists practicing
outside of their specialty.
This study is one of few that have examined the potential implications of
having subspecialists care for patients outside of their subspecialty. The
strengths of the present study include the following: (1) it had more than
5112 patients treated at 6 different hospitals and (2) a severity-of-illness
adjustment was performed to minimize the chance that differences in LOSs and
mortality rates could be attributed to differences in patient severity of
illness. 12 <http://archinte.ama-assn.org/issues/v162n5/rfull/#r12> , 13
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r13>
This study also has limitations. First, teams of physicians often care for
hospitalized patients, including different subspecialists. It can be
difficult to attribute the efficiency or quality of care to a single
physician or type of physician. However, the identified physician in the
study was the primary attending physician of record as coded by the
hospital. Although we attempted to control for the number of patients with a
particular condition treated by a physician, we only had access to the
information of patients treated in the hospitals that participated in this
study. It is possible that a physician might have admitted patients to
hospitals other than these 6. Therefore, the real volume of patients treated
by physicians in this study might be higher than what was measured.
Information regarding the volume of patients treated by different types of
physicians could be inaccurate and, thus, was not used as a variable in the
analyses. In addition, there were some differences in demographics between
those patients cared for by general internists, subspecialists practicing
within their subspecialty, and subspecialists practicing outside of their
subspecialty. However, patients' LOSs and mortality rates were adjusted for
patient severity of illness, which should account for any difference that
patient severity of illness or age might have had on LOS. In addition, the
LOS may impact hospital mortality rates. Finally, this study used the
patient as the unit of analysis rather than the hospital or the physician.
The observed differences in LOS may demonstrate that physicians caring for
patients outside of their chosen specialty are less familiar with patients
with these conditions because volume-outcome relationships have been shown
for many conditions in medicine, and subspecialists practicing outside of
their subspecialty may be a marker for low patient volume. Moreover,
subspecialists frequently care for patients outside of their chosen
subspecialty, because 25% of patients were cared for by subspecialists
practicing outside of their subspecialty. A recent study 8
<http://archinte.ama-assn.org/issues/v162n5/rfull/#r8>  showed that many
primary care physicians believe that the scope of conditions that they are
expected to treat is greater than it ought to be. Because many
subspecialists may perform primary care and treat hospitalized patients
outside of their subspecialty, it is possible that subspecialists may have
similar concerns that the scope of conditions that they treat outside of
their subspecialty is greater than it should be.
In conclusion, subspecialists commonly care for patients outside of their
subspecialty. Patients cared for by subspecialists practicing outside of
their subspecialty had longer LOSs and possibly higher mortality rates than
those cared for by subspecialists practicing within their subspecialty; they
also had longer LOSs when compared with those cared for by general
internists. If patients are cared for by subspecialists practicing outside
of their specialty, their LOSs, and possibly even mortality rates, may be
higher than those of patients cared for by subspecialists practicing within
their subspecialty.



Author/Article Information


From the Department of Health Services Research (Zynx Health, Inc),
Cedars-Sinai Health System, Beverly Hills, Calif (Drs Weingarten and Chiou);
and the Department of Medicine, University of California, Los Angeles, UCLA
School of Medicine (Drs Weingarten and Braunstein). Ms Lloyd is an
independent consultant.

Corresponding author and reprints: Scott R. Weingarten, MD, MPH, Zynx
Health, Inc, Cedars-Sinai Health System, 9100 Wilshire Blvd, Suite 655E,
Beverly Hills, CA 90212 (e-mail: [log in to unmask]
<mailto:[log in to unmask]> ).
Accepted for publication July 16, 2001.
We thank Dwain Harper, DO, for his assistance with this study.




REFERENCES



1. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr1>
Harrold LR, Field TS, Gurwitz JH.
Knowledge, patterns of care, and outcomes of care for general internists and
specialists.
J Gen Intern Med.
1999;14:499-511.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
10491236>
2. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr2>
Meyer GS, Jacoby I, Krakauer H, et al.
Gastroenterology workforce modeling.
JAMA.
1996;276:689-694.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
8769546>
3. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr3>
Rhee SO, Luke RD, Lyons TF, Payne BC.
Domain of practice and the quality of physician performance.
Med Care.
1981;19:14-23.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
7464306>
4. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr4>
Greenfield S, Nelson EC, Zubkoff M, et al.
Variations in resource utilization among medical specialties and systems of
care: results from the Medical Outcomes Study.
JAMA.
1992;267:1624-1630.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
1542172>
5. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr5>
Kassirer JP.
Access to specialty care.
N Engl J Med.
1994;331:1151-1153.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
7779175>
6. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr6>
Ayanian JZ, Guadagnoli E, McNeil BJ, Cleary CD.
Treatment and outcomes of acute myocardial infarction among patients of
cardiologists and generalist physicians.
Arch Intern Med.
1997;157:2570-2576.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
9531225>
7. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr7>
Jollis JG, DeLong ER, Peterson ED, et al.
Outcome of acute myocardial infarction according to the specialty of the
admitting physician.
N Engl J Med.
1996;335:1880-1887.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
8948564>
8. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr8>
St Peter RF, Reed MC, Kemper P, Blumenthal D.
Changes in the scope of care provided by primary care physicians.
N Engl J Med.
1999;341:1980-1985.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
10607816>
9. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr9>
Weiner JP.
Forecasting the effects of health reform on US physician workforce
requirement: evidence from HMO staffing patterns.
JAMA.
1994;272:222-230.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
7912746>
10. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr10>
Cooper RA.
Seeking a balanced physician workforce for the 21st century.
JAMA.
1994;272:680-687.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
8064983>
11. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr11>
Demaria AN, Lee TH, Leon DF, et al.
Effect of managed care on cardiovascular specialists: involvement, attitudes
and practice adaptations.
J Am Coll Cardiol.
1996;28:1884-1895.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
8962580>
12. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr12>
Rosenthal GE, Harper DL.
Cleveland Health Quality Choice: a model for collaborative community-based
outcomes assessment.
Jt Comm J Qual Improv.
1994;20:425-443.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
8000481>
13. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr13>
Rosenthal GE, Harper DL, Quinn LM, Cooper GS.
Severity-adjusted mortality and LOS in teaching and nonteaching hospitals:
results of a regional study.
JAMA.
1997;278:485-490.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
9256223>
14. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr14>
Rosenthal GE, Quinn LM, Harper DL.
Declines in hospital mortality associated with a regional initiative to
measure hospital performance.
Am J Med Qual.
1997;12:103-112.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
9161057>
15. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr15>
Reed Elsevier Inc and the American Board of Medical Specialists.
The Official ABMS Directory of Board Certified Medical Specialists.
30th ed. New Providence, NJ: Marquis Who's WhoA Division of Reed Elsevier
Inc; 1977.
16. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr16>
Hanley JA, McNeil BJ.
The meaning and use of the area under a receiver operating characteristic
(ROC) curve.
Radiology.
1982;143:29-36.
MEDLINE
<http://www.ncbi.nlm.nih.gov/htbin-post/Entrez/query?db=m&form=6&Dopt=r&uid=
7063747>
17. <http://archinte.ama-assn.org/issues/v162n5/rfull/#rr17>
SAS Software: SAS/Base and SAS/Stat, Version 6.12.
Cary, NC: SAS Institute Inc; 1996.


Edward E. Rylander, M.D.
Diplomat American Board of Family Practice.
Diplomat American Board of Palliative Medicine.